2024-07-27T06:22:33.231 INFO:root:teuthology version: 0.0.1.dev379+g53ce146 2024-07-27T06:22:33.258 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-27T06:22:33.354 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818877 branch: wip-vshankar-testing-20240725.121331-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/fsx}} email: vshankar@redhat.com first_in_suite: false job_id: '7818877' kernel: client: sha1: distro flavor: default kdb: 1 sha1: 3e9255e907100f88049f320ca33d977e29155d12 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240725.121331-debug ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug cephsqlite: 20 mgr/snap_schedule/allow_m_granularity: true mgr/snap_schedule/dump_on_update: true mgr/snap_schedule/log_level: debug mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: global: ms die on skipped message: false mds: mds_bal_replicate_threshold: 1 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request mgr-modules: - snap_schedule sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240725.121331-debug sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5532 sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 sleep_before_teardown: 0 subset: 1001/1024 suite: fs suite_branch: wip-vshankar-testing-20240725.121331-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGk9En5MHJFap7AWyyjRAwkJtqBs5Jxk1oNFBBfzOPZWm9bB8RXwxpiDh9imaOnF5LPVKKfOcpTkNQO1uaqLyO4= smithi047.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF7LdetHjVkjCWnwUsRWnOacaYeY7rLGcsWX4u5WWwZpGX/WDforQs4BD3hvgbf05T8/Ck86ym1MP2VcfX3Mxb4= smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDbeM6/VuQpbDhchUTaeok6oMkOlBwglbqAXWSIqFpFp5sbbGlJuIBK90t+q/YtPrGLwKCUdJIeKClRJwjkElT4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 53ce1462e129f6eb4071986336534c740fdebd31 timestamp: 2024-07-26_04:18:59 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-07-27T06:22:33.355 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa; will attempt to use it 2024-07-27T06:22:33.356 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks 2024-07-27T06:22:33.356 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-07-27T06:22:33.358 INFO:teuthology.task.internal:Checking packages... 2024-07-27T06:22:33.380 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528' 2024-07-27T06:22:33.380 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-07-27T06:22:33.380 INFO:teuthology.packaging:ref: None 2024-07-27T06:22:33.380 INFO:teuthology.packaging:tag: None 2024-07-27T06:22:33.380 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240725.121331-debug 2024-07-27T06:22:33.380 INFO:teuthology.packaging:sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:22:33.381 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240725.121331-debug 2024-07-27T06:22:33.532 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5320.ge8f88bb1 2024-07-27T06:22:33.533 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-07-27T06:22:33.539 INFO:teuthology.task.internal:no buildpackages task found 2024-07-27T06:22:33.539 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-07-27T06:22:33.556 INFO:teuthology.task.internal:Saving configuration 2024-07-27T06:22:33.577 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-07-27T06:22:33.584 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-07-27T06:22:33.610 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818877', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-27 06:17:13.824777', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGk9En5MHJFap7AWyyjRAwkJtqBs5Jxk1oNFBBfzOPZWm9bB8RXwxpiDh9imaOnF5LPVKKfOcpTkNQO1uaqLyO4='} 2024-07-27T06:22:33.631 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818877', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-27 06:17:13.823760', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF7LdetHjVkjCWnwUsRWnOacaYeY7rLGcsWX4u5WWwZpGX/WDforQs4BD3hvgbf05T8/Ck86ym1MP2VcfX3Mxb4='} 2024-07-27T06:22:33.651 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818877', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-27 06:17:13.822099', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDbeM6/VuQpbDhchUTaeok6oMkOlBwglbqAXWSIqFpFp5sbbGlJuIBK90t+q/YtPrGLwKCUdJIeKClRJwjkElT4='} 2024-07-27T06:22:33.651 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-07-27T06:22:33.658 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-07-27T06:22:33.658 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-07-27T06:22:33.658 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-07-27T06:22:33.658 INFO:teuthology.run_tasks:Running task console_log... 2024-07-27T06:22:33.748 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f734e85fac0>, signals=[15]) 2024-07-27T06:22:33.748 INFO:teuthology.run_tasks:Running task internal.connect... 2024-07-27T06:22:33.754 INFO:teuthology.task.internal:Opening connections... 2024-07-27T06:22:33.754 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-07-27T06:22:33.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T06:22:33.832 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2024-07-27T06:22:33.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T06:22:33.913 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-07-27T06:22:33.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T06:22:33.989 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-07-27T06:22:33.996 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-07-27T06:22:34.014 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-07-27T06:22:34.014 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-07-27T06:22:34.072 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-07-27T06:22:34.072 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-07-27T06:22:34.072 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-07-27T06:22:34.072 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-07-27T06:22:34.072 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-07-27T06:22:34.073 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-07-27T06:22:34.073 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-27T06:22:34.073 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-07-27T06:22:34.073 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-07-27T06:22:34.073 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-27T06:22:34.073 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-07-27T06:22:34.073 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-27T06:22:34.073 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-27T06:22:34.073 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-27T06:22:34.074 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-07-27T06:22:34.094 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2024-07-27T06:22:34.113 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2024-07-27T06:22:34.113 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2024-07-27T06:22:34.171 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="CentOS Stream" 2024-07-27T06:22:34.171 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="9" 2024-07-27T06:22:34.172 INFO:teuthology.orchestra.run.smithi047.stdout:ID="centos" 2024-07-27T06:22:34.172 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE="rhel fedora" 2024-07-27T06:22:34.172 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="9" 2024-07-27T06:22:34.172 INFO:teuthology.orchestra.run.smithi047.stdout:PLATFORM_ID="platform:el9" 2024-07-27T06:22:34.172 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-27T06:22:34.172 INFO:teuthology.orchestra.run.smithi047.stdout:ANSI_COLOR="0;31" 2024-07-27T06:22:34.172 INFO:teuthology.orchestra.run.smithi047.stdout:LOGO="fedora-logo-icon" 2024-07-27T06:22:34.172 INFO:teuthology.orchestra.run.smithi047.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-27T06:22:34.172 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://centos.org/" 2024-07-27T06:22:34.172 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-27T06:22:34.172 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-27T06:22:34.172 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-27T06:22:34.173 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2024-07-27T06:22:34.194 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-07-27T06:22:34.212 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-07-27T06:22:34.213 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-07-27T06:22:34.269 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-07-27T06:22:34.269 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="9" 2024-07-27T06:22:34.270 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-07-27T06:22:34.270 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-07-27T06:22:34.270 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="9" 2024-07-27T06:22:34.270 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el9" 2024-07-27T06:22:34.270 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-27T06:22:34.270 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-07-27T06:22:34.270 INFO:teuthology.orchestra.run.smithi077.stdout:LOGO="fedora-logo-icon" 2024-07-27T06:22:34.270 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-27T06:22:34.270 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-07-27T06:22:34.270 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-27T06:22:34.270 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-27T06:22:34.271 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-27T06:22:34.271 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-07-27T06:22:34.294 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-07-27T06:22:34.323 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-07-27T06:22:34.329 INFO:teuthology.task.internal:Checking for old test directory... 2024-07-27T06:22:34.329 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-07-27T06:22:34.332 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2024-07-27T06:22:34.334 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-07-27T06:22:34.353 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-07-27T06:22:34.406 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-07-27T06:22:34.407 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-07-27T06:22:34.410 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2024-07-27T06:22:34.413 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-07-27T06:22:34.437 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-07-27T06:22:34.519 INFO:teuthology.run_tasks:Running task kernel... 2024-07-27T06:22:34.533 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'} 2024-07-27T06:22:34.534 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-07-27T06:22:34.534 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-07-27T06:22:34.534 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-27T06:22:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T06:22:34.551 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-07-27T06:22:34.609 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-437.el9.x86_64 2024-07-27T06:22:34.610 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-437.el9.x86_64 2024-07-27T06:22:34.610 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-07-27T06:22:39.435 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 3.5 MB/s | 8.2 MB 00:02 2024-07-27T06:22:44.612 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 20 MB 00:04 2024-07-27T06:22:48.897 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-07-27T06:22:51.248 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 18 kB 00:01 2024-07-27T06:22:52.239 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 26 MB/s | 22 MB 00:00 2024-07-27T06:22:57.745 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-07-27T06:23:00.045 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-07-27T06:23:00.045 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-07-27T06:23:00.166 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-07-27T06:23:00.191 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-07-27T06:23:00.191 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-07-27T06:23:00.191 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-07-27T06:23:00.191 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-07-27T06:23:00.191 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-07-27T06:23:00.191 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-07-27T06:23:00.191 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-07-27T06:23:00.192 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-07-27T06:23:00.192 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-07-27T06:23:00.192 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:23:00.192 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-07-27T06:23:00.192 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-07-27T06:23:00.192 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-07-27T06:23:00.192 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:23:00.192 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 84 M 2024-07-27T06:23:00.193 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 126 M 2024-07-27T06:23:00.193 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-07-27T06:23:00.707 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 3.9 MB/s | 1.5 MB 00:00 2024-07-27T06:23:03.182 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 6.0 MB/s | 17 MB 00:02 2024-07-27T06:23:05.241 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-modules-core-5.14.0-480.el9.x86_6 6.6 MB/s | 30 MB 00:04 2024-07-27T06:23:07.607 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 4.9 MB/s | 36 MB 00:07 2024-07-27T06:23:07.608 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-07-27T06:23:07.608 INFO:teuthology.orchestra.run.smithi029.stdout:Total 11 MB/s | 84 MB 00:07 2024-07-27T06:23:08.371 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-07-27T06:23:08.782 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-07-27T06:23:08.782 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-07-27T06:23:09.795 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-07-27T06:23:09.796 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-07-27T06:23:12.386 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-07-27T06:23:13.181 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-07-27T06:23:13.297 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-27T06:23:16.093 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-27T06:23:16.312 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-27T06:23:21.522 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-27T06:23:21.586 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-07-27T06:23:27.088 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-07-27T06:24:00.710 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-07-27T06:24:00.753 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-07-27T06:24:01.176 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-07-27T06:24:01.176 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-07-27T06:24:01.176 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-27T06:24:01.177 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-27T06:24:02.222 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-07-27T06:24:02.222 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:24:02.222 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-07-27T06:24:02.222 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-480.el9.x86_64 2024-07-27T06:24:02.222 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-07-27T06:24:02.222 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-07-27T06:24:02.222 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-07-27T06:24:02.222 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:24:02.222 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-07-27T06:24:02.755 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-07-27T06:24:02.862 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-480.el9.x86_64 2024-07-27T06:24:02.863 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-07-27T06:24:02.863 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-07-27T06:24:02.863 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-07-27T06:24:02.863 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-07-27T06:24:02.863 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-07-27T06:24:02.863 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-07-27T06:24:02.863 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-07-27T06:24:02.924 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-480.el9.x86_64 2024-07-27T06:24:02.924 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-07-27T06:24:02.924 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-07-27T06:24:02.955 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-27T06:24:02.956 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2024-07-27T06:24:02.957 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-07-27T06:24:02.957 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-27T06:24:02.957 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-07-27T06:24:02.957 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-07-27T06:24:02.957 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:24:02.957 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-27T06:24:02.957 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-27T06:24:02.957 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-27T06:24:02.957 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-07-27T06:24:02.957 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:24:02.957 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-07-27T06:24:02.958 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-27T06:24:02.958 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-480.el9.x86_64 2024-07-27T06:24:02.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-27T06:24:03.684 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-07-27T06:24:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-27T06:24:05.352 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-07-27T06:24:05.354 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-07-27T06:24:05.383 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.0CyLA0RvLD 2024-07-27T06:24:05.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0CyLA0RvLD 2024-07-27T06:24:05.449 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.0CyLA0RvLD 2024-07-27T06:24:05.588 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.0CyLA0RvLD is 6KB 2024-07-27T06:24:05.638 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.0CyLA0RvLD 2024-07-27T06:24:05.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-07-27T06:24:05.715 INFO:teuthology.orchestra.run.smithi029.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-27T06:24:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-27T06:24:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-27T06:24:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-07-27T06:24:05.717 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-07-27T06:24:05.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-07-27T06:24:35.873 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-07-27T06:24:35.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-07-27T06:24:35.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T06:25:35.877 DEBUG:teuthology.orchestra.remote:timed out 2024-07-27T06:25:44.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-07-27T06:25:44.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T06:25:48.225 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-07-27T06:25:48.949 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-07-27T06:25:48.949 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-480.el9.x86_64"... 2024-07-27T06:25:48.950 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-07-27T06:25:48.979 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-480.el9.x86_64 2024-07-27T06:25:48.979 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-07-27T06:25:48.979 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-27T06:25:48.979 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-27T06:25:49.981 DEBUG:teuthology.parallel:result is None 2024-07-27T06:25:49.981 INFO:teuthology.run_tasks:Running task internal.base... 2024-07-27T06:25:49.989 INFO:teuthology.task.internal:Creating test directory... 2024-07-27T06:25:49.989 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-27T06:25:49.992 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-27T06:25:49.995 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-27T06:25:50.013 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-07-27T06:25:50.020 INFO:teuthology.run_tasks:Running task internal.archive... 2024-07-27T06:25:50.031 INFO:teuthology.task.internal:Creating archive directory... 2024-07-27T06:25:50.031 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-27T06:25:50.049 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-27T06:25:50.054 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-27T06:25:50.175 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-07-27T06:25:50.182 INFO:teuthology.task.internal:Enabling coredump saving... 2024-07-27T06:25:50.182 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-27T06:25:50.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T06:25:50.200 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-27T06:25:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T06:25:50.217 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-27T06:25:50.220 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-27T06:25:50.243 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-27T06:25:50.280 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-27T06:25:50.297 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-27T06:25:50.297 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-27T06:25:50.315 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-27T06:25:50.481 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-27T06:25:50.528 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-27T06:25:50.530 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-07-27T06:25:50.537 INFO:teuthology.task.internal:Configuring sudo... 2024-07-27T06:25:50.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-27T06:25:50.540 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-27T06:25:50.542 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-27T06:25:50.570 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-07-27T06:25:50.579 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-07-27T06:25:50.579 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-27T06:25:50.605 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-27T06:25:50.612 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-27T06:25:50.628 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-27T06:25:50.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-27T06:25:50.767 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-27T06:25:50.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-27T06:25:50.851 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:25:50.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-27T06:25:50.960 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-27T06:25:51.004 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-27T06:25:51.058 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-27T06:25:51.141 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-27T06:25:51.186 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-07-27T06:25:51.186 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-27T06:25:51.252 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-27T06:25:51.295 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-27T06:25:51.350 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-27T06:25:51.434 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-27T06:25:51.473 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-07-27T06:25:51.473 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-27T06:25:51.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-07-27T06:25:51.541 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2024-07-27T06:25:51.543 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-07-27T06:25:51.580 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-27T06:25:51.593 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-27T06:25:51.616 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-27T06:25:52.062 INFO:teuthology.run_tasks:Running task internal.timer... 2024-07-27T06:25:52.070 INFO:teuthology.task.internal:Starting timer... 2024-07-27T06:25:52.070 INFO:teuthology.run_tasks:Running task pcp... 2024-07-27T06:25:52.079 INFO:teuthology.run_tasks:Running task selinux... 2024-07-27T06:25:52.088 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-07-27T06:25:52.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-07-27T06:25:52.189 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-07-27T06:25:52.191 DEBUG:teuthology.orchestra.run.smithi047:> sudo service auditd rotate 2024-07-27T06:25:52.249 INFO:teuthology.orchestra.run.smithi047.stdout:Rotating logs: 2024-07-27T06:25:52.251 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-07-27T06:25:52.316 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-07-27T06:25:52.318 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-07-27T06:25:52.318 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-07-27T06:25:52.345 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-07-27T06:25:52.346 DEBUG:teuthology.orchestra.run.smithi047:> /usr/sbin/getenforce 2024-07-27T06:25:52.384 INFO:teuthology.orchestra.run.smithi047.stdout:Permissive 2024-07-27T06:25:52.384 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-07-27T06:25:52.416 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-07-27T06:25:52.416 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi047.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi077.front.sepia.ceph.com': 'permissive'} 2024-07-27T06:25:52.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-27T06:25:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T06:25:52.444 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-27T06:25:52.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T06:25:52.471 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-27T06:25:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T06:25:52.499 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-07-27T06:25:52.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-07-27T06:25:52.541 DEBUG:teuthology.orchestra.run.smithi047:> sudo /usr/sbin/setenforce permissive 2024-07-27T06:25:52.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-07-27T06:25:52.593 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-07-27T06:25:52.602 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-07-27T06:25:52.605 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-07-27T06:25:52.718 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-07-27T06:25:52.758 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-07-27T06:25:52.762 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi047.front.sepia.ceph.com,smithi077.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-07-27T06:30:45.465 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi047.front.sepia.ceph.com'), Remote(name='ubuntu@smithi077.front.sepia.ceph.com')] 2024-07-27T06:30:45.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-07-27T06:30:45.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T06:30:45.545 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-07-27T06:30:45.623 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-07-27T06:30:45.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-07-27T06:30:45.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T06:30:45.700 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-07-27T06:30:45.782 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-07-27T06:30:45.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-07-27T06:30:45.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-27T06:30:45.858 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-07-27T06:30:45.942 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-07-27T06:30:45.943 INFO:teuthology.run_tasks:Running task clock... 2024-07-27T06:30:45.953 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-07-27T06:30:45.954 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-27T06:30:45.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-27T06:30:45.957 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-27T06:30:45.957 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-27T06:30:45.959 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-27T06:30:45.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-27T06:30:45.987 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-27T06:30:45.987 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-27T06:30:46.001 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-27T06:30:46.003 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-27T06:30:46.012 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-27T06:30:46.026 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: ntpd: command not found 2024-07-27T06:30:46.026 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-27T06:30:46.031 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-07-27T06:30:46.038 INFO:teuthology.orchestra.run.smithi047.stdout:506 Cannot talk to daemon 2024-07-27T06:30:46.043 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-07-27T06:30:46.049 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-07-27T06:30:46.051 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-27T06:30:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-27T06:30:46.060 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-07-27T06:30:46.064 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-27T06:30:46.073 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-27T06:30:46.074 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-27T06:30:46.086 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-27T06:30:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:bash: line 1: ntpq: command not found 2024-07-27T06:30:46.121 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-27T06:30:46.121 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-07-27T06:30:46.121 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T06:30:46.121 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T06:30:46.121 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T06:30:46.121 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T06:30:46.134 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-07-27T06:30:46.136 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-07-27T06:30:46.137 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-27T06:30:46.137 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-07-27T06:30:46.137 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T06:30:46.138 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T06:30:46.138 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T06:30:46.138 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T06:30:46.140 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-27T06:30:46.140 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-07-27T06:30:46.140 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T06:30:46.140 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T06:30:46.140 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T06:30:46.141 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-27T06:30:46.141 INFO:teuthology.run_tasks:Running task install... 2024-07-27T06:30:46.149 DEBUG:teuthology.task.install:project ceph 2024-07-27T06:30:46.149 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'}} 2024-07-27T06:30:46.149 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'} 2024-07-27T06:30:46.149 INFO:teuthology.task.install:Using flavor: default 2024-07-27T06:30:46.155 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-07-27T06:30:46.155 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-07-27T06:30:46.155 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-27T06:30:46.156 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:30:46.158 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-27T06:30:46.158 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:30:46.159 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-27T06:30:46.159 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:30:46.316 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-27T06:30:46.316 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-27T06:30:46.321 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-27T06:30:46.321 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-27T06:30:46.356 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-27T06:30:46.356 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-27T06:30:46.428 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-27T06:30:46.428 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:30:46.428 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-27T06:30:46.433 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-27T06:30:46.433 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-07-27T06:30:46.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-27T06:30:46.456 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-27T06:30:46.456 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-27T06:30:46.459 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-27T06:30:46.459 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-27T06:30:46.464 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-27T06:30:46.465 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-07-27T06:30:46.465 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-27T06:30:46.490 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-27T06:30:46.491 DEBUG:teuthology.orchestra.run.smithi047:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-27T06:30:46.523 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-27T06:30:46.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-27T06:30:46.557 DEBUG:teuthology.orchestra.run.smithi047:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-27T06:30:46.596 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-27T06:30:46.601 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-27T06:30:46.631 DEBUG:teuthology.orchestra.run.smithi047:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-27T06:30:46.660 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2024-07-27T06:30:46.661 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2024-07-27T06:30:46.667 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-07-27T06:30:46.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-07-27T06:30:46.695 INFO:teuthology.orchestra.run.smithi047.stdout:check_obsoletes = 1 2024-07-27T06:30:46.696 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum clean all 2024-07-27T06:30:46.885 INFO:teuthology.orchestra.run.smithi077.stdout:45 files removed 2024-07-27T06:30:46.909 INFO:teuthology.orchestra.run.smithi029.stdout:45 files removed 2024-07-27T06:30:46.912 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-27T06:30:46.930 INFO:teuthology.orchestra.run.smithi047.stdout:45 files removed 2024-07-27T06:30:46.935 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-27T06:30:46.958 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-27T06:30:47.502 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 253 kB/s | 83 kB 00:00 2024-07-27T06:30:47.555 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 260 kB/s | 83 kB 00:00 2024-07-27T06:30:47.559 INFO:teuthology.orchestra.run.smithi047.stdout:ceph packages for x86_64 262 kB/s | 83 kB 00:00 2024-07-27T06:30:47.844 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 80 kB/s | 24 kB 00:00 2024-07-27T06:30:47.905 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 81 kB/s | 24 kB 00:00 2024-07-27T06:30:47.909 INFO:teuthology.orchestra.run.smithi047.stdout:ceph noarch packages 78 kB/s | 24 kB 00:00 2024-07-27T06:30:48.186 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-07-27T06:30:48.248 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 6.3 kB/s | 1.9 kB 00:00 2024-07-27T06:30:48.252 INFO:teuthology.orchestra.run.smithi047.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-07-27T06:30:48.969 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-07-27T06:30:49.460 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - BaseOS 7.0 MB/s | 8.2 MB 00:01 2024-07-27T06:30:49.609 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.2 MB 00:01 2024-07-27T06:30:50.837 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-07-27T06:30:52.720 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-07-27T06:30:56.211 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - AppStream 3.6 MB/s | 20 MB 00:05 2024-07-27T06:30:57.105 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.5 MB 00:02 2024-07-27T06:30:57.197 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-07-27T06:30:58.701 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 88 kB/s | 18 kB 00:00 2024-07-27T06:30:58.705 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 18 kB 00:00 2024-07-27T06:30:59.735 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 25 MB/s | 22 MB 00:00 2024-07-27T06:30:59.831 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:00 2024-07-27T06:31:01.522 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 6.5 MB 00:01 2024-07-27T06:31:03.129 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 18 kB 00:00 2024-07-27T06:31:04.100 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 27 MB/s | 22 MB 00:00 2024-07-27T06:31:05.083 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-07-27T06:31:05.472 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-07-27T06:31:07.625 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-27T06:31:07.626 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-27T06:31:07.637 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-27T06:31:07.638 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-27T06:31:07.638 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-27T06:31:07.639 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-27T06:31:07.641 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-27T06:31:07.643 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-27T06:31:07.644 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-27T06:31:07.707 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-07-27T06:31:07.724 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-07-27T06:31:07.724 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-07-27T06:31:07.724 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-07-27T06:31:07.724 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-07-27T06:31:07.724 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-27T06:31:07.725 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-27T06:31:07.726 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-27T06:31:07.726 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-27T06:31:07.726 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-27T06:31:07.726 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-27T06:31:07.726 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-27T06:31:07.726 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-27T06:31:07.726 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-27T06:31:07.726 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-27T06:31:07.726 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-27T06:31:07.726 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-27T06:31:07.726 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-27T06:31:07.726 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-27T06:31:07.727 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-27T06:31:07.727 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-27T06:31:07.727 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-27T06:31:07.727 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-27T06:31:07.727 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-27T06:31:07.727 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-27T06:31:07.727 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-27T06:31:07.727 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-27T06:31:07.727 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-27T06:31:07.727 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-27T06:31:07.727 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-07-27T06:31:07.727 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-27T06:31:07.728 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-27T06:31:07.729 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-27T06:31:07.730 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-27T06:31:07.730 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-27T06:31:07.730 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-27T06:31:07.730 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-27T06:31:07.730 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-27T06:31:07.730 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-27T06:31:07.730 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-27T06:31:07.730 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-27T06:31:07.730 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-27T06:31:07.730 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-27T06:31:07.730 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-27T06:31:07.730 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-27T06:31:07.731 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-27T06:31:07.731 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-27T06:31:07.731 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-27T06:31:07.731 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-27T06:31:07.731 INFO:teuthology.orchestra.run.smithi077.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-27T06:31:07.731 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-27T06:31:07.731 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-27T06:31:07.731 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-27T06:31:07.732 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-27T06:31:07.733 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-27T06:31:07.733 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-27T06:31:07.733 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-27T06:31:07.733 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-27T06:31:07.733 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-27T06:31:07.733 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-27T06:31:07.733 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-27T06:31:07.733 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-27T06:31:07.733 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-27T06:31:07.733 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-27T06:31:07.733 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-27T06:31:07.734 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-27T06:31:07.734 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-27T06:31:07.734 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-27T06:31:07.734 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-27T06:31:07.734 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-27T06:31:07.734 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-27T06:31:07.734 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-27T06:31:07.734 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-27T06:31:07.734 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-27T06:31:07.734 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-27T06:31:07.734 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-27T06:31:07.735 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-27T06:31:07.736 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-27T06:31:07.737 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-27T06:31:07.737 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-27T06:31:07.737 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-27T06:31:07.737 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-27T06:31:07.737 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-27T06:31:07.737 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-27T06:31:07.737 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-27T06:31:07.737 INFO:teuthology.orchestra.run.smithi077.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-27T06:31:07.737 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-27T06:31:07.737 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-27T06:31:07.737 INFO:teuthology.orchestra.run.smithi077.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-27T06:31:07.737 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-27T06:31:07.738 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-27T06:31:07.738 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2024-07-27T06:31:07.738 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-27T06:31:07.738 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:31:07.738 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-07-27T06:31:07.738 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-07-27T06:31:07.738 INFO:teuthology.orchestra.run.smithi077.stdout:Install 148 Packages 2024-07-27T06:31:07.738 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 12 Packages 2024-07-27T06:31:07.738 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:31:07.739 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 233 M 2024-07-27T06:31:07.739 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-07-27T06:31:07.953 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-27T06:31:07.954 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-27T06:31:07.965 INFO:teuthology.orchestra.run.smithi029.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-27T06:31:07.966 INFO:teuthology.orchestra.run.smithi029.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-27T06:31:07.967 INFO:teuthology.orchestra.run.smithi029.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-27T06:31:07.968 INFO:teuthology.orchestra.run.smithi029.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-27T06:31:07.970 INFO:teuthology.orchestra.run.smithi029.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-27T06:31:07.972 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-27T06:31:07.973 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-27T06:31:08.042 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-07-27T06:31:08.069 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-07-27T06:31:08.069 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-07-27T06:31:08.069 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-07-27T06:31:08.070 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-07-27T06:31:08.070 INFO:teuthology.orchestra.run.smithi029.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-27T06:31:08.070 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-27T06:31:08.070 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-27T06:31:08.070 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-27T06:31:08.070 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-27T06:31:08.070 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-27T06:31:08.070 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-27T06:31:08.070 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-27T06:31:08.070 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-27T06:31:08.070 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-27T06:31:08.071 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-27T06:31:08.072 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-27T06:31:08.073 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-27T06:31:08.073 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-07-27T06:31:08.073 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-27T06:31:08.073 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-27T06:31:08.073 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-27T06:31:08.073 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-27T06:31:08.073 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-27T06:31:08.073 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-27T06:31:08.073 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-27T06:31:08.073 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-27T06:31:08.073 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-27T06:31:08.073 INFO:teuthology.orchestra.run.smithi029.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-27T06:31:08.074 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-27T06:31:08.074 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-27T06:31:08.074 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-07-27T06:31:08.074 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-27T06:31:08.074 INFO:teuthology.orchestra.run.smithi029.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-27T06:31:08.074 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-27T06:31:08.074 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-27T06:31:08.074 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-27T06:31:08.074 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-27T06:31:08.074 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-27T06:31:08.074 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-27T06:31:08.075 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-27T06:31:08.075 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-27T06:31:08.075 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-27T06:31:08.075 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-27T06:31:08.075 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-27T06:31:08.075 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-27T06:31:08.075 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-27T06:31:08.075 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-27T06:31:08.075 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-27T06:31:08.075 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-27T06:31:08.075 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-27T06:31:08.075 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-27T06:31:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-27T06:31:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-27T06:31:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-27T06:31:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-27T06:31:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-27T06:31:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-27T06:31:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-27T06:31:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-27T06:31:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-27T06:31:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-27T06:31:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-27T06:31:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-27T06:31:08.077 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-27T06:31:08.078 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-27T06:31:08.079 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-27T06:31:08.079 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-27T06:31:08.079 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-27T06:31:08.079 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-27T06:31:08.079 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-27T06:31:08.079 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-27T06:31:08.079 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-27T06:31:08.079 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-27T06:31:08.079 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-27T06:31:08.079 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-27T06:31:08.079 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-27T06:31:08.080 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-27T06:31:08.080 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-27T06:31:08.080 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-27T06:31:08.080 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-27T06:31:08.080 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-27T06:31:08.080 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-27T06:31:08.080 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-27T06:31:08.080 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-27T06:31:08.080 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-27T06:31:08.080 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-27T06:31:08.080 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-27T06:31:08.080 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-27T06:31:08.081 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-27T06:31:08.082 INFO:teuthology.orchestra.run.smithi029.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-27T06:31:08.083 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-27T06:31:08.083 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-27T06:31:08.083 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-07-27T06:31:08.083 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-27T06:31:08.083 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:31:08.083 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-07-27T06:31:08.083 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-07-27T06:31:08.083 INFO:teuthology.orchestra.run.smithi029.stdout:Install 148 Packages 2024-07-27T06:31:08.083 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 12 Packages 2024-07-27T06:31:08.083 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:31:08.084 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 233 M 2024-07-27T06:31:08.084 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-07-27T06:31:08.559 INFO:teuthology.orchestra.run.smithi077.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-07-27T06:31:08.776 INFO:teuthology.orchestra.run.smithi077.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 3.9 MB/s | 872 kB 00:00 2024-07-27T06:31:08.859 INFO:teuthology.orchestra.run.smithi077.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 1.7 MB/s | 148 kB 00:00 2024-07-27T06:31:08.867 INFO:teuthology.orchestra.run.smithi029.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-07-27T06:31:08.943 INFO:teuthology.orchestra.run.smithi077.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 9.8 MB/s | 5.5 MB 00:00 2024-07-27T06:31:09.085 INFO:teuthology.orchestra.run.smithi077.stdout:(5/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 11 MB/s | 1.5 MB 00:00 2024-07-27T06:31:09.227 INFO:teuthology.orchestra.run.smithi077.stdout:(6/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 6.2 MB/s | 2.3 MB 00:00 2024-07-27T06:31:09.409 INFO:teuthology.orchestra.run.smithi029.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.6 MB/s | 872 kB 00:00 2024-07-27T06:31:09.410 INFO:teuthology.orchestra.run.smithi077.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 15 MB/s | 4.9 MB 00:00 2024-07-27T06:31:09.527 INFO:teuthology.orchestra.run.smithi029.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 1.2 MB/s | 148 kB 00:00 2024-07-27T06:31:09.598 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-07-27T06:31:09.669 INFO:teuthology.orchestra.run.smithi077.stdout:(8/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 15 MB/s | 20 MB 00:01 2024-07-27T06:31:09.719 INFO:teuthology.orchestra.run.smithi077.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 505 kB/s | 25 kB 00:00 2024-07-27T06:31:09.777 INFO:teuthology.orchestra.run.smithi029.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 4.7 MB/s | 5.5 MB 00:01 2024-07-27T06:31:10.103 INFO:teuthology.orchestra.run.smithi029.stdout:(5/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 4.7 MB/s | 1.5 MB 00:00 2024-07-27T06:31:10.162 INFO:teuthology.orchestra.run.smithi029.stdout:(6/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 3.6 MB/s | 2.3 MB 00:00 2024-07-27T06:31:10.178 INFO:teuthology.orchestra.run.smithi077.stdout:(10/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 14 MB/s | 10 MB 00:00 2024-07-27T06:31:10.270 INFO:teuthology.orchestra.run.smithi077.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 2.4 MB/s | 225 kB 00:00 2024-07-27T06:31:10.345 INFO:teuthology.orchestra.run.smithi077.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 360 kB/s | 27 kB 00:00 2024-07-27T06:31:10.421 INFO:teuthology.orchestra.run.smithi077.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 424 kB/s | 32 kB 00:00 2024-07-27T06:31:10.521 INFO:teuthology.orchestra.run.smithi077.stdout:(14/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 7.5 MB/s | 765 kB 00:00 2024-07-27T06:31:10.604 INFO:teuthology.orchestra.run.smithi077.stdout:(15/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 1.9 MB/s | 158 kB 00:00 2024-07-27T06:31:10.688 INFO:teuthology.orchestra.run.smithi077.stdout:(16/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.5 MB/s | 127 kB 00:00 2024-07-27T06:31:10.780 INFO:teuthology.orchestra.run.smithi077.stdout:(17/160): libradosstriper1-19.0.0-5320.ge8f88bb 5.4 MB/s | 503 kB 00:00 2024-07-27T06:31:10.871 INFO:teuthology.orchestra.run.smithi029.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 6.4 MB/s | 4.9 MB 00:00 2024-07-27T06:31:11.123 INFO:teuthology.orchestra.run.smithi077.stdout:(18/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 9.4 MB/s | 18 MB 00:01 2024-07-27T06:31:11.206 INFO:teuthology.orchestra.run.smithi077.stdout:(19/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 12 MB/s | 5.3 MB 00:00 2024-07-27T06:31:11.215 INFO:teuthology.orchestra.run.smithi029.stdout:(8/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 7.6 MB/s | 20 MB 00:02 2024-07-27T06:31:11.223 INFO:teuthology.orchestra.run.smithi077.stdout:(20/160): python3-ceph-argparse-19.0.0-5320.ge8 450 kB/s | 45 kB 00:00 2024-07-27T06:31:11.257 INFO:teuthology.orchestra.run.smithi077.stdout:(21/160): python3-ceph-common-19.0.0-5320.ge8f8 2.7 MB/s | 138 kB 00:00 2024-07-27T06:31:11.263 INFO:teuthology.orchestra.run.smithi029.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 503 kB/s | 25 kB 00:00 2024-07-27T06:31:11.282 INFO:teuthology.orchestra.run.smithi077.stdout:(22/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 3.1 MB/s | 188 kB 00:00 2024-07-27T06:31:11.315 INFO:teuthology.orchestra.run.smithi077.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 6.2 MB/s | 372 kB 00:00 2024-07-27T06:31:11.374 INFO:teuthology.orchestra.run.smithi077.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 3.8 MB/s | 355 kB 00:00 2024-07-27T06:31:11.399 INFO:teuthology.orchestra.run.smithi077.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 1.4 MB/s | 116 kB 00:00 2024-07-27T06:31:11.450 INFO:teuthology.orchestra.run.smithi077.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.1 MB/s | 86 kB 00:00 2024-07-27T06:31:11.550 INFO:teuthology.orchestra.run.smithi077.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 1.7 MB/s | 170 kB 00:00 2024-07-27T06:31:11.650 INFO:teuthology.orchestra.run.smithi077.stdout:(28/160): ceph-grafana-dashboards-19.0.0-5320.g 343 kB/s | 34 kB 00:00 2024-07-27T06:31:11.734 INFO:teuthology.orchestra.run.smithi077.stdout:(29/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 9.2 MB/s | 3.1 MB 00:00 2024-07-27T06:31:11.759 INFO:teuthology.orchestra.run.smithi077.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 1.3 MB/s | 149 kB 00:00 2024-07-27T06:31:12.162 INFO:teuthology.orchestra.run.smithi047.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-27T06:31:12.163 INFO:teuthology.orchestra.run.smithi047.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-27T06:31:12.172 INFO:teuthology.orchestra.run.smithi029.stdout:(10/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 8.8 MB/s | 18 MB 00:02 2024-07-27T06:31:12.175 INFO:teuthology.orchestra.run.smithi047.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-27T06:31:12.176 INFO:teuthology.orchestra.run.smithi047.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-27T06:31:12.177 INFO:teuthology.orchestra.run.smithi047.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-27T06:31:12.177 INFO:teuthology.orchestra.run.smithi047.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-27T06:31:12.179 INFO:teuthology.orchestra.run.smithi047.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-27T06:31:12.181 INFO:teuthology.orchestra.run.smithi047.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-27T06:31:12.182 INFO:teuthology.orchestra.run.smithi047.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-27T06:31:12.231 INFO:teuthology.orchestra.run.smithi029.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 3.8 MB/s | 225 kB 00:00 2024-07-27T06:31:12.250 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-07-27T06:31:12.272 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================= 2024-07-27T06:31:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-07-27T06:31:12.272 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================= 2024-07-27T06:31:12.272 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-07-27T06:31:12.272 INFO:teuthology.orchestra.run.smithi047.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-27T06:31:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-27T06:31:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-27T06:31:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-27T06:31:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-27T06:31:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-27T06:31:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-27T06:31:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-27T06:31:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-27T06:31:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-27T06:31:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-27T06:31:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-27T06:31:12.273 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-27T06:31:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-27T06:31:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-27T06:31:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-27T06:31:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-27T06:31:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-27T06:31:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-27T06:31:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-27T06:31:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-27T06:31:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-27T06:31:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-27T06:31:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-27T06:31:12.274 INFO:teuthology.orchestra.run.smithi047.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-27T06:31:12.275 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrading: 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-07-27T06:31:12.276 INFO:teuthology.orchestra.run.smithi047.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-27T06:31:12.277 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-27T06:31:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-27T06:31:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-27T06:31:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-27T06:31:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-27T06:31:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-27T06:31:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-27T06:31:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-27T06:31:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-27T06:31:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-27T06:31:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-27T06:31:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-27T06:31:12.278 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-27T06:31:12.279 INFO:teuthology.orchestra.run.smithi047.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-27T06:31:12.280 INFO:teuthology.orchestra.run.smithi047.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-27T06:31:12.280 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-27T06:31:12.280 INFO:teuthology.orchestra.run.smithi047.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-27T06:31:12.280 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-27T06:31:12.280 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-27T06:31:12.280 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-27T06:31:12.280 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-27T06:31:12.280 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-27T06:31:12.280 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-27T06:31:12.280 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-27T06:31:12.280 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-27T06:31:12.281 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-27T06:31:12.281 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-27T06:31:12.281 INFO:teuthology.orchestra.run.smithi047.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-27T06:31:12.281 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-27T06:31:12.281 INFO:teuthology.orchestra.run.smithi047.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-27T06:31:12.281 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-27T06:31:12.281 INFO:teuthology.orchestra.run.smithi047.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-27T06:31:12.281 INFO:teuthology.orchestra.run.smithi047.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-27T06:31:12.281 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-27T06:31:12.281 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-27T06:31:12.281 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-27T06:31:12.281 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-27T06:31:12.282 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-27T06:31:12.282 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-27T06:31:12.282 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-27T06:31:12.282 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-27T06:31:12.282 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-27T06:31:12.282 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-27T06:31:12.282 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-27T06:31:12.282 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-27T06:31:12.282 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-27T06:31:12.282 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-27T06:31:12.282 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-27T06:31:12.282 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-27T06:31:12.283 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-27T06:31:12.284 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout:Installing weak dependencies: 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-07-27T06:31:12.285 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================= 2024-07-27T06:31:12.286 INFO:teuthology.orchestra.run.smithi047.stdout:Install 148 Packages 2024-07-27T06:31:12.286 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrade 12 Packages 2024-07-27T06:31:12.286 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:31:12.286 INFO:teuthology.orchestra.run.smithi029.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 541 kB/s | 27 kB 00:00 2024-07-27T06:31:12.287 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 233 M 2024-07-27T06:31:12.287 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-07-27T06:31:12.333 INFO:teuthology.orchestra.run.smithi029.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 630 kB/s | 32 kB 00:00 2024-07-27T06:31:12.418 INFO:teuthology.orchestra.run.smithi077.stdout:(31/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 16 MB/s | 44 MB 00:02 2024-07-27T06:31:12.468 INFO:teuthology.orchestra.run.smithi077.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 5.6 MB/s | 284 kB 00:00 2024-07-27T06:31:12.516 INFO:teuthology.orchestra.run.smithi029.stdout:(14/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 6.3 MB/s | 10 MB 00:01 2024-07-27T06:31:12.527 INFO:teuthology.orchestra.run.smithi077.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 850 kB/s | 49 kB 00:00 2024-07-27T06:31:12.560 INFO:teuthology.orchestra.run.smithi029.stdout:(15/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 3.3 MB/s | 765 kB 00:00 2024-07-27T06:31:12.585 INFO:teuthology.orchestra.run.smithi077.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 282 kB/s | 16 kB 00:00 2024-07-27T06:31:12.593 INFO:teuthology.orchestra.run.smithi029.stdout:(16/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 2.0 MB/s | 158 kB 00:00 2024-07-27T06:31:12.627 INFO:teuthology.orchestra.run.smithi029.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.8 MB/s | 127 kB 00:00 2024-07-27T06:31:12.678 INFO:teuthology.orchestra.run.smithi029.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 5.8 MB/s | 503 kB 00:00 2024-07-27T06:31:12.737 INFO:teuthology.orchestra.run.smithi029.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 772 kB/s | 45 kB 00:00 2024-07-27T06:31:12.761 INFO:teuthology.orchestra.run.smithi077.stdout:(35/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 14 MB/s | 14 MB 00:01 2024-07-27T06:31:12.786 INFO:teuthology.orchestra.run.smithi077.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 1.3 MB/s | 274 kB 00:00 2024-07-27T06:31:12.804 INFO:teuthology.orchestra.run.smithi029.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 2.0 MB/s | 138 kB 00:00 2024-07-27T06:31:12.878 INFO:teuthology.orchestra.run.smithi077.stdout:(37/160): ceph-mgr-diskprediction-local-19.0.0- 6.6 MB/s | 7.4 MB 00:01 2024-07-27T06:31:12.882 INFO:teuthology.orchestra.run.smithi047.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 34 kB/s | 6.6 kB 00:00 2024-07-27T06:31:12.903 INFO:teuthology.orchestra.run.smithi077.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 156 kB/s | 18 kB 00:00 2024-07-27T06:31:12.937 INFO:teuthology.orchestra.run.smithi077.stdout:(39/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 4.3 MB/s | 771 kB 00:00 2024-07-27T06:31:12.946 INFO:teuthology.orchestra.run.smithi029.stdout:(21/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 17 MB/s | 5.3 MB 00:00 2024-07-27T06:31:12.972 INFO:teuthology.orchestra.run.smithi029.stdout:(22/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 1.1 MB/s | 188 kB 00:00 2024-07-27T06:31:13.006 INFO:teuthology.orchestra.run.smithi029.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 6.2 MB/s | 372 kB 00:00 2024-07-27T06:31:13.040 INFO:teuthology.orchestra.run.smithi029.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 5.2 MB/s | 355 kB 00:00 2024-07-27T06:31:13.054 INFO:teuthology.orchestra.run.smithi077.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 455 kB/s | 80 kB 00:00 2024-07-27T06:31:13.082 INFO:teuthology.orchestra.run.smithi029.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 1.5 MB/s | 116 kB 00:00 2024-07-27T06:31:13.124 INFO:teuthology.orchestra.run.smithi029.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.0 MB/s | 86 kB 00:00 2024-07-27T06:31:13.212 INFO:teuthology.orchestra.run.smithi077.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 709 kB/s | 195 kB 00:00 2024-07-27T06:31:13.225 INFO:teuthology.orchestra.run.smithi029.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 1.7 MB/s | 170 kB 00:00 2024-07-27T06:31:13.240 INFO:teuthology.orchestra.run.smithi047.stdout:(2/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 9.9 MB/s | 5.5 MB 00:00 2024-07-27T06:31:13.274 INFO:teuthology.orchestra.run.smithi047.stdout:(3/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 2.2 MB/s | 872 kB 00:00 2024-07-27T06:31:13.307 INFO:teuthology.orchestra.run.smithi047.stdout:(4/160): ceph-immutable-object-cache-19.0.0-532 2.2 MB/s | 148 kB 00:00 2024-07-27T06:31:13.334 INFO:teuthology.orchestra.run.smithi029.stdout:(28/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 12 MB/s | 3.1 MB 00:00 2024-07-27T06:31:13.354 INFO:teuthology.orchestra.run.smithi077.stdout:(42/160): python3-cryptography-36.0.1-4.el9.x86 8.7 MB/s | 1.2 MB 00:00 2024-07-27T06:31:13.359 INFO:teuthology.orchestra.run.smithi029.stdout:(29/160): ceph-grafana-dashboards-19.0.0-5320.g 256 kB/s | 34 kB 00:00 2024-07-27T06:31:13.413 INFO:teuthology.orchestra.run.smithi077.stdout:(43/160): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-07-27T06:31:13.420 INFO:teuthology.orchestra.run.smithi029.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 1.7 MB/s | 149 kB 00:00 2024-07-27T06:31:13.449 INFO:teuthology.orchestra.run.smithi047.stdout:(5/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 11 MB/s | 1.5 MB 00:00 2024-07-27T06:31:13.472 INFO:teuthology.orchestra.run.smithi077.stdout:(44/160): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-07-27T06:31:13.530 INFO:teuthology.orchestra.run.smithi077.stdout:(45/160): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-07-27T06:31:13.589 INFO:teuthology.orchestra.run.smithi077.stdout:(46/160): python3-urllib3-1.26.5-6.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-07-27T06:31:13.739 INFO:teuthology.orchestra.run.smithi077.stdout:(47/160): automake-1.16.2-8.el9.noarch.rpm 4.5 MB/s | 689 kB 00:00 2024-07-27T06:31:13.806 INFO:teuthology.orchestra.run.smithi077.stdout:(48/160): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-07-27T06:31:13.841 INFO:teuthology.orchestra.run.smithi047.stdout:(6/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 4.0 MB/s | 2.3 MB 00:00 2024-07-27T06:31:13.848 INFO:teuthology.orchestra.run.smithi077.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-07-27T06:31:13.882 INFO:teuthology.orchestra.run.smithi077.stdout:(50/160): elfutils-libelf-devel-0.191-4.el9.x86 653 kB/s | 22 kB 00:00 2024-07-27T06:31:13.932 INFO:teuthology.orchestra.run.smithi077.stdout:(51/160): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-07-27T06:31:13.966 INFO:teuthology.orchestra.run.smithi077.stdout:(52/160): flexiblas-3.0.4-8.el9.x86_64.rpm 984 kB/s | 33 kB 00:00 2024-07-27T06:31:14.029 INFO:teuthology.orchestra.run.smithi029.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-07-27T06:31:14.042 INFO:teuthology.orchestra.run.smithi047.stdout:(7/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 15 MB/s | 20 MB 00:01 2024-07-27T06:31:14.066 INFO:teuthology.orchestra.run.smithi077.stdout:(53/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-07-27T06:31:14.087 INFO:teuthology.orchestra.run.smithi029.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 4.8 MB/s | 284 kB 00:00 2024-07-27T06:31:14.100 INFO:teuthology.orchestra.run.smithi077.stdout:(54/160): flexiblas-openblas-openmp-3.0.4-8.el9 546 kB/s | 18 kB 00:00 2024-07-27T06:31:14.109 INFO:teuthology.orchestra.run.smithi047.stdout:(8/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 7.4 MB/s | 4.9 MB 00:00 2024-07-27T06:31:14.138 INFO:teuthology.orchestra.run.smithi029.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 981 kB/s | 49 kB 00:00 2024-07-27T06:31:14.150 INFO:teuthology.orchestra.run.smithi077.stdout:(55/160): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-07-27T06:31:14.159 INFO:teuthology.orchestra.run.smithi047.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 504 kB/s | 25 kB 00:00 2024-07-27T06:31:14.192 INFO:teuthology.orchestra.run.smithi077.stdout:(56/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 160 kB 00:00 2024-07-27T06:31:14.205 INFO:teuthology.orchestra.run.smithi029.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 246 kB/s | 16 kB 00:00 2024-07-27T06:31:14.234 INFO:teuthology.orchestra.run.smithi077.stdout:(57/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-07-27T06:31:14.284 INFO:teuthology.orchestra.run.smithi077.stdout:(58/160): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-07-27T06:31:14.297 INFO:teuthology.orchestra.run.smithi029.stdout:(35/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 2.9 MB/s | 274 kB 00:00 2024-07-27T06:31:14.343 INFO:teuthology.orchestra.run.smithi077.stdout:(59/160): libtool-2.4.6-46.el9.x86_64.rpm 9.7 MB/s | 581 kB 00:00 2024-07-27T06:31:14.401 INFO:teuthology.orchestra.run.smithi077.stdout:(60/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 436 kB/s | 25 kB 00:00 2024-07-27T06:31:14.406 INFO:teuthology.orchestra.run.smithi029.stdout:(36/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 6.9 MB/s | 771 kB 00:00 2024-07-27T06:31:14.460 INFO:teuthology.orchestra.run.smithi077.stdout:(61/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 814 kB/s | 47 kB 00:00 2024-07-27T06:31:14.490 INFO:teuthology.orchestra.run.smithi029.stdout:(37/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 219 kB/s | 18 kB 00:00 2024-07-27T06:31:14.502 INFO:teuthology.orchestra.run.smithi077.stdout:(62/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.9 MB/s | 292 kB 00:00 2024-07-27T06:31:14.544 INFO:teuthology.orchestra.run.smithi077.stdout:(63/160): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 188 kB 00:00 2024-07-27T06:31:14.586 INFO:teuthology.orchestra.run.smithi077.stdout:(64/160): openblas-0.3.21-2.el9.x86_64.rpm 845 kB/s | 35 kB 00:00 2024-07-27T06:31:14.634 INFO:teuthology.orchestra.run.smithi047.stdout:(10/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 18 MB/s | 10 MB 00:00 2024-07-27T06:31:14.693 INFO:teuthology.orchestra.run.smithi047.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 3.8 MB/s | 225 kB 00:00 2024-07-27T06:31:14.711 INFO:teuthology.orchestra.run.smithi077.stdout:(65/160): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-07-27T06:31:14.743 INFO:teuthology.orchestra.run.smithi047.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 543 kB/s | 27 kB 00:00 2024-07-27T06:31:14.801 INFO:teuthology.orchestra.run.smithi047.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 543 kB/s | 32 kB 00:00 2024-07-27T06:31:14.828 INFO:teuthology.orchestra.run.smithi077.stdout:(66/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 38 MB/s | 4.4 MB 00:00 2024-07-27T06:31:14.910 INFO:teuthology.orchestra.run.smithi047.stdout:(14/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 6.9 MB/s | 765 kB 00:00 2024-07-27T06:31:14.912 INFO:teuthology.orchestra.run.smithi077.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-07-27T06:31:14.977 INFO:teuthology.orchestra.run.smithi047.stdout:(15/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 2.3 MB/s | 158 kB 00:00 2024-07-27T06:31:14.987 INFO:teuthology.orchestra.run.smithi077.stdout:(68/160): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-07-27T06:31:15.037 INFO:teuthology.orchestra.run.smithi077.stdout:(69/160): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-07-27T06:31:15.053 INFO:teuthology.orchestra.run.smithi047.stdout:(16/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.6 MB/s | 127 kB 00:00 2024-07-27T06:31:15.136 INFO:teuthology.orchestra.run.smithi047.stdout:(17/160): libradosstriper1-19.0.0-5320.ge8f88bb 5.9 MB/s | 503 kB 00:00 2024-07-27T06:31:15.179 INFO:teuthology.orchestra.run.smithi077.stdout:(70/160): postgresql-server-13.14-1.el9.x86_64. 41 MB/s | 5.8 MB 00:00 2024-07-27T06:31:15.224 INFO:teuthology.orchestra.run.smithi029.stdout:(38/160): fuse-2.9.9-16.el9.x86_64.rpm 109 kB/s | 80 kB 00:00 2024-07-27T06:31:15.238 INFO:teuthology.orchestra.run.smithi077.stdout:(71/160): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-07-27T06:31:15.263 INFO:teuthology.orchestra.run.smithi077.stdout:(72/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 115 kB/s | 253 kB 00:02 2024-07-27T06:31:15.288 INFO:teuthology.orchestra.run.smithi077.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-07-27T06:31:15.331 INFO:teuthology.orchestra.run.smithi077.stdout:(74/160): libgfortran-11.4.1-3.el9.x86_64.rpm 331 kB/s | 803 kB 00:02 2024-07-27T06:31:15.356 INFO:teuthology.orchestra.run.smithi077.stdout:(75/160): python3-colorama-0.4.6-3.el9.noarch.r 914 kB/s | 61 kB 00:00 2024-07-27T06:31:15.398 INFO:teuthology.orchestra.run.smithi077.stdout:(76/160): python3-jinja2-2.11.3-6.el9.noarch.rp 5.8 MB/s | 248 kB 00:00 2024-07-27T06:31:15.440 INFO:teuthology.orchestra.run.smithi077.stdout:(77/160): python3-jsonpatch-1.21-16.el9.noarch. 614 kB/s | 26 kB 00:00 2024-07-27T06:31:15.462 INFO:teuthology.orchestra.run.smithi047.stdout:(18/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 11 MB/s | 18 MB 00:01 2024-07-27T06:31:15.482 INFO:teuthology.orchestra.run.smithi077.stdout:(78/160): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-07-27T06:31:15.507 INFO:teuthology.orchestra.run.smithi077.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 286 kB/s | 19 kB 00:00 2024-07-27T06:31:15.532 INFO:teuthology.orchestra.run.smithi077.stdout:(80/160): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-07-27T06:31:15.545 INFO:teuthology.orchestra.run.smithi047.stdout:(19/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 13 MB/s | 5.3 MB 00:00 2024-07-27T06:31:15.557 INFO:teuthology.orchestra.run.smithi077.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 688 kB/s | 35 kB 00:00 2024-07-27T06:31:15.570 INFO:teuthology.orchestra.run.smithi047.stdout:(20/160): python3-ceph-argparse-19.0.0-5320.ge8 416 kB/s | 45 kB 00:00 2024-07-27T06:31:15.595 INFO:teuthology.orchestra.run.smithi047.stdout:(21/160): python3-ceph-common-19.0.0-5320.ge8f8 2.7 MB/s | 138 kB 00:00 2024-07-27T06:31:15.621 INFO:teuthology.orchestra.run.smithi047.stdout:(22/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 3.7 MB/s | 188 kB 00:00 2024-07-27T06:31:15.641 INFO:teuthology.orchestra.run.smithi029.stdout:(39/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 6.1 MB/s | 14 MB 00:02 2024-07-27T06:31:15.646 INFO:teuthology.orchestra.run.smithi047.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 7.2 MB/s | 372 kB 00:00 2024-07-27T06:31:15.658 INFO:teuthology.orchestra.run.smithi077.stdout:(82/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 5.1 MB 00:00 2024-07-27T06:31:15.671 INFO:teuthology.orchestra.run.smithi047.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 6.9 MB/s | 355 kB 00:00 2024-07-27T06:31:15.683 INFO:teuthology.orchestra.run.smithi077.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 250 kB 00:00 2024-07-27T06:31:15.696 INFO:teuthology.orchestra.run.smithi047.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 2.3 MB/s | 116 kB 00:00 2024-07-27T06:31:15.708 INFO:teuthology.orchestra.run.smithi077.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-07-27T06:31:15.722 INFO:teuthology.orchestra.run.smithi047.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.7 MB/s | 86 kB 00:00 2024-07-27T06:31:15.734 INFO:teuthology.orchestra.run.smithi077.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-07-27T06:31:15.751 INFO:teuthology.orchestra.run.smithi077.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-07-27T06:31:15.789 INFO:teuthology.orchestra.run.smithi047.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 2.5 MB/s | 170 kB 00:00 2024-07-27T06:31:15.793 INFO:teuthology.orchestra.run.smithi077.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-07-27T06:31:15.819 INFO:teuthology.orchestra.run.smithi077.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-07-27T06:31:15.856 INFO:teuthology.orchestra.run.smithi047.stdout:(28/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 19 MB/s | 3.1 MB 00:00 2024-07-27T06:31:15.860 INFO:teuthology.orchestra.run.smithi077.stdout:(89/160): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-07-27T06:31:15.881 INFO:teuthology.orchestra.run.smithi047.stdout:(29/160): ceph-grafana-dashboards-19.0.0-5320.g 372 kB/s | 34 kB 00:00 2024-07-27T06:31:15.902 INFO:teuthology.orchestra.run.smithi077.stdout:(90/160): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-07-27T06:31:15.915 INFO:teuthology.orchestra.run.smithi047.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 2.5 MB/s | 149 kB 00:00 2024-07-27T06:31:15.928 INFO:teuthology.orchestra.run.smithi077.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 799 kB/s | 54 kB 00:00 2024-07-27T06:31:15.969 INFO:teuthology.orchestra.run.smithi077.stdout:(92/160): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-07-27T06:31:16.011 INFO:teuthology.orchestra.run.smithi077.stdout:(93/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-07-27T06:31:16.053 INFO:teuthology.orchestra.run.smithi077.stdout:(94/160): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-07-27T06:31:16.095 INFO:teuthology.orchestra.run.smithi077.stdout:(95/160): qatlib-service-24.02.0-1.el9.x86_64.r 856 kB/s | 36 kB 00:00 2024-07-27T06:31:16.137 INFO:teuthology.orchestra.run.smithi077.stdout:(96/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-07-27T06:31:16.179 INFO:teuthology.orchestra.run.smithi077.stdout:(97/160): socat-1.7.4.1-6.el9.x86_64.rpm 7.1 MB/s | 304 kB 00:00 2024-07-27T06:31:16.229 INFO:teuthology.orchestra.run.smithi077.stdout:(98/160): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-07-27T06:31:16.280 INFO:teuthology.orchestra.run.smithi077.stdout:(99/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rpm 1.9 MB/s | 96 kB 00:00 2024-07-27T06:31:16.322 INFO:teuthology.orchestra.run.smithi077.stdout:(100/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-07-27T06:31:16.364 INFO:teuthology.orchestra.run.smithi077.stdout:(101/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.0 MB/s | 45 kB 00:00 2024-07-27T06:31:16.406 INFO:teuthology.orchestra.run.smithi077.stdout:(102/160): lua-devel-5.4.4-4.el9.x86_64.rpm 532 kB/s | 22 kB 00:00 2024-07-27T06:31:16.423 INFO:teuthology.orchestra.run.smithi047.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 15 MB/s | 7.4 MB 00:00 2024-07-27T06:31:16.465 INFO:teuthology.orchestra.run.smithi077.stdout:(103/160): protobuf-compiler-3.14.0-13.el9.x86_ 15 MB/s | 863 kB 00:00 2024-07-27T06:31:16.492 INFO:teuthology.orchestra.run.smithi029.stdout:(40/160): libgfortran-11.4.1-3.el9.x86_64.rpm 634 kB/s | 803 kB 00:01 2024-07-27T06:31:16.499 INFO:teuthology.orchestra.run.smithi047.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 3.7 MB/s | 284 kB 00:00 2024-07-27T06:31:16.534 INFO:teuthology.orchestra.run.smithi077.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 8.0 MB/s | 548 kB 00:00 2024-07-27T06:31:16.557 INFO:teuthology.orchestra.run.smithi077.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 12 MB/s | 309 kB 00:00 2024-07-27T06:31:16.582 INFO:teuthology.orchestra.run.smithi077.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 776 kB/s | 19 kB 00:00 2024-07-27T06:31:16.592 INFO:teuthology.orchestra.run.smithi029.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 212 kB/s | 195 kB 00:00 2024-07-27T06:31:16.649 INFO:teuthology.orchestra.run.smithi047.stdout:(33/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 18 MB/s | 14 MB 00:00 2024-07-27T06:31:16.674 INFO:teuthology.orchestra.run.smithi047.stdout:(34/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 281 kB/s | 49 kB 00:00 2024-07-27T06:31:16.691 INFO:teuthology.orchestra.run.smithi077.stdout:(107/160): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-07-27T06:31:16.700 INFO:teuthology.orchestra.run.smithi047.stdout:(35/160): ceph-prometheus-alerts-19.0.0-5320.ge 328 kB/s | 16 kB 00:00 2024-07-27T06:31:16.716 INFO:teuthology.orchestra.run.smithi077.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-07-27T06:31:16.725 INFO:teuthology.orchestra.run.smithi047.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 5.3 MB/s | 274 kB 00:00 2024-07-27T06:31:16.742 INFO:teuthology.orchestra.run.smithi077.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-07-27T06:31:16.759 INFO:teuthology.orchestra.run.smithi047.stdout:(37/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 13 MB/s | 771 kB 00:00 2024-07-27T06:31:16.767 INFO:teuthology.orchestra.run.smithi077.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-07-27T06:31:16.784 INFO:teuthology.orchestra.run.smithi047.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 311 kB/s | 18 kB 00:00 2024-07-27T06:31:16.792 INFO:teuthology.orchestra.run.smithi077.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 6.0 MB/s | 151 kB 00:00 2024-07-27T06:31:16.826 INFO:teuthology.orchestra.run.smithi077.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 24 MB/s | 839 kB 00:00 2024-07-27T06:31:16.859 INFO:teuthology.orchestra.run.smithi077.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-07-27T06:31:16.927 INFO:teuthology.orchestra.run.smithi077.stdout:(114/160): python3-babel-2.9.1-2.el9.noarch.rpm 3.6 MB/s | 6.0 MB 00:01 2024-07-27T06:31:16.952 INFO:teuthology.orchestra.run.smithi077.stdout:(115/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 471 kB/s | 43 kB 00:00 2024-07-27T06:31:17.002 INFO:teuthology.orchestra.run.smithi077.stdout:(116/160): python3-certifi-2023.05.07-4.el9.noa 549 kB/s | 14 kB 00:00 2024-07-27T06:31:17.036 INFO:teuthology.orchestra.run.smithi077.stdout:(117/160): python3-cheroot-8.6.0-4.el9.noarch.r 5.4 MB/s | 172 kB 00:00 2024-07-27T06:31:17.061 INFO:teuthology.orchestra.run.smithi077.stdout:(118/160): python3-cachetools-4.2.4-1.el9.noarc 240 kB/s | 32 kB 00:00 2024-07-27T06:31:17.095 INFO:teuthology.orchestra.run.smithi077.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-07-27T06:31:17.120 INFO:teuthology.orchestra.run.smithi077.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.0 MB/s | 303 kB 00:00 2024-07-27T06:31:17.146 INFO:teuthology.orchestra.run.smithi077.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 4.2 MB/s | 216 kB 00:00 2024-07-27T06:31:17.168 INFO:teuthology.orchestra.run.smithi047.stdout:(39/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 15 MB/s | 44 MB 00:03 2024-07-27T06:31:17.171 INFO:teuthology.orchestra.run.smithi077.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 5.6 MB/s | 144 kB 00:00 2024-07-27T06:31:17.187 INFO:teuthology.orchestra.run.smithi077.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 648 kB/s | 11 kB 00:00 2024-07-27T06:31:17.213 INFO:teuthology.orchestra.run.smithi077.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 700 kB/s | 18 kB 00:00 2024-07-27T06:31:17.238 INFO:teuthology.orchestra.run.smithi077.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 916 kB/s | 23 kB 00:00 2024-07-27T06:31:17.264 INFO:teuthology.orchestra.run.smithi077.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 782 kB/s | 19 kB 00:00 2024-07-27T06:31:17.289 INFO:teuthology.orchestra.run.smithi077.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 780 kB/s | 20 kB 00:00 2024-07-27T06:31:17.318 INFO:teuthology.orchestra.run.smithi047.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 143 kB/s | 80 kB 00:00 2024-07-27T06:31:17.326 INFO:teuthology.orchestra.run.smithi029.stdout:(42/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 7.3 MB/s | 44 MB 00:06 2024-07-27T06:31:17.331 INFO:teuthology.orchestra.run.smithi077.stdout:(128/160): python3-grpcio-1.46.7-10.el9.x86_64. 9.7 MB/s | 2.0 MB 00:00 2024-07-27T06:31:17.357 INFO:teuthology.orchestra.run.smithi077.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-27T06:31:17.382 INFO:teuthology.orchestra.run.smithi077.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-07-27T06:31:17.407 INFO:teuthology.orchestra.run.smithi077.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-07-27T06:31:17.418 INFO:teuthology.orchestra.run.smithi047.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 782 kB/s | 195 kB 00:00 2024-07-27T06:31:17.441 INFO:teuthology.orchestra.run.smithi077.stdout:(132/160): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-07-27T06:31:17.466 INFO:teuthology.orchestra.run.smithi077.stdout:(133/160): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-07-27T06:31:17.485 INFO:teuthology.orchestra.run.smithi047.stdout:(42/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-07-27T06:31:17.499 INFO:teuthology.orchestra.run.smithi077.stdout:(134/160): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-07-27T06:31:17.525 INFO:teuthology.orchestra.run.smithi077.stdout:(135/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-07-27T06:31:17.527 INFO:teuthology.orchestra.run.smithi047.stdout:(43/160): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-07-27T06:31:17.553 INFO:teuthology.orchestra.run.smithi077.stdout:(136/160): python3-pyperclip-1.8.2-2.el9.noarch 558 kB/s | 28 kB 00:00 2024-07-27T06:31:17.576 INFO:teuthology.orchestra.run.smithi077.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-07-27T06:31:17.601 INFO:teuthology.orchestra.run.smithi077.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-07-27T06:31:17.626 INFO:teuthology.orchestra.run.smithi077.stdout:(139/160): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-07-27T06:31:17.652 INFO:teuthology.orchestra.run.smithi077.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 708 kB/s | 36 kB 00:00 2024-07-27T06:31:17.652 INFO:teuthology.orchestra.run.smithi047.stdout:(44/160): python3-cryptography-36.0.1-4.el9.x86 5.2 MB/s | 1.2 MB 00:00 2024-07-27T06:31:17.677 INFO:teuthology.orchestra.run.smithi077.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-07-27T06:31:17.677 INFO:teuthology.orchestra.run.smithi047.stdout:(45/160): python3-ply-3.11-14.el9.noarch.rpm 557 kB/s | 106 kB 00:00 2024-07-27T06:31:17.702 INFO:teuthology.orchestra.run.smithi077.stdout:(142/160): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-07-27T06:31:17.728 INFO:teuthology.orchestra.run.smithi077.stdout:(143/160): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-07-27T06:31:17.728 INFO:teuthology.orchestra.run.smithi047.stdout:(46/160): python3-pycparser-2.20-6.el9.noarch.r 774 kB/s | 135 kB 00:00 2024-07-27T06:31:17.753 INFO:teuthology.orchestra.run.smithi077.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 805 kB/s | 20 kB 00:00 2024-07-27T06:31:17.778 INFO:teuthology.orchestra.run.smithi077.stdout:(145/160): re2-20211101-3.el9.x86_64.rpm 7.5 MB/s | 192 kB 00:00 2024-07-27T06:31:17.794 INFO:teuthology.orchestra.run.smithi047.stdout:(47/160): python3-requests-2.25.1-8.el9.noarch. 879 kB/s | 125 kB 00:00 2024-07-27T06:31:17.803 INFO:teuthology.orchestra.run.smithi077.stdout:(146/160): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-07-27T06:31:17.845 INFO:teuthology.orchestra.run.smithi047.stdout:(48/160): python3-urllib3-1.26.5-6.el9.noarch.r 1.5 MB/s | 215 kB 00:00 2024-07-27T06:31:17.854 INFO:teuthology.orchestra.run.smithi077.stdout:(147/160): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-07-27T06:31:18.071 INFO:teuthology.orchestra.run.smithi029.stdout:(43/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 160 kB/s | 253 kB 00:01 2024-07-27T06:31:18.146 INFO:teuthology.orchestra.run.smithi029.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 130 kB/s | 106 kB 00:00 2024-07-27T06:31:18.187 INFO:teuthology.orchestra.run.smithi077.stdout:(148/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.1 MB/s | 16 MB 00:02 2024-07-27T06:31:18.246 INFO:teuthology.orchestra.run.smithi077.stdout:(149/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 8.0 MB/s | 3.1 MB 00:00 2024-07-27T06:31:18.296 INFO:teuthology.orchestra.run.smithi077.stdout:(150/160): librados2-19.0.0-5320.ge8f88bb1.el9. 6.8 MB/s | 3.4 MB 00:00 2024-07-27T06:31:18.329 INFO:teuthology.orchestra.run.smithi047.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 221 kB/s | 107 kB 00:00 2024-07-27T06:31:18.346 INFO:teuthology.orchestra.run.smithi029.stdout:(45/160): python3-cryptography-36.0.1-4.el9.x86 716 kB/s | 1.2 MB 00:01 2024-07-27T06:31:18.421 INFO:teuthology.orchestra.run.smithi077.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 326 kB/s | 41 kB 00:00 2024-07-27T06:31:18.437 INFO:teuthology.orchestra.run.smithi047.stdout:(50/160): elfutils-libelf-devel-0.191-4.el9.x86 200 kB/s | 22 kB 00:00 2024-07-27T06:31:18.504 INFO:teuthology.orchestra.run.smithi047.stdout:(51/160): automake-1.16.2-8.el9.noarch.rpm 919 kB/s | 689 kB 00:00 2024-07-27T06:31:18.571 INFO:teuthology.orchestra.run.smithi047.stdout:(52/160): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 943 kB 00:00 2024-07-27T06:31:18.621 INFO:teuthology.orchestra.run.smithi047.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 279 kB/s | 33 kB 00:00 2024-07-27T06:31:18.630 INFO:teuthology.orchestra.run.smithi077.stdout:(152/160): NetworkManager-tui-1.48.2-2.el9.x86_ 1.2 MB/s | 247 kB 00:00 2024-07-27T06:31:18.696 INFO:teuthology.orchestra.run.smithi047.stdout:(54/160): flex-2.6.4-9.el9.x86_64.rpm 1.2 MB/s | 313 kB 00:00 2024-07-27T06:31:18.722 INFO:teuthology.orchestra.run.smithi077.stdout:(153/160): NetworkManager-1.48.2-2.el9.x86_64.r 4.3 MB/s | 2.3 MB 00:00 2024-07-27T06:31:18.738 INFO:teuthology.orchestra.run.smithi047.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 155 kB/s | 18 kB 00:00 2024-07-27T06:31:18.747 INFO:teuthology.orchestra.run.smithi077.stdout:(154/160): elfutils-debuginfod-client-0.191-4.e 314 kB/s | 37 kB 00:00 2024-07-27T06:31:18.798 INFO:teuthology.orchestra.run.smithi077.stdout:(155/160): elfutils-libelf-0.191-4.el9.x86_64.r 2.7 MB/s | 207 kB 00:00 2024-07-27T06:31:18.822 INFO:teuthology.orchestra.run.smithi047.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 89 kB/s | 11 kB 00:00 2024-07-27T06:31:18.858 INFO:teuthology.orchestra.run.smithi047.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-07-27T06:31:18.905 INFO:teuthology.orchestra.run.smithi029.stdout:(46/160): python3-pycparser-2.20-6.el9.noarch.r 162 kB/s | 135 kB 00:00 2024-07-27T06:31:18.916 INFO:teuthology.orchestra.run.smithi047.stdout:(58/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.7 MB/s | 3.0 MB 00:00 2024-07-27T06:31:18.923 INFO:teuthology.orchestra.run.smithi077.stdout:(156/160): elfutils-libs-0.191-4.el9.x86_64.rpm 1.4 MB/s | 260 kB 00:00 2024-07-27T06:31:18.942 INFO:teuthology.orchestra.run.smithi047.stdout:(59/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 379 kB/s | 45 kB 00:00 2024-07-27T06:31:19.050 INFO:teuthology.orchestra.run.smithi047.stdout:(60/160): libtool-2.4.6-46.el9.x86_64.rpm 4.2 MB/s | 581 kB 00:00 2024-07-27T06:31:19.065 INFO:teuthology.orchestra.run.smithi077.stdout:(157/160): openssl-3.2.2-2.el9.x86_64.rpm 5.1 MB/s | 1.4 MB 00:00 2024-07-27T06:31:19.075 INFO:teuthology.orchestra.run.smithi047.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 190 kB/s | 25 kB 00:00 2024-07-27T06:31:19.081 INFO:teuthology.orchestra.run.smithi029.stdout:(47/160): python3-requests-2.25.1-8.el9.noarch. 134 kB/s | 125 kB 00:00 2024-07-27T06:31:19.101 INFO:teuthology.orchestra.run.smithi047.stdout:(62/160): librdkafka-1.6.1-102.el9.x86_64.rpm 2.7 MB/s | 662 kB 00:00 2024-07-27T06:31:19.168 INFO:teuthology.orchestra.run.smithi047.stdout:(63/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 405 kB/s | 47 kB 00:00 2024-07-27T06:31:19.194 INFO:teuthology.orchestra.run.smithi047.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-07-27T06:31:19.218 INFO:teuthology.orchestra.run.smithi047.stdout:(65/160): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-07-27T06:31:19.241 INFO:teuthology.orchestra.run.smithi077.stdout:(158/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 6.2 MB/s | 1.1 MB 00:00 2024-07-27T06:31:19.256 INFO:teuthology.orchestra.run.smithi029.stdout:(48/160): python3-urllib3-1.26.5-6.el9.noarch.r 244 kB/s | 215 kB 00:00 2024-07-27T06:31:19.277 INFO:teuthology.orchestra.run.smithi047.stdout:(66/160): openblas-0.3.21-2.el9.x86_64.rpm 324 kB/s | 35 kB 00:00 2024-07-27T06:31:19.419 INFO:teuthology.orchestra.run.smithi047.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-07-27T06:31:19.424 INFO:teuthology.orchestra.run.smithi077.stdout:(159/160): NetworkManager-libnm-1.48.2-2.el9.x8 1.5 MB/s | 1.8 MB 00:01 2024-07-27T06:31:19.561 INFO:teuthology.orchestra.run.smithi047.stdout:(68/160): postgresql-contrib-13.14-1.el9.x86_64 6.3 MB/s | 911 kB 00:00 2024-07-27T06:31:19.711 INFO:teuthology.orchestra.run.smithi047.stdout:(69/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 9.0 MB/s | 4.4 MB 00:00 2024-07-27T06:31:19.736 INFO:teuthology.orchestra.run.smithi047.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 760 kB/s | 133 kB 00:00 2024-07-27T06:31:19.790 INFO:teuthology.orchestra.run.smithi029.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 200 kB/s | 107 kB 00:00 2024-07-27T06:31:19.803 INFO:teuthology.orchestra.run.smithi047.stdout:(71/160): openblas-openmp-0.3.21-2.el9.x86_64.r 7.8 MB/s | 4.7 MB 00:00 2024-07-27T06:31:19.807 INFO:teuthology.orchestra.run.smithi077.stdout:(160/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 2.7 MB/s | 2.4 MB 00:00 2024-07-27T06:31:19.812 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-07-27T06:31:19.813 INFO:teuthology.orchestra.run.smithi077.stdout:Total 19 MB/s | 233 MB 00:12 2024-07-27T06:31:19.824 INFO:teuthology.orchestra.run.smithi029.stdout:(50/160): automake-1.16.2-8.el9.noarch.rpm 750 kB/s | 689 kB 00:00 2024-07-27T06:31:19.916 INFO:teuthology.orchestra.run.smithi029.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 173 kB/s | 22 kB 00:00 2024-07-27T06:31:19.929 INFO:teuthology.orchestra.run.smithi047.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 700 kB/s | 87 kB 00:00 2024-07-27T06:31:20.017 INFO:teuthology.orchestra.run.smithi029.stdout:(52/160): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 943 kB 00:00 2024-07-27T06:31:20.042 INFO:teuthology.orchestra.run.smithi029.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 259 kB/s | 33 kB 00:00 2024-07-27T06:31:20.068 INFO:teuthology.orchestra.run.smithi029.stdout:(54/160): flex-2.6.4-9.el9.x86_64.rpm 1.3 MB/s | 313 kB 00:00 2024-07-27T06:31:20.070 INFO:teuthology.orchestra.run.smithi047.stdout:(73/160): postgresql-server-13.14-1.el9.x86_64. 16 MB/s | 5.8 MB 00:00 2024-07-27T06:31:20.160 INFO:teuthology.orchestra.run.smithi029.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 155 kB/s | 18 kB 00:00 2024-07-27T06:31:20.179 INFO:teuthology.orchestra.run.smithi047.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 567 kB/s | 61 kB 00:00 2024-07-27T06:31:20.186 INFO:teuthology.orchestra.run.smithi029.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 11 kB 00:00 2024-07-27T06:31:20.254 INFO:teuthology.orchestra.run.smithi047.stdout:(75/160): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-07-27T06:31:20.278 INFO:teuthology.orchestra.run.smithi029.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-07-27T06:31:20.288 INFO:teuthology.orchestra.run.smithi047.stdout:(76/160): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-07-27T06:31:20.314 INFO:teuthology.orchestra.run.smithi029.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 355 kB/s | 45 kB 00:00 2024-07-27T06:31:20.363 INFO:teuthology.orchestra.run.smithi047.stdout:(77/160): python3-jinja2-2.11.3-6.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-07-27T06:31:20.388 INFO:teuthology.orchestra.run.smithi047.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 255 kB/s | 26 kB 00:00 2024-07-27T06:31:20.397 INFO:teuthology.orchestra.run.smithi029.stdout:(59/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.9 MB/s | 3.0 MB 00:00 2024-07-27T06:31:20.464 INFO:teuthology.orchestra.run.smithi047.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 191 kB/s | 19 kB 00:00 2024-07-27T06:31:20.497 INFO:teuthology.orchestra.run.smithi047.stdout:(80/160): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-07-27T06:31:20.515 INFO:teuthology.orchestra.run.smithi029.stdout:(60/160): libtool-2.4.6-46.el9.x86_64.rpm 2.8 MB/s | 581 kB 00:00 2024-07-27T06:31:20.540 INFO:teuthology.orchestra.run.smithi029.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 179 kB/s | 25 kB 00:00 2024-07-27T06:31:20.564 INFO:teuthology.orchestra.run.smithi047.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 346 kB/s | 35 kB 00:00 2024-07-27T06:31:20.574 INFO:teuthology.orchestra.run.smithi029.stdout:(62/160): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-07-27T06:31:20.650 INFO:teuthology.orchestra.run.smithi029.stdout:(63/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 352 kB/s | 47 kB 00:00 2024-07-27T06:31:20.673 INFO:teuthology.orchestra.run.smithi047.stdout:(82/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-07-27T06:31:20.683 INFO:teuthology.orchestra.run.smithi029.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-07-27T06:31:20.709 INFO:teuthology.orchestra.run.smithi029.stdout:(65/160): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-07-27T06:31:20.756 INFO:teuthology.orchestra.run.smithi047.stdout:(83/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-07-27T06:31:20.759 INFO:teuthology.orchestra.run.smithi029.stdout:(66/160): openblas-0.3.21-2.el9.x86_64.rpm 321 kB/s | 35 kB 00:00 2024-07-27T06:31:20.782 INFO:teuthology.orchestra.run.smithi047.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-07-27T06:31:20.865 INFO:teuthology.orchestra.run.smithi047.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 710 kB/s | 77 kB 00:00 2024-07-27T06:31:20.899 INFO:teuthology.orchestra.run.smithi047.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 362 kB/s | 42 kB 00:00 2024-07-27T06:31:20.983 INFO:teuthology.orchestra.run.smithi047.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 2.2 MB/s | 269 kB 00:00 2024-07-27T06:31:21.033 INFO:teuthology.orchestra.run.smithi047.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-07-27T06:31:21.045 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-07-27T06:31:21.060 INFO:teuthology.orchestra.run.smithi029.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-07-27T06:31:21.100 INFO:teuthology.orchestra.run.smithi047.stdout:(89/160): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-07-27T06:31:21.150 INFO:teuthology.orchestra.run.smithi047.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 436 kB/s | 51 kB 00:00 2024-07-27T06:31:21.161 INFO:teuthology.orchestra.run.smithi029.stdout:(68/160): openblas-openmp-0.3.21-2.el9.x86_64.r 9.9 MB/s | 4.7 MB 00:00 2024-07-27T06:31:21.209 INFO:teuthology.orchestra.run.smithi047.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 494 kB/s | 54 kB 00:00 2024-07-27T06:31:21.239 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-07-27T06:31:21.239 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-07-27T06:31:21.253 INFO:teuthology.orchestra.run.smithi029.stdout:(69/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 8.2 MB/s | 4.4 MB 00:00 2024-07-27T06:31:21.295 INFO:teuthology.orchestra.run.smithi029.stdout:(70/160): postgresql-contrib-13.14-1.el9.x86_64 3.8 MB/s | 911 kB 00:00 2024-07-27T06:31:21.321 INFO:teuthology.orchestra.run.smithi029.stdout:(71/160): postgresql-private-libs-13.14-1.el9.x 835 kB/s | 133 kB 00:00 2024-07-27T06:31:21.326 INFO:teuthology.orchestra.run.smithi047.stdout:(92/160): protobuf-3.14.0-13.el9.x86_64.rpm 650 kB/s | 1.0 MB 00:01 2024-07-27T06:31:21.352 INFO:teuthology.orchestra.run.smithi047.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 293 kB/s | 42 kB 00:00 2024-07-27T06:31:21.435 INFO:teuthology.orchestra.run.smithi047.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 408 kB/s | 44 kB 00:00 2024-07-27T06:31:21.446 INFO:teuthology.orchestra.run.smithi029.stdout:(72/160): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-07-27T06:31:21.460 INFO:teuthology.orchestra.run.smithi047.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-07-27T06:31:21.474 INFO:teuthology.orchestra.run.smithi029.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 573 kB/s | 87 kB 00:00 2024-07-27T06:31:21.536 INFO:teuthology.orchestra.run.smithi047.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 358 kB/s | 36 kB 00:00 2024-07-27T06:31:21.585 INFO:teuthology.orchestra.run.smithi047.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 378 kB/s | 47 kB 00:00 2024-07-27T06:31:21.624 INFO:teuthology.orchestra.run.smithi029.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 407 kB/s | 61 kB 00:00 2024-07-27T06:31:21.661 INFO:teuthology.orchestra.run.smithi047.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-07-27T06:31:21.703 INFO:teuthology.orchestra.run.smithi047.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 494 kB/s | 58 kB 00:00 2024-07-27T06:31:21.741 INFO:teuthology.orchestra.run.smithi029.stdout:(75/160): python3-devel-3.9.18-3.el9.x86_64.rpm 2.0 MB/s | 244 kB 00:00 2024-07-27T06:31:21.858 INFO:teuthology.orchestra.run.smithi029.stdout:(76/160): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-07-27T06:31:21.862 INFO:teuthology.orchestra.run.smithi047.stdout:(100/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-07-27T06:31:21.887 INFO:teuthology.orchestra.run.smithi047.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 346 kB/s | 64 kB 00:00 2024-07-27T06:31:21.892 INFO:teuthology.orchestra.run.smithi029.stdout:(77/160): python3-jinja2-2.11.3-6.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-07-27T06:31:21.903 INFO:teuthology.orchestra.run.smithi047.stdout:(102/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 394 kB/s | 96 kB 00:00 2024-07-27T06:31:21.970 INFO:teuthology.orchestra.run.smithi047.stdout:(103/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 413 kB/s | 45 kB 00:00 2024-07-27T06:31:22.001 INFO:teuthology.orchestra.run.smithi029.stdout:(78/160): postgresql-server-13.14-1.el9.x86_64. 7.7 MB/s | 5.8 MB 00:00 2024-07-27T06:31:22.027 INFO:teuthology.orchestra.run.smithi029.stdout:(79/160): python3-jsonpatch-1.21-16.el9.noarch. 152 kB/s | 26 kB 00:00 2024-07-27T06:31:22.029 INFO:teuthology.orchestra.run.smithi047.stdout:(104/160): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-07-27T06:31:22.052 INFO:teuthology.orchestra.run.smithi029.stdout:(80/160): python3-jsonpointer-2.0-4.el9.noarch. 120 kB/s | 19 kB 00:00 2024-07-27T06:31:22.088 INFO:teuthology.orchestra.run.smithi047.stdout:(105/160): abseil-cpp-20211102.0-3.el9.x86_64.r 4.6 MB/s | 548 kB 00:00 2024-07-27T06:31:22.113 INFO:teuthology.orchestra.run.smithi047.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 865 kB/s | 19 kB 00:00 2024-07-27T06:31:22.119 INFO:teuthology.orchestra.run.smithi029.stdout:(81/160): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-07-27T06:31:22.146 INFO:teuthology.orchestra.run.smithi047.stdout:(107/160): gperftools-libs-2.9.1-2.el9.x86_64.r 2.6 MB/s | 309 kB 00:00 2024-07-27T06:31:22.172 INFO:teuthology.orchestra.run.smithi047.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-07-27T06:31:22.178 INFO:teuthology.orchestra.run.smithi029.stdout:(82/160): python3-markupsafe-1.1.1-12.el9.x86_6 276 kB/s | 35 kB 00:00 2024-07-27T06:31:22.197 INFO:teuthology.orchestra.run.smithi047.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-07-27T06:31:22.222 INFO:teuthology.orchestra.run.smithi047.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-07-27T06:31:22.245 INFO:teuthology.orchestra.run.smithi029.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.9 MB/s | 250 kB 00:00 2024-07-27T06:31:22.248 INFO:teuthology.orchestra.run.smithi047.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 5.9 MB/s | 151 kB 00:00 2024-07-27T06:31:22.298 INFO:teuthology.orchestra.run.smithi047.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 16 MB/s | 839 kB 00:00 2024-07-27T06:31:22.313 INFO:teuthology.orchestra.run.smithi029.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-07-27T06:31:22.340 INFO:teuthology.orchestra.run.smithi047.stdout:(113/160): protobuf-compiler-3.14.0-13.el9.x86_ 1.9 MB/s | 863 kB 00:00 2024-07-27T06:31:22.398 INFO:teuthology.orchestra.run.smithi047.stdout:(114/160): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-07-27T06:31:22.413 INFO:teuthology.orchestra.run.smithi029.stdout:(85/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-07-27T06:31:22.424 INFO:teuthology.orchestra.run.smithi047.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.4 MB/s | 32 kB 00:00 2024-07-27T06:31:22.449 INFO:teuthology.orchestra.run.smithi047.stdout:(116/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 399 kB/s | 43 kB 00:00 2024-07-27T06:31:22.483 INFO:teuthology.orchestra.run.smithi047.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 239 kB/s | 14 kB 00:00 2024-07-27T06:31:22.500 INFO:teuthology.orchestra.run.smithi047.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-07-27T06:31:22.539 INFO:teuthology.orchestra.run.smithi029.stdout:(86/160): python3-packaging-20.9-5.el9.noarch.r 263 kB/s | 77 kB 00:00 2024-07-27T06:31:22.566 INFO:teuthology.orchestra.run.smithi029.stdout:(87/160): python3-prettytable-0.7.2-27.el9.noar 167 kB/s | 42 kB 00:00 2024-07-27T06:31:22.592 INFO:teuthology.orchestra.run.smithi047.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-07-27T06:31:22.600 INFO:teuthology.orchestra.run.smithi029.stdout:(88/160): python3-protobuf-3.14.0-13.el9.noarch 1.4 MB/s | 269 kB 00:00 2024-07-27T06:31:22.617 INFO:teuthology.orchestra.run.smithi047.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-07-27T06:31:22.651 INFO:teuthology.orchestra.run.smithi047.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 3.6 MB/s | 216 kB 00:00 2024-07-27T06:31:22.676 INFO:teuthology.orchestra.run.smithi047.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 5.7 MB/s | 144 kB 00:00 2024-07-27T06:31:22.692 INFO:teuthology.orchestra.run.smithi029.stdout:(89/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-07-27T06:31:22.701 INFO:teuthology.orchestra.run.smithi047.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 430 kB/s | 11 kB 00:00 2024-07-27T06:31:22.726 INFO:teuthology.orchestra.run.smithi029.stdout:(90/160): python3-pyasn1-modules-0.4.8-6.el9.no 1.7 MB/s | 279 kB 00:00 2024-07-27T06:31:22.727 INFO:teuthology.orchestra.run.smithi047.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 705 kB/s | 18 kB 00:00 2024-07-27T06:31:22.752 INFO:teuthology.orchestra.run.smithi047.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 932 kB/s | 23 kB 00:00 2024-07-27T06:31:22.760 INFO:teuthology.orchestra.run.smithi029.stdout:(91/160): python3-pytz-2021.1-5.el9.noarch.rpm 319 kB/s | 51 kB 00:00 2024-07-27T06:31:22.777 INFO:teuthology.orchestra.run.smithi047.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 782 kB/s | 19 kB 00:00 2024-07-27T06:31:22.803 INFO:teuthology.orchestra.run.smithi047.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 790 kB/s | 20 kB 00:00 2024-07-27T06:31:22.836 INFO:teuthology.orchestra.run.smithi029.stdout:(92/160): python3-requests-oauthlib-1.3.0-12.el 375 kB/s | 54 kB 00:00 2024-07-27T06:31:22.878 INFO:teuthology.orchestra.run.smithi047.stdout:(128/160): libarrow-9.0.0-11.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2024-07-27T06:31:22.903 INFO:teuthology.orchestra.run.smithi047.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-27T06:31:22.919 INFO:teuthology.orchestra.run.smithi029.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 262 kB/s | 42 kB 00:00 2024-07-27T06:31:22.929 INFO:teuthology.orchestra.run.smithi047.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-07-27T06:31:22.945 INFO:teuthology.orchestra.run.smithi029.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 404 kB/s | 44 kB 00:00 2024-07-27T06:31:22.954 INFO:teuthology.orchestra.run.smithi047.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-07-27T06:31:22.996 INFO:teuthology.orchestra.run.smithi047.stdout:(132/160): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-07-27T06:31:23.021 INFO:teuthology.orchestra.run.smithi047.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 658 kB/s | 16 kB 00:00 2024-07-27T06:31:23.054 INFO:teuthology.orchestra.run.smithi029.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-07-27T06:31:23.055 INFO:teuthology.orchestra.run.smithi047.stdout:(134/160): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-07-27T06:31:23.079 INFO:teuthology.orchestra.run.smithi029.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 267 kB/s | 36 kB 00:00 2024-07-27T06:31:23.080 INFO:teuthology.orchestra.run.smithi047.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-07-27T06:31:23.105 INFO:teuthology.orchestra.run.smithi047.stdout:(136/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-07-27T06:31:23.107 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-07-27T06:31:23.107 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-07-27T06:31:23.130 INFO:teuthology.orchestra.run.smithi047.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-07-27T06:31:23.156 INFO:teuthology.orchestra.run.smithi047.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-07-27T06:31:23.197 INFO:teuthology.orchestra.run.smithi029.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 333 kB/s | 47 kB 00:00 2024-07-27T06:31:23.198 INFO:teuthology.orchestra.run.smithi047.stdout:(139/160): python3-grpcio-1.46.7-10.el9.x86_64. 3.5 MB/s | 2.0 MB 00:00 2024-07-27T06:31:23.223 INFO:teuthology.orchestra.run.smithi047.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 534 kB/s | 36 kB 00:00 2024-07-27T06:31:23.248 INFO:teuthology.orchestra.run.smithi047.stdout:(141/160): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-07-27T06:31:23.273 INFO:teuthology.orchestra.run.smithi047.stdout:(142/160): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-07-27T06:31:23.299 INFO:teuthology.orchestra.run.smithi047.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-07-27T06:31:23.323 INFO:teuthology.orchestra.run.smithi029.stdout:(98/160): uuid-1.6.2-55.el9.x86_64.rpm 462 kB/s | 58 kB 00:00 2024-07-27T06:31:23.324 INFO:teuthology.orchestra.run.smithi047.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 799 kB/s | 20 kB 00:00 2024-07-27T06:31:23.350 INFO:teuthology.orchestra.run.smithi047.stdout:(145/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-07-27T06:31:23.375 INFO:teuthology.orchestra.run.smithi047.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-07-27T06:31:23.400 INFO:teuthology.orchestra.run.smithi047.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-07-27T06:31:23.456 INFO:teuthology.orchestra.run.smithi029.stdout:(99/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rpm 716 kB/s | 96 kB 00:00 2024-07-27T06:31:23.467 INFO:teuthology.orchestra.run.smithi047.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-07-27T06:31:23.573 INFO:teuthology.orchestra.run.smithi029.stdout:(100/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 544 kB/s | 64 kB 00:00 2024-07-27T06:31:23.676 INFO:teuthology.orchestra.run.smithi047.stdout:(149/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 11 MB/s | 3.1 MB 00:00 2024-07-27T06:31:23.699 INFO:teuthology.orchestra.run.smithi029.stdout:(101/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 357 kB/s | 45 kB 00:00 2024-07-27T06:31:23.726 INFO:teuthology.orchestra.run.smithi047.stdout:(150/160): librados2-19.0.0-5320.ge8f88bb1.el9. 9.6 MB/s | 3.4 MB 00:00 2024-07-27T06:31:23.793 INFO:teuthology.orchestra.run.smithi047.stdout:(151/160): NetworkManager-1.48.2-2.el9.x86_64.r 7.1 MB/s | 2.3 MB 00:00 2024-07-27T06:31:23.910 INFO:teuthology.orchestra.run.smithi047.stdout:(152/160): NetworkManager-team-1.48.2-2.el9.x86 259 kB/s | 41 kB 00:00 2024-07-27T06:31:23.952 INFO:teuthology.orchestra.run.smithi047.stdout:(153/160): NetworkManager-libnm-1.48.2-2.el9.x8 6.5 MB/s | 1.8 MB 00:00 2024-07-27T06:31:24.002 INFO:teuthology.orchestra.run.smithi047.stdout:(154/160): NetworkManager-tui-1.48.2-2.el9.x86_ 1.3 MB/s | 247 kB 00:00 2024-07-27T06:31:24.018 INFO:teuthology.orchestra.run.smithi029.stdout:(102/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-07-27T06:31:24.094 INFO:teuthology.orchestra.run.smithi047.stdout:(155/160): elfutils-debuginfod-client-0.191-4.e 200 kB/s | 37 kB 00:00 2024-07-27T06:31:24.194 INFO:teuthology.orchestra.run.smithi047.stdout:(156/160): elfutils-libelf-0.191-4.el9.x86_64.r 955 kB/s | 207 kB 00:00 2024-07-27T06:31:24.219 INFO:teuthology.orchestra.run.smithi047.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 1.3 MB/s | 260 kB 00:00 2024-07-27T06:31:24.243 INFO:teuthology.orchestra.run.smithi029.stdout:(103/160): socat-1.7.4.1-6.el9.x86_64.rpm 261 kB/s | 304 kB 00:01 2024-07-27T06:31:24.327 INFO:teuthology.orchestra.run.smithi029.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 6.4 MB/s | 548 kB 00:00 2024-07-27T06:31:24.345 INFO:teuthology.orchestra.run.smithi047.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 5.4 MB/s | 1.4 MB 00:00 2024-07-27T06:31:24.361 INFO:teuthology.orchestra.run.smithi029.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 9.0 MB/s | 309 kB 00:00 2024-07-27T06:31:24.387 INFO:teuthology.orchestra.run.smithi029.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 770 kB/s | 19 kB 00:00 2024-07-27T06:31:24.412 INFO:teuthology.orchestra.run.smithi029.stdout:(107/160): lua-devel-5.4.4-4.el9.x86_64.rpm 31 kB/s | 22 kB 00:00 2024-07-27T06:31:24.470 INFO:teuthology.orchestra.run.smithi047.stdout:(159/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 4.8 MB/s | 1.1 MB 00:00 2024-07-27T06:31:24.479 INFO:teuthology.orchestra.run.smithi029.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 387 kB/s | 26 kB 00:00 2024-07-27T06:31:24.513 INFO:teuthology.orchestra.run.smithi029.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-07-27T06:31:24.539 INFO:teuthology.orchestra.run.smithi029.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-07-27T06:31:24.570 INFO:teuthology.orchestra.run.smithi047.stdout:(160/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 6.4 MB/s | 2.4 MB 00:00 2024-07-27T06:31:24.573 INFO:teuthology.orchestra.run.smithi029.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-07-27T06:31:24.575 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-07-27T06:31:24.575 INFO:teuthology.orchestra.run.smithi047.stdout:Total 19 MB/s | 233 MB 00:12 2024-07-27T06:31:24.682 INFO:teuthology.orchestra.run.smithi029.stdout:(112/160): libarrow-9.0.0-11.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-07-27T06:31:24.724 INFO:teuthology.orchestra.run.smithi029.stdout:(113/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.4 MB/s | 839 kB 00:00 2024-07-27T06:31:24.758 INFO:teuthology.orchestra.run.smithi029.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-07-27T06:31:24.792 INFO:teuthology.orchestra.run.smithi029.stdout:(115/160): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-07-27T06:31:24.826 INFO:teuthology.orchestra.run.smithi029.stdout:(116/160): python3-cachetools-4.2.4-1.el9.noarc 476 kB/s | 32 kB 00:00 2024-07-27T06:31:24.853 INFO:teuthology.orchestra.run.smithi029.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 232 kB/s | 14 kB 00:00 2024-07-27T06:31:24.886 INFO:teuthology.orchestra.run.smithi029.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 2.8 MB/s | 172 kB 00:00 2024-07-27T06:31:24.920 INFO:teuthology.orchestra.run.smithi029.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-07-27T06:31:24.954 INFO:teuthology.orchestra.run.smithi029.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-07-27T06:31:25.005 INFO:teuthology.orchestra.run.smithi029.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 2.5 MB/s | 216 kB 00:00 2024-07-27T06:31:25.039 INFO:teuthology.orchestra.run.smithi029.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-07-27T06:31:25.073 INFO:teuthology.orchestra.run.smithi029.stdout:(123/160): protobuf-compiler-3.14.0-13.el9.x86_ 842 kB/s | 863 kB 00:01 2024-07-27T06:31:25.099 INFO:teuthology.orchestra.run.smithi029.stdout:(124/160): python3-jaraco-8.2.1-3.el9.noarch.rp 181 kB/s | 11 kB 00:00 2024-07-27T06:31:25.158 INFO:teuthology.orchestra.run.smithi029.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 792 kB/s | 23 kB 00:00 2024-07-27T06:31:25.191 INFO:teuthology.orchestra.run.smithi029.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 628 kB/s | 19 kB 00:00 2024-07-27T06:31:25.217 INFO:teuthology.orchestra.run.smithi029.stdout:(127/160): python3-jaraco-classes-3.2.1-5.el9.n 123 kB/s | 18 kB 00:00 2024-07-27T06:31:25.243 INFO:teuthology.orchestra.run.smithi029.stdout:(128/160): python3-jaraco-text-3.2.0-6.el9.noar 386 kB/s | 20 kB 00:00 2024-07-27T06:31:25.277 INFO:teuthology.orchestra.run.smithi029.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.3 MB/s | 46 kB 00:00 2024-07-27T06:31:25.302 INFO:teuthology.orchestra.run.smithi029.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-07-27T06:31:25.338 INFO:teuthology.orchestra.run.smithi029.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 1.6 MB/s | 58 kB 00:00 2024-07-27T06:31:25.372 INFO:teuthology.orchestra.run.smithi029.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 7.9 MB/s | 272 kB 00:00 2024-07-27T06:31:25.397 INFO:teuthology.orchestra.run.smithi029.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 649 kB/s | 16 kB 00:00 2024-07-27T06:31:25.423 INFO:teuthology.orchestra.run.smithi029.stdout:(134/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-07-27T06:31:25.448 INFO:teuthology.orchestra.run.smithi029.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-07-27T06:31:25.474 INFO:teuthology.orchestra.run.smithi029.stdout:(136/160): python3-repoze-lru-0.7-16.el9.noarch 1.2 MB/s | 31 kB 00:00 2024-07-27T06:31:25.516 INFO:teuthology.orchestra.run.smithi029.stdout:(137/160): python3-kubernetes-26.1.0-2.el9.noar 3.4 MB/s | 1.0 MB 00:00 2024-07-27T06:31:25.575 INFO:teuthology.orchestra.run.smithi029.stdout:(138/160): python3-grpcio-1.46.7-10.el9.x86_64. 3.3 MB/s | 2.0 MB 00:00 2024-07-27T06:31:25.600 INFO:teuthology.orchestra.run.smithi029.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 703 kB/s | 59 kB 00:00 2024-07-27T06:31:25.626 INFO:teuthology.orchestra.run.smithi029.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 705 kB/s | 36 kB 00:00 2024-07-27T06:31:25.652 INFO:teuthology.orchestra.run.smithi029.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-07-27T06:31:25.677 INFO:teuthology.orchestra.run.smithi029.stdout:(142/160): python3-routes-2.5.1-5.el9.noarch.rp 928 kB/s | 188 kB 00:00 2024-07-27T06:31:25.703 INFO:teuthology.orchestra.run.smithi029.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-07-27T06:31:25.728 INFO:teuthology.orchestra.run.smithi029.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 793 kB/s | 20 kB 00:00 2024-07-27T06:31:25.764 INFO:teuthology.orchestra.run.smithi029.stdout:(145/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-07-27T06:31:25.797 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-07-27T06:31:25.798 INFO:teuthology.orchestra.run.smithi029.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 2.7 MB/s | 192 kB 00:00 2024-07-27T06:31:25.832 INFO:teuthology.orchestra.run.smithi029.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 2.7 MB/s | 427 kB 00:00 2024-07-27T06:31:25.899 INFO:teuthology.orchestra.run.smithi029.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-07-27T06:31:26.011 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-07-27T06:31:26.011 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-07-27T06:31:26.116 INFO:teuthology.orchestra.run.smithi029.stdout:(149/160): librados2-19.0.0-5320.ge8f88bb1.el9. 11 MB/s | 3.4 MB 00:00 2024-07-27T06:31:26.234 INFO:teuthology.orchestra.run.smithi029.stdout:(150/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 7.8 MB/s | 3.1 MB 00:00 2024-07-27T06:31:26.951 INFO:teuthology.orchestra.run.smithi029.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 59 kB/s | 41 kB 00:00 2024-07-27T06:31:27.226 INFO:teuthology.orchestra.run.smithi029.stdout:(152/160): NetworkManager-1.48.2-2.el9.x86_64.r 1.7 MB/s | 2.3 MB 00:01 2024-07-27T06:31:27.352 INFO:teuthology.orchestra.run.smithi029.stdout:(153/160): NetworkManager-libnm-1.48.2-2.el9.x8 1.5 MB/s | 1.8 MB 00:01 2024-07-27T06:31:27.986 INFO:teuthology.orchestra.run.smithi029.stdout:(154/160): NetworkManager-tui-1.48.2-2.el9.x86_ 239 kB/s | 247 kB 00:01 2024-07-27T06:31:27.999 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-07-27T06:31:27.999 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-07-27T06:31:28.020 INFO:teuthology.orchestra.run.smithi029.stdout:(155/160): elfutils-debuginfod-client-0.191-4.e 48 kB/s | 37 kB 00:00 2024-07-27T06:31:28.287 INFO:teuthology.orchestra.run.smithi029.stdout:(156/160): elfutils-libelf-0.191-4.el9.x86_64.r 222 kB/s | 207 kB 00:00 2024-07-27T06:31:28.879 INFO:teuthology.orchestra.run.smithi029.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 291 kB/s | 260 kB 00:00 2024-07-27T06:31:29.213 INFO:teuthology.orchestra.run.smithi029.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 1.1 MB/s | 1.4 MB 00:01 2024-07-27T06:31:29.409 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-07-27T06:31:29.504 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-27T06:31:29.521 INFO:teuthology.orchestra.run.smithi029.stdout:(159/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 2.0 MB/s | 2.4 MB 00:01 2024-07-27T06:31:29.542 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-27T06:31:29.604 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-27T06:31:29.638 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-27T06:31:29.663 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-27T06:31:29.697 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-27T06:31:29.965 INFO:teuthology.orchestra.run.smithi029.stdout:(160/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 1.0 MB/s | 1.1 MB 00:01 2024-07-27T06:31:29.974 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-07-27T06:31:29.974 INFO:teuthology.orchestra.run.smithi029.stdout:Total 11 MB/s | 233 MB 00:21 2024-07-27T06:31:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-27T06:31:30.165 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-27T06:31:30.291 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-27T06:31:30.316 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-27T06:31:30.411 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-27T06:31:30.437 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-27T06:31:30.469 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-27T06:31:30.669 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-27T06:31:30.720 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-27T06:31:30.748 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-27T06:31:30.854 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-27T06:31:30.883 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-27T06:31:30.945 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-27T06:31:30.967 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-27T06:31:30.991 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-27T06:31:31.183 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-27T06:31:31.247 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-07-27T06:31:31.276 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-27T06:31:31.434 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-27T06:31:31.476 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-07-27T06:31:31.476 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-07-27T06:31:31.625 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-27T06:31:31.685 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-27T06:31:31.792 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-27T06:31:31.869 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-27T06:31:32.074 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-27T06:31:32.187 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-27T06:31:32.215 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-27T06:31:32.250 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-27T06:31:32.330 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-27T06:31:32.377 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-27T06:31:32.418 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-27T06:31:32.453 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-27T06:31:32.486 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-27T06:31:32.514 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-27T06:31:32.582 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-27T06:31:32.636 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-27T06:31:32.668 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-27T06:31:32.711 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-27T06:31:32.790 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-27T06:31:32.823 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-27T06:31:32.868 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-27T06:31:32.895 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-27T06:31:32.992 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-27T06:31:33.043 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-27T06:31:33.117 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-27T06:31:33.250 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-27T06:31:33.303 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-27T06:31:33.825 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-27T06:31:33.907 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-27T06:31:33.945 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-27T06:31:33.990 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-27T06:31:34.030 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-27T06:31:34.068 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-27T06:31:34.110 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-27T06:31:34.263 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-27T06:31:34.284 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-07-27T06:31:34.284 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-07-27T06:31:34.321 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-27T06:31:34.395 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-07-27T06:31:34.411 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-27T06:31:34.474 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-27T06:31:34.476 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-27T06:31:34.513 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-27T06:31:34.549 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-27T06:31:34.567 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-27T06:31:34.581 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-27T06:31:34.609 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-27T06:31:34.617 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-27T06:31:34.634 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-27T06:31:34.664 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-27T06:31:34.678 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-27T06:31:34.708 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-27T06:31:34.751 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-27T06:31:34.793 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-27T06:31:34.929 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-27T06:31:35.035 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-27T06:31:35.143 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-27T06:31:35.205 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-27T06:31:35.263 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-27T06:31:35.272 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-27T06:31:35.302 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-27T06:31:35.363 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-27T06:31:35.399 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-27T06:31:35.412 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-27T06:31:35.432 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-27T06:31:35.444 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-27T06:31:35.465 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-27T06:31:35.469 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-27T06:31:35.494 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-27T06:31:35.568 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-27T06:31:35.664 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-27T06:31:35.694 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-27T06:31:35.730 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-27T06:31:35.742 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-27T06:31:35.750 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-27T06:31:35.770 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-27T06:31:35.805 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-27T06:31:35.844 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-27T06:31:35.857 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-27T06:31:35.876 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-27T06:31:35.884 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-27T06:31:35.905 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-27T06:31:35.940 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-27T06:31:35.946 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-27T06:31:35.960 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-27T06:31:35.973 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-27T06:31:35.980 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-27T06:31:36.006 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-27T06:31:36.040 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-27T06:31:36.127 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-27T06:31:36.169 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-27T06:31:36.170 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-27T06:31:36.202 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-27T06:31:36.250 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-27T06:31:36.304 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-27T06:31:36.353 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-27T06:31:36.420 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-27T06:31:36.618 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-27T06:31:36.678 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-27T06:31:36.773 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-27T06:31:36.871 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-27T06:31:37.073 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-27T06:31:37.079 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-27T06:31:37.133 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-27T06:31:37.167 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-27T06:31:37.189 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-27T06:31:37.216 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-27T06:31:37.252 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-27T06:31:37.265 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-27T06:31:37.322 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-27T06:31:37.335 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-27T06:31:37.367 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-27T06:31:37.387 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-27T06:31:37.427 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-27T06:31:37.454 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-27T06:31:37.471 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-27T06:31:37.486 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-27T06:31:37.503 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-27T06:31:37.530 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-27T06:31:37.603 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-27T06:31:37.654 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-27T06:31:37.663 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-27T06:31:37.683 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-27T06:31:37.692 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-27T06:31:37.728 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-27T06:31:37.770 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-27T06:31:37.806 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-27T06:31:37.838 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-27T06:31:37.910 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-27T06:31:38.034 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-27T06:31:38.111 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-27T06:31:38.258 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-27T06:31:38.312 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-27T06:31:38.410 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-27T06:31:38.514 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-27T06:31:38.875 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-27T06:31:38.960 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-27T06:31:38.993 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-27T06:31:39.030 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-27T06:31:39.079 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-27T06:31:39.109 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-27T06:31:39.151 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-27T06:31:39.307 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-27T06:31:39.361 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-27T06:31:39.457 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-27T06:31:39.518 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-27T06:31:39.601 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-27T06:31:39.637 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-27T06:31:39.674 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-27T06:31:39.722 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-27T06:31:39.756 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-27T06:31:39.809 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-27T06:31:39.852 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-27T06:31:39.928 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-27T06:31:39.995 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-27T06:31:40.004 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-27T06:31:40.032 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-27T06:31:40.044 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-27T06:31:40.065 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-27T06:31:40.188 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-27T06:31:40.300 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-27T06:31:40.325 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-27T06:31:40.390 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-27T06:31:40.403 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-27T06:31:40.436 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-27T06:31:40.531 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-27T06:31:40.566 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-27T06:31:40.601 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-27T06:31:40.617 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-27T06:31:40.716 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-27T06:31:40.735 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-27T06:31:40.770 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-27T06:31:40.799 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-07-27T06:31:40.810 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-27T06:31:40.849 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-27T06:31:40.879 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-27T06:31:40.914 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-27T06:31:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-27T06:31:40.947 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-27T06:31:40.965 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-27T06:31:40.989 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-27T06:31:41.023 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-27T06:31:41.025 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-27T06:31:41.069 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-27T06:31:41.100 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-27T06:31:41.102 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-27T06:31:41.114 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-27T06:31:41.122 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-27T06:31:41.162 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-27T06:31:41.167 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-27T06:31:41.193 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-27T06:31:41.194 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-27T06:31:41.224 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-27T06:31:41.301 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-27T06:31:41.353 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-27T06:31:41.599 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-27T06:31:41.731 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-27T06:31:41.880 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-27T06:31:41.910 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-27T06:31:42.011 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-27T06:31:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-27T06:31:42.102 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-27T06:31:42.129 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-27T06:31:42.182 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-27T06:31:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-27T06:31:42.321 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-27T06:31:42.332 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-27T06:31:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-27T06:31:42.400 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-27T06:31:42.429 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-27T06:31:42.433 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-27T06:31:42.532 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-27T06:31:42.538 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-27T06:31:42.568 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-27T06:31:42.573 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-27T06:31:42.636 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-27T06:31:42.667 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-27T06:31:42.691 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-27T06:31:42.755 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-27T06:31:42.782 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-27T06:31:42.863 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-27T06:31:42.899 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-27T06:31:43.012 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-27T06:31:43.311 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-27T06:31:43.506 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-27T06:31:43.510 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-27T06:31:43.537 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-27T06:31:43.559 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-27T06:31:43.571 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-27T06:31:43.673 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-27T06:31:43.684 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-27T06:31:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-27T06:31:44.042 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-27T06:31:44.103 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-27T06:31:44.141 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-27T06:31:44.155 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-27T06:31:44.181 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-27T06:31:44.225 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-27T06:31:44.230 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-27T06:31:44.230 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-27T06:31:44.231 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-27T06:31:44.231 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:31:44.258 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-27T06:31:44.302 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-27T06:31:44.354 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-27T06:31:44.395 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-27T06:31:44.448 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-27T06:31:44.489 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-27T06:31:44.525 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-27T06:31:44.598 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-27T06:31:44.660 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-27T06:31:44.698 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-27T06:31:44.750 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-27T06:31:44.830 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-27T06:31:44.870 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-27T06:31:44.924 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-27T06:31:44.960 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-27T06:31:45.001 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-27T06:31:45.043 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-27T06:31:45.077 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-27T06:31:45.103 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-27T06:31:45.105 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-27T06:31:45.139 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-27T06:31:45.190 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-27T06:31:45.336 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-27T06:31:45.400 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-27T06:31:45.483 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-27T06:31:45.535 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-27T06:31:45.622 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-27T06:31:45.648 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-27T06:31:45.685 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-27T06:31:45.967 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-27T06:31:46.059 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-27T06:31:46.095 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-27T06:31:46.138 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-27T06:31:46.190 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-27T06:31:46.211 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-27T06:31:46.229 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-27T06:31:46.246 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-27T06:31:46.276 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-27T06:31:46.335 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-27T06:31:46.362 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-27T06:31:46.464 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-27T06:31:46.553 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-27T06:31:46.648 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-27T06:31:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-27T06:31:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-27T06:31:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-27T06:31:46.879 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-27T06:31:46.935 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-27T06:31:46.982 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-27T06:31:47.014 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-27T06:31:47.067 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-27T06:31:47.220 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-27T06:31:47.271 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-27T06:31:47.313 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-27T06:31:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-27T06:31:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-27T06:31:47.507 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-27T06:31:47.532 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-27T06:31:47.557 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-27T06:31:47.663 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-27T06:31:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-27T06:31:47.854 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-27T06:31:47.883 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-27T06:31:47.926 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-27T06:31:47.968 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-27T06:31:48.008 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-27T06:31:48.079 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-27T06:31:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-27T06:31:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-27T06:31:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-27T06:31:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-27T06:31:48.334 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-27T06:31:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-27T06:31:48.432 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-27T06:31:48.541 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-27T06:31:48.602 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-27T06:31:48.732 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-27T06:31:48.766 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-27T06:31:49.360 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-27T06:31:49.396 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-27T06:31:49.419 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-27T06:31:49.453 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-27T06:31:49.488 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-27T06:31:49.520 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-27T06:31:49.520 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-27T06:31:49.520 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-27T06:31:49.520 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:31:49.554 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-27T06:31:49.591 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-27T06:31:49.655 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-27T06:31:49.710 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-27T06:31:49.800 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-27T06:31:49.845 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-27T06:31:50.030 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-27T06:31:50.060 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-27T06:31:50.149 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-27T06:31:50.855 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-27T06:31:50.989 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-27T06:31:52.329 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-27T06:31:52.439 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-27T06:31:52.475 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-27T06:31:52.509 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-27T06:31:52.862 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-27T06:31:52.923 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-27T06:31:53.030 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-27T06:31:53.064 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-27T06:31:53.109 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-27T06:31:53.636 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-27T06:31:53.672 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-27T06:31:53.745 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-27T06:31:53.782 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-27T06:31:55.940 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-27T06:31:55.940 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2024-07-27T06:31:55.940 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2024-07-27T06:31:55.940 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2024-07-27T06:31:55.940 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2024-07-27T06:31:55.940 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2024-07-27T06:31:55.941 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /root 2024-07-27T06:31:55.941 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2024-07-27T06:31:55.941 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:31:56.115 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-27T06:31:56.131 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-27T06:31:56.131 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:31:56.131 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-27T06:31:56.131 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:31:56.189 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-27T06:31:56.231 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-27T06:31:56.827 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-27T06:31:56.873 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-27T06:31:56.969 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-27T06:31:56.969 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-27T06:31:56.969 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-27T06:31:56.970 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:31:57.000 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-27T06:31:57.710 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-27T06:31:57.736 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-27T06:31:57.736 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:31:57.736 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-27T06:31:57.736 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-27T06:31:57.736 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-27T06:31:57.737 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:31:57.960 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-27T06:31:57.983 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-27T06:31:57.983 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:31:57.983 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-27T06:31:57.983 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-27T06:31:57.983 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-27T06:31:57.983 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:31:58.444 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-27T06:31:58.467 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-27T06:31:58.467 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:31:58.467 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-27T06:31:58.468 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-27T06:31:58.468 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-27T06:31:58.468 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:31:58.566 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-27T06:31:58.614 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-27T06:31:58.670 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-27T06:31:58.865 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-27T06:31:58.912 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-27T06:31:58.989 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-27T06:31:59.078 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-27T06:31:59.225 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-27T06:31:59.262 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-27T06:31:59.293 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-27T06:31:59.321 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-27T06:31:59.757 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-27T06:31:59.786 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-27T06:31:59.818 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-27T06:31:59.883 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-27T06:31:59.909 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-27T06:32:01.622 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-27T06:32:01.622 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /sys 2024-07-27T06:32:01.622 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /proc 2024-07-27T06:32:01.623 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /mnt 2024-07-27T06:32:01.623 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /var/tmp 2024-07-27T06:32:01.623 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /home 2024-07-27T06:32:01.623 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /root 2024-07-27T06:32:01.623 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /tmp 2024-07-27T06:32:01.623 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:32:01.801 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-27T06:32:01.817 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-27T06:32:01.817 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:01.817 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-27T06:32:01.817 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:32:03.041 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-27T06:32:03.106 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-27T06:32:03.454 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-27T06:32:03.488 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-27T06:32:03.488 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:03.488 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-27T06:32:03.488 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-27T06:32:03.488 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-27T06:32:03.489 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:32:03.731 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-27T06:32:03.754 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-27T06:32:03.755 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:03.755 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-27T06:32:03.755 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-27T06:32:03.755 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-27T06:32:03.755 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:32:04.067 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-27T06:32:04.098 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-27T06:32:04.210 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-27T06:32:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-27T06:32:04.263 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-27T06:32:04.263 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:04.264 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-27T06:32:04.264 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-27T06:32:04.264 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-27T06:32:04.264 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:32:04.380 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-27T06:32:04.391 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-27T06:32:04.418 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-27T06:32:04.437 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-27T06:32:04.440 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-27T06:32:04.440 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:04.441 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-27T06:32:04.441 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-27T06:32:04.441 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-27T06:32:04.441 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:32:04.485 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-27T06:32:04.496 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-27T06:32:04.502 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-27T06:32:04.596 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-27T06:32:04.641 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-27T06:32:04.702 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-27T06:32:04.758 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-27T06:32:04.832 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-27T06:32:04.930 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-27T06:32:05.082 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-27T06:32:05.125 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-27T06:32:05.157 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-27T06:32:05.184 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-27T06:32:05.604 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-27T06:32:05.605 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-27T06:32:05.629 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-27T06:32:05.641 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-27T06:32:05.651 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-27T06:32:05.652 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:05.652 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-27T06:32:05.652 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-27T06:32:05.652 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-27T06:32:05.652 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:32:05.673 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-27T06:32:05.704 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-27T06:32:05.726 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-27T06:32:05.726 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:05.727 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-27T06:32:05.727 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:32:05.740 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-27T06:32:05.763 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-27T06:32:05.772 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-27T06:32:05.794 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-27T06:32:05.794 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:05.794 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-27T06:32:05.794 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-27T06:32:05.794 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-27T06:32:05.795 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:32:06.122 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-27T06:32:06.144 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-27T06:32:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-27T06:32:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-27T06:32:06.145 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-27T06:32:06.145 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:32:09.161 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-27T06:32:09.220 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-27T06:32:10.244 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-27T06:32:10.270 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-27T06:32:10.394 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-27T06:32:10.501 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-27T06:32:10.542 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-27T06:32:10.579 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-27T06:32:10.583 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-27T06:32:10.614 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-27T06:32:10.615 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-27T06:32:10.637 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-27T06:32:10.638 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:10.638 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-27T06:32:10.638 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-27T06:32:10.638 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-27T06:32:10.638 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:32:10.642 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-27T06:32:10.689 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-27T06:32:10.700 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-27T06:32:10.804 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-27T06:32:10.823 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-27T06:32:10.856 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-27T06:32:10.874 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-27T06:32:10.907 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-27T06:32:10.940 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-27T06:32:10.969 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-27T06:32:11.015 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-27T06:32:11.619 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-27T06:32:11.673 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-27T06:32:11.693 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-27T06:32:11.870 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-27T06:32:11.900 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-27T06:32:11.924 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-27T06:32:11.924 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:11.925 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-27T06:32:11.925 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-27T06:32:11.925 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-27T06:32:11.925 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:32:11.968 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-27T06:32:11.991 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-27T06:32:11.991 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:11.991 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-27T06:32:11.991 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:32:12.052 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-27T06:32:12.075 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-27T06:32:12.075 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:12.075 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-27T06:32:12.075 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-27T06:32:12.075 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-27T06:32:12.075 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:32:12.411 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-27T06:32:12.435 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-27T06:32:12.435 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:12.435 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-27T06:32:12.435 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-27T06:32:12.435 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-27T06:32:12.435 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:32:12.524 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-27T06:32:12.612 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-27T06:32:12.634 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-27T06:32:12.634 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-07-27T06:32:12.634 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-07-27T06:32:12.634 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-07-27T06:32:12.634 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-07-27T06:32:12.634 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-07-27T06:32:12.634 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2024-07-27T06:32:12.634 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-07-27T06:32:12.635 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:32:12.669 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-27T06:32:12.750 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-27T06:32:12.799 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-27T06:32:12.825 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-27T06:32:12.831 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-27T06:32:12.847 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-27T06:32:12.850 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:12.851 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-27T06:32:12.851 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:32:12.851 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-27T06:32:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-27T06:32:12.876 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-27T06:32:12.944 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-27T06:32:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-27T06:32:12.969 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-27T06:32:13.045 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-27T06:32:13.070 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-27T06:32:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-27T06:32:13.075 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-27T06:32:13.247 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-27T06:32:13.247 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-27T06:32:13.331 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-27T06:32:13.364 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-27T06:32:13.390 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-27T06:32:13.455 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-27T06:32:14.513 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-27T06:32:14.542 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-27T06:32:14.542 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:14.542 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-27T06:32:14.542 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-27T06:32:14.542 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-27T06:32:14.542 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:32:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-27T06:32:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-27T06:32:14.814 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:14.815 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-27T06:32:14.815 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-27T06:32:14.815 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-27T06:32:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:32:15.299 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-27T06:32:15.323 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-27T06:32:15.324 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:15.324 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-27T06:32:15.324 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-27T06:32:15.324 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-27T06:32:15.324 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:32:15.433 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-27T06:32:15.492 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-27T06:32:15.564 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-27T06:32:15.771 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-27T06:32:15.833 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-27T06:32:15.917 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-27T06:32:16.013 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-27T06:32:16.165 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-27T06:32:16.211 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-27T06:32:16.254 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-27T06:32:16.282 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-27T06:32:16.750 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-27T06:32:16.786 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-27T06:32:16.819 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-27T06:32:16.894 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-27T06:32:16.926 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-27T06:32:17.074 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-27T06:32:17.115 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-27T06:32:17.157 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-27T06:32:17.195 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-27T06:32:17.222 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-27T06:32:17.426 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-27T06:32:17.481 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-27T06:32:17.514 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-27T06:32:17.546 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-27T06:32:17.576 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-27T06:32:17.655 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-27T06:32:18.607 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-27T06:32:18.646 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-27T06:32:18.666 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-27T06:32:19.578 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-27T06:32:19.661 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-27T06:32:19.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-27T06:32:19.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-27T06:32:19.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-27T06:32:19.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-27T06:32:19.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-27T06:32:19.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-27T06:32:19.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-27T06:32:19.661 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-27T06:32:19.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-27T06:32:19.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-27T06:32:19.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-27T06:32:19.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-27T06:32:19.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-27T06:32:19.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-27T06:32:19.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-27T06:32:19.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-27T06:32:19.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-27T06:32:19.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-27T06:32:19.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-27T06:32:19.662 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-27T06:32:19.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-27T06:32:19.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-27T06:32:19.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-27T06:32:19.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-27T06:32:19.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-27T06:32:19.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-27T06:32:19.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-27T06:32:19.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-27T06:32:19.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-27T06:32:19.663 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-27T06:32:19.676 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-27T06:32:19.676 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-27T06:32:19.676 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-27T06:32:19.676 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-27T06:32:19.676 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-27T06:32:19.676 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-27T06:32:19.676 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-27T06:32:19.676 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-27T06:32:19.676 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-27T06:32:19.677 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-27T06:32:19.677 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-27T06:32:19.677 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-27T06:32:19.677 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-27T06:32:19.677 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-27T06:32:19.677 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-27T06:32:19.677 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-27T06:32:19.677 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-27T06:32:19.677 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-27T06:32:19.677 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-27T06:32:19.677 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-27T06:32:19.677 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-27T06:32:19.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-27T06:32:19.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-27T06:32:19.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-27T06:32:19.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-27T06:32:19.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-27T06:32:19.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-27T06:32:19.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-27T06:32:19.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-27T06:32:19.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-27T06:32:19.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-27T06:32:19.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-27T06:32:19.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-27T06:32:19.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-27T06:32:19.680 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-27T06:32:19.681 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-27T06:32:19.682 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-27T06:32:19.683 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-27T06:32:19.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-27T06:32:19.685 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-27T06:32:19.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-27T06:32:19.687 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-27T06:32:19.723 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-27T06:32:19.809 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-27T06:32:19.843 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-27T06:32:19.868 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-27T06:32:19.893 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-27T06:32:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-27T06:32:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-27T06:32:20.004 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-27T06:32:20.037 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-27T06:32:20.037 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-27T06:32:20.130 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-27T06:32:20.155 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-27T06:32:20.160 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-27T06:32:20.160 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-27T06:32:20.276 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-27T06:32:20.349 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-27T06:32:20.349 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-27T06:32:20.349 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-27T06:32:20.433 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-27T06:32:20.459 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-27T06:32:20.485 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-27T06:32:20.569 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-27T06:32:21.366 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-27T06:32:21.393 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-27T06:32:21.515 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-27T06:32:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-27T06:32:21.730 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-27T06:32:21.754 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-27T06:32:21.755 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:21.755 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-27T06:32:21.755 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-27T06:32:21.755 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-27T06:32:21.755 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:32:21.812 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-27T06:32:21.824 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-27T06:32:21.931 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-27T06:32:22.002 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-27T06:32:22.531 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-27T06:32:22.532 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-27T06:32:22.533 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-27T06:32:22.533 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-07-27T06:32:22.533 INFO:teuthology.orchestra.run.smithi077.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-27T06:32:22.533 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.2-8.el9.noarch 2024-07-27T06:32:22.533 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.7.4-5.el9.x86_64 2024-07-27T06:32:22.533 INFO:teuthology.orchestra.run.smithi077.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-27T06:32:22.533 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.533 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.533 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.533 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.533 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:22.533 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.534 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.4-9.el9.x86_64 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-27T06:32:22.535 INFO:teuthology.orchestra.run.smithi077.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.536 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: lua-5.4.4-4.el9.x86_64 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-27T06:32:22.537 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-13.14-1.el9.x86_64 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-27T06:32:22.538 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-27T06:32:22.539 INFO:teuthology.orchestra.run.smithi077.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-27T06:32:22.540 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply-3.11-14.el9.noarch 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-27T06:32:22.541 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-27T06:32:22.542 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-27T06:32:22.542 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-27T06:32:22.542 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-27T06:32:22.542 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.542 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.542 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-27T06:32:22.542 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-27T06:32:22.542 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-27T06:32:22.542 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.542 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-27T06:32:22.542 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-27T06:32:22.542 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.543 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.544 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:22.544 INFO:teuthology.orchestra.run.smithi077.stdout: re2-1:20211101-3.el9.x86_64 2024-07-27T06:32:22.544 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-27T06:32:22.544 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-27T06:32:22.544 INFO:teuthology.orchestra.run.smithi077.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-27T06:32:22.544 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-27T06:32:22.544 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-27T06:32:22.544 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-27T06:32:22.544 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:32:22.544 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-07-27T06:32:22.949 DEBUG:teuthology.parallel:result is None 2024-07-27T06:32:23.038 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-27T06:32:23.071 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-27T06:32:23.096 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-27T06:32:23.096 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:23.096 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-27T06:32:23.096 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-27T06:32:23.097 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-27T06:32:23.097 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:32:23.148 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-27T06:32:23.173 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-27T06:32:23.173 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:23.173 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-27T06:32:23.173 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:32:23.232 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-27T06:32:23.257 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-27T06:32:23.257 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:23.257 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-27T06:32:23.257 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-27T06:32:23.257 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-27T06:32:23.257 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:32:23.600 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-27T06:32:23.628 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-27T06:32:23.628 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-27T06:32:23.628 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-27T06:32:23.628 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-27T06:32:23.628 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-27T06:32:23.628 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:32:26.584 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-27T06:32:26.584 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-27T06:32:26.584 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-27T06:32:26.584 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-27T06:32:26.584 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-27T06:32:26.584 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-27T06:32:26.584 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-27T06:32:26.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-27T06:32:26.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-27T06:32:26.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-27T06:32:26.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-27T06:32:26.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-27T06:32:26.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-27T06:32:26.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-27T06:32:26.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-27T06:32:26.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-27T06:32:26.585 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-27T06:32:26.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-27T06:32:26.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-27T06:32:26.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-27T06:32:26.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-27T06:32:26.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-27T06:32:26.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-27T06:32:26.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-27T06:32:26.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-27T06:32:26.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-27T06:32:26.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-27T06:32:26.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-27T06:32:26.586 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-27T06:32:26.587 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-27T06:32:26.601 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-27T06:32:26.601 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-27T06:32:26.601 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-27T06:32:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-27T06:32:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-27T06:32:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-27T06:32:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-27T06:32:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-27T06:32:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-27T06:32:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-27T06:32:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-27T06:32:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-27T06:32:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-27T06:32:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-27T06:32:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-27T06:32:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-27T06:32:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-27T06:32:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-27T06:32:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-27T06:32:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-27T06:32:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-27T06:32:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-27T06:32:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-27T06:32:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-27T06:32:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-27T06:32:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-27T06:32:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-27T06:32:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-27T06:32:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-27T06:32:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-27T06:32:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-27T06:32:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-27T06:32:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-27T06:32:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-27T06:32:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-27T06:32:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-27T06:32:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-27T06:32:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-27T06:32:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-27T06:32:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-27T06:32:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-27T06:32:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-27T06:32:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-27T06:32:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-27T06:32:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-27T06:32:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-27T06:32:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-27T06:32:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-27T06:32:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-27T06:32:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-27T06:32:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-27T06:32:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-27T06:32:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-27T06:32:28.302 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-27T06:32:28.353 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-27T06:32:28.396 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-27T06:32:28.428 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-27T06:32:28.456 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-27T06:32:28.660 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-27T06:32:28.716 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-27T06:32:28.754 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-27T06:32:28.798 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-27T06:32:28.836 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-27T06:32:28.893 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-27T06:32:29.324 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-27T06:32:29.324 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:32:29.324 INFO:teuthology.orchestra.run.smithi047.stdout:Upgraded: 2024-07-27T06:32:29.324 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-27T06:32:29.324 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-27T06:32:29.324 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-27T06:32:29.324 INFO:teuthology.orchestra.run.smithi047.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-27T06:32:29.324 INFO:teuthology.orchestra.run.smithi047.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-27T06:32:29.324 INFO:teuthology.orchestra.run.smithi047.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-27T06:32:29.325 INFO:teuthology.orchestra.run.smithi047.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-27T06:32:29.325 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.325 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.325 INFO:teuthology.orchestra.run.smithi047.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-27T06:32:29.325 INFO:teuthology.orchestra.run.smithi047.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-27T06:32:29.325 INFO:teuthology.orchestra.run.smithi047.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-27T06:32:29.325 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-07-27T06:32:29.325 INFO:teuthology.orchestra.run.smithi047.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-27T06:32:29.325 INFO:teuthology.orchestra.run.smithi047.stdout: automake-1.16.2-8.el9.noarch 2024-07-27T06:32:29.325 INFO:teuthology.orchestra.run.smithi047.stdout: bison-3.7.4-5.el9.x86_64 2024-07-27T06:32:29.325 INFO:teuthology.orchestra.run.smithi047.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-27T06:32:29.326 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.326 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.326 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.326 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.326 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:29.326 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.326 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.326 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.326 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:29.326 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:29.326 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.327 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: flex-2.6.4-9.el9.x86_64 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.328 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-27T06:32:29.329 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-27T06:32:29.330 INFO:teuthology.orchestra.run.smithi047.stdout: lua-5.4.4-4.el9.x86_64 2024-07-27T06:32:29.330 INFO:teuthology.orchestra.run.smithi047.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-27T06:32:29.330 INFO:teuthology.orchestra.run.smithi047.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-27T06:32:29.330 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-27T06:32:29.330 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-27T06:32:29.330 INFO:teuthology.orchestra.run.smithi047.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-27T06:32:29.330 INFO:teuthology.orchestra.run.smithi047.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-27T06:32:29.330 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-13.14-1.el9.x86_64 2024-07-27T06:32:29.330 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-27T06:32:29.330 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-27T06:32:29.330 INFO:teuthology.orchestra.run.smithi047.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-27T06:32:29.330 INFO:teuthology.orchestra.run.smithi047.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-27T06:32:29.331 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-27T06:32:29.332 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-27T06:32:29.332 INFO:teuthology.orchestra.run.smithi047.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-27T06:32:29.332 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-27T06:32:29.332 INFO:teuthology.orchestra.run.smithi047.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-27T06:32:29.332 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-27T06:32:29.332 INFO:teuthology.orchestra.run.smithi047.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-27T06:32:29.332 INFO:teuthology.orchestra.run.smithi047.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-27T06:32:29.332 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-27T06:32:29.332 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-27T06:32:29.332 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-27T06:32:29.332 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-27T06:32:29.332 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-27T06:32:29.333 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ply-3.11-14.el9.noarch 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.334 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-27T06:32:29.335 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: re2-1:20211101-3.el9.x86_64 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-27T06:32:29.336 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-27T06:32:29.337 INFO:teuthology.orchestra.run.smithi047.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-27T06:32:29.337 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-07-27T06:32:29.337 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-07-27T06:32:29.581 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-27T06:32:29.642 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-27T06:32:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-27T06:32:29.841 DEBUG:teuthology.parallel:result is None 2024-07-27T06:32:30.587 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-27T06:32:30.704 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-27T06:32:30.746 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-27T06:32:30.840 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-27T06:32:30.878 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-27T06:32:30.911 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-27T06:32:30.945 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-27T06:32:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-27T06:32:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-27T06:32:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-27T06:32:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-27T06:32:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-27T06:32:31.203 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-27T06:32:31.228 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-27T06:32:31.234 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-27T06:32:31.234 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-27T06:32:31.431 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-27T06:32:31.432 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-27T06:32:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-27T06:32:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-27T06:32:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-27T06:32:31.673 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-27T06:32:43.442 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-27T06:32:43.442 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-27T06:32:43.442 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-27T06:32:43.442 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-27T06:32:43.442 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-27T06:32:43.442 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-27T06:32:43.442 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-27T06:32:43.442 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-27T06:32:43.442 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-27T06:32:43.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-27T06:32:43.444 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-27T06:32:43.444 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-27T06:32:43.444 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-27T06:32:43.444 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-27T06:32:43.444 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-27T06:32:43.444 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-27T06:32:43.444 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-27T06:32:43.456 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-27T06:32:43.456 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-27T06:32:43.456 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-27T06:32:43.456 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-27T06:32:43.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-27T06:32:43.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-27T06:32:43.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-27T06:32:43.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-27T06:32:43.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-27T06:32:43.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-27T06:32:43.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-27T06:32:43.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-27T06:32:43.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-27T06:32:43.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-27T06:32:43.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-27T06:32:43.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-27T06:32:43.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-27T06:32:43.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-27T06:32:43.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-27T06:32:43.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-27T06:32:43.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-27T06:32:43.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-27T06:32:43.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-27T06:32:43.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-27T06:32:43.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-27T06:32:43.467 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-27T06:32:43.467 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-27T06:32:43.467 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-27T06:32:43.467 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-27T06:32:46.217 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-27T06:32:46.217 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:32:46.218 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-07-27T06:32:46.218 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-27T06:32:46.218 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-27T06:32:46.218 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-27T06:32:46.218 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-27T06:32:46.219 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-27T06:32:46.219 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-27T06:32:46.219 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-27T06:32:46.219 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.219 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.220 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-27T06:32:46.220 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-27T06:32:46.220 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-27T06:32:46.220 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-07-27T06:32:46.220 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-27T06:32:46.220 INFO:teuthology.orchestra.run.smithi029.stdout: automake-1.16.2-8.el9.noarch 2024-07-27T06:32:46.221 INFO:teuthology.orchestra.run.smithi029.stdout: bison-3.7.4-5.el9.x86_64 2024-07-27T06:32:46.221 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-27T06:32:46.221 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.221 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.221 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.222 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.222 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:46.222 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.222 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.222 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.223 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:46.223 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:46.223 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:46.223 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:46.223 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:46.223 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.224 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.224 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:46.224 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.225 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.225 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.225 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:46.225 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:46.225 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.226 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.226 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-27T06:32:46.226 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-27T06:32:46.226 INFO:teuthology.orchestra.run.smithi029.stdout: flex-2.6.4-9.el9.x86_64 2024-07-27T06:32:46.226 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-27T06:32:46.226 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-27T06:32:46.227 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-27T06:32:46.227 INFO:teuthology.orchestra.run.smithi029.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-27T06:32:46.227 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-27T06:32:46.227 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-27T06:32:46.227 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-27T06:32:46.228 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-27T06:32:46.228 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-27T06:32:46.228 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.228 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.228 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.229 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-27T06:32:46.229 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-27T06:32:46.229 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-27T06:32:46.229 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-27T06:32:46.229 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-27T06:32:46.229 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.230 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.230 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-27T06:32:46.230 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.230 INFO:teuthology.orchestra.run.smithi029.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-27T06:32:46.231 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-27T06:32:46.231 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-27T06:32:46.231 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-27T06:32:46.231 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-27T06:32:46.231 INFO:teuthology.orchestra.run.smithi029.stdout: lua-5.4.4-4.el9.x86_64 2024-07-27T06:32:46.231 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-27T06:32:46.232 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-27T06:32:46.232 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-27T06:32:46.232 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-27T06:32:46.232 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-27T06:32:46.232 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-27T06:32:46.233 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-13.14-1.el9.x86_64 2024-07-27T06:32:46.233 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-27T06:32:46.233 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-27T06:32:46.233 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-27T06:32:46.233 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-27T06:32:46.234 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-27T06:32:46.234 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-27T06:32:46.234 INFO:teuthology.orchestra.run.smithi029.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-27T06:32:46.234 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-27T06:32:46.234 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-27T06:32:46.234 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-27T06:32:46.235 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.235 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.235 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.235 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-27T06:32:46.235 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-27T06:32:46.236 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-27T06:32:46.236 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-27T06:32:46.236 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-27T06:32:46.236 INFO:teuthology.orchestra.run.smithi029.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-27T06:32:46.236 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-27T06:32:46.236 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-27T06:32:46.237 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-27T06:32:46.237 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-27T06:32:46.237 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-27T06:32:46.237 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-27T06:32:46.237 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-27T06:32:46.238 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-27T06:32:46.238 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-27T06:32:46.238 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-27T06:32:46.238 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-27T06:32:46.238 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-27T06:32:46.238 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-27T06:32:46.239 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-27T06:32:46.239 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-27T06:32:46.239 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-27T06:32:46.239 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-27T06:32:46.239 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-27T06:32:46.240 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-27T06:32:46.240 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-27T06:32:46.240 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-27T06:32:46.240 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-27T06:32:46.240 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-27T06:32:46.240 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-27T06:32:46.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2024-07-27T06:32:46.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-27T06:32:46.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-27T06:32:46.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-27T06:32:46.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-27T06:32:46.241 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-27T06:32:46.242 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-27T06:32:46.242 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-27T06:32:46.242 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-27T06:32:46.243 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-27T06:32:46.243 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.243 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.243 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-27T06:32:46.243 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-27T06:32:46.243 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-27T06:32:46.244 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.244 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-27T06:32:46.244 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-27T06:32:46.244 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-27T06:32:46.244 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-27T06:32:46.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-27T06:32:46.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-27T06:32:46.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-27T06:32:46.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-27T06:32:46.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-27T06:32:46.245 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-27T06:32:46.246 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-27T06:32:46.246 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-27T06:32:46.246 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-27T06:32:46.246 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-27T06:32:46.246 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-27T06:32:46.247 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.247 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.247 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-27T06:32:46.247 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2024-07-27T06:32:46.247 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-27T06:32:46.248 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-27T06:32:46.248 INFO:teuthology.orchestra.run.smithi029.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-27T06:32:46.248 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-27T06:32:46.248 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-27T06:32:46.249 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-27T06:32:46.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:32:46.249 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-07-27T06:32:46.761 DEBUG:teuthology.parallel:result is None 2024-07-27T06:32:46.761 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:32:46.906 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-27T06:32:46.932 INFO:teuthology.orchestra.run.smithi029.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-27T06:32:46.932 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-27T06:32:46.932 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-27T06:32:46.934 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:32:47.078 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-27T06:32:47.105 INFO:teuthology.orchestra.run.smithi047.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-27T06:32:47.105 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-27T06:32:47.105 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-27T06:32:47.107 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:32:47.243 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-27T06:32:47.269 INFO:teuthology.orchestra.run.smithi077.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-27T06:32:47.269 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-27T06:32:47.269 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-27T06:32:47.271 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-07-27T06:32:47.271 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:32:47.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-27T06:32:47.305 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-07-27T06:32:47.305 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-27T06:32:47.337 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-07-27T06:32:47.338 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-27T06:32:47.369 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-07-27T06:32:47.369 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:32:47.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-07-27T06:32:47.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-27T06:32:47.468 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-07-27T06:32:47.468 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2024-07-27T06:32:47.498 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-27T06:32:47.566 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-07-27T06:32:47.566 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-07-27T06:32:47.594 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-27T06:32:47.662 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-07-27T06:32:47.662 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:32:47.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-27T06:32:47.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-27T06:32:47.764 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-07-27T06:32:47.764 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-27T06:32:47.793 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-27T06:32:47.858 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-07-27T06:32:47.858 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-27T06:32:47.883 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-27T06:32:47.946 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-07-27T06:32:47.947 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:32:47.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-07-27T06:32:47.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-27T06:32:48.048 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-07-27T06:32:48.049 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/stdin-killer 2024-07-27T06:32:48.074 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-27T06:32:48.138 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-07-27T06:32:48.138 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-07-27T06:32:48.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-27T06:32:48.225 INFO:teuthology.run_tasks:Running task cephadm... 2024-07-27T06:32:48.327 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug cephsqlite': 20, 'mgr/snap_schedule/allow_m_granularity': True, 'mgr/snap_schedule/dump_on_update': True, 'mgr/snap_schedule/log_level': 'debug'}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'global': {'ms die on skipped message': False}, 'mds': {'mds_bal_replicate_threshold': 1}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'mgr-modules': ['snap_schedule'], 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'subvols': {'create': 2}} 2024-07-27T06:32:48.328 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:32:48.328 INFO:tasks.cephadm:Cluster fsid is 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:32:48.328 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-07-27T06:32:48.329 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.29', 'mon.b': '172.21.15.47', 'mon.c': '172.21.15.77'} 2024-07-27T06:32:48.329 INFO:tasks.cephadm:First mon is mon.a on smithi029 2024-07-27T06:32:48.329 INFO:tasks.cephadm:First mgr is x 2024-07-27T06:32:48.329 INFO:tasks.cephadm:Normalizing hostnames... 2024-07-27T06:32:48.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-07-27T06:32:48.361 DEBUG:teuthology.orchestra.run.smithi047:> sudo hostname $(hostname -s) 2024-07-27T06:32:48.460 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname $(hostname -s) 2024-07-27T06:32:48.484 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-07-27T06:32:48.484 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:32:48.632 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 16:18:49.953482', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['arm64', 'x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.2.8+braggi08', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 15:10:42.298781', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-07-27T06:32:48.776 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-vshankar-testing-20240725.121331-debug, sha1 e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:32:48.777 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-27T06:32:48.777 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-27T06:32:48.777 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-27T06:32:49.157 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 27 06:32 /home/ubuntu/cephtest/cephadm 2024-07-27T06:32:49.157 DEBUG:teuthology.orchestra.run.smithi047:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-27T06:32:49.538 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 27 06:32 /home/ubuntu/cephtest/cephadm 2024-07-27T06:32:49.539 DEBUG:teuthology.orchestra.run.smithi077:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-27T06:32:49.919 INFO:teuthology.orchestra.run.smithi077.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 27 06:32 /home/ubuntu/cephtest/cephadm 2024-07-27T06:32:49.919 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-27T06:32:49.940 DEBUG:teuthology.orchestra.run.smithi047:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-27T06:32:49.959 DEBUG:teuthology.orchestra.run.smithi077:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-27T06:32:49.993 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 on all hosts... 2024-07-27T06:32:49.994 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-27T06:32:49.996 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-27T06:32:50.002 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-27T06:32:50.817 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-27T06:32:50.929 INFO:teuthology.orchestra.run.smithi047.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-27T06:32:51.018 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-27T06:33:24.863 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-07-27T06:33:24.864 INFO:teuthology.orchestra.run.smithi047.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-27T06:33:24.864 INFO:teuthology.orchestra.run.smithi047.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-27T06:33:24.864 INFO:teuthology.orchestra.run.smithi047.stdout: "repo_digests": [ 2024-07-27T06:33:24.864 INFO:teuthology.orchestra.run.smithi047.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-27T06:33:24.864 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-07-27T06:33:24.864 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-07-27T06:33:41.618 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-07-27T06:33:41.618 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-27T06:33:41.618 INFO:teuthology.orchestra.run.smithi077.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-27T06:33:41.618 INFO:teuthology.orchestra.run.smithi077.stdout: "repo_digests": [ 2024-07-27T06:33:41.618 INFO:teuthology.orchestra.run.smithi077.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-27T06:33:41.618 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-07-27T06:33:41.619 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-07-27T06:33:48.354 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:33:48.354 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-27T06:33:48.354 INFO:teuthology.orchestra.run.smithi029.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-27T06:33:48.354 INFO:teuthology.orchestra.run.smithi029.stdout: "repo_digests": [ 2024-07-27T06:33:48.354 INFO:teuthology.orchestra.run.smithi029.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-27T06:33:48.354 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-07-27T06:33:48.354 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:33:48.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-07-27T06:33:48.447 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph 2024-07-27T06:33:48.481 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2024-07-27T06:33:48.513 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-07-27T06:33:48.539 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 777 /etc/ceph 2024-07-27T06:33:48.568 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /etc/ceph 2024-07-27T06:33:48.594 INFO:tasks.cephadm:Writing seed config... 2024-07-27T06:33:48.596 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-07-27T06:33:48.596 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-07-27T06:33:48.596 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-07-27T06:33:48.596 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-07-27T06:33:48.596 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-07-27T06:33:48.596 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-07-27T06:33:48.596 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-07-27T06:33:48.596 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-07-27T06:33:48.596 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-07-27T06:33:48.596 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-07-27T06:33:48.596 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-07-27T06:33:48.597 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-07-27T06:33:48.598 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-07-27T06:33:48.598 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:33:48.598 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-07-27T06:33:48.615 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [mds] mds_bal_replicate_threshold = 1 2024-07-27T06:33:48.615 DEBUG:teuthology.orchestra.run.smithi029:mon.a> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mon.a.service 2024-07-27T06:33:48.658 DEBUG:teuthology.orchestra.run.smithi029:mgr.x> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mgr.x.service 2024-07-27T06:33:48.700 INFO:tasks.cephadm:Bootstrapping... 2024-07-27T06:33:48.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 -v bootstrap --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.29 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-07-27T06:33:48.914 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-07-27T06:33:48.914 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', '-v', 'bootstrap', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.29', '--skip-admin-label'] 2024-07-27T06:33:48.914 INFO:teuthology.orchestra.run.smithi029.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-07-27T06:33:48.915 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying podman|docker is present... 2024-07-27T06:33:48.940 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 5.1.2 2024-07-27T06:33:48.940 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying lvm2 is present... 2024-07-27T06:33:48.941 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying time synchronization is in place... 2024-07-27T06:33:48.948 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-27T06:33:48.948 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-27T06:33:48.955 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-27T06:33:48.955 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-07-27T06:33:48.962 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-07-27T06:33:48.969 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-07-27T06:33:48.969 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-07-27T06:33:48.969 INFO:teuthology.orchestra.run.smithi029.stdout:Repeating the final host check... 2024-07-27T06:33:48.993 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 5.1.2 2024-07-27T06:33:48.993 INFO:teuthology.orchestra.run.smithi029.stdout:podman (/bin/podman) version 5.1.2 is present 2024-07-27T06:33:48.993 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl is present 2024-07-27T06:33:48.993 INFO:teuthology.orchestra.run.smithi029.stdout:lvcreate is present 2024-07-27T06:33:49.000 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-27T06:33:49.000 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-27T06:33:49.006 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-27T06:33:49.006 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-07-27T06:33:49.013 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-07-27T06:33:49.020 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-07-27T06:33:49.020 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-07-27T06:33:49.020 INFO:teuthology.orchestra.run.smithi029.stdout:Host looks OK 2024-07-27T06:33:49.020 INFO:teuthology.orchestra.run.smithi029.stdout:Cluster fsid: 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:33:49.021 INFO:teuthology.orchestra.run.smithi029.stdout:Acquiring lock 140040693146768 on /run/cephadm/0a8a0a4c-4be2-11ef-bcc7-c7b262605968.lock 2024-07-27T06:33:49.021 INFO:teuthology.orchestra.run.smithi029.stdout:Lock 140040693146768 acquired on /run/cephadm/0a8a0a4c-4be2-11ef-bcc7-c7b262605968.lock 2024-07-27T06:33:49.021 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 3300 ... 2024-07-27T06:33:49.021 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 6789 ... 2024-07-27T06:33:49.021 INFO:teuthology.orchestra.run.smithi029.stdout:Base mon IP(s) is [172.21.15.29:3300, 172.21.15.29:6789], mon addrv is [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-07-27T06:33:49.025 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.29 metric 100 2024-07-27T06:33:49.025 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.29 metric 100 2024-07-27T06:33:49.027 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-07-27T06:33:49.028 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-07-27T06:33:49.028 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-07-27T06:33:49.030 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-07-27T06:33:49.030 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-07-27T06:33:49.030 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-27T06:33:49.030 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-07-27T06:33:49.030 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2024-07-27T06:33:49.030 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-27T06:33:49.031 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-07-27T06:33:49.031 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-07-27T06:33:49.031 INFO:teuthology.orchestra.run.smithi029.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-07-27T06:33:49.031 INFO:teuthology.orchestra.run.smithi029.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-07-27T06:33:49.032 INFO:teuthology.orchestra.run.smithi029.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-27T06:33:49.528 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-27T06:33:49.528 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-27T06:33:49.528 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Getting image source signatures 2024-07-27T06:33:49.528 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:27b44fafaf0d8ddf6712bf077b0e069fd1632e81c2016bcdf9194ad64e762303 2024-07-27T06:33:49.528 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:43735fbce9214f061a6894341d7822563f5aafe16fc7532189f4415cff6d2d91 2024-07-27T06:33:49.528 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying config sha256:ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-27T06:33:49.528 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Writing manifest to image destination 2024-07-27T06:33:50.464 INFO:teuthology.orchestra.run.smithi029.stdout:ceph: stdout ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:33:50.464 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph version: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:33:50.464 INFO:teuthology.orchestra.run.smithi029.stdout:Extracting ceph user uid/gid from container image... 2024-07-27T06:33:51.319 INFO:teuthology.orchestra.run.smithi029.stdout:stat: stdout 167 167 2024-07-27T06:33:51.320 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial keys... 2024-07-27T06:33:52.191 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQDPlKRm9Ey1JRAAJ5IEj6oYNr35eInTFxz9rA== 2024-07-27T06:33:53.129 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQDQlKRmEdqdHhAA7VI8FRNNFgc0r0J8kzSUCw== 2024-07-27T06:33:54.058 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQDRlKRmU9UQHBAAoWl6enpbVAMTg8X0A2XL/w== 2024-07-27T06:33:54.059 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial monmap... 2024-07-27T06:33:54.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-27T06:33:54.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-07-27T06:33:54.988 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:33:54.989 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-27T06:33:54.989 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool for a [v2:172.21.15.29:3300,v1:172.21.15.29:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-27T06:33:54.989 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = quincy 2024-07-27T06:33:54.989 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: set fsid to 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:33:54.989 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-27T06:33:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:33:54.989 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mon... 2024-07-27T06:33:56.036 INFO:teuthology.orchestra.run.smithi029.stdout:create mon.a on 2024-07-27T06:33:56.259 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-07-27T06:33:56.426 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-07-27T06:33:56.607 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968.target → /etc/systemd/system/ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968.target. 2024-07-27T06:33:56.608 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968.target → /etc/systemd/system/ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968.target. 2024-07-27T06:33:56.872 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mon.a 2024-07-27T06:33:56.873 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mon.a.service: Unit ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mon.a.service not loaded. 2024-07-27T06:33:57.055 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968.target.wants/ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mon.a.service → /etc/systemd/system/ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@.service. 2024-07-27T06:33:57.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:57 smithi029 systemd[1]: Starting Ceph mon.a for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:33:57.391 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:57 smithi029 podman[32220]: 2024-07-27 06:33:57.279780053 +0000 UTC m=+0.097190978 container create 81683426f6da612024aa5ad979df12f8ceb46fcd557267fe160babfed8dc3670 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724) 2024-07-27T06:33:57.391 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:57 smithi029 podman[32220]: 2024-07-27 06:33:57.201160358 +0000 UTC m=+0.018571293 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:33:57.584 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-27T06:33:57.584 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-07-27T06:33:57.592 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-27T06:33:57.592 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-07-27T06:33:57.592 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-07-27T06:33:57.592 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-07-27T06:33:57.593 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon to start... 2024-07-27T06:33:57.593 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon... 2024-07-27T06:33:57.795 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:57 smithi029 podman[32220]: 2024-07-27 06:33:57.483909577 +0000 UTC m=+0.301320500 container init 81683426f6da612024aa5ad979df12f8ceb46fcd557267fe160babfed8dc3670 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-07-27T06:33:57.795 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:57 smithi029 podman[32220]: 2024-07-27 06:33:57.487192849 +0000 UTC m=+0.304603772 container start 81683426f6da612024aa5ad979df12f8ceb46fcd557267fe160babfed8dc3670 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mon-a, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-27T06:33:57.795 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:57 smithi029 ceph-mon[32238]: mkfs 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:33:57.795 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:57 smithi029 ceph-mon[32238]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-27T06:33:57.795 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:57 smithi029 bash[32220]: 81683426f6da612024aa5ad979df12f8ceb46fcd557267fe160babfed8dc3670 2024-07-27T06:33:57.795 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:57 smithi029 systemd[1]: Started Ceph mon.a for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:33:58.597 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout cluster: 2024-07-27T06:33:58.597 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout id: 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:33:58.597 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-07-27T06:33:58.597 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-07-27T06:33:58.597 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout services: 2024-07-27T06:33:58.598 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.524922s) 2024-07-27T06:33:58.598 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-07-27T06:33:58.598 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-07-27T06:33:58.598 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-07-27T06:33:58.598 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout data: 2024-07-27T06:33:58.598 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-07-27T06:33:58.598 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-07-27T06:33:58.598 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-07-27T06:33:58.598 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pgs: 2024-07-27T06:33:58.598 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-07-27T06:33:58.598 INFO:teuthology.orchestra.run.smithi029.stdout:mon is available 2024-07-27T06:33:58.598 INFO:teuthology.orchestra.run.smithi029.stdout:Assimilating anything we can from ceph.conf... 2024-07-27T06:33:58.710 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:58 smithi029 ceph-mon[32238]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-27T06:33:58.710 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:58 smithi029 ceph-mon[32238]: monmap epoch 1 2024-07-27T06:33:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:58 smithi029 ceph-mon[32238]: fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:33:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:58 smithi029 ceph-mon[32238]: last_changed 2024-07-27T06:33:54.422980+0000 2024-07-27T06:33:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:58 smithi029 ceph-mon[32238]: created 2024-07-27T06:33:54.422980+0000 2024-07-27T06:33:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:58 smithi029 ceph-mon[32238]: min_mon_release 19 (squid) 2024-07-27T06:33:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:58 smithi029 ceph-mon[32238]: election_strategy: 1 2024-07-27T06:33:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:58 smithi029 ceph-mon[32238]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-07-27T06:33:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:58 smithi029 ceph-mon[32238]: fsmap 2024-07-27T06:33:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:58 smithi029 ceph-mon[32238]: osdmap e1: 0 total, 0 up, 0 in 2024-07-27T06:33:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:58 smithi029 ceph-mon[32238]: mgrmap e1: no daemons active 2024-07-27T06:33:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:58 smithi029 ceph-mon[32238]: from='client.? 172.21.15.29:0/1673366009' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-07-27T06:33:59.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-07-27T06:33:59.654 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-07-27T06:33:59.655 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-27T06:33:59.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-27T06:33:59.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-27T06:33:59.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-27T06:33:59.656 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-27T06:33:59.656 INFO:teuthology.orchestra.run.smithi029.stdout:Generating new minimal ceph.conf... 2024-07-27T06:33:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:59 smithi029 ceph-mon[32238]: from='client.? 172.21.15.29:0/2984253199' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-27T06:33:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:33:59 smithi029 ceph-mon[32238]: from='client.? 172.21.15.29:0/2984253199' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-07-27T06:34:00.696 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:00 smithi029 ceph-mon[32238]: from='client.? 172.21.15.29:0/802762183' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:34:00.699 INFO:teuthology.orchestra.run.smithi029.stdout:Restarting the monitor... 2024-07-27T06:34:00.959 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:00 smithi029 systemd[1]: Stopping Ceph mon.a for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:34:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mon-a[32234]: 2024-07-27T06:34:00.957+0000 7f2395e46640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-07-27T06:34:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mon-a[32234]: 2024-07-27T06:34:00.957+0000 7f2395e46640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-07-27T06:34:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:00 smithi029 podman[32482]: 2024-07-27 06:34:00.968761686 +0000 UTC m=+0.103560371 container died 81683426f6da612024aa5ad979df12f8ceb46fcd557267fe160babfed8dc3670 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-07-27T06:34:01.568 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:01 smithi029 podman[32482]: 2024-07-27 06:34:01.310955151 +0000 UTC m=+0.445753833 container cleanup 81683426f6da612024aa5ad979df12f8ceb46fcd557267fe160babfed8dc3670 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-07-27T06:34:01.568 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:01 smithi029 bash[32482]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mon-a 2024-07-27T06:34:01.568 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:01 smithi029 podman[32494]: 2024-07-27 06:34:01.479580488 +0000 UTC m=+0.505513099 container remove 81683426f6da612024aa5ad979df12f8ceb46fcd557267fe160babfed8dc3670 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mon-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.license=GPLv2) 2024-07-27T06:34:01.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:01 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mon.a.service: Deactivated successfully. 2024-07-27T06:34:01.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:01 smithi029 systemd[1]: Stopped Ceph mon.a for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:34:01.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:01 smithi029 systemd[1]: Starting Ceph mon.a for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:34:02.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:01 smithi029 podman[32591]: 2024-07-27 06:34:01.814924715 +0000 UTC m=+0.017938404 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:34:02.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:01 smithi029 podman[32591]: 2024-07-27 06:34:01.914321325 +0000 UTC m=+0.117335001 container create 11b5d831c49ce76de506c067dddf0803dbc506b464c9e52db3e353e44494004d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-07-27T06:34:02.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 podman[32591]: 2024-07-27 06:34:02.120506652 +0000 UTC m=+0.323520331 container init 11b5d831c49ce76de506c067dddf0803dbc506b464c9e52db3e353e44494004d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mon-a, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-27T06:34:02.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 podman[32591]: 2024-07-27 06:34:02.123779474 +0000 UTC m=+0.326793153 container start 11b5d831c49ce76de506c067dddf0803dbc506b464c9e52db3e353e44494004d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mon-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-27T06:34:02.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: set uid:gid to 167:167 (ceph:ceph) 2024-07-27T06:34:02.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-27T06:34:02.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: pidfile_write: ignore empty --pid-file 2024-07-27T06:34:02.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: load: jerasure load: lrc 2024-07-27T06:34:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: RocksDB version: 7.9.2 2024-07-27T06:34:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Git sha 0 2024-07-27T06:34:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-27T06:34:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: DB SUMMARY 2024-07-27T06:34:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: DB Session ID: VG1HLNSBISLY1VC5TNAC 2024-07-27T06:34:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: CURRENT file: CURRENT 2024-07-27T06:34:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: IDENTITY file: IDENTITY 2024-07-27T06:34:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-07-27T06:34:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-07-27T06:34:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 94975 ; 2024-07-27T06:34:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.error_if_exists: 0 2024-07-27T06:34:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.create_if_missing: 0 2024-07-27T06:34:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.paranoid_checks: 1 2024-07-27T06:34:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-27T06:34:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-27T06:34:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-27T06:34:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.env: 0x55b22b25a280 2024-07-27T06:34:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.fs: PosixFileSystem 2024-07-27T06:34:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.info_log: 0x55b22fcc5080 2024-07-27T06:34:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_file_opening_threads: 16 2024-07-27T06:34:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.statistics: (nil) 2024-07-27T06:34:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.use_fsync: 0 2024-07-27T06:34:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_log_file_size: 0 2024-07-27T06:34:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-27T06:34:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-27T06:34:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.keep_log_file_num: 1000 2024-07-27T06:34:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.recycle_log_file_num: 0 2024-07-27T06:34:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.allow_fallocate: 1 2024-07-27T06:34:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.allow_mmap_reads: 0 2024-07-27T06:34:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.allow_mmap_writes: 0 2024-07-27T06:34:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.use_direct_reads: 0 2024-07-27T06:34:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-27T06:34:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.create_missing_column_families: 0 2024-07-27T06:34:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.db_log_dir: 2024-07-27T06:34:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.wal_dir: 2024-07-27T06:34:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.advise_random_on_open: 1 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.db_write_buffer_size: 0 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.write_buffer_manager: 0x55b22fcc95e0 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.rate_limiter: (nil) 2024-07-27T06:34:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-27T06:34:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.wal_recovery_mode: 2 2024-07-27T06:34:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.enable_thread_tracking: 0 2024-07-27T06:34:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.enable_pipelined_write: 0 2024-07-27T06:34:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.unordered_write: 0 2024-07-27T06:34:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-27T06:34:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-27T06:34:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-27T06:34:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-27T06:34:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.row_cache: None 2024-07-27T06:34:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.wal_filter: None 2024-07-27T06:34:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.allow_ingest_behind: 0 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.two_write_queues: 0 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.manual_wal_flush: 0 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.wal_compression: 0 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.atomic_flush: 0 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.log_readahead_size: 0 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.best_efforts_recovery: 0 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-27T06:34:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.allow_data_in_errors: 0 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.db_host_id: __hostname__ 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_background_jobs: 2 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_background_compactions: -1 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_subcompactions: 1 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_total_wal_size: 0 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-27T06:34:02.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_open_files: -1 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bytes_per_sync: 0 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_readahead_size: 0 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_background_flushes: -1 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Compression algorithms supported: 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: kZSTD supported: 0 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: kXpressCompression supported: 0 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: kBZip2Compression supported: 0 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: kLZ4Compression supported: 1 2024-07-27T06:34:02.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: kZlibCompression supported: 1 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: kLZ4HCCompression supported: 1 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: kSnappyCompression supported: 1 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.merge_operator: 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_filter: None 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_filter_factory: None 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.sst_partitioner_factory: None 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-27T06:34:02.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b22fcc4c80) 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_type: 0 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_index_type: 0 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_shortening: 1 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: checksum: 4 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: no_block_cache: 0 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache: 0x55b22bc89610 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-07-27T06:34:02.175 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_options: 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: capacity : 536870912 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_shard_bits : 4 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: strict_capacity_limit : 0 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_compressed: (nil) 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: persistent_cache: (nil) 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size: 4096 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size_deviation: 10 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_restart_interval: 16 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_block_restart_interval: 1 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: metadata_block_size: 4096 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: partition_filters: 0 2024-07-27T06:34:02.176 INFO:journalctl@ceph.mon.a.smithi029.stdout: use_delta_encoding: 1 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout: filter_policy: bloomfilter 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout: whole_key_filtering: 1 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout: verify_compression: 0 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout: format_version: 5 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_index_compression: 1 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_align: 0 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout: max_auto_readahead_size: 262144 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout: prepopulate_block_cache: 0 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout: initial_auto_readahead_size: 8192 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.write_buffer_size: 33554432 2024-07-27T06:34:02.177 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_write_buffer_number: 2 2024-07-27T06:34:02.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compression: NoCompression 2024-07-27T06:34:02.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bottommost_compression: Disabled 2024-07-27T06:34:02.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.prefix_extractor: nullptr 2024-07-27T06:34:02.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-27T06:34:02.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.num_levels: 7 2024-07-27T06:34:02.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-27T06:34:02.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-27T06:34:02.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-27T06:34:02.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-27T06:34:02.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-27T06:34:02.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-27T06:34:02.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compression_opts.level: 32767 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compression_opts.strategy: 0 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-27T06:34:02.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compression_opts.enabled: false 2024-07-27T06:34:02.180 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-27T06:34:02.180 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-27T06:34:02.180 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-27T06:34:02.180 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.target_file_size_base: 67108864 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-27T06:34:02.181 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.arena_block_size: 1048576 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.disable_auto_compactions: 0 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-27T06:34:02.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-27T06:34:02.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-27T06:34:02.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-27T06:34:02.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-27T06:34:02.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-27T06:34:02.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.inplace_update_support: 0 2024-07-27T06:34:02.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-27T06:34:02.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-27T06:34:02.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-27T06:34:02.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-27T06:34:02.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.bloom_locality: 0 2024-07-27T06:34:02.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.max_successive_merges: 0 2024-07-27T06:34:02.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.paranoid_file_checks: 0 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.force_consistency_checks: 1 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.report_bg_io_stats: 0 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.ttl: 2592000 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.enable_blob_files: false 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.min_blob_size: 0 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.blob_file_size: 268435456 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-27T06:34:02.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.blob_file_starting_level: 0 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3d773c2f-8c60-4919-ada1-28286d977b0c 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: EVENT_LOG_v1 {"time_micros": 1722062042160211, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: EVENT_LOG_v1 {"time_micros": 1722062042161401, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 91126, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 282, "table_properties": {"data_size": 89174, "index_size": 272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11619, "raw_average_key_size": 48, "raw_value_size": 82648, "raw_average_value_size": 342, "num_data_blocks": 12, "num_entries": 241, "num_filter_entries": 241, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1722062042, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3d773c2f-8c60-4919-ada1-28286d977b0c", "db_session_id": "VG1HLNSBISLY1VC5TNAC", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: EVENT_LOG_v1 {"time_micros": 1722062042161505, "job": 1, "event": "recovery_finished"} 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-07-27T06:34:02.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b22fce6a80 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: DB pointer 0x55b230a86000 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** DB Stats ** 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-27T06:34:02.186 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-07-27T06:34:02.187 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-07-27T06:34:02.187 INFO:journalctl@ceph.mon.a.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-27T06:34:02.187 INFO:journalctl@ceph.mon.a.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-27T06:34:02.187 INFO:journalctl@ceph.mon.a.smithi029.stdout: L0 2/0 90.85 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-27T06:34:02.187 INFO:journalctl@ceph.mon.a.smithi029.stdout: Sum 2/0 90.85 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-27T06:34:02.187 INFO:journalctl@ceph.mon.a.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-27T06:34:02.187 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-07-27T06:34:02.187 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-07-27T06:34:02.187 INFO:journalctl@ceph.mon.a.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-27T06:34:02.187 INFO:journalctl@ceph.mon.a.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-27T06:34:02.187 INFO:journalctl@ceph.mon.a.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 120.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-27T06:34:02.187 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative compaction: 0.00 GB write, 24.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval compaction: 0.00 GB write, 24.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache BinnedLRUCache@0x55b22bc89610#2 capacity: 512.00 MB usage: 1.31 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-07-27T06:34:02.188 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.84 KB,0.000160933%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: starting mon.a rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: mon.a@-1(???) e1 preinit fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: mon.a@-1(???).mds e1 new map 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: mon.a@-1(???).mds e1 print_map 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout: e1 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout: btime 2024-07-27T06:33:57:535158+0000 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout: legacy client fscid: -1 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout: No filesystems configured 2024-07-27T06:34:02.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-27T06:34:02.190 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-27T06:34:02.190 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-27T06:34:02.190 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-27T06:34:02.190 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 ceph-mon[32610]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-07-27T06:34:02.260 INFO:teuthology.orchestra.run.smithi029.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-07-27T06:34:02.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 bash[32591]: 11b5d831c49ce76de506c067dddf0803dbc506b464c9e52db3e353e44494004d 2024-07-27T06:34:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:02 smithi029 systemd[1]: Started Ceph mon.a for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:34:03.281 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote config to /etc/ceph/ceph.conf 2024-07-27T06:34:03.310 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-07-27T06:34:03.310 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mgr... 2024-07-27T06:34:03.311 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:9283 ... 2024-07-27T06:34:03.311 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:8765 ... 2024-07-27T06:34:03.404 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:03 smithi029 ceph-mon[32610]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-27T06:34:03.404 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:03 smithi029 ceph-mon[32610]: monmap epoch 1 2024-07-27T06:34:03.404 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:03 smithi029 ceph-mon[32610]: fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:34:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:03 smithi029 ceph-mon[32610]: last_changed 2024-07-27T06:33:54.422980+0000 2024-07-27T06:34:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:03 smithi029 ceph-mon[32610]: created 2024-07-27T06:33:54.422980+0000 2024-07-27T06:34:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:03 smithi029 ceph-mon[32610]: min_mon_release 19 (squid) 2024-07-27T06:34:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:03 smithi029 ceph-mon[32610]: election_strategy: 1 2024-07-27T06:34:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:03 smithi029 ceph-mon[32610]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-07-27T06:34:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:03 smithi029 ceph-mon[32610]: fsmap 2024-07-27T06:34:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:03 smithi029 ceph-mon[32610]: osdmap e1: 0 total, 0 up, 0 in 2024-07-27T06:34:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:03 smithi029 ceph-mon[32610]: mgrmap e1: no daemons active 2024-07-27T06:34:03.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:03 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2031182657' entity='client.admin' 2024-07-27T06:34:03.581 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mgr.x 2024-07-27T06:34:03.581 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mgr.x.service: Unit ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mgr.x.service not loaded. 2024-07-27T06:34:03.733 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968.target.wants/ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@.service. 2024-07-27T06:34:04.251 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-27T06:34:04.252 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-07-27T06:34:04.259 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-27T06:34:04.259 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-07-27T06:34:04.259 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-07-27T06:34:04.260 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-07-27T06:34:04.268 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-27T06:34:04.268 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-07-27T06:34:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-27T06:34:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-07-27T06:34:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-07-27T06:34:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-07-27T06:34:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr to start... 2024-07-27T06:34:04.277 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr... 2024-07-27T06:34:04.389 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:04 smithi029 systemd[1]: Started Ceph mgr.x for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:34:04.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:04 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/4063923275' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-27T06:34:05.303 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-07-27T06:34:05.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-07-27T06:34:05.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:34:05.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-07-27T06:34:05.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-27T06:34:05.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-27T06:34:05.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-27T06:34:05.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:05.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-27T06:34:05.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-27T06:34:05.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-07-27T06:34:05.304 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-07-27T06:34:05.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-27T06:34:05.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-07-27T06:34:05.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-07-27T06:34:05.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-07-27T06:34:05.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-27T06:34:05.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-27T06:34:05.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-27T06:34:05.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-27T06:34:05.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:05.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-27T06:34:05.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-27T06:34:05.305 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-27T06:34:05.306 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "btime": "2024-07-27T06:33:57:535158+0000", 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-27T06:34:05.307 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-27T06:34:05.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-27T06:34:05.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-07-27T06:34:05.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-07-27T06:34:05.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-27T06:34:05.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:05.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-27T06:34:05.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-27T06:34:05.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-07-27T06:33:57.535934+0000", 2024-07-27T06:34:05.308 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-27T06:34:05.309 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:05.309 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-27T06:34:05.309 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-07-27T06:34:05.309 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (1/15)... 2024-07-27T06:34:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: Activating manager daemon x 2024-07-27T06:34:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: mgrmap e2: x(active, starting, since 0.00383958s) 2024-07-27T06:34:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: from='mgr.14100 172.21.15.29:0/3957526483' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-27T06:34:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: from='mgr.14100 172.21.15.29:0/3957526483' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-27T06:34:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: from='mgr.14100 172.21.15.29:0/3957526483' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-27T06:34:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: from='mgr.14100 172.21.15.29:0/3957526483' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:34:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: from='mgr.14100 172.21.15.29:0/3957526483' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-27T06:34:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: Manager daemon x is now available 2024-07-27T06:34:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: from='mgr.14100 172.21.15.29:0/3957526483' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-27T06:34:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: from='mgr.14100 172.21.15.29:0/3957526483' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-27T06:34:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: from='mgr.14100 172.21.15.29:0/3957526483' entity='mgr.x' 2024-07-27T06:34:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: from='mgr.14100 172.21.15.29:0/3957526483' entity='mgr.x' 2024-07-27T06:34:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:09 smithi029 ceph-mon[32610]: from='mgr.14100 172.21.15.29:0/3957526483' entity='mgr.x' 2024-07-27T06:34:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:11 smithi029 ceph-mon[32610]: mgrmap e3: x(active, since 1.00909s) 2024-07-27T06:34:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:12 smithi029 ceph-mon[32610]: mgrmap e4: x(active, since 2s) 2024-07-27T06:34:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:13 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1208419074' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-27T06:34:23.531 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-07-27T06:34:23.532 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-07-27T06:34:23.532 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:34:23.532 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-07-27T06:34:23.532 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-27T06:34:23.532 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-27T06:34:23.532 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-27T06:34:23.532 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:23.532 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-27T06:34:23.532 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-27T06:34:23.532 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-07-27T06:34:23.532 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-27T06:34:23.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-27T06:34:23.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-27T06:34:23.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-27T06:34:23.537 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-27T06:34:23.538 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "btime": "2024-07-27T06:33:57:535158+0000", 2024-07-27T06:34:23.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-27T06:34:23.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-27T06:34:23.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:23.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-27T06:34:23.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-27T06:34:23.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-27T06:34:23.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-27T06:34:23.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-27T06:34:23.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-27T06:34:23.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-07-27T06:34:23.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-07-27T06:34:23.539 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-27T06:34:23.540 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:23.540 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-27T06:34:23.540 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-27T06:34:23.540 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-07-27T06:33:57.535934+0000", 2024-07-27T06:34:23.540 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-27T06:34:23.540 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-07-27T06:34:23.540 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-27T06:34:23.540 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-07-27T06:34:23.540 INFO:teuthology.orchestra.run.smithi029.stdout:mgr is available 2024-07-27T06:34:28.188 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:27 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/527226375' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-27T06:34:28.452 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-07-27T06:34:28.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-07-27T06:34:28.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:34:28.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-27T06:34:28.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-07-27T06:34:28.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-27T06:34:28.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-27T06:34:28.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-27T06:34:28.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-27T06:34:28.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-07-27T06:34:28.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-27T06:34:28.454 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-27T06:34:28.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-07-27T06:34:28.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-07-27T06:34:28.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-27T06:34:28.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-27T06:34:28.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-27T06:34:28.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-27T06:34:28.455 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-27T06:34:28.455 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling cephadm module... 2024-07-27T06:34:29.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:29 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3550962386' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-07-27T06:34:29.462 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:29 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: ignoring --setuser ceph since I am not root 2024-07-27T06:34:29.462 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:29 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: ignoring --setgroup ceph since I am not root 2024-07-27T06:34:30.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:30 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3550962386' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-07-27T06:34:30.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:30 smithi029 ceph-mon[32610]: mgrmap e5: x(active, since 19s) 2024-07-27T06:34:31.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-07-27T06:34:31.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-07-27T06:34:31.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-27T06:34:31.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-27T06:34:31.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-27T06:34:31.008 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-07-27T06:34:31.008 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-07-27T06:34:31.009 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 5... 2024-07-27T06:34:31.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:31 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2893461654' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-27T06:34:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: Active manager daemon x restarted 2024-07-27T06:34:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: Activating manager daemon x 2024-07-27T06:34:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: osdmap e2: 0 total, 0 up, 0 in 2024-07-27T06:34:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: mgrmap e6: x(active, starting, since 0.00401017s) 2024-07-27T06:34:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:34:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-27T06:34:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-27T06:34:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-27T06:34:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-27T06:34:34.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: Manager daemon x is now available 2024-07-27T06:34:34.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:34.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:34.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:34:34.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:34:34.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-27T06:34:34.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:34 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-27T06:34:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:35 smithi029 ceph-mon[32610]: Found migration_current of "None". Setting to last migration. 2024-07-27T06:34:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:35 smithi029 ceph-mon[32610]: mgrmap e7: x(active, since 1.0061s) 2024-07-27T06:34:36.048 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-07-27T06:34:36.049 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-07-27T06:34:36.049 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-27T06:34:36.049 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-07-27T06:34:36.049 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 5 is available 2024-07-27T06:34:36.049 INFO:teuthology.orchestra.run.smithi029.stdout:Setting orchestrator backend to cephadm... 2024-07-27T06:34:36.957 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:36 smithi029 ceph-mon[32610]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-27T06:34:36.957 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:36 smithi029 ceph-mon[32610]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-27T06:34:36.957 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:36 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:36.957 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:36 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:36.958 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:36 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:36.958 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:36 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:34:37.642 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:34:37.363+0000 7fe6d9aa6640 -1 log_channel(cephadm) log [ERR] : [27/Jul/2024:06:34:37] ENGINE Error in HTTPServer.serve 2024-07-27T06:34:37.642 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: Traceback (most recent call last): 2024-07-27T06:34:37.642 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-27T06:34:37.642 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: self._connections.run(self.expiration_interval) 2024-07-27T06:34:37.643 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-27T06:34:37.643 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: self._run(expiration_interval) 2024-07-27T06:34:37.643 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-27T06:34:37.643 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: new_conn = self._from_server_socket(self.server.socket) 2024-07-27T06:34:37.643 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-27T06:34:37.643 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-27T06:34:37.643 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-27T06:34:37.643 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: s = self.context.wrap_socket( 2024-07-27T06:34:37.643 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-27T06:34:37.643 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: return self.sslsocket_class._create( 2024-07-27T06:34:37.643 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-27T06:34:37.643 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: self.do_handshake() 2024-07-27T06:34:37.644 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-27T06:34:37.644 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: self._sslobj.do_handshake() 2024-07-27T06:34:37.644 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-27T06:34:37.644 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:34:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-mon[32610]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:34:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-mon[32610]: mgrmap e8: x(active, since 2s) 2024-07-27T06:34:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:37 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:34:38.378 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout value unchanged 2024-07-27T06:34:38.378 INFO:teuthology.orchestra.run.smithi029.stdout:Generating ssh key... 2024-07-27T06:34:38.757 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:38 smithi029 ceph-mon[32610]: [27/Jul/2024:06:34:37] ENGINE Bus STARTING 2024-07-27T06:34:38.757 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:38 smithi029 ceph-mon[32610]: [27/Jul/2024:06:34:37] ENGINE Serving on https://172.21.15.29:7150 2024-07-27T06:34:38.758 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:38 smithi029 ceph-mon[32610]: [27/Jul/2024:06:34:37] ENGINE Error in HTTPServer.serve 2024-07-27T06:34:38.758 INFO:journalctl@ceph.mon.a.smithi029.stdout: Traceback (most recent call last): 2024-07-27T06:34:38.758 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-27T06:34:38.758 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._connections.run(self.expiration_interval) 2024-07-27T06:34:38.758 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-27T06:34:38.758 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._run(expiration_interval) 2024-07-27T06:34:38.758 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-27T06:34:38.758 INFO:journalctl@ceph.mon.a.smithi029.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-27T06:34:38.758 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-27T06:34:38.758 INFO:journalctl@ceph.mon.a.smithi029.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-27T06:34:38.759 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-27T06:34:38.759 INFO:journalctl@ceph.mon.a.smithi029.stdout: s = self.context.wrap_socket( 2024-07-27T06:34:38.759 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-27T06:34:38.759 INFO:journalctl@ceph.mon.a.smithi029.stdout: return self.sslsocket_class._create( 2024-07-27T06:34:38.759 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-27T06:34:38.759 INFO:journalctl@ceph.mon.a.smithi029.stdout: self.do_handshake() 2024-07-27T06:34:38.759 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-27T06:34:38.759 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._sslobj.do_handshake() 2024-07-27T06:34:38.759 INFO:journalctl@ceph.mon.a.smithi029.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-27T06:34:38.759 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:38 smithi029 ceph-mon[32610]: [27/Jul/2024:06:34:37] ENGINE Serving on http://172.21.15.29:8765 2024-07-27T06:34:38.759 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:38 smithi029 ceph-mon[32610]: [27/Jul/2024:06:34:37] ENGINE Bus STARTED 2024-07-27T06:34:38.760 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:38 smithi029 ceph-mon[32610]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:34:39.428 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: Generating public/private rsa key pair. 2024-07-27T06:34:39.429 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: Your identification has been saved in /tmp/tmpz1hnds_3/key 2024-07-27T06:34:39.429 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: Your public key has been saved in /tmp/tmpz1hnds_3/key.pub 2024-07-27T06:34:39.429 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: The key fingerprint is: 2024-07-27T06:34:39.429 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: SHA256:cqnJnyiUdIJplwtlDfYoivpBkyqfs9oKN6SqRs5Vyco ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:34:39.429 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: The key's randomart image is: 2024-07-27T06:34:39.429 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: +---[RSA 3072]----+ 2024-07-27T06:34:39.429 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: | oo | 2024-07-27T06:34:39.429 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: | .oo. | 2024-07-27T06:34:39.429 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: | .=o.o | 2024-07-27T06:34:39.429 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: |..=o=+. . | 2024-07-27T06:34:39.429 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: |o.*+o=. S | 2024-07-27T06:34:39.429 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: |.* E+. = | 2024-07-27T06:34:39.430 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: |X =. + | 2024-07-27T06:34:39.430 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: |+O.+. o . | 2024-07-27T06:34:39.430 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: |Bo*o .. o | 2024-07-27T06:34:39.430 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:39 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: +----[SHA256]-----+ 2024-07-27T06:34:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:40 smithi029 ceph-mon[32610]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:34:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:40 smithi029 ceph-mon[32610]: Generating ssh key... 2024-07-27T06:34:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:40 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:40 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:40.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMmjlnDqP7gkPiz+oBOffoDkyx+68axHHDWnHREaxf7NdlavvlJvAZ/w2UdJchhJnq/SGWb6dwEmMVwOiHyJuy9VmiZxeeKCimeBVX1hearH0Mk1qKn6yMSepm6CpnbwURX1qh2vHgU7pTPO710x5obOAQAJQ7vMqULjqfw07+sfSrOaCduaL3TVdg5qdoQvxgDwnJIGXPagxksbISP8m98eOVZ3Y5GZbC0imvQDeZjsAn6fFT/+NJGUVgcIdEyW6WopJynq3Hf0Y6zJg+UGCfkrb/z+bBefkEP6kewO+pxY+dS5YISw9DVFXng8J7N91UqdChmiQ+udxtg86+GLULLT6ll0xXTSNPsAIPCd9oWFlzqtr6NmoYGOF8dlm0Q+GC+8xRn7Grlhdk2qGq5ZRU90/KwpEjFkJpz7e2Eqhr2oy/olMBJyFqyIqB5qO0NoGdKQOGw7RC1tXP32Ut2ZU+VE99FBukC2lQ11f9bbZuzq4Rn/nH3+3FriIUgRYKRa8= ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:34:40.840 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-07-27T06:34:40.840 INFO:teuthology.orchestra.run.smithi029.stdout:Adding key to root@localhost authorized_keys... 2024-07-27T06:34:40.841 INFO:teuthology.orchestra.run.smithi029.stdout:Adding host smithi029... 2024-07-27T06:34:41.622 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:41 smithi029 ceph-mon[32610]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:34:42.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:42 smithi029 ceph-mon[32610]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:34:43.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:43 smithi029 ceph-mon[32610]: Deploying cephadm binary to smithi029 2024-07-27T06:34:45.884 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Added host 'smithi029' with addr '172.21.15.29' 2024-07-27T06:34:45.884 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mon service... 2024-07-27T06:34:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:46 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:46 smithi029 ceph-mon[32610]: Added host smithi029 2024-07-27T06:34:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:46 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:34:47.338 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-07-27T06:34:47.338 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mgr service... 2024-07-27T06:34:47.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:47 smithi029 ceph-mon[32610]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:34:47.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:47 smithi029 ceph-mon[32610]: Saving service mon spec with placement count:5 2024-07-27T06:34:47.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:47 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:48.572 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-07-27T06:34:49.330 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:48 smithi029 ceph-mon[32610]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:34:49.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:48 smithi029 ceph-mon[32610]: Saving service mgr spec with placement count:2 2024-07-27T06:34:49.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:48 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:50.546 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:50 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2044375654' entity='client.admin' 2024-07-27T06:34:50.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:50 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:50.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:50 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:52.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:51 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1714921920' entity='client.admin' 2024-07-27T06:34:52.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:51 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:52.003 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling the dashboard module... 2024-07-27T06:34:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:52 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2020260591' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-07-27T06:34:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:52 smithi029 ceph-mon[32610]: from='mgr.14116 172.21.15.29:0/4115397936' entity='mgr.x' 2024-07-27T06:34:53.212 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: ignoring --setuser ceph since I am not root 2024-07-27T06:34:53.212 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:34:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: ignoring --setgroup ceph since I am not root 2024-07-27T06:34:54.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:53 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2020260591' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-07-27T06:34:54.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:53 smithi029 ceph-mon[32610]: mgrmap e9: x(active, since 18s) 2024-07-27T06:34:54.771 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-07-27T06:34:54.772 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-07-27T06:34:54.772 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-27T06:34:54.772 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-27T06:34:54.772 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-27T06:34:54.772 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-07-27T06:34:54.772 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-07-27T06:34:54.772 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 9... 2024-07-27T06:34:55.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:54 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3163771635' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-27T06:34:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: Active manager daemon x restarted 2024-07-27T06:34:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: Activating manager daemon x 2024-07-27T06:34:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: osdmap e3: 0 total, 0 up, 0 in 2024-07-27T06:34:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: mgrmap e10: x(active, starting, since 0.00355054s) 2024-07-27T06:34:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:34:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-27T06:34:58.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-27T06:34:58.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-27T06:34:58.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-27T06:34:58.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: Manager daemon x is now available 2024-07-27T06:34:58.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:34:58.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-27T06:34:58.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:34:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-27T06:35:00.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-07-27T06:35:00.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-07-27T06:35:00.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-27T06:35:00.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-07-27T06:35:00.171 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 9 is available 2024-07-27T06:35:00.171 INFO:teuthology.orchestra.run.smithi029.stdout:Generating a dashboard self-signed certificate... 2024-07-27T06:35:00.541 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-mon[32610]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-27T06:35:00.541 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-mon[32610]: mgrmap e11: x(active, since 1.00701s) 2024-07-27T06:35:00.541 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-mon[32610]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-27T06:35:00.541 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:01.212 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:35:00.850+0000 7fc5ad219640 -1 log_channel(cephadm) log [ERR] : [27/Jul/2024:06:35:00] ENGINE Error in HTTPServer.serve 2024-07-27T06:35:01.213 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: Traceback (most recent call last): 2024-07-27T06:35:01.213 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-27T06:35:01.213 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: self._connections.run(self.expiration_interval) 2024-07-27T06:35:01.213 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-27T06:35:01.213 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: self._run(expiration_interval) 2024-07-27T06:35:01.213 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-27T06:35:01.213 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: new_conn = self._from_server_socket(self.server.socket) 2024-07-27T06:35:01.213 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-27T06:35:01.213 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-27T06:35:01.213 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-27T06:35:01.213 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: s = self.context.wrap_socket( 2024-07-27T06:35:01.213 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-27T06:35:01.214 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: return self.sslsocket_class._create( 2024-07-27T06:35:01.214 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-27T06:35:01.214 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: self.do_handshake() 2024-07-27T06:35:01.214 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-27T06:35:01.214 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: self._sslobj.do_handshake() 2024-07-27T06:35:01.214 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-27T06:35:01.214 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:35:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:35:01.478 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-07-27T06:35:01.478 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial admin user... 2024-07-27T06:35:01.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:01 smithi029 ceph-mon[32610]: [27/Jul/2024:06:35:00] ENGINE Bus STARTING 2024-07-27T06:35:01.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:01 smithi029 ceph-mon[32610]: mgrmap e12: x(active, since 2s) 2024-07-27T06:35:01.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:01 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:01.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:01 smithi029 ceph-mon[32610]: [27/Jul/2024:06:35:00] ENGINE Serving on http://172.21.15.29:8765 2024-07-27T06:35:01.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:01 smithi029 ceph-mon[32610]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:35:01.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:01 smithi029 ceph-mon[32610]: [27/Jul/2024:06:35:00] ENGINE Serving on https://172.21.15.29:7150 2024-07-27T06:35:01.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:01 smithi029 ceph-mon[32610]: [27/Jul/2024:06:35:00] ENGINE Bus STARTED 2024-07-27T06:35:01.858 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:01 smithi029 ceph-mon[32610]: [27/Jul/2024:06:35:00] ENGINE Error in HTTPServer.serve 2024-07-27T06:35:01.858 INFO:journalctl@ceph.mon.a.smithi029.stdout: Traceback (most recent call last): 2024-07-27T06:35:01.858 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-27T06:35:01.858 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._connections.run(self.expiration_interval) 2024-07-27T06:35:01.858 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-27T06:35:01.858 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._run(expiration_interval) 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: s = self.context.wrap_socket( 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: return self.sslsocket_class._create( 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: self.do_handshake() 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._sslobj.do_handshake() 2024-07-27T06:35:01.859 INFO:journalctl@ceph.mon.a.smithi029.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-27T06:35:01.860 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:01 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:01.860 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:01 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:03.020 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$yTKfPX8bFFOCwXnngr5iuuK05CF6q4wENeBBH2s3lCZBLGDpz1Bva", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1722062102, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-07-27T06:35:03.020 INFO:teuthology.orchestra.run.smithi029.stdout:Fetching dashboard port number... 2024-07-27T06:35:03.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:03 smithi029 ceph-mon[32610]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:35:03.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:03 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:04.592 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:04 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3458603424' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-07-27T06:35:04.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 8443 2024-07-27T06:35:04.852 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-27T06:35:04.852 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-07-27T06:35:04.861 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-27T06:35:04.861 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-07-27T06:35:04.861 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-07-27T06:35:04.861 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-07-27T06:35:04.863 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph Dashboard is now available at: 2024-07-27T06:35:04.863 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:35:04.863 INFO:teuthology.orchestra.run.smithi029.stdout: URL: https://smithi029.front.sepia.ceph.com:8443/ 2024-07-27T06:35:04.863 INFO:teuthology.orchestra.run.smithi029.stdout: User: admin 2024-07-27T06:35:04.863 INFO:teuthology.orchestra.run.smithi029.stdout: Password: va0rxg5mwk 2024-07-27T06:35:04.863 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:35:04.864 INFO:teuthology.orchestra.run.smithi029.stdout:Saving cluster configuration to /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config directory 2024-07-27T06:35:05.598 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:05 smithi029 ceph-mon[32610]: mgrmap e13: x(active, since 6s) 2024-07-27T06:35:06.162 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-07-27T06:35:06.162 INFO:teuthology.orchestra.run.smithi029.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-07-27T06:35:06.162 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:35:06.162 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-07-27T06:35:06.162 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:35:06.162 INFO:teuthology.orchestra.run.smithi029.stdout:Or, if you are only running a single cluster on this host: 2024-07-27T06:35:06.162 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:35:06.163 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-07-27T06:35:06.163 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:35:06.163 INFO:teuthology.orchestra.run.smithi029.stdout:Please consider enabling telemetry to help improve Ceph: 2024-07-27T06:35:06.163 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:35:06.163 INFO:teuthology.orchestra.run.smithi029.stdout: ceph telemetry on 2024-07-27T06:35:06.163 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:35:06.163 INFO:teuthology.orchestra.run.smithi029.stdout:For more information see: 2024-07-27T06:35:06.163 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:35:06.163 INFO:teuthology.orchestra.run.smithi029.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-07-27T06:35:06.163 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:35:06.163 INFO:teuthology.orchestra.run.smithi029.stdout:Bootstrap complete. 2024-07-27T06:35:06.209 INFO:tasks.cephadm:Fetching config... 2024-07-27T06:35:06.210 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:35:06.210 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-07-27T06:35:06.227 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-07-27T06:35:06.227 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:35:06.227 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-07-27T06:35:06.284 INFO:tasks.cephadm:Fetching mon keyring... 2024-07-27T06:35:06.284 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:35:06.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/keyring of=/dev/stdout 2024-07-27T06:35:06.351 INFO:tasks.cephadm:Fetching pub ssh key... 2024-07-27T06:35:06.351 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:35:06.351 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-07-27T06:35:06.408 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-07-27T06:35:06.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMmjlnDqP7gkPiz+oBOffoDkyx+68axHHDWnHREaxf7NdlavvlJvAZ/w2UdJchhJnq/SGWb6dwEmMVwOiHyJuy9VmiZxeeKCimeBVX1hearH0Mk1qKn6yMSepm6CpnbwURX1qh2vHgU7pTPO710x5obOAQAJQ7vMqULjqfw07+sfSrOaCduaL3TVdg5qdoQvxgDwnJIGXPagxksbISP8m98eOVZ3Y5GZbC0imvQDeZjsAn6fFT/+NJGUVgcIdEyW6WopJynq3Hf0Y6zJg+UGCfkrb/z+bBefkEP6kewO+pxY+dS5YISw9DVFXng8J7N91UqdChmiQ+udxtg86+GLULLT6ll0xXTSNPsAIPCd9oWFlzqtr6NmoYGOF8dlm0Q+GC+8xRn7Grlhdk2qGq5ZRU90/KwpEjFkJpz7e2Eqhr2oy/olMBJyFqyIqB5qO0NoGdKQOGw7RC1tXP32Ut2ZU+VE99FBukC2lQ11f9bbZuzq4Rn/nH3+3FriIUgRYKRa8= ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-27T06:35:06.487 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMmjlnDqP7gkPiz+oBOffoDkyx+68axHHDWnHREaxf7NdlavvlJvAZ/w2UdJchhJnq/SGWb6dwEmMVwOiHyJuy9VmiZxeeKCimeBVX1hearH0Mk1qKn6yMSepm6CpnbwURX1qh2vHgU7pTPO710x5obOAQAJQ7vMqULjqfw07+sfSrOaCduaL3TVdg5qdoQvxgDwnJIGXPagxksbISP8m98eOVZ3Y5GZbC0imvQDeZjsAn6fFT/+NJGUVgcIdEyW6WopJynq3Hf0Y6zJg+UGCfkrb/z+bBefkEP6kewO+pxY+dS5YISw9DVFXng8J7N91UqdChmiQ+udxtg86+GLULLT6ll0xXTSNPsAIPCd9oWFlzqtr6NmoYGOF8dlm0Q+GC+8xRn7Grlhdk2qGq5ZRU90/KwpEjFkJpz7e2Eqhr2oy/olMBJyFqyIqB5qO0NoGdKQOGw7RC1tXP32Ut2ZU+VE99FBukC2lQ11f9bbZuzq4Rn/nH3+3FriIUgRYKRa8= ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:35:06.499 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMmjlnDqP7gkPiz+oBOffoDkyx+68axHHDWnHREaxf7NdlavvlJvAZ/w2UdJchhJnq/SGWb6dwEmMVwOiHyJuy9VmiZxeeKCimeBVX1hearH0Mk1qKn6yMSepm6CpnbwURX1qh2vHgU7pTPO710x5obOAQAJQ7vMqULjqfw07+sfSrOaCduaL3TVdg5qdoQvxgDwnJIGXPagxksbISP8m98eOVZ3Y5GZbC0imvQDeZjsAn6fFT/+NJGUVgcIdEyW6WopJynq3Hf0Y6zJg+UGCfkrb/z+bBefkEP6kewO+pxY+dS5YISw9DVFXng8J7N91UqdChmiQ+udxtg86+GLULLT6ll0xXTSNPsAIPCd9oWFlzqtr6NmoYGOF8dlm0Q+GC+8xRn7Grlhdk2qGq5ZRU90/KwpEjFkJpz7e2Eqhr2oy/olMBJyFqyIqB5qO0NoGdKQOGw7RC1tXP32Ut2ZU+VE99FBukC2lQ11f9bbZuzq4Rn/nH3+3FriIUgRYKRa8= ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-27T06:35:06.539 INFO:teuthology.orchestra.run.smithi047.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMmjlnDqP7gkPiz+oBOffoDkyx+68axHHDWnHREaxf7NdlavvlJvAZ/w2UdJchhJnq/SGWb6dwEmMVwOiHyJuy9VmiZxeeKCimeBVX1hearH0Mk1qKn6yMSepm6CpnbwURX1qh2vHgU7pTPO710x5obOAQAJQ7vMqULjqfw07+sfSrOaCduaL3TVdg5qdoQvxgDwnJIGXPagxksbISP8m98eOVZ3Y5GZbC0imvQDeZjsAn6fFT/+NJGUVgcIdEyW6WopJynq3Hf0Y6zJg+UGCfkrb/z+bBefkEP6kewO+pxY+dS5YISw9DVFXng8J7N91UqdChmiQ+udxtg86+GLULLT6ll0xXTSNPsAIPCd9oWFlzqtr6NmoYGOF8dlm0Q+GC+8xRn7Grlhdk2qGq5ZRU90/KwpEjFkJpz7e2Eqhr2oy/olMBJyFqyIqB5qO0NoGdKQOGw7RC1tXP32Ut2ZU+VE99FBukC2lQ11f9bbZuzq4Rn/nH3+3FriIUgRYKRa8= ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:35:06.551 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMmjlnDqP7gkPiz+oBOffoDkyx+68axHHDWnHREaxf7NdlavvlJvAZ/w2UdJchhJnq/SGWb6dwEmMVwOiHyJuy9VmiZxeeKCimeBVX1hearH0Mk1qKn6yMSepm6CpnbwURX1qh2vHgU7pTPO710x5obOAQAJQ7vMqULjqfw07+sfSrOaCduaL3TVdg5qdoQvxgDwnJIGXPagxksbISP8m98eOVZ3Y5GZbC0imvQDeZjsAn6fFT/+NJGUVgcIdEyW6WopJynq3Hf0Y6zJg+UGCfkrb/z+bBefkEP6kewO+pxY+dS5YISw9DVFXng8J7N91UqdChmiQ+udxtg86+GLULLT6ll0xXTSNPsAIPCd9oWFlzqtr6NmoYGOF8dlm0Q+GC+8xRn7Grlhdk2qGq5ZRU90/KwpEjFkJpz7e2Eqhr2oy/olMBJyFqyIqB5qO0NoGdKQOGw7RC1tXP32Ut2ZU+VE99FBukC2lQ11f9bbZuzq4Rn/nH3+3FriIUgRYKRa8= ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-27T06:35:06.590 INFO:teuthology.orchestra.run.smithi077.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMmjlnDqP7gkPiz+oBOffoDkyx+68axHHDWnHREaxf7NdlavvlJvAZ/w2UdJchhJnq/SGWb6dwEmMVwOiHyJuy9VmiZxeeKCimeBVX1hearH0Mk1qKn6yMSepm6CpnbwURX1qh2vHgU7pTPO710x5obOAQAJQ7vMqULjqfw07+sfSrOaCduaL3TVdg5qdoQvxgDwnJIGXPagxksbISP8m98eOVZ3Y5GZbC0imvQDeZjsAn6fFT/+NJGUVgcIdEyW6WopJynq3Hf0Y6zJg+UGCfkrb/z+bBefkEP6kewO+pxY+dS5YISw9DVFXng8J7N91UqdChmiQ+udxtg86+GLULLT6ll0xXTSNPsAIPCd9oWFlzqtr6NmoYGOF8dlm0Q+GC+8xRn7Grlhdk2qGq5ZRU90/KwpEjFkJpz7e2Eqhr2oy/olMBJyFqyIqB5qO0NoGdKQOGw7RC1tXP32Ut2ZU+VE99FBukC2lQ11f9bbZuzq4Rn/nH3+3FriIUgRYKRa8= ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:35:06.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-07-27T06:35:06.726 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:06 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1102548832' entity='client.admin' 2024-07-27T06:35:06.865 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:35:10.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:10 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2585624605' entity='client.admin' 2024-07-27T06:35:10.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:10 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:10.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:10 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:10.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:10 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:35:10.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:10 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:10.446 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:10 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:35:10.446 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:10 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:10.474 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-07-27T06:35:10.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-07-27T06:35:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:35:13.161 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi047 2024-07-27T06:35:13.162 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-07-27T06:35:13.162 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/etc/ceph/ceph.conf 2024-07-27T06:35:13.182 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-07-27T06:35:13.182 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:35:13.240 INFO:tasks.cephadm:Adding host smithi047 to orchestrator... 2024-07-27T06:35:13.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch host add smithi047 2024-07-27T06:35:13.508 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:35:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:13 smithi029 ceph-mon[32610]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:35:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:13 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:13 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:35:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:13 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:13 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:35:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:13 smithi029 ceph-mon[32610]: Updating smithi029:/etc/ceph/ceph.conf 2024-07-27T06:35:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:13 smithi029 ceph-mon[32610]: Updating smithi029:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:13 smithi029 ceph-mon[32610]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:35:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:13 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:13 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:13.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:13 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:14.679 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:14 smithi029 ceph-mon[32610]: Updating smithi029:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:35:16.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:16 smithi029 ceph-mon[32610]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi047", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:35:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:17 smithi029 ceph-mon[32610]: Deploying cephadm binary to smithi047 2024-07-27T06:35:19.166 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi047' with addr '172.21.15.47' 2024-07-27T06:35:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:19 smithi029 ceph-mon[32610]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:19 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:19 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:35:19.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch host ls --format=json 2024-07-27T06:35:20.044 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:35:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:20 smithi029 ceph-mon[32610]: Added host smithi047 2024-07-27T06:35:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:20 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:20 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:21.596 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:21 smithi029 ceph-mon[32610]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:21.596 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:21 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:21.725 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:35:21.725 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.47", "hostname": "smithi047", "labels": [], "status": ""}] 2024-07-27T06:35:22.316 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi077 2024-07-27T06:35:22.317 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-07-27T06:35:22.317 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/etc/ceph/ceph.conf 2024-07-27T06:35:22.336 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-07-27T06:35:22.336 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:35:22.392 INFO:tasks.cephadm:Adding host smithi077 to orchestrator... 2024-07-27T06:35:22.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch host add smithi077 2024-07-27T06:35:22.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:22 smithi029 ceph-mon[32610]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:35:22.656 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:35:23.787 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:23 smithi029 ceph-mon[32610]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:23.788 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:23 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:23.788 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:23 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:23.788 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:23 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:23.788 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:23 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:23.788 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:23 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi047", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:35:23.788 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:23 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:23.788 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:23 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:35:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:24 smithi029 ceph-mon[32610]: Updating smithi047:/etc/ceph/ceph.conf 2024-07-27T06:35:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:24 smithi029 ceph-mon[32610]: Updating smithi047:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:24 smithi029 ceph-mon[32610]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:35:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:24 smithi029 ceph-mon[32610]: Updating smithi047:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:35:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:24 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:24 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:24 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:25 smithi029 ceph-mon[32610]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:25 smithi029 ceph-mon[32610]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:35:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:26 smithi029 ceph-mon[32610]: Deploying cephadm binary to smithi077 2024-07-27T06:35:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:27 smithi029 ceph-mon[32610]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:28.103 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi077' with addr '172.21.15.77' 2024-07-27T06:35:28.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch host ls --format=json 2024-07-27T06:35:28.955 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:35:29.330 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:29 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:29.330 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:29 smithi029 ceph-mon[32610]: Added host smithi077 2024-07-27T06:35:29.330 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:29 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:35:29.330 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:29 smithi029 ceph-mon[32610]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:29.330 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:29 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:29.330 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:29 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:30.722 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:35:30.723 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.47", "hostname": "smithi047", "labels": [], "status": ""}, {"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}] 2024-07-27T06:35:30.723 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:30 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:31.335 INFO:tasks.cephadm:Setting crush tunables to default 2024-07-27T06:35:31.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd crush tunables default 2024-07-27T06:35:31.594 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:35:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:31 smithi029 ceph-mon[32610]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:31 smithi029 ceph-mon[32610]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:35:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:35:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:35:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: Updating smithi077:/etc/ceph/ceph.conf 2024-07-27T06:35:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:35:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:35:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:33 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:34.225 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-07-27T06:35:34.534 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:34 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/431393186' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-27T06:35:34.847 INFO:tasks.cephadm:Adding mon.a on smithi029 2024-07-27T06:35:34.848 INFO:tasks.cephadm:Adding mon.b on smithi047 2024-07-27T06:35:34.848 INFO:tasks.cephadm:Adding mon.c on smithi077 2024-07-27T06:35:34.848 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch apply mon '3;smithi029:172.21.15.29=a;smithi047:172.21.15.47=b;smithi077:172.21.15.77=c' 2024-07-27T06:35:35.098 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:35 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/431393186' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-27T06:35:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:35 smithi029 ceph-mon[32610]: osdmap e4: 0 total, 0 up, 0 in 2024-07-27T06:35:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:35 smithi029 ceph-mon[32610]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:36.808 INFO:teuthology.orchestra.run.smithi077.stdout:Scheduled mon update... 2024-07-27T06:35:37.331 DEBUG:teuthology.orchestra.run.smithi047:mon.b> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mon.b.service 2024-07-27T06:35:37.334 DEBUG:teuthology.orchestra.run.smithi077:mon.c> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mon.c.service 2024-07-27T06:35:37.336 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-27T06:35:37.336 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph mon dump -f json 2024-07-27T06:35:38.102 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:37 smithi029 ceph-mon[32610]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:37 smithi029 ceph-mon[32610]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi047:172.21.15.47=b;smithi077:172.21.15.77=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:35:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:37 smithi029 ceph-mon[32610]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi047:172.21.15.47=b;smithi077:172.21.15.77=c;count:3 2024-07-27T06:35:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:37 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:37 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:35:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:37 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:37 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:35:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:37 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:38.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:37 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:35:38.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:37 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:38.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:37 smithi029 ceph-mon[32610]: Deploying daemon mon.c on smithi077 2024-07-27T06:35:40.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:39 smithi029 ceph-mon[32610]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:41.043 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:35:41.043 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":1,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","modified":"2024-07-27T06:33:54.422980Z","created":"2024-07-27T06:33:54.422980Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-07-27T06:35:41.043 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 1 2024-07-27T06:35:42.132 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: mon.c@-1(synchronizing).mds e1 print_map 2024-07-27T06:35:42.132 INFO:journalctl@ceph.mon.c.smithi077.stdout: e1 2024-07-27T06:35:42.132 INFO:journalctl@ceph.mon.c.smithi077.stdout: btime 2024-07-27T06:33:57:535158+0000 2024-07-27T06:35:42.132 INFO:journalctl@ceph.mon.c.smithi077.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-27T06:35:42.132 INFO:journalctl@ceph.mon.c.smithi077.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-27T06:35:42.132 INFO:journalctl@ceph.mon.c.smithi077.stdout: legacy client fscid: -1 2024-07-27T06:35:42.132 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-07-27T06:35:42.132 INFO:journalctl@ceph.mon.c.smithi077.stdout: No filesystems configured 2024-07-27T06:35:42.132 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-07-27T06:35:42.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:41 smithi029 ceph-mon[32610]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:41 smithi029 ceph-mon[32610]: from='client.? 172.21.15.77:0/445962937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-27T06:35:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-07-27T06:35:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-07-27T06:35:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-07-27T06:35:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-07-27T06:35:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-07-27T06:35:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-27T06:35:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-27T06:35:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-27T06:35:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-27T06:35:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: Deploying cephadm binary to smithi077 2024-07-27T06:35:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: Added host smithi077 2024-07-27T06:35:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:35:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:42.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:35:42.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:35:42.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:42.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:35:42.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: Updating smithi077:/etc/ceph/ceph.conf 2024-07-27T06:35:42.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:42.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:42.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:35:42.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:35:42.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/431393186' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-27T06:35:42.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/431393186' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-27T06:35:42.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: osdmap e4: 0 total, 0 up, 0 in 2024-07-27T06:35:42.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:42.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:42.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi047:172.21.15.47=b;smithi077:172.21.15.77=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:35:42.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi047:172.21.15.47=b;smithi077:172.21.15.77=c;count:3 2024-07-27T06:35:42.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:35:42.612 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:42.612 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:35:42.612 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:42.612 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:35:42.612 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:42.612 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: Deploying daemon mon.c on smithi077 2024-07-27T06:35:42.612 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:42.612 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:42.612 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: from='client.? 172.21.15.77:0/445962937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-27T06:35:42.613 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 ceph-mon[37153]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-07-27T06:35:42.613 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 bash[37135]: cee05e56bdd02d1e69785db74a487fcad1e5d060d52239193a321434340f55d1 2024-07-27T06:35:42.613 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:42 smithi077 systemd[1]: Started Ceph mon.c for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:35:42.710 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-27T06:35:42.711 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph mon dump -f json 2024-07-27T06:35:42.962 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.c/config 2024-07-27T06:35:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:35:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: mon.a calling monitor election 2024-07-27T06:35:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: mon.c calling monitor election 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: monmap epoch 2 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: last_changed 2024-07-27T06:35:42.137269+0000 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: created 2024-07-27T06:33:54.422980+0000 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: min_mon_release 19 (squid) 2024-07-27T06:35:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: election_strategy: 1 2024-07-27T06:35:47.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-07-27T06:35:47.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-07-27T06:35:47.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: fsmap 2024-07-27T06:35:47.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: osdmap e4: 0 total, 0 up, 0 in 2024-07-27T06:35:47.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: mgrmap e13: x(active, since 48s) 2024-07-27T06:35:47.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: overall HEALTH_OK 2024-07-27T06:35:47.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:47.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:47.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:47.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:35:47.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:47 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:35:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: mon.a calling monitor election 2024-07-27T06:35:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: mon.c calling monitor election 2024-07-27T06:35:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:47.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:47.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:47.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:47.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-27T06:35:47.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: monmap epoch 2 2024-07-27T06:35:47.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:35:47.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: last_changed 2024-07-27T06:35:42.137269+0000 2024-07-27T06:35:47.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: created 2024-07-27T06:33:54.422980+0000 2024-07-27T06:35:47.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: min_mon_release 19 (squid) 2024-07-27T06:35:47.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: election_strategy: 1 2024-07-27T06:35:47.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-07-27T06:35:47.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-07-27T06:35:47.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: fsmap 2024-07-27T06:35:47.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: osdmap e4: 0 total, 0 up, 0 in 2024-07-27T06:35:47.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: mgrmap e13: x(active, since 48s) 2024-07-27T06:35:47.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: overall HEALTH_OK 2024-07-27T06:35:47.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:47.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:47.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:47.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:35:47.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:47 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:47.734 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:35:47.734 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":2,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","modified":"2024-07-27T06:35:42.137269Z","created":"2024-07-27T06:33:54.422980Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-07-27T06:35:47.734 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 2 2024-07-27T06:35:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:48 smithi077 ceph-mon[37153]: Deploying daemon mon.b on smithi047 2024-07-27T06:35:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:48 smithi077 ceph-mon[37153]: from='client.? 172.21.15.77:0/52482692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-27T06:35:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:48 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:48 smithi029 ceph-mon[32610]: Deploying daemon mon.b on smithi047 2024-07-27T06:35:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:48 smithi029 ceph-mon[32610]: from='client.? 172.21.15.77:0/52482692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-27T06:35:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:48 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:49.255 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-27T06:35:49.255 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph mon dump -f json 2024-07-27T06:35:49.506 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.c/config 2024-07-27T06:35:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:49 smithi077 ceph-mon[37153]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:49.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:49 smithi029 ceph-mon[32610]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:51 smithi047 systemd[1]: Started Ceph mon.b for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:35:56.088 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-07-27T06:35:56.088 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":3,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","modified":"2024-07-27T06:35:51.075151Z","created":"2024-07-27T06:33:54.422980Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-07-27T06:35:56.089 INFO:teuthology.orchestra.run.smithi077.stderr:dumped monmap epoch 3 2024-07-27T06:35:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:35:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: mon.a calling monitor election 2024-07-27T06:35:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: mon.c calling monitor election 2024-07-27T06:35:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='client.? 172.21.15.77:0/1175539267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-27T06:35:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: mon.b calling monitor election 2024-07-27T06:35:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:56.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-27T06:35:56.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: monmap epoch 3 2024-07-27T06:35:56.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:35:56.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: last_changed 2024-07-27T06:35:51.075151+0000 2024-07-27T06:35:56.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: created 2024-07-27T06:33:54.422980+0000 2024-07-27T06:35:56.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: min_mon_release 19 (squid) 2024-07-27T06:35:56.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: election_strategy: 1 2024-07-27T06:35:56.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-07-27T06:35:56.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-07-27T06:35:56.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: 2: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.b 2024-07-27T06:35:56.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: fsmap 2024-07-27T06:35:56.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: osdmap e4: 0 total, 0 up, 0 in 2024-07-27T06:35:56.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: mgrmap e13: x(active, since 57s) 2024-07-27T06:35:56.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: overall HEALTH_OK 2024-07-27T06:35:56.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:56.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:56.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:56.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:56 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:35:56.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: mon.a calling monitor election 2024-07-27T06:35:56.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:56.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: mon.c calling monitor election 2024-07-27T06:35:56.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='client.? 172.21.15.77:0/1175539267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-27T06:35:56.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:56.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: mon.b calling monitor election 2024-07-27T06:35:56.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:56.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: monmap epoch 3 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: last_changed 2024-07-27T06:35:51.075151+0000 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: created 2024-07-27T06:33:54.422980+0000 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: min_mon_release 19 (squid) 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: election_strategy: 1 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: 2: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.b 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: fsmap 2024-07-27T06:35:56.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: osdmap e4: 0 total, 0 up, 0 in 2024-07-27T06:35:56.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: mgrmap e13: x(active, since 57s) 2024-07-27T06:35:56.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: overall HEALTH_OK 2024-07-27T06:35:56.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:56.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:56.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:56.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:56 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:35:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: mon.a calling monitor election 2024-07-27T06:35:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:35:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: mon.c calling monitor election 2024-07-27T06:35:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='client.? 172.21.15.77:0/1175539267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-27T06:35:56.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:56.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: mon.b calling monitor election 2024-07-27T06:35:56.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:56.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:56.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-27T06:35:56.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: monmap epoch 3 2024-07-27T06:35:56.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:35:56.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: last_changed 2024-07-27T06:35:51.075151+0000 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: created 2024-07-27T06:33:54.422980+0000 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: min_mon_release 19 (squid) 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: election_strategy: 1 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: 1: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.c 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: 2: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.b 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: fsmap 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: osdmap e4: 0 total, 0 up, 0 in 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: mgrmap e13: x(active, since 57s) 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: overall HEALTH_OK 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:56.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:56 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:56.651 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-07-27T06:35:56.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config generate-minimal-conf 2024-07-27T06:35:56.926 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:35:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:57 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:35:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:57 smithi077 ceph-mon[37153]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:57 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:57.377 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:57 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:35:57.377 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:57 smithi029 ceph-mon[32610]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:57.377 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:57 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:57 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:35:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:57 smithi047 ceph-mon[36835]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:57 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:35:58.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:58.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:35:58.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: Updating smithi029:/etc/ceph/ceph.conf 2024-07-27T06:35:58.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: Updating smithi047:/etc/ceph/ceph.conf 2024-07-27T06:35:58.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: Updating smithi077:/etc/ceph/ceph.conf 2024-07-27T06:35:58.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:58.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: Updating smithi047:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:58.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: Updating smithi029:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:58.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: Reconfiguring mon.a (unknown last config time)... 2024-07-27T06:35:58.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:35:58.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-27T06:35:58.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:58.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:58 smithi029 ceph-mon[32610]: Reconfiguring daemon mon.a on smithi029 2024-07-27T06:35:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:35:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: Updating smithi029:/etc/ceph/ceph.conf 2024-07-27T06:35:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: Updating smithi047:/etc/ceph/ceph.conf 2024-07-27T06:35:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: Updating smithi077:/etc/ceph/ceph.conf 2024-07-27T06:35:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:58.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: Updating smithi047:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:58.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: Updating smithi029:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:58.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: Reconfiguring mon.a (unknown last config time)... 2024-07-27T06:35:58.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:35:58.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-27T06:35:58.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:58.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:58 smithi077 ceph-mon[37153]: Reconfiguring daemon mon.a on smithi029 2024-07-27T06:35:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:35:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: Updating smithi029:/etc/ceph/ceph.conf 2024-07-27T06:35:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: Updating smithi047:/etc/ceph/ceph.conf 2024-07-27T06:35:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: Updating smithi077:/etc/ceph/ceph.conf 2024-07-27T06:35:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: Updating smithi047:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: Updating smithi029:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:35:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:58.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: Reconfiguring mon.a (unknown last config time)... 2024-07-27T06:35:58.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:35:58.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-27T06:35:58.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:58.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:58 smithi047 ceph-mon[36835]: Reconfiguring daemon mon.a on smithi029 2024-07-27T06:35:58.937 INFO:teuthology.orchestra.run.smithi029.stdout:# minimal ceph.conf for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:35:58.938 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-07-27T06:35:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:35:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: mon_host = [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] 2024-07-27T06:35:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:59 smithi077 ceph-mon[37153]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:59 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:35:59 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1766748486' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:59.674 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:59 smithi029 ceph-mon[32610]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:59.674 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:59 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:59.675 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:35:59 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1766748486' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:59 smithi047 ceph-mon[36835]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:35:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:59 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:35:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:35:59 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1766748486' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:35:59.706 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-07-27T06:35:59.706 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:35:59.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2024-07-27T06:35:59.737 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:35:59.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:35:59.804 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-07-27T06:35:59.804 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.conf 2024-07-27T06:35:59.837 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-07-27T06:35:59.837 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:35:59.903 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-07-27T06:35:59.903 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.conf 2024-07-27T06:35:59.934 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-07-27T06:35:59.935 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:36:00.003 INFO:tasks.cephadm:enabling module snap_schedule 2024-07-27T06:36:00.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- sudo ceph --cluster ceph mgr module enable snap_schedule 2024-07-27T06:36:00.271 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:01.431 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:01 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:36:01.432 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:01 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:36:01.432 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:01 smithi029 ceph-mon[32610]: Reconfiguring mon.b (monmap changed)... 2024-07-27T06:36:01.432 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:01 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:36:01.432 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:01 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-27T06:36:01.432 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:01 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:01.432 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:01 smithi029 ceph-mon[32610]: Reconfiguring daemon mon.b on smithi047 2024-07-27T06:36:01.432 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:01 smithi029 ceph-mon[32610]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:01 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:36:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:01 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:36:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:01 smithi047 ceph-mon[36835]: Reconfiguring mon.b (monmap changed)... 2024-07-27T06:36:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:01 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:36:01.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:01 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-27T06:36:01.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:01 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:01.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:01 smithi047 ceph-mon[36835]: Reconfiguring daemon mon.b on smithi047 2024-07-27T06:36:01.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:01 smithi047 ceph-mon[36835]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:01 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:36:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:01 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:36:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:01 smithi077 ceph-mon[37153]: Reconfiguring mon.b (monmap changed)... 2024-07-27T06:36:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:01 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:36:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:01 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-27T06:36:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:01 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:01 smithi077 ceph-mon[37153]: Reconfiguring daemon mon.b on smithi047 2024-07-27T06:36:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:01 smithi077 ceph-mon[37153]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:02.576 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:02 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:36:02.577 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:02 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:36:02.577 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:02 smithi077 ceph-mon[37153]: Reconfiguring mon.c (monmap changed)... 2024-07-27T06:36:02.577 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:02 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:36:02.577 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:02 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-27T06:36:02.577 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:02 smithi077 ceph-mon[37153]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:02.577 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:02 smithi077 ceph-mon[37153]: Reconfiguring daemon mon.c on smithi077 2024-07-27T06:36:02.577 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:02 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3022373979' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-27T06:36:02.676 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:02 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: ignoring --setuser ceph since I am not root 2024-07-27T06:36:02.677 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:02 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: ignoring --setgroup ceph since I am not root 2024-07-27T06:36:02.677 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:02 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:36:02.677 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:02 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:36:02.677 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:02 smithi029 ceph-mon[32610]: Reconfiguring mon.c (monmap changed)... 2024-07-27T06:36:02.677 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:02 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:36:02.677 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:02 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-27T06:36:02.677 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:02 smithi029 ceph-mon[32610]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:02.677 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:02 smithi029 ceph-mon[32610]: Reconfiguring daemon mon.c on smithi077 2024-07-27T06:36:02.677 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:02 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3022373979' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-27T06:36:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:02 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:36:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:02 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' 2024-07-27T06:36:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:02 smithi047 ceph-mon[36835]: Reconfiguring mon.c (monmap changed)... 2024-07-27T06:36:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:02 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:36:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:02 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-27T06:36:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:02 smithi047 ceph-mon[36835]: from='mgr.14148 172.21.15.29:0/969502786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:02 smithi047 ceph-mon[36835]: Reconfiguring daemon mon.c on smithi077 2024-07-27T06:36:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:02 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3022373979' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-27T06:36:02.972 INFO:tasks.cephadm:Adding mgr.x on smithi029 2024-07-27T06:36:02.973 INFO:tasks.cephadm:Adding mgr.y on smithi047 2024-07-27T06:36:02.973 INFO:tasks.cephadm:Adding mgr.z on smithi077 2024-07-27T06:36:02.973 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch apply mgr '3;smithi029=x;smithi047=y;smithi077=z' 2024-07-27T06:36:03.216 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.c/config 2024-07-27T06:36:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:03 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3022373979' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-27T06:36:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:03 smithi077 ceph-mon[37153]: mgrmap e14: x(active, since 64s) 2024-07-27T06:36:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:03 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3022373979' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-27T06:36:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:03 smithi047 ceph-mon[36835]: mgrmap e14: x(active, since 64s) 2024-07-27T06:36:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:03 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3022373979' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-27T06:36:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:03 smithi029 ceph-mon[32610]: mgrmap e14: x(active, since 64s) 2024-07-27T06:36:07.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: Active manager daemon x restarted 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: Activating manager daemon x 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: osdmap e5: 0 total, 0 up, 0 in 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: mgrmap e15: x(active, starting, since 0.00486205s) 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: Manager daemon x is now available 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:36:07.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-27T06:36:07.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: Active manager daemon x restarted 2024-07-27T06:36:07.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: Activating manager daemon x 2024-07-27T06:36:07.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: osdmap e5: 0 total, 0 up, 0 in 2024-07-27T06:36:07.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: mgrmap e15: x(active, starting, since 0.00486205s) 2024-07-27T06:36:07.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:36:07.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:36:07.915 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:36:07.915 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-27T06:36:07.915 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-27T06:36:07.915 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-27T06:36:07.915 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-27T06:36:07.915 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: Manager daemon x is now available 2024-07-27T06:36:07.915 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:36:07.915 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-27T06:36:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: Active manager daemon x restarted 2024-07-27T06:36:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: Activating manager daemon x 2024-07-27T06:36:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: osdmap e5: 0 total, 0 up, 0 in 2024-07-27T06:36:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: mgrmap e15: x(active, starting, since 0.00486205s) 2024-07-27T06:36:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:36:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:36:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:36:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-27T06:36:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-27T06:36:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-27T06:36:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-27T06:36:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: Manager daemon x is now available 2024-07-27T06:36:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:36:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-27T06:36:08.543 INFO:teuthology.orchestra.run.smithi077.stdout:Scheduled mgr update... 2024-07-27T06:36:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-27T06:36:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:08 smithi077 ceph-mon[37153]: mgrmap e16: x(active, since 1.009s) 2024-07-27T06:36:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-27T06:36:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:08 smithi047 ceph-mon[36835]: mgrmap e16: x(active, since 1.009s) 2024-07-27T06:36:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:08.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-27T06:36:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:08 smithi029 ceph-mon[32610]: mgrmap e16: x(active, since 1.009s) 2024-07-27T06:36:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:09.462 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:36:09.099+0000 7fdc4e944640 -1 log_channel(cephadm) log [ERR] : [27/Jul/2024:06:36:09] ENGINE Error in HTTPServer.serve 2024-07-27T06:36:09.462 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: Traceback (most recent call last): 2024-07-27T06:36:09.462 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-27T06:36:09.462 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: self._connections.run(self.expiration_interval) 2024-07-27T06:36:09.462 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-27T06:36:09.463 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: self._run(expiration_interval) 2024-07-27T06:36:09.463 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-27T06:36:09.463 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: new_conn = self._from_server_socket(self.server.socket) 2024-07-27T06:36:09.463 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-27T06:36:09.463 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-27T06:36:09.463 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-27T06:36:09.463 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: s = self.context.wrap_socket( 2024-07-27T06:36:09.463 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-27T06:36:09.463 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: return self.sslsocket_class._create( 2024-07-27T06:36:09.463 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-27T06:36:09.463 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: self.do_handshake() 2024-07-27T06:36:09.463 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-27T06:36:09.464 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: self._sslobj.do_handshake() 2024-07-27T06:36:09.464 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-27T06:36:09.464 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:36:09.800 DEBUG:teuthology.orchestra.run.smithi047:mgr.y> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mgr.y.service 2024-07-27T06:36:09.803 DEBUG:teuthology.orchestra.run.smithi077:mgr.z> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mgr.z.service 2024-07-27T06:36:09.805 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-07-27T06:36:09.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set client client_mount_timeout 600 2024-07-27T06:36:09.808 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-07-27T06:36:09.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set client debug_client 20 2024-07-27T06:36:09.851 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-07-27T06:36:09.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set client debug_ms 1 2024-07-27T06:36:09.894 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-07-27T06:36:09.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set client rados_mon_op_timeout 900 2024-07-27T06:36:09.937 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-07-27T06:36:09.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set client rados_osd_op_timeout 900 2024-07-27T06:36:09.980 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-07-27T06:36:09.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds debug_mds 20 2024-07-27T06:36:10.011 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: Saving service mgr spec with placement smithi029=x;smithi047=y;smithi077=z;count:3 2024-07-27T06:36:10.011 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: [27/Jul/2024:06:36:08] ENGINE Bus STARTING 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: [27/Jul/2024:06:36:09] ENGINE Serving on https://172.21.15.29:7150 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: [27/Jul/2024:06:36:09] ENGINE Error in HTTPServer.serve 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout: Traceback (most recent call last): 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._connections.run(self.expiration_interval) 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._run(expiration_interval) 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-27T06:36:10.012 INFO:journalctl@ceph.mon.a.smithi029.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-27T06:36:10.013 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-27T06:36:10.013 INFO:journalctl@ceph.mon.a.smithi029.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-27T06:36:10.013 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-27T06:36:10.013 INFO:journalctl@ceph.mon.a.smithi029.stdout: s = self.context.wrap_socket( 2024-07-27T06:36:10.013 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-27T06:36:10.013 INFO:journalctl@ceph.mon.a.smithi029.stdout: return self.sslsocket_class._create( 2024-07-27T06:36:10.013 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-27T06:36:10.013 INFO:journalctl@ceph.mon.a.smithi029.stdout: self.do_handshake() 2024-07-27T06:36:10.013 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-27T06:36:10.013 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._sslobj.do_handshake() 2024-07-27T06:36:10.013 INFO:journalctl@ceph.mon.a.smithi029.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-27T06:36:10.013 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: [27/Jul/2024:06:36:09] ENGINE Serving on http://172.21.15.29:8765 2024-07-27T06:36:10.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: [27/Jul/2024:06:36:09] ENGINE Bus STARTED 2024-07-27T06:36:10.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi047", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:36:10.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:36:10.015 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-07-27T06:36:10.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds debug_mds_balancer 20 2024-07-27T06:36:10.059 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-07-27T06:36:10.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds debug_ms 2 2024-07-27T06:36:10.102 INFO:tasks.cephadm:setting config [mds] mds_bal_fragment_size_max = 10000 2024-07-27T06:36:10.103 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds mds_bal_fragment_size_max 10000 2024-07-27T06:36:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: Saving service mgr spec with placement smithi029=x;smithi047=y;smithi077=z;count:3 2024-07-27T06:36:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: [27/Jul/2024:06:36:08] ENGINE Bus STARTING 2024-07-27T06:36:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: [27/Jul/2024:06:36:09] ENGINE Serving on https://172.21.15.29:7150 2024-07-27T06:36:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: [27/Jul/2024:06:36:09] ENGINE Error in HTTPServer.serve 2024-07-27T06:36:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-07-27T06:36:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-27T06:36:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout: self._connections.run(self.expiration_interval) 2024-07-27T06:36:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-27T06:36:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout: self._run(expiration_interval) 2024-07-27T06:36:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-27T06:36:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-27T06:36:10.108 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-27T06:36:10.108 INFO:journalctl@ceph.mon.c.smithi077.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-27T06:36:10.108 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-27T06:36:10.108 INFO:journalctl@ceph.mon.c.smithi077.stdout: s = self.context.wrap_socket( 2024-07-27T06:36:10.112 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-27T06:36:10.112 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.sslsocket_class._create( 2024-07-27T06:36:10.112 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-27T06:36:10.112 INFO:journalctl@ceph.mon.c.smithi077.stdout: self.do_handshake() 2024-07-27T06:36:10.112 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-27T06:36:10.112 INFO:journalctl@ceph.mon.c.smithi077.stdout: self._sslobj.do_handshake() 2024-07-27T06:36:10.112 INFO:journalctl@ceph.mon.c.smithi077.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-27T06:36:10.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: [27/Jul/2024:06:36:09] ENGINE Serving on http://172.21.15.29:8765 2024-07-27T06:36:10.112 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: [27/Jul/2024:06:36:09] ENGINE Bus STARTED 2024-07-27T06:36:10.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi047", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:36:10.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.113 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:36:10.122 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:10.123 INFO:tasks.cephadm:setting config [mds] mds_bal_merge_size = 5 2024-07-27T06:36:10.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds mds_bal_merge_size 5 2024-07-27T06:36:10.175 INFO:tasks.cephadm:setting config [mds] mds_bal_split_bits = 3 2024-07-27T06:36:10.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds mds_bal_split_bits 3 2024-07-27T06:36:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: Saving service mgr spec with placement smithi029=x;smithi047=y;smithi077=z;count:3 2024-07-27T06:36:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: [27/Jul/2024:06:36:08] ENGINE Bus STARTING 2024-07-27T06:36:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: [27/Jul/2024:06:36:09] ENGINE Serving on https://172.21.15.29:7150 2024-07-27T06:36:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: [27/Jul/2024:06:36:09] ENGINE Error in HTTPServer.serve 2024-07-27T06:36:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout: Traceback (most recent call last): 2024-07-27T06:36:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-27T06:36:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._connections.run(self.expiration_interval) 2024-07-27T06:36:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-27T06:36:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._run(expiration_interval) 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: s = self.context.wrap_socket( 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: return self.sslsocket_class._create( 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: self.do_handshake() 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: self._sslobj.do_handshake() 2024-07-27T06:36:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-27T06:36:10.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: [27/Jul/2024:06:36:09] ENGINE Serving on http://172.21.15.29:8765 2024-07-27T06:36:10.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: [27/Jul/2024:06:36:09] ENGINE Bus STARTED 2024-07-27T06:36:10.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi047", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:36:10.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:10.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:36:10.227 INFO:tasks.cephadm:setting config [mds] mds_bal_split_size = 100 2024-07-27T06:36:10.227 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds mds_bal_split_size 100 2024-07-27T06:36:10.274 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-07-27T06:36:10.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds mds_debug_frag True 2024-07-27T06:36:10.317 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-07-27T06:36:10.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds mds_debug_scatterstat True 2024-07-27T06:36:10.365 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-07-27T06:36:10.365 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds mds_op_complaint_time 180 2024-07-27T06:36:10.428 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-07-27T06:36:10.428 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds mds_verify_scatter True 2024-07-27T06:36:10.486 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-07-27T06:36:10.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds osd_op_complaint_time 180 2024-07-27T06:36:10.545 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-07-27T06:36:10.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds rados_mon_op_timeout 900 2024-07-27T06:36:10.595 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-07-27T06:36:10.595 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mds rados_osd_op_timeout 900 2024-07-27T06:36:10.651 INFO:tasks.cephadm:setting config [mgr] debug_cephsqlite = 20 2024-07-27T06:36:10.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mgr debug_cephsqlite 20 2024-07-27T06:36:10.706 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/allow_m_granularity = True 2024-07-27T06:36:10.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-07-27T06:36:10.755 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/dump_on_update = True 2024-07-27T06:36:10.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-07-27T06:36:10.807 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/log_level = debug 2024-07-27T06:36:10.807 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/log_level debug 2024-07-27T06:36:10.857 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-07-27T06:36:10.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mon mon_op_complaint_time 120 2024-07-27T06:36:10.912 INFO:tasks.cephadm:setting config [mon] mon_warn_on_pool_no_app = False 2024-07-27T06:36:10.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set mon mon_warn_on_pool_no_app False 2024-07-27T06:36:10.969 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-07-27T06:36:10.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config set osd osd_op_complaint_time 180 2024-07-27T06:36:11.026 DEBUG:tasks.cephadm:set 26 configs 2024-07-27T06:36:11.026 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi029') 2024-07-27T06:36:11.178 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:11.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:11.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:11.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:36:11.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:11.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:36:11.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:11 smithi029 ceph-mon[32610]: Updating smithi029:/etc/ceph/ceph.conf 2024-07-27T06:36:11.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:11 smithi029 ceph-mon[32610]: Updating smithi047:/etc/ceph/ceph.conf 2024-07-27T06:36:11.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:11 smithi029 ceph-mon[32610]: Updating smithi077:/etc/ceph/ceph.conf 2024-07-27T06:36:11.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:11 smithi029 ceph-mon[32610]: mgrmap e17: x(active, since 2s) 2024-07-27T06:36:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:36:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:36:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:11 smithi077 ceph-mon[37153]: Updating smithi029:/etc/ceph/ceph.conf 2024-07-27T06:36:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:11 smithi077 ceph-mon[37153]: Updating smithi047:/etc/ceph/ceph.conf 2024-07-27T06:36:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:11 smithi077 ceph-mon[37153]: Updating smithi077:/etc/ceph/ceph.conf 2024-07-27T06:36:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:11 smithi077 ceph-mon[37153]: mgrmap e17: x(active, since 2s) 2024-07-27T06:36:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:11.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:36:11.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:11.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:36:11.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:11 smithi047 ceph-mon[36835]: Updating smithi029:/etc/ceph/ceph.conf 2024-07-27T06:36:11.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:11 smithi047 ceph-mon[36835]: Updating smithi047:/etc/ceph/ceph.conf 2024-07-27T06:36:11.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:11 smithi047 ceph-mon[36835]: Updating smithi077:/etc/ceph/ceph.conf 2024-07-27T06:36:11.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:11 smithi047 ceph-mon[36835]: mgrmap e17: x(active, since 2s) 2024-07-27T06:36:12.458 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:36:12.458 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: Updating smithi047:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:36:12.458 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:36:12.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:36:12.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:36:12.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: Updating smithi047:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:36:12.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-27T06:36:12.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-27T06:36:12.460 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-27T06:36:12.460 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:36:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: Updating smithi047:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:36:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:36:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:36:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:36:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: Updating smithi047:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:36:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-27T06:36:12.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-27T06:36:12.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-27T06:36:12.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:36:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: Updating smithi047:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: Updating smithi047:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: Updating smithi077:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: Updating smithi047:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-27T06:36:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-27T06:36:12.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-27T06:36:12.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:12.761 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:12.779 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:12.993 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:12.993 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:12.994 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:12.999 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:13.002 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:13.008 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:13.009 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:13.011 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:13.012 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:13.013 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:13.099 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:13 smithi029 ceph-mon[32610]: Updating smithi029:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:36:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:13 smithi029 ceph-mon[32610]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:36:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:13 smithi029 ceph-mon[32610]: Updating smithi029:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:36:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:13 smithi029 ceph-mon[32610]: Deploying daemon mgr.z on smithi077 2024-07-27T06:36:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:13 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3566380804' entity='client.admin' 2024-07-27T06:36:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:13 smithi077 ceph-mon[37153]: Updating smithi029:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:36:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:13 smithi077 ceph-mon[37153]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:36:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:13 smithi077 ceph-mon[37153]: Updating smithi029:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:36:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:13 smithi077 ceph-mon[37153]: Deploying daemon mgr.z on smithi077 2024-07-27T06:36:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:13 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3566380804' entity='client.admin' 2024-07-27T06:36:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:13 smithi047 ceph-mon[36835]: Updating smithi029:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.conf 2024-07-27T06:36:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:13 smithi047 ceph-mon[36835]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-07-27T06:36:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:13 smithi047 ceph-mon[36835]: Updating smithi029:/var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/config/ceph.client.admin.keyring 2024-07-27T06:36:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:13 smithi047 ceph-mon[36835]: Deploying daemon mgr.z on smithi077 2024-07-27T06:36:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:13 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3566380804' entity='client.admin' 2024-07-27T06:36:13.968 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi029') 2024-07-27T06:36:13.983 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Jul 27 06:36:13 smithi077 systemd[1]: Starting Ceph mgr.z for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:36:14.263 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Jul 27 06:36:13 smithi077 podman[39929]: 2024-07-27 06:36:13.982149148 +0000 UTC m=+0.092725721 container create 67c05b33d6768e1f415a67896adddd5bb185689362ad2d0dc4855994cf4f38b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-07-27T06:36:14.263 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Jul 27 06:36:14 smithi077 podman[39929]: 2024-07-27 06:36:13.906234681 +0000 UTC m=+0.016811254 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:36:14.264 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Jul 27 06:36:14 smithi077 podman[39929]: 2024-07-27 06:36:14.226233245 +0000 UTC m=+0.336809832 container init 67c05b33d6768e1f415a67896adddd5bb185689362ad2d0dc4855994cf4f38b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=) 2024-07-27T06:36:14.264 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Jul 27 06:36:14 smithi077 podman[39929]: 2024-07-27 06:36:14.228898071 +0000 UTC m=+0.339474644 container start 67c05b33d6768e1f415a67896adddd5bb185689362ad2d0dc4855994cf4f38b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-z, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-27T06:36:14.332 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:14.367 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:14.367 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:14.606 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Jul 27 06:36:14 smithi077 ceph-mgr[39948]: -- 172.21.15.77:0/2729796808 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563f260b10e0 con 0x563f260cfa80 2024-07-27T06:36:14.607 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Jul 27 06:36:14 smithi077 bash[39929]: 67c05b33d6768e1f415a67896adddd5bb185689362ad2d0dc4855994cf4f38b7 2024-07-27T06:36:14.607 INFO:journalctl@ceph.mgr.z.smithi077.stdout:Jul 27 06:36:14 smithi077 systemd[1]: Started Ceph mgr.z for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:36:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-27T06:36:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-27T06:36:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-27T06:36:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:15 smithi047 ceph-mon[36835]: Deploying daemon mgr.y on smithi047 2024-07-27T06:36:15.489 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:15.489 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:15.489 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:15.489 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-27T06:36:15.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-27T06:36:15.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-27T06:36:15.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:15.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:15 smithi029 ceph-mon[32610]: Deploying daemon mgr.y on smithi047 2024-07-27T06:36:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-27T06:36:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-27T06:36:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-27T06:36:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:15 smithi077 ceph-mon[37153]: Deploying daemon mgr.y on smithi047 2024-07-27T06:36:16.144 INFO:journalctl@ceph.mgr.y.smithi047.stdout:Jul 27 06:36:16 smithi047 systemd[1]: Starting Ceph mgr.y for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:36:16.446 INFO:journalctl@ceph.mgr.y.smithi047.stdout:Jul 27 06:36:16 smithi047 podman[39164]: 2024-07-27 06:36:16.265752961 +0000 UTC m=+0.017180801 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:36:16.446 INFO:journalctl@ceph.mgr.y.smithi047.stdout:Jul 27 06:36:16 smithi047 podman[39164]: 2024-07-27 06:36:16.369761447 +0000 UTC m=+0.121189332 container create ef83de724406b4ed6f7cfcb48d2f483317c1d1f9607047f7a888ba8a76318893 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=) 2024-07-27T06:36:16.911 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:16.911 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:16.912 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:16.912 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:16.946 INFO:journalctl@ceph.mgr.y.smithi047.stdout:Jul 27 06:36:16 smithi047 podman[39164]: 2024-07-27 06:36:16.580181644 +0000 UTC m=+0.331609484 container init ef83de724406b4ed6f7cfcb48d2f483317c1d1f9607047f7a888ba8a76318893 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-y, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-07-27T06:36:16.946 INFO:journalctl@ceph.mgr.y.smithi047.stdout:Jul 27 06:36:16 smithi047 podman[39164]: 2024-07-27 06:36:16.582872236 +0000 UTC m=+0.334300078 container start ef83de724406b4ed6f7cfcb48d2f483317c1d1f9607047f7a888ba8a76318893 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-y, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-07-27T06:36:16.946 INFO:journalctl@ceph.mgr.y.smithi047.stdout:Jul 27 06:36:16 smithi047 ceph-mgr[39183]: -- 172.21.15.47:0/3500439 <== mon.2 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56447ba270e0 con 0x5644807c6480 2024-07-27T06:36:16.946 INFO:journalctl@ceph.mgr.y.smithi047.stdout:Jul 27 06:36:16 smithi047 bash[39164]: ef83de724406b4ed6f7cfcb48d2f483317c1d1f9607047f7a888ba8a76318893 2024-07-27T06:36:16.946 INFO:journalctl@ceph.mgr.y.smithi047.stdout:Jul 27 06:36:16 smithi047 systemd[1]: Started Ceph mgr.y for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:36:17.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:17.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:17.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:17.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:17.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:36:17.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:36:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:36:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.924 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:18.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:36:18.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:18 smithi029 ceph-mon[32610]: Reconfiguring mgr.x (unknown last config time)... 2024-07-27T06:36:18.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-27T06:36:18.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-27T06:36:18.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:36:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:18 smithi047 ceph-mon[36835]: Reconfiguring mgr.x (unknown last config time)... 2024-07-27T06:36:18.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-27T06:36:18.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-27T06:36:18.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:18.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:18 smithi047 ceph-mon[36835]: Reconfiguring daemon mgr.x on smithi029 2024-07-27T06:36:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:18 smithi029 ceph-mon[32610]: Reconfiguring daemon mgr.x on smithi029 2024-07-27T06:36:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:36:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:18 smithi077 ceph-mon[37153]: Reconfiguring mgr.x (unknown last config time)... 2024-07-27T06:36:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-27T06:36:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-27T06:36:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:18 smithi077 ceph-mon[37153]: Reconfiguring daemon mgr.x on smithi029 2024-07-27T06:36:19.501 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:19.502 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:19 smithi047 ceph-mon[36835]: Standby manager daemon z started 2024-07-27T06:36:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:19 smithi047 ceph-mon[36835]: from='mgr.? 172.21.15.77:0/2210154196' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-27T06:36:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:19 smithi047 ceph-mon[36835]: from='mgr.? 172.21.15.77:0/2210154196' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-27T06:36:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:19 smithi047 ceph-mon[36835]: from='mgr.? 172.21.15.77:0/2210154196' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-27T06:36:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:19 smithi047 ceph-mon[36835]: from='mgr.? 172.21.15.77:0/2210154196' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-27T06:36:20.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:19 smithi029 ceph-mon[32610]: Standby manager daemon z started 2024-07-27T06:36:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:19 smithi029 ceph-mon[32610]: from='mgr.? 172.21.15.77:0/2210154196' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-27T06:36:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:19 smithi029 ceph-mon[32610]: from='mgr.? 172.21.15.77:0/2210154196' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-27T06:36:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:19 smithi029 ceph-mon[32610]: from='mgr.? 172.21.15.77:0/2210154196' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-27T06:36:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:19 smithi029 ceph-mon[32610]: from='mgr.? 172.21.15.77:0/2210154196' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-27T06:36:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:19 smithi077 ceph-mon[37153]: Standby manager daemon z started 2024-07-27T06:36:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:19 smithi077 ceph-mon[37153]: from='mgr.? 172.21.15.77:0/2210154196' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-27T06:36:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:19 smithi077 ceph-mon[37153]: from='mgr.? 172.21.15.77:0/2210154196' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-27T06:36:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:19 smithi077 ceph-mon[37153]: from='mgr.? 172.21.15.77:0/2210154196' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-27T06:36:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:19 smithi077 ceph-mon[37153]: from='mgr.? 172.21.15.77:0/2210154196' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-27T06:36:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:20 smithi047 ceph-mon[36835]: mgrmap e18: x(active, since 12s), standbys: z 2024-07-27T06:36:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-27T06:36:21.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:20 smithi029 ceph-mon[32610]: mgrmap e18: x(active, since 12s), standbys: z 2024-07-27T06:36:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-27T06:36:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:20 smithi077 ceph-mon[37153]: mgrmap e18: x(active, since 12s), standbys: z 2024-07-27T06:36:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-27T06:36:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:21 smithi047 ceph-mon[36835]: Standby manager daemon y started 2024-07-27T06:36:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:21 smithi047 ceph-mon[36835]: from='mgr.? 172.21.15.47:0/1907153261' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-27T06:36:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:21 smithi047 ceph-mon[36835]: from='mgr.? 172.21.15.47:0/1907153261' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-27T06:36:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:21 smithi047 ceph-mon[36835]: from='mgr.? 172.21.15.47:0/1907153261' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-27T06:36:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:21 smithi047 ceph-mon[36835]: from='mgr.? 172.21.15.47:0/1907153261' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-27T06:36:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:21 smithi029 ceph-mon[32610]: Standby manager daemon y started 2024-07-27T06:36:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:21 smithi029 ceph-mon[32610]: from='mgr.? 172.21.15.47:0/1907153261' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-27T06:36:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:21 smithi029 ceph-mon[32610]: from='mgr.? 172.21.15.47:0/1907153261' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-27T06:36:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:21 smithi029 ceph-mon[32610]: from='mgr.? 172.21.15.47:0/1907153261' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-27T06:36:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:21 smithi029 ceph-mon[32610]: from='mgr.? 172.21.15.47:0/1907153261' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-27T06:36:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:21 smithi077 ceph-mon[37153]: Standby manager daemon y started 2024-07-27T06:36:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:21 smithi077 ceph-mon[37153]: from='mgr.? 172.21.15.47:0/1907153261' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-27T06:36:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:21 smithi077 ceph-mon[37153]: from='mgr.? 172.21.15.47:0/1907153261' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-27T06:36:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:21 smithi077 ceph-mon[37153]: from='mgr.? 172.21.15.47:0/1907153261' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-27T06:36:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:21 smithi077 ceph-mon[37153]: from='mgr.? 172.21.15.47:0/1907153261' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-27T06:36:23.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:22 smithi029 ceph-mon[32610]: mgrmap e19: x(active, since 14s), standbys: y, z 2024-07-27T06:36:23.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-27T06:36:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:22 smithi047 ceph-mon[36835]: mgrmap e19: x(active, since 14s), standbys: y, z 2024-07-27T06:36:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-27T06:36:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:22 smithi077 ceph-mon[37153]: mgrmap e19: x(active, since 14s), standbys: y, z 2024-07-27T06:36:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-27T06:36:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:26 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/532227396' entity='client.admin' 2024-07-27T06:36:26.857 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:26 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/532227396' entity='client.admin' 2024-07-27T06:36:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:26 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/532227396' entity='client.admin' 2024-07-27T06:36:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:27 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3699952292' entity='client.admin' 2024-07-27T06:36:27.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:27 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3699952292' entity='client.admin' 2024-07-27T06:36:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:27 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3699952292' entity='client.admin' 2024-07-27T06:36:28.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:28 smithi029 ceph-mon[32610]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:28 smithi077 ceph-mon[37153]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:28 smithi047 ceph-mon[36835]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:30 smithi077 ceph-mon[37153]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:30.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:30 smithi029 ceph-mon[32610]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:30 smithi047 ceph-mon[36835]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:31.929 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:31 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/741831125' entity='client.admin' 2024-07-27T06:36:31.930 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:31 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3500229001' entity='client.admin' 2024-07-27T06:36:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:31 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/741831125' entity='client.admin' 2024-07-27T06:36:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:31 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3500229001' entity='client.admin' 2024-07-27T06:36:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:31 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/741831125' entity='client.admin' 2024-07-27T06:36:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:31 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3500229001' entity='client.admin' 2024-07-27T06:36:33.610 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:33 smithi029 ceph-mon[32610]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:33.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:33 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/493442750' entity='client.admin' 2024-07-27T06:36:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:33 smithi047 ceph-mon[36835]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:33 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/493442750' entity='client.admin' 2024-07-27T06:36:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:33 smithi077 ceph-mon[37153]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:33 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/493442750' entity='client.admin' 2024-07-27T06:36:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:35 smithi029 ceph-mon[32610]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:35 smithi047 ceph-mon[36835]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:35 smithi077 ceph-mon[37153]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:36 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3577043311' entity='client.admin' 2024-07-27T06:36:36.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:36 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3577043311' entity='client.admin' 2024-07-27T06:36:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:36 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3577043311' entity='client.admin' 2024-07-27T06:36:37.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:37 smithi029 ceph-mon[32610]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:37 smithi047 ceph-mon[36835]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:37.757 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:37 smithi077 ceph-mon[37153]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:38 smithi029 ceph-mon[32610]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:38 smithi029 ceph-mon[32610]: Reconfiguring mon.c (monmap changed)... 2024-07-27T06:36:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:36:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-27T06:36:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:38 smithi029 ceph-mon[32610]: Reconfiguring daemon mon.c on smithi077 2024-07-27T06:36:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:38 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/23546413' entity='client.admin' 2024-07-27T06:36:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:38 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2870210062' entity='client.admin' 2024-07-27T06:36:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:38 smithi077 ceph-mon[37153]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:38 smithi077 ceph-mon[37153]: Reconfiguring mon.c (monmap changed)... 2024-07-27T06:36:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:36:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-27T06:36:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:38 smithi077 ceph-mon[37153]: Reconfiguring daemon mon.c on smithi077 2024-07-27T06:36:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:38 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/23546413' entity='client.admin' 2024-07-27T06:36:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:38 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2870210062' entity='client.admin' 2024-07-27T06:36:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:38 smithi047 ceph-mon[36835]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:38 smithi047 ceph-mon[36835]: Reconfiguring mon.c (monmap changed)... 2024-07-27T06:36:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-27T06:36:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-27T06:36:39.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:39.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:38 smithi047 ceph-mon[36835]: Reconfiguring daemon mon.c on smithi077 2024-07-27T06:36:39.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:38 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/23546413' entity='client.admin' 2024-07-27T06:36:39.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:38 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2870210062' entity='client.admin' 2024-07-27T06:36:40.343 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:40.343 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:40.343 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:36:40.343 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:40.343 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:36:40.343 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:36:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:36:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:36:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:36:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:41 smithi077 ceph-mon[37153]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:41 smithi047 ceph-mon[36835]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:41 smithi029 ceph-mon[32610]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:41.927 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi029') 2024-07-27T06:36:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:43 smithi077 ceph-mon[37153]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:43 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2756840676' entity='client.admin' 2024-07-27T06:36:43.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:43 smithi029 ceph-mon[32610]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:43.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:43 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2756840676' entity='client.admin' 2024-07-27T06:36:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:43 smithi047 ceph-mon[36835]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:43 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2756840676' entity='client.admin' 2024-07-27T06:36:44.521 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:36:44.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:44.899+0000 7f5d0bc19640 1 -- 172.21.15.29:0/1855658139 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5cf00b3b90 msgr2=0x7f5cf00b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:44.899+0000 7f5d0bc19640 1 --2- 172.21.15.29:0/1855658139 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5cf00b3b90 0x7f5cf00b60a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5d00005fd0 tx=0x7f5d0000b390 comp rx=0 tx=0).stop 2024-07-27T06:36:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:44.899+0000 7f5d0bc19640 1 -- 172.21.15.29:0/1855658139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d04104db0 msgr2=0x7f5d0419c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:44.899+0000 7f5d0bc19640 1 --2- 172.21.15.29:0/1855658139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d04104db0 0x7f5d0419c440 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5cf400d740 tx=0x7f5cf400dc00 comp rx=0 tx=0).stop 2024-07-27T06:36:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:44.899+0000 7f5d0bc19640 1 -- 172.21.15.29:0/1855658139 shutdown_connections 2024-07-27T06:36:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:44.899+0000 7f5d0bc19640 1 --2- 172.21.15.29:0/1855658139 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5cf00b3b90 0x7f5cf00b60a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:44.899+0000 7f5d0bc19640 1 --2- 172.21.15.29:0/1855658139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d0410d8c0 0x7f5d041a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:44.899+0000 7f5d0bc19640 1 --2- 172.21.15.29:0/1855658139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d04104db0 0x7f5d0419c440 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:44.899+0000 7f5d0bc19640 1 --2- 172.21.15.29:0/1855658139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d04104460 0x7f5d0419bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:44.899+0000 7f5d0bc19640 1 -- 172.21.15.29:0/1855658139 >> 172.21.15.29:0/1855658139 conn(0x7f5d040791a0 msgr2=0x7f5d04100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:44.899+0000 7f5d0bc19640 1 -- 172.21.15.29:0/1855658139 shutdown_connections 2024-07-27T06:36:44.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:44.899+0000 7f5d0bc19640 1 -- 172.21.15.29:0/1855658139 wait complete. 2024-07-27T06:36:45.077 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:44 smithi029 ceph-mon[32610]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:45.077 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:44 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1737872270' entity='client.admin' 2024-07-27T06:36:45.078 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:36:45.078 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:45.078 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:36:45.078 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:44 smithi077 ceph-mon[37153]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:44 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1737872270' entity='client.admin' 2024-07-27T06:36:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:36:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:36:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:44 smithi047 ceph-mon[36835]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:44 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1737872270' entity='client.admin' 2024-07-27T06:36:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:36:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:36:45.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:36:45.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:36:46.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:45 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1855658139' entity='client.admin' 2024-07-27T06:36:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:45 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1855658139' entity='client.admin' 2024-07-27T06:36:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:45 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1855658139' entity='client.admin' 2024-07-27T06:36:47.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:46 smithi029 ceph-mon[32610]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:47.123 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:46 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/376133328' entity='client.admin' 2024-07-27T06:36:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:46 smithi047 ceph-mon[36835]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:46 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/376133328' entity='client.admin' 2024-07-27T06:36:47.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:46 smithi077 ceph-mon[37153]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:47.360 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:46 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/376133328' entity='client.admin' 2024-07-27T06:36:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.222+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/1513988812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd28107e00 msgr2=0x7fcd28108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.222+0000 7fcd2d9b0640 1 --2- 172.21.15.29:0/1513988812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd28107e00 0x7fcd28108210 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009900 tx=0x7fcd1c02a410 comp rx=0 tx=0).stop 2024-07-27T06:36:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.222+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/1513988812 shutdown_connections 2024-07-27T06:36:48.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.222+0000 7fcd2d9b0640 1 --2- 172.21.15.29:0/1513988812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd28108750 0x7fcd2810ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.222+0000 7fcd2d9b0640 1 --2- 172.21.15.29:0/1513988812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd28107e00 0x7fcd28108210 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.222+0000 7fcd2d9b0640 1 --2- 172.21.15.29:0/1513988812 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd28102510 0x7fcd28102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.222+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/1513988812 >> 172.21.15.29:0/1513988812 conn(0x7fcd280791c0 msgr2=0x7fcd280795d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.222+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/1513988812 shutdown_connections 2024-07-27T06:36:48.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.222+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/1513988812 wait complete. 2024-07-27T06:36:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.223+0000 7fcd2d9b0640 1 Processor -- start 2024-07-27T06:36:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.223+0000 7fcd2d9b0640 1 -- start start 2024-07-27T06:36:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd2d9b0640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd28102510 0x7fcd28199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd2d9b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd28107e00 0x7fcd2819a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd2d9b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd28108750 0x7fcd2819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd2d9b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcd281122d0 con 0x7fcd28108750 2024-07-27T06:36:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd2d9b0640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fcd28112150 con 0x7fcd28102510 2024-07-27T06:36:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd2d9b0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcd28112450 con 0x7fcd28107e00 2024-07-27T06:36:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd28108750 0x7fcd2819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd28108750 0x7fcd2819f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57030/0 (socket says 172.21.15.29:57030) 2024-07-27T06:36:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd277fe640 1 -- 172.21.15.29:0/3296134792 learned_addr learned my addr 172.21.15.29:0/3296134792 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:36:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd26ffd640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd28102510 0x7fcd28199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd267fc640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd28107e00 0x7fcd2819a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd277fe640 1 -- 172.21.15.29:0/3296134792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd28107e00 msgr2=0x7fcd2819a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.224+0000 7fcd277fe640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd28107e00 0x7fcd2819a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.225+0000 7fcd277fe640 1 -- 172.21.15.29:0/3296134792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd28102510 msgr2=0x7fcd28199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.225+0000 7fcd277fe640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd28102510 0x7fcd28199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.225+0000 7fcd277fe640 1 -- 172.21.15.29:0/3296134792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd2819fd20 con 0x7fcd28108750 2024-07-27T06:36:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.225+0000 7fcd277fe640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd28108750 0x7fcd2819f6b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcd2001fab0 tx=0x7fcd2001feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.225+0000 7fcd2c9ae640 1 -- 172.21.15.29:0/3296134792 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd20004d20 con 0x7fcd28108750 2024-07-27T06:36:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.225+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/3296134792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd281a0010 con 0x7fcd28108750 2024-07-27T06:36:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.225+0000 7fcd2c9ae640 1 -- 172.21.15.29:0/3296134792 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcd20005020 con 0x7fcd28108750 2024-07-27T06:36:48.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.225+0000 7fcd2c9ae640 1 -- 172.21.15.29:0/3296134792 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd200226e0 con 0x7fcd28108750 2024-07-27T06:36:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.225+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/3296134792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd2806c810 con 0x7fcd28108750 2024-07-27T06:36:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.227+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/3296134792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd28103740 con 0x7fcd28108750 2024-07-27T06:36:48.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.228+0000 7fcd2c9ae640 1 -- 172.21.15.29:0/3296134792 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7fcd2000cb20 con 0x7fcd28108750 2024-07-27T06:36:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.229+0000 7fcd2c9ae640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcd100b3bc0 0x7fcd100b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.230+0000 7fcd26ffd640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcd100b3bc0 0x7fcd100b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.230+0000 7fcd2c9ae640 1 -- 172.21.15.29:0/3296134792 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fcd200f3170 con 0x7fcd28108750 2024-07-27T06:36:48.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.230+0000 7fcd26ffd640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcd100b3bc0 0x7fcd100b60d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcd1800abe0 tx=0x7fcd1800a2d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:48.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.237+0000 7fcd2c9ae640 1 -- 172.21.15.29:0/3296134792 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcd20027030 con 0x7fcd28108750 2024-07-27T06:36:48.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.421+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/3296134792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7fcd2810b550 con 0x7fcd28108750 2024-07-27T06:36:48.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.425+0000 7fcd2c9ae640 1 -- 172.21.15.29:0/3296134792 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v23)=0 v23) ==== 120+0+0 (secure 0 0 0) 0x7fcd200bc3c0 con 0x7fcd28108750 2024-07-27T06:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.427+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/3296134792 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcd100b3bc0 msgr2=0x7fcd100b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.427+0000 7fcd2d9b0640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcd100b3bc0 0x7fcd100b60d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcd1800abe0 tx=0x7fcd1800a2d0 comp rx=0 tx=0).stop 2024-07-27T06:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.427+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/3296134792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd28108750 msgr2=0x7fcd2819f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.427+0000 7fcd2d9b0640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd28108750 0x7fcd2819f6b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcd2001fab0 tx=0x7fcd2001feb0 comp rx=0 tx=0).stop 2024-07-27T06:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.427+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/3296134792 shutdown_connections 2024-07-27T06:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.427+0000 7fcd2d9b0640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd28108750 0x7fcd2819f6b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.427+0000 7fcd2d9b0640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcd100b3bc0 0x7fcd100b60d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.427+0000 7fcd2d9b0640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcd28107e00 0x7fcd2819a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.427+0000 7fcd2d9b0640 1 --2- 172.21.15.29:0/3296134792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd28102510 0x7fcd28199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.427+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/3296134792 >> 172.21.15.29:0/3296134792 conn(0x7fcd280791c0 msgr2=0x7fcd28101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.427+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/3296134792 shutdown_connections 2024-07-27T06:36:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:48.427+0000 7fcd2d9b0640 1 -- 172.21.15.29:0/3296134792 wait complete. 2024-07-27T06:36:48.947 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:48 smithi029 ceph-mon[32610]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:48.947 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:48 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3296134792' entity='client.admin' 2024-07-27T06:36:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:48 smithi047 ceph-mon[36835]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:48 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3296134792' entity='client.admin' 2024-07-27T06:36:49.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.281+0000 7f6d09a5a640 1 -- 172.21.15.29:0/3172613197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04108ed0 msgr2=0x7f6d041092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:49.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.281+0000 7f6d09a5a640 1 --2- 172.21.15.29:0/3172613197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04108ed0 0x7f6d041092e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009900 tx=0x7f6cf802a410 comp rx=0 tx=0).stop 2024-07-27T06:36:49.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.281+0000 7f6d09a5a640 1 -- 172.21.15.29:0/3172613197 shutdown_connections 2024-07-27T06:36:49.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.281+0000 7f6d09a5a640 1 --2- 172.21.15.29:0/3172613197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04109820 0x7f6d04110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.281+0000 7f6d09a5a640 1 --2- 172.21.15.29:0/3172613197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04108ed0 0x7f6d041092e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.281+0000 7f6d09a5a640 1 --2- 172.21.15.29:0/3172613197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d04103590 0x7f6d041039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.281+0000 7f6d09a5a640 1 -- 172.21.15.29:0/3172613197 >> 172.21.15.29:0/3172613197 conn(0x7f6d0407b280 msgr2=0x7f6d0407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.281+0000 7f6d09a5a640 1 -- 172.21.15.29:0/3172613197 shutdown_connections 2024-07-27T06:36:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.282+0000 7f6d09a5a640 1 -- 172.21.15.29:0/3172613197 wait complete. 2024-07-27T06:36:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.282+0000 7f6d09a5a640 1 Processor -- start 2024-07-27T06:36:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.282+0000 7f6d09a5a640 1 -- start start 2024-07-27T06:36:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d09a5a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04103590 0x7f6d04199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d09a5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d04108ed0 0x7f6d0419a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d09a5a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04109820 0x7f6d0419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d09a5a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6d04113370 con 0x7f6d04103590 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d09a5a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f6d041131f0 con 0x7f6d04109820 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d09a5a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6d041134f0 con 0x7f6d04108ed0 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04103590 0x7f6d04199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04103590 0x7f6d04199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58656/0 (socket says 172.21.15.29:58656) 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d02ffd640 1 -- 172.21.15.29:0/780280034 learned_addr learned my addr 172.21.15.29:0/780280034 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d027fc640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d04108ed0 0x7f6d0419a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d037fe640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04109820 0x7f6d0419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d02ffd640 1 -- 172.21.15.29:0/780280034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d04108ed0 msgr2=0x7f6d0419a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d02ffd640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d04108ed0 0x7f6d0419a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d02ffd640 1 -- 172.21.15.29:0/780280034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04109820 msgr2=0x7f6d0419f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d02ffd640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04109820 0x7f6d0419f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d02ffd640 1 -- 172.21.15.29:0/780280034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0419fe30 con 0x7f6d04103590 2024-07-27T06:36:49.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d02ffd640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04103590 0x7f6d04199ec0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400cce0 tx=0x7f6cf400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:49.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d08a58640 1 -- 172.21.15.29:0/780280034 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4004b20 con 0x7f6d04103590 2024-07-27T06:36:49.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d08a58640 1 -- 172.21.15.29:0/780280034 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6cf4004cc0 con 0x7f6d04103590 2024-07-27T06:36:49.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d08a58640 1 -- 172.21.15.29:0/780280034 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf400f690 con 0x7f6d04103590 2024-07-27T06:36:49.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.283+0000 7f6d09a5a640 1 -- 172.21.15.29:0/780280034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d041a0120 con 0x7f6d04103590 2024-07-27T06:36:49.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.284+0000 7f6d09a5a640 1 -- 172.21.15.29:0/780280034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d041a6880 con 0x7f6d04103590 2024-07-27T06:36:49.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.284+0000 7f6d09a5a640 1 -- 172.21.15.29:0/780280034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d041039a0 con 0x7f6d04103590 2024-07-27T06:36:49.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.286+0000 7f6d08a58640 1 -- 172.21.15.29:0/780280034 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f6cf4005540 con 0x7f6d04103590 2024-07-27T06:36:49.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.287+0000 7f6d08a58640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6cec0b3b90 0x7f6cec0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:49.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.287+0000 7f6d027fc640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6cec0b3b90 0x7f6cec0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:49.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.287+0000 7f6d08a58640 1 -- 172.21.15.29:0/780280034 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f6cf40e2650 con 0x7f6d04103590 2024-07-27T06:36:49.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.287+0000 7f6d027fc640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6cec0b3b90 0x7f6cec0b60a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009230 tx=0x7f6cf8032c60 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:49.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.290+0000 7f6d08a58640 1 -- 172.21.15.29:0/780280034 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6cf4014030 con 0x7f6d04103590 2024-07-27T06:36:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:48 smithi077 ceph-mon[37153]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:48 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3296134792' entity='client.admin' 2024-07-27T06:36:49.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.476+0000 7f6d09a5a640 1 -- 172.21.15.29:0/780280034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f6d0410b5d0 con 0x7f6d04103590 2024-07-27T06:36:49.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.480+0000 7f6d08a58640 1 -- 172.21.15.29:0/780280034 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v24)=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7f6cf40ab800 con 0x7f6d04103590 2024-07-27T06:36:49.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.484+0000 7f6d09a5a640 1 -- 172.21.15.29:0/780280034 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6cec0b3b90 msgr2=0x7f6cec0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:49.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.484+0000 7f6d09a5a640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6cec0b3b90 0x7f6cec0b60a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009230 tx=0x7f6cf8032c60 comp rx=0 tx=0).stop 2024-07-27T06:36:49.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.484+0000 7f6d09a5a640 1 -- 172.21.15.29:0/780280034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04103590 msgr2=0x7f6d04199ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:49.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.484+0000 7f6d09a5a640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04103590 0x7f6d04199ec0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400cce0 tx=0x7f6cf400a620 comp rx=0 tx=0).stop 2024-07-27T06:36:49.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.484+0000 7f6d09a5a640 1 -- 172.21.15.29:0/780280034 shutdown_connections 2024-07-27T06:36:49.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.484+0000 7f6d09a5a640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d04109820 0x7f6d0419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:49.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.484+0000 7f6d09a5a640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6cec0b3b90 0x7f6cec0b60a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:49.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.484+0000 7f6d09a5a640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d04108ed0 0x7f6d0419a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:49.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.484+0000 7f6d09a5a640 1 --2- 172.21.15.29:0/780280034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6d04103590 0x7f6d04199ec0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:49.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.484+0000 7f6d09a5a640 1 -- 172.21.15.29:0/780280034 >> 172.21.15.29:0/780280034 conn(0x7f6d0407b280 msgr2=0x7f6d041070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:49.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.485+0000 7f6d09a5a640 1 -- 172.21.15.29:0/780280034 shutdown_connections 2024-07-27T06:36:49.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.485+0000 7f6d09a5a640 1 -- 172.21.15.29:0/780280034 wait complete. 2024-07-27T06:36:50.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.911+0000 7f4d8de66640 1 -- 172.21.15.29:0/793924724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d881024d0 msgr2=0x7f4d881028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.911+0000 7f4d8de66640 1 --2- 172.21.15.29:0/793924724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d881024d0 0x7f4d881028e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009900 tx=0x7f4d7802a410 comp rx=0 tx=0).stop 2024-07-27T06:36:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.911+0000 7f4d8de66640 1 -- 172.21.15.29:0/793924724 shutdown_connections 2024-07-27T06:36:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.911+0000 7f4d8de66640 1 --2- 172.21.15.29:0/793924724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d88108760 0x7f4d8810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.911+0000 7f4d8de66640 1 --2- 172.21.15.29:0/793924724 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d88107e10 0x7f4d88108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.911+0000 7f4d8de66640 1 --2- 172.21.15.29:0/793924724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d881024d0 0x7f4d881028e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.911+0000 7f4d8de66640 1 -- 172.21.15.29:0/793924724 >> 172.21.15.29:0/793924724 conn(0x7f4d88079210 msgr2=0x7f4d88079620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.911+0000 7f4d8de66640 1 -- 172.21.15.29:0/793924724 shutdown_connections 2024-07-27T06:36:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.911+0000 7f4d8de66640 1 -- 172.21.15.29:0/793924724 wait complete. 2024-07-27T06:36:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.912+0000 7f4d8de66640 1 Processor -- start 2024-07-27T06:36:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.912+0000 7f4d8de66640 1 -- start start 2024-07-27T06:36:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.912+0000 7f4d8de66640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d881024d0 0x7f4d88199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.912+0000 7f4d8de66640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d88107e10 0x7f4d8819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.912+0000 7f4d8de66640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d88108760 0x7f4d8819f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.912+0000 7f4d8de66640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88112280 con 0x7f4d88107e10 2024-07-27T06:36:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.912+0000 7f4d8de66640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88112100 con 0x7f4d88108760 2024-07-27T06:36:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.912+0000 7f4d8de66640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88112400 con 0x7f4d881024d0 2024-07-27T06:36:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.912+0000 7f4d86ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d88107e10 0x7f4d8819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:50.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.912+0000 7f4d86ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d88107e10 0x7f4d8819a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58660/0 (socket says 172.21.15.29:58660) 2024-07-27T06:36:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.912+0000 7f4d86ffd640 1 -- 172.21.15.29:0/736567350 learned_addr learned my addr 172.21.15.29:0/736567350 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:36:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d877fe640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d881024d0 0x7f4d88199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d87fff640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d88108760 0x7f4d8819f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d86ffd640 1 -- 172.21.15.29:0/736567350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d881024d0 msgr2=0x7f4d88199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d86ffd640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d881024d0 0x7f4d88199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d86ffd640 1 -- 172.21.15.29:0/736567350 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d88108760 msgr2=0x7f4d8819f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d86ffd640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d88108760 0x7f4d8819f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d86ffd640 1 -- 172.21.15.29:0/736567350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8819fd10 con 0x7f4d88107e10 2024-07-27T06:36:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d86ffd640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d88107e10 0x7f4d8819a340 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00c970 tx=0x7f4d7c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d84ff9640 1 -- 172.21.15.29:0/736567350 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c007bb0 con 0x7f4d88107e10 2024-07-27T06:36:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d84ff9640 1 -- 172.21.15.29:0/736567350 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4d7c007d50 con 0x7f4d88107e10 2024-07-27T06:36:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d8de66640 1 -- 172.21.15.29:0/736567350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d881a0000 con 0x7f4d88107e10 2024-07-27T06:36:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d84ff9640 1 -- 172.21.15.29:0/736567350 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c004e20 con 0x7f4d88107e10 2024-07-27T06:36:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.913+0000 7f4d8de66640 1 -- 172.21.15.29:0/736567350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d881a6760 con 0x7f4d88107e10 2024-07-27T06:36:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.914+0000 7f4d8de66640 1 -- 172.21.15.29:0/736567350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d881028e0 con 0x7f4d88107e10 2024-07-27T06:36:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.916+0000 7f4d84ff9640 1 -- 172.21.15.29:0/736567350 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f4d7c004fe0 con 0x7f4d88107e10 2024-07-27T06:36:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.917+0000 7f4d84ff9640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4d700b3b90 0x7f4d700b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.917+0000 7f4d877fe640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4d700b3b90 0x7f4d700b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.917+0000 7f4d84ff9640 1 -- 172.21.15.29:0/736567350 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f4d7c0e2210 con 0x7f4d88107e10 2024-07-27T06:36:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.917+0000 7f4d877fe640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4d700b3b90 0x7f4d700b60a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009230 tx=0x7f4d780058d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:49.920+0000 7f4d84ff9640 1 -- 172.21.15.29:0/736567350 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4d7c014030 con 0x7f4d88107e10 2024-07-27T06:36:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.105+0000 7f4d8de66640 1 -- 172.21.15.29:0/736567350 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f4d881a6990 con 0x7f4d88107e10 2024-07-27T06:36:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.109+0000 7f4d84ff9640 1 -- 172.21.15.29:0/736567350 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v25)=0 v25) ==== 119+0+0 (secure 0 0 0) 0x7f4d7c0ab3c0 con 0x7f4d88107e10 2024-07-27T06:36:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.111+0000 7f4d8de66640 1 -- 172.21.15.29:0/736567350 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4d700b3b90 msgr2=0x7f4d700b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.112+0000 7f4d8de66640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4d700b3b90 0x7f4d700b60a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009230 tx=0x7f4d780058d0 comp rx=0 tx=0).stop 2024-07-27T06:36:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.112+0000 7f4d8de66640 1 -- 172.21.15.29:0/736567350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d88107e10 msgr2=0x7f4d8819a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.112+0000 7f4d8de66640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d88107e10 0x7f4d8819a340 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00c970 tx=0x7f4d7c00ce30 comp rx=0 tx=0).stop 2024-07-27T06:36:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.112+0000 7f4d8de66640 1 -- 172.21.15.29:0/736567350 shutdown_connections 2024-07-27T06:36:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.112+0000 7f4d8de66640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d88108760 0x7f4d8819f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.112+0000 7f4d8de66640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d88107e10 0x7f4d8819a340 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.112+0000 7f4d8de66640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4d700b3b90 0x7f4d700b60a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:50.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.112+0000 7f4d8de66640 1 --2- 172.21.15.29:0/736567350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d881024d0 0x7f4d88199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:50.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.112+0000 7f4d8de66640 1 -- 172.21.15.29:0/736567350 >> 172.21.15.29:0/736567350 conn(0x7f4d88079210 msgr2=0x7f4d88101580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:50.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.112+0000 7f4d8de66640 1 -- 172.21.15.29:0/736567350 shutdown_connections 2024-07-27T06:36:50.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:50.112+0000 7f4d8de66640 1 -- 172.21.15.29:0/736567350 wait complete. 2024-07-27T06:36:50.615 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:50 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/780280034' entity='client.admin' 2024-07-27T06:36:50.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:50 smithi029 ceph-mon[32610]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:50.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:50 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/736567350' entity='client.admin' 2024-07-27T06:36:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:50 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/780280034' entity='client.admin' 2024-07-27T06:36:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:50 smithi077 ceph-mon[37153]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:50 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/736567350' entity='client.admin' 2024-07-27T06:36:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:50 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/780280034' entity='client.admin' 2024-07-27T06:36:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:50 smithi047 ceph-mon[36835]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:50 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/736567350' entity='client.admin' 2024-07-27T06:36:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:52 smithi029 ceph-mon[32610]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:52 smithi077 ceph-mon[37153]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:52 smithi047 ceph-mon[36835]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:54.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.397+0000 7fef99557640 1 -- 172.21.15.29:0/819958915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef94102490 msgr2=0x7fef941028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:54.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.397+0000 7fef99557640 1 --2- 172.21.15.29:0/819958915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef94102490 0x7fef941028a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fef84009900 tx=0x7fef8402a410 comp rx=0 tx=0).stop 2024-07-27T06:36:54.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.397+0000 7fef99557640 1 -- 172.21.15.29:0/819958915 shutdown_connections 2024-07-27T06:36:54.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.397+0000 7fef99557640 1 --2- 172.21.15.29:0/819958915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef94108720 0x7fef9410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:54.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.397+0000 7fef99557640 1 --2- 172.21.15.29:0/819958915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef94107dd0 0x7fef941081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:54.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.397+0000 7fef99557640 1 --2- 172.21.15.29:0/819958915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef94102490 0x7fef941028a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:54.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.397+0000 7fef99557640 1 -- 172.21.15.29:0/819958915 >> 172.21.15.29:0/819958915 conn(0x7fef94079160 msgr2=0x7fef94079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:54.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.397+0000 7fef99557640 1 -- 172.21.15.29:0/819958915 shutdown_connections 2024-07-27T06:36:54.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.397+0000 7fef99557640 1 -- 172.21.15.29:0/819958915 wait complete. 2024-07-27T06:36:54.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.398+0000 7fef99557640 1 Processor -- start 2024-07-27T06:36:54.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef99557640 1 -- start start 2024-07-27T06:36:54.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef99557640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef94102490 0x7fef9419ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:54.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef99557640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef94107dd0 0x7fef9419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:54.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef99557640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef94108720 0x7fef9419b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:54.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef99557640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fef94112260 con 0x7fef94102490 2024-07-27T06:36:54.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef99557640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fef941120e0 con 0x7fef94108720 2024-07-27T06:36:54.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef99557640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fef941123e0 con 0x7fef94107dd0 2024-07-27T06:36:54.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef94102490 0x7fef9419ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:54.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef92ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef94102490 0x7fef9419ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58674/0 (socket says 172.21.15.29:58674) 2024-07-27T06:36:54.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef92ffd640 1 -- 172.21.15.29:0/3005745978 learned_addr learned my addr 172.21.15.29:0/3005745978 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:36:54.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef927fc640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef94107dd0 0x7fef9419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:54.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef937fe640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef94108720 0x7fef9419b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:54.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef92ffd640 1 -- 172.21.15.29:0/3005745978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef94107dd0 msgr2=0x7fef9419b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:54.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef92ffd640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef94107dd0 0x7fef9419b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:54.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef92ffd640 1 -- 172.21.15.29:0/3005745978 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef94108720 msgr2=0x7fef9419b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:54.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef92ffd640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef94108720 0x7fef9419b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:54.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef92ffd640 1 -- 172.21.15.29:0/3005745978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef941a5ea0 con 0x7fef94102490 2024-07-27T06:36:54.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.399+0000 7fef927fc640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef94107dd0 0x7fef9419b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:36:54.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.400+0000 7fef92ffd640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef94102490 0x7fef9419ab90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fef84006010 tx=0x7fef840040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:54.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.400+0000 7fef8bfff640 1 -- 172.21.15.29:0/3005745978 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef8403f030 con 0x7fef94102490 2024-07-27T06:36:54.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.400+0000 7fef8bfff640 1 -- 172.21.15.29:0/3005745978 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fef84003a80 con 0x7fef94102490 2024-07-27T06:36:54.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.400+0000 7fef99557640 1 -- 172.21.15.29:0/3005745978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef941a6130 con 0x7fef94102490 2024-07-27T06:36:54.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.400+0000 7fef8bfff640 1 -- 172.21.15.29:0/3005745978 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef84035070 con 0x7fef94102490 2024-07-27T06:36:54.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.400+0000 7fef99557640 1 -- 172.21.15.29:0/3005745978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef941a6610 con 0x7fef94102490 2024-07-27T06:36:54.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.401+0000 7fef99557640 1 -- 172.21.15.29:0/3005745978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef941028a0 con 0x7fef94102490 2024-07-27T06:36:54.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.403+0000 7fef8bfff640 1 -- 172.21.15.29:0/3005745978 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7fef84042030 con 0x7fef94102490 2024-07-27T06:36:54.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.403+0000 7fef8bfff640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fef800b3b90 0x7fef800b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:54.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.403+0000 7fef927fc640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fef800b3b90 0x7fef800b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:54.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.404+0000 7fef8bfff640 1 -- 172.21.15.29:0/3005745978 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fef84102530 con 0x7fef94102490 2024-07-27T06:36:54.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.404+0000 7fef927fc640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fef800b3b90 0x7fef800b60a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fef94100750 tx=0x7fef7c005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:54.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.408+0000 7fef8bfff640 1 -- 172.21.15.29:0/3005745978 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fef840cb780 con 0x7fef94102490 2024-07-27T06:36:54.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.592+0000 7fef99557640 1 -- 172.21.15.29:0/3005745978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7fef9410a4f0 con 0x7fef94102490 2024-07-27T06:36:54.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.596+0000 7fef8bfff640 1 -- 172.21.15.29:0/3005745978 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v26)=0 v26) ==== 125+0+0 (secure 0 0 0) 0x7fef840cb080 con 0x7fef94102490 2024-07-27T06:36:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.599+0000 7fef99557640 1 -- 172.21.15.29:0/3005745978 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fef800b3b90 msgr2=0x7fef800b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:54.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.599+0000 7fef99557640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fef800b3b90 0x7fef800b60a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fef94100750 tx=0x7fef7c005f90 comp rx=0 tx=0).stop 2024-07-27T06:36:54.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.599+0000 7fef99557640 1 -- 172.21.15.29:0/3005745978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef94102490 msgr2=0x7fef9419ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:54.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.599+0000 7fef99557640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef94102490 0x7fef9419ab90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fef84006010 tx=0x7fef840040c0 comp rx=0 tx=0).stop 2024-07-27T06:36:54.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.599+0000 7fef99557640 1 -- 172.21.15.29:0/3005745978 shutdown_connections 2024-07-27T06:36:54.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.599+0000 7fef99557640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef94108720 0x7fef9419b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:54.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.599+0000 7fef99557640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fef800b3b90 0x7fef800b60a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:54.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.599+0000 7fef99557640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef94107dd0 0x7fef9419b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:54.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.599+0000 7fef99557640 1 --2- 172.21.15.29:0/3005745978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef94102490 0x7fef9419ab90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:54.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.599+0000 7fef99557640 1 -- 172.21.15.29:0/3005745978 >> 172.21.15.29:0/3005745978 conn(0x7fef94079160 msgr2=0x7fef94100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:54.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.599+0000 7fef99557640 1 -- 172.21.15.29:0/3005745978 shutdown_connections 2024-07-27T06:36:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:54.599+0000 7fef99557640 1 -- 172.21.15.29:0/3005745978 wait complete. 2024-07-27T06:36:54.809 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:54 smithi029 ceph-mon[32610]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:54 smithi077 ceph-mon[37153]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:54 smithi047 ceph-mon[36835]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:55.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.303+0000 7efc7653e640 1 -- 172.21.15.29:0/798989826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc70103590 msgr2=0x7efc701039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:55.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.303+0000 7efc7653e640 1 --2- 172.21.15.29:0/798989826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc70103590 0x7efc701039a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efc60009900 tx=0x7efc6002a410 comp rx=0 tx=0).stop 2024-07-27T06:36:55.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.303+0000 7efc7653e640 1 -- 172.21.15.29:0/798989826 shutdown_connections 2024-07-27T06:36:55.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.303+0000 7efc7653e640 1 --2- 172.21.15.29:0/798989826 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc70109820 0x7efc70110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:55.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.303+0000 7efc7653e640 1 --2- 172.21.15.29:0/798989826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc70108ed0 0x7efc701092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:55.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.303+0000 7efc7653e640 1 --2- 172.21.15.29:0/798989826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc70103590 0x7efc701039a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:55.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.303+0000 7efc7653e640 1 -- 172.21.15.29:0/798989826 >> 172.21.15.29:0/798989826 conn(0x7efc7007b280 msgr2=0x7efc7007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:55.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.303+0000 7efc7653e640 1 -- 172.21.15.29:0/798989826 shutdown_connections 2024-07-27T06:36:55.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.303+0000 7efc7653e640 1 -- 172.21.15.29:0/798989826 wait complete. 2024-07-27T06:36:55.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.304+0000 7efc7653e640 1 Processor -- start 2024-07-27T06:36:55.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.304+0000 7efc7653e640 1 -- start start 2024-07-27T06:36:55.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.304+0000 7efc7653e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc70103590 0x7efc70199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:55.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.304+0000 7efc7653e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc70108ed0 0x7efc7019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:55.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.304+0000 7efc7653e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc70109820 0x7efc7019f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:55.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.304+0000 7efc7653e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efc70113390 con 0x7efc70108ed0 2024-07-27T06:36:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.304+0000 7efc7653e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7efc70113210 con 0x7efc70109820 2024-07-27T06:36:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.304+0000 7efc7653e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efc70113510 con 0x7efc70103590 2024-07-27T06:36:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc70108ed0 0x7efc7019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc70108ed0 0x7efc7019a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58686/0 (socket says 172.21.15.29:58686) 2024-07-27T06:36:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6f7fe640 1 -- 172.21.15.29:0/3834724057 learned_addr learned my addr 172.21.15.29:0/3834724057 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:36:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6ffff640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc70103590 0x7efc70199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc74adb640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc70109820 0x7efc7019f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6f7fe640 1 -- 172.21.15.29:0/3834724057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc70103590 msgr2=0x7efc70199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6f7fe640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc70103590 0x7efc70199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6f7fe640 1 -- 172.21.15.29:0/3834724057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc70109820 msgr2=0x7efc7019f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6f7fe640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc70109820 0x7efc7019f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:55.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6f7fe640 1 -- 172.21.15.29:0/3834724057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7019fdd0 con 0x7efc70108ed0 2024-07-27T06:36:55.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6f7fe640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc70108ed0 0x7efc7019a400 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efc6400d740 tx=0x7efc6400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:55.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6d7fa640 1 -- 172.21.15.29:0/3834724057 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc64017070 con 0x7efc70108ed0 2024-07-27T06:36:55.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6d7fa640 1 -- 172.21.15.29:0/3834724057 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efc6400de50 con 0x7efc70108ed0 2024-07-27T06:36:55.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc7653e640 1 -- 172.21.15.29:0/3834724057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc701a00c0 con 0x7efc70108ed0 2024-07-27T06:36:55.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.305+0000 7efc6d7fa640 1 -- 172.21.15.29:0/3834724057 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6400f620 con 0x7efc70108ed0 2024-07-27T06:36:55.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.306+0000 7efc7653e640 1 -- 172.21.15.29:0/3834724057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc70075020 con 0x7efc70108ed0 2024-07-27T06:36:55.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.306+0000 7efc7653e640 1 -- 172.21.15.29:0/3834724057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc701039a0 con 0x7efc70108ed0 2024-07-27T06:36:55.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.308+0000 7efc6d7fa640 1 -- 172.21.15.29:0/3834724057 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7efc6401a070 con 0x7efc70108ed0 2024-07-27T06:36:55.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.309+0000 7efc6d7fa640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efc580b3c10 0x7efc580b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:55.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.309+0000 7efc6ffff640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efc580b3c10 0x7efc580b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:55.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.309+0000 7efc6d7fa640 1 -- 172.21.15.29:0/3834724057 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7efc640e1980 con 0x7efc70108ed0 2024-07-27T06:36:55.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.310+0000 7efc6ffff640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efc580b3c10 0x7efc580b6120 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efc60009230 tx=0x7efc600058d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:55.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.312+0000 7efc6d7fa640 1 -- 172.21.15.29:0/3834724057 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7efc640aac50 con 0x7efc70108ed0 2024-07-27T06:36:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.499+0000 7efc7653e640 1 -- 172.21.15.29:0/3834724057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7efc7010b5a0 con 0x7efc70108ed0 2024-07-27T06:36:55.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.503+0000 7efc6d7fa640 1 -- 172.21.15.29:0/3834724057 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v27)=0 v27) ==== 120+0+0 (secure 0 0 0) 0x7efc640aa550 con 0x7efc70108ed0 2024-07-27T06:36:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.505+0000 7efc7653e640 1 -- 172.21.15.29:0/3834724057 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efc580b3c10 msgr2=0x7efc580b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.505+0000 7efc7653e640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efc580b3c10 0x7efc580b6120 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efc60009230 tx=0x7efc600058d0 comp rx=0 tx=0).stop 2024-07-27T06:36:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.505+0000 7efc7653e640 1 -- 172.21.15.29:0/3834724057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc70108ed0 msgr2=0x7efc7019a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.505+0000 7efc7653e640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc70108ed0 0x7efc7019a400 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efc6400d740 tx=0x7efc6400dc00 comp rx=0 tx=0).stop 2024-07-27T06:36:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.506+0000 7efc7653e640 1 -- 172.21.15.29:0/3834724057 shutdown_connections 2024-07-27T06:36:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.506+0000 7efc7653e640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc70109820 0x7efc7019f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.506+0000 7efc7653e640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc70108ed0 0x7efc7019a400 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.506+0000 7efc7653e640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efc580b3c10 0x7efc580b6120 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.506+0000 7efc7653e640 1 --2- 172.21.15.29:0/3834724057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efc70103590 0x7efc70199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.506+0000 7efc7653e640 1 -- 172.21.15.29:0/3834724057 >> 172.21.15.29:0/3834724057 conn(0x7efc7007b280 msgr2=0x7efc701070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.506+0000 7efc7653e640 1 -- 172.21.15.29:0/3834724057 shutdown_connections 2024-07-27T06:36:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:55.506+0000 7efc7653e640 1 -- 172.21.15.29:0/3834724057 wait complete. 2024-07-27T06:36:55.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:55 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3005745978' entity='client.admin' 2024-07-27T06:36:55.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:55 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3834724057' entity='client.admin' 2024-07-27T06:36:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:55 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3005745978' entity='client.admin' 2024-07-27T06:36:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:55 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3834724057' entity='client.admin' 2024-07-27T06:36:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:55 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3005745978' entity='client.admin' 2024-07-27T06:36:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:55 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3834724057' entity='client.admin' 2024-07-27T06:36:56.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.597+0000 7f2191ac7640 1 -- 172.21.15.29:0/3061361319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f218c107dd0 msgr2=0x7f218c1081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:56.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.597+0000 7f2191ac7640 1 --2- 172.21.15.29:0/3061361319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f218c107dd0 0x7f218c1081e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2180009900 tx=0x7f218002a410 comp rx=0 tx=0).stop 2024-07-27T06:36:56.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.597+0000 7f2191ac7640 1 -- 172.21.15.29:0/3061361319 shutdown_connections 2024-07-27T06:36:56.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.597+0000 7f2191ac7640 1 --2- 172.21.15.29:0/3061361319 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f218c108720 0x7f218c10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:56.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.597+0000 7f2191ac7640 1 --2- 172.21.15.29:0/3061361319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f218c107dd0 0x7f218c1081e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:56.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.597+0000 7f2191ac7640 1 --2- 172.21.15.29:0/3061361319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f218c102490 0x7f218c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:56.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.597+0000 7f2191ac7640 1 -- 172.21.15.29:0/3061361319 >> 172.21.15.29:0/3061361319 conn(0x7f218c079160 msgr2=0x7f218c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:56.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.597+0000 7f2191ac7640 1 -- 172.21.15.29:0/3061361319 shutdown_connections 2024-07-27T06:36:56.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.597+0000 7f2191ac7640 1 -- 172.21.15.29:0/3061361319 wait complete. 2024-07-27T06:36:56.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.598+0000 7f2191ac7640 1 Processor -- start 2024-07-27T06:36:56.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.598+0000 7f2191ac7640 1 -- start start 2024-07-27T06:36:56.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.598+0000 7f2191ac7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f218c102490 0x7f218c199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:56.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.598+0000 7f2191ac7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f218c107dd0 0x7f218c19a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.598+0000 7f2191ac7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f218c108720 0x7f218c19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.598+0000 7f2191ac7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f218c112230 con 0x7f218c102490 2024-07-27T06:36:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.598+0000 7f2191ac7640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f218c1120b0 con 0x7f218c107dd0 2024-07-27T06:36:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.598+0000 7f2191ac7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f218c1123b0 con 0x7f218c108720 2024-07-27T06:36:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f218b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f218c102490 0x7f218c199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f218b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f218c102490 0x7f218c199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58716/0 (socket says 172.21.15.29:58716) 2024-07-27T06:36:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f218b7fe640 1 -- 172.21.15.29:0/3289066489 learned_addr learned my addr 172.21.15.29:0/3289066489 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:36:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f218affd640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f218c107dd0 0x7f218c19a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f218bfff640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f218c108720 0x7f218c19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f218b7fe640 1 -- 172.21.15.29:0/3289066489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f218c108720 msgr2=0x7f218c19f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f218b7fe640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f218c108720 0x7f218c19f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f218b7fe640 1 -- 172.21.15.29:0/3289066489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f218c107dd0 msgr2=0x7f218c19a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f218b7fe640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f218c107dd0 0x7f218c19a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f218b7fe640 1 -- 172.21.15.29:0/3289066489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218c19fdc0 con 0x7f218c102490 2024-07-27T06:36:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f218b7fe640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f218c102490 0x7f218c199e50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f217c00ea80 tx=0x7f217c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f2188ff9640 1 -- 172.21.15.29:0/3289066489 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c00ce40 con 0x7f218c102490 2024-07-27T06:36:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f2188ff9640 1 -- 172.21.15.29:0/3289066489 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f217c005020 con 0x7f218c102490 2024-07-27T06:36:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.599+0000 7f2191ac7640 1 -- 172.21.15.29:0/3289066489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f218c1a00b0 con 0x7f218c102490 2024-07-27T06:36:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.600+0000 7f2188ff9640 1 -- 172.21.15.29:0/3289066489 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c0116d0 con 0x7f218c102490 2024-07-27T06:36:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.600+0000 7f2191ac7640 1 -- 172.21.15.29:0/3289066489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f218c1a6810 con 0x7f218c102490 2024-07-27T06:36:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.601+0000 7f2191ac7640 1 -- 172.21.15.29:0/3289066489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f218c103710 con 0x7f218c102490 2024-07-27T06:36:56.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.603+0000 7f2188ff9640 1 -- 172.21.15.29:0/3289066489 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f217c004a40 con 0x7f218c102490 2024-07-27T06:36:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.603+0000 7f2188ff9640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f21740b3c10 0x7f21740b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.603+0000 7f218affd640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f21740b3c10 0x7f21740b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.603+0000 7f2188ff9640 1 -- 172.21.15.29:0/3289066489 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f217c0e26b0 con 0x7f218c102490 2024-07-27T06:36:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.604+0000 7f218affd640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f21740b3c10 0x7f21740b6120 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2180004770 tx=0x7f21800058d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:56.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.609+0000 7f2188ff9640 1 -- 172.21.15.29:0/3289066489 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f217c016030 con 0x7f218c102490 2024-07-27T06:36:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:56 smithi029 ceph-mon[32610]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:56.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.797+0000 7f2191ac7640 1 -- 172.21.15.29:0/3289066489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7f218c19af80 con 0x7f218c102490 2024-07-27T06:36:56.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.801+0000 7f2188ff9640 1 -- 172.21.15.29:0/3289066489 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v28)=0 v28) ==== 116+0+0 (secure 0 0 0) 0x7f217c0ab980 con 0x7f218c102490 2024-07-27T06:36:56.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.804+0000 7f2191ac7640 1 -- 172.21.15.29:0/3289066489 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f21740b3c10 msgr2=0x7f21740b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:56.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.804+0000 7f2191ac7640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f21740b3c10 0x7f21740b6120 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2180004770 tx=0x7f21800058d0 comp rx=0 tx=0).stop 2024-07-27T06:36:56.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.804+0000 7f2191ac7640 1 -- 172.21.15.29:0/3289066489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f218c102490 msgr2=0x7f218c199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:56.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.804+0000 7f2191ac7640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f218c102490 0x7f218c199e50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f217c00ea80 tx=0x7f217c00ef40 comp rx=0 tx=0).stop 2024-07-27T06:36:56.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.804+0000 7f2191ac7640 1 -- 172.21.15.29:0/3289066489 shutdown_connections 2024-07-27T06:36:56.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.804+0000 7f2191ac7640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f21740b3c10 0x7f21740b6120 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:56.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.804+0000 7f2191ac7640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f218c108720 0x7f218c19f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:56.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.804+0000 7f2191ac7640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f218c107dd0 0x7f218c19a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:56.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.804+0000 7f2191ac7640 1 --2- 172.21.15.29:0/3289066489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f218c102490 0x7f218c199e50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:56.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.804+0000 7f2191ac7640 1 -- 172.21.15.29:0/3289066489 >> 172.21.15.29:0/3289066489 conn(0x7f218c079160 msgr2=0x7f218c0ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:56.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.804+0000 7f2191ac7640 1 -- 172.21.15.29:0/3289066489 shutdown_connections 2024-07-27T06:36:56.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:56.804+0000 7f2191ac7640 1 -- 172.21.15.29:0/3289066489 wait complete. 2024-07-27T06:36:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:56 smithi077 ceph-mon[37153]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:56 smithi047 ceph-mon[36835]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.616+0000 7f3c690e8640 1 -- 172.21.15.29:0/3561378997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c64108720 msgr2=0x7f3c6410ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.616+0000 7f3c690e8640 1 --2- 172.21.15.29:0/3561378997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c64108720 0x7f3c6410ef50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009ca0 tx=0x7f3c5c02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:36:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.616+0000 7f3c690e8640 1 -- 172.21.15.29:0/3561378997 shutdown_connections 2024-07-27T06:36:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.616+0000 7f3c690e8640 1 --2- 172.21.15.29:0/3561378997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c64108720 0x7f3c6410ef50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.616+0000 7f3c690e8640 1 --2- 172.21.15.29:0/3561378997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c64107dd0 0x7f3c641081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.616+0000 7f3c690e8640 1 --2- 172.21.15.29:0/3561378997 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c64102490 0x7f3c641028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.616+0000 7f3c690e8640 1 -- 172.21.15.29:0/3561378997 >> 172.21.15.29:0/3561378997 conn(0x7f3c64079160 msgr2=0x7f3c64079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.616+0000 7f3c690e8640 1 -- 172.21.15.29:0/3561378997 shutdown_connections 2024-07-27T06:36:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.616+0000 7f3c690e8640 1 -- 172.21.15.29:0/3561378997 wait complete. 2024-07-27T06:36:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.617+0000 7f3c690e8640 1 Processor -- start 2024-07-27T06:36:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.617+0000 7f3c690e8640 1 -- start start 2024-07-27T06:36:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.617+0000 7f3c690e8640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c64102490 0x7f3c64199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.617+0000 7f3c690e8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c64107dd0 0x7f3c6419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.617+0000 7f3c690e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c64108720 0x7f3c6419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.617+0000 7f3c690e8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3c64112170 con 0x7f3c64108720 2024-07-27T06:36:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c690e8640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3c64111ff0 con 0x7f3c64102490 2024-07-27T06:36:57.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c690e8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3c641122f0 con 0x7f3c64107dd0 2024-07-27T06:36:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c6359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c64108720 0x7f3c6419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c6359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c64108720 0x7f3c6419f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57008/0 (socket says 172.21.15.29:57008) 2024-07-27T06:36:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c6359e640 1 -- 172.21.15.29:0/376133328 learned_addr learned my addr 172.21.15.29:0/376133328 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:36:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c6259c640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c64107dd0 0x7f3c6419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c62d9d640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c64102490 0x7f3c64199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:57.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c6359e640 1 -- 172.21.15.29:0/376133328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c64107dd0 msgr2=0x7f3c6419a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c6359e640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c64107dd0 0x7f3c6419a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c6359e640 1 -- 172.21.15.29:0/376133328 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c64102490 msgr2=0x7f3c64199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c6359e640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c64102490 0x7f3c64199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c6359e640 1 -- 172.21.15.29:0/376133328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6419fc40 con 0x7f3c64108720 2024-07-27T06:36:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c6359e640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c64108720 0x7f3c6419f5d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c032ff0 tx=0x7f3c5c009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c4bfff640 1 -- 172.21.15.29:0/376133328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c003db0 con 0x7f3c64108720 2024-07-27T06:36:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c4bfff640 1 -- 172.21.15.29:0/376133328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(24 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f3c5c0044b0 con 0x7f3c64108720 2024-07-27T06:36:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c690e8640 1 -- 172.21.15.29:0/376133328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c6419fed0 con 0x7f3c64108720 2024-07-27T06:36:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.618+0000 7f3c4bfff640 1 -- 172.21.15.29:0/376133328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c030b00 con 0x7f3c64108720 2024-07-27T06:36:57.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.619+0000 7f3c690e8640 1 -- 172.21.15.29:0/376133328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c641a0410 con 0x7f3c64108720 2024-07-27T06:36:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.620+0000 7f3c690e8640 1 -- 172.21.15.29:0/376133328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c641028a0 con 0x7f3c64108720 2024-07-27T06:36:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.621+0000 7f3c4bfff640 1 -- 172.21.15.29:0/376133328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f3c5c030ca0 con 0x7f3c64108720 2024-07-27T06:36:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.622+0000 7f3c4bfff640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3c4c0b3bc0 0x7f3c4c0b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.622+0000 7f3c62d9d640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3c4c0b3bc0 0x7f3c4c0b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.622+0000 7f3c4bfff640 1 -- 172.21.15.29:0/376133328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f3c5c103600 con 0x7f3c64108720 2024-07-27T06:36:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.623+0000 7f3c62d9d640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3c4c0b3bc0 0x7f3c4c0b60d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009b60 tx=0x7f3c54005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.626+0000 7f3c4bfff640 1 -- 172.21.15.29:0/376133328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3c5c0cc7b0 con 0x7f3c64108720 2024-07-27T06:36:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.813+0000 7f3c690e8640 1 -- 172.21.15.29:0/376133328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f3c641a67e0 con 0x7f3c64108720 2024-07-27T06:36:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.817+0000 7f3c4bfff640 1 -- 172.21.15.29:0/376133328 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v22)=0 v22) ==== 123+0+0 (secure 0 0 0) 0x7f3c5c0cc0b0 con 0x7f3c64108720 2024-07-27T06:36:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.818+0000 7f3c4bfff640 1 -- 172.21.15.29:0/376133328 <== mon.0 v2:172.21.15.29:3300/0 8 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3c5c0cbbc0 con 0x7f3c64108720 2024-07-27T06:36:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.819+0000 7f3c690e8640 1 -- 172.21.15.29:0/376133328 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3c4c0b3bc0 msgr2=0x7f3c4c0b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.819+0000 7f3c690e8640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3c4c0b3bc0 0x7f3c4c0b60d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009b60 tx=0x7f3c54005f90 comp rx=0 tx=0).stop 2024-07-27T06:36:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.819+0000 7f3c690e8640 1 -- 172.21.15.29:0/376133328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c64108720 msgr2=0x7f3c6419f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.819+0000 7f3c690e8640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c64108720 0x7f3c6419f5d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c032ff0 tx=0x7f3c5c009360 comp rx=0 tx=0).stop 2024-07-27T06:36:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.819+0000 7f3c690e8640 1 -- 172.21.15.29:0/376133328 shutdown_connections 2024-07-27T06:36:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.819+0000 7f3c690e8640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c64108720 0x7f3c6419f5d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:57.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.819+0000 7f3c690e8640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3c4c0b3bc0 0x7f3c4c0b60d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.819+0000 7f3c690e8640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c64107dd0 0x7f3c6419a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.819+0000 7f3c690e8640 1 --2- 172.21.15.29:0/376133328 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c64102490 0x7f3c64199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.819+0000 7f3c690e8640 1 -- 172.21.15.29:0/376133328 >> 172.21.15.29:0/376133328 conn(0x7f3c64079160 msgr2=0x7f3c64101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.819+0000 7f3c690e8640 1 -- 172.21.15.29:0/376133328 shutdown_connections 2024-07-27T06:36:57.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:46.819+0000 7f3c690e8640 1 -- 172.21.15.29:0/376133328 wait complete. 2024-07-27T06:36:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:57 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3289066489' entity='client.admin' 2024-07-27T06:36:58.110 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:57 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3289066489' entity='client.admin' 2024-07-27T06:36:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:57 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3289066489' entity='client.admin' 2024-07-27T06:36:58.604 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi029') 2024-07-27T06:36:58.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.894+0000 7fd4ca976640 1 -- 172.21.15.29:0/2352858312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4c4105ce0 msgr2=0x7fd4c41060f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:58.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.894+0000 7fd4ca976640 1 --2- 172.21.15.29:0/2352858312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4c4105ce0 0x7fd4c41060f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009900 tx=0x7fd4b802a410 comp rx=0 tx=0).stop 2024-07-27T06:36:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.894+0000 7fd4ca976640 1 -- 172.21.15.29:0/2352858312 shutdown_connections 2024-07-27T06:36:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.894+0000 7fd4ca976640 1 --2- 172.21.15.29:0/2352858312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c4106630 0x7fd4c41100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.894+0000 7fd4ca976640 1 --2- 172.21.15.29:0/2352858312 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4c4105ce0 0x7fd4c41060f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.894+0000 7fd4ca976640 1 --2- 172.21.15.29:0/2352858312 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4c4101a90 0x7fd4c4101ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.894+0000 7fd4ca976640 1 -- 172.21.15.29:0/2352858312 >> 172.21.15.29:0/2352858312 conn(0x7fd4c40776f0 msgr2=0x7fd4c4077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.894+0000 7fd4ca976640 1 -- 172.21.15.29:0/2352858312 shutdown_connections 2024-07-27T06:36:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.894+0000 7fd4ca976640 1 -- 172.21.15.29:0/2352858312 wait complete. 2024-07-27T06:36:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.894+0000 7fd4ca976640 1 Processor -- start 2024-07-27T06:36:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4ca976640 1 -- start start 2024-07-27T06:36:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4ca976640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4c4101a90 0x7fd4c4199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4ca976640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4c4105ce0 0x7fd4c419a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:58.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4ca976640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c4106630 0x7fd4c419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4ca976640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c41133a0 con 0x7fd4c4105ce0 2024-07-27T06:36:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4ca976640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c4113220 con 0x7fd4c4101a90 2024-07-27T06:36:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4ca976640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c4113520 con 0x7fd4c4106630 2024-07-27T06:36:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4c4105ce0 0x7fd4c419a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4c37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4c4105ce0 0x7fd4c419a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41936/0 (socket says 172.21.15.29:41936) 2024-07-27T06:36:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4c37fe640 1 -- 172.21.15.29:0/1398454338 learned_addr learned my addr 172.21.15.29:0/1398454338 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:36:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4c3fff640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4c4101a90 0x7fd4c4199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4c8f13640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c4106630 0x7fd4c419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.895+0000 7fd4c37fe640 1 -- 172.21.15.29:0/1398454338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c4106630 msgr2=0x7fd4c419f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.896+0000 7fd4c37fe640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c4106630 0x7fd4c419f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.896+0000 7fd4c37fe640 1 -- 172.21.15.29:0/1398454338 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4c4101a90 msgr2=0x7fd4c4199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:58.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.896+0000 7fd4c37fe640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4c4101a90 0x7fd4c4199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.896+0000 7fd4c37fe640 1 -- 172.21.15.29:0/1398454338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c419fda0 con 0x7fd4c4105ce0 2024-07-27T06:36:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.896+0000 7fd4c37fe640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4c4105ce0 0x7fd4c419a3d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8005940 tx=0x7fd4b80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.896+0000 7fd4c17fa640 1 -- 172.21.15.29:0/1398454338 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b803f030 con 0x7fd4c4105ce0 2024-07-27T06:36:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.896+0000 7fd4ca976640 1 -- 172.21.15.29:0/1398454338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c41a0030 con 0x7fd4c4105ce0 2024-07-27T06:36:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.896+0000 7fd4c17fa640 1 -- 172.21.15.29:0/1398454338 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd4b8003c90 con 0x7fd4c4105ce0 2024-07-27T06:36:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.896+0000 7fd4ca976640 1 -- 172.21.15.29:0/1398454338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c41a67f0 con 0x7fd4c4105ce0 2024-07-27T06:36:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.897+0000 7fd4c17fa640 1 -- 172.21.15.29:0/1398454338 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8035070 con 0x7fd4c4105ce0 2024-07-27T06:36:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.897+0000 7fd4ca976640 1 -- 172.21.15.29:0/1398454338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c406b2e0 con 0x7fd4c4105ce0 2024-07-27T06:36:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.900+0000 7fd4c17fa640 1 -- 172.21.15.29:0/1398454338 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7fd4b8042030 con 0x7fd4c4105ce0 2024-07-27T06:36:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.901+0000 7fd4c17fa640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd4ac0b3b90 0x7fd4ac0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.901+0000 7fd4c3fff640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd4ac0b3b90 0x7fd4ac0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:58.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.901+0000 7fd4c17fa640 1 -- 172.21.15.29:0/1398454338 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fd4b81029c0 con 0x7fd4c4105ce0 2024-07-27T06:36:58.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.901+0000 7fd4c3fff640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd4ac0b3b90 0x7fd4ac0b60a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd4c406b580 tx=0x7fd4b400b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:58.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.904+0000 7fd4c17fa640 1 -- 172.21.15.29:0/1398454338 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd4b80cb5c0 con 0x7fd4c4105ce0 2024-07-27T06:36:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:58 smithi029 ceph-mon[32610]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.792+0000 7f768cdb8640 1 -- 172.21.15.29:0/3629220955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688102490 msgr2=0x7f76881028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.792+0000 7f768cdb8640 1 --2- 172.21.15.29:0/3629220955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688102490 0x7f76881028a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7678009ca0 tx=0x7f767802a8c0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.792+0000 7f768cdb8640 1 -- 172.21.15.29:0/3629220955 shutdown_connections 2024-07-27T06:36:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.792+0000 7f768cdb8640 1 --2- 172.21.15.29:0/3629220955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7688108720 0x7f768810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.792+0000 7f768cdb8640 1 --2- 172.21.15.29:0/3629220955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688107dd0 0x7f76881081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.792+0000 7f768cdb8640 1 --2- 172.21.15.29:0/3629220955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688102490 0x7f76881028a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.792+0000 7f768cdb8640 1 -- 172.21.15.29:0/3629220955 >> 172.21.15.29:0/3629220955 conn(0x7f7688079160 msgr2=0x7f7688079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.792+0000 7f768cdb8640 1 -- 172.21.15.29:0/3629220955 shutdown_connections 2024-07-27T06:36:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.792+0000 7f768cdb8640 1 -- 172.21.15.29:0/3629220955 wait complete. 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.793+0000 7f768cdb8640 1 Processor -- start 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.793+0000 7f768cdb8640 1 -- start start 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.793+0000 7f768cdb8640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7688102490 0x7f7688198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.793+0000 7f768cdb8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688107dd0 0x7f7688199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.793+0000 7f768cdb8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688108720 0x7f768819e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.793+0000 7f768cdb8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7688112260 con 0x7f7688108720 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.793+0000 7f768cdb8640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f76881120e0 con 0x7f7688102490 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.793+0000 7f768cdb8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f76881123e0 con 0x7f7688107dd0 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f7686d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688108720 0x7f768819e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f7686d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688108720 0x7f768819e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41910/0 (socket says 172.21.15.29:41910) 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f7686d9d640 1 -- 172.21.15.29:0/3191395326 learned_addr learned my addr 172.21.15.29:0/3191395326 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f768659c640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7688102490 0x7f7688198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f7685d9b640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688107dd0 0x7f7688199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f7686d9d640 1 -- 172.21.15.29:0/3191395326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688107dd0 msgr2=0x7f7688199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f7686d9d640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688107dd0 0x7f7688199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f7686d9d640 1 -- 172.21.15.29:0/3191395326 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7688102490 msgr2=0x7f7688198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f7686d9d640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7688102490 0x7f7688198cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f7686d9d640 1 -- 172.21.15.29:0/3191395326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768819ec40 con 0x7f7688108720 2024-07-27T06:36:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f7686d9d640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688108720 0x7f768819e510 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f768000d750 tx=0x7f768000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f767f7fe640 1 -- 172.21.15.29:0/3191395326 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680017070 con 0x7f7688108720 2024-07-27T06:36:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f768cdb8640 1 -- 172.21.15.29:0/3191395326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f768819ef30 con 0x7f7688108720 2024-07-27T06:36:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f767f7fe640 1 -- 172.21.15.29:0/3191395326 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f76800054b0 con 0x7f7688108720 2024-07-27T06:36:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f767f7fe640 1 -- 172.21.15.29:0/3191395326 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680004ba0 con 0x7f7688108720 2024-07-27T06:36:59.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.794+0000 7f768cdb8640 1 -- 172.21.15.29:0/3191395326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76881a5690 con 0x7f7688108720 2024-07-27T06:36:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.796+0000 7f768cdb8640 1 -- 172.21.15.29:0/3191395326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76881028a0 con 0x7f7688108720 2024-07-27T06:36:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.797+0000 7f767f7fe640 1 -- 172.21.15.29:0/3191395326 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f7680004d40 con 0x7f7688108720 2024-07-27T06:36:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.798+0000 7f767f7fe640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f76740b3b90 0x7f76740b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.798+0000 7f768659c640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f76740b3b90 0x7f76740b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.798+0000 7f767f7fe640 1 -- 172.21.15.29:0/3191395326 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f76800e1b40 con 0x7f7688108720 2024-07-27T06:36:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.799+0000 7f768659c640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f76740b3b90 0x7f76740b60a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7678006010 tx=0x7f7678031000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.803+0000 7f767f7fe640 1 -- 172.21.15.29:0/3191395326 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f76800aacf0 con 0x7f7688108720 2024-07-27T06:36:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.992+0000 7f768cdb8640 1 -- 172.21.15.29:0/3191395326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f768810a4f0 con 0x7f7688108720 2024-07-27T06:36:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.996+0000 7f767f7fe640 1 -- 172.21.15.29:0/3191395326 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v29)=0 v29) ==== 106+0+0 (secure 0 0 0) 0x7f76800aa5f0 con 0x7f7688108720 2024-07-27T06:36:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.999+0000 7f768cdb8640 1 -- 172.21.15.29:0/3191395326 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f76740b3b90 msgr2=0x7f76740b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.999+0000 7f768cdb8640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f76740b3b90 0x7f76740b60a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7678006010 tx=0x7f7678031000 comp rx=0 tx=0).stop 2024-07-27T06:36:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.999+0000 7f768cdb8640 1 -- 172.21.15.29:0/3191395326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688108720 msgr2=0x7f768819e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.999+0000 7f768cdb8640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688108720 0x7f768819e510 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f768000d750 tx=0x7f768000dc10 comp rx=0 tx=0).stop 2024-07-27T06:36:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.999+0000 7f768cdb8640 1 -- 172.21.15.29:0/3191395326 shutdown_connections 2024-07-27T06:36:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.999+0000 7f768cdb8640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688108720 0x7f768819e510 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.999+0000 7f768cdb8640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f76740b3b90 0x7f76740b60a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.999+0000 7f768cdb8640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688107dd0 0x7f7688199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.999+0000 7f768cdb8640 1 --2- 172.21.15.29:0/3191395326 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7688102490 0x7f7688198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.999+0000 7f768cdb8640 1 -- 172.21.15.29:0/3191395326 >> 172.21.15.29:0/3191395326 conn(0x7f7688079160 msgr2=0x7f7688100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.999+0000 7f768cdb8640 1 -- 172.21.15.29:0/3191395326 shutdown_connections 2024-07-27T06:36:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:58.999+0000 7f768cdb8640 1 -- 172.21.15.29:0/3191395326 wait complete. 2024-07-27T06:36:59.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.098+0000 7fd4ca976640 1 -- 172.21.15.29:0/1398454338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7fd4c4101ea0 con 0x7fd4c4105ce0 2024-07-27T06:36:59.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.102+0000 7fd4c17fa640 1 -- 172.21.15.29:0/1398454338 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v30)=0 v30) ==== 127+0+0 (secure 0 0 0) 0x7fd4b80cfb10 con 0x7fd4c4105ce0 2024-07-27T06:36:59.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.105+0000 7fd4ca976640 1 -- 172.21.15.29:0/1398454338 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd4ac0b3b90 msgr2=0x7fd4ac0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:59.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.105+0000 7fd4ca976640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd4ac0b3b90 0x7fd4ac0b60a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd4c406b580 tx=0x7fd4b400b390 comp rx=0 tx=0).stop 2024-07-27T06:36:59.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.105+0000 7fd4ca976640 1 -- 172.21.15.29:0/1398454338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4c4105ce0 msgr2=0x7fd4c419a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:59.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.105+0000 7fd4ca976640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4c4105ce0 0x7fd4c419a3d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8005940 tx=0x7fd4b80042f0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.105+0000 7fd4ca976640 1 -- 172.21.15.29:0/1398454338 shutdown_connections 2024-07-27T06:36:59.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.105+0000 7fd4ca976640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd4ac0b3b90 0x7fd4ac0b60a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.105+0000 7fd4ca976640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4c4106630 0x7fd4c419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.105+0000 7fd4ca976640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4c4105ce0 0x7fd4c419a3d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.105+0000 7fd4ca976640 1 --2- 172.21.15.29:0/1398454338 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4c4101a90 0x7fd4c4199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.105+0000 7fd4ca976640 1 -- 172.21.15.29:0/1398454338 >> 172.21.15.29:0/1398454338 conn(0x7fd4c40776f0 msgr2=0x7fd4c4106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:59.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.105+0000 7fd4ca976640 1 -- 172.21.15.29:0/1398454338 shutdown_connections 2024-07-27T06:36:59.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.105+0000 7fd4ca976640 1 -- 172.21.15.29:0/1398454338 wait complete. 2024-07-27T06:36:59.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:58 smithi077 ceph-mon[37153]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:58 smithi047 ceph-mon[36835]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:36:59.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.833+0000 7ff372e6a640 1 -- 172.21.15.29:0/4137028038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c109870 msgr2=0x7ff36c1100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:59.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.833+0000 7ff372e6a640 1 --2- 172.21.15.29:0/4137028038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c109870 0x7ff36c1100a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff364009ca0 tx=0x7ff36402a8c0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.833+0000 7ff372e6a640 1 -- 172.21.15.29:0/4137028038 shutdown_connections 2024-07-27T06:36:59.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.833+0000 7ff372e6a640 1 --2- 172.21.15.29:0/4137028038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c109870 0x7ff36c1100a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.833+0000 7ff372e6a640 1 --2- 172.21.15.29:0/4137028038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff36c108f20 0x7ff36c109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.833+0000 7ff372e6a640 1 --2- 172.21.15.29:0/4137028038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c1035e0 0x7ff36c1039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.833+0000 7ff372e6a640 1 -- 172.21.15.29:0/4137028038 >> 172.21.15.29:0/4137028038 conn(0x7ff36c0ff030 msgr2=0x7ff36c101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:36:59.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.833+0000 7ff372e6a640 1 -- 172.21.15.29:0/4137028038 shutdown_connections 2024-07-27T06:36:59.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.833+0000 7ff372e6a640 1 -- 172.21.15.29:0/4137028038 wait complete. 2024-07-27T06:36:59.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff372e6a640 1 Processor -- start 2024-07-27T06:36:59.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff372e6a640 1 -- start start 2024-07-27T06:36:59.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff372e6a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c1035e0 0x7ff36c199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff372e6a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c108f20 0x7ff36c19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff372e6a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff36c109870 0x7ff36c19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff372e6a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c1133e0 con 0x7ff36c108f20 2024-07-27T06:36:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff372e6a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c113260 con 0x7ff36c109870 2024-07-27T06:36:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff372e6a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c113560 con 0x7ff36c1035e0 2024-07-27T06:36:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff36bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c108f20 0x7ff36c19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff36bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c108f20 0x7ff36c19a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41968/0 (socket says 172.21.15.29:41968) 2024-07-27T06:36:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff36bfff640 1 -- 172.21.15.29:0/3302901985 learned_addr learned my addr 172.21.15.29:0/3302901985 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:36:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff371407640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff36c109870 0x7ff36c19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.834+0000 7ff370c06640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c1035e0 0x7ff36c199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.835+0000 7ff36bfff640 1 -- 172.21.15.29:0/3302901985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c1035e0 msgr2=0x7ff36c199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.835+0000 7ff36bfff640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c1035e0 0x7ff36c199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.835+0000 7ff36bfff640 1 -- 172.21.15.29:0/3302901985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff36c109870 msgr2=0x7ff36c19f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:36:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.835+0000 7ff36bfff640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff36c109870 0x7ff36c19f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:36:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.835+0000 7ff36bfff640 1 -- 172.21.15.29:0/3302901985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36c19fc80 con 0x7ff36c108f20 2024-07-27T06:36:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.835+0000 7ff36bfff640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c108f20 0x7ff36c19a370 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff36000d950 tx=0x7ff36000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.835+0000 7ff369ffb640 1 -- 172.21.15.29:0/3302901985 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff360017070 con 0x7ff36c108f20 2024-07-27T06:36:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.835+0000 7ff369ffb640 1 -- 172.21.15.29:0/3302901985 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff360009c30 con 0x7ff36c108f20 2024-07-27T06:36:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.835+0000 7ff372e6a640 1 -- 172.21.15.29:0/3302901985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff36c19ff70 con 0x7ff36c108f20 2024-07-27T06:36:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.835+0000 7ff369ffb640 1 -- 172.21.15.29:0/3302901985 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3600055d0 con 0x7ff36c108f20 2024-07-27T06:36:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.836+0000 7ff372e6a640 1 -- 172.21.15.29:0/3302901985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff36c1a6790 con 0x7ff36c108f20 2024-07-27T06:36:59.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.837+0000 7ff372e6a640 1 -- 172.21.15.29:0/3302901985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff36c1039f0 con 0x7ff36c108f20 2024-07-27T06:36:59.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.838+0000 7ff369ffb640 1 -- 172.21.15.29:0/3302901985 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7ff360004020 con 0x7ff36c108f20 2024-07-27T06:36:59.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.839+0000 7ff369ffb640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff3540b3c40 0x7ff3540b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:36:59.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.839+0000 7ff370c06640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff3540b3c40 0x7ff3540b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:36:59.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.839+0000 7ff369ffb640 1 -- 172.21.15.29:0/3302901985 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7ff3600e1b10 con 0x7ff36c108f20 2024-07-27T06:36:59.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.839+0000 7ff370c06640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff3540b3c40 0x7ff3540b6150 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00abe0 tx=0x7ff35c00a2d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:36:59.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:36:59.843+0000 7ff369ffb640 1 -- 172.21.15.29:0/3302901985 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff360020020 con 0x7ff36c108f20 2024-07-27T06:37:00.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.030+0000 7ff372e6a640 1 -- 172.21.15.29:0/3302901985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7ff36c10b660 con 0x7ff36c108f20 2024-07-27T06:37:00.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.034+0000 7ff369ffb640 1 -- 172.21.15.29:0/3302901985 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v31)=0 v31) ==== 123+0+0 (secure 0 0 0) 0x7ff3600aad60 con 0x7ff36c108f20 2024-07-27T06:37:00.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.035+0000 7ff369ffb640 1 -- 172.21.15.29:0/3302901985 <== mon.0 v2:172.21.15.29:3300/0 8 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3600e7080 con 0x7ff36c108f20 2024-07-27T06:37:00.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7ff372e6a640 1 -- 172.21.15.29:0/3302901985 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff3540b3c40 msgr2=0x7ff3540b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:00.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7ff372e6a640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff3540b3c40 0x7ff3540b6150 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff35c00abe0 tx=0x7ff35c00a2d0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7ff372e6a640 1 -- 172.21.15.29:0/3302901985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c108f20 msgr2=0x7ff36c19a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:00.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7ff372e6a640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c108f20 0x7ff36c19a370 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff36000d950 tx=0x7ff36000dd50 comp rx=0 tx=0).stop 2024-07-27T06:37:00.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7ff372e6a640 1 -- 172.21.15.29:0/3302901985 shutdown_connections 2024-07-27T06:37:00.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7ff372e6a640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff36c109870 0x7ff36c19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7fa1705f0640 1 -- 172.21.15.29:0/3584481926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa168109830 msgr2=0x7fa168110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:00.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7fa1705f0640 1 --2- 172.21.15.29:0/3584481926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa168109830 0x7fa168110060 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa160009ca0 tx=0x7fa16002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7ff372e6a640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff36c108f20 0x7ff36c19a370 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7ff372e6a640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff3540b3c40 0x7ff3540b6150 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7ff372e6a640 1 --2- 172.21.15.29:0/3302901985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c1035e0 0x7ff36c199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7ff372e6a640 1 -- 172.21.15.29:0/3302901985 >> 172.21.15.29:0/3302901985 conn(0x7ff36c0ff030 msgr2=0x7ff36c101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:00.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7fa1705f0640 1 -- 172.21.15.29:0/3584481926 shutdown_connections 2024-07-27T06:37:00.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7fa1705f0640 1 --2- 172.21.15.29:0/3584481926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa168109830 0x7fa168110060 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7fa1705f0640 1 --2- 172.21.15.29:0/3584481926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa168108ee0 0x7fa1681092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7fa1705f0640 1 --2- 172.21.15.29:0/3584481926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1681035a0 0x7fa1681039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7fa1705f0640 1 -- 172.21.15.29:0/3584481926 >> 172.21.15.29:0/3584481926 conn(0x7fa1680ff010 msgr2=0x7fa168101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:00.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7ff372e6a640 1 -- 172.21.15.29:0/3302901985 shutdown_connections 2024-07-27T06:37:00.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7fa1705f0640 1 -- 172.21.15.29:0/3584481926 shutdown_connections 2024-07-27T06:37:00.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7ff372e6a640 1 -- 172.21.15.29:0/3302901985 wait complete. 2024-07-27T06:37:00.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.037+0000 7fa1705f0640 1 -- 172.21.15.29:0/3584481926 wait complete. 2024-07-27T06:37:00.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.038+0000 7fa1705f0640 1 Processor -- start 2024-07-27T06:37:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.038+0000 7fa1705f0640 1 -- start start 2024-07-27T06:37:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.038+0000 7fa1705f0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1681035a0 0x7fa168199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.038+0000 7fa1705f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa168108ee0 0x7fa16819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.038+0000 7fa1705f0640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168109830 0x7fa16819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.038+0000 7fa1705f0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa168113500 con 0x7fa168108ee0 2024-07-27T06:37:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.038+0000 7fa1705f0640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fa168113380 con 0x7fa168109830 2024-07-27T06:37:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.038+0000 7fa1705f0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa168113680 con 0x7fa1681035a0 2024-07-27T06:37:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.038+0000 7fa16db8b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa168108ee0 0x7fa16819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.038+0000 7fa16db8b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa168108ee0 0x7fa16819a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41998/0 (socket says 172.21.15.29:41998) 2024-07-27T06:37:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.038+0000 7fa16db8b640 1 -- 172.21.15.29:0/710145544 learned_addr learned my addr 172.21.15.29:0/710145544 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:37:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa16eb8d640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168109830 0x7fa16819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa16e38c640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1681035a0 0x7fa168199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa16db8b640 1 -- 172.21.15.29:0/710145544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1681035a0 msgr2=0x7fa168199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa16db8b640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1681035a0 0x7fa168199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa16db8b640 1 -- 172.21.15.29:0/710145544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168109830 msgr2=0x7fa16819f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa16db8b640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168109830 0x7fa16819f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa16db8b640 1 -- 172.21.15.29:0/710145544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16819fe20 con 0x7fa168108ee0 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa16e38c640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1681035a0 0x7fa168199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa16db8b640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa168108ee0 0x7fa16819a450 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa15800e9c0 tx=0x7fa15800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa15f7fe640 1 -- 172.21.15.29:0/710145544 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15800ccc0 con 0x7fa168108ee0 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa15f7fe640 1 -- 172.21.15.29:0/710145544 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1580054d0 con 0x7fa168108ee0 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa1705f0640 1 -- 172.21.15.29:0/710145544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1681a0110 con 0x7fa168108ee0 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa15f7fe640 1 -- 172.21.15.29:0/710145544 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158010430 con 0x7fa168108ee0 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.039+0000 7fa1705f0640 1 -- 172.21.15.29:0/710145544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1681a6870 con 0x7fa168108ee0 2024-07-27T06:37:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.040+0000 7fa1705f0640 1 -- 172.21.15.29:0/710145544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1681039b0 con 0x7fa168108ee0 2024-07-27T06:37:00.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.043+0000 7fa15f7fe640 1 -- 172.21.15.29:0/710145544 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7fa158005000 con 0x7fa168108ee0 2024-07-27T06:37:00.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.044+0000 7fa15f7fe640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa1540b3b90 0x7fa1540b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:00.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.044+0000 7fa15f7fe640 1 -- 172.21.15.29:0/710145544 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa1580e2720 con 0x7fa168108ee0 2024-07-27T06:37:00.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.044+0000 7fa16e38c640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa1540b3b90 0x7fa1540b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:00.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.045+0000 7fa16e38c640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa1540b3b90 0x7fa1540b60a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa164009c20 tx=0x7fa164005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:00.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.046+0000 7fa15f7fe640 1 -- 172.21.15.29:0/710145544 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa158055600 con 0x7fa168108ee0 2024-07-27T06:37:00.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.232+0000 7fa1705f0640 1 -- 172.21.15.29:0/710145544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7fa16802e0c0 con 0x7fa168108ee0 2024-07-27T06:37:00.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.236+0000 7fa15f7fe640 1 -- 172.21.15.29:0/710145544 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v32)=0 v32) ==== 118+0+0 (secure 0 0 0) 0x7fa1580052f0 con 0x7fa168108ee0 2024-07-27T06:37:00.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.238+0000 7fa1705f0640 1 -- 172.21.15.29:0/710145544 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa1540b3b90 msgr2=0x7fa1540b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:00.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.238+0000 7fa1705f0640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa1540b3b90 0x7fa1540b60a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa164009c20 tx=0x7fa164005f90 comp rx=0 tx=0).stop 2024-07-27T06:37:00.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.238+0000 7fa1705f0640 1 -- 172.21.15.29:0/710145544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa168108ee0 msgr2=0x7fa16819a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:00.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.238+0000 7fa1705f0640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa168108ee0 0x7fa16819a450 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa15800e9c0 tx=0x7fa15800ee80 comp rx=0 tx=0).stop 2024-07-27T06:37:00.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.238+0000 7fa1705f0640 1 -- 172.21.15.29:0/710145544 shutdown_connections 2024-07-27T06:37:00.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.238+0000 7fa1705f0640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa168109830 0x7fa16819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.238+0000 7fa1705f0640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa168108ee0 0x7fa16819a450 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.238+0000 7fa1705f0640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa1540b3b90 0x7fa1540b60a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.238+0000 7fa1705f0640 1 --2- 172.21.15.29:0/710145544 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1681035a0 0x7fa168199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:00.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.238+0000 7fa1705f0640 1 -- 172.21.15.29:0/710145544 >> 172.21.15.29:0/710145544 conn(0x7fa1680ff010 msgr2=0x7fa168108550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:00.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.239+0000 7fa1705f0640 1 -- 172.21.15.29:0/710145544 shutdown_connections 2024-07-27T06:37:00.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:00.239+0000 7fa1705f0640 1 -- 172.21.15.29:0/710145544 wait complete. 2024-07-27T06:37:00.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:59 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3191395326' entity='client.admin' 2024-07-27T06:37:00.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:36:59 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1398454338' entity='client.admin' 2024-07-27T06:37:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:59 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3191395326' entity='client.admin' 2024-07-27T06:37:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:36:59 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1398454338' entity='client.admin' 2024-07-27T06:37:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:59 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3191395326' entity='client.admin' 2024-07-27T06:37:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:36:59 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1398454338' entity='client.admin' 2024-07-27T06:37:01.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:01 smithi029 ceph-mon[32610]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:01.291 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:01 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3302901985' entity='client.admin' 2024-07-27T06:37:01.291 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:01 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/710145544' entity='client.admin' 2024-07-27T06:37:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:01 smithi077 ceph-mon[37153]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:01 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3302901985' entity='client.admin' 2024-07-27T06:37:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:01 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/710145544' entity='client.admin' 2024-07-27T06:37:01.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.387+0000 7f4da8845640 1 -- 172.21.15.29:0/2112106841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da4107df0 msgr2=0x7f4da4108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:01.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.387+0000 7f4da8845640 1 --2- 172.21.15.29:0/2112106841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da4107df0 0x7f4da4108200 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009900 tx=0x7f4d9802a410 comp rx=0 tx=0).stop 2024-07-27T06:37:01.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.387+0000 7f4da8845640 1 -- 172.21.15.29:0/2112106841 shutdown_connections 2024-07-27T06:37:01.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.387+0000 7f4da8845640 1 --2- 172.21.15.29:0/2112106841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4da4108740 0x7f4da410ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:01.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.387+0000 7f4da8845640 1 --2- 172.21.15.29:0/2112106841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da4107df0 0x7f4da4108200 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:01.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.387+0000 7f4da8845640 1 --2- 172.21.15.29:0/2112106841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4da41024b0 0x7f4da41028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:01.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.387+0000 7f4da8845640 1 -- 172.21.15.29:0/2112106841 >> 172.21.15.29:0/2112106841 conn(0x7f4da4079170 msgr2=0x7f4da4079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:01.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.387+0000 7f4da8845640 1 -- 172.21.15.29:0/2112106841 shutdown_connections 2024-07-27T06:37:01.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.387+0000 7f4da8845640 1 -- 172.21.15.29:0/2112106841 wait complete. 2024-07-27T06:37:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.388+0000 7f4da8845640 1 Processor -- start 2024-07-27T06:37:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.388+0000 7f4da8845640 1 -- start start 2024-07-27T06:37:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da8845640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4da41024b0 0x7f4da4199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da8845640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da4107df0 0x7f4da419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da8845640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4da4108740 0x7f4da419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da8845640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4da41134d0 con 0x7f4da4107df0 2024-07-27T06:37:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da8845640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f4da4113350 con 0x7f4da41024b0 2024-07-27T06:37:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da8845640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4da4113650 con 0x7f4da4108740 2024-07-27T06:37:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da4107df0 0x7f4da419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da4107df0 0x7f4da419a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42008/0 (socket says 172.21.15.29:42008) 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da1d9b640 1 -- 172.21.15.29:0/3945620774 learned_addr learned my addr 172.21.15.29:0/3945620774 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da2d9d640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4da4108740 0x7f4da419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da259c640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4da41024b0 0x7f4da4199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da1d9b640 1 -- 172.21.15.29:0/3945620774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4da4108740 msgr2=0x7f4da419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da1d9b640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4da4108740 0x7f4da419f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da1d9b640 1 -- 172.21.15.29:0/3945620774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4da41024b0 msgr2=0x7f4da4199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da1d9b640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4da41024b0 0x7f4da4199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da1d9b640 1 -- 172.21.15.29:0/3945620774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da419fe10 con 0x7f4da4107df0 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da259c640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4da41024b0 0x7f4da4199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.389+0000 7f4da1d9b640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da4107df0 0x7f4da419a440 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4d98032fd0 tx=0x7f4d98004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.390+0000 7f4d937fe640 1 -- 172.21.15.29:0/3945620774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9803f030 con 0x7f4da4107df0 2024-07-27T06:37:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.390+0000 7f4d937fe640 1 -- 172.21.15.29:0/3945620774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d98004490 con 0x7f4da4107df0 2024-07-27T06:37:01.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.390+0000 7f4da8845640 1 -- 172.21.15.29:0/3945620774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da41a00a0 con 0x7f4da4107df0 2024-07-27T06:37:01.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.390+0000 7f4d937fe640 1 -- 172.21.15.29:0/3945620774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98035070 con 0x7f4da4107df0 2024-07-27T06:37:01.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.390+0000 7f4da8845640 1 -- 172.21.15.29:0/3945620774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da41a6860 con 0x7f4da4107df0 2024-07-27T06:37:01.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.391+0000 7f4da8845640 1 -- 172.21.15.29:0/3945620774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da4103730 con 0x7f4da4107df0 2024-07-27T06:37:01.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.392+0000 7f4d937fe640 1 -- 172.21.15.29:0/3945620774 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f4d98042030 con 0x7f4da4107df0 2024-07-27T06:37:01.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.393+0000 7f4d937fe640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4d8c0b3b90 0x7f4d8c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:01.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.393+0000 7f4da259c640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4d8c0b3b90 0x7f4d8c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:01.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.393+0000 7f4d937fe640 1 -- 172.21.15.29:0/3945620774 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f4d981025f0 con 0x7f4da4107df0 2024-07-27T06:37:01.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.394+0000 7f4da259c640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4d8c0b3b90 0x7f4d8c0b60a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d94005d00 tx=0x7f4d94005c70 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:01.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.397+0000 7f4d937fe640 1 -- 172.21.15.29:0/3945620774 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4d980cb7a0 con 0x7f4da4107df0 2024-07-27T06:37:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:01 smithi047 ceph-mon[36835]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:01 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3302901985' entity='client.admin' 2024-07-27T06:37:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:01 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/710145544' entity='client.admin' 2024-07-27T06:37:01.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.585+0000 7f4da8845640 1 -- 172.21.15.29:0/3945620774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f4da410a530 con 0x7f4da4107df0 2024-07-27T06:37:01.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.589+0000 7f4d937fe640 1 -- 172.21.15.29:0/3945620774 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v33)=0 v33) ==== 121+0+0 (secure 0 0 0) 0x7f4d980cb0a0 con 0x7f4da4107df0 2024-07-27T06:37:01.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.591+0000 7f4da8845640 1 -- 172.21.15.29:0/3945620774 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4d8c0b3b90 msgr2=0x7f4d8c0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:01.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.591+0000 7f4da8845640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4d8c0b3b90 0x7f4d8c0b60a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d94005d00 tx=0x7f4d94005c70 comp rx=0 tx=0).stop 2024-07-27T06:37:01.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.591+0000 7f4da8845640 1 -- 172.21.15.29:0/3945620774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da4107df0 msgr2=0x7f4da419a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:01.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.591+0000 7f4da8845640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da4107df0 0x7f4da419a440 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4d98032fd0 tx=0x7f4d98004270 comp rx=0 tx=0).stop 2024-07-27T06:37:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.591+0000 7f4da8845640 1 -- 172.21.15.29:0/3945620774 shutdown_connections 2024-07-27T06:37:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.591+0000 7f4da8845640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4d8c0b3b90 0x7f4d8c0b60a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.591+0000 7f4da8845640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4da4108740 0x7f4da419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.591+0000 7f4da8845640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da4107df0 0x7f4da419a440 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.591+0000 7f4da8845640 1 --2- 172.21.15.29:0/3945620774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4da41024b0 0x7f4da4199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.591+0000 7f4da8845640 1 -- 172.21.15.29:0/3945620774 >> 172.21.15.29:0/3945620774 conn(0x7f4da4079170 msgr2=0x7f4da4104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.591+0000 7f4da8845640 1 -- 172.21.15.29:0/3945620774 shutdown_connections 2024-07-27T06:37:01.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:01.591+0000 7f4da8845640 1 -- 172.21.15.29:0/3945620774 wait complete. 2024-07-27T06:37:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.173+0000 7f8d3a664640 1 -- 172.21.15.29:0/147700779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d34108ed0 msgr2=0x7f8d341092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.173+0000 7f8d3a664640 1 --2- 172.21.15.29:0/147700779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d34108ed0 0x7f8d341092e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009900 tx=0x7f8d2802a410 comp rx=0 tx=0).stop 2024-07-27T06:37:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.173+0000 7f8d3a664640 1 -- 172.21.15.29:0/147700779 shutdown_connections 2024-07-27T06:37:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.173+0000 7f8d3a664640 1 --2- 172.21.15.29:0/147700779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d34109820 0x7f8d34110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.173+0000 7f8d3a664640 1 --2- 172.21.15.29:0/147700779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d34108ed0 0x7f8d341092e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.173+0000 7f8d3a664640 1 --2- 172.21.15.29:0/147700779 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d34103590 0x7f8d341039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.173+0000 7f8d3a664640 1 -- 172.21.15.29:0/147700779 >> 172.21.15.29:0/147700779 conn(0x7f8d3407b280 msgr2=0x7f8d3407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.173+0000 7f8d3a664640 1 -- 172.21.15.29:0/147700779 shutdown_connections 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.173+0000 7f8d3a664640 1 -- 172.21.15.29:0/147700779 wait complete. 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d3a664640 1 Processor -- start 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d3a664640 1 -- start start 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d3a664640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d34103590 0x7f8d34199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d3a664640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d34108ed0 0x7f8d3419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d3a664640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d34109820 0x7f8d3419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d3a664640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8d34113490 con 0x7f8d34109820 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d3a664640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f8d34113310 con 0x7f8d34108ed0 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d3a664640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8d34113610 con 0x7f8d34103590 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d38c01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d34109820 0x7f8d3419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d337fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d34108ed0 0x7f8d3419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d38c01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d34109820 0x7f8d3419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42038/0 (socket says 172.21.15.29:42038) 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d38c01640 1 -- 172.21.15.29:0/415627752 learned_addr learned my addr 172.21.15.29:0/415627752 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d337fe640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d34108ed0 0x7f8d3419a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.29:34320/0 (socket says 172.21.15.29:34320) 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.174+0000 7f8d33fff640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d34103590 0x7f8d34199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.175+0000 7f8d38c01640 1 -- 172.21.15.29:0/415627752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d34103590 msgr2=0x7f8d34199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.175+0000 7f8d38c01640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d34103590 0x7f8d34199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.175+0000 7f8d38c01640 1 -- 172.21.15.29:0/415627752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d34108ed0 msgr2=0x7f8d3419a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.175+0000 7f8d38c01640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d34108ed0 0x7f8d3419a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.175+0000 7f8d38c01640 1 -- 172.21.15.29:0/415627752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d3419fed0 con 0x7f8d34109820 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.175+0000 7f8d38c01640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d34109820 0x7f8d3419f7a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00cad0 tx=0x7f8d2c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.175+0000 7f8d317fa640 1 -- 172.21.15.29:0/415627752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c00ae20 con 0x7f8d34109820 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.175+0000 7f8d317fa640 1 -- 172.21.15.29:0/415627752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d2c005020 con 0x7f8d34109820 2024-07-27T06:37:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.175+0000 7f8d317fa640 1 -- 172.21.15.29:0/415627752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c00f6b0 con 0x7f8d34109820 2024-07-27T06:37:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.175+0000 7f8d3a664640 1 -- 172.21.15.29:0/415627752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d341a01c0 con 0x7f8d34109820 2024-07-27T06:37:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.175+0000 7f8d3a664640 1 -- 172.21.15.29:0/415627752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d341a6920 con 0x7f8d34109820 2024-07-27T06:37:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.176+0000 7f8d3a664640 1 -- 172.21.15.29:0/415627752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d341039a0 con 0x7f8d34109820 2024-07-27T06:37:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.178+0000 7f8d317fa640 1 -- 172.21.15.29:0/415627752 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f8d2c0051d0 con 0x7f8d34109820 2024-07-27T06:37:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.178+0000 7f8d317fa640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8d1c0b3c10 0x7f8d1c0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.179+0000 7f8d33fff640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8d1c0b3c10 0x7f8d1c0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.179+0000 7f8d317fa640 1 -- 172.21.15.29:0/415627752 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8d2c0e2280 con 0x7f8d34109820 2024-07-27T06:37:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.179+0000 7f8d33fff640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8d1c0b3c10 0x7f8d1c0b6120 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8d24005d00 tx=0x7f8d2400b410 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.181+0000 7f8d317fa640 1 -- 172.21.15.29:0/415627752 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8d2c014030 con 0x7f8d34109820 2024-07-27T06:37:02.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.374+0000 7f8d3a664640 1 -- 172.21.15.29:0/415627752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f8d3419b090 con 0x7f8d34109820 2024-07-27T06:37:02.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.378+0000 7f8d317fa640 1 -- 172.21.15.29:0/415627752 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v34)=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7f8d3419b090 con 0x7f8d34109820 2024-07-27T06:37:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.380+0000 7f8d3a664640 1 -- 172.21.15.29:0/415627752 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8d1c0b3c10 msgr2=0x7f8d1c0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.380+0000 7f8d3a664640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8d1c0b3c10 0x7f8d1c0b6120 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8d24005d00 tx=0x7f8d2400b410 comp rx=0 tx=0).stop 2024-07-27T06:37:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.380+0000 7f8d3a664640 1 -- 172.21.15.29:0/415627752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d34109820 msgr2=0x7f8d3419f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.380+0000 7f8d3a664640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d34109820 0x7f8d3419f7a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00cad0 tx=0x7f8d2c00cf90 comp rx=0 tx=0).stop 2024-07-27T06:37:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.380+0000 7f8d3a664640 1 -- 172.21.15.29:0/415627752 shutdown_connections 2024-07-27T06:37:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.380+0000 7f8d3a664640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d34109820 0x7f8d3419f7a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:02.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.380+0000 7f8d3a664640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d34108ed0 0x7f8d3419a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.380+0000 7f8d3a664640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8d1c0b3c10 0x7f8d1c0b6120 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.380+0000 7f8d3a664640 1 --2- 172.21.15.29:0/415627752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d34103590 0x7f8d34199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.380+0000 7f8d3a664640 1 -- 172.21.15.29:0/415627752 >> 172.21.15.29:0/415627752 conn(0x7f8d3407b280 msgr2=0x7f8d34107020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.381+0000 7f8d3a664640 1 -- 172.21.15.29:0/415627752 shutdown_connections 2024-07-27T06:37:02.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:02.381+0000 7f8d3a664640 1 -- 172.21.15.29:0/415627752 wait complete. 2024-07-27T06:37:02.659 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:02 smithi029 ceph-mon[32610]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:02.659 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:02 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3945620774' entity='client.admin' 2024-07-27T06:37:02.659 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:02 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/415627752' entity='client.admin' 2024-07-27T06:37:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:02 smithi077 ceph-mon[37153]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:02 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3945620774' entity='client.admin' 2024-07-27T06:37:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:02 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/415627752' entity='client.admin' 2024-07-27T06:37:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:02 smithi047 ceph-mon[36835]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:02 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3945620774' entity='client.admin' 2024-07-27T06:37:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:02 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/415627752' entity='client.admin' 2024-07-27T06:37:03.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.314+0000 7f781f226640 1 -- 172.21.15.29:0/2960351558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7818107eb0 msgr2=0x7f781810e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:03.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.314+0000 7f781f226640 1 --2- 172.21.15.29:0/2960351558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7818107eb0 0x7f781810e6e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f781000adb0 tx=0x7f781002b7a0 comp rx=0 tx=0).stop 2024-07-27T06:37:03.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.314+0000 7f781f226640 1 -- 172.21.15.29:0/2960351558 shutdown_connections 2024-07-27T06:37:03.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.314+0000 7f781f226640 1 --2- 172.21.15.29:0/2960351558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7818107eb0 0x7f781810e6e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:03.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.314+0000 7f781f226640 1 --2- 172.21.15.29:0/2960351558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7818107560 0x7f7818107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:03.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.314+0000 7f781f226640 1 --2- 172.21.15.29:0/2960351558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7818101c20 0x7f7818102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.314+0000 7f781f226640 1 -- 172.21.15.29:0/2960351558 >> 172.21.15.29:0/2960351558 conn(0x7f7818077690 msgr2=0x7f7818077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.314+0000 7f781f226640 1 -- 172.21.15.29:0/2960351558 shutdown_connections 2024-07-27T06:37:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.314+0000 7f781f226640 1 -- 172.21.15.29:0/2960351558 wait complete. 2024-07-27T06:37:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.315+0000 7f781f226640 1 Processor -- start 2024-07-27T06:37:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.315+0000 7f781f226640 1 -- start start 2024-07-27T06:37:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.315+0000 7f781f226640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7818101c20 0x7f78180fe810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.315+0000 7f781f226640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7818107560 0x7f78180fed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.315+0000 7f781f226640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7818107eb0 0x7f78180ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.315+0000 7f781f226640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f78181112f0 con 0x7f7818107560 2024-07-27T06:37:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.315+0000 7f781f226640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7818111170 con 0x7f7818101c20 2024-07-27T06:37:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.315+0000 7f781f226640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7818111470 con 0x7f7818107eb0 2024-07-27T06:37:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.315+0000 7f7817fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7818107560 0x7f78180fed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f7817fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7818107560 0x7f78180fed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42052/0 (socket says 172.21.15.29:42052) 2024-07-27T06:37:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f7817fff640 1 -- 172.21.15.29:0/3133347786 learned_addr learned my addr 172.21.15.29:0/3133347786 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:37:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f781d7c3640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7818107eb0 0x7f78180ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f781cfc2640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7818101c20 0x7f78180fe810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f7817fff640 1 -- 172.21.15.29:0/3133347786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7818107eb0 msgr2=0x7f78180ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f7817fff640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7818107eb0 0x7f78180ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f7817fff640 1 -- 172.21.15.29:0/3133347786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7818101c20 msgr2=0x7f78180fe810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f7817fff640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7818101c20 0x7f78180fe810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f7817fff640 1 -- 172.21.15.29:0/3133347786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78181a8070 con 0x7f7818107560 2024-07-27T06:37:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f7817fff640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7818107560 0x7f78180fed50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f780400c8c0 tx=0x7f780400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f7815ffb640 1 -- 172.21.15.29:0/3133347786 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7804007bb0 con 0x7f7818107560 2024-07-27T06:37:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f781f226640 1 -- 172.21.15.29:0/3133347786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78181a8360 con 0x7f7818107560 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f7815ffb640 1 -- 172.21.15.29:0/3133347786 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7804007d50 con 0x7f7818107560 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f7815ffb640 1 -- 172.21.15.29:0/3133347786 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78040187c0 con 0x7f7818107560 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.316+0000 7f781f226640 1 -- 172.21.15.29:0/3133347786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78181a88f0 con 0x7f7818107560 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.317+0000 7f781f226640 1 -- 172.21.15.29:0/3133347786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7818102030 con 0x7f7818107560 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.319+0000 7f7815ffb640 1 -- 172.21.15.29:0/3133347786 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f780400fa20 con 0x7f7818107560 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.320+0000 7f7815ffb640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78080b3b90 0x7f78080b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.321+0000 7f781cfc2640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78080b3b90 0x7f78080b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.321+0000 7f7815ffb640 1 -- 172.21.15.29:0/3133347786 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f78040e2480 con 0x7f7818107560 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.321+0000 7f781cfc2640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78080b3b90 0x7f78080b60a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7818100260 tx=0x7f780c005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.323+0000 7f7815ffb640 1 -- 172.21.15.29:0/3133347786 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7804014030 con 0x7f7818107560 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.513+0000 7f781f226640 1 -- 172.21.15.29:0/3133347786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f78180ffb10 con 0x7f7818107560 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.517+0000 7f7815ffb640 1 -- 172.21.15.29:0/3133347786 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v35)=0 v35) ==== 117+0+0 (secure 0 0 0) 0x7f78040ab6c0 con 0x7f7818107560 2024-07-27T06:37:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.520+0000 7f781f226640 1 -- 172.21.15.29:0/3133347786 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78080b3b90 msgr2=0x7f78080b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.520+0000 7f781f226640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78080b3b90 0x7f78080b60a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7818100260 tx=0x7f780c005f90 comp rx=0 tx=0).stop 2024-07-27T06:37:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.520+0000 7f781f226640 1 -- 172.21.15.29:0/3133347786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7818107560 msgr2=0x7f78180fed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.520+0000 7f781f226640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7818107560 0x7f78180fed50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f780400c8c0 tx=0x7f780400cd80 comp rx=0 tx=0).stop 2024-07-27T06:37:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.520+0000 7f781f226640 1 -- 172.21.15.29:0/3133347786 shutdown_connections 2024-07-27T06:37:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.520+0000 7f781f226640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78080b3b90 0x7f78080b60a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.520+0000 7f781f226640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7818107eb0 0x7f78180ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.520+0000 7f781f226640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7818107560 0x7f78180fed50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.520+0000 7f781f226640 1 --2- 172.21.15.29:0/3133347786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7818101c20 0x7f78180fe810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.520+0000 7f781f226640 1 -- 172.21.15.29:0/3133347786 >> 172.21.15.29:0/3133347786 conn(0x7f7818077690 msgr2=0x7f7818100b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.521+0000 7f781f226640 1 -- 172.21.15.29:0/3133347786 shutdown_connections 2024-07-27T06:37:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:03.521+0000 7f781f226640 1 -- 172.21.15.29:0/3133347786 wait complete. 2024-07-27T06:37:04.159 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi029') 2024-07-27T06:37:04.159 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi029') 2024-07-27T06:37:04.159 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi029') 2024-07-27T06:37:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:04 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3133347786' entity='client.admin' 2024-07-27T06:37:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:04 smithi077 ceph-mon[37153]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:04.909 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:04 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3133347786' entity='client.admin' 2024-07-27T06:37:04.909 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:04 smithi029 ceph-mon[32610]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:04 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3133347786' entity='client.admin' 2024-07-27T06:37:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:04 smithi047 ceph-mon[36835]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:05.856 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.856 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.856 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.856 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.856 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.857 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.857 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.857 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.857 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.857 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.857 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.857 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.857 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'debug_cephsqlite', '20'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.858 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/allow_m_granularity', 'True'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.858 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/dump_on_update', 'True'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.858 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/log_level', 'debug'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.858 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.858 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.858 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0a8a0a4c-4be2-11ef-bcc7-c7b262605968', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi029') 2024-07-27T06:37:05.858 INFO:tasks.cephadm:Deploying OSDs... 2024-07-27T06:37:05.859 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:37:05.859 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-07-27T06:37:05.879 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-27T06:37:05.879 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-07-27T06:37:05.938 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-27T06:37:05.938 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T06:37:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-07-27T06:37:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:37:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T06:37:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-07-27 06:35:09.211755809 +0000 2024-07-27T06:37:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-07-27 06:30:02.484169427 +0000 2024-07-27T06:37:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-07-27 06:30:02.484169427 +0000 2024-07-27T06:37:05.939 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-07-27 06:30:02.484169427 +0000 2024-07-27T06:37:05.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-27T06:37:06.007 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-07-27T06:37:06.007 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-07-27T06:37:06.007 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000103939 s, 4.9 MB/s 2024-07-27T06:37:06.008 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-27T06:37:06.067 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-07-27T06:37:06.124 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-27T06:37:06.124 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T06:37:06.124 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 693 Links: 1 2024-07-27T06:37:06.124 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:37:06.124 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T06:37:06.124 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-07-27 06:35:09.253755133 +0000 2024-07-27T06:37:06.124 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-07-27 06:30:02.804164146 +0000 2024-07-27T06:37:06.124 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-07-27 06:30:02.804164146 +0000 2024-07-27T06:37:06.124 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-07-27 06:30:02.804164146 +0000 2024-07-27T06:37:06.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-27T06:37:06.190 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-07-27T06:37:06.190 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-07-27T06:37:06.191 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.5576e-05 s, 5.4 MB/s 2024-07-27T06:37:06.191 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-27T06:37:06.249 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-07-27T06:37:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-27T06:37:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T06:37:06.306 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 711 Links: 1 2024-07-27T06:37:06.306 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:37:06.306 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T06:37:06.306 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-07-27 06:35:09.293754489 +0000 2024-07-27T06:37:06.306 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-07-27 06:30:03.132158733 +0000 2024-07-27T06:37:06.307 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-07-27 06:30:03.132158733 +0000 2024-07-27T06:37:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-07-27 06:30:03.132158733 +0000 2024-07-27T06:37:06.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-27T06:37:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-07-27T06:37:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-07-27T06:37:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000112819 s, 4.5 MB/s 2024-07-27T06:37:06.373 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-27T06:37:06.431 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-07-27T06:37:06.488 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-27T06:37:06.488 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T06:37:06.488 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 729 Links: 1 2024-07-27T06:37:06.488 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:37:06.488 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T06:37:06.488 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-07-27 06:35:09.324753990 +0000 2024-07-27T06:37:06.488 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-07-27 06:30:03.451153468 +0000 2024-07-27T06:37:06.488 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-07-27 06:30:03.451153468 +0000 2024-07-27T06:37:06.488 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-07-27 06:30:03.451153468 +0000 2024-07-27T06:37:06.489 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-27T06:37:06.553 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-07-27T06:37:06.553 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-07-27T06:37:06.553 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000112182 s, 4.6 MB/s 2024-07-27T06:37:06.554 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-27T06:37:06.612 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-07-27T06:37:06.612 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/scratch_devs of=/dev/stdout 2024-07-27T06:37:06.631 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-27T06:37:06.631 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_1 2024-07-27T06:37:06.691 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-27T06:37:06.691 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T06:37:06.691 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-07-27T06:37:06.691 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:37:06.691 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T06:37:06.691 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-07-27 06:36:16.077876887 +0000 2024-07-27T06:37:06.691 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-07-27 06:30:04.027125180 +0000 2024-07-27T06:37:06.691 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-07-27 06:30:04.027125180 +0000 2024-07-27T06:37:06.691 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-07-27 06:30:04.027125180 +0000 2024-07-27T06:37:06.692 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-27T06:37:06.760 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-07-27T06:37:06.760 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-07-27T06:37:06.760 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000130788 s, 3.9 MB/s 2024-07-27T06:37:06.762 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-27T06:37:06.820 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_2 2024-07-27T06:37:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:06 smithi077 ceph-mon[37153]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:06.878 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-27T06:37:06.878 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T06:37:06.878 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-07-27T06:37:06.878 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:37:06.879 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T06:37:06.879 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-07-27 06:36:16.077876887 +0000 2024-07-27T06:37:06.879 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-07-27 06:30:04.355118295 +0000 2024-07-27T06:37:06.879 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-07-27 06:30:04.355118295 +0000 2024-07-27T06:37:06.879 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-07-27 06:30:04.355118295 +0000 2024-07-27T06:37:06.879 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-27T06:37:06.943 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:06 smithi047 ceph-mon[36835]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:06.945 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-07-27T06:37:06.945 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-07-27T06:37:06.945 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000124848 s, 4.1 MB/s 2024-07-27T06:37:06.947 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-27T06:37:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:06 smithi029 ceph-mon[32610]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:07.006 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_3 2024-07-27T06:37:07.064 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-27T06:37:07.064 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T06:37:07.064 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-07-27T06:37:07.064 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:37:07.065 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T06:37:07.065 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-07-27 06:36:16.077876887 +0000 2024-07-27T06:37:07.065 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-07-27 06:30:04.684111388 +0000 2024-07-27T06:37:07.065 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-07-27 06:30:04.684111388 +0000 2024-07-27T06:37:07.065 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-07-27 06:30:04.684111388 +0000 2024-07-27T06:37:07.065 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-27T06:37:07.130 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-07-27T06:37:07.130 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-07-27T06:37:07.131 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000140935 s, 3.6 MB/s 2024-07-27T06:37:07.132 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-27T06:37:07.190 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_4 2024-07-27T06:37:07.246 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-27T06:37:07.247 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T06:37:07.247 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-07-27T06:37:07.247 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:37:07.247 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T06:37:07.247 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-07-27 06:36:16.078876867 +0000 2024-07-27T06:37:07.247 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-07-27 06:30:05.003104687 +0000 2024-07-27T06:37:07.247 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-07-27 06:30:05.003104687 +0000 2024-07-27T06:37:07.247 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-07-27 06:30:05.003104687 +0000 2024-07-27T06:37:07.247 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-27T06:37:07.311 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-07-27T06:37:07.311 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-07-27T06:37:07.312 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000118008 s, 4.3 MB/s 2024-07-27T06:37:07.313 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-27T06:37:07.370 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-07-27T06:37:07.370 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-07-27T06:37:07.389 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-27T06:37:07.389 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-07-27T06:37:07.448 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-27T06:37:07.448 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T06:37:07.448 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-07-27T06:37:07.448 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:37:07.449 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T06:37:07.449 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-07-27 06:36:13.733519296 +0000 2024-07-27T06:37:07.449 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-07-27 06:29:53.163346606 +0000 2024-07-27T06:37:07.449 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-07-27 06:29:53.163346606 +0000 2024-07-27T06:37:07.449 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-07-27 06:29:53.163346606 +0000 2024-07-27T06:37:07.449 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-27T06:37:07.519 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-07-27T06:37:07.519 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-07-27T06:37:07.519 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000240914 s, 2.1 MB/s 2024-07-27T06:37:07.520 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-27T06:37:07.584 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-07-27T06:37:07.647 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-27T06:37:07.647 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T06:37:07.648 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-07-27T06:37:07.648 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:37:07.648 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T06:37:07.648 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-07-27 06:36:13.733519296 +0000 2024-07-27T06:37:07.648 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-07-27 06:29:53.505339644 +0000 2024-07-27T06:37:07.648 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-07-27 06:29:53.505339644 +0000 2024-07-27T06:37:07.648 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-07-27 06:29:53.505339644 +0000 2024-07-27T06:37:07.648 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-27T06:37:07.717 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-07-27T06:37:07.717 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-07-27T06:37:07.718 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000283394 s, 1.8 MB/s 2024-07-27T06:37:07.719 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-27T06:37:07.784 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-07-27T06:37:07.848 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-27T06:37:07.848 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T06:37:07.848 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-07-27T06:37:07.848 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:37:07.848 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T06:37:07.848 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-07-27 06:36:13.733519296 +0000 2024-07-27T06:37:07.848 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-07-27 06:29:53.832332988 +0000 2024-07-27T06:37:07.848 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-07-27 06:29:53.832332988 +0000 2024-07-27T06:37:07.848 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-07-27 06:29:53.832332988 +0000 2024-07-27T06:37:07.849 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-27T06:37:07.915 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-07-27T06:37:07.915 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-07-27T06:37:07.916 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.0002117 s, 2.4 MB/s 2024-07-27T06:37:07.916 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-27T06:37:07.977 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-07-27T06:37:08.036 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-27T06:37:08.037 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-27T06:37:08.037 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-07-27T06:37:08.037 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:37:08.037 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-07-27T06:37:08.037 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-07-27 06:36:13.734519277 +0000 2024-07-27T06:37:08.037 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-07-27 06:29:54.152326474 +0000 2024-07-27T06:37:08.037 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-07-27 06:29:54.152326474 +0000 2024-07-27T06:37:08.037 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-07-27 06:29:54.152326474 +0000 2024-07-27T06:37:08.037 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-27T06:37:08.101 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-07-27T06:37:08.102 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-07-27T06:37:08.102 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000222537 s, 2.3 MB/s 2024-07-27T06:37:08.103 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-27T06:37:08.166 INFO:tasks.cephadm:Deploying osd.0 on smithi029 with /dev/vg_nvme/lv_4... 2024-07-27T06:37:08.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-27T06:37:08.441 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:37:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:08 smithi077 ceph-mon[37153]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:08.903 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:08 smithi029 ceph-mon[32610]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:08 smithi047 ceph-mon[36835]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:10.212 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-27T06:37:10.289 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-27T06:37:10.299 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-07-27T06:37:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-07-27T06:37:10.300 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00839068 s, 1.2 GB/s 2024-07-27T06:37:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-07-27T06:37:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:10 smithi077 ceph-mon[37153]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:10.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:10 smithi029 ceph-mon[32610]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:10.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_4 2024-07-27T06:37:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:10 smithi047 ceph-mon[36835]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:11.173 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:37:12.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.691+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/1392298866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c8102510 msgr2=0x7fd6c8102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.691+0000 7fd6ccf7e640 1 --2- 172.21.15.29:0/1392298866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c8102510 0x7fd6c8102920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8009900 tx=0x7fd6b802a410 comp rx=0 tx=0).stop 2024-07-27T06:37:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.692+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/1392298866 shutdown_connections 2024-07-27T06:37:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.692+0000 7fd6ccf7e640 1 --2- 172.21.15.29:0/1392298866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c8108750 0x7fd6c810ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.692+0000 7fd6ccf7e640 1 --2- 172.21.15.29:0/1392298866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6c8107e00 0x7fd6c8108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.692+0000 7fd6ccf7e640 1 --2- 172.21.15.29:0/1392298866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c8102510 0x7fd6c8102920 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.692+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/1392298866 >> 172.21.15.29:0/1392298866 conn(0x7fd6c8079190 msgr2=0x7fd6c80795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:12.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.692+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/1392298866 shutdown_connections 2024-07-27T06:37:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.692+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/1392298866 wait complete. 2024-07-27T06:37:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6ccf7e640 1 Processor -- start 2024-07-27T06:37:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6ccf7e640 1 -- start start 2024-07-27T06:37:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6ccf7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c8102510 0x7fd6c819ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6ccf7e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6c8107e00 0x7fd6c819b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6ccf7e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c8108750 0x7fd6c819b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6ccf7e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c81123b0 con 0x7fd6c8102510 2024-07-27T06:37:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6ccf7e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c8112230 con 0x7fd6c8107e00 2024-07-27T06:37:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6ccf7e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c8112530 con 0x7fd6c8108750 2024-07-27T06:37:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c8102510 0x7fd6c819ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c8102510 0x7fd6c819ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42782/0 (socket says 172.21.15.29:42782) 2024-07-27T06:37:12.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6c659c640 1 -- 172.21.15.29:0/4064358194 learned_addr learned my addr 172.21.15.29:0/4064358194 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:37:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6c5d9b640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6c8107e00 0x7fd6c819b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.693+0000 7fd6c6d9d640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c8108750 0x7fd6c819b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:12.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.694+0000 7fd6c659c640 1 -- 172.21.15.29:0/4064358194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c8108750 msgr2=0x7fd6c819b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.694+0000 7fd6c659c640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c8108750 0x7fd6c819b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.694+0000 7fd6c659c640 1 -- 172.21.15.29:0/4064358194 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6c8107e00 msgr2=0x7fd6c819b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.694+0000 7fd6c659c640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6c8107e00 0x7fd6c819b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.694+0000 7fd6c659c640 1 -- 172.21.15.29:0/4064358194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c81a5fc0 con 0x7fd6c8102510 2024-07-27T06:37:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.694+0000 7fd6c659c640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c8102510 0x7fd6c819ad00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8005940 tx=0x7fd6b80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.694+0000 7fd6b77fe640 1 -- 172.21.15.29:0/4064358194 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b803f030 con 0x7fd6c8102510 2024-07-27T06:37:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.694+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/4064358194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c81a6250 con 0x7fd6c8102510 2024-07-27T06:37:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.694+0000 7fd6b77fe640 1 -- 172.21.15.29:0/4064358194 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6b8003cb0 con 0x7fd6c8102510 2024-07-27T06:37:12.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.694+0000 7fd6b77fe640 1 -- 172.21.15.29:0/4064358194 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8035070 con 0x7fd6c8102510 2024-07-27T06:37:12.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.694+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/4064358194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c81a6780 con 0x7fd6c8102510 2024-07-27T06:37:12.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.695+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/4064358194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6c8103740 con 0x7fd6c8102510 2024-07-27T06:37:12.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.697+0000 7fd6b77fe640 1 -- 172.21.15.29:0/4064358194 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7fd6b8042030 con 0x7fd6c8102510 2024-07-27T06:37:12.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.697+0000 7fd6b77fe640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd6b00b3b90 0x7fd6b00b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:12.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.697+0000 7fd6c5d9b640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd6b00b3b90 0x7fd6b00b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:12.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.697+0000 7fd6b77fe640 1 -- 172.21.15.29:0/4064358194 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fd6b8102ae0 con 0x7fd6c8102510 2024-07-27T06:37:12.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.698+0000 7fd6c5d9b640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd6b00b3b90 0x7fd6b00b60a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd6c81006f0 tx=0x7fd6bc00b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:12.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.700+0000 7fd6b77fe640 1 -- 172.21.15.29:0/4064358194 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd6b80cb6e0 con 0x7fd6c8102510 2024-07-27T06:37:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:12 smithi077 ceph-mon[37153]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:12.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:12.897+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/4064358194 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd6c8063230 con 0x7fd6b00b3b90 2024-07-27T06:37:12.899 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:12 smithi029 ceph-mon[32610]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:12 smithi047 ceph-mon[36835]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:13.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:13 smithi029 ceph-mon[32610]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:37:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:37:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:37:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:13 smithi047 ceph-mon[36835]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:37:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:37:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:37:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:13 smithi077 ceph-mon[37153]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:37:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:37:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:37:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:14.910 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:14 smithi029 ceph-mon[32610]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:14 smithi047 ceph-mon[36835]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:14 smithi077 ceph-mon[37153]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:15.873 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:15 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2353616048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06833d24-2e1a-4199-9338-7d6883d1bf9a"}]: dispatch 2024-07-27T06:37:15.873 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:15 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2353616048' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06833d24-2e1a-4199-9338-7d6883d1bf9a"}]': finished 2024-07-27T06:37:15.873 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:15 smithi029 ceph-mon[32610]: osdmap e6: 1 total, 0 up, 1 in 2024-07-27T06:37:15.874 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:15 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2353616048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06833d24-2e1a-4199-9338-7d6883d1bf9a"}]: dispatch 2024-07-27T06:37:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:15 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2353616048' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06833d24-2e1a-4199-9338-7d6883d1bf9a"}]': finished 2024-07-27T06:37:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:15 smithi047 ceph-mon[36835]: osdmap e6: 1 total, 0 up, 1 in 2024-07-27T06:37:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:15 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2353616048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06833d24-2e1a-4199-9338-7d6883d1bf9a"}]: dispatch 2024-07-27T06:37:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:15 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2353616048' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06833d24-2e1a-4199-9338-7d6883d1bf9a"}]': finished 2024-07-27T06:37:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:15 smithi077 ceph-mon[37153]: osdmap e6: 1 total, 0 up, 1 in 2024-07-27T06:37:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:16 smithi047 ceph-mon[36835]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:16 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/22325557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:37:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:16 smithi029 ceph-mon[32610]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:16.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:16 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/22325557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:37:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:16 smithi077 ceph-mon[37153]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:16 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/22325557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:37:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:18 smithi077 ceph-mon[37153]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:18.875 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:18 smithi029 ceph-mon[32610]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:18 smithi047 ceph-mon[36835]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:20 smithi047 ceph-mon[36835]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:20.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:20 smithi029 ceph-mon[32610]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:20 smithi077 ceph-mon[37153]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:22.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:22 smithi029 ceph-mon[32610]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:22.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-27T06:37:22.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:22.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:22 smithi029 ceph-mon[32610]: Deploying daemon osd.0 on smithi029 2024-07-27T06:37:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:22 smithi047 ceph-mon[36835]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-27T06:37:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:22 smithi047 ceph-mon[36835]: Deploying daemon osd.0 on smithi029 2024-07-27T06:37:23.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:22 smithi077 ceph-mon[37153]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:23.413 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-27T06:37:23.413 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:23.414 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:22 smithi077 ceph-mon[37153]: Deploying daemon osd.0 on smithi029 2024-07-27T06:37:24.749 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:24 smithi029 ceph-mon[32610]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:24 smithi047 ceph-mon[36835]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:24 smithi077 ceph-mon[37153]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:26 smithi029 ceph-mon[32610]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:26 smithi047 ceph-mon[36835]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:26 smithi077 ceph-mon[37153]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:37:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:37:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:37:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:28 smithi047 ceph-mon[36835]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:28.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:28 smithi029 ceph-mon[32610]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:28.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:28.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:28 smithi077 ceph-mon[37153]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.152+0000 7fd6b77fe640 1 -- 172.21.15.29:0/4064358194 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd6c8063230 con 0x7fd6b00b3b90 2024-07-27T06:37:31.154 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 0 on host 'smithi029' 2024-07-27T06:37:31.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.154+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/4064358194 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd6b00b3b90 msgr2=0x7fd6b00b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:31.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.154+0000 7fd6ccf7e640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd6b00b3b90 0x7fd6b00b60a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd6c81006f0 tx=0x7fd6bc00b390 comp rx=0 tx=0).stop 2024-07-27T06:37:31.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.154+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/4064358194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c8102510 msgr2=0x7fd6c819ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:31.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.154+0000 7fd6ccf7e640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c8102510 0x7fd6c819ad00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8005940 tx=0x7fd6b80042f0 comp rx=0 tx=0).stop 2024-07-27T06:37:31.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.154+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/4064358194 shutdown_connections 2024-07-27T06:37:31.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.154+0000 7fd6ccf7e640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd6b00b3b90 0x7fd6b00b60a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.154+0000 7fd6ccf7e640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6c8108750 0x7fd6c819b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.154+0000 7fd6ccf7e640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6c8107e00 0x7fd6c819b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.154+0000 7fd6ccf7e640 1 --2- 172.21.15.29:0/4064358194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6c8102510 0x7fd6c819ad00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.154+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/4064358194 >> 172.21.15.29:0/4064358194 conn(0x7fd6c8079190 msgr2=0x7fd6c8100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.154+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/4064358194 shutdown_connections 2024-07-27T06:37:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:37:31.154+0000 7fd6ccf7e640 1 -- 172.21.15.29:0/4064358194 wait complete. 2024-07-27T06:37:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:31 smithi047 ceph-mon[36835]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:37:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:37:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:31 smithi029 ceph-mon[32610]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:37:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:37:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:31 smithi077 ceph-mon[37153]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:37:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:37:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:31.879 DEBUG:teuthology.orchestra.run.smithi029:osd.0> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@osd.0.service 2024-07-27T06:37:31.885 INFO:tasks.cephadm:Deploying osd.1 on smithi047 with /dev/vg_nvme/lv_4... 2024-07-27T06:37:32.194 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-27T06:37:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.b/config 2024-07-27T06:37:32.471 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:32 smithi029 ceph-mon[32610]: from='osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-27T06:37:32.580 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:32 smithi047 ceph-mon[36835]: from='osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-27T06:37:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:32 smithi077 ceph-mon[37153]: from='osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-27T06:37:33.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:33 smithi029 ceph-mon[32610]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:33.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:33 smithi029 ceph-mon[32610]: from='osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-27T06:37:33.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:33 smithi029 ceph-mon[32610]: osdmap e7: 1 total, 0 up, 1 in 2024-07-27T06:37:33.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:33 smithi029 ceph-mon[32610]: from='osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-07-27T06:37:33.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:33.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jul 27 06:37:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-0[52010]: 2024-07-27T06:37:33.206+0000 7f1bb96e3640 -1 osd.0 0 waiting for initial osdmap 2024-07-27T06:37:33.388 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jul 27 06:37:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-0[52010]: 2024-07-27T06:37:33.209+0000 7f1bb54e1640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T06:37:33.485 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:33 smithi047 ceph-mon[36835]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:33.485 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:33 smithi047 ceph-mon[36835]: from='osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-27T06:37:33.485 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:33 smithi047 ceph-mon[36835]: osdmap e7: 1 total, 0 up, 1 in 2024-07-27T06:37:33.486 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:33 smithi047 ceph-mon[36835]: from='osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-07-27T06:37:33.486 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:33 smithi077 ceph-mon[37153]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:33 smithi077 ceph-mon[37153]: from='osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-27T06:37:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:33 smithi077 ceph-mon[37153]: osdmap e7: 1 total, 0 up, 1 in 2024-07-27T06:37:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:33 smithi077 ceph-mon[37153]: from='osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-07-27T06:37:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:34.100 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-27T06:37:34.182 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-27T06:37:34.192 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2024-07-27T06:37:34.192 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2024-07-27T06:37:34.192 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00839936 s, 1.2 GB/s 2024-07-27T06:37:34.192 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2024-07-27T06:37:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:34 smithi047 ceph-mon[36835]: from='osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-07-27T06:37:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:34 smithi047 ceph-mon[36835]: osdmap e8: 1 total, 0 up, 1 in 2024-07-27T06:37:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:34.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:34 smithi029 ceph-mon[32610]: from='osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-07-27T06:37:34.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:34 smithi029 ceph-mon[32610]: osdmap e8: 1 total, 0 up, 1 in 2024-07-27T06:37:34.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:34.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:34 smithi077 ceph-mon[37153]: from='osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-07-27T06:37:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:34 smithi077 ceph-mon[37153]: osdmap e8: 1 total, 0 up, 1 in 2024-07-27T06:37:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:34.863 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch daemon add osd smithi047:vg_nvme/lv_4 2024-07-27T06:37:35.132 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.b/config 2024-07-27T06:37:35.486 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: purged_snaps scrub starts 2024-07-27T06:37:35.486 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: purged_snaps scrub ok 2024-07-27T06:37:35.486 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:35.486 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641] boot 2024-07-27T06:37:35.487 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: osdmap e9: 1 total, 1 up, 1 in 2024-07-27T06:37:35.487 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:35.487 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:35.487 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:35.487 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:37:35.487 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:35.487 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:35.487 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:37:35.487 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: purged_snaps scrub starts 2024-07-27T06:37:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: purged_snaps scrub ok 2024-07-27T06:37:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641] boot 2024-07-27T06:37:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: osdmap e9: 1 total, 1 up, 1 in 2024-07-27T06:37:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:37:35.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:35.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:35.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:37:35.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: purged_snaps scrub starts 2024-07-27T06:37:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: purged_snaps scrub ok 2024-07-27T06:37:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-27T06:37:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: osd.0 [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641] boot 2024-07-27T06:37:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: osdmap e9: 1 total, 1 up, 1 in 2024-07-27T06:37:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-27T06:37:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:37:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:37:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:36 smithi047 ceph-mon[36835]: Detected new or changed devices on smithi029 2024-07-27T06:37:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:36 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 17092M 2024-07-27T06:37:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:36 smithi047 ceph-mon[36835]: osdmap e10: 1 total, 1 up, 1 in 2024-07-27T06:37:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:36 smithi077 ceph-mon[37153]: Detected new or changed devices on smithi029 2024-07-27T06:37:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:36 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 17092M 2024-07-27T06:37:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:36 smithi077 ceph-mon[37153]: osdmap e10: 1 total, 1 up, 1 in 2024-07-27T06:37:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:36 smithi029 ceph-mon[32610]: Detected new or changed devices on smithi029 2024-07-27T06:37:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:36 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 17092M 2024-07-27T06:37:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:36 smithi029 ceph-mon[32610]: osdmap e10: 1 total, 1 up, 1 in 2024-07-27T06:37:36.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.795+0000 7f44fb9e1640 1 -- 172.21.15.47:0/931881139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44f4102510 msgr2=0x7f44f4102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:36.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.795+0000 7f44fb9e1640 1 --2- 172.21.15.47:0/931881139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44f4102510 0x7f44f4102920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f44f0009900 tx=0x7f44f002a410 comp rx=0 tx=0).stop 2024-07-27T06:37:36.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.795+0000 7f44fb9e1640 1 -- 172.21.15.47:0/931881139 shutdown_connections 2024-07-27T06:37:36.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.795+0000 7f44fb9e1640 1 --2- 172.21.15.47:0/931881139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44f4108750 0x7f44f410ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:36.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.795+0000 7f44fb9e1640 1 --2- 172.21.15.47:0/931881139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44f4107e00 0x7f44f4108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:36.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.795+0000 7f44fb9e1640 1 --2- 172.21.15.47:0/931881139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44f4102510 0x7f44f4102920 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:36.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.795+0000 7f44fb9e1640 1 -- 172.21.15.47:0/931881139 >> 172.21.15.47:0/931881139 conn(0x7f44f40791e0 msgr2=0x7f44f40795f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:36.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.795+0000 7f44fb9e1640 1 -- 172.21.15.47:0/931881139 shutdown_connections 2024-07-27T06:37:36.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.795+0000 7f44fb9e1640 1 -- 172.21.15.47:0/931881139 wait complete. 2024-07-27T06:37:36.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.796+0000 7f44fb9e1640 1 Processor -- start 2024-07-27T06:37:36.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.796+0000 7f44fb9e1640 1 -- start start 2024-07-27T06:37:36.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.796+0000 7f44fb9e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44f4102510 0x7f44f4199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.796+0000 7f44fb9e1640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44f4107e00 0x7f44f419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.796+0000 7f44fb9e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44f4108750 0x7f44f419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.796+0000 7f44fb9e1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f44f4112420 con 0x7f44f4108750 2024-07-27T06:37:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.796+0000 7f44fb9e1640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f44f41122a0 con 0x7f44f4107e00 2024-07-27T06:37:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.796+0000 7f44fb9e1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f44f41125a0 con 0x7f44f4102510 2024-07-27T06:37:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.796+0000 7f44f8f7c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44f4107e00 0x7f44f419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.796+0000 7f44f8f7c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44f4107e00 0x7f44f419a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60928/0 (socket says 172.21.15.47:60928) 2024-07-27T06:37:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.796+0000 7f44f8f7c640 1 -- 172.21.15.47:0/2082868013 learned_addr learned my addr 172.21.15.47:0/2082868013 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-07-27T06:37:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.796+0000 7f44f9f7e640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44f4108750 0x7f44f419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.797+0000 7f44f977d640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44f4102510 0x7f44f4199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.797+0000 7f44f8f7c640 1 -- 172.21.15.47:0/2082868013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44f4102510 msgr2=0x7f44f4199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:36.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.797+0000 7f44f8f7c640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44f4102510 0x7f44f4199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:36.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.797+0000 7f44f8f7c640 1 -- 172.21.15.47:0/2082868013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44f4108750 msgr2=0x7f44f419f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:36.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.797+0000 7f44f8f7c640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44f4108750 0x7f44f419f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:36.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.797+0000 7f44f8f7c640 1 -- 172.21.15.47:0/2082868013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f419fe90 con 0x7f44f4107e00 2024-07-27T06:37:36.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.797+0000 7f44f8f7c640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44f4107e00 0x7f44f419a4c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f44e400d740 tx=0x7f44e400dc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:36.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.797+0000 7f44ea7fc640 1 -- 172.21.15.47:0/2082868013 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4017070 con 0x7f44f4107e00 2024-07-27T06:37:36.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.797+0000 7f44ea7fc640 1 -- 172.21.15.47:0/2082868013 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44e400de50 con 0x7f44f4107e00 2024-07-27T06:37:36.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.797+0000 7f44ea7fc640 1 -- 172.21.15.47:0/2082868013 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e400f620 con 0x7f44f4107e00 2024-07-27T06:37:36.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.797+0000 7f44fb9e1640 1 -- 172.21.15.47:0/2082868013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f41a0180 con 0x7f44f4107e00 2024-07-27T06:37:36.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.797+0000 7f44fb9e1640 1 -- 172.21.15.47:0/2082868013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f41a68e0 con 0x7f44f4107e00 2024-07-27T06:37:36.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.798+0000 7f44fb9e1640 1 -- 172.21.15.47:0/2082868013 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f4102920 con 0x7f44f4107e00 2024-07-27T06:37:36.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.801+0000 7f44ea7fc640 1 -- 172.21.15.47:0/2082868013 <== mon.2 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f44e401a070 con 0x7f44f4107e00 2024-07-27T06:37:36.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.801+0000 7f44ea7fc640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f44e00b3c10 0x7f44e00b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:36.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.802+0000 7f44ea7fc640 1 -- 172.21.15.47:0/2082868013 <== mon.2 v2:172.21.15.47:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1982+0+0 (secure 0 0 0) 0x7f44e40e1b20 con 0x7f44f4107e00 2024-07-27T06:37:36.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.802+0000 7f44f977d640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f44e00b3c10 0x7f44e00b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:36.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.802+0000 7f44f977d640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f44e00b3c10 0x7f44e00b6120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f44f0031440 tx=0x7f44f0009560 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:36.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.805+0000 7f44ea7fc640 1 -- 172.21.15.47:0/2082868013 <== mon.2 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f44e40aa520 con 0x7f44f4107e00 2024-07-27T06:37:36.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:36.996+0000 7f44fb9e1640 1 -- 172.21.15.47:0/2082868013 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f44f4063230 con 0x7f44e00b3c10 2024-07-27T06:37:37.482 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:37 smithi047 ceph-mon[36835]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:37.482 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:37:37.483 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:37:37.483 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:37.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:37 smithi077 ceph-mon[37153]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:37.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:37:37.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:37:37.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:37.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:37 smithi029 ceph-mon[32610]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:37:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:37:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:38.507 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:38 smithi047 ceph-mon[36835]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:37:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:38 smithi077 ceph-mon[37153]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:37:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:38 smithi029 ceph-mon[32610]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:37:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:39 smithi047 ceph-mon[36835]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:39 smithi077 ceph-mon[37153]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:39.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:39 smithi029 ceph-mon[32610]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:40 smithi047 ceph-mon[36835]: from='client.? 172.21.15.47:0/1961682379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "906aba19-80dc-41b5-953c-af6dca604b05"}]: dispatch 2024-07-27T06:37:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:40 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "906aba19-80dc-41b5-953c-af6dca604b05"}]: dispatch 2024-07-27T06:37:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:40 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "906aba19-80dc-41b5-953c-af6dca604b05"}]': finished 2024-07-27T06:37:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:40 smithi047 ceph-mon[36835]: osdmap e11: 2 total, 1 up, 2 in 2024-07-27T06:37:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:40 smithi047 ceph-mon[36835]: from='client.? 172.21.15.47:0/1627285550' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:37:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:40 smithi077 ceph-mon[37153]: from='client.? 172.21.15.47:0/1961682379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "906aba19-80dc-41b5-953c-af6dca604b05"}]: dispatch 2024-07-27T06:37:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:40 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "906aba19-80dc-41b5-953c-af6dca604b05"}]: dispatch 2024-07-27T06:37:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:40 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "906aba19-80dc-41b5-953c-af6dca604b05"}]': finished 2024-07-27T06:37:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:40 smithi077 ceph-mon[37153]: osdmap e11: 2 total, 1 up, 2 in 2024-07-27T06:37:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:40 smithi077 ceph-mon[37153]: from='client.? 172.21.15.47:0/1627285550' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:37:40.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:40 smithi029 ceph-mon[32610]: from='client.? 172.21.15.47:0/1961682379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "906aba19-80dc-41b5-953c-af6dca604b05"}]: dispatch 2024-07-27T06:37:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:40 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "906aba19-80dc-41b5-953c-af6dca604b05"}]: dispatch 2024-07-27T06:37:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:40 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "906aba19-80dc-41b5-953c-af6dca604b05"}]': finished 2024-07-27T06:37:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:40 smithi029 ceph-mon[32610]: osdmap e11: 2 total, 1 up, 2 in 2024-07-27T06:37:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:40 smithi029 ceph-mon[32610]: from='client.? 172.21.15.47:0/1627285550' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:37:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:41 smithi077 ceph-mon[37153]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:41 smithi047 ceph-mon[36835]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:41.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:41 smithi029 ceph-mon[32610]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:43 smithi047 ceph-mon[36835]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:43 smithi077 ceph-mon[37153]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:43.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:43 smithi029 ceph-mon[32610]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:45 smithi047 ceph-mon[36835]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:45.525 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:45 smithi029 ceph-mon[32610]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:45 smithi077 ceph-mon[37153]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:47.279 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:47 smithi047 ceph-mon[36835]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:47.279 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-27T06:37:47.279 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:47.279 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:47 smithi047 ceph-mon[36835]: Deploying daemon osd.1 on smithi047 2024-07-27T06:37:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:47 smithi077 ceph-mon[37153]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-27T06:37:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:47 smithi077 ceph-mon[37153]: Deploying daemon osd.1 on smithi047 2024-07-27T06:37:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:47 smithi029 ceph-mon[32610]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-27T06:37:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:47 smithi029 ceph-mon[32610]: Deploying daemon osd.1 on smithi047 2024-07-27T06:37:49.515 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:49 smithi029 ceph-mon[32610]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:49 smithi077 ceph-mon[37153]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:49.690 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:49 smithi047 ceph-mon[36835]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:51 smithi029 ceph-mon[32610]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:51 smithi077 ceph-mon[37153]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:51 smithi047 ceph-mon[36835]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:37:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:37:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:37:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:53.574 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:53 smithi047 ceph-mon[36835]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:53 smithi077 ceph-mon[37153]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:53 smithi029 ceph-mon[32610]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:54.996 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:54 smithi047 ceph-mon[36835]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:54.996 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:54.997 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:54.997 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:54.997 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:37:54.997 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:55.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.036+0000 7f44ea7fc640 1 -- 172.21.15.47:0/2082868013 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f44f4063230 con 0x7f44e00b3c10 2024-07-27T06:37:55.038 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 1 on host 'smithi047' 2024-07-27T06:37:55.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.038+0000 7f44fb9e1640 1 -- 172.21.15.47:0/2082868013 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f44e00b3c10 msgr2=0x7f44e00b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:55.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.038+0000 7f44fb9e1640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f44e00b3c10 0x7f44e00b6120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f44f0031440 tx=0x7f44f0009560 comp rx=0 tx=0).stop 2024-07-27T06:37:55.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.038+0000 7f44fb9e1640 1 -- 172.21.15.47:0/2082868013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44f4107e00 msgr2=0x7f44f419a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:55.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.038+0000 7f44fb9e1640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44f4107e00 0x7f44f419a4c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f44e400d740 tx=0x7f44e400dc00 comp rx=0 tx=0).stop 2024-07-27T06:37:55.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.038+0000 7f44fb9e1640 1 -- 172.21.15.47:0/2082868013 shutdown_connections 2024-07-27T06:37:55.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.038+0000 7f44fb9e1640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44f4108750 0x7f44f419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:55.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.038+0000 7f44fb9e1640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44f4107e00 0x7f44f419a4c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:55.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.038+0000 7f44fb9e1640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f44e00b3c10 0x7f44e00b6120 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:55.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.038+0000 7f44fb9e1640 1 --2- 172.21.15.47:0/2082868013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44f4102510 0x7f44f4199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:55.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.038+0000 7f44fb9e1640 1 -- 172.21.15.47:0/2082868013 >> 172.21.15.47:0/2082868013 conn(0x7f44f40791e0 msgr2=0x7f44f4101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:55.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.039+0000 7f44fb9e1640 1 -- 172.21.15.47:0/2082868013 shutdown_connections 2024-07-27T06:37:55.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:37:55.039+0000 7f44fb9e1640 1 -- 172.21.15.47:0/2082868013 wait complete. 2024-07-27T06:37:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:54 smithi029 ceph-mon[32610]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:37:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:54 smithi077 ceph-mon[37153]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:37:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:55.611 DEBUG:teuthology.orchestra.run.smithi047:osd.1> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@osd.1.service 2024-07-27T06:37:55.613 INFO:tasks.cephadm:Deploying osd.2 on smithi077 with /dev/vg_nvme/lv_4... 2024-07-27T06:37:55.614 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-27T06:37:55.868 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.c/config 2024-07-27T06:37:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:37:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:55 smithi047 ceph-mon[36835]: from='osd.1 [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-27T06:37:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:55 smithi047 ceph-mon[36835]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-27T06:37:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:37:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:55 smithi029 ceph-mon[32610]: from='osd.1 [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-27T06:37:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:55 smithi029 ceph-mon[32610]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-27T06:37:56.250 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:37:56.250 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:56.250 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:56.250 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:55 smithi077 ceph-mon[37153]: from='osd.1 [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-27T06:37:56.250 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:55 smithi077 ceph-mon[37153]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-27T06:37:57.205 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:57 smithi077 ceph-mon[37153]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:57.205 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:57 smithi077 ceph-mon[37153]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-27T06:37:57.206 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:57 smithi077 ceph-mon[37153]: from='osd.1 [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:37:57.206 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:57 smithi077 ceph-mon[37153]: osdmap e12: 2 total, 1 up, 2 in 2024-07-27T06:37:57.206 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:57.206 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:57 smithi077 ceph-mon[37153]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:37:57.344 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:57 smithi047 ceph-mon[36835]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:57 smithi047 ceph-mon[36835]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-27T06:37:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:57 smithi047 ceph-mon[36835]: from='osd.1 [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:37:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:57 smithi047 ceph-mon[36835]: osdmap e12: 2 total, 1 up, 2 in 2024-07-27T06:37:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:57 smithi047 ceph-mon[36835]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:37:57.448 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-27T06:37:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:57 smithi029 ceph-mon[32610]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:57 smithi029 ceph-mon[32610]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-27T06:37:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:57 smithi029 ceph-mon[32610]: from='osd.1 [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:37:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:57 smithi029 ceph-mon[32610]: osdmap e12: 2 total, 1 up, 2 in 2024-07-27T06:37:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:57 smithi029 ceph-mon[32610]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:37:57.547 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-27T06:37:57.561 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-07-27T06:37:57.561 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-07-27T06:37:57.561 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115085 s, 911 MB/s 2024-07-27T06:37:57.561 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-07-27T06:37:57.696 INFO:journalctl@ceph.osd.1.smithi047.stdout:Jul 27 06:37:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-1[42556]: 2024-07-27T06:37:57.535+0000 7f3a90fc4640 -1 osd.1 0 waiting for initial osdmap 2024-07-27T06:37:57.697 INFO:journalctl@ceph.osd.1.smithi047.stdout:Jul 27 06:37:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-1[42556]: 2024-07-27T06:37:57.539+0000 7f3a88dba640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T06:37:58.183 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_4 2024-07-27T06:37:58.306 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:58 smithi047 ceph-mon[36835]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-07-27T06:37:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:58 smithi047 ceph-mon[36835]: osdmap e13: 2 total, 1 up, 2 in 2024-07-27T06:37:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:58.307 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:58 smithi047 ceph-mon[36835]: from='osd.1 ' entity='osd.1' 2024-07-27T06:37:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:58 smithi077 ceph-mon[37153]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-07-27T06:37:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:58 smithi077 ceph-mon[37153]: osdmap e13: 2 total, 1 up, 2 in 2024-07-27T06:37:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:58 smithi077 ceph-mon[37153]: from='osd.1 ' entity='osd.1' 2024-07-27T06:37:58.437 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.c/config 2024-07-27T06:37:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:58 smithi029 ceph-mon[32610]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-07-27T06:37:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:58 smithi029 ceph-mon[32610]: osdmap e13: 2 total, 1 up, 2 in 2024-07-27T06:37:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:58 smithi029 ceph-mon[32610]: from='osd.1 ' entity='osd.1' 2024-07-27T06:37:59.339 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: purged_snaps scrub starts 2024-07-27T06:37:59.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: purged_snaps scrub ok 2024-07-27T06:37:59.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:59.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:59.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: osd.1 [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749] boot 2024-07-27T06:37:59.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: osdmap e14: 2 total, 2 up, 2 in 2024-07-27T06:37:59.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:59.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:59.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:59.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:37:59.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:59.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:59.341 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:37:59.341 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:37:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: purged_snaps scrub starts 2024-07-27T06:37:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: purged_snaps scrub ok 2024-07-27T06:37:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: osd.1 [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749] boot 2024-07-27T06:37:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: osdmap e14: 2 total, 2 up, 2 in 2024-07-27T06:37:59.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:59.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:59.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:59.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:37:59.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:59.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:59.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:37:59.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:37:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: purged_snaps scrub starts 2024-07-27T06:37:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: purged_snaps scrub ok 2024-07-27T06:37:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-27T06:37:59.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:59.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: osd.1 [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749] boot 2024-07-27T06:37:59.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: osdmap e14: 2 total, 2 up, 2 in 2024-07-27T06:37:59.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-27T06:37:59.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:59.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:59.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:37:59.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:59.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:37:59.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:37:59.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:37:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:37:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.847+0000 7f32be0f3640 1 -- 172.21.15.77:0/3466795380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32b8107dd0 msgr2=0x7f32b81081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.847+0000 7f32be0f3640 1 --2- 172.21.15.77:0/3466795380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32b8107dd0 0x7f32b81081e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009900 tx=0x7f32a802a410 comp rx=0 tx=0).stop 2024-07-27T06:37:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.847+0000 7f32be0f3640 1 -- 172.21.15.77:0/3466795380 shutdown_connections 2024-07-27T06:37:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.847+0000 7f32be0f3640 1 --2- 172.21.15.77:0/3466795380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32b8108720 0x7f32b810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.848+0000 7f32be0f3640 1 --2- 172.21.15.77:0/3466795380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32b8107dd0 0x7f32b81081e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.848+0000 7f32be0f3640 1 --2- 172.21.15.77:0/3466795380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b8102490 0x7f32b81028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.848+0000 7f32be0f3640 1 -- 172.21.15.77:0/3466795380 >> 172.21.15.77:0/3466795380 conn(0x7f32b8079150 msgr2=0x7f32b8079560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:37:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.848+0000 7f32be0f3640 1 -- 172.21.15.77:0/3466795380 shutdown_connections 2024-07-27T06:37:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.848+0000 7f32be0f3640 1 -- 172.21.15.77:0/3466795380 wait complete. 2024-07-27T06:37:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.848+0000 7f32be0f3640 1 Processor -- start 2024-07-27T06:37:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.848+0000 7f32be0f3640 1 -- start start 2024-07-27T06:37:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32be0f3640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b8102490 0x7f32b819ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32be0f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32b8107dd0 0x7f32b819b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32be0f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32b8108720 0x7f32b819b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32be0f3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f32b8112370 con 0x7f32b8107dd0 2024-07-27T06:37:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32be0f3640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f32b81121f0 con 0x7f32b8102490 2024-07-27T06:37:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32be0f3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f32b81124f0 con 0x7f32b8108720 2024-07-27T06:37:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32b7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32b8108720 0x7f32b819b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32b7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32b8108720 0x7f32b819b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55036/0 (socket says 172.21.15.77:55036) 2024-07-27T06:37:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32b7fff640 1 -- 172.21.15.77:0/2242638812 learned_addr learned my addr 172.21.15.77:0/2242638812 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-07-27T06:37:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32b77fe640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b8102490 0x7f32b819ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32b6ffd640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32b8107dd0 0x7f32b819b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32b7fff640 1 -- 172.21.15.77:0/2242638812 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b8102490 msgr2=0x7f32b819ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32b7fff640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b8102490 0x7f32b819ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32b7fff640 1 -- 172.21.15.77:0/2242638812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32b8107dd0 msgr2=0x7f32b819b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:37:59.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32b7fff640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32b8107dd0 0x7f32b819b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:37:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32b7fff640 1 -- 172.21.15.77:0/2242638812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b81a5f70 con 0x7f32b8108720 2024-07-27T06:37:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32b6ffd640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32b8107dd0 0x7f32b819b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:37:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.849+0000 7f32b7fff640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32b8108720 0x7f32b819b760 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f32b000cad0 tx=0x7f32b000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.850+0000 7f32b4ff9640 1 -- 172.21.15.77:0/2242638812 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b000ae20 con 0x7f32b8108720 2024-07-27T06:37:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.850+0000 7f32be0f3640 1 -- 172.21.15.77:0/2242638812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b81a6260 con 0x7f32b8108720 2024-07-27T06:37:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.850+0000 7f32b4ff9640 1 -- 172.21.15.77:0/2242638812 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32b0005020 con 0x7f32b8108720 2024-07-27T06:37:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.850+0000 7f32b4ff9640 1 -- 172.21.15.77:0/2242638812 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b000f6b0 con 0x7f32b8108720 2024-07-27T06:37:59.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.850+0000 7f32be0f3640 1 -- 172.21.15.77:0/2242638812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b81a6720 con 0x7f32b8108720 2024-07-27T06:37:59.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.851+0000 7f32be0f3640 1 -- 172.21.15.77:0/2242638812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32b8103710 con 0x7f32b8108720 2024-07-27T06:37:59.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.852+0000 7f32b4ff9640 1 -- 172.21.15.77:0/2242638812 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7f32b00051d0 con 0x7f32b8108720 2024-07-27T06:37:59.855 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.853+0000 7f32b4ff9640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f32a00b3bc0 0x7f32a00b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:37:59.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.854+0000 7f32b4ff9640 1 -- 172.21.15.77:0/2242638812 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2524+0+0 (secure 0 0 0) 0x7f32b00e2950 con 0x7f32b8108720 2024-07-27T06:37:59.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.854+0000 7f32b77fe640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f32a00b3bc0 0x7f32a00b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:37:59.856 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.854+0000 7f32b77fe640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f32a00b3bc0 0x7f32a00b60d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32b806c110 tx=0x7f32ac00b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:37:59.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:37:59.857+0000 7f32b4ff9640 1 -- 172.21.15.77:0/2242638812 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f32b0014030 con 0x7f32b8108720 2024-07-27T06:38:00.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:00.039+0000 7f32be0f3640 1 -- 172.21.15.77:0/2242638812 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f32b8063230 con 0x7f32a00b3bc0 2024-07-27T06:38:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:00 smithi077 ceph-mon[37153]: Detected new or changed devices on smithi047 2024-07-27T06:38:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:00 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 17093M 2024-07-27T06:38:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:38:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:38:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:00 smithi047 ceph-mon[36835]: Detected new or changed devices on smithi047 2024-07-27T06:38:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:00 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 17093M 2024-07-27T06:38:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:38:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:38:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:00 smithi029 ceph-mon[32610]: Detected new or changed devices on smithi047 2024-07-27T06:38:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:00 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 17093M 2024-07-27T06:38:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:38:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:38:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:01.273 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:01 smithi077 ceph-mon[37153]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:01.274 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:01 smithi077 ceph-mon[37153]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:38:01.274 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:01 smithi077 ceph-mon[37153]: osdmap e15: 2 total, 2 up, 2 in 2024-07-27T06:38:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:01 smithi047 ceph-mon[36835]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:01 smithi047 ceph-mon[36835]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:38:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:01 smithi047 ceph-mon[36835]: osdmap e15: 2 total, 2 up, 2 in 2024-07-27T06:38:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:01 smithi029 ceph-mon[32610]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:01 smithi029 ceph-mon[32610]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:38:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:01 smithi029 ceph-mon[32610]: osdmap e15: 2 total, 2 up, 2 in 2024-07-27T06:38:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:03 smithi077 ceph-mon[37153]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:03 smithi077 ceph-mon[37153]: from='client.? 172.21.15.77:0/107722267' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "194ae5a5-aebf-4cff-9b99-585a3073a269"}]: dispatch 2024-07-27T06:38:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:03 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "194ae5a5-aebf-4cff-9b99-585a3073a269"}]: dispatch 2024-07-27T06:38:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:03 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "194ae5a5-aebf-4cff-9b99-585a3073a269"}]': finished 2024-07-27T06:38:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:03 smithi077 ceph-mon[37153]: osdmap e16: 3 total, 2 up, 3 in 2024-07-27T06:38:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:03 smithi077 ceph-mon[37153]: from='client.? 172.21.15.77:0/2747133886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:38:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:03 smithi047 ceph-mon[36835]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:03 smithi047 ceph-mon[36835]: from='client.? 172.21.15.77:0/107722267' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "194ae5a5-aebf-4cff-9b99-585a3073a269"}]: dispatch 2024-07-27T06:38:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:03 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "194ae5a5-aebf-4cff-9b99-585a3073a269"}]: dispatch 2024-07-27T06:38:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:03 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "194ae5a5-aebf-4cff-9b99-585a3073a269"}]': finished 2024-07-27T06:38:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:03 smithi047 ceph-mon[36835]: osdmap e16: 3 total, 2 up, 3 in 2024-07-27T06:38:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:03 smithi047 ceph-mon[36835]: from='client.? 172.21.15.77:0/2747133886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:38:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:03 smithi029 ceph-mon[32610]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:03 smithi029 ceph-mon[32610]: from='client.? 172.21.15.77:0/107722267' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "194ae5a5-aebf-4cff-9b99-585a3073a269"}]: dispatch 2024-07-27T06:38:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:03 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "194ae5a5-aebf-4cff-9b99-585a3073a269"}]: dispatch 2024-07-27T06:38:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:03 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "194ae5a5-aebf-4cff-9b99-585a3073a269"}]': finished 2024-07-27T06:38:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:03 smithi029 ceph-mon[32610]: osdmap e16: 3 total, 2 up, 3 in 2024-07-27T06:38:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:03 smithi029 ceph-mon[32610]: from='client.? 172.21.15.77:0/2747133886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:38:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:05 smithi077 ceph-mon[37153]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:05 smithi047 ceph-mon[36835]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:05 smithi029 ceph-mon[32610]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:07.318 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:07 smithi077 ceph-mon[37153]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:07 smithi047 ceph-mon[36835]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:07 smithi029 ceph-mon[32610]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:09 smithi077 ceph-mon[37153]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-27T06:38:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:09 smithi047 ceph-mon[36835]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-27T06:38:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:09 smithi029 ceph-mon[32610]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-27T06:38:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:10 smithi077 ceph-mon[37153]: Deploying daemon osd.2 on smithi077 2024-07-27T06:38:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:10 smithi047 ceph-mon[36835]: Deploying daemon osd.2 on smithi077 2024-07-27T06:38:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:10 smithi029 ceph-mon[32610]: Deploying daemon osd.2 on smithi077 2024-07-27T06:38:11.255 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:11 smithi077 ceph-mon[37153]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:11 smithi047 ceph-mon[36835]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:11 smithi029 ceph-mon[32610]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:13 smithi077 ceph-mon[37153]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:13 smithi047 ceph-mon[36835]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:13 smithi029 ceph-mon[32610]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:38:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:38:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:38:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:15 smithi077 ceph-mon[37153]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:15 smithi047 ceph-mon[36835]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:15.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:15 smithi029 ceph-mon[32610]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:16 smithi077 ceph-mon[37153]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:16 smithi047 ceph-mon[36835]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:16 smithi029 ceph-mon[32610]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:17.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.287+0000 7f32b4ff9640 1 -- 172.21.15.77:0/2242638812 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f32b8063230 con 0x7f32a00b3bc0 2024-07-27T06:38:17.289 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 2 on host 'smithi077' 2024-07-27T06:38:17.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.290+0000 7f32be0f3640 1 -- 172.21.15.77:0/2242638812 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f32a00b3bc0 msgr2=0x7f32a00b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:38:17.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.290+0000 7f32be0f3640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f32a00b3bc0 0x7f32a00b60d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32b806c110 tx=0x7f32ac00b390 comp rx=0 tx=0).stop 2024-07-27T06:38:17.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.290+0000 7f32be0f3640 1 -- 172.21.15.77:0/2242638812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32b8108720 msgr2=0x7f32b819b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:38:17.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.290+0000 7f32be0f3640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32b8108720 0x7f32b819b760 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f32b000cad0 tx=0x7f32b000cf90 comp rx=0 tx=0).stop 2024-07-27T06:38:17.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.290+0000 7f32be0f3640 1 -- 172.21.15.77:0/2242638812 shutdown_connections 2024-07-27T06:38:17.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.290+0000 7f32be0f3640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f32a00b3bc0 0x7f32a00b60d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:17.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.290+0000 7f32be0f3640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32b8108720 0x7f32b819b760 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.290+0000 7f32be0f3640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32b8107dd0 0x7f32b819b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.290+0000 7f32be0f3640 1 --2- 172.21.15.77:0/2242638812 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f32b8102490 0x7f32b819ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.290+0000 7f32be0f3640 1 -- 172.21.15.77:0/2242638812 >> 172.21.15.77:0/2242638812 conn(0x7f32b8079150 msgr2=0x7f32b8103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:38:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.290+0000 7f32be0f3640 1 -- 172.21.15.77:0/2242638812 shutdown_connections 2024-07-27T06:38:17.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:38:17.290+0000 7f32be0f3640 1 -- 172.21.15.77:0/2242638812 wait complete. 2024-07-27T06:38:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:38:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:38:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:17.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:38:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:17.971 DEBUG:teuthology.orchestra.run.smithi077:osd.2> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@osd.2.service 2024-07-27T06:38:17.973 INFO:tasks.cephadm:Deploying osd.3 on smithi029 with /dev/vg_nvme/lv_3... 2024-07-27T06:38:17.973 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-27T06:38:18.251 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:38:18.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:18 smithi029 ceph-mon[32610]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:18.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:18 smithi029 ceph-mon[32610]: from='osd.2 [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-27T06:38:18.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:18 smithi029 ceph-mon[32610]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-27T06:38:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:18 smithi077 ceph-mon[37153]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:18 smithi077 ceph-mon[37153]: from='osd.2 [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-27T06:38:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:18 smithi077 ceph-mon[37153]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-27T06:38:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:18 smithi047 ceph-mon[36835]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:18 smithi047 ceph-mon[36835]: from='osd.2 [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-27T06:38:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:18 smithi047 ceph-mon[36835]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-27T06:38:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:19 smithi029 ceph-mon[32610]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-27T06:38:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:19 smithi029 ceph-mon[32610]: osdmap e17: 3 total, 2 up, 3 in 2024-07-27T06:38:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:19.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:19 smithi029 ceph-mon[32610]: from='osd.2 [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:38:19.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:19 smithi029 ceph-mon[32610]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:38:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:19 smithi077 ceph-mon[37153]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-27T06:38:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:19 smithi077 ceph-mon[37153]: osdmap e17: 3 total, 2 up, 3 in 2024-07-27T06:38:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:19 smithi077 ceph-mon[37153]: from='osd.2 [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:38:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:19 smithi077 ceph-mon[37153]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:38:19.857 INFO:journalctl@ceph.osd.2.smithi077.stdout:Jul 27 06:38:19 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-2[43506]: 2024-07-27T06:38:19.582+0000 7fee7673a640 -1 osd.2 0 waiting for initial osdmap 2024-07-27T06:38:19.857 INFO:journalctl@ceph.osd.2.smithi077.stdout:Jul 27 06:38:19 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-2[43506]: 2024-07-27T06:38:19.587+0000 7fee72538640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T06:38:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:19 smithi047 ceph-mon[36835]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-27T06:38:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:19 smithi047 ceph-mon[36835]: osdmap e17: 3 total, 2 up, 3 in 2024-07-27T06:38:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:19 smithi047 ceph-mon[36835]: from='osd.2 [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:38:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:19 smithi047 ceph-mon[36835]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:38:20.168 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-27T06:38:20.295 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-27T06:38:20.306 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-07-27T06:38:20.306 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-07-27T06:38:20.306 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00848597 s, 1.2 GB/s 2024-07-27T06:38:20.306 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 2024-07-27T06:38:20.306 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-07-27T06:38:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:20 smithi077 ceph-mon[37153]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:20 smithi077 ceph-mon[37153]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-07-27T06:38:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:20 smithi077 ceph-mon[37153]: osdmap e18: 3 total, 2 up, 3 in 2024-07-27T06:38:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:20 smithi047 ceph-mon[36835]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:20 smithi047 ceph-mon[36835]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-07-27T06:38:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:20 smithi047 ceph-mon[36835]: osdmap e18: 3 total, 2 up, 3 in 2024-07-27T06:38:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:20.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:20 smithi029 ceph-mon[32610]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:20 smithi029 ceph-mon[32610]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-07-27T06:38:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:20 smithi029 ceph-mon[32610]: osdmap e18: 3 total, 2 up, 3 in 2024-07-27T06:38:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:20.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_3 2024-07-27T06:38:21.257 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:38:21.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: purged_snaps scrub starts 2024-07-27T06:38:21.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: purged_snaps scrub ok 2024-07-27T06:38:21.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: osd.2 [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078] boot 2024-07-27T06:38:21.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: osdmap e19: 3 total, 3 up, 3 in 2024-07-27T06:38:21.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:21.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: Detected new or changed devices on smithi077 2024-07-27T06:38:21.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:21.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:21.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:38:21.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 17088M 2024-07-27T06:38:21.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:21.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:21.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:38:21.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: purged_snaps scrub starts 2024-07-27T06:38:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: purged_snaps scrub ok 2024-07-27T06:38:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: osd.2 [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078] boot 2024-07-27T06:38:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: osdmap e19: 3 total, 3 up, 3 in 2024-07-27T06:38:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: Detected new or changed devices on smithi077 2024-07-27T06:38:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:38:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 17088M 2024-07-27T06:38:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:38:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: purged_snaps scrub starts 2024-07-27T06:38:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: purged_snaps scrub ok 2024-07-27T06:38:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: osd.2 [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078] boot 2024-07-27T06:38:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: osdmap e19: 3 total, 3 up, 3 in 2024-07-27T06:38:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-27T06:38:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: Detected new or changed devices on smithi077 2024-07-27T06:38:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:38:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 17088M 2024-07-27T06:38:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:38:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:22.834 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 ceph-mon[32610]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:22.834 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-27T06:38:22.834 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-27T06:38:22.834 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 ceph-mon[32610]: osdmap e20: 3 total, 3 up, 3 in 2024-07-27T06:38:22.835 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-27T06:38:22.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56632]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-27T06:38:22.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56632]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-27T06:38:22.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56632]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-27T06:38:22.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56632]: pam_unix(sudo:session): session closed for user root 2024-07-27T06:38:22.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56635]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-27T06:38:22.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56635]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-27T06:38:22.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56635]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-27T06:38:22.835 INFO:journalctl@ceph.osd.0.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56635]: pam_unix(sudo:session): session closed for user root 2024-07-27T06:38:22.848 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 ceph-mon[37153]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:22.848 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-27T06:38:22.848 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-27T06:38:22.848 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 ceph-mon[37153]: osdmap e20: 3 total, 3 up, 3 in 2024-07-27T06:38:22.848 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-27T06:38:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:22 smithi047 ceph-mon[36835]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-27T06:38:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-27T06:38:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-27T06:38:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:22 smithi047 ceph-mon[36835]: osdmap e20: 3 total, 3 up, 3 in 2024-07-27T06:38:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-27T06:38:22.947 INFO:journalctl@ceph.osd.1.smithi047.stdout:Jul 27 06:38:22 smithi047 sudo[46649]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-27T06:38:22.947 INFO:journalctl@ceph.osd.1.smithi047.stdout:Jul 27 06:38:22 smithi047 sudo[46649]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-27T06:38:22.947 INFO:journalctl@ceph.osd.1.smithi047.stdout:Jul 27 06:38:22 smithi047 sudo[46649]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-27T06:38:22.948 INFO:journalctl@ceph.osd.1.smithi047.stdout:Jul 27 06:38:22 smithi047 sudo[46649]: pam_unix(sudo:session): session closed for user root 2024-07-27T06:38:22.948 INFO:journalctl@ceph.osd.1.smithi047.stdout:Jul 27 06:38:22 smithi047 sudo[46652]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-27T06:38:22.948 INFO:journalctl@ceph.osd.1.smithi047.stdout:Jul 27 06:38:22 smithi047 sudo[46652]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-27T06:38:22.948 INFO:journalctl@ceph.osd.1.smithi047.stdout:Jul 27 06:38:22 smithi047 sudo[46652]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-27T06:38:22.948 INFO:journalctl@ceph.osd.1.smithi047.stdout:Jul 27 06:38:22 smithi047 sudo[46652]: pam_unix(sudo:session): session closed for user root 2024-07-27T06:38:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.083+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3647758102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3d4103590 msgr2=0x7fb3d41039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:38:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.083+0000 7fb3d9c98640 1 --2- 172.21.15.29:0/3647758102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3d4103590 0x7fb3d41039a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009ca0 tx=0x7fb3c402a8c0 comp rx=0 tx=0).stop 2024-07-27T06:38:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.083+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3647758102 shutdown_connections 2024-07-27T06:38:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.083+0000 7fb3d9c98640 1 --2- 172.21.15.29:0/3647758102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3d4109820 0x7fb3d4110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.083+0000 7fb3d9c98640 1 --2- 172.21.15.29:0/3647758102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3d4108ed0 0x7fb3d41092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.083+0000 7fb3d9c98640 1 --2- 172.21.15.29:0/3647758102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3d4103590 0x7fb3d41039a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.083+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3647758102 >> 172.21.15.29:0/3647758102 conn(0x7fb3d407b280 msgr2=0x7fb3d407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:38:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.083+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3647758102 shutdown_connections 2024-07-27T06:38:23.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.083+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3647758102 wait complete. 2024-07-27T06:38:23.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56642]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-27T06:38:23.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56642]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-27T06:38:23.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56642]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-27T06:38:23.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56642]: pam_unix(sudo:session): session closed for user root 2024-07-27T06:38:23.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56645]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-27T06:38:23.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56645]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-27T06:38:23.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56645]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-27T06:38:23.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:22 smithi029 sudo[56645]: pam_unix(sudo:session): session closed for user root 2024-07-27T06:38:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.084+0000 7fb3d9c98640 1 Processor -- start 2024-07-27T06:38:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.085+0000 7fb3d9c98640 1 -- start start 2024-07-27T06:38:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.085+0000 7fb3d9c98640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3d4103590 0x7fb3d4199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:38:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.085+0000 7fb3d9c98640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3d4108ed0 0x7fb3d419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:38:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.085+0000 7fb3d9c98640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3d4109820 0x7fb3d419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:38:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.085+0000 7fb3d9c98640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d41134d0 con 0x7fb3d4109820 2024-07-27T06:38:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.085+0000 7fb3d9c98640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d4113350 con 0x7fb3d4108ed0 2024-07-27T06:38:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.085+0000 7fb3d9c98640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d4113650 con 0x7fb3d4103590 2024-07-27T06:38:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.085+0000 7fb3d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3d4103590 0x7fb3d4199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:38:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.085+0000 7fb3d37fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3d4103590 0x7fb3d4199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:57710/0 (socket says 172.21.15.29:57710) 2024-07-27T06:38:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.085+0000 7fb3d37fe640 1 -- 172.21.15.29:0/3521002036 learned_addr learned my addr 172.21.15.29:0/3521002036 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.085+0000 7fb3d3fff640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3d4109820 0x7fb3d419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.085+0000 7fb3d2ffd640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3d4108ed0 0x7fb3d419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.086+0000 7fb3d37fe640 1 -- 172.21.15.29:0/3521002036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3d4108ed0 msgr2=0x7fb3d419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.086+0000 7fb3d37fe640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3d4108ed0 0x7fb3d419a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.086+0000 7fb3d37fe640 1 -- 172.21.15.29:0/3521002036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3d4109820 msgr2=0x7fb3d419f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.086+0000 7fb3d37fe640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3d4109820 0x7fb3d419f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.086+0000 7fb3d37fe640 1 -- 172.21.15.29:0/3521002036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d419fd30 con 0x7fb3d4103590 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.086+0000 7fb3d37fe640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3d4103590 0x7fb3d4199e80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb3c402ade0 tx=0x7fb3c4003970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.086+0000 7fb3d0ff9640 1 -- 172.21.15.29:0/3521002036 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4003d60 con 0x7fb3d4103590 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.086+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3521002036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d419ffc0 con 0x7fb3d4103590 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.086+0000 7fb3d0ff9640 1 -- 172.21.15.29:0/3521002036 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3c40044f0 con 0x7fb3d4103590 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.086+0000 7fb3d0ff9640 1 -- 172.21.15.29:0/3521002036 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c402fd10 con 0x7fb3d4103590 2024-07-27T06:38:23.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.086+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3521002036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d41a0500 con 0x7fb3d4103590 2024-07-27T06:38:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.087+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3521002036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3d41039a0 con 0x7fb3d4103590 2024-07-27T06:38:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.089+0000 7fb3d0ff9640 1 -- 172.21.15.29:0/3521002036 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) ==== 151020+0+0 (secure 0 0 0) 0x7fb3c40046a0 con 0x7fb3d4103590 2024-07-27T06:38:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.090+0000 7fb3d0ff9640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb3bc0b3c40 0x7fb3bc0b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:38:23.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.090+0000 7fb3d2ffd640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb3bc0b3c40 0x7fb3bc0b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:38:23.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.090+0000 7fb3d0ff9640 1 -- 172.21.15.29:0/3521002036 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3452+0+0 (secure 0 0 0) 0x7fb3c4104090 con 0x7fb3d4103590 2024-07-27T06:38:23.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.090+0000 7fb3d2ffd640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb3bc0b3c40 0x7fb3bc0b6150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4105e80 tx=0x7fb3c800a2d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:38:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.093+0000 7fb3d0ff9640 1 -- 172.21.15.29:0/3521002036 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb3c40cca20 con 0x7fb3d4103590 2024-07-27T06:38:23.106 INFO:journalctl@ceph.osd.2.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47136]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-27T06:38:23.107 INFO:journalctl@ceph.osd.2.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47136]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-27T06:38:23.107 INFO:journalctl@ceph.osd.2.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47136]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-27T06:38:23.107 INFO:journalctl@ceph.osd.2.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47136]: pam_unix(sudo:session): session closed for user root 2024-07-27T06:38:23.107 INFO:journalctl@ceph.osd.2.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47139]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-27T06:38:23.107 INFO:journalctl@ceph.osd.2.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47139]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-27T06:38:23.107 INFO:journalctl@ceph.osd.2.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47139]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-27T06:38:23.107 INFO:journalctl@ceph.osd.2.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47139]: pam_unix(sudo:session): session closed for user root 2024-07-27T06:38:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47142]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-27T06:38:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47142]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-27T06:38:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47142]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-27T06:38:23.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47142]: pam_unix(sudo:session): session closed for user root 2024-07-27T06:38:23.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47145]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-27T06:38:23.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47145]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-27T06:38:23.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47145]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-27T06:38:23.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:22 smithi077 sudo[47145]: pam_unix(sudo:session): session closed for user root 2024-07-27T06:38:23.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.286+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3521002036 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb3d4063230 con 0x7fb3bc0b3c40 2024-07-27T06:38:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:22 smithi047 sudo[46655]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-27T06:38:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:22 smithi047 sudo[46655]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-27T06:38:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:22 smithi047 sudo[46655]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-27T06:38:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 sudo[46655]: pam_unix(sudo:session): session closed for user root 2024-07-27T06:38:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 sudo[46658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-27T06:38:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 sudo[46658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-27T06:38:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 sudo[46658]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-27T06:38:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 sudo[46658]: pam_unix(sudo:session): session closed for user root 2024-07-27T06:38:23.911 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-27T06:38:23.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: osdmap e21: 3 total, 3 up, 3 in 2024-07-27T06:38:23.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:23.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:23.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:23.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-27T06:38:23.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-27T06:38:23.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:38:23.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:38:23.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:38:23.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-27T06:38:23.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:38:23.912 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:38:23.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:38:23.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-27T06:38:23.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-27T06:38:23.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:38:23.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:38:23.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:38:23.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-27T06:38:23.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:38:23.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:38:23.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:38:23.913 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:23.909+0000 7fb3d0ff9640 1 -- 172.21.15.29:0/3521002036 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fb3c40c85b0 con 0x7fb3d4103590 2024-07-27T06:38:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-27T06:38:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: osdmap e21: 3 total, 3 up, 3 in 2024-07-27T06:38:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-27T06:38:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-27T06:38:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:38:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:38:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:38:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-27T06:38:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:38:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:38:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:38:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-27T06:38:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-27T06:38:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:38:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:38:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:38:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-27T06:38:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:38:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:38:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:38:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-27T06:38:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: osdmap e21: 3 total, 3 up, 3 in 2024-07-27T06:38:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-27T06:38:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-27T06:38:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:38:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:38:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:38:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-27T06:38:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:38:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:38:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:38:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-27T06:38:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-27T06:38:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-27T06:38:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-27T06:38:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-27T06:38:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-27T06:38:24.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:38:24.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:38:24.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:38:24.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:24 smithi029 ceph-mon[32610]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:24 smithi029 ceph-mon[32610]: osdmap e22: 3 total, 3 up, 3 in 2024-07-27T06:38:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:24 smithi029 ceph-mon[32610]: mgrmap e20: x(active, since 2m), standbys: y, z 2024-07-27T06:38:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:24 smithi047 ceph-mon[36835]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:24 smithi047 ceph-mon[36835]: osdmap e22: 3 total, 3 up, 3 in 2024-07-27T06:38:24.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:24 smithi047 ceph-mon[36835]: mgrmap e20: x(active, since 2m), standbys: y, z 2024-07-27T06:38:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:24 smithi077 ceph-mon[37153]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:24 smithi077 ceph-mon[37153]: osdmap e22: 3 total, 3 up, 3 in 2024-07-27T06:38:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:24 smithi077 ceph-mon[37153]: mgrmap e20: x(active, since 2m), standbys: y, z 2024-07-27T06:38:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:26 smithi029 ceph-mon[32610]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:26 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2622078858' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c22ce9c-963e-4f72-86e5-360e781526db"}]: dispatch 2024-07-27T06:38:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:26 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2622078858' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c22ce9c-963e-4f72-86e5-360e781526db"}]': finished 2024-07-27T06:38:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:26 smithi029 ceph-mon[32610]: osdmap e23: 4 total, 3 up, 4 in 2024-07-27T06:38:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-27T06:38:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:26 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/752145176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:38:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:26 smithi047 ceph-mon[36835]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:26 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2622078858' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c22ce9c-963e-4f72-86e5-360e781526db"}]: dispatch 2024-07-27T06:38:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:26 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2622078858' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c22ce9c-963e-4f72-86e5-360e781526db"}]': finished 2024-07-27T06:38:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:26 smithi047 ceph-mon[36835]: osdmap e23: 4 total, 3 up, 4 in 2024-07-27T06:38:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-27T06:38:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:26 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/752145176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:38:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:26 smithi077 ceph-mon[37153]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:26 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2622078858' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c22ce9c-963e-4f72-86e5-360e781526db"}]: dispatch 2024-07-27T06:38:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:26 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2622078858' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c22ce9c-963e-4f72-86e5-360e781526db"}]': finished 2024-07-27T06:38:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:26 smithi077 ceph-mon[37153]: osdmap e23: 4 total, 3 up, 4 in 2024-07-27T06:38:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-27T06:38:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:26 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/752145176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:38:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:28 smithi047 ceph-mon[36835]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:28 smithi029 ceph-mon[32610]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:28 smithi077 ceph-mon[37153]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:30 smithi047 ceph-mon[36835]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:30 smithi029 ceph-mon[32610]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:30 smithi077 ceph-mon[37153]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:32 smithi047 ceph-mon[36835]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:32 smithi029 ceph-mon[32610]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:32 smithi077 ceph-mon[37153]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:33.786 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-27T06:38:33.786 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:33.786 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:33 smithi029 ceph-mon[32610]: Deploying daemon osd.3 on smithi029 2024-07-27T06:38:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-27T06:38:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:33 smithi047 ceph-mon[36835]: Deploying daemon osd.3 on smithi029 2024-07-27T06:38:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-27T06:38:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:33 smithi077 ceph-mon[37153]: Deploying daemon osd.3 on smithi029 2024-07-27T06:38:34.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:34 smithi029 ceph-mon[32610]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:34 smithi047 ceph-mon[36835]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:34 smithi077 ceph-mon[37153]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:36 smithi047 ceph-mon[36835]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:36 smithi029 ceph-mon[32610]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:36 smithi077 ceph-mon[37153]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:38 smithi047 ceph-mon[36835]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:38.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:38 smithi029 ceph-mon[32610]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:38 smithi077 ceph-mon[37153]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:38:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:38:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:38:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:40.923 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:40 smithi029 ceph-mon[32610]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:40 smithi077 ceph-mon[37153]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:40 smithi047 ceph-mon[36835]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:38:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:42.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.549+0000 7fb3d0ff9640 1 -- 172.21.15.29:0/3521002036 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb3d4063230 con 0x7fb3bc0b3c40 2024-07-27T06:38:42.551 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 3 on host 'smithi029' 2024-07-27T06:38:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.552+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3521002036 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb3bc0b3c40 msgr2=0x7fb3bc0b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:38:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.552+0000 7fb3d9c98640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb3bc0b3c40 0x7fb3bc0b6150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4105e80 tx=0x7fb3c800a2d0 comp rx=0 tx=0).stop 2024-07-27T06:38:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.552+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3521002036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3d4103590 msgr2=0x7fb3d4199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:38:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.552+0000 7fb3d9c98640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3d4103590 0x7fb3d4199e80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb3c402ade0 tx=0x7fb3c4003970 comp rx=0 tx=0).stop 2024-07-27T06:38:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.552+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3521002036 shutdown_connections 2024-07-27T06:38:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.552+0000 7fb3d9c98640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3d4109820 0x7fb3d419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.552+0000 7fb3d9c98640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3d4108ed0 0x7fb3d419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.552+0000 7fb3d9c98640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb3bc0b3c40 0x7fb3bc0b6150 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.552+0000 7fb3d9c98640 1 --2- 172.21.15.29:0/3521002036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3d4103590 0x7fb3d4199e80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.552+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3521002036 >> 172.21.15.29:0/3521002036 conn(0x7fb3d407b280 msgr2=0x7fb3d4106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:38:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.553+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3521002036 shutdown_connections 2024-07-27T06:38:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:38:42.553+0000 7fb3d9c98640 1 -- 172.21.15.29:0/3521002036 wait complete. 2024-07-27T06:38:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:38:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:38:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:43.315 DEBUG:teuthology.orchestra.run.smithi029:osd.3> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@osd.3.service 2024-07-27T06:38:43.318 INFO:tasks.cephadm:Deploying osd.4 on smithi047 with /dev/vg_nvme/lv_3... 2024-07-27T06:38:43.318 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-27T06:38:43.589 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.b/config 2024-07-27T06:38:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:43 smithi077 ceph-mon[37153]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:38:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:43 smithi077 ceph-mon[37153]: from='osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-27T06:38:43.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:43 smithi029 ceph-mon[32610]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:43.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:38:43.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:43.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:43.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:43 smithi029 ceph-mon[32610]: from='osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-27T06:38:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:43 smithi047 ceph-mon[36835]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:38:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:43 smithi047 ceph-mon[36835]: from='osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-27T06:38:44.851 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:44 smithi047 ceph-mon[36835]: from='osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-27T06:38:44.852 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:44 smithi047 ceph-mon[36835]: osdmap e24: 4 total, 3 up, 4 in 2024-07-27T06:38:44.852 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:44 smithi047 ceph-mon[36835]: from='osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-07-27T06:38:44.852 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-27T06:38:44.852 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:44 smithi047 ceph-mon[36835]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:44 smithi077 ceph-mon[37153]: from='osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-27T06:38:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:44 smithi077 ceph-mon[37153]: osdmap e24: 4 total, 3 up, 4 in 2024-07-27T06:38:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:44 smithi077 ceph-mon[37153]: from='osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-07-27T06:38:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-27T06:38:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:44 smithi077 ceph-mon[37153]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:44.874 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:44 smithi029 ceph-mon[32610]: from='osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-27T06:38:44.875 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:44 smithi029 ceph-mon[32610]: osdmap e24: 4 total, 3 up, 4 in 2024-07-27T06:38:44.875 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:44 smithi029 ceph-mon[32610]: from='osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-07-27T06:38:44.875 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-27T06:38:44.875 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:44 smithi029 ceph-mon[32610]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:44.875 INFO:journalctl@ceph.osd.3.smithi029.stdout:Jul 27 06:38:44 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-3[58899]: 2024-07-27T06:38:44.560+0000 7f0ef11cc640 -1 osd.3 0 waiting for initial osdmap 2024-07-27T06:38:44.875 INFO:journalctl@ceph.osd.3.smithi029.stdout:Jul 27 06:38:44 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-3[58899]: 2024-07-27T06:38:44.567+0000 7f0eecfca640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T06:38:45.510 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-27T06:38:45.607 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-27T06:38:45.617 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2024-07-27T06:38:45.617 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2024-07-27T06:38:45.617 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00839829 s, 1.2 GB/s 2024-07-27T06:38:45.618 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2024-07-27T06:38:45.739 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:45 smithi047 ceph-mon[36835]: from='osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-07-27T06:38:45.739 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:45 smithi047 ceph-mon[36835]: osdmap e25: 4 total, 3 up, 4 in 2024-07-27T06:38:45.739 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-27T06:38:45.739 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:45 smithi047 ceph-mon[36835]: osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312] boot 2024-07-27T06:38:45.739 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:45 smithi047 ceph-mon[36835]: osdmap e26: 4 total, 4 up, 4 in 2024-07-27T06:38:45.739 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-27T06:38:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:45 smithi077 ceph-mon[37153]: from='osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-07-27T06:38:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:45 smithi077 ceph-mon[37153]: osdmap e25: 4 total, 3 up, 4 in 2024-07-27T06:38:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-27T06:38:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:45 smithi077 ceph-mon[37153]: osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312] boot 2024-07-27T06:38:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:45 smithi077 ceph-mon[37153]: osdmap e26: 4 total, 4 up, 4 in 2024-07-27T06:38:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-27T06:38:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:45 smithi029 ceph-mon[32610]: from='osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-07-27T06:38:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:45 smithi029 ceph-mon[32610]: osdmap e25: 4 total, 3 up, 4 in 2024-07-27T06:38:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-27T06:38:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:45 smithi029 ceph-mon[32610]: osd.3 [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312] boot 2024-07-27T06:38:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:45 smithi029 ceph-mon[32610]: osdmap e26: 4 total, 4 up, 4 in 2024-07-27T06:38:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-27T06:38:46.470 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch daemon add osd smithi047:vg_nvme/lv_3 2024-07-27T06:38:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:46 smithi047 ceph-mon[36835]: purged_snaps scrub starts 2024-07-27T06:38:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:46 smithi047 ceph-mon[36835]: purged_snaps scrub ok 2024-07-27T06:38:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:46 smithi047 ceph-mon[36835]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:38:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:38:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:38:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:46 smithi047 ceph-mon[36835]: osdmap e27: 4 total, 4 up, 4 in 2024-07-27T06:38:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:46 smithi029 ceph-mon[32610]: purged_snaps scrub starts 2024-07-27T06:38:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:46 smithi029 ceph-mon[32610]: purged_snaps scrub ok 2024-07-27T06:38:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:46 smithi029 ceph-mon[32610]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:38:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:38:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:38:46.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:46.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:46 smithi029 ceph-mon[32610]: osdmap e27: 4 total, 4 up, 4 in 2024-07-27T06:38:46.736 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.b/config 2024-07-27T06:38:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:46 smithi077 ceph-mon[37153]: purged_snaps scrub starts 2024-07-27T06:38:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:46 smithi077 ceph-mon[37153]: purged_snaps scrub ok 2024-07-27T06:38:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:46 smithi077 ceph-mon[37153]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-27T06:38:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:38:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:38:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:38:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:38:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:46 smithi077 ceph-mon[37153]: osdmap e27: 4 total, 4 up, 4 in 2024-07-27T06:38:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:47 smithi077 ceph-mon[37153]: Detected new or changed devices on smithi029 2024-07-27T06:38:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:47 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 8546M 2024-07-27T06:38:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:47 smithi077 ceph-mon[37153]: osdmap e28: 4 total, 4 up, 4 in 2024-07-27T06:38:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:47 smithi047 ceph-mon[36835]: Detected new or changed devices on smithi029 2024-07-27T06:38:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:47 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 8546M 2024-07-27T06:38:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:47 smithi047 ceph-mon[36835]: osdmap e28: 4 total, 4 up, 4 in 2024-07-27T06:38:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:47 smithi029 ceph-mon[32610]: Detected new or changed devices on smithi029 2024-07-27T06:38:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:47 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 8546M 2024-07-27T06:38:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:47 smithi029 ceph-mon[32610]: osdmap e28: 4 total, 4 up, 4 in 2024-07-27T06:38:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.431+0000 7fb519e32640 1 -- 172.21.15.47:0/3628399711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb514108f20 msgr2=0x7fb514109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:38:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.431+0000 7fb519e32640 1 --2- 172.21.15.47:0/3628399711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb514108f20 0x7fb514109330 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb508009900 tx=0x7fb50802a410 comp rx=0 tx=0).stop 2024-07-27T06:38:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.431+0000 7fb519e32640 1 -- 172.21.15.47:0/3628399711 shutdown_connections 2024-07-27T06:38:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.431+0000 7fb519e32640 1 --2- 172.21.15.47:0/3628399711 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514109870 0x7fb5141100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.431+0000 7fb519e32640 1 --2- 172.21.15.47:0/3628399711 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb514108f20 0x7fb514109330 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.431+0000 7fb519e32640 1 --2- 172.21.15.47:0/3628399711 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb514101ad0 0x7fb514101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.431+0000 7fb519e32640 1 -- 172.21.15.47:0/3628399711 >> 172.21.15.47:0/3628399711 conn(0x7fb514079870 msgr2=0x7fb514079c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:38:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.431+0000 7fb519e32640 1 -- 172.21.15.47:0/3628399711 shutdown_connections 2024-07-27T06:38:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.432+0000 7fb519e32640 1 -- 172.21.15.47:0/3628399711 wait complete. 2024-07-27T06:38:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.432+0000 7fb519e32640 1 Processor -- start 2024-07-27T06:38:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.432+0000 7fb519e32640 1 -- start start 2024-07-27T06:38:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb519e32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb514101ad0 0x7fb51419c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:38:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb519e32640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb514108f20 0x7fb51419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:38:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb519e32640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514109870 0x7fb5141a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:38:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb519e32640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb5141134e0 con 0x7fb514109870 2024-07-27T06:38:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb519e32640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fb514113360 con 0x7fb514108f20 2024-07-27T06:38:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb519e32640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb514113660 con 0x7fb514101ad0 2024-07-27T06:38:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb512ffd640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb514108f20 0x7fb51419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:38:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb512ffd640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb514108f20 0x7fb51419c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:39066/0 (socket says 172.21.15.47:39066) 2024-07-27T06:38:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb512ffd640 1 -- 172.21.15.47:0/844569233 learned_addr learned my addr 172.21.15.47:0/844569233 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-07-27T06:38:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb5137fe640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb514101ad0 0x7fb51419c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:38:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb513fff640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514109870 0x7fb5141a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:38:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb512ffd640 1 -- 172.21.15.47:0/844569233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb514101ad0 msgr2=0x7fb51419c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:38:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb512ffd640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb514101ad0 0x7fb51419c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb512ffd640 1 -- 172.21.15.47:0/844569233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514109870 msgr2=0x7fb5141a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:38:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb512ffd640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514109870 0x7fb5141a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:38:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb512ffd640 1 -- 172.21.15.47:0/844569233 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5141a2080 con 0x7fb514108f20 2024-07-27T06:38:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.433+0000 7fb512ffd640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb514108f20 0x7fb51419c6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb50802a930 tx=0x7fb508004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:38:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.434+0000 7fb510ff9640 1 -- 172.21.15.47:0/844569233 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50803f030 con 0x7fb514108f20 2024-07-27T06:38:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.434+0000 7fb510ff9640 1 -- 172.21.15.47:0/844569233 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb508003aa0 con 0x7fb514108f20 2024-07-27T06:38:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.434+0000 7fb519e32640 1 -- 172.21.15.47:0/844569233 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5141a2310 con 0x7fb514108f20 2024-07-27T06:38:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.434+0000 7fb510ff9640 1 -- 172.21.15.47:0/844569233 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb508035070 con 0x7fb514108f20 2024-07-27T06:38:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.434+0000 7fb519e32640 1 -- 172.21.15.47:0/844569233 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5141a8ad0 con 0x7fb514108f20 2024-07-27T06:38:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.435+0000 7fb519e32640 1 -- 172.21.15.47:0/844569233 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb51406b2b0 con 0x7fb514108f20 2024-07-27T06:38:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.436+0000 7fb510ff9640 1 -- 172.21.15.47:0/844569233 <== mon.2 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fb508042030 con 0x7fb514108f20 2024-07-27T06:38:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.437+0000 7fb510ff9640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb4fc0b3c20 0x7fb4fc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:38:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.437+0000 7fb510ff9640 1 -- 172.21.15.47:0/844569233 <== mon.2 v2:172.21.15.47:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3884+0+0 (secure 0 0 0) 0x7fb5081031c0 con 0x7fb514108f20 2024-07-27T06:38:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.437+0000 7fb5137fe640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb4fc0b3c20 0x7fb4fc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:38:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.438+0000 7fb5137fe640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb4fc0b3c20 0x7fb4fc0b6130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb50400bbe0 tx=0x7fb50400b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:38:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.441+0000 7fb510ff9640 1 -- 172.21.15.47:0/844569233 <== mon.2 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb5080cb960 con 0x7fb514108f20 2024-07-27T06:38:48.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:38:48.637+0000 7fb519e32640 1 -- 172.21.15.47:0/844569233 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb514107440 con 0x7fb4fc0b3c20 2024-07-27T06:38:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:48 smithi077 ceph-mon[37153]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:38:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:48 smithi047 ceph-mon[36835]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:38:48.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:48 smithi029 ceph-mon[32610]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:38:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:49 smithi047 ceph-mon[36835]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:38:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:38:49.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:38:49.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:49 smithi077 ceph-mon[37153]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:38:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:38:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:38:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:49.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:49 smithi029 ceph-mon[32610]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:38:49.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:38:49.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:38:49.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:50 smithi077 ceph-mon[37153]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-07-27T06:38:50.942 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:50 smithi047 ceph-mon[36835]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-07-27T06:38:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:50 smithi029 ceph-mon[32610]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-07-27T06:38:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:51 smithi077 ceph-mon[37153]: from='client.? 172.21.15.47:0/3799000634' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd4b585f-e7a0-4d82-873d-3a9269ebac3d"}]: dispatch 2024-07-27T06:38:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:51 smithi077 ceph-mon[37153]: from='client.? 172.21.15.47:0/3799000634' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd4b585f-e7a0-4d82-873d-3a9269ebac3d"}]': finished 2024-07-27T06:38:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:51 smithi077 ceph-mon[37153]: osdmap e29: 5 total, 4 up, 5 in 2024-07-27T06:38:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:38:51.929 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:51 smithi047 ceph-mon[36835]: from='client.? 172.21.15.47:0/3799000634' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd4b585f-e7a0-4d82-873d-3a9269ebac3d"}]: dispatch 2024-07-27T06:38:51.929 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:51 smithi047 ceph-mon[36835]: from='client.? 172.21.15.47:0/3799000634' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd4b585f-e7a0-4d82-873d-3a9269ebac3d"}]': finished 2024-07-27T06:38:51.929 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:51 smithi047 ceph-mon[36835]: osdmap e29: 5 total, 4 up, 5 in 2024-07-27T06:38:51.929 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:38:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:51 smithi029 ceph-mon[32610]: from='client.? 172.21.15.47:0/3799000634' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd4b585f-e7a0-4d82-873d-3a9269ebac3d"}]: dispatch 2024-07-27T06:38:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:51 smithi029 ceph-mon[32610]: from='client.? 172.21.15.47:0/3799000634' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd4b585f-e7a0-4d82-873d-3a9269ebac3d"}]': finished 2024-07-27T06:38:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:51 smithi029 ceph-mon[32610]: osdmap e29: 5 total, 4 up, 5 in 2024-07-27T06:38:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:38:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:52 smithi077 ceph-mon[37153]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-27T06:38:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:52 smithi077 ceph-mon[37153]: from='client.? 172.21.15.47:0/2987574770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:38:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:52 smithi047 ceph-mon[36835]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-27T06:38:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:52 smithi047 ceph-mon[36835]: from='client.? 172.21.15.47:0/2987574770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:38:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:52 smithi029 ceph-mon[32610]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-27T06:38:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:52 smithi029 ceph-mon[32610]: from='client.? 172.21.15.47:0/2987574770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:38:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:54 smithi077 ceph-mon[37153]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-07-27T06:38:54.869 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:54 smithi047 ceph-mon[36835]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-07-27T06:38:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:54 smithi029 ceph-mon[32610]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-07-27T06:38:56.789 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:56 smithi047 ceph-mon[36835]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-27T06:38:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:56 smithi077 ceph-mon[37153]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-27T06:38:56.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:56 smithi029 ceph-mon[32610]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-27T06:38:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:58 smithi077 ceph-mon[37153]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:38:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-27T06:38:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:58 smithi047 ceph-mon[36835]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:38:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-27T06:38:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:58 smithi029 ceph-mon[32610]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:38:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-27T06:38:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:38:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:38:59 smithi077 ceph-mon[37153]: Deploying daemon osd.4 on smithi047 2024-07-27T06:38:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:38:59 smithi047 ceph-mon[36835]: Deploying daemon osd.4 on smithi047 2024-07-27T06:38:59.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:38:59 smithi029 ceph-mon[32610]: Deploying daemon osd.4 on smithi047 2024-07-27T06:39:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:00 smithi077 ceph-mon[37153]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:00.875 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:00 smithi047 ceph-mon[36835]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:00.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:00 smithi029 ceph-mon[32610]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:02 smithi077 ceph-mon[37153]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:02 smithi047 ceph-mon[36835]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:02 smithi029 ceph-mon[32610]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:04 smithi077 ceph-mon[37153]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:04 smithi047 ceph-mon[36835]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:04 smithi029 ceph-mon[32610]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:06 smithi047 ceph-mon[36835]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:06 smithi077 ceph-mon[37153]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:06 smithi029 ceph-mon[32610]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:07.399 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 4 on host 'smithi047' 2024-07-27T06:39:07.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.397+0000 7fb510ff9640 1 -- 172.21.15.47:0/844569233 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb514107440 con 0x7fb4fc0b3c20 2024-07-27T06:39:07.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.400+0000 7fb519e32640 1 -- 172.21.15.47:0/844569233 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb4fc0b3c20 msgr2=0x7fb4fc0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:39:07.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.400+0000 7fb519e32640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb4fc0b3c20 0x7fb4fc0b6130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb50400bbe0 tx=0x7fb50400b390 comp rx=0 tx=0).stop 2024-07-27T06:39:07.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.400+0000 7fb519e32640 1 -- 172.21.15.47:0/844569233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb514108f20 msgr2=0x7fb51419c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:39:07.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.400+0000 7fb519e32640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb514108f20 0x7fb51419c6b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb50802a930 tx=0x7fb508004270 comp rx=0 tx=0).stop 2024-07-27T06:39:07.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.400+0000 7fb519e32640 1 -- 172.21.15.47:0/844569233 shutdown_connections 2024-07-27T06:39:07.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.400+0000 7fb519e32640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb514109870 0x7fb5141a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:07.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.400+0000 7fb519e32640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb514108f20 0x7fb51419c6b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:07.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.400+0000 7fb519e32640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb4fc0b3c20 0x7fb4fc0b6130 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:07.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.400+0000 7fb519e32640 1 --2- 172.21.15.47:0/844569233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb514101ad0 0x7fb51419c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:07.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.400+0000 7fb519e32640 1 -- 172.21.15.47:0/844569233 >> 172.21.15.47:0/844569233 conn(0x7fb514079870 msgr2=0x7fb514106d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:39:07.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.400+0000 7fb519e32640 1 -- 172.21.15.47:0/844569233 shutdown_connections 2024-07-27T06:39:07.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:39:07.400+0000 7fb519e32640 1 -- 172.21.15.47:0/844569233 wait complete. 2024-07-27T06:39:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:08.116 DEBUG:teuthology.orchestra.run.smithi047:osd.4> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@osd.4.service 2024-07-27T06:39:08.118 INFO:tasks.cephadm:Deploying osd.5 on smithi077 with /dev/vg_nvme/lv_3... 2024-07-27T06:39:08.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-27T06:39:08.377 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.c/config 2024-07-27T06:39:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:08 smithi047 ceph-mon[36835]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:08 smithi047 ceph-mon[36835]: from='osd.4 [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-27T06:39:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:08 smithi047 ceph-mon[36835]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-27T06:39:08.761 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:08 smithi077 ceph-mon[37153]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:08.761 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:08 smithi077 ceph-mon[37153]: from='osd.4 [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-27T06:39:08.761 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:08 smithi077 ceph-mon[37153]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-27T06:39:08.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:08 smithi029 ceph-mon[32610]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:08 smithi029 ceph-mon[32610]: from='osd.4 [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-27T06:39:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:08 smithi029 ceph-mon[32610]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-27T06:39:09.831 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:09 smithi047 ceph-mon[36835]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-27T06:39:09.832 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:09 smithi047 ceph-mon[36835]: from='osd.4 [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:39:09.832 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:09 smithi047 ceph-mon[36835]: osdmap e30: 5 total, 4 up, 5 in 2024-07-27T06:39:09.832 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:39:09.832 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:09 smithi047 ceph-mon[36835]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:39:09.832 INFO:journalctl@ceph.osd.4.smithi047.stdout:Jul 27 06:39:09 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-4[49441]: 2024-07-27T06:39:09.620+0000 7f4ae47d7640 -1 osd.4 0 waiting for initial osdmap 2024-07-27T06:39:09.832 INFO:journalctl@ceph.osd.4.smithi047.stdout:Jul 27 06:39:09 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-4[49441]: 2024-07-27T06:39:09.626+0000 7f4ae05d5640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T06:39:09.889 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:09 smithi077 ceph-mon[37153]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-27T06:39:09.890 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:09 smithi077 ceph-mon[37153]: from='osd.4 [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:39:09.890 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:09 smithi077 ceph-mon[37153]: osdmap e30: 5 total, 4 up, 5 in 2024-07-27T06:39:09.890 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:39:09.890 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:09 smithi077 ceph-mon[37153]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:39:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:09 smithi029 ceph-mon[32610]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-27T06:39:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:09 smithi029 ceph-mon[32610]: from='osd.4 [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:39:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:09 smithi029 ceph-mon[32610]: osdmap e30: 5 total, 4 up, 5 in 2024-07-27T06:39:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:39:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:09 smithi029 ceph-mon[32610]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:39:10.331 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-27T06:39:10.411 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-27T06:39:10.424 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-07-27T06:39:10.425 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-07-27T06:39:10.425 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114316 s, 917 MB/s 2024-07-27T06:39:10.425 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-07-27T06:39:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:10 smithi077 ceph-mon[37153]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:10 smithi077 ceph-mon[37153]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-07-27T06:39:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:10 smithi077 ceph-mon[37153]: osdmap e31: 5 total, 4 up, 5 in 2024-07-27T06:39:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:39:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:39:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:10 smithi047 ceph-mon[36835]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:10 smithi047 ceph-mon[36835]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-07-27T06:39:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:10 smithi047 ceph-mon[36835]: osdmap e31: 5 total, 4 up, 5 in 2024-07-27T06:39:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:39:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:39:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:10 smithi029 ceph-mon[32610]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-07-27T06:39:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:10 smithi029 ceph-mon[32610]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-07-27T06:39:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:10 smithi029 ceph-mon[32610]: osdmap e31: 5 total, 4 up, 5 in 2024-07-27T06:39:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:39:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:39:11.132 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_3 2024-07-27T06:39:11.383 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.c/config 2024-07-27T06:39:11.747 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: purged_snaps scrub starts 2024-07-27T06:39:11.747 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: purged_snaps scrub ok 2024-07-27T06:39:11.747 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: osd.4 [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720] boot 2024-07-27T06:39:11.747 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: osdmap e32: 5 total, 5 up, 5 in 2024-07-27T06:39:11.747 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:39:11.747 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: Detected new or changed devices on smithi047 2024-07-27T06:39:11.747 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:11.747 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:11.747 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:39:11.747 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:39:11.747 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 8546M 2024-07-27T06:39:11.748 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:11.748 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:11.748 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:11.748 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: purged_snaps scrub starts 2024-07-27T06:39:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: purged_snaps scrub ok 2024-07-27T06:39:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: osd.4 [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720] boot 2024-07-27T06:39:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: osdmap e32: 5 total, 5 up, 5 in 2024-07-27T06:39:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:39:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: Detected new or changed devices on smithi047 2024-07-27T06:39:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:39:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:39:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 8546M 2024-07-27T06:39:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: purged_snaps scrub starts 2024-07-27T06:39:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: purged_snaps scrub ok 2024-07-27T06:39:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: osd.4 [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720] boot 2024-07-27T06:39:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: osdmap e32: 5 total, 5 up, 5 in 2024-07-27T06:39:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-27T06:39:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: Detected new or changed devices on smithi047 2024-07-27T06:39:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:39:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:39:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 8546M 2024-07-27T06:39:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:12.666 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:12 smithi077 ceph-mon[37153]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:12.666 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:12 smithi077 ceph-mon[37153]: osdmap e33: 5 total, 5 up, 5 in 2024-07-27T06:39:12.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.883+0000 7f36d4e1b640 1 -- 172.21.15.77:0/1738315442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d0108780 msgr2=0x7f36d010efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:39:12.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.883+0000 7f36d4e1b640 1 --2- 172.21.15.77:0/1738315442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d0108780 0x7f36d010efb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009ca0 tx=0x7f36c802a8c0 comp rx=0 tx=0).stop 2024-07-27T06:39:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.883+0000 7f36d4e1b640 1 -- 172.21.15.77:0/1738315442 shutdown_connections 2024-07-27T06:39:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.883+0000 7f36d4e1b640 1 --2- 172.21.15.77:0/1738315442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d0108780 0x7f36d010efb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.883+0000 7f36d4e1b640 1 --2- 172.21.15.77:0/1738315442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36d0107e30 0x7f36d0108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.883+0000 7f36d4e1b640 1 --2- 172.21.15.77:0/1738315442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36d0102bc0 0x7f36d0102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.883+0000 7f36d4e1b640 1 -- 172.21.15.77:0/1738315442 >> 172.21.15.77:0/1738315442 conn(0x7f36d00799c0 msgr2=0x7f36d0079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:39:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.883+0000 7f36d4e1b640 1 -- 172.21.15.77:0/1738315442 shutdown_connections 2024-07-27T06:39:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.883+0000 7f36d4e1b640 1 -- 172.21.15.77:0/1738315442 wait complete. 2024-07-27T06:39:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.884+0000 7f36d4e1b640 1 Processor -- start 2024-07-27T06:39:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.884+0000 7f36d4e1b640 1 -- start start 2024-07-27T06:39:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.884+0000 7f36d4e1b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36d0102bc0 0x7f36d0198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:39:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.884+0000 7f36d4e1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d0107e30 0x7f36d0199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:39:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.884+0000 7f36d4e1b640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36d0108780 0x7f36d019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.884+0000 7f36d4e1b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36d0112450 con 0x7f36d0102bc0 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.884+0000 7f36d4e1b640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f36d01122d0 con 0x7f36d0108780 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.884+0000 7f36d4e1b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f36d01125d0 con 0x7f36d0107e30 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.884+0000 7f36c7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d0107e30 0x7f36d0199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.884+0000 7f36c7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d0107e30 0x7f36d0199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38282/0 (socket says 172.21.15.77:38282) 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.884+0000 7f36c7fff640 1 -- 172.21.15.77:0/729659852 learned_addr learned my addr 172.21.15.77:0/729659852 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.884+0000 7f36ce59c640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36d0102bc0 0x7f36d0198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.885+0000 7f36ced9d640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36d0108780 0x7f36d019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.885+0000 7f36c7fff640 1 -- 172.21.15.77:0/729659852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36d0108780 msgr2=0x7f36d019e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.885+0000 7f36c7fff640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36d0108780 0x7f36d019e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.885+0000 7f36c7fff640 1 -- 172.21.15.77:0/729659852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36d0102bc0 msgr2=0x7f36d0198de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.885+0000 7f36c7fff640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36d0102bc0 0x7f36d0198de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:12.889 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.885+0000 7f36c7fff640 1 -- 172.21.15.77:0/729659852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d019ed50 con 0x7f36d0107e30 2024-07-27T06:39:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.885+0000 7f36c7fff640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d0107e30 0x7f36d0199380 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f36b800d950 tx=0x7f36b800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:39:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.885+0000 7f36c77fe640 1 -- 172.21.15.77:0/729659852 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b8017070 con 0x7f36d0107e30 2024-07-27T06:39:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.885+0000 7f36d4e1b640 1 -- 172.21.15.77:0/729659852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d019f040 con 0x7f36d0107e30 2024-07-27T06:39:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.885+0000 7f36c77fe640 1 -- 172.21.15.77:0/729659852 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36b8009c30 con 0x7f36d0107e30 2024-07-27T06:39:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.885+0000 7f36c77fe640 1 -- 172.21.15.77:0/729659852 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b80047a0 con 0x7f36d0107e30 2024-07-27T06:39:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.885+0000 7f36d4e1b640 1 -- 172.21.15.77:0/729659852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d0101820 con 0x7f36d0107e30 2024-07-27T06:39:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.886+0000 7f36d4e1b640 1 -- 172.21.15.77:0/729659852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d0102fd0 con 0x7f36d0107e30 2024-07-27T06:39:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.887+0000 7f36c77fe640 1 -- 172.21.15.77:0/729659852 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f36b8004020 con 0x7f36d0107e30 2024-07-27T06:39:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.888+0000 7f36c77fe640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f36bc0b3c20 0x7f36bc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:39:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.888+0000 7f36c77fe640 1 -- 172.21.15.77:0/729659852 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4316+0+0 (secure 0 0 0) 0x7f36b80e2d00 con 0x7f36d0107e30 2024-07-27T06:39:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.888+0000 7f36ce59c640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f36bc0b3c20 0x7f36bc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:39:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.889+0000 7f36ce59c640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f36bc0b3c20 0x7f36bc0b6130 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f36c00058c0 tx=0x7f36c000a250 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:39:12.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:12.892+0000 7f36c77fe640 1 -- 172.21.15.77:0/729659852 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f36b8020020 con 0x7f36d0107e30 2024-07-27T06:39:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:12 smithi047 ceph-mon[36835]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:12 smithi047 ceph-mon[36835]: osdmap e33: 5 total, 5 up, 5 in 2024-07-27T06:39:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:12 smithi029 ceph-mon[32610]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:12 smithi029 ceph-mon[32610]: osdmap e33: 5 total, 5 up, 5 in 2024-07-27T06:39:13.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:13.072+0000 7f36d4e1b640 1 -- 172.21.15.77:0/729659852 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f36d01022d0 con 0x7f36bc0b3c20 2024-07-27T06:39:13.831 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:13 smithi077 ceph-mon[37153]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:39:13.831 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:39:13.831 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:39:13.831 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:13 smithi047 ceph-mon[36835]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:39:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:39:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:39:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:13 smithi029 ceph-mon[32610]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:39:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:39:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:39:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:14 smithi077 ceph-mon[37153]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:14 smithi047 ceph-mon[36835]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:14 smithi029 ceph-mon[32610]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:15.945 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:15 smithi077 ceph-mon[37153]: from='client.? 172.21.15.77:0/387632331' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a3eb14-dfaa-4523-8c28-d74c19293c8a"}]: dispatch 2024-07-27T06:39:15.946 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:15 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a3eb14-dfaa-4523-8c28-d74c19293c8a"}]: dispatch 2024-07-27T06:39:15.946 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:15 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40a3eb14-dfaa-4523-8c28-d74c19293c8a"}]': finished 2024-07-27T06:39:15.946 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:15 smithi077 ceph-mon[37153]: osdmap e34: 6 total, 5 up, 6 in 2024-07-27T06:39:15.946 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:15 smithi047 ceph-mon[36835]: from='client.? 172.21.15.77:0/387632331' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a3eb14-dfaa-4523-8c28-d74c19293c8a"}]: dispatch 2024-07-27T06:39:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:15 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a3eb14-dfaa-4523-8c28-d74c19293c8a"}]: dispatch 2024-07-27T06:39:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:15 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40a3eb14-dfaa-4523-8c28-d74c19293c8a"}]': finished 2024-07-27T06:39:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:15 smithi047 ceph-mon[36835]: osdmap e34: 6 total, 5 up, 6 in 2024-07-27T06:39:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:15 smithi029 ceph-mon[32610]: from='client.? 172.21.15.77:0/387632331' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a3eb14-dfaa-4523-8c28-d74c19293c8a"}]: dispatch 2024-07-27T06:39:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:15 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40a3eb14-dfaa-4523-8c28-d74c19293c8a"}]: dispatch 2024-07-27T06:39:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:15 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40a3eb14-dfaa-4523-8c28-d74c19293c8a"}]': finished 2024-07-27T06:39:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:15 smithi029 ceph-mon[32610]: osdmap e34: 6 total, 5 up, 6 in 2024-07-27T06:39:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:16 smithi047 ceph-mon[36835]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:16 smithi047 ceph-mon[36835]: from='client.? 172.21.15.77:0/2467607245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:39:16.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:16 smithi029 ceph-mon[32610]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:16 smithi029 ceph-mon[32610]: from='client.? 172.21.15.77:0/2467607245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:39:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:16 smithi077 ceph-mon[37153]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:16 smithi077 ceph-mon[37153]: from='client.? 172.21.15.77:0/2467607245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:39:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:18 smithi047 ceph-mon[36835]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:18 smithi029 ceph-mon[32610]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:18 smithi077 ceph-mon[37153]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:20.779 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:20 smithi077 ceph-mon[37153]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:20.903 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:20 smithi047 ceph-mon[36835]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:20 smithi029 ceph-mon[32610]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:22 smithi047 ceph-mon[36835]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:22.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:22 smithi029 ceph-mon[32610]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:22 smithi077 ceph-mon[37153]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-27T06:39:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:23 smithi047 ceph-mon[36835]: Deploying daemon osd.5 on smithi077 2024-07-27T06:39:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-27T06:39:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:23 smithi029 ceph-mon[32610]: Deploying daemon osd.5 on smithi077 2024-07-27T06:39:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-27T06:39:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:23 smithi077 ceph-mon[37153]: Deploying daemon osd.5 on smithi077 2024-07-27T06:39:24.715 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:24 smithi077 ceph-mon[37153]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:24 smithi047 ceph-mon[36835]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:24 smithi029 ceph-mon[32610]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:26.765 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:26 smithi077 ceph-mon[37153]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:26 smithi047 ceph-mon[36835]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:26.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:26 smithi029 ceph-mon[32610]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:28.746 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:28 smithi077 ceph-mon[37153]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:28.746 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:28.746 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:28.746 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:28 smithi047 ceph-mon[36835]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:28 smithi029 ceph-mon[32610]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:28.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:30.928 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:30 smithi077 ceph-mon[37153]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:30 smithi047 ceph-mon[36835]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:30 smithi029 ceph-mon[32610]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:31.502 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.501+0000 7f36c77fe640 1 -- 172.21.15.77:0/729659852 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f36d01022d0 con 0x7f36bc0b3c20 2024-07-27T06:39:31.503 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 5 on host 'smithi077' 2024-07-27T06:39:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.503+0000 7f36d4e1b640 1 -- 172.21.15.77:0/729659852 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f36bc0b3c20 msgr2=0x7f36bc0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:39:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.503+0000 7f36d4e1b640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f36bc0b3c20 0x7f36bc0b6130 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f36c00058c0 tx=0x7f36c000a250 comp rx=0 tx=0).stop 2024-07-27T06:39:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.503+0000 7f36d4e1b640 1 -- 172.21.15.77:0/729659852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d0107e30 msgr2=0x7f36d0199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:39:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.503+0000 7f36d4e1b640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d0107e30 0x7f36d0199380 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f36b800d950 tx=0x7f36b800de10 comp rx=0 tx=0).stop 2024-07-27T06:39:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.503+0000 7f36d4e1b640 1 -- 172.21.15.77:0/729659852 shutdown_connections 2024-07-27T06:39:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.503+0000 7f36d4e1b640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f36d0108780 0x7f36d019e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.503+0000 7f36d4e1b640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f36bc0b3c20 0x7f36bc0b6130 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.503+0000 7f36d4e1b640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36d0107e30 0x7f36d0199380 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.503+0000 7f36d4e1b640 1 --2- 172.21.15.77:0/729659852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36d0102bc0 0x7f36d0198de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.503+0000 7f36d4e1b640 1 -- 172.21.15.77:0/729659852 >> 172.21.15.77:0/729659852 conn(0x7f36d00799c0 msgr2=0x7f36d0103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:39:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.503+0000 7f36d4e1b640 1 -- 172.21.15.77:0/729659852 shutdown_connections 2024-07-27T06:39:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:39:31.503+0000 7f36d4e1b640 1 -- 172.21.15.77:0/729659852 wait complete. 2024-07-27T06:39:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:31.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:31.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:31.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:31.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:31.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:32.220 DEBUG:teuthology.orchestra.run.smithi077:osd.5> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@osd.5.service 2024-07-27T06:39:32.222 INFO:tasks.cephadm:Deploying osd.6 on smithi029 with /dev/vg_nvme/lv_2... 2024-07-27T06:39:32.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-27T06:39:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:39:32.897 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:32 smithi029 ceph-mon[32610]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:32.897 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:32 smithi029 ceph-mon[32610]: from='osd.5 [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-27T06:39:32.897 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:32 smithi029 ceph-mon[32610]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-27T06:39:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:32 smithi077 ceph-mon[37153]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:32 smithi077 ceph-mon[37153]: from='osd.5 [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-27T06:39:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:32 smithi077 ceph-mon[37153]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-27T06:39:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:32 smithi047 ceph-mon[36835]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:32 smithi047 ceph-mon[36835]: from='osd.5 [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-27T06:39:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:32 smithi047 ceph-mon[36835]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-27T06:39:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:33 smithi077 ceph-mon[37153]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-27T06:39:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:33 smithi077 ceph-mon[37153]: osdmap e35: 6 total, 5 up, 6 in 2024-07-27T06:39:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:33 smithi077 ceph-mon[37153]: from='osd.5 [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:39:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:33 smithi077 ceph-mon[37153]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:39:33.857 INFO:journalctl@ceph.osd.5.smithi077.stdout:Jul 27 06:39:33 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-5[50019]: 2024-07-27T06:39:33.714+0000 7fc36fdd3640 -1 osd.5 0 waiting for initial osdmap 2024-07-27T06:39:33.858 INFO:journalctl@ceph.osd.5.smithi077.stdout:Jul 27 06:39:33 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-5[50019]: 2024-07-27T06:39:33.725+0000 7fc36bbd1640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T06:39:33.957 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:33 smithi029 ceph-mon[32610]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-27T06:39:33.958 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:33 smithi029 ceph-mon[32610]: osdmap e35: 6 total, 5 up, 6 in 2024-07-27T06:39:33.958 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:33.958 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:33 smithi029 ceph-mon[32610]: from='osd.5 [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:39:33.958 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:33 smithi029 ceph-mon[32610]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:39:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:33 smithi047 ceph-mon[36835]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-27T06:39:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:33 smithi047 ceph-mon[36835]: osdmap e35: 6 total, 5 up, 6 in 2024-07-27T06:39:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:33 smithi047 ceph-mon[36835]: from='osd.5 [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:39:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:33 smithi047 ceph-mon[36835]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:39:34.502 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-27T06:39:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-27T06:39:34.603 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-07-27T06:39:34.603 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-07-27T06:39:34.603 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00866954 s, 1.2 GB/s 2024-07-27T06:39:34.603 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-07-27T06:39:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:34 smithi077 ceph-mon[37153]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:34 smithi077 ceph-mon[37153]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-07-27T06:39:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:34 smithi077 ceph-mon[37153]: osdmap e36: 6 total, 5 up, 6 in 2024-07-27T06:39:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:34 smithi077 ceph-mon[37153]: osd.5 [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780] boot 2024-07-27T06:39:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:34 smithi077 ceph-mon[37153]: osdmap e37: 6 total, 6 up, 6 in 2024-07-27T06:39:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:34 smithi029 ceph-mon[32610]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:34 smithi029 ceph-mon[32610]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-07-27T06:39:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:34 smithi029 ceph-mon[32610]: osdmap e36: 6 total, 5 up, 6 in 2024-07-27T06:39:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:34 smithi029 ceph-mon[32610]: osd.5 [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780] boot 2024-07-27T06:39:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:34 smithi029 ceph-mon[32610]: osdmap e37: 6 total, 6 up, 6 in 2024-07-27T06:39:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:34 smithi047 ceph-mon[36835]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-07-27T06:39:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:34 smithi047 ceph-mon[36835]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-07-27T06:39:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:34 smithi047 ceph-mon[36835]: osdmap e36: 6 total, 5 up, 6 in 2024-07-27T06:39:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:34 smithi047 ceph-mon[36835]: osd.5 [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780] boot 2024-07-27T06:39:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:34 smithi047 ceph-mon[36835]: osdmap e37: 6 total, 6 up, 6 in 2024-07-27T06:39:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-27T06:39:35.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_2 2024-07-27T06:39:35.716 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:39:35.994 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:35 smithi029 ceph-mon[32610]: purged_snaps scrub starts 2024-07-27T06:39:35.994 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:35 smithi029 ceph-mon[32610]: purged_snaps scrub ok 2024-07-27T06:39:35.994 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:35 smithi029 ceph-mon[32610]: Detected new or changed devices on smithi077 2024-07-27T06:39:35.994 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:35.994 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:35.994 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:39:35.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:39:35.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:35 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 8544M 2024-07-27T06:39:35.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:35.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:35.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:35.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:35 smithi077 ceph-mon[37153]: purged_snaps scrub starts 2024-07-27T06:39:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:35 smithi077 ceph-mon[37153]: purged_snaps scrub ok 2024-07-27T06:39:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:35 smithi077 ceph-mon[37153]: Detected new or changed devices on smithi077 2024-07-27T06:39:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:39:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:39:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:35 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 8544M 2024-07-27T06:39:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:35 smithi047 ceph-mon[36835]: purged_snaps scrub starts 2024-07-27T06:39:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:35 smithi047 ceph-mon[36835]: purged_snaps scrub ok 2024-07-27T06:39:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:35 smithi047 ceph-mon[36835]: Detected new or changed devices on smithi077 2024-07-27T06:39:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:39:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:39:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:35 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 8544M 2024-07-27T06:39:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:36.996 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:36 smithi029 ceph-mon[32610]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:36.996 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:36 smithi029 ceph-mon[32610]: osdmap e38: 6 total, 6 up, 6 in 2024-07-27T06:39:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:36 smithi077 ceph-mon[37153]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:36 smithi077 ceph-mon[37153]: osdmap e38: 6 total, 6 up, 6 in 2024-07-27T06:39:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:36 smithi047 ceph-mon[36835]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:36 smithi047 ceph-mon[36835]: osdmap e38: 6 total, 6 up, 6 in 2024-07-27T06:39:37.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.536+0000 7f311fc3f640 1 -- 172.21.15.29:0/3899103194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311810d840 msgr2=0x7f311810fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:39:37.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.536+0000 7f311fc3f640 1 --2- 172.21.15.29:0/3899103194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311810d840 0x7f311810fd50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3110009ca0 tx=0x7f311002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:39:37.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.536+0000 7f311fc3f640 1 -- 172.21.15.29:0/3899103194 shutdown_connections 2024-07-27T06:39:37.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.536+0000 7f311fc3f640 1 --2- 172.21.15.29:0/3899103194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f311810d840 0x7f311810fd50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:37.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.536+0000 7f311fc3f640 1 --2- 172.21.15.29:0/3899103194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3118104cc0 0x7f311810d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:37.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.536+0000 7f311fc3f640 1 --2- 172.21.15.29:0/3899103194 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3118104290 0x7f31181046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:37.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.536+0000 7f311fc3f640 1 -- 172.21.15.29:0/3899103194 >> 172.21.15.29:0/3899103194 conn(0x7f3118079170 msgr2=0x7f3118079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:39:37.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.536+0000 7f311fc3f640 1 -- 172.21.15.29:0/3899103194 shutdown_connections 2024-07-27T06:39:37.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.536+0000 7f311fc3f640 1 -- 172.21.15.29:0/3899103194 wait complete. 2024-07-27T06:39:37.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.537+0000 7f311fc3f640 1 Processor -- start 2024-07-27T06:39:37.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.537+0000 7f311fc3f640 1 -- start start 2024-07-27T06:39:37.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.537+0000 7f311fc3f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3118104290 0x7f3118199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:39:37.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.537+0000 7f311fc3f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3118104cc0 0x7f311819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:39:37.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.537+0000 7f311fc3f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f311810d840 0x7f311819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:39:37.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.537+0000 7f311fc3f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f31181134e0 con 0x7f3118104cc0 2024-07-27T06:39:37.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.537+0000 7f311fc3f640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3118113360 con 0x7f3118104290 2024-07-27T06:39:37.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.537+0000 7f311fc3f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3118113660 con 0x7f311810d840 2024-07-27T06:39:37.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.537+0000 7f311d1da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3118104cc0 0x7f311819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:39:37.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.537+0000 7f311d1da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3118104cc0 0x7f311819a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34380/0 (socket says 172.21.15.29:34380) 2024-07-27T06:39:37.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.537+0000 7f311d1da640 1 -- 172.21.15.29:0/3999461970 learned_addr learned my addr 172.21.15.29:0/3999461970 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:39:37.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f311e1dc640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f311810d840 0x7f311819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:39:37.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f311d9db640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3118104290 0x7f3118199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:39:37.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f311d1da640 1 -- 172.21.15.29:0/3999461970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f311810d840 msgr2=0x7f311819f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:39:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f311d1da640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f311810d840 0x7f311819f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f311d1da640 1 -- 172.21.15.29:0/3999461970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3118104290 msgr2=0x7f3118199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:39:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f311d1da640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3118104290 0x7f3118199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f311d1da640 1 -- 172.21.15.29:0/3999461970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311819fe80 con 0x7f3118104cc0 2024-07-27T06:39:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f311d9db640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3118104290 0x7f3118199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:39:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f311d1da640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3118104cc0 0x7f311819a4b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f310800d750 tx=0x7f310800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:39:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f310effd640 1 -- 172.21.15.29:0/3999461970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108017070 con 0x7f3118104cc0 2024-07-27T06:39:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f311fc3f640 1 -- 172.21.15.29:0/3999461970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31181a0170 con 0x7f3118104cc0 2024-07-27T06:39:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f310effd640 1 -- 172.21.15.29:0/3999461970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31080054d0 con 0x7f3118104cc0 2024-07-27T06:39:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f310effd640 1 -- 172.21.15.29:0/3999461970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108004c30 con 0x7f3118104cc0 2024-07-27T06:39:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.538+0000 7f311fc3f640 1 -- 172.21.15.29:0/3999461970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31181a68d0 con 0x7f3118104cc0 2024-07-27T06:39:37.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.539+0000 7f311fc3f640 1 -- 172.21.15.29:0/3999461970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f311806bc10 con 0x7f3118104cc0 2024-07-27T06:39:37.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.541+0000 7f310effd640 1 -- 172.21.15.29:0/3999461970 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f3108004dd0 con 0x7f3118104cc0 2024-07-27T06:39:37.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.542+0000 7f310effd640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f31040b3c20 0x7f31040b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:39:37.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.542+0000 7f311d9db640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f31040b3c20 0x7f31040b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:39:37.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.542+0000 7f310effd640 1 -- 172.21.15.29:0/3999461970 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4748+0+0 (secure 0 0 0) 0x7f31080e2bc0 con 0x7f3118104cc0 2024-07-27T06:39:37.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.543+0000 7f311d9db640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f31040b3c20 0x7f31040b6130 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3114005d00 tx=0x7f3114005c90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:39:37.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.546+0000 7f310effd640 1 -- 172.21.15.29:0/3999461970 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f31080aafe0 con 0x7f3118104cc0 2024-07-27T06:39:37.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:37.739+0000 7f311fc3f640 1 -- 172.21.15.29:0/3999461970 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f31181006c0 con 0x7f31040b3c20 2024-07-27T06:39:38.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:37 smithi077 ceph-mon[37153]: osdmap e39: 6 total, 6 up, 6 in 2024-07-27T06:39:38.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:37 smithi047 ceph-mon[36835]: osdmap e39: 6 total, 6 up, 6 in 2024-07-27T06:39:38.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:37 smithi029 ceph-mon[32610]: osdmap e39: 6 total, 6 up, 6 in 2024-07-27T06:39:39.021 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:38 smithi077 ceph-mon[37153]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:39.021 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:38 smithi077 ceph-mon[37153]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:39:39.021 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:39:39.022 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:39:39.022 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:38 smithi047 ceph-mon[36835]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:38 smithi047 ceph-mon[36835]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:39:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:39:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:39:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:38 smithi029 ceph-mon[32610]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:38 smithi029 ceph-mon[32610]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:39:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:39:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:39:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:40 smithi029 ceph-mon[32610]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-07-27T06:39:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:40 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2777056409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91e85be2-10d1-4209-aa33-17da8f8bd8bd"}]: dispatch 2024-07-27T06:39:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:40 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2777056409' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91e85be2-10d1-4209-aa33-17da8f8bd8bd"}]': finished 2024-07-27T06:39:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:40 smithi029 ceph-mon[32610]: osdmap e40: 7 total, 6 up, 7 in 2024-07-27T06:39:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:39:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:40 smithi077 ceph-mon[37153]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-07-27T06:39:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:40 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2777056409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91e85be2-10d1-4209-aa33-17da8f8bd8bd"}]: dispatch 2024-07-27T06:39:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:40 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2777056409' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91e85be2-10d1-4209-aa33-17da8f8bd8bd"}]': finished 2024-07-27T06:39:41.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:40 smithi077 ceph-mon[37153]: osdmap e40: 7 total, 6 up, 7 in 2024-07-27T06:39:41.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:39:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:40 smithi047 ceph-mon[36835]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-07-27T06:39:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:40 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2777056409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91e85be2-10d1-4209-aa33-17da8f8bd8bd"}]: dispatch 2024-07-27T06:39:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:40 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2777056409' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91e85be2-10d1-4209-aa33-17da8f8bd8bd"}]': finished 2024-07-27T06:39:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:40 smithi047 ceph-mon[36835]: osdmap e40: 7 total, 6 up, 7 in 2024-07-27T06:39:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:39:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:41 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2463228247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:39:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:41 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2463228247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:39:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:41 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2463228247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:39:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:42 smithi077 ceph-mon[37153]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-27T06:39:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:42 smithi047 ceph-mon[36835]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-27T06:39:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:42 smithi029 ceph-mon[32610]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-27T06:39:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:44 smithi077 ceph-mon[37153]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-07-27T06:39:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:44 smithi047 ceph-mon[36835]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-07-27T06:39:45.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:44 smithi029 ceph-mon[32610]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-07-27T06:39:47.012 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:46 smithi029 ceph-mon[32610]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-07-27T06:39:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:46 smithi077 ceph-mon[37153]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-07-27T06:39:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:46 smithi047 ceph-mon[36835]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-07-27T06:39:48.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:48 smithi029 ceph-mon[32610]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:39:48.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-27T06:39:48.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:48.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:48 smithi029 ceph-mon[32610]: Deploying daemon osd.6 on smithi029 2024-07-27T06:39:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:48 smithi077 ceph-mon[37153]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:39:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-27T06:39:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:48 smithi077 ceph-mon[37153]: Deploying daemon osd.6 on smithi029 2024-07-27T06:39:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:48 smithi047 ceph-mon[36835]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:39:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-27T06:39:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:48 smithi047 ceph-mon[36835]: Deploying daemon osd.6 on smithi029 2024-07-27T06:39:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:50 smithi077 ceph-mon[37153]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:51.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:50 smithi029 ceph-mon[32610]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:50 smithi047 ceph-mon[36835]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:52.861 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:52 smithi029 ceph-mon[32610]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:52 smithi077 ceph-mon[37153]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:52 smithi047 ceph-mon[36835]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:55.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:54 smithi029 ceph-mon[32610]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:55.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:55.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:55.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:54 smithi077 ceph-mon[37153]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:54 smithi047 ceph-mon[36835]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:57.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:56 smithi029 ceph-mon[32610]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:56 smithi077 ceph-mon[37153]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:56 smithi047 ceph-mon[36835]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:58.416 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 6 on host 'smithi029' 2024-07-27T06:39:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.415+0000 7f310effd640 1 -- 172.21.15.29:0/3999461970 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f31181006c0 con 0x7f31040b3c20 2024-07-27T06:39:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.418+0000 7f311fc3f640 1 -- 172.21.15.29:0/3999461970 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f31040b3c20 msgr2=0x7f31040b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:39:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.418+0000 7f311fc3f640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f31040b3c20 0x7f31040b6130 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3114005d00 tx=0x7f3114005c90 comp rx=0 tx=0).stop 2024-07-27T06:39:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.418+0000 7f311fc3f640 1 -- 172.21.15.29:0/3999461970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3118104cc0 msgr2=0x7f311819a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:39:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.418+0000 7f311fc3f640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3118104cc0 0x7f311819a4b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f310800d750 tx=0x7f310800dc10 comp rx=0 tx=0).stop 2024-07-27T06:39:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.418+0000 7f311fc3f640 1 -- 172.21.15.29:0/3999461970 shutdown_connections 2024-07-27T06:39:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.418+0000 7f311fc3f640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f31040b3c20 0x7f31040b6130 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.418+0000 7f311fc3f640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f311810d840 0x7f311819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.418+0000 7f311fc3f640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3118104cc0 0x7f311819a4b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.418+0000 7f311fc3f640 1 --2- 172.21.15.29:0/3999461970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3118104290 0x7f3118199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:39:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.418+0000 7f311fc3f640 1 -- 172.21.15.29:0/3999461970 >> 172.21.15.29:0/3999461970 conn(0x7f3118079170 msgr2=0x7f31180fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:39:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.418+0000 7f311fc3f640 1 -- 172.21.15.29:0/3999461970 shutdown_connections 2024-07-27T06:39:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:39:58.418+0000 7f311fc3f640 1 -- 172.21.15.29:0/3999461970 wait complete. 2024-07-27T06:39:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:58 smithi029 ceph-mon[32610]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:58 smithi029 ceph-mon[32610]: from='osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-27T06:39:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:39:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:58 smithi077 ceph-mon[37153]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:58 smithi077 ceph-mon[37153]: from='osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-27T06:39:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:39:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:58 smithi047 ceph-mon[36835]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:39:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:39:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:39:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:58 smithi047 ceph-mon[36835]: from='osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-27T06:39:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:39:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:39:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:39:59.181 DEBUG:teuthology.orchestra.run.smithi029:osd.6> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@osd.6.service 2024-07-27T06:39:59.183 INFO:tasks.cephadm:Deploying osd.7 on smithi047 with /dev/vg_nvme/lv_2... 2024-07-27T06:39:59.184 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-27T06:39:59.456 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.b/config 2024-07-27T06:40:00.174 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:00 smithi047 ceph-mon[36835]: from='osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-27T06:40:00.174 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:00 smithi047 ceph-mon[36835]: osdmap e41: 7 total, 6 up, 7 in 2024-07-27T06:40:00.174 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:00 smithi047 ceph-mon[36835]: from='osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-07-27T06:40:00.174 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:40:00.174 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:00 smithi047 ceph-mon[36835]: overall HEALTH_OK 2024-07-27T06:40:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:00 smithi077 ceph-mon[37153]: from='osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-27T06:40:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:00 smithi077 ceph-mon[37153]: osdmap e41: 7 total, 6 up, 7 in 2024-07-27T06:40:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:00 smithi077 ceph-mon[37153]: from='osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-07-27T06:40:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:40:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:00 smithi077 ceph-mon[37153]: overall HEALTH_OK 2024-07-27T06:40:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:00 smithi029 ceph-mon[32610]: from='osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-27T06:40:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:00 smithi029 ceph-mon[32610]: osdmap e41: 7 total, 6 up, 7 in 2024-07-27T06:40:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:00 smithi029 ceph-mon[32610]: from='osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-07-27T06:40:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:40:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:00 smithi029 ceph-mon[32610]: overall HEALTH_OK 2024-07-27T06:40:00.463 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jul 27 06:40:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-6[65353]: 2024-07-27T06:40:00.039+0000 7f87f722a640 -1 osd.6 0 waiting for initial osdmap 2024-07-27T06:40:00.463 INFO:journalctl@ceph.osd.6.smithi029.stdout:Jul 27 06:40:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-6[65353]: 2024-07-27T06:40:00.053+0000 7f87f3028640 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T06:40:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:01 smithi047 ceph-mon[36835]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:40:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:01 smithi047 ceph-mon[36835]: from='osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-07-27T06:40:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:01 smithi047 ceph-mon[36835]: osdmap e42: 7 total, 6 up, 7 in 2024-07-27T06:40:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:40:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:40:01.206 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-27T06:40:01.302 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-27T06:40:01.312 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2024-07-27T06:40:01.312 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2024-07-27T06:40:01.312 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00852816 s, 1.2 GB/s 2024-07-27T06:40:01.312 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2024-07-27T06:40:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:01 smithi077 ceph-mon[37153]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:40:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:01 smithi077 ceph-mon[37153]: from='osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-07-27T06:40:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:01 smithi077 ceph-mon[37153]: osdmap e42: 7 total, 6 up, 7 in 2024-07-27T06:40:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:40:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:40:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:01 smithi029 ceph-mon[32610]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-07-27T06:40:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:01 smithi029 ceph-mon[32610]: from='osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-07-27T06:40:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:01 smithi029 ceph-mon[32610]: osdmap e42: 7 total, 6 up, 7 in 2024-07-27T06:40:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:40:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:40:01.953 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch daemon add osd smithi047:vg_nvme/lv_2 2024-07-27T06:40:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:02 smithi047 ceph-mon[36835]: purged_snaps scrub starts 2024-07-27T06:40:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:02 smithi047 ceph-mon[36835]: purged_snaps scrub ok 2024-07-27T06:40:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:02 smithi047 ceph-mon[36835]: osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148] boot 2024-07-27T06:40:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:02 smithi047 ceph-mon[36835]: osdmap e43: 7 total, 7 up, 7 in 2024-07-27T06:40:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:40:02.226 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.b/config 2024-07-27T06:40:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:02 smithi077 ceph-mon[37153]: purged_snaps scrub starts 2024-07-27T06:40:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:02 smithi077 ceph-mon[37153]: purged_snaps scrub ok 2024-07-27T06:40:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:02 smithi077 ceph-mon[37153]: osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148] boot 2024-07-27T06:40:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:02 smithi077 ceph-mon[37153]: osdmap e43: 7 total, 7 up, 7 in 2024-07-27T06:40:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:40:02.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:02 smithi029 ceph-mon[32610]: purged_snaps scrub starts 2024-07-27T06:40:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:02 smithi029 ceph-mon[32610]: purged_snaps scrub ok 2024-07-27T06:40:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:02 smithi029 ceph-mon[32610]: osd.6 [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148] boot 2024-07-27T06:40:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:02 smithi029 ceph-mon[32610]: osdmap e43: 7 total, 7 up, 7 in 2024-07-27T06:40:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-27T06:40:03.313 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:03 smithi047 ceph-mon[36835]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:03.313 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:03 smithi047 ceph-mon[36835]: osdmap e44: 7 total, 7 up, 7 in 2024-07-27T06:40:03.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:03.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:03.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:03.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:03.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:03.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:03.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:03.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:03.314 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:03 smithi077 ceph-mon[37153]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:03 smithi077 ceph-mon[37153]: osdmap e44: 7 total, 7 up, 7 in 2024-07-27T06:40:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:03 smithi029 ceph-mon[32610]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:03 smithi029 ceph-mon[32610]: osdmap e44: 7 total, 7 up, 7 in 2024-07-27T06:40:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:03.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.743+0000 7fc14134b640 1 -- 172.21.15.47:0/752932661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc13c102490 msgr2=0x7fc13c1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:03.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.743+0000 7fc14134b640 1 --2- 172.21.15.47:0/752932661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc13c102490 0x7fc13c1028a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc12c009ca0 tx=0x7fc12c02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:40:03.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.743+0000 7fc14134b640 1 -- 172.21.15.47:0/752932661 shutdown_connections 2024-07-27T06:40:03.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.743+0000 7fc14134b640 1 --2- 172.21.15.47:0/752932661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc13c108720 0x7fc13c10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:03.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.743+0000 7fc14134b640 1 --2- 172.21.15.47:0/752932661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c107dd0 0x7fc13c1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:03.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.743+0000 7fc14134b640 1 --2- 172.21.15.47:0/752932661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc13c102490 0x7fc13c1028a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:03.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.743+0000 7fc14134b640 1 -- 172.21.15.47:0/752932661 >> 172.21.15.47:0/752932661 conn(0x7fc13c079160 msgr2=0x7fc13c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:40:03.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.743+0000 7fc14134b640 1 -- 172.21.15.47:0/752932661 shutdown_connections 2024-07-27T06:40:03.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.743+0000 7fc14134b640 1 -- 172.21.15.47:0/752932661 wait complete. 2024-07-27T06:40:03.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.744+0000 7fc14134b640 1 Processor -- start 2024-07-27T06:40:03.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.744+0000 7fc14134b640 1 -- start start 2024-07-27T06:40:03.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.744+0000 7fc14134b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c102490 0x7fc13c199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:40:03.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.744+0000 7fc14134b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc13c107dd0 0x7fc13c19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:40:03.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.744+0000 7fc14134b640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc13c108720 0x7fc13c19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:40:03.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.744+0000 7fc14134b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc13c1123a0 con 0x7fc13c102490 2024-07-27T06:40:03.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.744+0000 7fc14134b640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fc13c112220 con 0x7fc13c108720 2024-07-27T06:40:03.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.744+0000 7fc14134b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc13c112520 con 0x7fc13c107dd0 2024-07-27T06:40:03.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.744+0000 7fc13b7fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc13c108720 0x7fc13c19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:40:03.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.744+0000 7fc13b7fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc13c108720 0x7fc13c19f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:43332/0 (socket says 172.21.15.47:43332) 2024-07-27T06:40:03.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.744+0000 7fc13b7fe640 1 -- 172.21.15.47:0/703503236 learned_addr learned my addr 172.21.15.47:0/703503236 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-07-27T06:40:03.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.744+0000 7fc13affd640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c102490 0x7fc13c199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:40:03.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc13a7fc640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc13c107dd0 0x7fc13c19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:40:03.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc13b7fe640 1 -- 172.21.15.47:0/703503236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc13c107dd0 msgr2=0x7fc13c19a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:03.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc13b7fe640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc13c107dd0 0x7fc13c19a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:03.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc13b7fe640 1 -- 172.21.15.47:0/703503236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c102490 msgr2=0x7fc13c199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:03.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc13b7fe640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c102490 0x7fc13c199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:03.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc13b7fe640 1 -- 172.21.15.47:0/703503236 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc13c19fe70 con 0x7fc13c108720 2024-07-27T06:40:03.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc13a7fc640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc13c107dd0 0x7fc13c19a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:40:03.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc13affd640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c102490 0x7fc13c199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:40:03.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc13b7fe640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc13c108720 0x7fc13c19f740 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc13400d750 tx=0x7fc13400dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:40:03.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc123fff640 1 -- 172.21.15.47:0/703503236 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134017070 con 0x7fc13c108720 2024-07-27T06:40:03.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc14134b640 1 -- 172.21.15.47:0/703503236 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc13c1a0160 con 0x7fc13c108720 2024-07-27T06:40:03.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc123fff640 1 -- 172.21.15.47:0/703503236 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1340054d0 con 0x7fc13c108720 2024-07-27T06:40:03.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc123fff640 1 -- 172.21.15.47:0/703503236 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134004c60 con 0x7fc13c108720 2024-07-27T06:40:03.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.745+0000 7fc14134b640 1 -- 172.21.15.47:0/703503236 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc13c1a68c0 con 0x7fc13c108720 2024-07-27T06:40:03.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.747+0000 7fc14134b640 1 -- 172.21.15.47:0/703503236 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc13c1028a0 con 0x7fc13c108720 2024-07-27T06:40:03.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.748+0000 7fc123fff640 1 -- 172.21.15.47:0/703503236 <== mon.2 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fc134009bd0 con 0x7fc13c108720 2024-07-27T06:40:03.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.748+0000 7fc123fff640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc1240b3c20 0x7fc1240b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:40:03.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.749+0000 7fc123fff640 1 -- 172.21.15.47:0/703503236 <== mon.2 v2:172.21.15.47:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5180+0+0 (secure 0 0 0) 0x7fc1340e2950 con 0x7fc13c108720 2024-07-27T06:40:03.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.749+0000 7fc13affd640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc1240b3c20 0x7fc1240b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:40:03.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.749+0000 7fc13affd640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc1240b3c20 0x7fc1240b6130 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc12c005e30 tx=0x7fc12c009360 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:40:03.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.753+0000 7fc123fff640 1 -- 172.21.15.47:0/703503236 <== mon.2 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc1340aabe0 con 0x7fc13c108720 2024-07-27T06:40:03.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:03.945+0000 7fc14134b640 1 -- 172.21.15.47:0/703503236 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc13c06a8e0 con 0x7fc1240b3c20 2024-07-27T06:40:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:04 smithi077 ceph-mon[37153]: Detected new or changed devices on smithi029 2024-07-27T06:40:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:04 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 5697M 2024-07-27T06:40:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:40:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:40:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:04.372 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:04 smithi047 ceph-mon[36835]: Detected new or changed devices on smithi029 2024-07-27T06:40:04.372 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:04 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 5697M 2024-07-27T06:40:04.373 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:40:04.373 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:40:04.373 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:04 smithi029 ceph-mon[32610]: Detected new or changed devices on smithi029 2024-07-27T06:40:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:04 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 5697M 2024-07-27T06:40:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:40:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:40:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:05 smithi077 ceph-mon[37153]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:05 smithi077 ceph-mon[37153]: from='client.24416 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:40:05.386 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:05 smithi047 ceph-mon[36835]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:05.387 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:05 smithi047 ceph-mon[36835]: from='client.24416 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:40:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:05 smithi029 ceph-mon[32610]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:05 smithi029 ceph-mon[32610]: from='client.24416 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:40:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:07 smithi077 ceph-mon[37153]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:07 smithi077 ceph-mon[37153]: from='client.? 172.21.15.47:0/31877761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c655b450-0cc4-45d4-95ea-93a57fe8e4ff"}]: dispatch 2024-07-27T06:40:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:07 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c655b450-0cc4-45d4-95ea-93a57fe8e4ff"}]: dispatch 2024-07-27T06:40:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:07 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c655b450-0cc4-45d4-95ea-93a57fe8e4ff"}]': finished 2024-07-27T06:40:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:07 smithi077 ceph-mon[37153]: osdmap e45: 8 total, 7 up, 8 in 2024-07-27T06:40:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:07 smithi077 ceph-mon[37153]: from='client.? 172.21.15.47:0/3916696469' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:40:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:07 smithi047 ceph-mon[36835]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:07 smithi047 ceph-mon[36835]: from='client.? 172.21.15.47:0/31877761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c655b450-0cc4-45d4-95ea-93a57fe8e4ff"}]: dispatch 2024-07-27T06:40:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:07 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c655b450-0cc4-45d4-95ea-93a57fe8e4ff"}]: dispatch 2024-07-27T06:40:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:07 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c655b450-0cc4-45d4-95ea-93a57fe8e4ff"}]': finished 2024-07-27T06:40:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:07 smithi047 ceph-mon[36835]: osdmap e45: 8 total, 7 up, 8 in 2024-07-27T06:40:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:07 smithi047 ceph-mon[36835]: from='client.? 172.21.15.47:0/3916696469' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:40:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:07 smithi029 ceph-mon[32610]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:07 smithi029 ceph-mon[32610]: from='client.? 172.21.15.47:0/31877761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c655b450-0cc4-45d4-95ea-93a57fe8e4ff"}]: dispatch 2024-07-27T06:40:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:07 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c655b450-0cc4-45d4-95ea-93a57fe8e4ff"}]: dispatch 2024-07-27T06:40:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:07 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c655b450-0cc4-45d4-95ea-93a57fe8e4ff"}]': finished 2024-07-27T06:40:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:07 smithi029 ceph-mon[32610]: osdmap e45: 8 total, 7 up, 8 in 2024-07-27T06:40:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:07 smithi029 ceph-mon[32610]: from='client.? 172.21.15.47:0/3916696469' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:40:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:09 smithi077 ceph-mon[37153]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:09 smithi047 ceph-mon[36835]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:09.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:09 smithi029 ceph-mon[32610]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:11 smithi077 ceph-mon[37153]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:11.391 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:11 smithi047 ceph-mon[36835]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:11 smithi029 ceph-mon[32610]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:13 smithi047 ceph-mon[36835]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:13 smithi077 ceph-mon[37153]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:13 smithi029 ceph-mon[32610]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:14.339 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-27T06:40:14.339 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-27T06:40:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-27T06:40:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:15 smithi077 ceph-mon[37153]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:15 smithi077 ceph-mon[37153]: Deploying daemon osd.7 on smithi047 2024-07-27T06:40:15.363 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:15 smithi047 ceph-mon[36835]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:15.363 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:15 smithi047 ceph-mon[36835]: Deploying daemon osd.7 on smithi047 2024-07-27T06:40:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:15 smithi029 ceph-mon[32610]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:15 smithi029 ceph-mon[32610]: Deploying daemon osd.7 on smithi047 2024-07-27T06:40:17.321 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:17 smithi047 ceph-mon[36835]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:17 smithi077 ceph-mon[37153]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:17 smithi029 ceph-mon[32610]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:19 smithi077 ceph-mon[37153]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:40:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:19 smithi047 ceph-mon[36835]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:40:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:19 smithi029 ceph-mon[32610]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:40:20.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:20.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:20.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:20.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:21.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:21 smithi077 ceph-mon[37153]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:21.445 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:21 smithi047 ceph-mon[36835]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:21.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:21 smithi029 ceph-mon[32610]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:22.839 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 7 on host 'smithi047' 2024-07-27T06:40:22.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.837+0000 7fc123fff640 1 -- 172.21.15.47:0/703503236 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc13c06a8e0 con 0x7fc1240b3c20 2024-07-27T06:40:22.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.840+0000 7fc14134b640 1 -- 172.21.15.47:0/703503236 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc1240b3c20 msgr2=0x7fc1240b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:22.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.840+0000 7fc14134b640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc1240b3c20 0x7fc1240b6130 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc12c005e30 tx=0x7fc12c009360 comp rx=0 tx=0).stop 2024-07-27T06:40:22.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.840+0000 7fc14134b640 1 -- 172.21.15.47:0/703503236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc13c108720 msgr2=0x7fc13c19f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:22.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.840+0000 7fc14134b640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc13c108720 0x7fc13c19f740 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc13400d750 tx=0x7fc13400dc10 comp rx=0 tx=0).stop 2024-07-27T06:40:22.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.840+0000 7fc14134b640 1 -- 172.21.15.47:0/703503236 shutdown_connections 2024-07-27T06:40:22.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.840+0000 7fc14134b640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc13c108720 0x7fc13c19f740 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:22.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.840+0000 7fc14134b640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc1240b3c20 0x7fc1240b6130 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:22.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.840+0000 7fc14134b640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc13c107dd0 0x7fc13c19a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:22.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.840+0000 7fc14134b640 1 --2- 172.21.15.47:0/703503236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c102490 0x7fc13c199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:22.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.840+0000 7fc14134b640 1 -- 172.21.15.47:0/703503236 >> 172.21.15.47:0/703503236 conn(0x7fc13c079160 msgr2=0x7fc13c1014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:40:22.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.841+0000 7fc14134b640 1 -- 172.21.15.47:0/703503236 shutdown_connections 2024-07-27T06:40:22.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:40:22.841+0000 7fc14134b640 1 -- 172.21.15.47:0/703503236 wait complete. 2024-07-27T06:40:23.177 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:22 smithi047 ceph-mon[36835]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:23.177 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.177 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.177 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:23.177 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:23.177 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.178 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:40:23.178 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.178 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:22 smithi029 ceph-mon[32610]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:40:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:22 smithi077 ceph-mon[37153]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:40:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:23.526 DEBUG:teuthology.orchestra.run.smithi047:osd.7> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@osd.7.service 2024-07-27T06:40:23.529 INFO:tasks.cephadm:Deploying osd.8 on smithi077 with /dev/vg_nvme/lv_2... 2024-07-27T06:40:23.529 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-27T06:40:23.787 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.c/config 2024-07-27T06:40:24.166 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:23 smithi077 ceph-mon[37153]: from='osd.7 [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-27T06:40:24.166 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:23 smithi077 ceph-mon[37153]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-27T06:40:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:23 smithi047 ceph-mon[36835]: from='osd.7 [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-27T06:40:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:23 smithi047 ceph-mon[36835]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-27T06:40:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:23 smithi029 ceph-mon[32610]: from='osd.7 [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-27T06:40:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:23 smithi029 ceph-mon[32610]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-27T06:40:25.139 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:24 smithi077 ceph-mon[37153]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:25.139 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:24 smithi077 ceph-mon[37153]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-27T06:40:25.139 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:24 smithi077 ceph-mon[37153]: osdmap e46: 8 total, 7 up, 8 in 2024-07-27T06:40:25.139 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:24 smithi077 ceph-mon[37153]: from='osd.7 [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:40:25.139 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:25.139 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:24 smithi077 ceph-mon[37153]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:24 smithi029 ceph-mon[32610]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:24 smithi029 ceph-mon[32610]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-27T06:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:24 smithi029 ceph-mon[32610]: osdmap e46: 8 total, 7 up, 8 in 2024-07-27T06:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:24 smithi029 ceph-mon[32610]: from='osd.7 [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:24 smithi029 ceph-mon[32610]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:40:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:24 smithi047 ceph-mon[36835]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:24 smithi047 ceph-mon[36835]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-27T06:40:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:24 smithi047 ceph-mon[36835]: osdmap e46: 8 total, 7 up, 8 in 2024-07-27T06:40:25.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:24 smithi047 ceph-mon[36835]: from='osd.7 [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:40:25.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:25.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:24 smithi047 ceph-mon[36835]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:40:25.625 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-27T06:40:25.700 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-27T06:40:25.714 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-07-27T06:40:25.714 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-07-27T06:40:25.714 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115038 s, 912 MB/s 2024-07-27T06:40:25.714 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-07-27T06:40:25.773 INFO:journalctl@ceph.osd.7.smithi047.stdout:Jul 27 06:40:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-7[56121]: 2024-07-27T06:40:25.463+0000 7f3d6c810640 -1 osd.7 0 waiting for initial osdmap 2024-07-27T06:40:25.773 INFO:journalctl@ceph.osd.7.smithi047.stdout:Jul 27 06:40:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-7[56121]: 2024-07-27T06:40:25.474+0000 7f3d64606640 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T06:40:26.101 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:25 smithi047 ceph-mon[36835]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-07-27T06:40:26.101 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:25 smithi047 ceph-mon[36835]: osdmap e47: 8 total, 7 up, 8 in 2024-07-27T06:40:26.101 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:26.101 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:26.101 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:25 smithi047 ceph-mon[36835]: from='osd.7 ' entity='osd.7' 2024-07-27T06:40:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:25 smithi077 ceph-mon[37153]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-07-27T06:40:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:25 smithi077 ceph-mon[37153]: osdmap e47: 8 total, 7 up, 8 in 2024-07-27T06:40:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:25 smithi077 ceph-mon[37153]: from='osd.7 ' entity='osd.7' 2024-07-27T06:40:26.441 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_2 2024-07-27T06:40:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:25 smithi029 ceph-mon[32610]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-07-27T06:40:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:25 smithi029 ceph-mon[32610]: osdmap e47: 8 total, 7 up, 8 in 2024-07-27T06:40:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:25 smithi029 ceph-mon[32610]: from='osd.7 ' entity='osd.7' 2024-07-27T06:40:26.700 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.c/config 2024-07-27T06:40:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: purged_snaps scrub starts 2024-07-27T06:40:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: purged_snaps scrub ok 2024-07-27T06:40:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: osd.7 [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589] boot 2024-07-27T06:40:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: osdmap e48: 8 total, 8 up, 8 in 2024-07-27T06:40:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:27.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:27.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:27.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:27.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:27.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:27.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:27.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:27.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:27.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: purged_snaps scrub starts 2024-07-27T06:40:27.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: purged_snaps scrub ok 2024-07-27T06:40:27.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:27.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:27.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: osd.7 [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589] boot 2024-07-27T06:40:27.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: osdmap e48: 8 total, 8 up, 8 in 2024-07-27T06:40:27.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:27.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:27.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:27.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:27.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:27.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:27.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:27.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:27.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:27.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: purged_snaps scrub starts 2024-07-27T06:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: purged_snaps scrub ok 2024-07-27T06:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-07-27T06:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: osd.7 [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589] boot 2024-07-27T06:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: osdmap e48: 8 total, 8 up, 8 in 2024-07-27T06:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-27T06:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:27.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:27.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:27.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:27.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:27.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:27.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:27.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:27 smithi077 ceph-mon[37153]: Detected new or changed devices on smithi047 2024-07-27T06:40:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:27 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 5697M 2024-07-27T06:40:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:27 smithi077 ceph-mon[37153]: osdmap e49: 8 total, 8 up, 8 in 2024-07-27T06:40:28.348 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.338+0000 7f73a80e2640 1 -- 172.21.15.77:0/3189173414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a0102b20 msgr2=0x7f73a0107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.338+0000 7f73a80e2640 1 --2- 172.21.15.77:0/3189173414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a0102b20 0x7f73a0107050 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7390009900 tx=0x7f739002a410 comp rx=0 tx=0).stop 2024-07-27T06:40:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.339+0000 7f73a80e2640 1 -- 172.21.15.77:0/3189173414 shutdown_connections 2024-07-27T06:40:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.339+0000 7f73a80e2640 1 --2- 172.21.15.77:0/3189173414 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a0107610 0x7f73a0107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.339+0000 7f73a80e2640 1 --2- 172.21.15.77:0/3189173414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a0102b20 0x7f73a0107050 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.339+0000 7f73a80e2640 1 --2- 172.21.15.77:0/3189173414 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73a01021d0 0x7f73a01025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.339+0000 7f73a80e2640 1 -- 172.21.15.77:0/3189173414 >> 172.21.15.77:0/3189173414 conn(0x7f73a0079150 msgr2=0x7f73a0079560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:40:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.339+0000 7f73a80e2640 1 -- 172.21.15.77:0/3189173414 shutdown_connections 2024-07-27T06:40:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.339+0000 7f73a80e2640 1 -- 172.21.15.77:0/3189173414 wait complete. 2024-07-27T06:40:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.339+0000 7f73a80e2640 1 Processor -- start 2024-07-27T06:40:28.349 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.339+0000 7f73a80e2640 1 -- start start 2024-07-27T06:40:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a80e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a01021d0 0x7f73a01a0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:40:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a80e2640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73a0102b20 0x7f73a01a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:40:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a80e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a0107610 0x7f73a01a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:40:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a80e2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73a0119b00 con 0x7f73a01021d0 2024-07-27T06:40:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a80e2640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f73a0119980 con 0x7f73a0102b20 2024-07-27T06:40:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a80e2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f73a0119c80 con 0x7f73a0107610 2024-07-27T06:40:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a667f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a0107610 0x7f73a01a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:40:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a667f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a0107610 0x7f73a01a5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39808/0 (socket says 172.21.15.77:39808) 2024-07-27T06:40:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a667f640 1 -- 172.21.15.77:0/2876729946 learned_addr learned my addr 172.21.15.77:0/2876729946 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-07-27T06:40:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a5e7e640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a01021d0 0x7f73a01a0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:40:28.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a567d640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73a0102b20 0x7f73a01a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a667f640 1 -- 172.21.15.77:0/2876729946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73a0102b20 msgr2=0x7f73a01a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a667f640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73a0102b20 0x7f73a01a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a667f640 1 -- 172.21.15.77:0/2876729946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a01021d0 msgr2=0x7f73a01a0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a667f640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a01021d0 0x7f73a01a0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a667f640 1 -- 172.21.15.77:0/2876729946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a01a64c0 con 0x7f73a0107610 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a667f640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a0107610 0x7f73a01a5d90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f739800eaa0 tx=0x7f739800ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f7396ffd640 1 -- 172.21.15.77:0/2876729946 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739800cdb0 con 0x7f73a0107610 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f7396ffd640 1 -- 172.21.15.77:0/2876729946 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7398005020 con 0x7f73a0107610 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f7396ffd640 1 -- 172.21.15.77:0/2876729946 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73980116d0 con 0x7f73a0107610 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a80e2640 1 -- 172.21.15.77:0/2876729946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a01a67b0 con 0x7f73a0107610 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.340+0000 7f73a80e2640 1 -- 172.21.15.77:0/2876729946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a01acf10 con 0x7f73a0107610 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.341+0000 7f73a80e2640 1 -- 172.21.15.77:0/2876729946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73a006bcb0 con 0x7f73a0107610 2024-07-27T06:40:28.351 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.343+0000 7f7396ffd640 1 -- 172.21.15.77:0/2876729946 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f73980051d0 con 0x7f73a0107610 2024-07-27T06:40:28.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.344+0000 7f7396ffd640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f738c0b3c20 0x7f738c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:40:28.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.344+0000 7f7396ffd640 1 -- 172.21.15.77:0/2876729946 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5612+0+0 (secure 0 0 0) 0x7f73980e36e0 con 0x7f73a0107610 2024-07-27T06:40:28.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.344+0000 7f73a5e7e640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f738c0b3c20 0x7f738c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:40:28.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.345+0000 7f73a5e7e640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f738c0b3c20 0x7f738c0b6130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f73a006c3b0 tx=0x7f739c00b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:40:28.352 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.347+0000 7f7396ffd640 1 -- 172.21.15.77:0/2876729946 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7398016030 con 0x7f73a0107610 2024-07-27T06:40:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:27 smithi047 ceph-mon[36835]: Detected new or changed devices on smithi047 2024-07-27T06:40:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:27 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 5697M 2024-07-27T06:40:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:27 smithi047 ceph-mon[36835]: osdmap e49: 8 total, 8 up, 8 in 2024-07-27T06:40:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:27 smithi029 ceph-mon[32610]: Detected new or changed devices on smithi047 2024-07-27T06:40:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:27 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 5697M 2024-07-27T06:40:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:27 smithi029 ceph-mon[32610]: osdmap e49: 8 total, 8 up, 8 in 2024-07-27T06:40:28.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:28.527+0000 7f73a80e2640 1 -- 172.21.15.77:0/2876729946 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f73a0106740 con 0x7f738c0b3c20 2024-07-27T06:40:29.194 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:28 smithi077 ceph-mon[37153]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:29.194 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:28 smithi077 ceph-mon[37153]: osdmap e50: 8 total, 8 up, 8 in 2024-07-27T06:40:29.194 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:40:29.194 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:40:29.194 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:28 smithi047 ceph-mon[36835]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:28 smithi047 ceph-mon[36835]: osdmap e50: 8 total, 8 up, 8 in 2024-07-27T06:40:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:40:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:40:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:28 smithi029 ceph-mon[32610]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:28 smithi029 ceph-mon[32610]: osdmap e50: 8 total, 8 up, 8 in 2024-07-27T06:40:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:40:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:40:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:30.265 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:29 smithi077 ceph-mon[37153]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:40:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:29 smithi047 ceph-mon[36835]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:40:30.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:29 smithi029 ceph-mon[32610]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:40:31.262 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:30 smithi077 ceph-mon[37153]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:31.262 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:30 smithi077 ceph-mon[37153]: from='client.? 172.21.15.77:0/3721460686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7a95b51-f32c-433e-a197-feb6d47b6c71"}]: dispatch 2024-07-27T06:40:31.262 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:30 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7a95b51-f32c-433e-a197-feb6d47b6c71"}]: dispatch 2024-07-27T06:40:31.262 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:30 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7a95b51-f32c-433e-a197-feb6d47b6c71"}]': finished 2024-07-27T06:40:31.262 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:30 smithi077 ceph-mon[37153]: osdmap e51: 9 total, 8 up, 9 in 2024-07-27T06:40:31.262 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:30 smithi047 ceph-mon[36835]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:30 smithi047 ceph-mon[36835]: from='client.? 172.21.15.77:0/3721460686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7a95b51-f32c-433e-a197-feb6d47b6c71"}]: dispatch 2024-07-27T06:40:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:30 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7a95b51-f32c-433e-a197-feb6d47b6c71"}]: dispatch 2024-07-27T06:40:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:30 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7a95b51-f32c-433e-a197-feb6d47b6c71"}]': finished 2024-07-27T06:40:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:30 smithi047 ceph-mon[36835]: osdmap e51: 9 total, 8 up, 9 in 2024-07-27T06:40:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:31.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:30 smithi029 ceph-mon[32610]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:30 smithi029 ceph-mon[32610]: from='client.? 172.21.15.77:0/3721460686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7a95b51-f32c-433e-a197-feb6d47b6c71"}]: dispatch 2024-07-27T06:40:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:30 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7a95b51-f32c-433e-a197-feb6d47b6c71"}]: dispatch 2024-07-27T06:40:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:30 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7a95b51-f32c-433e-a197-feb6d47b6c71"}]': finished 2024-07-27T06:40:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:30 smithi029 ceph-mon[32610]: osdmap e51: 9 total, 8 up, 9 in 2024-07-27T06:40:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:31 smithi077 ceph-mon[37153]: from='client.? 172.21.15.77:0/3836097347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:40:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:31 smithi047 ceph-mon[36835]: from='client.? 172.21.15.77:0/3836097347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:40:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:31 smithi029 ceph-mon[32610]: from='client.? 172.21.15.77:0/3836097347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:40:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:32 smithi077 ceph-mon[37153]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:32 smithi047 ceph-mon[36835]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:33.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:32 smithi029 ceph-mon[32610]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:34 smithi077 ceph-mon[37153]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-07-27T06:40:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:34 smithi047 ceph-mon[36835]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-07-27T06:40:35.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:34 smithi029 ceph-mon[32610]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-07-27T06:40:37.235 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:36 smithi077 ceph-mon[37153]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-27T06:40:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:36 smithi047 ceph-mon[36835]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-27T06:40:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:36 smithi029 ceph-mon[32610]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-27T06:40:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:39 smithi077 ceph-mon[37153]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-07-27T06:40:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-27T06:40:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:39 smithi077 ceph-mon[37153]: Deploying daemon osd.8 on smithi077 2024-07-27T06:40:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:39 smithi047 ceph-mon[36835]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-07-27T06:40:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-27T06:40:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:39 smithi047 ceph-mon[36835]: Deploying daemon osd.8 on smithi077 2024-07-27T06:40:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:39 smithi029 ceph-mon[32610]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-07-27T06:40:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-27T06:40:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:39 smithi029 ceph-mon[32610]: Deploying daemon osd.8 on smithi077 2024-07-27T06:40:41.315 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:41 smithi077 ceph-mon[37153]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:40:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:41 smithi047 ceph-mon[36835]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:40:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:41 smithi029 ceph-mon[32610]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:40:43.328 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:43 smithi077 ceph-mon[37153]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-07-27T06:40:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:43 smithi047 ceph-mon[36835]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-07-27T06:40:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:43 smithi029 ceph-mon[32610]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-07-27T06:40:44.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:40:44.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:44.340 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:40:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:40:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:45.344 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:45 smithi077 ceph-mon[37153]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-07-27T06:40:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:45 smithi047 ceph-mon[36835]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-07-27T06:40:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:45 smithi029 ceph-mon[32610]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-07-27T06:40:47.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:47 smithi077 ceph-mon[37153]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:47.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.103+0000 7f7396ffd640 1 -- 172.21.15.77:0/2876729946 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f73a0106740 con 0x7f738c0b3c20 2024-07-27T06:40:47.104 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 8 on host 'smithi077' 2024-07-27T06:40:47.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.106+0000 7f73a80e2640 1 -- 172.21.15.77:0/2876729946 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f738c0b3c20 msgr2=0x7f738c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:47.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.106+0000 7f73a80e2640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f738c0b3c20 0x7f738c0b6130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f73a006c3b0 tx=0x7f739c00b390 comp rx=0 tx=0).stop 2024-07-27T06:40:47.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.106+0000 7f73a80e2640 1 -- 172.21.15.77:0/2876729946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a0107610 msgr2=0x7f73a01a5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:47.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.106+0000 7f73a80e2640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a0107610 0x7f73a01a5d90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f739800eaa0 tx=0x7f739800ef60 comp rx=0 tx=0).stop 2024-07-27T06:40:47.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.106+0000 7f73a80e2640 1 -- 172.21.15.77:0/2876729946 shutdown_connections 2024-07-27T06:40:47.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.106+0000 7f73a80e2640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f738c0b3c20 0x7f738c0b6130 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:47.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.106+0000 7f73a80e2640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73a0107610 0x7f73a01a5d90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:47.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.106+0000 7f73a80e2640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73a0102b20 0x7f73a01a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:47.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.106+0000 7f73a80e2640 1 --2- 172.21.15.77:0/2876729946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a01021d0 0x7f73a01a0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:47.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.106+0000 7f73a80e2640 1 -- 172.21.15.77:0/2876729946 >> 172.21.15.77:0/2876729946 conn(0x7f73a0079150 msgr2=0x7f73a0105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:40:47.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.106+0000 7f73a80e2640 1 -- 172.21.15.77:0/2876729946 shutdown_connections 2024-07-27T06:40:47.108 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:40:47.106+0000 7f73a80e2640 1 -- 172.21.15.77:0/2876729946 wait complete. 2024-07-27T06:40:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:47 smithi047 ceph-mon[36835]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:47.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:47 smithi029 ceph-mon[32610]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:47.812 DEBUG:teuthology.orchestra.run.smithi077:osd.8> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@osd.8.service 2024-07-27T06:40:47.814 INFO:tasks.cephadm:Deploying osd.9 on smithi029 with /dev/vg_nvme/lv_1... 2024-07-27T06:40:47.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-27T06:40:48.096 INFO:journalctl@ceph.osd.8.smithi077.stdout:Jul 27 06:40:47 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-8[56948]: 2024-07-27T06:40:47.856+0000 7f2296103640 -1 osd.8 0 log_to_monitors true 2024-07-27T06:40:48.100 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:40:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:48 smithi077 ceph-mon[37153]: from='osd.8 [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-27T06:40:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:48 smithi077 ceph-mon[37153]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-27T06:40:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:48.371 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:48.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:48.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:48 smithi029 ceph-mon[32610]: from='osd.8 [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-27T06:40:48.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:48 smithi029 ceph-mon[32610]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-27T06:40:48.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:48.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:48 smithi047 ceph-mon[36835]: from='osd.8 [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-27T06:40:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:48 smithi047 ceph-mon[36835]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-27T06:40:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:48.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:49.382 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:49 smithi077 ceph-mon[37153]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:49.382 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:49 smithi077 ceph-mon[37153]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-27T06:40:49.382 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:49 smithi077 ceph-mon[37153]: osdmap e52: 9 total, 8 up, 9 in 2024-07-27T06:40:49.382 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:49.382 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:49 smithi077 ceph-mon[37153]: from='osd.8 [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:40:49.382 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:49 smithi077 ceph-mon[37153]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:40:49.382 INFO:journalctl@ceph.osd.8.smithi077.stdout:Jul 27 06:40:49 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-8[56948]: 2024-07-27T06:40:49.116+0000 7f2292085640 -1 osd.8 0 waiting for initial osdmap 2024-07-27T06:40:49.383 INFO:journalctl@ceph.osd.8.smithi077.stdout:Jul 27 06:40:49 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-8[56948]: 2024-07-27T06:40:49.126+0000 7f228de83640 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T06:40:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:49 smithi047 ceph-mon[36835]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:49 smithi047 ceph-mon[36835]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-27T06:40:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:49 smithi047 ceph-mon[36835]: osdmap e52: 9 total, 8 up, 9 in 2024-07-27T06:40:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:49 smithi047 ceph-mon[36835]: from='osd.8 [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:40:49.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:49 smithi047 ceph-mon[36835]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:40:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:49 smithi029 ceph-mon[32610]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:49.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:49 smithi029 ceph-mon[32610]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-27T06:40:49.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:49 smithi029 ceph-mon[32610]: osdmap e52: 9 total, 8 up, 9 in 2024-07-27T06:40:49.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:49.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:49 smithi029 ceph-mon[32610]: from='osd.8 [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:40:49.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:49 smithi029 ceph-mon[32610]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:40:50.248 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-27T06:40:50.335 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-27T06:40:50.346 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-07-27T06:40:50.346 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-07-27T06:40:50.346 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00876257 s, 1.2 GB/s 2024-07-27T06:40:50.346 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:50 smithi029 ceph-mon[32610]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-07-27T06:40:50.346 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:50 smithi029 ceph-mon[32610]: osdmap e53: 9 total, 8 up, 9 in 2024-07-27T06:40:50.346 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:50.346 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:50.347 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-07-27T06:40:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:50 smithi047 ceph-mon[36835]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-07-27T06:40:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:50 smithi047 ceph-mon[36835]: osdmap e53: 9 total, 8 up, 9 in 2024-07-27T06:40:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:50 smithi077 ceph-mon[37153]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-07-27T06:40:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:50 smithi077 ceph-mon[37153]: osdmap e53: 9 total, 8 up, 9 in 2024-07-27T06:40:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:51.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_1 2024-07-27T06:40:51.361 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:40:51.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:51 smithi029 ceph-mon[32610]: purged_snaps scrub starts 2024-07-27T06:40:51.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:51 smithi029 ceph-mon[32610]: purged_snaps scrub ok 2024-07-27T06:40:51.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:51 smithi029 ceph-mon[32610]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:51.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:51 smithi029 ceph-mon[32610]: osd.8 [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905] boot 2024-07-27T06:40:51.402 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:51 smithi029 ceph-mon[32610]: osdmap e54: 9 total, 9 up, 9 in 2024-07-27T06:40:51.402 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:51 smithi047 ceph-mon[36835]: purged_snaps scrub starts 2024-07-27T06:40:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:51 smithi047 ceph-mon[36835]: purged_snaps scrub ok 2024-07-27T06:40:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:51 smithi047 ceph-mon[36835]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:51 smithi047 ceph-mon[36835]: osd.8 [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905] boot 2024-07-27T06:40:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:51 smithi047 ceph-mon[36835]: osdmap e54: 9 total, 9 up, 9 in 2024-07-27T06:40:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:51 smithi077 ceph-mon[37153]: purged_snaps scrub starts 2024-07-27T06:40:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:51 smithi077 ceph-mon[37153]: purged_snaps scrub ok 2024-07-27T06:40:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:51 smithi077 ceph-mon[37153]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-07-27T06:40:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:51 smithi077 ceph-mon[37153]: osd.8 [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905] boot 2024-07-27T06:40:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:51 smithi077 ceph-mon[37153]: osdmap e54: 9 total, 9 up, 9 in 2024-07-27T06:40:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-27T06:40:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: osdmap e55: 9 total, 9 up, 9 in 2024-07-27T06:40:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: Detected new or changed devices on smithi077 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 5696M 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:52.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:52.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: osdmap e55: 9 total, 9 up, 9 in 2024-07-27T06:40:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: Detected new or changed devices on smithi077 2024-07-27T06:40:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:52.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 5696M 2024-07-27T06:40:52.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:52.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:52.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:40:52.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:52.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:52.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: osdmap e55: 9 total, 9 up, 9 in 2024-07-27T06:40:52.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: Detected new or changed devices on smithi077 2024-07-27T06:40:52.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:52.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:40:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 5696M 2024-07-27T06:40:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:40:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:40:52.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:40:53.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:53 smithi029 ceph-mon[32610]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:40:53.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:53 smithi029 ceph-mon[32610]: osdmap e56: 9 total, 9 up, 9 in 2024-07-27T06:40:53.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.181+0000 7f637b87a640 1 -- 172.21.15.29:0/2953975983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f637410d860 msgr2=0x7f637410fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:53.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.181+0000 7f637b87a640 1 --2- 172.21.15.29:0/2953975983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f637410d860 0x7f637410fd30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f636c009ca0 tx=0x7f636c02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:40:53.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.181+0000 7f637b87a640 1 -- 172.21.15.29:0/2953975983 shutdown_connections 2024-07-27T06:40:53.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.181+0000 7f637b87a640 1 --2- 172.21.15.29:0/2953975983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f637410d860 0x7f637410fd30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:53.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.181+0000 7f637b87a640 1 --2- 172.21.15.29:0/2953975983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6374104ce0 0x7f637410d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:53.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.181+0000 7f637b87a640 1 --2- 172.21.15.29:0/2953975983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63741042b0 0x7f63741046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:53.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.181+0000 7f637b87a640 1 -- 172.21.15.29:0/2953975983 >> 172.21.15.29:0/2953975983 conn(0x7f6374079160 msgr2=0x7f6374079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:40:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.181+0000 7f637b87a640 1 -- 172.21.15.29:0/2953975983 shutdown_connections 2024-07-27T06:40:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.181+0000 7f637b87a640 1 -- 172.21.15.29:0/2953975983 wait complete. 2024-07-27T06:40:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.182+0000 7f637b87a640 1 Processor -- start 2024-07-27T06:40:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.182+0000 7f637b87a640 1 -- start start 2024-07-27T06:40:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.182+0000 7f637b87a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63741042b0 0x7f6374199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:40:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.182+0000 7f637b87a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6374104ce0 0x7f637419a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:40:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.182+0000 7f637b87a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f637410d860 0x7f637419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:40:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.182+0000 7f637b87a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f63741134a0 con 0x7f6374104ce0 2024-07-27T06:40:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.182+0000 7f637b87a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f6374113320 con 0x7f637410d860 2024-07-27T06:40:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.182+0000 7f637b87a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6374113620 con 0x7f63741042b0 2024-07-27T06:40:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f6378e15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6374104ce0 0x7f637419a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:40:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f6378e15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6374104ce0 0x7f637419a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53176/0 (socket says 172.21.15.29:53176) 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f6378e15640 1 -- 172.21.15.29:0/647294180 learned_addr learned my addr 172.21.15.29:0/647294180 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f6379616640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63741042b0 0x7f6374199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f6379e17640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f637410d860 0x7f637419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f6378e15640 1 -- 172.21.15.29:0/647294180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63741042b0 msgr2=0x7f6374199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f6378e15640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63741042b0 0x7f6374199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f6378e15640 1 -- 172.21.15.29:0/647294180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f637410d860 msgr2=0x7f637419f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f6378e15640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f637410d860 0x7f637419f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f6378e15640 1 -- 172.21.15.29:0/647294180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637419fe20 con 0x7f6374104ce0 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f6378e15640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6374104ce0 0x7f637419a450 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f636400c970 tx=0x7f636400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f636a7fc640 1 -- 172.21.15.29:0/647294180 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6364007bb0 con 0x7f6374104ce0 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f636a7fc640 1 -- 172.21.15.29:0/647294180 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6364007d50 con 0x7f6374104ce0 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f637b87a640 1 -- 172.21.15.29:0/647294180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63741a0110 con 0x7f6374104ce0 2024-07-27T06:40:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f636a7fc640 1 -- 172.21.15.29:0/647294180 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636400f450 con 0x7f6374104ce0 2024-07-27T06:40:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.183+0000 7f637b87a640 1 -- 172.21.15.29:0/647294180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63741a6870 con 0x7f6374104ce0 2024-07-27T06:40:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.184+0000 7f637b87a640 1 -- 172.21.15.29:0/647294180 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63741046c0 con 0x7f6374104ce0 2024-07-27T06:40:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.186+0000 7f636a7fc640 1 -- 172.21.15.29:0/647294180 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f6364004f20 con 0x7f6374104ce0 2024-07-27T06:40:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.187+0000 7f636a7fc640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63600b3ca0 0x7f63600b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:40:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.187+0000 7f6379616640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63600b3ca0 0x7f63600b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:40:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.187+0000 7f636a7fc640 1 -- 172.21.15.29:0/647294180 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6044+0+0 (secure 0 0 0) 0x7f63640e3450 con 0x7f6374104ce0 2024-07-27T06:40:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.187+0000 7f6379616640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63600b3ca0 0x7f63600b61b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6370005fd0 tx=0x7f637000b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:40:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.190+0000 7f636a7fc640 1 -- 172.21.15.29:0/647294180 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6364014030 con 0x7f6374104ce0 2024-07-27T06:40:53.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:40:53.383+0000 7f637b87a640 1 -- 172.21.15.29:0/647294180 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6374106a80 con 0x7f63600b3ca0 2024-07-27T06:40:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:53 smithi047 ceph-mon[36835]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:40:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:53 smithi047 ceph-mon[36835]: osdmap e56: 9 total, 9 up, 9 in 2024-07-27T06:40:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:53 smithi077 ceph-mon[37153]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:40:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:53 smithi077 ceph-mon[37153]: osdmap e56: 9 total, 9 up, 9 in 2024-07-27T06:40:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:54 smithi047 ceph-mon[36835]: from='client.14613 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:40:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:40:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:40:54.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:54 smithi029 ceph-mon[32610]: from='client.14613 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:40:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:40:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:40:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:54 smithi077 ceph-mon[37153]: from='client.14613 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:40:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:40:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:40:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:40:55.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:55 smithi029 ceph-mon[32610]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:40:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:55 smithi047 ceph-mon[36835]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:40:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:55 smithi077 ceph-mon[37153]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:40:56.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:56 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2241371875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f352242b-0d2e-41b3-9c83-436cf432b0df"}]: dispatch 2024-07-27T06:40:56.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:56 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2241371875' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f352242b-0d2e-41b3-9c83-436cf432b0df"}]': finished 2024-07-27T06:40:56.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:56 smithi029 ceph-mon[32610]: osdmap e57: 10 total, 9 up, 10 in 2024-07-27T06:40:56.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:40:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:56 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2241371875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f352242b-0d2e-41b3-9c83-436cf432b0df"}]: dispatch 2024-07-27T06:40:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:56 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2241371875' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f352242b-0d2e-41b3-9c83-436cf432b0df"}]': finished 2024-07-27T06:40:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:56 smithi047 ceph-mon[36835]: osdmap e57: 10 total, 9 up, 10 in 2024-07-27T06:40:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:40:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:56 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2241371875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f352242b-0d2e-41b3-9c83-436cf432b0df"}]: dispatch 2024-07-27T06:40:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:56 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2241371875' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f352242b-0d2e-41b3-9c83-436cf432b0df"}]': finished 2024-07-27T06:40:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:56 smithi077 ceph-mon[37153]: osdmap e57: 10 total, 9 up, 10 in 2024-07-27T06:40:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:40:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:57 smithi047 ceph-mon[36835]: pgmap v188: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:40:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:57 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2177757418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:40:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:57 smithi029 ceph-mon[32610]: pgmap v188: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:40:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:57 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2177757418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:40:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:57 smithi077 ceph-mon[37153]: pgmap v188: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:40:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:57 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2177757418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:40:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:40:59 smithi047 ceph-mon[36835]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-07-27T06:40:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:40:59 smithi029 ceph-mon[32610]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-07-27T06:40:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:40:59 smithi077 ceph-mon[37153]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-07-27T06:41:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:01 smithi047 ceph-mon[36835]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-27T06:41:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:01 smithi029 ceph-mon[32610]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-27T06:41:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:01 smithi077 ceph-mon[37153]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-27T06:41:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:03 smithi047 ceph-mon[36835]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-07-27T06:41:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:03 smithi029 ceph-mon[32610]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-07-27T06:41:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:03 smithi077 ceph-mon[37153]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-07-27T06:41:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-27T06:41:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-27T06:41:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-27T06:41:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:05.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:05 smithi029 ceph-mon[32610]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:41:05.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:05 smithi029 ceph-mon[32610]: Deploying daemon osd.9 on smithi029 2024-07-27T06:41:05.452 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:05 smithi047 ceph-mon[36835]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:41:05.452 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:05 smithi047 ceph-mon[36835]: Deploying daemon osd.9 on smithi029 2024-07-27T06:41:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:05 smithi077 ceph-mon[37153]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:41:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:05 smithi077 ceph-mon[37153]: Deploying daemon osd.9 on smithi029 2024-07-27T06:41:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:07 smithi047 ceph-mon[36835]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:41:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:07 smithi029 ceph-mon[32610]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:41:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:07 smithi077 ceph-mon[37153]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:41:09.394 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:09 smithi029 ceph-mon[32610]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-07-27T06:41:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:09 smithi047 ceph-mon[36835]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-07-27T06:41:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:09 smithi077 ceph-mon[37153]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-07-27T06:41:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:10.426 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:10.426 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:10.427 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:11 smithi047 ceph-mon[36835]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:41:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:11 smithi029 ceph-mon[32610]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:41:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:11 smithi077 ceph-mon[37153]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:41:13.134 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 9 on host 'smithi029' 2024-07-27T06:41:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.131+0000 7f636a7fc640 1 -- 172.21.15.29:0/647294180 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6374106a80 con 0x7f63600b3ca0 2024-07-27T06:41:13.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.134+0000 7f637b87a640 1 -- 172.21.15.29:0/647294180 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63600b3ca0 msgr2=0x7f63600b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:41:13.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.134+0000 7f637b87a640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63600b3ca0 0x7f63600b61b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6370005fd0 tx=0x7f637000b390 comp rx=0 tx=0).stop 2024-07-27T06:41:13.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.134+0000 7f637b87a640 1 -- 172.21.15.29:0/647294180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6374104ce0 msgr2=0x7f637419a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:41:13.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.134+0000 7f637b87a640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6374104ce0 0x7f637419a450 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f636400c970 tx=0x7f636400ce30 comp rx=0 tx=0).stop 2024-07-27T06:41:13.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.134+0000 7f637b87a640 1 -- 172.21.15.29:0/647294180 shutdown_connections 2024-07-27T06:41:13.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.134+0000 7f637b87a640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f637410d860 0x7f637419f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:13.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.134+0000 7f637b87a640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6374104ce0 0x7f637419a450 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:13.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.134+0000 7f637b87a640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63600b3ca0 0x7f63600b61b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:13.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.134+0000 7f637b87a640 1 --2- 172.21.15.29:0/647294180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63741042b0 0x7f6374199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:13.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.134+0000 7f637b87a640 1 -- 172.21.15.29:0/647294180 >> 172.21.15.29:0/647294180 conn(0x7f6374079160 msgr2=0x7f6374106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:41:13.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.134+0000 7f637b87a640 1 -- 172.21.15.29:0/647294180 shutdown_connections 2024-07-27T06:41:13.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:41:13.134+0000 7f637b87a640 1 -- 172.21.15.29:0/647294180 wait complete. 2024-07-27T06:41:13.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:13 smithi029 ceph-mon[32610]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:41:13.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:13.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:13 smithi047 ceph-mon[36835]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:41:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:13 smithi077 ceph-mon[37153]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:41:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:14.295 DEBUG:teuthology.orchestra.run.smithi029:osd.9> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@osd.9.service 2024-07-27T06:41:14.299 INFO:tasks.cephadm:Deploying osd.10 on smithi047 with /dev/vg_nvme/lv_1... 2024-07-27T06:41:14.299 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-27T06:41:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:14 smithi047 ceph-mon[36835]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-27T06:41:14.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:14 smithi029 ceph-mon[32610]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-27T06:41:14.574 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.b/config 2024-07-27T06:41:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:14 smithi077 ceph-mon[37153]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-27T06:41:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:15 smithi047 ceph-mon[36835]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:41:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:15 smithi047 ceph-mon[36835]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-27T06:41:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:15 smithi047 ceph-mon[36835]: osdmap e58: 10 total, 9 up, 10 in 2024-07-27T06:41:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:15 smithi047 ceph-mon[36835]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-07-27T06:41:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:15.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:15 smithi029 ceph-mon[32610]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:41:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:15 smithi029 ceph-mon[32610]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-27T06:41:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:15 smithi029 ceph-mon[32610]: osdmap e58: 10 total, 9 up, 10 in 2024-07-27T06:41:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:15 smithi029 ceph-mon[32610]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-07-27T06:41:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:15 smithi077 ceph-mon[37153]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:41:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:15 smithi077 ceph-mon[37153]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-27T06:41:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:15 smithi077 ceph-mon[37153]: osdmap e58: 10 total, 9 up, 10 in 2024-07-27T06:41:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:15 smithi077 ceph-mon[37153]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-07-27T06:41:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:15.962 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jul 27 06:41:15 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-9[71869]: 2024-07-27T06:41:15.656+0000 7fbe6af04640 -1 osd.9 0 waiting for initial osdmap 2024-07-27T06:41:15.963 INFO:journalctl@ceph.osd.9.smithi029.stdout:Jul 27 06:41:15 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-9[71869]: 2024-07-27T06:41:15.669+0000 7fbe62cfa640 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T06:41:16.320 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:16 smithi047 ceph-mon[36835]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-07-27T06:41:16.320 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:16 smithi047 ceph-mon[36835]: osdmap e59: 10 total, 9 up, 10 in 2024-07-27T06:41:16.320 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:16.320 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:16.320 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:16 smithi047 ceph-mon[36835]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' 2024-07-27T06:41:16.442 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:16 smithi029 ceph-mon[32610]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-07-27T06:41:16.442 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:16 smithi029 ceph-mon[32610]: osdmap e59: 10 total, 9 up, 10 in 2024-07-27T06:41:16.443 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:16.443 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:16.443 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:16 smithi029 ceph-mon[32610]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' 2024-07-27T06:41:16.463 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-27T06:41:16.554 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-27T06:41:16.564 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10+0 records in 2024-07-27T06:41:16.564 INFO:teuthology.orchestra.run.smithi047.stderr:10+0 records out 2024-07-27T06:41:16.565 INFO:teuthology.orchestra.run.smithi047.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00835109 s, 1.3 GB/s 2024-07-27T06:41:16.565 INFO:teuthology.orchestra.run.smithi047.stderr:--> Zapping successful for: 2024-07-27T06:41:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:16 smithi077 ceph-mon[37153]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-07-27T06:41:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:16 smithi077 ceph-mon[37153]: osdmap e59: 10 total, 9 up, 10 in 2024-07-27T06:41:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:16 smithi077 ceph-mon[37153]: from='osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384]' entity='osd.9' 2024-07-27T06:41:17.338 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch daemon add osd smithi047:vg_nvme/lv_1 2024-07-27T06:41:17.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:17 smithi029 ceph-mon[32610]: purged_snaps scrub starts 2024-07-27T06:41:17.439 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:17 smithi029 ceph-mon[32610]: purged_snaps scrub ok 2024-07-27T06:41:17.439 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:17 smithi029 ceph-mon[32610]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:41:17.439 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:17.439 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:17 smithi029 ceph-mon[32610]: osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384] boot 2024-07-27T06:41:17.439 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:17 smithi029 ceph-mon[32610]: osdmap e60: 10 total, 10 up, 10 in 2024-07-27T06:41:17.439 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:17 smithi047 ceph-mon[36835]: purged_snaps scrub starts 2024-07-27T06:41:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:17 smithi047 ceph-mon[36835]: purged_snaps scrub ok 2024-07-27T06:41:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:17 smithi047 ceph-mon[36835]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:41:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:17 smithi047 ceph-mon[36835]: osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384] boot 2024-07-27T06:41:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:17 smithi047 ceph-mon[36835]: osdmap e60: 10 total, 10 up, 10 in 2024-07-27T06:41:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:17.600 INFO:teuthology.orchestra.run.smithi047.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.b/config 2024-07-27T06:41:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:17 smithi077 ceph-mon[37153]: purged_snaps scrub starts 2024-07-27T06:41:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:17 smithi077 ceph-mon[37153]: purged_snaps scrub ok 2024-07-27T06:41:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:17 smithi077 ceph-mon[37153]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-07-27T06:41:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:17 smithi077 ceph-mon[37153]: osd.9 [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384] boot 2024-07-27T06:41:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:17 smithi077 ceph-mon[37153]: osdmap e60: 10 total, 10 up, 10 in 2024-07-27T06:41:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-27T06:41:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.324+0000 7f033feb4640 1 -- 172.21.15.47:0/2554463771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0338109820 msgr2=0x7f0338110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:41:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.324+0000 7f033feb4640 1 --2- 172.21.15.47:0/2554463771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0338109820 0x7f0338110050 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0330009ca0 tx=0x7f033002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:41:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.324+0000 7f033feb4640 1 -- 172.21.15.47:0/2554463771 shutdown_connections 2024-07-27T06:41:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.324+0000 7f033feb4640 1 --2- 172.21.15.47:0/2554463771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0338109820 0x7f0338110050 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.324+0000 7f033feb4640 1 --2- 172.21.15.47:0/2554463771 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0338108ed0 0x7f03381092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.324+0000 7f033feb4640 1 --2- 172.21.15.47:0/2554463771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0338103590 0x7f03381039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.324+0000 7f033feb4640 1 -- 172.21.15.47:0/2554463771 >> 172.21.15.47:0/2554463771 conn(0x7f033807b280 msgr2=0x7f033807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:41:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.324+0000 7f033feb4640 1 -- 172.21.15.47:0/2554463771 shutdown_connections 2024-07-27T06:41:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.324+0000 7f033feb4640 1 -- 172.21.15.47:0/2554463771 wait complete. 2024-07-27T06:41:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033feb4640 1 Processor -- start 2024-07-27T06:41:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033feb4640 1 -- start start 2024-07-27T06:41:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033feb4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0338103590 0x7f0338199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:41:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033feb4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0338108ed0 0x7f033819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:41:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033feb4640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0338109820 0x7f033819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:41:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033feb4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03381134f0 con 0x7f0338108ed0 2024-07-27T06:41:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033feb4640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f0338113370 con 0x7f0338109820 2024-07-27T06:41:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033feb4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0338113670 con 0x7f0338103590 2024-07-27T06:41:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033e451640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0338109820 0x7f033819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:41:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033e451640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0338109820 0x7f033819f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50112/0 (socket says 172.21.15.47:50112) 2024-07-27T06:41:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033e451640 1 -- 172.21.15.47:0/3076822917 learned_addr learned my addr 172.21.15.47:0/3076822917 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-07-27T06:41:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033d44f640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0338108ed0 0x7f033819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:41:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.325+0000 7f033dc50640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0338103590 0x7f0338199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:41:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.326+0000 7f033e451640 1 -- 172.21.15.47:0/3076822917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0338103590 msgr2=0x7f0338199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:41:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.326+0000 7f033e451640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0338103590 0x7f0338199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.326+0000 7f033e451640 1 -- 172.21.15.47:0/3076822917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0338108ed0 msgr2=0x7f033819a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:41:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.326+0000 7f033e451640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0338108ed0 0x7f033819a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.326+0000 7f033e451640 1 -- 172.21.15.47:0/3076822917 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f033819fdb0 con 0x7f0338109820 2024-07-27T06:41:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.326+0000 7f033e451640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0338109820 0x7f033819f680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0330005cc0 tx=0x7f0330031700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:41:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.326+0000 7f032effd640 1 -- 172.21.15.47:0/3076822917 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0330003b70 con 0x7f0338109820 2024-07-27T06:41:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.326+0000 7f032effd640 1 -- 172.21.15.47:0/3076822917 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03300318f0 con 0x7f0338109820 2024-07-27T06:41:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.326+0000 7f033feb4640 1 -- 172.21.15.47:0/3076822917 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03381a0040 con 0x7f0338109820 2024-07-27T06:41:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.326+0000 7f032effd640 1 -- 172.21.15.47:0/3076822917 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033002f950 con 0x7f0338109820 2024-07-27T06:41:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.326+0000 7f033feb4640 1 -- 172.21.15.47:0/3076822917 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03381a6800 con 0x7f0338109820 2024-07-27T06:41:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.327+0000 7f033feb4640 1 -- 172.21.15.47:0/3076822917 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03381039a0 con 0x7f0338109820 2024-07-27T06:41:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.329+0000 7f032effd640 1 -- 172.21.15.47:0/3076822917 <== mon.2 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f0330004b40 con 0x7f0338109820 2024-07-27T06:41:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.329+0000 7f032effd640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f03240b3c20 0x7f03240b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:41:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.330+0000 7f032effd640 1 -- 172.21.15.47:0/3076822917 <== mon.2 v2:172.21.15.47:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6476+0+0 (secure 0 0 0) 0x7f0330104ae0 con 0x7f0338109820 2024-07-27T06:41:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.330+0000 7f033dc50640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f03240b3c20 0x7f03240b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:41:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.330+0000 7f033dc50640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f03240b3c20 0x7f03240b6130 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0334009b60 tx=0x7f0334005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:41:19.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.333+0000 7f032effd640 1 -- 172.21.15.47:0/3076822917 <== mon.2 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f03300cc860 con 0x7f0338109820 2024-07-27T06:41:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: osdmap e61: 10 total, 10 up, 10 in 2024-07-27T06:41:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:41:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: osdmap e61: 10 total, 10 up, 10 in 2024-07-27T06:41:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:19.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:19.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:19.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:41:19.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:19.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:19.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:19.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:19.525+0000 7f033feb4640 1 -- 172.21.15.47:0/3076822917 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0338063230 con 0x7f03240b3c20 2024-07-27T06:41:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: osdmap e61: 10 total, 10 up, 10 in 2024-07-27T06:41:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:41:19.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:19.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:19.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:20 smithi029 ceph-mon[32610]: Detected new or changed devices on smithi029 2024-07-27T06:41:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:20 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 4273M 2024-07-27T06:41:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:41:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:41:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:41:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:20 smithi077 ceph-mon[37153]: Detected new or changed devices on smithi029 2024-07-27T06:41:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:20 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 4273M 2024-07-27T06:41:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:41:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:41:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:41:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:20 smithi047 ceph-mon[36835]: Detected new or changed devices on smithi029 2024-07-27T06:41:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:20 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 4273M 2024-07-27T06:41:20.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:41:20.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:20.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:41:20.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:41:20.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:21 smithi077 ceph-mon[37153]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:41:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:21 smithi077 ceph-mon[37153]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:21 smithi029 ceph-mon[32610]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:41:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:21 smithi029 ceph-mon[32610]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:21.500 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:21 smithi047 ceph-mon[36835]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi047:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:41:21.501 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:21 smithi047 ceph-mon[36835]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:22 smithi077 ceph-mon[37153]: from='client.? 172.21.15.47:0/2775938533' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4148c2df-c3e9-4a47-88f7-36cf3eab8d99"}]: dispatch 2024-07-27T06:41:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:22 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4148c2df-c3e9-4a47-88f7-36cf3eab8d99"}]: dispatch 2024-07-27T06:41:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:22 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4148c2df-c3e9-4a47-88f7-36cf3eab8d99"}]': finished 2024-07-27T06:41:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:22 smithi077 ceph-mon[37153]: osdmap e62: 11 total, 10 up, 11 in 2024-07-27T06:41:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:22.628 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:22 smithi047 ceph-mon[36835]: from='client.? 172.21.15.47:0/2775938533' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4148c2df-c3e9-4a47-88f7-36cf3eab8d99"}]: dispatch 2024-07-27T06:41:22.628 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:22 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4148c2df-c3e9-4a47-88f7-36cf3eab8d99"}]: dispatch 2024-07-27T06:41:22.628 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:22 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4148c2df-c3e9-4a47-88f7-36cf3eab8d99"}]': finished 2024-07-27T06:41:22.628 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:22 smithi047 ceph-mon[36835]: osdmap e62: 11 total, 10 up, 11 in 2024-07-27T06:41:22.628 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:22.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:22 smithi029 ceph-mon[32610]: from='client.? 172.21.15.47:0/2775938533' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4148c2df-c3e9-4a47-88f7-36cf3eab8d99"}]: dispatch 2024-07-27T06:41:22.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:22 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4148c2df-c3e9-4a47-88f7-36cf3eab8d99"}]: dispatch 2024-07-27T06:41:22.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:22 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4148c2df-c3e9-4a47-88f7-36cf3eab8d99"}]': finished 2024-07-27T06:41:22.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:22 smithi029 ceph-mon[32610]: osdmap e62: 11 total, 10 up, 11 in 2024-07-27T06:41:22.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:23 smithi077 ceph-mon[37153]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:23 smithi077 ceph-mon[37153]: from='client.? 172.21.15.47:0/3322589285' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:41:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:23 smithi047 ceph-mon[36835]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:23 smithi047 ceph-mon[36835]: from='client.? 172.21.15.47:0/3322589285' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:41:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:23 smithi029 ceph-mon[32610]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:23 smithi029 ceph-mon[32610]: from='client.? 172.21.15.47:0/3322589285' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:41:25.564 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:25 smithi047 ceph-mon[36835]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:25 smithi077 ceph-mon[37153]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:25.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:25 smithi029 ceph-mon[32610]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:27.324 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:27 smithi047 ceph-mon[36835]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:27 smithi077 ceph-mon[37153]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:27 smithi029 ceph-mon[32610]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:29 smithi047 ceph-mon[36835]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-27T06:41:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:29 smithi077 ceph-mon[37153]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-27T06:41:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:29.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:29 smithi029 ceph-mon[32610]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-27T06:41:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:30.340 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:30 smithi047 ceph-mon[36835]: Deploying daemon osd.10 on smithi047 2024-07-27T06:41:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:30 smithi077 ceph-mon[37153]: Deploying daemon osd.10 on smithi047 2024-07-27T06:41:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:30 smithi029 ceph-mon[32610]: Deploying daemon osd.10 on smithi047 2024-07-27T06:41:31.273 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:31 smithi047 ceph-mon[36835]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:31 smithi077 ceph-mon[37153]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:31.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:31 smithi029 ceph-mon[32610]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:33 smithi077 ceph-mon[37153]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:33 smithi047 ceph-mon[36835]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:33.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:33 smithi029 ceph-mon[32610]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:35.334 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:35 smithi047 ceph-mon[36835]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:35.334 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:35.334 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:35.335 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:35 smithi077 ceph-mon[37153]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:35 smithi029 ceph-mon[32610]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:37.558 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:37 smithi047 ceph-mon[36835]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:37 smithi077 ceph-mon[37153]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:37 smithi029 ceph-mon[32610]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:38.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.302+0000 7f032effd640 1 -- 172.21.15.47:0/3076822917 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f0338063230 con 0x7f03240b3c20 2024-07-27T06:41:38.304 INFO:teuthology.orchestra.run.smithi047.stdout:Created osd(s) 10 on host 'smithi047' 2024-07-27T06:41:38.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.304+0000 7f033feb4640 1 -- 172.21.15.47:0/3076822917 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f03240b3c20 msgr2=0x7f03240b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:41:38.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.304+0000 7f033feb4640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f03240b3c20 0x7f03240b6130 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0334009b60 tx=0x7f0334005f90 comp rx=0 tx=0).stop 2024-07-27T06:41:38.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.304+0000 7f033feb4640 1 -- 172.21.15.47:0/3076822917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0338109820 msgr2=0x7f033819f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:41:38.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.304+0000 7f033feb4640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0338109820 0x7f033819f680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0330005cc0 tx=0x7f0330031700 comp rx=0 tx=0).stop 2024-07-27T06:41:38.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.304+0000 7f033feb4640 1 -- 172.21.15.47:0/3076822917 shutdown_connections 2024-07-27T06:41:38.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.304+0000 7f033feb4640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0338109820 0x7f033819f680 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:38.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.304+0000 7f033feb4640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0338108ed0 0x7f033819a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:38.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.304+0000 7f033feb4640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f03240b3c20 0x7f03240b6130 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:38.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.304+0000 7f033feb4640 1 --2- 172.21.15.47:0/3076822917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0338103590 0x7f0338199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:38.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.304+0000 7f033feb4640 1 -- 172.21.15.47:0/3076822917 >> 172.21.15.47:0/3076822917 conn(0x7f033807b280 msgr2=0x7f0338106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:41:38.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.304+0000 7f033feb4640 1 -- 172.21.15.47:0/3076822917 shutdown_connections 2024-07-27T06:41:38.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-07-27T06:41:38.304+0000 7f033feb4640 1 -- 172.21.15.47:0/3076822917 wait complete. 2024-07-27T06:41:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:41:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:38 smithi077 ceph-mon[37153]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:38 smithi077 ceph-mon[37153]: from='osd.10 [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-27T06:41:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:38 smithi077 ceph-mon[37153]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-27T06:41:38.913 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.913 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.913 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:38.913 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:41:38.913 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:38 smithi047 ceph-mon[36835]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:38.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:38.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:38 smithi047 ceph-mon[36835]: from='osd.10 [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-27T06:41:38.914 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:38 smithi047 ceph-mon[36835]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-27T06:41:38.943 DEBUG:teuthology.orchestra.run.smithi047:osd.10> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@osd.10.service 2024-07-27T06:41:38.945 INFO:tasks.cephadm:Deploying osd.11 on smithi077 with /dev/vg_nvme/lv_1... 2024-07-27T06:41:38.946 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-27T06:41:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:41:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:38 smithi029 ceph-mon[32610]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:38 smithi029 ceph-mon[32610]: from='osd.10 [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-27T06:41:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:38 smithi029 ceph-mon[32610]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-27T06:41:39.205 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.c/config 2024-07-27T06:41:40.523 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:40 smithi047 ceph-mon[36835]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-27T06:41:40.523 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:40 smithi047 ceph-mon[36835]: from='osd.10 [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:41:40.523 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:40 smithi047 ceph-mon[36835]: osdmap e63: 11 total, 10 up, 11 in 2024-07-27T06:41:40.523 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:40.523 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:40 smithi047 ceph-mon[36835]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:41:40.523 INFO:journalctl@ceph.osd.10.smithi047.stdout:Jul 27 06:41:40 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-10[63012]: 2024-07-27T06:41:40.310+0000 7f49888a3640 -1 osd.10 0 waiting for initial osdmap 2024-07-27T06:41:40.523 INFO:journalctl@ceph.osd.10.smithi047.stdout:Jul 27 06:41:40 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-10[63012]: 2024-07-27T06:41:40.323+0000 7f49846a1640 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T06:41:40.562 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:40 smithi077 ceph-mon[37153]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-27T06:41:40.562 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:40 smithi077 ceph-mon[37153]: from='osd.10 [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:41:40.562 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:40 smithi077 ceph-mon[37153]: osdmap e63: 11 total, 10 up, 11 in 2024-07-27T06:41:40.562 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:40.562 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:40 smithi077 ceph-mon[37153]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:41:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:40 smithi029 ceph-mon[32610]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-27T06:41:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:40 smithi029 ceph-mon[32610]: from='osd.10 [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:41:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:40 smithi029 ceph-mon[32610]: osdmap e63: 11 total, 10 up, 11 in 2024-07-27T06:41:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:40 smithi029 ceph-mon[32610]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]: dispatch 2024-07-27T06:41:41.060 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-27T06:41:41.158 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-27T06:41:41.170 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-07-27T06:41:41.170 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-07-27T06:41:41.170 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113652 s, 923 MB/s 2024-07-27T06:41:41.170 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-07-27T06:41:41.470 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:41 smithi077 ceph-mon[37153]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:41.470 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:41 smithi077 ceph-mon[37153]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-07-27T06:41:41.470 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:41 smithi077 ceph-mon[37153]: osdmap e64: 11 total, 10 up, 11 in 2024-07-27T06:41:41.471 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:41.471 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:41.604 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:41 smithi047 ceph-mon[36835]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:41.604 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:41 smithi047 ceph-mon[36835]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-07-27T06:41:41.604 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:41 smithi047 ceph-mon[36835]: osdmap e64: 11 total, 10 up, 11 in 2024-07-27T06:41:41.605 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:41.605 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:41 smithi029 ceph-mon[32610]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:41 smithi029 ceph-mon[32610]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi047", "root=default"]}]': finished 2024-07-27T06:41:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:41 smithi029 ceph-mon[32610]: osdmap e64: 11 total, 10 up, 11 in 2024-07-27T06:41:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:41.811 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_1 2024-07-27T06:41:42.076 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.c/config 2024-07-27T06:41:42.452 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: purged_snaps scrub starts 2024-07-27T06:41:42.452 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: purged_snaps scrub ok 2024-07-27T06:41:42.452 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: osd.10 [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850] boot 2024-07-27T06:41:42.452 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: osdmap e65: 11 total, 11 up, 11 in 2024-07-27T06:41:42.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:42.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:42.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:42.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:42.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:42.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:42.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:42.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:42.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:42.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:41:42.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:42.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:42 smithi077 ceph-mon[37153]: osdmap e66: 11 total, 11 up, 11 in 2024-07-27T06:41:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: purged_snaps scrub starts 2024-07-27T06:41:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: purged_snaps scrub ok 2024-07-27T06:41:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: osd.10 [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850] boot 2024-07-27T06:41:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: osdmap e65: 11 total, 11 up, 11 in 2024-07-27T06:41:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:41:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:42.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:42 smithi047 ceph-mon[36835]: osdmap e66: 11 total, 11 up, 11 in 2024-07-27T06:41:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: purged_snaps scrub starts 2024-07-27T06:41:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: purged_snaps scrub ok 2024-07-27T06:41:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: osd.10 [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850] boot 2024-07-27T06:41:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: osdmap e65: 11 total, 11 up, 11 in 2024-07-27T06:41:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-27T06:41:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:41:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:41:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:42 smithi029 ceph-mon[32610]: osdmap e66: 11 total, 11 up, 11 in 2024-07-27T06:41:43.587 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:43 smithi077 ceph-mon[37153]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:43.588 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:43 smithi077 ceph-mon[37153]: Detected new or changed devices on smithi047 2024-07-27T06:41:43.588 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:43 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 4273M 2024-07-27T06:41:43.588 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:43 smithi077 ceph-mon[37153]: osdmap e67: 11 total, 11 up, 11 in 2024-07-27T06:41:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:43 smithi047 ceph-mon[36835]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:43 smithi047 ceph-mon[36835]: Detected new or changed devices on smithi047 2024-07-27T06:41:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:43 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 4273M 2024-07-27T06:41:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:43 smithi047 ceph-mon[36835]: osdmap e67: 11 total, 11 up, 11 in 2024-07-27T06:41:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:43 smithi029 ceph-mon[32610]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-07-27T06:41:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:43 smithi029 ceph-mon[32610]: Detected new or changed devices on smithi047 2024-07-27T06:41:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:43 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 4273M 2024-07-27T06:41:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:43 smithi029 ceph-mon[32610]: osdmap e67: 11 total, 11 up, 11 in 2024-07-27T06:41:43.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.799+0000 7fc2599d8640 1 -- 172.21.15.77:0/2755445179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc254106630 msgr2=0x7fc2541100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:41:43.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.799+0000 7fc2599d8640 1 --2- 172.21.15.77:0/2755445179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc254106630 0x7fc2541100a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009ca0 tx=0x7fc24c02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:41:43.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.799+0000 7fc2599d8640 1 -- 172.21.15.77:0/2755445179 shutdown_connections 2024-07-27T06:41:43.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.799+0000 7fc2599d8640 1 --2- 172.21.15.77:0/2755445179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc254106630 0x7fc2541100a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:43.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.799+0000 7fc2599d8640 1 --2- 172.21.15.77:0/2755445179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc254105ce0 0x7fc2541060f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:43.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.799+0000 7fc2599d8640 1 --2- 172.21.15.77:0/2755445179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc254101af0 0x7fc254101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:43.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.799+0000 7fc2599d8640 1 -- 172.21.15.77:0/2755445179 >> 172.21.15.77:0/2755445179 conn(0x7fc254079870 msgr2=0x7fc254079c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:41:43.818 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.799+0000 7fc2599d8640 1 -- 172.21.15.77:0/2755445179 shutdown_connections 2024-07-27T06:41:43.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.799+0000 7fc2599d8640 1 -- 172.21.15.77:0/2755445179 wait complete. 2024-07-27T06:41:43.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.800+0000 7fc2599d8640 1 Processor -- start 2024-07-27T06:41:43.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.800+0000 7fc2599d8640 1 -- start start 2024-07-27T06:41:43.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.800+0000 7fc2599d8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc254101af0 0x7fc254199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:41:43.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2599d8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc254105ce0 0x7fc25419a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:41:43.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2599d8640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc254106630 0x7fc25419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:41:43.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2599d8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc254113590 con 0x7fc254105ce0 2024-07-27T06:41:43.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2599d8640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fc254113410 con 0x7fc254106630 2024-07-27T06:41:43.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2599d8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc254113710 con 0x7fc254101af0 2024-07-27T06:41:43.819 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc252ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc254101af0 0x7fc254199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:41:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc252ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc254101af0 0x7fc254199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36072/0 (socket says 172.21.15.77:36072) 2024-07-27T06:41:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc252ffd640 1 -- 172.21.15.77:0/3630474651 learned_addr learned my addr 172.21.15.77:0/3630474651 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-07-27T06:41:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2527fc640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc254105ce0 0x7fc25419a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:41:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2537fe640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc254106630 0x7fc25419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:41:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc252ffd640 1 -- 172.21.15.77:0/3630474651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc254106630 msgr2=0x7fc25419f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:41:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc252ffd640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc254106630 0x7fc25419f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc252ffd640 1 -- 172.21.15.77:0/3630474651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc254105ce0 msgr2=0x7fc25419a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:41:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc252ffd640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc254105ce0 0x7fc25419a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:41:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc252ffd640 1 -- 172.21.15.77:0/3630474651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25419fe30 con 0x7fc254101af0 2024-07-27T06:41:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2537fe640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc254106630 0x7fc25419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:41:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc252ffd640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc254101af0 0x7fc254199ec0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc24400b4a0 tx=0x7fc24400b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:41:43.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2589d6640 1 -- 172.21.15.77:0/3630474651 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244014070 con 0x7fc254101af0 2024-07-27T06:41:43.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2599d8640 1 -- 172.21.15.77:0/3630474651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2541a0120 con 0x7fc254101af0 2024-07-27T06:41:43.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2589d6640 1 -- 172.21.15.77:0/3630474651 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc244010ac0 con 0x7fc254101af0 2024-07-27T06:41:43.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2589d6640 1 -- 172.21.15.77:0/3630474651 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244017c30 con 0x7fc254101af0 2024-07-27T06:41:43.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.801+0000 7fc2599d8640 1 -- 172.21.15.77:0/3630474651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2541a6880 con 0x7fc254101af0 2024-07-27T06:41:43.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.802+0000 7fc2599d8640 1 -- 172.21.15.77:0/3630474651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc254101f00 con 0x7fc254101af0 2024-07-27T06:41:43.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.804+0000 7fc2589d6640 1 -- 172.21.15.77:0/3630474651 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fc244017070 con 0x7fc254101af0 2024-07-27T06:41:43.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.805+0000 7fc2589d6640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc23c0b3c20 0x7fc23c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:41:43.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.805+0000 7fc2589d6640 1 -- 172.21.15.77:0/3630474651 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6909+0+0 (secure 0 0 0) 0x7fc2440e2ec0 con 0x7fc254101af0 2024-07-27T06:41:43.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.805+0000 7fc2527fc640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc23c0b3c20 0x7fc23c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:41:43.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.806+0000 7fc2527fc640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc23c0b3c20 0x7fc23c0b6130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc254108c00 tx=0x7fc24800b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:41:43.821 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.808+0000 7fc2589d6640 1 -- 172.21.15.77:0/3630474651 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc2440aaa90 con 0x7fc254101af0 2024-07-27T06:41:43.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:41:43.990+0000 7fc2599d8640 1 -- 172.21.15.77:0/3630474651 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc254063230 con 0x7fc23c0b3c20 2024-07-27T06:41:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:41:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:41:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:41:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:41:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-27T06:41:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-27T06:41:44.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:45 smithi077 ceph-mon[37153]: pgmap v223: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:41:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:45 smithi077 ceph-mon[37153]: from='client.24562 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:41:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:45 smithi047 ceph-mon[36835]: pgmap v223: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:41:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:45 smithi047 ceph-mon[36835]: from='client.24562 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:41:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:45 smithi029 ceph-mon[32610]: pgmap v223: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:41:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:45 smithi029 ceph-mon[32610]: from='client.24562 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:41:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:47 smithi077 ceph-mon[37153]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:41:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:47 smithi077 ceph-mon[37153]: from='client.? 172.21.15.77:0/3685239020' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e8b1da3-5f65-4fe9-8722-b35921cbd8c9"}]: dispatch 2024-07-27T06:41:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:47 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e8b1da3-5f65-4fe9-8722-b35921cbd8c9"}]: dispatch 2024-07-27T06:41:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:47 smithi077 ceph-mon[37153]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e8b1da3-5f65-4fe9-8722-b35921cbd8c9"}]': finished 2024-07-27T06:41:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:47 smithi077 ceph-mon[37153]: osdmap e68: 12 total, 11 up, 12 in 2024-07-27T06:41:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:41:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:47 smithi077 ceph-mon[37153]: from='client.? 172.21.15.77:0/2896410941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:41:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:47 smithi047 ceph-mon[36835]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:41:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:47 smithi047 ceph-mon[36835]: from='client.? 172.21.15.77:0/3685239020' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e8b1da3-5f65-4fe9-8722-b35921cbd8c9"}]: dispatch 2024-07-27T06:41:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:47 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e8b1da3-5f65-4fe9-8722-b35921cbd8c9"}]: dispatch 2024-07-27T06:41:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:47 smithi047 ceph-mon[36835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e8b1da3-5f65-4fe9-8722-b35921cbd8c9"}]': finished 2024-07-27T06:41:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:47 smithi047 ceph-mon[36835]: osdmap e68: 12 total, 11 up, 12 in 2024-07-27T06:41:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:41:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:47 smithi047 ceph-mon[36835]: from='client.? 172.21.15.77:0/2896410941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:41:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:47 smithi029 ceph-mon[32610]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:41:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:47 smithi029 ceph-mon[32610]: from='client.? 172.21.15.77:0/3685239020' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e8b1da3-5f65-4fe9-8722-b35921cbd8c9"}]: dispatch 2024-07-27T06:41:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:47 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e8b1da3-5f65-4fe9-8722-b35921cbd8c9"}]: dispatch 2024-07-27T06:41:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:47 smithi029 ceph-mon[32610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e8b1da3-5f65-4fe9-8722-b35921cbd8c9"}]': finished 2024-07-27T06:41:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:47 smithi029 ceph-mon[32610]: osdmap e68: 12 total, 11 up, 12 in 2024-07-27T06:41:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:41:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:47 smithi029 ceph-mon[32610]: from='client.? 172.21.15.77:0/2896410941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-27T06:41:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:49 smithi077 ceph-mon[37153]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-27T06:41:49.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:49 smithi047 ceph-mon[36835]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-27T06:41:49.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:49 smithi029 ceph-mon[32610]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-27T06:41:51.588 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:51 smithi077 ceph-mon[37153]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-27T06:41:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:51 smithi047 ceph-mon[36835]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-27T06:41:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:51 smithi029 ceph-mon[32610]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-27T06:41:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:53 smithi077 ceph-mon[37153]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-07-27T06:41:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:53 smithi047 ceph-mon[36835]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-07-27T06:41:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:53 smithi029 ceph-mon[32610]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-07-27T06:41:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-27T06:41:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-27T06:41:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:54.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-27T06:41:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:41:55.582 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:55 smithi077 ceph-mon[37153]: Deploying daemon osd.11 on smithi077 2024-07-27T06:41:55.583 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:55 smithi077 ceph-mon[37153]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:41:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:55 smithi047 ceph-mon[36835]: Deploying daemon osd.11 on smithi077 2024-07-27T06:41:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:55 smithi047 ceph-mon[36835]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:41:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:55 smithi029 ceph-mon[32610]: Deploying daemon osd.11 on smithi077 2024-07-27T06:41:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:55 smithi029 ceph-mon[32610]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:41:57.504 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:57 smithi077 ceph-mon[37153]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:41:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:57 smithi047 ceph-mon[36835]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:41:57.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:57 smithi029 ceph-mon[32610]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-27T06:41:59.581 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:59 smithi077 ceph-mon[37153]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-07-27T06:41:59.581 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:59.581 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:59.581 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:41:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:59 smithi047 ceph-mon[36835]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-07-27T06:41:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:41:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:59 smithi029 ceph-mon[32610]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-07-27T06:41:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:41:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:41:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:41:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:01.544 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:01 smithi077 ceph-mon[37153]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:42:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:01 smithi047 ceph-mon[36835]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:42:01.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:01 smithi029 ceph-mon[32610]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:42:02.626 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 11 on host 'smithi077' 2024-07-27T06:42:02.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.625+0000 7fc2589d6640 1 -- 172.21.15.77:0/3630474651 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fc254063230 con 0x7fc23c0b3c20 2024-07-27T06:42:02.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.627+0000 7fc2599d8640 1 -- 172.21.15.77:0/3630474651 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc23c0b3c20 msgr2=0x7fc23c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:02.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.627+0000 7fc2599d8640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc23c0b3c20 0x7fc23c0b6130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc254108c00 tx=0x7fc24800b390 comp rx=0 tx=0).stop 2024-07-27T06:42:02.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.627+0000 7fc2599d8640 1 -- 172.21.15.77:0/3630474651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc254101af0 msgr2=0x7fc254199ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:02.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.627+0000 7fc2599d8640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc254101af0 0x7fc254199ec0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc24400b4a0 tx=0x7fc24400b960 comp rx=0 tx=0).stop 2024-07-27T06:42:02.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.627+0000 7fc2599d8640 1 -- 172.21.15.77:0/3630474651 shutdown_connections 2024-07-27T06:42:02.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.627+0000 7fc2599d8640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc254106630 0x7fc25419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:02.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.627+0000 7fc2599d8640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc254105ce0 0x7fc25419a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:02.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.627+0000 7fc2599d8640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc23c0b3c20 0x7fc23c0b6130 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:02.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.627+0000 7fc2599d8640 1 --2- 172.21.15.77:0/3630474651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc254101af0 0x7fc254199ec0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:02.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.627+0000 7fc2599d8640 1 -- 172.21.15.77:0/3630474651 >> 172.21.15.77:0/3630474651 conn(0x7fc254079870 msgr2=0x7fc254106f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:02.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.628+0000 7fc2599d8640 1 -- 172.21.15.77:0/3630474651 shutdown_connections 2024-07-27T06:42:02.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:02.628+0000 7fc2599d8640 1 -- 172.21.15.77:0/3630474651 wait complete. 2024-07-27T06:42:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:02 smithi077 ceph-mon[37153]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:42:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:42:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:42:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:02 smithi047 ceph-mon[36835]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:42:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:42:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:42:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:02 smithi029 ceph-mon[32610]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:42:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:42:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:42:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:03.258 DEBUG:teuthology.orchestra.run.smithi077:osd.11> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@osd.11.service 2024-07-27T06:42:03.261 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-07-27T06:42:03.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd stat -f json 2024-07-27T06:42:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:42:03.928 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:03 smithi029 ceph-mon[32610]: from='osd.11 [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-27T06:42:03.928 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:03 smithi029 ceph-mon[32610]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-27T06:42:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:03 smithi077 ceph-mon[37153]: from='osd.11 [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-27T06:42:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:03 smithi077 ceph-mon[37153]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-27T06:42:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:03 smithi047 ceph-mon[36835]: from='osd.11 [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-27T06:42:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:03 smithi047 ceph-mon[36835]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-27T06:42:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:04 smithi029 ceph-mon[32610]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:42:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:04 smithi029 ceph-mon[32610]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-27T06:42:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:04 smithi029 ceph-mon[32610]: osdmap e69: 12 total, 11 up, 12 in 2024-07-27T06:42:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:42:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:04 smithi029 ceph-mon[32610]: from='osd.11 [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:42:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:04 smithi029 ceph-mon[32610]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:42:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:04 smithi077 ceph-mon[37153]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:42:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:04 smithi077 ceph-mon[37153]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-27T06:42:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:04 smithi077 ceph-mon[37153]: osdmap e69: 12 total, 11 up, 12 in 2024-07-27T06:42:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:42:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:04 smithi077 ceph-mon[37153]: from='osd.11 [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:42:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:04 smithi077 ceph-mon[37153]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:42:05.107 INFO:journalctl@ceph.osd.11.smithi077.stdout:Jul 27 06:42:04 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-11[63952]: 2024-07-27T06:42:04.761+0000 7f4fda572640 -1 osd.11 0 waiting for initial osdmap 2024-07-27T06:42:05.107 INFO:journalctl@ceph.osd.11.smithi077.stdout:Jul 27 06:42:04 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-osd-11[63952]: 2024-07-27T06:42:04.775+0000 7f4fd6370640 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-27T06:42:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:04 smithi047 ceph-mon[36835]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:42:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:04 smithi047 ceph-mon[36835]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-27T06:42:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:04 smithi047 ceph-mon[36835]: osdmap e69: 12 total, 11 up, 12 in 2024-07-27T06:42:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:42:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:04 smithi047 ceph-mon[36835]: from='osd.11 [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:42:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:04 smithi047 ceph-mon[36835]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-07-27T06:42:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.386+0000 7f0ae68de640 1 -- 172.21.15.29:0/3542604933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae0108720 msgr2=0x7f0ae010ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.386+0000 7f0ae68de640 1 --2- 172.21.15.29:0/3542604933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae0108720 0x7f0ae010ef50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009be0 tx=0x7f0ad802a6c0 comp rx=0 tx=0).stop 2024-07-27T06:42:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.386+0000 7f0ae68de640 1 -- 172.21.15.29:0/3542604933 shutdown_connections 2024-07-27T06:42:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.386+0000 7f0ae68de640 1 --2- 172.21.15.29:0/3542604933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae0108720 0x7f0ae010ef50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.386+0000 7f0ae68de640 1 --2- 172.21.15.29:0/3542604933 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ae0107dd0 0x7f0ae01081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.386+0000 7f0ae68de640 1 --2- 172.21.15.29:0/3542604933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae0102490 0x7f0ae01028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.386+0000 7f0ae68de640 1 -- 172.21.15.29:0/3542604933 >> 172.21.15.29:0/3542604933 conn(0x7f0ae0079160 msgr2=0x7f0ae0079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.386+0000 7f0ae68de640 1 -- 172.21.15.29:0/3542604933 shutdown_connections 2024-07-27T06:42:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.386+0000 7f0ae68de640 1 -- 172.21.15.29:0/3542604933 wait complete. 2024-07-27T06:42:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.387+0000 7f0ae68de640 1 Processor -- start 2024-07-27T06:42:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.387+0000 7f0ae68de640 1 -- start start 2024-07-27T06:42:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.387+0000 7f0ae68de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae0102490 0x7f0ae0198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.387+0000 7f0ae68de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae0107dd0 0x7f0ae01992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.387+0000 7f0ae68de640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ae0108720 0x7f0ae019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.387+0000 7f0ae68de640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae01123a0 con 0x7f0ae0102490 2024-07-27T06:42:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.387+0000 7f0ae68de640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae0112220 con 0x7f0ae0108720 2024-07-27T06:42:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.387+0000 7f0ae68de640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae0112520 con 0x7f0ae0107dd0 2024-07-27T06:42:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.387+0000 7f0adffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae0102490 0x7f0ae0198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.387+0000 7f0adffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae0102490 0x7f0ae0198d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50450/0 (socket says 172.21.15.29:50450) 2024-07-27T06:42:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.387+0000 7f0adffff640 1 -- 172.21.15.29:0/3788883843 learned_addr learned my addr 172.21.15.29:0/3788883843 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:42:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.387+0000 7f0ad7fff640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae0107dd0 0x7f0ae01992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0ae4e7b640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ae0108720 0x7f0ae019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0adffff640 1 -- 172.21.15.29:0/3788883843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae0107dd0 msgr2=0x7f0ae01992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0adffff640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae0107dd0 0x7f0ae01992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0adffff640 1 -- 172.21.15.29:0/3788883843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ae0108720 msgr2=0x7f0ae019e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0adffff640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ae0108720 0x7f0ae019e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0adffff640 1 -- 172.21.15.29:0/3788883843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae019ecc0 con 0x7f0ae0102490 2024-07-27T06:42:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0ad7fff640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae0107dd0 0x7f0ae01992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:42:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0adffff640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae0102490 0x7f0ae0198d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0ad000e9c0 tx=0x7f0ad000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0addffb640 1 -- 172.21.15.29:0/3788883843 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad000cde0 con 0x7f0ae0102490 2024-07-27T06:42:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0ae68de640 1 -- 172.21.15.29:0/3788883843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae019efb0 con 0x7f0ae0102490 2024-07-27T06:42:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0addffb640 1 -- 172.21.15.29:0/3788883843 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ad00035f0 con 0x7f0ae0102490 2024-07-27T06:42:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0addffb640 1 -- 172.21.15.29:0/3788883843 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0010690 con 0x7f0ae0102490 2024-07-27T06:42:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.388+0000 7f0ae68de640 1 -- 172.21.15.29:0/3788883843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae01a5710 con 0x7f0ae0102490 2024-07-27T06:42:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.389+0000 7f0ae68de640 1 -- 172.21.15.29:0/3788883843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae01028a0 con 0x7f0ae0102490 2024-07-27T06:42:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.391+0000 7f0addffb640 1 -- 172.21.15.29:0/3788883843 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f0ad0004000 con 0x7f0ae0102490 2024-07-27T06:42:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.392+0000 7f0addffb640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0acc0b3c20 0x7f0acc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.393+0000 7f0ad7fff640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0acc0b3c20 0x7f0acc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.393+0000 7f0addffb640 1 -- 172.21.15.29:0/3788883843 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 7062+0+0 (secure 0 0 0) 0x7f0ad00e3be0 con 0x7f0ae0102490 2024-07-27T06:42:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.393+0000 7f0ad7fff640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0acc0b3c20 0x7f0acc0b6130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0ae01069d0 tx=0x7f0ac800b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.395+0000 7f0addffb640 1 -- 172.21.15.29:0/3788883843 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0ad00ab700 con 0x7f0ae0102490 2024-07-27T06:42:05.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.580+0000 7f0ae68de640 1 -- 172.21.15.29:0/3788883843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f0ae0199e80 con 0x7f0ae0102490 2024-07-27T06:42:05.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.581+0000 7f0addffb640 1 -- 172.21.15.29:0/3788883843 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) ==== 74+0+133 (secure 0 0 0) 0x7f0ad00ab000 con 0x7f0ae0102490 2024-07-27T06:42:05.582 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:42:05.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.584+0000 7f0ae68de640 1 -- 172.21.15.29:0/3788883843 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0acc0b3c20 msgr2=0x7f0acc0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:05.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.584+0000 7f0ae68de640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0acc0b3c20 0x7f0acc0b6130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0ae01069d0 tx=0x7f0ac800b390 comp rx=0 tx=0).stop 2024-07-27T06:42:05.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.584+0000 7f0ae68de640 1 -- 172.21.15.29:0/3788883843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae0102490 msgr2=0x7f0ae0198d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:05.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.584+0000 7f0ae68de640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae0102490 0x7f0ae0198d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0ad000e9c0 tx=0x7f0ad000ee80 comp rx=0 tx=0).stop 2024-07-27T06:42:05.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.584+0000 7f0ae68de640 1 -- 172.21.15.29:0/3788883843 shutdown_connections 2024-07-27T06:42:05.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.584+0000 7f0ae68de640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ae0108720 0x7f0ae019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:05.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.584+0000 7f0ae68de640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0acc0b3c20 0x7f0acc0b6130 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:05.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.584+0000 7f0ae68de640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ae0107dd0 0x7f0ae01992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:05.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.584+0000 7f0ae68de640 1 --2- 172.21.15.29:0/3788883843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ae0102490 0x7f0ae0198d50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:05.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.584+0000 7f0ae68de640 1 -- 172.21.15.29:0/3788883843 >> 172.21.15.29:0/3788883843 conn(0x7f0ae0079160 msgr2=0x7f0ae0104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:05.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.584+0000 7f0ae68de640 1 -- 172.21.15.29:0/3788883843 shutdown_connections 2024-07-27T06:42:05.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:05.585+0000 7f0ae68de640 1 -- 172.21.15.29:0/3788883843 wait complete. 2024-07-27T06:42:05.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:05 smithi029 ceph-mon[32610]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-07-27T06:42:05.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:05 smithi029 ceph-mon[32610]: osdmap e70: 12 total, 11 up, 12 in 2024-07-27T06:42:05.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:42:05.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:42:05.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:05 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3788883843' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-27T06:42:06.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:05 smithi077 ceph-mon[37153]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-07-27T06:42:06.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:05 smithi077 ceph-mon[37153]: osdmap e70: 12 total, 11 up, 12 in 2024-07-27T06:42:06.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:42:06.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:42:06.073 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:05 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3788883843' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-27T06:42:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:05 smithi047 ceph-mon[36835]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-07-27T06:42:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:05 smithi047 ceph-mon[36835]: osdmap e70: 12 total, 11 up, 12 in 2024-07-27T06:42:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:42:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:42:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:05 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3788883843' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-27T06:42:06.293 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":70,"num_osds":12,"num_up_osds":11,"osd_up_since":1722062501,"num_in_osds":12,"osd_in_since":1722062506,"num_remapped_pgs":0} 2024-07-27T06:42:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: purged_snaps scrub starts 2024-07-27T06:42:06.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: purged_snaps scrub ok 2024-07-27T06:42:06.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:42:06.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: osd.11 [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327] boot 2024-07-27T06:42:06.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: osdmap e71: 12 total, 12 up, 12 in 2024-07-27T06:42:06.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:42:06.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: Detected new or changed devices on smithi077 2024-07-27T06:42:06.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:06.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:06.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:42:06.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:42:06.965 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:42:06.965 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:42:06.965 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 4272M 2024-07-27T06:42:06.965 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:06.965 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:06.965 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:42:06.966 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: purged_snaps scrub starts 2024-07-27T06:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: purged_snaps scrub ok 2024-07-27T06:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: osd.11 [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327] boot 2024-07-27T06:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: osdmap e71: 12 total, 12 up, 12 in 2024-07-27T06:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: Detected new or changed devices on smithi077 2024-07-27T06:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:42:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:42:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 4272M 2024-07-27T06:42:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:42:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: purged_snaps scrub starts 2024-07-27T06:42:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: purged_snaps scrub ok 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: osd.11 [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327] boot 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: osdmap e71: 12 total, 12 up, 12 in 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: Detected new or changed devices on smithi077 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 4272M 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:42:07.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:07.295 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd stat -f json 2024-07-27T06:42:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:42:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:07 smithi029 ceph-mon[32610]: osdmap e72: 12 total, 12 up, 12 in 2024-07-27T06:42:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:07 smithi077 ceph-mon[37153]: osdmap e72: 12 total, 12 up, 12 in 2024-07-27T06:42:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:07 smithi047 ceph-mon[36835]: osdmap e72: 12 total, 12 up, 12 in 2024-07-27T06:42:09.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:08 smithi047 ceph-mon[36835]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:08 smithi077 ceph-mon[37153]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:09.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:08 smithi029 ceph-mon[32610]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.531+0000 7f97c70d5640 1 -- 172.21.15.29:0/2193674624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c0107dd0 msgr2=0x7f97c01081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.531+0000 7f97c70d5640 1 --2- 172.21.15.29:0/2193674624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c0107dd0 0x7f97c01081e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009900 tx=0x7f97b402a410 comp rx=0 tx=0).stop 2024-07-27T06:42:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.531+0000 7f97c70d5640 1 -- 172.21.15.29:0/2193674624 shutdown_connections 2024-07-27T06:42:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.531+0000 7f97c70d5640 1 --2- 172.21.15.29:0/2193674624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c0108720 0x7f97c010ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.531+0000 7f97c70d5640 1 --2- 172.21.15.29:0/2193674624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c0107dd0 0x7f97c01081e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.531+0000 7f97c70d5640 1 --2- 172.21.15.29:0/2193674624 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97c0102490 0x7f97c01028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.531+0000 7f97c70d5640 1 -- 172.21.15.29:0/2193674624 >> 172.21.15.29:0/2193674624 conn(0x7f97c0079160 msgr2=0x7f97c0079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.531+0000 7f97c70d5640 1 -- 172.21.15.29:0/2193674624 shutdown_connections 2024-07-27T06:42:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.532+0000 7f97c70d5640 1 -- 172.21.15.29:0/2193674624 wait complete. 2024-07-27T06:42:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.532+0000 7f97c70d5640 1 Processor -- start 2024-07-27T06:42:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.532+0000 7f97c70d5640 1 -- start start 2024-07-27T06:42:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.532+0000 7f97c70d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c0102490 0x7f97c0199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.532+0000 7f97c70d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c0107dd0 0x7f97c019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c70d5640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97c0108720 0x7f97c019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c70d5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f97c01134b0 con 0x7f97c0102490 2024-07-27T06:42:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c70d5640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f97c0113330 con 0x7f97c0108720 2024-07-27T06:42:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c70d5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f97c0113630 con 0x7f97c0107dd0 2024-07-27T06:42:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c4e71640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c0102490 0x7f97c0199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c4e71640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c0102490 0x7f97c0199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57718/0 (socket says 172.21.15.29:57718) 2024-07-27T06:42:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c4e71640 1 -- 172.21.15.29:0/1175484726 learned_addr learned my addr 172.21.15.29:0/1175484726 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:42:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c5672640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97c0108720 0x7f97c019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97bffff640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c0107dd0 0x7f97c019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c4e71640 1 -- 172.21.15.29:0/1175484726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c0107dd0 msgr2=0x7f97c019a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:09.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c4e71640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c0107dd0 0x7f97c019a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c4e71640 1 -- 172.21.15.29:0/1175484726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97c0108720 msgr2=0x7f97c019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c4e71640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97c0108720 0x7f97c019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c4e71640 1 -- 172.21.15.29:0/1175484726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c019feb0 con 0x7f97c0102490 2024-07-27T06:42:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c5672640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97c0108720 0x7f97c019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:42:09.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c4e71640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c0102490 0x7f97c0199f40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f97b000e9f0 tx=0x7f97b000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97bdffb640 1 -- 172.21.15.29:0/1175484726 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b000cd80 con 0x7f97c0102490 2024-07-27T06:42:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97bdffb640 1 -- 172.21.15.29:0/1175484726 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97b00108e0 con 0x7f97c0102490 2024-07-27T06:42:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97bdffb640 1 -- 172.21.15.29:0/1175484726 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0018980 con 0x7f97c0102490 2024-07-27T06:42:09.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.533+0000 7f97c70d5640 1 -- 172.21.15.29:0/1175484726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c01a01a0 con 0x7f97c0102490 2024-07-27T06:42:09.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.534+0000 7f97c70d5640 1 -- 172.21.15.29:0/1175484726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c01a6900 con 0x7f97c0102490 2024-07-27T06:42:09.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.535+0000 7f97c70d5640 1 -- 172.21.15.29:0/1175484726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c01028a0 con 0x7f97c0102490 2024-07-27T06:42:09.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.536+0000 7f97bdffb640 1 -- 172.21.15.29:0/1175484726 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f97b0010430 con 0x7f97c0102490 2024-07-27T06:42:09.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.537+0000 7f97bdffb640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f97a80b3ca0 0x7f97a80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:09.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.537+0000 7f97bffff640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f97a80b3ca0 0x7f97a80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:09.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.537+0000 7f97bdffb640 1 -- 172.21.15.29:0/1175484726 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f97b00e4190 con 0x7f97c0102490 2024-07-27T06:42:09.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.538+0000 7f97bffff640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f97a80b3ca0 0x7f97a80b61b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f97b4000c00 tx=0x7f97b4031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:09.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.540+0000 7f97bdffb640 1 -- 172.21.15.29:0/1175484726 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f97b00abb10 con 0x7f97c0102490 2024-07-27T06:42:09.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.725+0000 7f97c70d5640 1 -- 172.21.15.29:0/1175484726 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f97c0109f80 con 0x7f97c0102490 2024-07-27T06:42:09.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.725+0000 7f97bdffb640 1 -- 172.21.15.29:0/1175484726 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7f97b00ab410 con 0x7f97c0102490 2024-07-27T06:42:09.727 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:42:09.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.727+0000 7f97c70d5640 1 -- 172.21.15.29:0/1175484726 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f97a80b3ca0 msgr2=0x7f97a80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:09.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.727+0000 7f97c70d5640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f97a80b3ca0 0x7f97a80b61b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f97b4000c00 tx=0x7f97b4031400 comp rx=0 tx=0).stop 2024-07-27T06:42:09.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.727+0000 7f97c70d5640 1 -- 172.21.15.29:0/1175484726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c0102490 msgr2=0x7f97c0199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.727+0000 7f97c70d5640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c0102490 0x7f97c0199f40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f97b000e9f0 tx=0x7f97b000eeb0 comp rx=0 tx=0).stop 2024-07-27T06:42:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.727+0000 7f97c70d5640 1 -- 172.21.15.29:0/1175484726 shutdown_connections 2024-07-27T06:42:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.727+0000 7f97c70d5640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97c0108720 0x7f97c019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.727+0000 7f97c70d5640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f97a80b3ca0 0x7f97a80b61b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.727+0000 7f97c70d5640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f97c0107dd0 0x7f97c019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.727+0000 7f97c70d5640 1 --2- 172.21.15.29:0/1175484726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97c0102490 0x7f97c0199f40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.727+0000 7f97c70d5640 1 -- 172.21.15.29:0/1175484726 >> 172.21.15.29:0/1175484726 conn(0x7f97c0079160 msgr2=0x7f97c0104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.727+0000 7f97c70d5640 1 -- 172.21.15.29:0/1175484726 shutdown_connections 2024-07-27T06:42:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:09.728+0000 7f97c70d5640 1 -- 172.21.15.29:0/1175484726 wait complete. 2024-07-27T06:42:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:09 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1175484726' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-27T06:42:10.140 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:09 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1175484726' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-27T06:42:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:09 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1175484726' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-27T06:42:10.571 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1722062525,"num_in_osds":12,"osd_in_since":1722062506,"num_remapped_pgs":0} 2024-07-27T06:42:10.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd dump --format=json 2024-07-27T06:42:10.843 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:42:11.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:10 smithi029 ceph-mon[32610]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:10 smithi077 ceph-mon[37153]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:10 smithi047 ceph-mon[36835]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.749+0000 7f073936a640 1 -- 172.21.15.29:0/2584481008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0734101af0 msgr2=0x7f0734101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.749+0000 7f073936a640 1 --2- 172.21.15.29:0/2584481008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0734101af0 0x7f0734101f00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0724009ca0 tx=0x7f072402a8c0 comp rx=0 tx=0).stop 2024-07-27T06:42:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.749+0000 7f073936a640 1 -- 172.21.15.29:0/2584481008 shutdown_connections 2024-07-27T06:42:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.749+0000 7f073936a640 1 --2- 172.21.15.29:0/2584481008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0734106630 0x7f07341100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.749+0000 7f073936a640 1 --2- 172.21.15.29:0/2584481008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0734105ce0 0x7f07341060f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.749+0000 7f073936a640 1 --2- 172.21.15.29:0/2584481008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0734101af0 0x7f0734101f00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.749+0000 7f073936a640 1 -- 172.21.15.29:0/2584481008 >> 172.21.15.29:0/2584481008 conn(0x7f0734079870 msgr2=0x7f0734079c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.749+0000 7f073936a640 1 -- 172.21.15.29:0/2584481008 shutdown_connections 2024-07-27T06:42:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.749+0000 7f073936a640 1 -- 172.21.15.29:0/2584481008 wait complete. 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.750+0000 7f073936a640 1 Processor -- start 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.750+0000 7f073936a640 1 -- start start 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f073936a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0734101af0 0x7f0734102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f073936a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0734105ce0 0x7f0734102fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f073936a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0734106630 0x7f0734103510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f073936a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0734113590 con 0x7f0734101af0 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f073936a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f0734113410 con 0x7f0734105ce0 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f073936a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0734113710 con 0x7f0734106630 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f0732ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0734101af0 0x7f0734102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f0732ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0734101af0 0x7f0734102a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57738/0 (socket says 172.21.15.29:57738) 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f0732ffd640 1 -- 172.21.15.29:0/3199682520 learned_addr learned my addr 172.21.15.29:0/3199682520 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f07327fc640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0734105ce0 0x7f0734102fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f07337fe640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0734106630 0x7f0734103510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f0732ffd640 1 -- 172.21.15.29:0/3199682520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0734106630 msgr2=0x7f0734103510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f0732ffd640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0734106630 0x7f0734103510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f0732ffd640 1 -- 172.21.15.29:0/3199682520 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0734105ce0 msgr2=0x7f0734102fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f0732ffd640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0734105ce0 0x7f0734102fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f0732ffd640 1 -- 172.21.15.29:0/3199682520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07341a81d0 con 0x7f0734101af0 2024-07-27T06:42:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f07337fe640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0734106630 0x7f0734103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:42:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.751+0000 7f0732ffd640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0734101af0 0x7f0734102a90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f072402ade0 tx=0x7f0724003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.752+0000 7f071bfff640 1 -- 172.21.15.29:0/3199682520 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724003d60 con 0x7f0734101af0 2024-07-27T06:42:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.752+0000 7f071bfff640 1 -- 172.21.15.29:0/3199682520 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07240044f0 con 0x7f0734101af0 2024-07-27T06:42:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.752+0000 7f073936a640 1 -- 172.21.15.29:0/3199682520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07341a8460 con 0x7f0734101af0 2024-07-27T06:42:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.752+0000 7f071bfff640 1 -- 172.21.15.29:0/3199682520 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072402fd10 con 0x7f0734101af0 2024-07-27T06:42:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.752+0000 7f073936a640 1 -- 172.21.15.29:0/3199682520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07341a8a20 con 0x7f0734101af0 2024-07-27T06:42:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.753+0000 7f073936a640 1 -- 172.21.15.29:0/3199682520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0734101f00 con 0x7f0734101af0 2024-07-27T06:42:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.755+0000 7f071bfff640 1 -- 172.21.15.29:0/3199682520 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f07240046a0 con 0x7f0734101af0 2024-07-27T06:42:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.756+0000 7f071bfff640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f071c0b3c20 0x7f071c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.756+0000 7f07327fc640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f071c0b3c20 0x7f071c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.756+0000 7f071bfff640 1 -- 172.21.15.29:0/3199682520 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f07241052a0 con 0x7f0734101af0 2024-07-27T06:42:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.756+0000 7f07327fc640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f071c0b3c20 0x7f071c0b6130 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0728005e10 tx=0x7f0728005da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:12.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.759+0000 7f071bfff640 1 -- 172.21.15.29:0/3199682520 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f07240ccc20 con 0x7f0734101af0 2024-07-27T06:42:12.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.947+0000 7f073936a640 1 -- 172.21.15.29:0/3199682520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f073402e0c0 con 0x7f0734101af0 2024-07-27T06:42:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.948+0000 7f071bfff640 1 -- 172.21.15.29:0/3199682520 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18701 (secure 0 0 0) 0x7f07240cc520 con 0x7f0734101af0 2024-07-27T06:42:12.950 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:42:12.950 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":72,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","created":"2024-07-27T06:33:57.535327+0000","modified":"2024-07-27T06:42:06.764177+0000","last_up_change":"2024-07-27T06:42:05.757156+0000","last_in_change":"2024-07-27T06:41:46.423732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T06:38:21.606527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06833d24-2e1a-4199-9338-7d6883d1bf9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6803","nonce":386121641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6805","nonce":386121641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6809","nonce":386121641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6807","nonce":386121641}]},"public_addr":"172.21.15.29:6803/386121641","cluster_addr":"172.21.15.29:6805/386121641","heartbeat_back_addr":"172.21.15.29:6809/386121641","heartbeat_front_addr":"172.21.15.29:6807/386121641","state":["exists","up"]},{"osd":1,"uuid":"906aba19-80dc-41b5-953c-af6dca604b05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6801","nonce":2795953749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6803","nonce":2795953749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6807","nonce":2795953749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6805","nonce":2795953749}]},"public_addr":"172.21.15.47:6801/2795953749","cluster_addr":"172.21.15.47:6803/2795953749","heartbeat_back_addr":"172.21.15.47:6807/2795953749","heartbeat_front_addr":"172.21.15.47:6805/2795953749","state":["exists","up"]},{"osd":2,"uuid":"194ae5a5-aebf-4cff-9b99-585a3073a269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6801","nonce":2202565078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6803","nonce":2202565078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6807","nonce":2202565078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6805","nonce":2202565078}]},"public_addr":"172.21.15.77:6801/2202565078","cluster_addr":"172.21.15.77:6803/2202565078","heartbeat_back_addr":"172.21.15.77:6807/2202565078","heartbeat_front_addr":"172.21.15.77:6805/2202565078","state":["exists","up"]},{"osd":3,"uuid":"3c22ce9c-963e-4f72-86e5-360e781526db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6811","nonce":3757476312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6813","nonce":3757476312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6817","nonce":3757476312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6815","nonce":3757476312}]},"public_addr":"172.21.15.29:6811/3757476312","cluster_addr":"172.21.15.29:6813/3757476312","heartbeat_back_addr":"172.21.15.29:6817/3757476312","heartbeat_front_addr":"172.21.15.29:6815/3757476312","state":["exists","up"]},{"osd":4,"uuid":"cd4b585f-e7a0-4d82-873d-3a9269ebac3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6809","nonce":1379363720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6811","nonce":1379363720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6815","nonce":1379363720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6813","nonce":1379363720}]},"public_addr":"172.21.15.47:6809/1379363720","cluster_addr":"172.21.15.47:6811/1379363720","heartbeat_back_addr":"172.21.15.47:6815/1379363720","heartbeat_front_addr":"172.21.15.47:6813/1379363720","state":["exists","up"]},{"osd":5,"uuid":"40a3eb14-dfaa-4523-8c28-d74c19293c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6809","nonce":1338438780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6811","nonce":1338438780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6815","nonce":1338438780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6813","nonce":1338438780}]},"public_addr":"172.21.15.77:6809/1338438780","cluster_addr":"172.21.15.77:6811/1338438780","heartbeat_back_addr":"172.21.15.77:6815/1338438780","heartbeat_front_addr":"172.21.15.77:6813/1338438780","state":["exists","up"]},{"osd":6,"uuid":"91e85be2-10d1-4209-aa33-17da8f8bd8bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6819","nonce":2679402148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6821","nonce":2679402148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6825","nonce":2679402148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6823","nonce":2679402148}]},"public_addr":"172.21.15.29:6819/2679402148","cluster_addr":"172.21.15.29:6821/2679402148","heartbeat_back_addr":"172.21.15.29:6825/2679402148","heartbeat_front_addr":"172.21.15.29:6823/2679402148","state":["exists","up"]},{"osd":7,"uuid":"c655b450-0cc4-45d4-95ea-93a57fe8e4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6817","nonce":2439195589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6819","nonce":2439195589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6823","nonce":2439195589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6821","nonce":2439195589}]},"public_addr":"172.21.15.47:6817/2439195589","cluster_addr":"172.21.15.47:6819/2439195589","heartbeat_back_addr":"172.21.15.47:6823/2439195589","heartbeat_front_addr":"172.21.15.47:6821/2439195589","state":["exists","up"]},{"osd":8,"uuid":"a7a95b51-f32c-433e-a197-feb6d47b6c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6817","nonce":1422290905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6819","nonce":1422290905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6823","nonce":1422290905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6821","nonce":1422290905}]},"public_addr":"172.21.15.77:6817/1422290905","cluster_addr":"172.21.15.77:6819/1422290905","heartbeat_back_addr":"172.21.15.77:6823/1422290905","heartbeat_front_addr":"172.21.15.77:6821/1422290905","state":["exists","up"]},{"osd":9,"uuid":"f352242b-0d2e-41b3-9c83-436cf432b0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6827","nonce":1307316384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6829","nonce":1307316384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6833","nonce":1307316384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6831","nonce":1307316384}]},"public_addr":"172.21.15.29:6827/1307316384","cluster_addr":"172.21.15.29:6829/1307316384","heartbeat_back_addr":"172.21.15.29:6833/1307316384","heartbeat_front_addr":"172.21.15.29:6831/1307316384","state":["exists","up"]},{"osd":10,"uuid":"4148c2df-c3e9-4a47-88f7-36cf3eab8d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6825","nonce":2133726850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6827","nonce":2133726850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6831","nonce":2133726850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6829","nonce":2133726850}]},"public_addr":"172.21.15.47:6825/2133726850","cluster_addr":"172.21.15.47:6827/2133726850","heartbeat_back_addr":"172.21.15.47:6831/2133726850","heartbeat_front_addr":"172.21.15.47:6829/2133726850","state":["exists","up"]},{"osd":11,"uuid":"5e8b1da3-5f65-4fe9-8722-b35921cbd8c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6825","nonce":3228299327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6827","nonce":3228299327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6831","nonce":3228299327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6829","nonce":3228299327}]},"public_addr":"172.21.15.77:6825/3228299327","cluster_addr":"172.21.15.77:6827/3228299327","heartbeat_back_addr":"172.21.15.77:6831/3228299327","heartbeat_front_addr":"172.21.15.77:6829/3228299327","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:32.525206+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:56.489640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:18.887845+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:43.727678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:08.755553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:32.923461+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:59.298439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:23.944848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:48.883050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:14.566310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:39.557809+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:42:03.813903+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2378279024":"2024-07-28T06:36:07.527503+0000","172.21.15.29:6801/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3527133965":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2262661620":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/3156863587":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2424241521":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3881857930":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/398508875":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6801/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6800/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2679856281":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6801/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2708687265":"2024-07-28T06:34:58.250198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T06:42:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.950+0000 7f073936a640 1 -- 172.21.15.29:0/3199682520 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f071c0b3c20 msgr2=0x7f071c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.950+0000 7f073936a640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f071c0b3c20 0x7f071c0b6130 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0728005e10 tx=0x7f0728005da0 comp rx=0 tx=0).stop 2024-07-27T06:42:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.950+0000 7f073936a640 1 -- 172.21.15.29:0/3199682520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0734101af0 msgr2=0x7f0734102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.950+0000 7f073936a640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0734101af0 0x7f0734102a90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f072402ade0 tx=0x7f0724003970 comp rx=0 tx=0).stop 2024-07-27T06:42:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.950+0000 7f073936a640 1 -- 172.21.15.29:0/3199682520 shutdown_connections 2024-07-27T06:42:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.950+0000 7f073936a640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f071c0b3c20 0x7f071c0b6130 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.950+0000 7f073936a640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0734106630 0x7f0734103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.950+0000 7f073936a640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0734105ce0 0x7f0734102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.950+0000 7f073936a640 1 --2- 172.21.15.29:0/3199682520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0734101af0 0x7f0734102a90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.950+0000 7f073936a640 1 -- 172.21.15.29:0/3199682520 >> 172.21.15.29:0/3199682520 conn(0x7f0734079870 msgr2=0x7f0734107b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.951+0000 7f073936a640 1 -- 172.21.15.29:0/3199682520 shutdown_connections 2024-07-27T06:42:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:12.951+0000 7f073936a640 1 -- 172.21.15.29:0/3199682520 wait complete. 2024-07-27T06:42:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:12 smithi077 ceph-mon[37153]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:12 smithi047 ceph-mon[36835]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:13.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:12 smithi029 ceph-mon[32610]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:13.773 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-27T06:38:21.606527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-27T06:42:13.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-07-27T06:42:14.045 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:42:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:13 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3199682520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:42:14.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:13 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3199682520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:42:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:13 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3199682520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:42:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:14 smithi077 ceph-mon[37153]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:14 smithi047 ceph-mon[36835]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:15.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:14 smithi029 ceph-mon[32610]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:15.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.839+0000 7f40153fa640 1 -- 172.21.15.29:0/2531763041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010107dd0 msgr2=0x7f40101081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.839+0000 7f40153fa640 1 --2- 172.21.15.29:0/2531763041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010107dd0 0x7f40101081e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4004009900 tx=0x7f400402a410 comp rx=0 tx=0).stop 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.839+0000 7f40153fa640 1 -- 172.21.15.29:0/2531763041 shutdown_connections 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.839+0000 7f40153fa640 1 --2- 172.21.15.29:0/2531763041 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4010108720 0x7f401010ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.839+0000 7f40153fa640 1 --2- 172.21.15.29:0/2531763041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010107dd0 0x7f40101081e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.839+0000 7f40153fa640 1 --2- 172.21.15.29:0/2531763041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4010102490 0x7f40101028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.839+0000 7f40153fa640 1 -- 172.21.15.29:0/2531763041 >> 172.21.15.29:0/2531763041 conn(0x7f4010079150 msgr2=0x7f4010079560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.839+0000 7f40153fa640 1 -- 172.21.15.29:0/2531763041 shutdown_connections 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.839+0000 7f40153fa640 1 -- 172.21.15.29:0/2531763041 wait complete. 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.840+0000 7f40153fa640 1 Processor -- start 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.840+0000 7f40153fa640 1 -- start start 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.840+0000 7f40153fa640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4010102490 0x7f4010199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.840+0000 7f40153fa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4010107dd0 0x7f401019a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.840+0000 7f40153fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010108720 0x7f401019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.840+0000 7f40153fa640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f40101134f0 con 0x7f4010108720 2024-07-27T06:42:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.840+0000 7f40153fa640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f4010113370 con 0x7f4010102490 2024-07-27T06:42:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.840+0000 7f40153fa640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4010113670 con 0x7f4010107dd0 2024-07-27T06:42:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.840+0000 7f400f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010108720 0x7f401019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f400f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010108720 0x7f401019f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57768/0 (socket says 172.21.15.29:57768) 2024-07-27T06:42:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f400f7fe640 1 -- 172.21.15.29:0/1728877835 learned_addr learned my addr 172.21.15.29:0/1728877835 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:42:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f400effd640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4010102490 0x7f4010199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f400e7fc640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4010107dd0 0x7f401019a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f400f7fe640 1 -- 172.21.15.29:0/1728877835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4010107dd0 msgr2=0x7f401019a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f400f7fe640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4010107dd0 0x7f401019a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f400f7fe640 1 -- 172.21.15.29:0/1728877835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4010102490 msgr2=0x7f4010199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f400f7fe640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4010102490 0x7f4010199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f400f7fe640 1 -- 172.21.15.29:0/1728877835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401019ff50 con 0x7f4010108720 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f400f7fe640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010108720 0x7f401019f820 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4008009fd0 tx=0x7f400800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f3ff7fff640 1 -- 172.21.15.29:0/1728877835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008011830 con 0x7f4010108720 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f3ff7fff640 1 -- 172.21.15.29:0/1728877835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4008016070 con 0x7f4010108720 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f40153fa640 1 -- 172.21.15.29:0/1728877835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40101a0240 con 0x7f4010108720 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f3ff7fff640 1 -- 172.21.15.29:0/1728877835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008010de0 con 0x7f4010108720 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.841+0000 7f40153fa640 1 -- 172.21.15.29:0/1728877835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40101a69a0 con 0x7f4010108720 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.842+0000 7f40153fa640 1 -- 172.21.15.29:0/1728877835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4010103710 con 0x7f4010108720 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.845+0000 7f3ff7fff640 1 -- 172.21.15.29:0/1728877835 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f40080119d0 con 0x7f4010108720 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.846+0000 7f3ff7fff640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3ff80b3c20 0x7f3ff80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.846+0000 7f400effd640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3ff80b3c20 0x7f3ff80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.846+0000 7f3ff7fff640 1 -- 172.21.15.29:0/1728877835 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f40080e3560 con 0x7f4010108720 2024-07-27T06:42:15.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.846+0000 7f400effd640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3ff80b3c20 0x7f3ff80b6130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4000005fd0 tx=0x7f4000009680 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:15.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:15.849+0000 7f3ff7fff640 1 -- 172.21.15.29:0/1728877835 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f40080aaf80 con 0x7f4010108720 2024-07-27T06:42:16.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.030+0000 7f40153fa640 1 -- 172.21.15.29:0/1728877835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f401019b110 con 0x7f4010108720 2024-07-27T06:42:16.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.031+0000 7f3ff7fff640 1 -- 172.21.15.29:0/1728877835 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f40080b3250 con 0x7f4010108720 2024-07-27T06:42:16.033 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-07-27T06:42:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.033+0000 7f40153fa640 1 -- 172.21.15.29:0/1728877835 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3ff80b3c20 msgr2=0x7f3ff80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.033+0000 7f40153fa640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3ff80b3c20 0x7f3ff80b6130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4000005fd0 tx=0x7f4000009680 comp rx=0 tx=0).stop 2024-07-27T06:42:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.033+0000 7f40153fa640 1 -- 172.21.15.29:0/1728877835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010108720 msgr2=0x7f401019f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.033+0000 7f40153fa640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010108720 0x7f401019f820 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4008009fd0 tx=0x7f400800b8e0 comp rx=0 tx=0).stop 2024-07-27T06:42:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.033+0000 7f40153fa640 1 -- 172.21.15.29:0/1728877835 shutdown_connections 2024-07-27T06:42:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.033+0000 7f40153fa640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4010108720 0x7f401019f820 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.033+0000 7f40153fa640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3ff80b3c20 0x7f3ff80b6130 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.033+0000 7f40153fa640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4010107dd0 0x7f401019a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:16.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.033+0000 7f40153fa640 1 --2- 172.21.15.29:0/1728877835 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4010102490 0x7f4010199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:16.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.033+0000 7f40153fa640 1 -- 172.21.15.29:0/1728877835 >> 172.21.15.29:0/1728877835 conn(0x7f4010079150 msgr2=0x7f4010103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:16.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.033+0000 7f40153fa640 1 -- 172.21.15.29:0/1728877835 shutdown_connections 2024-07-27T06:42:16.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:16.033+0000 7f40153fa640 1 -- 172.21.15.29:0/1728877835 wait complete. 2024-07-27T06:42:16.721 INFO:tasks.cephadm:Adding mds.a on smithi029 2024-07-27T06:42:16.721 INFO:tasks.cephadm:Adding mds.d on smithi029 2024-07-27T06:42:16.721 INFO:tasks.cephadm:Adding mds.g on smithi029 2024-07-27T06:42:16.721 INFO:tasks.cephadm:Adding mds.j on smithi029 2024-07-27T06:42:16.722 INFO:tasks.cephadm:Adding mds.b on smithi047 2024-07-27T06:42:16.722 INFO:tasks.cephadm:Adding mds.e on smithi047 2024-07-27T06:42:16.722 INFO:tasks.cephadm:Adding mds.h on smithi047 2024-07-27T06:42:16.722 INFO:tasks.cephadm:Adding mds.k on smithi047 2024-07-27T06:42:16.722 INFO:tasks.cephadm:Adding mds.c on smithi077 2024-07-27T06:42:16.722 INFO:tasks.cephadm:Adding mds.f on smithi077 2024-07-27T06:42:16.722 INFO:tasks.cephadm:Adding mds.i on smithi077 2024-07-27T06:42:16.722 INFO:tasks.cephadm:Adding mds.l on smithi077 2024-07-27T06:42:16.722 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph orch apply mds all '12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l' 2024-07-27T06:42:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:16 smithi029 ceph-mon[32610]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:16 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1728877835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-27T06:42:16.986 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.c/config 2024-07-27T06:42:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:16 smithi077 ceph-mon[37153]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:16 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1728877835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-27T06:42:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:16 smithi047 ceph-mon[36835]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:16 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1728877835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-27T06:42:18.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.545+0000 7f483e005640 1 -- 172.21.15.77:0/1472759143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4838101810 msgr2=0x7f4838101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:18.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.545+0000 7f483e005640 1 --2- 172.21.15.77:0/1472759143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4838101810 0x7f4838101c20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4820009900 tx=0x7f482002a410 comp rx=0 tx=0).stop 2024-07-27T06:42:18.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.545+0000 7f483e005640 1 -- 172.21.15.77:0/1472759143 shutdown_connections 2024-07-27T06:42:18.577 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.545+0000 7f483e005640 1 --2- 172.21.15.77:0/1472759143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4838102160 0x7f483810ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.545+0000 7f483e005640 1 --2- 172.21.15.77:0/1472759143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4838101810 0x7f4838101c20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.545+0000 7f483e005640 1 --2- 172.21.15.77:0/1472759143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4838108120 0x7f4838108530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.545+0000 7f483e005640 1 -- 172.21.15.77:0/1472759143 >> 172.21.15.77:0/1472759143 conn(0x7f4838077690 msgr2=0x7f4838077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.545+0000 7f483e005640 1 -- 172.21.15.77:0/1472759143 shutdown_connections 2024-07-27T06:42:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.545+0000 7f483e005640 1 -- 172.21.15.77:0/1472759143 wait complete. 2024-07-27T06:42:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.545+0000 7f483e005640 1 Processor -- start 2024-07-27T06:42:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f483e005640 1 -- start start 2024-07-27T06:42:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f483e005640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4838101810 0x7f4838199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f483e005640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4838102160 0x7f483819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f483e005640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4838108120 0x7f483819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f483e005640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4838113510 con 0x7f4838108120 2024-07-27T06:42:18.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f483e005640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f4838113390 con 0x7f4838102160 2024-07-27T06:42:18.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f483e005640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4838113690 con 0x7f4838101810 2024-07-27T06:42:18.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f48377fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4838101810 0x7f4838199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:18.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f48377fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4838101810 0x7f4838199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55056/0 (socket says 172.21.15.77:55056) 2024-07-27T06:42:18.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f48377fe640 1 -- 172.21.15.77:0/3287545260 learned_addr learned my addr 172.21.15.77:0/3287545260 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-07-27T06:42:18.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f4836ffd640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4838102160 0x7f483819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:18.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f4837fff640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4838108120 0x7f483819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:18.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f48377fe640 1 -- 172.21.15.77:0/3287545260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4838102160 msgr2=0x7f483819a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:18.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f48377fe640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4838102160 0x7f483819a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f48377fe640 1 -- 172.21.15.77:0/3287545260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4838108120 msgr2=0x7f483819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f48377fe640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4838108120 0x7f483819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f48377fe640 1 -- 172.21.15.77:0/3287545260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f483819fe90 con 0x7f4838101810 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.546+0000 7f4837fff640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4838108120 0x7f483819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.547+0000 7f48377fe640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4838101810 0x7f4838199f20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f483000ba50 tx=0x7f483000bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.547+0000 7f4834ff9640 1 -- 172.21.15.77:0/3287545260 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830019070 con 0x7f4838101810 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.547+0000 7f4834ff9640 1 -- 172.21.15.77:0/3287545260 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f483000cbe0 con 0x7f4838101810 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.547+0000 7f483e005640 1 -- 172.21.15.77:0/3287545260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48381a0180 con 0x7f4838101810 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.547+0000 7f4834ff9640 1 -- 172.21.15.77:0/3287545260 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48300075d0 con 0x7f4838101810 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.547+0000 7f483e005640 1 -- 172.21.15.77:0/3287545260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48381a68e0 con 0x7f4838101810 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.548+0000 7f483e005640 1 -- 172.21.15.77:0/3287545260 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4838101c20 con 0x7f4838101810 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.549+0000 7f4834ff9640 1 -- 172.21.15.77:0/3287545260 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f483001c070 con 0x7f4838101810 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.550+0000 7f4834ff9640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f48240b3ca0 0x7f48240b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:18.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.550+0000 7f4836ffd640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f48240b3ca0 0x7f48240b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:18.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.551+0000 7f4834ff9640 1 -- 172.21.15.77:0/3287545260 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f48300e3390 con 0x7f4838101810 2024-07-27T06:42:18.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.551+0000 7f4836ffd640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f48240b3ca0 0x7f48240b61b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4820009230 tx=0x7f4820031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:18.581 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.554+0000 7f4834ff9640 1 -- 172.21.15.77:0/3287545260 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f48300aa6c0 con 0x7f4838101810 2024-07-27T06:42:18.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.737+0000 7f483e005640 1 -- 172.21.15.77:0/3287545260 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l", "target": ["mon-mgr", ""]}) -- 0x7f4838063230 con 0x7f48240b3ca0 2024-07-27T06:42:18.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.746+0000 7f4834ff9640 1 -- 172.21.15.77:0/3287545260 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f4838063230 con 0x7f48240b3ca0 2024-07-27T06:42:18.747 INFO:teuthology.orchestra.run.smithi077.stdout:Scheduled mds.all update... 2024-07-27T06:42:18.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.748+0000 7f483e005640 1 -- 172.21.15.77:0/3287545260 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f48240b3ca0 msgr2=0x7f48240b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:18.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.748+0000 7f483e005640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f48240b3ca0 0x7f48240b61b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4820009230 tx=0x7f4820031400 comp rx=0 tx=0).stop 2024-07-27T06:42:18.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.748+0000 7f483e005640 1 -- 172.21.15.77:0/3287545260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4838101810 msgr2=0x7f4838199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:18.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.748+0000 7f483e005640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4838101810 0x7f4838199f20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f483000ba50 tx=0x7f483000bf10 comp rx=0 tx=0).stop 2024-07-27T06:42:18.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.748+0000 7f483e005640 1 -- 172.21.15.77:0/3287545260 shutdown_connections 2024-07-27T06:42:18.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.748+0000 7f483e005640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4838108120 0x7f483819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:18.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.748+0000 7f483e005640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4838102160 0x7f483819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:18.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.748+0000 7f483e005640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f48240b3ca0 0x7f48240b61b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:18.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.748+0000 7f483e005640 1 --2- 172.21.15.77:0/3287545260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4838101810 0x7f4838199f20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:18.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.748+0000 7f483e005640 1 -- 172.21.15.77:0/3287545260 >> 172.21.15.77:0/3287545260 conn(0x7f4838077690 msgr2=0x7f4838104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:18.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.748+0000 7f483e005640 1 -- 172.21.15.77:0/3287545260 shutdown_connections 2024-07-27T06:42:18.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-07-27T06:42:18.748+0000 7f483e005640 1 -- 172.21.15.77:0/3287545260 wait complete. 2024-07-27T06:42:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:18 smithi077 ceph-mon[37153]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:42:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:18 smithi047 ceph-mon[36835]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:42:19.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:18 smithi029 ceph-mon[32610]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:42:19.349 DEBUG:teuthology.orchestra.run.smithi029:mds.a> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.a.service 2024-07-27T06:42:19.352 DEBUG:teuthology.orchestra.run.smithi029:mds.d> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service 2024-07-27T06:42:19.396 DEBUG:teuthology.orchestra.run.smithi029:mds.g> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.g.service 2024-07-27T06:42:19.439 DEBUG:teuthology.orchestra.run.smithi029:mds.j> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service 2024-07-27T06:42:19.482 DEBUG:teuthology.orchestra.run.smithi047:mds.b> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.b.service 2024-07-27T06:42:19.485 DEBUG:teuthology.orchestra.run.smithi047:mds.e> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.e.service 2024-07-27T06:42:19.528 DEBUG:teuthology.orchestra.run.smithi047:mds.h> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service 2024-07-27T06:42:19.570 DEBUG:teuthology.orchestra.run.smithi047:mds.k> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.k.service 2024-07-27T06:42:19.614 DEBUG:teuthology.orchestra.run.smithi077:mds.c> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.c.service 2024-07-27T06:42:19.616 DEBUG:teuthology.orchestra.run.smithi077:mds.f> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.f.service 2024-07-27T06:42:19.659 DEBUG:teuthology.orchestra.run.smithi077:mds.i> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service 2024-07-27T06:42:19.701 DEBUG:teuthology.orchestra.run.smithi077:mds.l> sudo journalctl -f -n 0 -u ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.l.service 2024-07-27T06:42:19.744 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-07-27T06:42:19.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:42:19.966 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:19 smithi077 ceph-mon[37153]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:42:19.966 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:19 smithi077 ceph-mon[37153]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l;count:12 2024-07-27T06:42:19.966 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:19.966 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:42:19.966 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:19.967 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:19.967 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:19.967 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:20.008 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:42:20.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:19 smithi029 ceph-mon[32610]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:42:20.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:19 smithi029 ceph-mon[32610]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l;count:12 2024-07-27T06:42:20.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:20.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:42:20.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:20.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:20.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:20.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:19 smithi047 ceph-mon[36835]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:42:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:19 smithi047 ceph-mon[36835]: Saving service mds.all spec with placement smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l;count:12 2024-07-27T06:42:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:42:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:20.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:20.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:20.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:20.948 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:20 smithi077 ceph-mon[37153]: Deploying daemon mds.l on smithi077 2024-07-27T06:42:20.948 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:20 smithi077 ceph-mon[37153]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:21.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:20 smithi029 ceph-mon[32610]: Deploying daemon mds.l on smithi077 2024-07-27T06:42:21.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:20 smithi029 ceph-mon[32610]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:20 smithi047 ceph-mon[36835]: Deploying daemon mds.l on smithi077 2024-07-27T06:42:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:20 smithi047 ceph-mon[36835]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:21.596 INFO:journalctl@ceph.mds.l.smithi077.stdout:Jul 27 06:42:21 smithi077 systemd[1]: Starting Ceph mds.l for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:42:21.847 INFO:journalctl@ceph.mds.l.smithi077.stdout:Jul 27 06:42:21 smithi077 podman[68427]: 2024-07-27 06:42:21.497536392 +0000 UTC m=+0.017028310 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:42:21.847 INFO:journalctl@ceph.mds.l.smithi077.stdout:Jul 27 06:42:21 smithi077 podman[68427]: 2024-07-27 06:42:21.64193921 +0000 UTC m=+0.161431118 container create 3ba94332af1944bab5048fbc95e409d7150c1b81b78b8149b27e2243b17ac8c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-l, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-07-27T06:42:21.847 INFO:journalctl@ceph.mds.l.smithi077.stdout:Jul 27 06:42:21 smithi077 podman[68427]: 2024-07-27 06:42:21.844395819 +0000 UTC m=+0.363887740 container init 3ba94332af1944bab5048fbc95e409d7150c1b81b78b8149b27e2243b17ac8c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-l, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-07-27T06:42:21.849 INFO:journalctl@ceph.mds.l.smithi077.stdout:Jul 27 06:42:21 smithi077 podman[68427]: 2024-07-27 06:42:21.846931946 +0000 UTC m=+0.366423854 container start 3ba94332af1944bab5048fbc95e409d7150c1b81b78b8149b27e2243b17ac8c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-l, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0) 2024-07-27T06:42:22.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.027+0000 7f7258908640 1 -- 172.21.15.29:0/3128198486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254109820 msgr2=0x7f7254110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.027+0000 7f7258908640 1 --2- 172.21.15.29:0/3128198486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254109820 0x7f7254110050 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f724c009ca0 tx=0x7f724c02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:42:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.027+0000 7f7258908640 1 -- 172.21.15.29:0/3128198486 shutdown_connections 2024-07-27T06:42:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.027+0000 7f7258908640 1 --2- 172.21.15.29:0/3128198486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254109820 0x7f7254110050 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.027+0000 7f7258908640 1 --2- 172.21.15.29:0/3128198486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7254108ed0 0x7f72541092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.027+0000 7f7258908640 1 --2- 172.21.15.29:0/3128198486 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7254103590 0x7f72541039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.027+0000 7f7258908640 1 -- 172.21.15.29:0/3128198486 >> 172.21.15.29:0/3128198486 conn(0x7f725407b280 msgr2=0x7f725407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.027+0000 7f7258908640 1 -- 172.21.15.29:0/3128198486 shutdown_connections 2024-07-27T06:42:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.027+0000 7f7258908640 1 -- 172.21.15.29:0/3128198486 wait complete. 2024-07-27T06:42:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f7258908640 1 Processor -- start 2024-07-27T06:42:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f7258908640 1 -- start start 2024-07-27T06:42:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f7258908640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7254103590 0x7f7254199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f7258908640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7254108ed0 0x7f725419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f7258908640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254109820 0x7f725419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f7258908640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f72541134d0 con 0x7f7254109820 2024-07-27T06:42:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f7258908640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7254113350 con 0x7f7254108ed0 2024-07-27T06:42:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f7258908640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7254113650 con 0x7f7254103590 2024-07-27T06:42:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f7252d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254109820 0x7f725419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f7252d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254109820 0x7f725419f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58978/0 (socket says 172.21.15.29:58978) 2024-07-27T06:42:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f7252d9d640 1 -- 172.21.15.29:0/3896308405 learned_addr learned my addr 172.21.15.29:0/3896308405 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:42:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f7251d9b640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7254108ed0 0x7f725419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.028+0000 7f725259c640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7254103590 0x7f7254199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.029+0000 7f7252d9d640 1 -- 172.21.15.29:0/3896308405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7254103590 msgr2=0x7f7254199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:22.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.029+0000 7f7252d9d640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7254103590 0x7f7254199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:22.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.029+0000 7f7252d9d640 1 -- 172.21.15.29:0/3896308405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7254108ed0 msgr2=0x7f725419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:22.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.029+0000 7f7252d9d640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7254108ed0 0x7f725419a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:22.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.029+0000 7f7252d9d640 1 -- 172.21.15.29:0/3896308405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725419fdf0 con 0x7f7254109820 2024-07-27T06:42:22.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.029+0000 7f7251d9b640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7254108ed0 0x7f725419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:42:22.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.029+0000 7f7252d9d640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254109820 0x7f725419f6c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f724c005cc0 tx=0x7f724c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:22.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.029+0000 7f72437fe640 1 -- 172.21.15.29:0/3896308405 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c003bf0 con 0x7f7254109820 2024-07-27T06:42:22.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.029+0000 7f72437fe640 1 -- 172.21.15.29:0/3896308405 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f724c003d90 con 0x7f7254109820 2024-07-27T06:42:22.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.029+0000 7f7258908640 1 -- 172.21.15.29:0/3896308405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72541a0080 con 0x7f7254109820 2024-07-27T06:42:22.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.029+0000 7f72437fe640 1 -- 172.21.15.29:0/3896308405 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c02fbc0 con 0x7f7254109820 2024-07-27T06:42:22.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.029+0000 7f7258908640 1 -- 172.21.15.29:0/3896308405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72541a6840 con 0x7f7254109820 2024-07-27T06:42:22.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.030+0000 7f7258908640 1 -- 172.21.15.29:0/3896308405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72541039a0 con 0x7f7254109820 2024-07-27T06:42:22.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.032+0000 7f72437fe640 1 -- 172.21.15.29:0/3896308405 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f724c02fd60 con 0x7f7254109820 2024-07-27T06:42:22.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.033+0000 7f72437fe640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f723c0b3c20 0x7f723c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:22.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.033+0000 7f725259c640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f723c0b3c20 0x7f723c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:22.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.033+0000 7f72437fe640 1 -- 172.21.15.29:0/3896308405 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f724c105210 con 0x7f7254109820 2024-07-27T06:42:22.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.033+0000 7f725259c640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f723c0b3c20 0x7f723c0b6130 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7244005d00 tx=0x7f724400b410 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:22.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.036+0000 7f72437fe640 1 -- 172.21.15.29:0/3896308405 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f724c082600 con 0x7f7254109820 2024-07-27T06:42:22.106 INFO:journalctl@ceph.mds.l.smithi077.stdout:Jul 27 06:42:21 smithi077 ceph-mds[68447]: -- 172.21.15.77:0/80965331 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5634537fd860 con 0x56345799fa80 2024-07-27T06:42:22.107 INFO:journalctl@ceph.mds.l.smithi077.stdout:Jul 27 06:42:21 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-l[68443]: starting mds.l at 2024-07-27T06:42:22.107 INFO:journalctl@ceph.mds.l.smithi077.stdout:Jul 27 06:42:21 smithi077 bash[68427]: 3ba94332af1944bab5048fbc95e409d7150c1b81b78b8149b27e2243b17ac8c8 2024-07-27T06:42:22.107 INFO:journalctl@ceph.mds.l.smithi077.stdout:Jul 27 06:42:21 smithi077 systemd[1]: Started Ceph mds.l for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:42:22.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.256+0000 7f7258908640 1 -- 172.21.15.29:0/3896308405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f725419afb0 con 0x7f7254109820 2024-07-27T06:42:22.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.256+0000 7f72437fe640 1 -- 172.21.15.29:0/3896308405 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1435 (secure 0 0 0) 0x7f725419afb0 con 0x7f7254109820 2024-07-27T06:42:22.258 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:42:22.258 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":2,"btime":"2024-07-27T06:42:21:885509+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-07-27T06:42:22.258 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 2 2024-07-27T06:42:22.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.258+0000 7f7258908640 1 -- 172.21.15.29:0/3896308405 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f723c0b3c20 msgr2=0x7f723c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:22.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.258+0000 7f7258908640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f723c0b3c20 0x7f723c0b6130 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7244005d00 tx=0x7f724400b410 comp rx=0 tx=0).stop 2024-07-27T06:42:22.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.258+0000 7f7258908640 1 -- 172.21.15.29:0/3896308405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254109820 msgr2=0x7f725419f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:22.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.258+0000 7f7258908640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254109820 0x7f725419f6c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f724c005cc0 tx=0x7f724c02f380 comp rx=0 tx=0).stop 2024-07-27T06:42:22.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.258+0000 7f7258908640 1 -- 172.21.15.29:0/3896308405 shutdown_connections 2024-07-27T06:42:22.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.258+0000 7f7258908640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7254109820 0x7f725419f6c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:22.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.258+0000 7f7258908640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7254108ed0 0x7f725419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:22.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.258+0000 7f7258908640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f723c0b3c20 0x7f723c0b6130 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:22.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.258+0000 7f7258908640 1 --2- 172.21.15.29:0/3896308405 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7254103590 0x7f7254199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:22.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.258+0000 7f7258908640 1 -- 172.21.15.29:0/3896308405 >> 172.21.15.29:0/3896308405 conn(0x7f725407b280 msgr2=0x7f7254106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:22.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.258+0000 7f7258908640 1 -- 172.21.15.29:0/3896308405 shutdown_connections 2024-07-27T06:42:22.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:22.258+0000 7f7258908640 1 -- 172.21.15.29:0/3896308405 wait complete. 2024-07-27T06:42:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:22 smithi047 ceph-mon[36835]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:22 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] up:boot 2024-07-27T06:42:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:22 smithi047 ceph-mon[36835]: fsmap 1 up:standby 2024-07-27T06:42:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-27T06:42:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:22 smithi047 ceph-mon[36835]: Deploying daemon mds.k on smithi047 2024-07-27T06:42:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:22 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3896308405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:22 smithi029 ceph-mon[32610]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:22 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] up:boot 2024-07-27T06:42:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:22 smithi029 ceph-mon[32610]: fsmap 1 up:standby 2024-07-27T06:42:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-27T06:42:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:22 smithi029 ceph-mon[32610]: Deploying daemon mds.k on smithi047 2024-07-27T06:42:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:22 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3896308405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:22 smithi077 ceph-mon[37153]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:22 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] up:boot 2024-07-27T06:42:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:22 smithi077 ceph-mon[37153]: fsmap 1 up:standby 2024-07-27T06:42:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-27T06:42:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:22 smithi077 ceph-mon[37153]: Deploying daemon mds.k on smithi047 2024-07-27T06:42:23.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:22 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3896308405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:24.196 INFO:journalctl@ceph.mds.k.smithi047.stdout:Jul 27 06:42:24 smithi047 systemd[1]: Starting Ceph mds.k for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:42:24.586 INFO:journalctl@ceph.mds.k.smithi047.stdout:Jul 27 06:42:24 smithi047 podman[67714]: 2024-07-27 06:42:24.20602963 +0000 UTC m=+0.017738070 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:42:24.586 INFO:journalctl@ceph.mds.k.smithi047.stdout:Jul 27 06:42:24 smithi047 podman[67714]: 2024-07-27 06:42:24.30572313 +0000 UTC m=+0.117431602 container create d6773413f40baee0e7132fc5e913b270180eff8f1ef436cd090d2527327ee269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2) 2024-07-27T06:42:24.586 INFO:journalctl@ceph.mds.k.smithi047.stdout:Jul 27 06:42:24 smithi047 podman[67714]: 2024-07-27 06:42:24.549868667 +0000 UTC m=+0.361577102 container init d6773413f40baee0e7132fc5e913b270180eff8f1ef436cd090d2527327ee269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-27T06:42:24.587 INFO:journalctl@ceph.mds.k.smithi047.stdout:Jul 27 06:42:24 smithi047 podman[67714]: 2024-07-27 06:42:24.55265673 +0000 UTC m=+0.364365170 container start d6773413f40baee0e7132fc5e913b270180eff8f1ef436cd090d2527327ee269 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-k, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-07-27T06:42:24.895 INFO:journalctl@ceph.mds.k.smithi047.stdout:Jul 27 06:42:24 smithi047 ceph-mds[67733]: -- 172.21.15.47:0/1441806273 <== mon.2 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56241599b860 con 0x562419b3d600 2024-07-27T06:42:24.895 INFO:journalctl@ceph.mds.k.smithi047.stdout:Jul 27 06:42:24 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-k[67729]: starting mds.k at 2024-07-27T06:42:24.895 INFO:journalctl@ceph.mds.k.smithi047.stdout:Jul 27 06:42:24 smithi047 bash[67714]: d6773413f40baee0e7132fc5e913b270180eff8f1ef436cd090d2527327ee269 2024-07-27T06:42:24.895 INFO:journalctl@ceph.mds.k.smithi047.stdout:Jul 27 06:42:24 smithi047 systemd[1]: Started Ceph mds.k for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:42:25.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:42:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:24 smithi029 ceph-mon[32610]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:24 smithi047 ceph-mon[36835]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:24 smithi077 ceph-mon[37153]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:25.958 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:42:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:25 smithi047 ceph-mon[36835]: Deploying daemon mds.j on smithi029 2024-07-27T06:42:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:25 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6832/2402830639,v1:172.21.15.47:6833/2402830639] up:boot 2024-07-27T06:42:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:25 smithi047 ceph-mon[36835]: fsmap 2 up:standby 2024-07-27T06:42:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-27T06:42:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:25 smithi029 ceph-mon[32610]: Deploying daemon mds.j on smithi029 2024-07-27T06:42:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:25 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6832/2402830639,v1:172.21.15.47:6833/2402830639] up:boot 2024-07-27T06:42:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:25 smithi029 ceph-mon[32610]: fsmap 2 up:standby 2024-07-27T06:42:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-27T06:42:26.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:25 smithi077 ceph-mon[37153]: Deploying daemon mds.j on smithi029 2024-07-27T06:42:26.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:25 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6832/2402830639,v1:172.21.15.47:6833/2402830639] up:boot 2024-07-27T06:42:26.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:25 smithi077 ceph-mon[37153]: fsmap 2 up:standby 2024-07-27T06:42:26.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-27T06:42:26.984 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:26 smithi047 ceph-mon[36835]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:27.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:26 smithi029 ceph-mon[32610]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:26 smithi077 ceph-mon[37153]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.210+0000 7f04e4388640 1 -- 172.21.15.29:0/3676366002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04dc104330 msgr2=0x7f04dc104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.210+0000 7f04e4388640 1 --2- 172.21.15.29:0/3676366002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04dc104330 0x7f04dc104740 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009900 tx=0x7f04d802a410 comp rx=0 tx=0).stop 2024-07-27T06:42:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.210+0000 7f04e4388640 1 -- 172.21.15.29:0/3676366002 shutdown_connections 2024-07-27T06:42:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.210+0000 7f04e4388640 1 --2- 172.21.15.29:0/3676366002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04dc10d8e0 0x7f04dc10fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.210+0000 7f04e4388640 1 --2- 172.21.15.29:0/3676366002 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04dc104c80 0x7f04dc10d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.210+0000 7f04e4388640 1 --2- 172.21.15.29:0/3676366002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04dc104330 0x7f04dc104740 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.210+0000 7f04e4388640 1 -- 172.21.15.29:0/3676366002 >> 172.21.15.29:0/3676366002 conn(0x7f04dc079190 msgr2=0x7f04dc0795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.210+0000 7f04e4388640 1 -- 172.21.15.29:0/3676366002 shutdown_connections 2024-07-27T06:42:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.211+0000 7f04e4388640 1 -- 172.21.15.29:0/3676366002 wait complete. 2024-07-27T06:42:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.211+0000 7f04e4388640 1 Processor -- start 2024-07-27T06:42:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.211+0000 7f04e4388640 1 -- start start 2024-07-27T06:42:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e4388640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04dc104330 0x7f04dc102240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e4388640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04dc104c80 0x7f04dc102780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e4388640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04dc10d8e0 0x7f04dc102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e4388640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc113500 con 0x7f04dc10d8e0 2024-07-27T06:42:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e4388640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc113380 con 0x7f04dc104c80 2024-07-27T06:42:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e4388640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc113680 con 0x7f04dc104330 2024-07-27T06:42:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e2925640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04dc10d8e0 0x7f04dc102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e2925640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04dc10d8e0 0x7f04dc102cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58992/0 (socket says 172.21.15.29:58992) 2024-07-27T06:42:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e2925640 1 -- 172.21.15.29:0/2965271848 learned_addr learned my addr 172.21.15.29:0/2965271848 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:42:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e1923640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04dc104c80 0x7f04dc102780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e2124640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04dc104330 0x7f04dc102240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e2925640 1 -- 172.21.15.29:0/2965271848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04dc104330 msgr2=0x7f04dc102240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e2925640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04dc104330 0x7f04dc102240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e2925640 1 -- 172.21.15.29:0/2965271848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04dc104c80 msgr2=0x7f04dc102780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e2925640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04dc104c80 0x7f04dc102780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e2925640 1 -- 172.21.15.29:0/2965271848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04dc1a8230 con 0x7f04dc10d8e0 2024-07-27T06:42:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.212+0000 7f04e2925640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04dc10d8e0 0x7f04dc102cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f04d400e9d0 tx=0x7f04d400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.213+0000 7f04d37fe640 1 -- 172.21.15.29:0/2965271848 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d400cd80 con 0x7f04dc10d8e0 2024-07-27T06:42:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.213+0000 7f04d37fe640 1 -- 172.21.15.29:0/2965271848 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04d4005020 con 0x7f04dc10d8e0 2024-07-27T06:42:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.213+0000 7f04e4388640 1 -- 172.21.15.29:0/2965271848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04dc1a8520 con 0x7f04dc10d8e0 2024-07-27T06:42:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.213+0000 7f04d37fe640 1 -- 172.21.15.29:0/2965271848 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d40116d0 con 0x7f04dc10d8e0 2024-07-27T06:42:28.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.213+0000 7f04e4388640 1 -- 172.21.15.29:0/2965271848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04dc1a8a60 con 0x7f04dc10d8e0 2024-07-27T06:42:28.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.214+0000 7f04e4388640 1 -- 172.21.15.29:0/2965271848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04dc06bb80 con 0x7f04dc10d8e0 2024-07-27T06:42:28.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.216+0000 7f04d37fe640 1 -- 172.21.15.29:0/2965271848 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f04d40051d0 con 0x7f04dc10d8e0 2024-07-27T06:42:28.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.217+0000 7f04d37fe640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f04c80b3c20 0x7f04c80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:28.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.217+0000 7f04e2124640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f04c80b3c20 0x7f04c80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:28.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.217+0000 7f04d37fe640 1 -- 172.21.15.29:0/2965271848 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f04d40e3d00 con 0x7f04dc10d8e0 2024-07-27T06:42:28.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.217+0000 7f04e2124640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f04c80b3c20 0x7f04c80b6130 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009230 tx=0x7f04d8031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:28.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.220+0000 7f04d37fe640 1 -- 172.21.15.29:0/2965271848 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f04d4016030 con 0x7f04dc10d8e0 2024-07-27T06:42:28.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.469+0000 7f04e4388640 1 -- 172.21.15.29:0/2965271848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04dc103540 con 0x7f04dc10d8e0 2024-07-27T06:42:28.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.469+0000 7f04d37fe640 1 -- 172.21.15.29:0/2965271848 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2260 (secure 0 0 0) 0x7f04dc103540 con 0x7f04dc10d8e0 2024-07-27T06:42:28.471 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:42:28.471 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":3,"btime":"2024-07-27T06:42:24:889486+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24542,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-07-27T06:42:28.471 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 3 2024-07-27T06:42:28.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.471+0000 7f04e4388640 1 -- 172.21.15.29:0/2965271848 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f04c80b3c20 msgr2=0x7f04c80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:28.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.471+0000 7f04e4388640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f04c80b3c20 0x7f04c80b6130 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009230 tx=0x7f04d8031400 comp rx=0 tx=0).stop 2024-07-27T06:42:28.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.471+0000 7f04e4388640 1 -- 172.21.15.29:0/2965271848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04dc10d8e0 msgr2=0x7f04dc102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:28.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.471+0000 7f04e4388640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04dc10d8e0 0x7f04dc102cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f04d400e9d0 tx=0x7f04d400ee90 comp rx=0 tx=0).stop 2024-07-27T06:42:28.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.471+0000 7f04e4388640 1 -- 172.21.15.29:0/2965271848 shutdown_connections 2024-07-27T06:42:28.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.471+0000 7f04e4388640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04dc10d8e0 0x7f04dc102cc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:28.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.471+0000 7f04e4388640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04dc104c80 0x7f04dc102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:28.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.471+0000 7f04e4388640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f04c80b3c20 0x7f04c80b6130 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:28.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.471+0000 7f04e4388640 1 --2- 172.21.15.29:0/2965271848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04dc104330 0x7f04dc102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:28.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.471+0000 7f04e4388640 1 -- 172.21.15.29:0/2965271848 >> 172.21.15.29:0/2965271848 conn(0x7f04dc079190 msgr2=0x7f04dc107480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:28.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.471+0000 7f04e4388640 1 -- 172.21.15.29:0/2965271848 shutdown_connections 2024-07-27T06:42:28.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:28.471+0000 7f04e4388640 1 -- 172.21.15.29:0/2965271848 wait complete. 2024-07-27T06:42:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:28 smithi047 ceph-mon[36835]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:29.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:28 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2965271848' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:28 smithi029 ceph-mon[32610]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:28 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2965271848' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:28 smithi077 ceph-mon[37153]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:28 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2965271848' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:29.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:42:29 smithi029 systemd[1]: Started Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:42:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:30 smithi077 ceph-mon[37153]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:30 smithi077 ceph-mon[37153]: Deploying daemon mds.d on smithi029 2024-07-27T06:42:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:30 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:boot 2024-07-27T06:42:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:30 smithi077 ceph-mon[37153]: fsmap 3 up:standby 2024-07-27T06:42:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:42:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:30 smithi047 ceph-mon[36835]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:30 smithi047 ceph-mon[36835]: Deploying daemon mds.d on smithi029 2024-07-27T06:42:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:30 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:boot 2024-07-27T06:42:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:30 smithi047 ceph-mon[36835]: fsmap 3 up:standby 2024-07-27T06:42:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:42:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:30 smithi029 ceph-mon[32610]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:30 smithi029 ceph-mon[32610]: Deploying daemon mds.d on smithi029 2024-07-27T06:42:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:30 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:boot 2024-07-27T06:42:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:30 smithi029 ceph-mon[32610]: fsmap 3 up:standby 2024-07-27T06:42:31.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:42:31.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:42:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:42:32.107 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:42:31 smithi029 systemd[1]: Starting Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:42:32.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:42:32 smithi029 podman[77598]: 2024-07-27 06:42:32.288812089 +0000 UTC m=+0.022691436 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:42:33.314 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:33 smithi029 ceph-mon[32610]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:33.576 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:42:33 smithi029 podman[77598]: 2024-07-27 06:42:33.431303642 +0000 UTC m=+1.165183022 container create d25557f5b03dff4cb63d83b456b53cf4d74f078b6f9eb0ea2266311e49e28069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-27T06:42:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:33 smithi077 ceph-mon[37153]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:33 smithi047 ceph-mon[36835]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:33.944 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:42:33 smithi029 podman[77598]: 2024-07-27 06:42:33.706117323 +0000 UTC m=+1.439996676 container init d25557f5b03dff4cb63d83b456b53cf4d74f078b6f9eb0ea2266311e49e28069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-27T06:42:33.944 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:42:33 smithi029 podman[77598]: 2024-07-27 06:42:33.709517157 +0000 UTC m=+1.443396489 container start d25557f5b03dff4cb63d83b456b53cf4d74f078b6f9eb0ea2266311e49e28069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-07-27T06:42:33.944 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:42:33 smithi029 ceph-mds[77653]: -- 172.21.15.29:0/2294596634 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aaae6af860 con 0x55aab2851a80 2024-07-27T06:42:33.945 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:42:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: starting mds.d at 2024-07-27T06:42:33.945 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:42:33 smithi029 bash[77598]: d25557f5b03dff4cb63d83b456b53cf4d74f078b6f9eb0ea2266311e49e28069 2024-07-27T06:42:33.945 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:42:33 smithi029 systemd[1]: Started Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:42:34.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.368+0000 7f93411da640 1 -- 172.21.15.29:0/2318293064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933c102490 msgr2=0x7f933c1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:34.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.368+0000 7f93411da640 1 --2- 172.21.15.29:0/2318293064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933c102490 0x7f933c1028a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f932c009900 tx=0x7f932c02a410 comp rx=0 tx=0).stop 2024-07-27T06:42:34.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.368+0000 7f93411da640 1 -- 172.21.15.29:0/2318293064 shutdown_connections 2024-07-27T06:42:34.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.368+0000 7f93411da640 1 --2- 172.21.15.29:0/2318293064 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f933c108720 0x7f933c10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:34.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.368+0000 7f93411da640 1 --2- 172.21.15.29:0/2318293064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f933c107dd0 0x7f933c1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:34.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.368+0000 7f93411da640 1 --2- 172.21.15.29:0/2318293064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933c102490 0x7f933c1028a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:34.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.368+0000 7f93411da640 1 -- 172.21.15.29:0/2318293064 >> 172.21.15.29:0/2318293064 conn(0x7f933c079160 msgr2=0x7f933c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:34.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.368+0000 7f93411da640 1 -- 172.21.15.29:0/2318293064 shutdown_connections 2024-07-27T06:42:34.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.368+0000 7f93411da640 1 -- 172.21.15.29:0/2318293064 wait complete. 2024-07-27T06:42:34.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.369+0000 7f93411da640 1 Processor -- start 2024-07-27T06:42:34.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.369+0000 7f93411da640 1 -- start start 2024-07-27T06:42:34.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.369+0000 7f93411da640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f933c102490 0x7f933c198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:34.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f93411da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933c107dd0 0x7f933c1993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:34.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f93411da640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f933c108720 0x7f933c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:34.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f93411da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f933c112380 con 0x7f933c107dd0 2024-07-27T06:42:34.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f93411da640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f933c112200 con 0x7f933c102490 2024-07-27T06:42:34.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f93411da640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f933c112500 con 0x7f933c108720 2024-07-27T06:42:34.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f933a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933c107dd0 0x7f933c1993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:34.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f933a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933c107dd0 0x7f933c1993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46540/0 (socket says 172.21.15.29:46540) 2024-07-27T06:42:34.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f933a59c640 1 -- 172.21.15.29:0/2790472090 learned_addr learned my addr 172.21.15.29:0/2790472090 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:42:34.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f933ad9d640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f933c102490 0x7f933c198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:34.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f933b59e640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f933c108720 0x7f933c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:34.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f933a59c640 1 -- 172.21.15.29:0/2790472090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f933c108720 msgr2=0x7f933c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f933a59c640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f933c108720 0x7f933c19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f933a59c640 1 -- 172.21.15.29:0/2790472090 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f933c102490 msgr2=0x7f933c198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f933a59c640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f933c102490 0x7f933c198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f933a59c640 1 -- 172.21.15.29:0/2790472090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933c19edc0 con 0x7f933c107dd0 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f933a59c640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933c107dd0 0x7f933c1993f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f933000c970 tx=0x7f933000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f9323fff640 1 -- 172.21.15.29:0/2790472090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9330007bb0 con 0x7f933c107dd0 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f93411da640 1 -- 172.21.15.29:0/2790472090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f933c19f0b0 con 0x7f933c107dd0 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f9323fff640 1 -- 172.21.15.29:0/2790472090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9330007d50 con 0x7f933c107dd0 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.370+0000 7f9323fff640 1 -- 172.21.15.29:0/2790472090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933000f450 con 0x7f933c107dd0 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.371+0000 7f93411da640 1 -- 172.21.15.29:0/2790472090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f933c1a5810 con 0x7f933c107dd0 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.372+0000 7f93411da640 1 -- 172.21.15.29:0/2790472090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f933c1028a0 con 0x7f933c107dd0 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.373+0000 7f9323fff640 1 -- 172.21.15.29:0/2790472090 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f9330004ea0 con 0x7f933c107dd0 2024-07-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.374+0000 7f9323fff640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f93240b3c20 0x7f93240b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.374+0000 7f933ad9d640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f93240b3c20 0x7f93240b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.374+0000 7f9323fff640 1 -- 172.21.15.29:0/2790472090 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f93300e39f0 con 0x7f933c107dd0 2024-07-27T06:42:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.374+0000 7f933ad9d640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f93240b3c20 0x7f93240b6130 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f932c031440 tx=0x7f932c009560 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.377+0000 7f9323fff640 1 -- 172.21.15.29:0/2790472090 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9330014030 con 0x7f933c107dd0 2024-07-27T06:42:34.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.596+0000 7f93411da640 1 -- 172.21.15.29:0/2790472090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f933c199f80 con 0x7f933c107dd0 2024-07-27T06:42:34.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.597+0000 7f9323fff640 1 -- 172.21.15.29:0/2790472090 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3910 (secure 0 0 0) 0x7f93300ab410 con 0x7f933c107dd0 2024-07-27T06:42:34.598 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:42:34.599 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"btime":"2024-07-27T06:42:34:277838+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24542,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-07-27T06:42:34.599 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 5 2024-07-27T06:42:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.599+0000 7f93411da640 1 -- 172.21.15.29:0/2790472090 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f93240b3c20 msgr2=0x7f93240b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.599+0000 7f93411da640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f93240b3c20 0x7f93240b6130 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f932c031440 tx=0x7f932c009560 comp rx=0 tx=0).stop 2024-07-27T06:42:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.599+0000 7f93411da640 1 -- 172.21.15.29:0/2790472090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933c107dd0 msgr2=0x7f933c1993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.599+0000 7f93411da640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933c107dd0 0x7f933c1993f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f933000c970 tx=0x7f933000ce30 comp rx=0 tx=0).stop 2024-07-27T06:42:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.599+0000 7f93411da640 1 -- 172.21.15.29:0/2790472090 shutdown_connections 2024-07-27T06:42:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.599+0000 7f93411da640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f93240b3c20 0x7f93240b6130 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.599+0000 7f93411da640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f933c108720 0x7f933c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.599+0000 7f93411da640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933c107dd0 0x7f933c1993f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.599+0000 7f93411da640 1 --2- 172.21.15.29:0/2790472090 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f933c102490 0x7f933c198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.599+0000 7f93411da640 1 -- 172.21.15.29:0/2790472090 >> 172.21.15.29:0/2790472090 conn(0x7f933c079160 msgr2=0x7f933c100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.599+0000 7f93411da640 1 -- 172.21.15.29:0/2790472090 shutdown_connections 2024-07-27T06:42:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:34.599+0000 7f93411da640 1 -- 172.21.15.29:0/2790472090 wait complete. 2024-07-27T06:42:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:34 smithi047 ceph-mon[36835]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:34 smithi047 ceph-mon[36835]: Deploying daemon mds.e on smithi047 2024-07-27T06:42:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:34 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:boot 2024-07-27T06:42:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:34 smithi047 ceph-mon[36835]: fsmap 4 up:standby 2024-07-27T06:42:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:42:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:34 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2790472090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:34 smithi029 ceph-mon[32610]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:34 smithi029 ceph-mon[32610]: Deploying daemon mds.e on smithi047 2024-07-27T06:42:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:34 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:boot 2024-07-27T06:42:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:34 smithi029 ceph-mon[32610]: fsmap 4 up:standby 2024-07-27T06:42:35.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:42:35.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:34 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2790472090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:34 smithi077 ceph-mon[37153]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:34 smithi077 ceph-mon[37153]: Deploying daemon mds.e on smithi047 2024-07-27T06:42:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:34 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:boot 2024-07-27T06:42:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:34 smithi077 ceph-mon[37153]: fsmap 4 up:standby 2024-07-27T06:42:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:42:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:34 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2790472090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:36.157 INFO:journalctl@ceph.mds.e.smithi047.stdout:Jul 27 06:42:35 smithi047 systemd[1]: Starting Ceph mds.e for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:42:36.431 INFO:journalctl@ceph.mds.e.smithi047.stdout:Jul 27 06:42:36 smithi047 podman[68030]: 2024-07-27 06:42:36.05934413 +0000 UTC m=+0.018148591 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:42:36.431 INFO:journalctl@ceph.mds.e.smithi047.stdout:Jul 27 06:42:36 smithi047 podman[68030]: 2024-07-27 06:42:36.186096551 +0000 UTC m=+0.144901039 container create 818f7d59409cb2a4cef0988ae517f423995eaea8a61299891b08f8640787526a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-27T06:42:36.431 INFO:journalctl@ceph.mds.e.smithi047.stdout:Jul 27 06:42:36 smithi047 podman[68030]: 2024-07-27 06:42:36.430170432 +0000 UTC m=+0.388974879 container init 818f7d59409cb2a4cef0988ae517f423995eaea8a61299891b08f8640787526a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-07-27T06:42:36.696 INFO:journalctl@ceph.mds.e.smithi047.stdout:Jul 27 06:42:36 smithi047 podman[68030]: 2024-07-27 06:42:36.433179214 +0000 UTC m=+0.391983660 container start 818f7d59409cb2a4cef0988ae517f423995eaea8a61299891b08f8640787526a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-e, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-07-27T06:42:36.696 INFO:journalctl@ceph.mds.e.smithi047.stdout:Jul 27 06:42:36 smithi047 ceph-mds[68049]: -- 172.21.15.47:0/777140766 <== mon.2 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5613ce623860 con 0x5613d27c5600 2024-07-27T06:42:36.696 INFO:journalctl@ceph.mds.e.smithi047.stdout:Jul 27 06:42:36 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-e[68045]: starting mds.e at 2024-07-27T06:42:36.696 INFO:journalctl@ceph.mds.e.smithi047.stdout:Jul 27 06:42:36 smithi047 bash[68030]: 818f7d59409cb2a4cef0988ae517f423995eaea8a61299891b08f8640787526a 2024-07-27T06:42:36.697 INFO:journalctl@ceph.mds.e.smithi047.stdout:Jul 27 06:42:36 smithi047 systemd[1]: Started Ceph mds.e for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:42:37.190 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:36 smithi077 ceph-mon[37153]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:37.190 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:37.190 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:37.190 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:37.190 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:37.190 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:37.191 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:36 smithi047 ceph-mon[36835]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:37.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:36 smithi029 ceph-mon[32610]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:37.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:42:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:42:38.077 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:37 smithi077 ceph-mon[37153]: Deploying daemon mds.f on smithi077 2024-07-27T06:42:38.078 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:37 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6834/2900591116,v1:172.21.15.47:6835/2900591116] up:boot 2024-07-27T06:42:38.078 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:37 smithi077 ceph-mon[37153]: fsmap 5 up:standby 2024-07-27T06:42:38.078 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-27T06:42:38.123 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:37 smithi029 ceph-mon[32610]: Deploying daemon mds.f on smithi077 2024-07-27T06:42:38.123 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:37 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6834/2900591116,v1:172.21.15.47:6835/2900591116] up:boot 2024-07-27T06:42:38.123 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:37 smithi029 ceph-mon[32610]: fsmap 5 up:standby 2024-07-27T06:42:38.123 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-27T06:42:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:37 smithi047 ceph-mon[36835]: Deploying daemon mds.f on smithi077 2024-07-27T06:42:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:37 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6834/2900591116,v1:172.21.15.47:6835/2900591116] up:boot 2024-07-27T06:42:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:37 smithi047 ceph-mon[36835]: fsmap 5 up:standby 2024-07-27T06:42:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-27T06:42:38.607 INFO:journalctl@ceph.mds.f.smithi077.stdout:Jul 27 06:42:38 smithi077 systemd[1]: Starting Ceph mds.f for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:42:39.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:38 smithi077 ceph-mon[37153]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:39.033 INFO:journalctl@ceph.mds.f.smithi077.stdout:Jul 27 06:42:38 smithi077 podman[68731]: 2024-07-27 06:42:38.641222628 +0000 UTC m=+0.016721625 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:42:39.033 INFO:journalctl@ceph.mds.f.smithi077.stdout:Jul 27 06:42:38 smithi077 podman[68731]: 2024-07-27 06:42:38.754547135 +0000 UTC m=+0.130046123 container create e312408b0360806f5bbe11d42e6bc4a16395a89c46a41834c8a8ed81bd6522df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-f, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux ) 2024-07-27T06:42:39.033 INFO:journalctl@ceph.mds.f.smithi077.stdout:Jul 27 06:42:39 smithi077 podman[68731]: 2024-07-27 06:42:39.032689248 +0000 UTC m=+0.408188236 container init e312408b0360806f5bbe11d42e6bc4a16395a89c46a41834c8a8ed81bd6522df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True) 2024-07-27T06:42:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:38 smithi047 ceph-mon[36835]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:39.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:38 smithi029 ceph-mon[32610]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:39.356 INFO:journalctl@ceph.mds.f.smithi077.stdout:Jul 27 06:42:39 smithi077 podman[68731]: 2024-07-27 06:42:39.036011525 +0000 UTC m=+0.411510518 container start e312408b0360806f5bbe11d42e6bc4a16395a89c46a41834c8a8ed81bd6522df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-07-27T06:42:39.357 INFO:journalctl@ceph.mds.f.smithi077.stdout:Jul 27 06:42:39 smithi077 ceph-mds[68751]: -- 172.21.15.77:0/2814061025 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558fff2df860 con 0x559003481600 2024-07-27T06:42:39.357 INFO:journalctl@ceph.mds.f.smithi077.stdout:Jul 27 06:42:39 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-f[68747]: starting mds.f at 2024-07-27T06:42:39.357 INFO:journalctl@ceph.mds.f.smithi077.stdout:Jul 27 06:42:39 smithi077 bash[68731]: e312408b0360806f5bbe11d42e6bc4a16395a89c46a41834c8a8ed81bd6522df 2024-07-27T06:42:39.357 INFO:journalctl@ceph.mds.f.smithi077.stdout:Jul 27 06:42:39 smithi077 systemd[1]: Started Ceph mds.f for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:42:39.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.519+0000 7fc694785640 1 -- 172.21.15.29:0/2660580820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc68c107e30 msgr2=0x7fc68c108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:39.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.519+0000 7fc694785640 1 --2- 172.21.15.29:0/2660580820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc68c107e30 0x7fc68c108240 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc67c009900 tx=0x7fc67c02a410 comp rx=0 tx=0).stop 2024-07-27T06:42:39.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.519+0000 7fc694785640 1 -- 172.21.15.29:0/2660580820 shutdown_connections 2024-07-27T06:42:39.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.519+0000 7fc694785640 1 --2- 172.21.15.29:0/2660580820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc68c108780 0x7fc68c10efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:39.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.519+0000 7fc694785640 1 --2- 172.21.15.29:0/2660580820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc68c107e30 0x7fc68c108240 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:39.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.519+0000 7fc694785640 1 --2- 172.21.15.29:0/2660580820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc68c102bc0 0x7fc68c102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:39.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.519+0000 7fc694785640 1 -- 172.21.15.29:0/2660580820 >> 172.21.15.29:0/2660580820 conn(0x7fc68c0799c0 msgr2=0x7fc68c079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:39.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.519+0000 7fc694785640 1 -- 172.21.15.29:0/2660580820 shutdown_connections 2024-07-27T06:42:39.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.519+0000 7fc694785640 1 -- 172.21.15.29:0/2660580820 wait complete. 2024-07-27T06:42:39.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.520+0000 7fc694785640 1 Processor -- start 2024-07-27T06:42:39.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.520+0000 7fc694785640 1 -- start start 2024-07-27T06:42:39.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.520+0000 7fc694785640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc68c102bc0 0x7fc68c19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:39.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.520+0000 7fc694785640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc68c107e30 0x7fc68c19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:39.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.520+0000 7fc694785640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc68c108780 0x7fc68c19f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:39.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.520+0000 7fc694785640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c112410 con 0x7fc68c108780 2024-07-27T06:42:39.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.520+0000 7fc694785640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c112290 con 0x7fc68c102bc0 2024-07-27T06:42:39.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.520+0000 7fc694785640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c112590 con 0x7fc68c107e30 2024-07-27T06:42:39.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.520+0000 7fc692d22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc68c108780 0x7fc68c19f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:39.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.520+0000 7fc692d22640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc68c108780 0x7fc68c19f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51122/0 (socket says 172.21.15.29:51122) 2024-07-27T06:42:39.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.520+0000 7fc692d22640 1 -- 172.21.15.29:0/833107420 learned_addr learned my addr 172.21.15.29:0/833107420 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:42:39.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc691d20640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc68c107e30 0x7fc68c19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:39.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc692521640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc68c102bc0 0x7fc68c19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:39.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc692d22640 1 -- 172.21.15.29:0/833107420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc68c107e30 msgr2=0x7fc68c19a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:39.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc692d22640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc68c107e30 0x7fc68c19a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:39.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc692d22640 1 -- 172.21.15.29:0/833107420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc68c102bc0 msgr2=0x7fc68c19a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:39.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc692d22640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc68c102bc0 0x7fc68c19a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:39.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc692d22640 1 -- 172.21.15.29:0/833107420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc68c19ffc0 con 0x7fc68c108780 2024-07-27T06:42:39.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc692d22640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc68c108780 0x7fc68c19f890 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc684009fd0 tx=0x7fc68400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:39.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc6837fe640 1 -- 172.21.15.29:0/833107420 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc684017070 con 0x7fc68c108780 2024-07-27T06:42:39.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc6837fe640 1 -- 172.21.15.29:0/833107420 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc684003e20 con 0x7fc68c108780 2024-07-27T06:42:39.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc694785640 1 -- 172.21.15.29:0/833107420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc68c1a02b0 con 0x7fc68c108780 2024-07-27T06:42:39.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc6837fe640 1 -- 172.21.15.29:0/833107420 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc684010460 con 0x7fc68c108780 2024-07-27T06:42:39.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.521+0000 7fc694785640 1 -- 172.21.15.29:0/833107420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc68c1a6a10 con 0x7fc68c108780 2024-07-27T06:42:39.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.522+0000 7fc694785640 1 -- 172.21.15.29:0/833107420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc68c102fd0 con 0x7fc68c108780 2024-07-27T06:42:39.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.524+0000 7fc6837fe640 1 -- 172.21.15.29:0/833107420 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fc684003970 con 0x7fc68c108780 2024-07-27T06:42:39.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.525+0000 7fc6837fe640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc6780b3ca0 0x7fc6780b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:39.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.525+0000 7fc692521640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc6780b3ca0 0x7fc6780b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:39.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.526+0000 7fc6837fe640 1 -- 172.21.15.29:0/833107420 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fc6840e3480 con 0x7fc68c108780 2024-07-27T06:42:39.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.526+0000 7fc692521640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc6780b3ca0 0x7fc6780b61b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc688009bd0 tx=0x7fc688005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:39.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.529+0000 7fc6837fe640 1 -- 172.21.15.29:0/833107420 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc6840aa870 con 0x7fc68c108780 2024-07-27T06:42:39.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.749+0000 7fc694785640 1 -- 172.21.15.29:0/833107420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc68c19b180 con 0x7fc68c108780 2024-07-27T06:42:39.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.749+0000 7fc6837fe640 1 -- 172.21.15.29:0/833107420 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4735 (secure 0 0 0) 0x7fc68c19b180 con 0x7fc68c108780 2024-07-27T06:42:39.751 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:42:39.751 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":6,"btime":"2024-07-27T06:42:36:870896+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24542,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24566,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-07-27T06:42:39.751 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 6 2024-07-27T06:42:39.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.751+0000 7fc694785640 1 -- 172.21.15.29:0/833107420 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc6780b3ca0 msgr2=0x7fc6780b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:39.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.751+0000 7fc694785640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc6780b3ca0 0x7fc6780b61b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc688009bd0 tx=0x7fc688005f90 comp rx=0 tx=0).stop 2024-07-27T06:42:39.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.751+0000 7fc694785640 1 -- 172.21.15.29:0/833107420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc68c108780 msgr2=0x7fc68c19f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:39.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.751+0000 7fc694785640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc68c108780 0x7fc68c19f890 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc684009fd0 tx=0x7fc68400b260 comp rx=0 tx=0).stop 2024-07-27T06:42:39.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.751+0000 7fc694785640 1 -- 172.21.15.29:0/833107420 shutdown_connections 2024-07-27T06:42:39.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.751+0000 7fc694785640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc68c108780 0x7fc68c19f890 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:39.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.751+0000 7fc694785640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc6780b3ca0 0x7fc6780b61b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:39.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.751+0000 7fc694785640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc68c107e30 0x7fc68c19a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:39.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.751+0000 7fc694785640 1 --2- 172.21.15.29:0/833107420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc68c102bc0 0x7fc68c19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:39.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.751+0000 7fc694785640 1 -- 172.21.15.29:0/833107420 >> 172.21.15.29:0/833107420 conn(0x7fc68c0799c0 msgr2=0x7fc68c100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:39.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.751+0000 7fc694785640 1 -- 172.21.15.29:0/833107420 shutdown_connections 2024-07-27T06:42:39.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:39.751+0000 7fc694785640 1 -- 172.21.15.29:0/833107420 wait complete. 2024-07-27T06:42:40.341 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:40.341 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:40.341 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:40.341 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:40.342 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:40.342 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:40.342 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:40 smithi077 ceph-mon[37153]: Deploying daemon mds.c on smithi077 2024-07-27T06:42:40.342 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:40 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/833107420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:40.342 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:40 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] up:boot 2024-07-27T06:42:40.342 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:40 smithi077 ceph-mon[37153]: fsmap 6 up:standby 2024-07-27T06:42:40.342 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-27T06:42:40.397 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:40.397 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:40.397 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:40.397 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:40.397 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:40.397 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:40.397 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:40 smithi029 ceph-mon[32610]: Deploying daemon mds.c on smithi077 2024-07-27T06:42:40.397 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:40 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/833107420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:40.397 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:40 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] up:boot 2024-07-27T06:42:40.398 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:40 smithi029 ceph-mon[32610]: fsmap 6 up:standby 2024-07-27T06:42:40.398 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-27T06:42:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:40 smithi047 ceph-mon[36835]: Deploying daemon mds.c on smithi077 2024-07-27T06:42:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:40 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/833107420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:40 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] up:boot 2024-07-27T06:42:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:40 smithi047 ceph-mon[36835]: fsmap 6 up:standby 2024-07-27T06:42:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-27T06:42:41.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:41 smithi029 ceph-mon[32610]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:41.470 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:41 smithi077 ceph-mon[37153]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:41 smithi047 ceph-mon[36835]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:41.856 INFO:journalctl@ceph.mds.c.smithi077.stdout:Jul 27 06:42:41 smithi077 systemd[1]: Started Ceph mds.c for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:42:42.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:42:42.701 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:42:43.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:42 smithi029 ceph-mon[32610]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:43.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:43.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:43.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:43.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:43.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:43.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:43.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:42 smithi029 ceph-mon[32610]: Deploying daemon mds.h on smithi047 2024-07-27T06:42:43.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:42 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] up:boot 2024-07-27T06:42:43.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:42 smithi029 ceph-mon[32610]: fsmap 7 up:standby 2024-07-27T06:42:43.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-27T06:42:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:42 smithi077 ceph-mon[37153]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:42 smithi077 ceph-mon[37153]: Deploying daemon mds.h on smithi047 2024-07-27T06:42:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:42 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] up:boot 2024-07-27T06:42:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:42 smithi077 ceph-mon[37153]: fsmap 7 up:standby 2024-07-27T06:42:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-27T06:42:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:42 smithi047 ceph-mon[36835]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:42 smithi047 ceph-mon[36835]: Deploying daemon mds.h on smithi047 2024-07-27T06:42:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:42 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] up:boot 2024-07-27T06:42:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:42 smithi047 ceph-mon[36835]: fsmap 7 up:standby 2024-07-27T06:42:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-27T06:42:43.946 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:42:43 smithi047 systemd[1]: Starting Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:42:44.245 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:42:43 smithi047 podman[68330]: 2024-07-27 06:42:43.890633094 +0000 UTC m=+0.017550951 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:42:44.245 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:42:44 smithi047 podman[68330]: 2024-07-27 06:42:44.024612532 +0000 UTC m=+0.151530379 container create d8f4ee956badb55e0699ae09ded162d0cd9f9944acff00bd81f3e1b45513f034 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-07-27T06:42:44.245 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:42:44 smithi047 podman[68330]: 2024-07-27 06:42:44.244217859 +0000 UTC m=+0.371135725 container init d8f4ee956badb55e0699ae09ded162d0cd9f9944acff00bd81f3e1b45513f034 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS) 2024-07-27T06:42:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.452+0000 7f18614c2640 1 -- 172.21.15.29:0/43212851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185c108720 msgr2=0x7f185c10ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.452+0000 7f18614c2640 1 --2- 172.21.15.29:0/43212851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185c108720 0x7f185c10ef50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1854009900 tx=0x7f185402a410 comp rx=0 tx=0).stop 2024-07-27T06:42:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.452+0000 7f18614c2640 1 -- 172.21.15.29:0/43212851 shutdown_connections 2024-07-27T06:42:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.452+0000 7f18614c2640 1 --2- 172.21.15.29:0/43212851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185c108720 0x7f185c10ef50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.452+0000 7f18614c2640 1 --2- 172.21.15.29:0/43212851 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f185c107dd0 0x7f185c1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.452+0000 7f18614c2640 1 --2- 172.21.15.29:0/43212851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c102490 0x7f185c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.452+0000 7f18614c2640 1 -- 172.21.15.29:0/43212851 >> 172.21.15.29:0/43212851 conn(0x7f185c079190 msgr2=0x7f185c0795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.452+0000 7f18614c2640 1 -- 172.21.15.29:0/43212851 shutdown_connections 2024-07-27T06:42:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.452+0000 7f18614c2640 1 -- 172.21.15.29:0/43212851 wait complete. 2024-07-27T06:42:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.453+0000 7f18614c2640 1 Processor -- start 2024-07-27T06:42:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.453+0000 7f18614c2640 1 -- start start 2024-07-27T06:42:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.453+0000 7f18614c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c102490 0x7f185c199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.453+0000 7f18614c2640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f185c107dd0 0x7f185c19a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.453+0000 7f18614c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185c108720 0x7f185c19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.453+0000 7f18614c2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f185c1134f0 con 0x7f185c108720 2024-07-27T06:42:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.453+0000 7f18614c2640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f185c113370 con 0x7f185c107dd0 2024-07-27T06:42:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.453+0000 7f18614c2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f185c113670 con 0x7f185c102490 2024-07-27T06:42:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f185b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185c108720 0x7f185c19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f185b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185c108720 0x7f185c19f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51140/0 (socket says 172.21.15.29:51140) 2024-07-27T06:42:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f185affd640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c102490 0x7f185c199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f185b7fe640 1 -- 172.21.15.29:0/3107871938 learned_addr learned my addr 172.21.15.29:0/3107871938 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:42:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f185a7fc640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f185c107dd0 0x7f185c19a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f185b7fe640 1 -- 172.21.15.29:0/3107871938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c102490 msgr2=0x7f185c199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f185b7fe640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c102490 0x7f185c199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f185b7fe640 1 -- 172.21.15.29:0/3107871938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f185c107dd0 msgr2=0x7f185c19a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f185b7fe640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f185c107dd0 0x7f185c19a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f185b7fe640 1 -- 172.21.15.29:0/3107871938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185c19feb0 con 0x7f185c108720 2024-07-27T06:42:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f185a7fc640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f185c107dd0 0x7f185c19a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:42:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f185b7fe640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185c108720 0x7f185c19f780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1854004770 tx=0x7f1854004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f1843fff640 1 -- 172.21.15.29:0/3107871938 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185403f030 con 0x7f185c108720 2024-07-27T06:42:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f18614c2640 1 -- 172.21.15.29:0/3107871938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f185c1a0140 con 0x7f185c108720 2024-07-27T06:42:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f1843fff640 1 -- 172.21.15.29:0/3107871938 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1854004440 con 0x7f185c108720 2024-07-27T06:42:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f1843fff640 1 -- 172.21.15.29:0/3107871938 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1854035070 con 0x7f185c108720 2024-07-27T06:42:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.454+0000 7f18614c2640 1 -- 172.21.15.29:0/3107871938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f185c1a6900 con 0x7f185c108720 2024-07-27T06:42:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.455+0000 7f18614c2640 1 -- 172.21.15.29:0/3107871938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f185c103710 con 0x7f185c108720 2024-07-27T06:42:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.458+0000 7f1843fff640 1 -- 172.21.15.29:0/3107871938 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f1854042030 con 0x7f185c108720 2024-07-27T06:42:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.459+0000 7f1843fff640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f18440b3c20 0x7f18440b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.459+0000 7f185affd640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f18440b3c20 0x7f18440b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.459+0000 7f1843fff640 1 -- 172.21.15.29:0/3107871938 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f1854103f20 con 0x7f185c108720 2024-07-27T06:42:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.459+0000 7f185affd640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f18440b3c20 0x7f18440b6130 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f184c005fd0 tx=0x7f184c009500 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.462+0000 7f1843fff640 1 -- 172.21.15.29:0/3107871938 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f18540cb8a0 con 0x7f185c108720 2024-07-27T06:42:44.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.689+0000 7f18614c2640 1 -- 172.21.15.29:0/3107871938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f185c1028a0 con 0x7f185c108720 2024-07-27T06:42:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.689+0000 7f1843fff640 1 -- 172.21.15.29:0/3107871938 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6382 (secure 0 0 0) 0x7f18540cb1a0 con 0x7f185c108720 2024-07-27T06:42:44.691 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:42:44.691 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"btime":"2024-07-27T06:42:42:208023+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24542,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24566,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24646,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24658,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-07-27T06:42:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2024-07-27T06:42:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.692+0000 7f18614c2640 1 -- 172.21.15.29:0/3107871938 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f18440b3c20 msgr2=0x7f18440b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.692+0000 7f18614c2640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f18440b3c20 0x7f18440b6130 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f184c005fd0 tx=0x7f184c009500 comp rx=0 tx=0).stop 2024-07-27T06:42:44.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.692+0000 7f18614c2640 1 -- 172.21.15.29:0/3107871938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185c108720 msgr2=0x7f185c19f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:44.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.692+0000 7f18614c2640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185c108720 0x7f185c19f780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1854004770 tx=0x7f1854004040 comp rx=0 tx=0).stop 2024-07-27T06:42:44.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.692+0000 7f18614c2640 1 -- 172.21.15.29:0/3107871938 shutdown_connections 2024-07-27T06:42:44.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.692+0000 7f18614c2640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f185c108720 0x7f185c19f780 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:44.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.692+0000 7f18614c2640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f185c107dd0 0x7f185c19a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:44.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.692+0000 7f18614c2640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f18440b3c20 0x7f18440b6130 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:44.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.692+0000 7f18614c2640 1 --2- 172.21.15.29:0/3107871938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f185c102490 0x7f185c199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:44.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.692+0000 7f18614c2640 1 -- 172.21.15.29:0/3107871938 >> 172.21.15.29:0/3107871938 conn(0x7f185c079190 msgr2=0x7f185c103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:44.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.692+0000 7f18614c2640 1 -- 172.21.15.29:0/3107871938 shutdown_connections 2024-07-27T06:42:44.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:44.692+0000 7f18614c2640 1 -- 172.21.15.29:0/3107871938 wait complete. 2024-07-27T06:42:44.696 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:42:44 smithi047 podman[68330]: 2024-07-27 06:42:44.247069409 +0000 UTC m=+0.373987261 container start d8f4ee956badb55e0699ae09ded162d0cd9f9944acff00bd81f3e1b45513f034 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-07-27T06:42:44.696 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:42:44 smithi047 ceph-mds[68350]: -- 172.21.15.47:0/2588400587 <== mon.2 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d4e7e99860 con 0x55d4ec03b180 2024-07-27T06:42:44.696 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:42:44 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: starting mds.h at 2024-07-27T06:42:44.696 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:42:44 smithi047 bash[68330]: d8f4ee956badb55e0699ae09ded162d0cd9f9944acff00bd81f3e1b45513f034 2024-07-27T06:42:44.696 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:42:44 smithi047 systemd[1]: Started Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:42:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:45 smithi077 ceph-mon[37153]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:45 smithi077 ceph-mon[37153]: Deploying daemon mds.i on smithi077 2024-07-27T06:42:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:45 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3107871938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:45 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:boot 2024-07-27T06:42:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:45 smithi077 ceph-mon[37153]: fsmap 8 up:standby 2024-07-27T06:42:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:42:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:45 smithi047 ceph-mon[36835]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:45 smithi047 ceph-mon[36835]: Deploying daemon mds.i on smithi077 2024-07-27T06:42:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:45 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3107871938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:45 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:boot 2024-07-27T06:42:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:45 smithi047 ceph-mon[36835]: fsmap 8 up:standby 2024-07-27T06:42:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:42:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:45 smithi029 ceph-mon[32610]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:45 smithi029 ceph-mon[32610]: Deploying daemon mds.i on smithi077 2024-07-27T06:42:45.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:45 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3107871938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:45.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:45 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:boot 2024-07-27T06:42:45.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:45 smithi029 ceph-mon[32610]: fsmap 8 up:standby 2024-07-27T06:42:45.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:42:46.357 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:42:46 smithi077 systemd[1]: Starting Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:42:46.703 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:42:46 smithi077 podman[69346]: 2024-07-27 06:42:46.337804944 +0000 UTC m=+0.017284064 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:42:46.703 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:42:46 smithi077 podman[69346]: 2024-07-27 06:42:46.440833058 +0000 UTC m=+0.120312219 container create 12d1fea815231b4a7ee0b98aa2eb7e4c055b4cb0bf7889434f6769af724287fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-27T06:42:46.703 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:42:46 smithi077 podman[69346]: 2024-07-27 06:42:46.70197814 +0000 UTC m=+0.381457260 container init 12d1fea815231b4a7ee0b98aa2eb7e4c055b4cb0bf7889434f6769af724287fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-07-27T06:42:47.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:42:46 smithi077 podman[69346]: 2024-07-27 06:42:46.704617113 +0000 UTC m=+0.384096232 container start 12d1fea815231b4a7ee0b98aa2eb7e4c055b4cb0bf7889434f6769af724287fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-27T06:42:47.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:42:46 smithi077 ceph-mds[69365]: -- 172.21.15.77:0/3419371670 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5563bc551860 con 0x5563c06f3a80 2024-07-27T06:42:47.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:42:46 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: starting mds.i at 2024-07-27T06:42:47.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:42:46 smithi077 bash[69346]: 12d1fea815231b4a7ee0b98aa2eb7e4c055b4cb0bf7889434f6769af724287fa 2024-07-27T06:42:47.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:42:46 smithi077 systemd[1]: Started Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:42:47.344 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:42:47.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:47 smithi029 ceph-mon[32610]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:47.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:47.587 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:47.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:47.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:47.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:47.588 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:47 smithi047 ceph-mon[36835]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:47 smithi077 ceph-mon[37153]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:42:48.681 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:48 smithi029 ceph-mon[32610]: Deploying daemon mds.g on smithi029 2024-07-27T06:42:48.681 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:48 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] up:boot 2024-07-27T06:42:48.682 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:48 smithi029 ceph-mon[32610]: fsmap 9 up:standby 2024-07-27T06:42:48.682 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:42:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:48 smithi047 ceph-mon[36835]: Deploying daemon mds.g on smithi029 2024-07-27T06:42:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:48 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] up:boot 2024-07-27T06:42:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:48 smithi047 ceph-mon[36835]: fsmap 9 up:standby 2024-07-27T06:42:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:42:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:48 smithi077 ceph-mon[37153]: Deploying daemon mds.g on smithi029 2024-07-27T06:42:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:48 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] up:boot 2024-07-27T06:42:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:48 smithi077 ceph-mon[37153]: fsmap 9 up:standby 2024-07-27T06:42:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:42:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:49 smithi047 ceph-mon[36835]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:49.770 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:49 smithi029 ceph-mon[32610]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:49 smithi077 ceph-mon[37153]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.935+0000 7ff55ef89640 1 -- 172.21.15.29:0/2626278854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff558102490 msgr2=0x7ff5581028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.935+0000 7ff55ef89640 1 --2- 172.21.15.29:0/2626278854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff558102490 0x7ff5581028a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff548009900 tx=0x7ff54802a410 comp rx=0 tx=0).stop 2024-07-27T06:42:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.936+0000 7ff55ef89640 1 -- 172.21.15.29:0/2626278854 shutdown_connections 2024-07-27T06:42:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.936+0000 7ff55ef89640 1 --2- 172.21.15.29:0/2626278854 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff558108720 0x7ff55810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.936+0000 7ff55ef89640 1 --2- 172.21.15.29:0/2626278854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff558107dd0 0x7ff5581081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.936+0000 7ff55ef89640 1 --2- 172.21.15.29:0/2626278854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff558102490 0x7ff5581028a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:50.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.936+0000 7ff55ef89640 1 -- 172.21.15.29:0/2626278854 >> 172.21.15.29:0/2626278854 conn(0x7ff558079160 msgr2=0x7ff558079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.936+0000 7ff55ef89640 1 -- 172.21.15.29:0/2626278854 shutdown_connections 2024-07-27T06:42:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.936+0000 7ff55ef89640 1 -- 172.21.15.29:0/2626278854 wait complete. 2024-07-27T06:42:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.936+0000 7ff55ef89640 1 Processor -- start 2024-07-27T06:42:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff55ef89640 1 -- start start 2024-07-27T06:42:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff55ef89640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff558102490 0x7ff558103c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff55ef89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff558107dd0 0x7ff5581041c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff55ef89640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff558108720 0x7ff558104700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff55ef89640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff558112380 con 0x7ff558107dd0 2024-07-27T06:42:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff55ef89640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7ff558112200 con 0x7ff558108720 2024-07-27T06:42:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff55ef89640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff558112500 con 0x7ff558102490 2024-07-27T06:42:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff557fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff558107dd0 0x7ff5581041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff557fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff558107dd0 0x7ff5581041c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45538/0 (socket says 172.21.15.29:45538) 2024-07-27T06:42:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff557fff640 1 -- 172.21.15.29:0/560141871 learned_addr learned my addr 172.21.15.29:0/560141871 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:42:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff55cd25640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff558102490 0x7ff558103c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff55d526640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff558108720 0x7ff558104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff557fff640 1 -- 172.21.15.29:0/560141871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff558102490 msgr2=0x7ff558103c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff557fff640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff558102490 0x7ff558103c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff557fff640 1 -- 172.21.15.29:0/560141871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff558108720 msgr2=0x7ff558104700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff557fff640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff558108720 0x7ff558104700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff557fff640 1 -- 172.21.15.29:0/560141871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5581a8160 con 0x7ff558107dd0 2024-07-27T06:42:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.937+0000 7ff55d526640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff558108720 0x7ff558104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:42:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.938+0000 7ff557fff640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff558107dd0 0x7ff5581041c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff54c005fd0 tx=0x7ff54c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.938+0000 7ff555ffb640 1 -- 172.21.15.29:0/560141871 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c017070 con 0x7ff558107dd0 2024-07-27T06:42:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.938+0000 7ff55ef89640 1 -- 172.21.15.29:0/560141871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5581a8450 con 0x7ff558107dd0 2024-07-27T06:42:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.938+0000 7ff555ffb640 1 -- 172.21.15.29:0/560141871 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff54c009c30 con 0x7ff558107dd0 2024-07-27T06:42:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.938+0000 7ff555ffb640 1 -- 172.21.15.29:0/560141871 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54c003d30 con 0x7ff558107dd0 2024-07-27T06:42:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.938+0000 7ff55ef89640 1 -- 172.21.15.29:0/560141871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5581a8990 con 0x7ff558107dd0 2024-07-27T06:42:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.939+0000 7ff55ef89640 1 -- 172.21.15.29:0/560141871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5581028a0 con 0x7ff558107dd0 2024-07-27T06:42:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.942+0000 7ff555ffb640 1 -- 172.21.15.29:0/560141871 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7ff54c003ed0 con 0x7ff558107dd0 2024-07-27T06:42:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.943+0000 7ff555ffb640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff5400b3c20 0x7ff5400b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.943+0000 7ff55cd25640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff5400b3c20 0x7ff5400b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.943+0000 7ff555ffb640 1 -- 172.21.15.29:0/560141871 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7ff54c0e3ec0 con 0x7ff558107dd0 2024-07-27T06:42:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.943+0000 7ff55cd25640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff5400b3c20 0x7ff5400b6130 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff548000c00 tx=0x7ff548031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:50.945+0000 7ff555ffb640 1 -- 172.21.15.29:0/560141871 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff54c020030 con 0x7ff558107dd0 2024-07-27T06:42:51.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.173+0000 7ff55ef89640 1 -- 172.21.15.29:0/560141871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5581a8d90 con 0x7ff558107dd0 2024-07-27T06:42:51.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.174+0000 7ff555ffb640 1 -- 172.21.15.29:0/560141871 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8031 (secure 0 0 0) 0x7ff54c009de0 con 0x7ff558107dd0 2024-07-27T06:42:51.175 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:42:51.175 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":10,"btime":"2024-07-27T06:42:47:389433+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24542,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24566,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24584,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24646,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24658,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24670,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-07-27T06:42:51.176 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 10 2024-07-27T06:42:51.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.176+0000 7ff55ef89640 1 -- 172.21.15.29:0/560141871 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff5400b3c20 msgr2=0x7ff5400b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:51.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.176+0000 7ff55ef89640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff5400b3c20 0x7ff5400b6130 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff548000c00 tx=0x7ff548031400 comp rx=0 tx=0).stop 2024-07-27T06:42:51.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.176+0000 7ff55ef89640 1 -- 172.21.15.29:0/560141871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff558107dd0 msgr2=0x7ff5581041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:51.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.176+0000 7ff55ef89640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff558107dd0 0x7ff5581041c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff54c005fd0 tx=0x7ff54c00dde0 comp rx=0 tx=0).stop 2024-07-27T06:42:51.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.176+0000 7ff55ef89640 1 -- 172.21.15.29:0/560141871 shutdown_connections 2024-07-27T06:42:51.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.176+0000 7ff55ef89640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff558108720 0x7ff558104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:51.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.176+0000 7ff55ef89640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff558107dd0 0x7ff5581041c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:51.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.176+0000 7ff55ef89640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff5400b3c20 0x7ff5400b6130 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:51.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.176+0000 7ff55ef89640 1 --2- 172.21.15.29:0/560141871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff558102490 0x7ff558103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:51.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.176+0000 7ff55ef89640 1 -- 172.21.15.29:0/560141871 >> 172.21.15.29:0/560141871 conn(0x7ff558079160 msgr2=0x7ff558101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:51.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.176+0000 7ff55ef89640 1 -- 172.21.15.29:0/560141871 shutdown_connections 2024-07-27T06:42:51.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:51.176+0000 7ff55ef89640 1 -- 172.21.15.29:0/560141871 wait complete. 2024-07-27T06:42:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:51 smithi029 ceph-mon[32610]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:51.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:51.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:51.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:51 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/560141871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:51 smithi047 ceph-mon[36835]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:51.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:51.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:51.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:51.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:51 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/560141871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:51 smithi077 ceph-mon[37153]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:51.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:51.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:51 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/560141871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:52 smithi077 ceph-mon[37153]: Deploying daemon mds.a on smithi029 2024-07-27T06:42:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:52 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6838/1639930501,v1:172.21.15.29:6839/1639930501] up:boot 2024-07-27T06:42:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:52 smithi077 ceph-mon[37153]: fsmap 10 up:standby 2024-07-27T06:42:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-27T06:42:52.662 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:52 smithi029 ceph-mon[32610]: Deploying daemon mds.a on smithi029 2024-07-27T06:42:52.663 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:52 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6838/1639930501,v1:172.21.15.29:6839/1639930501] up:boot 2024-07-27T06:42:52.663 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:52 smithi029 ceph-mon[32610]: fsmap 10 up:standby 2024-07-27T06:42:52.663 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-27T06:42:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:52 smithi047 ceph-mon[36835]: Deploying daemon mds.a on smithi029 2024-07-27T06:42:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:52 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6838/1639930501,v1:172.21.15.29:6839/1639930501] up:boot 2024-07-27T06:42:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:52 smithi047 ceph-mon[36835]: fsmap 10 up:standby 2024-07-27T06:42:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-27T06:42:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:53 smithi047 ceph-mon[36835]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:53.707 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:53 smithi029 ceph-mon[32610]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:53.708 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jul 27 06:42:53 smithi029 systemd[1]: Starting Ceph mds.a for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:42:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:53 smithi077 ceph-mon[37153]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:53.962 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jul 27 06:42:53 smithi029 podman[78769]: 2024-07-27 06:42:53.609395632 +0000 UTC m=+0.017862140 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:42:53.962 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jul 27 06:42:53 smithi029 podman[78769]: 2024-07-27 06:42:53.721661225 +0000 UTC m=+0.130127716 container create 5c7d8d8a910a904c83c7ee8e45e7752ca68384baf30756a112c9c14088aa7a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-a, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-07-27T06:42:54.054 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:42:54.332 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:42:54.456 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jul 27 06:42:53 smithi029 podman[78769]: 2024-07-27 06:42:53.97007897 +0000 UTC m=+0.378545464 container init 5c7d8d8a910a904c83c7ee8e45e7752ca68384baf30756a112c9c14088aa7a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0) 2024-07-27T06:42:54.456 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jul 27 06:42:53 smithi029 podman[78769]: 2024-07-27 06:42:53.973036308 +0000 UTC m=+0.381502801 container start 5c7d8d8a910a904c83c7ee8e45e7752ca68384baf30756a112c9c14088aa7a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-a, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-07-27T06:42:54.456 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jul 27 06:42:54 smithi029 ceph-mds[78788]: -- 172.21.15.29:0/2949070392 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5625cefbbc20 con 0x5625d315da80 2024-07-27T06:42:54.456 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jul 27 06:42:54 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-a[78784]: starting mds.a at 2024-07-27T06:42:54.456 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jul 27 06:42:54 smithi029 bash[78769]: 5c7d8d8a910a904c83c7ee8e45e7752ca68384baf30756a112c9c14088aa7a82 2024-07-27T06:42:54.456 INFO:journalctl@ceph.mds.a.smithi029.stdout:Jul 27 06:42:54 smithi029 systemd[1]: Started Ceph mds.a for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:42:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:55 smithi047 ceph-mon[36835]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:55 smithi047 ceph-mon[36835]: Deploying daemon mds.b on smithi047 2024-07-27T06:42:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:55 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6840/2982668187,v1:172.21.15.29:6841/2982668187] up:boot 2024-07-27T06:42:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:55 smithi047 ceph-mon[36835]: fsmap 11 up:standby 2024-07-27T06:42:55.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-27T06:42:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:55 smithi029 ceph-mon[32610]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:55.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:55.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:55 smithi029 ceph-mon[32610]: Deploying daemon mds.b on smithi047 2024-07-27T06:42:55.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:55 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6840/2982668187,v1:172.21.15.29:6841/2982668187] up:boot 2024-07-27T06:42:55.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:55 smithi029 ceph-mon[32610]: fsmap 11 up:standby 2024-07-27T06:42:55.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-27T06:42:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:55 smithi077 ceph-mon[37153]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-27T06:42:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-27T06:42:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:55 smithi077 ceph-mon[37153]: Deploying daemon mds.b on smithi047 2024-07-27T06:42:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:55 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6840/2982668187,v1:172.21.15.29:6841/2982668187] up:boot 2024-07-27T06:42:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:55 smithi077 ceph-mon[37153]: fsmap 11 up:standby 2024-07-27T06:42:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-27T06:42:56.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.153+0000 7f3abd58c640 1 -- 172.21.15.29:0/1326076383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ab8107dd0 msgr2=0x7f3ab81081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.153+0000 7f3abd58c640 1 --2- 172.21.15.29:0/1326076383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ab8107dd0 0x7f3ab81081e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009900 tx=0x7f3aac02a410 comp rx=0 tx=0).stop 2024-07-27T06:42:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.153+0000 7f3abd58c640 1 -- 172.21.15.29:0/1326076383 shutdown_connections 2024-07-27T06:42:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.153+0000 7f3abd58c640 1 --2- 172.21.15.29:0/1326076383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ab8108720 0x7f3ab810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.153+0000 7f3abd58c640 1 --2- 172.21.15.29:0/1326076383 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ab8107dd0 0x7f3ab81081e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.153+0000 7f3abd58c640 1 --2- 172.21.15.29:0/1326076383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ab8102490 0x7f3ab81028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.153+0000 7f3abd58c640 1 -- 172.21.15.29:0/1326076383 >> 172.21.15.29:0/1326076383 conn(0x7f3ab8079160 msgr2=0x7f3ab8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.153+0000 7f3abd58c640 1 -- 172.21.15.29:0/1326076383 shutdown_connections 2024-07-27T06:42:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.153+0000 7f3abd58c640 1 -- 172.21.15.29:0/1326076383 wait complete. 2024-07-27T06:42:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.154+0000 7f3abd58c640 1 Processor -- start 2024-07-27T06:42:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.154+0000 7f3abd58c640 1 -- start start 2024-07-27T06:42:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.154+0000 7f3abd58c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ab8102490 0x7f3ab819c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.154+0000 7f3abd58c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ab8107dd0 0x7f3ab819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.154+0000 7f3abd58c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ab8108720 0x7f3ab81a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.154+0000 7f3abd58c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab81123d0 con 0x7f3ab8102490 2024-07-27T06:42:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.154+0000 7f3abd58c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab8112250 con 0x7f3ab8107dd0 2024-07-27T06:42:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.154+0000 7f3abd58c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab8112550 con 0x7f3ab8108720 2024-07-27T06:42:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ab8102490 0x7f3ab819c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ab8102490 0x7f3ab819c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45562/0 (socket says 172.21.15.29:45562) 2024-07-27T06:42:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab6ffd640 1 -- 172.21.15.29:0/2262298376 learned_addr learned my addr 172.21.15.29:0/2262298376 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:42:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab67fc640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ab8107dd0 0x7f3ab819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab77fe640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ab8108720 0x7f3ab81a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab6ffd640 1 -- 172.21.15.29:0/2262298376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ab8108720 msgr2=0x7f3ab81a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab6ffd640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ab8108720 0x7f3ab81a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:56.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab6ffd640 1 -- 172.21.15.29:0/2262298376 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ab8107dd0 msgr2=0x7f3ab819c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab6ffd640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ab8107dd0 0x7f3ab819c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab6ffd640 1 -- 172.21.15.29:0/2262298376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab81a2100 con 0x7f3ab8102490 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab77fe640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ab8108720 0x7f3ab81a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab67fc640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ab8107dd0 0x7f3ab819c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3ab6ffd640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ab8102490 0x7f3ab819c190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800e9f0 tx=0x7f3aa800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3a9ffff640 1 -- 172.21.15.29:0/2262298376 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa800cd80 con 0x7f3ab8102490 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3a9ffff640 1 -- 172.21.15.29:0/2262298376 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aa80108e0 con 0x7f3ab8102490 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3a9ffff640 1 -- 172.21.15.29:0/2262298376 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8018980 con 0x7f3ab8102490 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3abd58c640 1 -- 172.21.15.29:0/2262298376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab81a23f0 con 0x7f3ab8102490 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.155+0000 7f3abd58c640 1 -- 172.21.15.29:0/2262298376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab81a8b50 con 0x7f3ab8102490 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.157+0000 7f3abd58c640 1 -- 172.21.15.29:0/2262298376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ab81028a0 con 0x7f3ab8102490 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.158+0000 7f3a9ffff640 1 -- 172.21.15.29:0/2262298376 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f3aa8010430 con 0x7f3ab8102490 2024-07-27T06:42:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.159+0000 7f3a9ffff640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3aa00b3c20 0x7f3aa00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:42:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.159+0000 7f3ab67fc640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3aa00b3c20 0x7f3aa00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:42:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.160+0000 7f3a9ffff640 1 -- 172.21.15.29:0/2262298376 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f3aa80e41b0 con 0x7f3ab8102490 2024-07-27T06:42:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.160+0000 7f3ab67fc640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3aa00b3c20 0x7f3aa00b6130 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3aac000c00 tx=0x7f3aac031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:42:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.163+0000 7f3a9ffff640 1 -- 172.21.15.29:0/2262298376 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3aa80abb30 con 0x7f3ab8102490 2024-07-27T06:42:56.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.381+0000 7f3abd58c640 1 -- 172.21.15.29:0/2262298376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ab81a8e40 con 0x7f3ab8102490 2024-07-27T06:42:56.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.382+0000 7f3a9ffff640 1 -- 172.21.15.29:0/2262298376 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9683 (secure 0 0 0) 0x7f3aa80ab430 con 0x7f3ab8102490 2024-07-27T06:42:56.383 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:42:56.384 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":12,"btime":"2024-07-27T06:42:54:417207+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14781,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14793,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24542,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24566,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24584,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24646,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24658,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24670,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-07-27T06:42:56.385 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 12 2024-07-27T06:42:56.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.384+0000 7f3abd58c640 1 -- 172.21.15.29:0/2262298376 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3aa00b3c20 msgr2=0x7f3aa00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:56.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.384+0000 7f3abd58c640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3aa00b3c20 0x7f3aa00b6130 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3aac000c00 tx=0x7f3aac031400 comp rx=0 tx=0).stop 2024-07-27T06:42:56.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.384+0000 7f3abd58c640 1 -- 172.21.15.29:0/2262298376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ab8102490 msgr2=0x7f3ab819c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:42:56.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.384+0000 7f3abd58c640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ab8102490 0x7f3ab819c190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800e9f0 tx=0x7f3aa800eeb0 comp rx=0 tx=0).stop 2024-07-27T06:42:56.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.384+0000 7f3abd58c640 1 -- 172.21.15.29:0/2262298376 shutdown_connections 2024-07-27T06:42:56.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.384+0000 7f3abd58c640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3aa00b3c20 0x7f3aa00b6130 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:56.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.384+0000 7f3abd58c640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ab8108720 0x7f3ab81a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:56.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.384+0000 7f3abd58c640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ab8107dd0 0x7f3ab819c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:56.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.384+0000 7f3abd58c640 1 --2- 172.21.15.29:0/2262298376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ab8102490 0x7f3ab819c190 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:42:56.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.384+0000 7f3abd58c640 1 -- 172.21.15.29:0/2262298376 >> 172.21.15.29:0/2262298376 conn(0x7f3ab8079160 msgr2=0x7f3ab806a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:42:56.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.384+0000 7f3abd58c640 1 -- 172.21.15.29:0/2262298376 shutdown_connections 2024-07-27T06:42:56.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:42:56.384+0000 7f3abd58c640 1 -- 172.21.15.29:0/2262298376 wait complete. 2024-07-27T06:42:56.442 INFO:journalctl@ceph.mds.b.smithi047.stdout:Jul 27 06:42:56 smithi047 systemd[1]: Starting Ceph mds.b for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:42:56.696 INFO:journalctl@ceph.mds.b.smithi047.stdout:Jul 27 06:42:56 smithi047 podman[68637]: 2024-07-27 06:42:56.344465607 +0000 UTC m=+0.018179191 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:42:56.696 INFO:journalctl@ceph.mds.b.smithi047.stdout:Jul 27 06:42:56 smithi047 podman[68637]: 2024-07-27 06:42:56.464696414 +0000 UTC m=+0.138409991 container create 6f166d060dc5adbff28266f3ed2cc0615e451597338225a54199696ef81a5de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_CLEAN=True, ceph=True) 2024-07-27T06:42:56.697 INFO:journalctl@ceph.mds.b.smithi047.stdout:Jul 27 06:42:56 smithi047 podman[68637]: 2024-07-27 06:42:56.667660714 +0000 UTC m=+0.341374311 container init 6f166d060dc5adbff28266f3ed2cc0615e451597338225a54199696ef81a5de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-07-27T06:42:56.697 INFO:journalctl@ceph.mds.b.smithi047.stdout:Jul 27 06:42:56 smithi047 podman[68637]: 2024-07-27 06:42:56.670283525 +0000 UTC m=+0.343997106 container start 6f166d060dc5adbff28266f3ed2cc0615e451597338225a54199696ef81a5de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-07-27T06:42:57.127 INFO:journalctl@ceph.mds.b.smithi047.stdout:Jul 27 06:42:56 smithi047 ceph-mds[68656]: -- 172.21.15.47:0/3236948413 <== mon.2 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd01325860 con 0x55bd054c7a80 2024-07-27T06:42:57.127 INFO:journalctl@ceph.mds.b.smithi047.stdout:Jul 27 06:42:56 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-b[68652]: starting mds.b at 2024-07-27T06:42:57.127 INFO:journalctl@ceph.mds.b.smithi047.stdout:Jul 27 06:42:56 smithi047 bash[68637]: 6f166d060dc5adbff28266f3ed2cc0615e451597338225a54199696ef81a5de8 2024-07-27T06:42:57.127 INFO:journalctl@ceph.mds.b.smithi047.stdout:Jul 27 06:42:56 smithi047 systemd[1]: Started Ceph mds.b for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:42:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:57 smithi047 ceph-mon[36835]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:57 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2262298376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:42:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:57 smithi029 ceph-mon[32610]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:57 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2262298376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:42:57.600 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:57 smithi077 ceph-mon[37153]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:57.600 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:57 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2262298376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:42:57.600 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.600 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.600 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.600 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.601 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:57.601 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:42:58.325 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:58 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] up:boot 2024-07-27T06:42:58.325 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:58 smithi047 ceph-mon[36835]: fsmap 12 up:standby 2024-07-27T06:42:58.325 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-27T06:42:58.325 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:58 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] up:boot 2024-07-27T06:42:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:58 smithi029 ceph-mon[32610]: fsmap 12 up:standby 2024-07-27T06:42:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-27T06:42:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:58 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] up:boot 2024-07-27T06:42:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:58 smithi077 ceph-mon[37153]: fsmap 12 up:standby 2024-07-27T06:42:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-27T06:42:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:42:59.562 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:59 smithi047 ceph-mon[36835]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:59.562 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.562 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:59.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:42:59.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.563 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:42:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:42:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:59 smithi077 ceph-mon[37153]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:59.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:42:59.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:42:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:42:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:59 smithi029 ceph-mon[32610]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:42:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:42:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:42:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:42:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:42:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:42:59.815 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:01 smithi077 ceph-mon[37153]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:01.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:01 smithi029 ceph-mon[32610]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:01.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:01.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:01.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:01.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:01 smithi047 ceph-mon[36835]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:01.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:01.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.599+0000 7f4d07dc6640 1 -- 172.21.15.29:0/1809394260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d00109820 msgr2=0x7f4d00110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.599+0000 7f4d07dc6640 1 --2- 172.21.15.29:0/1809394260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d00109820 0x7f4d00110050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8009ca0 tx=0x7f4cf802a8c0 comp rx=0 tx=0).stop 2024-07-27T06:43:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.599+0000 7f4d07dc6640 1 -- 172.21.15.29:0/1809394260 shutdown_connections 2024-07-27T06:43:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.599+0000 7f4d07dc6640 1 --2- 172.21.15.29:0/1809394260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d00109820 0x7f4d00110050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.599+0000 7f4d07dc6640 1 --2- 172.21.15.29:0/1809394260 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d00108ed0 0x7f4d001092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.599+0000 7f4d07dc6640 1 --2- 172.21.15.29:0/1809394260 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d00103590 0x7f4d001039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.599+0000 7f4d07dc6640 1 -- 172.21.15.29:0/1809394260 >> 172.21.15.29:0/1809394260 conn(0x7f4d0007b280 msgr2=0x7f4d0007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.599+0000 7f4d07dc6640 1 -- 172.21.15.29:0/1809394260 shutdown_connections 2024-07-27T06:43:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.599+0000 7f4d07dc6640 1 -- 172.21.15.29:0/1809394260 wait complete. 2024-07-27T06:43:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d07dc6640 1 Processor -- start 2024-07-27T06:43:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d07dc6640 1 -- start start 2024-07-27T06:43:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d07dc6640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d00103590 0x7f4d00075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d07dc6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d00108ed0 0x7f4d00075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d07dc6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d00109820 0x7f4d00075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d07dc6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d00113540 con 0x7f4d00108ed0 2024-07-27T06:43:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d07dc6640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f4d001133c0 con 0x7f4d00103590 2024-07-27T06:43:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d07dc6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4d001136c0 con 0x7f4d00109820 2024-07-27T06:43:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d05361640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d00108ed0 0x7f4d00075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d05361640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d00108ed0 0x7f4d00075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51824/0 (socket says 172.21.15.29:51824) 2024-07-27T06:43:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d05361640 1 -- 172.21.15.29:0/3632113255 learned_addr learned my addr 172.21.15.29:0/3632113255 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d06363640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d00109820 0x7f4d00075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d05b62640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d00103590 0x7f4d00075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d05361640 1 -- 172.21.15.29:0/3632113255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d00109820 msgr2=0x7f4d00075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d05361640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d00109820 0x7f4d00075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d05361640 1 -- 172.21.15.29:0/3632113255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d00103590 msgr2=0x7f4d00075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d05361640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d00103590 0x7f4d00075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.600+0000 7f4d05361640 1 -- 172.21.15.29:0/3632113255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d001a8190 con 0x7f4d00108ed0 2024-07-27T06:43:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.601+0000 7f4d05361640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d00108ed0 0x7f4d00075680 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4cf000d740 tx=0x7f4cf000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.601+0000 7f4cf6ffd640 1 -- 172.21.15.29:0/3632113255 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0017070 con 0x7f4d00108ed0 2024-07-27T06:43:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.601+0000 7f4d07dc6640 1 -- 172.21.15.29:0/3632113255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d001a8480 con 0x7f4d00108ed0 2024-07-27T06:43:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.601+0000 7f4cf6ffd640 1 -- 172.21.15.29:0/3632113255 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cf000de50 con 0x7f4d00108ed0 2024-07-27T06:43:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.601+0000 7f4cf6ffd640 1 -- 172.21.15.29:0/3632113255 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf000f620 con 0x7f4d00108ed0 2024-07-27T06:43:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.601+0000 7f4d07dc6640 1 -- 172.21.15.29:0/3632113255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d001a8a90 con 0x7f4d00108ed0 2024-07-27T06:43:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.602+0000 7f4d07dc6640 1 -- 172.21.15.29:0/3632113255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d001039a0 con 0x7f4d00108ed0 2024-07-27T06:43:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.604+0000 7f4cf6ffd640 1 -- 172.21.15.29:0/3632113255 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f4cf001a070 con 0x7f4d00108ed0 2024-07-27T06:43:01.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.604+0000 7f4cf6ffd640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4cec0b3c20 0x7f4cec0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:01.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.604+0000 7f4d05b62640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4cec0b3c20 0x7f4cec0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:01.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.605+0000 7f4cf6ffd640 1 -- 172.21.15.29:0/3632113255 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f4cf00e3280 con 0x7f4d00108ed0 2024-07-27T06:43:01.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.605+0000 7f4d05b62640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4cec0b3c20 0x7f4cec0b6130 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4d00076b90 tx=0x7f4cfc008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:01.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.607+0000 7f4cf6ffd640 1 -- 172.21.15.29:0/3632113255 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4cf00aaca0 con 0x7f4d00108ed0 2024-07-27T06:43:01.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.868+0000 7f4d07dc6640 1 -- 172.21.15.29:0/3632113255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d00076440 con 0x7f4d00108ed0 2024-07-27T06:43:01.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.869+0000 7f4cf6ffd640 1 -- 172.21.15.29:0/3632113255 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10506 (secure 0 0 0) 0x7f4d00076440 con 0x7f4d00108ed0 2024-07-27T06:43:01.870 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:43:01.870 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":13,"btime":"2024-07-27T06:42:57:125586+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14781,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14793,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24542,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24566,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24584,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24613,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24646,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24658,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24670,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-07-27T06:43:01.870 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 13 2024-07-27T06:43:01.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.871+0000 7f4d07dc6640 1 -- 172.21.15.29:0/3632113255 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4cec0b3c20 msgr2=0x7f4cec0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:01.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.871+0000 7f4d07dc6640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4cec0b3c20 0x7f4cec0b6130 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4d00076b90 tx=0x7f4cfc008040 comp rx=0 tx=0).stop 2024-07-27T06:43:01.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.871+0000 7f4d07dc6640 1 -- 172.21.15.29:0/3632113255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d00108ed0 msgr2=0x7f4d00075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:01.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.871+0000 7f4d07dc6640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d00108ed0 0x7f4d00075680 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4cf000d740 tx=0x7f4cf000dc00 comp rx=0 tx=0).stop 2024-07-27T06:43:01.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.871+0000 7f4d07dc6640 1 -- 172.21.15.29:0/3632113255 shutdown_connections 2024-07-27T06:43:01.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.871+0000 7f4d07dc6640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4cec0b3c20 0x7f4cec0b6130 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:01.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.871+0000 7f4d07dc6640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4d00109820 0x7f4d00075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:01.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.871+0000 7f4d07dc6640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d00108ed0 0x7f4d00075680 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:01.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.871+0000 7f4d07dc6640 1 --2- 172.21.15.29:0/3632113255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d00103590 0x7f4d00075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:01.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.871+0000 7f4d07dc6640 1 -- 172.21.15.29:0/3632113255 >> 172.21.15.29:0/3632113255 conn(0x7f4d0007b280 msgr2=0x7f4d00106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:01.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.871+0000 7f4d07dc6640 1 -- 172.21.15.29:0/3632113255 shutdown_connections 2024-07-27T06:43:01.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:01.871+0000 7f4d07dc6640 1 -- 172.21.15.29:0/3632113255 wait complete. 2024-07-27T06:43:02.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-07-27T06:43:02.887 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:02 smithi029 ceph-mon[32610]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:43:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:43:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:02 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3632113255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:43:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:02 smithi077 ceph-mon[37153]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:43:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:43:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:02 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3632113255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:43:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:02 smithi047 ceph-mon[36835]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:43:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:43:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:43:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:02 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3632113255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:43:04.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.593+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/2745362257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dcc107e00 msgr2=0x7f9dcc108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:04.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.593+0000 7f9dd3bf2640 1 --2- 172.21.15.29:0/2745362257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dcc107e00 0x7f9dcc108210 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009900 tx=0x7f9dbc02a410 comp rx=0 tx=0).stop 2024-07-27T06:43:04.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.593+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/2745362257 shutdown_connections 2024-07-27T06:43:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.593+0000 7f9dd3bf2640 1 --2- 172.21.15.29:0/2745362257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9dcc108750 0x7f9dcc10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.593+0000 7f9dd3bf2640 1 --2- 172.21.15.29:0/2745362257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dcc107e00 0x7f9dcc108210 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.593+0000 7f9dd3bf2640 1 --2- 172.21.15.29:0/2745362257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dcc102510 0x7f9dcc102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.593+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/2745362257 >> 172.21.15.29:0/2745362257 conn(0x7f9dcc079190 msgr2=0x7f9dcc0795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.593+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/2745362257 shutdown_connections 2024-07-27T06:43:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.593+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/2745362257 wait complete. 2024-07-27T06:43:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.594+0000 7f9dd3bf2640 1 Processor -- start 2024-07-27T06:43:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.594+0000 7f9dd3bf2640 1 -- start start 2024-07-27T06:43:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.594+0000 7f9dd3bf2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dcc102510 0x7f9dcc198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.594+0000 7f9dd3bf2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dcc107e00 0x7f9dcc199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.594+0000 7f9dd3bf2640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9dcc108750 0x7f9dcc19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.594+0000 7f9dd3bf2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc112480 con 0x7f9dcc102510 2024-07-27T06:43:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.594+0000 7f9dd3bf2640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc112300 con 0x7f9dcc108750 2024-07-27T06:43:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.594+0000 7f9dd3bf2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc112600 con 0x7f9dcc107e00 2024-07-27T06:43:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.594+0000 7f9dd198e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dcc102510 0x7f9dcc198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.594+0000 7f9dd198e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dcc102510 0x7f9dcc198ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51846/0 (socket says 172.21.15.29:51846) 2024-07-27T06:43:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dd198e640 1 -- 172.21.15.29:0/1930129125 learned_addr learned my addr 172.21.15.29:0/1930129125 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dd118d640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dcc107e00 0x7f9dcc199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dd218f640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9dcc108750 0x7f9dcc19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dd198e640 1 -- 172.21.15.29:0/1930129125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dcc107e00 msgr2=0x7f9dcc199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dd198e640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dcc107e00 0x7f9dcc199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dd198e640 1 -- 172.21.15.29:0/1930129125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9dcc108750 msgr2=0x7f9dcc19e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dd198e640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9dcc108750 0x7f9dcc19e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dd198e640 1 -- 172.21.15.29:0/1930129125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dcc19ee40 con 0x7f9dcc102510 2024-07-27T06:43:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dd118d640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dcc107e00 0x7f9dcc199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:43:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dd198e640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dcc102510 0x7f9dcc198ed0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800b4a0 tx=0x7f9dc800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dc2ffd640 1 -- 172.21.15.29:0/1930129125 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8014070 con 0x7f9dcc102510 2024-07-27T06:43:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/1930129125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dcc19f130 con 0x7f9dcc102510 2024-07-27T06:43:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dc2ffd640 1 -- 172.21.15.29:0/1930129125 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dc8010ac0 con 0x7f9dcc102510 2024-07-27T06:43:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dc2ffd640 1 -- 172.21.15.29:0/1930129125 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8017c30 con 0x7f9dcc102510 2024-07-27T06:43:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.595+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/1930129125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dcc1a5890 con 0x7f9dcc102510 2024-07-27T06:43:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.596+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/1930129125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dcc103740 con 0x7f9dcc102510 2024-07-27T06:43:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.598+0000 7f9dc2ffd640 1 -- 172.21.15.29:0/1930129125 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f9dc8017070 con 0x7f9dcc102510 2024-07-27T06:43:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.599+0000 7f9dc2ffd640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9db80b3c20 0x7f9db80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.599+0000 7f9dd118d640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9db80b3c20 0x7f9db80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:04.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.599+0000 7f9dc2ffd640 1 -- 172.21.15.29:0/1930129125 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f9dc80e3100 con 0x7f9dcc102510 2024-07-27T06:43:04.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.599+0000 7f9dd118d640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9db80b3c20 0x7f9db80b6130 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc000c00 tx=0x7f9dbc031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:04.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.602+0000 7f9dc2ffd640 1 -- 172.21.15.29:0/1930129125 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9dc80aab20 con 0x7f9dcc102510 2024-07-27T06:43:04.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.788+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/1930129125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f9dcc19a000 con 0x7f9dcc102510 2024-07-27T06:43:04.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.789+0000 7f9dc2ffd640 1 -- 172.21.15.29:0/1930129125 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7f9dcc19a000 con 0x7f9dcc102510 2024-07-27T06:43:04.789 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:43:04.790 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2024-07-27T06:43:04.790 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-07-27T06:43:04.790 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2024-07-27T06:43:04.790 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2024-07-27T06:43:04.790 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:43:04.790 INFO:teuthology.orchestra.run.smithi029.stdout:] 2024-07-27T06:43:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.791+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/1930129125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9db80b3c20 msgr2=0x7f9db80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.791+0000 7f9dd3bf2640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9db80b3c20 0x7f9db80b6130 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc000c00 tx=0x7f9dbc031400 comp rx=0 tx=0).stop 2024-07-27T06:43:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.791+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/1930129125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dcc102510 msgr2=0x7f9dcc198ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.791+0000 7f9dd3bf2640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dcc102510 0x7f9dcc198ed0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800b4a0 tx=0x7f9dc800b960 comp rx=0 tx=0).stop 2024-07-27T06:43:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.791+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/1930129125 shutdown_connections 2024-07-27T06:43:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.791+0000 7f9dd3bf2640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9dcc108750 0x7f9dcc19e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.791+0000 7f9dd3bf2640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9db80b3c20 0x7f9db80b6130 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.791+0000 7f9dd3bf2640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9dcc107e00 0x7f9dcc199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.791+0000 7f9dd3bf2640 1 --2- 172.21.15.29:0/1930129125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9dcc102510 0x7f9dcc198ed0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.791+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/1930129125 >> 172.21.15.29:0/1930129125 conn(0x7f9dcc079190 msgr2=0x7f9dcc104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:04.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.791+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/1930129125 shutdown_connections 2024-07-27T06:43:04.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:04.791+0000 7f9dd3bf2640 1 -- 172.21.15.29:0/1930129125 wait complete. 2024-07-27T06:43:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:04 smithi029 ceph-mon[32610]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:04 smithi077 ceph-mon[37153]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:04 smithi047 ceph-mon[36835]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:05.534 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-27T06:43:05.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-27T06:43:05.805 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:05.928 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:05 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1930129125' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-27T06:43:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:05 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1930129125' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-27T06:43:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:05 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1930129125' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-27T06:43:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:06 smithi029 ceph-mon[32610]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:06 smithi077 ceph-mon[37153]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:06 smithi047 ceph-mon[36835]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.556+0000 7efe9d2fd640 1 -- 172.21.15.29:0/712524239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe98107de0 msgr2=0x7efe981081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.556+0000 7efe9d2fd640 1 --2- 172.21.15.29:0/712524239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe98107de0 0x7efe981081f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009900 tx=0x7efe8c02a410 comp rx=0 tx=0).stop 2024-07-27T06:43:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.556+0000 7efe9d2fd640 1 -- 172.21.15.29:0/712524239 shutdown_connections 2024-07-27T06:43:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.556+0000 7efe9d2fd640 1 --2- 172.21.15.29:0/712524239 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe98108730 0x7efe9810ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.556+0000 7efe9d2fd640 1 --2- 172.21.15.29:0/712524239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe98107de0 0x7efe981081f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.556+0000 7efe9d2fd640 1 --2- 172.21.15.29:0/712524239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe981024a0 0x7efe981028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.556+0000 7efe9d2fd640 1 -- 172.21.15.29:0/712524239 >> 172.21.15.29:0/712524239 conn(0x7efe98079160 msgr2=0x7efe98079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.556+0000 7efe9d2fd640 1 -- 172.21.15.29:0/712524239 shutdown_connections 2024-07-27T06:43:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.556+0000 7efe9d2fd640 1 -- 172.21.15.29:0/712524239 wait complete. 2024-07-27T06:43:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.557+0000 7efe9d2fd640 1 Processor -- start 2024-07-27T06:43:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.557+0000 7efe9d2fd640 1 -- start start 2024-07-27T06:43:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.557+0000 7efe9d2fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe981024a0 0x7efe98199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.557+0000 7efe9d2fd640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe98107de0 0x7efe9819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.557+0000 7efe9d2fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe98108730 0x7efe9819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.557+0000 7efe9d2fd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efe981123e0 con 0x7efe98108730 2024-07-27T06:43:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.557+0000 7efe9d2fd640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7efe98112260 con 0x7efe98107de0 2024-07-27T06:43:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.557+0000 7efe9d2fd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efe98112560 con 0x7efe981024a0 2024-07-27T06:43:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe98108730 0x7efe9819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe98108730 0x7efe9819f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51866/0 (socket says 172.21.15.29:51866) 2024-07-27T06:43:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe977fe640 1 -- 172.21.15.29:0/3027371359 learned_addr learned my addr 172.21.15.29:0/3027371359 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe96ffd640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe981024a0 0x7efe98199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe967fc640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe98107de0 0x7efe9819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe977fe640 1 -- 172.21.15.29:0/3027371359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe981024a0 msgr2=0x7efe98199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe977fe640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe981024a0 0x7efe98199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe977fe640 1 -- 172.21.15.29:0/3027371359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe98107de0 msgr2=0x7efe9819a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe977fe640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe98107de0 0x7efe9819a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe977fe640 1 -- 172.21.15.29:0/3027371359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe9819fed0 con 0x7efe98108730 2024-07-27T06:43:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe977fe640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe98108730 0x7efe9819f7a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efe9000cad0 tx=0x7efe9000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe7ffff640 1 -- 172.21.15.29:0/3027371359 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9000ad60 con 0x7efe98108730 2024-07-27T06:43:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe9d2fd640 1 -- 172.21.15.29:0/3027371359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe981a01c0 con 0x7efe98108730 2024-07-27T06:43:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe7ffff640 1 -- 172.21.15.29:0/3027371359 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe90005020 con 0x7efe98108730 2024-07-27T06:43:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe7ffff640 1 -- 172.21.15.29:0/3027371359 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9000f6b0 con 0x7efe98108730 2024-07-27T06:43:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.558+0000 7efe9d2fd640 1 -- 172.21.15.29:0/3027371359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe981a6920 con 0x7efe98108730 2024-07-27T06:43:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.560+0000 7efe9d2fd640 1 -- 172.21.15.29:0/3027371359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe98103720 con 0x7efe98108730 2024-07-27T06:43:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.562+0000 7efe7ffff640 1 -- 172.21.15.29:0/3027371359 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7efe900051d0 con 0x7efe98108730 2024-07-27T06:43:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.562+0000 7efe7ffff640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efe800b3c20 0x7efe800b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.562+0000 7efe96ffd640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efe800b3c20 0x7efe800b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.563+0000 7efe7ffff640 1 -- 172.21.15.29:0/3027371359 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7efe900e3bf0 con 0x7efe98108730 2024-07-27T06:43:07.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.563+0000 7efe96ffd640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efe800b3c20 0x7efe800b6130 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efe88009990 tx=0x7efe88009110 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:07.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.565+0000 7efe7ffff640 1 -- 172.21.15.29:0/3027371359 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7efe90014030 con 0x7efe98108730 2024-07-27T06:43:07.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:07.751+0000 7efe9d2fd640 1 -- 172.21.15.29:0/3027371359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7efe981028b0 con 0x7efe98108730 2024-07-27T06:43:08.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.752+0000 7efe7ffff640 1 -- 172.21.15.29:0/3027371359 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7efe900ab610 con 0x7efe98108730 2024-07-27T06:43:08.753 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2024-07-27T06:43:08.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.756+0000 7efe9d2fd640 1 -- 172.21.15.29:0/3027371359 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efe800b3c20 msgr2=0x7efe800b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:08.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.756+0000 7efe9d2fd640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efe800b3c20 0x7efe800b6130 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efe88009990 tx=0x7efe88009110 comp rx=0 tx=0).stop 2024-07-27T06:43:08.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.756+0000 7efe9d2fd640 1 -- 172.21.15.29:0/3027371359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe98108730 msgr2=0x7efe9819f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:08.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.756+0000 7efe9d2fd640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe98108730 0x7efe9819f7a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efe9000cad0 tx=0x7efe9000ced0 comp rx=0 tx=0).stop 2024-07-27T06:43:08.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.757+0000 7efe9d2fd640 1 -- 172.21.15.29:0/3027371359 shutdown_connections 2024-07-27T06:43:08.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.757+0000 7efe9d2fd640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe98108730 0x7efe9819f7a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:08.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.757+0000 7efe9d2fd640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe98107de0 0x7efe9819a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:08.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.757+0000 7efe9d2fd640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efe800b3c20 0x7efe800b6130 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:08.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.757+0000 7efe9d2fd640 1 --2- 172.21.15.29:0/3027371359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe981024a0 0x7efe98199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:08.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.757+0000 7efe9d2fd640 1 -- 172.21.15.29:0/3027371359 >> 172.21.15.29:0/3027371359 conn(0x7efe98079160 msgr2=0x7efe98101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:08.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.757+0000 7efe9d2fd640 1 -- 172.21.15.29:0/3027371359 shutdown_connections 2024-07-27T06:43:08.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:08.757+0000 7efe9d2fd640 1 -- 172.21.15.29:0/3027371359 wait complete. 2024-07-27T06:43:09.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:08 smithi029 ceph-mon[32610]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:09.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:08 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3027371359' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-27T06:43:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:08 smithi077 ceph-mon[37153]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:08 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3027371359' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-27T06:43:09.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:08 smithi047 ceph-mon[36835]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:09.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:08 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3027371359' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-27T06:43:09.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-07-27T06:43:09.727 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:09.996 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:09 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3027371359' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-27T06:43:09.996 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:09 smithi047 ceph-mon[36835]: osdmap e73: 12 total, 12 up, 12 in 2024-07-27T06:43:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:09 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3027371359' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-27T06:43:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:09 smithi077 ceph-mon[37153]: osdmap e73: 12 total, 12 up, 12 in 2024-07-27T06:43:10.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:09 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3027371359' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-27T06:43:10.108 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:09 smithi029 ceph-mon[32610]: osdmap e73: 12 total, 12 up, 12 in 2024-07-27T06:43:10.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:10 smithi029 ceph-mon[32610]: pgmap v272: 33 pgs: 1 creating+peering, 31 unknown, 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:10.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:10 smithi029 ceph-mon[32610]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-27T06:43:10.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:10 smithi029 ceph-mon[32610]: osdmap e74: 12 total, 12 up, 12 in 2024-07-27T06:43:11.019 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:10 smithi077 ceph-mon[37153]: pgmap v272: 33 pgs: 1 creating+peering, 31 unknown, 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:11.019 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:10 smithi077 ceph-mon[37153]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-27T06:43:11.019 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:10 smithi077 ceph-mon[37153]: osdmap e74: 12 total, 12 up, 12 in 2024-07-27T06:43:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:10 smithi047 ceph-mon[36835]: pgmap v272: 33 pgs: 1 creating+peering, 31 unknown, 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:10 smithi047 ceph-mon[36835]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-27T06:43:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:10 smithi047 ceph-mon[36835]: osdmap e74: 12 total, 12 up, 12 in 2024-07-27T06:43:11.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.457+0000 7fcf02021640 1 -- 172.21.15.29:0/3333789932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc108ed0 msgr2=0x7fcefc1092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:11.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.457+0000 7fcf02021640 1 --2- 172.21.15.29:0/3333789932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc108ed0 0x7fcefc1092e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009900 tx=0x7fcee402a410 comp rx=0 tx=0).stop 2024-07-27T06:43:11.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.457+0000 7fcf02021640 1 -- 172.21.15.29:0/3333789932 shutdown_connections 2024-07-27T06:43:11.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.457+0000 7fcf02021640 1 --2- 172.21.15.29:0/3333789932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc109820 0x7fcefc110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:11.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.457+0000 7fcf02021640 1 --2- 172.21.15.29:0/3333789932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc108ed0 0x7fcefc1092e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:11.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.457+0000 7fcf02021640 1 --2- 172.21.15.29:0/3333789932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc103590 0x7fcefc1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:11.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.457+0000 7fcf02021640 1 -- 172.21.15.29:0/3333789932 >> 172.21.15.29:0/3333789932 conn(0x7fcefc07b280 msgr2=0x7fcefc07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:11.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.457+0000 7fcf02021640 1 -- 172.21.15.29:0/3333789932 shutdown_connections 2024-07-27T06:43:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.457+0000 7fcf02021640 1 -- 172.21.15.29:0/3333789932 wait complete. 2024-07-27T06:43:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.458+0000 7fcf02021640 1 Processor -- start 2024-07-27T06:43:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.458+0000 7fcf02021640 1 -- start start 2024-07-27T06:43:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.458+0000 7fcf02021640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc103590 0x7fcefc199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.458+0000 7fcf02021640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc108ed0 0x7fcefc19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcf02021640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc109820 0x7fcefc19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcf02021640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc113500 con 0x7fcefc103590 2024-07-27T06:43:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcf02021640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc113380 con 0x7fcefc109820 2024-07-27T06:43:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcf02021640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc113680 con 0x7fcefc108ed0 2024-07-27T06:43:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcefb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc103590 0x7fcefc199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcefb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc103590 0x7fcefc199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54642/0 (socket says 172.21.15.29:54642) 2024-07-27T06:43:11.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcefb7fe640 1 -- 172.21.15.29:0/880856268 learned_addr learned my addr 172.21.15.29:0/880856268 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcefaffd640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc108ed0 0x7fcefc19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcefbfff640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc109820 0x7fcefc19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcefb7fe640 1 -- 172.21.15.29:0/880856268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc108ed0 msgr2=0x7fcefc19a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcefb7fe640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc108ed0 0x7fcefc19a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcefb7fe640 1 -- 172.21.15.29:0/880856268 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc109820 msgr2=0x7fcefc19f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcefb7fe640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc109820 0x7fcefc19f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcefb7fe640 1 -- 172.21.15.29:0/880856268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcefc19fe40 con 0x7fcefc103590 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcefb7fe640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc103590 0x7fcefc199ed0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fceec00e9f0 tx=0x7fceec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.459+0000 7fcef8ff9640 1 -- 172.21.15.29:0/880856268 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec00cd80 con 0x7fcefc103590 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.460+0000 7fcef8ff9640 1 -- 172.21.15.29:0/880856268 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceec0054d0 con 0x7fcefc103590 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.460+0000 7fcf02021640 1 -- 172.21.15.29:0/880856268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcefc1a0130 con 0x7fcefc103590 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.460+0000 7fcef8ff9640 1 -- 172.21.15.29:0/880856268 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec010690 con 0x7fcefc103590 2024-07-27T06:43:11.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.460+0000 7fcf02021640 1 -- 172.21.15.29:0/880856268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcefc1a6890 con 0x7fcefc103590 2024-07-27T06:43:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.460+0000 7fcf02021640 1 -- 172.21.15.29:0/880856268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcefc104810 con 0x7fcefc103590 2024-07-27T06:43:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.462+0000 7fcef8ff9640 1 -- 172.21.15.29:0/880856268 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fceec004e50 con 0x7fcefc103590 2024-07-27T06:43:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.463+0000 7fcef8ff9640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcee80b3c20 0x7fcee80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:11.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.463+0000 7fcefaffd640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcee80b3c20 0x7fcee80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:11.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.463+0000 7fcef8ff9640 1 -- 172.21.15.29:0/880856268 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7712+0+0 (secure 0 0 0) 0x7fceec0e4050 con 0x7fcefc103590 2024-07-27T06:43:11.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.464+0000 7fcefaffd640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcee80b3c20 0x7fcee80b6130 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcee402aa00 tx=0x7fcee4005a20 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:11.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.467+0000 7fcef8ff9640 1 -- 172.21.15.29:0/880856268 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fceec055600 con 0x7fcefc103590 2024-07-27T06:43:11.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.652+0000 7fcf02021640 1 -- 172.21.15.29:0/880856268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fcefc1039a0 con 0x7fcefc103590 2024-07-27T06:43:11.773 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2024-07-27T06:43:11.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.771+0000 7fcef8ff9640 1 -- 172.21.15.29:0/880856268 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v76) ==== 170+0+0 (secure 0 0 0) 0x7fceec0ab900 con 0x7fcefc103590 2024-07-27T06:43:11.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.774+0000 7fcf02021640 1 -- 172.21.15.29:0/880856268 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcee80b3c20 msgr2=0x7fcee80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:11.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.774+0000 7fcf02021640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcee80b3c20 0x7fcee80b6130 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcee402aa00 tx=0x7fcee4005a20 comp rx=0 tx=0).stop 2024-07-27T06:43:11.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.774+0000 7fcf02021640 1 -- 172.21.15.29:0/880856268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc103590 msgr2=0x7fcefc199ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.774+0000 7fcf02021640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc103590 0x7fcefc199ed0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fceec00e9f0 tx=0x7fceec00edf0 comp rx=0 tx=0).stop 2024-07-27T06:43:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.774+0000 7fcf02021640 1 -- 172.21.15.29:0/880856268 shutdown_connections 2024-07-27T06:43:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.774+0000 7fcf02021640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcefc109820 0x7fcefc19f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.774+0000 7fcf02021640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcee80b3c20 0x7fcee80b6130 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.774+0000 7fcf02021640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc108ed0 0x7fcefc19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.774+0000 7fcf02021640 1 --2- 172.21.15.29:0/880856268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcefc103590 0x7fcefc199ed0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.774+0000 7fcf02021640 1 -- 172.21.15.29:0/880856268 >> 172.21.15.29:0/880856268 conn(0x7fcefc07b280 msgr2=0x7fcefc104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:11.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.774+0000 7fcf02021640 1 -- 172.21.15.29:0/880856268 shutdown_connections 2024-07-27T06:43:11.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:11.775+0000 7fcf02021640 1 -- 172.21.15.29:0/880856268 wait complete. 2024-07-27T06:43:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:11 smithi029 ceph-mon[32610]: osdmap e75: 12 total, 12 up, 12 in 2024-07-27T06:43:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:11 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/880856268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-27T06:43:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:11 smithi077 ceph-mon[37153]: osdmap e75: 12 total, 12 up, 12 in 2024-07-27T06:43:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:11 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/880856268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-27T06:43:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:11 smithi047 ceph-mon[36835]: osdmap e75: 12 total, 12 up, 12 in 2024-07-27T06:43:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:11 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/880856268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-27T06:43:12.539 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-27T06:43:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:12 smithi077 ceph-mon[37153]: pgmap v275: 33 pgs: 2 creating+activating, 6 creating+peering, 9 unknown, 16 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:12 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/880856268' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-27T06:43:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:12 smithi077 ceph-mon[37153]: osdmap e76: 12 total, 12 up, 12 in 2024-07-27T06:43:13.110 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:12 smithi029 ceph-mon[32610]: pgmap v275: 33 pgs: 2 creating+activating, 6 creating+peering, 9 unknown, 16 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:13.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:12 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/880856268' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-27T06:43:13.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:12 smithi029 ceph-mon[32610]: osdmap e76: 12 total, 12 up, 12 in 2024-07-27T06:43:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:12 smithi047 ceph-mon[36835]: pgmap v275: 33 pgs: 2 creating+activating, 6 creating+peering, 9 unknown, 16 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:12 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/880856268' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-27T06:43:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:12 smithi047 ceph-mon[36835]: osdmap e76: 12 total, 12 up, 12 in 2024-07-27T06:43:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:13 smithi029 ceph-mon[32610]: osdmap e77: 12 total, 12 up, 12 in 2024-07-27T06:43:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:13 smithi077 ceph-mon[37153]: osdmap e77: 12 total, 12 up, 12 in 2024-07-27T06:43:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:13 smithi047 ceph-mon[36835]: osdmap e77: 12 total, 12 up, 12 in 2024-07-27T06:43:14.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.568+0000 7f700ebd4640 1 -- 172.21.15.29:0/3527711813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7008103590 msgr2=0x7f70081039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:14.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.568+0000 7f700ebd4640 1 --2- 172.21.15.29:0/3527711813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7008103590 0x7f70081039a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009900 tx=0x7f6ff802a410 comp rx=0 tx=0).stop 2024-07-27T06:43:14.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.568+0000 7f700ebd4640 1 -- 172.21.15.29:0/3527711813 shutdown_connections 2024-07-27T06:43:14.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.568+0000 7f700ebd4640 1 --2- 172.21.15.29:0/3527711813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008109820 0x7f7008110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:14.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.568+0000 7f700ebd4640 1 --2- 172.21.15.29:0/3527711813 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7008108ed0 0x7f70081092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:14.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.568+0000 7f700ebd4640 1 --2- 172.21.15.29:0/3527711813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7008103590 0x7f70081039a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:14.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.568+0000 7f700ebd4640 1 -- 172.21.15.29:0/3527711813 >> 172.21.15.29:0/3527711813 conn(0x7f700807b280 msgr2=0x7f700807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:14.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.568+0000 7f700ebd4640 1 -- 172.21.15.29:0/3527711813 shutdown_connections 2024-07-27T06:43:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.568+0000 7f700ebd4640 1 -- 172.21.15.29:0/3527711813 wait complete. 2024-07-27T06:43:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.569+0000 7f700ebd4640 1 Processor -- start 2024-07-27T06:43:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.569+0000 7f700ebd4640 1 -- start start 2024-07-27T06:43:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.569+0000 7f700ebd4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7008103590 0x7f7008199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.569+0000 7f700ebd4640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7008108ed0 0x7f700819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.569+0000 7f700ebd4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008109820 0x7f700819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.569+0000 7f700ebd4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70081134f0 con 0x7f7008103590 2024-07-27T06:43:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.569+0000 7f700ebd4640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7008113370 con 0x7f7008108ed0 2024-07-27T06:43:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.569+0000 7f700ebd4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7008113670 con 0x7f7008109820 2024-07-27T06:43:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f700c970640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7008103590 0x7f7008199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f700c970640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7008103590 0x7f7008199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54652/0 (socket says 172.21.15.29:54652) 2024-07-27T06:43:14.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f700c970640 1 -- 172.21.15.29:0/1206626313 learned_addr learned my addr 172.21.15.29:0/1206626313 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f7007fff640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7008108ed0 0x7f700819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f700d171640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008109820 0x7f700819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f700c970640 1 -- 172.21.15.29:0/1206626313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008109820 msgr2=0x7f700819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f700c970640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008109820 0x7f700819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f700c970640 1 -- 172.21.15.29:0/1206626313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7008108ed0 msgr2=0x7f700819a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f700c970640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7008108ed0 0x7f700819a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f700c970640 1 -- 172.21.15.29:0/1206626313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f700819fe90 con 0x7f7008103590 2024-07-27T06:43:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f700d171640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008109820 0x7f700819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:43:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f7007fff640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7008108ed0 0x7f700819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:43:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f700c970640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7008103590 0x7f7008199f20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8005940 tx=0x7f6ff8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f7005ffb640 1 -- 172.21.15.29:0/1206626313 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff803f030 con 0x7f7008103590 2024-07-27T06:43:14.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f700ebd4640 1 -- 172.21.15.29:0/1206626313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70081a0120 con 0x7f7008103590 2024-07-27T06:43:14.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f7005ffb640 1 -- 172.21.15.29:0/1206626313 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ff8003aa0 con 0x7f7008103590 2024-07-27T06:43:14.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.570+0000 7f7005ffb640 1 -- 172.21.15.29:0/1206626313 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8035070 con 0x7f7008103590 2024-07-27T06:43:14.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.571+0000 7f700ebd4640 1 -- 172.21.15.29:0/1206626313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70081a68e0 con 0x7f7008103590 2024-07-27T06:43:14.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.571+0000 7f700ebd4640 1 -- 172.21.15.29:0/1206626313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70081039a0 con 0x7f7008103590 2024-07-27T06:43:14.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.573+0000 7f7005ffb640 1 -- 172.21.15.29:0/1206626313 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f6ff8042030 con 0x7f7008103590 2024-07-27T06:43:14.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.574+0000 7f7005ffb640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6ff00b3c20 0x7f6ff00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:14.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.575+0000 7f7007fff640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6ff00b3c20 0x7f6ff00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:14.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.575+0000 7f7005ffb640 1 -- 172.21.15.29:0/1206626313 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8094+0+0 (secure 0 0 0) 0x7f6ff8104520 con 0x7f7008103590 2024-07-27T06:43:14.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.575+0000 7f7007fff640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6ff00b3c20 0x7f6ff00b6130 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc005d00 tx=0x7f6ffc009550 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:14.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.578+0000 7f7005ffb640 1 -- 172.21.15.29:0/1206626313 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6ff80cbc30 con 0x7f7008103590 2024-07-27T06:43:14.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:14.799+0000 7f700ebd4640 1 -- 172.21.15.29:0/1206626313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f700819b050 con 0x7f7008103590 2024-07-27T06:43:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:14 smithi029 ceph-mon[32610]: pgmap v278: 97 pgs: 2 creating+activating, 20 creating+peering, 50 unknown, 25 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:14 smithi029 ceph-mon[32610]: osdmap e78: 12 total, 12 up, 12 in 2024-07-27T06:43:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:14 smithi077 ceph-mon[37153]: pgmap v278: 97 pgs: 2 creating+activating, 20 creating+peering, 50 unknown, 25 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:14 smithi077 ceph-mon[37153]: osdmap e78: 12 total, 12 up, 12 in 2024-07-27T06:43:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:14 smithi047 ceph-mon[36835]: pgmap v278: 97 pgs: 2 creating+activating, 20 creating+peering, 50 unknown, 25 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:14 smithi047 ceph-mon[36835]: osdmap e78: 12 total, 12 up, 12 in 2024-07-27T06:43:15.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.796+0000 7f7005ffb640 1 -- 172.21.15.29:0/1206626313 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f6ff80cb530 con 0x7f7008103590 2024-07-27T06:43:15.798 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 2 and data pool 3 2024-07-27T06:43:15.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.799+0000 7f700ebd4640 1 -- 172.21.15.29:0/1206626313 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6ff00b3c20 msgr2=0x7f6ff00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:15.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.799+0000 7f700ebd4640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6ff00b3c20 0x7f6ff00b6130 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc005d00 tx=0x7f6ffc009550 comp rx=0 tx=0).stop 2024-07-27T06:43:15.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.799+0000 7f700ebd4640 1 -- 172.21.15.29:0/1206626313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7008103590 msgr2=0x7f7008199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:15.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.799+0000 7f700ebd4640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7008103590 0x7f7008199f20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8005940 tx=0x7f6ff8004270 comp rx=0 tx=0).stop 2024-07-27T06:43:15.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.799+0000 7f700ebd4640 1 -- 172.21.15.29:0/1206626313 shutdown_connections 2024-07-27T06:43:15.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.799+0000 7f700ebd4640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6ff00b3c20 0x7f6ff00b6130 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:15.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.799+0000 7f700ebd4640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7008109820 0x7f700819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:15.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.799+0000 7f700ebd4640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7008108ed0 0x7f700819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:15.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.799+0000 7f700ebd4640 1 --2- 172.21.15.29:0/1206626313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7008103590 0x7f7008199f20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:15.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.799+0000 7f700ebd4640 1 -- 172.21.15.29:0/1206626313 >> 172.21.15.29:0/1206626313 conn(0x7f700807b280 msgr2=0x7f7008107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:15.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.799+0000 7f700ebd4640 1 -- 172.21.15.29:0/1206626313 shutdown_connections 2024-07-27T06:43:15.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:15.799+0000 7f700ebd4640 1 -- 172.21.15.29:0/1206626313 wait complete. 2024-07-27T06:43:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:15 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1206626313' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-27T06:43:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:15 smithi077 ceph-mon[37153]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:43:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:15 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1206626313' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-27T06:43:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:15 smithi047 ceph-mon[36835]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:43:16.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:15 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1206626313' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-27T06:43:16.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:15 smithi029 ceph-mon[32610]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:43:16.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd dump --format=json 2024-07-27T06:43:16.858 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:16 smithi077 ceph-mon[37153]: pgmap v280: 97 pgs: 14 creating+peering, 19 unknown, 64 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:16 smithi077 ceph-mon[37153]: osdmap e79: 12 total, 12 up, 12 in 2024-07-27T06:43:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:16 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1206626313' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-27T06:43:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:16 smithi077 ceph-mon[37153]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-07-27T06:43:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:16 smithi077 ceph-mon[37153]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:43:17.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:16 smithi029 ceph-mon[32610]: pgmap v280: 97 pgs: 14 creating+peering, 19 unknown, 64 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:17.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:16 smithi029 ceph-mon[32610]: osdmap e79: 12 total, 12 up, 12 in 2024-07-27T06:43:17.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:16 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1206626313' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-27T06:43:17.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:16 smithi029 ceph-mon[32610]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-07-27T06:43:17.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:16 smithi029 ceph-mon[32610]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:43:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:16 smithi047 ceph-mon[36835]: pgmap v280: 97 pgs: 14 creating+peering, 19 unknown, 64 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-27T06:43:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:16 smithi047 ceph-mon[36835]: osdmap e79: 12 total, 12 up, 12 in 2024-07-27T06:43:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:16 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1206626313' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-27T06:43:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:16 smithi047 ceph-mon[36835]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-07-27T06:43:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:16 smithi047 ceph-mon[36835]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:43:17.801 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:43:17 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2024-07-27T06:43:17.402+0000 7fd17da7c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-27T06:43:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:17 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] up:active 2024-07-27T06:43:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:17 smithi029 ceph-mon[32610]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-07-27T06:43:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:17 smithi029 ceph-mon[32610]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-07-27T06:43:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:17 smithi029 ceph-mon[32610]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-27T06:43:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:17 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] up:active 2024-07-27T06:43:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:17 smithi077 ceph-mon[37153]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-07-27T06:43:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:17 smithi077 ceph-mon[37153]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-07-27T06:43:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:17 smithi077 ceph-mon[37153]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-27T06:43:18.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:17 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] up:active 2024-07-27T06:43:18.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:17 smithi047 ceph-mon[36835]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-07-27T06:43:18.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:17 smithi047 ceph-mon[36835]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-07-27T06:43:18.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:17 smithi047 ceph-mon[36835]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-27T06:43:18.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.588+0000 7fc7a1909640 1 -- 172.21.15.29:0/2038624572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc79c107e00 msgr2=0x7fc79c108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.588+0000 7fc7a1909640 1 --2- 172.21.15.29:0/2038624572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc79c107e00 0x7fc79c108210 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc790009900 tx=0x7fc79002a410 comp rx=0 tx=0).stop 2024-07-27T06:43:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.588+0000 7fc7a1909640 1 -- 172.21.15.29:0/2038624572 shutdown_connections 2024-07-27T06:43:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.588+0000 7fc7a1909640 1 --2- 172.21.15.29:0/2038624572 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc79c108750 0x7fc79c10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.588+0000 7fc7a1909640 1 --2- 172.21.15.29:0/2038624572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc79c107e00 0x7fc79c108210 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.588+0000 7fc7a1909640 1 --2- 172.21.15.29:0/2038624572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc79c102510 0x7fc79c102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.588+0000 7fc7a1909640 1 -- 172.21.15.29:0/2038624572 >> 172.21.15.29:0/2038624572 conn(0x7fc79c079190 msgr2=0x7fc79c0795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.588+0000 7fc7a1909640 1 -- 172.21.15.29:0/2038624572 shutdown_connections 2024-07-27T06:43:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.588+0000 7fc7a1909640 1 -- 172.21.15.29:0/2038624572 wait complete. 2024-07-27T06:43:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.589+0000 7fc7a1909640 1 Processor -- start 2024-07-27T06:43:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.589+0000 7fc7a1909640 1 -- start start 2024-07-27T06:43:18.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc7a1909640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc79c102510 0x7fc79c198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc7a1909640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc79c107e00 0x7fc79c199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc7a1909640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc79c108750 0x7fc79c19e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc7a1909640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c112380 con 0x7fc79c108750 2024-07-27T06:43:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc7a1909640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c112200 con 0x7fc79c102510 2024-07-27T06:43:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc7a1909640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c112500 con 0x7fc79c107e00 2024-07-27T06:43:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc79b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc79c108750 0x7fc79c19e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc79b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc79c108750 0x7fc79c19e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54672/0 (socket says 172.21.15.29:54672) 2024-07-27T06:43:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc79b7fe640 1 -- 172.21.15.29:0/2682450108 learned_addr learned my addr 172.21.15.29:0/2682450108 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc79affd640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc79c102510 0x7fc79c198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc79a7fc640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc79c107e00 0x7fc79c199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc79b7fe640 1 -- 172.21.15.29:0/2682450108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc79c107e00 msgr2=0x7fc79c199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:18.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc79b7fe640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc79c107e00 0x7fc79c199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc79b7fe640 1 -- 172.21.15.29:0/2682450108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc79c102510 msgr2=0x7fc79c198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc79b7fe640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc79c102510 0x7fc79c198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc79b7fe640 1 -- 172.21.15.29:0/2682450108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79c19ece0 con 0x7fc79c108750 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc79b7fe640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc79c108750 0x7fc79c19e5b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc79400eb10 tx=0x7fc79400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.590+0000 7fc7a0907640 1 -- 172.21.15.29:0/2682450108 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc794004b20 con 0x7fc79c108750 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.591+0000 7fc7a0907640 1 -- 172.21.15.29:0/2682450108 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc794004cc0 con 0x7fc79c108750 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.591+0000 7fc7a0907640 1 -- 172.21.15.29:0/2682450108 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc794010670 con 0x7fc79c108750 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.591+0000 7fc7a1909640 1 -- 172.21.15.29:0/2682450108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc79c19efd0 con 0x7fc79c108750 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.591+0000 7fc7a1909640 1 -- 172.21.15.29:0/2682450108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79c1a5730 con 0x7fc79c108750 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.592+0000 7fc7a1909640 1 -- 172.21.15.29:0/2682450108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc79c103740 con 0x7fc79c108750 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.593+0000 7fc7a0907640 1 -- 172.21.15.29:0/2682450108 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fc794005000 con 0x7fc79c108750 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.594+0000 7fc7a0907640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc7840b3ca0 0x7fc7840b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.594+0000 7fc79affd640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc7840b3ca0 0x7fc7840b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:18.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.594+0000 7fc7a0907640 1 -- 172.21.15.29:0/2682450108 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7fc7940e42c0 con 0x7fc79c108750 2024-07-27T06:43:18.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.595+0000 7fc79affd640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc7840b3ca0 0x7fc7840b61b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc78c005d00 tx=0x7fc78c00b410 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:18.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.597+0000 7fc7a0907640 1 -- 172.21.15.29:0/2682450108 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc794055600 con 0x7fc79c108750 2024-07-27T06:43:18.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.782+0000 7fc7a1909640 1 -- 172.21.15.29:0/2682450108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc79c102920 con 0x7fc79c108750 2024-07-27T06:43:18.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.783+0000 7fc7a0907640 1 -- 172.21.15.29:0/2682450108 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22464 (secure 0 0 0) 0x7fc7940ab930 con 0x7fc79c108750 2024-07-27T06:43:18.785 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:43:18.786 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":79,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","created":"2024-07-27T06:33:57.535327+0000","modified":"2024-07-27T06:43:14.801257+0000","last_up_change":"2024-07-27T06:42:05.757156+0000","last_in_change":"2024-07-27T06:41:46.423732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T06:38:21.606527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-27T06:43:07.755490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-27T06:43:11.658694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06833d24-2e1a-4199-9338-7d6883d1bf9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6803","nonce":386121641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6805","nonce":386121641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6809","nonce":386121641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6807","nonce":386121641}]},"public_addr":"172.21.15.29:6803/386121641","cluster_addr":"172.21.15.29:6805/386121641","heartbeat_back_addr":"172.21.15.29:6809/386121641","heartbeat_front_addr":"172.21.15.29:6807/386121641","state":["exists","up"]},{"osd":1,"uuid":"906aba19-80dc-41b5-953c-af6dca604b05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6801","nonce":2795953749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6803","nonce":2795953749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6807","nonce":2795953749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6805","nonce":2795953749}]},"public_addr":"172.21.15.47:6801/2795953749","cluster_addr":"172.21.15.47:6803/2795953749","heartbeat_back_addr":"172.21.15.47:6807/2795953749","heartbeat_front_addr":"172.21.15.47:6805/2795953749","state":["exists","up"]},{"osd":2,"uuid":"194ae5a5-aebf-4cff-9b99-585a3073a269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6801","nonce":2202565078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6803","nonce":2202565078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6807","nonce":2202565078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6805","nonce":2202565078}]},"public_addr":"172.21.15.77:6801/2202565078","cluster_addr":"172.21.15.77:6803/2202565078","heartbeat_back_addr":"172.21.15.77:6807/2202565078","heartbeat_front_addr":"172.21.15.77:6805/2202565078","state":["exists","up"]},{"osd":3,"uuid":"3c22ce9c-963e-4f72-86e5-360e781526db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6811","nonce":3757476312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6813","nonce":3757476312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6817","nonce":3757476312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6815","nonce":3757476312}]},"public_addr":"172.21.15.29:6811/3757476312","cluster_addr":"172.21.15.29:6813/3757476312","heartbeat_back_addr":"172.21.15.29:6817/3757476312","heartbeat_front_addr":"172.21.15.29:6815/3757476312","state":["exists","up"]},{"osd":4,"uuid":"cd4b585f-e7a0-4d82-873d-3a9269ebac3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6809","nonce":1379363720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6811","nonce":1379363720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6815","nonce":1379363720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6813","nonce":1379363720}]},"public_addr":"172.21.15.47:6809/1379363720","cluster_addr":"172.21.15.47:6811/1379363720","heartbeat_back_addr":"172.21.15.47:6815/1379363720","heartbeat_front_addr":"172.21.15.47:6813/1379363720","state":["exists","up"]},{"osd":5,"uuid":"40a3eb14-dfaa-4523-8c28-d74c19293c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6809","nonce":1338438780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6811","nonce":1338438780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6815","nonce":1338438780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6813","nonce":1338438780}]},"public_addr":"172.21.15.77:6809/1338438780","cluster_addr":"172.21.15.77:6811/1338438780","heartbeat_back_addr":"172.21.15.77:6815/1338438780","heartbeat_front_addr":"172.21.15.77:6813/1338438780","state":["exists","up"]},{"osd":6,"uuid":"91e85be2-10d1-4209-aa33-17da8f8bd8bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6819","nonce":2679402148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6821","nonce":2679402148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6825","nonce":2679402148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6823","nonce":2679402148}]},"public_addr":"172.21.15.29:6819/2679402148","cluster_addr":"172.21.15.29:6821/2679402148","heartbeat_back_addr":"172.21.15.29:6825/2679402148","heartbeat_front_addr":"172.21.15.29:6823/2679402148","state":["exists","up"]},{"osd":7,"uuid":"c655b450-0cc4-45d4-95ea-93a57fe8e4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6817","nonce":2439195589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6819","nonce":2439195589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6823","nonce":2439195589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6821","nonce":2439195589}]},"public_addr":"172.21.15.47:6817/2439195589","cluster_addr":"172.21.15.47:6819/2439195589","heartbeat_back_addr":"172.21.15.47:6823/2439195589","heartbeat_front_addr":"172.21.15.47:6821/2439195589","state":["exists","up"]},{"osd":8,"uuid":"a7a95b51-f32c-433e-a197-feb6d47b6c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6817","nonce":1422290905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6819","nonce":1422290905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6823","nonce":1422290905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6821","nonce":1422290905}]},"public_addr":"172.21.15.77:6817/1422290905","cluster_addr":"172.21.15.77:6819/1422290905","heartbeat_back_addr":"172.21.15.77:6823/1422290905","heartbeat_front_addr":"172.21.15.77:6821/1422290905","state":["exists","up"]},{"osd":9,"uuid":"f352242b-0d2e-41b3-9c83-436cf432b0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6827","nonce":1307316384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6829","nonce":1307316384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6833","nonce":1307316384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6831","nonce":1307316384}]},"public_addr":"172.21.15.29:6827/1307316384","cluster_addr":"172.21.15.29:6829/1307316384","heartbeat_back_addr":"172.21.15.29:6833/1307316384","heartbeat_front_addr":"172.21.15.29:6831/1307316384","state":["exists","up"]},{"osd":10,"uuid":"4148c2df-c3e9-4a47-88f7-36cf3eab8d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6825","nonce":2133726850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6827","nonce":2133726850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6831","nonce":2133726850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6829","nonce":2133726850}]},"public_addr":"172.21.15.47:6825/2133726850","cluster_addr":"172.21.15.47:6827/2133726850","heartbeat_back_addr":"172.21.15.47:6831/2133726850","heartbeat_front_addr":"172.21.15.47:6829/2133726850","state":["exists","up"]},{"osd":11,"uuid":"5e8b1da3-5f65-4fe9-8722-b35921cbd8c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6825","nonce":3228299327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6827","nonce":3228299327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6831","nonce":3228299327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6829","nonce":3228299327}]},"public_addr":"172.21.15.77:6825/3228299327","cluster_addr":"172.21.15.77:6827/3228299327","heartbeat_back_addr":"172.21.15.77:6831/3228299327","heartbeat_front_addr":"172.21.15.77:6829/3228299327","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:32.525206+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:56.489640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:18.887845+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:43.727678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:08.755553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:32.923461+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:59.298439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:23.944848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:48.883050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:14.566310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:39.557809+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:42:03.813903+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2378279024":"2024-07-28T06:36:07.527503+0000","172.21.15.29:6801/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3527133965":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2262661620":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/3156863587":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2424241521":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3881857930":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/398508875":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6801/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6800/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2679856281":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6801/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2708687265":"2024-07-28T06:34:58.250198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T06:43:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.786+0000 7fc7a1909640 1 -- 172.21.15.29:0/2682450108 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc7840b3ca0 msgr2=0x7fc7840b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.786+0000 7fc7a1909640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc7840b3ca0 0x7fc7840b61b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc78c005d00 tx=0x7fc78c00b410 comp rx=0 tx=0).stop 2024-07-27T06:43:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.786+0000 7fc7a1909640 1 -- 172.21.15.29:0/2682450108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc79c108750 msgr2=0x7fc79c19e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.786+0000 7fc7a1909640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc79c108750 0x7fc79c19e5b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc79400eb10 tx=0x7fc79400c2d0 comp rx=0 tx=0).stop 2024-07-27T06:43:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.786+0000 7fc7a1909640 1 -- 172.21.15.29:0/2682450108 shutdown_connections 2024-07-27T06:43:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.786+0000 7fc7a1909640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc79c108750 0x7fc79c19e5b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.786+0000 7fc7a1909640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc7840b3ca0 0x7fc7840b61b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.786+0000 7fc7a1909640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc79c107e00 0x7fc79c199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.786+0000 7fc7a1909640 1 --2- 172.21.15.29:0/2682450108 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc79c102510 0x7fc79c198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.786+0000 7fc7a1909640 1 -- 172.21.15.29:0/2682450108 >> 172.21.15.29:0/2682450108 conn(0x7fc79c079190 msgr2=0x7fc79c104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.786+0000 7fc7a1909640 1 -- 172.21.15.29:0/2682450108 shutdown_connections 2024-07-27T06:43:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:18.786+0000 7fc7a1909640 1 -- 172.21.15.29:0/2682450108 wait complete. 2024-07-27T06:43:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:18 smithi029 ceph-mon[32610]: pgmap v282: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 2 op/s 2024-07-27T06:43:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:18 smithi029 ceph-mon[32610]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-27T06:43:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:18 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:43:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:18 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2682450108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:43:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:18 smithi077 ceph-mon[37153]: pgmap v282: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 2 op/s 2024-07-27T06:43:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:18 smithi077 ceph-mon[37153]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-27T06:43:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:18 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:43:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:18 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2682450108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:43:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:18 smithi047 ceph-mon[36835]: pgmap v282: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 2 op/s 2024-07-27T06:43:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:18 smithi047 ceph-mon[36835]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-27T06:43:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:18 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:43:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:18 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2682450108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:43:19.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd dump --format=json 2024-07-27T06:43:19.881 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:20 smithi077 ceph-mon[37153]: pgmap v283: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 2 op/s 2024-07-27T06:43:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:20 smithi047 ceph-mon[36835]: pgmap v283: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 2 op/s 2024-07-27T06:43:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:20 smithi029 ceph-mon[32610]: pgmap v283: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 2 op/s 2024-07-27T06:43:21.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.834+0000 7f98b10c1640 1 -- 172.21.15.29:0/2203472640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac1024b0 msgr2=0x7f98ac1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:21.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.834+0000 7f98b10c1640 1 --2- 172.21.15.29:0/2203472640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac1024b0 0x7f98ac1028c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f989c009900 tx=0x7f989c02a3f0 comp rx=0 tx=0).stop 2024-07-27T06:43:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.834+0000 7f98b10c1640 1 -- 172.21.15.29:0/2203472640 shutdown_connections 2024-07-27T06:43:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.834+0000 7f98b10c1640 1 --2- 172.21.15.29:0/2203472640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98ac108740 0x7f98ac10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.834+0000 7f98b10c1640 1 --2- 172.21.15.29:0/2203472640 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98ac107df0 0x7f98ac108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.834+0000 7f98b10c1640 1 --2- 172.21.15.29:0/2203472640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac1024b0 0x7f98ac1028c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.834+0000 7f98b10c1640 1 -- 172.21.15.29:0/2203472640 >> 172.21.15.29:0/2203472640 conn(0x7f98ac079170 msgr2=0x7f98ac079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.834+0000 7f98b10c1640 1 -- 172.21.15.29:0/2203472640 shutdown_connections 2024-07-27T06:43:21.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.834+0000 7f98b10c1640 1 -- 172.21.15.29:0/2203472640 wait complete. 2024-07-27T06:43:21.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.835+0000 7f98b10c1640 1 Processor -- start 2024-07-27T06:43:21.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.835+0000 7f98b10c1640 1 -- start start 2024-07-27T06:43:21.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.835+0000 7f98b10c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98ac1024b0 0x7f98ac103ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:21.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.835+0000 7f98b10c1640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98ac107df0 0x7f98ac1041e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:21.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.835+0000 7f98b10c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac108740 0x7f98ac104720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:21.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.835+0000 7f98b10c1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac1123a0 con 0x7f98ac108740 2024-07-27T06:43:21.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.835+0000 7f98b10c1640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac112220 con 0x7f98ac107df0 2024-07-27T06:43:21.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.835+0000 7f98b10c1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f98ac112520 con 0x7f98ac1024b0 2024-07-27T06:43:21.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.835+0000 7f98ab59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac108740 0x7f98ac104720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:21.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.835+0000 7f98ab59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac108740 0x7f98ac104720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51428/0 (socket says 172.21.15.29:51428) 2024-07-27T06:43:21.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.835+0000 7f98ab59e640 1 -- 172.21.15.29:0/3353314033 learned_addr learned my addr 172.21.15.29:0/3353314033 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:21.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.835+0000 7f98a3fff640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98ac107df0 0x7f98ac1041e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:21.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98aad9d640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98ac1024b0 0x7f98ac103ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:21.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98ab59e640 1 -- 172.21.15.29:0/3353314033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98ac1024b0 msgr2=0x7f98ac103ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:21.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98ab59e640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98ac1024b0 0x7f98ac103ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:21.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98ab59e640 1 -- 172.21.15.29:0/3353314033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98ac107df0 msgr2=0x7f98ac1041e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:21.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98ab59e640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98ac107df0 0x7f98ac1041e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:21.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98ab59e640 1 -- 172.21.15.29:0/3353314033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98ac1a80f0 con 0x7f98ac108740 2024-07-27T06:43:21.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98aad9d640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98ac1024b0 0x7f98ac103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:43:21.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98ab59e640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac108740 0x7f98ac104720 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f98a400cce0 tx=0x7f98a400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:21.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98a8d99640 1 -- 172.21.15.29:0/3353314033 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a400f450 con 0x7f98ac108740 2024-07-27T06:43:21.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98a8d99640 1 -- 172.21.15.29:0/3353314033 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a400faa0 con 0x7f98ac108740 2024-07-27T06:43:21.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98b10c1640 1 -- 172.21.15.29:0/3353314033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98ac1a83e0 con 0x7f98ac108740 2024-07-27T06:43:21.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98a8d99640 1 -- 172.21.15.29:0/3353314033 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a4018b70 con 0x7f98ac108740 2024-07-27T06:43:21.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.836+0000 7f98b10c1640 1 -- 172.21.15.29:0/3353314033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98ac1a88d0 con 0x7f98ac108740 2024-07-27T06:43:21.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.837+0000 7f98b10c1640 1 -- 172.21.15.29:0/3353314033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98ac103730 con 0x7f98ac108740 2024-07-27T06:43:21.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.839+0000 7f98a8d99640 1 -- 172.21.15.29:0/3353314033 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f98a4018d10 con 0x7f98ac108740 2024-07-27T06:43:21.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.840+0000 7f98a8d99640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f98980b3c20 0x7f98980b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:21.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.840+0000 7f98aad9d640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f98980b3c20 0x7f98980b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:21.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.840+0000 7f98a8d99640 1 -- 172.21.15.29:0/3353314033 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7f98a40e41d0 con 0x7f98ac108740 2024-07-27T06:43:21.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.841+0000 7f98aad9d640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f98980b3c20 0x7f98980b6130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f989c000c00 tx=0x7f989c009680 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:21.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:21.843+0000 7f98a8d99640 1 -- 172.21.15.29:0/3353314033 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f98a4014030 con 0x7f98ac108740 2024-07-27T06:43:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.026+0000 7f98b10c1640 1 -- 172.21.15.29:0/3353314033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98ac104fa0 con 0x7f98ac108740 2024-07-27T06:43:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.027+0000 7f98a8d99640 1 -- 172.21.15.29:0/3353314033 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22464 (secure 0 0 0) 0x7f98ac104fa0 con 0x7f98ac108740 2024-07-27T06:43:22.029 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:43:22.030 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":79,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","created":"2024-07-27T06:33:57.535327+0000","modified":"2024-07-27T06:43:14.801257+0000","last_up_change":"2024-07-27T06:42:05.757156+0000","last_in_change":"2024-07-27T06:41:46.423732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T06:38:21.606527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-27T06:43:07.755490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-27T06:43:11.658694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06833d24-2e1a-4199-9338-7d6883d1bf9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6803","nonce":386121641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6805","nonce":386121641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6809","nonce":386121641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6807","nonce":386121641}]},"public_addr":"172.21.15.29:6803/386121641","cluster_addr":"172.21.15.29:6805/386121641","heartbeat_back_addr":"172.21.15.29:6809/386121641","heartbeat_front_addr":"172.21.15.29:6807/386121641","state":["exists","up"]},{"osd":1,"uuid":"906aba19-80dc-41b5-953c-af6dca604b05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6801","nonce":2795953749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6803","nonce":2795953749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6807","nonce":2795953749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6805","nonce":2795953749}]},"public_addr":"172.21.15.47:6801/2795953749","cluster_addr":"172.21.15.47:6803/2795953749","heartbeat_back_addr":"172.21.15.47:6807/2795953749","heartbeat_front_addr":"172.21.15.47:6805/2795953749","state":["exists","up"]},{"osd":2,"uuid":"194ae5a5-aebf-4cff-9b99-585a3073a269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6801","nonce":2202565078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6803","nonce":2202565078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6807","nonce":2202565078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6805","nonce":2202565078}]},"public_addr":"172.21.15.77:6801/2202565078","cluster_addr":"172.21.15.77:6803/2202565078","heartbeat_back_addr":"172.21.15.77:6807/2202565078","heartbeat_front_addr":"172.21.15.77:6805/2202565078","state":["exists","up"]},{"osd":3,"uuid":"3c22ce9c-963e-4f72-86e5-360e781526db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6811","nonce":3757476312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6813","nonce":3757476312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6817","nonce":3757476312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6815","nonce":3757476312}]},"public_addr":"172.21.15.29:6811/3757476312","cluster_addr":"172.21.15.29:6813/3757476312","heartbeat_back_addr":"172.21.15.29:6817/3757476312","heartbeat_front_addr":"172.21.15.29:6815/3757476312","state":["exists","up"]},{"osd":4,"uuid":"cd4b585f-e7a0-4d82-873d-3a9269ebac3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6809","nonce":1379363720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6811","nonce":1379363720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6815","nonce":1379363720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6813","nonce":1379363720}]},"public_addr":"172.21.15.47:6809/1379363720","cluster_addr":"172.21.15.47:6811/1379363720","heartbeat_back_addr":"172.21.15.47:6815/1379363720","heartbeat_front_addr":"172.21.15.47:6813/1379363720","state":["exists","up"]},{"osd":5,"uuid":"40a3eb14-dfaa-4523-8c28-d74c19293c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6809","nonce":1338438780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6811","nonce":1338438780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6815","nonce":1338438780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6813","nonce":1338438780}]},"public_addr":"172.21.15.77:6809/1338438780","cluster_addr":"172.21.15.77:6811/1338438780","heartbeat_back_addr":"172.21.15.77:6815/1338438780","heartbeat_front_addr":"172.21.15.77:6813/1338438780","state":["exists","up"]},{"osd":6,"uuid":"91e85be2-10d1-4209-aa33-17da8f8bd8bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6819","nonce":2679402148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6821","nonce":2679402148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6825","nonce":2679402148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6823","nonce":2679402148}]},"public_addr":"172.21.15.29:6819/2679402148","cluster_addr":"172.21.15.29:6821/2679402148","heartbeat_back_addr":"172.21.15.29:6825/2679402148","heartbeat_front_addr":"172.21.15.29:6823/2679402148","state":["exists","up"]},{"osd":7,"uuid":"c655b450-0cc4-45d4-95ea-93a57fe8e4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6817","nonce":2439195589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6819","nonce":2439195589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6823","nonce":2439195589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6821","nonce":2439195589}]},"public_addr":"172.21.15.47:6817/2439195589","cluster_addr":"172.21.15.47:6819/2439195589","heartbeat_back_addr":"172.21.15.47:6823/2439195589","heartbeat_front_addr":"172.21.15.47:6821/2439195589","state":["exists","up"]},{"osd":8,"uuid":"a7a95b51-f32c-433e-a197-feb6d47b6c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6817","nonce":1422290905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6819","nonce":1422290905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6823","nonce":1422290905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6821","nonce":1422290905}]},"public_addr":"172.21.15.77:6817/1422290905","cluster_addr":"172.21.15.77:6819/1422290905","heartbeat_back_addr":"172.21.15.77:6823/1422290905","heartbeat_front_addr":"172.21.15.77:6821/1422290905","state":["exists","up"]},{"osd":9,"uuid":"f352242b-0d2e-41b3-9c83-436cf432b0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6827","nonce":1307316384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6829","nonce":1307316384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6833","nonce":1307316384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6831","nonce":1307316384}]},"public_addr":"172.21.15.29:6827/1307316384","cluster_addr":"172.21.15.29:6829/1307316384","heartbeat_back_addr":"172.21.15.29:6833/1307316384","heartbeat_front_addr":"172.21.15.29:6831/1307316384","state":["exists","up"]},{"osd":10,"uuid":"4148c2df-c3e9-4a47-88f7-36cf3eab8d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6825","nonce":2133726850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6827","nonce":2133726850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6831","nonce":2133726850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6829","nonce":2133726850}]},"public_addr":"172.21.15.47:6825/2133726850","cluster_addr":"172.21.15.47:6827/2133726850","heartbeat_back_addr":"172.21.15.47:6831/2133726850","heartbeat_front_addr":"172.21.15.47:6829/2133726850","state":["exists","up"]},{"osd":11,"uuid":"5e8b1da3-5f65-4fe9-8722-b35921cbd8c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6825","nonce":3228299327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6827","nonce":3228299327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6831","nonce":3228299327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6829","nonce":3228299327}]},"public_addr":"172.21.15.77:6825/3228299327","cluster_addr":"172.21.15.77:6827/3228299327","heartbeat_back_addr":"172.21.15.77:6831/3228299327","heartbeat_front_addr":"172.21.15.77:6829/3228299327","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:32.525206+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:56.489640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:18.887845+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:43.727678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:08.755553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:32.923461+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:59.298439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:23.944848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:48.883050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:14.566310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:39.557809+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:42:03.813903+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2378279024":"2024-07-28T06:36:07.527503+0000","172.21.15.29:6801/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3527133965":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2262661620":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/3156863587":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2424241521":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3881857930":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/398508875":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6801/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6800/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2679856281":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6801/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2708687265":"2024-07-28T06:34:58.250198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T06:43:22.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.029+0000 7f98b10c1640 1 -- 172.21.15.29:0/3353314033 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f98980b3c20 msgr2=0x7f98980b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:22.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.029+0000 7f98b10c1640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f98980b3c20 0x7f98980b6130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f989c000c00 tx=0x7f989c009680 comp rx=0 tx=0).stop 2024-07-27T06:43:22.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.029+0000 7f98b10c1640 1 -- 172.21.15.29:0/3353314033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac108740 msgr2=0x7f98ac104720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:22.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.029+0000 7f98b10c1640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac108740 0x7f98ac104720 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f98a400cce0 tx=0x7f98a400a620 comp rx=0 tx=0).stop 2024-07-27T06:43:22.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.029+0000 7f98b10c1640 1 -- 172.21.15.29:0/3353314033 shutdown_connections 2024-07-27T06:43:22.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.029+0000 7f98b10c1640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98ac108740 0x7f98ac104720 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:22.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.029+0000 7f98b10c1640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98ac107df0 0x7f98ac1041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:22.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.029+0000 7f98b10c1640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f98980b3c20 0x7f98980b6130 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:22.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.029+0000 7f98b10c1640 1 --2- 172.21.15.29:0/3353314033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98ac1024b0 0x7f98ac103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:22.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.029+0000 7f98b10c1640 1 -- 172.21.15.29:0/3353314033 >> 172.21.15.29:0/3353314033 conn(0x7f98ac079170 msgr2=0x7f98ac101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:22.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.029+0000 7f98b10c1640 1 -- 172.21.15.29:0/3353314033 shutdown_connections 2024-07-27T06:43:22.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:22.029+0000 7f98b10c1640 1 -- 172.21.15.29:0/3353314033 wait complete. 2024-07-27T06:43:22.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-07-27T06:43:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:22 smithi077 ceph-mon[37153]: pgmap v284: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-27T06:43:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:22 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3353314033' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:43:23.164 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:22 smithi047 ceph-mon[36835]: pgmap v284: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-27T06:43:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:22 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3353314033' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:43:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:22 smithi029 ceph-mon[32610]: pgmap v284: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-27T06:43:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:22 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3353314033' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:43:25.034 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:24 smithi029 ceph-mon[32610]: pgmap v285: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-07-27T06:43:25.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.029+0000 7f5899836640 1 -- 172.21.15.29:0/3404997971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894108750 msgr2=0x7f589410ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.029+0000 7f5899836640 1 --2- 172.21.15.29:0/3404997971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894108750 0x7f589410ef80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f588c009ca0 tx=0x7f588c02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:43:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.029+0000 7f5899836640 1 -- 172.21.15.29:0/3404997971 shutdown_connections 2024-07-27T06:43:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.029+0000 7f5899836640 1 --2- 172.21.15.29:0/3404997971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894108750 0x7f589410ef80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.029+0000 7f5899836640 1 --2- 172.21.15.29:0/3404997971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5894107e00 0x7f5894108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.029+0000 7f5899836640 1 --2- 172.21.15.29:0/3404997971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5894102510 0x7f5894102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.029+0000 7f5899836640 1 -- 172.21.15.29:0/3404997971 >> 172.21.15.29:0/3404997971 conn(0x7f5894079190 msgr2=0x7f58940795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.029+0000 7f5899836640 1 -- 172.21.15.29:0/3404997971 shutdown_connections 2024-07-27T06:43:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.029+0000 7f5899836640 1 -- 172.21.15.29:0/3404997971 wait complete. 2024-07-27T06:43:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.029+0000 7f5899836640 1 Processor -- start 2024-07-27T06:43:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.030+0000 7f5899836640 1 -- start start 2024-07-27T06:43:25.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.030+0000 7f5899836640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894102510 0x7f589410fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.030+0000 7f5899836640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5894107e00 0x7f589410ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.030+0000 7f5899836640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5894108750 0x7f5894110490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.030+0000 7f5899836640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f58941134e0 con 0x7f5894102510 2024-07-27T06:43:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.030+0000 7f5899836640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f5894113360 con 0x7f5894108750 2024-07-27T06:43:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.030+0000 7f5899836640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5894113660 con 0x7f5894107e00 2024-07-27T06:43:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.030+0000 7f5892ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894102510 0x7f589410fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.030+0000 7f5892ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894102510 0x7f589410fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51448/0 (socket says 172.21.15.29:51448) 2024-07-27T06:43:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.030+0000 7f5892ffd640 1 -- 172.21.15.29:0/3520221475 learned_addr learned my addr 172.21.15.29:0/3520221475 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.030+0000 7f58927fc640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5894107e00 0x7f589410ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.030+0000 7f58937fe640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5894108750 0x7f5894110490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:25.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.031+0000 7f5892ffd640 1 -- 172.21.15.29:0/3520221475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5894107e00 msgr2=0x7f589410ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.031+0000 7f5892ffd640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5894107e00 0x7f589410ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.031+0000 7f5892ffd640 1 -- 172.21.15.29:0/3520221475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5894108750 msgr2=0x7f5894110490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.031+0000 7f5892ffd640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5894108750 0x7f5894110490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.031+0000 7f5892ffd640 1 -- 172.21.15.29:0/3520221475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58941a7f80 con 0x7f5894102510 2024-07-27T06:43:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.031+0000 7f5892ffd640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894102510 0x7f589410fa10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f588400c920 tx=0x7f588400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.031+0000 7f5898834640 1 -- 172.21.15.29:0/3520221475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5884007bb0 con 0x7f5894102510 2024-07-27T06:43:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.031+0000 7f5898834640 1 -- 172.21.15.29:0/3520221475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5884007d50 con 0x7f5894102510 2024-07-27T06:43:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.031+0000 7f5899836640 1 -- 172.21.15.29:0/3520221475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58941a8270 con 0x7f5894102510 2024-07-27T06:43:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.031+0000 7f5898834640 1 -- 172.21.15.29:0/3520221475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5884018730 con 0x7f5894102510 2024-07-27T06:43:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.031+0000 7f5899836640 1 -- 172.21.15.29:0/3520221475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58941a87b0 con 0x7f5894102510 2024-07-27T06:43:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.032+0000 7f5899836640 1 -- 172.21.15.29:0/3520221475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5894103740 con 0x7f5894102510 2024-07-27T06:43:25.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.034+0000 7f5898834640 1 -- 172.21.15.29:0/3520221475 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f588400f450 con 0x7f5894102510 2024-07-27T06:43:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.034+0000 7f5898834640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f587c0b3c20 0x7f587c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.035+0000 7f58927fc640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f587c0b3c20 0x7f587c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.035+0000 7f5898834640 1 -- 172.21.15.29:0/3520221475 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7f58840e3f90 con 0x7f5894102510 2024-07-27T06:43:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.035+0000 7f58927fc640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f587c0b3c20 0x7f587c0b6130 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5894105950 tx=0x7f588800a2d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.038+0000 7f5898834640 1 -- 172.21.15.29:0/3520221475 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5884014030 con 0x7f5894102510 2024-07-27T06:43:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:24 smithi077 ceph-mon[37153]: pgmap v285: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-07-27T06:43:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:24 smithi047 ceph-mon[36835]: pgmap v285: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-07-27T06:43:25.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.258+0000 7f5899836640 1 -- 172.21.15.29:0/3520221475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5894102920 con 0x7f5894102510 2024-07-27T06:43:25.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.817+0000 7f5898834640 1 -- 172.21.15.29:0/3520221475 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f58840ab690 con 0x7f5894102510 2024-07-27T06:43:25.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.819+0000 7f5899836640 1 -- 172.21.15.29:0/3520221475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f587c0b3c20 msgr2=0x7f587c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:25.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.819+0000 7f5899836640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f587c0b3c20 0x7f587c0b6130 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5894105950 tx=0x7f588800a2d0 comp rx=0 tx=0).stop 2024-07-27T06:43:25.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.819+0000 7f5899836640 1 -- 172.21.15.29:0/3520221475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894102510 msgr2=0x7f589410fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:25.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.819+0000 7f5899836640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894102510 0x7f589410fa10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f588400c920 tx=0x7f588400cd20 comp rx=0 tx=0).stop 2024-07-27T06:43:25.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.820+0000 7f5899836640 1 -- 172.21.15.29:0/3520221475 shutdown_connections 2024-07-27T06:43:25.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.820+0000 7f5899836640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5894108750 0x7f5894110490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:25.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.820+0000 7f5899836640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f587c0b3c20 0x7f587c0b6130 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:25.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.820+0000 7f5899836640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5894107e00 0x7f589410ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:25.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.820+0000 7f5899836640 1 --2- 172.21.15.29:0/3520221475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5894102510 0x7f589410fa10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:25.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.820+0000 7f5899836640 1 -- 172.21.15.29:0/3520221475 >> 172.21.15.29:0/3520221475 conn(0x7f5894079190 msgr2=0x7f5894103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:25.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.820+0000 7f5899836640 1 -- 172.21.15.29:0/3520221475 shutdown_connections 2024-07-27T06:43:25.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:25.820+0000 7f5899836640 1 -- 172.21.15.29:0/3520221475 wait complete. 2024-07-27T06:43:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:25 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3520221475' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-27T06:43:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:25 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3520221475' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-27T06:43:26.207 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:25 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3520221475' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-27T06:43:26.589 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-07-27T06:43:26.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-07-27T06:43:26.867 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:26 smithi077 ceph-mon[37153]: pgmap v286: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-07-27T06:43:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:26 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3520221475' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-27T06:43:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:26 smithi077 ceph-mon[37153]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-07-27T06:43:27.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:26 smithi029 ceph-mon[32610]: pgmap v286: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-07-27T06:43:27.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:26 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3520221475' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-27T06:43:27.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:26 smithi029 ceph-mon[32610]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-07-27T06:43:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:26 smithi047 ceph-mon[36835]: pgmap v286: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-07-27T06:43:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:26 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3520221475' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-27T06:43:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:26 smithi047 ceph-mon[36835]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-07-27T06:43:28.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.761+0000 7fdb94d87640 1 -- 172.21.15.29:0/166764613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb90103590 msgr2=0x7fdb901039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.761+0000 7fdb94d87640 1 --2- 172.21.15.29:0/166764613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb90103590 0x7fdb901039a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdb80009900 tx=0x7fdb8002a410 comp rx=0 tx=0).stop 2024-07-27T06:43:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.762+0000 7fdb94d87640 1 -- 172.21.15.29:0/166764613 shutdown_connections 2024-07-27T06:43:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.762+0000 7fdb94d87640 1 --2- 172.21.15.29:0/166764613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb90109820 0x7fdb90110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.762+0000 7fdb94d87640 1 --2- 172.21.15.29:0/166764613 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb90108ed0 0x7fdb901092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.762+0000 7fdb94d87640 1 --2- 172.21.15.29:0/166764613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb90103590 0x7fdb901039a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.762+0000 7fdb94d87640 1 -- 172.21.15.29:0/166764613 >> 172.21.15.29:0/166764613 conn(0x7fdb9007b280 msgr2=0x7fdb9007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.762+0000 7fdb94d87640 1 -- 172.21.15.29:0/166764613 shutdown_connections 2024-07-27T06:43:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.762+0000 7fdb94d87640 1 -- 172.21.15.29:0/166764613 wait complete. 2024-07-27T06:43:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb94d87640 1 Processor -- start 2024-07-27T06:43:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb94d87640 1 -- start start 2024-07-27T06:43:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb94d87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb90103590 0x7fdb90199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb94d87640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb90108ed0 0x7fdb9019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb94d87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb90109820 0x7fdb9019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb94d87640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdb901134f0 con 0x7fdb90109820 2024-07-27T06:43:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb94d87640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90113370 con 0x7fdb90108ed0 2024-07-27T06:43:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb94d87640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90113670 con 0x7fdb90103590 2024-07-27T06:43:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb8ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb90109820 0x7fdb9019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb8e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb90103590 0x7fdb90199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb8ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb90109820 0x7fdb9019f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51466/0 (socket says 172.21.15.29:51466) 2024-07-27T06:43:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb8e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb90103590 0x7fdb90199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:33936/0 (socket says 172.21.15.29:33936) 2024-07-27T06:43:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb8ed9d640 1 -- 172.21.15.29:0/770019822 learned_addr learned my addr 172.21.15.29:0/770019822 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.763+0000 7fdb8dd9b640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb90108ed0 0x7fdb9019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.764+0000 7fdb8ed9d640 1 -- 172.21.15.29:0/770019822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb90103590 msgr2=0x7fdb90199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.764+0000 7fdb8ed9d640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb90103590 0x7fdb90199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.764+0000 7fdb8ed9d640 1 -- 172.21.15.29:0/770019822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb90108ed0 msgr2=0x7fdb9019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.764+0000 7fdb8ed9d640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb90108ed0 0x7fdb9019a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.764+0000 7fdb8ed9d640 1 -- 172.21.15.29:0/770019822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9019fe90 con 0x7fdb90109820 2024-07-27T06:43:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.764+0000 7fdb8dd9b640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb90108ed0 0x7fdb9019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:43:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.764+0000 7fdb8ed9d640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb90109820 0x7fdb9019f760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800eaa0 tx=0x7fdb8800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.764+0000 7fdb877fe640 1 -- 172.21.15.29:0/770019822 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8800cdb0 con 0x7fdb90109820 2024-07-27T06:43:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.764+0000 7fdb94d87640 1 -- 172.21.15.29:0/770019822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb901a0180 con 0x7fdb90109820 2024-07-27T06:43:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.764+0000 7fdb877fe640 1 -- 172.21.15.29:0/770019822 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb88005020 con 0x7fdb90109820 2024-07-27T06:43:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.764+0000 7fdb877fe640 1 -- 172.21.15.29:0/770019822 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb880116d0 con 0x7fdb90109820 2024-07-27T06:43:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.764+0000 7fdb94d87640 1 -- 172.21.15.29:0/770019822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb901a68e0 con 0x7fdb90109820 2024-07-27T06:43:28.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.765+0000 7fdb94d87640 1 -- 172.21.15.29:0/770019822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb901039a0 con 0x7fdb90109820 2024-07-27T06:43:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.767+0000 7fdb877fe640 1 -- 172.21.15.29:0/770019822 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fdb880054a0 con 0x7fdb90109820 2024-07-27T06:43:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.768+0000 7fdb877fe640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdb7c0b3c20 0x7fdb7c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.768+0000 7fdb8e59c640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdb7c0b3c20 0x7fdb7c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.768+0000 7fdb877fe640 1 -- 172.21.15.29:0/770019822 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7fdb880e40c0 con 0x7fdb90109820 2024-07-27T06:43:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.768+0000 7fdb8e59c640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdb7c0b3c20 0x7fdb7c0b6130 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdb80031440 tx=0x7fdb80009560 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.771+0000 7fdb877fe640 1 -- 172.21.15.29:0/770019822 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdb88016030 con 0x7fdb90109820 2024-07-27T06:43:29.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:28.999+0000 7fdb94d87640 1 -- 172.21.15.29:0/770019822 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fdb9010b600 con 0x7fdb90109820 2024-07-27T06:43:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:28 smithi077 ceph-mon[37153]: pgmap v287: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-07-27T06:43:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:28 smithi047 ceph-mon[36835]: pgmap v287: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-07-27T06:43:29.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:28 smithi029 ceph-mon[32610]: pgmap v287: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-07-27T06:43:29.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.828+0000 7fdb877fe640 1 -- 172.21.15.29:0/770019822 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fdb880ab730 con 0x7fdb90109820 2024-07-27T06:43:29.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.831+0000 7fdb94d87640 1 -- 172.21.15.29:0/770019822 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdb7c0b3c20 msgr2=0x7fdb7c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:29.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.831+0000 7fdb94d87640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdb7c0b3c20 0x7fdb7c0b6130 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdb80031440 tx=0x7fdb80009560 comp rx=0 tx=0).stop 2024-07-27T06:43:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.831+0000 7fdb94d87640 1 -- 172.21.15.29:0/770019822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb90109820 msgr2=0x7fdb9019f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.831+0000 7fdb94d87640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb90109820 0x7fdb9019f760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800eaa0 tx=0x7fdb8800ef60 comp rx=0 tx=0).stop 2024-07-27T06:43:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.831+0000 7fdb94d87640 1 -- 172.21.15.29:0/770019822 shutdown_connections 2024-07-27T06:43:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.831+0000 7fdb94d87640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb90109820 0x7fdb9019f760 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.831+0000 7fdb94d87640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb90108ed0 0x7fdb9019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.831+0000 7fdb94d87640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdb7c0b3c20 0x7fdb7c0b6130 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.831+0000 7fdb94d87640 1 --2- 172.21.15.29:0/770019822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb90103590 0x7fdb90199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.831+0000 7fdb94d87640 1 -- 172.21.15.29:0/770019822 >> 172.21.15.29:0/770019822 conn(0x7fdb9007b280 msgr2=0x7fdb90107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.831+0000 7fdb94d87640 1 -- 172.21.15.29:0/770019822 shutdown_connections 2024-07-27T06:43:29.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:29.831+0000 7fdb94d87640 1 -- 172.21.15.29:0/770019822 wait complete. 2024-07-27T06:43:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:29 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/770019822' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-27T06:43:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:29 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/770019822' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-27T06:43:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:29 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/770019822' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-27T06:43:30.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-07-27T06:43:31.091 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:30 smithi077 ceph-mon[37153]: pgmap v288: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 2 op/s 2024-07-27T06:43:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:30 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-27T06:43:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:30 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/770019822' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-27T06:43:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:30 smithi077 ceph-mon[37153]: daemon mds.c assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-27T06:43:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:30 smithi077 ceph-mon[37153]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-07-27T06:43:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:30 smithi077 ceph-mon[37153]: fsmap cephfs:2 {0=i=up:active,1=c=up:creating} 10 up:standby 2024-07-27T06:43:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:30 smithi077 ceph-mon[37153]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-27T06:43:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:30 smithi047 ceph-mon[36835]: pgmap v288: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 2 op/s 2024-07-27T06:43:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:30 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-27T06:43:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:30 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/770019822' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-27T06:43:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:30 smithi047 ceph-mon[36835]: daemon mds.c assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-27T06:43:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:30 smithi047 ceph-mon[36835]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-07-27T06:43:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:30 smithi047 ceph-mon[36835]: fsmap cephfs:2 {0=i=up:active,1=c=up:creating} 10 up:standby 2024-07-27T06:43:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:30 smithi047 ceph-mon[36835]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-27T06:43:31.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:30 smithi029 ceph-mon[32610]: pgmap v288: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 2 op/s 2024-07-27T06:43:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:30 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-27T06:43:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:30 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/770019822' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-27T06:43:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:30 smithi029 ceph-mon[32610]: daemon mds.c assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-27T06:43:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:30 smithi029 ceph-mon[32610]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-07-27T06:43:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:30 smithi029 ceph-mon[32610]: fsmap cephfs:2 {0=i=up:active,1=c=up:creating} 10 up:standby 2024-07-27T06:43:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:30 smithi029 ceph-mon[32610]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-07-27T06:43:32.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:31 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] up:active 2024-07-27T06:43:32.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:31 smithi029 ceph-mon[32610]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-27T06:43:32.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:31 smithi029 ceph-mon[32610]: fsmap cephfs:2 {0=i=up:active,1=c=up:active} 10 up:standby 2024-07-27T06:43:32.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:31 smithi029 ceph-mon[32610]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:creating} 9 up:standby 2024-07-27T06:43:32.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:31 smithi029 ceph-mon[32610]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-07-27T06:43:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:31 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] up:active 2024-07-27T06:43:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:31 smithi077 ceph-mon[37153]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-27T06:43:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:31 smithi077 ceph-mon[37153]: fsmap cephfs:2 {0=i=up:active,1=c=up:active} 10 up:standby 2024-07-27T06:43:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:31 smithi077 ceph-mon[37153]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:creating} 9 up:standby 2024-07-27T06:43:32.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:31 smithi077 ceph-mon[37153]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-07-27T06:43:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:31 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] up:active 2024-07-27T06:43:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:31 smithi047 ceph-mon[36835]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-27T06:43:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:31 smithi047 ceph-mon[36835]: fsmap cephfs:2 {0=i=up:active,1=c=up:active} 10 up:standby 2024-07-27T06:43:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:31 smithi047 ceph-mon[36835]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:creating} 9 up:standby 2024-07-27T06:43:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:31 smithi047 ceph-mon[36835]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-07-27T06:43:32.849 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:43:32 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2024-07-27T06:43:32.403+0000 7fd17da7c640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-07-27T06:43:32.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.955+0000 7f41c9656640 1 -- 172.21.15.29:0/2513606172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c4108170 msgr2=0x7f41c4108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.955+0000 7f41c9656640 1 --2- 172.21.15.29:0/2513606172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c4108170 0x7f41c4108580 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f41b4009900 tx=0x7f41b402a410 comp rx=0 tx=0).stop 2024-07-27T06:43:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.955+0000 7f41c9656640 1 -- 172.21.15.29:0/2513606172 shutdown_connections 2024-07-27T06:43:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.955+0000 7f41c9656640 1 --2- 172.21.15.29:0/2513606172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c41021b0 0x7f41c410ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.955+0000 7f41c9656640 1 --2- 172.21.15.29:0/2513606172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41c4101860 0x7f41c4101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.955+0000 7f41c9656640 1 --2- 172.21.15.29:0/2513606172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c4108170 0x7f41c4108580 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.955+0000 7f41c9656640 1 -- 172.21.15.29:0/2513606172 >> 172.21.15.29:0/2513606172 conn(0x7f41c40799c0 msgr2=0x7f41c4079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.955+0000 7f41c9656640 1 -- 172.21.15.29:0/2513606172 shutdown_connections 2024-07-27T06:43:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.955+0000 7f41c9656640 1 -- 172.21.15.29:0/2513606172 wait complete. 2024-07-27T06:43:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41c9656640 1 Processor -- start 2024-07-27T06:43:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41c9656640 1 -- start start 2024-07-27T06:43:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41c9656640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c4101860 0x7f41c419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41c9656640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c41021b0 0x7f41c419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41c9656640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41c4108170 0x7f41c419f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41c9656640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4113580 con 0x7f41c41021b0 2024-07-27T06:43:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41c9656640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4113400 con 0x7f41c4108170 2024-07-27T06:43:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41c9656640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4113700 con 0x7f41c4101860 2024-07-27T06:43:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c41021b0 0x7f41c419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c41021b0 0x7f41c419a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57866/0 (socket says 172.21.15.29:57866) 2024-07-27T06:43:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41bbfff640 1 -- 172.21.15.29:0/3002655114 learned_addr learned my addr 172.21.15.29:0/3002655114 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41c2ffd640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c4101860 0x7f41c419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.956+0000 7f41c37fe640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41c4108170 0x7f41c419f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.957+0000 7f41bbfff640 1 -- 172.21.15.29:0/3002655114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c4101860 msgr2=0x7f41c419a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.957+0000 7f41bbfff640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c4101860 0x7f41c419a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.957+0000 7f41bbfff640 1 -- 172.21.15.29:0/3002655114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41c4108170 msgr2=0x7f41c419f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.957+0000 7f41bbfff640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41c4108170 0x7f41c419f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.957+0000 7f41bbfff640 1 -- 172.21.15.29:0/3002655114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c419ffe0 con 0x7f41c41021b0 2024-07-27T06:43:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.957+0000 7f41bbfff640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c41021b0 0x7f41c419a610 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f41ac006fd0 tx=0x7f41ac00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.957+0000 7f41c0ff9640 1 -- 172.21.15.29:0/3002655114 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac011830 con 0x7f41c41021b0 2024-07-27T06:43:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.957+0000 7f41c0ff9640 1 -- 172.21.15.29:0/3002655114 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41ac016070 con 0x7f41c41021b0 2024-07-27T06:43:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.957+0000 7f41c9656640 1 -- 172.21.15.29:0/3002655114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c41a02d0 con 0x7f41c41021b0 2024-07-27T06:43:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.957+0000 7f41c0ff9640 1 -- 172.21.15.29:0/3002655114 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac010e00 con 0x7f41c41021b0 2024-07-27T06:43:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.957+0000 7f41c9656640 1 -- 172.21.15.29:0/3002655114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c41a6a30 con 0x7f41c41021b0 2024-07-27T06:43:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.958+0000 7f41c9656640 1 -- 172.21.15.29:0/3002655114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41c4101c70 con 0x7f41c41021b0 2024-07-27T06:43:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.960+0000 7f41c0ff9640 1 -- 172.21.15.29:0/3002655114 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f41ac0119d0 con 0x7f41c41021b0 2024-07-27T06:43:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.961+0000 7f41c0ff9640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f41b00b3ca0 0x7f41b00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.961+0000 7f41c2ffd640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f41b00b3ca0 0x7f41b00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.961+0000 7f41c0ff9640 1 -- 172.21.15.29:0/3002655114 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7f41ac0e3870 con 0x7f41c41021b0 2024-07-27T06:43:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.961+0000 7f41c2ffd640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f41b00b3ca0 0x7f41b00b61b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f41b4031440 tx=0x7f41b4009560 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:32.964+0000 7f41c0ff9640 1 -- 172.21.15.29:0/3002655114 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f41ac0aae80 con 0x7f41c41021b0 2024-07-27T06:43:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:32 smithi077 ceph-mon[37153]: pgmap v289: 97 pgs: 97 active+clean; 580 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 5 op/s 2024-07-27T06:43:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:32 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] up:active 2024-07-27T06:43:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:32 smithi077 ceph-mon[37153]: daemon mds.b assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-27T06:43:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:32 smithi077 ceph-mon[37153]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:active} 9 up:standby 2024-07-27T06:43:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:32 smithi077 ceph-mon[37153]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=b=up:creating} 8 up:standby 2024-07-27T06:43:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:32 smithi077 ceph-mon[37153]: daemon mds.b is now active in filesystem cephfs as rank 3 2024-07-27T06:43:33.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.186+0000 7f41c9656640 1 -- 172.21.15.29:0/3002655114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f41c410b5f0 con 0x7f41c41021b0 2024-07-27T06:43:33.188 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:32 smithi029 ceph-mon[32610]: pgmap v289: 97 pgs: 97 active+clean; 580 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 5 op/s 2024-07-27T06:43:33.188 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:32 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] up:active 2024-07-27T06:43:33.188 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:32 smithi029 ceph-mon[32610]: daemon mds.b assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-27T06:43:33.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:32 smithi029 ceph-mon[32610]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:active} 9 up:standby 2024-07-27T06:43:33.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:32 smithi029 ceph-mon[32610]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=b=up:creating} 8 up:standby 2024-07-27T06:43:33.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:32 smithi029 ceph-mon[32610]: daemon mds.b is now active in filesystem cephfs as rank 3 2024-07-27T06:43:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:32 smithi047 ceph-mon[36835]: pgmap v289: 97 pgs: 97 active+clean; 580 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 5 op/s 2024-07-27T06:43:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:32 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] up:active 2024-07-27T06:43:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:32 smithi047 ceph-mon[36835]: daemon mds.b assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-27T06:43:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:32 smithi047 ceph-mon[36835]: fsmap cephfs:3 {0=i=up:active,1=c=up:active,2=f=up:active} 9 up:standby 2024-07-27T06:43:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:32 smithi047 ceph-mon[36835]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=b=up:creating} 8 up:standby 2024-07-27T06:43:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:32 smithi047 ceph-mon[36835]: daemon mds.b is now active in filesystem cephfs as rank 3 2024-07-27T06:43:33.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.869+0000 7f41c0ff9640 1 -- 172.21.15.29:0/3002655114 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v26) ==== 121+0+0 (secure 0 0 0) 0x7f41ac0aa780 con 0x7f41c41021b0 2024-07-27T06:43:33.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.872+0000 7f41c9656640 1 -- 172.21.15.29:0/3002655114 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f41b00b3ca0 msgr2=0x7f41b00b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:33.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.872+0000 7f41c9656640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f41b00b3ca0 0x7f41b00b61b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f41b4031440 tx=0x7f41b4009560 comp rx=0 tx=0).stop 2024-07-27T06:43:33.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.872+0000 7f41c9656640 1 -- 172.21.15.29:0/3002655114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c41021b0 msgr2=0x7f41c419a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:33.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.872+0000 7f41c9656640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c41021b0 0x7f41c419a610 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f41ac006fd0 tx=0x7f41ac00b8e0 comp rx=0 tx=0).stop 2024-07-27T06:43:33.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.872+0000 7f41c9656640 1 -- 172.21.15.29:0/3002655114 shutdown_connections 2024-07-27T06:43:33.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.872+0000 7f41c9656640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f41c4108170 0x7f41c419f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:33.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.872+0000 7f41c9656640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41c41021b0 0x7f41c419a610 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.872+0000 7f41c9656640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f41b00b3ca0 0x7f41b00b61b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.872+0000 7f41c9656640 1 --2- 172.21.15.29:0/3002655114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f41c4101860 0x7f41c419a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.872+0000 7f41c9656640 1 -- 172.21.15.29:0/3002655114 >> 172.21.15.29:0/3002655114 conn(0x7f41c40799c0 msgr2=0x7f41c4107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.873+0000 7f41c9656640 1 -- 172.21.15.29:0/3002655114 shutdown_connections 2024-07-27T06:43:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:33.873+0000 7f41c9656640 1 -- 172.21.15.29:0/3002655114 wait complete. 2024-07-27T06:43:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:33 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] up:active 2024-07-27T06:43:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:33 smithi077 ceph-mon[37153]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-27T06:43:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:33 smithi077 ceph-mon[37153]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-27T06:43:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:33 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:43:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:33 smithi077 ceph-mon[37153]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=b=up:active} 8 up:standby 2024-07-27T06:43:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:33 smithi077 ceph-mon[37153]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-27T06:43:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:33 smithi077 ceph-mon[37153]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-07-27T06:43:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:33 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3002655114' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-27T06:43:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:33 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] up:active 2024-07-27T06:43:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:33 smithi047 ceph-mon[36835]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-27T06:43:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:33 smithi047 ceph-mon[36835]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-27T06:43:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:33 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:43:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:33 smithi047 ceph-mon[36835]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=b=up:active} 8 up:standby 2024-07-27T06:43:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:33 smithi047 ceph-mon[36835]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-27T06:43:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:33 smithi047 ceph-mon[36835]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-07-27T06:43:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:33 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3002655114' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-27T06:43:34.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:33 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] up:active 2024-07-27T06:43:34.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:33 smithi029 ceph-mon[32610]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-27T06:43:34.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:33 smithi029 ceph-mon[32610]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-27T06:43:34.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:33 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:43:34.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:33 smithi029 ceph-mon[32610]: fsmap cephfs:4 {0=i=up:active,1=c=up:active,2=f=up:active,3=b=up:active} 8 up:standby 2024-07-27T06:43:34.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:33 smithi029 ceph-mon[32610]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-27T06:43:34.203 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:33 smithi029 ceph-mon[32610]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-07-27T06:43:34.203 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:33 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3002655114' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-27T06:43:34.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-07-27T06:43:34.908 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:34 smithi047 ceph-mon[36835]: pgmap v290: 97 pgs: 97 active+clean; 580 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 6 op/s 2024-07-27T06:43:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:34 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3002655114' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-27T06:43:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:34 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] up:active 2024-07-27T06:43:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:34 smithi047 ceph-mon[36835]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-27T06:43:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:34 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:43:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:34 smithi029 ceph-mon[32610]: pgmap v290: 97 pgs: 97 active+clean; 580 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 6 op/s 2024-07-27T06:43:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:34 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3002655114' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-27T06:43:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:34 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] up:active 2024-07-27T06:43:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:34 smithi029 ceph-mon[32610]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-27T06:43:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:34 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:43:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:34 smithi077 ceph-mon[37153]: pgmap v290: 97 pgs: 97 active+clean; 580 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 6 op/s 2024-07-27T06:43:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:34 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3002655114' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-27T06:43:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:34 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] up:active 2024-07-27T06:43:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:34 smithi077 ceph-mon[37153]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-27T06:43:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:34 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:43:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.781+0000 7f1ef0612640 1 -- 172.21.15.29:0/1210917955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee81024a0 msgr2=0x7f1ee81028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.781+0000 7f1ef0612640 1 --2- 172.21.15.29:0/1210917955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee81024a0 0x7f1ee81028b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4009900 tx=0x7f1ee402a410 comp rx=0 tx=0).stop 2024-07-27T06:43:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.781+0000 7f1ef0612640 1 -- 172.21.15.29:0/1210917955 shutdown_connections 2024-07-27T06:43:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.781+0000 7f1ef0612640 1 --2- 172.21.15.29:0/1210917955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ee8108730 0x7f1ee810ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.781+0000 7f1ef0612640 1 --2- 172.21.15.29:0/1210917955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee8107de0 0x7f1ee81081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.781+0000 7f1ef0612640 1 --2- 172.21.15.29:0/1210917955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee81024a0 0x7f1ee81028b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.781+0000 7f1ef0612640 1 -- 172.21.15.29:0/1210917955 >> 172.21.15.29:0/1210917955 conn(0x7f1ee8079140 msgr2=0x7f1ee8079550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:36.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.781+0000 7f1ef0612640 1 -- 172.21.15.29:0/1210917955 shutdown_connections 2024-07-27T06:43:36.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.781+0000 7f1ef0612640 1 -- 172.21.15.29:0/1210917955 wait complete. 2024-07-27T06:43:36.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.782+0000 7f1ef0612640 1 Processor -- start 2024-07-27T06:43:36.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.782+0000 7f1ef0612640 1 -- start start 2024-07-27T06:43:36.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.782+0000 7f1ef0612640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee81024a0 0x7f1ee8199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.782+0000 7f1ef0612640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ee8107de0 0x7f1ee819a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.782+0000 7f1ef0612640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee8108730 0x7f1ee819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.782+0000 7f1ef0612640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee8112430 con 0x7f1ee81024a0 2024-07-27T06:43:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.782+0000 7f1ef0612640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee81122b0 con 0x7f1ee8108730 2024-07-27T06:43:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.782+0000 7f1ef0612640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee81125b0 con 0x7f1ee8107de0 2024-07-27T06:43:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.782+0000 7f1eee3ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee81024a0 0x7f1ee8199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.782+0000 7f1eee3ae640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee81024a0 0x7f1ee8199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57896/0 (socket says 172.21.15.29:57896) 2024-07-27T06:43:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1eee3ae640 1 -- 172.21.15.29:0/342313462 learned_addr learned my addr 172.21.15.29:0/342313462 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1eeebaf640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee8108730 0x7f1ee819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1eedbad640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ee8107de0 0x7f1ee819a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1eee3ae640 1 -- 172.21.15.29:0/342313462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ee8107de0 msgr2=0x7f1ee819a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1eee3ae640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ee8107de0 0x7f1ee819a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:36.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1eee3ae640 1 -- 172.21.15.29:0/342313462 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee8108730 msgr2=0x7f1ee819f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:36.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1eee3ae640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee8108730 0x7f1ee819f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:36.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1eee3ae640 1 -- 172.21.15.29:0/342313462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee819fe60 con 0x7f1ee81024a0 2024-07-27T06:43:36.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1eee3ae640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee81024a0 0x7f1ee8199ef0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4005ab0 tx=0x7f1ee4003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:36.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1edb7fe640 1 -- 172.21.15.29:0/342313462 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee403f070 con 0x7f1ee81024a0 2024-07-27T06:43:36.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1ef0612640 1 -- 172.21.15.29:0/342313462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ee81a00f0 con 0x7f1ee81024a0 2024-07-27T06:43:36.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1edb7fe640 1 -- 172.21.15.29:0/342313462 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ee4031d60 con 0x7f1ee81024a0 2024-07-27T06:43:36.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.783+0000 7f1edb7fe640 1 -- 172.21.15.29:0/342313462 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4035030 con 0x7f1ee81024a0 2024-07-27T06:43:36.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.784+0000 7f1ef0612640 1 -- 172.21.15.29:0/342313462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ee81a68b0 con 0x7f1ee81024a0 2024-07-27T06:43:36.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.785+0000 7f1ef0612640 1 -- 172.21.15.29:0/342313462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ee8103720 con 0x7f1ee81024a0 2024-07-27T06:43:36.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.786+0000 7f1edb7fe640 1 -- 172.21.15.29:0/342313462 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7f1ee4042070 con 0x7f1ee81024a0 2024-07-27T06:43:36.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.787+0000 7f1edb7fe640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1ed40b3c20 0x7f1ed40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:36.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.787+0000 7f1eedbad640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1ed40b3c20 0x7f1ed40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:36.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.787+0000 7f1edb7fe640 1 -- 172.21.15.29:0/342313462 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7f1ee4104a90 con 0x7f1ee81024a0 2024-07-27T06:43:36.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.788+0000 7f1eedbad640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1ed40b3c20 0x7f1ed40b6130 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8079600 tx=0x7f1edc00b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:36.790+0000 7f1edb7fe640 1 -- 172.21.15.29:0/342313462 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1ee40cba50 con 0x7f1ee81024a0 2024-07-27T06:43:37.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.015+0000 7f1ef0612640 1 -- 172.21.15.29:0/342313462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f1ee810b4c0 con 0x7f1ee81024a0 2024-07-27T06:43:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:36 smithi047 ceph-mon[36835]: pgmap v291: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-07-27T06:43:37.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:36 smithi029 ceph-mon[32610]: pgmap v291: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-07-27T06:43:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:36 smithi077 ceph-mon[37153]: pgmap v291: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-07-27T06:43:37.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.880+0000 7f1edb7fe640 1 -- 172.21.15.29:0/342313462 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v28) ==== 115+0+0 (secure 0 0 0) 0x7f1ee40cffa0 con 0x7f1ee81024a0 2024-07-27T06:43:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.883+0000 7f1ef0612640 1 -- 172.21.15.29:0/342313462 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1ed40b3c20 msgr2=0x7f1ed40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.883+0000 7f1ef0612640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1ed40b3c20 0x7f1ed40b6130 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8079600 tx=0x7f1edc00b390 comp rx=0 tx=0).stop 2024-07-27T06:43:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.883+0000 7f1ef0612640 1 -- 172.21.15.29:0/342313462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee81024a0 msgr2=0x7f1ee8199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.883+0000 7f1ef0612640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee81024a0 0x7f1ee8199ef0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4005ab0 tx=0x7f1ee4003cd0 comp rx=0 tx=0).stop 2024-07-27T06:43:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.883+0000 7f1ef0612640 1 -- 172.21.15.29:0/342313462 shutdown_connections 2024-07-27T06:43:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.883+0000 7f1ef0612640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee8108730 0x7f1ee819f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.883+0000 7f1ef0612640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1ed40b3c20 0x7f1ed40b6130 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:37.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.883+0000 7f1ef0612640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ee8107de0 0x7f1ee819a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:37.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.883+0000 7f1ef0612640 1 --2- 172.21.15.29:0/342313462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ee81024a0 0x7f1ee8199ef0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:37.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.883+0000 7f1ef0612640 1 -- 172.21.15.29:0/342313462 >> 172.21.15.29:0/342313462 conn(0x7f1ee8079140 msgr2=0x7f1ee81014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:37.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.883+0000 7f1ef0612640 1 -- 172.21.15.29:0/342313462 shutdown_connections 2024-07-27T06:43:37.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:37.883+0000 7f1ef0612640 1 -- 172.21.15.29:0/342313462 wait complete. 2024-07-27T06:43:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:37 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/342313462' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-27T06:43:38.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:37 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/342313462' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-27T06:43:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:37 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/342313462' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-27T06:43:38.663 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-07-27T06:43:38.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-07-27T06:43:38.940 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:38 smithi047 ceph-mon[36835]: pgmap v292: 97 pgs: 97 active+clean; 584 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-07-27T06:43:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:38 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/342313462' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-27T06:43:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:38 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:43:39.201 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:38 smithi029 ceph-mon[32610]: pgmap v292: 97 pgs: 97 active+clean; 584 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-07-27T06:43:39.201 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:38 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/342313462' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-27T06:43:39.201 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:38 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:43:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:38 smithi077 ceph-mon[37153]: pgmap v292: 97 pgs: 97 active+clean; 584 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-07-27T06:43:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:38 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/342313462' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-27T06:43:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:38 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:43:40.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.777+0000 7fdb6fcee640 1 -- 172.21.15.29:0/3679519020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb68102490 msgr2=0x7fdb681028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:40.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.777+0000 7fdb6fcee640 1 --2- 172.21.15.29:0/3679519020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb68102490 0x7fdb681028a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009900 tx=0x7fdb6402a3f0 comp rx=0 tx=0).stop 2024-07-27T06:43:40.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.777+0000 7fdb6fcee640 1 -- 172.21.15.29:0/3679519020 shutdown_connections 2024-07-27T06:43:40.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.777+0000 7fdb6fcee640 1 --2- 172.21.15.29:0/3679519020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb68108720 0x7fdb6810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:40.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.777+0000 7fdb6fcee640 1 --2- 172.21.15.29:0/3679519020 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb68107dd0 0x7fdb681081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:40.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.777+0000 7fdb6fcee640 1 --2- 172.21.15.29:0/3679519020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb68102490 0x7fdb681028a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:40.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.777+0000 7fdb6fcee640 1 -- 172.21.15.29:0/3679519020 >> 172.21.15.29:0/3679519020 conn(0x7fdb68079160 msgr2=0x7fdb68079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:40.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.777+0000 7fdb6fcee640 1 -- 172.21.15.29:0/3679519020 shutdown_connections 2024-07-27T06:43:40.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.777+0000 7fdb6fcee640 1 -- 172.21.15.29:0/3679519020 wait complete. 2024-07-27T06:43:40.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.778+0000 7fdb6fcee640 1 Processor -- start 2024-07-27T06:43:40.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.778+0000 7fdb6fcee640 1 -- start start 2024-07-27T06:43:40.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.778+0000 7fdb6fcee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb68102490 0x7fdb68199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:40.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.778+0000 7fdb6fcee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb68107dd0 0x7fdb6819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:40.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.778+0000 7fdb6fcee640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb68108720 0x7fdb6819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:40.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.778+0000 7fdb6fcee640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdb681134b0 con 0x7fdb68107dd0 2024-07-27T06:43:40.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.778+0000 7fdb6fcee640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fdb68113330 con 0x7fdb68108720 2024-07-27T06:43:40.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.778+0000 7fdb6fcee640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdb68113630 con 0x7fdb68102490 2024-07-27T06:43:40.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6d289640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb68107dd0 0x7fdb6819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:40.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6d289640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb68107dd0 0x7fdb6819a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57062/0 (socket says 172.21.15.29:57062) 2024-07-27T06:43:40.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6d289640 1 -- 172.21.15.29:0/1256300125 learned_addr learned my addr 172.21.15.29:0/1256300125 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:40.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6da8a640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb68102490 0x7fdb68199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:40.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6e28b640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb68108720 0x7fdb6819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:40.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6d289640 1 -- 172.21.15.29:0/1256300125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb68102490 msgr2=0x7fdb68199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:40.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6d289640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb68102490 0x7fdb68199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:40.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6d289640 1 -- 172.21.15.29:0/1256300125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb68108720 msgr2=0x7fdb6819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:40.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6d289640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb68108720 0x7fdb6819f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:40.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6d289640 1 -- 172.21.15.29:0/1256300125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6819fcf0 con 0x7fdb68107dd0 2024-07-27T06:43:40.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6e28b640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb68108720 0x7fdb6819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:43:40.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6d289640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb68107dd0 0x7fdb6819a3e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdb5800c970 tx=0x7fdb5800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:40.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb5effd640 1 -- 172.21.15.29:0/1256300125 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb58014070 con 0x7fdb68107dd0 2024-07-27T06:43:40.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb5effd640 1 -- 172.21.15.29:0/1256300125 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb58004440 con 0x7fdb68107dd0 2024-07-27T06:43:40.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6fcee640 1 -- 172.21.15.29:0/1256300125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb6819ffe0 con 0x7fdb68107dd0 2024-07-27T06:43:40.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb5effd640 1 -- 172.21.15.29:0/1256300125 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb58003e50 con 0x7fdb68107dd0 2024-07-27T06:43:40.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.779+0000 7fdb6fcee640 1 -- 172.21.15.29:0/1256300125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb681a6800 con 0x7fdb68107dd0 2024-07-27T06:43:40.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.781+0000 7fdb6fcee640 1 -- 172.21.15.29:0/1256300125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb681028a0 con 0x7fdb68107dd0 2024-07-27T06:43:40.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.782+0000 7fdb5effd640 1 -- 172.21.15.29:0/1256300125 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 20) ==== 151076+0+0 (secure 0 0 0) 0x7fdb58017070 con 0x7fdb68107dd0 2024-07-27T06:43:40.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.783+0000 7fdb5effd640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdb540b3df0 0x7fdb540b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:40.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.783+0000 7fdb6da8a640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdb540b3df0 0x7fdb540b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:40.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.783+0000 7fdb5effd640 1 -- 172.21.15.29:0/1256300125 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7fdb580e3700 con 0x7fdb68107dd0 2024-07-27T06:43:40.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.784+0000 7fdb6da8a640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdb540b3df0 0x7fdb540b6300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdb64031440 tx=0x7fdb64005e80 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:40.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.787+0000 7fdb5effd640 1 -- 172.21.15.29:0/1256300125 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdb5801e030 con 0x7fdb68107dd0 2024-07-27T06:43:41.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:40.999+0000 7fdb6fcee640 1 -- 172.21.15.29:0/1256300125 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fdb68063230 con 0x7fdb540b3df0 2024-07-27T06:43:41.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.032+0000 7fdb5effd640 1 -- 172.21.15.29:0/1256300125 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdb68063230 con 0x7fdb540b3df0 2024-07-27T06:43:41.035 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:40 smithi029 ceph-mon[32610]: pgmap v293: 97 pgs: 97 active+clean; 584 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-07-27T06:43:41.035 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:43:41 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:43:41.004+0000 7fdc5634d640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-27T06:43:41.035 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:43:41 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:43:41.004+0000 7fdc5634d640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-27T06:43:41.035 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:43:41 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:43:41.004+0000 7fdc5634d640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-27T06:43:41.035 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:43:41 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:43:41.004+0000 7fdc5634d640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-27T06:43:41.035 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:43:41 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:43:41.004+0000 7fdc5634d640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-27T06:43:41.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.035+0000 7fdb6fcee640 1 -- 172.21.15.29:0/1256300125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdb540b3df0 msgr2=0x7fdb540b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:41.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.035+0000 7fdb6fcee640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdb540b3df0 0x7fdb540b6300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdb64031440 tx=0x7fdb64005e80 comp rx=0 tx=0).stop 2024-07-27T06:43:41.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.035+0000 7fdb6fcee640 1 -- 172.21.15.29:0/1256300125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb68107dd0 msgr2=0x7fdb6819a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:41.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.035+0000 7fdb6fcee640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb68107dd0 0x7fdb6819a3e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdb5800c970 tx=0x7fdb5800ce30 comp rx=0 tx=0).stop 2024-07-27T06:43:41.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.035+0000 7fdb6fcee640 1 -- 172.21.15.29:0/1256300125 shutdown_connections 2024-07-27T06:43:41.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.035+0000 7fdb6fcee640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb68108720 0x7fdb6819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:41.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.035+0000 7fdb6fcee640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb68107dd0 0x7fdb6819a3e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:41.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.035+0000 7fdb6fcee640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdb540b3df0 0x7fdb540b6300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:41.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.035+0000 7fdb6fcee640 1 --2- 172.21.15.29:0/1256300125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb68102490 0x7fdb68199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:41.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.035+0000 7fdb6fcee640 1 -- 172.21.15.29:0/1256300125 >> 172.21.15.29:0/1256300125 conn(0x7fdb68079160 msgr2=0x7fdb68100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:41.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.035+0000 7fdb6fcee640 1 -- 172.21.15.29:0/1256300125 shutdown_connections 2024-07-27T06:43:41.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:41.035+0000 7fdb6fcee640 1 -- 172.21.15.29:0/1256300125 wait complete. 2024-07-27T06:43:41.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:40 smithi077 ceph-mon[37153]: pgmap v293: 97 pgs: 97 active+clean; 584 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-07-27T06:43:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:40 smithi047 ceph-mon[36835]: pgmap v293: 97 pgs: 97 active+clean; 584 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-07-27T06:43:41.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-07-27T06:43:42.189 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:42.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:41 smithi047 ceph-mon[36835]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:42.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:41 smithi029 ceph-mon[32610]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:41 smithi077 ceph-mon[37153]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:43.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:42 smithi029 ceph-mon[32610]: pgmap v294: 97 pgs: 97 active+clean; 585 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.1 KiB/s wr, 20 op/s 2024-07-27T06:43:43.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:42 smithi029 ceph-mon[32610]: mgrmap e21: x(active, since 7m), standbys: y, z 2024-07-27T06:43:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:42 smithi047 ceph-mon[36835]: pgmap v294: 97 pgs: 97 active+clean; 585 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.1 KiB/s wr, 20 op/s 2024-07-27T06:43:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:42 smithi047 ceph-mon[36835]: mgrmap e21: x(active, since 7m), standbys: y, z 2024-07-27T06:43:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:42 smithi077 ceph-mon[37153]: pgmap v294: 97 pgs: 97 active+clean; 585 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.1 KiB/s wr, 20 op/s 2024-07-27T06:43:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:42 smithi077 ceph-mon[37153]: mgrmap e21: x(active, since 7m), standbys: y, z 2024-07-27T06:43:44.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.143+0000 7f359773a640 1 -- 172.21.15.29:0/99505116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f359010d840 msgr2=0x7f359010fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:44.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.143+0000 7f359773a640 1 --2- 172.21.15.29:0/99505116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f359010d840 0x7f359010fd30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3588009900 tx=0x7f358802a410 comp rx=0 tx=0).stop 2024-07-27T06:43:44.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.143+0000 7f359773a640 1 -- 172.21.15.29:0/99505116 shutdown_connections 2024-07-27T06:43:44.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.143+0000 7f359773a640 1 --2- 172.21.15.29:0/99505116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f359010d840 0x7f359010fd30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:44.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.143+0000 7f359773a640 1 --2- 172.21.15.29:0/99505116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3590104d30 0x7f359010d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:44.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.143+0000 7f359773a640 1 --2- 172.21.15.29:0/99505116 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35901043e0 0x7f35901047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.143+0000 7f359773a640 1 -- 172.21.15.29:0/99505116 >> 172.21.15.29:0/99505116 conn(0x7f3590079160 msgr2=0x7f3590079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.143+0000 7f359773a640 1 -- 172.21.15.29:0/99505116 shutdown_connections 2024-07-27T06:43:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.143+0000 7f359773a640 1 -- 172.21.15.29:0/99505116 wait complete. 2024-07-27T06:43:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.144+0000 7f359773a640 1 Processor -- start 2024-07-27T06:43:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.144+0000 7f359773a640 1 -- start start 2024-07-27T06:43:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.144+0000 7f359773a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35901043e0 0x7f3590199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.144+0000 7f359773a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3590104d30 0x7f359019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.144+0000 7f359773a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f359010d840 0x7f359019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.144+0000 7f359773a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f35901134a0 con 0x7f359010d840 2024-07-27T06:43:44.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.144+0000 7f359773a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3590113320 con 0x7f35901043e0 2024-07-27T06:43:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.144+0000 7f359773a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3590113620 con 0x7f3590104d30 2024-07-27T06:43:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.144+0000 7f3595cd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f359010d840 0x7f359019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f3595cd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f359010d840 0x7f359019f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57096/0 (socket says 172.21.15.29:57096) 2024-07-27T06:43:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f3595cd7640 1 -- 172.21.15.29:0/3702312309 learned_addr learned my addr 172.21.15.29:0/3702312309 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f35954d6640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35901043e0 0x7f3590199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f3594cd5640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3590104d30 0x7f359019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f3595cd7640 1 -- 172.21.15.29:0/3702312309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3590104d30 msgr2=0x7f359019a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f3595cd7640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3590104d30 0x7f359019a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f3595cd7640 1 -- 172.21.15.29:0/3702312309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35901043e0 msgr2=0x7f3590199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f3595cd7640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35901043e0 0x7f3590199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f3595cd7640 1 -- 172.21.15.29:0/3702312309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359019fe80 con 0x7f359010d840 2024-07-27T06:43:44.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f3594cd5640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3590104d30 0x7f359019a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:43:44.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f3595cd7640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f359010d840 0x7f359019f750 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3588004770 tx=0x7f3588004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:44.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f358e7fc640 1 -- 172.21.15.29:0/3702312309 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358803f030 con 0x7f359010d840 2024-07-27T06:43:44.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f358e7fc640 1 -- 172.21.15.29:0/3702312309 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3588003aa0 con 0x7f359010d840 2024-07-27T06:43:44.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f359773a640 1 -- 172.21.15.29:0/3702312309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35901a0110 con 0x7f359010d840 2024-07-27T06:43:44.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f358e7fc640 1 -- 172.21.15.29:0/3702312309 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588035070 con 0x7f359010d840 2024-07-27T06:43:44.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.145+0000 7f359773a640 1 -- 172.21.15.29:0/3702312309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35901a68d0 con 0x7f359010d840 2024-07-27T06:43:44.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.147+0000 7f359773a640 1 -- 172.21.15.29:0/3702312309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35901047f0 con 0x7f359010d840 2024-07-27T06:43:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.149+0000 7f358e7fc640 1 -- 172.21.15.29:0/3702312309 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f3588042030 con 0x7f359010d840 2024-07-27T06:43:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.149+0000 7f358e7fc640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f35780b3cb0 0x7f35780b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.150+0000 7f35954d6640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f35780b3cb0 0x7f35780b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.150+0000 7f358e7fc640 1 -- 172.21.15.29:0/3702312309 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7f3588104b80 con 0x7f359010d840 2024-07-27T06:43:44.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.150+0000 7f35954d6640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f35780b3cb0 0x7f35780b61c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f359006b690 tx=0x7f3580009670 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.153+0000 7f358e7fc640 1 -- 172.21.15.29:0/3702312309 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3588049030 con 0x7f359010d840 2024-07-27T06:43:44.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.365+0000 7f359773a640 1 -- 172.21.15.29:0/3702312309 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3590063230 con 0x7f35780b3cb0 2024-07-27T06:43:44.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.396+0000 7f358e7fc640 1 -- 172.21.15.29:0/3702312309 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3590063230 con 0x7f35780b3cb0 2024-07-27T06:43:44.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.399+0000 7f359773a640 1 -- 172.21.15.29:0/3702312309 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f35780b3cb0 msgr2=0x7f35780b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:44.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.399+0000 7f359773a640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f35780b3cb0 0x7f35780b61c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f359006b690 tx=0x7f3580009670 comp rx=0 tx=0).stop 2024-07-27T06:43:44.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.399+0000 7f359773a640 1 -- 172.21.15.29:0/3702312309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f359010d840 msgr2=0x7f359019f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:44.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.399+0000 7f359773a640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f359010d840 0x7f359019f750 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3588004770 tx=0x7f3588004270 comp rx=0 tx=0).stop 2024-07-27T06:43:44.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.399+0000 7f359773a640 1 -- 172.21.15.29:0/3702312309 shutdown_connections 2024-07-27T06:43:44.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.399+0000 7f359773a640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f359010d840 0x7f359019f750 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:44.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.399+0000 7f359773a640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f35780b3cb0 0x7f35780b61c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:44.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.399+0000 7f359773a640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3590104d30 0x7f359019a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:44.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.399+0000 7f359773a640 1 --2- 172.21.15.29:0/3702312309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35901043e0 0x7f3590199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:44.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.399+0000 7f359773a640 1 -- 172.21.15.29:0/3702312309 >> 172.21.15.29:0/3702312309 conn(0x7f3590079160 msgr2=0x7f3590100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:44.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.399+0000 7f359773a640 1 -- 172.21.15.29:0/3702312309 shutdown_connections 2024-07-27T06:43:44.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:44.399+0000 7f359773a640 1 -- 172.21.15.29:0/3702312309 wait complete. 2024-07-27T06:43:45.076 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:44 smithi029 ceph-mon[32610]: pgmap v295: 97 pgs: 97 active+clean; 592 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 4.2 KiB/s wr, 20 op/s 2024-07-27T06:43:45.076 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:44 smithi029 ceph-mon[32610]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:45.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-07-27T06:43:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:44 smithi047 ceph-mon[36835]: pgmap v295: 97 pgs: 97 active+clean; 592 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 4.2 KiB/s wr, 20 op/s 2024-07-27T06:43:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:44 smithi047 ceph-mon[36835]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:44 smithi077 ceph-mon[37153]: pgmap v295: 97 pgs: 97 active+clean; 592 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 4.2 KiB/s wr, 20 op/s 2024-07-27T06:43:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:44 smithi077 ceph-mon[37153]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:45.374 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:46 smithi029 ceph-mon[32610]: pgmap v296: 97 pgs: 97 active+clean; 592 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 4.1 KiB/s wr, 17 op/s 2024-07-27T06:43:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:46 smithi047 ceph-mon[36835]: pgmap v296: 97 pgs: 97 active+clean; 592 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 4.1 KiB/s wr, 17 op/s 2024-07-27T06:43:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.270+0000 7f0efc42c640 1 -- 172.21.15.29:0/2714915030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4102510 msgr2=0x7f0ef4102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.270+0000 7f0efc42c640 1 --2- 172.21.15.29:0/2714915030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4102510 0x7f0ef4102920 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009900 tx=0x7f0ef002a410 comp rx=0 tx=0).stop 2024-07-27T06:43:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.270+0000 7f0efc42c640 1 -- 172.21.15.29:0/2714915030 shutdown_connections 2024-07-27T06:43:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.270+0000 7f0efc42c640 1 --2- 172.21.15.29:0/2714915030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ef4108750 0x7f0ef410ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.270+0000 7f0efc42c640 1 --2- 172.21.15.29:0/2714915030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef4107e00 0x7f0ef4108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.270+0000 7f0efc42c640 1 --2- 172.21.15.29:0/2714915030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4102510 0x7f0ef4102920 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.270+0000 7f0efc42c640 1 -- 172.21.15.29:0/2714915030 >> 172.21.15.29:0/2714915030 conn(0x7f0ef4079190 msgr2=0x7f0ef40795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.270+0000 7f0efc42c640 1 -- 172.21.15.29:0/2714915030 shutdown_connections 2024-07-27T06:43:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.270+0000 7f0efc42c640 1 -- 172.21.15.29:0/2714915030 wait complete. 2024-07-27T06:43:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0efc42c640 1 Processor -- start 2024-07-27T06:43:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0efc42c640 1 -- start start 2024-07-27T06:43:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0efc42c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ef4102510 0x7f0ef410fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0efc42c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4107e00 0x7f0ef410ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0efc42c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef4108750 0x7f0ef4110490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0efc42c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4113520 con 0x7f0ef4107e00 2024-07-27T06:43:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0efc42c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41133a0 con 0x7f0ef4102510 2024-07-27T06:43:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0efc42c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef41136a0 con 0x7f0ef4108750 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0ef99c7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4107e00 0x7f0ef410ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0efa1c8640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ef4102510 0x7f0ef410fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0ef99c7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4107e00 0x7f0ef410ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57108/0 (socket says 172.21.15.29:57108) 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0ef99c7640 1 -- 172.21.15.29:0/948449839 learned_addr learned my addr 172.21.15.29:0/948449839 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.271+0000 7f0efa9c9640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef4108750 0x7f0ef4110490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.272+0000 7f0ef99c7640 1 -- 172.21.15.29:0/948449839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef4108750 msgr2=0x7f0ef4110490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.272+0000 7f0ef99c7640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef4108750 0x7f0ef4110490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.272+0000 7f0ef99c7640 1 -- 172.21.15.29:0/948449839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ef4102510 msgr2=0x7f0ef410fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.272+0000 7f0ef99c7640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ef4102510 0x7f0ef410fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.272+0000 7f0ef99c7640 1 -- 172.21.15.29:0/948449839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef41a81b0 con 0x7f0ef4107e00 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.272+0000 7f0ef99c7640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4107e00 0x7f0ef410ff50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400b450 tx=0x7f0ee400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.272+0000 7f0eeb7fe640 1 -- 172.21.15.29:0/948449839 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4015020 con 0x7f0ef4107e00 2024-07-27T06:43:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.272+0000 7f0efc42c640 1 -- 172.21.15.29:0/948449839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef41a84a0 con 0x7f0ef4107e00 2024-07-27T06:43:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.272+0000 7f0eeb7fe640 1 -- 172.21.15.29:0/948449839 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ee4011d40 con 0x7f0ef4107e00 2024-07-27T06:43:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.272+0000 7f0eeb7fe640 1 -- 172.21.15.29:0/948449839 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee400f340 con 0x7f0ef4107e00 2024-07-27T06:43:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.272+0000 7f0efc42c640 1 -- 172.21.15.29:0/948449839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef41a8ab0 con 0x7f0ef4107e00 2024-07-27T06:43:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.273+0000 7f0efc42c640 1 -- 172.21.15.29:0/948449839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef4103740 con 0x7f0ef4107e00 2024-07-27T06:43:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.275+0000 7f0eeb7fe640 1 -- 172.21.15.29:0/948449839 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f0ee4018020 con 0x7f0ef4107e00 2024-07-27T06:43:47.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.276+0000 7f0eeb7fe640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0ee00b3cb0 0x7f0ee00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:47.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.276+0000 7f0eeb7fe640 1 -- 172.21.15.29:0/948449839 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7f0ee40e3d30 con 0x7f0ef4107e00 2024-07-27T06:43:47.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.276+0000 7f0efa1c8640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0ee00b3cb0 0x7f0ee00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:47.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.276+0000 7f0efa1c8640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0ee00b3cb0 0x7f0ee00b61c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0000c00 tx=0x7f0ef0031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.279+0000 7f0eeb7fe640 1 -- 172.21.15.29:0/948449839 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0ee401f020 con 0x7f0ef4107e00 2024-07-27T06:43:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:46 smithi077 ceph-mon[37153]: pgmap v296: 97 pgs: 97 active+clean; 592 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 4.1 KiB/s wr, 17 op/s 2024-07-27T06:43:47.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.490+0000 7f0efc42c640 1 -- 172.21.15.29:0/948449839 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f0ef4063230 con 0x7f0ee00b3cb0 2024-07-27T06:43:47.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.495+0000 7f0eeb7fe640 1 -- 172.21.15.29:0/948449839 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0ef4063230 con 0x7f0ee00b3cb0 2024-07-27T06:43:47.497 INFO:teuthology.orchestra.run.smithi029.stdout:/volumes/qa/sv_0/41f7407c-9cfe-4bd0-b635-544fa15c38ed 2024-07-27T06:43:47.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.497+0000 7f0efc42c640 1 -- 172.21.15.29:0/948449839 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0ee00b3cb0 msgr2=0x7f0ee00b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:47.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.497+0000 7f0efc42c640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0ee00b3cb0 0x7f0ee00b61c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0000c00 tx=0x7f0ef0031400 comp rx=0 tx=0).stop 2024-07-27T06:43:47.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.497+0000 7f0efc42c640 1 -- 172.21.15.29:0/948449839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4107e00 msgr2=0x7f0ef410ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.497+0000 7f0efc42c640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4107e00 0x7f0ef410ff50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400b450 tx=0x7f0ee400b910 comp rx=0 tx=0).stop 2024-07-27T06:43:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.497+0000 7f0efc42c640 1 -- 172.21.15.29:0/948449839 shutdown_connections 2024-07-27T06:43:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.497+0000 7f0efc42c640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0ee00b3cb0 0x7f0ee00b61c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.497+0000 7f0efc42c640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef4108750 0x7f0ef4110490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.497+0000 7f0efc42c640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ef4107e00 0x7f0ef410ff50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.497+0000 7f0efc42c640 1 --2- 172.21.15.29:0/948449839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ef4102510 0x7f0ef410fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.497+0000 7f0efc42c640 1 -- 172.21.15.29:0/948449839 >> 172.21.15.29:0/948449839 conn(0x7f0ef4079190 msgr2=0x7f0ef4100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.498+0000 7f0efc42c640 1 -- 172.21.15.29:0/948449839 shutdown_connections 2024-07-27T06:43:47.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:47.498+0000 7f0efc42c640 1 -- 172.21.15.29:0/948449839 wait complete. 2024-07-27T06:43:48.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-07-27T06:43:48.464 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:48 smithi047 ceph-mon[36835]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:48 smithi047 ceph-mon[36835]: pgmap v297: 97 pgs: 97 active+clean; 611 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 4.3 KiB/s wr, 17 op/s 2024-07-27T06:43:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:48 smithi029 ceph-mon[32610]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:48 smithi029 ceph-mon[32610]: pgmap v297: 97 pgs: 97 active+clean; 611 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 4.3 KiB/s wr, 17 op/s 2024-07-27T06:43:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:48 smithi077 ceph-mon[37153]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:48 smithi077 ceph-mon[37153]: pgmap v297: 97 pgs: 97 active+clean; 611 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 4.3 KiB/s wr, 17 op/s 2024-07-27T06:43:50.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.150+0000 7fd781439640 1 -- 172.21.15.29:0/1605360832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1024d0 msgr2=0x7fd77c1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:50.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.150+0000 7fd781439640 1 --2- 172.21.15.29:0/1605360832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1024d0 0x7fd77c1028e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009900 tx=0x7fd76c02a410 comp rx=0 tx=0).stop 2024-07-27T06:43:50.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.150+0000 7fd781439640 1 -- 172.21.15.29:0/1605360832 shutdown_connections 2024-07-27T06:43:50.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.150+0000 7fd781439640 1 --2- 172.21.15.29:0/1605360832 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd77c108760 0x7fd77c10ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:50.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.150+0000 7fd781439640 1 --2- 172.21.15.29:0/1605360832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77c107e10 0x7fd77c108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.150+0000 7fd781439640 1 --2- 172.21.15.29:0/1605360832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1024d0 0x7fd77c1028e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.150+0000 7fd781439640 1 -- 172.21.15.29:0/1605360832 >> 172.21.15.29:0/1605360832 conn(0x7fd77c079180 msgr2=0x7fd77c079590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.150+0000 7fd781439640 1 -- 172.21.15.29:0/1605360832 shutdown_connections 2024-07-27T06:43:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.150+0000 7fd781439640 1 -- 172.21.15.29:0/1605360832 wait complete. 2024-07-27T06:43:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.151+0000 7fd781439640 1 Processor -- start 2024-07-27T06:43:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.151+0000 7fd781439640 1 -- start start 2024-07-27T06:43:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd781439640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1024d0 0x7fd77c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd781439640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd77c107e10 0x7fd77c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd781439640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77c108760 0x7fd77c19f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd781439640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c1124e0 con 0x7fd77c1024d0 2024-07-27T06:43:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd781439640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c112360 con 0x7fd77c107e10 2024-07-27T06:43:50.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd781439640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c112660 con 0x7fd77c108760 2024-07-27T06:43:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd77affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1024d0 0x7fd77c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd77affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1024d0 0x7fd77c19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51114/0 (socket says 172.21.15.29:51114) 2024-07-27T06:43:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd77affd640 1 -- 172.21.15.29:0/839640478 learned_addr learned my addr 172.21.15.29:0/839640478 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd77a7fc640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd77c107e10 0x7fd77c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd77b7fe640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77c108760 0x7fd77c19f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd77affd640 1 -- 172.21.15.29:0/839640478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77c108760 msgr2=0x7fd77c19f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd77affd640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77c108760 0x7fd77c19f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd77affd640 1 -- 172.21.15.29:0/839640478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd77c107e10 msgr2=0x7fd77c19a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd77affd640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd77c107e10 0x7fd77c19a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd77affd640 1 -- 172.21.15.29:0/839640478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77c1a00d0 con 0x7fd77c1024d0 2024-07-27T06:43:50.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd77a7fc640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd77c107e10 0x7fd77c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:43:50.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.152+0000 7fd77affd640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1024d0 0x7fd77c19a160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd76c006010 tx=0x7fd76c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:50.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.153+0000 7fd763fff640 1 -- 172.21.15.29:0/839640478 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c03f030 con 0x7fd77c1024d0 2024-07-27T06:43:50.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.153+0000 7fd781439640 1 -- 172.21.15.29:0/839640478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77c1a0360 con 0x7fd77c1024d0 2024-07-27T06:43:50.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.153+0000 7fd763fff640 1 -- 172.21.15.29:0/839640478 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd76c003800 con 0x7fd77c1024d0 2024-07-27T06:43:50.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.153+0000 7fd763fff640 1 -- 172.21.15.29:0/839640478 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c035070 con 0x7fd77c1024d0 2024-07-27T06:43:50.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.153+0000 7fd781439640 1 -- 172.21.15.29:0/839640478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77c1a6b20 con 0x7fd77c1024d0 2024-07-27T06:43:50.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.154+0000 7fd781439640 1 -- 172.21.15.29:0/839640478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd77c102960 con 0x7fd77c1024d0 2024-07-27T06:43:50.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.155+0000 7fd763fff640 1 -- 172.21.15.29:0/839640478 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fd76c0039b0 con 0x7fd77c1024d0 2024-07-27T06:43:50.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.156+0000 7fd763fff640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd7640b3cb0 0x7fd7640b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:50.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.156+0000 7fd77a7fc640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd7640b3cb0 0x7fd7640b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:50.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.156+0000 7fd763fff640 1 -- 172.21.15.29:0/839640478 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7fd76c104460 con 0x7fd77c1024d0 2024-07-27T06:43:50.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.157+0000 7fd77a7fc640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd7640b3cb0 0x7fd7640b61c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd77c079a70 tx=0x7fd770008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:50.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.161+0000 7fd763fff640 1 -- 172.21.15.29:0/839640478 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd76c0cb970 con 0x7fd77c1024d0 2024-07-27T06:43:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.372+0000 7fd781439640 1 -- 172.21.15.29:0/839640478 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fd77c063230 con 0x7fd7640b3cb0 2024-07-27T06:43:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.398+0000 7fd763fff640 1 -- 172.21.15.29:0/839640478 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd77c063230 con 0x7fd7640b3cb0 2024-07-27T06:43:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.400+0000 7fd781439640 1 -- 172.21.15.29:0/839640478 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd7640b3cb0 msgr2=0x7fd7640b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.400+0000 7fd781439640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd7640b3cb0 0x7fd7640b61c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd77c079a70 tx=0x7fd770008040 comp rx=0 tx=0).stop 2024-07-27T06:43:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.400+0000 7fd781439640 1 -- 172.21.15.29:0/839640478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1024d0 msgr2=0x7fd77c19a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.400+0000 7fd781439640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1024d0 0x7fd77c19a160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd76c006010 tx=0x7fd76c0042f0 comp rx=0 tx=0).stop 2024-07-27T06:43:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.400+0000 7fd781439640 1 -- 172.21.15.29:0/839640478 shutdown_connections 2024-07-27T06:43:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.400+0000 7fd781439640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd7640b3cb0 0x7fd7640b61c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.400+0000 7fd781439640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77c108760 0x7fd77c19f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.400+0000 7fd781439640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd77c107e10 0x7fd77c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.400+0000 7fd781439640 1 --2- 172.21.15.29:0/839640478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd77c1024d0 0x7fd77c19a160 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.400+0000 7fd781439640 1 -- 172.21.15.29:0/839640478 >> 172.21.15.29:0/839640478 conn(0x7fd77c079180 msgr2=0x7fd77c101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:50.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.401+0000 7fd781439640 1 -- 172.21.15.29:0/839640478 shutdown_connections 2024-07-27T06:43:50.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:50.401+0000 7fd781439640 1 -- 172.21.15.29:0/839640478 wait complete. 2024-07-27T06:43:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:50 smithi029 ceph-mon[32610]: pgmap v298: 97 pgs: 97 active+clean; 611 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.1 KiB/s wr, 10 op/s 2024-07-27T06:43:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:50 smithi029 ceph-mon[32610]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:50 smithi047 ceph-mon[36835]: pgmap v298: 97 pgs: 97 active+clean; 611 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.1 KiB/s wr, 10 op/s 2024-07-27T06:43:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:50 smithi047 ceph-mon[36835]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:51.199 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-07-27T06:43:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:50 smithi077 ceph-mon[37153]: pgmap v298: 97 pgs: 97 active+clean; 611 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.1 KiB/s wr, 10 op/s 2024-07-27T06:43:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:50 smithi077 ceph-mon[37153]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:51.467 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:53.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:52 smithi029 ceph-mon[32610]: pgmap v299: 97 pgs: 97 active+clean; 611 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.1 KiB/s wr, 12 op/s 2024-07-27T06:43:53.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.114+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/1590698483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0107e00 msgr2=0x7fe4e0108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:53.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.114+0000 7fe4e4dc7640 1 --2- 172.21.15.29:0/1590698483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0107e00 0x7fe4e0108210 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4009900 tx=0x7fe4d402a410 comp rx=0 tx=0).stop 2024-07-27T06:43:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.114+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/1590698483 shutdown_connections 2024-07-27T06:43:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.114+0000 7fe4e4dc7640 1 --2- 172.21.15.29:0/1590698483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e0108750 0x7fe4e010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.114+0000 7fe4e4dc7640 1 --2- 172.21.15.29:0/1590698483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0107e00 0x7fe4e0108210 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.114+0000 7fe4e4dc7640 1 --2- 172.21.15.29:0/1590698483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4e0102510 0x7fe4e0102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.114+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/1590698483 >> 172.21.15.29:0/1590698483 conn(0x7fe4e0079190 msgr2=0x7fe4e00795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.114+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/1590698483 shutdown_connections 2024-07-27T06:43:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.114+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/1590698483 wait complete. 2024-07-27T06:43:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.115+0000 7fe4e4dc7640 1 Processor -- start 2024-07-27T06:43:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.115+0000 7fe4e4dc7640 1 -- start start 2024-07-27T06:43:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4e4dc7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e0102510 0x7fe4e0199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4e4dc7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0107e00 0x7fe4e019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:53.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4e4dc7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4e0108750 0x7fe4e019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4ddd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0107e00 0x7fe4e019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4ddd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0107e00 0x7fe4e019a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51132/0 (socket says 172.21.15.29:51132) 2024-07-27T06:43:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4ddd9b640 1 -- 172.21.15.29:0/491201778 learned_addr learned my addr 172.21.15.29:0/491201778 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4ded9d640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4e0108750 0x7fe4e019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4de59c640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e0102510 0x7fe4e0199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4e4dc7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e0112380 con 0x7fe4e0107e00 2024-07-27T06:43:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/491201778 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e0112200 con 0x7fe4e0108750 2024-07-27T06:43:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/491201778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e0112500 con 0x7fe4e0102510 2024-07-27T06:43:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4ddd9b640 1 -- 172.21.15.29:0/491201778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e0102510 msgr2=0x7fe4e0199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4ddd9b640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e0102510 0x7fe4e0199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4ddd9b640 1 -- 172.21.15.29:0/491201778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4e0108750 msgr2=0x7fe4e019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:53.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4ddd9b640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4e0108750 0x7fe4e019f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.116+0000 7fe4ddd9b640 1 -- 172.21.15.29:0/491201778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e019fe50 con 0x7fe4e0107e00 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.117+0000 7fe4ddd9b640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0107e00 0x7fe4e019a480 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4032fd0 tx=0x7fe4d4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.117+0000 7fe4cf7fe640 1 -- 172.21.15.29:0/491201778 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d403f030 con 0x7fe4e0107e00 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.117+0000 7fe4cf7fe640 1 -- 172.21.15.29:0/491201778 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4d4004490 con 0x7fe4e0107e00 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.117+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/491201778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e01a00e0 con 0x7fe4e0107e00 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.117+0000 7fe4cf7fe640 1 -- 172.21.15.29:0/491201778 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4035070 con 0x7fe4e0107e00 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.117+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/491201778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e01a68a0 con 0x7fe4e0107e00 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.118+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/491201778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4e0103740 con 0x7fe4e0107e00 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.119+0000 7fe4cf7fe640 1 -- 172.21.15.29:0/491201778 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fe4d4042030 con 0x7fe4e0107e00 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.120+0000 7fe4cf7fe640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe4c80b3cb0 0x7fe4c80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.120+0000 7fe4de59c640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe4c80b3cb0 0x7fe4c80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.120+0000 7fe4cf7fe640 1 -- 172.21.15.29:0/491201778 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7fe4d4104650 con 0x7fe4e0107e00 2024-07-27T06:43:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.121+0000 7fe4de59c640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe4c80b3cb0 0x7fe4c80b61c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00058c0 tx=0x7fe4d000a2d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:53.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.124+0000 7fe4cf7fe640 1 -- 172.21.15.29:0/491201778 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe4d40cb510 con 0x7fe4e0107e00 2024-07-27T06:43:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:52 smithi047 ceph-mon[36835]: pgmap v299: 97 pgs: 97 active+clean; 611 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.1 KiB/s wr, 12 op/s 2024-07-27T06:43:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.335+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/491201778 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fe4e0063230 con 0x7fe4c80b3cb0 2024-07-27T06:43:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.339+0000 7fe4cf7fe640 1 -- 172.21.15.29:0/491201778 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fe4e0063230 con 0x7fe4c80b3cb0 2024-07-27T06:43:53.340 INFO:teuthology.orchestra.run.smithi029.stdout:/volumes/qa/sv_1/617f3b02-55d2-4590-8cf9-6ef1a11d6f17 2024-07-27T06:43:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.341+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/491201778 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe4c80b3cb0 msgr2=0x7fe4c80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.341+0000 7fe4e4dc7640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe4c80b3cb0 0x7fe4c80b61c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe4d00058c0 tx=0x7fe4d000a2d0 comp rx=0 tx=0).stop 2024-07-27T06:43:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.341+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/491201778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0107e00 msgr2=0x7fe4e019a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.341+0000 7fe4e4dc7640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0107e00 0x7fe4e019a480 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4032fd0 tx=0x7fe4d4004270 comp rx=0 tx=0).stop 2024-07-27T06:43:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.342+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/491201778 shutdown_connections 2024-07-27T06:43:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.342+0000 7fe4e4dc7640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe4e0108750 0x7fe4e019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.342+0000 7fe4e4dc7640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4e0107e00 0x7fe4e019a480 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.342+0000 7fe4e4dc7640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe4c80b3cb0 0x7fe4c80b61c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.342+0000 7fe4e4dc7640 1 --2- 172.21.15.29:0/491201778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e0102510 0x7fe4e0199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.342+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/491201778 >> 172.21.15.29:0/491201778 conn(0x7fe4e0079190 msgr2=0x7fe4e006b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.342+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/491201778 shutdown_connections 2024-07-27T06:43:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:53.342+0000 7fe4e4dc7640 1 -- 172.21.15.29:0/491201778 wait complete. 2024-07-27T06:43:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:52 smithi077 ceph-mon[37153]: pgmap v299: 97 pgs: 97 active+clean; 611 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.1 KiB/s wr, 12 op/s 2024-07-27T06:43:53.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:53 smithi029 ceph-mon[32610]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:54.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:43:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:53 smithi047 ceph-mon[36835]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:54.301 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:53 smithi077 ceph-mon[37153]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:43:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:54 smithi029 ceph-mon[32610]: pgmap v300: 97 pgs: 97 active+clean; 634 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-07-27T06:43:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:54 smithi047 ceph-mon[36835]: pgmap v300: 97 pgs: 97 active+clean; 634 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-07-27T06:43:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:54 smithi077 ceph-mon[37153]: pgmap v300: 97 pgs: 97 active+clean; 634 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-07-27T06:43:56.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.067+0000 7f8474a10640 1 -- 172.21.15.29:0/2495892964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8470107dd0 msgr2=0x7f84701081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:56.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.067+0000 7f8474a10640 1 --2- 172.21.15.29:0/2495892964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8470107dd0 0x7f84701081e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8458009900 tx=0x7f845802a410 comp rx=0 tx=0).stop 2024-07-27T06:43:56.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.067+0000 7f8474a10640 1 -- 172.21.15.29:0/2495892964 shutdown_connections 2024-07-27T06:43:56.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.067+0000 7f8474a10640 1 --2- 172.21.15.29:0/2495892964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8470108720 0x7f847010ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.067+0000 7f8474a10640 1 --2- 172.21.15.29:0/2495892964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8470107dd0 0x7f84701081e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.067+0000 7f8474a10640 1 --2- 172.21.15.29:0/2495892964 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8470102490 0x7f84701028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.067+0000 7f8474a10640 1 -- 172.21.15.29:0/2495892964 >> 172.21.15.29:0/2495892964 conn(0x7f8470079160 msgr2=0x7f8470079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.067+0000 7f8474a10640 1 -- 172.21.15.29:0/2495892964 shutdown_connections 2024-07-27T06:43:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.067+0000 7f8474a10640 1 -- 172.21.15.29:0/2495892964 wait complete. 2024-07-27T06:43:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.068+0000 7f8474a10640 1 Processor -- start 2024-07-27T06:43:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.068+0000 7f8474a10640 1 -- start start 2024-07-27T06:43:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.068+0000 7f8474a10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8470102490 0x7f8470199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.068+0000 7f8474a10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8470107dd0 0x7f847019a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.068+0000 7f8474a10640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8470108720 0x7f847019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:56.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.068+0000 7f8474a10640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f84701134b0 con 0x7f8470107dd0 2024-07-27T06:43:56.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.068+0000 7f8474a10640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f8470113330 con 0x7f8470108720 2024-07-27T06:43:56.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.068+0000 7f8474a10640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8470113630 con 0x7f8470102490 2024-07-27T06:43:56.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.068+0000 7f846dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8470107dd0 0x7f847019a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:56.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f846dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8470107dd0 0x7f847019a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51154/0 (socket says 172.21.15.29:51154) 2024-07-27T06:43:56.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f846dd9b640 1 -- 172.21.15.29:0/4109304195 learned_addr learned my addr 172.21.15.29:0/4109304195 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:56.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f846ed9d640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8470108720 0x7f847019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:56.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f846e59c640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8470102490 0x7f8470199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:56.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f846dd9b640 1 -- 172.21.15.29:0/4109304195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8470102490 msgr2=0x7f8470199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:56.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f846dd9b640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8470102490 0x7f8470199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:56.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f846dd9b640 1 -- 172.21.15.29:0/4109304195 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8470108720 msgr2=0x7f847019f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:56.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f846dd9b640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8470108720 0x7f847019f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f846dd9b640 1 -- 172.21.15.29:0/4109304195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847019fe10 con 0x7f8470107dd0 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f846e59c640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8470102490 0x7f8470199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f846dd9b640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8470107dd0 0x7f847019a440 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8458032fd0 tx=0x7f8458004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f84677fe640 1 -- 172.21.15.29:0/4109304195 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845803f030 con 0x7f8470107dd0 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f84677fe640 1 -- 172.21.15.29:0/4109304195 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8458004490 con 0x7f8470107dd0 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f8474a10640 1 -- 172.21.15.29:0/4109304195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84701a00a0 con 0x7f8470107dd0 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f84677fe640 1 -- 172.21.15.29:0/4109304195 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8458035070 con 0x7f8470107dd0 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.069+0000 7f8474a10640 1 -- 172.21.15.29:0/4109304195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84701a6860 con 0x7f8470107dd0 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.070+0000 7f8474a10640 1 -- 172.21.15.29:0/4109304195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8470103710 con 0x7f8470107dd0 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.072+0000 7f84677fe640 1 -- 172.21.15.29:0/4109304195 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f8458042030 con 0x7f8470107dd0 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.073+0000 7f84677fe640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f845c0b3cb0 0x7f845c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.073+0000 7f846e59c640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f845c0b3cb0 0x7f845c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.073+0000 7f84677fe640 1 -- 172.21.15.29:0/4109304195 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7f8458104700 con 0x7f8470107dd0 2024-07-27T06:43:56.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.074+0000 7f846e59c640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f845c0b3cb0 0x7f845c0b61c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8468005fd0 tx=0x7f8468009520 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:56.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.076+0000 7f84677fe640 1 -- 172.21.15.29:0/4109304195 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f84580cbc10 con 0x7f8470107dd0 2024-07-27T06:43:56.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.297+0000 7f8474a10640 1 -- 172.21.15.29:0/4109304195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84701028a0 con 0x7f8470107dd0 2024-07-27T06:43:56.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.298+0000 7f84677fe640 1 -- 172.21.15.29:0/4109304195 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11957 (secure 0 0 0) 0x7f84701028a0 con 0x7f8470107dd0 2024-07-27T06:43:56.299 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:43:56.299 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"btime":"2024-07-27T06:43:37:875457+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:43:37.017169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24613,24617,24646,24658,24670]},"id":1}]} 2024-07-27T06:43:56.299 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-07-27T06:43:56.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.300+0000 7f8474a10640 1 -- 172.21.15.29:0/4109304195 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f845c0b3cb0 msgr2=0x7f845c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:56.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.300+0000 7f8474a10640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f845c0b3cb0 0x7f845c0b61c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8468005fd0 tx=0x7f8468009520 comp rx=0 tx=0).stop 2024-07-27T06:43:56.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.300+0000 7f8474a10640 1 -- 172.21.15.29:0/4109304195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8470107dd0 msgr2=0x7f847019a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:56.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.300+0000 7f8474a10640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8470107dd0 0x7f847019a440 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8458032fd0 tx=0x7f8458004270 comp rx=0 tx=0).stop 2024-07-27T06:43:56.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.301+0000 7f8474a10640 1 -- 172.21.15.29:0/4109304195 shutdown_connections 2024-07-27T06:43:56.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.301+0000 7f8474a10640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8470108720 0x7f847019f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.301+0000 7f8474a10640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8470107dd0 0x7f847019a440 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.301+0000 7f8474a10640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f845c0b3cb0 0x7f845c0b61c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.301+0000 7f8474a10640 1 --2- 172.21.15.29:0/4109304195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8470102490 0x7f8470199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.301+0000 7f8474a10640 1 -- 172.21.15.29:0/4109304195 >> 172.21.15.29:0/4109304195 conn(0x7f8470079160 msgr2=0x7f8470103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.301+0000 7f8474a10640 1 -- 172.21.15.29:0/4109304195 shutdown_connections 2024-07-27T06:43:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:56.301+0000 7f8474a10640 1 -- 172.21.15.29:0/4109304195 wait complete. 2024-07-27T06:43:56.989 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd dump --format=json 2024-07-27T06:43:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:56 smithi047 ceph-mon[36835]: pgmap v301: 97 pgs: 97 active+clean; 634 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-27T06:43:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:56 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/4109304195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:43:57.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:56 smithi029 ceph-mon[32610]: pgmap v301: 97 pgs: 97 active+clean; 634 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-27T06:43:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:56 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/4109304195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:43:57.257 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:43:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:56 smithi077 ceph-mon[37153]: pgmap v301: 97 pgs: 97 active+clean; 634 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-27T06:43:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:56 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/4109304195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:43:59.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:58 smithi029 ceph-mon[32610]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.9 KiB/s wr, 15 op/s 2024-07-27T06:43:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:58 smithi047 ceph-mon[36835]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.9 KiB/s wr, 15 op/s 2024-07-27T06:43:59.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.175+0000 7f1d32330640 1 -- 172.21.15.29:0/1901558186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d2c107df0 msgr2=0x7f1d2c108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:59.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.175+0000 7f1d32330640 1 --2- 172.21.15.29:0/1901558186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d2c107df0 0x7f1d2c108200 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009900 tx=0x7f1d2002a410 comp rx=0 tx=0).stop 2024-07-27T06:43:59.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.175+0000 7f1d32330640 1 -- 172.21.15.29:0/1901558186 shutdown_connections 2024-07-27T06:43:59.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.175+0000 7f1d32330640 1 --2- 172.21.15.29:0/1901558186 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c108740 0x7f1d2c10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:59.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.175+0000 7f1d32330640 1 --2- 172.21.15.29:0/1901558186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d2c107df0 0x7f1d2c108200 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:59.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.175+0000 7f1d32330640 1 --2- 172.21.15.29:0/1901558186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d2c1024b0 0x7f1d2c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:59.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.175+0000 7f1d32330640 1 -- 172.21.15.29:0/1901558186 >> 172.21.15.29:0/1901558186 conn(0x7f1d2c079170 msgr2=0x7f1d2c079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:59.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.175+0000 7f1d32330640 1 -- 172.21.15.29:0/1901558186 shutdown_connections 2024-07-27T06:43:59.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.175+0000 7f1d32330640 1 -- 172.21.15.29:0/1901558186 wait complete. 2024-07-27T06:43:59.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.176+0000 7f1d32330640 1 Processor -- start 2024-07-27T06:43:59.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.176+0000 7f1d32330640 1 -- start start 2024-07-27T06:43:59.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.177+0000 7f1d32330640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c1024b0 0x7f1d2c199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:59.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.177+0000 7f1d32330640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d2c107df0 0x7f1d2c19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:59.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.177+0000 7f1d32330640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d2c108740 0x7f1d2c19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:59.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.177+0000 7f1d32330640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2c1134d0 con 0x7f1d2c108740 2024-07-27T06:43:59.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.177+0000 7f1d32330640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2c113350 con 0x7f1d2c1024b0 2024-07-27T06:43:59.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.177+0000 7f1d32330640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2c113650 con 0x7f1d2c107df0 2024-07-27T06:43:59.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.177+0000 7f1d308cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d2c108740 0x7f1d2c19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:59.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.177+0000 7f1d308cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d2c108740 0x7f1d2c19f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34032/0 (socket says 172.21.15.29:34032) 2024-07-27T06:43:59.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.177+0000 7f1d308cd640 1 -- 172.21.15.29:0/2682378240 learned_addr learned my addr 172.21.15.29:0/2682378240 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:43:59.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.177+0000 7f1d2bfff640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c1024b0 0x7f1d2c199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:59.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.177+0000 7f1d2b7fe640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d2c107df0 0x7f1d2c19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:59.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d308cd640 1 -- 172.21.15.29:0/2682378240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d2c107df0 msgr2=0x7f1d2c19a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:59.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d308cd640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d2c107df0 0x7f1d2c19a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:59.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d308cd640 1 -- 172.21.15.29:0/2682378240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c1024b0 msgr2=0x7f1d2c199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:59.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d308cd640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c1024b0 0x7f1d2c199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:59.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d308cd640 1 -- 172.21.15.29:0/2682378240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2c19fe30 con 0x7f1d2c108740 2024-07-27T06:43:59.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d2b7fe640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d2c107df0 0x7f1d2c19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:43:59.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d2bfff640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c1024b0 0x7f1d2c199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:43:59.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d308cd640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d2c108740 0x7f1d2c19f700 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400cad0 tx=0x7f1d2400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:59.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d297fa640 1 -- 172.21.15.29:0/2682378240 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2400ae20 con 0x7f1d2c108740 2024-07-27T06:43:59.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d297fa640 1 -- 172.21.15.29:0/2682378240 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d24005020 con 0x7f1d2c108740 2024-07-27T06:43:59.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d32330640 1 -- 172.21.15.29:0/2682378240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d2c1a0120 con 0x7f1d2c108740 2024-07-27T06:43:59.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d297fa640 1 -- 172.21.15.29:0/2682378240 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2400f6b0 con 0x7f1d2c108740 2024-07-27T06:43:59.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.178+0000 7f1d32330640 1 -- 172.21.15.29:0/2682378240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d2c1a6880 con 0x7f1d2c108740 2024-07-27T06:43:59.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.179+0000 7f1d32330640 1 -- 172.21.15.29:0/2682378240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d2c103730 con 0x7f1d2c108740 2024-07-27T06:43:59.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.181+0000 7f1d297fa640 1 -- 172.21.15.29:0/2682378240 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f1d240051d0 con 0x7f1d2c108740 2024-07-27T06:43:59.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.181+0000 7f1d297fa640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1d140b3cb0 0x7f1d140b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:43:59.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.182+0000 7f1d297fa640 1 -- 172.21.15.29:0/2682378240 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7f1d240e4130 con 0x7f1d2c108740 2024-07-27T06:43:59.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.182+0000 7f1d2bfff640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1d140b3cb0 0x7f1d140b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:43:59.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.182+0000 7f1d2bfff640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1d140b3cb0 0x7f1d140b61c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009fc0 tx=0x7f1d1c005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:43:59.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.185+0000 7f1d297fa640 1 -- 172.21.15.29:0/2682378240 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1d24014030 con 0x7f1d2c108740 2024-07-27T06:43:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:58 smithi077 ceph-mon[37153]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.9 KiB/s wr, 15 op/s 2024-07-27T06:43:59.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.368+0000 7f1d32330640 1 -- 172.21.15.29:0/2682378240 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1d2c1028c0 con 0x7f1d2c108740 2024-07-27T06:43:59.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.369+0000 7f1d297fa640 1 -- 172.21.15.29:0/2682378240 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22464 (secure 0 0 0) 0x7f1d2c1028c0 con 0x7f1d2c108740 2024-07-27T06:43:59.371 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:43:59.371 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":79,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","created":"2024-07-27T06:33:57.535327+0000","modified":"2024-07-27T06:43:14.801257+0000","last_up_change":"2024-07-27T06:42:05.757156+0000","last_in_change":"2024-07-27T06:41:46.423732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T06:38:21.606527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-27T06:43:07.755490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-27T06:43:11.658694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06833d24-2e1a-4199-9338-7d6883d1bf9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6803","nonce":386121641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6805","nonce":386121641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6809","nonce":386121641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6807","nonce":386121641}]},"public_addr":"172.21.15.29:6803/386121641","cluster_addr":"172.21.15.29:6805/386121641","heartbeat_back_addr":"172.21.15.29:6809/386121641","heartbeat_front_addr":"172.21.15.29:6807/386121641","state":["exists","up"]},{"osd":1,"uuid":"906aba19-80dc-41b5-953c-af6dca604b05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6801","nonce":2795953749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6803","nonce":2795953749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6807","nonce":2795953749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6805","nonce":2795953749}]},"public_addr":"172.21.15.47:6801/2795953749","cluster_addr":"172.21.15.47:6803/2795953749","heartbeat_back_addr":"172.21.15.47:6807/2795953749","heartbeat_front_addr":"172.21.15.47:6805/2795953749","state":["exists","up"]},{"osd":2,"uuid":"194ae5a5-aebf-4cff-9b99-585a3073a269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6801","nonce":2202565078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6803","nonce":2202565078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6807","nonce":2202565078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6805","nonce":2202565078}]},"public_addr":"172.21.15.77:6801/2202565078","cluster_addr":"172.21.15.77:6803/2202565078","heartbeat_back_addr":"172.21.15.77:6807/2202565078","heartbeat_front_addr":"172.21.15.77:6805/2202565078","state":["exists","up"]},{"osd":3,"uuid":"3c22ce9c-963e-4f72-86e5-360e781526db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6811","nonce":3757476312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6813","nonce":3757476312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6817","nonce":3757476312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6815","nonce":3757476312}]},"public_addr":"172.21.15.29:6811/3757476312","cluster_addr":"172.21.15.29:6813/3757476312","heartbeat_back_addr":"172.21.15.29:6817/3757476312","heartbeat_front_addr":"172.21.15.29:6815/3757476312","state":["exists","up"]},{"osd":4,"uuid":"cd4b585f-e7a0-4d82-873d-3a9269ebac3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6809","nonce":1379363720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6811","nonce":1379363720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6815","nonce":1379363720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6813","nonce":1379363720}]},"public_addr":"172.21.15.47:6809/1379363720","cluster_addr":"172.21.15.47:6811/1379363720","heartbeat_back_addr":"172.21.15.47:6815/1379363720","heartbeat_front_addr":"172.21.15.47:6813/1379363720","state":["exists","up"]},{"osd":5,"uuid":"40a3eb14-dfaa-4523-8c28-d74c19293c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6809","nonce":1338438780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6811","nonce":1338438780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6815","nonce":1338438780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6813","nonce":1338438780}]},"public_addr":"172.21.15.77:6809/1338438780","cluster_addr":"172.21.15.77:6811/1338438780","heartbeat_back_addr":"172.21.15.77:6815/1338438780","heartbeat_front_addr":"172.21.15.77:6813/1338438780","state":["exists","up"]},{"osd":6,"uuid":"91e85be2-10d1-4209-aa33-17da8f8bd8bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6819","nonce":2679402148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6821","nonce":2679402148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6825","nonce":2679402148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6823","nonce":2679402148}]},"public_addr":"172.21.15.29:6819/2679402148","cluster_addr":"172.21.15.29:6821/2679402148","heartbeat_back_addr":"172.21.15.29:6825/2679402148","heartbeat_front_addr":"172.21.15.29:6823/2679402148","state":["exists","up"]},{"osd":7,"uuid":"c655b450-0cc4-45d4-95ea-93a57fe8e4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6817","nonce":2439195589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6819","nonce":2439195589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6823","nonce":2439195589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6821","nonce":2439195589}]},"public_addr":"172.21.15.47:6817/2439195589","cluster_addr":"172.21.15.47:6819/2439195589","heartbeat_back_addr":"172.21.15.47:6823/2439195589","heartbeat_front_addr":"172.21.15.47:6821/2439195589","state":["exists","up"]},{"osd":8,"uuid":"a7a95b51-f32c-433e-a197-feb6d47b6c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6817","nonce":1422290905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6819","nonce":1422290905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6823","nonce":1422290905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6821","nonce":1422290905}]},"public_addr":"172.21.15.77:6817/1422290905","cluster_addr":"172.21.15.77:6819/1422290905","heartbeat_back_addr":"172.21.15.77:6823/1422290905","heartbeat_front_addr":"172.21.15.77:6821/1422290905","state":["exists","up"]},{"osd":9,"uuid":"f352242b-0d2e-41b3-9c83-436cf432b0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6827","nonce":1307316384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6829","nonce":1307316384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6833","nonce":1307316384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6831","nonce":1307316384}]},"public_addr":"172.21.15.29:6827/1307316384","cluster_addr":"172.21.15.29:6829/1307316384","heartbeat_back_addr":"172.21.15.29:6833/1307316384","heartbeat_front_addr":"172.21.15.29:6831/1307316384","state":["exists","up"]},{"osd":10,"uuid":"4148c2df-c3e9-4a47-88f7-36cf3eab8d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6825","nonce":2133726850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6827","nonce":2133726850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6831","nonce":2133726850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6829","nonce":2133726850}]},"public_addr":"172.21.15.47:6825/2133726850","cluster_addr":"172.21.15.47:6827/2133726850","heartbeat_back_addr":"172.21.15.47:6831/2133726850","heartbeat_front_addr":"172.21.15.47:6829/2133726850","state":["exists","up"]},{"osd":11,"uuid":"5e8b1da3-5f65-4fe9-8722-b35921cbd8c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6825","nonce":3228299327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6827","nonce":3228299327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6831","nonce":3228299327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6829","nonce":3228299327}]},"public_addr":"172.21.15.77:6825/3228299327","cluster_addr":"172.21.15.77:6827/3228299327","heartbeat_back_addr":"172.21.15.77:6831/3228299327","heartbeat_front_addr":"172.21.15.77:6829/3228299327","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:32.525206+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:56.489640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:18.887845+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:43.727678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:08.755553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:32.923461+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:59.298439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:23.944848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:48.883050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:14.566310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:39.557809+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:42:03.813903+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2378279024":"2024-07-28T06:36:07.527503+0000","172.21.15.29:6801/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3527133965":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2262661620":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/3156863587":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2424241521":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3881857930":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/398508875":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6801/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6800/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2679856281":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6801/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2708687265":"2024-07-28T06:34:58.250198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T06:43:59.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.371+0000 7f1d32330640 1 -- 172.21.15.29:0/2682378240 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1d140b3cb0 msgr2=0x7f1d140b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:59.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.371+0000 7f1d32330640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1d140b3cb0 0x7f1d140b61c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009fc0 tx=0x7f1d1c005f90 comp rx=0 tx=0).stop 2024-07-27T06:43:59.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.371+0000 7f1d32330640 1 -- 172.21.15.29:0/2682378240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d2c108740 msgr2=0x7f1d2c19f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:43:59.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.371+0000 7f1d32330640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d2c108740 0x7f1d2c19f700 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400cad0 tx=0x7f1d2400cf90 comp rx=0 tx=0).stop 2024-07-27T06:43:59.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.372+0000 7f1d32330640 1 -- 172.21.15.29:0/2682378240 shutdown_connections 2024-07-27T06:43:59.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.372+0000 7f1d32330640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d2c108740 0x7f1d2c19f700 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:59.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.372+0000 7f1d32330640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1d140b3cb0 0x7f1d140b61c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:59.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.372+0000 7f1d32330640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d2c107df0 0x7f1d2c19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:59.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.372+0000 7f1d32330640 1 --2- 172.21.15.29:0/2682378240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d2c1024b0 0x7f1d2c199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:43:59.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.372+0000 7f1d32330640 1 -- 172.21.15.29:0/2682378240 >> 172.21.15.29:0/2682378240 conn(0x7f1d2c079170 msgr2=0x7f1d2c103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:43:59.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.372+0000 7f1d32330640 1 -- 172.21.15.29:0/2682378240 shutdown_connections 2024-07-27T06:43:59.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:43:59.372+0000 7f1d32330640 1 -- 172.21.15.29:0/2682378240 wait complete. 2024-07-27T06:44:00.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:43:59 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2682378240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:44:00.129 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-27T06:44:00.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph pg dump --format=json 2024-07-27T06:44:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:43:59 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2682378240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:44:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:43:59 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2682378240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:44:00.401 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:00 smithi047 ceph-mon[36835]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.9 KiB/s wr, 10 op/s 2024-07-27T06:44:01.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:00 smithi029 ceph-mon[32610]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.9 KiB/s wr, 10 op/s 2024-07-27T06:44:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:00 smithi077 ceph-mon[37153]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.9 KiB/s wr, 10 op/s 2024-07-27T06:44:02.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:44:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:44:02.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.267+0000 7fab43b62640 1 -- 172.21.15.29:0/1533675266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab3c108ed0 msgr2=0x7fab3c1092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:02.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.267+0000 7fab43b62640 1 --2- 172.21.15.29:0/1533675266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab3c108ed0 0x7fab3c1092e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fab2c009900 tx=0x7fab2c02a410 comp rx=0 tx=0).stop 2024-07-27T06:44:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.267+0000 7fab43b62640 1 -- 172.21.15.29:0/1533675266 shutdown_connections 2024-07-27T06:44:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.267+0000 7fab43b62640 1 --2- 172.21.15.29:0/1533675266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab3c109820 0x7fab3c110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.267+0000 7fab43b62640 1 --2- 172.21.15.29:0/1533675266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab3c108ed0 0x7fab3c1092e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.267+0000 7fab43b62640 1 --2- 172.21.15.29:0/1533675266 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab3c103590 0x7fab3c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.267+0000 7fab43b62640 1 -- 172.21.15.29:0/1533675266 >> 172.21.15.29:0/1533675266 conn(0x7fab3c07b280 msgr2=0x7fab3c07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.268+0000 7fab43b62640 1 -- 172.21.15.29:0/1533675266 shutdown_connections 2024-07-27T06:44:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.268+0000 7fab43b62640 1 -- 172.21.15.29:0/1533675266 wait complete. 2024-07-27T06:44:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.268+0000 7fab43b62640 1 Processor -- start 2024-07-27T06:44:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.268+0000 7fab43b62640 1 -- start start 2024-07-27T06:44:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.268+0000 7fab43b62640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab3c103590 0x7fab3c199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:02.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.268+0000 7fab43b62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab3c108ed0 0x7fab3c19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:02.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab43b62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab3c109820 0x7fab3c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:02.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab43b62640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c1134b0 con 0x7fab3c108ed0 2024-07-27T06:44:02.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab43b62640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c113330 con 0x7fab3c103590 2024-07-27T06:44:02.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab43b62640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c113630 con 0x7fab3c109820 2024-07-27T06:44:02.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab410fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab3c108ed0 0x7fab3c19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:02.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab410fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab3c108ed0 0x7fab3c19a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34044/0 (socket says 172.21.15.29:34044) 2024-07-27T06:44:02.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab410fd640 1 -- 172.21.15.29:0/2522012223 learned_addr learned my addr 172.21.15.29:0/2522012223 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:02.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab420ff640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab3c109820 0x7fab3c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:02.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab418fe640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab3c103590 0x7fab3c199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:02.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab410fd640 1 -- 172.21.15.29:0/2522012223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab3c109820 msgr2=0x7fab3c19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:02.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab410fd640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab3c109820 0x7fab3c19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:02.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab410fd640 1 -- 172.21.15.29:0/2522012223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab3c103590 msgr2=0x7fab3c199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:02.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab410fd640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab3c103590 0x7fab3c199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:02.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab410fd640 1 -- 172.21.15.29:0/2522012223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3c19fe50 con 0x7fab3c108ed0 2024-07-27T06:44:02.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab420ff640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab3c109820 0x7fab3c19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:44:02.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab410fd640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab3c108ed0 0x7fab3c19a480 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fab2c02a930 tx=0x7fab2c031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:02.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab32ffd640 1 -- 172.21.15.29:0/2522012223 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c03f030 con 0x7fab3c108ed0 2024-07-27T06:44:02.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab32ffd640 1 -- 172.21.15.29:0/2522012223 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab2c004490 con 0x7fab3c108ed0 2024-07-27T06:44:02.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.269+0000 7fab43b62640 1 -- 172.21.15.29:0/2522012223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3c1a00e0 con 0x7fab3c108ed0 2024-07-27T06:44:02.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.270+0000 7fab32ffd640 1 -- 172.21.15.29:0/2522012223 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c035070 con 0x7fab3c108ed0 2024-07-27T06:44:02.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.270+0000 7fab43b62640 1 -- 172.21.15.29:0/2522012223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab3c1a68a0 con 0x7fab3c108ed0 2024-07-27T06:44:02.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.271+0000 7fab43b62640 1 -- 172.21.15.29:0/2522012223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab3c104810 con 0x7fab3c108ed0 2024-07-27T06:44:02.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.272+0000 7fab32ffd640 1 -- 172.21.15.29:0/2522012223 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fab2c042030 con 0x7fab3c108ed0 2024-07-27T06:44:02.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.273+0000 7fab32ffd640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fab280b3cb0 0x7fab280b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:02.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.273+0000 7fab418fe640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fab280b3cb0 0x7fab280b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:02.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.273+0000 7fab32ffd640 1 -- 172.21.15.29:0/2522012223 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7fab2c1040a0 con 0x7fab3c108ed0 2024-07-27T06:44:02.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.273+0000 7fab418fe640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fab280b3cb0 0x7fab280b61c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fab38009bd0 tx=0x7fab38005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:02.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.277+0000 7fab32ffd640 1 -- 172.21.15.29:0/2522012223 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fab2c0caf60 con 0x7fab3c108ed0 2024-07-27T06:44:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:44:02.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.492+0000 7fab43b62640 1 -- 172.21.15.29:0/2522012223 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fab3c063230 con 0x7fab280b3cb0 2024-07-27T06:44:02.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.497+0000 7fab32ffd640 1 -- 172.21.15.29:0/2522012223 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258903 (secure 0 0 0) 0x7fab3c063230 con 0x7fab280b3cb0 2024-07-27T06:44:02.500 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:44:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-07-27T06:44:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.501+0000 7fab43b62640 1 -- 172.21.15.29:0/2522012223 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fab280b3cb0 msgr2=0x7fab280b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.501+0000 7fab43b62640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fab280b3cb0 0x7fab280b61c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fab38009bd0 tx=0x7fab38005f90 comp rx=0 tx=0).stop 2024-07-27T06:44:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.501+0000 7fab43b62640 1 -- 172.21.15.29:0/2522012223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab3c108ed0 msgr2=0x7fab3c19a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.501+0000 7fab43b62640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab3c108ed0 0x7fab3c19a480 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fab2c02a930 tx=0x7fab2c031a20 comp rx=0 tx=0).stop 2024-07-27T06:44:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.501+0000 7fab43b62640 1 -- 172.21.15.29:0/2522012223 shutdown_connections 2024-07-27T06:44:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.501+0000 7fab43b62640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fab280b3cb0 0x7fab280b61c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.501+0000 7fab43b62640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fab3c109820 0x7fab3c19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.501+0000 7fab43b62640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab3c108ed0 0x7fab3c19a480 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:02.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.501+0000 7fab43b62640 1 --2- 172.21.15.29:0/2522012223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fab3c103590 0x7fab3c199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:02.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.501+0000 7fab43b62640 1 -- 172.21.15.29:0/2522012223 >> 172.21.15.29:0/2522012223 conn(0x7fab3c07b280 msgr2=0x7fab3c106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:02.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.502+0000 7fab43b62640 1 -- 172.21.15.29:0/2522012223 shutdown_connections 2024-07-27T06:44:02.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:02.502+0000 7fab43b62640 1 -- 172.21.15.29:0/2522012223 wait complete. 2024-07-27T06:44:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:02 smithi047 ceph-mon[36835]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.9 KiB/s wr, 11 op/s 2024-07-27T06:44:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:02 smithi029 ceph-mon[32610]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.9 KiB/s wr, 11 op/s 2024-07-27T06:44:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:02 smithi077 ceph-mon[37153]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.9 KiB/s wr, 11 op/s 2024-07-27T06:44:03.659 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-07-27T06:44:01.632137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654287,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":375,"num_read_kb":284,"num_write":368,"num_write_kb":2284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":359844,"kb_used_data":10692,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581980,"statfs":{"total":1151940427776,"available":1151571947520,"internally_reserved":0,"allocated":10948608,"data_stored":7064553,"data_compressed":107991,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28124,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":123,"num_read_kb":84,"num_write":20,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003349"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812421+0000","last_change":"2024-07-27T06:43:12.802353+0000","last_active":"2024-07-27T06:43:15.812421+0000","last_peered":"2024-07-27T06:43:15.812421+0000","last_clean":"2024-07-27T06:43:15.812421+0000","last_became_active":"2024-07-27T06:43:12.802028+0000","last_became_peered":"2024-07-27T06:43:12.802028+0000","last_unstale":"2024-07-27T06:43:15.812421+0000","last_undegraded":"2024-07-27T06:43:15.812421+0000","last_fullsized":"2024-07-27T06:43:15.812421+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:22:24.292567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812061+0000","last_change":"2024-07-27T06:43:12.806269+0000","last_active":"2024-07-27T06:43:15.812061+0000","last_peered":"2024-07-27T06:43:15.812061+0000","last_clean":"2024-07-27T06:43:15.812061+0000","last_became_active":"2024-07-27T06:43:12.803232+0000","last_became_peered":"2024-07-27T06:43:12.803232+0000","last_unstale":"2024-07-27T06:43:15.812061+0000","last_undegraded":"2024-07-27T06:43:15.812061+0000","last_fullsized":"2024-07-27T06:43:15.812061+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:57:53.673850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811879+0000","last_change":"2024-07-27T06:43:12.805542+0000","last_active":"2024-07-27T06:43:15.811879+0000","last_peered":"2024-07-27T06:43:15.811879+0000","last_clean":"2024-07-27T06:43:15.811879+0000","last_became_active":"2024-07-27T06:43:12.804951+0000","last_became_peered":"2024-07-27T06:43:12.804951+0000","last_unstale":"2024-07-27T06:43:15.811879+0000","last_undegraded":"2024-07-27T06:43:15.811879+0000","last_fullsized":"2024-07-27T06:43:15.811879+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:06:40.469695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794334+0000","last_change":"2024-07-27T06:43:12.796950+0000","last_active":"2024-07-27T06:43:15.794334+0000","last_peered":"2024-07-27T06:43:15.794334+0000","last_clean":"2024-07-27T06:43:15.794334+0000","last_became_active":"2024-07-27T06:43:12.796549+0000","last_became_peered":"2024-07-27T06:43:12.796549+0000","last_unstale":"2024-07-27T06:43:15.794334+0000","last_undegraded":"2024-07-27T06:43:15.794334+0000","last_fullsized":"2024-07-27T06:43:15.794334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:30:36.391046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808022+0000","last_change":"2024-07-27T06:43:12.799838+0000","last_active":"2024-07-27T06:43:15.808022+0000","last_peered":"2024-07-27T06:43:15.808022+0000","last_clean":"2024-07-27T06:43:15.808022+0000","last_became_active":"2024-07-27T06:43:12.799462+0000","last_became_peered":"2024-07-27T06:43:12.799462+0000","last_unstale":"2024-07-27T06:43:15.808022+0000","last_undegraded":"2024-07-27T06:43:15.808022+0000","last_fullsized":"2024-07-27T06:43:15.808022+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:45:19.371259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.806561+0000","last_change":"2024-07-27T06:43:12.797187+0000","last_active":"2024-07-27T06:43:15.806561+0000","last_peered":"2024-07-27T06:43:15.806561+0000","last_clean":"2024-07-27T06:43:15.806561+0000","last_became_active":"2024-07-27T06:43:12.796633+0000","last_became_peered":"2024-07-27T06:43:12.796633+0000","last_unstale":"2024-07-27T06:43:15.806561+0000","last_undegraded":"2024-07-27T06:43:15.806561+0000","last_fullsized":"2024-07-27T06:43:15.806561+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:22:20.882899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810075+0000","last_change":"2024-07-27T06:43:12.793786+0000","last_active":"2024-07-27T06:43:15.810075+0000","last_peered":"2024-07-27T06:43:15.810075+0000","last_clean":"2024-07-27T06:43:15.810075+0000","last_became_active":"2024-07-27T06:43:12.793413+0000","last_became_peered":"2024-07-27T06:43:12.793413+0000","last_unstale":"2024-07-27T06:43:15.810075+0000","last_undegraded":"2024-07-27T06:43:15.810075+0000","last_fullsized":"2024-07-27T06:43:15.810075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:14.015373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.807834+0000","last_change":"2024-07-27T06:43:12.791304+0000","last_active":"2024-07-27T06:43:15.807834+0000","last_peered":"2024-07-27T06:43:15.807834+0000","last_clean":"2024-07-27T06:43:15.807834+0000","last_became_active":"2024-07-27T06:43:12.790801+0000","last_became_peered":"2024-07-27T06:43:12.790801+0000","last_unstale":"2024-07-27T06:43:15.807834+0000","last_undegraded":"2024-07-27T06:43:15.807834+0000","last_fullsized":"2024-07-27T06:43:15.807834+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:12:37.606748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811264+0000","last_change":"2024-07-27T06:43:12.797778+0000","last_active":"2024-07-27T06:43:15.811264+0000","last_peered":"2024-07-27T06:43:15.811264+0000","last_clean":"2024-07-27T06:43:15.811264+0000","last_became_active":"2024-07-27T06:43:12.796784+0000","last_became_peered":"2024-07-27T06:43:12.796784+0000","last_unstale":"2024-07-27T06:43:15.811264+0000","last_undegraded":"2024-07-27T06:43:15.811264+0000","last_fullsized":"2024-07-27T06:43:15.811264+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:52:22.275282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.807073+0000","last_change":"2024-07-27T06:43:12.798970+0000","last_active":"2024-07-27T06:43:15.807073+0000","last_peered":"2024-07-27T06:43:15.807073+0000","last_clean":"2024-07-27T06:43:15.807073+0000","last_became_active":"2024-07-27T06:43:12.798549+0000","last_became_peered":"2024-07-27T06:43:12.798549+0000","last_unstale":"2024-07-27T06:43:15.807073+0000","last_undegraded":"2024-07-27T06:43:15.807073+0000","last_fullsized":"2024-07-27T06:43:15.807073+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:58:32.464429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.809869+0000","last_change":"2024-07-27T06:43:12.809405+0000","last_active":"2024-07-27T06:43:15.809869+0000","last_peered":"2024-07-27T06:43:15.809869+0000","last_clean":"2024-07-27T06:43:15.809869+0000","last_became_active":"2024-07-27T06:43:12.809030+0000","last_became_peered":"2024-07-27T06:43:12.809030+0000","last_unstale":"2024-07-27T06:43:15.809869+0000","last_undegraded":"2024-07-27T06:43:15.809869+0000","last_fullsized":"2024-07-27T06:43:15.809869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:46:35.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794487+0000","last_change":"2024-07-27T06:43:12.802647+0000","last_active":"2024-07-27T06:43:15.794487+0000","last_peered":"2024-07-27T06:43:15.794487+0000","last_clean":"2024-07-27T06:43:15.794487+0000","last_became_active":"2024-07-27T06:43:12.802247+0000","last_became_peered":"2024-07-27T06:43:12.802247+0000","last_unstale":"2024-07-27T06:43:15.794487+0000","last_undegraded":"2024-07-27T06:43:15.794487+0000","last_fullsized":"2024-07-27T06:43:15.794487+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:44:24.564852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.793711+0000","last_change":"2024-07-27T06:43:12.809140+0000","last_active":"2024-07-27T06:43:15.793711+0000","last_peered":"2024-07-27T06:43:15.793711+0000","last_clean":"2024-07-27T06:43:15.793711+0000","last_became_active":"2024-07-27T06:43:12.808692+0000","last_became_peered":"2024-07-27T06:43:12.808692+0000","last_unstale":"2024-07-27T06:43:15.793711+0000","last_undegraded":"2024-07-27T06:43:15.793711+0000","last_fullsized":"2024-07-27T06:43:15.793711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:37:45.326234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811596+0000","last_change":"2024-07-27T06:43:12.802838+0000","last_active":"2024-07-27T06:43:15.811596+0000","last_peered":"2024-07-27T06:43:15.811596+0000","last_clean":"2024-07-27T06:43:15.811596+0000","last_became_active":"2024-07-27T06:43:12.802503+0000","last_became_peered":"2024-07-27T06:43:12.802503+0000","last_unstale":"2024-07-27T06:43:15.811596+0000","last_undegraded":"2024-07-27T06:43:15.811596+0000","last_fullsized":"2024-07-27T06:43:15.811596+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:41:03.428162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811733+0000","last_change":"2024-07-27T06:43:12.801626+0000","last_active":"2024-07-27T06:43:15.811733+0000","last_peered":"2024-07-27T06:43:15.811733+0000","last_clean":"2024-07-27T06:43:15.811733+0000","last_became_active":"2024-07-27T06:43:12.801266+0000","last_became_peered":"2024-07-27T06:43:12.801266+0000","last_unstale":"2024-07-27T06:43:15.811733+0000","last_undegraded":"2024-07-27T06:43:15.811733+0000","last_fullsized":"2024-07-27T06:43:15.811733+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:54:31.934223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812785+0000","last_change":"2024-07-27T06:43:12.804543+0000","last_active":"2024-07-27T06:43:15.812785+0000","last_peered":"2024-07-27T06:43:15.812785+0000","last_clean":"2024-07-27T06:43:15.812785+0000","last_became_active":"2024-07-27T06:43:12.804173+0000","last_became_peered":"2024-07-27T06:43:12.804173+0000","last_unstale":"2024-07-27T06:43:15.812785+0000","last_undegraded":"2024-07-27T06:43:15.812785+0000","last_fullsized":"2024-07-27T06:43:15.812785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:18:25.337333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812620+0000","last_change":"2024-07-27T06:43:12.798604+0000","last_active":"2024-07-27T06:43:15.812620+0000","last_peered":"2024-07-27T06:43:15.812620+0000","last_clean":"2024-07-27T06:43:15.812620+0000","last_became_active":"2024-07-27T06:43:12.798228+0000","last_became_peered":"2024-07-27T06:43:12.798228+0000","last_unstale":"2024-07-27T06:43:15.812620+0000","last_undegraded":"2024-07-27T06:43:15.812620+0000","last_fullsized":"2024-07-27T06:43:15.812620+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:04:35.851399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.807464+0000","last_change":"2024-07-27T06:43:12.800452+0000","last_active":"2024-07-27T06:43:15.807464+0000","last_peered":"2024-07-27T06:43:15.807464+0000","last_clean":"2024-07-27T06:43:15.807464+0000","last_became_active":"2024-07-27T06:43:12.800084+0000","last_became_peered":"2024-07-27T06:43:12.800084+0000","last_unstale":"2024-07-27T06:43:15.807464+0000","last_undegraded":"2024-07-27T06:43:15.807464+0000","last_fullsized":"2024-07-27T06:43:15.807464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:54:27.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.814057+0000","last_change":"2024-07-27T06:43:12.800761+0000","last_active":"2024-07-27T06:43:15.814057+0000","last_peered":"2024-07-27T06:43:15.814057+0000","last_clean":"2024-07-27T06:43:15.814057+0000","last_became_active":"2024-07-27T06:43:12.800414+0000","last_became_peered":"2024-07-27T06:43:12.800414+0000","last_unstale":"2024-07-27T06:43:15.814057+0000","last_undegraded":"2024-07-27T06:43:15.814057+0000","last_fullsized":"2024-07-27T06:43:15.814057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:02:47.247522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811101+0000","last_change":"2024-07-27T06:43:12.795470+0000","last_active":"2024-07-27T06:43:15.811101+0000","last_peered":"2024-07-27T06:43:15.811101+0000","last_clean":"2024-07-27T06:43:15.811101+0000","last_became_active":"2024-07-27T06:43:12.795136+0000","last_became_peered":"2024-07-27T06:43:12.795136+0000","last_unstale":"2024-07-27T06:43:15.811101+0000","last_undegraded":"2024-07-27T06:43:15.811101+0000","last_fullsized":"2024-07-27T06:43:15.811101+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:01:22.178699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.793740+0000","last_change":"2024-07-27T06:43:12.803897+0000","last_active":"2024-07-27T06:43:15.793740+0000","last_peered":"2024-07-27T06:43:15.793740+0000","last_clean":"2024-07-27T06:43:15.793740+0000","last_became_active":"2024-07-27T06:43:12.803546+0000","last_became_peered":"2024-07-27T06:43:12.803546+0000","last_unstale":"2024-07-27T06:43:15.793740+0000","last_undegraded":"2024-07-27T06:43:15.793740+0000","last_fullsized":"2024-07-27T06:43:15.793740+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:13:50.816831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.813198+0000","last_change":"2024-07-27T06:43:12.803605+0000","last_active":"2024-07-27T06:43:15.813198+0000","last_peered":"2024-07-27T06:43:15.813198+0000","last_clean":"2024-07-27T06:43:15.813198+0000","last_became_active":"2024-07-27T06:43:12.803144+0000","last_became_peered":"2024-07-27T06:43:12.803144+0000","last_unstale":"2024-07-27T06:43:15.813198+0000","last_undegraded":"2024-07-27T06:43:15.813198+0000","last_fullsized":"2024-07-27T06:43:15.813198+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:34:10.372185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808876+0000","last_change":"2024-07-27T06:43:12.788570+0000","last_active":"2024-07-27T06:43:15.808876+0000","last_peered":"2024-07-27T06:43:15.808876+0000","last_clean":"2024-07-27T06:43:15.808876+0000","last_became_active":"2024-07-27T06:43:12.788199+0000","last_became_peered":"2024-07-27T06:43:12.788199+0000","last_unstale":"2024-07-27T06:43:15.808876+0000","last_undegraded":"2024-07-27T06:43:15.808876+0000","last_fullsized":"2024-07-27T06:43:15.808876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:44:26.751094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794071+0000","last_change":"2024-07-27T06:43:12.792866+0000","last_active":"2024-07-27T06:43:15.794071+0000","last_peered":"2024-07-27T06:43:15.794071+0000","last_clean":"2024-07-27T06:43:15.794071+0000","last_became_active":"2024-07-27T06:43:12.792389+0000","last_became_peered":"2024-07-27T06:43:12.792389+0000","last_unstale":"2024-07-27T06:43:15.794071+0000","last_undegraded":"2024-07-27T06:43:15.794071+0000","last_fullsized":"2024-07-27T06:43:15.794071+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:10:37.367642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.807780+0000","last_change":"2024-07-27T06:43:12.797002+0000","last_active":"2024-07-27T06:43:15.807780+0000","last_peered":"2024-07-27T06:43:15.807780+0000","last_clean":"2024-07-27T06:43:15.807780+0000","last_became_active":"2024-07-27T06:43:12.796651+0000","last_became_peered":"2024-07-27T06:43:12.796651+0000","last_unstale":"2024-07-27T06:43:15.807780+0000","last_undegraded":"2024-07-27T06:43:15.807780+0000","last_fullsized":"2024-07-27T06:43:15.807780+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:28:06.934042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.813220+0000","last_change":"2024-07-27T06:43:12.798549+0000","last_active":"2024-07-27T06:43:15.813220+0000","last_peered":"2024-07-27T06:43:15.813220+0000","last_clean":"2024-07-27T06:43:15.813220+0000","last_became_active":"2024-07-27T06:43:12.798201+0000","last_became_peered":"2024-07-27T06:43:12.798201+0000","last_unstale":"2024-07-27T06:43:15.813220+0000","last_undegraded":"2024-07-27T06:43:15.813220+0000","last_fullsized":"2024-07-27T06:43:15.813220+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:59:47.525536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811494+0000","last_change":"2024-07-27T06:43:12.791053+0000","last_active":"2024-07-27T06:43:15.811494+0000","last_peered":"2024-07-27T06:43:15.811494+0000","last_clean":"2024-07-27T06:43:15.811494+0000","last_became_active":"2024-07-27T06:43:12.790718+0000","last_became_peered":"2024-07-27T06:43:12.790718+0000","last_unstale":"2024-07-27T06:43:15.811494+0000","last_undegraded":"2024-07-27T06:43:15.811494+0000","last_fullsized":"2024-07-27T06:43:15.811494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:09.455399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.793523+0000","last_change":"2024-07-27T06:43:12.804180+0000","last_active":"2024-07-27T06:43:15.793523+0000","last_peered":"2024-07-27T06:43:15.793523+0000","last_clean":"2024-07-27T06:43:15.793523+0000","last_became_active":"2024-07-27T06:43:12.803881+0000","last_became_peered":"2024-07-27T06:43:12.803881+0000","last_unstale":"2024-07-27T06:43:15.793523+0000","last_undegraded":"2024-07-27T06:43:15.793523+0000","last_fullsized":"2024-07-27T06:43:15.793523+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:01:05.009233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811748+0000","last_change":"2024-07-27T06:43:12.809150+0000","last_active":"2024-07-27T06:43:15.811748+0000","last_peered":"2024-07-27T06:43:15.811748+0000","last_clean":"2024-07-27T06:43:15.811748+0000","last_became_active":"2024-07-27T06:43:12.808428+0000","last_became_peered":"2024-07-27T06:43:12.808428+0000","last_unstale":"2024-07-27T06:43:15.811748+0000","last_undegraded":"2024-07-27T06:43:15.811748+0000","last_fullsized":"2024-07-27T06:43:15.811748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:07:13.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810154+0000","last_change":"2024-07-27T06:43:12.793562+0000","last_active":"2024-07-27T06:43:15.810154+0000","last_peered":"2024-07-27T06:43:15.810154+0000","last_clean":"2024-07-27T06:43:15.810154+0000","last_became_active":"2024-07-27T06:43:12.793031+0000","last_became_peered":"2024-07-27T06:43:12.793031+0000","last_unstale":"2024-07-27T06:43:15.810154+0000","last_undegraded":"2024-07-27T06:43:15.810154+0000","last_fullsized":"2024-07-27T06:43:15.810154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:27:46.227980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811513+0000","last_change":"2024-07-27T06:43:12.797282+0000","last_active":"2024-07-27T06:43:15.811513+0000","last_peered":"2024-07-27T06:43:15.811513+0000","last_clean":"2024-07-27T06:43:15.811513+0000","last_became_active":"2024-07-27T06:43:12.796921+0000","last_became_peered":"2024-07-27T06:43:12.796921+0000","last_unstale":"2024-07-27T06:43:15.811513+0000","last_undegraded":"2024-07-27T06:43:15.811513+0000","last_fullsized":"2024-07-27T06:43:15.811513+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:19:39.940577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811833+0000","last_change":"2024-07-27T06:43:12.787866+0000","last_active":"2024-07-27T06:43:15.811833+0000","last_peered":"2024-07-27T06:43:15.811833+0000","last_clean":"2024-07-27T06:43:15.811833+0000","last_became_active":"2024-07-27T06:43:12.787513+0000","last_became_peered":"2024-07-27T06:43:12.787513+0000","last_unstale":"2024-07-27T06:43:15.811833+0000","last_undegraded":"2024-07-27T06:43:15.811833+0000","last_fullsized":"2024-07-27T06:43:15.811833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:48:57.876588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.809386+0000","last_change":"2024-07-27T06:43:12.791158+0000","last_active":"2024-07-27T06:43:15.809386+0000","last_peered":"2024-07-27T06:43:15.809386+0000","last_clean":"2024-07-27T06:43:15.809386+0000","last_became_active":"2024-07-27T06:43:12.790771+0000","last_became_peered":"2024-07-27T06:43:12.790771+0000","last_unstale":"2024-07-27T06:43:15.809386+0000","last_undegraded":"2024-07-27T06:43:15.809386+0000","last_fullsized":"2024-07-27T06:43:15.809386+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:10:18.739265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"79'6","reported_seq":58,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:57.928677+0000","last_change":"2024-07-27T06:43:09.769281+0000","last_active":"2024-07-27T06:43:57.928677+0000","last_peered":"2024-07-27T06:43:57.928677+0000","last_clean":"2024-07-27T06:43:57.928677+0000","last_became_active":"2024-07-27T06:43:09.768864+0000","last_became_peered":"2024-07-27T06:43:09.768864+0000","last_unstale":"2024-07-27T06:43:57.928677+0000","last_undegraded":"2024-07-27T06:43:57.928677+0000","last_fullsized":"2024-07-27T06:43:57.928677+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:55:14.191180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794374+0000","last_change":"2024-07-27T06:43:12.794484+0000","last_active":"2024-07-27T06:43:15.794374+0000","last_peered":"2024-07-27T06:43:15.794374+0000","last_clean":"2024-07-27T06:43:15.794374+0000","last_became_active":"2024-07-27T06:43:12.794147+0000","last_became_peered":"2024-07-27T06:43:12.794147+0000","last_unstale":"2024-07-27T06:43:15.794374+0000","last_undegraded":"2024-07-27T06:43:15.794374+0000","last_fullsized":"2024-07-27T06:43:15.794374+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T18:04:34.745156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:31.864525+0000","last_change":"2024-07-27T06:43:09.772227+0000","last_active":"2024-07-27T06:43:31.864525+0000","last_peered":"2024-07-27T06:43:31.864525+0000","last_clean":"2024-07-27T06:43:31.864525+0000","last_became_active":"2024-07-27T06:43:09.771812+0000","last_became_peered":"2024-07-27T06:43:09.771812+0000","last_unstale":"2024-07-27T06:43:31.864525+0000","last_undegraded":"2024-07-27T06:43:31.864525+0000","last_fullsized":"2024-07-27T06:43:31.864525+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:27:28.054590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:30.853588+0000","last_change":"2024-07-27T06:43:09.766490+0000","last_active":"2024-07-27T06:43:30.853588+0000","last_peered":"2024-07-27T06:43:30.853588+0000","last_clean":"2024-07-27T06:43:30.853588+0000","last_became_active":"2024-07-27T06:43:09.765973+0000","last_became_peered":"2024-07-27T06:43:09.765973+0000","last_unstale":"2024-07-27T06:43:30.853588+0000","last_undegraded":"2024-07-27T06:43:30.853588+0000","last_fullsized":"2024-07-27T06:43:30.853588+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:00:02.913080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808080+0000","last_change":"2024-07-27T06:43:12.805234+0000","last_active":"2024-07-27T06:43:15.808080+0000","last_peered":"2024-07-27T06:43:15.808080+0000","last_clean":"2024-07-27T06:43:15.808080+0000","last_became_active":"2024-07-27T06:43:12.804908+0000","last_became_peered":"2024-07-27T06:43:12.804908+0000","last_unstale":"2024-07-27T06:43:15.808080+0000","last_undegraded":"2024-07-27T06:43:15.808080+0000","last_fullsized":"2024-07-27T06:43:15.808080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:09:34.988755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.873019+0000","last_change":"2024-07-27T06:43:09.774201+0000","last_active":"2024-07-27T06:43:32.873019+0000","last_peered":"2024-07-27T06:43:32.873019+0000","last_clean":"2024-07-27T06:43:32.873019+0000","last_became_active":"2024-07-27T06:43:09.773817+0000","last_became_peered":"2024-07-27T06:43:09.773817+0000","last_unstale":"2024-07-27T06:43:32.873019+0000","last_undegraded":"2024-07-27T06:43:32.873019+0000","last_fullsized":"2024-07-27T06:43:32.873019+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:06.166251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.806897+0000","last_change":"2024-07-27T06:43:12.800174+0000","last_active":"2024-07-27T06:43:15.806897+0000","last_peered":"2024-07-27T06:43:15.806897+0000","last_clean":"2024-07-27T06:43:15.806897+0000","last_became_active":"2024-07-27T06:43:12.799769+0000","last_became_peered":"2024-07-27T06:43:12.799769+0000","last_unstale":"2024-07-27T06:43:15.806897+0000","last_undegraded":"2024-07-27T06:43:15.806897+0000","last_fullsized":"2024-07-27T06:43:15.806897+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:43:06.890372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":75,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:56.926397+0000","last_change":"2024-07-27T06:43:09.776612+0000","last_active":"2024-07-27T06:43:56.926397+0000","last_peered":"2024-07-27T06:43:56.926397+0000","last_clean":"2024-07-27T06:43:56.926397+0000","last_became_active":"2024-07-27T06:43:09.775761+0000","last_became_peered":"2024-07-27T06:43:09.775761+0000","last_unstale":"2024-07-27T06:43:56.926397+0000","last_undegraded":"2024-07-27T06:43:56.926397+0000","last_fullsized":"2024-07-27T06:43:56.926397+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:42:52.132357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":48,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810601+0000","last_change":"2024-07-27T06:43:12.797200+0000","last_active":"2024-07-27T06:43:15.810601+0000","last_peered":"2024-07-27T06:43:15.810601+0000","last_clean":"2024-07-27T06:43:15.810601+0000","last_became_active":"2024-07-27T06:43:12.796288+0000","last_became_peered":"2024-07-27T06:43:12.796288+0000","last_unstale":"2024-07-27T06:43:15.810601+0000","last_undegraded":"2024-07-27T06:43:15.810601+0000","last_fullsized":"2024-07-27T06:43:15.810601+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:31:27.603883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.873048+0000","last_change":"2024-07-27T06:43:09.772658+0000","last_active":"2024-07-27T06:43:32.873048+0000","last_peered":"2024-07-27T06:43:32.873048+0000","last_clean":"2024-07-27T06:43:32.873048+0000","last_became_active":"2024-07-27T06:43:09.772277+0000","last_became_peered":"2024-07-27T06:43:09.772277+0000","last_unstale":"2024-07-27T06:43:32.873048+0000","last_undegraded":"2024-07-27T06:43:32.873048+0000","last_fullsized":"2024-07-27T06:43:32.873048+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:09:16.854718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812087+0000","last_change":"2024-07-27T06:43:12.809643+0000","last_active":"2024-07-27T06:43:15.812087+0000","last_peered":"2024-07-27T06:43:15.812087+0000","last_clean":"2024-07-27T06:43:15.812087+0000","last_became_active":"2024-07-27T06:43:12.809020+0000","last_became_peered":"2024-07-27T06:43:12.809020+0000","last_unstale":"2024-07-27T06:43:15.812087+0000","last_undegraded":"2024-07-27T06:43:15.812087+0000","last_fullsized":"2024-07-27T06:43:15.812087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:08:03.062914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":29,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:31.863658+0000","last_change":"2024-07-27T06:43:10.534042+0000","last_active":"2024-07-27T06:43:31.863658+0000","last_peered":"2024-07-27T06:43:31.863658+0000","last_clean":"2024-07-27T06:43:31.863658+0000","last_became_active":"2024-07-27T06:43:10.533615+0000","last_became_peered":"2024-07-27T06:43:10.533615+0000","last_unstale":"2024-07-27T06:43:31.863658+0000","last_undegraded":"2024-07-27T06:43:31.863658+0000","last_fullsized":"2024-07-27T06:43:31.863658+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:09:32.235633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810039+0000","last_change":"2024-07-27T06:43:12.798764+0000","last_active":"2024-07-27T06:43:15.810039+0000","last_peered":"2024-07-27T06:43:15.810039+0000","last_clean":"2024-07-27T06:43:15.810039+0000","last_became_active":"2024-07-27T06:43:12.797500+0000","last_became_peered":"2024-07-27T06:43:12.797500+0000","last_unstale":"2024-07-27T06:43:15.810039+0000","last_undegraded":"2024-07-27T06:43:15.810039+0000","last_fullsized":"2024-07-27T06:43:15.810039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:12:09.368711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'30","reported_seq":109,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:56.931202+0000","last_change":"2024-07-27T06:43:09.775954+0000","last_active":"2024-07-27T06:43:56.931202+0000","last_peered":"2024-07-27T06:43:56.931202+0000","last_clean":"2024-07-27T06:43:56.931202+0000","last_became_active":"2024-07-27T06:43:09.775498+0000","last_became_peered":"2024-07-27T06:43:09.775498+0000","last_unstale":"2024-07-27T06:43:56.931202+0000","last_undegraded":"2024-07-27T06:43:56.931202+0000","last_fullsized":"2024-07-27T06:43:56.931202+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:08:46.965525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56090,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":56,"num_read_kb":82,"num_write":32,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808773+0000","last_change":"2024-07-27T06:43:12.797220+0000","last_active":"2024-07-27T06:43:15.808773+0000","last_peered":"2024-07-27T06:43:15.808773+0000","last_clean":"2024-07-27T06:43:15.808773+0000","last_became_active":"2024-07-27T06:43:12.796889+0000","last_became_peered":"2024-07-27T06:43:12.796889+0000","last_unstale":"2024-07-27T06:43:15.808773+0000","last_undegraded":"2024-07-27T06:43:15.808773+0000","last_fullsized":"2024-07-27T06:43:15.808773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:24:36.275738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:33.884067+0000","last_change":"2024-07-27T06:43:09.776600+0000","last_active":"2024-07-27T06:43:33.884067+0000","last_peered":"2024-07-27T06:43:33.884067+0000","last_clean":"2024-07-27T06:43:33.884067+0000","last_became_active":"2024-07-27T06:43:09.776163+0000","last_became_peered":"2024-07-27T06:43:09.776163+0000","last_unstale":"2024-07-27T06:43:33.884067+0000","last_undegraded":"2024-07-27T06:43:33.884067+0000","last_fullsized":"2024-07-27T06:43:33.884067+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:22:15.947916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.793174+0000","last_change":"2024-07-27T06:43:12.804984+0000","last_active":"2024-07-27T06:43:15.793174+0000","last_peered":"2024-07-27T06:43:15.793174+0000","last_clean":"2024-07-27T06:43:15.793174+0000","last_became_active":"2024-07-27T06:43:12.804644+0000","last_became_peered":"2024-07-27T06:43:12.804644+0000","last_unstale":"2024-07-27T06:43:15.793174+0000","last_undegraded":"2024-07-27T06:43:15.793174+0000","last_fullsized":"2024-07-27T06:43:15.793174+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.355981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":29,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.872774+0000","last_change":"2024-07-27T06:43:09.772489+0000","last_active":"2024-07-27T06:43:32.872774+0000","last_peered":"2024-07-27T06:43:32.872774+0000","last_clean":"2024-07-27T06:43:32.872774+0000","last_became_active":"2024-07-27T06:43:09.772069+0000","last_became_peered":"2024-07-27T06:43:09.772069+0000","last_unstale":"2024-07-27T06:43:32.872774+0000","last_undegraded":"2024-07-27T06:43:32.872774+0000","last_fullsized":"2024-07-27T06:43:32.872774+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:15:30.549971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810397+0000","last_change":"2024-07-27T06:43:12.803376+0000","last_active":"2024-07-27T06:43:15.810397+0000","last_peered":"2024-07-27T06:43:15.810397+0000","last_clean":"2024-07-27T06:43:15.810397+0000","last_became_active":"2024-07-27T06:43:12.802961+0000","last_became_peered":"2024-07-27T06:43:12.802961+0000","last_unstale":"2024-07-27T06:43:15.810397+0000","last_undegraded":"2024-07-27T06:43:15.810397+0000","last_fullsized":"2024-07-27T06:43:15.810397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:00:24.649877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":57,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:58.929501+0000","last_change":"2024-07-27T06:43:09.773182+0000","last_active":"2024-07-27T06:43:58.929501+0000","last_peered":"2024-07-27T06:43:58.929501+0000","last_clean":"2024-07-27T06:43:58.929501+0000","last_became_active":"2024-07-27T06:43:09.772126+0000","last_became_peered":"2024-07-27T06:43:09.772126+0000","last_unstale":"2024-07-27T06:43:58.929501+0000","last_undegraded":"2024-07-27T06:43:58.929501+0000","last_fullsized":"2024-07-27T06:43:58.929501+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:45:26.598799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.809741+0000","last_change":"2024-07-27T06:43:12.799385+0000","last_active":"2024-07-27T06:43:15.809741+0000","last_peered":"2024-07-27T06:43:15.809741+0000","last_clean":"2024-07-27T06:43:15.809741+0000","last_became_active":"2024-07-27T06:43:12.798639+0000","last_became_peered":"2024-07-27T06:43:12.798639+0000","last_unstale":"2024-07-27T06:43:15.809741+0000","last_undegraded":"2024-07-27T06:43:15.809741+0000","last_fullsized":"2024-07-27T06:43:15.809741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:56:29.462491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":41,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.872940+0000","last_change":"2024-07-27T06:43:10.535429+0000","last_active":"2024-07-27T06:43:32.872940+0000","last_peered":"2024-07-27T06:43:32.872940+0000","last_clean":"2024-07-27T06:43:32.872940+0000","last_became_active":"2024-07-27T06:43:10.534952+0000","last_became_peered":"2024-07-27T06:43:10.534952+0000","last_unstale":"2024-07-27T06:43:32.872940+0000","last_undegraded":"2024-07-27T06:43:32.872940+0000","last_fullsized":"2024-07-27T06:43:32.872940+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:54:20.018263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794089+0000","last_change":"2024-07-27T06:43:12.804266+0000","last_active":"2024-07-27T06:43:15.794089+0000","last_peered":"2024-07-27T06:43:15.794089+0000","last_clean":"2024-07-27T06:43:15.794089+0000","last_became_active":"2024-07-27T06:43:12.803888+0000","last_became_peered":"2024-07-27T06:43:12.803888+0000","last_unstale":"2024-07-27T06:43:15.794089+0000","last_undegraded":"2024-07-27T06:43:15.794089+0000","last_fullsized":"2024-07-27T06:43:15.794089+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:48:00.490592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.848213+0000","last_change":"2024-07-27T06:43:09.777656+0000","last_active":"2024-07-27T06:43:32.848213+0000","last_peered":"2024-07-27T06:43:32.848213+0000","last_clean":"2024-07-27T06:43:32.848213+0000","last_became_active":"2024-07-27T06:43:09.776620+0000","last_became_peered":"2024-07-27T06:43:09.776620+0000","last_unstale":"2024-07-27T06:43:32.848213+0000","last_undegraded":"2024-07-27T06:43:32.848213+0000","last_fullsized":"2024-07-27T06:43:32.848213+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:50.203623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811868+0000","last_change":"2024-07-27T06:43:12.796179+0000","last_active":"2024-07-27T06:43:15.811868+0000","last_peered":"2024-07-27T06:43:15.811868+0000","last_clean":"2024-07-27T06:43:15.811868+0000","last_became_active":"2024-07-27T06:43:12.795701+0000","last_became_peered":"2024-07-27T06:43:12.795701+0000","last_unstale":"2024-07-27T06:43:15.811868+0000","last_undegraded":"2024-07-27T06:43:15.811868+0000","last_fullsized":"2024-07-27T06:43:15.811868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:20:12.236769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":28,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.871192+0000","last_change":"2024-07-27T06:43:09.769949+0000","last_active":"2024-07-27T06:43:32.871192+0000","last_peered":"2024-07-27T06:43:32.871192+0000","last_clean":"2024-07-27T06:43:32.871192+0000","last_became_active":"2024-07-27T06:43:09.769602+0000","last_became_peered":"2024-07-27T06:43:09.769602+0000","last_unstale":"2024-07-27T06:43:32.871192+0000","last_undegraded":"2024-07-27T06:43:32.871192+0000","last_fullsized":"2024-07-27T06:43:32.871192+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:28:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810922+0000","last_change":"2024-07-27T06:43:12.802230+0000","last_active":"2024-07-27T06:43:15.810922+0000","last_peered":"2024-07-27T06:43:15.810922+0000","last_clean":"2024-07-27T06:43:15.810922+0000","last_became_active":"2024-07-27T06:43:12.800944+0000","last_became_peered":"2024-07-27T06:43:12.800944+0000","last_unstale":"2024-07-27T06:43:15.810922+0000","last_undegraded":"2024-07-27T06:43:15.810922+0000","last_fullsized":"2024-07-27T06:43:15.810922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:05:54.585955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.871177+0000","last_change":"2024-07-27T06:43:09.768239+0000","last_active":"2024-07-27T06:43:32.871177+0000","last_peered":"2024-07-27T06:43:32.871177+0000","last_clean":"2024-07-27T06:43:32.871177+0000","last_became_active":"2024-07-27T06:43:09.767584+0000","last_became_peered":"2024-07-27T06:43:09.767584+0000","last_unstale":"2024-07-27T06:43:32.871177+0000","last_undegraded":"2024-07-27T06:43:32.871177+0000","last_fullsized":"2024-07-27T06:43:32.871177+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:35:39.552246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"79'1","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810434+0000","last_change":"2024-07-27T06:43:12.803779+0000","last_active":"2024-07-27T06:43:15.810434+0000","last_peered":"2024-07-27T06:43:15.810434+0000","last_clean":"2024-07-27T06:43:15.810434+0000","last_became_active":"2024-07-27T06:43:12.803359+0000","last_became_peered":"2024-07-27T06:43:12.803359+0000","last_unstale":"2024-07-27T06:43:15.810434+0000","last_undegraded":"2024-07-27T06:43:15.810434+0000","last_fullsized":"2024-07-27T06:43:15.810434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:32:53.167039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'6","reported_seq":60,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:44:00.000894+0000","last_change":"2024-07-27T06:43:10.533751+0000","last_active":"2024-07-27T06:44:00.000894+0000","last_peered":"2024-07-27T06:44:00.000894+0000","last_clean":"2024-07-27T06:44:00.000894+0000","last_became_active":"2024-07-27T06:43:10.533245+0000","last_became_peered":"2024-07-27T06:43:10.533245+0000","last_unstale":"2024-07-27T06:44:00.000894+0000","last_undegraded":"2024-07-27T06:44:00.000894+0000","last_fullsized":"2024-07-27T06:44:00.000894+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:43:01.469725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810147+0000","last_change":"2024-07-27T06:43:12.805830+0000","last_active":"2024-07-27T06:43:15.810147+0000","last_peered":"2024-07-27T06:43:15.810147+0000","last_clean":"2024-07-27T06:43:15.810147+0000","last_became_active":"2024-07-27T06:43:12.805324+0000","last_became_peered":"2024-07-27T06:43:12.805324+0000","last_unstale":"2024-07-27T06:43:15.810147+0000","last_undegraded":"2024-07-27T06:43:15.810147+0000","last_fullsized":"2024-07-27T06:43:15.810147+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:38:31.430676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":99,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810142+0000","last_change":"2024-07-27T06:41:43.549132+0000","last_active":"2024-07-27T06:43:15.810142+0000","last_peered":"2024-07-27T06:43:15.810142+0000","last_clean":"2024-07-27T06:43:15.810142+0000","last_became_active":"2024-07-27T06:41:43.542635+0000","last_became_peered":"2024-07-27T06:41:43.542635+0000","last_unstale":"2024-07-27T06:43:15.810142+0000","last_undegraded":"2024-07-27T06:43:15.810142+0000","last_fullsized":"2024-07-27T06:43:15.810142+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_clean_scrub_stamp":"2024-07-27T06:38:21.639907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:10:06.417387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:33.885316+0000","last_change":"2024-07-27T06:43:09.768774+0000","last_active":"2024-07-27T06:43:33.885316+0000","last_peered":"2024-07-27T06:43:33.885316+0000","last_clean":"2024-07-27T06:43:33.885316+0000","last_became_active":"2024-07-27T06:43:09.768345+0000","last_became_peered":"2024-07-27T06:43:09.768345+0000","last_unstale":"2024-07-27T06:43:33.885316+0000","last_undegraded":"2024-07-27T06:43:33.885316+0000","last_fullsized":"2024-07-27T06:43:33.885316+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:40:49.411746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808005+0000","last_change":"2024-07-27T06:43:12.797817+0000","last_active":"2024-07-27T06:43:15.808005+0000","last_peered":"2024-07-27T06:43:15.808005+0000","last_clean":"2024-07-27T06:43:15.808005+0000","last_became_active":"2024-07-27T06:43:12.797483+0000","last_became_peered":"2024-07-27T06:43:12.797483+0000","last_unstale":"2024-07-27T06:43:15.808005+0000","last_undegraded":"2024-07-27T06:43:15.808005+0000","last_fullsized":"2024-07-27T06:43:15.808005+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:24:44.539505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:30.853385+0000","last_change":"2024-07-27T06:43:09.776880+0000","last_active":"2024-07-27T06:43:30.853385+0000","last_peered":"2024-07-27T06:43:30.853385+0000","last_clean":"2024-07-27T06:43:30.853385+0000","last_became_active":"2024-07-27T06:43:09.776486+0000","last_became_peered":"2024-07-27T06:43:09.776486+0000","last_unstale":"2024-07-27T06:43:30.853385+0000","last_undegraded":"2024-07-27T06:43:30.853385+0000","last_fullsized":"2024-07-27T06:43:30.853385+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:55:25.436405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.793149+0000","last_change":"2024-07-27T06:43:12.799755+0000","last_active":"2024-07-27T06:43:15.793149+0000","last_peered":"2024-07-27T06:43:15.793149+0000","last_clean":"2024-07-27T06:43:15.793149+0000","last_became_active":"2024-07-27T06:43:12.799412+0000","last_became_peered":"2024-07-27T06:43:12.799412+0000","last_unstale":"2024-07-27T06:43:15.793149+0000","last_undegraded":"2024-07-27T06:43:15.793149+0000","last_fullsized":"2024-07-27T06:43:15.793149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:45:50.545431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":57,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:59.930350+0000","last_change":"2024-07-27T06:43:10.533039+0000","last_active":"2024-07-27T06:43:59.930350+0000","last_peered":"2024-07-27T06:43:59.930350+0000","last_clean":"2024-07-27T06:43:59.930350+0000","last_became_active":"2024-07-27T06:43:10.532583+0000","last_became_peered":"2024-07-27T06:43:10.532583+0000","last_unstale":"2024-07-27T06:43:59.930350+0000","last_undegraded":"2024-07-27T06:43:59.930350+0000","last_fullsized":"2024-07-27T06:43:59.930350+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:04:51.954463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794070+0000","last_change":"2024-07-27T06:43:12.795452+0000","last_active":"2024-07-27T06:43:15.794070+0000","last_peered":"2024-07-27T06:43:15.794070+0000","last_clean":"2024-07-27T06:43:15.794070+0000","last_became_active":"2024-07-27T06:43:12.795019+0000","last_became_peered":"2024-07-27T06:43:12.795019+0000","last_unstale":"2024-07-27T06:43:15.794070+0000","last_undegraded":"2024-07-27T06:43:15.794070+0000","last_fullsized":"2024-07-27T06:43:15.794070+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:03:18.016269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":55,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:55.937018+0000","last_change":"2024-07-27T06:43:09.776214+0000","last_active":"2024-07-27T06:43:55.937018+0000","last_peered":"2024-07-27T06:43:55.937018+0000","last_clean":"2024-07-27T06:43:55.937018+0000","last_became_active":"2024-07-27T06:43:09.775719+0000","last_became_peered":"2024-07-27T06:43:09.775719+0000","last_unstale":"2024-07-27T06:43:55.937018+0000","last_undegraded":"2024-07-27T06:43:55.937018+0000","last_fullsized":"2024-07-27T06:43:55.937018+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:57:20.381599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.806627+0000","last_change":"2024-07-27T06:43:12.794909+0000","last_active":"2024-07-27T06:43:15.806627+0000","last_peered":"2024-07-27T06:43:15.806627+0000","last_clean":"2024-07-27T06:43:15.806627+0000","last_became_active":"2024-07-27T06:43:12.794497+0000","last_became_peered":"2024-07-27T06:43:12.794497+0000","last_unstale":"2024-07-27T06:43:15.806627+0000","last_undegraded":"2024-07-27T06:43:15.806627+0000","last_fullsized":"2024-07-27T06:43:15.806627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:42:49.760126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.807290+0000","last_change":"2024-07-27T06:43:10.534240+0000","last_active":"2024-07-27T06:43:15.807290+0000","last_peered":"2024-07-27T06:43:15.807290+0000","last_clean":"2024-07-27T06:43:15.807290+0000","last_became_active":"2024-07-27T06:43:10.533596+0000","last_became_peered":"2024-07-27T06:43:10.533596+0000","last_unstale":"2024-07-27T06:43:15.807290+0000","last_undegraded":"2024-07-27T06:43:15.807290+0000","last_fullsized":"2024-07-27T06:43:15.807290+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:16.098839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.806800+0000","last_change":"2024-07-27T06:43:12.805248+0000","last_active":"2024-07-27T06:43:15.806800+0000","last_peered":"2024-07-27T06:43:15.806800+0000","last_clean":"2024-07-27T06:43:15.806800+0000","last_became_active":"2024-07-27T06:43:12.803693+0000","last_became_peered":"2024-07-27T06:43:12.803693+0000","last_unstale":"2024-07-27T06:43:15.806800+0000","last_undegraded":"2024-07-27T06:43:15.806800+0000","last_fullsized":"2024-07-27T06:43:15.806800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:17.466465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":28,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.873421+0000","last_change":"2024-07-27T06:43:10.533701+0000","last_active":"2024-07-27T06:43:32.873421+0000","last_peered":"2024-07-27T06:43:32.873421+0000","last_clean":"2024-07-27T06:43:32.873421+0000","last_became_active":"2024-07-27T06:43:10.533109+0000","last_became_peered":"2024-07-27T06:43:10.533109+0000","last_unstale":"2024-07-27T06:43:32.873421+0000","last_undegraded":"2024-07-27T06:43:32.873421+0000","last_fullsized":"2024-07-27T06:43:32.873421+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:32:52.906691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.793621+0000","last_change":"2024-07-27T06:43:12.798479+0000","last_active":"2024-07-27T06:43:15.793621+0000","last_peered":"2024-07-27T06:43:15.793621+0000","last_clean":"2024-07-27T06:43:15.793621+0000","last_became_active":"2024-07-27T06:43:12.798124+0000","last_became_peered":"2024-07-27T06:43:12.798124+0000","last_unstale":"2024-07-27T06:43:15.793621+0000","last_undegraded":"2024-07-27T06:43:15.793621+0000","last_fullsized":"2024-07-27T06:43:15.793621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:14:55.267213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"79'14","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:53.810521+0000","last_change":"2024-07-27T06:43:09.770122+0000","last_active":"2024-07-27T06:43:53.810521+0000","last_peered":"2024-07-27T06:43:53.810521+0000","last_clean":"2024-07-27T06:43:53.810521+0000","last_became_active":"2024-07-27T06:43:09.769784+0000","last_became_peered":"2024-07-27T06:43:09.769784+0000","last_unstale":"2024-07-27T06:43:53.810521+0000","last_undegraded":"2024-07-27T06:43:53.810521+0000","last_fullsized":"2024-07-27T06:43:53.810521+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:47:00.398135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808033+0000","last_change":"2024-07-27T06:43:12.799603+0000","last_active":"2024-07-27T06:43:15.808033+0000","last_peered":"2024-07-27T06:43:15.808033+0000","last_clean":"2024-07-27T06:43:15.808033+0000","last_became_active":"2024-07-27T06:43:12.799189+0000","last_became_peered":"2024-07-27T06:43:12.799189+0000","last_unstale":"2024-07-27T06:43:15.808033+0000","last_undegraded":"2024-07-27T06:43:15.808033+0000","last_fullsized":"2024-07-27T06:43:15.808033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:32:30.603258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.872453+0000","last_change":"2024-07-27T06:43:10.004726+0000","last_active":"2024-07-27T06:43:32.872453+0000","last_peered":"2024-07-27T06:43:32.872453+0000","last_clean":"2024-07-27T06:43:32.872453+0000","last_became_active":"2024-07-27T06:43:10.004367+0000","last_became_peered":"2024-07-27T06:43:10.004367+0000","last_unstale":"2024-07-27T06:43:32.872453+0000","last_undegraded":"2024-07-27T06:43:32.872453+0000","last_fullsized":"2024-07-27T06:43:32.872453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:22:39.942391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812071+0000","last_change":"2024-07-27T06:43:12.796511+0000","last_active":"2024-07-27T06:43:15.812071+0000","last_peered":"2024-07-27T06:43:15.812071+0000","last_clean":"2024-07-27T06:43:15.812071+0000","last_became_active":"2024-07-27T06:43:12.796152+0000","last_became_peered":"2024-07-27T06:43:12.796152+0000","last_unstale":"2024-07-27T06:43:15.812071+0000","last_undegraded":"2024-07-27T06:43:15.812071+0000","last_fullsized":"2024-07-27T06:43:15.812071+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:03:02.508990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:29.847443+0000","last_change":"2024-07-27T06:43:10.533656+0000","last_active":"2024-07-27T06:43:29.847443+0000","last_peered":"2024-07-27T06:43:29.847443+0000","last_clean":"2024-07-27T06:43:29.847443+0000","last_became_active":"2024-07-27T06:43:10.533295+0000","last_became_peered":"2024-07-27T06:43:10.533295+0000","last_unstale":"2024-07-27T06:43:29.847443+0000","last_undegraded":"2024-07-27T06:43:29.847443+0000","last_fullsized":"2024-07-27T06:43:29.847443+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:11:00.032511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.807280+0000","last_change":"2024-07-27T06:43:12.795106+0000","last_active":"2024-07-27T06:43:15.807280+0000","last_peered":"2024-07-27T06:43:15.807280+0000","last_clean":"2024-07-27T06:43:15.807280+0000","last_became_active":"2024-07-27T06:43:12.794729+0000","last_became_peered":"2024-07-27T06:43:12.794729+0000","last_unstale":"2024-07-27T06:43:15.807280+0000","last_undegraded":"2024-07-27T06:43:15.807280+0000","last_fullsized":"2024-07-27T06:43:15.807280+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:19:16.542363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.871541+0000","last_change":"2024-07-27T06:43:09.770016+0000","last_active":"2024-07-27T06:43:32.871541+0000","last_peered":"2024-07-27T06:43:32.871541+0000","last_clean":"2024-07-27T06:43:32.871541+0000","last_became_active":"2024-07-27T06:43:09.769672+0000","last_became_peered":"2024-07-27T06:43:09.769672+0000","last_unstale":"2024-07-27T06:43:32.871541+0000","last_undegraded":"2024-07-27T06:43:32.871541+0000","last_fullsized":"2024-07-27T06:43:32.871541+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:30:55.737494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812064+0000","last_change":"2024-07-27T06:43:12.793690+0000","last_active":"2024-07-27T06:43:15.812064+0000","last_peered":"2024-07-27T06:43:15.812064+0000","last_clean":"2024-07-27T06:43:15.812064+0000","last_became_active":"2024-07-27T06:43:12.793352+0000","last_became_peered":"2024-07-27T06:43:12.793352+0000","last_unstale":"2024-07-27T06:43:15.812064+0000","last_undegraded":"2024-07-27T06:43:15.812064+0000","last_fullsized":"2024-07-27T06:43:15.812064+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:22:47.606990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:31.864434+0000","last_change":"2024-07-27T06:43:09.770022+0000","last_active":"2024-07-27T06:43:31.864434+0000","last_peered":"2024-07-27T06:43:31.864434+0000","last_clean":"2024-07-27T06:43:31.864434+0000","last_became_active":"2024-07-27T06:43:09.769559+0000","last_became_peered":"2024-07-27T06:43:09.769559+0000","last_unstale":"2024-07-27T06:43:31.864434+0000","last_undegraded":"2024-07-27T06:43:31.864434+0000","last_fullsized":"2024-07-27T06:43:31.864434+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:33:01.252989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808333+0000","last_change":"2024-07-27T06:43:12.792648+0000","last_active":"2024-07-27T06:43:15.808333+0000","last_peered":"2024-07-27T06:43:15.808333+0000","last_clean":"2024-07-27T06:43:15.808333+0000","last_became_active":"2024-07-27T06:43:12.792338+0000","last_became_peered":"2024-07-27T06:43:12.792338+0000","last_unstale":"2024-07-27T06:43:15.808333+0000","last_undegraded":"2024-07-27T06:43:15.808333+0000","last_fullsized":"2024-07-27T06:43:15.808333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:36:52.605957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":28,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:29.847280+0000","last_change":"2024-07-27T06:43:09.772603+0000","last_active":"2024-07-27T06:43:29.847280+0000","last_peered":"2024-07-27T06:43:29.847280+0000","last_clean":"2024-07-27T06:43:29.847280+0000","last_became_active":"2024-07-27T06:43:09.772239+0000","last_became_peered":"2024-07-27T06:43:09.772239+0000","last_unstale":"2024-07-27T06:43:29.847280+0000","last_undegraded":"2024-07-27T06:43:29.847280+0000","last_fullsized":"2024-07-27T06:43:29.847280+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:33:32.905030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810927+0000","last_change":"2024-07-27T06:43:12.809347+0000","last_active":"2024-07-27T06:43:15.810927+0000","last_peered":"2024-07-27T06:43:15.810927+0000","last_clean":"2024-07-27T06:43:15.810927+0000","last_became_active":"2024-07-27T06:43:12.808980+0000","last_became_peered":"2024-07-27T06:43:12.808980+0000","last_unstale":"2024-07-27T06:43:15.810927+0000","last_undegraded":"2024-07-27T06:43:15.810927+0000","last_fullsized":"2024-07-27T06:43:15.810927+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:05:14.132641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794469+0000","last_change":"2024-07-27T06:43:12.803767+0000","last_active":"2024-07-27T06:43:15.794469+0000","last_peered":"2024-07-27T06:43:15.794469+0000","last_clean":"2024-07-27T06:43:15.794469+0000","last_became_active":"2024-07-27T06:43:12.803420+0000","last_became_peered":"2024-07-27T06:43:12.803420+0000","last_unstale":"2024-07-27T06:43:15.794469+0000","last_undegraded":"2024-07-27T06:43:15.794469+0000","last_fullsized":"2024-07-27T06:43:15.794469+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:09:56.623458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":29,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.872773+0000","last_change":"2024-07-27T06:43:09.776000+0000","last_active":"2024-07-27T06:43:32.872773+0000","last_peered":"2024-07-27T06:43:32.872773+0000","last_clean":"2024-07-27T06:43:32.872773+0000","last_became_active":"2024-07-27T06:43:09.775522+0000","last_became_peered":"2024-07-27T06:43:09.775522+0000","last_unstale":"2024-07-27T06:43:32.872773+0000","last_undegraded":"2024-07-27T06:43:32.872773+0000","last_fullsized":"2024-07-27T06:43:32.872773+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:39:42.650177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.809785+0000","last_change":"2024-07-27T06:43:12.807062+0000","last_active":"2024-07-27T06:43:15.809785+0000","last_peered":"2024-07-27T06:43:15.809785+0000","last_clean":"2024-07-27T06:43:15.809785+0000","last_became_active":"2024-07-27T06:43:12.805816+0000","last_became_peered":"2024-07-27T06:43:12.805816+0000","last_unstale":"2024-07-27T06:43:15.809785+0000","last_undegraded":"2024-07-27T06:43:15.809785+0000","last_fullsized":"2024-07-27T06:43:15.809785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:44:16.261663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:33.884014+0000","last_change":"2024-07-27T06:43:09.776285+0000","last_active":"2024-07-27T06:43:33.884014+0000","last_peered":"2024-07-27T06:43:33.884014+0000","last_clean":"2024-07-27T06:43:33.884014+0000","last_became_active":"2024-07-27T06:43:09.775055+0000","last_became_peered":"2024-07-27T06:43:09.775055+0000","last_unstale":"2024-07-27T06:43:33.884014+0000","last_undegraded":"2024-07-27T06:43:33.884014+0000","last_fullsized":"2024-07-27T06:43:33.884014+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:30:44.728547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812075+0000","last_change":"2024-07-27T06:43:12.809135+0000","last_active":"2024-07-27T06:43:15.812075+0000","last_peered":"2024-07-27T06:43:15.812075+0000","last_clean":"2024-07-27T06:43:15.812075+0000","last_became_active":"2024-07-27T06:43:12.806731+0000","last_became_peered":"2024-07-27T06:43:12.806731+0000","last_unstale":"2024-07-27T06:43:15.812075+0000","last_undegraded":"2024-07-27T06:43:15.812075+0000","last_fullsized":"2024-07-27T06:43:15.812075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:47:50.397517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.818330+0000","last_change":"2024-07-27T06:43:10.533608+0000","last_active":"2024-07-27T06:43:15.818330+0000","last_peered":"2024-07-27T06:43:15.818330+0000","last_clean":"2024-07-27T06:43:15.818330+0000","last_became_active":"2024-07-27T06:43:10.532663+0000","last_became_peered":"2024-07-27T06:43:10.532663+0000","last_unstale":"2024-07-27T06:43:15.818330+0000","last_undegraded":"2024-07-27T06:43:15.818330+0000","last_fullsized":"2024-07-27T06:43:15.818330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:05:33.155427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808322+0000","last_change":"2024-07-27T06:43:12.789890+0000","last_active":"2024-07-27T06:43:15.808322+0000","last_peered":"2024-07-27T06:43:15.808322+0000","last_clean":"2024-07-27T06:43:15.808322+0000","last_became_active":"2024-07-27T06:43:12.789561+0000","last_became_peered":"2024-07-27T06:43:12.789561+0000","last_unstale":"2024-07-27T06:43:15.808322+0000","last_undegraded":"2024-07-27T06:43:15.808322+0000","last_fullsized":"2024-07-27T06:43:15.808322+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:35:49.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":61,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:55.936436+0000","last_change":"2024-07-27T06:43:09.773471+0000","last_active":"2024-07-27T06:43:55.936436+0000","last_peered":"2024-07-27T06:43:55.936436+0000","last_clean":"2024-07-27T06:43:55.936436+0000","last_became_active":"2024-07-27T06:43:09.773213+0000","last_became_peered":"2024-07-27T06:43:09.773213+0000","last_unstale":"2024-07-27T06:43:55.936436+0000","last_undegraded":"2024-07-27T06:43:55.936436+0000","last_fullsized":"2024-07-27T06:43:55.936436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:54:38.770198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63685,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":294,"num_read_kb":215,"num_write":243,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":191055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283116,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678041,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":426724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874268,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1073452,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037795,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":888832,"data_stored":426263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234024,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":426641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430251,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":1019550,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593778,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":872448,"data_stored":426371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790007,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":1018837,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953532,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":872448,"data_stored":425841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":425805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378695,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":483985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542219,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":929792,"data_stored":483374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705744,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":427710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":58233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":57826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T06:44:03.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph pg dump --format=json 2024-07-27T06:44:04.044 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:04.424 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:04 smithi029 ceph-mon[32610]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:44:04.424 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:04.425 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:04.425 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:04.425 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:04 smithi077 ceph-mon[37153]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:44:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:04 smithi047 ceph-mon[36835]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:44:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:04.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:05 smithi029 ceph-mon[32610]: pgmap v305: 97 pgs: 97 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-07-27T06:44:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:05 smithi047 ceph-mon[36835]: pgmap v305: 97 pgs: 97 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-07-27T06:44:05.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.735+0000 7f65d3b20640 1 -- 172.21.15.29:0/1145013233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc106d20 msgr2=0x7f65cc107130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:05.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.736+0000 7f65d3b20640 1 --2- 172.21.15.29:0/1145013233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc106d20 0x7f65cc107130 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f65bc009900 tx=0x7f65bc02a410 comp rx=0 tx=0).stop 2024-07-27T06:44:05.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.736+0000 7f65d3b20640 1 -- 172.21.15.29:0/1145013233 shutdown_connections 2024-07-27T06:44:05.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.736+0000 7f65d3b20640 1 --2- 172.21.15.29:0/1145013233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65cc107670 0x7f65cc10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:05.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.736+0000 7f65d3b20640 1 --2- 172.21.15.29:0/1145013233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc106d20 0x7f65cc107130 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:05.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.736+0000 7f65d3b20640 1 --2- 172.21.15.29:0/1145013233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65cc108030 0x7f65cc108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:05.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.736+0000 7f65d3b20640 1 -- 172.21.15.29:0/1145013233 >> 172.21.15.29:0/1145013233 conn(0x7f65cc077230 msgr2=0x7f65cc077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:05.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.736+0000 7f65d3b20640 1 -- 172.21.15.29:0/1145013233 shutdown_connections 2024-07-27T06:44:05.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.736+0000 7f65d3b20640 1 -- 172.21.15.29:0/1145013233 wait complete. 2024-07-27T06:44:05.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.736+0000 7f65d3b20640 1 Processor -- start 2024-07-27T06:44:05.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.736+0000 7f65d3b20640 1 -- start start 2024-07-27T06:44:05.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d3b20640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65cc106d20 0x7f65cc10fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:05.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d3b20640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65cc107670 0x7f65cc10ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d3b20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc108030 0x7f65cc1104b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d3b20640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f65cc113500 con 0x7f65cc108030 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d3b20640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f65cc113380 con 0x7f65cc107670 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d3b20640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f65cc113680 con 0x7f65cc106d20 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d20bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc108030 0x7f65cc1104b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d20bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc108030 0x7f65cc1104b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34052/0 (socket says 172.21.15.29:34052) 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d20bd640 1 -- 172.21.15.29:0/606639294 learned_addr learned my addr 172.21.15.29:0/606639294 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d18bc640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65cc106d20 0x7f65cc10fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d10bb640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65cc107670 0x7f65cc10ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d20bd640 1 -- 172.21.15.29:0/606639294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65cc106d20 msgr2=0x7f65cc10fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d20bd640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65cc106d20 0x7f65cc10fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d20bd640 1 -- 172.21.15.29:0/606639294 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65cc107670 msgr2=0x7f65cc10ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d20bd640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65cc107670 0x7f65cc10ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:05.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d20bd640 1 -- 172.21.15.29:0/606639294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65cc1a8230 con 0x7f65cc108030 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.737+0000 7f65d20bd640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc108030 0x7f65cc1104b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009fd0 tx=0x7f65c400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.738+0000 7f65c2ffd640 1 -- 172.21.15.29:0/606639294 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c4017070 con 0x7f65cc108030 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.738+0000 7f65d3b20640 1 -- 172.21.15.29:0/606639294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65cc1a8520 con 0x7f65cc108030 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.738+0000 7f65c2ffd640 1 -- 172.21.15.29:0/606639294 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65c4003e20 con 0x7f65cc108030 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.738+0000 7f65c2ffd640 1 -- 172.21.15.29:0/606639294 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c4010460 con 0x7f65cc108030 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.738+0000 7f65d3b20640 1 -- 172.21.15.29:0/606639294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65cc1a89e0 con 0x7f65cc108030 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.739+0000 7f65d3b20640 1 -- 172.21.15.29:0/606639294 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65cc107130 con 0x7f65cc108030 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.741+0000 7f65c2ffd640 1 -- 172.21.15.29:0/606639294 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f65c4003970 con 0x7f65cc108030 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.742+0000 7f65c2ffd640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f65b80b3cb0 0x7f65b80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.742+0000 7f65d18bc640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f65b80b3cb0 0x7f65b80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.742+0000 7f65c2ffd640 1 -- 172.21.15.29:0/606639294 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7f65c40e3ba0 con 0x7f65cc108030 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.742+0000 7f65d18bc640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f65b80b3cb0 0x7f65b80b61c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f65cc111480 tx=0x7f65c800b410 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.745+0000 7f65c2ffd640 1 -- 172.21.15.29:0/606639294 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f65c40ab050 con 0x7f65cc108030 2024-07-27T06:44:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:05 smithi077 ceph-mon[37153]: pgmap v305: 97 pgs: 97 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-07-27T06:44:05.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.932+0000 7f65d3b20640 1 -- 172.21.15.29:0/606639294 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f65cc063230 con 0x7f65b80b3cb0 2024-07-27T06:44:05.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.941+0000 7f65c2ffd640 1 -- 172.21.15.29:0/606639294 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258902 (secure 0 0 0) 0x7f65cc063230 con 0x7f65b80b3cb0 2024-07-27T06:44:05.943 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:44:05.945 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-07-27T06:44:05.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.944+0000 7f65d3b20640 1 -- 172.21.15.29:0/606639294 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f65b80b3cb0 msgr2=0x7f65b80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:05.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.944+0000 7f65d3b20640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f65b80b3cb0 0x7f65b80b61c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f65cc111480 tx=0x7f65c800b410 comp rx=0 tx=0).stop 2024-07-27T06:44:05.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.944+0000 7f65d3b20640 1 -- 172.21.15.29:0/606639294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc108030 msgr2=0x7f65cc1104b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:05.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.944+0000 7f65d3b20640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc108030 0x7f65cc1104b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009fd0 tx=0x7f65c400b260 comp rx=0 tx=0).stop 2024-07-27T06:44:05.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.944+0000 7f65d3b20640 1 -- 172.21.15.29:0/606639294 shutdown_connections 2024-07-27T06:44:05.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.944+0000 7f65d3b20640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc108030 0x7f65cc1104b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:05.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.944+0000 7f65d3b20640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65cc107670 0x7f65cc10ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:05.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.944+0000 7f65d3b20640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f65b80b3cb0 0x7f65b80b61c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:05.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.944+0000 7f65d3b20640 1 --2- 172.21.15.29:0/606639294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65cc106d20 0x7f65cc10fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:05.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.944+0000 7f65d3b20640 1 -- 172.21.15.29:0/606639294 >> 172.21.15.29:0/606639294 conn(0x7f65cc077230 msgr2=0x7f65cc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:05.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.944+0000 7f65d3b20640 1 -- 172.21.15.29:0/606639294 shutdown_connections 2024-07-27T06:44:05.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:05.944+0000 7f65d3b20640 1 -- 172.21.15.29:0/606639294 wait complete. 2024-07-27T06:44:06.664 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-07-27T06:44:05.633288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658544,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":411,"num_read_kb":309,"num_write":371,"num_write_kb":2289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":359852,"kb_used_data":10700,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581972,"statfs":{"total":1151940427776,"available":1151571939328,"internally_reserved":0,"allocated":10956800,"data_stored":7073067,"data_compressed":107991,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":61,"num_write":10,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003430"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812421+0000","last_change":"2024-07-27T06:43:12.802353+0000","last_active":"2024-07-27T06:43:15.812421+0000","last_peered":"2024-07-27T06:43:15.812421+0000","last_clean":"2024-07-27T06:43:15.812421+0000","last_became_active":"2024-07-27T06:43:12.802028+0000","last_became_peered":"2024-07-27T06:43:12.802028+0000","last_unstale":"2024-07-27T06:43:15.812421+0000","last_undegraded":"2024-07-27T06:43:15.812421+0000","last_fullsized":"2024-07-27T06:43:15.812421+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:22:24.292567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812061+0000","last_change":"2024-07-27T06:43:12.806269+0000","last_active":"2024-07-27T06:43:15.812061+0000","last_peered":"2024-07-27T06:43:15.812061+0000","last_clean":"2024-07-27T06:43:15.812061+0000","last_became_active":"2024-07-27T06:43:12.803232+0000","last_became_peered":"2024-07-27T06:43:12.803232+0000","last_unstale":"2024-07-27T06:43:15.812061+0000","last_undegraded":"2024-07-27T06:43:15.812061+0000","last_fullsized":"2024-07-27T06:43:15.812061+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:57:53.673850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811879+0000","last_change":"2024-07-27T06:43:12.805542+0000","last_active":"2024-07-27T06:43:15.811879+0000","last_peered":"2024-07-27T06:43:15.811879+0000","last_clean":"2024-07-27T06:43:15.811879+0000","last_became_active":"2024-07-27T06:43:12.804951+0000","last_became_peered":"2024-07-27T06:43:12.804951+0000","last_unstale":"2024-07-27T06:43:15.811879+0000","last_undegraded":"2024-07-27T06:43:15.811879+0000","last_fullsized":"2024-07-27T06:43:15.811879+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:06:40.469695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794334+0000","last_change":"2024-07-27T06:43:12.796950+0000","last_active":"2024-07-27T06:43:15.794334+0000","last_peered":"2024-07-27T06:43:15.794334+0000","last_clean":"2024-07-27T06:43:15.794334+0000","last_became_active":"2024-07-27T06:43:12.796549+0000","last_became_peered":"2024-07-27T06:43:12.796549+0000","last_unstale":"2024-07-27T06:43:15.794334+0000","last_undegraded":"2024-07-27T06:43:15.794334+0000","last_fullsized":"2024-07-27T06:43:15.794334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:30:36.391046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808022+0000","last_change":"2024-07-27T06:43:12.799838+0000","last_active":"2024-07-27T06:43:15.808022+0000","last_peered":"2024-07-27T06:43:15.808022+0000","last_clean":"2024-07-27T06:43:15.808022+0000","last_became_active":"2024-07-27T06:43:12.799462+0000","last_became_peered":"2024-07-27T06:43:12.799462+0000","last_unstale":"2024-07-27T06:43:15.808022+0000","last_undegraded":"2024-07-27T06:43:15.808022+0000","last_fullsized":"2024-07-27T06:43:15.808022+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:45:19.371259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.806561+0000","last_change":"2024-07-27T06:43:12.797187+0000","last_active":"2024-07-27T06:43:15.806561+0000","last_peered":"2024-07-27T06:43:15.806561+0000","last_clean":"2024-07-27T06:43:15.806561+0000","last_became_active":"2024-07-27T06:43:12.796633+0000","last_became_peered":"2024-07-27T06:43:12.796633+0000","last_unstale":"2024-07-27T06:43:15.806561+0000","last_undegraded":"2024-07-27T06:43:15.806561+0000","last_fullsized":"2024-07-27T06:43:15.806561+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:22:20.882899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810075+0000","last_change":"2024-07-27T06:43:12.793786+0000","last_active":"2024-07-27T06:43:15.810075+0000","last_peered":"2024-07-27T06:43:15.810075+0000","last_clean":"2024-07-27T06:43:15.810075+0000","last_became_active":"2024-07-27T06:43:12.793413+0000","last_became_peered":"2024-07-27T06:43:12.793413+0000","last_unstale":"2024-07-27T06:43:15.810075+0000","last_undegraded":"2024-07-27T06:43:15.810075+0000","last_fullsized":"2024-07-27T06:43:15.810075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:14.015373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.807834+0000","last_change":"2024-07-27T06:43:12.791304+0000","last_active":"2024-07-27T06:43:15.807834+0000","last_peered":"2024-07-27T06:43:15.807834+0000","last_clean":"2024-07-27T06:43:15.807834+0000","last_became_active":"2024-07-27T06:43:12.790801+0000","last_became_peered":"2024-07-27T06:43:12.790801+0000","last_unstale":"2024-07-27T06:43:15.807834+0000","last_undegraded":"2024-07-27T06:43:15.807834+0000","last_fullsized":"2024-07-27T06:43:15.807834+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:12:37.606748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811264+0000","last_change":"2024-07-27T06:43:12.797778+0000","last_active":"2024-07-27T06:43:15.811264+0000","last_peered":"2024-07-27T06:43:15.811264+0000","last_clean":"2024-07-27T06:43:15.811264+0000","last_became_active":"2024-07-27T06:43:12.796784+0000","last_became_peered":"2024-07-27T06:43:12.796784+0000","last_unstale":"2024-07-27T06:43:15.811264+0000","last_undegraded":"2024-07-27T06:43:15.811264+0000","last_fullsized":"2024-07-27T06:43:15.811264+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:52:22.275282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.807073+0000","last_change":"2024-07-27T06:43:12.798970+0000","last_active":"2024-07-27T06:43:15.807073+0000","last_peered":"2024-07-27T06:43:15.807073+0000","last_clean":"2024-07-27T06:43:15.807073+0000","last_became_active":"2024-07-27T06:43:12.798549+0000","last_became_peered":"2024-07-27T06:43:12.798549+0000","last_unstale":"2024-07-27T06:43:15.807073+0000","last_undegraded":"2024-07-27T06:43:15.807073+0000","last_fullsized":"2024-07-27T06:43:15.807073+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:58:32.464429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.809869+0000","last_change":"2024-07-27T06:43:12.809405+0000","last_active":"2024-07-27T06:43:15.809869+0000","last_peered":"2024-07-27T06:43:15.809869+0000","last_clean":"2024-07-27T06:43:15.809869+0000","last_became_active":"2024-07-27T06:43:12.809030+0000","last_became_peered":"2024-07-27T06:43:12.809030+0000","last_unstale":"2024-07-27T06:43:15.809869+0000","last_undegraded":"2024-07-27T06:43:15.809869+0000","last_fullsized":"2024-07-27T06:43:15.809869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:46:35.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794487+0000","last_change":"2024-07-27T06:43:12.802647+0000","last_active":"2024-07-27T06:43:15.794487+0000","last_peered":"2024-07-27T06:43:15.794487+0000","last_clean":"2024-07-27T06:43:15.794487+0000","last_became_active":"2024-07-27T06:43:12.802247+0000","last_became_peered":"2024-07-27T06:43:12.802247+0000","last_unstale":"2024-07-27T06:43:15.794487+0000","last_undegraded":"2024-07-27T06:43:15.794487+0000","last_fullsized":"2024-07-27T06:43:15.794487+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:44:24.564852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.793711+0000","last_change":"2024-07-27T06:43:12.809140+0000","last_active":"2024-07-27T06:43:15.793711+0000","last_peered":"2024-07-27T06:43:15.793711+0000","last_clean":"2024-07-27T06:43:15.793711+0000","last_became_active":"2024-07-27T06:43:12.808692+0000","last_became_peered":"2024-07-27T06:43:12.808692+0000","last_unstale":"2024-07-27T06:43:15.793711+0000","last_undegraded":"2024-07-27T06:43:15.793711+0000","last_fullsized":"2024-07-27T06:43:15.793711+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:37:45.326234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811596+0000","last_change":"2024-07-27T06:43:12.802838+0000","last_active":"2024-07-27T06:43:15.811596+0000","last_peered":"2024-07-27T06:43:15.811596+0000","last_clean":"2024-07-27T06:43:15.811596+0000","last_became_active":"2024-07-27T06:43:12.802503+0000","last_became_peered":"2024-07-27T06:43:12.802503+0000","last_unstale":"2024-07-27T06:43:15.811596+0000","last_undegraded":"2024-07-27T06:43:15.811596+0000","last_fullsized":"2024-07-27T06:43:15.811596+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:41:03.428162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811733+0000","last_change":"2024-07-27T06:43:12.801626+0000","last_active":"2024-07-27T06:43:15.811733+0000","last_peered":"2024-07-27T06:43:15.811733+0000","last_clean":"2024-07-27T06:43:15.811733+0000","last_became_active":"2024-07-27T06:43:12.801266+0000","last_became_peered":"2024-07-27T06:43:12.801266+0000","last_unstale":"2024-07-27T06:43:15.811733+0000","last_undegraded":"2024-07-27T06:43:15.811733+0000","last_fullsized":"2024-07-27T06:43:15.811733+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:54:31.934223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812785+0000","last_change":"2024-07-27T06:43:12.804543+0000","last_active":"2024-07-27T06:43:15.812785+0000","last_peered":"2024-07-27T06:43:15.812785+0000","last_clean":"2024-07-27T06:43:15.812785+0000","last_became_active":"2024-07-27T06:43:12.804173+0000","last_became_peered":"2024-07-27T06:43:12.804173+0000","last_unstale":"2024-07-27T06:43:15.812785+0000","last_undegraded":"2024-07-27T06:43:15.812785+0000","last_fullsized":"2024-07-27T06:43:15.812785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:18:25.337333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812620+0000","last_change":"2024-07-27T06:43:12.798604+0000","last_active":"2024-07-27T06:43:15.812620+0000","last_peered":"2024-07-27T06:43:15.812620+0000","last_clean":"2024-07-27T06:43:15.812620+0000","last_became_active":"2024-07-27T06:43:12.798228+0000","last_became_peered":"2024-07-27T06:43:12.798228+0000","last_unstale":"2024-07-27T06:43:15.812620+0000","last_undegraded":"2024-07-27T06:43:15.812620+0000","last_fullsized":"2024-07-27T06:43:15.812620+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:04:35.851399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.807464+0000","last_change":"2024-07-27T06:43:12.800452+0000","last_active":"2024-07-27T06:43:15.807464+0000","last_peered":"2024-07-27T06:43:15.807464+0000","last_clean":"2024-07-27T06:43:15.807464+0000","last_became_active":"2024-07-27T06:43:12.800084+0000","last_became_peered":"2024-07-27T06:43:12.800084+0000","last_unstale":"2024-07-27T06:43:15.807464+0000","last_undegraded":"2024-07-27T06:43:15.807464+0000","last_fullsized":"2024-07-27T06:43:15.807464+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:54:27.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.814057+0000","last_change":"2024-07-27T06:43:12.800761+0000","last_active":"2024-07-27T06:43:15.814057+0000","last_peered":"2024-07-27T06:43:15.814057+0000","last_clean":"2024-07-27T06:43:15.814057+0000","last_became_active":"2024-07-27T06:43:12.800414+0000","last_became_peered":"2024-07-27T06:43:12.800414+0000","last_unstale":"2024-07-27T06:43:15.814057+0000","last_undegraded":"2024-07-27T06:43:15.814057+0000","last_fullsized":"2024-07-27T06:43:15.814057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:02:47.247522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811101+0000","last_change":"2024-07-27T06:43:12.795470+0000","last_active":"2024-07-27T06:43:15.811101+0000","last_peered":"2024-07-27T06:43:15.811101+0000","last_clean":"2024-07-27T06:43:15.811101+0000","last_became_active":"2024-07-27T06:43:12.795136+0000","last_became_peered":"2024-07-27T06:43:12.795136+0000","last_unstale":"2024-07-27T06:43:15.811101+0000","last_undegraded":"2024-07-27T06:43:15.811101+0000","last_fullsized":"2024-07-27T06:43:15.811101+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:01:22.178699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.793740+0000","last_change":"2024-07-27T06:43:12.803897+0000","last_active":"2024-07-27T06:43:15.793740+0000","last_peered":"2024-07-27T06:43:15.793740+0000","last_clean":"2024-07-27T06:43:15.793740+0000","last_became_active":"2024-07-27T06:43:12.803546+0000","last_became_peered":"2024-07-27T06:43:12.803546+0000","last_unstale":"2024-07-27T06:43:15.793740+0000","last_undegraded":"2024-07-27T06:43:15.793740+0000","last_fullsized":"2024-07-27T06:43:15.793740+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:13:50.816831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.813198+0000","last_change":"2024-07-27T06:43:12.803605+0000","last_active":"2024-07-27T06:43:15.813198+0000","last_peered":"2024-07-27T06:43:15.813198+0000","last_clean":"2024-07-27T06:43:15.813198+0000","last_became_active":"2024-07-27T06:43:12.803144+0000","last_became_peered":"2024-07-27T06:43:12.803144+0000","last_unstale":"2024-07-27T06:43:15.813198+0000","last_undegraded":"2024-07-27T06:43:15.813198+0000","last_fullsized":"2024-07-27T06:43:15.813198+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:34:10.372185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808876+0000","last_change":"2024-07-27T06:43:12.788570+0000","last_active":"2024-07-27T06:43:15.808876+0000","last_peered":"2024-07-27T06:43:15.808876+0000","last_clean":"2024-07-27T06:43:15.808876+0000","last_became_active":"2024-07-27T06:43:12.788199+0000","last_became_peered":"2024-07-27T06:43:12.788199+0000","last_unstale":"2024-07-27T06:43:15.808876+0000","last_undegraded":"2024-07-27T06:43:15.808876+0000","last_fullsized":"2024-07-27T06:43:15.808876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:44:26.751094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794071+0000","last_change":"2024-07-27T06:43:12.792866+0000","last_active":"2024-07-27T06:43:15.794071+0000","last_peered":"2024-07-27T06:43:15.794071+0000","last_clean":"2024-07-27T06:43:15.794071+0000","last_became_active":"2024-07-27T06:43:12.792389+0000","last_became_peered":"2024-07-27T06:43:12.792389+0000","last_unstale":"2024-07-27T06:43:15.794071+0000","last_undegraded":"2024-07-27T06:43:15.794071+0000","last_fullsized":"2024-07-27T06:43:15.794071+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:10:37.367642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.807780+0000","last_change":"2024-07-27T06:43:12.797002+0000","last_active":"2024-07-27T06:43:15.807780+0000","last_peered":"2024-07-27T06:43:15.807780+0000","last_clean":"2024-07-27T06:43:15.807780+0000","last_became_active":"2024-07-27T06:43:12.796651+0000","last_became_peered":"2024-07-27T06:43:12.796651+0000","last_unstale":"2024-07-27T06:43:15.807780+0000","last_undegraded":"2024-07-27T06:43:15.807780+0000","last_fullsized":"2024-07-27T06:43:15.807780+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:28:06.934042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.813220+0000","last_change":"2024-07-27T06:43:12.798549+0000","last_active":"2024-07-27T06:43:15.813220+0000","last_peered":"2024-07-27T06:43:15.813220+0000","last_clean":"2024-07-27T06:43:15.813220+0000","last_became_active":"2024-07-27T06:43:12.798201+0000","last_became_peered":"2024-07-27T06:43:12.798201+0000","last_unstale":"2024-07-27T06:43:15.813220+0000","last_undegraded":"2024-07-27T06:43:15.813220+0000","last_fullsized":"2024-07-27T06:43:15.813220+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:59:47.525536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811494+0000","last_change":"2024-07-27T06:43:12.791053+0000","last_active":"2024-07-27T06:43:15.811494+0000","last_peered":"2024-07-27T06:43:15.811494+0000","last_clean":"2024-07-27T06:43:15.811494+0000","last_became_active":"2024-07-27T06:43:12.790718+0000","last_became_peered":"2024-07-27T06:43:12.790718+0000","last_unstale":"2024-07-27T06:43:15.811494+0000","last_undegraded":"2024-07-27T06:43:15.811494+0000","last_fullsized":"2024-07-27T06:43:15.811494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:09.455399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.793523+0000","last_change":"2024-07-27T06:43:12.804180+0000","last_active":"2024-07-27T06:43:15.793523+0000","last_peered":"2024-07-27T06:43:15.793523+0000","last_clean":"2024-07-27T06:43:15.793523+0000","last_became_active":"2024-07-27T06:43:12.803881+0000","last_became_peered":"2024-07-27T06:43:12.803881+0000","last_unstale":"2024-07-27T06:43:15.793523+0000","last_undegraded":"2024-07-27T06:43:15.793523+0000","last_fullsized":"2024-07-27T06:43:15.793523+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:01:05.009233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811748+0000","last_change":"2024-07-27T06:43:12.809150+0000","last_active":"2024-07-27T06:43:15.811748+0000","last_peered":"2024-07-27T06:43:15.811748+0000","last_clean":"2024-07-27T06:43:15.811748+0000","last_became_active":"2024-07-27T06:43:12.808428+0000","last_became_peered":"2024-07-27T06:43:12.808428+0000","last_unstale":"2024-07-27T06:43:15.811748+0000","last_undegraded":"2024-07-27T06:43:15.811748+0000","last_fullsized":"2024-07-27T06:43:15.811748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:07:13.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810154+0000","last_change":"2024-07-27T06:43:12.793562+0000","last_active":"2024-07-27T06:43:15.810154+0000","last_peered":"2024-07-27T06:43:15.810154+0000","last_clean":"2024-07-27T06:43:15.810154+0000","last_became_active":"2024-07-27T06:43:12.793031+0000","last_became_peered":"2024-07-27T06:43:12.793031+0000","last_unstale":"2024-07-27T06:43:15.810154+0000","last_undegraded":"2024-07-27T06:43:15.810154+0000","last_fullsized":"2024-07-27T06:43:15.810154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:27:46.227980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811513+0000","last_change":"2024-07-27T06:43:12.797282+0000","last_active":"2024-07-27T06:43:15.811513+0000","last_peered":"2024-07-27T06:43:15.811513+0000","last_clean":"2024-07-27T06:43:15.811513+0000","last_became_active":"2024-07-27T06:43:12.796921+0000","last_became_peered":"2024-07-27T06:43:12.796921+0000","last_unstale":"2024-07-27T06:43:15.811513+0000","last_undegraded":"2024-07-27T06:43:15.811513+0000","last_fullsized":"2024-07-27T06:43:15.811513+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:19:39.940577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811833+0000","last_change":"2024-07-27T06:43:12.787866+0000","last_active":"2024-07-27T06:43:15.811833+0000","last_peered":"2024-07-27T06:43:15.811833+0000","last_clean":"2024-07-27T06:43:15.811833+0000","last_became_active":"2024-07-27T06:43:12.787513+0000","last_became_peered":"2024-07-27T06:43:12.787513+0000","last_unstale":"2024-07-27T06:43:15.811833+0000","last_undegraded":"2024-07-27T06:43:15.811833+0000","last_fullsized":"2024-07-27T06:43:15.811833+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:48:57.876588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.809386+0000","last_change":"2024-07-27T06:43:12.791158+0000","last_active":"2024-07-27T06:43:15.809386+0000","last_peered":"2024-07-27T06:43:15.809386+0000","last_clean":"2024-07-27T06:43:15.809386+0000","last_became_active":"2024-07-27T06:43:12.790771+0000","last_became_peered":"2024-07-27T06:43:12.790771+0000","last_unstale":"2024-07-27T06:43:15.809386+0000","last_undegraded":"2024-07-27T06:43:15.809386+0000","last_fullsized":"2024-07-27T06:43:15.809386+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:10:18.739265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"79'6","reported_seq":63,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:44:02.935784+0000","last_change":"2024-07-27T06:43:09.769281+0000","last_active":"2024-07-27T06:44:02.935784+0000","last_peered":"2024-07-27T06:44:02.935784+0000","last_clean":"2024-07-27T06:44:02.935784+0000","last_became_active":"2024-07-27T06:43:09.768864+0000","last_became_peered":"2024-07-27T06:43:09.768864+0000","last_unstale":"2024-07-27T06:44:02.935784+0000","last_undegraded":"2024-07-27T06:44:02.935784+0000","last_fullsized":"2024-07-27T06:44:02.935784+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:55:14.191180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794374+0000","last_change":"2024-07-27T06:43:12.794484+0000","last_active":"2024-07-27T06:43:15.794374+0000","last_peered":"2024-07-27T06:43:15.794374+0000","last_clean":"2024-07-27T06:43:15.794374+0000","last_became_active":"2024-07-27T06:43:12.794147+0000","last_became_peered":"2024-07-27T06:43:12.794147+0000","last_unstale":"2024-07-27T06:43:15.794374+0000","last_undegraded":"2024-07-27T06:43:15.794374+0000","last_fullsized":"2024-07-27T06:43:15.794374+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T18:04:34.745156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:31.864525+0000","last_change":"2024-07-27T06:43:09.772227+0000","last_active":"2024-07-27T06:43:31.864525+0000","last_peered":"2024-07-27T06:43:31.864525+0000","last_clean":"2024-07-27T06:43:31.864525+0000","last_became_active":"2024-07-27T06:43:09.771812+0000","last_became_peered":"2024-07-27T06:43:09.771812+0000","last_unstale":"2024-07-27T06:43:31.864525+0000","last_undegraded":"2024-07-27T06:43:31.864525+0000","last_fullsized":"2024-07-27T06:43:31.864525+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:27:28.054590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:30.853588+0000","last_change":"2024-07-27T06:43:09.766490+0000","last_active":"2024-07-27T06:43:30.853588+0000","last_peered":"2024-07-27T06:43:30.853588+0000","last_clean":"2024-07-27T06:43:30.853588+0000","last_became_active":"2024-07-27T06:43:09.765973+0000","last_became_peered":"2024-07-27T06:43:09.765973+0000","last_unstale":"2024-07-27T06:43:30.853588+0000","last_undegraded":"2024-07-27T06:43:30.853588+0000","last_fullsized":"2024-07-27T06:43:30.853588+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:00:02.913080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808080+0000","last_change":"2024-07-27T06:43:12.805234+0000","last_active":"2024-07-27T06:43:15.808080+0000","last_peered":"2024-07-27T06:43:15.808080+0000","last_clean":"2024-07-27T06:43:15.808080+0000","last_became_active":"2024-07-27T06:43:12.804908+0000","last_became_peered":"2024-07-27T06:43:12.804908+0000","last_unstale":"2024-07-27T06:43:15.808080+0000","last_undegraded":"2024-07-27T06:43:15.808080+0000","last_fullsized":"2024-07-27T06:43:15.808080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:09:34.988755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.873019+0000","last_change":"2024-07-27T06:43:09.774201+0000","last_active":"2024-07-27T06:43:32.873019+0000","last_peered":"2024-07-27T06:43:32.873019+0000","last_clean":"2024-07-27T06:43:32.873019+0000","last_became_active":"2024-07-27T06:43:09.773817+0000","last_became_peered":"2024-07-27T06:43:09.773817+0000","last_unstale":"2024-07-27T06:43:32.873019+0000","last_undegraded":"2024-07-27T06:43:32.873019+0000","last_fullsized":"2024-07-27T06:43:32.873019+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:06.166251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.806897+0000","last_change":"2024-07-27T06:43:12.800174+0000","last_active":"2024-07-27T06:43:15.806897+0000","last_peered":"2024-07-27T06:43:15.806897+0000","last_clean":"2024-07-27T06:43:15.806897+0000","last_became_active":"2024-07-27T06:43:12.799769+0000","last_became_peered":"2024-07-27T06:43:12.799769+0000","last_unstale":"2024-07-27T06:43:15.806897+0000","last_undegraded":"2024-07-27T06:43:15.806897+0000","last_fullsized":"2024-07-27T06:43:15.806897+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:43:06.890372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":85,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:44:01.933707+0000","last_change":"2024-07-27T06:43:09.776612+0000","last_active":"2024-07-27T06:44:01.933707+0000","last_peered":"2024-07-27T06:44:01.933707+0000","last_clean":"2024-07-27T06:44:01.933707+0000","last_became_active":"2024-07-27T06:43:09.775761+0000","last_became_peered":"2024-07-27T06:43:09.775761+0000","last_unstale":"2024-07-27T06:44:01.933707+0000","last_undegraded":"2024-07-27T06:44:01.933707+0000","last_fullsized":"2024-07-27T06:44:01.933707+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:42:52.132357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":58,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810601+0000","last_change":"2024-07-27T06:43:12.797200+0000","last_active":"2024-07-27T06:43:15.810601+0000","last_peered":"2024-07-27T06:43:15.810601+0000","last_clean":"2024-07-27T06:43:15.810601+0000","last_became_active":"2024-07-27T06:43:12.796288+0000","last_became_peered":"2024-07-27T06:43:12.796288+0000","last_unstale":"2024-07-27T06:43:15.810601+0000","last_undegraded":"2024-07-27T06:43:15.810601+0000","last_fullsized":"2024-07-27T06:43:15.810601+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:31:27.603883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.873048+0000","last_change":"2024-07-27T06:43:09.772658+0000","last_active":"2024-07-27T06:43:32.873048+0000","last_peered":"2024-07-27T06:43:32.873048+0000","last_clean":"2024-07-27T06:43:32.873048+0000","last_became_active":"2024-07-27T06:43:09.772277+0000","last_became_peered":"2024-07-27T06:43:09.772277+0000","last_unstale":"2024-07-27T06:43:32.873048+0000","last_undegraded":"2024-07-27T06:43:32.873048+0000","last_fullsized":"2024-07-27T06:43:32.873048+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:09:16.854718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812087+0000","last_change":"2024-07-27T06:43:12.809643+0000","last_active":"2024-07-27T06:43:15.812087+0000","last_peered":"2024-07-27T06:43:15.812087+0000","last_clean":"2024-07-27T06:43:15.812087+0000","last_became_active":"2024-07-27T06:43:12.809020+0000","last_became_peered":"2024-07-27T06:43:12.809020+0000","last_unstale":"2024-07-27T06:43:15.812087+0000","last_undegraded":"2024-07-27T06:43:15.812087+0000","last_fullsized":"2024-07-27T06:43:15.812087+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:08:03.062914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":29,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:31.863658+0000","last_change":"2024-07-27T06:43:10.534042+0000","last_active":"2024-07-27T06:43:31.863658+0000","last_peered":"2024-07-27T06:43:31.863658+0000","last_clean":"2024-07-27T06:43:31.863658+0000","last_became_active":"2024-07-27T06:43:10.533615+0000","last_became_peered":"2024-07-27T06:43:10.533615+0000","last_unstale":"2024-07-27T06:43:31.863658+0000","last_undegraded":"2024-07-27T06:43:31.863658+0000","last_fullsized":"2024-07-27T06:43:31.863658+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:09:32.235633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810039+0000","last_change":"2024-07-27T06:43:12.798764+0000","last_active":"2024-07-27T06:43:15.810039+0000","last_peered":"2024-07-27T06:43:15.810039+0000","last_clean":"2024-07-27T06:43:15.810039+0000","last_became_active":"2024-07-27T06:43:12.797500+0000","last_became_peered":"2024-07-27T06:43:12.797500+0000","last_unstale":"2024-07-27T06:43:15.810039+0000","last_undegraded":"2024-07-27T06:43:15.810039+0000","last_fullsized":"2024-07-27T06:43:15.810039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:12:09.368711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"79'31","reported_seq":121,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:44:01.939179+0000","last_change":"2024-07-27T06:43:09.775954+0000","last_active":"2024-07-27T06:44:01.939179+0000","last_peered":"2024-07-27T06:44:01.939179+0000","last_clean":"2024-07-27T06:44:01.939179+0000","last_became_active":"2024-07-27T06:43:09.775498+0000","last_became_peered":"2024-07-27T06:43:09.775498+0000","last_unstale":"2024-07-27T06:44:01.939179+0000","last_undegraded":"2024-07-27T06:44:01.939179+0000","last_fullsized":"2024-07-27T06:44:01.939179+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:08:46.965525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60347,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":67,"num_read_kb":92,"num_write":33,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808773+0000","last_change":"2024-07-27T06:43:12.797220+0000","last_active":"2024-07-27T06:43:15.808773+0000","last_peered":"2024-07-27T06:43:15.808773+0000","last_clean":"2024-07-27T06:43:15.808773+0000","last_became_active":"2024-07-27T06:43:12.796889+0000","last_became_peered":"2024-07-27T06:43:12.796889+0000","last_unstale":"2024-07-27T06:43:15.808773+0000","last_undegraded":"2024-07-27T06:43:15.808773+0000","last_fullsized":"2024-07-27T06:43:15.808773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:24:36.275738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:33.884067+0000","last_change":"2024-07-27T06:43:09.776600+0000","last_active":"2024-07-27T06:43:33.884067+0000","last_peered":"2024-07-27T06:43:33.884067+0000","last_clean":"2024-07-27T06:43:33.884067+0000","last_became_active":"2024-07-27T06:43:09.776163+0000","last_became_peered":"2024-07-27T06:43:09.776163+0000","last_unstale":"2024-07-27T06:43:33.884067+0000","last_undegraded":"2024-07-27T06:43:33.884067+0000","last_fullsized":"2024-07-27T06:43:33.884067+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:22:15.947916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.793174+0000","last_change":"2024-07-27T06:43:12.804984+0000","last_active":"2024-07-27T06:43:15.793174+0000","last_peered":"2024-07-27T06:43:15.793174+0000","last_clean":"2024-07-27T06:43:15.793174+0000","last_became_active":"2024-07-27T06:43:12.804644+0000","last_became_peered":"2024-07-27T06:43:12.804644+0000","last_unstale":"2024-07-27T06:43:15.793174+0000","last_undegraded":"2024-07-27T06:43:15.793174+0000","last_fullsized":"2024-07-27T06:43:15.793174+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.355981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":29,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.872774+0000","last_change":"2024-07-27T06:43:09.772489+0000","last_active":"2024-07-27T06:43:32.872774+0000","last_peered":"2024-07-27T06:43:32.872774+0000","last_clean":"2024-07-27T06:43:32.872774+0000","last_became_active":"2024-07-27T06:43:09.772069+0000","last_became_peered":"2024-07-27T06:43:09.772069+0000","last_unstale":"2024-07-27T06:43:32.872774+0000","last_undegraded":"2024-07-27T06:43:32.872774+0000","last_fullsized":"2024-07-27T06:43:32.872774+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:15:30.549971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810397+0000","last_change":"2024-07-27T06:43:12.803376+0000","last_active":"2024-07-27T06:43:15.810397+0000","last_peered":"2024-07-27T06:43:15.810397+0000","last_clean":"2024-07-27T06:43:15.810397+0000","last_became_active":"2024-07-27T06:43:12.802961+0000","last_became_peered":"2024-07-27T06:43:12.802961+0000","last_unstale":"2024-07-27T06:43:15.810397+0000","last_undegraded":"2024-07-27T06:43:15.810397+0000","last_fullsized":"2024-07-27T06:43:15.810397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:00:24.649877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":57,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:58.929501+0000","last_change":"2024-07-27T06:43:09.773182+0000","last_active":"2024-07-27T06:43:58.929501+0000","last_peered":"2024-07-27T06:43:58.929501+0000","last_clean":"2024-07-27T06:43:58.929501+0000","last_became_active":"2024-07-27T06:43:09.772126+0000","last_became_peered":"2024-07-27T06:43:09.772126+0000","last_unstale":"2024-07-27T06:43:58.929501+0000","last_undegraded":"2024-07-27T06:43:58.929501+0000","last_fullsized":"2024-07-27T06:43:58.929501+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:45:26.598799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.809741+0000","last_change":"2024-07-27T06:43:12.799385+0000","last_active":"2024-07-27T06:43:15.809741+0000","last_peered":"2024-07-27T06:43:15.809741+0000","last_clean":"2024-07-27T06:43:15.809741+0000","last_became_active":"2024-07-27T06:43:12.798639+0000","last_became_peered":"2024-07-27T06:43:12.798639+0000","last_unstale":"2024-07-27T06:43:15.809741+0000","last_undegraded":"2024-07-27T06:43:15.809741+0000","last_fullsized":"2024-07-27T06:43:15.809741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:56:29.462491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":41,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.872940+0000","last_change":"2024-07-27T06:43:10.535429+0000","last_active":"2024-07-27T06:43:32.872940+0000","last_peered":"2024-07-27T06:43:32.872940+0000","last_clean":"2024-07-27T06:43:32.872940+0000","last_became_active":"2024-07-27T06:43:10.534952+0000","last_became_peered":"2024-07-27T06:43:10.534952+0000","last_unstale":"2024-07-27T06:43:32.872940+0000","last_undegraded":"2024-07-27T06:43:32.872940+0000","last_fullsized":"2024-07-27T06:43:32.872940+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:54:20.018263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794089+0000","last_change":"2024-07-27T06:43:12.804266+0000","last_active":"2024-07-27T06:43:15.794089+0000","last_peered":"2024-07-27T06:43:15.794089+0000","last_clean":"2024-07-27T06:43:15.794089+0000","last_became_active":"2024-07-27T06:43:12.803888+0000","last_became_peered":"2024-07-27T06:43:12.803888+0000","last_unstale":"2024-07-27T06:43:15.794089+0000","last_undegraded":"2024-07-27T06:43:15.794089+0000","last_fullsized":"2024-07-27T06:43:15.794089+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:48:00.490592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.848213+0000","last_change":"2024-07-27T06:43:09.777656+0000","last_active":"2024-07-27T06:43:32.848213+0000","last_peered":"2024-07-27T06:43:32.848213+0000","last_clean":"2024-07-27T06:43:32.848213+0000","last_became_active":"2024-07-27T06:43:09.776620+0000","last_became_peered":"2024-07-27T06:43:09.776620+0000","last_unstale":"2024-07-27T06:43:32.848213+0000","last_undegraded":"2024-07-27T06:43:32.848213+0000","last_fullsized":"2024-07-27T06:43:32.848213+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:50.203623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.811868+0000","last_change":"2024-07-27T06:43:12.796179+0000","last_active":"2024-07-27T06:43:15.811868+0000","last_peered":"2024-07-27T06:43:15.811868+0000","last_clean":"2024-07-27T06:43:15.811868+0000","last_became_active":"2024-07-27T06:43:12.795701+0000","last_became_peered":"2024-07-27T06:43:12.795701+0000","last_unstale":"2024-07-27T06:43:15.811868+0000","last_undegraded":"2024-07-27T06:43:15.811868+0000","last_fullsized":"2024-07-27T06:43:15.811868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:20:12.236769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":28,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.871192+0000","last_change":"2024-07-27T06:43:09.769949+0000","last_active":"2024-07-27T06:43:32.871192+0000","last_peered":"2024-07-27T06:43:32.871192+0000","last_clean":"2024-07-27T06:43:32.871192+0000","last_became_active":"2024-07-27T06:43:09.769602+0000","last_became_peered":"2024-07-27T06:43:09.769602+0000","last_unstale":"2024-07-27T06:43:32.871192+0000","last_undegraded":"2024-07-27T06:43:32.871192+0000","last_fullsized":"2024-07-27T06:43:32.871192+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:28:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810922+0000","last_change":"2024-07-27T06:43:12.802230+0000","last_active":"2024-07-27T06:43:15.810922+0000","last_peered":"2024-07-27T06:43:15.810922+0000","last_clean":"2024-07-27T06:43:15.810922+0000","last_became_active":"2024-07-27T06:43:12.800944+0000","last_became_peered":"2024-07-27T06:43:12.800944+0000","last_unstale":"2024-07-27T06:43:15.810922+0000","last_undegraded":"2024-07-27T06:43:15.810922+0000","last_fullsized":"2024-07-27T06:43:15.810922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:05:54.585955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.871177+0000","last_change":"2024-07-27T06:43:09.768239+0000","last_active":"2024-07-27T06:43:32.871177+0000","last_peered":"2024-07-27T06:43:32.871177+0000","last_clean":"2024-07-27T06:43:32.871177+0000","last_became_active":"2024-07-27T06:43:09.767584+0000","last_became_peered":"2024-07-27T06:43:09.767584+0000","last_unstale":"2024-07-27T06:43:32.871177+0000","last_undegraded":"2024-07-27T06:43:32.871177+0000","last_fullsized":"2024-07-27T06:43:32.871177+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:35:39.552246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"79'1","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810434+0000","last_change":"2024-07-27T06:43:12.803779+0000","last_active":"2024-07-27T06:43:15.810434+0000","last_peered":"2024-07-27T06:43:15.810434+0000","last_clean":"2024-07-27T06:43:15.810434+0000","last_became_active":"2024-07-27T06:43:12.803359+0000","last_became_peered":"2024-07-27T06:43:12.803359+0000","last_unstale":"2024-07-27T06:43:15.810434+0000","last_undegraded":"2024-07-27T06:43:15.810434+0000","last_fullsized":"2024-07-27T06:43:15.810434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:32:53.167039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'6","reported_seq":60,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:44:00.000894+0000","last_change":"2024-07-27T06:43:10.533751+0000","last_active":"2024-07-27T06:44:00.000894+0000","last_peered":"2024-07-27T06:44:00.000894+0000","last_clean":"2024-07-27T06:44:00.000894+0000","last_became_active":"2024-07-27T06:43:10.533245+0000","last_became_peered":"2024-07-27T06:43:10.533245+0000","last_unstale":"2024-07-27T06:44:00.000894+0000","last_undegraded":"2024-07-27T06:44:00.000894+0000","last_fullsized":"2024-07-27T06:44:00.000894+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:43:01.469725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810147+0000","last_change":"2024-07-27T06:43:12.805830+0000","last_active":"2024-07-27T06:43:15.810147+0000","last_peered":"2024-07-27T06:43:15.810147+0000","last_clean":"2024-07-27T06:43:15.810147+0000","last_became_active":"2024-07-27T06:43:12.805324+0000","last_became_peered":"2024-07-27T06:43:12.805324+0000","last_unstale":"2024-07-27T06:43:15.810147+0000","last_undegraded":"2024-07-27T06:43:15.810147+0000","last_fullsized":"2024-07-27T06:43:15.810147+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:38:31.430676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":99,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810142+0000","last_change":"2024-07-27T06:41:43.549132+0000","last_active":"2024-07-27T06:43:15.810142+0000","last_peered":"2024-07-27T06:43:15.810142+0000","last_clean":"2024-07-27T06:43:15.810142+0000","last_became_active":"2024-07-27T06:41:43.542635+0000","last_became_peered":"2024-07-27T06:41:43.542635+0000","last_unstale":"2024-07-27T06:43:15.810142+0000","last_undegraded":"2024-07-27T06:43:15.810142+0000","last_fullsized":"2024-07-27T06:43:15.810142+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_clean_scrub_stamp":"2024-07-27T06:38:21.639907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:10:06.417387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":32,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:33.885316+0000","last_change":"2024-07-27T06:43:09.768774+0000","last_active":"2024-07-27T06:43:33.885316+0000","last_peered":"2024-07-27T06:43:33.885316+0000","last_clean":"2024-07-27T06:43:33.885316+0000","last_became_active":"2024-07-27T06:43:09.768345+0000","last_became_peered":"2024-07-27T06:43:09.768345+0000","last_unstale":"2024-07-27T06:43:33.885316+0000","last_undegraded":"2024-07-27T06:43:33.885316+0000","last_fullsized":"2024-07-27T06:43:33.885316+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:40:49.411746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808005+0000","last_change":"2024-07-27T06:43:12.797817+0000","last_active":"2024-07-27T06:43:15.808005+0000","last_peered":"2024-07-27T06:43:15.808005+0000","last_clean":"2024-07-27T06:43:15.808005+0000","last_became_active":"2024-07-27T06:43:12.797483+0000","last_became_peered":"2024-07-27T06:43:12.797483+0000","last_unstale":"2024-07-27T06:43:15.808005+0000","last_undegraded":"2024-07-27T06:43:15.808005+0000","last_fullsized":"2024-07-27T06:43:15.808005+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:24:44.539505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:30.853385+0000","last_change":"2024-07-27T06:43:09.776880+0000","last_active":"2024-07-27T06:43:30.853385+0000","last_peered":"2024-07-27T06:43:30.853385+0000","last_clean":"2024-07-27T06:43:30.853385+0000","last_became_active":"2024-07-27T06:43:09.776486+0000","last_became_peered":"2024-07-27T06:43:09.776486+0000","last_unstale":"2024-07-27T06:43:30.853385+0000","last_undegraded":"2024-07-27T06:43:30.853385+0000","last_fullsized":"2024-07-27T06:43:30.853385+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:55:25.436405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.793149+0000","last_change":"2024-07-27T06:43:12.799755+0000","last_active":"2024-07-27T06:43:15.793149+0000","last_peered":"2024-07-27T06:43:15.793149+0000","last_clean":"2024-07-27T06:43:15.793149+0000","last_became_active":"2024-07-27T06:43:12.799412+0000","last_became_peered":"2024-07-27T06:43:12.799412+0000","last_unstale":"2024-07-27T06:43:15.793149+0000","last_undegraded":"2024-07-27T06:43:15.793149+0000","last_fullsized":"2024-07-27T06:43:15.793149+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:45:50.545431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":57,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:59.930350+0000","last_change":"2024-07-27T06:43:10.533039+0000","last_active":"2024-07-27T06:43:59.930350+0000","last_peered":"2024-07-27T06:43:59.930350+0000","last_clean":"2024-07-27T06:43:59.930350+0000","last_became_active":"2024-07-27T06:43:10.532583+0000","last_became_peered":"2024-07-27T06:43:10.532583+0000","last_unstale":"2024-07-27T06:43:59.930350+0000","last_undegraded":"2024-07-27T06:43:59.930350+0000","last_fullsized":"2024-07-27T06:43:59.930350+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:04:51.954463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794070+0000","last_change":"2024-07-27T06:43:12.795452+0000","last_active":"2024-07-27T06:43:15.794070+0000","last_peered":"2024-07-27T06:43:15.794070+0000","last_clean":"2024-07-27T06:43:15.794070+0000","last_became_active":"2024-07-27T06:43:12.795019+0000","last_became_peered":"2024-07-27T06:43:12.795019+0000","last_unstale":"2024-07-27T06:43:15.794070+0000","last_undegraded":"2024-07-27T06:43:15.794070+0000","last_fullsized":"2024-07-27T06:43:15.794070+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:03:18.016269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":60,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:44:00.944352+0000","last_change":"2024-07-27T06:43:09.776214+0000","last_active":"2024-07-27T06:44:00.944352+0000","last_peered":"2024-07-27T06:44:00.944352+0000","last_clean":"2024-07-27T06:44:00.944352+0000","last_became_active":"2024-07-27T06:43:09.775719+0000","last_became_peered":"2024-07-27T06:43:09.775719+0000","last_unstale":"2024-07-27T06:44:00.944352+0000","last_undegraded":"2024-07-27T06:44:00.944352+0000","last_fullsized":"2024-07-27T06:44:00.944352+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:57:20.381599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.806627+0000","last_change":"2024-07-27T06:43:12.794909+0000","last_active":"2024-07-27T06:43:15.806627+0000","last_peered":"2024-07-27T06:43:15.806627+0000","last_clean":"2024-07-27T06:43:15.806627+0000","last_became_active":"2024-07-27T06:43:12.794497+0000","last_became_peered":"2024-07-27T06:43:12.794497+0000","last_unstale":"2024-07-27T06:43:15.806627+0000","last_undegraded":"2024-07-27T06:43:15.806627+0000","last_fullsized":"2024-07-27T06:43:15.806627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:42:49.760126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.807290+0000","last_change":"2024-07-27T06:43:10.534240+0000","last_active":"2024-07-27T06:43:15.807290+0000","last_peered":"2024-07-27T06:43:15.807290+0000","last_clean":"2024-07-27T06:43:15.807290+0000","last_became_active":"2024-07-27T06:43:10.533596+0000","last_became_peered":"2024-07-27T06:43:10.533596+0000","last_unstale":"2024-07-27T06:43:15.807290+0000","last_undegraded":"2024-07-27T06:43:15.807290+0000","last_fullsized":"2024-07-27T06:43:15.807290+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:16.098839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":18,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.806800+0000","last_change":"2024-07-27T06:43:12.805248+0000","last_active":"2024-07-27T06:43:15.806800+0000","last_peered":"2024-07-27T06:43:15.806800+0000","last_clean":"2024-07-27T06:43:15.806800+0000","last_became_active":"2024-07-27T06:43:12.803693+0000","last_became_peered":"2024-07-27T06:43:12.803693+0000","last_unstale":"2024-07-27T06:43:15.806800+0000","last_undegraded":"2024-07-27T06:43:15.806800+0000","last_fullsized":"2024-07-27T06:43:15.806800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:17.466465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":28,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.873421+0000","last_change":"2024-07-27T06:43:10.533701+0000","last_active":"2024-07-27T06:43:32.873421+0000","last_peered":"2024-07-27T06:43:32.873421+0000","last_clean":"2024-07-27T06:43:32.873421+0000","last_became_active":"2024-07-27T06:43:10.533109+0000","last_became_peered":"2024-07-27T06:43:10.533109+0000","last_unstale":"2024-07-27T06:43:32.873421+0000","last_undegraded":"2024-07-27T06:43:32.873421+0000","last_fullsized":"2024-07-27T06:43:32.873421+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:32:52.906691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.793621+0000","last_change":"2024-07-27T06:43:12.798479+0000","last_active":"2024-07-27T06:43:15.793621+0000","last_peered":"2024-07-27T06:43:15.793621+0000","last_clean":"2024-07-27T06:43:15.793621+0000","last_became_active":"2024-07-27T06:43:12.798124+0000","last_became_peered":"2024-07-27T06:43:12.798124+0000","last_unstale":"2024-07-27T06:43:15.793621+0000","last_undegraded":"2024-07-27T06:43:15.793621+0000","last_fullsized":"2024-07-27T06:43:15.793621+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:14:55.267213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"79'15","reported_seq":38,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:56.811241+0000","last_change":"2024-07-27T06:43:09.770122+0000","last_active":"2024-07-27T06:43:56.811241+0000","last_peered":"2024-07-27T06:43:56.811241+0000","last_clean":"2024-07-27T06:43:56.811241+0000","last_became_active":"2024-07-27T06:43:09.769784+0000","last_became_peered":"2024-07-27T06:43:09.769784+0000","last_unstale":"2024-07-27T06:43:56.811241+0000","last_undegraded":"2024-07-27T06:43:56.811241+0000","last_fullsized":"2024-07-27T06:43:56.811241+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:47:00.398135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808033+0000","last_change":"2024-07-27T06:43:12.799603+0000","last_active":"2024-07-27T06:43:15.808033+0000","last_peered":"2024-07-27T06:43:15.808033+0000","last_clean":"2024-07-27T06:43:15.808033+0000","last_became_active":"2024-07-27T06:43:12.799189+0000","last_became_peered":"2024-07-27T06:43:12.799189+0000","last_unstale":"2024-07-27T06:43:15.808033+0000","last_undegraded":"2024-07-27T06:43:15.808033+0000","last_fullsized":"2024-07-27T06:43:15.808033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:32:30.603258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":31,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.872453+0000","last_change":"2024-07-27T06:43:10.004726+0000","last_active":"2024-07-27T06:43:32.872453+0000","last_peered":"2024-07-27T06:43:32.872453+0000","last_clean":"2024-07-27T06:43:32.872453+0000","last_became_active":"2024-07-27T06:43:10.004367+0000","last_became_peered":"2024-07-27T06:43:10.004367+0000","last_unstale":"2024-07-27T06:43:32.872453+0000","last_undegraded":"2024-07-27T06:43:32.872453+0000","last_fullsized":"2024-07-27T06:43:32.872453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:22:39.942391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812071+0000","last_change":"2024-07-27T06:43:12.796511+0000","last_active":"2024-07-27T06:43:15.812071+0000","last_peered":"2024-07-27T06:43:15.812071+0000","last_clean":"2024-07-27T06:43:15.812071+0000","last_became_active":"2024-07-27T06:43:12.796152+0000","last_became_peered":"2024-07-27T06:43:12.796152+0000","last_unstale":"2024-07-27T06:43:15.812071+0000","last_undegraded":"2024-07-27T06:43:15.812071+0000","last_fullsized":"2024-07-27T06:43:15.812071+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:03:02.508990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:29.847443+0000","last_change":"2024-07-27T06:43:10.533656+0000","last_active":"2024-07-27T06:43:29.847443+0000","last_peered":"2024-07-27T06:43:29.847443+0000","last_clean":"2024-07-27T06:43:29.847443+0000","last_became_active":"2024-07-27T06:43:10.533295+0000","last_became_peered":"2024-07-27T06:43:10.533295+0000","last_unstale":"2024-07-27T06:43:29.847443+0000","last_undegraded":"2024-07-27T06:43:29.847443+0000","last_fullsized":"2024-07-27T06:43:29.847443+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:11:00.032511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.807280+0000","last_change":"2024-07-27T06:43:12.795106+0000","last_active":"2024-07-27T06:43:15.807280+0000","last_peered":"2024-07-27T06:43:15.807280+0000","last_clean":"2024-07-27T06:43:15.807280+0000","last_became_active":"2024-07-27T06:43:12.794729+0000","last_became_peered":"2024-07-27T06:43:12.794729+0000","last_unstale":"2024-07-27T06:43:15.807280+0000","last_undegraded":"2024-07-27T06:43:15.807280+0000","last_fullsized":"2024-07-27T06:43:15.807280+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:19:16.542363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.871541+0000","last_change":"2024-07-27T06:43:09.770016+0000","last_active":"2024-07-27T06:43:32.871541+0000","last_peered":"2024-07-27T06:43:32.871541+0000","last_clean":"2024-07-27T06:43:32.871541+0000","last_became_active":"2024-07-27T06:43:09.769672+0000","last_became_peered":"2024-07-27T06:43:09.769672+0000","last_unstale":"2024-07-27T06:43:32.871541+0000","last_undegraded":"2024-07-27T06:43:32.871541+0000","last_fullsized":"2024-07-27T06:43:32.871541+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:30:55.737494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812064+0000","last_change":"2024-07-27T06:43:12.793690+0000","last_active":"2024-07-27T06:43:15.812064+0000","last_peered":"2024-07-27T06:43:15.812064+0000","last_clean":"2024-07-27T06:43:15.812064+0000","last_became_active":"2024-07-27T06:43:12.793352+0000","last_became_peered":"2024-07-27T06:43:12.793352+0000","last_unstale":"2024-07-27T06:43:15.812064+0000","last_undegraded":"2024-07-27T06:43:15.812064+0000","last_fullsized":"2024-07-27T06:43:15.812064+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:22:47.606990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:31.864434+0000","last_change":"2024-07-27T06:43:09.770022+0000","last_active":"2024-07-27T06:43:31.864434+0000","last_peered":"2024-07-27T06:43:31.864434+0000","last_clean":"2024-07-27T06:43:31.864434+0000","last_became_active":"2024-07-27T06:43:09.769559+0000","last_became_peered":"2024-07-27T06:43:09.769559+0000","last_unstale":"2024-07-27T06:43:31.864434+0000","last_undegraded":"2024-07-27T06:43:31.864434+0000","last_fullsized":"2024-07-27T06:43:31.864434+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:33:01.252989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808333+0000","last_change":"2024-07-27T06:43:12.792648+0000","last_active":"2024-07-27T06:43:15.808333+0000","last_peered":"2024-07-27T06:43:15.808333+0000","last_clean":"2024-07-27T06:43:15.808333+0000","last_became_active":"2024-07-27T06:43:12.792338+0000","last_became_peered":"2024-07-27T06:43:12.792338+0000","last_unstale":"2024-07-27T06:43:15.808333+0000","last_undegraded":"2024-07-27T06:43:15.808333+0000","last_fullsized":"2024-07-27T06:43:15.808333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:36:52.605957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":28,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:29.847280+0000","last_change":"2024-07-27T06:43:09.772603+0000","last_active":"2024-07-27T06:43:29.847280+0000","last_peered":"2024-07-27T06:43:29.847280+0000","last_clean":"2024-07-27T06:43:29.847280+0000","last_became_active":"2024-07-27T06:43:09.772239+0000","last_became_peered":"2024-07-27T06:43:09.772239+0000","last_unstale":"2024-07-27T06:43:29.847280+0000","last_undegraded":"2024-07-27T06:43:29.847280+0000","last_fullsized":"2024-07-27T06:43:29.847280+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:33:32.905030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.810927+0000","last_change":"2024-07-27T06:43:12.809347+0000","last_active":"2024-07-27T06:43:15.810927+0000","last_peered":"2024-07-27T06:43:15.810927+0000","last_clean":"2024-07-27T06:43:15.810927+0000","last_became_active":"2024-07-27T06:43:12.808980+0000","last_became_peered":"2024-07-27T06:43:12.808980+0000","last_unstale":"2024-07-27T06:43:15.810927+0000","last_undegraded":"2024-07-27T06:43:15.810927+0000","last_fullsized":"2024-07-27T06:43:15.810927+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:05:14.132641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.794469+0000","last_change":"2024-07-27T06:43:12.803767+0000","last_active":"2024-07-27T06:43:15.794469+0000","last_peered":"2024-07-27T06:43:15.794469+0000","last_clean":"2024-07-27T06:43:15.794469+0000","last_became_active":"2024-07-27T06:43:12.803420+0000","last_became_peered":"2024-07-27T06:43:12.803420+0000","last_unstale":"2024-07-27T06:43:15.794469+0000","last_undegraded":"2024-07-27T06:43:15.794469+0000","last_fullsized":"2024-07-27T06:43:15.794469+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:09:56.623458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":29,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:32.872773+0000","last_change":"2024-07-27T06:43:09.776000+0000","last_active":"2024-07-27T06:43:32.872773+0000","last_peered":"2024-07-27T06:43:32.872773+0000","last_clean":"2024-07-27T06:43:32.872773+0000","last_became_active":"2024-07-27T06:43:09.775522+0000","last_became_peered":"2024-07-27T06:43:09.775522+0000","last_unstale":"2024-07-27T06:43:32.872773+0000","last_undegraded":"2024-07-27T06:43:32.872773+0000","last_fullsized":"2024-07-27T06:43:32.872773+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:39:42.650177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.809785+0000","last_change":"2024-07-27T06:43:12.807062+0000","last_active":"2024-07-27T06:43:15.809785+0000","last_peered":"2024-07-27T06:43:15.809785+0000","last_clean":"2024-07-27T06:43:15.809785+0000","last_became_active":"2024-07-27T06:43:12.805816+0000","last_became_peered":"2024-07-27T06:43:12.805816+0000","last_unstale":"2024-07-27T06:43:15.809785+0000","last_undegraded":"2024-07-27T06:43:15.809785+0000","last_fullsized":"2024-07-27T06:43:15.809785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:44:16.261663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:33.884014+0000","last_change":"2024-07-27T06:43:09.776285+0000","last_active":"2024-07-27T06:43:33.884014+0000","last_peered":"2024-07-27T06:43:33.884014+0000","last_clean":"2024-07-27T06:43:33.884014+0000","last_became_active":"2024-07-27T06:43:09.775055+0000","last_became_peered":"2024-07-27T06:43:09.775055+0000","last_unstale":"2024-07-27T06:43:33.884014+0000","last_undegraded":"2024-07-27T06:43:33.884014+0000","last_fullsized":"2024-07-27T06:43:33.884014+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:30:44.728547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.812075+0000","last_change":"2024-07-27T06:43:12.809135+0000","last_active":"2024-07-27T06:43:15.812075+0000","last_peered":"2024-07-27T06:43:15.812075+0000","last_clean":"2024-07-27T06:43:15.812075+0000","last_became_active":"2024-07-27T06:43:12.806731+0000","last_became_peered":"2024-07-27T06:43:12.806731+0000","last_unstale":"2024-07-27T06:43:15.812075+0000","last_undegraded":"2024-07-27T06:43:15.812075+0000","last_fullsized":"2024-07-27T06:43:15.812075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:47:50.397517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.818330+0000","last_change":"2024-07-27T06:43:10.533608+0000","last_active":"2024-07-27T06:43:15.818330+0000","last_peered":"2024-07-27T06:43:15.818330+0000","last_clean":"2024-07-27T06:43:15.818330+0000","last_became_active":"2024-07-27T06:43:10.532663+0000","last_became_peered":"2024-07-27T06:43:10.532663+0000","last_unstale":"2024-07-27T06:43:15.818330+0000","last_undegraded":"2024-07-27T06:43:15.818330+0000","last_fullsized":"2024-07-27T06:43:15.818330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:05:33.155427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:43:15.808322+0000","last_change":"2024-07-27T06:43:12.789890+0000","last_active":"2024-07-27T06:43:15.808322+0000","last_peered":"2024-07-27T06:43:15.808322+0000","last_clean":"2024-07-27T06:43:15.808322+0000","last_became_active":"2024-07-27T06:43:12.789561+0000","last_became_peered":"2024-07-27T06:43:12.789561+0000","last_unstale":"2024-07-27T06:43:15.808322+0000","last_undegraded":"2024-07-27T06:43:15.808322+0000","last_fullsized":"2024-07-27T06:43:15.808322+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:35:49.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":66,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-07-27T06:44:00.943764+0000","last_change":"2024-07-27T06:43:09.773471+0000","last_active":"2024-07-27T06:44:00.943764+0000","last_peered":"2024-07-27T06:44:00.943764+0000","last_clean":"2024-07-27T06:44:00.943764+0000","last_became_active":"2024-07-27T06:43:09.773213+0000","last_became_peered":"2024-07-27T06:43:09.773213+0000","last_unstale":"2024-07-27T06:44:00.943764+0000","last_undegraded":"2024-07-27T06:44:00.943764+0000","last_fullsized":"2024-07-27T06:44:00.943764+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:54:38.770198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67942,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":330,"num_read_kb":240,"num_write":246,"num_write_kb":137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":199569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283116,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678041,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":426724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874269,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1077709,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037796,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36204,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708948,"statfs":{"total":95995035648,"available":95957962752,"internally_reserved":0,"allocated":888832,"data_stored":426263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234025,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":426641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430252,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":1019550,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593778,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":872448,"data_stored":426371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790008,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":1018837,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953532,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":872448,"data_stored":425841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":425805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378695,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":483985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542220,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":933888,"data_stored":487631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705745,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":427710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T06:44:06.668 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-27T06:44:06.668 INFO:tasks.cephadm:Setting up client nodes... 2024-07-27T06:44:06.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-07-27T06:44:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:07.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:07 smithi029 ceph-mon[32610]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-07-27T06:44:07.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:07 smithi029 ceph-mon[32610]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:44:07.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:07.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:07.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:44:07.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:44:07.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:07 smithi077 ceph-mon[37153]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-07-27T06:44:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:07 smithi077 ceph-mon[37153]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:44:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:44:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:44:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:07 smithi047 ceph-mon[36835]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-07-27T06:44:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:07 smithi047 ceph-mon[36835]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:44:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:44:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:44:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:44:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-07-27T06:44:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-07-27T06:44:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7, 5, 9]}]: dispatch 2024-07-27T06:44:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-07-27T06:44:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8, 5, 9]}]: dispatch 2024-07-27T06:44:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2024-07-27T06:44:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-07-27T06:44:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-07-27T06:44:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-07-27T06:44:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-07-27T06:44:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7, 5, 9]}]: dispatch 2024-07-27T06:44:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-07-27T06:44:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8, 5, 9]}]: dispatch 2024-07-27T06:44:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2024-07-27T06:44:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-07-27T06:44:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-07-27T06:44:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-07-27T06:44:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-07-27T06:44:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7, 5, 9]}]: dispatch 2024-07-27T06:44:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-07-27T06:44:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8, 5, 9]}]: dispatch 2024-07-27T06:44:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2024-07-27T06:44:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-07-27T06:44:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-07-27T06:44:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.780+0000 7fd65911c640 1 -- 172.21.15.29:0/4202203814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd654109830 msgr2=0x7fd654110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.780+0000 7fd65911c640 1 --2- 172.21.15.29:0/4202203814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd654109830 0x7fd654110060 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009ca0 tx=0x7fd64c02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:44:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.780+0000 7fd65911c640 1 -- 172.21.15.29:0/4202203814 shutdown_connections 2024-07-27T06:44:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.780+0000 7fd65911c640 1 --2- 172.21.15.29:0/4202203814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd654109830 0x7fd654110060 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.780+0000 7fd65911c640 1 --2- 172.21.15.29:0/4202203814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd654108ee0 0x7fd6541092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.780+0000 7fd65911c640 1 --2- 172.21.15.29:0/4202203814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6541035a0 0x7fd6541039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.780+0000 7fd65911c640 1 -- 172.21.15.29:0/4202203814 >> 172.21.15.29:0/4202203814 conn(0x7fd6540fefd0 msgr2=0x7fd654101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.780+0000 7fd65911c640 1 -- 172.21.15.29:0/4202203814 shutdown_connections 2024-07-27T06:44:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.780+0000 7fd65911c640 1 -- 172.21.15.29:0/4202203814 wait complete. 2024-07-27T06:44:08.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.781+0000 7fd65911c640 1 Processor -- start 2024-07-27T06:44:08.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.781+0000 7fd65911c640 1 -- start start 2024-07-27T06:44:08.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.781+0000 7fd65911c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6541035a0 0x7fd654199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:08.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.781+0000 7fd65911c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd654108ee0 0x7fd65419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:08.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.781+0000 7fd65911c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd654109830 0x7fd65419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.781+0000 7fd65911c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd654113500 con 0x7fd654108ee0 2024-07-27T06:44:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.781+0000 7fd65911c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd654113380 con 0x7fd6541035a0 2024-07-27T06:44:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.781+0000 7fd65911c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd654113680 con 0x7fd654109830 2024-07-27T06:44:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.781+0000 7fd65259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd654108ee0 0x7fd65419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.781+0000 7fd65259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd654108ee0 0x7fd65419a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34068/0 (socket says 172.21.15.29:34068) 2024-07-27T06:44:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.781+0000 7fd65259c640 1 -- 172.21.15.29:0/1636006524 learned_addr learned my addr 172.21.15.29:0/1636006524 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.781+0000 7fd652d9d640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6541035a0 0x7fd654199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd65359e640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd654109830 0x7fd65419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd65259c640 1 -- 172.21.15.29:0/1636006524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd654109830 msgr2=0x7fd65419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd65259c640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd654109830 0x7fd65419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd65259c640 1 -- 172.21.15.29:0/1636006524 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6541035a0 msgr2=0x7fd654199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:08.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd65259c640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6541035a0 0x7fd654199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd65259c640 1 -- 172.21.15.29:0/1636006524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65419fe80 con 0x7fd654108ee0 2024-07-27T06:44:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd65359e640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd654109830 0x7fd65419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:44:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd65259c640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd654108ee0 0x7fd65419a4b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd64800e9c0 tx=0x7fd64800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd63bfff640 1 -- 172.21.15.29:0/1636006524 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64800ccc0 con 0x7fd654108ee0 2024-07-27T06:44:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd65911c640 1 -- 172.21.15.29:0/1636006524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6541a0170 con 0x7fd654108ee0 2024-07-27T06:44:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd63bfff640 1 -- 172.21.15.29:0/1636006524 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6480054d0 con 0x7fd654108ee0 2024-07-27T06:44:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd63bfff640 1 -- 172.21.15.29:0/1636006524 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd648010430 con 0x7fd654108ee0 2024-07-27T06:44:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.782+0000 7fd65911c640 1 -- 172.21.15.29:0/1636006524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6541a68d0 con 0x7fd654108ee0 2024-07-27T06:44:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.784+0000 7fd65911c640 1 -- 172.21.15.29:0/1636006524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6541039b0 con 0x7fd654108ee0 2024-07-27T06:44:08.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.785+0000 7fd63bfff640 1 -- 172.21.15.29:0/1636006524 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fd648005020 con 0x7fd654108ee0 2024-07-27T06:44:08.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.787+0000 7fd63bfff640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd63c0b3cb0 0x7fd63c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:08.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.787+0000 7fd652d9d640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd63c0b3cb0 0x7fd63c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:08.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.787+0000 7fd63bfff640 1 -- 172.21.15.29:0/1636006524 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8442+0+0 (secure 0 0 0) 0x7fd6480e4200 con 0x7fd654108ee0 2024-07-27T06:44:08.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.787+0000 7fd652d9d640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd63c0b3cb0 0x7fd63c0b61c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd644005d00 tx=0x7fd64400b410 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:08.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:08.791+0000 7fd63bfff640 1 -- 172.21.15.29:0/1636006524 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd6480ab620 con 0x7fd654108ee0 2024-07-27T06:44:09.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.046+0000 7fd65911c640 1 -- 172.21.15.29:0/1636006524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fd65402e0c0 con 0x7fd654108ee0 2024-07-27T06:44:09.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.049+0000 7fd63bfff640 1 -- 172.21.15.29:0/1636006524 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fd6480aaf20 con 0x7fd654108ee0 2024-07-27T06:44:09.049 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-07-27T06:44:09.049 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQA5l6RmJCXQAhAA4gM8WV2I/ov38+qfTy1v5A== 2024-07-27T06:44:09.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.051+0000 7fd65911c640 1 -- 172.21.15.29:0/1636006524 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd63c0b3cb0 msgr2=0x7fd63c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:09.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.051+0000 7fd65911c640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd63c0b3cb0 0x7fd63c0b61c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd644005d00 tx=0x7fd64400b410 comp rx=0 tx=0).stop 2024-07-27T06:44:09.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.051+0000 7fd65911c640 1 -- 172.21.15.29:0/1636006524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd654108ee0 msgr2=0x7fd65419a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:09.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.051+0000 7fd65911c640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd654108ee0 0x7fd65419a4b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd64800e9c0 tx=0x7fd64800ee80 comp rx=0 tx=0).stop 2024-07-27T06:44:09.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.051+0000 7fd65911c640 1 -- 172.21.15.29:0/1636006524 shutdown_connections 2024-07-27T06:44:09.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.051+0000 7fd65911c640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd63c0b3cb0 0x7fd63c0b61c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:09.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.051+0000 7fd65911c640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd654109830 0x7fd65419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:09.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.051+0000 7fd65911c640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd654108ee0 0x7fd65419a4b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:09.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.051+0000 7fd65911c640 1 --2- 172.21.15.29:0/1636006524 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd6541035a0 0x7fd654199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:09.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.051+0000 7fd65911c640 1 -- 172.21.15.29:0/1636006524 >> 172.21.15.29:0/1636006524 conn(0x7fd6540fefd0 msgr2=0x7fd654108550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:09.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.051+0000 7fd65911c640 1 -- 172.21.15.29:0/1636006524 shutdown_connections 2024-07-27T06:44:09.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:09.051+0000 7fd65911c640 1 -- 172.21.15.29:0/1636006524 wait complete. 2024-07-27T06:44:09.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:09 smithi029 ceph-mon[32610]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-07-27T06:44:09.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-07-27T06:44:09.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-07-27T06:44:09.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7, 5, 9]}]': finished 2024-07-27T06:44:09.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-07-27T06:44:09.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8, 5, 9]}]': finished 2024-07-27T06:44:09.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2024-07-27T06:44:09.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-07-27T06:44:09.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-07-27T06:44:09.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:09 smithi029 ceph-mon[32610]: osdmap e80: 12 total, 12 up, 12 in 2024-07-27T06:44:09.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:09 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1636006524' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-27T06:44:09.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:09 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1636006524' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-27T06:44:09.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:09 smithi047 ceph-mon[36835]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-07-27T06:44:09.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-07-27T06:44:09.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-07-27T06:44:09.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7, 5, 9]}]': finished 2024-07-27T06:44:09.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-07-27T06:44:09.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8, 5, 9]}]': finished 2024-07-27T06:44:09.599 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2024-07-27T06:44:09.599 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-07-27T06:44:09.599 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-07-27T06:44:09.599 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:09 smithi047 ceph-mon[36835]: osdmap e80: 12 total, 12 up, 12 in 2024-07-27T06:44:09.599 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:09 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1636006524' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-27T06:44:09.599 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:09 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1636006524' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-27T06:44:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:09 smithi077 ceph-mon[37153]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-07-27T06:44:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-07-27T06:44:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-07-27T06:44:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7, 5, 9]}]': finished 2024-07-27T06:44:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-07-27T06:44:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8, 5, 9]}]': finished 2024-07-27T06:44:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2024-07-27T06:44:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-07-27T06:44:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-07-27T06:44:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:09 smithi077 ceph-mon[37153]: osdmap e80: 12 total, 12 up, 12 in 2024-07-27T06:44:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:09 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1636006524' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-27T06:44:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:09 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1636006524' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-27T06:44:09.775 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:44:09.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-07-27T06:44:09.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-07-27T06:44:09.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph config log 1 --format=json 2024-07-27T06:44:10.120 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:10.531 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:10 smithi029 ceph-mon[32610]: osdmap e81: 12 total, 12 up, 12 in 2024-07-27T06:44:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:10 smithi077 ceph-mon[37153]: osdmap e81: 12 total, 12 up, 12 in 2024-07-27T06:44:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:10 smithi047 ceph-mon[36835]: osdmap e81: 12 total, 12 up, 12 in 2024-07-27T06:44:11.500 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:11 smithi029 ceph-mon[32610]: pgmap v310: 97 pgs: 1 peering, 96 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 639 B/s wr, 10 op/s 2024-07-27T06:44:11.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:11 smithi077 ceph-mon[37153]: pgmap v310: 97 pgs: 1 peering, 96 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 639 B/s wr, 10 op/s 2024-07-27T06:44:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:11 smithi047 ceph-mon[36835]: pgmap v310: 97 pgs: 1 peering, 96 active+clean; 643 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 639 B/s wr, 10 op/s 2024-07-27T06:44:11.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.908+0000 7f058d735640 1 -- 172.21.15.29:0/2790437160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0588107e00 msgr2=0x7f0588108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.908+0000 7f058d735640 1 --2- 172.21.15.29:0/2790437160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0588107e00 0x7f0588108210 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f057c009900 tx=0x7f057c02a410 comp rx=0 tx=0).stop 2024-07-27T06:44:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.908+0000 7f058d735640 1 -- 172.21.15.29:0/2790437160 shutdown_connections 2024-07-27T06:44:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.908+0000 7f058d735640 1 --2- 172.21.15.29:0/2790437160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0588108750 0x7f058810ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.908+0000 7f058d735640 1 --2- 172.21.15.29:0/2790437160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0588107e00 0x7f0588108210 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.908+0000 7f058d735640 1 --2- 172.21.15.29:0/2790437160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0588102510 0x7f0588102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.908+0000 7f058d735640 1 -- 172.21.15.29:0/2790437160 >> 172.21.15.29:0/2790437160 conn(0x7f0588079190 msgr2=0x7f05880795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.908+0000 7f058d735640 1 -- 172.21.15.29:0/2790437160 shutdown_connections 2024-07-27T06:44:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.908+0000 7f058d735640 1 -- 172.21.15.29:0/2790437160 wait complete. 2024-07-27T06:44:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.909+0000 7f058d735640 1 Processor -- start 2024-07-27T06:44:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.909+0000 7f058d735640 1 -- start start 2024-07-27T06:44:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.909+0000 7f058d735640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0588102510 0x7f0588199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.909+0000 7f058d735640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0588107e00 0x7f058819a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.909+0000 7f058d735640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0588108750 0x7f058819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.909+0000 7f058d735640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f05881134e0 con 0x7f0588108750 2024-07-27T06:44:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.909+0000 7f058d735640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f0588113360 con 0x7f0588102510 2024-07-27T06:44:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.909+0000 7f058d735640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0588113660 con 0x7f0588107e00 2024-07-27T06:44:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.909+0000 7f05877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0588108750 0x7f058819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.909+0000 7f05877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0588108750 0x7f058819f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40726/0 (socket says 172.21.15.29:40726) 2024-07-27T06:44:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.909+0000 7f05877fe640 1 -- 172.21.15.29:0/123601074 learned_addr learned my addr 172.21.15.29:0/123601074 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.909+0000 7f0586ffd640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0588102510 0x7f0588199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.910+0000 7f05867fc640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0588107e00 0x7f058819a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.910+0000 7f05877fe640 1 -- 172.21.15.29:0/123601074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0588107e00 msgr2=0x7f058819a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.910+0000 7f05877fe640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0588107e00 0x7f058819a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.910+0000 7f05877fe640 1 -- 172.21.15.29:0/123601074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0588102510 msgr2=0x7f0588199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.910+0000 7f05877fe640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0588102510 0x7f0588199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.910+0000 7f05877fe640 1 -- 172.21.15.29:0/123601074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058819fe00 con 0x7f0588108750 2024-07-27T06:44:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.910+0000 7f05877fe640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0588108750 0x7f058819f6d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f058000eaa0 tx=0x7f058000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.910+0000 7f056ffff640 1 -- 172.21.15.29:0/123601074 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058000cdb0 con 0x7f0588108750 2024-07-27T06:44:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.910+0000 7f056ffff640 1 -- 172.21.15.29:0/123601074 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0580005020 con 0x7f0588108750 2024-07-27T06:44:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.910+0000 7f058d735640 1 -- 172.21.15.29:0/123601074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05881a00f0 con 0x7f0588108750 2024-07-27T06:44:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.910+0000 7f056ffff640 1 -- 172.21.15.29:0/123601074 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05800116d0 con 0x7f0588108750 2024-07-27T06:44:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.910+0000 7f058d735640 1 -- 172.21.15.29:0/123601074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05881a6850 con 0x7f0588108750 2024-07-27T06:44:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.911+0000 7f058d735640 1 -- 172.21.15.29:0/123601074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0588103740 con 0x7f0588108750 2024-07-27T06:44:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.913+0000 7f056ffff640 1 -- 172.21.15.29:0/123601074 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f05800054a0 con 0x7f0588108750 2024-07-27T06:44:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.914+0000 7f056ffff640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f05700b3cb0 0x7f05700b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.914+0000 7f0586ffd640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f05700b3cb0 0x7f05700b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.914+0000 7f056ffff640 1 -- 172.21.15.29:0/123601074 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f05800e4370 con 0x7f0588108750 2024-07-27T06:44:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.914+0000 7f0586ffd640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f05700b3cb0 0x7f05700b61c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0578005d00 tx=0x7f057800b410 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:11.917+0000 7f056ffff640 1 -- 172.21.15.29:0/123601074 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0580016030 con 0x7f0588108750 2024-07-27T06:44:12.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.100+0000 7f058d735640 1 -- 172.21.15.29:0/123601074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f0588102920 con 0x7f0588108750 2024-07-27T06:44:12.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.100+0000 7f056ffff640 1 -- 172.21.15.29:0/123601074 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v51) ==== 86+0+133 (secure 0 0 0) 0x7f05800ab0e0 con 0x7f0588108750 2024-07-27T06:44:12.103 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:44:12.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.103+0000 7f058d735640 1 -- 172.21.15.29:0/123601074 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f05700b3cb0 msgr2=0x7f05700b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:12.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.103+0000 7f058d735640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f05700b3cb0 0x7f05700b61c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0578005d00 tx=0x7f057800b410 comp rx=0 tx=0).stop 2024-07-27T06:44:12.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.103+0000 7f058d735640 1 -- 172.21.15.29:0/123601074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0588108750 msgr2=0x7f058819f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.103+0000 7f058d735640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0588108750 0x7f058819f6d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f058000eaa0 tx=0x7f058000ef60 comp rx=0 tx=0).stop 2024-07-27T06:44:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.103+0000 7f058d735640 1 -- 172.21.15.29:0/123601074 shutdown_connections 2024-07-27T06:44:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.103+0000 7f058d735640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0588108750 0x7f058819f6d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.103+0000 7f058d735640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f05700b3cb0 0x7f05700b61c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.103+0000 7f058d735640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0588107e00 0x7f058819a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.103+0000 7f058d735640 1 --2- 172.21.15.29:0/123601074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0588102510 0x7f0588199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.103+0000 7f058d735640 1 -- 172.21.15.29:0/123601074 >> 172.21.15.29:0/123601074 conn(0x7f0588079190 msgr2=0x7f0588104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.103+0000 7f058d735640 1 -- 172.21.15.29:0/123601074 shutdown_connections 2024-07-27T06:44:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:12.103+0000 7f058d735640 1 -- 172.21.15.29:0/123601074 wait complete. 2024-07-27T06:44:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:12 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/123601074' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-27T06:44:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:12 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/123601074' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-27T06:44:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:12 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/123601074' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-27T06:44:12.856 INFO:teuthology.orchestra.run.smithi029.stdout:[{"version":51,"timestamp":"2024-07-27T06:42:56.845509+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2024-07-27T06:44:12.856 INFO:tasks.ceph_manager:config epoch is 51 2024-07-27T06:44:12.856 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-07-27T06:44:12.856 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-07-27T06:44:12.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph mgr dump --format=json 2024-07-27T06:44:13.125 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:13.559 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:13 smithi029 ceph-mon[32610]: pgmap v311: 97 pgs: 1 peering, 96 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 3 op/s; 195 B/s, 1 keys/s, 1 objects/s recovering 2024-07-27T06:44:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:13 smithi077 ceph-mon[37153]: pgmap v311: 97 pgs: 1 peering, 96 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 3 op/s; 195 B/s, 1 keys/s, 1 objects/s recovering 2024-07-27T06:44:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:13 smithi047 ceph-mon[36835]: pgmap v311: 97 pgs: 1 peering, 96 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 3 op/s; 195 B/s, 1 keys/s, 1 objects/s recovering 2024-07-27T06:44:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.896+0000 7f768d2d3640 1 -- 172.21.15.29:0/325339817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688108ed0 msgr2=0x7f76881092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.896+0000 7f768d2d3640 1 --2- 172.21.15.29:0/325339817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688108ed0 0x7f76881092e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f767c009900 tx=0x7f767c02a410 comp rx=0 tx=0).stop 2024-07-27T06:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.896+0000 7f768d2d3640 1 -- 172.21.15.29:0/325339817 shutdown_connections 2024-07-27T06:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.896+0000 7f768d2d3640 1 --2- 172.21.15.29:0/325339817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7688109820 0x7f7688110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.896+0000 7f768d2d3640 1 --2- 172.21.15.29:0/325339817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688108ed0 0x7f76881092e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.896+0000 7f768d2d3640 1 --2- 172.21.15.29:0/325339817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688103590 0x7f76881039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.896+0000 7f768d2d3640 1 -- 172.21.15.29:0/325339817 >> 172.21.15.29:0/325339817 conn(0x7f768807b280 msgr2=0x7f768807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.896+0000 7f768d2d3640 1 -- 172.21.15.29:0/325339817 shutdown_connections 2024-07-27T06:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.896+0000 7f768d2d3640 1 -- 172.21.15.29:0/325339817 wait complete. 2024-07-27T06:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.897+0000 7f768d2d3640 1 Processor -- start 2024-07-27T06:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.897+0000 7f768d2d3640 1 -- start start 2024-07-27T06:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f768d2d3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688103590 0x7f768819c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:14.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f768d2d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108ed0 0x7f768819c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f768d2d3640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7688109820 0x7f76881a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f768d2d3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7688113490 con 0x7f7688103590 2024-07-27T06:44:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f768d2d3640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7688113310 con 0x7f7688109820 2024-07-27T06:44:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f768d2d3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7688113610 con 0x7f7688108ed0 2024-07-27T06:44:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f7686ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688103590 0x7f768819c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f7686ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688103590 0x7f768819c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40738/0 (socket says 172.21.15.29:40738) 2024-07-27T06:44:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f7686ffd640 1 -- 172.21.15.29:0/911153609 learned_addr learned my addr 172.21.15.29:0/911153609 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f76877fe640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7688109820 0x7f76881a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f76867fc640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108ed0 0x7f768819c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f7686ffd640 1 -- 172.21.15.29:0/911153609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108ed0 msgr2=0x7f768819c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f7686ffd640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108ed0 0x7f768819c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f7686ffd640 1 -- 172.21.15.29:0/911153609 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7688109820 msgr2=0x7f76881a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f7686ffd640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7688109820 0x7f76881a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f7686ffd640 1 -- 172.21.15.29:0/911153609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76881a2030 con 0x7f7688103590 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f76867fc640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108ed0 0x7f768819c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f7686ffd640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688103590 0x7f768819c0c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f767800d950 tx=0x7f767800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f766ffff640 1 -- 172.21.15.29:0/911153609 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7678017070 con 0x7f7688103590 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.898+0000 7f768d2d3640 1 -- 172.21.15.29:0/911153609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76881a2320 con 0x7f7688103590 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.899+0000 7f766ffff640 1 -- 172.21.15.29:0/911153609 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7678009c30 con 0x7f7688103590 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.899+0000 7f768d2d3640 1 -- 172.21.15.29:0/911153609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76881a8a80 con 0x7f7688103590 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.899+0000 7f766ffff640 1 -- 172.21.15.29:0/911153609 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767800f620 con 0x7f7688103590 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.900+0000 7f768d2d3640 1 -- 172.21.15.29:0/911153609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76881039a0 con 0x7f7688103590 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.901+0000 7f766ffff640 1 -- 172.21.15.29:0/911153609 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f7678010460 con 0x7f7688103590 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.902+0000 7f766ffff640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f76700b3cb0 0x7f76700b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.902+0000 7f76867fc640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f76700b3cb0 0x7f76700b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:14.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.902+0000 7f766ffff640 1 -- 172.21.15.29:0/911153609 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f76780e3c60 con 0x7f7688103590 2024-07-27T06:44:14.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.903+0000 7f76867fc640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f76700b3cb0 0x7f76700b61c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f767c000c00 tx=0x7f767c031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:14.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:14.907+0000 7f766ffff640 1 -- 172.21.15.29:0/911153609 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f76780ab020 con 0x7f7688103590 2024-07-27T06:44:15.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.144+0000 7f768d2d3640 1 -- 172.21.15.29:0/911153609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f76881a8de0 con 0x7f7688103590 2024-07-27T06:44:15.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.149+0000 7f766ffff640 1 -- 172.21.15.29:0/911153609 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v21) ==== 74+0+291035 (secure 0 0 0) 0x7f76780cbe20 con 0x7f7688103590 2024-07-27T06:44:15.155 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:44:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.152+0000 7f768d2d3640 1 -- 172.21.15.29:0/911153609 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f76700b3cb0 msgr2=0x7f76700b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.152+0000 7f768d2d3640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f76700b3cb0 0x7f76700b61c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f767c000c00 tx=0x7f767c031400 comp rx=0 tx=0).stop 2024-07-27T06:44:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.152+0000 7f768d2d3640 1 -- 172.21.15.29:0/911153609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688103590 msgr2=0x7f768819c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.152+0000 7f768d2d3640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688103590 0x7f768819c0c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f767800d950 tx=0x7f767800de10 comp rx=0 tx=0).stop 2024-07-27T06:44:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.152+0000 7f768d2d3640 1 -- 172.21.15.29:0/911153609 shutdown_connections 2024-07-27T06:44:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.152+0000 7f768d2d3640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7688109820 0x7f76881a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.152+0000 7f768d2d3640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f76700b3cb0 0x7f76700b61c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.152+0000 7f768d2d3640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7688108ed0 0x7f768819c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:15.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.152+0000 7f768d2d3640 1 --2- 172.21.15.29:0/911153609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7688103590 0x7f768819c0c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:15.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.152+0000 7f768d2d3640 1 -- 172.21.15.29:0/911153609 >> 172.21.15.29:0/911153609 conn(0x7f768807b280 msgr2=0x7f7688106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:15.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.152+0000 7f768d2d3640 1 -- 172.21.15.29:0/911153609 shutdown_connections 2024-07-27T06:44:15.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:15.153+0000 7f768d2d3640 1 -- 172.21.15.29:0/911153609 wait complete. 2024-07-27T06:44:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:15 smithi029 ceph-mon[32610]: pgmap v312: 97 pgs: 1 peering, 96 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 7 op/s; 195 B/s, 1 keys/s, 1 objects/s recovering 2024-07-27T06:44:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:15 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/911153609' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-27T06:44:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:15 smithi077 ceph-mon[37153]: pgmap v312: 97 pgs: 1 peering, 96 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 7 op/s; 195 B/s, 1 keys/s, 1 objects/s recovering 2024-07-27T06:44:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:15 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/911153609' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-27T06:44:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:15 smithi047 ceph-mon[36835]: pgmap v312: 97 pgs: 1 peering, 96 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 7 op/s; 195 B/s, 1 keys/s, 1 objects/s recovering 2024-07-27T06:44:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:15 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/911153609' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-27T06:44:15.869 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"flags":0,"active_gid":14208,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":2980069896},{"type":"v1","addr":"172.21.15.29:6801","nonce":2980069896}]},"active_addr":"172.21.15.29:6801/2980069896","active_change":"2024-07-27T06:36:07.527632+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24125,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24136,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.29:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1345510174}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2150676079}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":878765798}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2165176928}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1152281543}]}]} 2024-07-27T06:44:15.875 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-07-27T06:44:15.875 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-07-27T06:44:15.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd dump --format=json 2024-07-27T06:44:16.151 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:17.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:17 smithi029 ceph-mon[32610]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2 op/s; 195 B/s, 1 keys/s, 1 objects/s recovering 2024-07-27T06:44:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:17 smithi077 ceph-mon[37153]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2 op/s; 195 B/s, 1 keys/s, 1 objects/s recovering 2024-07-27T06:44:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:17 smithi047 ceph-mon[36835]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2 op/s; 195 B/s, 1 keys/s, 1 objects/s recovering 2024-07-27T06:44:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.027+0000 7f345d654640 1 -- 172.21.15.29:0/1139244153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f345810fb30 msgr2=0x7f3458111f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.027+0000 7f345d654640 1 --2- 172.21.15.29:0/1139244153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f345810fb30 0x7f3458111f50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3450009ca0 tx=0x7f345002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:44:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.027+0000 7f345d654640 1 -- 172.21.15.29:0/1139244153 shutdown_connections 2024-07-27T06:44:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.027+0000 7f345d654640 1 --2- 172.21.15.29:0/1139244153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f345810fb30 0x7f3458111f50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.027+0000 7f345d654640 1 --2- 172.21.15.29:0/1139244153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3458075af0 0x7f345810f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.027+0000 7f345d654640 1 --2- 172.21.15.29:0/1139244153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34580751a0 0x7f34580755b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.027+0000 7f345d654640 1 -- 172.21.15.29:0/1139244153 >> 172.21.15.29:0/1139244153 conn(0x7f345807b280 msgr2=0x7f345807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.027+0000 7f345d654640 1 -- 172.21.15.29:0/1139244153 shutdown_connections 2024-07-27T06:44:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.027+0000 7f345d654640 1 -- 172.21.15.29:0/1139244153 wait complete. 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.027+0000 7f345d654640 1 Processor -- start 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f345d654640 1 -- start start 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f345d654640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34580751a0 0x7f345819c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f345d654640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3458075af0 0x7f345819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f345d654640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f345810fb30 0x7f34581a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f345d654640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f34581156d0 con 0x7f345810fb30 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f345d654640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3458115550 con 0x7f3458075af0 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f345d654640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3458115850 con 0x7f34580751a0 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f34577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f345810fb30 0x7f34581a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f34577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f345810fb30 0x7f34581a19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40748/0 (socket says 172.21.15.29:40748) 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f34577fe640 1 -- 172.21.15.29:0/2141332132 learned_addr learned my addr 172.21.15.29:0/2141332132 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f3456ffd640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34580751a0 0x7f345819c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f34567fc640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3458075af0 0x7f345819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f34577fe640 1 -- 172.21.15.29:0/2141332132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34580751a0 msgr2=0x7f345819c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f34577fe640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34580751a0 0x7f345819c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f34577fe640 1 -- 172.21.15.29:0/2141332132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3458075af0 msgr2=0x7f345819c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f34577fe640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3458075af0 0x7f345819c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f34577fe640 1 -- 172.21.15.29:0/2141332132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34581a2110 con 0x7f345810fb30 2024-07-27T06:44:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.028+0000 7f34567fc640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3458075af0 0x7f345819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:44:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.029+0000 7f34577fe640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f345810fb30 0x7f34581a19e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3450009430 tx=0x7f34500316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.029+0000 7f344ffff640 1 -- 172.21.15.29:0/2141332132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3450003b70 con 0x7f345810fb30 2024-07-27T06:44:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.029+0000 7f345d654640 1 -- 172.21.15.29:0/2141332132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34581a23a0 con 0x7f345810fb30 2024-07-27T06:44:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.029+0000 7f344ffff640 1 -- 172.21.15.29:0/2141332132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3450003d10 con 0x7f345810fb30 2024-07-27T06:44:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.029+0000 7f344ffff640 1 -- 172.21.15.29:0/2141332132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345002fb20 con 0x7f345810fb30 2024-07-27T06:44:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.029+0000 7f345d654640 1 -- 172.21.15.29:0/2141332132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34581a8b60 con 0x7f345810fb30 2024-07-27T06:44:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.030+0000 7f345d654640 1 -- 172.21.15.29:0/2141332132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34580755b0 con 0x7f345810fb30 2024-07-27T06:44:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.032+0000 7f344ffff640 1 -- 172.21.15.29:0/2141332132 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f3450004580 con 0x7f345810fb30 2024-07-27T06:44:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.033+0000 7f344ffff640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f34440b3cb0 0x7f34440b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.033+0000 7f3456ffd640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f34440b3cb0 0x7f34440b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.033+0000 7f344ffff640 1 -- 172.21.15.29:0/2141332132 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f3450105410 con 0x7f345810fb30 2024-07-27T06:44:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.033+0000 7f3456ffd640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f34440b3cb0 0x7f34440b61c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3448005fd0 tx=0x7f3448005850 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.036+0000 7f344ffff640 1 -- 172.21.15.29:0/2141332132 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f345003b410 con 0x7f345810fb30 2024-07-27T06:44:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.219+0000 7f345d654640 1 -- 172.21.15.29:0/2141332132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f34581a8d30 con 0x7f345810fb30 2024-07-27T06:44:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.221+0000 7f344ffff640 1 -- 172.21.15.29:0/2141332132 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22933 (secure 0 0 0) 0x7f34500cc940 con 0x7f345810fb30 2024-07-27T06:44:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:44:18.223 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","created":"2024-07-27T06:33:57.535327+0000","modified":"2024-07-27T06:44:09.220470+0000","last_up_change":"2024-07-27T06:42:05.757156+0000","last_in_change":"2024-07-27T06:41:46.423732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T06:38:21.606527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-27T06:43:07.755490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-27T06:43:11.658694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06833d24-2e1a-4199-9338-7d6883d1bf9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6803","nonce":386121641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6805","nonce":386121641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6809","nonce":386121641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6807","nonce":386121641}]},"public_addr":"172.21.15.29:6803/386121641","cluster_addr":"172.21.15.29:6805/386121641","heartbeat_back_addr":"172.21.15.29:6809/386121641","heartbeat_front_addr":"172.21.15.29:6807/386121641","state":["exists","up"]},{"osd":1,"uuid":"906aba19-80dc-41b5-953c-af6dca604b05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6801","nonce":2795953749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6803","nonce":2795953749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6807","nonce":2795953749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6805","nonce":2795953749}]},"public_addr":"172.21.15.47:6801/2795953749","cluster_addr":"172.21.15.47:6803/2795953749","heartbeat_back_addr":"172.21.15.47:6807/2795953749","heartbeat_front_addr":"172.21.15.47:6805/2795953749","state":["exists","up"]},{"osd":2,"uuid":"194ae5a5-aebf-4cff-9b99-585a3073a269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6801","nonce":2202565078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6803","nonce":2202565078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6807","nonce":2202565078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6805","nonce":2202565078}]},"public_addr":"172.21.15.77:6801/2202565078","cluster_addr":"172.21.15.77:6803/2202565078","heartbeat_back_addr":"172.21.15.77:6807/2202565078","heartbeat_front_addr":"172.21.15.77:6805/2202565078","state":["exists","up"]},{"osd":3,"uuid":"3c22ce9c-963e-4f72-86e5-360e781526db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6811","nonce":3757476312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6813","nonce":3757476312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6817","nonce":3757476312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6815","nonce":3757476312}]},"public_addr":"172.21.15.29:6811/3757476312","cluster_addr":"172.21.15.29:6813/3757476312","heartbeat_back_addr":"172.21.15.29:6817/3757476312","heartbeat_front_addr":"172.21.15.29:6815/3757476312","state":["exists","up"]},{"osd":4,"uuid":"cd4b585f-e7a0-4d82-873d-3a9269ebac3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6809","nonce":1379363720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6811","nonce":1379363720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6815","nonce":1379363720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6813","nonce":1379363720}]},"public_addr":"172.21.15.47:6809/1379363720","cluster_addr":"172.21.15.47:6811/1379363720","heartbeat_back_addr":"172.21.15.47:6815/1379363720","heartbeat_front_addr":"172.21.15.47:6813/1379363720","state":["exists","up"]},{"osd":5,"uuid":"40a3eb14-dfaa-4523-8c28-d74c19293c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6809","nonce":1338438780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6811","nonce":1338438780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6815","nonce":1338438780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6813","nonce":1338438780}]},"public_addr":"172.21.15.77:6809/1338438780","cluster_addr":"172.21.15.77:6811/1338438780","heartbeat_back_addr":"172.21.15.77:6815/1338438780","heartbeat_front_addr":"172.21.15.77:6813/1338438780","state":["exists","up"]},{"osd":6,"uuid":"91e85be2-10d1-4209-aa33-17da8f8bd8bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6819","nonce":2679402148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6821","nonce":2679402148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6825","nonce":2679402148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6823","nonce":2679402148}]},"public_addr":"172.21.15.29:6819/2679402148","cluster_addr":"172.21.15.29:6821/2679402148","heartbeat_back_addr":"172.21.15.29:6825/2679402148","heartbeat_front_addr":"172.21.15.29:6823/2679402148","state":["exists","up"]},{"osd":7,"uuid":"c655b450-0cc4-45d4-95ea-93a57fe8e4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6817","nonce":2439195589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6819","nonce":2439195589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6823","nonce":2439195589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6821","nonce":2439195589}]},"public_addr":"172.21.15.47:6817/2439195589","cluster_addr":"172.21.15.47:6819/2439195589","heartbeat_back_addr":"172.21.15.47:6823/2439195589","heartbeat_front_addr":"172.21.15.47:6821/2439195589","state":["exists","up"]},{"osd":8,"uuid":"a7a95b51-f32c-433e-a197-feb6d47b6c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6817","nonce":1422290905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6819","nonce":1422290905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6823","nonce":1422290905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6821","nonce":1422290905}]},"public_addr":"172.21.15.77:6817/1422290905","cluster_addr":"172.21.15.77:6819/1422290905","heartbeat_back_addr":"172.21.15.77:6823/1422290905","heartbeat_front_addr":"172.21.15.77:6821/1422290905","state":["exists","up"]},{"osd":9,"uuid":"f352242b-0d2e-41b3-9c83-436cf432b0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6827","nonce":1307316384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6829","nonce":1307316384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6833","nonce":1307316384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6831","nonce":1307316384}]},"public_addr":"172.21.15.29:6827/1307316384","cluster_addr":"172.21.15.29:6829/1307316384","heartbeat_back_addr":"172.21.15.29:6833/1307316384","heartbeat_front_addr":"172.21.15.29:6831/1307316384","state":["exists","up"]},{"osd":10,"uuid":"4148c2df-c3e9-4a47-88f7-36cf3eab8d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6825","nonce":2133726850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6827","nonce":2133726850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6831","nonce":2133726850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6829","nonce":2133726850}]},"public_addr":"172.21.15.47:6825/2133726850","cluster_addr":"172.21.15.47:6827/2133726850","heartbeat_back_addr":"172.21.15.47:6831/2133726850","heartbeat_front_addr":"172.21.15.47:6829/2133726850","state":["exists","up"]},{"osd":11,"uuid":"5e8b1da3-5f65-4fe9-8722-b35921cbd8c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6825","nonce":3228299327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6827","nonce":3228299327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6831","nonce":3228299327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6829","nonce":3228299327}]},"public_addr":"172.21.15.77:6825/3228299327","cluster_addr":"172.21.15.77:6827/3228299327","heartbeat_back_addr":"172.21.15.77:6831/3228299327","heartbeat_front_addr":"172.21.15.77:6829/3228299327","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:32.525206+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:56.489640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:18.887845+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:43.727678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:08.755553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:32.923461+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:59.298439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:23.944848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:48.883050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:14.566310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:39.557809+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:42:03.813903+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7},{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2378279024":"2024-07-28T06:36:07.527503+0000","172.21.15.29:6801/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3527133965":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2262661620":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/3156863587":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2424241521":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3881857930":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/398508875":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6801/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6800/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2679856281":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6801/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2708687265":"2024-07-28T06:34:58.250198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T06:44:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.223+0000 7f345d654640 1 -- 172.21.15.29:0/2141332132 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f34440b3cb0 msgr2=0x7f34440b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.223+0000 7f345d654640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f34440b3cb0 0x7f34440b61c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3448005fd0 tx=0x7f3448005850 comp rx=0 tx=0).stop 2024-07-27T06:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.223+0000 7f345d654640 1 -- 172.21.15.29:0/2141332132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f345810fb30 msgr2=0x7f34581a19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.223+0000 7f345d654640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f345810fb30 0x7f34581a19e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3450009430 tx=0x7f34500316f0 comp rx=0 tx=0).stop 2024-07-27T06:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.223+0000 7f345d654640 1 -- 172.21.15.29:0/2141332132 shutdown_connections 2024-07-27T06:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.223+0000 7f345d654640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f345810fb30 0x7f34581a19e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.223+0000 7f345d654640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3458075af0 0x7f345819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.223+0000 7f345d654640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f34440b3cb0 0x7f34440b61c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.223+0000 7f345d654640 1 --2- 172.21.15.29:0/2141332132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34580751a0 0x7f345819c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.223+0000 7f345d654640 1 -- 172.21.15.29:0/2141332132 >> 172.21.15.29:0/2141332132 conn(0x7f345807b280 msgr2=0x7f34581096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.223+0000 7f345d654640 1 -- 172.21.15.29:0/2141332132 shutdown_connections 2024-07-27T06:44:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:18.223+0000 7f345d654640 1 -- 172.21.15.29:0/2141332132 wait complete. 2024-07-27T06:44:18.554 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:18 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2141332132' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:44:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:18 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2141332132' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:44:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:18 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2141332132' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:44:18.954 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-07-27T06:44:18.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd dump --format=json 2024-07-27T06:44:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:19.599 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:19 smithi029 ceph-mon[32610]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 6 op/s; 165 B/s, 1 keys/s, 1 objects/s recovering 2024-07-27T06:44:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:19 smithi077 ceph-mon[37153]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 6 op/s; 165 B/s, 1 keys/s, 1 objects/s recovering 2024-07-27T06:44:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:19 smithi047 ceph-mon[36835]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 6 op/s; 165 B/s, 1 keys/s, 1 objects/s recovering 2024-07-27T06:44:21.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.965+0000 7f9eadab3640 1 -- 172.21.15.29:0/36524804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea81024b0 msgr2=0x7f9ea81028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:21.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.965+0000 7f9eadab3640 1 --2- 172.21.15.29:0/36524804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea81024b0 0x7f9ea81028c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009ca0 tx=0x7f9e9802a8c0 comp rx=0 tx=0).stop 2024-07-27T06:44:21.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.965+0000 7f9eadab3640 1 -- 172.21.15.29:0/36524804 shutdown_connections 2024-07-27T06:44:21.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.965+0000 7f9eadab3640 1 --2- 172.21.15.29:0/36524804 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ea8108740 0x7f9ea810ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:21.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.965+0000 7f9eadab3640 1 --2- 172.21.15.29:0/36524804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ea8107df0 0x7f9ea8108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:21.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.965+0000 7f9eadab3640 1 --2- 172.21.15.29:0/36524804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea81024b0 0x7f9ea81028c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:21.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.965+0000 7f9eadab3640 1 -- 172.21.15.29:0/36524804 >> 172.21.15.29:0/36524804 conn(0x7f9ea80791a0 msgr2=0x7f9ea80795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:21.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.965+0000 7f9eadab3640 1 -- 172.21.15.29:0/36524804 shutdown_connections 2024-07-27T06:44:21.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.965+0000 7f9eadab3640 1 -- 172.21.15.29:0/36524804 wait complete. 2024-07-27T06:44:21.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.966+0000 7f9eadab3640 1 Processor -- start 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.966+0000 7f9eadab3640 1 -- start start 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.966+0000 7f9eadab3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea81024b0 0x7f9ea819ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.966+0000 7f9eadab3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ea8107df0 0x7f9ea819b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.966+0000 7f9eadab3640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ea8108740 0x7f9ea819b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.966+0000 7f9eadab3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea8112460 con 0x7f9ea81024b0 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.966+0000 7f9eadab3640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea81122e0 con 0x7f9ea8108740 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.966+0000 7f9eadab3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea81125e0 con 0x7f9ea8107df0 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.966+0000 7f9ea77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea81024b0 0x7f9ea819ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.966+0000 7f9ea77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea81024b0 0x7f9ea819ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45942/0 (socket says 172.21.15.29:45942) 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.966+0000 7f9ea77fe640 1 -- 172.21.15.29:0/1109183525 learned_addr learned my addr 172.21.15.29:0/1109183525 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.966+0000 7f9ea7fff640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ea8108740 0x7f9ea819b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9ea6ffd640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ea8107df0 0x7f9ea819b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:21.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9ea77fe640 1 -- 172.21.15.29:0/1109183525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ea8107df0 msgr2=0x7f9ea819b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:21.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9ea77fe640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ea8107df0 0x7f9ea819b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:21.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9ea77fe640 1 -- 172.21.15.29:0/1109183525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ea8108740 msgr2=0x7f9ea819b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:21.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9ea77fe640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ea8108740 0x7f9ea819b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:21.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9ea77fe640 1 -- 172.21.15.29:0/1109183525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea81a5fa0 con 0x7f9ea81024b0 2024-07-27T06:44:21.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9ea7fff640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ea8108740 0x7f9ea819b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:44:21.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9ea77fe640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea81024b0 0x7f9ea819ad10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9e98005e30 tx=0x7f9e98031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:21.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9ea4ff9640 1 -- 172.21.15.29:0/1109183525 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e98003b70 con 0x7f9ea81024b0 2024-07-27T06:44:21.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9ea4ff9640 1 -- 172.21.15.29:0/1109183525 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e98003d10 con 0x7f9ea81024b0 2024-07-27T06:44:21.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9eadab3640 1 -- 172.21.15.29:0/1109183525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea81a6230 con 0x7f9ea81024b0 2024-07-27T06:44:21.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9ea4ff9640 1 -- 172.21.15.29:0/1109183525 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9802fa30 con 0x7f9ea81024b0 2024-07-27T06:44:21.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.967+0000 7f9eadab3640 1 -- 172.21.15.29:0/1109183525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea81a6770 con 0x7f9ea81024b0 2024-07-27T06:44:21.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.968+0000 7f9eadab3640 1 -- 172.21.15.29:0/1109183525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea8103730 con 0x7f9ea81024b0 2024-07-27T06:44:21.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.970+0000 7f9ea4ff9640 1 -- 172.21.15.29:0/1109183525 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f9e9802fbd0 con 0x7f9ea81024b0 2024-07-27T06:44:21.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.971+0000 7f9ea4ff9640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9e940b3ce0 0x7f9e940b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:21.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.971+0000 7f9ea6ffd640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9e940b3ce0 0x7f9e940b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:21.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.971+0000 7f9ea4ff9640 1 -- 172.21.15.29:0/1109183525 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f9e98105240 con 0x7f9ea81024b0 2024-07-27T06:44:21.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.971+0000 7f9ea6ffd640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9e940b3ce0 0x7f9e940b61f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9ea81079b0 tx=0x7f9e90005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:21.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:20.974+0000 7f9ea4ff9640 1 -- 172.21.15.29:0/1109183525 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9e980cc770 con 0x7f9ea81024b0 2024-07-27T06:44:21.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.158+0000 7f9eadab3640 1 -- 172.21.15.29:0/1109183525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ea806b9c0 con 0x7f9ea81024b0 2024-07-27T06:44:21.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.160+0000 7f9ea4ff9640 1 -- 172.21.15.29:0/1109183525 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22933 (secure 0 0 0) 0x7f9e980cc070 con 0x7f9ea81024b0 2024-07-27T06:44:21.162 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:44:21.162 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","created":"2024-07-27T06:33:57.535327+0000","modified":"2024-07-27T06:44:09.220470+0000","last_up_change":"2024-07-27T06:42:05.757156+0000","last_in_change":"2024-07-27T06:41:46.423732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T06:38:21.606527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-27T06:43:07.755490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-27T06:43:11.658694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06833d24-2e1a-4199-9338-7d6883d1bf9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6803","nonce":386121641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6805","nonce":386121641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6809","nonce":386121641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6807","nonce":386121641}]},"public_addr":"172.21.15.29:6803/386121641","cluster_addr":"172.21.15.29:6805/386121641","heartbeat_back_addr":"172.21.15.29:6809/386121641","heartbeat_front_addr":"172.21.15.29:6807/386121641","state":["exists","up"]},{"osd":1,"uuid":"906aba19-80dc-41b5-953c-af6dca604b05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6801","nonce":2795953749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6803","nonce":2795953749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6807","nonce":2795953749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6805","nonce":2795953749}]},"public_addr":"172.21.15.47:6801/2795953749","cluster_addr":"172.21.15.47:6803/2795953749","heartbeat_back_addr":"172.21.15.47:6807/2795953749","heartbeat_front_addr":"172.21.15.47:6805/2795953749","state":["exists","up"]},{"osd":2,"uuid":"194ae5a5-aebf-4cff-9b99-585a3073a269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6801","nonce":2202565078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6803","nonce":2202565078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6807","nonce":2202565078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6805","nonce":2202565078}]},"public_addr":"172.21.15.77:6801/2202565078","cluster_addr":"172.21.15.77:6803/2202565078","heartbeat_back_addr":"172.21.15.77:6807/2202565078","heartbeat_front_addr":"172.21.15.77:6805/2202565078","state":["exists","up"]},{"osd":3,"uuid":"3c22ce9c-963e-4f72-86e5-360e781526db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6811","nonce":3757476312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6813","nonce":3757476312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6817","nonce":3757476312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6815","nonce":3757476312}]},"public_addr":"172.21.15.29:6811/3757476312","cluster_addr":"172.21.15.29:6813/3757476312","heartbeat_back_addr":"172.21.15.29:6817/3757476312","heartbeat_front_addr":"172.21.15.29:6815/3757476312","state":["exists","up"]},{"osd":4,"uuid":"cd4b585f-e7a0-4d82-873d-3a9269ebac3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6809","nonce":1379363720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6811","nonce":1379363720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6815","nonce":1379363720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6813","nonce":1379363720}]},"public_addr":"172.21.15.47:6809/1379363720","cluster_addr":"172.21.15.47:6811/1379363720","heartbeat_back_addr":"172.21.15.47:6815/1379363720","heartbeat_front_addr":"172.21.15.47:6813/1379363720","state":["exists","up"]},{"osd":5,"uuid":"40a3eb14-dfaa-4523-8c28-d74c19293c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6809","nonce":1338438780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6811","nonce":1338438780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6815","nonce":1338438780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6813","nonce":1338438780}]},"public_addr":"172.21.15.77:6809/1338438780","cluster_addr":"172.21.15.77:6811/1338438780","heartbeat_back_addr":"172.21.15.77:6815/1338438780","heartbeat_front_addr":"172.21.15.77:6813/1338438780","state":["exists","up"]},{"osd":6,"uuid":"91e85be2-10d1-4209-aa33-17da8f8bd8bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6819","nonce":2679402148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6821","nonce":2679402148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6825","nonce":2679402148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6823","nonce":2679402148}]},"public_addr":"172.21.15.29:6819/2679402148","cluster_addr":"172.21.15.29:6821/2679402148","heartbeat_back_addr":"172.21.15.29:6825/2679402148","heartbeat_front_addr":"172.21.15.29:6823/2679402148","state":["exists","up"]},{"osd":7,"uuid":"c655b450-0cc4-45d4-95ea-93a57fe8e4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6817","nonce":2439195589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6819","nonce":2439195589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6823","nonce":2439195589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6821","nonce":2439195589}]},"public_addr":"172.21.15.47:6817/2439195589","cluster_addr":"172.21.15.47:6819/2439195589","heartbeat_back_addr":"172.21.15.47:6823/2439195589","heartbeat_front_addr":"172.21.15.47:6821/2439195589","state":["exists","up"]},{"osd":8,"uuid":"a7a95b51-f32c-433e-a197-feb6d47b6c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6817","nonce":1422290905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6819","nonce":1422290905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6823","nonce":1422290905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6821","nonce":1422290905}]},"public_addr":"172.21.15.77:6817/1422290905","cluster_addr":"172.21.15.77:6819/1422290905","heartbeat_back_addr":"172.21.15.77:6823/1422290905","heartbeat_front_addr":"172.21.15.77:6821/1422290905","state":["exists","up"]},{"osd":9,"uuid":"f352242b-0d2e-41b3-9c83-436cf432b0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6827","nonce":1307316384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6829","nonce":1307316384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6833","nonce":1307316384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6831","nonce":1307316384}]},"public_addr":"172.21.15.29:6827/1307316384","cluster_addr":"172.21.15.29:6829/1307316384","heartbeat_back_addr":"172.21.15.29:6833/1307316384","heartbeat_front_addr":"172.21.15.29:6831/1307316384","state":["exists","up"]},{"osd":10,"uuid":"4148c2df-c3e9-4a47-88f7-36cf3eab8d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6825","nonce":2133726850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6827","nonce":2133726850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6831","nonce":2133726850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6829","nonce":2133726850}]},"public_addr":"172.21.15.47:6825/2133726850","cluster_addr":"172.21.15.47:6827/2133726850","heartbeat_back_addr":"172.21.15.47:6831/2133726850","heartbeat_front_addr":"172.21.15.47:6829/2133726850","state":["exists","up"]},{"osd":11,"uuid":"5e8b1da3-5f65-4fe9-8722-b35921cbd8c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6825","nonce":3228299327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6827","nonce":3228299327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6831","nonce":3228299327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6829","nonce":3228299327}]},"public_addr":"172.21.15.77:6825/3228299327","cluster_addr":"172.21.15.77:6827/3228299327","heartbeat_back_addr":"172.21.15.77:6831/3228299327","heartbeat_front_addr":"172.21.15.77:6829/3228299327","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:32.525206+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:56.489640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:18.887845+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:43.727678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:08.755553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:32.923461+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:59.298439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:23.944848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:48.883050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:14.566310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:39.557809+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:42:03.813903+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7},{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2378279024":"2024-07-28T06:36:07.527503+0000","172.21.15.29:6801/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3527133965":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2262661620":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/3156863587":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2424241521":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3881857930":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/398508875":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6801/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6800/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2679856281":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6801/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2708687265":"2024-07-28T06:34:58.250198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T06:44:21.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.162+0000 7f9eadab3640 1 -- 172.21.15.29:0/1109183525 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9e940b3ce0 msgr2=0x7f9e940b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:21.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.162+0000 7f9eadab3640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9e940b3ce0 0x7f9e940b61f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9ea81079b0 tx=0x7f9e90005f90 comp rx=0 tx=0).stop 2024-07-27T06:44:21.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.162+0000 7f9eadab3640 1 -- 172.21.15.29:0/1109183525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea81024b0 msgr2=0x7f9ea819ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:21.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.162+0000 7f9eadab3640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea81024b0 0x7f9ea819ad10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9e98005e30 tx=0x7f9e98031840 comp rx=0 tx=0).stop 2024-07-27T06:44:21.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.162+0000 7f9eadab3640 1 -- 172.21.15.29:0/1109183525 shutdown_connections 2024-07-27T06:44:21.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.162+0000 7f9eadab3640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ea8108740 0x7f9ea819b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:21.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.162+0000 7f9eadab3640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9e940b3ce0 0x7f9e940b61f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:21.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.162+0000 7f9eadab3640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ea8107df0 0x7f9ea819b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:21.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.162+0000 7f9eadab3640 1 --2- 172.21.15.29:0/1109183525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ea81024b0 0x7f9ea819ad10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:21.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.162+0000 7f9eadab3640 1 -- 172.21.15.29:0/1109183525 >> 172.21.15.29:0/1109183525 conn(0x7f9ea80791a0 msgr2=0x7f9ea8105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:21.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.162+0000 7f9eadab3640 1 -- 172.21.15.29:0/1109183525 shutdown_connections 2024-07-27T06:44:21.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:21.162+0000 7f9eadab3640 1 -- 172.21.15.29:0/1109183525 wait complete. 2024-07-27T06:44:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:21 smithi077 ceph-mon[37153]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 5 op/s; 149 B/s, 0 keys/s, 0 objects/s recovering 2024-07-27T06:44:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:21 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1109183525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:44:21.638 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:21 smithi029 ceph-mon[32610]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 5 op/s; 149 B/s, 0 keys/s, 0 objects/s recovering 2024-07-27T06:44:21.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:21 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1109183525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:44:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:21 smithi047 ceph-mon[36835]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 5 op/s; 149 B/s, 0 keys/s, 0 objects/s recovering 2024-07-27T06:44:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:21 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1109183525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:44:22.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-07-27T06:44:22.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-07-27T06:44:22.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-07-27T06:44:22.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-07-27T06:44:22.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-07-27T06:44:22.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-07-27T06:44:22.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-07-27T06:44:22.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-07-27T06:44:22.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-07-27T06:44:22.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-07-27T06:44:22.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-07-27T06:44:22.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-07-27T06:44:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:22.800 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:22.805 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:23.516 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:23 smithi029 ceph-mon[32610]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s; 130 B/s, 0 keys/s, 0 objects/s recovering 2024-07-27T06:44:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:23 smithi077 ceph-mon[37153]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s; 130 B/s, 0 keys/s, 0 objects/s recovering 2024-07-27T06:44:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:23 smithi047 ceph-mon[36835]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 6 op/s; 130 B/s, 0 keys/s, 0 objects/s recovering 2024-07-27T06:44:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:25 smithi077 ceph-mon[37153]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:44:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:25 smithi047 ceph-mon[36835]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:44:25.708 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:25 smithi029 ceph-mon[32610]: pgmap v317: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:44:25.713 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:25.714 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:25.714 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:25.718 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:25.719 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:27.348 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:27 smithi077 ceph-mon[37153]: pgmap v318: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:27 smithi047 ceph-mon[36835]: pgmap v318: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:27 smithi029 ceph-mon[32610]: pgmap v318: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:29 smithi077 ceph-mon[37153]: pgmap v319: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:44:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:29 smithi047 ceph-mon[36835]: pgmap v319: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:44:29.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:29 smithi029 ceph-mon[32610]: pgmap v319: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:44:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:31 smithi077 ceph-mon[37153]: pgmap v320: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:31 smithi047 ceph-mon[36835]: pgmap v320: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:31 smithi029 ceph-mon[32610]: pgmap v320: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:32.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.718+0000 7f3b56004640 1 -- 172.21.15.29:0/2945784027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b50103590 msgr2=0x7f3b501039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.718+0000 7f3b56004640 1 --2- 172.21.15.29:0/2945784027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b50103590 0x7f3b501039a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3b40009900 tx=0x7f3b4002a410 comp rx=0 tx=0).stop 2024-07-27T06:44:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.718+0000 7f3b56004640 1 -- 172.21.15.29:0/2945784027 shutdown_connections 2024-07-27T06:44:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.718+0000 7f3b56004640 1 --2- 172.21.15.29:0/2945784027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b50109820 0x7f3b50110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.718+0000 7f3b56004640 1 --2- 172.21.15.29:0/2945784027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b50108ed0 0x7f3b501092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.718+0000 7f3b56004640 1 --2- 172.21.15.29:0/2945784027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b50103590 0x7f3b501039a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.718+0000 7f3b56004640 1 -- 172.21.15.29:0/2945784027 >> 172.21.15.29:0/2945784027 conn(0x7f3b5007b280 msgr2=0x7f3b5007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.718+0000 7f3b56004640 1 -- 172.21.15.29:0/2945784027 shutdown_connections 2024-07-27T06:44:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.718+0000 7f3b56004640 1 -- 172.21.15.29:0/2945784027 wait complete. 2024-07-27T06:44:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b56004640 1 Processor -- start 2024-07-27T06:44:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b56004640 1 -- start start 2024-07-27T06:44:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b56004640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b50103590 0x7f3b50075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b56004640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b50108ed0 0x7f3b50075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b56004640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b50109820 0x7f3b50075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b56004640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3b501134d0 con 0x7f3b50109820 2024-07-27T06:44:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b56004640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3b50113350 con 0x7f3b50103590 2024-07-27T06:44:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b56004640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3b50113650 con 0x7f3b50108ed0 2024-07-27T06:44:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b50109820 0x7f3b50075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b50109820 0x7f3b50075bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36058/0 (socket says 172.21.15.29:36058) 2024-07-27T06:44:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b4ffff640 1 -- 172.21.15.29:0/2849940198 learned_addr learned my addr 172.21.15.29:0/2849940198 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b47fff640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b50108ed0 0x7f3b50075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b4f7fe640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b50103590 0x7f3b50075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b4ffff640 1 -- 172.21.15.29:0/2849940198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b50108ed0 msgr2=0x7f3b50075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b4ffff640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b50108ed0 0x7f3b50075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b4ffff640 1 -- 172.21.15.29:0/2849940198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b50103590 msgr2=0x7f3b50075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b4ffff640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b50103590 0x7f3b50075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b4ffff640 1 -- 172.21.15.29:0/2849940198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b501a8090 con 0x7f3b50109820 2024-07-27T06:44:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.719+0000 7f3b4f7fe640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b50103590 0x7f3b50075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:44:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.720+0000 7f3b4ffff640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b50109820 0x7f3b50075bc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800eaa0 tx=0x7f3b4800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.720+0000 7f3b4d7fa640 1 -- 172.21.15.29:0/2849940198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b48004d20 con 0x7f3b50109820 2024-07-27T06:44:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.720+0000 7f3b4d7fa640 1 -- 172.21.15.29:0/2849940198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b48005020 con 0x7f3b50109820 2024-07-27T06:44:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.720+0000 7f3b56004640 1 -- 172.21.15.29:0/2849940198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b501a8380 con 0x7f3b50109820 2024-07-27T06:44:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.720+0000 7f3b4d7fa640 1 -- 172.21.15.29:0/2849940198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b480116b0 con 0x7f3b50109820 2024-07-27T06:44:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.720+0000 7f3b56004640 1 -- 172.21.15.29:0/2849940198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b501a88c0 con 0x7f3b50109820 2024-07-27T06:44:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.721+0000 7f3b56004640 1 -- 172.21.15.29:0/2849940198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3b50063540 con 0x7f3b50109820 2024-07-27T06:44:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.723+0000 7f3b4d7fa640 1 -- 172.21.15.29:0/2849940198 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f3b480051d0 con 0x7f3b50109820 2024-07-27T06:44:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.723+0000 7f3b4d7fa640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3b3c0b3d30 0x7f3b3c0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.723+0000 7f3b4f7fe640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3b3c0b3d30 0x7f3b3c0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.724+0000 7f3b4d7fa640 1 -- 172.21.15.29:0/2849940198 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f3b480e44d0 con 0x7f3b50109820 2024-07-27T06:44:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.724+0000 7f3b4d7fa640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148] conn(0x7f3b3c0bdbb0 0x7f3b3c0c00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.724+0000 7f3b4d7fa640 1 -- 172.21.15.29:0/2849940198 --> [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b48014ec0 con 0x7f3b3c0bdbb0 2024-07-27T06:44:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.724+0000 7f3b4d7fa640 1 -- 172.21.15.29:0/2849940198 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f3b480e4920 con 0x7f3b50109820 2024-07-27T06:44:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.724+0000 7f3b47fff640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148] conn(0x7f3b3c0bdbb0 0x7f3b3c0c00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.724+0000 7f3b4f7fe640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3b3c0b3d30 0x7f3b3c0b6240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3b40000c00 tx=0x7f3b40009680 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.724+0000 7f3b47fff640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148] conn(0x7f3b3c0bdbb0 0x7f3b3c0c00a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:32.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.725+0000 7f3b4d7fa640 1 -- 172.21.15.29:0/2849940198 <== osd.6 v2:172.21.15.29:6818/2679402148 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3b48014ec0 con 0x7f3b3c0bdbb0 2024-07-27T06:44:32.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.740+0000 7f3b56004640 1 -- 172.21.15.29:0/2849940198 --> [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3b500639c0 con 0x7f3b3c0bdbb0 2024-07-27T06:44:32.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.740+0000 7f3b4d7fa640 1 -- 172.21.15.29:0/2849940198 <== osd.6 v2:172.21.15.29:6818/2679402148 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3b500639c0 con 0x7f3b3c0bdbb0 2024-07-27T06:44:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.741+0000 7f3b56004640 1 -- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148] conn(0x7f3b3c0bdbb0 msgr2=0x7f3b3c0c00a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.741+0000 7f3b56004640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148] conn(0x7f3b3c0bdbb0 0x7f3b3c0c00a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.741+0000 7f3b56004640 1 -- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3b3c0b3d30 msgr2=0x7f3b3c0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.741+0000 7f3b56004640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3b3c0b3d30 0x7f3b3c0b6240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3b40000c00 tx=0x7f3b40009680 comp rx=0 tx=0).stop 2024-07-27T06:44:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.741+0000 7f3b56004640 1 -- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b50109820 msgr2=0x7f3b50075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.741+0000 7f3b56004640 1 --2- 172.21.15.29:0/2849940198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b50109820 0x7f3b50075bc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800eaa0 tx=0x7f3b4800eea0 comp rx=0 tx=0).stop 2024-07-27T06:44:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.741+0000 7f3b4ffff640 1 -- 172.21.15.29:0/2849940198 reap_dead start 2024-07-27T06:44:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.741+0000 7f3b56004640 1 -- 172.21.15.29:0/2849940198 shutdown_connections 2024-07-27T06:44:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.741+0000 7f3b56004640 1 -- 172.21.15.29:0/2849940198 >> 172.21.15.29:0/2849940198 conn(0x7f3b5007b280 msgr2=0x7f3b50106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.741+0000 7f3b56004640 1 -- 172.21.15.29:0/2849940198 shutdown_connections 2024-07-27T06:44:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:32.741+0000 7f3b56004640 1 -- 172.21.15.29:0/2849940198 wait complete. 2024-07-27T06:44:33.430 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:33 smithi029 ceph-mon[32610]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:44:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:33 smithi077 ceph-mon[37153]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:44:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:33 smithi047 ceph-mon[36835]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:44:34.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.359+0000 7f80cb1a7640 1 -- 172.21.15.29:0/2265734211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c4107e50 msgr2=0x7f80c4108260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.359+0000 7f80cb1a7640 1 --2- 172.21.15.29:0/2265734211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c4107e50 0x7f80c4108260 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009900 tx=0x7f80b002a410 comp rx=0 tx=0).stop 2024-07-27T06:44:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.359+0000 7f80cb1a7640 1 -- 172.21.15.29:0/2265734211 shutdown_connections 2024-07-27T06:44:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.359+0000 7f80cb1a7640 1 --2- 172.21.15.29:0/2265734211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80c41087a0 0x7f80c410efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.359+0000 7f80cb1a7640 1 --2- 172.21.15.29:0/2265734211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c4107e50 0x7f80c4108260 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.359+0000 7f80cb1a7640 1 --2- 172.21.15.29:0/2265734211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c4102510 0x7f80c4102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.359+0000 7f80cb1a7640 1 -- 172.21.15.29:0/2265734211 >> 172.21.15.29:0/2265734211 conn(0x7f80c40791a0 msgr2=0x7f80c40795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.359+0000 7f80cb1a7640 1 -- 172.21.15.29:0/2265734211 shutdown_connections 2024-07-27T06:44:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.359+0000 7f80cb1a7640 1 -- 172.21.15.29:0/2265734211 wait complete. 2024-07-27T06:44:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.359+0000 7f80cb1a7640 1 Processor -- start 2024-07-27T06:44:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.359+0000 7f80cb1a7640 1 -- start start 2024-07-27T06:44:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80cb1a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c4102510 0x7f80c419c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80cb1a7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c4107e50 0x7f80c419c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80cb1a7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80c41087a0 0x7f80c41a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80cb1a7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f80c4112430 con 0x7f80c4102510 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80cb1a7640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f80c41122b0 con 0x7f80c41087a0 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80cb1a7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f80c41125b0 con 0x7f80c4107e50 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80c8f43640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c4102510 0x7f80c419c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80c8f43640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c4102510 0x7f80c419c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36076/0 (socket says 172.21.15.29:36076) 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80c8f43640 1 -- 172.21.15.29:0/1049138346 learned_addr learned my addr 172.21.15.29:0/1049138346 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80c3fff640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c4107e50 0x7f80c419c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80c8f43640 1 -- 172.21.15.29:0/1049138346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c4107e50 msgr2=0x7f80c419c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80c8f43640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80c4107e50 0x7f80c419c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80c9744640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80c41087a0 0x7f80c41a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80c8f43640 1 -- 172.21.15.29:0/1049138346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80c41087a0 msgr2=0x7f80c41a19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80c8f43640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80c41087a0 0x7f80c41a19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80c8f43640 1 -- 172.21.15.29:0/1049138346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c41a2120 con 0x7f80c4102510 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80c9744640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80c41087a0 0x7f80c41a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.360+0000 7f80c8f43640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c4102510 0x7f80c419c1b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f80b800d770 tx=0x7f80b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.361+0000 7f80c1ffb640 1 -- 172.21.15.29:0/1049138346 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8017070 con 0x7f80c4102510 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.361+0000 7f80cb1a7640 1 -- 172.21.15.29:0/1049138346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c41a2410 con 0x7f80c4102510 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.361+0000 7f80c1ffb640 1 -- 172.21.15.29:0/1049138346 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80b80054d0 con 0x7f80c4102510 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.361+0000 7f80c1ffb640 1 -- 172.21.15.29:0/1049138346 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8004c30 con 0x7f80c4102510 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.361+0000 7f80cb1a7640 1 -- 172.21.15.29:0/1049138346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c41a8b70 con 0x7f80c4102510 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.362+0000 7f80cb1a7640 1 -- 172.21.15.29:0/1049138346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f80c40634c0 con 0x7f80c4102510 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.363+0000 7f80c1ffb640 1 -- 172.21.15.29:0/1049138346 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f80b8004dd0 con 0x7f80c4102510 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.364+0000 7f80c1ffb640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f80b40b3cb0 0x7f80b40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.364+0000 7f80c3fff640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f80b40b3cb0 0x7f80b40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:34.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.364+0000 7f80c1ffb640 1 -- 172.21.15.29:0/1049138346 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f80b80e3e40 con 0x7f80c4102510 2024-07-27T06:44:34.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.364+0000 7f80c1ffb640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905] conn(0x7f80b40bdb30 0x7f80b40c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:34.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.364+0000 7f80c1ffb640 1 -- 172.21.15.29:0/1049138346 --> [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80b8001d30 con 0x7f80b40bdb30 2024-07-27T06:44:34.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.364+0000 7f80c1ffb640 1 -- 172.21.15.29:0/1049138346 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f80b80e4290 con 0x7f80c4102510 2024-07-27T06:44:34.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.365+0000 7f80c3fff640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f80b40b3cb0 0x7f80b40b61c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f80b0006010 tx=0x7f80b0000bc0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:34.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.365+0000 7f80c9744640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905] conn(0x7f80b40bdb30 0x7f80b40c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:34.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.365+0000 7f80c9744640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905] conn(0x7f80b40bdb30 0x7f80b40c0020 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:34.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.366+0000 7f80c1ffb640 1 -- 172.21.15.29:0/1049138346 <== osd.8 v2:172.21.15.77:6816/1422290905 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80b8001d30 con 0x7f80b40bdb30 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.380+0000 7f80cb1a7640 1 -- 172.21.15.29:0/1049138346 --> [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f80c4063940 con 0x7f80b40bdb30 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.380+0000 7f80c1ffb640 1 -- 172.21.15.29:0/1049138346 <== osd.8 v2:172.21.15.77:6816/1422290905 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f80c4063940 con 0x7f80b40bdb30 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.381+0000 7f80cb1a7640 1 -- 172.21.15.29:0/1049138346 >> [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905] conn(0x7f80b40bdb30 msgr2=0x7f80b40c0020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.381+0000 7f80cb1a7640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905] conn(0x7f80b40bdb30 0x7f80b40c0020 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.381+0000 7f80cb1a7640 1 -- 172.21.15.29:0/1049138346 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f80b40b3cb0 msgr2=0x7f80b40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.381+0000 7f80cb1a7640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f80b40b3cb0 0x7f80b40b61c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f80b0006010 tx=0x7f80b0000bc0 comp rx=0 tx=0).stop 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.381+0000 7f80cb1a7640 1 -- 172.21.15.29:0/1049138346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c4102510 msgr2=0x7f80c419c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.381+0000 7f80cb1a7640 1 --2- 172.21.15.29:0/1049138346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80c4102510 0x7f80c419c1b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f80b800d770 tx=0x7f80b800dc30 comp rx=0 tx=0).stop 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.381+0000 7f80c9744640 1 -- 172.21.15.29:0/1049138346 reap_dead start 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.381+0000 7f80cb1a7640 1 -- 172.21.15.29:0/1049138346 shutdown_connections 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.381+0000 7f80cb1a7640 1 -- 172.21.15.29:0/1049138346 >> 172.21.15.29:0/1049138346 conn(0x7f80c40791a0 msgr2=0x7f80c406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.381+0000 7f80cb1a7640 1 -- 172.21.15.29:0/1049138346 shutdown_connections 2024-07-27T06:44:34.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:34.381+0000 7f80cb1a7640 1 -- 172.21.15.29:0/1049138346 wait complete. 2024-07-27T06:44:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.345+0000 7f63e1d3c640 1 -- 172.21.15.29:0/4259944596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63dc109880 msgr2=0x7f63dc1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.345+0000 7f63e1d3c640 1 --2- 172.21.15.29:0/4259944596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63dc109880 0x7f63dc1100b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009ca0 tx=0x7f63d402a8e0 comp rx=0 tx=0).stop 2024-07-27T06:44:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.346+0000 7f63e1d3c640 1 -- 172.21.15.29:0/4259944596 shutdown_connections 2024-07-27T06:44:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.346+0000 7f63e1d3c640 1 --2- 172.21.15.29:0/4259944596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63dc109880 0x7f63dc1100b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:35.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.346+0000 7f63e1d3c640 1 --2- 172.21.15.29:0/4259944596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63dc108f30 0x7f63dc109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.346+0000 7f63e1d3c640 1 --2- 172.21.15.29:0/4259944596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63dc1019f0 0x7f63dc101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.346+0000 7f63e1d3c640 1 -- 172.21.15.29:0/4259944596 >> 172.21.15.29:0/4259944596 conn(0x7f63dc0799c0 msgr2=0x7f63dc079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.346+0000 7f63e1d3c640 1 -- 172.21.15.29:0/4259944596 shutdown_connections 2024-07-27T06:44:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.346+0000 7f63e1d3c640 1 -- 172.21.15.29:0/4259944596 wait complete. 2024-07-27T06:44:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.346+0000 7f63e1d3c640 1 Processor -- start 2024-07-27T06:44:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63e1d3c640 1 -- start start 2024-07-27T06:44:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63e1d3c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63dc1019f0 0x7f63dc102990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63e1d3c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63dc108f30 0x7f63dc102ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63e1d3c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63dc109880 0x7f63dc103410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63e1d3c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc113580 con 0x7f63dc109880 2024-07-27T06:44:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63e1d3c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc113400 con 0x7f63dc108f30 2024-07-27T06:44:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63e1d3c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc113700 con 0x7f63dc1019f0 2024-07-27T06:44:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63dbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63dc109880 0x7f63dc103410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63dbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63dc109880 0x7f63dc103410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36084/0 (socket says 172.21.15.29:36084) 2024-07-27T06:44:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63dbfff640 1 -- 172.21.15.29:0/1464395112 learned_addr learned my addr 172.21.15.29:0/1464395112 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63db7fe640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63dc1019f0 0x7f63dc102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63daffd640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63dc108f30 0x7f63dc102ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63dbfff640 1 -- 172.21.15.29:0/1464395112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63dc1019f0 msgr2=0x7f63dc102990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63dbfff640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63dc1019f0 0x7f63dc102990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63dbfff640 1 -- 172.21.15.29:0/1464395112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63dc108f30 msgr2=0x7f63dc102ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63dbfff640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63dc108f30 0x7f63dc102ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.347+0000 7f63dbfff640 1 -- 172.21.15.29:0/1464395112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc1a8190 con 0x7f63dc109880 2024-07-27T06:44:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.348+0000 7f63dbfff640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63dc109880 0x7f63dc103410 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f63d4005cc0 tx=0x7f63d40039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.348+0000 7f63d8ff9640 1 -- 172.21.15.29:0/1464395112 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d4003de0 con 0x7f63dc109880 2024-07-27T06:44:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.348+0000 7f63d8ff9640 1 -- 172.21.15.29:0/1464395112 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63d40044f0 con 0x7f63dc109880 2024-07-27T06:44:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.348+0000 7f63e1d3c640 1 -- 172.21.15.29:0/1464395112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63dc1a8420 con 0x7f63dc109880 2024-07-27T06:44:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.348+0000 7f63d8ff9640 1 -- 172.21.15.29:0/1464395112 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d4030d90 con 0x7f63dc109880 2024-07-27T06:44:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.348+0000 7f63e1d3c640 1 -- 172.21.15.29:0/1464395112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63dc1a89b0 con 0x7f63dc109880 2024-07-27T06:44:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.349+0000 7f63e1d3c640 1 -- 172.21.15.29:0/1464395112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f63dc063580 con 0x7f63dc109880 2024-07-27T06:44:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.350+0000 7f63d8ff9640 1 -- 172.21.15.29:0/1464395112 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f63d4004040 con 0x7f63dc109880 2024-07-27T06:44:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.351+0000 7f63d8ff9640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63c40b3d30 0x7f63c40b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.351+0000 7f63db7fe640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63c40b3d30 0x7f63c40b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.351+0000 7f63d8ff9640 1 -- 172.21.15.29:0/1464395112 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f63d4105660 con 0x7f63dc109880 2024-07-27T06:44:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.351+0000 7f63d8ff9640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078] conn(0x7f63c40bdbb0 0x7f63c40c00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.351+0000 7f63d8ff9640 1 -- 172.21.15.29:0/1464395112 --> [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f63d403eec0 con 0x7f63c40bdbb0 2024-07-27T06:44:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.351+0000 7f63d8ff9640 1 -- 172.21.15.29:0/1464395112 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f63d4105a10 con 0x7f63dc109880 2024-07-27T06:44:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.352+0000 7f63db7fe640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63c40b3d30 0x7f63c40b6240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f63dc1043e0 tx=0x7f63cc00b410 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.352+0000 7f63daffd640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078] conn(0x7f63c40bdbb0 0x7f63c40c00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.352+0000 7f63daffd640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078] conn(0x7f63c40bdbb0 0x7f63c40c00a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.353+0000 7f63d8ff9640 1 -- 172.21.15.29:0/1464395112 <== osd.2 v2:172.21.15.77:6800/2202565078 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63d403eec0 con 0x7f63c40bdbb0 2024-07-27T06:44:35.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.367+0000 7f63e1d3c640 1 -- 172.21.15.29:0/1464395112 --> [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f63dc063980 con 0x7f63c40bdbb0 2024-07-27T06:44:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.367+0000 7f63d8ff9640 1 -- 172.21.15.29:0/1464395112 <== osd.2 v2:172.21.15.77:6800/2202565078 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f63dc063980 con 0x7f63c40bdbb0 2024-07-27T06:44:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.368+0000 7f63e1d3c640 1 -- 172.21.15.29:0/1464395112 >> [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078] conn(0x7f63c40bdbb0 msgr2=0x7f63c40c00a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.368+0000 7f63e1d3c640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078] conn(0x7f63c40bdbb0 0x7f63c40c00a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.368+0000 7f63e1d3c640 1 -- 172.21.15.29:0/1464395112 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63c40b3d30 msgr2=0x7f63c40b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.368+0000 7f63e1d3c640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63c40b3d30 0x7f63c40b6240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f63dc1043e0 tx=0x7f63cc00b410 comp rx=0 tx=0).stop 2024-07-27T06:44:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.368+0000 7f63e1d3c640 1 -- 172.21.15.29:0/1464395112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63dc109880 msgr2=0x7f63dc103410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.368+0000 7f63e1d3c640 1 --2- 172.21.15.29:0/1464395112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63dc109880 0x7f63dc103410 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f63d4005cc0 tx=0x7f63d40039f0 comp rx=0 tx=0).stop 2024-07-27T06:44:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.368+0000 7f63dbfff640 1 -- 172.21.15.29:0/1464395112 reap_dead start 2024-07-27T06:44:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.368+0000 7f63e1d3c640 1 -- 172.21.15.29:0/1464395112 shutdown_connections 2024-07-27T06:44:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.368+0000 7f63e1d3c640 1 -- 172.21.15.29:0/1464395112 >> 172.21.15.29:0/1464395112 conn(0x7f63dc0799c0 msgr2=0x7f63dc106ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.368+0000 7f63e1d3c640 1 -- 172.21.15.29:0/1464395112 shutdown_connections 2024-07-27T06:44:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.368+0000 7f63e1d3c640 1 -- 172.21.15.29:0/1464395112 wait complete. 2024-07-27T06:44:35.579 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:35 smithi029 ceph-mon[32610]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-07-27T06:44:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:35 smithi077 ceph-mon[37153]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-07-27T06:44:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:35 smithi047 ceph-mon[36835]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-07-27T06:44:36.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.046+0000 7fec0afd6640 1 -- 172.21.15.29:0/3861074294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec04102510 msgr2=0x7fec04102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.046+0000 7fec0afd6640 1 --2- 172.21.15.29:0/3861074294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec04102510 0x7fec04102920 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fec00009900 tx=0x7fec0002a410 comp rx=0 tx=0).stop 2024-07-27T06:44:36.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.046+0000 7fec0afd6640 1 -- 172.21.15.29:0/3861074294 shutdown_connections 2024-07-27T06:44:36.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.046+0000 7fec0afd6640 1 --2- 172.21.15.29:0/3861074294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec04108750 0x7fec0410ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.046+0000 7fec0afd6640 1 --2- 172.21.15.29:0/3861074294 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec04107e00 0x7fec04108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.046+0000 7fec0afd6640 1 --2- 172.21.15.29:0/3861074294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec04102510 0x7fec04102920 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.046+0000 7fec0afd6640 1 -- 172.21.15.29:0/3861074294 >> 172.21.15.29:0/3861074294 conn(0x7fec04079190 msgr2=0x7fec040795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:36.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.046+0000 7fec0afd6640 1 -- 172.21.15.29:0/3861074294 shutdown_connections 2024-07-27T06:44:36.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.046+0000 7fec0afd6640 1 -- 172.21.15.29:0/3861074294 wait complete. 2024-07-27T06:44:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.047+0000 7fec0afd6640 1 Processor -- start 2024-07-27T06:44:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.047+0000 7fec0afd6640 1 -- start start 2024-07-27T06:44:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7fec0afd6640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec04102510 0x7fec0419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7fec0afd6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec04107e00 0x7fec0419b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7fec0afd6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec04108750 0x7fec0419b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7fec0afd6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fec041123b0 con 0x7fec04107e00 2024-07-27T06:44:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7fec0afd6640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fec04112230 con 0x7fec04102510 2024-07-27T06:44:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7fec0afd6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fec04112530 con 0x7fec04108750 2024-07-27T06:44:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7febfbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec04107e00 0x7fec0419b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7febfbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec04107e00 0x7fec0419b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36144/0 (socket says 172.21.15.29:36144) 2024-07-27T06:44:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7febfbfff640 1 -- 172.21.15.29:0/1294397102 learned_addr learned my addr 172.21.15.29:0/1294397102 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:36.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7fec09573640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec04108750 0x7fec0419b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7fec08d72640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec04102510 0x7fec0419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7febfbfff640 1 -- 172.21.15.29:0/1294397102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec04108750 msgr2=0x7fec0419b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7febfbfff640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec04108750 0x7fec0419b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7febfbfff640 1 -- 172.21.15.29:0/1294397102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec04102510 msgr2=0x7fec0419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7febfbfff640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec04102510 0x7fec0419aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7febfbfff640 1 -- 172.21.15.29:0/1294397102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec041a5ed0 con 0x7fec04107e00 2024-07-27T06:44:36.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.048+0000 7febfbfff640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec04107e00 0x7fec0419b1e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7febf400c970 tx=0x7febf400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:36.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.049+0000 7febf9ffb640 1 -- 172.21.15.29:0/1294397102 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf4014070 con 0x7fec04107e00 2024-07-27T06:44:36.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.049+0000 7febf9ffb640 1 -- 172.21.15.29:0/1294397102 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febf4004440 con 0x7fec04107e00 2024-07-27T06:44:36.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.049+0000 7fec0afd6640 1 -- 172.21.15.29:0/1294397102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec041a61c0 con 0x7fec04107e00 2024-07-27T06:44:36.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.049+0000 7febf9ffb640 1 -- 172.21.15.29:0/1294397102 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf4003e50 con 0x7fec04107e00 2024-07-27T06:44:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.049+0000 7fec0afd6640 1 -- 172.21.15.29:0/1294397102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec041a6700 con 0x7fec04107e00 2024-07-27T06:44:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.050+0000 7fec0afd6640 1 -- 172.21.15.29:0/1294397102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fec04063580 con 0x7fec04107e00 2024-07-27T06:44:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.051+0000 7febf9ffb640 1 -- 172.21.15.29:0/1294397102 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7febf4017070 con 0x7fec04107e00 2024-07-27T06:44:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.052+0000 7febf9ffb640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7febec0b3e50 0x7febec0b6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.052+0000 7fec08d72640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7febec0b3e50 0x7febec0b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.052+0000 7febf9ffb640 1 -- 172.21.15.29:0/1294397102 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7febf40e3ee0 con 0x7fec04107e00 2024-07-27T06:44:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.052+0000 7febf9ffb640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850] conn(0x7febec0bdcd0 0x7febec0c01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.052+0000 7febf9ffb640 1 -- 172.21.15.29:0/1294397102 --> [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7febf4003e50 con 0x7febec0bdcd0 2024-07-27T06:44:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.052+0000 7febf9ffb640 1 -- 172.21.15.29:0/1294397102 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7febf40e4290 con 0x7fec04107e00 2024-07-27T06:44:36.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.053+0000 7fec08d72640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7febec0b3e50 0x7febec0b6360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec00006010 tx=0x7fec00000bc0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:36.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.053+0000 7fec09573640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850] conn(0x7febec0bdcd0 0x7febec0c01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.053+0000 7fec09573640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850] conn(0x7febec0bdcd0 0x7febec0c01c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:36.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.054+0000 7febf9ffb640 1 -- 172.21.15.29:0/1294397102 <== osd.10 v2:172.21.15.47:6824/2133726850 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7febf4003e50 con 0x7febec0bdcd0 2024-07-27T06:44:36.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.068+0000 7fec0afd6640 1 -- 172.21.15.29:0/1294397102 --> [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fec04063980 con 0x7febec0bdcd0 2024-07-27T06:44:36.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.069+0000 7febf9ffb640 1 -- 172.21.15.29:0/1294397102 <== osd.10 v2:172.21.15.47:6824/2133726850 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fec04063980 con 0x7febec0bdcd0 2024-07-27T06:44:36.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.069+0000 7fec0afd6640 1 -- 172.21.15.29:0/1294397102 >> [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850] conn(0x7febec0bdcd0 msgr2=0x7febec0c01c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.069+0000 7fec0afd6640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850] conn(0x7febec0bdcd0 0x7febec0c01c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.069+0000 7fec0afd6640 1 -- 172.21.15.29:0/1294397102 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7febec0b3e50 msgr2=0x7febec0b6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.069+0000 7fec0afd6640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7febec0b3e50 0x7febec0b6360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec00006010 tx=0x7fec00000bc0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.069+0000 7fec0afd6640 1 -- 172.21.15.29:0/1294397102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec04107e00 msgr2=0x7fec0419b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.069+0000 7fec0afd6640 1 --2- 172.21.15.29:0/1294397102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec04107e00 0x7fec0419b1e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7febf400c970 tx=0x7febf400ce30 comp rx=0 tx=0).stop 2024-07-27T06:44:36.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.069+0000 7fec09573640 1 -- 172.21.15.29:0/1294397102 reap_dead start 2024-07-27T06:44:36.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.069+0000 7fec0afd6640 1 -- 172.21.15.29:0/1294397102 shutdown_connections 2024-07-27T06:44:36.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.069+0000 7fec0afd6640 1 -- 172.21.15.29:0/1294397102 >> 172.21.15.29:0/1294397102 conn(0x7fec04079190 msgr2=0x7fec04100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:36.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.069+0000 7fec0afd6640 1 -- 172.21.15.29:0/1294397102 shutdown_connections 2024-07-27T06:44:36.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:36.069+0000 7fec0afd6640 1 -- 172.21.15.29:0/1294397102 wait complete. 2024-07-27T06:44:36.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.578+0000 7f6df4476640 1 -- 172.21.15.29:0/411017923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dec108730 msgr2=0x7f6dec10ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.578+0000 7f6df4476640 1 --2- 172.21.15.29:0/411017923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dec108730 0x7f6dec10ef60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009ca0 tx=0x7f6de402a8c0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.578+0000 7f6df4476640 1 -- 172.21.15.29:0/411017923 shutdown_connections 2024-07-27T06:44:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.578+0000 7f6df4476640 1 --2- 172.21.15.29:0/411017923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dec108730 0x7f6dec10ef60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.578+0000 7f6df4476640 1 --2- 172.21.15.29:0/411017923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6dec107de0 0x7f6dec1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.578+0000 7f6df4476640 1 --2- 172.21.15.29:0/411017923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec1024f0 0x7f6dec102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.578+0000 7f6df4476640 1 -- 172.21.15.29:0/411017923 >> 172.21.15.29:0/411017923 conn(0x7f6dec079150 msgr2=0x7f6dec079560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.578+0000 7f6df4476640 1 -- 172.21.15.29:0/411017923 shutdown_connections 2024-07-27T06:44:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.578+0000 7f6df4476640 1 -- 172.21.15.29:0/411017923 wait complete. 2024-07-27T06:44:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df4476640 1 Processor -- start 2024-07-27T06:44:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df4476640 1 -- start start 2024-07-27T06:44:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df4476640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec1024f0 0x7f6dec199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df4476640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dec107de0 0x7f6dec19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df4476640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6dec108730 0x7f6dec19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df4476640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec112400 con 0x7f6dec107de0 2024-07-27T06:44:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df4476640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec112280 con 0x7f6dec108730 2024-07-27T06:44:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df4476640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec112580 con 0x7f6dec1024f0 2024-07-27T06:44:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df1a11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dec107de0 0x7f6dec19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df1a11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dec107de0 0x7f6dec19a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36102/0 (socket says 172.21.15.29:36102) 2024-07-27T06:44:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df1a11640 1 -- 172.21.15.29:0/1342473024 learned_addr learned my addr 172.21.15.29:0/1342473024 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df2212640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec1024f0 0x7f6dec199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df2a13640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6dec108730 0x7f6dec19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df1a11640 1 -- 172.21.15.29:0/1342473024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec1024f0 msgr2=0x7f6dec199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df1a11640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6dec1024f0 0x7f6dec199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df1a11640 1 -- 172.21.15.29:0/1342473024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6dec108730 msgr2=0x7f6dec19f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.579+0000 7f6df1a11640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6dec108730 0x7f6dec19f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.580+0000 7f6df1a11640 1 -- 172.21.15.29:0/1342473024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec19fdb0 con 0x7f6dec107de0 2024-07-27T06:44:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.580+0000 7f6df1a11640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dec107de0 0x7f6dec19a3e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6de000d950 tx=0x7f6de000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.580+0000 7f6deb7fe640 1 -- 172.21.15.29:0/1342473024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0017070 con 0x7f6dec107de0 2024-07-27T06:44:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.580+0000 7f6deb7fe640 1 -- 172.21.15.29:0/1342473024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de0009c30 con 0x7f6dec107de0 2024-07-27T06:44:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.580+0000 7f6df4476640 1 -- 172.21.15.29:0/1342473024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dec1a00a0 con 0x7f6dec107de0 2024-07-27T06:44:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.580+0000 7f6deb7fe640 1 -- 172.21.15.29:0/1342473024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de00047a0 con 0x7f6dec107de0 2024-07-27T06:44:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.580+0000 7f6df4476640 1 -- 172.21.15.29:0/1342473024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dec1a6800 con 0x7f6dec107de0 2024-07-27T06:44:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.581+0000 7f6df4476640 1 -- 172.21.15.29:0/1342473024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6dec06b5a0 con 0x7f6dec107de0 2024-07-27T06:44:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.583+0000 7f6deb7fe640 1 -- 172.21.15.29:0/1342473024 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f6de0004020 con 0x7f6dec107de0 2024-07-27T06:44:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.583+0000 7f6deb7fe640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6dd40b3d30 0x7f6dd40b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.583+0000 7f6df2212640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6dd40b3d30 0x7f6dd40b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.583+0000 7f6deb7fe640 1 -- 172.21.15.29:0/1342473024 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f6de00e4070 con 0x7f6dec107de0 2024-07-27T06:44:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.584+0000 7f6deb7fe640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384] conn(0x7f6dd40bdbb0 0x7f6dd40c00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.584+0000 7f6deb7fe640 1 -- 172.21.15.29:0/1342473024 --> [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6de0024e80 con 0x7f6dd40bdbb0 2024-07-27T06:44:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.584+0000 7f6deb7fe640 1 -- 172.21.15.29:0/1342473024 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f6de00e44c0 con 0x7f6dec107de0 2024-07-27T06:44:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.584+0000 7f6df2a13640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384] conn(0x7f6dd40bdbb0 0x7f6dd40c00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.584+0000 7f6df2212640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6dd40b3d30 0x7f6dd40b6240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc005fd0 tx=0x7f6ddc00b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.584+0000 7f6df2a13640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384] conn(0x7f6dd40bdbb0 0x7f6dd40c00a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.585+0000 7f6deb7fe640 1 -- 172.21.15.29:0/1342473024 <== osd.9 v2:172.21.15.29:6826/1307316384 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6de0024e80 con 0x7f6dd40bdbb0 2024-07-27T06:44:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6df4476640 1 -- 172.21.15.29:0/1342473024 --> [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6dec103720 con 0x7f6dd40bdbb0 2024-07-27T06:44:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6deb7fe640 1 -- 172.21.15.29:0/1342473024 <== osd.9 v2:172.21.15.29:6826/1307316384 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6dec103720 con 0x7f6dd40bdbb0 2024-07-27T06:44:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6df4476640 1 -- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384] conn(0x7f6dd40bdbb0 msgr2=0x7f6dd40c00a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6df4476640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384] conn(0x7f6dd40bdbb0 0x7f6dd40c00a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6df4476640 1 -- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6dd40b3d30 msgr2=0x7f6dd40b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6df4476640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6dd40b3d30 0x7f6dd40b6240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc005fd0 tx=0x7f6ddc00b390 comp rx=0 tx=0).stop 2024-07-27T06:44:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6df4476640 1 -- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dec107de0 msgr2=0x7f6dec19a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6df4476640 1 --2- 172.21.15.29:0/1342473024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dec107de0 0x7f6dec19a3e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6de000d950 tx=0x7f6de000de10 comp rx=0 tx=0).stop 2024-07-27T06:44:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6df2a13640 1 -- 172.21.15.29:0/1342473024 reap_dead start 2024-07-27T06:44:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6df4476640 1 -- 172.21.15.29:0/1342473024 shutdown_connections 2024-07-27T06:44:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6df4476640 1 -- 172.21.15.29:0/1342473024 >> 172.21.15.29:0/1342473024 conn(0x7f6dec079150 msgr2=0x7f6dec101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6df4476640 1 -- 172.21.15.29:0/1342473024 shutdown_connections 2024-07-27T06:44:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.599+0000 7f6df4476640 1 -- 172.21.15.29:0/1342473024 wait complete. 2024-07-27T06:44:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.698+0000 7f7561eeb640 1 -- 172.21.15.29:0/618660592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c108740 msgr2=0x7f755c10ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.698+0000 7f7561eeb640 1 --2- 172.21.15.29:0/618660592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c108740 0x7f755c10ef70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7554009ca0 tx=0x7f755402a8c0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.698+0000 7f7561eeb640 1 -- 172.21.15.29:0/618660592 shutdown_connections 2024-07-27T06:44:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.698+0000 7f7561eeb640 1 --2- 172.21.15.29:0/618660592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c108740 0x7f755c10ef70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.698+0000 7f7561eeb640 1 --2- 172.21.15.29:0/618660592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f755c107df0 0x7f755c108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.698+0000 7f7561eeb640 1 --2- 172.21.15.29:0/618660592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f755c1024b0 0x7f755c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.698+0000 7f7561eeb640 1 -- 172.21.15.29:0/618660592 >> 172.21.15.29:0/618660592 conn(0x7f755c079170 msgr2=0x7f755c079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.698+0000 7f7561eeb640 1 -- 172.21.15.29:0/618660592 shutdown_connections 2024-07-27T06:44:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.699+0000 7f7561eeb640 1 -- 172.21.15.29:0/618660592 wait complete. 2024-07-27T06:44:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.699+0000 7f7561eeb640 1 Processor -- start 2024-07-27T06:44:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.699+0000 7f7561eeb640 1 -- start start 2024-07-27T06:44:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f7561eeb640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f755c1024b0 0x7f755c199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f7561eeb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c107df0 0x7f755c19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f7561eeb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f755c108740 0x7f755c19f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f7561eeb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f755c1123a0 con 0x7f755c107df0 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f7561eeb640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f755c112220 con 0x7f755c1024b0 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f7561eeb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f755c112520 con 0x7f755c108740 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f755affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c107df0 0x7f755c19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f755affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c107df0 0x7f755c19a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36116/0 (socket says 172.21.15.29:36116) 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f755affd640 1 -- 172.21.15.29:0/2559638756 learned_addr learned my addr 172.21.15.29:0/2559638756 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f755affd640 1 -- 172.21.15.29:0/2559638756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f755c108740 msgr2=0x7f755c19f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f755b7fe640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f755c1024b0 0x7f755c199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f755affd640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f755c108740 0x7f755c19f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f755affd640 1 -- 172.21.15.29:0/2559638756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f755c1024b0 msgr2=0x7f755c199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f755affd640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f755c1024b0 0x7f755c199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f755affd640 1 -- 172.21.15.29:0/2559638756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f755c19fdd0 con 0x7f755c107df0 2024-07-27T06:44:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f755affd640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c107df0 0x7f755c19a400 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f755000d770 tx=0x7f755000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f7558ff9640 1 -- 172.21.15.29:0/2559638756 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7550017070 con 0x7f755c107df0 2024-07-27T06:44:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.700+0000 7f7561eeb640 1 -- 172.21.15.29:0/2559638756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f755c1a00c0 con 0x7f755c107df0 2024-07-27T06:44:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.701+0000 7f7558ff9640 1 -- 172.21.15.29:0/2559638756 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7550009c30 con 0x7f755c107df0 2024-07-27T06:44:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.701+0000 7f7558ff9640 1 -- 172.21.15.29:0/2559638756 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f755000f620 con 0x7f755c107df0 2024-07-27T06:44:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.701+0000 7f7561eeb640 1 -- 172.21.15.29:0/2559638756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f755c1a6820 con 0x7f755c107df0 2024-07-27T06:44:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.701+0000 7f7561eeb640 1 -- 172.21.15.29:0/2559638756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f755c104d60 con 0x7f755c107df0 2024-07-27T06:44:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.703+0000 7f7558ff9640 1 -- 172.21.15.29:0/2559638756 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f755001a070 con 0x7f755c107df0 2024-07-27T06:44:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.704+0000 7f7558ff9640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f75440b3cb0 0x7f75440b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.704+0000 7f755b7fe640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f75440b3cb0 0x7f75440b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.704+0000 7f7558ff9640 1 -- 172.21.15.29:0/2559638756 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f75500e3f40 con 0x7f755c107df0 2024-07-27T06:44:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.704+0000 7f7558ff9640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720] conn(0x7f75440bdb30 0x7f75440c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.704+0000 7f7558ff9640 1 -- 172.21.15.29:0/2559638756 --> [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75440c06f0 con 0x7f75440bdb30 2024-07-27T06:44:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.704+0000 7f7558ff9640 1 -- 172.21.15.29:0/2559638756 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f75500e42f0 con 0x7f755c107df0 2024-07-27T06:44:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.704+0000 7f755b7fe640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f75440b3cb0 0x7f75440b61c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f754c009aa0 tx=0x7f754c005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.704+0000 7f755bfff640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720] conn(0x7f75440bdb30 0x7f75440c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.705+0000 7f755bfff640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720] conn(0x7f75440bdb30 0x7f75440c0020 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.706+0000 7f7558ff9640 1 -- 172.21.15.29:0/2559638756 <== osd.4 v2:172.21.15.47:6808/1379363720 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75440c06f0 con 0x7f75440bdb30 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.720+0000 7f7561eeb640 1 -- 172.21.15.29:0/2559638756 --> [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f755c06adc0 con 0x7f75440bdb30 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.721+0000 7f7558ff9640 1 -- 172.21.15.29:0/2559638756 <== osd.4 v2:172.21.15.47:6808/1379363720 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f755c06adc0 con 0x7f75440bdb30 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.721+0000 7f7561eeb640 1 -- 172.21.15.29:0/2559638756 >> [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720] conn(0x7f75440bdb30 msgr2=0x7f75440c0020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.721+0000 7f7561eeb640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720] conn(0x7f75440bdb30 0x7f75440c0020 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.721+0000 7f7561eeb640 1 -- 172.21.15.29:0/2559638756 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f75440b3cb0 msgr2=0x7f75440b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.721+0000 7f7561eeb640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f75440b3cb0 0x7f75440b61c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f754c009aa0 tx=0x7f754c005f90 comp rx=0 tx=0).stop 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.721+0000 7f7561eeb640 1 -- 172.21.15.29:0/2559638756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c107df0 msgr2=0x7f755c19a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.721+0000 7f7561eeb640 1 --2- 172.21.15.29:0/2559638756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f755c107df0 0x7f755c19a400 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f755000d770 tx=0x7f755000dc30 comp rx=0 tx=0).stop 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.721+0000 7f755bfff640 1 -- 172.21.15.29:0/2559638756 reap_dead start 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.721+0000 7f7561eeb640 1 -- 172.21.15.29:0/2559638756 shutdown_connections 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.721+0000 7f7561eeb640 1 -- 172.21.15.29:0/2559638756 >> 172.21.15.29:0/2559638756 conn(0x7f755c079170 msgr2=0x7f755c101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.721+0000 7f7561eeb640 1 -- 172.21.15.29:0/2559638756 shutdown_connections 2024-07-27T06:44:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:35.721+0000 7f7561eeb640 1 -- 172.21.15.29:0/2559638756 wait complete. 2024-07-27T06:44:37.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.504+0000 7fb092343640 1 -- 172.21.15.29:0/833108012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb08c108ed0 msgr2=0x7fb08c1092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.504+0000 7fb092343640 1 --2- 172.21.15.29:0/833108012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb08c108ed0 0x7fb08c1092e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb080009900 tx=0x7fb08002a410 comp rx=0 tx=0).stop 2024-07-27T06:44:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.504+0000 7fb092343640 1 -- 172.21.15.29:0/833108012 shutdown_connections 2024-07-27T06:44:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.504+0000 7fb092343640 1 --2- 172.21.15.29:0/833108012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb08c109820 0x7fb08c110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.504+0000 7fb092343640 1 --2- 172.21.15.29:0/833108012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb08c108ed0 0x7fb08c1092e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.504+0000 7fb092343640 1 --2- 172.21.15.29:0/833108012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb08c103590 0x7fb08c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.504+0000 7fb092343640 1 -- 172.21.15.29:0/833108012 >> 172.21.15.29:0/833108012 conn(0x7fb08c07b280 msgr2=0x7fb08c07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.504+0000 7fb092343640 1 -- 172.21.15.29:0/833108012 shutdown_connections 2024-07-27T06:44:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.504+0000 7fb092343640 1 -- 172.21.15.29:0/833108012 wait complete. 2024-07-27T06:44:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.505+0000 7fb092343640 1 Processor -- start 2024-07-27T06:44:37.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.505+0000 7fb092343640 1 -- start start 2024-07-27T06:44:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.505+0000 7fb092343640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb08c103590 0x7fb08c199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.505+0000 7fb092343640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb08c108ed0 0x7fb08c19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.505+0000 7fb092343640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb08c109820 0x7fb08c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.505+0000 7fb092343640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb08c113490 con 0x7fb08c109820 2024-07-27T06:44:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.505+0000 7fb092343640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fb08c113310 con 0x7fb08c103590 2024-07-27T06:44:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.505+0000 7fb092343640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb08c113610 con 0x7fb08c108ed0 2024-07-27T06:44:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb0908e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb08c109820 0x7fb08c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb08bfff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb08c103590 0x7fb08c199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb0908e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb08c109820 0x7fb08c19f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36160/0 (socket says 172.21.15.29:36160) 2024-07-27T06:44:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb0908e0640 1 -- 172.21.15.29:0/875069116 learned_addr learned my addr 172.21.15.29:0/875069116 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb08b7fe640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb08c108ed0 0x7fb08c19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb0908e0640 1 -- 172.21.15.29:0/875069116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb08c108ed0 msgr2=0x7fb08c19a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb0908e0640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb08c108ed0 0x7fb08c19a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb0908e0640 1 -- 172.21.15.29:0/875069116 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb08c103590 msgr2=0x7fb08c199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb0908e0640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb08c103590 0x7fb08c199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb0908e0640 1 -- 172.21.15.29:0/875069116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb08c19fe90 con 0x7fb08c109820 2024-07-27T06:44:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb0908e0640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb08c109820 0x7fb08c19f760 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb08400eaa0 tx=0x7fb08400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb0897fa640 1 -- 172.21.15.29:0/875069116 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08400cdb0 con 0x7fb08c109820 2024-07-27T06:44:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb0897fa640 1 -- 172.21.15.29:0/875069116 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb084005020 con 0x7fb08c109820 2024-07-27T06:44:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb092343640 1 -- 172.21.15.29:0/875069116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb08c1a0180 con 0x7fb08c109820 2024-07-27T06:44:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.506+0000 7fb0897fa640 1 -- 172.21.15.29:0/875069116 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0840116d0 con 0x7fb08c109820 2024-07-27T06:44:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.507+0000 7fb092343640 1 -- 172.21.15.29:0/875069116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb08c1a68e0 con 0x7fb08c109820 2024-07-27T06:44:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.507+0000 7fb092343640 1 -- 172.21.15.29:0/875069116 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb08c063540 con 0x7fb08c109820 2024-07-27T06:44:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.509+0000 7fb0897fa640 1 -- 172.21.15.29:0/875069116 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fb0840051d0 con 0x7fb08c109820 2024-07-27T06:44:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.510+0000 7fb0897fa640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb0740b3cb0 0x7fb0740b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.510+0000 7fb08bfff640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb0740b3cb0 0x7fb0740b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.510+0000 7fb0897fa640 1 -- 172.21.15.29:0/875069116 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7fb0840e4420 con 0x7fb08c109820 2024-07-27T06:44:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.510+0000 7fb0897fa640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641] conn(0x7fb0740bdb30 0x7fb0740c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.510+0000 7fb0897fa640 1 -- 172.21.15.29:0/875069116 --> [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb084014ec0 con 0x7fb0740bdb30 2024-07-27T06:44:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.510+0000 7fb0897fa640 1 -- 172.21.15.29:0/875069116 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fb0840e4870 con 0x7fb08c109820 2024-07-27T06:44:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.510+0000 7fb08b7fe640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641] conn(0x7fb0740bdb30 0x7fb0740c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.511+0000 7fb08bfff640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb0740b3cb0 0x7fb0740b61c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb07c005d00 tx=0x7fb07c00b410 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.511+0000 7fb08b7fe640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641] conn(0x7fb0740bdb30 0x7fb0740c0020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:37.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.512+0000 7fb0897fa640 1 -- 172.21.15.29:0/875069116 <== osd.0 v2:172.21.15.29:6802/386121641 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb084014ec0 con 0x7fb0740bdb30 2024-07-27T06:44:37.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.526+0000 7fb092343640 1 -- 172.21.15.29:0/875069116 --> [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb08c0639f0 con 0x7fb0740bdb30 2024-07-27T06:44:37.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.527+0000 7fb0897fa640 1 -- 172.21.15.29:0/875069116 <== osd.0 v2:172.21.15.29:6802/386121641 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb08c0639f0 con 0x7fb0740bdb30 2024-07-27T06:44:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.527+0000 7fb092343640 1 -- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641] conn(0x7fb0740bdb30 msgr2=0x7fb0740c0020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.527+0000 7fb092343640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641] conn(0x7fb0740bdb30 0x7fb0740c0020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.527+0000 7fb092343640 1 -- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb0740b3cb0 msgr2=0x7fb0740b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.527+0000 7fb092343640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb0740b3cb0 0x7fb0740b61c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb07c005d00 tx=0x7fb07c00b410 comp rx=0 tx=0).stop 2024-07-27T06:44:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.527+0000 7fb092343640 1 -- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb08c109820 msgr2=0x7fb08c19f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.527+0000 7fb092343640 1 --2- 172.21.15.29:0/875069116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb08c109820 0x7fb08c19f760 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb08400eaa0 tx=0x7fb08400ef60 comp rx=0 tx=0).stop 2024-07-27T06:44:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.527+0000 7fb0908e0640 1 -- 172.21.15.29:0/875069116 reap_dead start 2024-07-27T06:44:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.527+0000 7fb092343640 1 -- 172.21.15.29:0/875069116 shutdown_connections 2024-07-27T06:44:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.527+0000 7fb092343640 1 -- 172.21.15.29:0/875069116 >> 172.21.15.29:0/875069116 conn(0x7fb08c07b280 msgr2=0x7fb08c107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.527+0000 7fb092343640 1 -- 172.21.15.29:0/875069116 shutdown_connections 2024-07-27T06:44:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:37.527+0000 7fb092343640 1 -- 172.21.15.29:0/875069116 wait complete. 2024-07-27T06:44:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:37 smithi077 ceph-mon[37153]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:44:37.643 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:37 smithi029 ceph-mon[32610]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:44:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:37 smithi047 ceph-mon[36835]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:44:38.590 INFO:teuthology.orchestra.run.smithi029.stdout:231928234031 2024-07-27T06:44:38.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-07-27T06:44:39.023 INFO:teuthology.orchestra.run.smithi029.stdout:184683593785 2024-07-27T06:44:39.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-07-27T06:44:39.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:39 smithi029 ceph-mon[32610]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:44:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:39 smithi077 ceph-mon[37153]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:44:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:39 smithi047 ceph-mon[36835]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:44:40.667 INFO:teuthology.orchestra.run.smithi029.stdout:137438953540 2024-07-27T06:44:40.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-07-27T06:44:41.554 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:41 smithi029 ceph-mon[32610]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:41.584 INFO:teuthology.orchestra.run.smithi029.stdout:257698037803 2024-07-27T06:44:41.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-07-27T06:44:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:41 smithi077 ceph-mon[37153]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:41 smithi047 ceph-mon[36835]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:43 smithi029 ceph-mon[32610]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:44:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:43 smithi077 ceph-mon[37153]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:44:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:43 smithi047 ceph-mon[36835]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:44:43.976 INFO:teuthology.orchestra.run.smithi029.stdout:38654705752 2024-07-27T06:44:43.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-07-27T06:44:44.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.674+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2430818121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f48108ed0 msgr2=0x7f1f481092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:44.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.674+0000 7f1f4dfe8640 1 --2- 172.21.15.29:0/2430818121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f48108ed0 0x7f1f481092e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009900 tx=0x7f1f3002a410 comp rx=0 tx=0).stop 2024-07-27T06:44:44.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.674+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2430818121 shutdown_connections 2024-07-27T06:44:44.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.674+0000 7f1f4dfe8640 1 --2- 172.21.15.29:0/2430818121 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f48109820 0x7f1f48110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:44.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.674+0000 7f1f4dfe8640 1 --2- 172.21.15.29:0/2430818121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f48108ed0 0x7f1f481092e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:44.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.674+0000 7f1f4dfe8640 1 --2- 172.21.15.29:0/2430818121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f48103590 0x7f1f481039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:44.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.674+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2430818121 >> 172.21.15.29:0/2430818121 conn(0x7f1f4807b280 msgr2=0x7f1f4807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:44.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.674+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2430818121 shutdown_connections 2024-07-27T06:44:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.674+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2430818121 wait complete. 2024-07-27T06:44:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f4dfe8640 1 Processor -- start 2024-07-27T06:44:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f4dfe8640 1 -- start start 2024-07-27T06:44:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f4dfe8640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f48103590 0x7f1f48199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f4dfe8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f48108ed0 0x7f1f4819a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f4dfe8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f48109820 0x7f1f4819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f4dfe8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1f48113500 con 0x7f1f48109820 2024-07-27T06:44:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f4dfe8640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f1f48113380 con 0x7f1f48103590 2024-07-27T06:44:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f4dfe8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1f48113680 con 0x7f1f48108ed0 2024-07-27T06:44:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f48109820 0x7f1f4819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f47fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f48109820 0x7f1f4819f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43400/0 (socket says 172.21.15.29:43400) 2024-07-27T06:44:44.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f47fff640 1 -- 172.21.15.29:0/2109040117 learned_addr learned my addr 172.21.15.29:0/2109040117 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f46ffd640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f48108ed0 0x7f1f4819a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.675+0000 7f1f477fe640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f48103590 0x7f1f48199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.676+0000 7f1f47fff640 1 -- 172.21.15.29:0/2109040117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f48108ed0 msgr2=0x7f1f4819a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.676+0000 7f1f47fff640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f48108ed0 0x7f1f4819a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.676+0000 7f1f47fff640 1 -- 172.21.15.29:0/2109040117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f48103590 msgr2=0x7f1f48199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.676+0000 7f1f47fff640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f48103590 0x7f1f48199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.676+0000 7f1f47fff640 1 -- 172.21.15.29:0/2109040117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f4819ff00 con 0x7f1f48109820 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.676+0000 7f1f47fff640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f48109820 0x7f1f4819f7d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000d750 tx=0x7f1f4000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.676+0000 7f1f44ff9640 1 -- 172.21.15.29:0/2109040117 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40017070 con 0x7f1f48109820 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.676+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2109040117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f481a01f0 con 0x7f1f48109820 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.676+0000 7f1f44ff9640 1 -- 172.21.15.29:0/2109040117 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f400054d0 con 0x7f1f48109820 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.676+0000 7f1f44ff9640 1 -- 172.21.15.29:0/2109040117 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40004c60 con 0x7f1f48109820 2024-07-27T06:44:44.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.676+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2109040117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f481a6950 con 0x7f1f48109820 2024-07-27T06:44:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.677+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2109040117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1f48063580 con 0x7f1f48109820 2024-07-27T06:44:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.679+0000 7f1f44ff9640 1 -- 172.21.15.29:0/2109040117 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f1f40009bd0 con 0x7f1f48109820 2024-07-27T06:44:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.679+0000 7f1f44ff9640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1f340b3cb0 0x7f1f340b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.679+0000 7f1f477fe640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1f340b3cb0 0x7f1f340b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.679+0000 7f1f44ff9640 1 -- 172.21.15.29:0/2109040117 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f1f400e3be0 con 0x7f1f48109820 2024-07-27T06:44:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.680+0000 7f1f44ff9640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780] conn(0x7f1f340bdb30 0x7f1f340c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.680+0000 7f1f44ff9640 1 -- 172.21.15.29:0/2109040117 --> [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f340c06f0 con 0x7f1f340bdb30 2024-07-27T06:44:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.680+0000 7f1f44ff9640 1 -- 172.21.15.29:0/2109040117 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f1f400e4030 con 0x7f1f48109820 2024-07-27T06:44:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.680+0000 7f1f477fe640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1f340b3cb0 0x7f1f340b61c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1f3800a400 tx=0x7f1f3800d040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.680+0000 7f1f46ffd640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780] conn(0x7f1f340bdb30 0x7f1f340c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.681+0000 7f1f46ffd640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780] conn(0x7f1f340bdb30 0x7f1f340c0020 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:44.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.681+0000 7f1f44ff9640 1 -- 172.21.15.29:0/2109040117 <== osd.5 v2:172.21.15.77:6808/1338438780 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f340c06f0 con 0x7f1f340bdb30 2024-07-27T06:44:44.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.695+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2109040117 --> [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1f48063a00 con 0x7f1f340bdb30 2024-07-27T06:44:44.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.696+0000 7f1f44ff9640 1 -- 172.21.15.29:0/2109040117 <== osd.5 v2:172.21.15.77:6808/1338438780 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1f48063a00 con 0x7f1f340bdb30 2024-07-27T06:44:44.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.696+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2109040117 >> [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780] conn(0x7f1f340bdb30 msgr2=0x7f1f340c0020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:44.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.696+0000 7f1f4dfe8640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780] conn(0x7f1f340bdb30 0x7f1f340c0020 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:44.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.696+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2109040117 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1f340b3cb0 msgr2=0x7f1f340b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:44.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.696+0000 7f1f4dfe8640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1f340b3cb0 0x7f1f340b61c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1f3800a400 tx=0x7f1f3800d040 comp rx=0 tx=0).stop 2024-07-27T06:44:44.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.696+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2109040117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f48109820 msgr2=0x7f1f4819f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:44.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.696+0000 7f1f4dfe8640 1 --2- 172.21.15.29:0/2109040117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f48109820 0x7f1f4819f7d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000d750 tx=0x7f1f4000dc10 comp rx=0 tx=0).stop 2024-07-27T06:44:44.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.696+0000 7f1f47fff640 1 -- 172.21.15.29:0/2109040117 reap_dead start 2024-07-27T06:44:44.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.696+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2109040117 shutdown_connections 2024-07-27T06:44:44.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.696+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2109040117 >> 172.21.15.29:0/2109040117 conn(0x7f1f4807b280 msgr2=0x7f1f48106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:44.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.696+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2109040117 shutdown_connections 2024-07-27T06:44:44.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:44.696+0000 7f1f4dfe8640 1 -- 172.21.15.29:0/2109040117 wait complete. 2024-07-27T06:44:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:45 smithi077 ceph-mon[37153]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:44:45.617 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:45 smithi029 ceph-mon[32610]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:44:45.618 INFO:teuthology.orchestra.run.smithi029.stdout:279172874276 2024-07-27T06:44:45.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-07-27T06:44:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:45 smithi047 ceph-mon[36835]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:44:46.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.133+0000 7f039fd1f640 1 -- 172.21.15.29:0/1015973380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0398102490 msgr2=0x7f03981028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:46.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.133+0000 7f039fd1f640 1 --2- 172.21.15.29:0/1015973380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0398102490 0x7f03981028a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0394009900 tx=0x7f039402a410 comp rx=0 tx=0).stop 2024-07-27T06:44:46.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.133+0000 7f039fd1f640 1 -- 172.21.15.29:0/1015973380 shutdown_connections 2024-07-27T06:44:46.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.133+0000 7f039fd1f640 1 --2- 172.21.15.29:0/1015973380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0398108720 0x7f039810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.133+0000 7f039fd1f640 1 --2- 172.21.15.29:0/1015973380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0398107dd0 0x7f03981081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.133+0000 7f039fd1f640 1 --2- 172.21.15.29:0/1015973380 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0398102490 0x7f03981028a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.133+0000 7f039fd1f640 1 -- 172.21.15.29:0/1015973380 >> 172.21.15.29:0/1015973380 conn(0x7f0398079160 msgr2=0x7f0398079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:46.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.133+0000 7f039fd1f640 1 -- 172.21.15.29:0/1015973380 shutdown_connections 2024-07-27T06:44:46.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.133+0000 7f039fd1f640 1 -- 172.21.15.29:0/1015973380 wait complete. 2024-07-27T06:44:46.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.134+0000 7f039fd1f640 1 Processor -- start 2024-07-27T06:44:46.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.134+0000 7f039fd1f640 1 -- start start 2024-07-27T06:44:46.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.134+0000 7f039fd1f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0398102490 0x7f039819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:46.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.134+0000 7f039fd1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0398107dd0 0x7f039819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:46.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.134+0000 7f039fd1f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0398108720 0x7f039819f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:46.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.134+0000 7f039fd1f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0398113570 con 0x7f0398107dd0 2024-07-27T06:44:46.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.134+0000 7f039fd1f640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f03981133f0 con 0x7f0398102490 2024-07-27T06:44:46.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.134+0000 7f039fd1f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f03981136f0 con 0x7f0398108720 2024-07-27T06:44:46.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.134+0000 7f039d2ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0398107dd0 0x7f039819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:46.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039d2ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0398107dd0 0x7f039819a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43428/0 (socket says 172.21.15.29:43428) 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039d2ba640 1 -- 172.21.15.29:0/1246450428 learned_addr learned my addr 172.21.15.29:0/1246450428 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039dabb640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0398102490 0x7f039819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039e2bc640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0398108720 0x7f039819f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039d2ba640 1 -- 172.21.15.29:0/1246450428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0398108720 msgr2=0x7f039819f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039d2ba640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0398108720 0x7f039819f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039d2ba640 1 -- 172.21.15.29:0/1246450428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0398102490 msgr2=0x7f039819a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039d2ba640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0398102490 0x7f039819a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039d2ba640 1 -- 172.21.15.29:0/1246450428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f039819ffd0 con 0x7f0398107dd0 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039e2bc640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0398108720 0x7f039819f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039d2ba640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0398107dd0 0x7f039819a600 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f038800b450 tx=0x7f038800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f038effd640 1 -- 172.21.15.29:0/1246450428 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0388015020 con 0x7f0398107dd0 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039fd1f640 1 -- 172.21.15.29:0/1246450428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03981a02c0 con 0x7f0398107dd0 2024-07-27T06:44:46.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f038effd640 1 -- 172.21.15.29:0/1246450428 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0388011d40 con 0x7f0398107dd0 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f038effd640 1 -- 172.21.15.29:0/1246450428 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038800f340 con 0x7f0398107dd0 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.135+0000 7f039fd1f640 1 -- 172.21.15.29:0/1246450428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03981a6a20 con 0x7f0398107dd0 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.136+0000 7f039fd1f640 1 -- 172.21.15.29:0/1246450428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0398063580 con 0x7f0398107dd0 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.138+0000 7f038effd640 1 -- 172.21.15.29:0/1246450428 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f0388018020 con 0x7f0398107dd0 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.139+0000 7f038effd640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f03840b3cb0 0x7f03840b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.139+0000 7f039dabb640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f03840b3cb0 0x7f03840b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.139+0000 7f038effd640 1 -- 172.21.15.29:0/1246450428 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f03880e3f50 con 0x7f0398107dd0 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.139+0000 7f038effd640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327] conn(0x7f03840bdb30 0x7f03840c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.139+0000 7f038effd640 1 -- 172.21.15.29:0/1246450428 --> [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0388017e50 con 0x7f03840bdb30 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.139+0000 7f038effd640 1 -- 172.21.15.29:0/1246450428 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f03880e4300 con 0x7f0398107dd0 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.139+0000 7f039dabb640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f03840b3cb0 0x7f03840b61c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0394031440 tx=0x7f0394005e80 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.139+0000 7f039e2bc640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327] conn(0x7f03840bdb30 0x7f03840c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.140+0000 7f039e2bc640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327] conn(0x7f03840bdb30 0x7f03840c0020 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:46.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.141+0000 7f038effd640 1 -- 172.21.15.29:0/1246450428 <== osd.11 v2:172.21.15.77:6824/3228299327 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0388017e50 con 0x7f03840bdb30 2024-07-27T06:44:46.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.157+0000 7f039fd1f640 1 -- 172.21.15.29:0/1246450428 --> [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f03980639b0 con 0x7f03840bdb30 2024-07-27T06:44:46.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.158+0000 7f038effd640 1 -- 172.21.15.29:0/1246450428 <== osd.11 v2:172.21.15.77:6824/3228299327 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f03980639b0 con 0x7f03840bdb30 2024-07-27T06:44:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.158+0000 7f039fd1f640 1 -- 172.21.15.29:0/1246450428 >> [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327] conn(0x7f03840bdb30 msgr2=0x7f03840c0020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.158+0000 7f039fd1f640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327] conn(0x7f03840bdb30 0x7f03840c0020 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.158+0000 7f039fd1f640 1 -- 172.21.15.29:0/1246450428 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f03840b3cb0 msgr2=0x7f03840b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.158+0000 7f039fd1f640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f03840b3cb0 0x7f03840b61c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0394031440 tx=0x7f0394005e80 comp rx=0 tx=0).stop 2024-07-27T06:44:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.158+0000 7f039fd1f640 1 -- 172.21.15.29:0/1246450428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0398107dd0 msgr2=0x7f039819a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.158+0000 7f039fd1f640 1 --2- 172.21.15.29:0/1246450428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0398107dd0 0x7f039819a600 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f038800b450 tx=0x7f038800b910 comp rx=0 tx=0).stop 2024-07-27T06:44:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.158+0000 7f039e2bc640 1 -- 172.21.15.29:0/1246450428 reap_dead start 2024-07-27T06:44:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.158+0000 7f039fd1f640 1 -- 172.21.15.29:0/1246450428 shutdown_connections 2024-07-27T06:44:46.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.158+0000 7f039fd1f640 1 -- 172.21.15.29:0/1246450428 >> 172.21.15.29:0/1246450428 conn(0x7f0398079160 msgr2=0x7f0398100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:46.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.158+0000 7f039fd1f640 1 -- 172.21.15.29:0/1246450428 shutdown_connections 2024-07-27T06:44:46.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.158+0000 7f039fd1f640 1 -- 172.21.15.29:0/1246450428 wait complete. 2024-07-27T06:44:46.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.420+0000 7f6bfdca3640 1 -- 172.21.15.29:0/3975567349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf81042d0 msgr2=0x7f6bf81046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:46.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.420+0000 7f6bfdca3640 1 --2- 172.21.15.29:0/3975567349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf81042d0 0x7f6bf81046e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009900 tx=0x7f6be802a3f0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.420+0000 7f6bfdca3640 1 -- 172.21.15.29:0/3975567349 shutdown_connections 2024-07-27T06:44:46.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.420+0000 7f6bfdca3640 1 --2- 172.21.15.29:0/3975567349 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bf810d880 0x7f6bf810fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.420+0000 7f6bfdca3640 1 --2- 172.21.15.29:0/3975567349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf8104d00 0x7f6bf810d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.420+0000 7f6bfdca3640 1 --2- 172.21.15.29:0/3975567349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf81042d0 0x7f6bf81046e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.420+0000 7f6bfdca3640 1 -- 172.21.15.29:0/3975567349 >> 172.21.15.29:0/3975567349 conn(0x7f6bf8079170 msgr2=0x7f6bf8079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.420+0000 7f6bfdca3640 1 -- 172.21.15.29:0/3975567349 shutdown_connections 2024-07-27T06:44:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.420+0000 7f6bfdca3640 1 -- 172.21.15.29:0/3975567349 wait complete. 2024-07-27T06:44:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.420+0000 7f6bfdca3640 1 Processor -- start 2024-07-27T06:44:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bfdca3640 1 -- start start 2024-07-27T06:44:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bfdca3640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bf81042d0 0x7f6bf8199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bfdca3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf8104d00 0x7f6bf819a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bfdca3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf810d880 0x7f6bf819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bfdca3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf81134d0 con 0x7f6bf810d880 2024-07-27T06:44:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bfdca3640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf8113350 con 0x7f6bf81042d0 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bfdca3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf8113650 con 0x7f6bf8104d00 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bf7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf810d880 0x7f6bf819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bf7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf810d880 0x7f6bf819f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43450/0 (socket says 172.21.15.29:43450) 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bf7fff640 1 -- 172.21.15.29:0/2347512917 learned_addr learned my addr 172.21.15.29:0/2347512917 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bf77fe640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bf81042d0 0x7f6bf8199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bf6ffd640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf8104d00 0x7f6bf819a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bf7fff640 1 -- 172.21.15.29:0/2347512917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf8104d00 msgr2=0x7f6bf819a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bf7fff640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf8104d00 0x7f6bf819a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bf7fff640 1 -- 172.21.15.29:0/2347512917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bf81042d0 msgr2=0x7f6bf8199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bf7fff640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bf81042d0 0x7f6bf8199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bf7fff640 1 -- 172.21.15.29:0/2347512917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf819fe50 con 0x7f6bf810d880 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bf77fe640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bf81042d0 0x7f6bf8199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:44:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.421+0000 7f6bf6ffd640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bf8104d00 0x7f6bf819a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.422+0000 7f6bf7fff640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf810d880 0x7f6bf819f7e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000cce0 tx=0x7f6bf000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.422+0000 7f6bf4ff9640 1 -- 172.21.15.29:0/2347512917 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf000f450 con 0x7f6bf810d880 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.422+0000 7f6bf4ff9640 1 -- 172.21.15.29:0/2347512917 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bf000faa0 con 0x7f6bf810d880 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.422+0000 7f6bfdca3640 1 -- 172.21.15.29:0/2347512917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf81a0140 con 0x7f6bf810d880 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.422+0000 7f6bf4ff9640 1 -- 172.21.15.29:0/2347512917 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf0018b70 con 0x7f6bf810d880 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.422+0000 7f6bfdca3640 1 -- 172.21.15.29:0/2347512917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf81a6960 con 0x7f6bf810d880 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.423+0000 7f6bfdca3640 1 -- 172.21.15.29:0/2347512917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6bf8063540 con 0x7f6bf810d880 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.424+0000 7f6bf4ff9640 1 -- 172.21.15.29:0/2347512917 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f6bf0018d10 con 0x7f6bf810d880 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.425+0000 7f6bf4ff9640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6be00b3ce0 0x7f6be00b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.425+0000 7f6bf77fe640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6be00b3ce0 0x7f6be00b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.425+0000 7f6bf4ff9640 1 -- 172.21.15.29:0/2347512917 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f6bf00e45b0 con 0x7f6bf810d880 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.425+0000 7f6bf4ff9640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749] conn(0x7f6be00bdb60 0x7f6be00c0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.425+0000 7f6bf4ff9640 1 -- 172.21.15.29:0/2347512917 --> [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6bf0024ea0 con 0x7f6be00bdb60 2024-07-27T06:44:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.425+0000 7f6bf4ff9640 1 -- 172.21.15.29:0/2347512917 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f6bf00e4a00 con 0x7f6bf810d880 2024-07-27T06:44:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.426+0000 7f6bf77fe640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6be00b3ce0 0x7f6be00b61f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6be8000c00 tx=0x7f6be8032da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.426+0000 7f6bf6ffd640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749] conn(0x7f6be00bdb60 0x7f6be00c0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.426+0000 7f6bf6ffd640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749] conn(0x7f6be00bdb60 0x7f6be00c0050 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.427+0000 7f6bf4ff9640 1 -- 172.21.15.29:0/2347512917 <== osd.1 v2:172.21.15.47:6800/2795953749 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bf0024ea0 con 0x7f6be00bdb60 2024-07-27T06:44:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.441+0000 7f6bfdca3640 1 -- 172.21.15.29:0/2347512917 --> [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6bf80639c0 con 0x7f6be00bdb60 2024-07-27T06:44:46.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.442+0000 7f6bf4ff9640 1 -- 172.21.15.29:0/2347512917 <== osd.1 v2:172.21.15.47:6800/2795953749 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6bf80639c0 con 0x7f6be00bdb60 2024-07-27T06:44:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.442+0000 7f6bfdca3640 1 -- 172.21.15.29:0/2347512917 >> [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749] conn(0x7f6be00bdb60 msgr2=0x7f6be00c0050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.442+0000 7f6bfdca3640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749] conn(0x7f6be00bdb60 0x7f6be00c0050 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.442+0000 7f6bfdca3640 1 -- 172.21.15.29:0/2347512917 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6be00b3ce0 msgr2=0x7f6be00b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.442+0000 7f6bfdca3640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6be00b3ce0 0x7f6be00b61f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6be8000c00 tx=0x7f6be8032da0 comp rx=0 tx=0).stop 2024-07-27T06:44:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.442+0000 7f6bfdca3640 1 -- 172.21.15.29:0/2347512917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf810d880 msgr2=0x7f6bf819f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.442+0000 7f6bfdca3640 1 --2- 172.21.15.29:0/2347512917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bf810d880 0x7f6bf819f7e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6bf000cce0 tx=0x7f6bf000a620 comp rx=0 tx=0).stop 2024-07-27T06:44:46.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.442+0000 7f6bf7fff640 1 -- 172.21.15.29:0/2347512917 reap_dead start 2024-07-27T06:44:46.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.442+0000 7f6bfdca3640 1 -- 172.21.15.29:0/2347512917 shutdown_connections 2024-07-27T06:44:46.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.442+0000 7f6bfdca3640 1 -- 172.21.15.29:0/2347512917 >> 172.21.15.29:0/2347512917 conn(0x7f6bf8079170 msgr2=0x7f6bf8100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:46.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.442+0000 7f6bfdca3640 1 -- 172.21.15.29:0/2347512917 shutdown_connections 2024-07-27T06:44:46.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:46.442+0000 7f6bfdca3640 1 -- 172.21.15.29:0/2347512917 wait complete. 2024-07-27T06:44:47.402 INFO:teuthology.orchestra.run.smithi029.stdout:81604378702 2024-07-27T06:44:47.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-07-27T06:44:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:47 smithi077 ceph-mon[37153]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:44:47.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:47 smithi029 ceph-mon[32610]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:44:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:47 smithi047 ceph-mon[36835]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:44:48.335 INFO:teuthology.orchestra.run.smithi029.stdout:60129542229 2024-07-27T06:44:48.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-07-27T06:44:48.689 INFO:teuthology.orchestra.run.smithi029.stdout:158913790016 2024-07-27T06:44:48.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-07-27T06:44:49.665 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:49 smithi029 ceph-mon[32610]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-07-27T06:44:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:49 smithi047 ceph-mon[36835]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-07-27T06:44:49.701 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:49.701 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:49 smithi077 ceph-mon[37153]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-07-27T06:44:50.162 INFO:teuthology.orchestra.run.smithi029.stdout:304942678051 2024-07-27T06:44:50.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-07-27T06:44:50.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.864+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/3371141591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb8107dd0 msgr2=0x7f0bb81081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:50.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.864+0000 7f0bbdb1f640 1 --2- 172.21.15.29:0/3371141591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb8107dd0 0x7f0bb81081e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0bac009900 tx=0x7f0bac02a410 comp rx=0 tx=0).stop 2024-07-27T06:44:50.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.864+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/3371141591 shutdown_connections 2024-07-27T06:44:50.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.864+0000 7f0bbdb1f640 1 --2- 172.21.15.29:0/3371141591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bb8108720 0x7f0bb810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:50.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.864+0000 7f0bbdb1f640 1 --2- 172.21.15.29:0/3371141591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb8107dd0 0x7f0bb81081e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:50.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.864+0000 7f0bbdb1f640 1 --2- 172.21.15.29:0/3371141591 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb8102490 0x7f0bb81028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:50.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.864+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/3371141591 >> 172.21.15.29:0/3371141591 conn(0x7f0bb8079160 msgr2=0x7f0bb8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:50.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.864+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/3371141591 shutdown_connections 2024-07-27T06:44:50.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.864+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/3371141591 wait complete. 2024-07-27T06:44:50.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.865+0000 7f0bbdb1f640 1 Processor -- start 2024-07-27T06:44:50.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.865+0000 7f0bbdb1f640 1 -- start start 2024-07-27T06:44:50.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.865+0000 7f0bbdb1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb8102490 0x7f0bb819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:50.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.865+0000 7f0bbdb1f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bb8107dd0 0x7f0bb819b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:50.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.865+0000 7f0bbdb1f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb8108720 0x7f0bb819b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:50.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.865+0000 7f0bbdb1f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb8112370 con 0x7f0bb8102490 2024-07-27T06:44:50.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.865+0000 7f0bbdb1f640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb81121f0 con 0x7f0bb8108720 2024-07-27T06:44:50.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.865+0000 7f0bbdb1f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb81124f0 con 0x7f0bb8107dd0 2024-07-27T06:44:50.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.865+0000 7f0bb77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb8102490 0x7f0bb819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:50.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.865+0000 7f0bb77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb8102490 0x7f0bb819ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55126/0 (socket says 172.21.15.29:55126) 2024-07-27T06:44:50.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.865+0000 7f0bb77fe640 1 -- 172.21.15.29:0/2121261487 learned_addr learned my addr 172.21.15.29:0/2121261487 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:50.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bb7fff640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb8108720 0x7f0bb819b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:50.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bb6ffd640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bb8107dd0 0x7f0bb819b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:50.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bb77fe640 1 -- 172.21.15.29:0/2121261487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bb8107dd0 msgr2=0x7f0bb819b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:50.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bb77fe640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bb8107dd0 0x7f0bb819b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:50.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bb77fe640 1 -- 172.21.15.29:0/2121261487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb8108720 msgr2=0x7f0bb819b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:50.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bb77fe640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb8108720 0x7f0bb819b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:50.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bb77fe640 1 -- 172.21.15.29:0/2121261487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb81a5fb0 con 0x7f0bb8102490 2024-07-27T06:44:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bb7fff640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bb8108720 0x7f0bb819b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:44:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bb77fe640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb8102490 0x7f0bb819ad20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0ba800e9f0 tx=0x7f0ba800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bb4ff9640 1 -- 172.21.15.29:0/2121261487 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba800cd80 con 0x7f0bb8102490 2024-07-27T06:44:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bb4ff9640 1 -- 172.21.15.29:0/2121261487 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ba80108e0 con 0x7f0bb8102490 2024-07-27T06:44:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/2121261487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bb81a62a0 con 0x7f0bb8102490 2024-07-27T06:44:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bb4ff9640 1 -- 172.21.15.29:0/2121261487 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8018980 con 0x7f0bb8102490 2024-07-27T06:44:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.866+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/2121261487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bb81a67b0 con 0x7f0bb8102490 2024-07-27T06:44:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.867+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/2121261487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0bb8063580 con 0x7f0bb8102490 2024-07-27T06:44:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.869+0000 7f0bb4ff9640 1 -- 172.21.15.29:0/2121261487 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f0ba8010430 con 0x7f0bb8102490 2024-07-27T06:44:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.869+0000 7f0bb4ff9640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0ba00b3ce0 0x7f0ba00b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.870+0000 7f0bb6ffd640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0ba00b3ce0 0x7f0ba00b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:50.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.870+0000 7f0bb4ff9640 1 -- 172.21.15.29:0/2121261487 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f0ba80e4a20 con 0x7f0bb8102490 2024-07-27T06:44:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.870+0000 7f0bb4ff9640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589] conn(0x7f0ba00bdb60 0x7f0ba00c0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.870+0000 7f0bb4ff9640 1 -- 172.21.15.29:0/2121261487 --> [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ba00c0720 con 0x7f0ba00bdb60 2024-07-27T06:44:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.870+0000 7f0bb4ff9640 1 -- 172.21.15.29:0/2121261487 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f0ba80e4e70 con 0x7f0bb8102490 2024-07-27T06:44:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.870+0000 7f0bb6ffd640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0ba00b3ce0 0x7f0ba00b61f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0bac000c00 tx=0x7f0bac031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.870+0000 7f0bb7fff640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589] conn(0x7f0ba00bdb60 0x7f0ba00c0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.871+0000 7f0bb7fff640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589] conn(0x7f0ba00bdb60 0x7f0ba00c0050 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.872+0000 7f0bb4ff9640 1 -- 172.21.15.29:0/2121261487 <== osd.7 v2:172.21.15.47:6816/2439195589 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ba00c0720 con 0x7f0ba00bdb60 2024-07-27T06:44:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.885+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/2121261487 --> [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0bb80639b0 con 0x7f0ba00bdb60 2024-07-27T06:44:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.886+0000 7f0bb4ff9640 1 -- 172.21.15.29:0/2121261487 <== osd.7 v2:172.21.15.47:6816/2439195589 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0bb80639b0 con 0x7f0ba00bdb60 2024-07-27T06:44:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.886+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/2121261487 >> [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589] conn(0x7f0ba00bdb60 msgr2=0x7f0ba00c0050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.886+0000 7f0bbdb1f640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589] conn(0x7f0ba00bdb60 0x7f0ba00c0050 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:50.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.886+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/2121261487 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0ba00b3ce0 msgr2=0x7f0ba00b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:50.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.886+0000 7f0bbdb1f640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f0ba00b3ce0 0x7f0ba00b61f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0bac000c00 tx=0x7f0bac031400 comp rx=0 tx=0).stop 2024-07-27T06:44:50.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.886+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/2121261487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb8102490 msgr2=0x7f0bb819ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:50.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.886+0000 7f0bbdb1f640 1 --2- 172.21.15.29:0/2121261487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0bb8102490 0x7f0bb819ad20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0ba800e9f0 tx=0x7f0ba800eeb0 comp rx=0 tx=0).stop 2024-07-27T06:44:50.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.886+0000 7f0bb7fff640 1 -- 172.21.15.29:0/2121261487 reap_dead start 2024-07-27T06:44:50.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.886+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/2121261487 shutdown_connections 2024-07-27T06:44:50.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.886+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/2121261487 >> 172.21.15.29:0/2121261487 conn(0x7f0bb8079160 msgr2=0x7f0bb8104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:50.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.886+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/2121261487 shutdown_connections 2024-07-27T06:44:50.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:50.886+0000 7f0bbdb1f640 1 -- 172.21.15.29:0/2121261487 wait complete. 2024-07-27T06:44:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:51 smithi047 ceph-mon[36835]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:51 smithi029 ceph-mon[32610]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:51.732 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:51.732 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:51.732 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:51.733 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:51.780 INFO:teuthology.orchestra.run.smithi029.stdout:206158430262 2024-07-27T06:44:51.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-07-27T06:44:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:51 smithi077 ceph-mon[37153]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:53.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:53 smithi047 ceph-mon[36835]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-07-27T06:44:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:53 smithi029 ceph-mon[32610]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-07-27T06:44:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:53 smithi077 ceph-mon[37153]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-07-27T06:44:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.547+0000 7fbc0100a640 1 -- 172.21.15.29:0/613772671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbfc107e00 msgr2=0x7fbbfc108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.547+0000 7fbc0100a640 1 --2- 172.21.15.29:0/613772671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbfc107e00 0x7fbbfc108210 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009900 tx=0x7fbbf002a410 comp rx=0 tx=0).stop 2024-07-27T06:44:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.547+0000 7fbc0100a640 1 -- 172.21.15.29:0/613772671 shutdown_connections 2024-07-27T06:44:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.547+0000 7fbc0100a640 1 --2- 172.21.15.29:0/613772671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbfc108750 0x7fbbfc10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.547+0000 7fbc0100a640 1 --2- 172.21.15.29:0/613772671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbfc107e00 0x7fbbfc108210 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.547+0000 7fbc0100a640 1 --2- 172.21.15.29:0/613772671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbfc1024c0 0x7fbbfc1028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.547+0000 7fbc0100a640 1 -- 172.21.15.29:0/613772671 >> 172.21.15.29:0/613772671 conn(0x7fbbfc079180 msgr2=0x7fbbfc079590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.547+0000 7fbc0100a640 1 -- 172.21.15.29:0/613772671 shutdown_connections 2024-07-27T06:44:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.547+0000 7fbc0100a640 1 -- 172.21.15.29:0/613772671 wait complete. 2024-07-27T06:44:55.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.548+0000 7fbc0100a640 1 Processor -- start 2024-07-27T06:44:55.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.548+0000 7fbc0100a640 1 -- start start 2024-07-27T06:44:55.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.548+0000 7fbc0100a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbfc1024c0 0x7fbbfc0ff020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:55.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.548+0000 7fbc0100a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbfc107e00 0x7fbbfc0ff560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:55.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbc0100a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbfc108750 0x7fbbfc0ffaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbc0100a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbbfc112380 con 0x7fbbfc108750 2024-07-27T06:44:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbc0100a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fbbfc112200 con 0x7fbbfc1024c0 2024-07-27T06:44:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbc0100a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbbfc112500 con 0x7fbbfc107e00 2024-07-27T06:44:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbbfb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbfc108750 0x7fbbfc0ffaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbbfb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbfc108750 0x7fbbfc0ffaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55134/0 (socket says 172.21.15.29:55134) 2024-07-27T06:44:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbbfb59e640 1 -- 172.21.15.29:0/2774182074 learned_addr learned my addr 172.21.15.29:0/2774182074 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbbfad9d640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbfc1024c0 0x7fbbfc0ff020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbbfb59e640 1 -- 172.21.15.29:0/2774182074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbfc107e00 msgr2=0x7fbbfc0ff560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbbfb59e640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbfc107e00 0x7fbbfc0ff560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbbfb59e640 1 -- 172.21.15.29:0/2774182074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbfc1024c0 msgr2=0x7fbbfc0ff020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbbfb59e640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbfc1024c0 0x7fbbfc0ff020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:55.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.549+0000 7fbbfb59e640 1 -- 172.21.15.29:0/2774182074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbfc1a8160 con 0x7fbbfc108750 2024-07-27T06:44:55.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.550+0000 7fbbfb59e640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbfc108750 0x7fbbfc0ffaa0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4016fd0 tx=0x7fbbf400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:55.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.550+0000 7fbbe3fff640 1 -- 172.21.15.29:0/2774182074 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4004b20 con 0x7fbbfc108750 2024-07-27T06:44:55.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.550+0000 7fbbe3fff640 1 -- 172.21.15.29:0/2774182074 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbf4004cc0 con 0x7fbbfc108750 2024-07-27T06:44:55.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.550+0000 7fbc0100a640 1 -- 172.21.15.29:0/2774182074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbfc199c10 con 0x7fbbfc108750 2024-07-27T06:44:55.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.550+0000 7fbbe3fff640 1 -- 172.21.15.29:0/2774182074 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf401e790 con 0x7fbbfc108750 2024-07-27T06:44:55.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.550+0000 7fbc0100a640 1 -- 172.21.15.29:0/2774182074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbfc19a100 con 0x7fbbfc108750 2024-07-27T06:44:55.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.551+0000 7fbc0100a640 1 -- 172.21.15.29:0/2774182074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbbfc063580 con 0x7fbbfc108750 2024-07-27T06:44:55.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.553+0000 7fbbe3fff640 1 -- 172.21.15.29:0/2774182074 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fbbf4005020 con 0x7fbbfc108750 2024-07-27T06:44:55.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.554+0000 7fbbe3fff640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbbe40b3cb0 0x7fbbe40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:55.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.554+0000 7fbbfad9d640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbbe40b3cb0 0x7fbbe40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:55.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.554+0000 7fbbe3fff640 1 -- 172.21.15.29:0/2774182074 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7fbbf40ecd40 con 0x7fbbfc108750 2024-07-27T06:44:55.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.554+0000 7fbbe3fff640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312] conn(0x7fbbe40bdb30 0x7fbbe40c0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:55.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.554+0000 7fbbe3fff640 1 -- 172.21.15.29:0/2774182074 --> [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbbf4019ec0 con 0x7fbbe40bdb30 2024-07-27T06:44:55.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.554+0000 7fbbe3fff640 1 -- 172.21.15.29:0/2774182074 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fbbf40ed190 con 0x7fbbfc108750 2024-07-27T06:44:55.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.554+0000 7fbbfa59c640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312] conn(0x7fbbe40bdb30 0x7fbbe40c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.555+0000 7fbbfad9d640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbbe40b3cb0 0x7fbbe40b61c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc100a70 tx=0x7fbbec00b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.555+0000 7fbbfa59c640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312] conn(0x7fbbe40bdb30 0x7fbbe40c0020 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:55.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.556+0000 7fbbe3fff640 1 -- 172.21.15.29:0/2774182074 <== osd.3 v2:172.21.15.29:6810/3757476312 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbbf4019ec0 con 0x7fbbe40bdb30 2024-07-27T06:44:55.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.570+0000 7fbc0100a640 1 -- 172.21.15.29:0/2774182074 --> [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbbfc063980 con 0x7fbbe40bdb30 2024-07-27T06:44:55.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.570+0000 7fbbe3fff640 1 -- 172.21.15.29:0/2774182074 <== osd.3 v2:172.21.15.29:6810/3757476312 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbbfc063980 con 0x7fbbe40bdb30 2024-07-27T06:44:55.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.570+0000 7fbc0100a640 1 -- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312] conn(0x7fbbe40bdb30 msgr2=0x7fbbe40c0020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:55.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.570+0000 7fbc0100a640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312] conn(0x7fbbe40bdb30 0x7fbbe40c0020 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:55.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.570+0000 7fbc0100a640 1 -- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbbe40b3cb0 msgr2=0x7fbbe40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:55.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.570+0000 7fbc0100a640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbbe40b3cb0 0x7fbbe40b61c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc100a70 tx=0x7fbbec00b390 comp rx=0 tx=0).stop 2024-07-27T06:44:55.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.570+0000 7fbc0100a640 1 -- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbfc108750 msgr2=0x7fbbfc0ffaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:55.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.570+0000 7fbc0100a640 1 --2- 172.21.15.29:0/2774182074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbfc108750 0x7fbbfc0ffaa0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4016fd0 tx=0x7fbbf400c940 comp rx=0 tx=0).stop 2024-07-27T06:44:55.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.570+0000 7fbbfb59e640 1 -- 172.21.15.29:0/2774182074 reap_dead start 2024-07-27T06:44:55.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.571+0000 7fbc0100a640 1 -- 172.21.15.29:0/2774182074 shutdown_connections 2024-07-27T06:44:55.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.571+0000 7fbc0100a640 1 -- 172.21.15.29:0/2774182074 >> 172.21.15.29:0/2774182074 conn(0x7fbbfc079180 msgr2=0x7fbbfc1014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:55.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.571+0000 7fbc0100a640 1 -- 172.21.15.29:0/2774182074 shutdown_connections 2024-07-27T06:44:55.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:55.571+0000 7fbc0100a640 1 -- 172.21.15.29:0/2774182074 wait complete. 2024-07-27T06:44:55.654 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:55 smithi029 ceph-mon[32610]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:44:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:55 smithi047 ceph-mon[36835]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:44:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:55 smithi077 ceph-mon[37153]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:44:56.145 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:56.145 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:56.489 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:44:56.956 INFO:teuthology.orchestra.run.smithi029.stdout:111669149772 2024-07-27T06:44:56.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-07-27T06:44:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:57 smithi029 ceph-mon[32610]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:57 smithi047 ceph-mon[36835]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:57 smithi077 ceph-mon[37153]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:44:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.386+0000 7fb699dc7640 1 -- 172.21.15.29:0/3105506787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb694108740 msgr2=0x7fb69410ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.386+0000 7fb699dc7640 1 --2- 172.21.15.29:0/3105506787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb694108740 0x7fb69410ef70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009ca0 tx=0x7fb68c02a8e0 comp rx=0 tx=0).stop 2024-07-27T06:44:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.386+0000 7fb699dc7640 1 -- 172.21.15.29:0/3105506787 shutdown_connections 2024-07-27T06:44:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.386+0000 7fb699dc7640 1 --2- 172.21.15.29:0/3105506787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb694108740 0x7fb69410ef70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.386+0000 7fb699dc7640 1 --2- 172.21.15.29:0/3105506787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb694107df0 0x7fb694108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.386+0000 7fb699dc7640 1 --2- 172.21.15.29:0/3105506787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb694102500 0x7fb694102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.386+0000 7fb699dc7640 1 -- 172.21.15.29:0/3105506787 >> 172.21.15.29:0/3105506787 conn(0x7fb694079100 msgr2=0x7fb694079510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.387+0000 7fb699dc7640 1 -- 172.21.15.29:0/3105506787 shutdown_connections 2024-07-27T06:44:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.387+0000 7fb699dc7640 1 -- 172.21.15.29:0/3105506787 wait complete. 2024-07-27T06:44:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.387+0000 7fb699dc7640 1 Processor -- start 2024-07-27T06:44:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb699dc7640 1 -- start start 2024-07-27T06:44:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb699dc7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb694102500 0x7fb694198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb699dc7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb694107df0 0x7fb694199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb699dc7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb694108740 0x7fb69419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb699dc7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb694112440 con 0x7fb694107df0 2024-07-27T06:44:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb699dc7640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fb6941122c0 con 0x7fb694108740 2024-07-27T06:44:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb699dc7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb6941125c0 con 0x7fb694102500 2024-07-27T06:44:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb692ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb694107df0 0x7fb694199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb692ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb694107df0 0x7fb694199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55378/0 (socket says 172.21.15.29:55378) 2024-07-27T06:44:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb692ffd640 1 -- 172.21.15.29:0/2320914797 learned_addr learned my addr 172.21.15.29:0/2320914797 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:44:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb6937fe640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb694102500 0x7fb694198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb693fff640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb694108740 0x7fb69419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb692ffd640 1 -- 172.21.15.29:0/2320914797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb694102500 msgr2=0x7fb694198db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb692ffd640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb694102500 0x7fb694198db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb692ffd640 1 -- 172.21.15.29:0/2320914797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb694108740 msgr2=0x7fb69419e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb692ffd640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb694108740 0x7fb69419e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.388+0000 7fb692ffd640 1 -- 172.21.15.29:0/2320914797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69419ed20 con 0x7fb694107df0 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.389+0000 7fb692ffd640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb694107df0 0x7fb694199350 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb68800d8f0 tx=0x7fb68800ddb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.389+0000 7fb690ff9640 1 -- 172.21.15.29:0/2320914797 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68801b070 con 0x7fb694107df0 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.389+0000 7fb699dc7640 1 -- 172.21.15.29:0/2320914797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb69419f010 con 0x7fb694107df0 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.389+0000 7fb690ff9640 1 -- 172.21.15.29:0/2320914797 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb68800ed60 con 0x7fb694107df0 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.389+0000 7fb690ff9640 1 -- 172.21.15.29:0/2320914797 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6880147e0 con 0x7fb694107df0 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.389+0000 7fb699dc7640 1 -- 172.21.15.29:0/2320914797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb69406b570 con 0x7fb694107df0 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.390+0000 7fb699dc7640 1 -- 172.21.15.29:0/2320914797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb694103730 con 0x7fb694107df0 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.392+0000 7fb690ff9640 1 -- 172.21.15.29:0/2320914797 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fb68801e070 con 0x7fb694107df0 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.393+0000 7fb690ff9640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb67c0b3d30 0x7fb67c0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.393+0000 7fb6937fe640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb67c0b3d30 0x7fb67c0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:44:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.393+0000 7fb690ff9640 1 -- 172.21.15.29:0/2320914797 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7fb6880e5920 con 0x7fb694107df0 2024-07-27T06:44:59.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.393+0000 7fb6937fe640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb67c0b3d30 0x7fb67c0b6240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb684005d00 tx=0x7fb68400b410 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:44:59.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.396+0000 7fb690ff9640 1 -- 172.21.15.29:0/2320914797 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb6880ac800 con 0x7fb694107df0 2024-07-27T06:44:59.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.584+0000 7fb699dc7640 1 -- 172.21.15.29:0/2320914797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fb694199ee0 con 0x7fb694107df0 2024-07-27T06:44:59.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:44:59 smithi029 ceph-mon[32610]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:44:59.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.584+0000 7fb690ff9640 1 -- 172.21.15.29:0/2320914797 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb6880b0d50 con 0x7fb694107df0 2024-07-27T06:44:59.586 INFO:teuthology.orchestra.run.smithi029.stdout:81604378707 2024-07-27T06:44:59.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.586+0000 7fb699dc7640 1 -- 172.21.15.29:0/2320914797 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb67c0b3d30 msgr2=0x7fb67c0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:59.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.586+0000 7fb699dc7640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb67c0b3d30 0x7fb67c0b6240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb684005d00 tx=0x7fb68400b410 comp rx=0 tx=0).stop 2024-07-27T06:44:59.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.586+0000 7fb699dc7640 1 -- 172.21.15.29:0/2320914797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb694107df0 msgr2=0x7fb694199350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:44:59.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.586+0000 7fb699dc7640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb694107df0 0x7fb694199350 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb68800d8f0 tx=0x7fb68800ddb0 comp rx=0 tx=0).stop 2024-07-27T06:44:59.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.587+0000 7fb699dc7640 1 -- 172.21.15.29:0/2320914797 shutdown_connections 2024-07-27T06:44:59.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.587+0000 7fb699dc7640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb694108740 0x7fb69419e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:59.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.587+0000 7fb699dc7640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb694107df0 0x7fb694199350 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:59.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.587+0000 7fb699dc7640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb67c0b3d30 0x7fb67c0b6240 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:59.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.587+0000 7fb699dc7640 1 --2- 172.21.15.29:0/2320914797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb694102500 0x7fb694198db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:44:59.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.587+0000 7fb699dc7640 1 -- 172.21.15.29:0/2320914797 >> 172.21.15.29:0/2320914797 conn(0x7fb694079100 msgr2=0x7fb694105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:44:59.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.587+0000 7fb699dc7640 1 -- 172.21.15.29:0/2320914797 shutdown_connections 2024-07-27T06:44:59.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:44:59.587+0000 7fb699dc7640 1 -- 172.21.15.29:0/2320914797 wait complete. 2024-07-27T06:44:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:44:59 smithi047 ceph-mon[36835]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:44:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:44:59 smithi077 ceph-mon[37153]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:45:00.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:00 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2320914797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-27T06:45:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:00 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2320914797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-27T06:45:00.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.738+0000 7f2ef607d640 1 -- 172.21.15.29:0/3918892018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef0108720 msgr2=0x7f2ef010ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:00.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.738+0000 7f2ef607d640 1 --2- 172.21.15.29:0/3918892018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef0108720 0x7f2ef010ef50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009ca0 tx=0x7f2ee802a8c0 comp rx=0 tx=0).stop 2024-07-27T06:45:00.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.738+0000 7f2ef607d640 1 -- 172.21.15.29:0/3918892018 shutdown_connections 2024-07-27T06:45:00.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.738+0000 7f2ef607d640 1 --2- 172.21.15.29:0/3918892018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef0108720 0x7f2ef010ef50 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:00.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.738+0000 7f2ef607d640 1 --2- 172.21.15.29:0/3918892018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ef0107dd0 0x7f2ef01081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:00.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.738+0000 7f2ef607d640 1 --2- 172.21.15.29:0/3918892018 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0102490 0x7f2ef01028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:00.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.738+0000 7f2ef607d640 1 -- 172.21.15.29:0/3918892018 >> 172.21.15.29:0/3918892018 conn(0x7f2ef0079160 msgr2=0x7f2ef0079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:00.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.738+0000 7f2ef607d640 1 -- 172.21.15.29:0/3918892018 shutdown_connections 2024-07-27T06:45:00.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.738+0000 7f2ef607d640 1 -- 172.21.15.29:0/3918892018 wait complete. 2024-07-27T06:45:00.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2ef607d640 1 Processor -- start 2024-07-27T06:45:00.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2ef607d640 1 -- start start 2024-07-27T06:45:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2ef607d640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ef0102490 0x7f2ef0199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2ef607d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef0107dd0 0x7f2ef019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2ef607d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0108720 0x7f2ef019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2ef607d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef01123a0 con 0x7f2ef0107dd0 2024-07-27T06:45:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2ef607d640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef0112220 con 0x7f2ef0102490 2024-07-27T06:45:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2ef607d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef0112520 con 0x7f2ef0108720 2024-07-27T06:45:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2eeeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef0107dd0 0x7f2ef019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2eeeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef0107dd0 0x7f2ef019a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55398/0 (socket says 172.21.15.29:55398) 2024-07-27T06:45:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2eeeffd640 1 -- 172.21.15.29:0/614226123 learned_addr learned my addr 172.21.15.29:0/614226123 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2eef7fe640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ef0102490 0x7f2ef0199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.739+0000 7f2eeffff640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0108720 0x7f2ef019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.740+0000 7f2eeeffd640 1 -- 172.21.15.29:0/614226123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0108720 msgr2=0x7f2ef019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.740+0000 7f2eeeffd640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0108720 0x7f2ef019f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.740+0000 7f2eeeffd640 1 -- 172.21.15.29:0/614226123 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ef0102490 msgr2=0x7f2ef0199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.740+0000 7f2eeeffd640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ef0102490 0x7f2ef0199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.740+0000 7f2eeeffd640 1 -- 172.21.15.29:0/614226123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef019fe50 con 0x7f2ef0107dd0 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.740+0000 7f2eeffff640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0108720 0x7f2ef019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.740+0000 7f2eeeffd640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef0107dd0 0x7f2ef019a480 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400d770 tx=0x7f2ee400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.740+0000 7f2eecff9640 1 -- 172.21.15.29:0/614226123 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4017070 con 0x7f2ef0107dd0 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.740+0000 7f2eecff9640 1 -- 172.21.15.29:0/614226123 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ee4009c30 con 0x7f2ef0107dd0 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.740+0000 7f2ef607d640 1 -- 172.21.15.29:0/614226123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ef01a0140 con 0x7f2ef0107dd0 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.740+0000 7f2eecff9640 1 -- 172.21.15.29:0/614226123 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee400f620 con 0x7f2ef0107dd0 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.740+0000 7f2ef607d640 1 -- 172.21.15.29:0/614226123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ef01a68a0 con 0x7f2ef0107dd0 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.741+0000 7f2ef607d640 1 -- 172.21.15.29:0/614226123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ef01028a0 con 0x7f2ef0107dd0 2024-07-27T06:45:00.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.743+0000 7f2eecff9640 1 -- 172.21.15.29:0/614226123 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7f2ee401a070 con 0x7f2ef0107dd0 2024-07-27T06:45:00.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.744+0000 7f2eecff9640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2ed80b3cb0 0x7f2ed80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:00.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.744+0000 7f2eef7fe640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2ed80b3cb0 0x7f2ed80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:00.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.744+0000 7f2eecff9640 1 -- 172.21.15.29:0/614226123 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7f2ee40e3900 con 0x7f2ef0107dd0 2024-07-27T06:45:00.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.744+0000 7f2eef7fe640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2ed80b3cb0 0x7f2ed80b61c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00058c0 tx=0x7f2ee000a250 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:00.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.747+0000 7f2eecff9640 1 -- 172.21.15.29:0/614226123 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2ee4009de0 con 0x7f2ef0107dd0 2024-07-27T06:45:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:00 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2320914797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-27T06:45:00.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.933+0000 7f2ef607d640 1 -- 172.21.15.29:0/614226123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2ef002e0c0 con 0x7f2ef0107dd0 2024-07-27T06:45:00.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.933+0000 7f2eecff9640 1 -- 172.21.15.29:0/614226123 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2ee40aad20 con 0x7f2ef0107dd0 2024-07-27T06:45:00.934 INFO:teuthology.orchestra.run.smithi029.stdout:137438953544 2024-07-27T06:45:00.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.935+0000 7f2ef607d640 1 -- 172.21.15.29:0/614226123 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2ed80b3cb0 msgr2=0x7f2ed80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:00.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.935+0000 7f2ef607d640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2ed80b3cb0 0x7f2ed80b61c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00058c0 tx=0x7f2ee000a250 comp rx=0 tx=0).stop 2024-07-27T06:45:00.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.935+0000 7f2ef607d640 1 -- 172.21.15.29:0/614226123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef0107dd0 msgr2=0x7f2ef019a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:00.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.935+0000 7f2ef607d640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef0107dd0 0x7f2ef019a480 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400d770 tx=0x7f2ee400dc30 comp rx=0 tx=0).stop 2024-07-27T06:45:00.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.936+0000 7f2ef607d640 1 -- 172.21.15.29:0/614226123 shutdown_connections 2024-07-27T06:45:00.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.936+0000 7f2ef607d640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2ed80b3cb0 0x7f2ed80b61c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:00.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.936+0000 7f2ef607d640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ef0108720 0x7f2ef019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:00.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.936+0000 7f2ef607d640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ef0107dd0 0x7f2ef019a480 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:00.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.936+0000 7f2ef607d640 1 --2- 172.21.15.29:0/614226123 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2ef0102490 0x7f2ef0199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:00.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.936+0000 7f2ef607d640 1 -- 172.21.15.29:0/614226123 >> 172.21.15.29:0/614226123 conn(0x7f2ef0079160 msgr2=0x7f2ef006ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:00.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.936+0000 7f2ef607d640 1 -- 172.21.15.29:0/614226123 shutdown_connections 2024-07-27T06:45:00.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:00.936+0000 7f2ef607d640 1 -- 172.21.15.29:0/614226123 wait complete. 2024-07-27T06:45:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:01 smithi047 ceph-mon[36835]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:01 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/614226123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-27T06:45:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:01 smithi029 ceph-mon[32610]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:01 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/614226123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-27T06:45:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:01 smithi077 ceph-mon[37153]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:01 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/614226123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-27T06:45:02.489 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953540 got 137438953544 for osd.4 2024-07-27T06:45:02.490 DEBUG:teuthology.parallel:result is None 2024-07-27T06:45:03.439 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378702 got 81604378707 for osd.2 2024-07-27T06:45:03.439 DEBUG:teuthology.parallel:result is None 2024-07-27T06:45:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:03 smithi047 ceph-mon[36835]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:45:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:03 smithi029 ceph-mon[32610]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:45:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:03 smithi077 ceph-mon[37153]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:45:04.677 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:04.677 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:05 smithi047 ceph-mon[36835]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-27T06:45:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:05 smithi029 ceph-mon[32610]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-27T06:45:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:05 smithi077 ceph-mon[37153]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-27T06:45:06.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.017+0000 7fd5f124d640 1 -- 172.21.15.29:0/3552163528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5ec107dd0 msgr2=0x7fd5ec1081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:06.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.017+0000 7fd5f124d640 1 --2- 172.21.15.29:0/3552163528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5ec107dd0 0x7fd5ec1081e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009900 tx=0x7fd5e002a410 comp rx=0 tx=0).stop 2024-07-27T06:45:06.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.017+0000 7fd5f124d640 1 -- 172.21.15.29:0/3552163528 shutdown_connections 2024-07-27T06:45:06.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.017+0000 7fd5f124d640 1 --2- 172.21.15.29:0/3552163528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec108720 0x7fd5ec10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:06.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.017+0000 7fd5f124d640 1 --2- 172.21.15.29:0/3552163528 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5ec107dd0 0x7fd5ec1081e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:06.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.017+0000 7fd5f124d640 1 --2- 172.21.15.29:0/3552163528 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5ec102490 0x7fd5ec1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:06.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.017+0000 7fd5f124d640 1 -- 172.21.15.29:0/3552163528 >> 172.21.15.29:0/3552163528 conn(0x7fd5ec079160 msgr2=0x7fd5ec079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:06.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.017+0000 7fd5f124d640 1 -- 172.21.15.29:0/3552163528 shutdown_connections 2024-07-27T06:45:06.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.017+0000 7fd5f124d640 1 -- 172.21.15.29:0/3552163528 wait complete. 2024-07-27T06:45:06.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5f124d640 1 Processor -- start 2024-07-27T06:45:06.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5f124d640 1 -- start start 2024-07-27T06:45:06.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5f124d640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5ec102490 0x7fd5ec104b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:06.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5f124d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5ec107dd0 0x7fd5ec105050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:06.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5f124d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec108720 0x7fd5ec105590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:06.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5f124d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec112370 con 0x7fd5ec107dd0 2024-07-27T06:45:06.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5f124d640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec1121f0 con 0x7fd5ec102490 2024-07-27T06:45:06.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5f124d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec1124f0 con 0x7fd5ec108720 2024-07-27T06:45:06.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5ea59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5ec107dd0 0x7fd5ec105050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:06.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5ea59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5ec107dd0 0x7fd5ec105050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55418/0 (socket says 172.21.15.29:55418) 2024-07-27T06:45:06.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5ea59c640 1 -- 172.21.15.29:0/3859178375 learned_addr learned my addr 172.21.15.29:0/3859178375 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:06.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5ead9d640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5ec102490 0x7fd5ec104b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:06.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.018+0000 7fd5eb59e640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec108720 0x7fd5ec105590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:06.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.019+0000 7fd5ea59c640 1 -- 172.21.15.29:0/3859178375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec108720 msgr2=0x7fd5ec105590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:06.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.019+0000 7fd5ea59c640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec108720 0x7fd5ec105590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:06.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.019+0000 7fd5ea59c640 1 -- 172.21.15.29:0/3859178375 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5ec102490 msgr2=0x7fd5ec104b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:06.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.019+0000 7fd5ea59c640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5ec102490 0x7fd5ec104b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:06.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.019+0000 7fd5ea59c640 1 -- 172.21.15.29:0/3859178375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5ec1a8140 con 0x7fd5ec107dd0 2024-07-27T06:45:06.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.019+0000 7fd5eb59e640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec108720 0x7fd5ec105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:45:06.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.019+0000 7fd5ea59c640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5ec107dd0 0x7fd5ec105050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0006010 tx=0x7fd5e0004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:06.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.019+0000 7fd5d3fff640 1 -- 172.21.15.29:0/3859178375 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e003f030 con 0x7fd5ec107dd0 2024-07-27T06:45:06.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.019+0000 7fd5d3fff640 1 -- 172.21.15.29:0/3859178375 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5e0003aa0 con 0x7fd5ec107dd0 2024-07-27T06:45:06.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.019+0000 7fd5d3fff640 1 -- 172.21.15.29:0/3859178375 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0035070 con 0x7fd5ec107dd0 2024-07-27T06:45:06.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.019+0000 7fd5f124d640 1 -- 172.21.15.29:0/3859178375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5ec1a83d0 con 0x7fd5ec107dd0 2024-07-27T06:45:06.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.019+0000 7fd5f124d640 1 -- 172.21.15.29:0/3859178375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5ec1a88b0 con 0x7fd5ec107dd0 2024-07-27T06:45:06.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.020+0000 7fd5f124d640 1 -- 172.21.15.29:0/3859178375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5ec1028a0 con 0x7fd5ec107dd0 2024-07-27T06:45:06.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.022+0000 7fd5d3fff640 1 -- 172.21.15.29:0/3859178375 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) ==== 151127+0+0 (secure 0 0 0) 0x7fd5e0042030 con 0x7fd5ec107dd0 2024-07-27T06:45:06.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.022+0000 7fd5d3fff640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd5d40b3cb0 0x7fd5d40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:06.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.023+0000 7fd5ead9d640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd5d40b3cb0 0x7fd5d40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:06.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.023+0000 7fd5d3fff640 1 -- 172.21.15.29:0/3859178375 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8442+0+0 (secure 0 0 0) 0x7fd5e0104330 con 0x7fd5ec107dd0 2024-07-27T06:45:06.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.023+0000 7fd5ead9d640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd5d40b3cb0 0x7fd5d40b61c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec106560 tx=0x7fd5dc005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:06.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.027+0000 7fd5d3fff640 1 -- 172.21.15.29:0/3859178375 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd5e00cb750 con 0x7fd5ec107dd0 2024-07-27T06:45:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.221+0000 7fd5f124d640 1 -- 172.21.15.29:0/3859178375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fd5ec10af90 con 0x7fd5ec107dd0 2024-07-27T06:45:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.221+0000 7fd5d3fff640 1 -- 172.21.15.29:0/3859178375 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fd5e00cb050 con 0x7fd5ec107dd0 2024-07-27T06:45:06.222 INFO:teuthology.orchestra.run.smithi029.stdout:279172874282 2024-07-27T06:45:06.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.223+0000 7fd5f124d640 1 -- 172.21.15.29:0/3859178375 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd5d40b3cb0 msgr2=0x7fd5d40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:06.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.223+0000 7fd5f124d640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd5d40b3cb0 0x7fd5d40b61c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec106560 tx=0x7fd5dc005f90 comp rx=0 tx=0).stop 2024-07-27T06:45:06.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.223+0000 7fd5f124d640 1 -- 172.21.15.29:0/3859178375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5ec107dd0 msgr2=0x7fd5ec105050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:06.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.223+0000 7fd5f124d640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5ec107dd0 0x7fd5ec105050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0006010 tx=0x7fd5e0004060 comp rx=0 tx=0).stop 2024-07-27T06:45:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.223+0000 7fd5f124d640 1 -- 172.21.15.29:0/3859178375 shutdown_connections 2024-07-27T06:45:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.223+0000 7fd5f124d640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd5d40b3cb0 0x7fd5d40b61c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.223+0000 7fd5f124d640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5ec108720 0x7fd5ec105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.223+0000 7fd5f124d640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5ec107dd0 0x7fd5ec105050 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.223+0000 7fd5f124d640 1 --2- 172.21.15.29:0/3859178375 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5ec102490 0x7fd5ec104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.223+0000 7fd5f124d640 1 -- 172.21.15.29:0/3859178375 >> 172.21.15.29:0/3859178375 conn(0x7fd5ec079160 msgr2=0x7fd5ec06ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.224+0000 7fd5f124d640 1 -- 172.21.15.29:0/3859178375 shutdown_connections 2024-07-27T06:45:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:06.224+0000 7fd5f124d640 1 -- 172.21.15.29:0/3859178375 wait complete. 2024-07-27T06:45:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:06 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3859178375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-27T06:45:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:06 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3859178375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-27T06:45:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:06 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3859178375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-27T06:45:07.646 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:07 smithi029 ceph-mon[32610]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:45:07.646 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:45:07.665 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:07 smithi077 ceph-mon[37153]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:45:07.665 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:45:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:07 smithi047 ceph-mon[36835]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:45:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:45:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-07-27T06:45:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-07-27T06:45:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:08.710 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-07-27T06:45:08.710 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-07-27T06:45:08.710 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:08.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-07-27T06:45:08.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-07-27T06:45:08.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.258+0000 7f816bf71640 1 -- 172.21.15.29:0/1243248349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f816410d840 msgr2=0x7f816410fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.258+0000 7f816bf71640 1 --2- 172.21.15.29:0/1243248349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f816410d840 0x7f816410fd30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f815c009ca0 tx=0x7f815c02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:45:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.258+0000 7f816bf71640 1 -- 172.21.15.29:0/1243248349 shutdown_connections 2024-07-27T06:45:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.258+0000 7f816bf71640 1 --2- 172.21.15.29:0/1243248349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f816410d840 0x7f816410fd30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.258+0000 7f816bf71640 1 --2- 172.21.15.29:0/1243248349 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8164104d30 0x7f816410d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.258+0000 7f816bf71640 1 --2- 172.21.15.29:0/1243248349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81641043e0 0x7f81641047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.258+0000 7f816bf71640 1 -- 172.21.15.29:0/1243248349 >> 172.21.15.29:0/1243248349 conn(0x7f8164079160 msgr2=0x7f8164079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.258+0000 7f816bf71640 1 -- 172.21.15.29:0/1243248349 shutdown_connections 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.258+0000 7f816bf71640 1 -- 172.21.15.29:0/1243248349 wait complete. 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.258+0000 7f816bf71640 1 Processor -- start 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f816bf71640 1 -- start start 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f816bf71640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81641043e0 0x7f8164199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f816bf71640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8164104d30 0x7f816419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f816bf71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f816410d840 0x7f816419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f816bf71640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f81641134c0 con 0x7f81641043e0 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f816bf71640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f8164113340 con 0x7f8164104d30 2024-07-27T06:45:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f816bf71640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8164113640 con 0x7f816410d840 2024-07-27T06:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f8169d0d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81641043e0 0x7f8164199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f8169d0d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81641043e0 0x7f8164199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49256/0 (socket says 172.21.15.29:49256) 2024-07-27T06:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f8169d0d640 1 -- 172.21.15.29:0/3872723636 learned_addr learned my addr 172.21.15.29:0/3872723636 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f816950c640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8164104d30 0x7f816419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f8169d0d640 1 -- 172.21.15.29:0/3872723636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f816410d840 msgr2=0x7f816419f730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-27T06:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f8169d0d640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f816410d840 0x7f816419f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f8169d0d640 1 -- 172.21.15.29:0/3872723636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8164104d30 msgr2=0x7f816419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f8169d0d640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8164104d30 0x7f816419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.259+0000 7f8169d0d640 1 -- 172.21.15.29:0/3872723636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f816419fe60 con 0x7f81641043e0 2024-07-27T06:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.260+0000 7f8169d0d640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81641043e0 0x7f8164199ef0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f816000e9c0 tx=0x7f816000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.260+0000 7f815affd640 1 -- 172.21.15.29:0/3872723636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816000cd80 con 0x7f81641043e0 2024-07-27T06:45:09.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.260+0000 7f815affd640 1 -- 172.21.15.29:0/3872723636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81600035f0 con 0x7f81641043e0 2024-07-27T06:45:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.260+0000 7f816bf71640 1 -- 172.21.15.29:0/3872723636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81641a0150 con 0x7f81641043e0 2024-07-27T06:45:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.260+0000 7f815affd640 1 -- 172.21.15.29:0/3872723636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160010690 con 0x7f81641043e0 2024-07-27T06:45:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.260+0000 7f816bf71640 1 -- 172.21.15.29:0/3872723636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81641a68b0 con 0x7f81641043e0 2024-07-27T06:45:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.261+0000 7f816bf71640 1 -- 172.21.15.29:0/3872723636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81641047f0 con 0x7f81641043e0 2024-07-27T06:45:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.263+0000 7f815affd640 1 -- 172.21.15.29:0/3872723636 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f8160004000 con 0x7f81641043e0 2024-07-27T06:45:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.264+0000 7f815affd640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f81500b3c20 0x7f81500b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.264+0000 7f816950c640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f81500b3c20 0x7f81500b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.264+0000 7f815affd640 1 -- 172.21.15.29:0/3872723636 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8500+0+0 (secure 0 0 0) 0x7f81600e4520 con 0x7f81641043e0 2024-07-27T06:45:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.264+0000 7f816950c640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f81500b3c20 0x7f81500b6130 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f816410f490 tx=0x7f815400b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:09.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.271+0000 7f815affd640 1 -- 172.21.15.29:0/3872723636 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f81600aba00 con 0x7f81641043e0 2024-07-27T06:45:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.462+0000 7f816bf71640 1 -- 172.21.15.29:0/3872723636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f816419b020 con 0x7f81641043e0 2024-07-27T06:45:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.462+0000 7f815affd640 1 -- 172.21.15.29:0/3872723636 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f81600ab300 con 0x7f81641043e0 2024-07-27T06:45:09.463 INFO:teuthology.orchestra.run.smithi029.stdout:184683593792 2024-07-27T06:45:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.464+0000 7f816bf71640 1 -- 172.21.15.29:0/3872723636 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f81500b3c20 msgr2=0x7f81500b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.464+0000 7f816bf71640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f81500b3c20 0x7f81500b6130 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f816410f490 tx=0x7f815400b390 comp rx=0 tx=0).stop 2024-07-27T06:45:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.464+0000 7f816bf71640 1 -- 172.21.15.29:0/3872723636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81641043e0 msgr2=0x7f8164199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.464+0000 7f816bf71640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81641043e0 0x7f8164199ef0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f816000e9c0 tx=0x7f816000ee80 comp rx=0 tx=0).stop 2024-07-27T06:45:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.464+0000 7f816bf71640 1 -- 172.21.15.29:0/3872723636 shutdown_connections 2024-07-27T06:45:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.464+0000 7f816bf71640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f81500b3c20 0x7f81500b6130 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.464+0000 7f816bf71640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f816410d840 0x7f816419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.464+0000 7f816bf71640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8164104d30 0x7f816419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.464+0000 7f816bf71640 1 --2- 172.21.15.29:0/3872723636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81641043e0 0x7f8164199ef0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.465+0000 7f816bf71640 1 -- 172.21.15.29:0/3872723636 >> 172.21.15.29:0/3872723636 conn(0x7f8164079160 msgr2=0x7f81640fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.465+0000 7f816bf71640 1 -- 172.21.15.29:0/3872723636 shutdown_connections 2024-07-27T06:45:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:09.465+0000 7f816bf71640 1 -- 172.21.15.29:0/3872723636 wait complete. 2024-07-27T06:45:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:09 smithi047 ceph-mon[36835]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:45:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-07-27T06:45:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-07-27T06:45:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:09 smithi047 ceph-mon[36835]: osdmap e82: 12 total, 12 up, 12 in 2024-07-27T06:45:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:09 smithi047 ceph-mon[36835]: mgrmap e22: x(active, since 9m), standbys: y, z 2024-07-27T06:45:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:09 smithi029 ceph-mon[32610]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:45:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-07-27T06:45:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-07-27T06:45:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:09 smithi029 ceph-mon[32610]: osdmap e82: 12 total, 12 up, 12 in 2024-07-27T06:45:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:09 smithi029 ceph-mon[32610]: mgrmap e22: x(active, since 9m), standbys: y, z 2024-07-27T06:45:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:09 smithi077 ceph-mon[37153]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:45:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-07-27T06:45:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-07-27T06:45:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:09 smithi077 ceph-mon[37153]: osdmap e82: 12 total, 12 up, 12 in 2024-07-27T06:45:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:09 smithi077 ceph-mon[37153]: mgrmap e22: x(active, since 9m), standbys: y, z 2024-07-27T06:45:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:10.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.602+0000 7f3746f58640 1 -- 172.21.15.29:0/204981504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37401020a0 msgr2=0x7f37401024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:10.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.602+0000 7f3746f58640 1 --2- 172.21.15.29:0/204981504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37401020a0 0x7f37401024b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3734009860 tx=0x7f373402a3d0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.602+0000 7f3746f58640 1 -- 172.21.15.29:0/204981504 shutdown_connections 2024-07-27T06:45:10.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.602+0000 7f3746f58640 1 --2- 172.21.15.29:0/204981504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f374010fc50 0x7f3740112070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.602+0000 7f3746f58640 1 --2- 172.21.15.29:0/204981504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37401029f0 0x7f374010f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.602+0000 7f3746f58640 1 --2- 172.21.15.29:0/204981504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37401020a0 0x7f37401024b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.602+0000 7f3746f58640 1 -- 172.21.15.29:0/204981504 >> 172.21.15.29:0/204981504 conn(0x7f3740079210 msgr2=0x7f3740079620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.602+0000 7f3746f58640 1 -- 172.21.15.29:0/204981504 shutdown_connections 2024-07-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.602+0000 7f3746f58640 1 -- 172.21.15.29:0/204981504 wait complete. 2024-07-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.602+0000 7f3746f58640 1 Processor -- start 2024-07-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f3746f58640 1 -- start start 2024-07-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f3746f58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37401020a0 0x7f374019c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f3746f58640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37401029f0 0x7f374019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f3746f58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f374010fc50 0x7f37401a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f3746f58640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f37401156f0 con 0x7f37401020a0 2024-07-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f3746f58640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3740115570 con 0x7f37401029f0 2024-07-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f3746f58640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3740115870 con 0x7f374010fc50 2024-07-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f3744cf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37401020a0 0x7f374019c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f373ffff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37401029f0 0x7f374019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f373ffff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37401029f0 0x7f374019c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.29:43966/0 (socket says 172.21.15.29:43966) 2024-07-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f373ffff640 1 -- 172.21.15.29:0/3890738671 learned_addr learned my addr 172.21.15.29:0/3890738671 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f37454f5640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f374010fc50 0x7f37401a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f373ffff640 1 -- 172.21.15.29:0/3890738671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f374010fc50 msgr2=0x7f37401a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f373ffff640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f374010fc50 0x7f37401a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f373ffff640 1 -- 172.21.15.29:0/3890738671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37401020a0 msgr2=0x7f374019c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f373ffff640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37401020a0 0x7f374019c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.603+0000 7f373ffff640 1 -- 172.21.15.29:0/3890738671 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37401a2030 con 0x7f37401029f0 2024-07-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.604+0000 7f373ffff640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37401029f0 0x7f374019c660 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f372c00e9c0 tx=0x7f372c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.604+0000 7f373dffb640 1 -- 172.21.15.29:0/3890738671 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c00cde0 con 0x7f37401029f0 2024-07-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.604+0000 7f373dffb640 1 -- 172.21.15.29:0/3890738671 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f372c0035f0 con 0x7f37401029f0 2024-07-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.604+0000 7f373dffb640 1 -- 172.21.15.29:0/3890738671 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c010690 con 0x7f37401029f0 2024-07-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.604+0000 7f3746f58640 1 -- 172.21.15.29:0/3890738671 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37401a2320 con 0x7f37401029f0 2024-07-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.604+0000 7f3746f58640 1 -- 172.21.15.29:0/3890738671 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37401a8a80 con 0x7f37401029f0 2024-07-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.605+0000 7f3746f58640 1 -- 172.21.15.29:0/3890738671 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37401024b0 con 0x7f37401029f0 2024-07-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.607+0000 7f373dffb640 1 -- 172.21.15.29:0/3890738671 <== mon.2 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f372c004000 con 0x7f37401029f0 2024-07-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.607+0000 7f373dffb640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f37300b3ca0 0x7f37300b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.608+0000 7f3744cf4640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f37300b3ca0 0x7f37300b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.608+0000 7f373dffb640 1 -- 172.21.15.29:0/3890738671 <== mon.2 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f372c0e3fc0 con 0x7f37401029f0 2024-07-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.608+0000 7f3744cf4640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f37300b3ca0 0x7f37300b61b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f37340095f0 tx=0x7f3734005c90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.612+0000 7f373dffb640 1 -- 172.21.15.29:0/3890738671 <== mon.2 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f372c009510 con 0x7f37401029f0 2024-07-27T06:45:10.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:10 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3872723636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-27T06:45:10.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:10 smithi029 ceph-mon[32610]: osdmap e83: 12 total, 12 up, 12 in 2024-07-27T06:45:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:10 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3872723636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-27T06:45:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:10 smithi047 ceph-mon[36835]: osdmap e83: 12 total, 12 up, 12 in 2024-07-27T06:45:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.479+0000 7f4d0010c640 1 -- 172.21.15.29:0/772155841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf8107dd0 msgr2=0x7f4cf81081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.479+0000 7f4d0010c640 1 --2- 172.21.15.29:0/772155841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf8107dd0 0x7f4cf81081e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009900 tx=0x7f4ce802a410 comp rx=0 tx=0).stop 2024-07-27T06:45:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.479+0000 7f4d0010c640 1 -- 172.21.15.29:0/772155841 shutdown_connections 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.479+0000 7f4d0010c640 1 --2- 172.21.15.29:0/772155841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf8108720 0x7f4cf810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.479+0000 7f4d0010c640 1 --2- 172.21.15.29:0/772155841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf8107dd0 0x7f4cf81081e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.479+0000 7f4d0010c640 1 --2- 172.21.15.29:0/772155841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cf8102490 0x7f4cf81028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.479+0000 7f4d0010c640 1 -- 172.21.15.29:0/772155841 >> 172.21.15.29:0/772155841 conn(0x7f4cf8079160 msgr2=0x7f4cf8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.480+0000 7f4d0010c640 1 -- 172.21.15.29:0/772155841 shutdown_connections 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.480+0000 7f4d0010c640 1 -- 172.21.15.29:0/772155841 wait complete. 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.480+0000 7f4d0010c640 1 Processor -- start 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.480+0000 7f4d0010c640 1 -- start start 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4d0010c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cf8102490 0x7f4cf8199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4d0010c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf8107dd0 0x7f4cf819a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4d0010c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf8108720 0x7f4cf819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4d0010c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf8112350 con 0x7f4cf8107dd0 2024-07-27T06:45:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4d0010c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf81121d0 con 0x7f4cf8102490 2024-07-27T06:45:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4d0010c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf81124d0 con 0x7f4cf8108720 2024-07-27T06:45:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4cfd6a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf8107dd0 0x7f4cf819a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4cfd6a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf8107dd0 0x7f4cf819a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49296/0 (socket says 172.21.15.29:49296) 2024-07-27T06:45:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4cfd6a7640 1 -- 172.21.15.29:0/2626660166 learned_addr learned my addr 172.21.15.29:0/2626660166 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4cfdea8640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cf8102490 0x7f4cf8199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4cfe6a9640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf8108720 0x7f4cf819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4cfd6a7640 1 -- 172.21.15.29:0/2626660166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf8108720 msgr2=0x7f4cf819f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4cfd6a7640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf8108720 0x7f4cf819f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4cfd6a7640 1 -- 172.21.15.29:0/2626660166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cf8102490 msgr2=0x7f4cf8199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4cfd6a7640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cf8102490 0x7f4cf8199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4cfd6a7640 1 -- 172.21.15.29:0/2626660166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf819ff00 con 0x7f4cf8107dd0 2024-07-27T06:45:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4cfdea8640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cf8102490 0x7f4cf8199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:45:10.770 INFO:teuthology.orchestra.run.smithi029.stdout:38654705758 2024-07-27T06:45:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.481+0000 7f4cfd6a7640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf8107dd0 0x7f4cf819a530 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8006010 tx=0x7f4ce8004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.483+0000 7f4ceeffd640 1 -- 172.21.15.29:0/2626660166 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce803f030 con 0x7f4cf8107dd0 2024-07-27T06:45:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.483+0000 7f4d0010c640 1 -- 172.21.15.29:0/2626660166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf81a0190 con 0x7f4cf8107dd0 2024-07-27T06:45:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.483+0000 7f4ceeffd640 1 -- 172.21.15.29:0/2626660166 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ce8003aa0 con 0x7f4cf8107dd0 2024-07-27T06:45:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.483+0000 7f4d0010c640 1 -- 172.21.15.29:0/2626660166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf81a6950 con 0x7f4cf8107dd0 2024-07-27T06:45:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.483+0000 7f4ceeffd640 1 -- 172.21.15.29:0/2626660166 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8035070 con 0x7f4cf8107dd0 2024-07-27T06:45:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.485+0000 7f4d0010c640 1 -- 172.21.15.29:0/2626660166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cf81028a0 con 0x7f4cf8107dd0 2024-07-27T06:45:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.486+0000 7f4ceeffd640 1 -- 172.21.15.29:0/2626660166 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f4ce8042030 con 0x7f4cf8107dd0 2024-07-27T06:45:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.487+0000 7f4ceeffd640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4ce40b3c20 0x7f4ce40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.487+0000 7f4cfdea8640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4ce40b3c20 0x7f4ce40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.487+0000 7f4ceeffd640 1 -- 172.21.15.29:0/2626660166 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f4ce8103e10 con 0x7f4cf8107dd0 2024-07-27T06:45:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.487+0000 7f4cfdea8640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4ce40b3c20 0x7f4ce40b6130 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40095f0 tx=0x7f4cf4005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.495+0000 7f4ceeffd640 1 -- 172.21.15.29:0/2626660166 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4ce8107030 con 0x7f4cf8107dd0 2024-07-27T06:45:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.685+0000 7f4d0010c640 1 -- 172.21.15.29:0/2626660166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f4cf81a6b20 con 0x7f4cf8107dd0 2024-07-27T06:45:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.686+0000 7f4ceeffd640 1 -- 172.21.15.29:0/2626660166 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4ce80caca0 con 0x7f4cf8107dd0 2024-07-27T06:45:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.688+0000 7f4d0010c640 1 -- 172.21.15.29:0/2626660166 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4ce40b3c20 msgr2=0x7f4ce40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.688+0000 7f4d0010c640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4ce40b3c20 0x7f4ce40b6130 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40095f0 tx=0x7f4cf4005f90 comp rx=0 tx=0).stop 2024-07-27T06:45:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.688+0000 7f4d0010c640 1 -- 172.21.15.29:0/2626660166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf8107dd0 msgr2=0x7f4cf819a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.688+0000 7f4d0010c640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf8107dd0 0x7f4cf819a530 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8006010 tx=0x7f4ce8004060 comp rx=0 tx=0).stop 2024-07-27T06:45:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.688+0000 7f4d0010c640 1 -- 172.21.15.29:0/2626660166 shutdown_connections 2024-07-27T06:45:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.688+0000 7f4d0010c640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4ce40b3c20 0x7f4ce40b6130 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.688+0000 7f4d0010c640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf8108720 0x7f4cf819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.688+0000 7f4d0010c640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cf8107dd0 0x7f4cf819a530 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.688+0000 7f4d0010c640 1 --2- 172.21.15.29:0/2626660166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4cf8102490 0x7f4cf8199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.688+0000 7f4d0010c640 1 -- 172.21.15.29:0/2626660166 >> 172.21.15.29:0/2626660166 conn(0x7f4cf8079160 msgr2=0x7f4cf8101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.688+0000 7f4d0010c640 1 -- 172.21.15.29:0/2626660166 shutdown_connections 2024-07-27T06:45:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.688+0000 7f4d0010c640 1 -- 172.21.15.29:0/2626660166 wait complete. 2024-07-27T06:45:10.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.799+0000 7f3746f58640 1 -- 172.21.15.29:0/3890738671 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f37401a8cf0 con 0x7f37401029f0 2024-07-27T06:45:10.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.799+0000 7f373dffb640 1 -- 172.21.15.29:0/3890738671 <== mon.2 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f372c0ab500 con 0x7f37401029f0 2024-07-27T06:45:10.801 INFO:teuthology.orchestra.run.smithi029.stdout:231928234038 2024-07-27T06:45:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.801+0000 7f3746f58640 1 -- 172.21.15.29:0/3890738671 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f37300b3ca0 msgr2=0x7f37300b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.801+0000 7f3746f58640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f37300b3ca0 0x7f37300b61b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f37340095f0 tx=0x7f3734005c90 comp rx=0 tx=0).stop 2024-07-27T06:45:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.801+0000 7f3746f58640 1 -- 172.21.15.29:0/3890738671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37401029f0 msgr2=0x7f374019c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.801+0000 7f3746f58640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37401029f0 0x7f374019c660 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f372c00e9c0 tx=0x7f372c00ee80 comp rx=0 tx=0).stop 2024-07-27T06:45:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.801+0000 7f3746f58640 1 -- 172.21.15.29:0/3890738671 shutdown_connections 2024-07-27T06:45:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.801+0000 7f3746f58640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f37300b3ca0 0x7f37300b61b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.801+0000 7f3746f58640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f374010fc50 0x7f37401a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.801+0000 7f3746f58640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f37401029f0 0x7f374019c660 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.801+0000 7f3746f58640 1 --2- 172.21.15.29:0/3890738671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37401020a0 0x7f374019c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.801+0000 7f3746f58640 1 -- 172.21.15.29:0/3890738671 >> 172.21.15.29:0/3890738671 conn(0x7f3740079210 msgr2=0x7f3740110f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.801+0000 7f3746f58640 1 -- 172.21.15.29:0/3890738671 shutdown_connections 2024-07-27T06:45:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.802+0000 7f3746f58640 1 -- 172.21.15.29:0/3890738671 wait complete. 2024-07-27T06:45:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:10 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3872723636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-27T06:45:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:10 smithi077 ceph-mon[37153]: osdmap e83: 12 total, 12 up, 12 in 2024-07-27T06:45:11.465 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593785 got 184683593792 for osd.6 2024-07-27T06:45:11.466 DEBUG:teuthology.parallel:result is None 2024-07-27T06:45:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:11 smithi029 ceph-mon[32610]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-07-27T06:45:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:11 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1314546027' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-27T06:45:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:11 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2626660166' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-27T06:45:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:11 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3890738671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-27T06:45:11.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:11 smithi077 ceph-mon[37153]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-07-27T06:45:11.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:11 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1314546027' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-27T06:45:11.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:11 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2626660166' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-27T06:45:11.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:11 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3890738671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-27T06:45:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:11 smithi047 ceph-mon[36835]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-07-27T06:45:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:11 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1314546027' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-27T06:45:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:11 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2626660166' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-27T06:45:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:11 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3890738671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-27T06:45:12.112 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234031 got 231928234038 for osd.8 2024-07-27T06:45:12.112 DEBUG:teuthology.parallel:result is None 2024-07-27T06:45:13.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.477+0000 7fbfc0145640 1 -- 172.21.15.29:0/3540816486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb8103590 msgr2=0x7fbfb81039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:13.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.477+0000 7fbfc0145640 1 --2- 172.21.15.29:0/3540816486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb8103590 0x7fbfb81039a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4009900 tx=0x7fbfb402a410 comp rx=0 tx=0).stop 2024-07-27T06:45:13.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.477+0000 7fbfc0145640 1 -- 172.21.15.29:0/3540816486 shutdown_connections 2024-07-27T06:45:13.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.477+0000 7fbfc0145640 1 --2- 172.21.15.29:0/3540816486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb8109820 0x7fbfb8110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.477+0000 7fbfc0145640 1 --2- 172.21.15.29:0/3540816486 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfb8108ed0 0x7fbfb81092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.477+0000 7fbfc0145640 1 --2- 172.21.15.29:0/3540816486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb8103590 0x7fbfb81039a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.477+0000 7fbfc0145640 1 -- 172.21.15.29:0/3540816486 >> 172.21.15.29:0/3540816486 conn(0x7fbfb807b280 msgr2=0x7fbfb807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:13.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.477+0000 7fbfc0145640 1 -- 172.21.15.29:0/3540816486 shutdown_connections 2024-07-27T06:45:13.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.477+0000 7fbfc0145640 1 -- 172.21.15.29:0/3540816486 wait complete. 2024-07-27T06:45:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.477+0000 7fbfc0145640 1 Processor -- start 2024-07-27T06:45:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfc0145640 1 -- start start 2024-07-27T06:45:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfc0145640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb8103590 0x7fbfb819c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfc0145640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb8108ed0 0x7fbfb819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfc0145640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfb8109820 0x7fbfb81a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfc0145640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb81134f0 con 0x7fbfb8103590 2024-07-27T06:45:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfc0145640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb8113370 con 0x7fbfb8109820 2024-07-27T06:45:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfc0145640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb8113670 con 0x7fbfb8108ed0 2024-07-27T06:45:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfbdee1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb8103590 0x7fbfb819c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfbdee1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb8103590 0x7fbfb819c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49346/0 (socket says 172.21.15.29:49346) 2024-07-27T06:45:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfbdee1640 1 -- 172.21.15.29:0/3683136475 learned_addr learned my addr 172.21.15.29:0/3683136475 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfbd6e0640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb8108ed0 0x7fbfb819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfbe6e2640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfb8109820 0x7fbfb81a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfbdee1640 1 -- 172.21.15.29:0/3683136475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb8108ed0 msgr2=0x7fbfb819c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfbdee1640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb8108ed0 0x7fbfb819c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfbdee1640 1 -- 172.21.15.29:0/3683136475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfb8109820 msgr2=0x7fbfb81a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfbdee1640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfb8109820 0x7fbfb81a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfbdee1640 1 -- 172.21.15.29:0/3683136475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb81a2050 con 0x7fbfb8103590 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfbe6e2640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfb8109820 0x7fbfb81a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.478+0000 7fbfbdee1640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb8103590 0x7fbfb819c0e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4005940 tx=0x7fbfb4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.479+0000 7fbfaeffd640 1 -- 172.21.15.29:0/3683136475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb403f030 con 0x7fbfb8103590 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.479+0000 7fbfaeffd640 1 -- 172.21.15.29:0/3683136475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb4003aa0 con 0x7fbfb8103590 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.479+0000 7fbfc0145640 1 -- 172.21.15.29:0/3683136475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb81a22e0 con 0x7fbfb8103590 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.479+0000 7fbfaeffd640 1 -- 172.21.15.29:0/3683136475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb4035070 con 0x7fbfb8103590 2024-07-27T06:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.479+0000 7fbfc0145640 1 -- 172.21.15.29:0/3683136475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb81a8aa0 con 0x7fbfb8103590 2024-07-27T06:45:13.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.480+0000 7fbfc0145640 1 -- 172.21.15.29:0/3683136475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfb81039a0 con 0x7fbfb8103590 2024-07-27T06:45:13.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.482+0000 7fbfaeffd640 1 -- 172.21.15.29:0/3683136475 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fbfb4042030 con 0x7fbfb8103590 2024-07-27T06:45:13.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.483+0000 7fbfaeffd640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbfa40b3c20 0x7fbfa40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:13.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.483+0000 7fbfbd6e0640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbfa40b3c20 0x7fbfa40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:13.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.483+0000 7fbfaeffd640 1 -- 172.21.15.29:0/3683136475 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fbfb4104760 con 0x7fbfb8103590 2024-07-27T06:45:13.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.483+0000 7fbfbd6e0640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbfa40b3c20 0x7fbfa40b6130 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8105f00 tx=0x7fbfa800b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:13.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.486+0000 7fbfaeffd640 1 -- 172.21.15.29:0/3683136475 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbfb40cbc40 con 0x7fbfb8103590 2024-07-27T06:45:13.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:13 smithi029 ceph-mon[32610]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:13.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.361+0000 7f434a31f640 1 -- 172.21.15.29:0/3013305308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4344108750 msgr2=0x7f434410ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.361+0000 7f434a31f640 1 --2- 172.21.15.29:0/3013305308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4344108750 0x7f434410ef80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f433c009ca0 tx=0x7f433c02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.362+0000 7f434a31f640 1 -- 172.21.15.29:0/3013305308 shutdown_connections 2024-07-27T06:45:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.362+0000 7f434a31f640 1 --2- 172.21.15.29:0/3013305308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4344108750 0x7f434410ef80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.362+0000 7f434a31f640 1 --2- 172.21.15.29:0/3013305308 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4344107e00 0x7f4344108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.362+0000 7f434a31f640 1 --2- 172.21.15.29:0/3013305308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4344102510 0x7f4344102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.362+0000 7f434a31f640 1 -- 172.21.15.29:0/3013305308 >> 172.21.15.29:0/3013305308 conn(0x7f4344079190 msgr2=0x7f43440795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.362+0000 7f434a31f640 1 -- 172.21.15.29:0/3013305308 shutdown_connections 2024-07-27T06:45:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.362+0000 7f434a31f640 1 -- 172.21.15.29:0/3013305308 wait complete. 2024-07-27T06:45:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.362+0000 7f434a31f640 1 Processor -- start 2024-07-27T06:45:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.362+0000 7f434a31f640 1 -- start start 2024-07-27T06:45:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f434a31f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4344102510 0x7f4344199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f434a31f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4344107e00 0x7f434419a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f434a31f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4344108750 0x7f434419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f434a31f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f43441134e0 con 0x7f4344107e00 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f434a31f640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f4344113360 con 0x7f4344108750 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f434a31f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4344113660 con 0x7f4344102510 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f43437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4344107e00 0x7f434419a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f43437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4344107e00 0x7f434419a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49280/0 (socket says 172.21.15.29:49280) 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f43437fe640 1 -- 172.21.15.29:0/1314546027 learned_addr learned my addr 172.21.15.29:0/1314546027 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f43488bc640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4344108750 0x7f434419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f4343fff640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4344102510 0x7f4344199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f43437fe640 1 -- 172.21.15.29:0/1314546027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4344102510 msgr2=0x7f4344199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f43437fe640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4344102510 0x7f4344199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f43437fe640 1 -- 172.21.15.29:0/1314546027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4344108750 msgr2=0x7f434419f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:13.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f43437fe640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4344108750 0x7f434419f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f43437fe640 1 -- 172.21.15.29:0/1314546027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434419fda0 con 0x7f4344107e00 2024-07-27T06:45:13.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.363+0000 7f43437fe640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4344107e00 0x7f434419a3d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f433800e9c0 tx=0x7f433800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:13.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.364+0000 7f43417fa640 1 -- 172.21.15.29:0/1314546027 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433800cd80 con 0x7f4344107e00 2024-07-27T06:45:13.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.364+0000 7f43417fa640 1 -- 172.21.15.29:0/1314546027 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43380054d0 con 0x7f4344107e00 2024-07-27T06:45:13.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.364+0000 7f434a31f640 1 -- 172.21.15.29:0/1314546027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43441a0090 con 0x7f4344107e00 2024-07-27T06:45:13.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.364+0000 7f43417fa640 1 -- 172.21.15.29:0/1314546027 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4338010430 con 0x7f4344107e00 2024-07-27T06:45:13.629 INFO:teuthology.orchestra.run.smithi029.stdout:158913790021 2024-07-27T06:45:13.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.364+0000 7f434a31f640 1 -- 172.21.15.29:0/1314546027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43441a67f0 con 0x7f4344107e00 2024-07-27T06:45:13.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.365+0000 7f434a31f640 1 -- 172.21.15.29:0/1314546027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4344103740 con 0x7f4344107e00 2024-07-27T06:45:13.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.366+0000 7f43417fa640 1 -- 172.21.15.29:0/1314546027 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f4338004fb0 con 0x7f4344107e00 2024-07-27T06:45:13.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.367+0000 7f43417fa640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f432c0b3ca0 0x7f432c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:13.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.367+0000 7f4343fff640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f432c0b3ca0 0x7f432c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:13.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.367+0000 7f43417fa640 1 -- 172.21.15.29:0/1314546027 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f43380e43a0 con 0x7f4344107e00 2024-07-27T06:45:13.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.368+0000 7f4343fff640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f432c0b3ca0 0x7f432c0b61b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4334005fd0 tx=0x7f4334005850 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.370+0000 7f43417fa640 1 -- 172.21.15.29:0/1314546027 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4338055600 con 0x7f4344107e00 2024-07-27T06:45:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.569+0000 7f434a31f640 1 -- 172.21.15.29:0/1314546027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f4344102920 con 0x7f4344107e00 2024-07-27T06:45:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.570+0000 7f43417fa640 1 -- 172.21.15.29:0/1314546027 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4344102920 con 0x7f4344107e00 2024-07-27T06:45:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.572+0000 7f434a31f640 1 -- 172.21.15.29:0/1314546027 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f432c0b3ca0 msgr2=0x7f432c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.572+0000 7f434a31f640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f432c0b3ca0 0x7f432c0b61b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4334005fd0 tx=0x7f4334005850 comp rx=0 tx=0).stop 2024-07-27T06:45:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.572+0000 7f434a31f640 1 -- 172.21.15.29:0/1314546027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4344107e00 msgr2=0x7f434419a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.572+0000 7f434a31f640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4344107e00 0x7f434419a3d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f433800e9c0 tx=0x7f433800ee80 comp rx=0 tx=0).stop 2024-07-27T06:45:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.572+0000 7f434a31f640 1 -- 172.21.15.29:0/1314546027 shutdown_connections 2024-07-27T06:45:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.572+0000 7f434a31f640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4344108750 0x7f434419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.572+0000 7f434a31f640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4344107e00 0x7f434419a3d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.572+0000 7f434a31f640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f432c0b3ca0 0x7f432c0b61b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.572+0000 7f434a31f640 1 --2- 172.21.15.29:0/1314546027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4344102510 0x7f4344199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.572+0000 7f434a31f640 1 -- 172.21.15.29:0/1314546027 >> 172.21.15.29:0/1314546027 conn(0x7f4344079190 msgr2=0x7f4344103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:13.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.572+0000 7f434a31f640 1 -- 172.21.15.29:0/1314546027 shutdown_connections 2024-07-27T06:45:13.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:10.572+0000 7f434a31f640 1 -- 172.21.15.29:0/1314546027 wait complete. 2024-07-27T06:45:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.674+0000 7fbfc0145640 1 -- 172.21.15.29:0/3683136475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbfb81a8ca0 con 0x7fbfb8103590 2024-07-27T06:45:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.674+0000 7fbfaeffd640 1 -- 172.21.15.29:0/3683136475 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbfb40cb540 con 0x7fbfb8103590 2024-07-27T06:45:13.676 INFO:teuthology.orchestra.run.smithi029.stdout:60129542234 2024-07-27T06:45:13.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.676+0000 7fbfc0145640 1 -- 172.21.15.29:0/3683136475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbfa40b3c20 msgr2=0x7fbfa40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:13.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.676+0000 7fbfc0145640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbfa40b3c20 0x7fbfa40b6130 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8105f00 tx=0x7fbfa800b390 comp rx=0 tx=0).stop 2024-07-27T06:45:13.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.676+0000 7fbfc0145640 1 -- 172.21.15.29:0/3683136475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb8103590 msgr2=0x7fbfb819c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:13.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.676+0000 7fbfc0145640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb8103590 0x7fbfb819c0e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4005940 tx=0x7fbfb4004270 comp rx=0 tx=0).stop 2024-07-27T06:45:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.676+0000 7fbfc0145640 1 -- 172.21.15.29:0/3683136475 shutdown_connections 2024-07-27T06:45:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.676+0000 7fbfc0145640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfb8109820 0x7fbfb81a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.676+0000 7fbfc0145640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbfa40b3c20 0x7fbfa40b6130 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.676+0000 7fbfc0145640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbfb8108ed0 0x7fbfb819c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.676+0000 7fbfc0145640 1 --2- 172.21.15.29:0/3683136475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfb8103590 0x7fbfb819c0e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.676+0000 7fbfc0145640 1 -- 172.21.15.29:0/3683136475 >> 172.21.15.29:0/3683136475 conn(0x7fbfb807b280 msgr2=0x7fbfb8106d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.677+0000 7fbfc0145640 1 -- 172.21.15.29:0/3683136475 shutdown_connections 2024-07-27T06:45:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:13.677+0000 7fbfc0145640 1 -- 172.21.15.29:0/3683136475 wait complete. 2024-07-27T06:45:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:13 smithi077 ceph-mon[37153]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:13 smithi047 ceph-mon[36835]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:14.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:14 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3683136475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-27T06:45:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:14 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3683136475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-27T06:45:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:14 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3683136475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-27T06:45:15.022 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790016 got 158913790021 for osd.5 2024-07-27T06:45:15.022 DEBUG:teuthology.parallel:result is None 2024-07-27T06:45:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.475+0000 7ff75be48640 1 -- 172.21.15.29:0/4255681973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff754107df0 msgr2=0x7ff754108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.475+0000 7ff75be48640 1 --2- 172.21.15.29:0/4255681973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff754107df0 0x7ff754108200 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff744009900 tx=0x7ff74402a410 comp rx=0 tx=0).stop 2024-07-27T06:45:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.475+0000 7ff75be48640 1 -- 172.21.15.29:0/4255681973 shutdown_connections 2024-07-27T06:45:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.475+0000 7ff75be48640 1 --2- 172.21.15.29:0/4255681973 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff754108740 0x7ff75410ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.475+0000 7ff75be48640 1 --2- 172.21.15.29:0/4255681973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff754107df0 0x7ff754108200 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.475+0000 7ff75be48640 1 --2- 172.21.15.29:0/4255681973 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7541024b0 0x7ff7541028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.475+0000 7ff75be48640 1 -- 172.21.15.29:0/4255681973 >> 172.21.15.29:0/4255681973 conn(0x7ff754079170 msgr2=0x7ff754079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.475+0000 7ff75be48640 1 -- 172.21.15.29:0/4255681973 shutdown_connections 2024-07-27T06:45:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.475+0000 7ff75be48640 1 -- 172.21.15.29:0/4255681973 wait complete. 2024-07-27T06:45:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff75be48640 1 Processor -- start 2024-07-27T06:45:15.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff75be48640 1 -- start start 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff75be48640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7541024b0 0x7ff754198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff75be48640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff754107df0 0x7ff754199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff75be48640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff754108740 0x7ff75419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff75be48640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff754112390 con 0x7ff754108740 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff75be48640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7ff754112210 con 0x7ff754107df0 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff75be48640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff754112510 con 0x7ff7541024b0 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff75a3e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff754108740 0x7ff75419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff75a3e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff754108740 0x7ff75419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49358/0 (socket says 172.21.15.29:49358) 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff75a3e5640 1 -- 172.21.15.29:0/694532981 learned_addr learned my addr 172.21.15.29:0/694532981 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff759be4640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7541024b0 0x7ff754198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.476+0000 7ff7593e3640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff754107df0 0x7ff754199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.477+0000 7ff75a3e5640 1 -- 172.21.15.29:0/694532981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7541024b0 msgr2=0x7ff754198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:15.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.477+0000 7ff75a3e5640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7541024b0 0x7ff754198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.477+0000 7ff75a3e5640 1 -- 172.21.15.29:0/694532981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff754107df0 msgr2=0x7ff754199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.477+0000 7ff75a3e5640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff754107df0 0x7ff754199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.477+0000 7ff75a3e5640 1 -- 172.21.15.29:0/694532981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75419edd0 con 0x7ff754108740 2024-07-27T06:45:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.477+0000 7ff75a3e5640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff754108740 0x7ff75419e6a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00cad0 tx=0x7ff74c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.477+0000 7ff74affd640 1 -- 172.21.15.29:0/694532981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c00ae20 con 0x7ff754108740 2024-07-27T06:45:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.477+0000 7ff75be48640 1 -- 172.21.15.29:0/694532981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75419f0c0 con 0x7ff754108740 2024-07-27T06:45:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.477+0000 7ff74affd640 1 -- 172.21.15.29:0/694532981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff74c005020 con 0x7ff754108740 2024-07-27T06:45:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.477+0000 7ff74affd640 1 -- 172.21.15.29:0/694532981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c00f6b0 con 0x7ff754108740 2024-07-27T06:45:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.477+0000 7ff75be48640 1 -- 172.21.15.29:0/694532981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7541a5820 con 0x7ff754108740 2024-07-27T06:45:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.478+0000 7ff75be48640 1 -- 172.21.15.29:0/694532981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff754103730 con 0x7ff754108740 2024-07-27T06:45:15.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.480+0000 7ff74affd640 1 -- 172.21.15.29:0/694532981 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7ff74c0051d0 con 0x7ff754108740 2024-07-27T06:45:15.130 INFO:teuthology.orchestra.run.smithi029.stdout:304942678056 2024-07-27T06:45:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.480+0000 7ff74affd640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff7400b3c20 0x7ff7400b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.481+0000 7ff74affd640 1 -- 172.21.15.29:0/694532981 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7ff74c0e4030 con 0x7ff754108740 2024-07-27T06:45:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.480+0000 7ff759be4640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff7400b3c20 0x7ff7400b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.481+0000 7ff759be4640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff7400b3c20 0x7ff7400b6130 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff7500058c0 tx=0x7ff75000a250 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.483+0000 7ff74affd640 1 -- 172.21.15.29:0/694532981 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff74c014030 con 0x7ff754108740 2024-07-27T06:45:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.669+0000 7ff75be48640 1 -- 172.21.15.29:0/694532981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7ff7541028c0 con 0x7ff754108740 2024-07-27T06:45:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.669+0000 7ff74affd640 1 -- 172.21.15.29:0/694532981 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff7541028c0 con 0x7ff754108740 2024-07-27T06:45:15.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.672+0000 7ff75be48640 1 -- 172.21.15.29:0/694532981 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff7400b3c20 msgr2=0x7ff7400b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.672+0000 7ff75be48640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff7400b3c20 0x7ff7400b6130 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff7500058c0 tx=0x7ff75000a250 comp rx=0 tx=0).stop 2024-07-27T06:45:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.672+0000 7ff75be48640 1 -- 172.21.15.29:0/694532981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff754108740 msgr2=0x7ff75419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.672+0000 7ff75be48640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff754108740 0x7ff75419e6a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00cad0 tx=0x7ff74c00cf90 comp rx=0 tx=0).stop 2024-07-27T06:45:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.672+0000 7ff75be48640 1 -- 172.21.15.29:0/694532981 shutdown_connections 2024-07-27T06:45:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.672+0000 7ff75be48640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff754108740 0x7ff75419e6a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.672+0000 7ff75be48640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff754107df0 0x7ff754199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.672+0000 7ff75be48640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff7400b3c20 0x7ff7400b6130 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.672+0000 7ff75be48640 1 --2- 172.21.15.29:0/694532981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7541024b0 0x7ff754198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.672+0000 7ff75be48640 1 -- 172.21.15.29:0/694532981 >> 172.21.15.29:0/694532981 conn(0x7ff754079170 msgr2=0x7ff754103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.672+0000 7ff75be48640 1 -- 172.21.15.29:0/694532981 shutdown_connections 2024-07-27T06:45:15.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:14.672+0000 7ff75be48640 1 -- 172.21.15.29:0/694532981 wait complete. 2024-07-27T06:45:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:15 smithi029 ceph-mon[32610]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:15 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/694532981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-27T06:45:15.825 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705752 got 38654705758 for osd.0 2024-07-27T06:45:15.826 DEBUG:teuthology.parallel:result is None 2024-07-27T06:45:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:15 smithi077 ceph-mon[37153]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:15 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/694532981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-27T06:45:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:15 smithi047 ceph-mon[36835]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:15 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/694532981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-27T06:45:16.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.108+0000 7f7276fe7640 1 -- 172.21.15.29:0/3140706097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7270107dd0 msgr2=0x7f72701081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.108+0000 7f7276fe7640 1 --2- 172.21.15.29:0/3140706097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7270107dd0 0x7f72701081e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f725c009900 tx=0x7f725c02a410 comp rx=0 tx=0).stop 2024-07-27T06:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.109+0000 7f7276fe7640 1 -- 172.21.15.29:0/3140706097 shutdown_connections 2024-07-27T06:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.109+0000 7f7276fe7640 1 --2- 172.21.15.29:0/3140706097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7270108720 0x7f727010ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.109+0000 7f7276fe7640 1 --2- 172.21.15.29:0/3140706097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7270107dd0 0x7f72701081e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.109+0000 7f7276fe7640 1 --2- 172.21.15.29:0/3140706097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7270102490 0x7f72701028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.109+0000 7f7276fe7640 1 -- 172.21.15.29:0/3140706097 >> 172.21.15.29:0/3140706097 conn(0x7f7270079160 msgr2=0x7f7270079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.109+0000 7f7276fe7640 1 -- 172.21.15.29:0/3140706097 shutdown_connections 2024-07-27T06:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.109+0000 7f7276fe7640 1 -- 172.21.15.29:0/3140706097 wait complete. 2024-07-27T06:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.109+0000 7f7276fe7640 1 Processor -- start 2024-07-27T06:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7276fe7640 1 -- start start 2024-07-27T06:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7276fe7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7270102490 0x7f727019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:16.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7276fe7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7270107dd0 0x7f727019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7276fe7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7270108720 0x7f727019f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7276fe7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7270112350 con 0x7f7270102490 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7276fe7640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f72701121d0 con 0x7f7270107dd0 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7276fe7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72701124d0 con 0x7f7270108720 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7274d83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7270102490 0x7f727019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7274d83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7270102490 0x7f727019a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49384/0 (socket says 172.21.15.29:49384) 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7274d83640 1 -- 172.21.15.29:0/2789039459 learned_addr learned my addr 172.21.15.29:0/2789039459 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f726ffff640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7270107dd0 0x7f727019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7275584640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7270108720 0x7f727019f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7274d83640 1 -- 172.21.15.29:0/2789039459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7270108720 msgr2=0x7f727019f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7274d83640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7270108720 0x7f727019f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7274d83640 1 -- 172.21.15.29:0/2789039459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7270107dd0 msgr2=0x7f727019a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7274d83640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7270107dd0 0x7f727019a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.110+0000 7f7274d83640 1 -- 172.21.15.29:0/2789039459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727019ff00 con 0x7f7270102490 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.111+0000 7f7275584640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7270108720 0x7f727019f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.111+0000 7f7274d83640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7270102490 0x7f727019a050 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f726400d950 tx=0x7f726400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.111+0000 7f726dffb640 1 -- 172.21.15.29:0/2789039459 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7264017070 con 0x7f7270102490 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.111+0000 7f7276fe7640 1 -- 172.21.15.29:0/2789039459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72701a01f0 con 0x7f7270102490 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.111+0000 7f726dffb640 1 -- 172.21.15.29:0/2789039459 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7264009c30 con 0x7f7270102490 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.111+0000 7f726dffb640 1 -- 172.21.15.29:0/2789039459 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726400f620 con 0x7f7270102490 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.111+0000 7f7276fe7640 1 -- 172.21.15.29:0/2789039459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72701a6a10 con 0x7f7270102490 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.112+0000 7f7276fe7640 1 -- 172.21.15.29:0/2789039459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72701028a0 con 0x7f7270102490 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.114+0000 7f726dffb640 1 -- 172.21.15.29:0/2789039459 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f7264010460 con 0x7f7270102490 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.115+0000 7f726dffb640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f72600b3c50 0x7f72600b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.115+0000 7f726ffff640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f72600b3c50 0x7f72600b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:16.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.115+0000 7f726dffb640 1 -- 172.21.15.29:0/2789039459 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f72640e39f0 con 0x7f7270102490 2024-07-27T06:45:16.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.115+0000 7f726ffff640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f72600b3c50 0x7f72600b6160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f725c000c00 tx=0x7f725c031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:16.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.118+0000 7f726dffb640 1 -- 172.21.15.29:0/2789039459 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f72640aaf30 con 0x7f7270102490 2024-07-27T06:45:16.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.307+0000 7f7276fe7640 1 -- 172.21.15.29:0/2789039459 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f727010b520 con 0x7f7270102490 2024-07-27T06:45:16.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.308+0000 7f726dffb640 1 -- 172.21.15.29:0/2789039459 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f72640b3020 con 0x7f7270102490 2024-07-27T06:45:16.310 INFO:teuthology.orchestra.run.smithi029.stdout:257698037811 2024-07-27T06:45:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.310+0000 7f7276fe7640 1 -- 172.21.15.29:0/2789039459 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f72600b3c50 msgr2=0x7f72600b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.310+0000 7f7276fe7640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f72600b3c50 0x7f72600b6160 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f725c000c00 tx=0x7f725c031400 comp rx=0 tx=0).stop 2024-07-27T06:45:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.310+0000 7f7276fe7640 1 -- 172.21.15.29:0/2789039459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7270102490 msgr2=0x7f727019a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.310+0000 7f7276fe7640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7270102490 0x7f727019a050 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f726400d950 tx=0x7f726400de10 comp rx=0 tx=0).stop 2024-07-27T06:45:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.310+0000 7f7276fe7640 1 -- 172.21.15.29:0/2789039459 shutdown_connections 2024-07-27T06:45:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.310+0000 7f7276fe7640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f72600b3c50 0x7f72600b6160 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.310+0000 7f7276fe7640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7270108720 0x7f727019f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.310+0000 7f7276fe7640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7270107dd0 0x7f727019a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.310+0000 7f7276fe7640 1 --2- 172.21.15.29:0/2789039459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7270102490 0x7f727019a050 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.310+0000 7f7276fe7640 1 -- 172.21.15.29:0/2789039459 >> 172.21.15.29:0/2789039459 conn(0x7f7270079160 msgr2=0x7f727006ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.310+0000 7f7276fe7640 1 -- 172.21.15.29:0/2789039459 shutdown_connections 2024-07-27T06:45:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:16.310+0000 7f7276fe7640 1 -- 172.21.15.29:0/2789039459 wait complete. 2024-07-27T06:45:16.621 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678051 got 304942678056 for osd.11 2024-07-27T06:45:16.622 DEBUG:teuthology.parallel:result is None 2024-07-27T06:45:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:16 smithi077 ceph-mon[37153]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:16 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2789039459' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-27T06:45:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:16 smithi047 ceph-mon[36835]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:16 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2789039459' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-27T06:45:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:16 smithi029 ceph-mon[32610]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:16 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2789039459' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-27T06:45:16.997 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874276 got 279172874282 for osd.10 2024-07-27T06:45:16.998 DEBUG:teuthology.parallel:result is None 2024-07-27T06:45:17.428 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542229 got 60129542234 for osd.1 2024-07-27T06:45:17.429 DEBUG:teuthology.parallel:result is None 2024-07-27T06:45:18.697 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037803 got 257698037811 for osd.9 2024-07-27T06:45:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:18.697 DEBUG:teuthology.parallel:result is None 2024-07-27T06:45:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:18 smithi029 ceph-mon[32610]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 995 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:18 smithi077 ceph-mon[37153]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 995 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:18 smithi047 ceph-mon[36835]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 995 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.923+0000 7ff3fb37b640 1 -- 172.21.15.29:0/1455866612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f4102020 msgr2=0x7ff3f4102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.923+0000 7ff3fb37b640 1 --2- 172.21.15.29:0/1455866612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f4102020 0x7ff3f4102430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009900 tx=0x7ff3f002a480 comp rx=0 tx=0).stop 2024-07-27T06:45:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.923+0000 7ff3fb37b640 1 -- 172.21.15.29:0/1455866612 shutdown_connections 2024-07-27T06:45:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.923+0000 7ff3fb37b640 1 --2- 172.21.15.29:0/1455866612 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3f410fbd0 0x7ff3f4111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.923+0000 7ff3fb37b640 1 --2- 172.21.15.29:0/1455866612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f4102970 0x7ff3f410f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.923+0000 7ff3fb37b640 1 --2- 172.21.15.29:0/1455866612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f4102020 0x7ff3f4102430 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.923+0000 7ff3fb37b640 1 -- 172.21.15.29:0/1455866612 >> 172.21.15.29:0/1455866612 conn(0x7ff3f4079170 msgr2=0x7ff3f4079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.923+0000 7ff3fb37b640 1 -- 172.21.15.29:0/1455866612 shutdown_connections 2024-07-27T06:45:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.923+0000 7ff3fb37b640 1 -- 172.21.15.29:0/1455866612 wait complete. 2024-07-27T06:45:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3fb37b640 1 Processor -- start 2024-07-27T06:45:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3fb37b640 1 -- start start 2024-07-27T06:45:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3fb37b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f4102020 0x7ff3f419c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3fb37b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f4102970 0x7ff3f419c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3fb37b640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3f410fbd0 0x7ff3f41a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3fb37b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f4115660 con 0x7ff3f4102970 2024-07-27T06:45:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3fb37b640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f41154e0 con 0x7ff3f410fbd0 2024-07-27T06:45:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3fb37b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f41157e0 con 0x7ff3f4102020 2024-07-27T06:45:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3f8916640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f4102970 0x7ff3f419c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3f8916640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f4102970 0x7ff3f419c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38018/0 (socket says 172.21.15.29:38018) 2024-07-27T06:45:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3f8916640 1 -- 172.21.15.29:0/3983103113 learned_addr learned my addr 172.21.15.29:0/3983103113 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3f9117640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f4102020 0x7ff3f419c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3f9918640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3f410fbd0 0x7ff3f41a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.924+0000 7ff3f8916640 1 -- 172.21.15.29:0/3983103113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f4102020 msgr2=0x7ff3f419c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.925+0000 7ff3f8916640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f4102020 0x7ff3f419c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.925+0000 7ff3f8916640 1 -- 172.21.15.29:0/3983103113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3f410fbd0 msgr2=0x7ff3f41a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.925+0000 7ff3f8916640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3f410fbd0 0x7ff3f41a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.925+0000 7ff3f8916640 1 -- 172.21.15.29:0/3983103113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f41a2040 con 0x7ff3f4102970 2024-07-27T06:45:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.925+0000 7ff3f8916640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f4102970 0x7ff3f419c670 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400d950 tx=0x7ff3e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.925+0000 7ff3ea7fc640 1 -- 172.21.15.29:0/3983103113 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e4017070 con 0x7ff3f4102970 2024-07-27T06:45:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.925+0000 7ff3ea7fc640 1 -- 172.21.15.29:0/3983103113 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3e4009c30 con 0x7ff3f4102970 2024-07-27T06:45:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.925+0000 7ff3fb37b640 1 -- 172.21.15.29:0/3983103113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f41a2330 con 0x7ff3f4102970 2024-07-27T06:45:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.925+0000 7ff3ea7fc640 1 -- 172.21.15.29:0/3983103113 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e400f620 con 0x7ff3f4102970 2024-07-27T06:45:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.925+0000 7ff3fb37b640 1 -- 172.21.15.29:0/3983103113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f41a8a90 con 0x7ff3f4102970 2024-07-27T06:45:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.926+0000 7ff3fb37b640 1 -- 172.21.15.29:0/3983103113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3f406bc70 con 0x7ff3f4102970 2024-07-27T06:45:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.928+0000 7ff3ea7fc640 1 -- 172.21.15.29:0/3983103113 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7ff3e4005030 con 0x7ff3f4102970 2024-07-27T06:45:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.929+0000 7ff3ea7fc640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff3e00b3c20 0x7ff3e00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.929+0000 7ff3f9117640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff3e00b3c20 0x7ff3e00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.929+0000 7ff3ea7fc640 1 -- 172.21.15.29:0/3983103113 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7ff3e40e38d0 con 0x7ff3f4102970 2024-07-27T06:45:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.929+0000 7ff3f9117640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff3e00b3c20 0x7ff3e00b6130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0031440 tx=0x7ff3f0009560 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:19.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:19.932+0000 7ff3ea7fc640 1 -- 172.21.15.29:0/3983103113 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff3e4009de0 con 0x7ff3f4102970 2024-07-27T06:45:20.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.122+0000 7ff3fb37b640 1 -- 172.21.15.29:0/3983103113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ff3f419d200 con 0x7ff3f4102970 2024-07-27T06:45:20.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.122+0000 7ff3ea7fc640 1 -- 172.21.15.29:0/3983103113 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff3e40aa7c0 con 0x7ff3f4102970 2024-07-27T06:45:20.124 INFO:teuthology.orchestra.run.smithi029.stdout:206158430268 2024-07-27T06:45:20.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.124+0000 7ff3fb37b640 1 -- 172.21.15.29:0/3983103113 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff3e00b3c20 msgr2=0x7ff3e00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:20.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.124+0000 7ff3fb37b640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff3e00b3c20 0x7ff3e00b6130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0031440 tx=0x7ff3f0009560 comp rx=0 tx=0).stop 2024-07-27T06:45:20.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.124+0000 7ff3fb37b640 1 -- 172.21.15.29:0/3983103113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f4102970 msgr2=0x7ff3f419c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:20.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.124+0000 7ff3fb37b640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f4102970 0x7ff3f419c670 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400d950 tx=0x7ff3e400de10 comp rx=0 tx=0).stop 2024-07-27T06:45:20.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.124+0000 7ff3fb37b640 1 -- 172.21.15.29:0/3983103113 shutdown_connections 2024-07-27T06:45:20.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.124+0000 7ff3fb37b640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff3f410fbd0 0x7ff3f41a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:20.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.124+0000 7ff3fb37b640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3f4102970 0x7ff3f419c670 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:20.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.124+0000 7ff3fb37b640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff3e00b3c20 0x7ff3e00b6130 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:20.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.124+0000 7ff3fb37b640 1 --2- 172.21.15.29:0/3983103113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f4102020 0x7ff3f419c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:20.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.124+0000 7ff3fb37b640 1 -- 172.21.15.29:0/3983103113 >> 172.21.15.29:0/3983103113 conn(0x7ff3f4079170 msgr2=0x7ff3f4109800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:20.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.124+0000 7ff3fb37b640 1 -- 172.21.15.29:0/3983103113 shutdown_connections 2024-07-27T06:45:20.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:20.124+0000 7ff3fb37b640 1 -- 172.21.15.29:0/3983103113 wait complete. 2024-07-27T06:45:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:20 smithi029 ceph-mon[32610]: pgmap v347: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 905 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:20 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3983103113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-27T06:45:21.097 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430262 got 206158430268 for osd.7 2024-07-27T06:45:21.097 DEBUG:teuthology.parallel:result is None 2024-07-27T06:45:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:20 smithi077 ceph-mon[37153]: pgmap v347: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 905 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:20 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3983103113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-27T06:45:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:20 smithi047 ceph-mon[36835]: pgmap v347: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 905 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:20 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3983103113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-27T06:45:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.978+0000 7f229f890640 1 -- 172.21.15.29:0/3111572767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2298108ed0 msgr2=0x7f22981092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.978+0000 7f229f890640 1 --2- 172.21.15.29:0/3111572767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2298108ed0 0x7f22981092e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2288009900 tx=0x7f228802a3f0 comp rx=0 tx=0).stop 2024-07-27T06:45:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.978+0000 7f229f890640 1 -- 172.21.15.29:0/3111572767 shutdown_connections 2024-07-27T06:45:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.978+0000 7f229f890640 1 --2- 172.21.15.29:0/3111572767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2298109820 0x7f2298110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.978+0000 7f229f890640 1 --2- 172.21.15.29:0/3111572767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2298108ed0 0x7f22981092e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.978+0000 7f229f890640 1 --2- 172.21.15.29:0/3111572767 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2298103590 0x7f22981039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.978+0000 7f229f890640 1 -- 172.21.15.29:0/3111572767 >> 172.21.15.29:0/3111572767 conn(0x7f229807b280 msgr2=0x7f229807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.978+0000 7f229f890640 1 -- 172.21.15.29:0/3111572767 shutdown_connections 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.979+0000 7f229f890640 1 -- 172.21.15.29:0/3111572767 wait complete. 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.979+0000 7f229f890640 1 Processor -- start 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.979+0000 7f229f890640 1 -- start start 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.979+0000 7f229f890640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2298103590 0x7f2298199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.979+0000 7f229f890640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2298108ed0 0x7f229819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.979+0000 7f229f890640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2298109820 0x7f229819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229f890640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f22981134e0 con 0x7f2298109820 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229f890640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f2298113360 con 0x7f2298103590 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229f890640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2298113660 con 0x7f2298108ed0 2024-07-27T06:45:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229de2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2298109820 0x7f229819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229de2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2298109820 0x7f229819f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38042/0 (socket says 172.21.15.29:38042) 2024-07-27T06:45:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229de2d640 1 -- 172.21.15.29:0/1210861980 learned_addr learned my addr 172.21.15.29:0/1210861980 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229d62c640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2298103590 0x7f2298199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229ce2b640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2298108ed0 0x7f229819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229de2d640 1 -- 172.21.15.29:0/1210861980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2298108ed0 msgr2=0x7f229819a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229de2d640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2298108ed0 0x7f229819a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229de2d640 1 -- 172.21.15.29:0/1210861980 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2298103590 msgr2=0x7f2298199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229de2d640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2298103590 0x7f2298199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229de2d640 1 -- 172.21.15.29:0/1210861980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229819fdc0 con 0x7f2298109820 2024-07-27T06:45:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229de2d640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2298109820 0x7f229819f750 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f229000cce0 tx=0x7f229000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f228e7fc640 1 -- 172.21.15.29:0/1210861980 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229000f450 con 0x7f2298109820 2024-07-27T06:45:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f229f890640 1 -- 172.21.15.29:0/1210861980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22981a00b0 con 0x7f2298109820 2024-07-27T06:45:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f228e7fc640 1 -- 172.21.15.29:0/1210861980 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f229000faa0 con 0x7f2298109820 2024-07-27T06:45:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.980+0000 7f228e7fc640 1 -- 172.21.15.29:0/1210861980 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2290018a80 con 0x7f2298109820 2024-07-27T06:45:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.981+0000 7f229f890640 1 -- 172.21.15.29:0/1210861980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22981a68d0 con 0x7f2298109820 2024-07-27T06:45:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.981+0000 7f229f890640 1 -- 172.21.15.29:0/1210861980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22981039a0 con 0x7f2298109820 2024-07-27T06:45:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.983+0000 7f228e7fc640 1 -- 172.21.15.29:0/1210861980 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f229000fc50 con 0x7f2298109820 2024-07-27T06:45:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.984+0000 7f228e7fc640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f22840b3c50 0x7f22840b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.984+0000 7f229d62c640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f22840b3c50 0x7f22840b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.984+0000 7f228e7fc640 1 -- 172.21.15.29:0/1210861980 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f22900e4510 con 0x7f2298109820 2024-07-27T06:45:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.985+0000 7f229d62c640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f22840b3c50 0x7f22840b6160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f229400abe0 tx=0x7f229400a2d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:21.988+0000 7f228e7fc640 1 -- 172.21.15.29:0/1210861980 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2290014030 con 0x7f2298109820 2024-07-27T06:45:22.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.174+0000 7f229f890640 1 -- 172.21.15.29:0/1210861980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f229819b040 con 0x7f2298109820 2024-07-27T06:45:22.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.174+0000 7f228e7fc640 1 -- 172.21.15.29:0/1210861980 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f229819b040 con 0x7f2298109820 2024-07-27T06:45:22.176 INFO:teuthology.orchestra.run.smithi029.stdout:111669149778 2024-07-27T06:45:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.176+0000 7f229f890640 1 -- 172.21.15.29:0/1210861980 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f22840b3c50 msgr2=0x7f22840b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.176+0000 7f229f890640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f22840b3c50 0x7f22840b6160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f229400abe0 tx=0x7f229400a2d0 comp rx=0 tx=0).stop 2024-07-27T06:45:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.176+0000 7f229f890640 1 -- 172.21.15.29:0/1210861980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2298109820 msgr2=0x7f229819f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.176+0000 7f229f890640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2298109820 0x7f229819f750 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f229000cce0 tx=0x7f229000a620 comp rx=0 tx=0).stop 2024-07-27T06:45:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.176+0000 7f229f890640 1 -- 172.21.15.29:0/1210861980 shutdown_connections 2024-07-27T06:45:22.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.176+0000 7f229f890640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2298109820 0x7f229819f750 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.176+0000 7f229f890640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f22840b3c50 0x7f22840b6160 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.176+0000 7f229f890640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2298108ed0 0x7f229819a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.176+0000 7f229f890640 1 --2- 172.21.15.29:0/1210861980 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2298103590 0x7f2298199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.176+0000 7f229f890640 1 -- 172.21.15.29:0/1210861980 >> 172.21.15.29:0/1210861980 conn(0x7f229807b280 msgr2=0x7f2298106f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.177+0000 7f229f890640 1 -- 172.21.15.29:0/1210861980 shutdown_connections 2024-07-27T06:45:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:22.177+0000 7f229f890640 1 -- 172.21.15.29:0/1210861980 wait complete. 2024-07-27T06:45:23.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:22 smithi029 ceph-mon[32610]: pgmap v348: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:23.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:22 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1210861980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-27T06:45:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:22 smithi077 ceph-mon[37153]: pgmap v348: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:22 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1210861980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-27T06:45:23.123 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149772 got 111669149778 for osd.3 2024-07-27T06:45:23.124 DEBUG:teuthology.parallel:result is None 2024-07-27T06:45:23.124 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-27T06:45:23.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph pg dump --format=json 2024-07-27T06:45:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:22 smithi047 ceph-mon[36835]: pgmap v348: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-07-27T06:45:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:22 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1210861980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-27T06:45:23.439 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.180+0000 7fc807006640 1 -- 172.21.15.29:0/1680490376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc800108ed0 msgr2=0x7fc8001092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.180+0000 7fc807006640 1 --2- 172.21.15.29:0/1680490376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc800108ed0 0x7fc8001092e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009900 tx=0x7fc7f002a410 comp rx=0 tx=0).stop 2024-07-27T06:45:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.180+0000 7fc807006640 1 -- 172.21.15.29:0/1680490376 shutdown_connections 2024-07-27T06:45:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.180+0000 7fc807006640 1 --2- 172.21.15.29:0/1680490376 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc800109820 0x7fc800110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.180+0000 7fc807006640 1 --2- 172.21.15.29:0/1680490376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc800108ed0 0x7fc8001092e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.180+0000 7fc807006640 1 --2- 172.21.15.29:0/1680490376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc800103590 0x7fc8001039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.180+0000 7fc807006640 1 -- 172.21.15.29:0/1680490376 >> 172.21.15.29:0/1680490376 conn(0x7fc80007b280 msgr2=0x7fc80007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.180+0000 7fc807006640 1 -- 172.21.15.29:0/1680490376 shutdown_connections 2024-07-27T06:45:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.180+0000 7fc807006640 1 -- 172.21.15.29:0/1680490376 wait complete. 2024-07-27T06:45:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc807006640 1 Processor -- start 2024-07-27T06:45:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc807006640 1 -- start start 2024-07-27T06:45:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc807006640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc800103590 0x7fc800199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc807006640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc800108ed0 0x7fc80019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc807006640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc800109820 0x7fc80019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc807006640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc800113520 con 0x7fc800103590 2024-07-27T06:45:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc807006640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fc8001133a0 con 0x7fc800108ed0 2024-07-27T06:45:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc807006640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc8001136a0 con 0x7fc800109820 2024-07-27T06:45:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc804da2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc800103590 0x7fc800199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc804da2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc800103590 0x7fc800199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38068/0 (socket says 172.21.15.29:38068) 2024-07-27T06:45:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc804da2640 1 -- 172.21.15.29:0/3203877866 learned_addr learned my addr 172.21.15.29:0/3203877866 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc7f7fff640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc800108ed0 0x7fc80019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc8055a3640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc800109820 0x7fc80019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc804da2640 1 -- 172.21.15.29:0/3203877866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc800109820 msgr2=0x7fc80019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc804da2640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc800109820 0x7fc80019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc804da2640 1 -- 172.21.15.29:0/3203877866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc800108ed0 msgr2=0x7fc80019a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc804da2640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc800108ed0 0x7fc80019a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.181+0000 7fc804da2640 1 -- 172.21.15.29:0/3203877866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80019fe80 con 0x7fc800103590 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.182+0000 7fc804da2640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc800103590 0x7fc800199f10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00c970 tx=0x7fc7fc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.182+0000 7fc7f5ffb640 1 -- 172.21.15.29:0/3203877866 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc007bb0 con 0x7fc800103590 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.182+0000 7fc7f5ffb640 1 -- 172.21.15.29:0/3203877866 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7fc007d50 con 0x7fc800103590 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.182+0000 7fc7f5ffb640 1 -- 172.21.15.29:0/3203877866 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc00f450 con 0x7fc800103590 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.182+0000 7fc807006640 1 -- 172.21.15.29:0/3203877866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8001a0170 con 0x7fc800103590 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.182+0000 7fc807006640 1 -- 172.21.15.29:0/3203877866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8001a68d0 con 0x7fc800103590 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.183+0000 7fc807006640 1 -- 172.21.15.29:0/3203877866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8001039a0 con 0x7fc800103590 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.184+0000 7fc7f5ffb640 1 -- 172.21.15.29:0/3203877866 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fc7fc004f20 con 0x7fc800103590 2024-07-27T06:45:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.185+0000 7fc7f5ffb640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc7e80b3c20 0x7fc7e80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.185+0000 7fc7f7fff640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc7e80b3c20 0x7fc7e80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.185+0000 7fc7f5ffb640 1 -- 172.21.15.29:0/3203877866 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fc7fc0e3aa0 con 0x7fc800103590 2024-07-27T06:45:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.186+0000 7fc7f7fff640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc7e80b3c20 0x7fc7e80b6130 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009230 tx=0x7fc7f0031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.190+0000 7fc7f5ffb640 1 -- 172.21.15.29:0/3203877866 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc7fc014030 con 0x7fc800103590 2024-07-27T06:45:25.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.371+0000 7fc807006640 1 -- 172.21.15.29:0/3203877866 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc80010b670 con 0x7fc7e80b3c20 2024-07-27T06:45:25.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.376+0000 7fc7f5ffb640 1 -- 172.21.15.29:0/3203877866 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259151 (secure 0 0 0) 0x7fc80010b670 con 0x7fc7e80b3c20 2024-07-27T06:45:25.380 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:45:25.382 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-07-27T06:45:25.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.378+0000 7fc807006640 1 -- 172.21.15.29:0/3203877866 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc7e80b3c20 msgr2=0x7fc7e80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.378+0000 7fc807006640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc7e80b3c20 0x7fc7e80b6130 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009230 tx=0x7fc7f0031400 comp rx=0 tx=0).stop 2024-07-27T06:45:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.378+0000 7fc807006640 1 -- 172.21.15.29:0/3203877866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc800103590 msgr2=0x7fc800199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.378+0000 7fc807006640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc800103590 0x7fc800199f10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00c970 tx=0x7fc7fc00ce30 comp rx=0 tx=0).stop 2024-07-27T06:45:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.379+0000 7fc807006640 1 -- 172.21.15.29:0/3203877866 shutdown_connections 2024-07-27T06:45:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.379+0000 7fc807006640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc7e80b3c20 0x7fc7e80b6130 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.379+0000 7fc807006640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc800109820 0x7fc80019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.379+0000 7fc807006640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc800108ed0 0x7fc80019a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.379+0000 7fc807006640 1 --2- 172.21.15.29:0/3203877866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc800103590 0x7fc800199f10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.379+0000 7fc807006640 1 -- 172.21.15.29:0/3203877866 >> 172.21.15.29:0/3203877866 conn(0x7fc80007b280 msgr2=0x7fc800104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.379+0000 7fc807006640 1 -- 172.21.15.29:0/3203877866 shutdown_connections 2024-07-27T06:45:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:25.379+0000 7fc807006640 1 -- 172.21.15.29:0/3203877866 wait complete. 2024-07-27T06:45:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:25 smithi029 ceph-mon[32610]: pgmap v349: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-27T06:45:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:45:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:45:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:25 smithi077 ceph-mon[37153]: pgmap v349: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-27T06:45:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:45:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:45:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:25 smithi047 ceph-mon[36835]: pgmap v349: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-27T06:45:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:45:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:45:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:45:26.125 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-07-27T06:45:23.656215+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659216,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1170,"num_read_kb":709,"num_write":382,"num_write_kb":2298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2953401,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":815,"ondisk_log_size":815,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360496,"kb_used_data":11280,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581328,"statfs":{"total":1151940427776,"available":1151571279872,"internally_reserved":0,"allocated":11550720,"data_stored":7499988,"data_compressed":247335,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":132,"num_read_kb":72,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003552"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483739+0000","last_change":"2024-07-27T06:43:12.802353+0000","last_active":"2024-07-27T06:45:09.483739+0000","last_peered":"2024-07-27T06:45:09.483739+0000","last_clean":"2024-07-27T06:45:09.483739+0000","last_became_active":"2024-07-27T06:43:12.802028+0000","last_became_peered":"2024-07-27T06:43:12.802028+0000","last_unstale":"2024-07-27T06:45:09.483739+0000","last_undegraded":"2024-07-27T06:45:09.483739+0000","last_fullsized":"2024-07-27T06:45:09.483739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:22:24.292567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862408+0000","last_change":"2024-07-27T06:43:12.806269+0000","last_active":"2024-07-27T06:45:09.862408+0000","last_peered":"2024-07-27T06:45:09.862408+0000","last_clean":"2024-07-27T06:45:09.862408+0000","last_became_active":"2024-07-27T06:43:12.803232+0000","last_became_peered":"2024-07-27T06:43:12.803232+0000","last_unstale":"2024-07-27T06:45:09.862408+0000","last_undegraded":"2024-07-27T06:45:09.862408+0000","last_fullsized":"2024-07-27T06:45:09.862408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:57:53.673850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862259+0000","last_change":"2024-07-27T06:43:12.805542+0000","last_active":"2024-07-27T06:45:09.862259+0000","last_peered":"2024-07-27T06:45:09.862259+0000","last_clean":"2024-07-27T06:45:09.862259+0000","last_became_active":"2024-07-27T06:43:12.804951+0000","last_became_peered":"2024-07-27T06:43:12.804951+0000","last_unstale":"2024-07-27T06:45:09.862259+0000","last_undegraded":"2024-07-27T06:45:09.862259+0000","last_fullsized":"2024-07-27T06:45:09.862259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:06:40.469695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862158+0000","last_change":"2024-07-27T06:43:12.796950+0000","last_active":"2024-07-27T06:45:09.862158+0000","last_peered":"2024-07-27T06:45:09.862158+0000","last_clean":"2024-07-27T06:45:09.862158+0000","last_became_active":"2024-07-27T06:43:12.796549+0000","last_became_peered":"2024-07-27T06:43:12.796549+0000","last_unstale":"2024-07-27T06:45:09.862158+0000","last_undegraded":"2024-07-27T06:45:09.862158+0000","last_fullsized":"2024-07-27T06:45:09.862158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:30:36.391046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485939+0000","last_change":"2024-07-27T06:43:12.799838+0000","last_active":"2024-07-27T06:45:09.485939+0000","last_peered":"2024-07-27T06:45:09.485939+0000","last_clean":"2024-07-27T06:45:09.485939+0000","last_became_active":"2024-07-27T06:43:12.799462+0000","last_became_peered":"2024-07-27T06:43:12.799462+0000","last_unstale":"2024-07-27T06:45:09.485939+0000","last_undegraded":"2024-07-27T06:45:09.485939+0000","last_fullsized":"2024-07-27T06:45:09.485939+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:45:19.371259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481556+0000","last_change":"2024-07-27T06:43:12.797187+0000","last_active":"2024-07-27T06:45:09.481556+0000","last_peered":"2024-07-27T06:45:09.481556+0000","last_clean":"2024-07-27T06:45:09.481556+0000","last_became_active":"2024-07-27T06:43:12.796633+0000","last_became_peered":"2024-07-27T06:43:12.796633+0000","last_unstale":"2024-07-27T06:45:09.481556+0000","last_undegraded":"2024-07-27T06:45:09.481556+0000","last_fullsized":"2024-07-27T06:45:09.481556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:22:20.882899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488365+0000","last_change":"2024-07-27T06:43:12.793786+0000","last_active":"2024-07-27T06:45:09.488365+0000","last_peered":"2024-07-27T06:45:09.488365+0000","last_clean":"2024-07-27T06:45:09.488365+0000","last_became_active":"2024-07-27T06:43:12.793413+0000","last_became_peered":"2024-07-27T06:43:12.793413+0000","last_unstale":"2024-07-27T06:45:09.488365+0000","last_undegraded":"2024-07-27T06:45:09.488365+0000","last_fullsized":"2024-07-27T06:45:09.488365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:14.015373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485304+0000","last_change":"2024-07-27T06:43:12.791304+0000","last_active":"2024-07-27T06:45:09.485304+0000","last_peered":"2024-07-27T06:45:09.485304+0000","last_clean":"2024-07-27T06:45:09.485304+0000","last_became_active":"2024-07-27T06:43:12.790801+0000","last_became_peered":"2024-07-27T06:43:12.790801+0000","last_unstale":"2024-07-27T06:45:09.485304+0000","last_undegraded":"2024-07-27T06:45:09.485304+0000","last_fullsized":"2024-07-27T06:45:09.485304+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:12:37.606748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822470+0000","last_change":"2024-07-27T06:43:12.797778+0000","last_active":"2024-07-27T06:45:09.822470+0000","last_peered":"2024-07-27T06:45:09.822470+0000","last_clean":"2024-07-27T06:45:09.822470+0000","last_became_active":"2024-07-27T06:43:12.796784+0000","last_became_peered":"2024-07-27T06:43:12.796784+0000","last_unstale":"2024-07-27T06:45:09.822470+0000","last_undegraded":"2024-07-27T06:45:09.822470+0000","last_fullsized":"2024-07-27T06:45:09.822470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:52:22.275282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.482052+0000","last_change":"2024-07-27T06:43:12.798970+0000","last_active":"2024-07-27T06:45:09.482052+0000","last_peered":"2024-07-27T06:45:09.482052+0000","last_clean":"2024-07-27T06:45:09.482052+0000","last_became_active":"2024-07-27T06:43:12.798549+0000","last_became_peered":"2024-07-27T06:43:12.798549+0000","last_unstale":"2024-07-27T06:45:09.482052+0000","last_undegraded":"2024-07-27T06:45:09.482052+0000","last_fullsized":"2024-07-27T06:45:09.482052+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:58:32.464429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488545+0000","last_change":"2024-07-27T06:43:12.809405+0000","last_active":"2024-07-27T06:45:09.488545+0000","last_peered":"2024-07-27T06:45:09.488545+0000","last_clean":"2024-07-27T06:45:09.488545+0000","last_became_active":"2024-07-27T06:43:12.809030+0000","last_became_peered":"2024-07-27T06:43:12.809030+0000","last_unstale":"2024-07-27T06:45:09.488545+0000","last_undegraded":"2024-07-27T06:45:09.488545+0000","last_fullsized":"2024-07-27T06:45:09.488545+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:46:35.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862145+0000","last_change":"2024-07-27T06:43:12.802647+0000","last_active":"2024-07-27T06:45:09.862145+0000","last_peered":"2024-07-27T06:45:09.862145+0000","last_clean":"2024-07-27T06:45:09.862145+0000","last_became_active":"2024-07-27T06:43:12.802247+0000","last_became_peered":"2024-07-27T06:43:12.802247+0000","last_unstale":"2024-07-27T06:45:09.862145+0000","last_undegraded":"2024-07-27T06:45:09.862145+0000","last_fullsized":"2024-07-27T06:45:09.862145+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:44:24.564852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481567+0000","last_change":"2024-07-27T06:43:12.809140+0000","last_active":"2024-07-27T06:45:09.481567+0000","last_peered":"2024-07-27T06:45:09.481567+0000","last_clean":"2024-07-27T06:45:09.481567+0000","last_became_active":"2024-07-27T06:43:12.808692+0000","last_became_peered":"2024-07-27T06:43:12.808692+0000","last_unstale":"2024-07-27T06:45:09.481567+0000","last_undegraded":"2024-07-27T06:45:09.481567+0000","last_fullsized":"2024-07-27T06:45:09.481567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:37:45.326234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862160+0000","last_change":"2024-07-27T06:43:12.802838+0000","last_active":"2024-07-27T06:45:09.862160+0000","last_peered":"2024-07-27T06:45:09.862160+0000","last_clean":"2024-07-27T06:45:09.862160+0000","last_became_active":"2024-07-27T06:43:12.802503+0000","last_became_peered":"2024-07-27T06:43:12.802503+0000","last_unstale":"2024-07-27T06:45:09.862160+0000","last_undegraded":"2024-07-27T06:45:09.862160+0000","last_fullsized":"2024-07-27T06:45:09.862160+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:41:03.428162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862037+0000","last_change":"2024-07-27T06:43:12.801626+0000","last_active":"2024-07-27T06:45:09.862037+0000","last_peered":"2024-07-27T06:45:09.862037+0000","last_clean":"2024-07-27T06:45:09.862037+0000","last_became_active":"2024-07-27T06:43:12.801266+0000","last_became_peered":"2024-07-27T06:43:12.801266+0000","last_unstale":"2024-07-27T06:45:09.862037+0000","last_undegraded":"2024-07-27T06:45:09.862037+0000","last_fullsized":"2024-07-27T06:45:09.862037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:54:31.934223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.821914+0000","last_change":"2024-07-27T06:43:12.804543+0000","last_active":"2024-07-27T06:45:09.821914+0000","last_peered":"2024-07-27T06:45:09.821914+0000","last_clean":"2024-07-27T06:45:09.821914+0000","last_became_active":"2024-07-27T06:43:12.804173+0000","last_became_peered":"2024-07-27T06:43:12.804173+0000","last_unstale":"2024-07-27T06:45:09.821914+0000","last_undegraded":"2024-07-27T06:45:09.821914+0000","last_fullsized":"2024-07-27T06:45:09.821914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:18:25.337333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483727+0000","last_change":"2024-07-27T06:43:12.798604+0000","last_active":"2024-07-27T06:45:09.483727+0000","last_peered":"2024-07-27T06:45:09.483727+0000","last_clean":"2024-07-27T06:45:09.483727+0000","last_became_active":"2024-07-27T06:43:12.798228+0000","last_became_peered":"2024-07-27T06:43:12.798228+0000","last_unstale":"2024-07-27T06:45:09.483727+0000","last_undegraded":"2024-07-27T06:45:09.483727+0000","last_fullsized":"2024-07-27T06:45:09.483727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:04:35.851399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633300+0000","last_change":"2024-07-27T06:43:12.800452+0000","last_active":"2024-07-27T06:45:09.633300+0000","last_peered":"2024-07-27T06:45:09.633300+0000","last_clean":"2024-07-27T06:45:09.633300+0000","last_became_active":"2024-07-27T06:43:12.800084+0000","last_became_peered":"2024-07-27T06:43:12.800084+0000","last_unstale":"2024-07-27T06:45:09.633300+0000","last_undegraded":"2024-07-27T06:45:09.633300+0000","last_fullsized":"2024-07-27T06:45:09.633300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:54:27.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485716+0000","last_change":"2024-07-27T06:43:12.800761+0000","last_active":"2024-07-27T06:45:09.485716+0000","last_peered":"2024-07-27T06:45:09.485716+0000","last_clean":"2024-07-27T06:45:09.485716+0000","last_became_active":"2024-07-27T06:43:12.800414+0000","last_became_peered":"2024-07-27T06:43:12.800414+0000","last_unstale":"2024-07-27T06:45:09.485716+0000","last_undegraded":"2024-07-27T06:45:09.485716+0000","last_fullsized":"2024-07-27T06:45:09.485716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:02:47.247522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483308+0000","last_change":"2024-07-27T06:43:12.795470+0000","last_active":"2024-07-27T06:45:09.483308+0000","last_peered":"2024-07-27T06:45:09.483308+0000","last_clean":"2024-07-27T06:45:09.483308+0000","last_became_active":"2024-07-27T06:43:12.795136+0000","last_became_peered":"2024-07-27T06:43:12.795136+0000","last_unstale":"2024-07-27T06:45:09.483308+0000","last_undegraded":"2024-07-27T06:45:09.483308+0000","last_fullsized":"2024-07-27T06:45:09.483308+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:01:22.178699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481693+0000","last_change":"2024-07-27T06:43:12.803897+0000","last_active":"2024-07-27T06:45:09.481693+0000","last_peered":"2024-07-27T06:45:09.481693+0000","last_clean":"2024-07-27T06:45:09.481693+0000","last_became_active":"2024-07-27T06:43:12.803546+0000","last_became_peered":"2024-07-27T06:43:12.803546+0000","last_unstale":"2024-07-27T06:45:09.481693+0000","last_undegraded":"2024-07-27T06:45:09.481693+0000","last_fullsized":"2024-07-27T06:45:09.481693+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:13:50.816831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822134+0000","last_change":"2024-07-27T06:44:09.238514+0000","last_active":"2024-07-27T06:45:09.822134+0000","last_peered":"2024-07-27T06:45:09.822134+0000","last_clean":"2024-07-27T06:45:09.822134+0000","last_became_active":"2024-07-27T06:44:09.237903+0000","last_became_peered":"2024-07-27T06:44:09.237903+0000","last_unstale":"2024-07-27T06:45:09.822134+0000","last_undegraded":"2024-07-27T06:45:09.822134+0000","last_fullsized":"2024-07-27T06:45:09.822134+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:20:59.887491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485364+0000","last_change":"2024-07-27T06:43:12.788570+0000","last_active":"2024-07-27T06:45:09.485364+0000","last_peered":"2024-07-27T06:45:09.485364+0000","last_clean":"2024-07-27T06:45:09.485364+0000","last_became_active":"2024-07-27T06:43:12.788199+0000","last_became_peered":"2024-07-27T06:43:12.788199+0000","last_unstale":"2024-07-27T06:45:09.485364+0000","last_undegraded":"2024-07-27T06:45:09.485364+0000","last_fullsized":"2024-07-27T06:45:09.485364+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:44:26.751094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862687+0000","last_change":"2024-07-27T06:43:12.792866+0000","last_active":"2024-07-27T06:45:09.862687+0000","last_peered":"2024-07-27T06:45:09.862687+0000","last_clean":"2024-07-27T06:45:09.862687+0000","last_became_active":"2024-07-27T06:43:12.792389+0000","last_became_peered":"2024-07-27T06:43:12.792389+0000","last_unstale":"2024-07-27T06:45:09.862687+0000","last_undegraded":"2024-07-27T06:45:09.862687+0000","last_fullsized":"2024-07-27T06:45:09.862687+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:10:37.367642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485763+0000","last_change":"2024-07-27T06:43:12.797002+0000","last_active":"2024-07-27T06:45:09.485763+0000","last_peered":"2024-07-27T06:45:09.485763+0000","last_clean":"2024-07-27T06:45:09.485763+0000","last_became_active":"2024-07-27T06:43:12.796651+0000","last_became_peered":"2024-07-27T06:43:12.796651+0000","last_unstale":"2024-07-27T06:45:09.485763+0000","last_undegraded":"2024-07-27T06:45:09.485763+0000","last_fullsized":"2024-07-27T06:45:09.485763+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:28:06.934042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485804+0000","last_change":"2024-07-27T06:43:12.798549+0000","last_active":"2024-07-27T06:45:09.485804+0000","last_peered":"2024-07-27T06:45:09.485804+0000","last_clean":"2024-07-27T06:45:09.485804+0000","last_became_active":"2024-07-27T06:43:12.798201+0000","last_became_peered":"2024-07-27T06:43:12.798201+0000","last_unstale":"2024-07-27T06:45:09.485804+0000","last_undegraded":"2024-07-27T06:45:09.485804+0000","last_fullsized":"2024-07-27T06:45:09.485804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:59:47.525536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483583+0000","last_change":"2024-07-27T06:43:12.791053+0000","last_active":"2024-07-27T06:45:09.483583+0000","last_peered":"2024-07-27T06:45:09.483583+0000","last_clean":"2024-07-27T06:45:09.483583+0000","last_became_active":"2024-07-27T06:43:12.790718+0000","last_became_peered":"2024-07-27T06:43:12.790718+0000","last_unstale":"2024-07-27T06:45:09.483583+0000","last_undegraded":"2024-07-27T06:45:09.483583+0000","last_fullsized":"2024-07-27T06:45:09.483583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:09.455399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481758+0000","last_change":"2024-07-27T06:43:12.804180+0000","last_active":"2024-07-27T06:45:09.481758+0000","last_peered":"2024-07-27T06:45:09.481758+0000","last_clean":"2024-07-27T06:45:09.481758+0000","last_became_active":"2024-07-27T06:43:12.803881+0000","last_became_peered":"2024-07-27T06:43:12.803881+0000","last_unstale":"2024-07-27T06:45:09.481758+0000","last_undegraded":"2024-07-27T06:45:09.481758+0000","last_fullsized":"2024-07-27T06:45:09.481758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:01:05.009233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862727+0000","last_change":"2024-07-27T06:43:12.809150+0000","last_active":"2024-07-27T06:45:09.862727+0000","last_peered":"2024-07-27T06:45:09.862727+0000","last_clean":"2024-07-27T06:45:09.862727+0000","last_became_active":"2024-07-27T06:43:12.808428+0000","last_became_peered":"2024-07-27T06:43:12.808428+0000","last_unstale":"2024-07-27T06:45:09.862727+0000","last_undegraded":"2024-07-27T06:45:09.862727+0000","last_fullsized":"2024-07-27T06:45:09.862727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:07:13.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486325+0000","last_change":"2024-07-27T06:44:09.233785+0000","last_active":"2024-07-27T06:45:09.486325+0000","last_peered":"2024-07-27T06:45:09.486325+0000","last_clean":"2024-07-27T06:45:09.486325+0000","last_became_active":"2024-07-27T06:44:09.233184+0000","last_became_peered":"2024-07-27T06:44:09.233184+0000","last_unstale":"2024-07-27T06:45:09.486325+0000","last_undegraded":"2024-07-27T06:45:09.486325+0000","last_fullsized":"2024-07-27T06:45:09.486325+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:12:48.556383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486129+0000","last_change":"2024-07-27T06:44:09.320182+0000","last_active":"2024-07-27T06:45:09.486129+0000","last_peered":"2024-07-27T06:45:09.486129+0000","last_clean":"2024-07-27T06:45:09.486129+0000","last_became_active":"2024-07-27T06:44:09.319661+0000","last_became_peered":"2024-07-27T06:44:09.319661+0000","last_unstale":"2024-07-27T06:45:09.486129+0000","last_undegraded":"2024-07-27T06:45:09.486129+0000","last_fullsized":"2024-07-27T06:45:09.486129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:52:11.006680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483342+0000","last_change":"2024-07-27T06:43:12.787866+0000","last_active":"2024-07-27T06:45:09.483342+0000","last_peered":"2024-07-27T06:45:09.483342+0000","last_clean":"2024-07-27T06:45:09.483342+0000","last_became_active":"2024-07-27T06:43:12.787513+0000","last_became_peered":"2024-07-27T06:43:12.787513+0000","last_unstale":"2024-07-27T06:45:09.483342+0000","last_undegraded":"2024-07-27T06:45:09.483342+0000","last_fullsized":"2024-07-27T06:45:09.483342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:48:57.876588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485327+0000","last_change":"2024-07-27T06:43:12.791158+0000","last_active":"2024-07-27T06:45:09.485327+0000","last_peered":"2024-07-27T06:45:09.485327+0000","last_clean":"2024-07-27T06:45:09.485327+0000","last_became_active":"2024-07-27T06:43:12.790771+0000","last_became_peered":"2024-07-27T06:43:12.790771+0000","last_unstale":"2024-07-27T06:45:09.485327+0000","last_undegraded":"2024-07-27T06:45:09.485327+0000","last_fullsized":"2024-07-27T06:45:09.485327+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:10:18.739265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'7","reported_seq":148,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:18.051261+0000","last_change":"2024-07-27T06:43:09.769281+0000","last_active":"2024-07-27T06:45:18.051261+0000","last_peered":"2024-07-27T06:45:18.051261+0000","last_clean":"2024-07-27T06:45:18.051261+0000","last_became_active":"2024-07-27T06:43:09.768864+0000","last_became_peered":"2024-07-27T06:43:09.768864+0000","last_unstale":"2024-07-27T06:45:18.051261+0000","last_undegraded":"2024-07-27T06:45:18.051261+0000","last_fullsized":"2024-07-27T06:45:18.051261+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:55:14.191180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862898+0000","last_change":"2024-07-27T06:44:09.319930+0000","last_active":"2024-07-27T06:45:09.862898+0000","last_peered":"2024-07-27T06:45:09.862898+0000","last_clean":"2024-07-27T06:45:09.862898+0000","last_became_active":"2024-07-27T06:44:09.319535+0000","last_became_peered":"2024-07-27T06:44:09.319535+0000","last_unstale":"2024-07-27T06:45:09.862898+0000","last_undegraded":"2024-07-27T06:45:09.862898+0000","last_fullsized":"2024-07-27T06:45:09.862898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:39:52.887860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481606+0000","last_change":"2024-07-27T06:43:09.772227+0000","last_active":"2024-07-27T06:45:09.481606+0000","last_peered":"2024-07-27T06:45:09.481606+0000","last_clean":"2024-07-27T06:45:09.481606+0000","last_became_active":"2024-07-27T06:43:09.771812+0000","last_became_peered":"2024-07-27T06:43:09.771812+0000","last_unstale":"2024-07-27T06:45:09.481606+0000","last_undegraded":"2024-07-27T06:45:09.481606+0000","last_fullsized":"2024-07-27T06:45:09.481606+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:27:28.054590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.480968+0000","last_change":"2024-07-27T06:43:09.766490+0000","last_active":"2024-07-27T06:45:09.480968+0000","last_peered":"2024-07-27T06:45:09.480968+0000","last_clean":"2024-07-27T06:45:09.480968+0000","last_became_active":"2024-07-27T06:43:09.765973+0000","last_became_peered":"2024-07-27T06:43:09.765973+0000","last_unstale":"2024-07-27T06:45:09.480968+0000","last_undegraded":"2024-07-27T06:45:09.480968+0000","last_fullsized":"2024-07-27T06:45:09.480968+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:00:02.913080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485390+0000","last_change":"2024-07-27T06:43:12.805234+0000","last_active":"2024-07-27T06:45:09.485390+0000","last_peered":"2024-07-27T06:45:09.485390+0000","last_clean":"2024-07-27T06:45:09.485390+0000","last_became_active":"2024-07-27T06:43:12.804908+0000","last_became_peered":"2024-07-27T06:43:12.804908+0000","last_unstale":"2024-07-27T06:45:09.485390+0000","last_undegraded":"2024-07-27T06:45:09.485390+0000","last_fullsized":"2024-07-27T06:45:09.485390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:09:34.988755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862144+0000","last_change":"2024-07-27T06:43:09.774201+0000","last_active":"2024-07-27T06:45:09.862144+0000","last_peered":"2024-07-27T06:45:09.862144+0000","last_clean":"2024-07-27T06:45:09.862144+0000","last_became_active":"2024-07-27T06:43:09.773817+0000","last_became_peered":"2024-07-27T06:43:09.773817+0000","last_unstale":"2024-07-27T06:45:09.862144+0000","last_undegraded":"2024-07-27T06:45:09.862144+0000","last_fullsized":"2024-07-27T06:45:09.862144+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:06.166251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633044+0000","last_change":"2024-07-27T06:43:12.800174+0000","last_active":"2024-07-27T06:45:09.633044+0000","last_peered":"2024-07-27T06:45:09.633044+0000","last_clean":"2024-07-27T06:45:09.633044+0000","last_became_active":"2024-07-27T06:43:12.799769+0000","last_became_peered":"2024-07-27T06:43:12.799769+0000","last_unstale":"2024-07-27T06:45:09.633044+0000","last_undegraded":"2024-07-27T06:45:09.633044+0000","last_fullsized":"2024-07-27T06:45:09.633044+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:43:06.890372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":254,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:21.400760+0000","last_change":"2024-07-27T06:43:09.776612+0000","last_active":"2024-07-27T06:45:21.400760+0000","last_peered":"2024-07-27T06:45:21.400760+0000","last_clean":"2024-07-27T06:45:21.400760+0000","last_became_active":"2024-07-27T06:43:09.775761+0000","last_became_peered":"2024-07-27T06:43:09.775761+0000","last_unstale":"2024-07-27T06:45:21.400760+0000","last_undegraded":"2024-07-27T06:45:21.400760+0000","last_fullsized":"2024-07-27T06:45:21.400760+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:42:52.132357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":217,"num_read_kb":217,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822431+0000","last_change":"2024-07-27T06:43:12.797200+0000","last_active":"2024-07-27T06:45:09.822431+0000","last_peered":"2024-07-27T06:45:09.822431+0000","last_clean":"2024-07-27T06:45:09.822431+0000","last_became_active":"2024-07-27T06:43:12.796288+0000","last_became_peered":"2024-07-27T06:43:12.796288+0000","last_unstale":"2024-07-27T06:45:09.822431+0000","last_undegraded":"2024-07-27T06:45:09.822431+0000","last_fullsized":"2024-07-27T06:45:09.822431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:31:27.603883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488625+0000","last_change":"2024-07-27T06:43:09.772658+0000","last_active":"2024-07-27T06:45:09.488625+0000","last_peered":"2024-07-27T06:45:09.488625+0000","last_clean":"2024-07-27T06:45:09.488625+0000","last_became_active":"2024-07-27T06:43:09.772277+0000","last_became_peered":"2024-07-27T06:43:09.772277+0000","last_unstale":"2024-07-27T06:45:09.488625+0000","last_undegraded":"2024-07-27T06:45:09.488625+0000","last_fullsized":"2024-07-27T06:45:09.488625+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:09:16.854718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862494+0000","last_change":"2024-07-27T06:43:12.809643+0000","last_active":"2024-07-27T06:45:09.862494+0000","last_peered":"2024-07-27T06:45:09.862494+0000","last_clean":"2024-07-27T06:45:09.862494+0000","last_became_active":"2024-07-27T06:43:12.809020+0000","last_became_peered":"2024-07-27T06:43:12.809020+0000","last_unstale":"2024-07-27T06:45:09.862494+0000","last_undegraded":"2024-07-27T06:45:09.862494+0000","last_fullsized":"2024-07-27T06:45:09.862494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:08:03.062914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862958+0000","last_change":"2024-07-27T06:43:10.534042+0000","last_active":"2024-07-27T06:45:09.862958+0000","last_peered":"2024-07-27T06:45:09.862958+0000","last_clean":"2024-07-27T06:45:09.862958+0000","last_became_active":"2024-07-27T06:43:10.533615+0000","last_became_peered":"2024-07-27T06:43:10.533615+0000","last_unstale":"2024-07-27T06:45:09.862958+0000","last_undegraded":"2024-07-27T06:45:09.862958+0000","last_fullsized":"2024-07-27T06:45:09.862958+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:09:32.235633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822162+0000","last_change":"2024-07-27T06:43:12.798764+0000","last_active":"2024-07-27T06:45:09.822162+0000","last_peered":"2024-07-27T06:45:09.822162+0000","last_clean":"2024-07-27T06:45:09.822162+0000","last_became_active":"2024-07-27T06:43:12.797500+0000","last_became_peered":"2024-07-27T06:43:12.797500+0000","last_unstale":"2024-07-27T06:45:09.822162+0000","last_undegraded":"2024-07-27T06:45:09.822162+0000","last_fullsized":"2024-07-27T06:45:09.822162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:12:09.368711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'34","reported_seq":293,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:22.057403+0000","last_change":"2024-07-27T06:43:09.775954+0000","last_active":"2024-07-27T06:45:22.057403+0000","last_peered":"2024-07-27T06:45:22.057403+0000","last_clean":"2024-07-27T06:45:22.057403+0000","last_became_active":"2024-07-27T06:43:09.775498+0000","last_became_peered":"2024-07-27T06:43:09.775498+0000","last_unstale":"2024-07-27T06:45:22.057403+0000","last_undegraded":"2024-07-27T06:45:22.057403+0000","last_fullsized":"2024-07-27T06:45:22.057403+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:08:46.965525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60596,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":228,"num_read_kb":173,"num_write":36,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486502+0000","last_change":"2024-07-27T06:43:12.797220+0000","last_active":"2024-07-27T06:45:09.486502+0000","last_peered":"2024-07-27T06:45:09.486502+0000","last_clean":"2024-07-27T06:45:09.486502+0000","last_became_active":"2024-07-27T06:43:12.796889+0000","last_became_peered":"2024-07-27T06:43:12.796889+0000","last_unstale":"2024-07-27T06:45:09.486502+0000","last_undegraded":"2024-07-27T06:45:09.486502+0000","last_fullsized":"2024-07-27T06:45:09.486502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:24:36.275738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485602+0000","last_change":"2024-07-27T06:43:09.776600+0000","last_active":"2024-07-27T06:45:09.485602+0000","last_peered":"2024-07-27T06:45:09.485602+0000","last_clean":"2024-07-27T06:45:09.485602+0000","last_became_active":"2024-07-27T06:43:09.776163+0000","last_became_peered":"2024-07-27T06:43:09.776163+0000","last_unstale":"2024-07-27T06:45:09.485602+0000","last_undegraded":"2024-07-27T06:45:09.485602+0000","last_fullsized":"2024-07-27T06:45:09.485602+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:22:15.947916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481153+0000","last_change":"2024-07-27T06:43:12.804984+0000","last_active":"2024-07-27T06:45:09.481153+0000","last_peered":"2024-07-27T06:45:09.481153+0000","last_clean":"2024-07-27T06:45:09.481153+0000","last_became_active":"2024-07-27T06:43:12.804644+0000","last_became_peered":"2024-07-27T06:43:12.804644+0000","last_unstale":"2024-07-27T06:45:09.481153+0000","last_undegraded":"2024-07-27T06:45:09.481153+0000","last_fullsized":"2024-07-27T06:45:09.481153+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.355981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862176+0000","last_change":"2024-07-27T06:43:09.772489+0000","last_active":"2024-07-27T06:45:09.862176+0000","last_peered":"2024-07-27T06:45:09.862176+0000","last_clean":"2024-07-27T06:45:09.862176+0000","last_became_active":"2024-07-27T06:43:09.772069+0000","last_became_peered":"2024-07-27T06:43:09.772069+0000","last_unstale":"2024-07-27T06:45:09.862176+0000","last_undegraded":"2024-07-27T06:45:09.862176+0000","last_fullsized":"2024-07-27T06:45:09.862176+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:15:30.549971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488522+0000","last_change":"2024-07-27T06:43:12.803376+0000","last_active":"2024-07-27T06:45:09.488522+0000","last_peered":"2024-07-27T06:45:09.488522+0000","last_clean":"2024-07-27T06:45:09.488522+0000","last_became_active":"2024-07-27T06:43:12.802961+0000","last_became_peered":"2024-07-27T06:43:12.802961+0000","last_unstale":"2024-07-27T06:45:09.488522+0000","last_undegraded":"2024-07-27T06:45:09.488522+0000","last_fullsized":"2024-07-27T06:45:09.488522+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:00:24.649877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":145,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:19.052582+0000","last_change":"2024-07-27T06:43:09.773182+0000","last_active":"2024-07-27T06:45:19.052582+0000","last_peered":"2024-07-27T06:45:19.052582+0000","last_clean":"2024-07-27T06:45:19.052582+0000","last_became_active":"2024-07-27T06:43:09.772126+0000","last_became_peered":"2024-07-27T06:43:09.772126+0000","last_unstale":"2024-07-27T06:45:19.052582+0000","last_undegraded":"2024-07-27T06:45:19.052582+0000","last_fullsized":"2024-07-27T06:45:19.052582+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:45:26.598799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":110,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488346+0000","last_change":"2024-07-27T06:43:12.799385+0000","last_active":"2024-07-27T06:45:09.488346+0000","last_peered":"2024-07-27T06:45:09.488346+0000","last_clean":"2024-07-27T06:45:09.488346+0000","last_became_active":"2024-07-27T06:43:12.798639+0000","last_became_peered":"2024-07-27T06:43:12.798639+0000","last_unstale":"2024-07-27T06:45:09.488346+0000","last_undegraded":"2024-07-27T06:45:09.488346+0000","last_fullsized":"2024-07-27T06:45:09.488346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:56:29.462491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485244+0000","last_change":"2024-07-27T06:43:10.535429+0000","last_active":"2024-07-27T06:45:09.485244+0000","last_peered":"2024-07-27T06:45:09.485244+0000","last_clean":"2024-07-27T06:45:09.485244+0000","last_became_active":"2024-07-27T06:43:10.534952+0000","last_became_peered":"2024-07-27T06:43:10.534952+0000","last_unstale":"2024-07-27T06:45:09.485244+0000","last_undegraded":"2024-07-27T06:45:09.485244+0000","last_fullsized":"2024-07-27T06:45:09.485244+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:54:20.018263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862639+0000","last_change":"2024-07-27T06:43:12.804266+0000","last_active":"2024-07-27T06:45:09.862639+0000","last_peered":"2024-07-27T06:45:09.862639+0000","last_clean":"2024-07-27T06:45:09.862639+0000","last_became_active":"2024-07-27T06:43:12.803888+0000","last_became_peered":"2024-07-27T06:43:12.803888+0000","last_unstale":"2024-07-27T06:45:09.862639+0000","last_undegraded":"2024-07-27T06:45:09.862639+0000","last_fullsized":"2024-07-27T06:45:09.862639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:48:00.490592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485746+0000","last_change":"2024-07-27T06:43:09.777656+0000","last_active":"2024-07-27T06:45:09.485746+0000","last_peered":"2024-07-27T06:45:09.485746+0000","last_clean":"2024-07-27T06:45:09.485746+0000","last_became_active":"2024-07-27T06:43:09.776620+0000","last_became_peered":"2024-07-27T06:43:09.776620+0000","last_unstale":"2024-07-27T06:45:09.485746+0000","last_undegraded":"2024-07-27T06:45:09.485746+0000","last_fullsized":"2024-07-27T06:45:09.485746+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:50.203623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483488+0000","last_change":"2024-07-27T06:43:12.796179+0000","last_active":"2024-07-27T06:45:09.483488+0000","last_peered":"2024-07-27T06:45:09.483488+0000","last_clean":"2024-07-27T06:45:09.483488+0000","last_became_active":"2024-07-27T06:43:12.795701+0000","last_became_peered":"2024-07-27T06:43:12.795701+0000","last_unstale":"2024-07-27T06:45:09.483488+0000","last_undegraded":"2024-07-27T06:45:09.483488+0000","last_fullsized":"2024-07-27T06:45:09.483488+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:20:12.236769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481579+0000","last_change":"2024-07-27T06:43:09.769949+0000","last_active":"2024-07-27T06:45:09.481579+0000","last_peered":"2024-07-27T06:45:09.481579+0000","last_clean":"2024-07-27T06:45:09.481579+0000","last_became_active":"2024-07-27T06:43:09.769602+0000","last_became_peered":"2024-07-27T06:43:09.769602+0000","last_unstale":"2024-07-27T06:45:09.481579+0000","last_undegraded":"2024-07-27T06:45:09.481579+0000","last_fullsized":"2024-07-27T06:45:09.481579+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:28:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488763+0000","last_change":"2024-07-27T06:43:12.802230+0000","last_active":"2024-07-27T06:45:09.488763+0000","last_peered":"2024-07-27T06:45:09.488763+0000","last_clean":"2024-07-27T06:45:09.488763+0000","last_became_active":"2024-07-27T06:43:12.800944+0000","last_became_peered":"2024-07-27T06:43:12.800944+0000","last_unstale":"2024-07-27T06:45:09.488763+0000","last_undegraded":"2024-07-27T06:45:09.488763+0000","last_fullsized":"2024-07-27T06:45:09.488763+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:05:54.585955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862208+0000","last_change":"2024-07-27T06:43:09.768239+0000","last_active":"2024-07-27T06:45:09.862208+0000","last_peered":"2024-07-27T06:45:09.862208+0000","last_clean":"2024-07-27T06:45:09.862208+0000","last_became_active":"2024-07-27T06:43:09.767584+0000","last_became_peered":"2024-07-27T06:43:09.767584+0000","last_unstale":"2024-07-27T06:45:09.862208+0000","last_undegraded":"2024-07-27T06:45:09.862208+0000","last_fullsized":"2024-07-27T06:45:09.862208+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:35:39.552246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488502+0000","last_change":"2024-07-27T06:43:12.803779+0000","last_active":"2024-07-27T06:45:09.488502+0000","last_peered":"2024-07-27T06:45:09.488502+0000","last_clean":"2024-07-27T06:45:09.488502+0000","last_became_active":"2024-07-27T06:43:12.803359+0000","last_became_peered":"2024-07-27T06:43:12.803359+0000","last_unstale":"2024-07-27T06:45:09.488502+0000","last_undegraded":"2024-07-27T06:45:09.488502+0000","last_fullsized":"2024-07-27T06:45:09.488502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:32:53.167039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'7","reported_seq":98,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:19.397744+0000","last_change":"2024-07-27T06:44:09.241782+0000","last_active":"2024-07-27T06:45:19.397744+0000","last_peered":"2024-07-27T06:45:19.397744+0000","last_clean":"2024-07-27T06:45:19.397744+0000","last_became_active":"2024-07-27T06:44:09.234692+0000","last_became_peered":"2024-07-27T06:44:09.234692+0000","last_unstale":"2024-07-27T06:45:19.397744+0000","last_undegraded":"2024-07-27T06:45:19.397744+0000","last_fullsized":"2024-07-27T06:45:19.397744+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:45:02.155686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1561,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488659+0000","last_change":"2024-07-27T06:43:12.805830+0000","last_active":"2024-07-27T06:45:09.488659+0000","last_peered":"2024-07-27T06:45:09.488659+0000","last_clean":"2024-07-27T06:45:09.488659+0000","last_became_active":"2024-07-27T06:43:12.805324+0000","last_became_peered":"2024-07-27T06:43:12.805324+0000","last_unstale":"2024-07-27T06:45:09.488659+0000","last_undegraded":"2024-07-27T06:45:09.488659+0000","last_fullsized":"2024-07-27T06:45:09.488659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:38:31.430676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":108,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.489020+0000","last_change":"2024-07-27T06:41:43.549132+0000","last_active":"2024-07-27T06:45:09.489020+0000","last_peered":"2024-07-27T06:45:09.489020+0000","last_clean":"2024-07-27T06:45:09.489020+0000","last_became_active":"2024-07-27T06:41:43.542635+0000","last_became_peered":"2024-07-27T06:41:43.542635+0000","last_unstale":"2024-07-27T06:45:09.489020+0000","last_undegraded":"2024-07-27T06:45:09.489020+0000","last_fullsized":"2024-07-27T06:45:09.489020+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_clean_scrub_stamp":"2024-07-27T06:38:21.639907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:10:06.417387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.861700+0000","last_change":"2024-07-27T06:43:09.768774+0000","last_active":"2024-07-27T06:45:09.861700+0000","last_peered":"2024-07-27T06:45:09.861700+0000","last_clean":"2024-07-27T06:45:09.861700+0000","last_became_active":"2024-07-27T06:43:09.768345+0000","last_became_peered":"2024-07-27T06:43:09.768345+0000","last_unstale":"2024-07-27T06:45:09.861700+0000","last_undegraded":"2024-07-27T06:45:09.861700+0000","last_fullsized":"2024-07-27T06:45:09.861700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:40:49.411746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485670+0000","last_change":"2024-07-27T06:43:12.797817+0000","last_active":"2024-07-27T06:45:09.485670+0000","last_peered":"2024-07-27T06:45:09.485670+0000","last_clean":"2024-07-27T06:45:09.485670+0000","last_became_active":"2024-07-27T06:43:12.797483+0000","last_became_peered":"2024-07-27T06:43:12.797483+0000","last_unstale":"2024-07-27T06:45:09.485670+0000","last_undegraded":"2024-07-27T06:45:09.485670+0000","last_fullsized":"2024-07-27T06:45:09.485670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:24:44.539505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488887+0000","last_change":"2024-07-27T06:43:09.776880+0000","last_active":"2024-07-27T06:45:09.488887+0000","last_peered":"2024-07-27T06:45:09.488887+0000","last_clean":"2024-07-27T06:45:09.488887+0000","last_became_active":"2024-07-27T06:43:09.776486+0000","last_became_peered":"2024-07-27T06:43:09.776486+0000","last_unstale":"2024-07-27T06:45:09.488887+0000","last_undegraded":"2024-07-27T06:45:09.488887+0000","last_fullsized":"2024-07-27T06:45:09.488887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:55:25.436405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.480961+0000","last_change":"2024-07-27T06:43:12.799755+0000","last_active":"2024-07-27T06:45:09.480961+0000","last_peered":"2024-07-27T06:45:09.480961+0000","last_clean":"2024-07-27T06:45:09.480961+0000","last_became_active":"2024-07-27T06:43:12.799412+0000","last_became_peered":"2024-07-27T06:43:12.799412+0000","last_unstale":"2024-07-27T06:45:09.480961+0000","last_undegraded":"2024-07-27T06:45:09.480961+0000","last_fullsized":"2024-07-27T06:45:09.480961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:45:50.545431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":146,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:20.053473+0000","last_change":"2024-07-27T06:43:10.533039+0000","last_active":"2024-07-27T06:45:20.053473+0000","last_peered":"2024-07-27T06:45:20.053473+0000","last_clean":"2024-07-27T06:45:20.053473+0000","last_became_active":"2024-07-27T06:43:10.532583+0000","last_became_peered":"2024-07-27T06:43:10.532583+0000","last_unstale":"2024-07-27T06:45:20.053473+0000","last_undegraded":"2024-07-27T06:45:20.053473+0000","last_fullsized":"2024-07-27T06:45:20.053473+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:04:51.954463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":109,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633016+0000","last_change":"2024-07-27T06:44:09.238218+0000","last_active":"2024-07-27T06:45:09.633016+0000","last_peered":"2024-07-27T06:45:09.633016+0000","last_clean":"2024-07-27T06:45:09.633016+0000","last_became_active":"2024-07-27T06:44:09.237841+0000","last_became_peered":"2024-07-27T06:44:09.237841+0000","last_unstale":"2024-07-27T06:45:09.633016+0000","last_undegraded":"2024-07-27T06:45:09.633016+0000","last_fullsized":"2024-07-27T06:45:09.633016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:19.101854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"81'7","reported_seq":150,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:20.064380+0000","last_change":"2024-07-27T06:43:09.776214+0000","last_active":"2024-07-27T06:45:20.064380+0000","last_peered":"2024-07-27T06:45:20.064380+0000","last_clean":"2024-07-27T06:45:20.064380+0000","last_became_active":"2024-07-27T06:43:09.775719+0000","last_became_peered":"2024-07-27T06:43:09.775719+0000","last_unstale":"2024-07-27T06:45:20.064380+0000","last_undegraded":"2024-07-27T06:45:20.064380+0000","last_fullsized":"2024-07-27T06:45:20.064380+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:57:20.381599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481735+0000","last_change":"2024-07-27T06:43:12.794909+0000","last_active":"2024-07-27T06:45:09.481735+0000","last_peered":"2024-07-27T06:45:09.481735+0000","last_clean":"2024-07-27T06:45:09.481735+0000","last_became_active":"2024-07-27T06:43:12.794497+0000","last_became_peered":"2024-07-27T06:43:12.794497+0000","last_unstale":"2024-07-27T06:45:09.481735+0000","last_undegraded":"2024-07-27T06:45:09.481735+0000","last_fullsized":"2024-07-27T06:45:09.481735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:42:49.760126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633255+0000","last_change":"2024-07-27T06:43:10.534240+0000","last_active":"2024-07-27T06:45:09.633255+0000","last_peered":"2024-07-27T06:45:09.633255+0000","last_clean":"2024-07-27T06:45:09.633255+0000","last_became_active":"2024-07-27T06:43:10.533596+0000","last_became_peered":"2024-07-27T06:43:10.533596+0000","last_unstale":"2024-07-27T06:45:09.633255+0000","last_undegraded":"2024-07-27T06:45:09.633255+0000","last_fullsized":"2024-07-27T06:45:09.633255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:16.098839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481991+0000","last_change":"2024-07-27T06:43:12.805248+0000","last_active":"2024-07-27T06:45:09.481991+0000","last_peered":"2024-07-27T06:45:09.481991+0000","last_clean":"2024-07-27T06:45:09.481991+0000","last_became_active":"2024-07-27T06:43:12.803693+0000","last_became_peered":"2024-07-27T06:43:12.803693+0000","last_unstale":"2024-07-27T06:45:09.481991+0000","last_undegraded":"2024-07-27T06:45:09.481991+0000","last_fullsized":"2024-07-27T06:45:09.481991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:17.466465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486692+0000","last_change":"2024-07-27T06:43:10.533701+0000","last_active":"2024-07-27T06:45:09.486692+0000","last_peered":"2024-07-27T06:45:09.486692+0000","last_clean":"2024-07-27T06:45:09.486692+0000","last_became_active":"2024-07-27T06:43:10.533109+0000","last_became_peered":"2024-07-27T06:43:10.533109+0000","last_unstale":"2024-07-27T06:45:09.486692+0000","last_undegraded":"2024-07-27T06:45:09.486692+0000","last_fullsized":"2024-07-27T06:45:09.486692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:32:52.906691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481918+0000","last_change":"2024-07-27T06:43:12.798479+0000","last_active":"2024-07-27T06:45:09.481918+0000","last_peered":"2024-07-27T06:45:09.481918+0000","last_clean":"2024-07-27T06:45:09.481918+0000","last_became_active":"2024-07-27T06:43:12.798124+0000","last_became_peered":"2024-07-27T06:43:12.798124+0000","last_unstale":"2024-07-27T06:45:09.481918+0000","last_undegraded":"2024-07-27T06:45:09.481918+0000","last_fullsized":"2024-07-27T06:45:09.481918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:14:55.267213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'16","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633427+0000","last_change":"2024-07-27T06:43:09.770122+0000","last_active":"2024-07-27T06:45:09.633427+0000","last_peered":"2024-07-27T06:45:09.633427+0000","last_clean":"2024-07-27T06:45:09.633427+0000","last_became_active":"2024-07-27T06:43:09.769784+0000","last_became_peered":"2024-07-27T06:43:09.769784+0000","last_unstale":"2024-07-27T06:45:09.633427+0000","last_undegraded":"2024-07-27T06:45:09.633427+0000","last_fullsized":"2024-07-27T06:45:09.633427+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:47:00.398135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485844+0000","last_change":"2024-07-27T06:43:12.799603+0000","last_active":"2024-07-27T06:45:09.485844+0000","last_peered":"2024-07-27T06:45:09.485844+0000","last_clean":"2024-07-27T06:45:09.485844+0000","last_became_active":"2024-07-27T06:43:12.799189+0000","last_became_peered":"2024-07-27T06:43:12.799189+0000","last_unstale":"2024-07-27T06:45:09.485844+0000","last_undegraded":"2024-07-27T06:45:09.485844+0000","last_fullsized":"2024-07-27T06:45:09.485844+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:32:30.603258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862220+0000","last_change":"2024-07-27T06:43:10.004726+0000","last_active":"2024-07-27T06:45:09.862220+0000","last_peered":"2024-07-27T06:45:09.862220+0000","last_clean":"2024-07-27T06:45:09.862220+0000","last_became_active":"2024-07-27T06:43:10.004367+0000","last_became_peered":"2024-07-27T06:43:10.004367+0000","last_unstale":"2024-07-27T06:45:09.862220+0000","last_undegraded":"2024-07-27T06:45:09.862220+0000","last_fullsized":"2024-07-27T06:45:09.862220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:22:39.942391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862359+0000","last_change":"2024-07-27T06:43:12.796511+0000","last_active":"2024-07-27T06:45:09.862359+0000","last_peered":"2024-07-27T06:45:09.862359+0000","last_clean":"2024-07-27T06:45:09.862359+0000","last_became_active":"2024-07-27T06:43:12.796152+0000","last_became_peered":"2024-07-27T06:43:12.796152+0000","last_unstale":"2024-07-27T06:45:09.862359+0000","last_undegraded":"2024-07-27T06:45:09.862359+0000","last_fullsized":"2024-07-27T06:45:09.862359+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:03:02.508990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822077+0000","last_change":"2024-07-27T06:43:10.533656+0000","last_active":"2024-07-27T06:45:09.822077+0000","last_peered":"2024-07-27T06:45:09.822077+0000","last_clean":"2024-07-27T06:45:09.822077+0000","last_became_active":"2024-07-27T06:43:10.533295+0000","last_became_peered":"2024-07-27T06:43:10.533295+0000","last_unstale":"2024-07-27T06:45:09.822077+0000","last_undegraded":"2024-07-27T06:45:09.822077+0000","last_fullsized":"2024-07-27T06:45:09.822077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:11:00.032511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485166+0000","last_change":"2024-07-27T06:43:12.795106+0000","last_active":"2024-07-27T06:45:09.485166+0000","last_peered":"2024-07-27T06:45:09.485166+0000","last_clean":"2024-07-27T06:45:09.485166+0000","last_became_active":"2024-07-27T06:43:12.794729+0000","last_became_peered":"2024-07-27T06:43:12.794729+0000","last_unstale":"2024-07-27T06:45:09.485166+0000","last_undegraded":"2024-07-27T06:45:09.485166+0000","last_fullsized":"2024-07-27T06:45:09.485166+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:19:16.542363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485673+0000","last_change":"2024-07-27T06:43:09.770016+0000","last_active":"2024-07-27T06:45:09.485673+0000","last_peered":"2024-07-27T06:45:09.485673+0000","last_clean":"2024-07-27T06:45:09.485673+0000","last_became_active":"2024-07-27T06:43:09.769672+0000","last_became_peered":"2024-07-27T06:43:09.769672+0000","last_unstale":"2024-07-27T06:45:09.485673+0000","last_undegraded":"2024-07-27T06:45:09.485673+0000","last_fullsized":"2024-07-27T06:45:09.485673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:30:55.737494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483331+0000","last_change":"2024-07-27T06:43:12.793690+0000","last_active":"2024-07-27T06:45:09.483331+0000","last_peered":"2024-07-27T06:45:09.483331+0000","last_clean":"2024-07-27T06:45:09.483331+0000","last_became_active":"2024-07-27T06:43:12.793352+0000","last_became_peered":"2024-07-27T06:43:12.793352+0000","last_unstale":"2024-07-27T06:45:09.483331+0000","last_undegraded":"2024-07-27T06:45:09.483331+0000","last_fullsized":"2024-07-27T06:45:09.483331+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:22:47.606990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.491196+0000","last_change":"2024-07-27T06:45:09.490738+0000","last_active":"2024-07-27T06:45:09.491196+0000","last_peered":"2024-07-27T06:45:09.491196+0000","last_clean":"2024-07-27T06:45:09.491196+0000","last_became_active":"2024-07-27T06:45:09.487147+0000","last_became_peered":"2024-07-27T06:45:09.487147+0000","last_unstale":"2024-07-27T06:45:09.491196+0000","last_undegraded":"2024-07-27T06:45:09.491196+0000","last_fullsized":"2024-07-27T06:45:09.491196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:46.732987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485502+0000","last_change":"2024-07-27T06:43:12.792648+0000","last_active":"2024-07-27T06:45:09.485502+0000","last_peered":"2024-07-27T06:45:09.485502+0000","last_clean":"2024-07-27T06:45:09.485502+0000","last_became_active":"2024-07-27T06:43:12.792338+0000","last_became_peered":"2024-07-27T06:43:12.792338+0000","last_unstale":"2024-07-27T06:45:09.485502+0000","last_undegraded":"2024-07-27T06:45:09.485502+0000","last_fullsized":"2024-07-27T06:45:09.485502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:36:52.605957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.827215+0000","last_change":"2024-07-27T06:45:09.827140+0000","last_active":"2024-07-27T06:45:09.827215+0000","last_peered":"2024-07-27T06:45:09.827215+0000","last_clean":"2024-07-27T06:45:09.827215+0000","last_became_active":"2024-07-27T06:45:09.823119+0000","last_became_peered":"2024-07-27T06:45:09.823119+0000","last_unstale":"2024-07-27T06:45:09.827215+0000","last_undegraded":"2024-07-27T06:45:09.827215+0000","last_fullsized":"2024-07-27T06:45:09.827215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:32:19.682193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488730+0000","last_change":"2024-07-27T06:43:12.809347+0000","last_active":"2024-07-27T06:45:09.488730+0000","last_peered":"2024-07-27T06:45:09.488730+0000","last_clean":"2024-07-27T06:45:09.488730+0000","last_became_active":"2024-07-27T06:43:12.808980+0000","last_became_peered":"2024-07-27T06:43:12.808980+0000","last_unstale":"2024-07-27T06:45:09.488730+0000","last_undegraded":"2024-07-27T06:45:09.488730+0000","last_fullsized":"2024-07-27T06:45:09.488730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:05:14.132641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633228+0000","last_change":"2024-07-27T06:44:09.237908+0000","last_active":"2024-07-27T06:45:09.633228+0000","last_peered":"2024-07-27T06:45:09.633228+0000","last_clean":"2024-07-27T06:45:09.633228+0000","last_became_active":"2024-07-27T06:44:09.237503+0000","last_became_peered":"2024-07-27T06:44:09.237503+0000","last_unstale":"2024-07-27T06:45:09.633228+0000","last_undegraded":"2024-07-27T06:45:09.633228+0000","last_fullsized":"2024-07-27T06:45:09.633228+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:29.654244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486331+0000","last_change":"2024-07-27T06:43:09.776000+0000","last_active":"2024-07-27T06:45:09.486331+0000","last_peered":"2024-07-27T06:45:09.486331+0000","last_clean":"2024-07-27T06:45:09.486331+0000","last_became_active":"2024-07-27T06:43:09.775522+0000","last_became_peered":"2024-07-27T06:43:09.775522+0000","last_unstale":"2024-07-27T06:45:09.486331+0000","last_undegraded":"2024-07-27T06:45:09.486331+0000","last_fullsized":"2024-07-27T06:45:09.486331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:39:42.650177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822023+0000","last_change":"2024-07-27T06:43:12.807062+0000","last_active":"2024-07-27T06:45:09.822023+0000","last_peered":"2024-07-27T06:45:09.822023+0000","last_clean":"2024-07-27T06:45:09.822023+0000","last_became_active":"2024-07-27T06:43:12.805816+0000","last_became_peered":"2024-07-27T06:43:12.805816+0000","last_unstale":"2024-07-27T06:45:09.822023+0000","last_undegraded":"2024-07-27T06:45:09.822023+0000","last_fullsized":"2024-07-27T06:45:09.822023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:44:16.261663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483406+0000","last_change":"2024-07-27T06:44:09.250070+0000","last_active":"2024-07-27T06:45:09.483406+0000","last_peered":"2024-07-27T06:45:09.483406+0000","last_clean":"2024-07-27T06:45:09.483406+0000","last_became_active":"2024-07-27T06:44:09.237290+0000","last_became_peered":"2024-07-27T06:44:09.237290+0000","last_unstale":"2024-07-27T06:45:09.483406+0000","last_undegraded":"2024-07-27T06:45:09.483406+0000","last_fullsized":"2024-07-27T06:45:09.483406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:51.590487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862423+0000","last_change":"2024-07-27T06:43:12.809135+0000","last_active":"2024-07-27T06:45:09.862423+0000","last_peered":"2024-07-27T06:45:09.862423+0000","last_clean":"2024-07-27T06:45:09.862423+0000","last_became_active":"2024-07-27T06:43:12.806731+0000","last_became_peered":"2024-07-27T06:43:12.806731+0000","last_unstale":"2024-07-27T06:45:09.862423+0000","last_undegraded":"2024-07-27T06:45:09.862423+0000","last_fullsized":"2024-07-27T06:45:09.862423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:47:50.397517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486050+0000","last_change":"2024-07-27T06:43:10.533608+0000","last_active":"2024-07-27T06:45:09.486050+0000","last_peered":"2024-07-27T06:45:09.486050+0000","last_clean":"2024-07-27T06:45:09.486050+0000","last_became_active":"2024-07-27T06:43:10.532663+0000","last_became_peered":"2024-07-27T06:43:10.532663+0000","last_unstale":"2024-07-27T06:45:09.486050+0000","last_undegraded":"2024-07-27T06:45:09.486050+0000","last_fullsized":"2024-07-27T06:45:09.486050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:05:33.155427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485883+0000","last_change":"2024-07-27T06:43:12.789890+0000","last_active":"2024-07-27T06:45:09.485883+0000","last_peered":"2024-07-27T06:45:09.485883+0000","last_clean":"2024-07-27T06:45:09.485883+0000","last_became_active":"2024-07-27T06:43:12.789561+0000","last_became_peered":"2024-07-27T06:43:12.789561+0000","last_unstale":"2024-07-27T06:45:09.485883+0000","last_undegraded":"2024-07-27T06:45:09.485883+0000","last_fullsized":"2024-07-27T06:45:09.485883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:35:49.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"81'14","reported_seq":156,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:21.065059+0000","last_change":"2024-07-27T06:43:09.773471+0000","last_active":"2024-07-27T06:45:21.065059+0000","last_peered":"2024-07-27T06:45:21.065059+0000","last_clean":"2024-07-27T06:45:21.065059+0000","last_became_active":"2024-07-27T06:43:09.773213+0000","last_became_peered":"2024-07-27T06:43:09.773213+0000","last_unstale":"2024-07-27T06:45:21.065059+0000","last_undegraded":"2024-07-27T06:45:21.065059+0000","last_fullsized":"2024-07-27T06:45:21.065059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:54:38.770198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":110,"num_read_kb":110,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68614,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1089,"num_read_kb":640,"num_write":257,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1561,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":205842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283116,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678058,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":461919,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874286,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":1113012,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037813,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":937984,"data_stored":461317,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234041,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":461695,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430269,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":987136,"data_stored":1054886,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593795,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":461566,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790024,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":1054032,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953549,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":929792,"data_stored":462597,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":460859,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378712,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":522125,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542236,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":983040,"data_stored":523075,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705761,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":462905,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":61930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T06:45:26.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph pg dump --format=json 2024-07-27T06:45:26.400 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:26 smithi029 ceph-mon[32610]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:45:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:26 smithi077 ceph-mon[37153]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:45:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:26 smithi047 ceph-mon[36835]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:45:27.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:27 smithi029 ceph-mon[32610]: pgmap v350: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:27 smithi077 ceph-mon[37153]: pgmap v350: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:27 smithi047 ceph-mon[36835]: pgmap v350: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:28.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.150+0000 7fef7eae2640 1 -- 172.21.15.29:0/1351241390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef781024a0 msgr2=0x7fef781028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:28.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.150+0000 7fef7eae2640 1 --2- 172.21.15.29:0/1351241390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef781024a0 0x7fef781028b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fef70009ca0 tx=0x7fef7002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:45:28.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.151+0000 7fef7eae2640 1 -- 172.21.15.29:0/1351241390 shutdown_connections 2024-07-27T06:45:28.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.151+0000 7fef7eae2640 1 --2- 172.21.15.29:0/1351241390 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef78108730 0x7fef7810ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:28.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.151+0000 7fef7eae2640 1 --2- 172.21.15.29:0/1351241390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef78107de0 0x7fef781081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:28.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.151+0000 7fef7eae2640 1 --2- 172.21.15.29:0/1351241390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef781024a0 0x7fef781028b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:28.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.151+0000 7fef7eae2640 1 -- 172.21.15.29:0/1351241390 >> 172.21.15.29:0/1351241390 conn(0x7fef78079140 msgr2=0x7fef78079550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:28.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.151+0000 7fef7eae2640 1 -- 172.21.15.29:0/1351241390 shutdown_connections 2024-07-27T06:45:28.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.151+0000 7fef7eae2640 1 -- 172.21.15.29:0/1351241390 wait complete. 2024-07-27T06:45:28.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.151+0000 7fef7eae2640 1 Processor -- start 2024-07-27T06:45:28.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.151+0000 7fef7eae2640 1 -- start start 2024-07-27T06:45:28.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7eae2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef781024a0 0x7fef78199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:28.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7eae2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef78107de0 0x7fef7819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:28.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7eae2640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef78108730 0x7fef7819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:28.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7eae2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fef78112400 con 0x7fef781024a0 2024-07-27T06:45:28.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7eae2640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fef78112280 con 0x7fef78108730 2024-07-27T06:45:28.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7eae2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fef78112580 con 0x7fef78107de0 2024-07-27T06:45:28.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7c87e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef781024a0 0x7fef78199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:28.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7c87e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef781024a0 0x7fef78199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38092/0 (socket says 172.21.15.29:38092) 2024-07-27T06:45:28.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7c87e640 1 -- 172.21.15.29:0/3927699988 learned_addr learned my addr 172.21.15.29:0/3927699988 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:28.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef77fff640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef78107de0 0x7fef7819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:28.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7d07f640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef78108730 0x7fef7819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:28.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7c87e640 1 -- 172.21.15.29:0/3927699988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef78107de0 msgr2=0x7fef7819a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:28.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7c87e640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef78107de0 0x7fef7819a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:28.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7c87e640 1 -- 172.21.15.29:0/3927699988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef78108730 msgr2=0x7fef7819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:28.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7c87e640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef78108730 0x7fef7819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:28.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7c87e640 1 -- 172.21.15.29:0/3927699988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7819fe90 con 0x7fef781024a0 2024-07-27T06:45:28.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef77fff640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef78107de0 0x7fef7819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:45:28.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7d07f640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef78108730 0x7fef7819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:45:28.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7c87e640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef781024a0 0x7fef78199f20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fef70006010 tx=0x7fef700319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:28.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef75ffb640 1 -- 172.21.15.29:0/3927699988 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70003bf0 con 0x7fef781024a0 2024-07-27T06:45:28.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef7eae2640 1 -- 172.21.15.29:0/3927699988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef781a0120 con 0x7fef781024a0 2024-07-27T06:45:28.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef75ffb640 1 -- 172.21.15.29:0/3927699988 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef70003d90 con 0x7fef781024a0 2024-07-27T06:45:28.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.152+0000 7fef75ffb640 1 -- 172.21.15.29:0/3927699988 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7002fc20 con 0x7fef781024a0 2024-07-27T06:45:28.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.153+0000 7fef7eae2640 1 -- 172.21.15.29:0/3927699988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef781a68e0 con 0x7fef781024a0 2024-07-27T06:45:28.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.154+0000 7fef7eae2640 1 -- 172.21.15.29:0/3927699988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef78103720 con 0x7fef781024a0 2024-07-27T06:45:28.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.155+0000 7fef75ffb640 1 -- 172.21.15.29:0/3927699988 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fef700046a0 con 0x7fef781024a0 2024-07-27T06:45:28.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.156+0000 7fef75ffb640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fef640b3c20 0x7fef640b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:28.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.156+0000 7fef77fff640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fef640b3c20 0x7fef640b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:28.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.156+0000 7fef75ffb640 1 -- 172.21.15.29:0/3927699988 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fef70105550 con 0x7fef781024a0 2024-07-27T06:45:28.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.156+0000 7fef77fff640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fef640b3c20 0x7fef640b6130 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fef68005fd0 tx=0x7fef68005ea0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:28.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.159+0000 7fef75ffb640 1 -- 172.21.15.29:0/3927699988 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fef700cca90 con 0x7fef781024a0 2024-07-27T06:45:28.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.339+0000 7fef7eae2640 1 -- 172.21.15.29:0/3927699988 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fef78063230 con 0x7fef640b3c20 2024-07-27T06:45:28.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.344+0000 7fef75ffb640 1 -- 172.21.15.29:0/3927699988 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259150 (secure 0 0 0) 0x7fef78063230 con 0x7fef640b3c20 2024-07-27T06:45:28.348 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:45:28.349 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-07-27T06:45:28.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.347+0000 7fef7eae2640 1 -- 172.21.15.29:0/3927699988 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fef640b3c20 msgr2=0x7fef640b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:28.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.347+0000 7fef7eae2640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fef640b3c20 0x7fef640b6130 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fef68005fd0 tx=0x7fef68005ea0 comp rx=0 tx=0).stop 2024-07-27T06:45:28.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.347+0000 7fef7eae2640 1 -- 172.21.15.29:0/3927699988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef781024a0 msgr2=0x7fef78199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:28.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.347+0000 7fef7eae2640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef781024a0 0x7fef78199f20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fef70006010 tx=0x7fef700319f0 comp rx=0 tx=0).stop 2024-07-27T06:45:28.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.347+0000 7fef7eae2640 1 -- 172.21.15.29:0/3927699988 shutdown_connections 2024-07-27T06:45:28.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.347+0000 7fef7eae2640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef78108730 0x7fef7819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:28.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.347+0000 7fef7eae2640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fef640b3c20 0x7fef640b6130 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:28.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.347+0000 7fef7eae2640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fef78107de0 0x7fef7819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:28.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.347+0000 7fef7eae2640 1 --2- 172.21.15.29:0/3927699988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef781024a0 0x7fef78199f20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:28.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.347+0000 7fef7eae2640 1 -- 172.21.15.29:0/3927699988 >> 172.21.15.29:0/3927699988 conn(0x7fef78079140 msgr2=0x7fef78101510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:28.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.347+0000 7fef7eae2640 1 -- 172.21.15.29:0/3927699988 shutdown_connections 2024-07-27T06:45:28.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:28.347+0000 7fef7eae2640 1 -- 172.21.15.29:0/3927699988 wait complete. 2024-07-27T06:45:29.109 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-07-27T06:45:27.657414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659216,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1220,"num_read_kb":734,"num_write":382,"num_write_kb":2298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2953401,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":815,"ondisk_log_size":815,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360496,"kb_used_data":11280,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581328,"statfs":{"total":1151940427776,"available":1151571279872,"internally_reserved":0,"allocated":11550720,"data_stored":7499988,"data_compressed":247335,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":145,"num_read_kb":75,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003723"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483739+0000","last_change":"2024-07-27T06:43:12.802353+0000","last_active":"2024-07-27T06:45:09.483739+0000","last_peered":"2024-07-27T06:45:09.483739+0000","last_clean":"2024-07-27T06:45:09.483739+0000","last_became_active":"2024-07-27T06:43:12.802028+0000","last_became_peered":"2024-07-27T06:43:12.802028+0000","last_unstale":"2024-07-27T06:45:09.483739+0000","last_undegraded":"2024-07-27T06:45:09.483739+0000","last_fullsized":"2024-07-27T06:45:09.483739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:22:24.292567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862408+0000","last_change":"2024-07-27T06:43:12.806269+0000","last_active":"2024-07-27T06:45:09.862408+0000","last_peered":"2024-07-27T06:45:09.862408+0000","last_clean":"2024-07-27T06:45:09.862408+0000","last_became_active":"2024-07-27T06:43:12.803232+0000","last_became_peered":"2024-07-27T06:43:12.803232+0000","last_unstale":"2024-07-27T06:45:09.862408+0000","last_undegraded":"2024-07-27T06:45:09.862408+0000","last_fullsized":"2024-07-27T06:45:09.862408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:57:53.673850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862259+0000","last_change":"2024-07-27T06:43:12.805542+0000","last_active":"2024-07-27T06:45:09.862259+0000","last_peered":"2024-07-27T06:45:09.862259+0000","last_clean":"2024-07-27T06:45:09.862259+0000","last_became_active":"2024-07-27T06:43:12.804951+0000","last_became_peered":"2024-07-27T06:43:12.804951+0000","last_unstale":"2024-07-27T06:45:09.862259+0000","last_undegraded":"2024-07-27T06:45:09.862259+0000","last_fullsized":"2024-07-27T06:45:09.862259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:06:40.469695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862158+0000","last_change":"2024-07-27T06:43:12.796950+0000","last_active":"2024-07-27T06:45:09.862158+0000","last_peered":"2024-07-27T06:45:09.862158+0000","last_clean":"2024-07-27T06:45:09.862158+0000","last_became_active":"2024-07-27T06:43:12.796549+0000","last_became_peered":"2024-07-27T06:43:12.796549+0000","last_unstale":"2024-07-27T06:45:09.862158+0000","last_undegraded":"2024-07-27T06:45:09.862158+0000","last_fullsized":"2024-07-27T06:45:09.862158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:30:36.391046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485939+0000","last_change":"2024-07-27T06:43:12.799838+0000","last_active":"2024-07-27T06:45:09.485939+0000","last_peered":"2024-07-27T06:45:09.485939+0000","last_clean":"2024-07-27T06:45:09.485939+0000","last_became_active":"2024-07-27T06:43:12.799462+0000","last_became_peered":"2024-07-27T06:43:12.799462+0000","last_unstale":"2024-07-27T06:45:09.485939+0000","last_undegraded":"2024-07-27T06:45:09.485939+0000","last_fullsized":"2024-07-27T06:45:09.485939+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:45:19.371259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481556+0000","last_change":"2024-07-27T06:43:12.797187+0000","last_active":"2024-07-27T06:45:09.481556+0000","last_peered":"2024-07-27T06:45:09.481556+0000","last_clean":"2024-07-27T06:45:09.481556+0000","last_became_active":"2024-07-27T06:43:12.796633+0000","last_became_peered":"2024-07-27T06:43:12.796633+0000","last_unstale":"2024-07-27T06:45:09.481556+0000","last_undegraded":"2024-07-27T06:45:09.481556+0000","last_fullsized":"2024-07-27T06:45:09.481556+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:22:20.882899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488365+0000","last_change":"2024-07-27T06:43:12.793786+0000","last_active":"2024-07-27T06:45:09.488365+0000","last_peered":"2024-07-27T06:45:09.488365+0000","last_clean":"2024-07-27T06:45:09.488365+0000","last_became_active":"2024-07-27T06:43:12.793413+0000","last_became_peered":"2024-07-27T06:43:12.793413+0000","last_unstale":"2024-07-27T06:45:09.488365+0000","last_undegraded":"2024-07-27T06:45:09.488365+0000","last_fullsized":"2024-07-27T06:45:09.488365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:14.015373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485304+0000","last_change":"2024-07-27T06:43:12.791304+0000","last_active":"2024-07-27T06:45:09.485304+0000","last_peered":"2024-07-27T06:45:09.485304+0000","last_clean":"2024-07-27T06:45:09.485304+0000","last_became_active":"2024-07-27T06:43:12.790801+0000","last_became_peered":"2024-07-27T06:43:12.790801+0000","last_unstale":"2024-07-27T06:45:09.485304+0000","last_undegraded":"2024-07-27T06:45:09.485304+0000","last_fullsized":"2024-07-27T06:45:09.485304+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:12:37.606748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822470+0000","last_change":"2024-07-27T06:43:12.797778+0000","last_active":"2024-07-27T06:45:09.822470+0000","last_peered":"2024-07-27T06:45:09.822470+0000","last_clean":"2024-07-27T06:45:09.822470+0000","last_became_active":"2024-07-27T06:43:12.796784+0000","last_became_peered":"2024-07-27T06:43:12.796784+0000","last_unstale":"2024-07-27T06:45:09.822470+0000","last_undegraded":"2024-07-27T06:45:09.822470+0000","last_fullsized":"2024-07-27T06:45:09.822470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:52:22.275282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.482052+0000","last_change":"2024-07-27T06:43:12.798970+0000","last_active":"2024-07-27T06:45:09.482052+0000","last_peered":"2024-07-27T06:45:09.482052+0000","last_clean":"2024-07-27T06:45:09.482052+0000","last_became_active":"2024-07-27T06:43:12.798549+0000","last_became_peered":"2024-07-27T06:43:12.798549+0000","last_unstale":"2024-07-27T06:45:09.482052+0000","last_undegraded":"2024-07-27T06:45:09.482052+0000","last_fullsized":"2024-07-27T06:45:09.482052+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:58:32.464429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488545+0000","last_change":"2024-07-27T06:43:12.809405+0000","last_active":"2024-07-27T06:45:09.488545+0000","last_peered":"2024-07-27T06:45:09.488545+0000","last_clean":"2024-07-27T06:45:09.488545+0000","last_became_active":"2024-07-27T06:43:12.809030+0000","last_became_peered":"2024-07-27T06:43:12.809030+0000","last_unstale":"2024-07-27T06:45:09.488545+0000","last_undegraded":"2024-07-27T06:45:09.488545+0000","last_fullsized":"2024-07-27T06:45:09.488545+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:46:35.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862145+0000","last_change":"2024-07-27T06:43:12.802647+0000","last_active":"2024-07-27T06:45:09.862145+0000","last_peered":"2024-07-27T06:45:09.862145+0000","last_clean":"2024-07-27T06:45:09.862145+0000","last_became_active":"2024-07-27T06:43:12.802247+0000","last_became_peered":"2024-07-27T06:43:12.802247+0000","last_unstale":"2024-07-27T06:45:09.862145+0000","last_undegraded":"2024-07-27T06:45:09.862145+0000","last_fullsized":"2024-07-27T06:45:09.862145+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:44:24.564852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481567+0000","last_change":"2024-07-27T06:43:12.809140+0000","last_active":"2024-07-27T06:45:09.481567+0000","last_peered":"2024-07-27T06:45:09.481567+0000","last_clean":"2024-07-27T06:45:09.481567+0000","last_became_active":"2024-07-27T06:43:12.808692+0000","last_became_peered":"2024-07-27T06:43:12.808692+0000","last_unstale":"2024-07-27T06:45:09.481567+0000","last_undegraded":"2024-07-27T06:45:09.481567+0000","last_fullsized":"2024-07-27T06:45:09.481567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:37:45.326234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862160+0000","last_change":"2024-07-27T06:43:12.802838+0000","last_active":"2024-07-27T06:45:09.862160+0000","last_peered":"2024-07-27T06:45:09.862160+0000","last_clean":"2024-07-27T06:45:09.862160+0000","last_became_active":"2024-07-27T06:43:12.802503+0000","last_became_peered":"2024-07-27T06:43:12.802503+0000","last_unstale":"2024-07-27T06:45:09.862160+0000","last_undegraded":"2024-07-27T06:45:09.862160+0000","last_fullsized":"2024-07-27T06:45:09.862160+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:41:03.428162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862037+0000","last_change":"2024-07-27T06:43:12.801626+0000","last_active":"2024-07-27T06:45:09.862037+0000","last_peered":"2024-07-27T06:45:09.862037+0000","last_clean":"2024-07-27T06:45:09.862037+0000","last_became_active":"2024-07-27T06:43:12.801266+0000","last_became_peered":"2024-07-27T06:43:12.801266+0000","last_unstale":"2024-07-27T06:45:09.862037+0000","last_undegraded":"2024-07-27T06:45:09.862037+0000","last_fullsized":"2024-07-27T06:45:09.862037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:54:31.934223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.821914+0000","last_change":"2024-07-27T06:43:12.804543+0000","last_active":"2024-07-27T06:45:09.821914+0000","last_peered":"2024-07-27T06:45:09.821914+0000","last_clean":"2024-07-27T06:45:09.821914+0000","last_became_active":"2024-07-27T06:43:12.804173+0000","last_became_peered":"2024-07-27T06:43:12.804173+0000","last_unstale":"2024-07-27T06:45:09.821914+0000","last_undegraded":"2024-07-27T06:45:09.821914+0000","last_fullsized":"2024-07-27T06:45:09.821914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:18:25.337333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483727+0000","last_change":"2024-07-27T06:43:12.798604+0000","last_active":"2024-07-27T06:45:09.483727+0000","last_peered":"2024-07-27T06:45:09.483727+0000","last_clean":"2024-07-27T06:45:09.483727+0000","last_became_active":"2024-07-27T06:43:12.798228+0000","last_became_peered":"2024-07-27T06:43:12.798228+0000","last_unstale":"2024-07-27T06:45:09.483727+0000","last_undegraded":"2024-07-27T06:45:09.483727+0000","last_fullsized":"2024-07-27T06:45:09.483727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:04:35.851399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633300+0000","last_change":"2024-07-27T06:43:12.800452+0000","last_active":"2024-07-27T06:45:09.633300+0000","last_peered":"2024-07-27T06:45:09.633300+0000","last_clean":"2024-07-27T06:45:09.633300+0000","last_became_active":"2024-07-27T06:43:12.800084+0000","last_became_peered":"2024-07-27T06:43:12.800084+0000","last_unstale":"2024-07-27T06:45:09.633300+0000","last_undegraded":"2024-07-27T06:45:09.633300+0000","last_fullsized":"2024-07-27T06:45:09.633300+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:54:27.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485716+0000","last_change":"2024-07-27T06:43:12.800761+0000","last_active":"2024-07-27T06:45:09.485716+0000","last_peered":"2024-07-27T06:45:09.485716+0000","last_clean":"2024-07-27T06:45:09.485716+0000","last_became_active":"2024-07-27T06:43:12.800414+0000","last_became_peered":"2024-07-27T06:43:12.800414+0000","last_unstale":"2024-07-27T06:45:09.485716+0000","last_undegraded":"2024-07-27T06:45:09.485716+0000","last_fullsized":"2024-07-27T06:45:09.485716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:02:47.247522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483308+0000","last_change":"2024-07-27T06:43:12.795470+0000","last_active":"2024-07-27T06:45:09.483308+0000","last_peered":"2024-07-27T06:45:09.483308+0000","last_clean":"2024-07-27T06:45:09.483308+0000","last_became_active":"2024-07-27T06:43:12.795136+0000","last_became_peered":"2024-07-27T06:43:12.795136+0000","last_unstale":"2024-07-27T06:45:09.483308+0000","last_undegraded":"2024-07-27T06:45:09.483308+0000","last_fullsized":"2024-07-27T06:45:09.483308+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:01:22.178699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481693+0000","last_change":"2024-07-27T06:43:12.803897+0000","last_active":"2024-07-27T06:45:09.481693+0000","last_peered":"2024-07-27T06:45:09.481693+0000","last_clean":"2024-07-27T06:45:09.481693+0000","last_became_active":"2024-07-27T06:43:12.803546+0000","last_became_peered":"2024-07-27T06:43:12.803546+0000","last_unstale":"2024-07-27T06:45:09.481693+0000","last_undegraded":"2024-07-27T06:45:09.481693+0000","last_fullsized":"2024-07-27T06:45:09.481693+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:13:50.816831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822134+0000","last_change":"2024-07-27T06:44:09.238514+0000","last_active":"2024-07-27T06:45:09.822134+0000","last_peered":"2024-07-27T06:45:09.822134+0000","last_clean":"2024-07-27T06:45:09.822134+0000","last_became_active":"2024-07-27T06:44:09.237903+0000","last_became_peered":"2024-07-27T06:44:09.237903+0000","last_unstale":"2024-07-27T06:45:09.822134+0000","last_undegraded":"2024-07-27T06:45:09.822134+0000","last_fullsized":"2024-07-27T06:45:09.822134+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:20:59.887491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485364+0000","last_change":"2024-07-27T06:43:12.788570+0000","last_active":"2024-07-27T06:45:09.485364+0000","last_peered":"2024-07-27T06:45:09.485364+0000","last_clean":"2024-07-27T06:45:09.485364+0000","last_became_active":"2024-07-27T06:43:12.788199+0000","last_became_peered":"2024-07-27T06:43:12.788199+0000","last_unstale":"2024-07-27T06:45:09.485364+0000","last_undegraded":"2024-07-27T06:45:09.485364+0000","last_fullsized":"2024-07-27T06:45:09.485364+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:44:26.751094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862750+0000","last_change":"2024-07-27T06:43:12.792866+0000","last_active":"2024-07-27T06:45:09.862750+0000","last_peered":"2024-07-27T06:45:09.862750+0000","last_clean":"2024-07-27T06:45:09.862750+0000","last_became_active":"2024-07-27T06:43:12.792389+0000","last_became_peered":"2024-07-27T06:43:12.792389+0000","last_unstale":"2024-07-27T06:45:09.862750+0000","last_undegraded":"2024-07-27T06:45:09.862750+0000","last_fullsized":"2024-07-27T06:45:09.862750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:10:37.367642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485830+0000","last_change":"2024-07-27T06:43:12.797002+0000","last_active":"2024-07-27T06:45:09.485830+0000","last_peered":"2024-07-27T06:45:09.485830+0000","last_clean":"2024-07-27T06:45:09.485830+0000","last_became_active":"2024-07-27T06:43:12.796651+0000","last_became_peered":"2024-07-27T06:43:12.796651+0000","last_unstale":"2024-07-27T06:45:09.485830+0000","last_undegraded":"2024-07-27T06:45:09.485830+0000","last_fullsized":"2024-07-27T06:45:09.485830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:28:06.934042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485867+0000","last_change":"2024-07-27T06:43:12.798549+0000","last_active":"2024-07-27T06:45:09.485867+0000","last_peered":"2024-07-27T06:45:09.485867+0000","last_clean":"2024-07-27T06:45:09.485867+0000","last_became_active":"2024-07-27T06:43:12.798201+0000","last_became_peered":"2024-07-27T06:43:12.798201+0000","last_unstale":"2024-07-27T06:45:09.485867+0000","last_undegraded":"2024-07-27T06:45:09.485867+0000","last_fullsized":"2024-07-27T06:45:09.485867+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:59:47.525536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483656+0000","last_change":"2024-07-27T06:43:12.791053+0000","last_active":"2024-07-27T06:45:09.483656+0000","last_peered":"2024-07-27T06:45:09.483656+0000","last_clean":"2024-07-27T06:45:09.483656+0000","last_became_active":"2024-07-27T06:43:12.790718+0000","last_became_peered":"2024-07-27T06:43:12.790718+0000","last_unstale":"2024-07-27T06:45:09.483656+0000","last_undegraded":"2024-07-27T06:45:09.483656+0000","last_fullsized":"2024-07-27T06:45:09.483656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:09.455399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481811+0000","last_change":"2024-07-27T06:43:12.804180+0000","last_active":"2024-07-27T06:45:09.481811+0000","last_peered":"2024-07-27T06:45:09.481811+0000","last_clean":"2024-07-27T06:45:09.481811+0000","last_became_active":"2024-07-27T06:43:12.803881+0000","last_became_peered":"2024-07-27T06:43:12.803881+0000","last_unstale":"2024-07-27T06:45:09.481811+0000","last_undegraded":"2024-07-27T06:45:09.481811+0000","last_fullsized":"2024-07-27T06:45:09.481811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:01:05.009233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862727+0000","last_change":"2024-07-27T06:43:12.809150+0000","last_active":"2024-07-27T06:45:09.862727+0000","last_peered":"2024-07-27T06:45:09.862727+0000","last_clean":"2024-07-27T06:45:09.862727+0000","last_became_active":"2024-07-27T06:43:12.808428+0000","last_became_peered":"2024-07-27T06:43:12.808428+0000","last_unstale":"2024-07-27T06:45:09.862727+0000","last_undegraded":"2024-07-27T06:45:09.862727+0000","last_fullsized":"2024-07-27T06:45:09.862727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:07:13.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486325+0000","last_change":"2024-07-27T06:44:09.233785+0000","last_active":"2024-07-27T06:45:09.486325+0000","last_peered":"2024-07-27T06:45:09.486325+0000","last_clean":"2024-07-27T06:45:09.486325+0000","last_became_active":"2024-07-27T06:44:09.233184+0000","last_became_peered":"2024-07-27T06:44:09.233184+0000","last_unstale":"2024-07-27T06:45:09.486325+0000","last_undegraded":"2024-07-27T06:45:09.486325+0000","last_fullsized":"2024-07-27T06:45:09.486325+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:12:48.556383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486129+0000","last_change":"2024-07-27T06:44:09.320182+0000","last_active":"2024-07-27T06:45:09.486129+0000","last_peered":"2024-07-27T06:45:09.486129+0000","last_clean":"2024-07-27T06:45:09.486129+0000","last_became_active":"2024-07-27T06:44:09.319661+0000","last_became_peered":"2024-07-27T06:44:09.319661+0000","last_unstale":"2024-07-27T06:45:09.486129+0000","last_undegraded":"2024-07-27T06:45:09.486129+0000","last_fullsized":"2024-07-27T06:45:09.486129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:52:11.006680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483342+0000","last_change":"2024-07-27T06:43:12.787866+0000","last_active":"2024-07-27T06:45:09.483342+0000","last_peered":"2024-07-27T06:45:09.483342+0000","last_clean":"2024-07-27T06:45:09.483342+0000","last_became_active":"2024-07-27T06:43:12.787513+0000","last_became_peered":"2024-07-27T06:43:12.787513+0000","last_unstale":"2024-07-27T06:45:09.483342+0000","last_undegraded":"2024-07-27T06:45:09.483342+0000","last_fullsized":"2024-07-27T06:45:09.483342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:48:57.876588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485327+0000","last_change":"2024-07-27T06:43:12.791158+0000","last_active":"2024-07-27T06:45:09.485327+0000","last_peered":"2024-07-27T06:45:09.485327+0000","last_clean":"2024-07-27T06:45:09.485327+0000","last_became_active":"2024-07-27T06:43:12.790771+0000","last_became_peered":"2024-07-27T06:43:12.790771+0000","last_unstale":"2024-07-27T06:45:09.485327+0000","last_undegraded":"2024-07-27T06:45:09.485327+0000","last_fullsized":"2024-07-27T06:45:09.485327+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:10:18.739265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'7","reported_seq":153,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:23.060173+0000","last_change":"2024-07-27T06:43:09.769281+0000","last_active":"2024-07-27T06:45:23.060173+0000","last_peered":"2024-07-27T06:45:23.060173+0000","last_clean":"2024-07-27T06:45:23.060173+0000","last_became_active":"2024-07-27T06:43:09.768864+0000","last_became_peered":"2024-07-27T06:43:09.768864+0000","last_unstale":"2024-07-27T06:45:23.060173+0000","last_undegraded":"2024-07-27T06:45:23.060173+0000","last_fullsized":"2024-07-27T06:45:23.060173+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:55:14.191180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862898+0000","last_change":"2024-07-27T06:44:09.319930+0000","last_active":"2024-07-27T06:45:09.862898+0000","last_peered":"2024-07-27T06:45:09.862898+0000","last_clean":"2024-07-27T06:45:09.862898+0000","last_became_active":"2024-07-27T06:44:09.319535+0000","last_became_peered":"2024-07-27T06:44:09.319535+0000","last_unstale":"2024-07-27T06:45:09.862898+0000","last_undegraded":"2024-07-27T06:45:09.862898+0000","last_fullsized":"2024-07-27T06:45:09.862898+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:39:52.887860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481606+0000","last_change":"2024-07-27T06:43:09.772227+0000","last_active":"2024-07-27T06:45:09.481606+0000","last_peered":"2024-07-27T06:45:09.481606+0000","last_clean":"2024-07-27T06:45:09.481606+0000","last_became_active":"2024-07-27T06:43:09.771812+0000","last_became_peered":"2024-07-27T06:43:09.771812+0000","last_unstale":"2024-07-27T06:45:09.481606+0000","last_undegraded":"2024-07-27T06:45:09.481606+0000","last_fullsized":"2024-07-27T06:45:09.481606+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:27:28.054590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.480968+0000","last_change":"2024-07-27T06:43:09.766490+0000","last_active":"2024-07-27T06:45:09.480968+0000","last_peered":"2024-07-27T06:45:09.480968+0000","last_clean":"2024-07-27T06:45:09.480968+0000","last_became_active":"2024-07-27T06:43:09.765973+0000","last_became_peered":"2024-07-27T06:43:09.765973+0000","last_unstale":"2024-07-27T06:45:09.480968+0000","last_undegraded":"2024-07-27T06:45:09.480968+0000","last_fullsized":"2024-07-27T06:45:09.480968+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:00:02.913080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485390+0000","last_change":"2024-07-27T06:43:12.805234+0000","last_active":"2024-07-27T06:45:09.485390+0000","last_peered":"2024-07-27T06:45:09.485390+0000","last_clean":"2024-07-27T06:45:09.485390+0000","last_became_active":"2024-07-27T06:43:12.804908+0000","last_became_peered":"2024-07-27T06:43:12.804908+0000","last_unstale":"2024-07-27T06:45:09.485390+0000","last_undegraded":"2024-07-27T06:45:09.485390+0000","last_fullsized":"2024-07-27T06:45:09.485390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:09:34.988755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862144+0000","last_change":"2024-07-27T06:43:09.774201+0000","last_active":"2024-07-27T06:45:09.862144+0000","last_peered":"2024-07-27T06:45:09.862144+0000","last_clean":"2024-07-27T06:45:09.862144+0000","last_became_active":"2024-07-27T06:43:09.773817+0000","last_became_peered":"2024-07-27T06:43:09.773817+0000","last_unstale":"2024-07-27T06:45:09.862144+0000","last_undegraded":"2024-07-27T06:45:09.862144+0000","last_fullsized":"2024-07-27T06:45:09.862144+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:06.166251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633044+0000","last_change":"2024-07-27T06:43:12.800174+0000","last_active":"2024-07-27T06:45:09.633044+0000","last_peered":"2024-07-27T06:45:09.633044+0000","last_clean":"2024-07-27T06:45:09.633044+0000","last_became_active":"2024-07-27T06:43:12.799769+0000","last_became_peered":"2024-07-27T06:43:12.799769+0000","last_unstale":"2024-07-27T06:45:09.633044+0000","last_undegraded":"2024-07-27T06:45:09.633044+0000","last_fullsized":"2024-07-27T06:45:09.633044+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:43:06.890372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":264,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:26.408691+0000","last_change":"2024-07-27T06:43:09.776612+0000","last_active":"2024-07-27T06:45:26.408691+0000","last_peered":"2024-07-27T06:45:26.408691+0000","last_clean":"2024-07-27T06:45:26.408691+0000","last_became_active":"2024-07-27T06:43:09.775761+0000","last_became_peered":"2024-07-27T06:43:09.775761+0000","last_unstale":"2024-07-27T06:45:26.408691+0000","last_undegraded":"2024-07-27T06:45:26.408691+0000","last_fullsized":"2024-07-27T06:45:26.408691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:42:52.132357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":227,"num_read_kb":227,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822431+0000","last_change":"2024-07-27T06:43:12.797200+0000","last_active":"2024-07-27T06:45:09.822431+0000","last_peered":"2024-07-27T06:45:09.822431+0000","last_clean":"2024-07-27T06:45:09.822431+0000","last_became_active":"2024-07-27T06:43:12.796288+0000","last_became_peered":"2024-07-27T06:43:12.796288+0000","last_unstale":"2024-07-27T06:45:09.822431+0000","last_undegraded":"2024-07-27T06:45:09.822431+0000","last_fullsized":"2024-07-27T06:45:09.822431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:31:27.603883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488625+0000","last_change":"2024-07-27T06:43:09.772658+0000","last_active":"2024-07-27T06:45:09.488625+0000","last_peered":"2024-07-27T06:45:09.488625+0000","last_clean":"2024-07-27T06:45:09.488625+0000","last_became_active":"2024-07-27T06:43:09.772277+0000","last_became_peered":"2024-07-27T06:43:09.772277+0000","last_unstale":"2024-07-27T06:45:09.488625+0000","last_undegraded":"2024-07-27T06:45:09.488625+0000","last_fullsized":"2024-07-27T06:45:09.488625+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:09:16.854718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862494+0000","last_change":"2024-07-27T06:43:12.809643+0000","last_active":"2024-07-27T06:45:09.862494+0000","last_peered":"2024-07-27T06:45:09.862494+0000","last_clean":"2024-07-27T06:45:09.862494+0000","last_became_active":"2024-07-27T06:43:12.809020+0000","last_became_peered":"2024-07-27T06:43:12.809020+0000","last_unstale":"2024-07-27T06:45:09.862494+0000","last_undegraded":"2024-07-27T06:45:09.862494+0000","last_fullsized":"2024-07-27T06:45:09.862494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:08:03.062914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862958+0000","last_change":"2024-07-27T06:43:10.534042+0000","last_active":"2024-07-27T06:45:09.862958+0000","last_peered":"2024-07-27T06:45:09.862958+0000","last_clean":"2024-07-27T06:45:09.862958+0000","last_became_active":"2024-07-27T06:43:10.533615+0000","last_became_peered":"2024-07-27T06:43:10.533615+0000","last_unstale":"2024-07-27T06:45:09.862958+0000","last_undegraded":"2024-07-27T06:45:09.862958+0000","last_fullsized":"2024-07-27T06:45:09.862958+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:09:32.235633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822162+0000","last_change":"2024-07-27T06:43:12.798764+0000","last_active":"2024-07-27T06:45:09.822162+0000","last_peered":"2024-07-27T06:45:09.822162+0000","last_clean":"2024-07-27T06:45:09.822162+0000","last_became_active":"2024-07-27T06:43:12.797500+0000","last_became_peered":"2024-07-27T06:43:12.797500+0000","last_unstale":"2024-07-27T06:45:09.822162+0000","last_undegraded":"2024-07-27T06:45:09.822162+0000","last_fullsized":"2024-07-27T06:45:09.822162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:12:09.368711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'34","reported_seq":303,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:27.065702+0000","last_change":"2024-07-27T06:43:09.775954+0000","last_active":"2024-07-27T06:45:27.065702+0000","last_peered":"2024-07-27T06:45:27.065702+0000","last_clean":"2024-07-27T06:45:27.065702+0000","last_became_active":"2024-07-27T06:43:09.775498+0000","last_became_peered":"2024-07-27T06:43:09.775498+0000","last_unstale":"2024-07-27T06:45:27.065702+0000","last_undegraded":"2024-07-27T06:45:27.065702+0000","last_fullsized":"2024-07-27T06:45:27.065702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:08:46.965525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60596,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":238,"num_read_kb":178,"num_write":36,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486502+0000","last_change":"2024-07-27T06:43:12.797220+0000","last_active":"2024-07-27T06:45:09.486502+0000","last_peered":"2024-07-27T06:45:09.486502+0000","last_clean":"2024-07-27T06:45:09.486502+0000","last_became_active":"2024-07-27T06:43:12.796889+0000","last_became_peered":"2024-07-27T06:43:12.796889+0000","last_unstale":"2024-07-27T06:45:09.486502+0000","last_undegraded":"2024-07-27T06:45:09.486502+0000","last_fullsized":"2024-07-27T06:45:09.486502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:24:36.275738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485602+0000","last_change":"2024-07-27T06:43:09.776600+0000","last_active":"2024-07-27T06:45:09.485602+0000","last_peered":"2024-07-27T06:45:09.485602+0000","last_clean":"2024-07-27T06:45:09.485602+0000","last_became_active":"2024-07-27T06:43:09.776163+0000","last_became_peered":"2024-07-27T06:43:09.776163+0000","last_unstale":"2024-07-27T06:45:09.485602+0000","last_undegraded":"2024-07-27T06:45:09.485602+0000","last_fullsized":"2024-07-27T06:45:09.485602+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:22:15.947916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481153+0000","last_change":"2024-07-27T06:43:12.804984+0000","last_active":"2024-07-27T06:45:09.481153+0000","last_peered":"2024-07-27T06:45:09.481153+0000","last_clean":"2024-07-27T06:45:09.481153+0000","last_became_active":"2024-07-27T06:43:12.804644+0000","last_became_peered":"2024-07-27T06:43:12.804644+0000","last_unstale":"2024-07-27T06:45:09.481153+0000","last_undegraded":"2024-07-27T06:45:09.481153+0000","last_fullsized":"2024-07-27T06:45:09.481153+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.355981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862176+0000","last_change":"2024-07-27T06:43:09.772489+0000","last_active":"2024-07-27T06:45:09.862176+0000","last_peered":"2024-07-27T06:45:09.862176+0000","last_clean":"2024-07-27T06:45:09.862176+0000","last_became_active":"2024-07-27T06:43:09.772069+0000","last_became_peered":"2024-07-27T06:43:09.772069+0000","last_unstale":"2024-07-27T06:45:09.862176+0000","last_undegraded":"2024-07-27T06:45:09.862176+0000","last_fullsized":"2024-07-27T06:45:09.862176+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:15:30.549971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488522+0000","last_change":"2024-07-27T06:43:12.803376+0000","last_active":"2024-07-27T06:45:09.488522+0000","last_peered":"2024-07-27T06:45:09.488522+0000","last_clean":"2024-07-27T06:45:09.488522+0000","last_became_active":"2024-07-27T06:43:12.802961+0000","last_became_peered":"2024-07-27T06:43:12.802961+0000","last_unstale":"2024-07-27T06:45:09.488522+0000","last_undegraded":"2024-07-27T06:45:09.488522+0000","last_fullsized":"2024-07-27T06:45:09.488522+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:00:24.649877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":150,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:24.060866+0000","last_change":"2024-07-27T06:43:09.773182+0000","last_active":"2024-07-27T06:45:24.060866+0000","last_peered":"2024-07-27T06:45:24.060866+0000","last_clean":"2024-07-27T06:45:24.060866+0000","last_became_active":"2024-07-27T06:43:09.772126+0000","last_became_peered":"2024-07-27T06:43:09.772126+0000","last_unstale":"2024-07-27T06:45:24.060866+0000","last_undegraded":"2024-07-27T06:45:24.060866+0000","last_fullsized":"2024-07-27T06:45:24.060866+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:45:26.598799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488346+0000","last_change":"2024-07-27T06:43:12.799385+0000","last_active":"2024-07-27T06:45:09.488346+0000","last_peered":"2024-07-27T06:45:09.488346+0000","last_clean":"2024-07-27T06:45:09.488346+0000","last_became_active":"2024-07-27T06:43:12.798639+0000","last_became_peered":"2024-07-27T06:43:12.798639+0000","last_unstale":"2024-07-27T06:45:09.488346+0000","last_undegraded":"2024-07-27T06:45:09.488346+0000","last_fullsized":"2024-07-27T06:45:09.488346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:56:29.462491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485244+0000","last_change":"2024-07-27T06:43:10.535429+0000","last_active":"2024-07-27T06:45:09.485244+0000","last_peered":"2024-07-27T06:45:09.485244+0000","last_clean":"2024-07-27T06:45:09.485244+0000","last_became_active":"2024-07-27T06:43:10.534952+0000","last_became_peered":"2024-07-27T06:43:10.534952+0000","last_unstale":"2024-07-27T06:45:09.485244+0000","last_undegraded":"2024-07-27T06:45:09.485244+0000","last_fullsized":"2024-07-27T06:45:09.485244+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:54:20.018263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862639+0000","last_change":"2024-07-27T06:43:12.804266+0000","last_active":"2024-07-27T06:45:09.862639+0000","last_peered":"2024-07-27T06:45:09.862639+0000","last_clean":"2024-07-27T06:45:09.862639+0000","last_became_active":"2024-07-27T06:43:12.803888+0000","last_became_peered":"2024-07-27T06:43:12.803888+0000","last_unstale":"2024-07-27T06:45:09.862639+0000","last_undegraded":"2024-07-27T06:45:09.862639+0000","last_fullsized":"2024-07-27T06:45:09.862639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:48:00.490592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485827+0000","last_change":"2024-07-27T06:43:09.777656+0000","last_active":"2024-07-27T06:45:09.485827+0000","last_peered":"2024-07-27T06:45:09.485827+0000","last_clean":"2024-07-27T06:45:09.485827+0000","last_became_active":"2024-07-27T06:43:09.776620+0000","last_became_peered":"2024-07-27T06:43:09.776620+0000","last_unstale":"2024-07-27T06:45:09.485827+0000","last_undegraded":"2024-07-27T06:45:09.485827+0000","last_fullsized":"2024-07-27T06:45:09.485827+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:50.203623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483488+0000","last_change":"2024-07-27T06:43:12.796179+0000","last_active":"2024-07-27T06:45:09.483488+0000","last_peered":"2024-07-27T06:45:09.483488+0000","last_clean":"2024-07-27T06:45:09.483488+0000","last_became_active":"2024-07-27T06:43:12.795701+0000","last_became_peered":"2024-07-27T06:43:12.795701+0000","last_unstale":"2024-07-27T06:45:09.483488+0000","last_undegraded":"2024-07-27T06:45:09.483488+0000","last_fullsized":"2024-07-27T06:45:09.483488+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:20:12.236769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481627+0000","last_change":"2024-07-27T06:43:09.769949+0000","last_active":"2024-07-27T06:45:09.481627+0000","last_peered":"2024-07-27T06:45:09.481627+0000","last_clean":"2024-07-27T06:45:09.481627+0000","last_became_active":"2024-07-27T06:43:09.769602+0000","last_became_peered":"2024-07-27T06:43:09.769602+0000","last_unstale":"2024-07-27T06:45:09.481627+0000","last_undegraded":"2024-07-27T06:45:09.481627+0000","last_fullsized":"2024-07-27T06:45:09.481627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:28:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488869+0000","last_change":"2024-07-27T06:43:12.802230+0000","last_active":"2024-07-27T06:45:09.488869+0000","last_peered":"2024-07-27T06:45:09.488869+0000","last_clean":"2024-07-27T06:45:09.488869+0000","last_became_active":"2024-07-27T06:43:12.800944+0000","last_became_peered":"2024-07-27T06:43:12.800944+0000","last_unstale":"2024-07-27T06:45:09.488869+0000","last_undegraded":"2024-07-27T06:45:09.488869+0000","last_fullsized":"2024-07-27T06:45:09.488869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:05:54.585955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862270+0000","last_change":"2024-07-27T06:43:09.768239+0000","last_active":"2024-07-27T06:45:09.862270+0000","last_peered":"2024-07-27T06:45:09.862270+0000","last_clean":"2024-07-27T06:45:09.862270+0000","last_became_active":"2024-07-27T06:43:09.767584+0000","last_became_peered":"2024-07-27T06:43:09.767584+0000","last_unstale":"2024-07-27T06:45:09.862270+0000","last_undegraded":"2024-07-27T06:45:09.862270+0000","last_fullsized":"2024-07-27T06:45:09.862270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:35:39.552246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488571+0000","last_change":"2024-07-27T06:43:12.803779+0000","last_active":"2024-07-27T06:45:09.488571+0000","last_peered":"2024-07-27T06:45:09.488571+0000","last_clean":"2024-07-27T06:45:09.488571+0000","last_became_active":"2024-07-27T06:43:12.803359+0000","last_became_peered":"2024-07-27T06:43:12.803359+0000","last_unstale":"2024-07-27T06:45:09.488571+0000","last_undegraded":"2024-07-27T06:45:09.488571+0000","last_fullsized":"2024-07-27T06:45:09.488571+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:32:53.167039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'7","reported_seq":103,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:24.406095+0000","last_change":"2024-07-27T06:44:09.241782+0000","last_active":"2024-07-27T06:45:24.406095+0000","last_peered":"2024-07-27T06:45:24.406095+0000","last_clean":"2024-07-27T06:45:24.406095+0000","last_became_active":"2024-07-27T06:44:09.234692+0000","last_became_peered":"2024-07-27T06:44:09.234692+0000","last_unstale":"2024-07-27T06:45:24.406095+0000","last_undegraded":"2024-07-27T06:45:24.406095+0000","last_fullsized":"2024-07-27T06:45:24.406095+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:45:02.155686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":81,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1561,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488800+0000","last_change":"2024-07-27T06:43:12.805830+0000","last_active":"2024-07-27T06:45:09.488800+0000","last_peered":"2024-07-27T06:45:09.488800+0000","last_clean":"2024-07-27T06:45:09.488800+0000","last_became_active":"2024-07-27T06:43:12.805324+0000","last_became_peered":"2024-07-27T06:43:12.805324+0000","last_unstale":"2024-07-27T06:45:09.488800+0000","last_undegraded":"2024-07-27T06:45:09.488800+0000","last_fullsized":"2024-07-27T06:45:09.488800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:38:31.430676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":108,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.489020+0000","last_change":"2024-07-27T06:41:43.549132+0000","last_active":"2024-07-27T06:45:09.489020+0000","last_peered":"2024-07-27T06:45:09.489020+0000","last_clean":"2024-07-27T06:45:09.489020+0000","last_became_active":"2024-07-27T06:41:43.542635+0000","last_became_peered":"2024-07-27T06:41:43.542635+0000","last_unstale":"2024-07-27T06:45:09.489020+0000","last_undegraded":"2024-07-27T06:45:09.489020+0000","last_fullsized":"2024-07-27T06:45:09.489020+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_clean_scrub_stamp":"2024-07-27T06:38:21.639907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:10:06.417387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.861700+0000","last_change":"2024-07-27T06:43:09.768774+0000","last_active":"2024-07-27T06:45:09.861700+0000","last_peered":"2024-07-27T06:45:09.861700+0000","last_clean":"2024-07-27T06:45:09.861700+0000","last_became_active":"2024-07-27T06:43:09.768345+0000","last_became_peered":"2024-07-27T06:43:09.768345+0000","last_unstale":"2024-07-27T06:45:09.861700+0000","last_undegraded":"2024-07-27T06:45:09.861700+0000","last_fullsized":"2024-07-27T06:45:09.861700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:40:49.411746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485670+0000","last_change":"2024-07-27T06:43:12.797817+0000","last_active":"2024-07-27T06:45:09.485670+0000","last_peered":"2024-07-27T06:45:09.485670+0000","last_clean":"2024-07-27T06:45:09.485670+0000","last_became_active":"2024-07-27T06:43:12.797483+0000","last_became_peered":"2024-07-27T06:43:12.797483+0000","last_unstale":"2024-07-27T06:45:09.485670+0000","last_undegraded":"2024-07-27T06:45:09.485670+0000","last_fullsized":"2024-07-27T06:45:09.485670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:24:44.539505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488887+0000","last_change":"2024-07-27T06:43:09.776880+0000","last_active":"2024-07-27T06:45:09.488887+0000","last_peered":"2024-07-27T06:45:09.488887+0000","last_clean":"2024-07-27T06:45:09.488887+0000","last_became_active":"2024-07-27T06:43:09.776486+0000","last_became_peered":"2024-07-27T06:43:09.776486+0000","last_unstale":"2024-07-27T06:45:09.488887+0000","last_undegraded":"2024-07-27T06:45:09.488887+0000","last_fullsized":"2024-07-27T06:45:09.488887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:55:25.436405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.480961+0000","last_change":"2024-07-27T06:43:12.799755+0000","last_active":"2024-07-27T06:45:09.480961+0000","last_peered":"2024-07-27T06:45:09.480961+0000","last_clean":"2024-07-27T06:45:09.480961+0000","last_became_active":"2024-07-27T06:43:12.799412+0000","last_became_peered":"2024-07-27T06:43:12.799412+0000","last_unstale":"2024-07-27T06:45:09.480961+0000","last_undegraded":"2024-07-27T06:45:09.480961+0000","last_fullsized":"2024-07-27T06:45:09.480961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:45:50.545431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":151,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:25.061804+0000","last_change":"2024-07-27T06:43:10.533039+0000","last_active":"2024-07-27T06:45:25.061804+0000","last_peered":"2024-07-27T06:45:25.061804+0000","last_clean":"2024-07-27T06:45:25.061804+0000","last_became_active":"2024-07-27T06:43:10.532583+0000","last_became_peered":"2024-07-27T06:43:10.532583+0000","last_unstale":"2024-07-27T06:45:25.061804+0000","last_undegraded":"2024-07-27T06:45:25.061804+0000","last_fullsized":"2024-07-27T06:45:25.061804+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:04:51.954463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633016+0000","last_change":"2024-07-27T06:44:09.238218+0000","last_active":"2024-07-27T06:45:09.633016+0000","last_peered":"2024-07-27T06:45:09.633016+0000","last_clean":"2024-07-27T06:45:09.633016+0000","last_became_active":"2024-07-27T06:44:09.237841+0000","last_became_peered":"2024-07-27T06:44:09.237841+0000","last_unstale":"2024-07-27T06:45:09.633016+0000","last_undegraded":"2024-07-27T06:45:09.633016+0000","last_fullsized":"2024-07-27T06:45:09.633016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:19.101854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"81'7","reported_seq":155,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:25.071878+0000","last_change":"2024-07-27T06:43:09.776214+0000","last_active":"2024-07-27T06:45:25.071878+0000","last_peered":"2024-07-27T06:45:25.071878+0000","last_clean":"2024-07-27T06:45:25.071878+0000","last_became_active":"2024-07-27T06:43:09.775719+0000","last_became_peered":"2024-07-27T06:43:09.775719+0000","last_unstale":"2024-07-27T06:45:25.071878+0000","last_undegraded":"2024-07-27T06:45:25.071878+0000","last_fullsized":"2024-07-27T06:45:25.071878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:57:20.381599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481735+0000","last_change":"2024-07-27T06:43:12.794909+0000","last_active":"2024-07-27T06:45:09.481735+0000","last_peered":"2024-07-27T06:45:09.481735+0000","last_clean":"2024-07-27T06:45:09.481735+0000","last_became_active":"2024-07-27T06:43:12.794497+0000","last_became_peered":"2024-07-27T06:43:12.794497+0000","last_unstale":"2024-07-27T06:45:09.481735+0000","last_undegraded":"2024-07-27T06:45:09.481735+0000","last_fullsized":"2024-07-27T06:45:09.481735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:42:49.760126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633255+0000","last_change":"2024-07-27T06:43:10.534240+0000","last_active":"2024-07-27T06:45:09.633255+0000","last_peered":"2024-07-27T06:45:09.633255+0000","last_clean":"2024-07-27T06:45:09.633255+0000","last_became_active":"2024-07-27T06:43:10.533596+0000","last_became_peered":"2024-07-27T06:43:10.533596+0000","last_unstale":"2024-07-27T06:45:09.633255+0000","last_undegraded":"2024-07-27T06:45:09.633255+0000","last_fullsized":"2024-07-27T06:45:09.633255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:16.098839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481991+0000","last_change":"2024-07-27T06:43:12.805248+0000","last_active":"2024-07-27T06:45:09.481991+0000","last_peered":"2024-07-27T06:45:09.481991+0000","last_clean":"2024-07-27T06:45:09.481991+0000","last_became_active":"2024-07-27T06:43:12.803693+0000","last_became_peered":"2024-07-27T06:43:12.803693+0000","last_unstale":"2024-07-27T06:45:09.481991+0000","last_undegraded":"2024-07-27T06:45:09.481991+0000","last_fullsized":"2024-07-27T06:45:09.481991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:17.466465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486692+0000","last_change":"2024-07-27T06:43:10.533701+0000","last_active":"2024-07-27T06:45:09.486692+0000","last_peered":"2024-07-27T06:45:09.486692+0000","last_clean":"2024-07-27T06:45:09.486692+0000","last_became_active":"2024-07-27T06:43:10.533109+0000","last_became_peered":"2024-07-27T06:43:10.533109+0000","last_unstale":"2024-07-27T06:45:09.486692+0000","last_undegraded":"2024-07-27T06:45:09.486692+0000","last_fullsized":"2024-07-27T06:45:09.486692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:32:52.906691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481918+0000","last_change":"2024-07-27T06:43:12.798479+0000","last_active":"2024-07-27T06:45:09.481918+0000","last_peered":"2024-07-27T06:45:09.481918+0000","last_clean":"2024-07-27T06:45:09.481918+0000","last_became_active":"2024-07-27T06:43:12.798124+0000","last_became_peered":"2024-07-27T06:43:12.798124+0000","last_unstale":"2024-07-27T06:45:09.481918+0000","last_undegraded":"2024-07-27T06:45:09.481918+0000","last_fullsized":"2024-07-27T06:45:09.481918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:14:55.267213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'16","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633427+0000","last_change":"2024-07-27T06:43:09.770122+0000","last_active":"2024-07-27T06:45:09.633427+0000","last_peered":"2024-07-27T06:45:09.633427+0000","last_clean":"2024-07-27T06:45:09.633427+0000","last_became_active":"2024-07-27T06:43:09.769784+0000","last_became_peered":"2024-07-27T06:43:09.769784+0000","last_unstale":"2024-07-27T06:45:09.633427+0000","last_undegraded":"2024-07-27T06:45:09.633427+0000","last_fullsized":"2024-07-27T06:45:09.633427+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:47:00.398135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485891+0000","last_change":"2024-07-27T06:43:12.799603+0000","last_active":"2024-07-27T06:45:09.485891+0000","last_peered":"2024-07-27T06:45:09.485891+0000","last_clean":"2024-07-27T06:45:09.485891+0000","last_became_active":"2024-07-27T06:43:12.799189+0000","last_became_peered":"2024-07-27T06:43:12.799189+0000","last_unstale":"2024-07-27T06:45:09.485891+0000","last_undegraded":"2024-07-27T06:45:09.485891+0000","last_fullsized":"2024-07-27T06:45:09.485891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:32:30.603258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862220+0000","last_change":"2024-07-27T06:43:10.004726+0000","last_active":"2024-07-27T06:45:09.862220+0000","last_peered":"2024-07-27T06:45:09.862220+0000","last_clean":"2024-07-27T06:45:09.862220+0000","last_became_active":"2024-07-27T06:43:10.004367+0000","last_became_peered":"2024-07-27T06:43:10.004367+0000","last_unstale":"2024-07-27T06:45:09.862220+0000","last_undegraded":"2024-07-27T06:45:09.862220+0000","last_fullsized":"2024-07-27T06:45:09.862220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:22:39.942391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862427+0000","last_change":"2024-07-27T06:43:12.796511+0000","last_active":"2024-07-27T06:45:09.862427+0000","last_peered":"2024-07-27T06:45:09.862427+0000","last_clean":"2024-07-27T06:45:09.862427+0000","last_became_active":"2024-07-27T06:43:12.796152+0000","last_became_peered":"2024-07-27T06:43:12.796152+0000","last_unstale":"2024-07-27T06:45:09.862427+0000","last_undegraded":"2024-07-27T06:45:09.862427+0000","last_fullsized":"2024-07-27T06:45:09.862427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:03:02.508990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822077+0000","last_change":"2024-07-27T06:43:10.533656+0000","last_active":"2024-07-27T06:45:09.822077+0000","last_peered":"2024-07-27T06:45:09.822077+0000","last_clean":"2024-07-27T06:45:09.822077+0000","last_became_active":"2024-07-27T06:43:10.533295+0000","last_became_peered":"2024-07-27T06:43:10.533295+0000","last_unstale":"2024-07-27T06:45:09.822077+0000","last_undegraded":"2024-07-27T06:45:09.822077+0000","last_fullsized":"2024-07-27T06:45:09.822077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:11:00.032511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485237+0000","last_change":"2024-07-27T06:43:12.795106+0000","last_active":"2024-07-27T06:45:09.485237+0000","last_peered":"2024-07-27T06:45:09.485237+0000","last_clean":"2024-07-27T06:45:09.485237+0000","last_became_active":"2024-07-27T06:43:12.794729+0000","last_became_peered":"2024-07-27T06:43:12.794729+0000","last_unstale":"2024-07-27T06:45:09.485237+0000","last_undegraded":"2024-07-27T06:45:09.485237+0000","last_fullsized":"2024-07-27T06:45:09.485237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:19:16.542363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485673+0000","last_change":"2024-07-27T06:43:09.770016+0000","last_active":"2024-07-27T06:45:09.485673+0000","last_peered":"2024-07-27T06:45:09.485673+0000","last_clean":"2024-07-27T06:45:09.485673+0000","last_became_active":"2024-07-27T06:43:09.769672+0000","last_became_peered":"2024-07-27T06:43:09.769672+0000","last_unstale":"2024-07-27T06:45:09.485673+0000","last_undegraded":"2024-07-27T06:45:09.485673+0000","last_fullsized":"2024-07-27T06:45:09.485673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:30:55.737494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483659+0000","last_change":"2024-07-27T06:43:12.793690+0000","last_active":"2024-07-27T06:45:09.483659+0000","last_peered":"2024-07-27T06:45:09.483659+0000","last_clean":"2024-07-27T06:45:09.483659+0000","last_became_active":"2024-07-27T06:43:12.793352+0000","last_became_peered":"2024-07-27T06:43:12.793352+0000","last_unstale":"2024-07-27T06:45:09.483659+0000","last_undegraded":"2024-07-27T06:45:09.483659+0000","last_fullsized":"2024-07-27T06:45:09.483659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:22:47.606990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.491196+0000","last_change":"2024-07-27T06:45:09.490738+0000","last_active":"2024-07-27T06:45:09.491196+0000","last_peered":"2024-07-27T06:45:09.491196+0000","last_clean":"2024-07-27T06:45:09.491196+0000","last_became_active":"2024-07-27T06:45:09.487147+0000","last_became_peered":"2024-07-27T06:45:09.487147+0000","last_unstale":"2024-07-27T06:45:09.491196+0000","last_undegraded":"2024-07-27T06:45:09.491196+0000","last_fullsized":"2024-07-27T06:45:09.491196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:46.732987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485502+0000","last_change":"2024-07-27T06:43:12.792648+0000","last_active":"2024-07-27T06:45:09.485502+0000","last_peered":"2024-07-27T06:45:09.485502+0000","last_clean":"2024-07-27T06:45:09.485502+0000","last_became_active":"2024-07-27T06:43:12.792338+0000","last_became_peered":"2024-07-27T06:43:12.792338+0000","last_unstale":"2024-07-27T06:45:09.485502+0000","last_undegraded":"2024-07-27T06:45:09.485502+0000","last_fullsized":"2024-07-27T06:45:09.485502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:36:52.605957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.827632+0000","last_change":"2024-07-27T06:45:09.827140+0000","last_active":"2024-07-27T06:45:09.827632+0000","last_peered":"2024-07-27T06:45:09.827632+0000","last_clean":"2024-07-27T06:45:09.827632+0000","last_became_active":"2024-07-27T06:45:09.823119+0000","last_became_peered":"2024-07-27T06:45:09.823119+0000","last_unstale":"2024-07-27T06:45:09.827632+0000","last_undegraded":"2024-07-27T06:45:09.827632+0000","last_fullsized":"2024-07-27T06:45:09.827632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:32:19.682193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488730+0000","last_change":"2024-07-27T06:43:12.809347+0000","last_active":"2024-07-27T06:45:09.488730+0000","last_peered":"2024-07-27T06:45:09.488730+0000","last_clean":"2024-07-27T06:45:09.488730+0000","last_became_active":"2024-07-27T06:43:12.808980+0000","last_became_peered":"2024-07-27T06:43:12.808980+0000","last_unstale":"2024-07-27T06:45:09.488730+0000","last_undegraded":"2024-07-27T06:45:09.488730+0000","last_fullsized":"2024-07-27T06:45:09.488730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:05:14.132641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633228+0000","last_change":"2024-07-27T06:44:09.237908+0000","last_active":"2024-07-27T06:45:09.633228+0000","last_peered":"2024-07-27T06:45:09.633228+0000","last_clean":"2024-07-27T06:45:09.633228+0000","last_became_active":"2024-07-27T06:44:09.237503+0000","last_became_peered":"2024-07-27T06:44:09.237503+0000","last_unstale":"2024-07-27T06:45:09.633228+0000","last_undegraded":"2024-07-27T06:45:09.633228+0000","last_fullsized":"2024-07-27T06:45:09.633228+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:29.654244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486331+0000","last_change":"2024-07-27T06:43:09.776000+0000","last_active":"2024-07-27T06:45:09.486331+0000","last_peered":"2024-07-27T06:45:09.486331+0000","last_clean":"2024-07-27T06:45:09.486331+0000","last_became_active":"2024-07-27T06:43:09.775522+0000","last_became_peered":"2024-07-27T06:43:09.775522+0000","last_unstale":"2024-07-27T06:45:09.486331+0000","last_undegraded":"2024-07-27T06:45:09.486331+0000","last_fullsized":"2024-07-27T06:45:09.486331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:39:42.650177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822023+0000","last_change":"2024-07-27T06:43:12.807062+0000","last_active":"2024-07-27T06:45:09.822023+0000","last_peered":"2024-07-27T06:45:09.822023+0000","last_clean":"2024-07-27T06:45:09.822023+0000","last_became_active":"2024-07-27T06:43:12.805816+0000","last_became_peered":"2024-07-27T06:43:12.805816+0000","last_unstale":"2024-07-27T06:45:09.822023+0000","last_undegraded":"2024-07-27T06:45:09.822023+0000","last_fullsized":"2024-07-27T06:45:09.822023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:44:16.261663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483406+0000","last_change":"2024-07-27T06:44:09.250070+0000","last_active":"2024-07-27T06:45:09.483406+0000","last_peered":"2024-07-27T06:45:09.483406+0000","last_clean":"2024-07-27T06:45:09.483406+0000","last_became_active":"2024-07-27T06:44:09.237290+0000","last_became_peered":"2024-07-27T06:44:09.237290+0000","last_unstale":"2024-07-27T06:45:09.483406+0000","last_undegraded":"2024-07-27T06:45:09.483406+0000","last_fullsized":"2024-07-27T06:45:09.483406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:51.590487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862423+0000","last_change":"2024-07-27T06:43:12.809135+0000","last_active":"2024-07-27T06:45:09.862423+0000","last_peered":"2024-07-27T06:45:09.862423+0000","last_clean":"2024-07-27T06:45:09.862423+0000","last_became_active":"2024-07-27T06:43:12.806731+0000","last_became_peered":"2024-07-27T06:43:12.806731+0000","last_unstale":"2024-07-27T06:45:09.862423+0000","last_undegraded":"2024-07-27T06:45:09.862423+0000","last_fullsized":"2024-07-27T06:45:09.862423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:47:50.397517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486050+0000","last_change":"2024-07-27T06:43:10.533608+0000","last_active":"2024-07-27T06:45:09.486050+0000","last_peered":"2024-07-27T06:45:09.486050+0000","last_clean":"2024-07-27T06:45:09.486050+0000","last_became_active":"2024-07-27T06:43:10.532663+0000","last_became_peered":"2024-07-27T06:43:10.532663+0000","last_unstale":"2024-07-27T06:45:09.486050+0000","last_undegraded":"2024-07-27T06:45:09.486050+0000","last_fullsized":"2024-07-27T06:45:09.486050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:05:33.155427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485883+0000","last_change":"2024-07-27T06:43:12.789890+0000","last_active":"2024-07-27T06:45:09.485883+0000","last_peered":"2024-07-27T06:45:09.485883+0000","last_clean":"2024-07-27T06:45:09.485883+0000","last_became_active":"2024-07-27T06:43:12.789561+0000","last_became_peered":"2024-07-27T06:43:12.789561+0000","last_unstale":"2024-07-27T06:45:09.485883+0000","last_undegraded":"2024-07-27T06:45:09.485883+0000","last_fullsized":"2024-07-27T06:45:09.485883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:35:49.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"81'14","reported_seq":161,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:26.072635+0000","last_change":"2024-07-27T06:43:09.773471+0000","last_active":"2024-07-27T06:45:26.072635+0000","last_peered":"2024-07-27T06:45:26.072635+0000","last_clean":"2024-07-27T06:45:26.072635+0000","last_became_active":"2024-07-27T06:43:09.773213+0000","last_became_peered":"2024-07-27T06:43:09.773213+0000","last_unstale":"2024-07-27T06:45:26.072635+0000","last_undegraded":"2024-07-27T06:45:26.072635+0000","last_fullsized":"2024-07-27T06:45:26.072635+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:54:38.770198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68614,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1139,"num_read_kb":665,"num_write":257,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1561,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":205842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283116,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678059,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":461919,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874287,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":1113012,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037814,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":937984,"data_stored":461317,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234042,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":461695,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430270,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":987136,"data_stored":1054886,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593796,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":461566,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790025,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":1054032,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":929792,"data_stored":462597,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":460859,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378713,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":522125,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542237,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":983040,"data_stored":523075,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705762,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":462905,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":61930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T06:45:29.113 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-27T06:45:29.113 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-07-27T06:45:29.113 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-07-27T06:45:29.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph health --format=json 2024-07-27T06:45:29.382 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:29 smithi029 ceph-mon[32610]: pgmap v351: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:45:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:29 smithi029 ceph-mon[32610]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:45:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:29 smithi077 ceph-mon[37153]: pgmap v351: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:45:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:29 smithi077 ceph-mon[37153]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:45:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:29 smithi047 ceph-mon[36835]: pgmap v351: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:45:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:29 smithi047 ceph-mon[36835]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:45:31.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.316+0000 7fcf7c898640 1 -- 172.21.15.29:0/1169889547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf78108740 msgr2=0x7fcf7810ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.316+0000 7fcf7c898640 1 --2- 172.21.15.29:0/1169889547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf78108740 0x7fcf7810ef70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009ca0 tx=0x7fcf7002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:45:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.316+0000 7fcf7c898640 1 -- 172.21.15.29:0/1169889547 shutdown_connections 2024-07-27T06:45:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.316+0000 7fcf7c898640 1 --2- 172.21.15.29:0/1169889547 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf78108740 0x7fcf7810ef70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.316+0000 7fcf7c898640 1 --2- 172.21.15.29:0/1169889547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf78107df0 0x7fcf78108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.316+0000 7fcf7c898640 1 --2- 172.21.15.29:0/1169889547 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf781024b0 0x7fcf781028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.316+0000 7fcf7c898640 1 -- 172.21.15.29:0/1169889547 >> 172.21.15.29:0/1169889547 conn(0x7fcf78079170 msgr2=0x7fcf78079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.316+0000 7fcf7c898640 1 -- 172.21.15.29:0/1169889547 shutdown_connections 2024-07-27T06:45:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.316+0000 7fcf7c898640 1 -- 172.21.15.29:0/1169889547 wait complete. 2024-07-27T06:45:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.317+0000 7fcf7c898640 1 Processor -- start 2024-07-27T06:45:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.317+0000 7fcf7c898640 1 -- start start 2024-07-27T06:45:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.317+0000 7fcf7c898640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf781024b0 0x7fcf78199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.317+0000 7fcf7c898640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf78107df0 0x7fcf7819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.317+0000 7fcf7c898640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf78108740 0x7fcf7819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.317+0000 7fcf7c898640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcf781123c0 con 0x7fcf781024b0 2024-07-27T06:45:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.317+0000 7fcf7c898640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fcf78112240 con 0x7fcf78107df0 2024-07-27T06:45:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.317+0000 7fcf7c898640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcf78112540 con 0x7fcf78108740 2024-07-27T06:45:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf7659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf781024b0 0x7fcf78199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf7659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf781024b0 0x7fcf78199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53538/0 (socket says 172.21.15.29:53538) 2024-07-27T06:45:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf7659c640 1 -- 172.21.15.29:0/3253570788 learned_addr learned my addr 172.21.15.29:0/3253570788 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf75d9b640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf78107df0 0x7fcf7819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf76d9d640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf78108740 0x7fcf7819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf7659c640 1 -- 172.21.15.29:0/3253570788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf78108740 msgr2=0x7fcf7819f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf7659c640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf78108740 0x7fcf7819f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf7659c640 1 -- 172.21.15.29:0/3253570788 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf78107df0 msgr2=0x7fcf7819a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf7659c640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf78107df0 0x7fcf7819a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf7659c640 1 -- 172.21.15.29:0/3253570788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf7819fdd0 con 0x7fcf781024b0 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf75d9b640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf78107df0 0x7fcf7819a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf76d9d640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf78108740 0x7fcf7819f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf7659c640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf781024b0 0x7fcf78199e60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800d950 tx=0x7fcf6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf677fe640 1 -- 172.21.15.29:0/3253570788 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf68017070 con 0x7fcf781024b0 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf677fe640 1 -- 172.21.15.29:0/3253570788 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf68009c30 con 0x7fcf781024b0 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf7c898640 1 -- 172.21.15.29:0/3253570788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf781a00c0 con 0x7fcf781024b0 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf677fe640 1 -- 172.21.15.29:0/3253570788 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6800f620 con 0x7fcf781024b0 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.318+0000 7fcf7c898640 1 -- 172.21.15.29:0/3253570788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf781a6820 con 0x7fcf781024b0 2024-07-27T06:45:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.319+0000 7fcf7c898640 1 -- 172.21.15.29:0/3253570788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf78103730 con 0x7fcf781024b0 2024-07-27T06:45:31.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.321+0000 7fcf677fe640 1 -- 172.21.15.29:0/3253570788 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fcf68005030 con 0x7fcf781024b0 2024-07-27T06:45:31.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.322+0000 7fcf677fe640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcf600b3c20 0x7fcf600b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:31.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.322+0000 7fcf75d9b640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcf600b3c20 0x7fcf600b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:31.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.322+0000 7fcf677fe640 1 -- 172.21.15.29:0/3253570788 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fcf680e3a90 con 0x7fcf781024b0 2024-07-27T06:45:31.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.322+0000 7fcf75d9b640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcf600b3c20 0x7fcf600b6130 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcf781075a0 tx=0x7fcf6c005e50 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:31.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.325+0000 7fcf677fe640 1 -- 172.21.15.29:0/3253570788 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcf680aaf70 con 0x7fcf781024b0 2024-07-27T06:45:31.347 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:31 smithi029 ceph-mon[32610]: pgmap v352: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:31.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.547+0000 7fcf7c898640 1 -- 172.21.15.29:0/3253570788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fcf7819af90 con 0x7fcf781024b0 2024-07-27T06:45:31.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.547+0000 7fcf677fe640 1 -- 172.21.15.29:0/3253570788 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fcf68009de0 con 0x7fcf781024b0 2024-07-27T06:45:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:45:31.549 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-07-27T06:45:31.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.549+0000 7fcf7c898640 1 -- 172.21.15.29:0/3253570788 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcf600b3c20 msgr2=0x7fcf600b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:31.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.549+0000 7fcf7c898640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcf600b3c20 0x7fcf600b6130 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcf781075a0 tx=0x7fcf6c005e50 comp rx=0 tx=0).stop 2024-07-27T06:45:31.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.549+0000 7fcf7c898640 1 -- 172.21.15.29:0/3253570788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf781024b0 msgr2=0x7fcf78199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:31.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.549+0000 7fcf7c898640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf781024b0 0x7fcf78199e60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800d950 tx=0x7fcf6800de10 comp rx=0 tx=0).stop 2024-07-27T06:45:31.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.549+0000 7fcf7c898640 1 -- 172.21.15.29:0/3253570788 shutdown_connections 2024-07-27T06:45:31.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.549+0000 7fcf7c898640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcf600b3c20 0x7fcf600b6130 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:31.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.549+0000 7fcf7c898640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf78108740 0x7fcf7819f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:31.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.549+0000 7fcf7c898640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcf78107df0 0x7fcf7819a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:31.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.549+0000 7fcf7c898640 1 --2- 172.21.15.29:0/3253570788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf781024b0 0x7fcf78199e60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:31.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.549+0000 7fcf7c898640 1 -- 172.21.15.29:0/3253570788 >> 172.21.15.29:0/3253570788 conn(0x7fcf78079170 msgr2=0x7fcf7806b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:31.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.549+0000 7fcf7c898640 1 -- 172.21.15.29:0/3253570788 shutdown_connections 2024-07-27T06:45:31.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:31.549+0000 7fcf7c898640 1 -- 172.21.15.29:0/3253570788 wait complete. 2024-07-27T06:45:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:31 smithi077 ceph-mon[37153]: pgmap v352: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:31 smithi047 ceph-mon[36835]: pgmap v352: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:32.311 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-07-27T06:45:32.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:45:32.576 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:32 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3253570788' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-27T06:45:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:32 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3253570788' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-27T06:45:32.699 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:32 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3253570788' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-27T06:45:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:33 smithi077 ceph-mon[37153]: pgmap v353: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:45:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:33 smithi047 ceph-mon[36835]: pgmap v353: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:45:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:33 smithi029 ceph-mon[32610]: pgmap v353: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:45:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.310+0000 7f17f6472640 1 -- 172.21.15.29:0/1874274110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17f0104350 msgr2=0x7f17f0104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:34.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.310+0000 7f17f6472640 1 --2- 172.21.15.29:0/1874274110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17f0104350 0x7f17f0104760 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009900 tx=0x7f17e002a410 comp rx=0 tx=0).stop 2024-07-27T06:45:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.310+0000 7f17f6472640 1 -- 172.21.15.29:0/1874274110 shutdown_connections 2024-07-27T06:45:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.310+0000 7f17f6472640 1 --2- 172.21.15.29:0/1874274110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f010d900 0x7f17f010fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.310+0000 7f17f6472640 1 --2- 172.21.15.29:0/1874274110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17f0104ca0 0x7f17f010d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.310+0000 7f17f6472640 1 --2- 172.21.15.29:0/1874274110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17f0104350 0x7f17f0104760 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.310+0000 7f17f6472640 1 -- 172.21.15.29:0/1874274110 >> 172.21.15.29:0/1874274110 conn(0x7f17f0079190 msgr2=0x7f17f00795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.310+0000 7f17f6472640 1 -- 172.21.15.29:0/1874274110 shutdown_connections 2024-07-27T06:45:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.311+0000 7f17f6472640 1 -- 172.21.15.29:0/1874274110 wait complete. 2024-07-27T06:45:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.311+0000 7f17f6472640 1 Processor -- start 2024-07-27T06:45:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.311+0000 7f17f6472640 1 -- start start 2024-07-27T06:45:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f6472640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f0104350 0x7f17f0199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f6472640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17f0104ca0 0x7f17f019a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:34.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f6472640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17f010d900 0x7f17f019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f6472640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0113500 con 0x7f17f010d900 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f6472640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0113380 con 0x7f17f0104ca0 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f6472640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0113680 con 0x7f17f0104350 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f4a0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17f010d900 0x7f17f019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f4a0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17f010d900 0x7f17f019f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53554/0 (socket says 172.21.15.29:53554) 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f4a0f640 1 -- 172.21.15.29:0/27388646 learned_addr learned my addr 172.21.15.29:0/27388646 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17ef7fe640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17f0104ca0 0x7f17f019a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17effff640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f0104350 0x7f17f0199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f4a0f640 1 -- 172.21.15.29:0/27388646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f0104350 msgr2=0x7f17f0199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f4a0f640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f0104350 0x7f17f0199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f4a0f640 1 -- 172.21.15.29:0/27388646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17f0104ca0 msgr2=0x7f17f019a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f4a0f640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17f0104ca0 0x7f17f019a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:34.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f4a0f640 1 -- 172.21.15.29:0/27388646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f019ff00 con 0x7f17f010d900 2024-07-27T06:45:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17effff640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f0104350 0x7f17f0199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:45:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17ef7fe640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17f0104ca0 0x7f17f019a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:45:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.312+0000 7f17f4a0f640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17f010d900 0x7f17f019f7d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f17e800e9d0 tx=0x7f17e800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.313+0000 7f17ed7fa640 1 -- 172.21.15.29:0/27388646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e800cd80 con 0x7f17f010d900 2024-07-27T06:45:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.313+0000 7f17f6472640 1 -- 172.21.15.29:0/27388646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f01a01f0 con 0x7f17f010d900 2024-07-27T06:45:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.313+0000 7f17ed7fa640 1 -- 172.21.15.29:0/27388646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e8005020 con 0x7f17f010d900 2024-07-27T06:45:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.313+0000 7f17ed7fa640 1 -- 172.21.15.29:0/27388646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e80116d0 con 0x7f17f010d900 2024-07-27T06:45:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.313+0000 7f17f6472640 1 -- 172.21.15.29:0/27388646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f01a6950 con 0x7f17f010d900 2024-07-27T06:45:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.314+0000 7f17f6472640 1 -- 172.21.15.29:0/27388646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f006bb60 con 0x7f17f010d900 2024-07-27T06:45:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.315+0000 7f17ed7fa640 1 -- 172.21.15.29:0/27388646 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f17e80051d0 con 0x7f17f010d900 2024-07-27T06:45:34.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.316+0000 7f17ed7fa640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17d80b3c20 0x7f17d80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:34.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.316+0000 7f17effff640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17d80b3c20 0x7f17d80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:34.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.316+0000 7f17ed7fa640 1 -- 172.21.15.29:0/27388646 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f17e80e4330 con 0x7f17f010d900 2024-07-27T06:45:34.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.317+0000 7f17effff640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17d80b3c20 0x7f17d80b6130 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f17e00096c0 tx=0x7f17e0031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:34.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.319+0000 7f17ed7fa640 1 -- 172.21.15.29:0/27388646 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f17e8016030 con 0x7f17f010d900 2024-07-27T06:45:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.546+0000 7f17f6472640 1 -- 172.21.15.29:0/27388646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17f0104760 con 0x7f17f010d900 2024-07-27T06:45:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.547+0000 7f17ed7fa640 1 -- 172.21.15.29:0/27388646 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11957 (secure 0 0 0) 0x7f17f0104760 con 0x7f17f010d900 2024-07-27T06:45:34.549 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:45:34.549 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"btime":"2024-07-27T06:43:37:875457+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:43:37.017169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24613,24617,24646,24658,24670]},"id":1}]} 2024-07-27T06:45:34.549 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-07-27T06:45:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.549+0000 7f17f6472640 1 -- 172.21.15.29:0/27388646 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17d80b3c20 msgr2=0x7f17d80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.549+0000 7f17f6472640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17d80b3c20 0x7f17d80b6130 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f17e00096c0 tx=0x7f17e0031400 comp rx=0 tx=0).stop 2024-07-27T06:45:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.549+0000 7f17f6472640 1 -- 172.21.15.29:0/27388646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17f010d900 msgr2=0x7f17f019f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.549+0000 7f17f6472640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17f010d900 0x7f17f019f7d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f17e800e9d0 tx=0x7f17e800ee90 comp rx=0 tx=0).stop 2024-07-27T06:45:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.549+0000 7f17f6472640 1 -- 172.21.15.29:0/27388646 shutdown_connections 2024-07-27T06:45:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.549+0000 7f17f6472640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17f010d900 0x7f17f019f7d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.549+0000 7f17f6472640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17f0104ca0 0x7f17f019a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.549+0000 7f17f6472640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17d80b3c20 0x7f17d80b6130 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.549+0000 7f17f6472640 1 --2- 172.21.15.29:0/27388646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f0104350 0x7f17f0199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:34.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.549+0000 7f17f6472640 1 -- 172.21.15.29:0/27388646 >> 172.21.15.29:0/27388646 conn(0x7f17f0079190 msgr2=0x7f17f0106460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:34.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.549+0000 7f17f6472640 1 -- 172.21.15.29:0/27388646 shutdown_connections 2024-07-27T06:45:34.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:34.550+0000 7f17f6472640 1 -- 172.21.15.29:0/27388646 wait complete. 2024-07-27T06:45:35.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:45:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:35 smithi077 ceph-mon[37153]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:45:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:35 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/27388646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:45:35.607 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:35 smithi047 ceph-mon[36835]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:45:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:35 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/27388646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:45:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:35 smithi029 ceph-mon[32610]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:45:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:35 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/27388646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:45:37.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.329+0000 7f649b3bb640 1 -- 172.21.15.29:0/4008562563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f649410fc30 msgr2=0x7f6494112050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:37.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.329+0000 7f649b3bb640 1 --2- 172.21.15.29:0/4008562563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f649410fc30 0x7f6494112050 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f648c009ca0 tx=0x7f648c02a950 comp rx=0 tx=0).stop 2024-07-27T06:45:37.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.329+0000 7f649b3bb640 1 -- 172.21.15.29:0/4008562563 shutdown_connections 2024-07-27T06:45:37.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.329+0000 7f649b3bb640 1 --2- 172.21.15.29:0/4008562563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f649410fc30 0x7f6494112050 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:37.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.330+0000 7f649b3bb640 1 --2- 172.21.15.29:0/4008562563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64941029d0 0x7f649410f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:37.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.330+0000 7f649b3bb640 1 --2- 172.21.15.29:0/4008562563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6494102080 0x7f6494102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:37.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.330+0000 7f649b3bb640 1 -- 172.21.15.29:0/4008562563 >> 172.21.15.29:0/4008562563 conn(0x7f6494079190 msgr2=0x7f64940795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:37.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.330+0000 7f649b3bb640 1 -- 172.21.15.29:0/4008562563 shutdown_connections 2024-07-27T06:45:37.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.330+0000 7f649b3bb640 1 -- 172.21.15.29:0/4008562563 wait complete. 2024-07-27T06:45:37.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.330+0000 7f649b3bb640 1 Processor -- start 2024-07-27T06:45:37.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.330+0000 7f649b3bb640 1 -- start start 2024-07-27T06:45:37.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f649b3bb640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6494102080 0x7f649419c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:37.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f649b3bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64941029d0 0x7f649419c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:37.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f649b3bb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f649410fc30 0x7f64941a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:37.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f649b3bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6494115680 con 0x7f64941029d0 2024-07-27T06:45:37.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f649b3bb640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f6494115500 con 0x7f6494102080 2024-07-27T06:45:37.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f649b3bb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6494115800 con 0x7f649410fc30 2024-07-27T06:45:37.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f6498956640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64941029d0 0x7f649419c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:37.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f6498956640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64941029d0 0x7f649419c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53558/0 (socket says 172.21.15.29:53558) 2024-07-27T06:45:37.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f6498956640 1 -- 172.21.15.29:0/4149243125 learned_addr learned my addr 172.21.15.29:0/4149243125 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:37.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f6498956640 1 -- 172.21.15.29:0/4149243125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f649410fc30 msgr2=0x7f64941a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f6498956640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f649410fc30 0x7f64941a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f6499157640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6494102080 0x7f649419c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f6498956640 1 -- 172.21.15.29:0/4149243125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6494102080 msgr2=0x7f649419c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f6498956640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6494102080 0x7f649419c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f6498956640 1 -- 172.21.15.29:0/4149243125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64941a1ee0 con 0x7f64941029d0 2024-07-27T06:45:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f6498956640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64941029d0 0x7f649419c5d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f648400c920 tx=0x7f648400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f648a7fc640 1 -- 172.21.15.29:0/4149243125 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484007c70 con 0x7f64941029d0 2024-07-27T06:45:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f648a7fc640 1 -- 172.21.15.29:0/4149243125 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6484007e10 con 0x7f64941029d0 2024-07-27T06:45:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f649b3bb640 1 -- 172.21.15.29:0/4149243125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64941a21d0 con 0x7f64941029d0 2024-07-27T06:45:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f648a7fc640 1 -- 172.21.15.29:0/4149243125 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484018820 con 0x7f64941029d0 2024-07-27T06:45:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.331+0000 7f649b3bb640 1 -- 172.21.15.29:0/4149243125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64941a89f0 con 0x7f64941029d0 2024-07-27T06:45:37.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.333+0000 7f649b3bb640 1 -- 172.21.15.29:0/4149243125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f649406bc20 con 0x7f64941029d0 2024-07-27T06:45:37.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.334+0000 7f648a7fc640 1 -- 172.21.15.29:0/4149243125 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f648400f450 con 0x7f64941029d0 2024-07-27T06:45:37.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.335+0000 7f648a7fc640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f64800b3cd0 0x7f64800b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:37.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.335+0000 7f6499157640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f64800b3cd0 0x7f64800b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:37.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.335+0000 7f648a7fc640 1 -- 172.21.15.29:0/4149243125 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f64840e40c0 con 0x7f64941029d0 2024-07-27T06:45:37.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.336+0000 7f6499157640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f64800b3cd0 0x7f64800b61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f64900058c0 tx=0x7f6490005c60 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:37.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.338+0000 7f648a7fc640 1 -- 172.21.15.29:0/4149243125 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6484014030 con 0x7f64941029d0 2024-07-27T06:45:37.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:37 smithi029 ceph-mon[32610]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:37.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.558+0000 7f649b3bb640 1 -- 172.21.15.29:0/4149243125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6494102490 con 0x7f64941029d0 2024-07-27T06:45:37.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.559+0000 7f648a7fc640 1 -- 172.21.15.29:0/4149243125 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11957 (secure 0 0 0) 0x7f64840ab7b0 con 0x7f64941029d0 2024-07-27T06:45:37.561 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:45:37.561 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"btime":"2024-07-27T06:43:37:875457+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:43:37.017169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24613,24617,24646,24658,24670]},"id":1}]} 2024-07-27T06:45:37.561 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-07-27T06:45:37.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.561+0000 7f649b3bb640 1 -- 172.21.15.29:0/4149243125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f64800b3cd0 msgr2=0x7f64800b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:37.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.561+0000 7f649b3bb640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f64800b3cd0 0x7f64800b61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f64900058c0 tx=0x7f6490005c60 comp rx=0 tx=0).stop 2024-07-27T06:45:37.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.561+0000 7f649b3bb640 1 -- 172.21.15.29:0/4149243125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64941029d0 msgr2=0x7f649419c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.561+0000 7f649b3bb640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64941029d0 0x7f649419c5d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f648400c920 tx=0x7f648400cde0 comp rx=0 tx=0).stop 2024-07-27T06:45:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.561+0000 7f649b3bb640 1 -- 172.21.15.29:0/4149243125 shutdown_connections 2024-07-27T06:45:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.561+0000 7f649b3bb640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f64800b3cd0 0x7f64800b61e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.561+0000 7f649b3bb640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f649410fc30 0x7f64941a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.561+0000 7f649b3bb640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64941029d0 0x7f649419c5d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.561+0000 7f649b3bb640 1 --2- 172.21.15.29:0/4149243125 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6494102080 0x7f649419c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.561+0000 7f649b3bb640 1 -- 172.21.15.29:0/4149243125 >> 172.21.15.29:0/4149243125 conn(0x7f6494079190 msgr2=0x7f64941086a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.561+0000 7f649b3bb640 1 -- 172.21.15.29:0/4149243125 shutdown_connections 2024-07-27T06:45:37.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:37.561+0000 7f649b3bb640 1 -- 172.21.15.29:0/4149243125 wait complete. 2024-07-27T06:45:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:37 smithi077 ceph-mon[37153]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:37 smithi047 ceph-mon[36835]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:38.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd dump --format=json 2024-07-27T06:45:38.549 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:38 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/4149243125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:45:38.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:38 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/4149243125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:45:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:38 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/4149243125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:45:39.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:39 smithi047 ceph-mon[36835]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:45:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:39 smithi077 ceph-mon[37153]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:45:39.707 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:39 smithi029 ceph-mon[32610]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:45:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.218+0000 7f5287a3c640 1 -- 172.21.15.29:0/1027215917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5280107df0 msgr2=0x7f5280108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.218+0000 7f5287a3c640 1 --2- 172.21.15.29:0/1027215917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5280107df0 0x7f5280108200 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f5270009900 tx=0x7f527002a410 comp rx=0 tx=0).stop 2024-07-27T06:45:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.218+0000 7f5287a3c640 1 -- 172.21.15.29:0/1027215917 shutdown_connections 2024-07-27T06:45:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.218+0000 7f5287a3c640 1 --2- 172.21.15.29:0/1027215917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5280108740 0x7f528010ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.218+0000 7f5287a3c640 1 --2- 172.21.15.29:0/1027215917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5280107df0 0x7f5280108200 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.218+0000 7f5287a3c640 1 --2- 172.21.15.29:0/1027215917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f52801024b0 0x7f52801028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.218+0000 7f5287a3c640 1 -- 172.21.15.29:0/1027215917 >> 172.21.15.29:0/1027215917 conn(0x7f5280079170 msgr2=0x7f5280079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.218+0000 7f5287a3c640 1 -- 172.21.15.29:0/1027215917 shutdown_connections 2024-07-27T06:45:40.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.218+0000 7f5287a3c640 1 -- 172.21.15.29:0/1027215917 wait complete. 2024-07-27T06:45:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.219+0000 7f5287a3c640 1 Processor -- start 2024-07-27T06:45:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.219+0000 7f5287a3c640 1 -- start start 2024-07-27T06:45:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.219+0000 7f5287a3c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52801024b0 0x7f5280199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.219+0000 7f5287a3c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5280107df0 0x7f528019a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.219+0000 7f5287a3c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5280108740 0x7f528019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.219+0000 7f5287a3c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5280112370 con 0x7f5280108740 2024-07-27T06:45:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.219+0000 7f5287a3c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f52801121f0 con 0x7f5280107df0 2024-07-27T06:45:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.219+0000 7f5287a3c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f52801124f0 con 0x7f52801024b0 2024-07-27T06:45:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.219+0000 7f5285fd9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5280108740 0x7f528019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f5285fd9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5280108740 0x7f528019f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49962/0 (socket says 172.21.15.29:49962) 2024-07-27T06:45:40.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f5285fd9640 1 -- 172.21.15.29:0/3006313969 learned_addr learned my addr 172.21.15.29:0/3006313969 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f52857d8640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52801024b0 0x7f5280199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f5284fd7640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5280107df0 0x7f528019a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f5285fd9640 1 -- 172.21.15.29:0/3006313969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52801024b0 msgr2=0x7f5280199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f5285fd9640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52801024b0 0x7f5280199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f5285fd9640 1 -- 172.21.15.29:0/3006313969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5280107df0 msgr2=0x7f528019a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f5285fd9640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5280107df0 0x7f528019a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f5285fd9640 1 -- 172.21.15.29:0/3006313969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528019fe40 con 0x7f5280108740 2024-07-27T06:45:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f5285fd9640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5280108740 0x7f528019f710 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f527800eaa0 tx=0x7f527800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f52767fc640 1 -- 172.21.15.29:0/3006313969 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527800cdb0 con 0x7f5280108740 2024-07-27T06:45:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f52767fc640 1 -- 172.21.15.29:0/3006313969 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5278005020 con 0x7f5280108740 2024-07-27T06:45:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f5287a3c640 1 -- 172.21.15.29:0/3006313969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52801a0130 con 0x7f5280108740 2024-07-27T06:45:40.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f52767fc640 1 -- 172.21.15.29:0/3006313969 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52780116d0 con 0x7f5280108740 2024-07-27T06:45:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.220+0000 7f5287a3c640 1 -- 172.21.15.29:0/3006313969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52801a6890 con 0x7f5280108740 2024-07-27T06:45:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.221+0000 7f5287a3c640 1 -- 172.21.15.29:0/3006313969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5280103730 con 0x7f5280108740 2024-07-27T06:45:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.223+0000 7f52767fc640 1 -- 172.21.15.29:0/3006313969 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f52780051d0 con 0x7f5280108740 2024-07-27T06:45:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.224+0000 7f52767fc640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f526c0b3c20 0x7f526c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.224+0000 7f52857d8640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f526c0b3c20 0x7f526c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.224+0000 7f52767fc640 1 -- 172.21.15.29:0/3006313969 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f52780e42b0 con 0x7f5280108740 2024-07-27T06:45:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.224+0000 7f52857d8640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f526c0b3c20 0x7f526c0b6130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f527c00bbe0 tx=0x7f527c00b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:40.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.227+0000 7f52767fc640 1 -- 172.21.15.29:0/3006313969 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5278016030 con 0x7f5280108740 2024-07-27T06:45:40.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.411+0000 7f5287a3c640 1 -- 172.21.15.29:0/3006313969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f52801028c0 con 0x7f5280108740 2024-07-27T06:45:40.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.413+0000 7f52767fc640 1 -- 172.21.15.29:0/3006313969 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+23026 (secure 0 0 0) 0x7f52801028c0 con 0x7f5280108740 2024-07-27T06:45:40.415 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:45:40.415 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","created":"2024-07-27T06:33:57.535327+0000","modified":"2024-07-27T06:45:09.477077+0000","last_up_change":"2024-07-27T06:42:05.757156+0000","last_in_change":"2024-07-27T06:41:46.423732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T06:38:21.606527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-27T06:43:07.755490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-27T06:43:11.658694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06833d24-2e1a-4199-9338-7d6883d1bf9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6803","nonce":386121641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6805","nonce":386121641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6809","nonce":386121641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6807","nonce":386121641}]},"public_addr":"172.21.15.29:6803/386121641","cluster_addr":"172.21.15.29:6805/386121641","heartbeat_back_addr":"172.21.15.29:6809/386121641","heartbeat_front_addr":"172.21.15.29:6807/386121641","state":["exists","up"]},{"osd":1,"uuid":"906aba19-80dc-41b5-953c-af6dca604b05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6801","nonce":2795953749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6803","nonce":2795953749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6807","nonce":2795953749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6805","nonce":2795953749}]},"public_addr":"172.21.15.47:6801/2795953749","cluster_addr":"172.21.15.47:6803/2795953749","heartbeat_back_addr":"172.21.15.47:6807/2795953749","heartbeat_front_addr":"172.21.15.47:6805/2795953749","state":["exists","up"]},{"osd":2,"uuid":"194ae5a5-aebf-4cff-9b99-585a3073a269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6801","nonce":2202565078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6803","nonce":2202565078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6807","nonce":2202565078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6805","nonce":2202565078}]},"public_addr":"172.21.15.77:6801/2202565078","cluster_addr":"172.21.15.77:6803/2202565078","heartbeat_back_addr":"172.21.15.77:6807/2202565078","heartbeat_front_addr":"172.21.15.77:6805/2202565078","state":["exists","up"]},{"osd":3,"uuid":"3c22ce9c-963e-4f72-86e5-360e781526db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6811","nonce":3757476312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6813","nonce":3757476312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6817","nonce":3757476312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6815","nonce":3757476312}]},"public_addr":"172.21.15.29:6811/3757476312","cluster_addr":"172.21.15.29:6813/3757476312","heartbeat_back_addr":"172.21.15.29:6817/3757476312","heartbeat_front_addr":"172.21.15.29:6815/3757476312","state":["exists","up"]},{"osd":4,"uuid":"cd4b585f-e7a0-4d82-873d-3a9269ebac3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6809","nonce":1379363720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6811","nonce":1379363720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6815","nonce":1379363720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6813","nonce":1379363720}]},"public_addr":"172.21.15.47:6809/1379363720","cluster_addr":"172.21.15.47:6811/1379363720","heartbeat_back_addr":"172.21.15.47:6815/1379363720","heartbeat_front_addr":"172.21.15.47:6813/1379363720","state":["exists","up"]},{"osd":5,"uuid":"40a3eb14-dfaa-4523-8c28-d74c19293c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6809","nonce":1338438780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6811","nonce":1338438780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6815","nonce":1338438780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6813","nonce":1338438780}]},"public_addr":"172.21.15.77:6809/1338438780","cluster_addr":"172.21.15.77:6811/1338438780","heartbeat_back_addr":"172.21.15.77:6815/1338438780","heartbeat_front_addr":"172.21.15.77:6813/1338438780","state":["exists","up"]},{"osd":6,"uuid":"91e85be2-10d1-4209-aa33-17da8f8bd8bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6819","nonce":2679402148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6821","nonce":2679402148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6825","nonce":2679402148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6823","nonce":2679402148}]},"public_addr":"172.21.15.29:6819/2679402148","cluster_addr":"172.21.15.29:6821/2679402148","heartbeat_back_addr":"172.21.15.29:6825/2679402148","heartbeat_front_addr":"172.21.15.29:6823/2679402148","state":["exists","up"]},{"osd":7,"uuid":"c655b450-0cc4-45d4-95ea-93a57fe8e4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6817","nonce":2439195589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6819","nonce":2439195589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6823","nonce":2439195589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6821","nonce":2439195589}]},"public_addr":"172.21.15.47:6817/2439195589","cluster_addr":"172.21.15.47:6819/2439195589","heartbeat_back_addr":"172.21.15.47:6823/2439195589","heartbeat_front_addr":"172.21.15.47:6821/2439195589","state":["exists","up"]},{"osd":8,"uuid":"a7a95b51-f32c-433e-a197-feb6d47b6c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6817","nonce":1422290905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6819","nonce":1422290905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6823","nonce":1422290905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6821","nonce":1422290905}]},"public_addr":"172.21.15.77:6817/1422290905","cluster_addr":"172.21.15.77:6819/1422290905","heartbeat_back_addr":"172.21.15.77:6823/1422290905","heartbeat_front_addr":"172.21.15.77:6821/1422290905","state":["exists","up"]},{"osd":9,"uuid":"f352242b-0d2e-41b3-9c83-436cf432b0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6827","nonce":1307316384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6829","nonce":1307316384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6833","nonce":1307316384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6831","nonce":1307316384}]},"public_addr":"172.21.15.29:6827/1307316384","cluster_addr":"172.21.15.29:6829/1307316384","heartbeat_back_addr":"172.21.15.29:6833/1307316384","heartbeat_front_addr":"172.21.15.29:6831/1307316384","state":["exists","up"]},{"osd":10,"uuid":"4148c2df-c3e9-4a47-88f7-36cf3eab8d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6825","nonce":2133726850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6827","nonce":2133726850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6831","nonce":2133726850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6829","nonce":2133726850}]},"public_addr":"172.21.15.47:6825/2133726850","cluster_addr":"172.21.15.47:6827/2133726850","heartbeat_back_addr":"172.21.15.47:6831/2133726850","heartbeat_front_addr":"172.21.15.47:6829/2133726850","state":["exists","up"]},{"osd":11,"uuid":"5e8b1da3-5f65-4fe9-8722-b35921cbd8c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6825","nonce":3228299327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6827","nonce":3228299327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6831","nonce":3228299327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6829","nonce":3228299327}]},"public_addr":"172.21.15.77:6825/3228299327","cluster_addr":"172.21.15.77:6827/3228299327","heartbeat_back_addr":"172.21.15.77:6831/3228299327","heartbeat_front_addr":"172.21.15.77:6829/3228299327","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:32.525206+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:56.489640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:18.887845+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:43.727678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:08.755553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:32.923461+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:59.298439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:23.944848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:48.883050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:14.566310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:39.557809+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:42:03.813903+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7},{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2378279024":"2024-07-28T06:36:07.527503+0000","172.21.15.29:6801/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3527133965":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2262661620":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/3156863587":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2424241521":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3881857930":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/398508875":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6801/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6800/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2679856281":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6801/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2708687265":"2024-07-28T06:34:58.250198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T06:45:40.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.415+0000 7f5287a3c640 1 -- 172.21.15.29:0/3006313969 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f526c0b3c20 msgr2=0x7f526c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:40.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.415+0000 7f5287a3c640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f526c0b3c20 0x7f526c0b6130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f527c00bbe0 tx=0x7f527c00b390 comp rx=0 tx=0).stop 2024-07-27T06:45:40.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.415+0000 7f5287a3c640 1 -- 172.21.15.29:0/3006313969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5280108740 msgr2=0x7f528019f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:40.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.415+0000 7f5287a3c640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5280108740 0x7f528019f710 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f527800eaa0 tx=0x7f527800ef60 comp rx=0 tx=0).stop 2024-07-27T06:45:40.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.415+0000 7f5287a3c640 1 -- 172.21.15.29:0/3006313969 shutdown_connections 2024-07-27T06:45:40.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.415+0000 7f5287a3c640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5280108740 0x7f528019f710 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:40.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.415+0000 7f5287a3c640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5280107df0 0x7f528019a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:40.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.415+0000 7f5287a3c640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f526c0b3c20 0x7f526c0b6130 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:40.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.415+0000 7f5287a3c640 1 --2- 172.21.15.29:0/3006313969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52801024b0 0x7f5280199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:40.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.415+0000 7f5287a3c640 1 -- 172.21.15.29:0/3006313969 >> 172.21.15.29:0/3006313969 conn(0x7f5280079170 msgr2=0x7f528006b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:40.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.415+0000 7f5287a3c640 1 -- 172.21.15.29:0/3006313969 shutdown_connections 2024-07-27T06:45:40.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:40.415+0000 7f5287a3c640 1 -- 172.21.15.29:0/3006313969 wait complete. 2024-07-27T06:45:41.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:45:41.449 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:41 smithi029 ceph-mon[32610]: pgmap v357: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:41 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3006313969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:45:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:41 smithi077 ceph-mon[37153]: pgmap v357: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:41 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3006313969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:45:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:41 smithi047 ceph-mon[36835]: pgmap v357: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:45:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:41 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3006313969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:45:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.199+0000 7fd467a1b640 1 -- 172.21.15.29:0/3336417145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd460109820 msgr2=0x7fd460110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.199+0000 7fd467a1b640 1 --2- 172.21.15.29:0/3336417145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd460109820 0x7fd460110050 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd458009ca0 tx=0x7fd45802a8c0 comp rx=0 tx=0).stop 2024-07-27T06:45:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.199+0000 7fd467a1b640 1 -- 172.21.15.29:0/3336417145 shutdown_connections 2024-07-27T06:45:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.199+0000 7fd467a1b640 1 --2- 172.21.15.29:0/3336417145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd460109820 0x7fd460110050 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.199+0000 7fd467a1b640 1 --2- 172.21.15.29:0/3336417145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd460108ed0 0x7fd4601092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.199+0000 7fd467a1b640 1 --2- 172.21.15.29:0/3336417145 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd460103590 0x7fd4601039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.199+0000 7fd467a1b640 1 -- 172.21.15.29:0/3336417145 >> 172.21.15.29:0/3336417145 conn(0x7fd46007b280 msgr2=0x7fd46007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.199+0000 7fd467a1b640 1 -- 172.21.15.29:0/3336417145 shutdown_connections 2024-07-27T06:45:43.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.199+0000 7fd467a1b640 1 -- 172.21.15.29:0/3336417145 wait complete. 2024-07-27T06:45:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd467a1b640 1 Processor -- start 2024-07-27T06:45:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd467a1b640 1 -- start start 2024-07-27T06:45:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd467a1b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd460103590 0x7fd460199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd467a1b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd460108ed0 0x7fd46019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd467a1b640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd460109820 0x7fd46019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd467a1b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd4601134f0 con 0x7fd460103590 2024-07-27T06:45:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd467a1b640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd460113370 con 0x7fd460109820 2024-07-27T06:45:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd467a1b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd460113670 con 0x7fd460108ed0 2024-07-27T06:45:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd4657b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd460103590 0x7fd460199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd4657b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd460103590 0x7fd460199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49986/0 (socket says 172.21.15.29:49986) 2024-07-27T06:45:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd4657b7640 1 -- 172.21.15.29:0/3431560994 learned_addr learned my addr 172.21.15.29:0/3431560994 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd465fb8640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd460109820 0x7fd46019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd464fb6640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd460108ed0 0x7fd46019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd4657b7640 1 -- 172.21.15.29:0/3431560994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd460108ed0 msgr2=0x7fd46019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd4657b7640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd460108ed0 0x7fd46019a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd4657b7640 1 -- 172.21.15.29:0/3431560994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd460109820 msgr2=0x7fd46019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.200+0000 7fd4657b7640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd460109820 0x7fd46019f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.201+0000 7fd4657b7640 1 -- 172.21.15.29:0/3431560994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd46019fe90 con 0x7fd460103590 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.201+0000 7fd4657b7640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd460103590 0x7fd460199f20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00d950 tx=0x7fd45c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.201+0000 7fd4527fc640 1 -- 172.21.15.29:0/3431560994 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c017030 con 0x7fd460103590 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.201+0000 7fd4527fc640 1 -- 172.21.15.29:0/3431560994 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd45c004020 con 0x7fd460103590 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.201+0000 7fd467a1b640 1 -- 172.21.15.29:0/3431560994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4601a0180 con 0x7fd460103590 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.201+0000 7fd4527fc640 1 -- 172.21.15.29:0/3431560994 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c0047a0 con 0x7fd460103590 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.201+0000 7fd467a1b640 1 -- 172.21.15.29:0/3431560994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4601a68e0 con 0x7fd460103590 2024-07-27T06:45:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.202+0000 7fd467a1b640 1 -- 172.21.15.29:0/3431560994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4601039a0 con 0x7fd460103590 2024-07-27T06:45:43.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.204+0000 7fd4527fc640 1 -- 172.21.15.29:0/3431560994 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fd45c009a30 con 0x7fd460103590 2024-07-27T06:45:43.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.204+0000 7fd4527fc640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd44c0b3c20 0x7fd44c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:43.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.204+0000 7fd464fb6640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd44c0b3c20 0x7fd44c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:43.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.205+0000 7fd4527fc640 1 -- 172.21.15.29:0/3431560994 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd45c0e3ec0 con 0x7fd460103590 2024-07-27T06:45:43.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.205+0000 7fd464fb6640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd44c0b3c20 0x7fd44c0b6130 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd454005fd0 tx=0x7fd45400b510 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:43.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.207+0000 7fd4527fc640 1 -- 172.21.15.29:0/3431560994 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd45c0ab400 con 0x7fd460103590 2024-07-27T06:45:43.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.429+0000 7fd467a1b640 1 -- 172.21.15.29:0/3431560994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd46019b050 con 0x7fd460103590 2024-07-27T06:45:43.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.430+0000 7fd4527fc640 1 -- 172.21.15.29:0/3431560994 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11957 (secure 0 0 0) 0x7fd45c0aad00 con 0x7fd460103590 2024-07-27T06:45:43.432 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:45:43.433 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"btime":"2024-07-27T06:43:37:875457+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:43:37.017169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24613,24617,24646,24658,24670]},"id":1}]} 2024-07-27T06:45:43.433 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-07-27T06:45:43.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.432+0000 7fd467a1b640 1 -- 172.21.15.29:0/3431560994 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd44c0b3c20 msgr2=0x7fd44c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.432+0000 7fd467a1b640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd44c0b3c20 0x7fd44c0b6130 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd454005fd0 tx=0x7fd45400b510 comp rx=0 tx=0).stop 2024-07-27T06:45:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.432+0000 7fd467a1b640 1 -- 172.21.15.29:0/3431560994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd460103590 msgr2=0x7fd460199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.432+0000 7fd467a1b640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd460103590 0x7fd460199f20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00d950 tx=0x7fd45c00de10 comp rx=0 tx=0).stop 2024-07-27T06:45:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.432+0000 7fd467a1b640 1 -- 172.21.15.29:0/3431560994 shutdown_connections 2024-07-27T06:45:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.432+0000 7fd467a1b640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd460109820 0x7fd46019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.432+0000 7fd467a1b640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd44c0b3c20 0x7fd44c0b6130 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.432+0000 7fd467a1b640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd460108ed0 0x7fd46019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.432+0000 7fd467a1b640 1 --2- 172.21.15.29:0/3431560994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd460103590 0x7fd460199f20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.432+0000 7fd467a1b640 1 -- 172.21.15.29:0/3431560994 >> 172.21.15.29:0/3431560994 conn(0x7fd46007b280 msgr2=0x7fd460106f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.432+0000 7fd467a1b640 1 -- 172.21.15.29:0/3431560994 shutdown_connections 2024-07-27T06:45:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:43.432+0000 7fd467a1b640 1 -- 172.21.15.29:0/3431560994 wait complete. 2024-07-27T06:45:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:43 smithi077 ceph-mon[37153]: pgmap v358: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:45:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:43 smithi047 ceph-mon[36835]: pgmap v358: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:45:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:43 smithi029 ceph-mon[32610]: pgmap v358: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:45:44.230 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-27T06:43:14.801207+0000', 'modified': '2024-07-27T06:43:37.017169+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24670, 'mds_1': 24658, 'mds_2': 24646, 'mds_3': 24617, 'mds_4': 24613}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14781': {'gid': 14781, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6839/1639930501', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6838', 'nonce': 1639930501}, {'type': 'v1', 'addr': '172.21.15.29:6839', 'nonce': 1639930501}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14793': {'gid': 14793, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6841/2982668187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6840', 'nonce': 2982668187}, {'type': 'v1', 'addr': '172.21.15.29:6841', 'nonce': 2982668187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24542': {'gid': 24542, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6833/2402830639', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6832', 'nonce': 2402830639}, {'type': 'v1', 'addr': '172.21.15.47:6833', 'nonce': 2402830639}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24566': {'gid': 24566, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6835/2900591116', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6834', 'nonce': 2900591116}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 2900591116}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24584': {'gid': 24584, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6837/2573307312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6836', 'nonce': 2573307312}, {'type': 'v1', 'addr': '172.21.15.47:6837', 'nonce': 2573307312}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24613': {'gid': 24613, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.77:6833/1085922080', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6832', 'nonce': 1085922080}, {'type': 'v1', 'addr': '172.21.15.77:6833', 'nonce': 1085922080}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24617': {'gid': 24617, 'name': 'b', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.47:6839/670107266', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6838', 'nonce': 670107266}, {'type': 'v1', 'addr': '172.21.15.47:6839', 'nonce': 670107266}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24646': {'gid': 24646, 'name': 'f', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.77:6835/439792826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 439792826}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 439792826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24658': {'gid': 24658, 'name': 'c', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.77:6837/2959628937', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2959628937}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2959628937}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24670': {'gid': 24670, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.77:6839/269899139', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 269899139}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 269899139}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24670, 'qdb_cluster': [24613, 24617, 24646, 24658, 24670]} 2024-07-27T06:45:44.231 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-27T06:45:44.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.l status 2024-07-27T06:45:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:44 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3431560994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:45:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:44 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3431560994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:45:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:44 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3431560994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:45:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:45 smithi077 ceph-mon[37153]: pgmap v359: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-27T06:45:45.685 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:45 smithi029 ceph-mon[32610]: pgmap v359: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-27T06:45:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:45 smithi047 ceph-mon[36835]: pgmap v359: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-27T06:45:46.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.212+0000 7fda35b3f640 1 -- 172.21.15.29:0/1112733191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda30108ef0 msgr2=0x7fda30109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:46.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.212+0000 7fda35b3f640 1 --2- 172.21.15.29:0/1112733191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda30108ef0 0x7fda30109300 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fda24009900 tx=0x7fda2402a410 comp rx=0 tx=0).stop 2024-07-27T06:45:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.212+0000 7fda35b3f640 1 -- 172.21.15.29:0/1112733191 shutdown_connections 2024-07-27T06:45:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.212+0000 7fda35b3f640 1 --2- 172.21.15.29:0/1112733191 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda30109840 0x7fda30110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.212+0000 7fda35b3f640 1 --2- 172.21.15.29:0/1112733191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda30108ef0 0x7fda30109300 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.212+0000 7fda35b3f640 1 --2- 172.21.15.29:0/1112733191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda30101a10 0x7fda30101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.212+0000 7fda35b3f640 1 -- 172.21.15.29:0/1112733191 >> 172.21.15.29:0/1112733191 conn(0x7fda300775f0 msgr2=0x7fda30077a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.212+0000 7fda35b3f640 1 -- 172.21.15.29:0/1112733191 shutdown_connections 2024-07-27T06:45:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.212+0000 7fda35b3f640 1 -- 172.21.15.29:0/1112733191 wait complete. 2024-07-27T06:45:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.213+0000 7fda35b3f640 1 Processor -- start 2024-07-27T06:45:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.213+0000 7fda35b3f640 1 -- start start 2024-07-27T06:45:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.213+0000 7fda35b3f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda30101a10 0x7fda301029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.213+0000 7fda35b3f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda30108ef0 0x7fda30102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.213+0000 7fda35b3f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda30109840 0x7fda30103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.213+0000 7fda35b3f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fda301134c0 con 0x7fda30109840 2024-07-27T06:45:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.213+0000 7fda35b3f640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fda30113340 con 0x7fda30108ef0 2024-07-27T06:45:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.213+0000 7fda35b3f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fda30113640 con 0x7fda30101a10 2024-07-27T06:45:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda30109840 0x7fda30103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda30109840 0x7fda30103430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50012/0 (socket says 172.21.15.29:50012) 2024-07-27T06:45:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2ffff640 1 -- 172.21.15.29:0/3177831831 learned_addr learned my addr 172.21.15.29:0/3177831831 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2effd640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda30108ef0 0x7fda30102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda30101a10 0x7fda301029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2ffff640 1 -- 172.21.15.29:0/3177831831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda30101a10 msgr2=0x7fda301029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2ffff640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda30101a10 0x7fda301029b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2ffff640 1 -- 172.21.15.29:0/3177831831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda30108ef0 msgr2=0x7fda30102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2ffff640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda30108ef0 0x7fda30102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2ffff640 1 -- 172.21.15.29:0/3177831831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda301a8170 con 0x7fda30109840 2024-07-27T06:45:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2ffff640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda30109840 0x7fda30103430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fda2800eaa0 tx=0x7fda2800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2cff9640 1 -- 172.21.15.29:0/3177831831 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2800cdb0 con 0x7fda30109840 2024-07-27T06:45:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda35b3f640 1 -- 172.21.15.29:0/3177831831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda301a8460 con 0x7fda30109840 2024-07-27T06:45:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2cff9640 1 -- 172.21.15.29:0/3177831831 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda28005020 con 0x7fda30109840 2024-07-27T06:45:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda2cff9640 1 -- 172.21.15.29:0/3177831831 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda280116d0 con 0x7fda30109840 2024-07-27T06:45:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.214+0000 7fda35b3f640 1 -- 172.21.15.29:0/3177831831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda301a8970 con 0x7fda30109840 2024-07-27T06:45:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.217+0000 7fda2cff9640 1 -- 172.21.15.29:0/3177831831 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fda280051d0 con 0x7fda30109840 2024-07-27T06:45:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.217+0000 7fda3c0c3740 1 Processor -- start 2024-07-27T06:45:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.217+0000 7fda3c0c3740 1 -- start start 2024-07-27T06:45:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.217+0000 7fda3c0c3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3210af60 0x561e324b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.217+0000 7fda3c0c3740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320e20a0 0x561e324b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.217+0000 7fda3c0c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 0x561e324b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.217+0000 7fda3c0c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561e32426940 con 0x561e320ea280 2024-07-27T06:45:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.217+0000 7fda3c0c3740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x561e32415e40 con 0x561e320e20a0 2024-07-27T06:45:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.217+0000 7fda3c0c3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561e32419e80 con 0x561e3210af60 2024-07-27T06:45:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 0x561e324b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 0x561e324b0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50014/0 (socket says 172.21.15.29:50014) 2024-07-27T06:45:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2effd640 1 -- 172.21.15.29:0/3385251032 learned_addr learned my addr 172.21.15.29:0/3385251032 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/3385251032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320e20a0 0x561e324b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2effd640 1 --2- 172.21.15.29:0/3385251032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3210af60 0x561e324b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2cff9640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fda180b3c20 0x7fda180b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2f7fe640 1 -- 172.21.15.29:0/3385251032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3210af60 msgr2=0x561e324b03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/3385251032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3210af60 0x561e324b03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2f7fe640 1 -- 172.21.15.29:0/3385251032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 msgr2=0x561e324b0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/3385251032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 0x561e324b0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2f7fe640 1 -- 172.21.15.29:0/3385251032 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e323bca60 con 0x561e320e20a0 2024-07-27T06:45:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2effd640 1 --2- 172.21.15.29:0/3385251032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 0x561e324b0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:45:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2cff9640 1 -- 172.21.15.29:0/3177831831 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fda280e4010 con 0x7fda30109840 2024-07-27T06:45:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2ffff640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fda180b3c20 0x7fda180b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.218+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/3385251032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320e20a0 0x561e324b08e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fda2000bbc0 tx=0x7fda2000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f13654e0872e622 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:46.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.219+0000 7fda15ffb640 1 -- 172.21.15.29:0/3385251032 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda20015070 con 0x561e320e20a0 2024-07-27T06:45:46.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.219+0000 7fda2ffff640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fda180b3c20 0x7fda180b6130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fda280e1a00 tx=0x7fda2800cf50 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:46.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.219+0000 7fda15ffb640 1 -- 172.21.15.29:0/3385251032 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda200035f0 con 0x561e320e20a0 2024-07-27T06:45:46.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.219+0000 7fda15ffb640 1 -- 172.21.15.29:0/3385251032 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda200116a0 con 0x561e320e20a0 2024-07-27T06:45:46.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.219+0000 7fda3c0c3740 1 -- 172.21.15.29:0/3385251032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320e20a0 msgr2=0x561e324b08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:46.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.219+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/3385251032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320e20a0 0x561e324b08e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fda2000bbc0 tx=0x7fda2000f730 comp rx=0 tx=0).stop 2024-07-27T06:45:46.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.219+0000 7fda3c0c3740 1 -- 172.21.15.29:0/3385251032 shutdown_connections 2024-07-27T06:45:46.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.219+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/3385251032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 0x561e324b0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.219+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/3385251032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320e20a0 0x561e324b08e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.219+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/3385251032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e3210af60 0x561e324b03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.219+0000 7fda3c0c3740 1 -- 172.21.15.29:0/3385251032 >> 172.21.15.29:0/3385251032 conn(0x561e320ff350 msgr2=0x561e324ae290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.220+0000 7fda3c0c3740 1 -- 172.21.15.29:0/3385251032 shutdown_connections 2024-07-27T06:45:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.220+0000 7fda3c0c3740 1 -- 172.21.15.29:0/3385251032 wait complete. 2024-07-27T06:45:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.220+0000 7fda3c0c3740 1 -- start start 2024-07-27T06:45:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.220+0000 7fda3c0c3740 1 Processor -- start 2024-07-27T06:45:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.220+0000 7fda3c0c3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320e20a0 0x561e324bbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda3c0c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 0x561e324bc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda3c0c3740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e3210af60 0x561e324c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda3c0c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561e322795f0 con 0x561e320ea280 2024-07-27T06:45:46.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda3c0c3740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x561e323b88b0 con 0x561e3210af60 2024-07-27T06:45:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda3c0c3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561e324beb80 con 0x561e320e20a0 2024-07-27T06:45:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda3c0c3740 10 client.0 _ll_register_callbacks cb 0x561e323c27f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:45:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 0x561e324bc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 0x561e324bc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50028/0 (socket says 172.21.15.29:50028) 2024-07-27T06:45:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda2f7fe640 1 -- 172.21.15.29:0/2506015129 learned_addr learned my addr 172.21.15.29:0/2506015129 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda2effd640 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e3210af60 0x561e324c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda2effd640 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320e20a0 0x561e324bbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda2f7fe640 1 -- 172.21.15.29:0/2506015129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320e20a0 msgr2=0x561e324bbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320e20a0 0x561e324bbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda2f7fe640 1 -- 172.21.15.29:0/2506015129 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e3210af60 msgr2=0x561e324c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e3210af60 0x561e324c7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda2f7fe640 1 -- 172.21.15.29:0/2506015129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e32176540 con 0x561e320ea280 2024-07-27T06:45:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 0x561e324bc570 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fda2000bee0 tx=0x7fda2000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6e8fd0358eba9e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda0effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:45:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda0effd640 1 -- 172.21.15.29:0/2506015129 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda20028060 con 0x561e320ea280 2024-07-27T06:45:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda0effd640 1 -- 172.21.15.29:0/2506015129 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fda200039b0 con 0x561e320ea280 2024-07-27T06:45:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.221+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2506015129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda200035f0 con 0x561e320ea280 2024-07-27T06:45:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.222+0000 7fda0effd640 1 -- 172.21.15.29:0/2506015129 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda20015070 con 0x561e320ea280 2024-07-27T06:45:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.222+0000 7fda3c0c3740 10 client.15159 fetch_fsmap learned FSMap version 28 2024-07-27T06:45:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.222+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2506015129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561e324cac10 con 0x561e320ea280 2024-07-27T06:45:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.222+0000 7fda0effd640 1 -- 172.21.15.29:0/2506015129 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda200035f0 con 0x561e320ea280 2024-07-27T06:45:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.222+0000 7fda0effd640 1 -- 172.21.15.29:0/2506015129 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fda200117f0 con 0x561e320ea280 2024-07-27T06:45:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.222+0000 7fda3c0c3740 10 client.15159 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:45:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.222+0000 7fda3c0c3740 10 client.15159 resolve_mds: resolved name 'l' to daemon mds.l 2024-07-27T06:45:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.222+0000 7fda3c0c3740 20 client.15159 populate_metadata read hostname 'smithi029' 2024-07-27T06:45:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.222+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x561e320c98e0 0x561e324ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.222+0000 7fda3c0c3740 4 client.15159 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:45:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.222+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2506015129 --> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e32428890 con 0x561e320c98e0 2024-07-27T06:45:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.222+0000 7fda2effd640 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x561e320c98e0 0x561e324ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.223+0000 7fda0effd640 10 client.15159 ms_handle_connect on v2:172.21.15.77:6832/1085922080 2024-07-27T06:45:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.223+0000 7fda2effd640 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x561e320c98e0 0x561e324ce480 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3230bfeff4500228 server_cookie=b6e7701b4d474341 in_seq=0 out_seq=0 2024-07-27T06:45:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda0effd640 1 -- 172.21.15.29:0/2506015129 <== mds.4 v2:172.21.15.77:6832/1085922080 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561e32428890 con 0x561e320c98e0 2024-07-27T06:45:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda0effd640 10 client.15159 handle_command_reply: tid=0 2024-07-27T06:45:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda3c0c3740 1 client.15159 shutdown 2024-07-27T06:45:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2506015129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 msgr2=0x561e324bc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 0x561e324bc570 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fda2000bee0 tx=0x7fda2000bf10 comp rx=0 tx=0).stop 2024-07-27T06:45:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2506015129 shutdown_connections 2024-07-27T06:45:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x561e320c98e0 0x561e324ce480 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e3210af60 0x561e324c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320ea280 0x561e324bc570 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda0effd640 0 client.15159 ms_handle_reset on v2:172.21.15.77:6832/1085922080 2024-07-27T06:45:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/2506015129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320e20a0 0x561e324bbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2506015129 >> 172.21.15.29:0/2506015129 conn(0x561e320ff350 msgr2=0x561e324b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:46.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2506015129 shutdown_connections 2024-07-27T06:45:46.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2506015129 wait complete. 2024-07-27T06:45:46.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.224+0000 7fda3c0c3740 20 client.15159 trim_cache size 0 max 16384 2024-07-27T06:45:46.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.243+0000 7fda3c0c3740 1 Processor -- start 2024-07-27T06:45:46.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.243+0000 7fda3c0c3740 1 -- start start 2024-07-27T06:45:46.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.243+0000 7fda3c0c3740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320c98e0 0x561e324be300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.243+0000 7fda3c0c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320e20a0 0x561e324c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.243+0000 7fda3c0c3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320ea280 0x561e324ba650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.243+0000 7fda3c0c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561e324cc7c0 con 0x561e320e20a0 2024-07-27T06:45:46.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.243+0000 7fda3c0c3740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x561e324cce90 con 0x561e320c98e0 2024-07-27T06:45:46.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.243+0000 7fda3c0c3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561e324cd560 con 0x561e320ea280 2024-07-27T06:45:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.243+0000 7fda2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320e20a0 0x561e324c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.243+0000 7fda2f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320e20a0 0x561e324c7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50030/0 (socket says 172.21.15.29:50030) 2024-07-27T06:45:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.243+0000 7fda2f7fe640 1 -- 172.21.15.29:0/3044314774 learned_addr learned my addr 172.21.15.29:0/3044314774 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.244+0000 7fda2f7fe640 1 -- 172.21.15.29:0/3044314774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320ea280 msgr2=0x561e324ba650 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-27T06:45:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.244+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/3044314774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320ea280 0x561e324ba650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.244+0000 7fda2f7fe640 1 -- 172.21.15.29:0/3044314774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320c98e0 msgr2=0x561e324be300 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-27T06:45:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.244+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/3044314774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320c98e0 0x561e324be300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.244+0000 7fda2f7fe640 1 -- 172.21.15.29:0/3044314774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e324aeb80 con 0x561e320e20a0 2024-07-27T06:45:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.244+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/3044314774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320e20a0 0x561e324c7de0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fda2000f770 tx=0x7fda2000b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bea92351aa7307 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.244+0000 7fda157fa640 1 -- 172.21.15.29:0/3044314774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda20028070 con 0x561e320e20a0 2024-07-27T06:45:46.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.244+0000 7fda157fa640 1 -- 172.21.15.29:0/3044314774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda20019410 con 0x561e320e20a0 2024-07-27T06:45:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.244+0000 7fda157fa640 1 -- 172.21.15.29:0/3044314774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda20003ae0 con 0x561e320e20a0 2024-07-27T06:45:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.245+0000 7fda3c0c3740 1 -- 172.21.15.29:0/3044314774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320e20a0 msgr2=0x561e324c7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.245+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/3044314774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320e20a0 0x561e324c7de0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fda2000f770 tx=0x7fda2000b3a0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.245+0000 7fda3c0c3740 1 -- 172.21.15.29:0/3044314774 shutdown_connections 2024-07-27T06:45:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.245+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/3044314774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320ea280 0x561e324ba650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.245+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/3044314774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320e20a0 0x561e324c7de0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.245+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/3044314774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320c98e0 0x561e324be300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.245+0000 7fda3c0c3740 1 -- 172.21.15.29:0/3044314774 >> 172.21.15.29:0/3044314774 conn(0x561e320ff350 msgr2=0x561e324b6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.245+0000 7fda3c0c3740 1 -- 172.21.15.29:0/3044314774 shutdown_connections 2024-07-27T06:45:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.245+0000 7fda3c0c3740 1 -- 172.21.15.29:0/3044314774 wait complete. 2024-07-27T06:45:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.245+0000 7fda3c0c3740 1 -- start start 2024-07-27T06:45:46.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda3c0c3740 1 Processor -- start 2024-07-27T06:45:46.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda3c0c3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320c98e0 0x561e324cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda3c0c3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320e20a0 0x561e324d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda3c0c3740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320ea280 0x561e3253df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda3c0c3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561e324ce300 con 0x561e320c98e0 2024-07-27T06:45:46.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda3c0c3740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x561e324cdc30 con 0x561e320ea280 2024-07-27T06:45:46.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda3c0c3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561e324ce9d0 con 0x561e320e20a0 2024-07-27T06:45:46.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda3c0c3740 10 client.0 _ll_register_callbacks cb 0x7fda2000ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:45:46.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda2effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320c98e0 0x561e324cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda2effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320c98e0 0x561e324cfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50038/0 (socket says 172.21.15.29:50038) 2024-07-27T06:45:46.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda2effd640 1 -- 172.21.15.29:0/2968106960 learned_addr learned my addr 172.21.15.29:0/2968106960 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:46.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda2effd640 1 --2- 172.21.15.29:0/2968106960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320ea280 0x561e3253df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda2effd640 1 -- 172.21.15.29:0/2968106960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320e20a0 msgr2=0x561e324d0030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T06:45:46.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda2effd640 1 --2- 172.21.15.29:0/2968106960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320e20a0 0x561e324d0030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda2effd640 1 -- 172.21.15.29:0/2968106960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320ea280 msgr2=0x561e3253df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:46.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.246+0000 7fda2effd640 1 --2- 172.21.15.29:0/2968106960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320ea280 0x561e3253df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.247+0000 7fda2effd640 1 -- 172.21.15.29:0/2968106960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e3253e460 con 0x561e320c98e0 2024-07-27T06:45:46.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.247+0000 7fda2effd640 1 --2- 172.21.15.29:0/2968106960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320c98e0 0x561e324cfab0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fda2402a930 tx=0x7fda2402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02c885016a40f35 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:46.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.247+0000 7fda0effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:45:46.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.247+0000 7fda0effd640 1 -- 172.21.15.29:0/2968106960 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2404e030 con 0x561e320c98e0 2024-07-27T06:45:46.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.247+0000 7fda0effd640 1 -- 172.21.15.29:0/2968106960 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fda24003b20 con 0x561e320c98e0 2024-07-27T06:45:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.247+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2968106960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561e32424330 con 0x561e320c98e0 2024-07-27T06:45:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.247+0000 7fda0effd640 1 -- 172.21.15.29:0/2968106960 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda24035040 con 0x561e320c98e0 2024-07-27T06:45:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.247+0000 7fda3c0c3740 10 client.15165 fetch_fsmap learned FSMap version 28 2024-07-27T06:45:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.247+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2968106960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561e3253f610 con 0x561e320c98e0 2024-07-27T06:45:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.247+0000 7fda0effd640 1 -- 172.21.15.29:0/2968106960 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561e32424330 con 0x561e320c98e0 2024-07-27T06:45:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.248+0000 7fda0effd640 1 -- 172.21.15.29:0/2968106960 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fda2403a040 con 0x561e320c98e0 2024-07-27T06:45:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.248+0000 7fda3c0c3740 10 client.15165 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:45:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.248+0000 7fda3c0c3740 10 client.15165 resolve_mds: resolved name 'l' to daemon mds.l 2024-07-27T06:45:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.248+0000 7fda3c0c3740 20 client.15165 populate_metadata read hostname 'smithi029' 2024-07-27T06:45:46.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.248+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/2968106960 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x561e3210af60 0x561e32542ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.248+0000 7fda3c0c3740 4 client.15165 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:45:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.248+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2968106960 --> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] -- command(tid 0: {"prefix": "status"}) -- 0x561e32421a70 con 0x561e3210af60 2024-07-27T06:45:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.248+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/2968106960 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x561e3210af60 0x561e32542ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.248+0000 7fda2f7fe640 1 --2- 172.21.15.29:0/2968106960 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x561e3210af60 0x561e32542ee0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b67cfa80b45334ab server_cookie=ca3bbea507d6eaa0 in_seq=0 out_seq=0 2024-07-27T06:45:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.248+0000 7fda0effd640 10 client.15165 ms_handle_connect on v2:172.21.15.77:6832/1085922080 2024-07-27T06:45:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda0effd640 1 -- 172.21.15.29:0/2968106960 <== mds.4 v2:172.21.15.77:6832/1085922080 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561e32421a70 con 0x561e3210af60 2024-07-27T06:45:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda0effd640 10 client.15165 handle_command_reply: tid=0 2024-07-27T06:45:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda3c0c3740 1 client.15165 shutdown 2024-07-27T06:45:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2968106960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320c98e0 msgr2=0x561e324cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/2968106960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320c98e0 0x561e324cfab0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fda2402a930 tx=0x7fda2402a410 comp rx=0 tx=0).stop 2024-07-27T06:45:46.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2968106960 shutdown_connections 2024-07-27T06:45:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/2968106960 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x561e3210af60 0x561e32542ee0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/2968106960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561e320ea280 0x561e3253df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda0effd640 0 client.15165 ms_handle_reset on v2:172.21.15.77:6832/1085922080 2024-07-27T06:45:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/2968106960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561e320e20a0 0x561e324d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda3c0c3740 1 --2- 172.21.15.29:0/2968106960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561e320c98e0 0x561e324cfab0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2968106960 >> 172.21.15.29:0/2968106960 conn(0x561e320ff350 msgr2=0x561e324ba580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2968106960 shutdown_connections 2024-07-27T06:45:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda3c0c3740 1 -- 172.21.15.29:0/2968106960 wait complete. 2024-07-27T06:45:46.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.249+0000 7fda3c0c3740 20 client.15165 trim_cache size 0 max 16384 2024-07-27T06:45:46.261 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:45:46.261 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:45:46.261 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 4, 2024-07-27T06:45:46.261 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24613, 2024-07-27T06:45:46.261 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:45:46.261 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:45:46.261 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:45:46.261 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 133.387759593, 2024-07-27T06:45:46.261 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-07-27T06:45:46.261 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:45:46.262 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 79, 2024-07-27T06:45:46.262 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 204.36752805099999 2024-07-27T06:45:46.262 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:45:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.250+0000 7fda35b3f640 1 -- 172.21.15.29:0/3177831831 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fda180b3c20 msgr2=0x7fda180b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.250+0000 7fda35b3f640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fda180b3c20 0x7fda180b6130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fda280e1a00 tx=0x7fda2800cf50 comp rx=0 tx=0).stop 2024-07-27T06:45:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.250+0000 7fda35b3f640 1 -- 172.21.15.29:0/3177831831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda30109840 msgr2=0x7fda30103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.250+0000 7fda35b3f640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda30109840 0x7fda30103430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fda2800eaa0 tx=0x7fda2800ef60 comp rx=0 tx=0).stop 2024-07-27T06:45:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.250+0000 7fda35b3f640 1 -- 172.21.15.29:0/3177831831 shutdown_connections 2024-07-27T06:45:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.250+0000 7fda35b3f640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda30109840 0x7fda30103430 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.250+0000 7fda35b3f640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda30108ef0 0x7fda30102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.250+0000 7fda35b3f640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fda180b3c20 0x7fda180b6130 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.250+0000 7fda35b3f640 1 --2- 172.21.15.29:0/3177831831 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda30101a10 0x7fda301029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:46.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.250+0000 7fda35b3f640 1 -- 172.21.15.29:0/3177831831 >> 172.21.15.29:0/3177831831 conn(0x7fda300775f0 msgr2=0x7fda30106f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:46.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.250+0000 7fda35b3f640 1 -- 172.21.15.29:0/3177831831 shutdown_connections 2024-07-27T06:45:46.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:46.250+0000 7fda35b3f640 1 -- 172.21.15.29:0/3177831831 wait complete. 2024-07-27T06:45:47.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.b status 2024-07-27T06:45:47.305 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:47.553 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:47 smithi029 ceph-mon[32610]: pgmap v360: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:45:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:47 smithi077 ceph-mon[37153]: pgmap v360: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:45:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:47 smithi047 ceph-mon[36835]: pgmap v360: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:45:49.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.174+0000 7f174a857640 1 -- 172.21.15.29:0/4082024901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1744109820 msgr2=0x7f1744110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:49.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.174+0000 7f174a857640 1 --2- 172.21.15.29:0/4082024901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1744109820 0x7f1744110050 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f173c009ca0 tx=0x7f173c02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.174+0000 7f174a857640 1 -- 172.21.15.29:0/4082024901 shutdown_connections 2024-07-27T06:45:49.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.174+0000 7f174a857640 1 --2- 172.21.15.29:0/4082024901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1744109820 0x7f1744110050 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.174+0000 7f174a857640 1 --2- 172.21.15.29:0/4082024901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1744108ed0 0x7f17441092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.174+0000 7f174a857640 1 --2- 172.21.15.29:0/4082024901 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1744103590 0x7f17441039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.174+0000 7f174a857640 1 -- 172.21.15.29:0/4082024901 >> 172.21.15.29:0/4082024901 conn(0x7f174407b280 msgr2=0x7f174407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:49.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.174+0000 7f174a857640 1 -- 172.21.15.29:0/4082024901 shutdown_connections 2024-07-27T06:45:49.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.174+0000 7f174a857640 1 -- 172.21.15.29:0/4082024901 wait complete. 2024-07-27T06:45:49.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.175+0000 7f174a857640 1 Processor -- start 2024-07-27T06:45:49.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.175+0000 7f174a857640 1 -- start start 2024-07-27T06:45:49.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f174a857640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1744103590 0x7f1744199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f174a857640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1744108ed0 0x7f174419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f174a857640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1744109820 0x7f174419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f174a857640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17441134d0 con 0x7f1744109820 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f174a857640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f1744113350 con 0x7f1744108ed0 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f174a857640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1744113650 con 0x7f1744103590 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f1748df4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1744109820 0x7f174419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f1748df4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1744109820 0x7f174419f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47558/0 (socket says 172.21.15.29:47558) 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f1748df4640 1 -- 172.21.15.29:0/941818592 learned_addr learned my addr 172.21.15.29:0/941818592 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f1743fff640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1744103590 0x7f1744199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f17437fe640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1744108ed0 0x7f174419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f1748df4640 1 -- 172.21.15.29:0/941818592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1744103590 msgr2=0x7f1744199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f1748df4640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1744103590 0x7f1744199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f1748df4640 1 -- 172.21.15.29:0/941818592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1744108ed0 msgr2=0x7f174419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f1748df4640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1744108ed0 0x7f174419a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f1748df4640 1 -- 172.21.15.29:0/941818592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174419fdf0 con 0x7f1744109820 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f17437fe640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1744108ed0 0x7f174419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.176+0000 7f1748df4640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1744109820 0x7f174419f6c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f173c0098d0 tx=0x7f173c0317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.177+0000 7f17417fa640 1 -- 172.21.15.29:0/941818592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c003b70 con 0x7f1744109820 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.177+0000 7f17417fa640 1 -- 172.21.15.29:0/941818592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f173c003d10 con 0x7f1744109820 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.177+0000 7f174a857640 1 -- 172.21.15.29:0/941818592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17441a0080 con 0x7f1744109820 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.177+0000 7f17417fa640 1 -- 172.21.15.29:0/941818592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c02fa20 con 0x7f1744109820 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.177+0000 7f174a857640 1 -- 172.21.15.29:0/941818592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17441a6840 con 0x7f1744109820 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.179+0000 7f1750ddb740 1 Processor -- start 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.180+0000 7f17417fa640 1 -- 172.21.15.29:0/941818592 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f173c02fc00 con 0x7f1744109820 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.180+0000 7f1750ddb740 1 -- start start 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.180+0000 7f1750ddb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 0x55daf8d2f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.180+0000 7f1750ddb740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf89610a0 0x55daf8d2f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.180+0000 7f1750ddb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf8969280 0x55daf8d2fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.180+0000 7f1750ddb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55daf8ca5920 con 0x55daf8989f60 2024-07-27T06:45:49.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.180+0000 7f1750ddb740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55daf8c94e20 con 0x55daf89610a0 2024-07-27T06:45:49.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.180+0000 7f1750ddb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55daf8c98e60 con 0x55daf8969280 2024-07-27T06:45:49.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.180+0000 7f17437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 0x55daf8d2f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.180+0000 7f17437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 0x55daf8d2f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47572/0 (socket says 172.21.15.29:47572) 2024-07-27T06:45:49.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.180+0000 7f17437fe640 1 -- 172.21.15.29:0/1726226928 learned_addr learned my addr 172.21.15.29:0/1726226928 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:49.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.180+0000 7f1743fff640 1 --2- 172.21.15.29:0/1726226928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf89610a0 0x55daf8d2f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f17437fe640 1 --2- 172.21.15.29:0/1726226928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf8969280 0x55daf8d2fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f17437fe640 1 -- 172.21.15.29:0/1726226928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf8969280 msgr2=0x55daf8d2fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:49.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f17437fe640 1 --2- 172.21.15.29:0/1726226928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf8969280 0x55daf8d2fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f17437fe640 1 -- 172.21.15.29:0/1726226928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf89610a0 msgr2=0x55daf8d2f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:49.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f17437fe640 1 --2- 172.21.15.29:0/1726226928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf89610a0 0x55daf8d2f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f17437fe640 1 -- 172.21.15.29:0/1726226928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daf8c3ba40 con 0x55daf8989f60 2024-07-27T06:45:49.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f17417fa640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f172c0b3c20 0x7f172c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f17437fe640 1 --2- 172.21.15.29:0/1726226928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 0x55daf8d2f380 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f173800bfd0 tx=0x7f173800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86b39c9ed9480df server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:49.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f1743fff640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f172c0b3c20 0x7f172c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f17417fa640 1 -- 172.21.15.29:0/941818592 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f173c1052f0 con 0x7f1744109820 2024-07-27T06:45:49.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f172a7fc640 1 -- 172.21.15.29:0/1726226928 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738019070 con 0x55daf8989f60 2024-07-27T06:45:49.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f172a7fc640 1 -- 172.21.15.29:0/1726226928 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17380054d0 con 0x55daf8989f60 2024-07-27T06:45:49.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.181+0000 7f172a7fc640 1 -- 172.21.15.29:0/1726226928 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738004c30 con 0x55daf8989f60 2024-07-27T06:45:49.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.182+0000 7f1750ddb740 1 -- 172.21.15.29:0/1726226928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 msgr2=0x55daf8d2f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:49.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.182+0000 7f1743fff640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f172c0b3c20 0x7f172c0b6130 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55daf8d2e000 tx=0x7f1734005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:49.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.182+0000 7f1750ddb740 1 --2- 172.21.15.29:0/1726226928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 0x55daf8d2f380 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f173800bfd0 tx=0x7f173800b650 comp rx=0 tx=0).stop 2024-07-27T06:45:49.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.182+0000 7f1750ddb740 1 -- 172.21.15.29:0/1726226928 shutdown_connections 2024-07-27T06:45:49.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.182+0000 7f1750ddb740 1 --2- 172.21.15.29:0/1726226928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf8969280 0x55daf8d2fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.182+0000 7f1750ddb740 1 --2- 172.21.15.29:0/1726226928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf89610a0 0x55daf8d2f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.182+0000 7f1750ddb740 1 --2- 172.21.15.29:0/1726226928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 0x55daf8d2f380 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.182+0000 7f1750ddb740 1 -- 172.21.15.29:0/1726226928 >> 172.21.15.29:0/1726226928 conn(0x55daf897e350 msgr2=0x55daf8d2d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:49.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.182+0000 7f1750ddb740 1 -- 172.21.15.29:0/1726226928 shutdown_connections 2024-07-27T06:45:49.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.182+0000 7f1750ddb740 1 -- 172.21.15.29:0/1726226928 wait complete. 2024-07-27T06:45:49.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.182+0000 7f1750ddb740 1 -- start start 2024-07-27T06:45:49.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f1750ddb740 1 Processor -- start 2024-07-27T06:45:49.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f1750ddb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf89610a0 0x55daf8d3afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f1750ddb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 0x55daf8d3b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f1750ddb740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf8989f60 0x55daf8d46e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f1750ddb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55daf8af85f0 con 0x55daf8969280 2024-07-27T06:45:49.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f1750ddb740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55daf8c37890 con 0x55daf8989f60 2024-07-27T06:45:49.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f1750ddb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55daf8d3db60 con 0x55daf89610a0 2024-07-27T06:45:49.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f1750ddb740 10 client.0 _ll_register_callbacks cb 0x55daf8c417d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:45:49.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f17437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 0x55daf8d3b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f17437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 0x55daf8d3b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47574/0 (socket says 172.21.15.29:47574) 2024-07-27T06:45:49.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f17437fe640 1 -- 172.21.15.29:0/1673110085 learned_addr learned my addr 172.21.15.29:0/1673110085 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:49.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f1748df4640 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf8989f60 0x55daf8d46e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.183+0000 7f1743fff640 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf89610a0 0x55daf8d3afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.184+0000 7f17437fe640 1 -- 172.21.15.29:0/1673110085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf89610a0 msgr2=0x55daf8d3afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.184+0000 7f17437fe640 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf89610a0 0x55daf8d3afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.184+0000 7f17437fe640 1 -- 172.21.15.29:0/1673110085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf8989f60 msgr2=0x55daf8d46e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.184+0000 7f17437fe640 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf8989f60 0x55daf8d46e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.184+0000 7f17437fe640 1 -- 172.21.15.29:0/1673110085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daf89f5540 con 0x55daf8969280 2024-07-27T06:45:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.184+0000 7f17437fe640 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 0x55daf8d3b520 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1738004250 tx=0x7f1738004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113c7b4c6e941cf2 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.184+0000 7f17237fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:45:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.184+0000 7f17237fe640 1 -- 172.21.15.29:0/1673110085 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173802b060 con 0x55daf8969280 2024-07-27T06:45:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.184+0000 7f17237fe640 1 -- 172.21.15.29:0/1673110085 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f1738003800 con 0x55daf8969280 2024-07-27T06:45:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.184+0000 7f1750ddb740 1 -- 172.21.15.29:0/1673110085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17380054d0 con 0x55daf8969280 2024-07-27T06:45:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.184+0000 7f17237fe640 1 -- 172.21.15.29:0/1673110085 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1738019040 con 0x55daf8969280 2024-07-27T06:45:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.185+0000 7f1750ddb740 10 client.15177 fetch_fsmap learned FSMap version 28 2024-07-27T06:45:49.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.185+0000 7f1750ddb740 1 -- 172.21.15.29:0/1673110085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55daf8d49bd0 con 0x55daf8969280 2024-07-27T06:45:49.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.185+0000 7f17237fe640 1 -- 172.21.15.29:0/1673110085 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17380054d0 con 0x55daf8969280 2024-07-27T06:45:49.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.185+0000 7f17237fe640 1 -- 172.21.15.29:0/1673110085 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1738004ad0 con 0x55daf8969280 2024-07-27T06:45:49.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.185+0000 7f1750ddb740 10 client.15177 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:45:49.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.185+0000 7f1750ddb740 10 client.15177 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-27T06:45:49.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.185+0000 7f1750ddb740 20 client.15177 populate_metadata read hostname 'smithi029' 2024-07-27T06:45:49.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.185+0000 7f1750ddb740 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55daf89488e0 0x55daf8d4d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.185+0000 7f1750ddb740 4 client.15177 mds_command: new command op to 24617 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:45:49.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.185+0000 7f1750ddb740 1 -- 172.21.15.29:0/1673110085 --> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55daf8ca7870 con 0x55daf89488e0 2024-07-27T06:45:49.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.185+0000 7f1743fff640 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55daf89488e0 0x55daf8d4d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.186+0000 7f1743fff640 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55daf89488e0 0x55daf8d4d210 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=687dac008359ad12 server_cookie=6ca912236d292077 in_seq=0 out_seq=0 2024-07-27T06:45:49.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.186+0000 7f17237fe640 10 client.15177 ms_handle_connect on v2:172.21.15.47:6838/670107266 2024-07-27T06:45:49.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f17237fe640 1 -- 172.21.15.29:0/1673110085 <== mds.3 v2:172.21.15.47:6838/670107266 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55daf8ca7870 con 0x55daf89488e0 2024-07-27T06:45:49.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f17237fe640 10 client.15177 handle_command_reply: tid=0 2024-07-27T06:45:49.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f1750ddb740 1 client.15177 shutdown 2024-07-27T06:45:49.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f1750ddb740 1 -- 172.21.15.29:0/1673110085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 msgr2=0x55daf8d3b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:49.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f1750ddb740 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 0x55daf8d3b520 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1738004250 tx=0x7f1738004280 comp rx=0 tx=0).stop 2024-07-27T06:45:49.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f1750ddb740 1 -- 172.21.15.29:0/1673110085 shutdown_connections 2024-07-27T06:45:49.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f1750ddb740 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55daf89488e0 0x55daf8d4d210 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f1750ddb740 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf8989f60 0x55daf8d46e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f1750ddb740 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 0x55daf8d3b520 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f1750ddb740 1 --2- 172.21.15.29:0/1673110085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf89610a0 0x55daf8d3afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f1750ddb740 1 -- 172.21.15.29:0/1673110085 >> 172.21.15.29:0/1673110085 conn(0x55daf897e350 msgr2=0x55daf8d32560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:49.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f17237fe640 0 client.15177 ms_handle_reset on v2:172.21.15.47:6838/670107266 2024-07-27T06:45:49.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f1750ddb740 1 -- 172.21.15.29:0/1673110085 shutdown_connections 2024-07-27T06:45:49.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f1750ddb740 1 -- 172.21.15.29:0/1673110085 wait complete. 2024-07-27T06:45:49.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.187+0000 7f1750ddb740 20 client.15177 trim_cache size 0 max 16384 2024-07-27T06:45:49.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.205+0000 7f1750ddb740 1 Processor -- start 2024-07-27T06:45:49.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.205+0000 7f1750ddb740 1 -- start start 2024-07-27T06:45:49.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f1750ddb740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf89610a0 0x55daf8d34600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f1750ddb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 0x55daf8d34b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f1750ddb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf8989f60 0x55daf8d35f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f1750ddb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55daf8d4b550 con 0x55daf8969280 2024-07-27T06:45:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f1750ddb740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55daf8d4bc20 con 0x55daf89610a0 2024-07-27T06:45:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f1750ddb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55daf8d4c2f0 con 0x55daf8989f60 2024-07-27T06:45:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f17437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 0x55daf8d34b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f17437fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 0x55daf8d34b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47588/0 (socket says 172.21.15.29:47588) 2024-07-27T06:45:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f17437fe640 1 -- 172.21.15.29:0/618757066 learned_addr learned my addr 172.21.15.29:0/618757066 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f1748df4640 1 --2- 172.21.15.29:0/618757066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf8989f60 0x55daf8d35f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f1743fff640 1 --2- 172.21.15.29:0/618757066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf89610a0 0x55daf8d34600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f17437fe640 1 -- 172.21.15.29:0/618757066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf8989f60 msgr2=0x55daf8d35f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f17437fe640 1 --2- 172.21.15.29:0/618757066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf8989f60 0x55daf8d35f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f17437fe640 1 -- 172.21.15.29:0/618757066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf89610a0 msgr2=0x55daf8d34600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f17437fe640 1 --2- 172.21.15.29:0/618757066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf89610a0 0x55daf8d34600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f17437fe640 1 -- 172.21.15.29:0/618757066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daf8d28a40 con 0x55daf8969280 2024-07-27T06:45:49.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.206+0000 7f17437fe640 1 --2- 172.21.15.29:0/618757066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 0x55daf8d34b40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1738006010 tx=0x7f173800b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd40d69ee2a7305 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:49.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.207+0000 7f1729ffb640 1 -- 172.21.15.29:0/618757066 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173802b050 con 0x55daf8969280 2024-07-27T06:45:49.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.207+0000 7f1729ffb640 1 -- 172.21.15.29:0/618757066 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1738004640 con 0x55daf8969280 2024-07-27T06:45:49.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.207+0000 7f1729ffb640 1 -- 172.21.15.29:0/618757066 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738003800 con 0x55daf8969280 2024-07-27T06:45:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.207+0000 7f1750ddb740 1 -- 172.21.15.29:0/618757066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 msgr2=0x55daf8d34b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.207+0000 7f1750ddb740 1 --2- 172.21.15.29:0/618757066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 0x55daf8d34b40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1738006010 tx=0x7f173800b760 comp rx=0 tx=0).stop 2024-07-27T06:45:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.207+0000 7f1750ddb740 1 -- 172.21.15.29:0/618757066 shutdown_connections 2024-07-27T06:45:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.207+0000 7f1750ddb740 1 --2- 172.21.15.29:0/618757066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf8989f60 0x55daf8d35f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.207+0000 7f1750ddb740 1 --2- 172.21.15.29:0/618757066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8969280 0x55daf8d34b40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.207+0000 7f1750ddb740 1 --2- 172.21.15.29:0/618757066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf89610a0 0x55daf8d34600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.207+0000 7f1750ddb740 1 -- 172.21.15.29:0/618757066 >> 172.21.15.29:0/618757066 conn(0x55daf897e350 msgr2=0x55daf8d3afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.208+0000 7f1750ddb740 1 -- 172.21.15.29:0/618757066 shutdown_connections 2024-07-27T06:45:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.208+0000 7f1750ddb740 1 -- 172.21.15.29:0/618757066 wait complete. 2024-07-27T06:45:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.208+0000 7f1750ddb740 1 -- start start 2024-07-27T06:45:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.208+0000 7f1750ddb740 1 Processor -- start 2024-07-27T06:45:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1750ddb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf89610a0 0x55daf8d2b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1750ddb740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf8969280 0x55daf8d2bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1750ddb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 0x55daf8d2c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1750ddb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55daf8d4d090 con 0x55daf8989f60 2024-07-27T06:45:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1750ddb740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55daf8d4c9c0 con 0x55daf8969280 2024-07-27T06:45:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1750ddb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55daf8d4d760 con 0x55daf89610a0 2024-07-27T06:45:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1750ddb740 10 client.0 _ll_register_callbacks cb 0x7f17380044f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:45:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1748df4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 0x55daf8d2c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1748df4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 0x55daf8d2c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47592/0 (socket says 172.21.15.29:47592) 2024-07-27T06:45:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f17437fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf8969280 0x55daf8d2bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1748df4640 1 -- 172.21.15.29:0/2908566644 learned_addr learned my addr 172.21.15.29:0/2908566644 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1743fff640 1 --2- 172.21.15.29:0/2908566644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf89610a0 0x55daf8d2b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1748df4640 1 -- 172.21.15.29:0/2908566644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf89610a0 msgr2=0x55daf8d2b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1748df4640 1 --2- 172.21.15.29:0/2908566644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf89610a0 0x55daf8d2b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1748df4640 1 -- 172.21.15.29:0/2908566644 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf8969280 msgr2=0x55daf8d2bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1748df4640 1 --2- 172.21.15.29:0/2908566644 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf8969280 0x55daf8d2bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1748df4640 1 -- 172.21.15.29:0/2908566644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daf8d54460 con 0x55daf8989f60 2024-07-27T06:45:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f1748df4640 1 --2- 172.21.15.29:0/2908566644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 0x55daf8d2c2b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f173c0c9340 tx=0x7f173c0cb840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae1d4a075195af6 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.209+0000 7f17237fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:45:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f17237fe640 1 -- 172.21.15.29:0/2908566644 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c0e5070 con 0x55daf8989f60 2024-07-27T06:45:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f17237fe640 1 -- 172.21.15.29:0/2908566644 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f173c091300 con 0x55daf8989f60 2024-07-27T06:45:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f1750ddb740 1 -- 172.21.15.29:0/2908566644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1738004640 con 0x55daf8989f60 2024-07-27T06:45:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f17237fe640 1 -- 172.21.15.29:0/2908566644 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f173c0b3040 con 0x55daf8989f60 2024-07-27T06:45:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f1750ddb740 10 client.15183 fetch_fsmap learned FSMap version 28 2024-07-27T06:45:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f1750ddb740 1 -- 172.21.15.29:0/2908566644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55daf8d556e0 con 0x55daf8989f60 2024-07-27T06:45:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f17237fe640 1 -- 172.21.15.29:0/2908566644 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1738004640 con 0x55daf8989f60 2024-07-27T06:45:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f17237fe640 1 -- 172.21.15.29:0/2908566644 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f173c0cc3c0 con 0x55daf8989f60 2024-07-27T06:45:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f1750ddb740 10 client.15183 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:45:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f1750ddb740 10 client.15183 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-27T06:45:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f1750ddb740 20 client.15183 populate_metadata read hostname 'smithi029' 2024-07-27T06:45:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f1750ddb740 1 --2- 172.21.15.29:0/2908566644 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55daf891dc90 0x55daf8db9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f1750ddb740 4 client.15183 mds_command: new command op to 24617 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:45:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.210+0000 7f1750ddb740 1 -- 172.21.15.29:0/2908566644 --> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] -- command(tid 0: {"prefix": "status"}) -- 0x55daf8ca3310 con 0x55daf891dc90 2024-07-27T06:45:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.211+0000 7f1743fff640 1 --2- 172.21.15.29:0/2908566644 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55daf891dc90 0x55daf8db9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.211+0000 7f1743fff640 1 --2- 172.21.15.29:0/2908566644 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55daf891dc90 0x55daf8db9130 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a732044e6cac22e2 server_cookie=94fe676cc07910e2 in_seq=0 out_seq=0 2024-07-27T06:45:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.211+0000 7f17237fe640 10 client.15183 ms_handle_connect on v2:172.21.15.47:6838/670107266 2024-07-27T06:45:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.211+0000 7f17237fe640 1 -- 172.21.15.29:0/2908566644 <== mds.3 v2:172.21.15.47:6838/670107266 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55daf8ca3310 con 0x55daf891dc90 2024-07-27T06:45:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.211+0000 7f17237fe640 10 client.15183 handle_command_reply: tid=0 2024-07-27T06:45:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f1750ddb740 1 client.15183 shutdown 2024-07-27T06:45:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f1750ddb740 1 -- 172.21.15.29:0/2908566644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 msgr2=0x55daf8d2c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f1750ddb740 1 --2- 172.21.15.29:0/2908566644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 0x55daf8d2c2b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f173c0c9340 tx=0x7f173c0cb840 comp rx=0 tx=0).stop 2024-07-27T06:45:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f1750ddb740 1 -- 172.21.15.29:0/2908566644 shutdown_connections 2024-07-27T06:45:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f1750ddb740 1 --2- 172.21.15.29:0/2908566644 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55daf891dc90 0x55daf8db9130 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f1750ddb740 1 --2- 172.21.15.29:0/2908566644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55daf8989f60 0x55daf8d2c2b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f1750ddb740 1 --2- 172.21.15.29:0/2908566644 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55daf8969280 0x55daf8d2bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f17237fe640 0 client.15183 ms_handle_reset on v2:172.21.15.47:6838/670107266 2024-07-27T06:45:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f1750ddb740 1 --2- 172.21.15.29:0/2908566644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55daf89610a0 0x55daf8d2b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f1750ddb740 1 -- 172.21.15.29:0/2908566644 >> 172.21.15.29:0/2908566644 conn(0x55daf897e350 msgr2=0x55daf8d39240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:49.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f1750ddb740 1 -- 172.21.15.29:0/2908566644 shutdown_connections 2024-07-27T06:45:49.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f1750ddb740 1 -- 172.21.15.29:0/2908566644 wait complete. 2024-07-27T06:45:49.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.212+0000 7f1750ddb740 20 client.15183 trim_cache size 0 max 16384 2024-07-27T06:45:49.221 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:45:49.221 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:45:49.221 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 3, 2024-07-27T06:45:49.221 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24617, 2024-07-27T06:45:49.221 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:45:49.222 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:45:49.222 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:45:49.222 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 137.35936296700001, 2024-07-27T06:45:49.222 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-07-27T06:45:49.222 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:45:49.222 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 79, 2024-07-27T06:45:49.222 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 172.50599606899999 2024-07-27T06:45:49.222 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:45:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.213+0000 7f174a857640 1 -- 172.21.15.29:0/941818592 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f172c0b3c20 msgr2=0x7f172c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.213+0000 7f174a857640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f172c0b3c20 0x7f172c0b6130 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55daf8d2e000 tx=0x7f1734005f90 comp rx=0 tx=0).stop 2024-07-27T06:45:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.213+0000 7f174a857640 1 -- 172.21.15.29:0/941818592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1744109820 msgr2=0x7f174419f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.213+0000 7f174a857640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1744109820 0x7f174419f6c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f173c0098d0 tx=0x7f173c0317b0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.213+0000 7f174a857640 1 -- 172.21.15.29:0/941818592 shutdown_connections 2024-07-27T06:45:49.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.213+0000 7f174a857640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1744109820 0x7f174419f6c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.213+0000 7f174a857640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1744108ed0 0x7f174419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.213+0000 7f174a857640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f172c0b3c20 0x7f172c0b6130 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.213+0000 7f174a857640 1 --2- 172.21.15.29:0/941818592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1744103590 0x7f1744199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.213+0000 7f174a857640 1 -- 172.21.15.29:0/941818592 >> 172.21.15.29:0/941818592 conn(0x7f174407b280 msgr2=0x7f1744106ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.213+0000 7f174a857640 1 -- 172.21.15.29:0/941818592 shutdown_connections 2024-07-27T06:45:49.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:49.213+0000 7f174a857640 1 -- 172.21.15.29:0/941818592 wait complete. 2024-07-27T06:45:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:49 smithi029 ceph-mon[32610]: pgmap v361: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:45:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:49 smithi077 ceph-mon[37153]: pgmap v361: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:45:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:49 smithi047 ceph-mon[36835]: pgmap v361: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:45:49.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.f status 2024-07-27T06:45:50.216 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:51 smithi077 ceph-mon[37153]: pgmap v362: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:45:51.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:51 smithi029 ceph-mon[32610]: pgmap v362: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:45:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:51 smithi047 ceph-mon[36835]: pgmap v362: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:45:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.021+0000 7f7fee4f8640 1 -- 172.21.15.29:0/1883927909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe8107de0 msgr2=0x7f7fe81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.021+0000 7f7fee4f8640 1 --2- 172.21.15.29:0/1883927909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe8107de0 0x7f7fe81081f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009900 tx=0x7f7fdc02a410 comp rx=0 tx=0).stop 2024-07-27T06:45:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.022+0000 7f7fee4f8640 1 -- 172.21.15.29:0/1883927909 shutdown_connections 2024-07-27T06:45:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.022+0000 7f7fee4f8640 1 --2- 172.21.15.29:0/1883927909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fe8108730 0x7f7fe810ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.022+0000 7f7fee4f8640 1 --2- 172.21.15.29:0/1883927909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe8107de0 0x7f7fe81081f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.022+0000 7f7fee4f8640 1 --2- 172.21.15.29:0/1883927909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fe81024a0 0x7f7fe81028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.022+0000 7f7fee4f8640 1 -- 172.21.15.29:0/1883927909 >> 172.21.15.29:0/1883927909 conn(0x7f7fe8079140 msgr2=0x7f7fe8079550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.022+0000 7f7fee4f8640 1 -- 172.21.15.29:0/1883927909 shutdown_connections 2024-07-27T06:45:52.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.022+0000 7f7fee4f8640 1 -- 172.21.15.29:0/1883927909 wait complete. 2024-07-27T06:45:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.022+0000 7f7fee4f8640 1 Processor -- start 2024-07-27T06:45:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.022+0000 7f7fee4f8640 1 -- start start 2024-07-27T06:45:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fee4f8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fe81024a0 0x7f7fe8198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fee4f8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe8107de0 0x7f7fe8199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fee4f8640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fe8108730 0x7f7fe819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fee4f8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe81123a0 con 0x7f7fe8107de0 2024-07-27T06:45:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fee4f8640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe8112220 con 0x7f7fe8108730 2024-07-27T06:45:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fee4f8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7fe8112520 con 0x7f7fe81024a0 2024-07-27T06:45:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fe77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe8107de0 0x7f7fe8199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fe77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe8107de0 0x7f7fe8199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47596/0 (socket says 172.21.15.29:47596) 2024-07-27T06:45:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fe77fe640 1 -- 172.21.15.29:0/3017953853 learned_addr learned my addr 172.21.15.29:0/3017953853 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fe81024a0 0x7f7fe8198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7feca95640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fe8108730 0x7f7fe819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fe77fe640 1 -- 172.21.15.29:0/3017953853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fe81024a0 msgr2=0x7f7fe8198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fe77fe640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fe81024a0 0x7f7fe8198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fe77fe640 1 -- 172.21.15.29:0/3017953853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fe8108730 msgr2=0x7f7fe819e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fe77fe640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fe8108730 0x7f7fe819e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7fe77fe640 1 -- 172.21.15.29:0/3017953853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fe819eca0 con 0x7f7fe8107de0 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.023+0000 7f7feca95640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fe8108730 0x7f7fe819e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.024+0000 7f7fe77fe640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe8107de0 0x7f7fe8199390 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc02a930 tx=0x7f7fdc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.024+0000 7f7fe57fa640 1 -- 172.21.15.29:0/3017953853 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc03f030 con 0x7f7fe8107de0 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.024+0000 7f7fe57fa640 1 -- 172.21.15.29:0/3017953853 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fdc003aa0 con 0x7f7fe8107de0 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.024+0000 7f7fe57fa640 1 -- 172.21.15.29:0/3017953853 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc035070 con 0x7f7fe8107de0 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.024+0000 7f7fee4f8640 1 -- 172.21.15.29:0/3017953853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fe819ef30 con 0x7f7fe8107de0 2024-07-27T06:45:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.024+0000 7f7fee4f8640 1 -- 172.21.15.29:0/3017953853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fe819f3c0 con 0x7f7fe8107de0 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.026+0000 7f7fe57fa640 1 -- 172.21.15.29:0/3017953853 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f7fdc042030 con 0x7f7fe8107de0 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7ff4a7c740 1 Processor -- start 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7ff4a7c740 1 -- start start 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7ff4a7c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de4df60 0x563e0e1f3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7ff4a7c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de250a0 0x563e0e1f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7ff4a7c740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de2d280 0x563e0e1f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7ff4a7c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563e0e169920 con 0x563e0de4df60 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7ff4a7c740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x563e0e158e20 con 0x563e0de2d280 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7ff4a7c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e0e15ce60 con 0x563e0de250a0 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7feca95640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de4df60 0x563e0e1f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7feca95640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de4df60 0x563e0e1f3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47602/0 (socket says 172.21.15.29:47602) 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7feca95640 1 -- 172.21.15.29:0/739431866 learned_addr learned my addr 172.21.15.29:0/739431866 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/739431866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de250a0 0x563e0e1f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7fe77fe640 1 --2- 172.21.15.29:0/739431866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de2d280 0x563e0e1f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7feca95640 1 -- 172.21.15.29:0/739431866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de250a0 msgr2=0x563e0e1f38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7feca95640 1 --2- 172.21.15.29:0/739431866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de250a0 0x563e0e1f38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7feca95640 1 -- 172.21.15.29:0/739431866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de2d280 msgr2=0x563e0e1f3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7feca95640 1 --2- 172.21.15.29:0/739431866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de2d280 0x563e0e1f3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7feca95640 1 -- 172.21.15.29:0/739431866 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e0e0ffa40 con 0x563e0de4df60 2024-07-27T06:45:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7fe77fe640 1 --2- 172.21.15.29:0/739431866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de2d280 0x563e0e1f3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:45:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.027+0000 7f7fe57fa640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7fd00b3c50 0x7f7fd00b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.028+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7fd00b3c50 0x7f7fd00b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.028+0000 7f7feca95640 1 --2- 172.21.15.29:0/739431866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de4df60 0x563e0e1f3380 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009f10 tx=0x7f7fe000e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9959515b08b2c82 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.028+0000 7f7fe57fa640 1 -- 172.21.15.29:0/3017953853 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7fdc104460 con 0x7f7fe8107de0 2024-07-27T06:45:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.028+0000 7f7fce7fc640 1 -- 172.21.15.29:0/739431866 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe0016070 con 0x563e0de4df60 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.028+0000 7f7fce7fc640 1 -- 172.21.15.29:0/739431866 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fe0004d20 con 0x563e0de4df60 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.028+0000 7f7fce7fc640 1 -- 172.21.15.29:0/739431866 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe00117a0 con 0x563e0de4df60 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.029+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/739431866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de4df60 msgr2=0x563e0e1f3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.029+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/739431866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de4df60 0x563e0e1f3380 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009f10 tx=0x7f7fe000e6c0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.029+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7fd00b3c50 0x7f7fd00b6160 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800ad30 tx=0x7f7fd800a2d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.029+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/739431866 shutdown_connections 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.029+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/739431866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de2d280 0x563e0e1f3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.029+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/739431866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de250a0 0x563e0e1f38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.029+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/739431866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de4df60 0x563e0e1f3380 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.029+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/739431866 >> 172.21.15.29:0/739431866 conn(0x563e0de42350 msgr2=0x563e0e1f1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.029+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/739431866 shutdown_connections 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.029+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/739431866 wait complete. 2024-07-27T06:45:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.029+0000 7f7ff4a7c740 1 -- start start 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7ff4a7c740 1 Processor -- start 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7ff4a7c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 0x563e0e201130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7ff4a7c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de2d280 0x563e0e201670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7ff4a7c740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de4df60 0x563e0e20d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7ff4a7c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563e0dfbc5f0 con 0x563e0de250a0 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7ff4a7c740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x563e0e0fb890 con 0x563e0de4df60 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7ff4a7c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e0e203cc0 con 0x563e0de2d280 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7ff4a7c740 10 client.0 _ll_register_callbacks cb 0x563e0e1057d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7fe7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 0x563e0e201130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7fe7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 0x563e0e201130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47610/0 (socket says 172.21.15.29:47610) 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7fe7fff640 1 -- 172.21.15.29:0/1132444251 learned_addr learned my addr 172.21.15.29:0/1132444251 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7feca95640 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de4df60 0x563e0e20d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7fe77fe640 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de2d280 0x563e0e201670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7fe7fff640 1 -- 172.21.15.29:0/1132444251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de2d280 msgr2=0x563e0e201670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de2d280 0x563e0e201670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7fe7fff640 1 -- 172.21.15.29:0/1132444251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de4df60 msgr2=0x563e0e20d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de4df60 0x563e0e20d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.030+0000 7f7fe7fff640 1 -- 172.21.15.29:0/1132444251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e0deb9540 con 0x563e0de250a0 2024-07-27T06:45:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.031+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 0x563e0e201130 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7fd801cc10 tx=0x7f7fd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee8c6acccf7cc9e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.031+0000 7f7fc77fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:45:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.031+0000 7f7fc77fe640 1 -- 172.21.15.29:0/1132444251 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd802c060 con 0x563e0de250a0 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.031+0000 7f7fc77fe640 1 -- 172.21.15.29:0/1132444251 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7fd8016030 con 0x563e0de250a0 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.031+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1132444251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fe0004d20 con 0x563e0de250a0 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.031+0000 7f7fc77fe640 1 -- 172.21.15.29:0/1132444251 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7fd800aea0 con 0x563e0de250a0 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.031+0000 7f7ff4a7c740 10 client.15195 fetch_fsmap learned FSMap version 28 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.031+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1132444251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563e0e20fd50 con 0x563e0de250a0 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.031+0000 7f7fc77fe640 1 -- 172.21.15.29:0/1132444251 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fe0004d20 con 0x563e0de250a0 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.031+0000 7f7fc77fe640 1 -- 172.21.15.29:0/1132444251 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7fd801a7b0 con 0x563e0de250a0 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.031+0000 7f7ff4a7c740 10 client.15195 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.032+0000 7f7ff4a7c740 10 client.15195 resolve_mds: resolved name 'f' to daemon mds.f 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.032+0000 7f7ff4a7c740 20 client.15195 populate_metadata read hostname 'smithi029' 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.032+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x563e0de0c8e0 0x563e0e213810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.032+0000 7f7ff4a7c740 4 client.15195 mds_command: new command op to 24646 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:45:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.032+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1132444251 --> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e0e16b870 con 0x563e0de0c8e0 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.032+0000 7f7fe77fe640 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x563e0de0c8e0 0x563e0e213810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.032+0000 7f7fc77fe640 10 client.15195 ms_handle_connect on v2:172.21.15.77:6834/439792826 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.032+0000 7f7fe77fe640 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x563e0de0c8e0 0x563e0e213810 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=807e101da6b2105f server_cookie=d589cf423ee85c40 in_seq=0 out_seq=0 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.033+0000 7f7fc77fe640 1 -- 172.21.15.29:0/1132444251 <== mds.2 v2:172.21.15.77:6834/439792826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563e0e16b870 con 0x563e0de0c8e0 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.033+0000 7f7fc77fe640 10 client.15195 handle_command_reply: tid=0 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.033+0000 7f7ff4a7c740 1 client.15195 shutdown 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.034+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1132444251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 msgr2=0x563e0e201130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.034+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 0x563e0e201130 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7fd801cc10 tx=0x7f7fd8004730 comp rx=0 tx=0).stop 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.034+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1132444251 shutdown_connections 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.034+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x563e0de0c8e0 0x563e0e213810 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.034+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de4df60 0x563e0e20d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.034+0000 7f7fc77fe640 0 client.15195 ms_handle_reset on v2:172.21.15.77:6834/439792826 2024-07-27T06:45:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.034+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de2d280 0x563e0e201670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.034+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1132444251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 0x563e0e201130 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.034+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1132444251 >> 172.21.15.29:0/1132444251 conn(0x563e0de42350 msgr2=0x563e0e1f4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.034+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1132444251 shutdown_connections 2024-07-27T06:45:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.034+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1132444251 wait complete. 2024-07-27T06:45:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.034+0000 7f7ff4a7c740 20 client.15195 trim_cache size 0 max 16384 2024-07-27T06:45:52.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.052+0000 7f7ff4a7c740 1 Processor -- start 2024-07-27T06:45:52.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.052+0000 7f7ff4a7c740 1 -- start start 2024-07-27T06:45:52.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.052+0000 7f7ff4a7c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 0x563e0e20cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.052+0000 7f7ff4a7c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de2d280 0x563e0e1f8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.052+0000 7f7ff4a7c740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de4df60 0x563e0e1f8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.052+0000 7f7ff4a7c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563e0e211b50 con 0x563e0de250a0 2024-07-27T06:45:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.052+0000 7f7ff4a7c740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x563e0e212220 con 0x563e0de4df60 2024-07-27T06:45:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.052+0000 7f7ff4a7c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e0e2128f0 con 0x563e0de2d280 2024-07-27T06:45:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.052+0000 7f7fe7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 0x563e0e20cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.052+0000 7f7fe7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 0x563e0e20cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47618/0 (socket says 172.21.15.29:47618) 2024-07-27T06:45:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.052+0000 7f7fe7fff640 1 -- 172.21.15.29:0/1885387807 learned_addr learned my addr 172.21.15.29:0/1885387807 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.053+0000 7f7fe77fe640 1 --2- 172.21.15.29:0/1885387807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de2d280 0x563e0e1f8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.053+0000 7f7feca95640 1 --2- 172.21.15.29:0/1885387807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de4df60 0x563e0e1f8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.053+0000 7f7fe7fff640 1 -- 172.21.15.29:0/1885387807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de2d280 msgr2=0x563e0e1f8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.053+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/1885387807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de2d280 0x563e0e1f8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.053+0000 7f7fe7fff640 1 -- 172.21.15.29:0/1885387807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de4df60 msgr2=0x563e0e1f8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.053+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/1885387807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de4df60 0x563e0e1f8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.053+0000 7f7fe7fff640 1 -- 172.21.15.29:0/1885387807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e0e1f61a0 con 0x563e0de250a0 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.053+0000 7f7feca95640 1 --2- 172.21.15.29:0/1885387807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de4df60 0x563e0e1f8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.053+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/1885387807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 0x563e0e20cf90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8010c90 tx=0x7f7fd801afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450ece8509f70ee1 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.053+0000 7f7fcdffb640 1 -- 172.21.15.29:0/1885387807 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd801cdc0 con 0x563e0de250a0 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.053+0000 7f7fcdffb640 1 -- 172.21.15.29:0/1885387807 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fd803d070 con 0x563e0de250a0 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.053+0000 7f7fcdffb640 1 -- 172.21.15.29:0/1885387807 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8027420 con 0x563e0de250a0 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.054+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1885387807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 msgr2=0x563e0e20cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.054+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1885387807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 0x563e0e20cf90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8010c90 tx=0x7f7fd801afd0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.054+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1885387807 shutdown_connections 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.054+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1885387807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de4df60 0x563e0e1f8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.054+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1885387807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de2d280 0x563e0e1f8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.054+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1885387807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de250a0 0x563e0e20cf90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.054+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1885387807 >> 172.21.15.29:0/1885387807 conn(0x563e0de42350 msgr2=0x563e0e1f5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.054+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1885387807 shutdown_connections 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.054+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1885387807 wait complete. 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.054+0000 7f7ff4a7c740 1 -- start start 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7ff4a7c740 1 Processor -- start 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7ff4a7c740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de250a0 0x563e0e2151c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7ff4a7c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de2d280 0x563e0e215760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7ff4a7c740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de4df60 0x563e0e21a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7ff4a7c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563e0e213690 con 0x563e0de2d280 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7ff4a7c740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x563e0e212fc0 con 0x563e0de250a0 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7ff4a7c740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e0e213d60 con 0x563e0de4df60 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7ff4a7c740 10 client.0 _ll_register_callbacks cb 0x7f7fd801c890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:45:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7fe77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de2d280 0x563e0e215760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7fe77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de2d280 0x563e0e215760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47624/0 (socket says 172.21.15.29:47624) 2024-07-27T06:45:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7fe77fe640 1 -- 172.21.15.29:0/1691522133 learned_addr learned my addr 172.21.15.29:0/1691522133 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7feca95640 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de4df60 0x563e0e21a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de250a0 0x563e0e2151c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7fe77fe640 1 -- 172.21.15.29:0/1691522133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de4df60 msgr2=0x563e0e21a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7fe77fe640 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de4df60 0x563e0e21a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7fe77fe640 1 -- 172.21.15.29:0/1691522133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de250a0 msgr2=0x563e0e2151c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7fe77fe640 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de250a0 0x563e0e2151c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7fe77fe640 1 -- 172.21.15.29:0/1691522133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e0e21ad40 con 0x563e0de2d280 2024-07-27T06:45:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.055+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de250a0 0x563e0e2151c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:45:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.056+0000 7f7fe77fe640 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de2d280 0x563e0e215760 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0cafa0 tx=0x7f7fdc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26936c2096f5e34 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.056+0000 7f7fc77fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.056+0000 7f7fc77fe640 1 -- 172.21.15.29:0/1691522133 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc049040 con 0x563e0de2d280 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.056+0000 7f7fc77fe640 1 -- 172.21.15.29:0/1691522133 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7fdc003c50 con 0x563e0de2d280 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.056+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1691522133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fd803d070 con 0x563e0de2d280 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.056+0000 7f7fc77fe640 1 -- 172.21.15.29:0/1691522133 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7fdc03b460 con 0x563e0de2d280 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.056+0000 7f7ff4a7c740 10 client.15201 fetch_fsmap learned FSMap version 28 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.056+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1691522133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563e0e21c090 con 0x563e0de2d280 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.056+0000 7f7fc77fe640 1 -- 172.21.15.29:0/1691522133 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fd803d070 con 0x563e0de2d280 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.056+0000 7f7fc77fe640 1 -- 172.21.15.29:0/1691522133 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7fdc0c7530 con 0x563e0de2d280 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.056+0000 7f7ff4a7c740 10 client.15201 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.057+0000 7f7ff4a7c740 10 client.15201 resolve_mds: resolved name 'f' to daemon mds.f 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.057+0000 7f7ff4a7c740 20 client.15201 populate_metadata read hostname 'smithi029' 2024-07-27T06:45:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.057+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x563e0dde1c90 0x563e0e27faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.057+0000 7f7ff4a7c740 4 client.15201 mds_command: new command op to 24646 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.057+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1691522133 --> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] -- command(tid 0: {"prefix": "status"}) -- 0x563e0e167310 con 0x563e0dde1c90 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.057+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x563e0dde1c90 0x563e0e27faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.057+0000 7f7fc77fe640 10 client.15201 ms_handle_connect on v2:172.21.15.77:6834/439792826 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.057+0000 7f7fe7fff640 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x563e0dde1c90 0x563e0e27faf0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2fc911f1344e36fa server_cookie=768b047067769bdc in_seq=0 out_seq=0 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7fc77fe640 1 -- 172.21.15.29:0/1691522133 <== mds.2 v2:172.21.15.77:6834/439792826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563e0e167310 con 0x563e0dde1c90 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7fc77fe640 10 client.15201 handle_command_reply: tid=0 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7ff4a7c740 1 client.15201 shutdown 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1691522133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de2d280 msgr2=0x563e0e215760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de2d280 0x563e0e215760 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0cafa0 tx=0x7f7fdc000bc0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1691522133 shutdown_connections 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x563e0dde1c90 0x563e0e27faf0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0de4df60 0x563e0e21a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0de2d280 0x563e0e215760 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7fc77fe640 0 client.15201 ms_handle_reset on v2:172.21.15.77:6834/439792826 2024-07-27T06:45:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7ff4a7c740 1 --2- 172.21.15.29:0/1691522133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0de250a0 0x563e0e2151c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1691522133 >> 172.21.15.29:0/1691522133 conn(0x563e0de42350 msgr2=0x563e0e1f5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1691522133 shutdown_connections 2024-07-27T06:45:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7ff4a7c740 1 -- 172.21.15.29:0/1691522133 wait complete. 2024-07-27T06:45:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.058+0000 7f7ff4a7c740 20 client.15201 trim_cache size 0 max 16384 2024-07-27T06:45:52.069 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:45:52.069 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:45:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 2, 2024-07-27T06:45:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24646, 2024-07-27T06:45:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:45:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:45:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:45:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 141.21460985600001, 2024-07-27T06:45:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-07-27T06:45:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:45:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 79, 2024-07-27T06:45:52.070 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 192.987040737 2024-07-27T06:45:52.070 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:45:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.059+0000 7f7fee4f8640 1 -- 172.21.15.29:0/3017953853 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7fd00b3c50 msgr2=0x7f7fd00b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.059+0000 7f7fee4f8640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7fd00b3c50 0x7f7fd00b6160 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800ad30 tx=0x7f7fd800a2d0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.059+0000 7f7fee4f8640 1 -- 172.21.15.29:0/3017953853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe8107de0 msgr2=0x7f7fe8199390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.059+0000 7f7fee4f8640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe8107de0 0x7f7fe8199390 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc02a930 tx=0x7f7fdc004270 comp rx=0 tx=0).stop 2024-07-27T06:45:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.059+0000 7f7fee4f8640 1 -- 172.21.15.29:0/3017953853 shutdown_connections 2024-07-27T06:45:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.059+0000 7f7fee4f8640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7fe8108730 0x7f7fe819e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.059+0000 7f7fee4f8640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fe8107de0 0x7f7fe8199390 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.059+0000 7f7fee4f8640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7fd00b3c50 0x7f7fd00b6160 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.059+0000 7f7fee4f8640 1 --2- 172.21.15.29:0/3017953853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fe81024a0 0x7f7fe8198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.059+0000 7f7fee4f8640 1 -- 172.21.15.29:0/3017953853 >> 172.21.15.29:0/3017953853 conn(0x7f7fe8079140 msgr2=0x7f7fe8105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.059+0000 7f7fee4f8640 1 -- 172.21.15.29:0/3017953853 shutdown_connections 2024-07-27T06:45:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:52.059+0000 7f7fee4f8640 1 -- 172.21.15.29:0/3017953853 wait complete. 2024-07-27T06:45:52.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.c status 2024-07-27T06:45:53.075 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:53.458 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:53 smithi029 ceph-mon[32610]: pgmap v363: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:45:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:53 smithi077 ceph-mon[37153]: pgmap v363: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:45:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:53 smithi047 ceph-mon[36835]: pgmap v363: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:45:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.847+0000 7f6b93173640 1 -- 172.21.15.29:0/186574194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b8c105d10 msgr2=0x7f6b8c106120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:54.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.847+0000 7f6b93173640 1 --2- 172.21.15.29:0/186574194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b8c105d10 0x7f6b8c106120 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6b74009900 tx=0x7f6b7402a410 comp rx=0 tx=0).stop 2024-07-27T06:45:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.847+0000 7f6b93173640 1 -- 172.21.15.29:0/186574194 shutdown_connections 2024-07-27T06:45:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.847+0000 7f6b93173640 1 --2- 172.21.15.29:0/186574194 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b8c106660 0x7f6b8c1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.847+0000 7f6b93173640 1 --2- 172.21.15.29:0/186574194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b8c105d10 0x7f6b8c106120 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.847+0000 7f6b93173640 1 --2- 172.21.15.29:0/186574194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b8c101a30 0x7f6b8c101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.847+0000 7f6b93173640 1 -- 172.21.15.29:0/186574194 >> 172.21.15.29:0/186574194 conn(0x7f6b8c079870 msgr2=0x7f6b8c079c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.847+0000 7f6b93173640 1 -- 172.21.15.29:0/186574194 shutdown_connections 2024-07-27T06:45:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.847+0000 7f6b93173640 1 -- 172.21.15.29:0/186574194 wait complete. 2024-07-27T06:45:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.848+0000 7f6b93173640 1 Processor -- start 2024-07-27T06:45:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.848+0000 7f6b93173640 1 -- start start 2024-07-27T06:45:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.848+0000 7f6b93173640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b8c101a30 0x7f6b8c199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b93173640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b8c105d10 0x7f6b8c19a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b93173640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b8c106660 0x7f6b8c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b93173640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c113540 con 0x7f6b8c105d10 2024-07-27T06:45:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b93173640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c1133c0 con 0x7f6b8c101a30 2024-07-27T06:45:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b93173640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c1136c0 con 0x7f6b8c106660 2024-07-27T06:45:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b8c105d10 0x7f6b8c19a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b8c105d10 0x7f6b8c19a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47638/0 (socket says 172.21.15.29:47638) 2024-07-27T06:45:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b8bfff640 1 -- 172.21.15.29:0/809633293 learned_addr learned my addr 172.21.15.29:0/809633293 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b91710640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b8c106660 0x7f6b8c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b90f0f640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b8c101a30 0x7f6b8c199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b8bfff640 1 -- 172.21.15.29:0/809633293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b8c106660 msgr2=0x7f6b8c19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b8bfff640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b8c106660 0x7f6b8c19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b8bfff640 1 -- 172.21.15.29:0/809633293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b8c101a30 msgr2=0x7f6b8c199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b8bfff640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b8c101a30 0x7f6b8c199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b8bfff640 1 -- 172.21.15.29:0/809633293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c19fea0 con 0x7f6b8c105d10 2024-07-27T06:45:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b8bfff640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b8c105d10 0x7f6b8c19a4d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6b74005ab0 tx=0x7f6b74003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b89ffb640 1 -- 172.21.15.29:0/809633293 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7403f070 con 0x7f6b8c105d10 2024-07-27T06:45:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b93173640 1 -- 172.21.15.29:0/809633293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8c1a0130 con 0x7f6b8c105d10 2024-07-27T06:45:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b89ffb640 1 -- 172.21.15.29:0/809633293 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b74031b50 con 0x7f6b8c105d10 2024-07-27T06:45:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.849+0000 7f6b89ffb640 1 -- 172.21.15.29:0/809633293 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b74035030 con 0x7f6b8c105d10 2024-07-27T06:45:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.850+0000 7f6b93173640 1 -- 172.21.15.29:0/809633293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8c1a68f0 con 0x7f6b8c105d10 2024-07-27T06:45:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.852+0000 7f6b996f7740 1 Processor -- start 2024-07-27T06:45:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.852+0000 7f6b996f7740 1 -- start start 2024-07-27T06:45:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.852+0000 7f6b89ffb640 1 -- 172.21.15.29:0/809633293 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f6b74042070 con 0x7f6b8c105d10 2024-07-27T06:45:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b996f7740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x562370255380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b996f7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x5623702558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b996f7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x562370255e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b996f7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5623701cb920 con 0x56236fe870a0 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b996f7740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5623701bae20 con 0x56236feaff60 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b996f7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5623701bee60 con 0x56236fe8f280 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b90f0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x5623702558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b90f0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x5623702558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47646/0 (socket says 172.21.15.29:47646) 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b90f0f640 1 -- 172.21.15.29:0/2341006846 learned_addr learned my addr 172.21.15.29:0/2341006846 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b8bfff640 1 --2- 172.21.15.29:0/2341006846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x562370255e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b91710640 1 --2- 172.21.15.29:0/2341006846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x562370255380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b90f0f640 1 -- 172.21.15.29:0/2341006846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 msgr2=0x562370255e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b90f0f640 1 --2- 172.21.15.29:0/2341006846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x562370255e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b90f0f640 1 -- 172.21.15.29:0/2341006846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 msgr2=0x562370255380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:54.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b90f0f640 1 --2- 172.21.15.29:0/2341006846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x562370255380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b90f0f640 1 -- 172.21.15.29:0/2341006846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562370161a40 con 0x56236fe870a0 2024-07-27T06:45:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b91710640 1 --2- 172.21.15.29:0/2341006846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x562370255380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:45:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.853+0000 7f6b90f0f640 1 --2- 172.21.15.29:0/2341006846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x5623702558c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009cc0 tx=0x7f6b7c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4dd3903fc224dde server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.854+0000 7f6b89ffb640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6b780b3c20 0x7f6b780b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.854+0000 7f6b82ffd640 1 -- 172.21.15.29:0/2341006846 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c01b470 con 0x56236fe870a0 2024-07-27T06:45:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.854+0000 7f6b82ffd640 1 -- 172.21.15.29:0/2341006846 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b7c016070 con 0x56236fe870a0 2024-07-27T06:45:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.854+0000 7f6b82ffd640 1 -- 172.21.15.29:0/2341006846 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c012a80 con 0x56236fe870a0 2024-07-27T06:45:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.854+0000 7f6b89ffb640 1 -- 172.21.15.29:0/809633293 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f6b74104570 con 0x7f6b8c105d10 2024-07-27T06:45:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.854+0000 7f6b91710640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6b780b3c20 0x7f6b780b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.855+0000 7f6b996f7740 1 -- 172.21.15.29:0/2341006846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 msgr2=0x5623702558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.855+0000 7f6b91710640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6b780b3c20 0x7f6b780b6130 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6b84005fd0 tx=0x7f6b840094a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.855+0000 7f6b996f7740 1 --2- 172.21.15.29:0/2341006846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x5623702558c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009cc0 tx=0x7f6b7c0117b0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.855+0000 7f6b996f7740 1 -- 172.21.15.29:0/2341006846 shutdown_connections 2024-07-27T06:45:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.855+0000 7f6b996f7740 1 --2- 172.21.15.29:0/2341006846 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x562370255e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.855+0000 7f6b996f7740 1 --2- 172.21.15.29:0/2341006846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x5623702558c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.855+0000 7f6b996f7740 1 --2- 172.21.15.29:0/2341006846 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x562370255380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.855+0000 7f6b996f7740 1 -- 172.21.15.29:0/2341006846 >> 172.21.15.29:0/2341006846 conn(0x56236fea4350 msgr2=0x562370253270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.855+0000 7f6b996f7740 1 -- 172.21.15.29:0/2341006846 shutdown_connections 2024-07-27T06:45:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.855+0000 7f6b996f7740 1 -- 172.21.15.29:0/2341006846 wait complete. 2024-07-27T06:45:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.855+0000 7f6b996f7740 1 -- start start 2024-07-27T06:45:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.855+0000 7f6b996f7740 1 Processor -- start 2024-07-27T06:45:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b996f7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x5623702652c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b996f7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x562370265800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b996f7740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x5623702711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b996f7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56237001e5f0 con 0x56236fe870a0 2024-07-27T06:45:54.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b996f7740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x56237015d890 con 0x56236feaff60 2024-07-27T06:45:54.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b996f7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562370267e90 con 0x56236fe8f280 2024-07-27T06:45:54.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b996f7740 10 client.0 _ll_register_callbacks cb 0x5623701677d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:45:54.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b90f0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x5623702652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b90f0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x5623702652c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47648/0 (socket says 172.21.15.29:47648) 2024-07-27T06:45:54.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b90f0f640 1 -- 172.21.15.29:0/3433776110 learned_addr learned my addr 172.21.15.29:0/3433776110 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b8bfff640 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x562370265800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b91710640 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x5623702711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b90f0f640 1 -- 172.21.15.29:0/3433776110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 msgr2=0x562370265800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b90f0f640 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x562370265800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b90f0f640 1 -- 172.21.15.29:0/3433776110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 msgr2=0x5623702711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b90f0f640 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x5623702711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b90f0f640 1 -- 172.21.15.29:0/3433776110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56236ff1b540 con 0x56236fe870a0 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.856+0000 7f6b91710640 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x5623702711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.857+0000 7f6b90f0f640 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x5623702652c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0125c0 tx=0x7f6b7c012f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2f7235972bc6ec server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.857+0000 7f6b6bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.857+0000 7f6b6bfff640 1 -- 172.21.15.29:0/3433776110 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c028020 con 0x56236fe870a0 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.857+0000 7f6b6bfff640 1 -- 172.21.15.29:0/3433776110 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f6b7c004220 con 0x56236fe870a0 2024-07-27T06:45:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.857+0000 7f6b996f7740 1 -- 172.21.15.29:0/3433776110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b7c016070 con 0x56236fe870a0 2024-07-27T06:45:54.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.857+0000 7f6b6bfff640 1 -- 172.21.15.29:0/3433776110 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6b7c004420 con 0x56236fe870a0 2024-07-27T06:45:54.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.857+0000 7f6b996f7740 10 client.15213 fetch_fsmap learned FSMap version 28 2024-07-27T06:45:54.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.857+0000 7f6b996f7740 1 -- 172.21.15.29:0/3433776110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562370273e00 con 0x56236fe870a0 2024-07-27T06:45:54.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.857+0000 7f6b6bfff640 1 -- 172.21.15.29:0/3433776110 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b7c016070 con 0x56236fe870a0 2024-07-27T06:45:54.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.857+0000 7f6b6bfff640 1 -- 172.21.15.29:0/3433776110 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6b7c023070 con 0x56236fe870a0 2024-07-27T06:45:54.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.857+0000 7f6b996f7740 10 client.15213 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:45:54.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.858+0000 7f6b996f7740 10 client.15213 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-27T06:45:54.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.858+0000 7f6b996f7740 20 client.15213 populate_metadata read hostname 'smithi029' 2024-07-27T06:45:54.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.858+0000 7f6b996f7740 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56236fe6e8e0 0x5623702774d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.858+0000 7f6b996f7740 4 client.15213 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:45:54.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.858+0000 7f6b996f7740 1 -- 172.21.15.29:0/3433776110 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623701cd870 con 0x56236fe6e8e0 2024-07-27T06:45:54.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.858+0000 7f6b8bfff640 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56236fe6e8e0 0x5623702774d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.858+0000 7f6b8bfff640 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56236fe6e8e0 0x5623702774d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9fbe399884a126c server_cookie=93aabe1e0ea6b1af in_seq=0 out_seq=0 2024-07-27T06:45:54.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.858+0000 7f6b6bfff640 10 client.15213 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:45:54.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.859+0000 7f6b6bfff640 1 -- 172.21.15.29:0/3433776110 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5623701cd870 con 0x56236fe6e8e0 2024-07-27T06:45:54.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.859+0000 7f6b6bfff640 10 client.15213 handle_command_reply: tid=0 2024-07-27T06:45:54.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.859+0000 7f6b996f7740 1 client.15213 shutdown 2024-07-27T06:45:54.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.859+0000 7f6b996f7740 1 -- 172.21.15.29:0/3433776110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 msgr2=0x5623702652c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:54.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.859+0000 7f6b996f7740 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x5623702652c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0125c0 tx=0x7f6b7c012f60 comp rx=0 tx=0).stop 2024-07-27T06:45:54.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.859+0000 7f6b996f7740 1 -- 172.21.15.29:0/3433776110 shutdown_connections 2024-07-27T06:45:54.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.859+0000 7f6b996f7740 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56236fe6e8e0 0x5623702774d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.859+0000 7f6b996f7740 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x5623702711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.859+0000 7f6b996f7740 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x562370265800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.859+0000 7f6b6bfff640 0 client.15213 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.859+0000 7f6b996f7740 1 --2- 172.21.15.29:0/3433776110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x5623702652c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.859+0000 7f6b996f7740 1 -- 172.21.15.29:0/3433776110 >> 172.21.15.29:0/3433776110 conn(0x56236fea4350 msgr2=0x5623702584d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.860+0000 7f6b996f7740 1 -- 172.21.15.29:0/3433776110 shutdown_connections 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.860+0000 7f6b996f7740 1 -- 172.21.15.29:0/3433776110 wait complete. 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.860+0000 7f6b996f7740 20 client.15213 trim_cache size 0 max 16384 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b996f7740 1 Processor -- start 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b996f7740 1 -- start start 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b996f7740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236fe870a0 0x562370276730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b996f7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x5623702751b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b996f7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236feaff60 0x562370271160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b996f7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562370275810 con 0x56236feaff60 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b996f7740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x562370275ee0 con 0x56236fe870a0 2024-07-27T06:45:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b996f7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5623702765b0 con 0x56236fe8f280 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b91710640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236feaff60 0x562370271160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b91710640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236feaff60 0x562370271160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47664/0 (socket says 172.21.15.29:47664) 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b91710640 1 -- 172.21.15.29:0/4026304187 learned_addr learned my addr 172.21.15.29:0/4026304187 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b8bfff640 1 --2- 172.21.15.29:0/4026304187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x5623702751b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b91710640 1 -- 172.21.15.29:0/4026304187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 msgr2=0x5623702751b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b90f0f640 1 --2- 172.21.15.29:0/4026304187 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236fe870a0 0x562370276730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b91710640 1 --2- 172.21.15.29:0/4026304187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x5623702751b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b91710640 1 -- 172.21.15.29:0/4026304187 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236fe870a0 msgr2=0x562370276730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b91710640 1 --2- 172.21.15.29:0/4026304187 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236fe870a0 0x562370276730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b91710640 1 -- 172.21.15.29:0/4026304187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562370259430 con 0x56236feaff60 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.878+0000 7f6b8bfff640 1 --2- 172.21.15.29:0/4026304187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x5623702751b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.879+0000 7f6b91710640 1 --2- 172.21.15.29:0/4026304187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236feaff60 0x562370271160 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6b84012680 tx=0x7f6b84014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dce9f3979362c00 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:54.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.879+0000 7f6b827fc640 1 -- 172.21.15.29:0/4026304187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8401e770 con 0x56236feaff60 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.879+0000 7f6b827fc640 1 -- 172.21.15.29:0/4026304187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b8401edc0 con 0x56236feaff60 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.879+0000 7f6b827fc640 1 -- 172.21.15.29:0/4026304187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8401c410 con 0x56236feaff60 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.880+0000 7f6b996f7740 1 -- 172.21.15.29:0/4026304187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236feaff60 msgr2=0x562370271160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.880+0000 7f6b996f7740 1 --2- 172.21.15.29:0/4026304187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236feaff60 0x562370271160 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6b84012680 tx=0x7f6b84014000 comp rx=0 tx=0).stop 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.880+0000 7f6b996f7740 1 -- 172.21.15.29:0/4026304187 shutdown_connections 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.880+0000 7f6b996f7740 1 --2- 172.21.15.29:0/4026304187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236feaff60 0x562370271160 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.880+0000 7f6b996f7740 1 --2- 172.21.15.29:0/4026304187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x5623702751b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.880+0000 7f6b996f7740 1 --2- 172.21.15.29:0/4026304187 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236fe870a0 0x562370276730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.880+0000 7f6b996f7740 1 -- 172.21.15.29:0/4026304187 >> 172.21.15.29:0/4026304187 conn(0x56236fea4350 msgr2=0x5623702602a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.880+0000 7f6b996f7740 1 -- 172.21.15.29:0/4026304187 shutdown_connections 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.880+0000 7f6b996f7740 1 -- 172.21.15.29:0/4026304187 wait complete. 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.880+0000 7f6b996f7740 1 -- start start 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.880+0000 7f6b996f7740 1 Processor -- start 2024-07-27T06:45:54.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b996f7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x562370278930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b996f7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x562370278ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b996f7740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x56237027df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b996f7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x562370277350 con 0x56236fe870a0 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b996f7740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x562370276c80 con 0x56236feaff60 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b996f7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562370277a20 con 0x56236fe8f280 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b996f7740 10 client.0 _ll_register_callbacks cb 0x7f6b7c012bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b90f0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x562370278930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b90f0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x562370278930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47680/0 (socket says 172.21.15.29:47680) 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b90f0f640 1 -- 172.21.15.29:0/968354129 learned_addr learned my addr 172.21.15.29:0/968354129 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b91710640 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x56237027df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b8bfff640 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x562370278ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b90f0f640 1 -- 172.21.15.29:0/968354129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 msgr2=0x562370278ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b90f0f640 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x562370278ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b90f0f640 1 -- 172.21.15.29:0/968354129 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 msgr2=0x56237027df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:54.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b90f0f640 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x56237027df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b90f0f640 1 -- 172.21.15.29:0/968354129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56237027e5a0 con 0x56236fe870a0 2024-07-27T06:45:54.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b90f0f640 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x562370278930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c004960 tx=0x7f6b7c01b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a36c9857d33aa8 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:54.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b6bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:45:54.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b6bfff640 1 -- 172.21.15.29:0/968354129 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c03a060 con 0x56236fe870a0 2024-07-27T06:45:54.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b6bfff640 1 -- 172.21.15.29:0/968354129 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f6b7c004c50 con 0x56236fe870a0 2024-07-27T06:45:54.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.881+0000 7f6b996f7740 1 -- 172.21.15.29:0/968354129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b8401edc0 con 0x56236fe870a0 2024-07-27T06:45:54.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.882+0000 7f6b6bfff640 1 -- 172.21.15.29:0/968354129 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6b7c028040 con 0x56236fe870a0 2024-07-27T06:45:54.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.882+0000 7f6b996f7740 10 client.15219 fetch_fsmap learned FSMap version 28 2024-07-27T06:45:54.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.882+0000 7f6b996f7740 1 -- 172.21.15.29:0/968354129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56237027f9f0 con 0x56236fe870a0 2024-07-27T06:45:54.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.882+0000 7f6b6bfff640 1 -- 172.21.15.29:0/968354129 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b8401edc0 con 0x56236fe870a0 2024-07-27T06:45:54.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.882+0000 7f6b6bfff640 1 -- 172.21.15.29:0/968354129 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6b7c023040 con 0x56236fe870a0 2024-07-27T06:45:54.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.882+0000 7f6b996f7740 10 client.15219 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:45:54.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.882+0000 7f6b996f7740 10 client.15219 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-27T06:45:54.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.882+0000 7f6b996f7740 20 client.15219 populate_metadata read hostname 'smithi029' 2024-07-27T06:45:54.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.882+0000 7f6b996f7740 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56236fe43c90 0x5623702e3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:54.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.882+0000 7f6b996f7740 4 client.15219 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:45:54.931 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:45:54.931 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:45:54.931 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 1, 2024-07-27T06:45:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24658, 2024-07-27T06:45:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:45:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:45:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:45:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 145.04678251199999, 2024-07-27T06:45:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-07-27T06:45:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:45:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 79, 2024-07-27T06:45:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 193.25753332400001 2024-07-27T06:45:54.932 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:45:54.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.882+0000 7f6b996f7740 1 -- 172.21.15.29:0/968354129 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "status"}) -- 0x5623701c9310 con 0x56236fe43c90 2024-07-27T06:45:54.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.882+0000 7f6b8bfff640 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56236fe43c90 0x5623702e3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:54.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b8bfff640 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56236fe43c90 0x5623702e3450 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=197918e1d4bd331c server_cookie=4d00208052ac7c3a in_seq=0 out_seq=0 2024-07-27T06:45:54.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b6bfff640 10 client.15219 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:45:54.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b6bfff640 1 -- 172.21.15.29:0/968354129 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5623701c9310 con 0x56236fe43c90 2024-07-27T06:45:54.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b6bfff640 10 client.15219 handle_command_reply: tid=0 2024-07-27T06:45:54.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b996f7740 1 client.15219 shutdown 2024-07-27T06:45:54.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b996f7740 1 -- 172.21.15.29:0/968354129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 msgr2=0x562370278930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:54.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b996f7740 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x562370278930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c004960 tx=0x7f6b7c01b7a0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b996f7740 1 -- 172.21.15.29:0/968354129 shutdown_connections 2024-07-27T06:45:54.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b996f7740 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56236fe43c90 0x5623702e3450 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b6bfff640 0 client.15219 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:45:54.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b996f7740 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56236feaff60 0x56237027df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b996f7740 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56236fe8f280 0x562370278ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b996f7740 1 --2- 172.21.15.29:0/968354129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56236fe870a0 0x562370278930 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.883+0000 7f6b996f7740 1 -- 172.21.15.29:0/968354129 >> 172.21.15.29:0/968354129 conn(0x56236fea4350 msgr2=0x562370263880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:54.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b996f7740 1 -- 172.21.15.29:0/968354129 shutdown_connections 2024-07-27T06:45:54.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b996f7740 1 -- 172.21.15.29:0/968354129 wait complete. 2024-07-27T06:45:54.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b996f7740 20 client.15219 trim_cache size 0 max 16384 2024-07-27T06:45:54.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b93173640 1 -- 172.21.15.29:0/809633293 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6b780b3c20 msgr2=0x7f6b780b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:54.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b93173640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6b780b3c20 0x7f6b780b6130 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6b84005fd0 tx=0x7f6b840094a0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b93173640 1 -- 172.21.15.29:0/809633293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b8c105d10 msgr2=0x7f6b8c19a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:54.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b93173640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b8c105d10 0x7f6b8c19a4d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6b74005ab0 tx=0x7f6b74003b90 comp rx=0 tx=0).stop 2024-07-27T06:45:54.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b93173640 1 -- 172.21.15.29:0/809633293 shutdown_connections 2024-07-27T06:45:54.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b93173640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f6b780b3c20 0x7f6b780b6130 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b93173640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b8c106660 0x7f6b8c19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b93173640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b8c105d10 0x7f6b8c19a4d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b93173640 1 --2- 172.21.15.29:0/809633293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b8c101a30 0x7f6b8c199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:54.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.884+0000 7f6b93173640 1 -- 172.21.15.29:0/809633293 >> 172.21.15.29:0/809633293 conn(0x7f6b8c079870 msgr2=0x7f6b8c106f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:54.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.885+0000 7f6b93173640 1 -- 172.21.15.29:0/809633293 shutdown_connections 2024-07-27T06:45:54.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:54.885+0000 7f6b93173640 1 -- 172.21.15.29:0/809633293 wait complete. 2024-07-27T06:45:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:55 smithi077 ceph-mon[37153]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:45:55.659 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:55 smithi029 ceph-mon[32610]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:45:55.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.i status 2024-07-27T06:45:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:55 smithi047 ceph-mon[36835]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:45:55.958 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:57 smithi029 ceph-mon[32610]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:45:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:57 smithi077 ceph-mon[37153]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:45:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:57 smithi047 ceph-mon[36835]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:45:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.739+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/3939803709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb8108720 msgr2=0x7f5fb810ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.739+0000 7f5fbfb5e640 1 --2- 172.21.15.29:0/3939803709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb8108720 0x7f5fb810ef50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0009ca0 tx=0x7f5fb002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.740+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/3939803709 shutdown_connections 2024-07-27T06:45:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.740+0000 7f5fbfb5e640 1 --2- 172.21.15.29:0/3939803709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb8108720 0x7f5fb810ef50 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.740+0000 7f5fbfb5e640 1 --2- 172.21.15.29:0/3939803709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fb8107dd0 0x7f5fb81081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.740+0000 7f5fbfb5e640 1 --2- 172.21.15.29:0/3939803709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8102490 0x7f5fb81028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.740+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/3939803709 >> 172.21.15.29:0/3939803709 conn(0x7f5fb8079160 msgr2=0x7f5fb8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.740+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/3939803709 shutdown_connections 2024-07-27T06:45:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.740+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/3939803709 wait complete. 2024-07-27T06:45:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.740+0000 7f5fbfb5e640 1 Processor -- start 2024-07-27T06:45:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.740+0000 7f5fbfb5e640 1 -- start start 2024-07-27T06:45:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbfb5e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fb8102490 0x7f5fb8199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbfb5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb8107dd0 0x7f5fb819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbfb5e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8108720 0x7f5fb819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbfb5e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb81134b0 con 0x7f5fb8107dd0 2024-07-27T06:45:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbfb5e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb8113330 con 0x7f5fb8102490 2024-07-27T06:45:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbfb5e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb8113630 con 0x7f5fb8108720 2024-07-27T06:45:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbd0f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb8107dd0 0x7f5fb819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbd0f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb8107dd0 0x7f5fb819a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47698/0 (socket says 172.21.15.29:47698) 2024-07-27T06:45:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbd0f9640 1 -- 172.21.15.29:0/2988966841 learned_addr learned my addr 172.21.15.29:0/2988966841 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbd8fa640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fb8102490 0x7f5fb8199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbe0fb640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8108720 0x7f5fb819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbd0f9640 1 -- 172.21.15.29:0/2988966841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8108720 msgr2=0x7f5fb819f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbd0f9640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8108720 0x7f5fb819f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbd0f9640 1 -- 172.21.15.29:0/2988966841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fb8102490 msgr2=0x7f5fb8199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbd0f9640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fb8102490 0x7f5fb8199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbd0f9640 1 -- 172.21.15.29:0/2988966841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb819fe10 con 0x7f5fb8107dd0 2024-07-27T06:45:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbe0fb640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8108720 0x7f5fb819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:45:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbd8fa640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fb8102490 0x7f5fb8199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:45:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbd0f9640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb8107dd0 0x7f5fb819a440 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800d770 tx=0x7f5fa800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5faeffd640 1 -- 172.21.15.29:0/2988966841 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8017070 con 0x7f5fb8107dd0 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5faeffd640 1 -- 172.21.15.29:0/2988966841 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fa8009c30 con 0x7f5fb8107dd0 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.741+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/2988966841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb81a0100 con 0x7f5fb8107dd0 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.742+0000 7f5faeffd640 1 -- 172.21.15.29:0/2988966841 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa800f620 con 0x7f5fb8107dd0 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.742+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/2988966841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb81a6860 con 0x7f5fb8107dd0 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.745+0000 7f5fc60e2740 1 Processor -- start 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.745+0000 7f5fc60e2740 1 -- start start 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.745+0000 7f5fc60e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab0224f60 0x55aab05ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.745+0000 7f5faeffd640 1 -- 172.21.15.29:0/2988966841 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f5fa801a070 con 0x7f5fb8107dd0 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.745+0000 7f5fc60e2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 0x55aab05ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.745+0000 7f5fc60e2740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab0204280 0x55aab05cae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.745+0000 7f5fc60e2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55aab0540940 con 0x55aab01fc0a0 2024-07-27T06:45:57.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.745+0000 7f5fc60e2740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55aab052fe40 con 0x55aab0204280 2024-07-27T06:45:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.745+0000 7f5fc60e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55aab0533e80 con 0x55aab0224f60 2024-07-27T06:45:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.745+0000 7f5fbd8fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 0x55aab05ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.745+0000 7f5fbd8fa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 0x55aab05ca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47708/0 (socket says 172.21.15.29:47708) 2024-07-27T06:45:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.745+0000 7f5fbd8fa640 1 -- 172.21.15.29:0/3600143399 learned_addr learned my addr 172.21.15.29:0/3600143399 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5fbd0f9640 1 --2- 172.21.15.29:0/3600143399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab0204280 0x55aab05cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5fbe0fb640 1 --2- 172.21.15.29:0/3600143399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab0224f60 0x55aab05ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5fbd8fa640 1 -- 172.21.15.29:0/3600143399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab0224f60 msgr2=0x55aab05ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5fbd8fa640 1 --2- 172.21.15.29:0/3600143399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab0224f60 0x55aab05ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5fbd8fa640 1 -- 172.21.15.29:0/3600143399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab0204280 msgr2=0x55aab05cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5fbd8fa640 1 --2- 172.21.15.29:0/3600143399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab0204280 0x55aab05cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5fbd8fa640 1 -- 172.21.15.29:0/3600143399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aab04d6a60 con 0x55aab01fc0a0 2024-07-27T06:45:57.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5fbd8fa640 1 --2- 172.21.15.29:0/3600143399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 0x55aab05ca8e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4009d40 tx=0x7f5fb40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250b5e9f86b97ab server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5fa3fff640 1 -- 172.21.15.29:0/3600143399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4012650 con 0x55aab01fc0a0 2024-07-27T06:45:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5faeffd640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5fa40b3c20 0x7f5fa40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5fa3fff640 1 -- 172.21.15.29:0/3600143399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fb4009e30 con 0x55aab01fc0a0 2024-07-27T06:45:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5fa3fff640 1 -- 172.21.15.29:0/3600143399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb400fc20 con 0x55aab01fc0a0 2024-07-27T06:45:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.746+0000 7f5fbe0fb640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5fa40b3c20 0x7f5fa40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.747+0000 7f5faeffd640 1 -- 172.21.15.29:0/2988966841 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f5fa80e38d0 con 0x7f5fb8107dd0 2024-07-27T06:45:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.747+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3600143399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 msgr2=0x55aab05ca8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.747+0000 7f5fbe0fb640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5fa40b3c20 0x7f5fa40b6130 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5fb002f500 tx=0x7f5fb002f310 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.747+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/3600143399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 0x55aab05ca8e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4009d40 tx=0x7f5fb40117b0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.747+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3600143399 shutdown_connections 2024-07-27T06:45:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.747+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/3600143399 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab0204280 0x55aab05cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.747+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/3600143399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 0x55aab05ca8e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.747+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/3600143399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab0224f60 0x55aab05ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.747+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3600143399 >> 172.21.15.29:0/3600143399 conn(0x55aab0219350 msgr2=0x55aab05c8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.747+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3600143399 shutdown_connections 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.747+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3600143399 wait complete. 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.748+0000 7f5fc60e2740 1 -- start start 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.748+0000 7f5fc60e2740 1 Processor -- start 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.748+0000 7f5fc60e2740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab01fc0a0 0x55aab05d5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.748+0000 7f5fc60e2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 0x55aab05d64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.748+0000 7f5fc60e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab0224f60 0x55aab05e1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.748+0000 7f5fc60e2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55aab03935f0 con 0x55aab0204280 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.748+0000 7f5fc60e2740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55aab04d28b0 con 0x55aab01fc0a0 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.748+0000 7f5fc60e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55aab05d8b80 con 0x55aab0224f60 2024-07-27T06:45:57.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.748+0000 7f5fc60e2740 10 client.0 _ll_register_callbacks cb 0x55aab04dc7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fbd0f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 0x55aab05d64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fbd0f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 0x55aab05d64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47722/0 (socket says 172.21.15.29:47722) 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fbd0f9640 1 -- 172.21.15.29:0/3517703891 learned_addr learned my addr 172.21.15.29:0/3517703891 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fbd0f9640 1 -- 172.21.15.29:0/3517703891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab0224f60 msgr2=0x55aab05e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fbd0f9640 1 --2- 172.21.15.29:0/3517703891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab0224f60 0x55aab05e1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fbd0f9640 1 -- 172.21.15.29:0/3517703891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab01fc0a0 msgr2=0x55aab05d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fbd8fa640 1 --2- 172.21.15.29:0/3517703891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab01fc0a0 0x55aab05d5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fbd0f9640 1 --2- 172.21.15.29:0/3517703891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab01fc0a0 0x55aab05d5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fbd0f9640 1 -- 172.21.15.29:0/3517703891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aab0290540 con 0x55aab0204280 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fbd0f9640 1 --2- 172.21.15.29:0/3517703891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 0x55aab05d64c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80aa6a0 tx=0x7f5fa80a6c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4a4dbf1f46e842 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fa17fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fa17fa640 1 -- 172.21.15.29:0/3517703891 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8021030 con 0x55aab0204280 2024-07-27T06:45:57.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fa17fa640 1 -- 172.21.15.29:0/3517703891 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f5fa80a9970 con 0x55aab0204280 2024-07-27T06:45:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3517703891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fb4009e30 con 0x55aab0204280 2024-07-27T06:45:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fa17fa640 1 -- 172.21.15.29:0/3517703891 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5fa80aa060 con 0x55aab0204280 2024-07-27T06:45:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fc60e2740 10 client.15231 fetch_fsmap learned FSMap version 28 2024-07-27T06:45:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.749+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3517703891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55aab05e4b90 con 0x55aab0204280 2024-07-27T06:45:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.750+0000 7f5fa17fa640 1 -- 172.21.15.29:0/3517703891 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fb4009e30 con 0x55aab0204280 2024-07-27T06:45:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.750+0000 7f5fa17fa640 1 -- 172.21.15.29:0/3517703891 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5fa80a93e0 con 0x55aab0204280 2024-07-27T06:45:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.750+0000 7f5fc60e2740 10 client.15231 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:45:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.750+0000 7f5fc60e2740 10 client.15231 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-27T06:45:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.750+0000 7f5fc60e2740 20 client.15231 populate_metadata read hostname 'smithi029' 2024-07-27T06:45:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.750+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/3517703891 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55aab01e38e0 0x55aab05e86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.750+0000 7f5fc60e2740 4 client.15231 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:45:57.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.750+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3517703891 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aab0542890 con 0x55aab01e38e0 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.750+0000 7f5fbd8fa640 1 --2- 172.21.15.29:0/3517703891 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55aab01e38e0 0x55aab05e86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.751+0000 7f5fa17fa640 10 client.15231 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.751+0000 7f5fbd8fa640 1 --2- 172.21.15.29:0/3517703891 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55aab01e38e0 0x55aab05e86d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed051cb3a12deba5 server_cookie=b64c6c64cd91cc61 in_seq=0 out_seq=0 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.751+0000 7f5fa17fa640 1 -- 172.21.15.29:0/3517703891 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55aab0542890 con 0x55aab01e38e0 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.751+0000 7f5fa17fa640 10 client.15231 handle_command_reply: tid=0 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fc60e2740 1 client.15231 shutdown 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3517703891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 msgr2=0x55aab05d64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/3517703891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 0x55aab05d64c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80aa6a0 tx=0x7f5fa80a6c20 comp rx=0 tx=0).stop 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3517703891 shutdown_connections 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/3517703891 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55aab01e38e0 0x55aab05e86d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/3517703891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab0224f60 0x55aab05e1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/3517703891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 0x55aab05d64c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fa17fa640 0 client.15231 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:45:57.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/3517703891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab01fc0a0 0x55aab05d5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3517703891 >> 172.21.15.29:0/3517703891 conn(0x55aab0219350 msgr2=0x55aab05cd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:57.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3517703891 shutdown_connections 2024-07-27T06:45:57.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fc60e2740 1 -- 172.21.15.29:0/3517703891 wait complete. 2024-07-27T06:45:57.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.752+0000 7f5fc60e2740 20 client.15231 trim_cache size 0 max 16384 2024-07-27T06:45:57.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.770+0000 7f5fc60e2740 1 Processor -- start 2024-07-27T06:45:57.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.770+0000 7f5fc60e2740 1 -- start start 2024-07-27T06:45:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fc60e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab01e38e0 0x55aab05d4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fc60e2740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab01fc0a0 0x55aab05d4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fc60e2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 0x55aab05cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fc60e2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55aab05e6a10 con 0x55aab0204280 2024-07-27T06:45:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fc60e2740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55aab05e70e0 con 0x55aab01fc0a0 2024-07-27T06:45:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fc60e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55aab05e77b0 con 0x55aab01e38e0 2024-07-27T06:45:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fbe0fb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 0x55aab05cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fbe0fb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 0x55aab05cf4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47726/0 (socket says 172.21.15.29:47726) 2024-07-27T06:45:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fbe0fb640 1 -- 172.21.15.29:0/1934742044 learned_addr learned my addr 172.21.15.29:0/1934742044 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fbd8fa640 1 --2- 172.21.15.29:0/1934742044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab01e38e0 0x55aab05d4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fbd0f9640 1 --2- 172.21.15.29:0/1934742044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab01fc0a0 0x55aab05d4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fbe0fb640 1 -- 172.21.15.29:0/1934742044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab01e38e0 msgr2=0x55aab05d4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fbe0fb640 1 --2- 172.21.15.29:0/1934742044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab01e38e0 0x55aab05d4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fbe0fb640 1 -- 172.21.15.29:0/1934742044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab01fc0a0 msgr2=0x55aab05d4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fbe0fb640 1 --2- 172.21.15.29:0/1934742044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab01fc0a0 0x55aab05d4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fbe0fb640 1 -- 172.21.15.29:0/1934742044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aab05d6920 con 0x55aab0204280 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.771+0000 7f5fbe0fb640 1 --2- 172.21.15.29:0/1934742044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 0x55aab05cf4b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0032d70 tx=0x7f5fb003e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9eeb21ce42825e3 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.772+0000 7f5fa37fe640 1 -- 172.21.15.29:0/1934742044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb004e070 con 0x55aab0204280 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.772+0000 7f5fa37fe640 1 -- 172.21.15.29:0/1934742044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fb0036040 con 0x55aab0204280 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.772+0000 7f5fa37fe640 1 -- 172.21.15.29:0/1934742044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb00309b0 con 0x55aab0204280 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.773+0000 7f5fc60e2740 1 -- 172.21.15.29:0/1934742044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 msgr2=0x55aab05cf4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.773+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/1934742044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 0x55aab05cf4b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0032d70 tx=0x7f5fb003e040 comp rx=0 tx=0).stop 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.773+0000 7f5fc60e2740 1 -- 172.21.15.29:0/1934742044 shutdown_connections 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.773+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/1934742044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab0204280 0x55aab05cf4b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.773+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/1934742044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab01fc0a0 0x55aab05d4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.773+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/1934742044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab01e38e0 0x55aab05d4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.773+0000 7f5fc60e2740 1 -- 172.21.15.29:0/1934742044 >> 172.21.15.29:0/1934742044 conn(0x55aab0219350 msgr2=0x55aab05d0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:57.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.773+0000 7f5fc60e2740 1 -- 172.21.15.29:0/1934742044 shutdown_connections 2024-07-27T06:45:57.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.773+0000 7f5fc60e2740 1 -- 172.21.15.29:0/1934742044 wait complete. 2024-07-27T06:45:57.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.773+0000 7f5fc60e2740 1 -- start start 2024-07-27T06:45:57.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.773+0000 7f5fc60e2740 1 Processor -- start 2024-07-27T06:45:57.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fc60e2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab01e38e0 0x55aab05f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fc60e2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 0x55aab05f1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fc60e2740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab0204280 0x55aab05d2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fc60e2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55aab05e8550 con 0x55aab01fc0a0 2024-07-27T06:45:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fc60e2740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55aab05e7e80 con 0x55aab0204280 2024-07-27T06:45:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fc60e2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55aab05e8c20 con 0x55aab01e38e0 2024-07-27T06:45:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fc60e2740 10 client.0 _ll_register_callbacks cb 0x7f5fa8005b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:45:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fbd0f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 0x55aab05f1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fbe0fb640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab0204280 0x55aab05d2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fbd0f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 0x55aab05f1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47738/0 (socket says 172.21.15.29:47738) 2024-07-27T06:45:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fbe0fb640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab0204280 0x55aab05d2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.29:49754/0 (socket says 172.21.15.29:49754) 2024-07-27T06:45:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fbd0f9640 1 -- 172.21.15.29:0/2071986566 learned_addr learned my addr 172.21.15.29:0/2071986566 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:45:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.774+0000 7f5fbd8fa640 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab01e38e0 0x55aab05f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.775+0000 7f5fbd0f9640 1 -- 172.21.15.29:0/2071986566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab01e38e0 msgr2=0x55aab05f0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.775+0000 7f5fbd0f9640 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab01e38e0 0x55aab05f0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.775+0000 7f5fbd0f9640 1 -- 172.21.15.29:0/2071986566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab0204280 msgr2=0x55aab05d2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.775+0000 7f5fbd0f9640 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab0204280 0x55aab05d2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.775+0000 7f5fbd0f9640 1 -- 172.21.15.29:0/2071986566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aab05d30b0 con 0x55aab01fc0a0 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.775+0000 7f5fbe0fb640 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab0204280 0x55aab05d2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.775+0000 7f5fbd0f9640 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 0x55aab05f1240 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80a9820 tx=0x7f5fa80a9b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6c17aef35ec8ef server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.775+0000 7f5fa17fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fa17fa640 1 -- 172.21.15.29:0/2071986566 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8014030 con 0x55aab01fc0a0 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fa17fa640 1 -- 172.21.15.29:0/2071986566 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f5fa80a6290 con 0x55aab01fc0a0 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fc60e2740 1 -- 172.21.15.29:0/2071986566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fb0036040 con 0x55aab01fc0a0 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fa17fa640 1 -- 172.21.15.29:0/2071986566 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5fa806e050 con 0x55aab01fc0a0 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fc60e2740 10 client.15237 fetch_fsmap learned FSMap version 28 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fc60e2740 1 -- 172.21.15.29:0/2071986566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55aab05ee460 con 0x55aab01fc0a0 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fa17fa640 1 -- 172.21.15.29:0/2071986566 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fb0036040 con 0x55aab01fc0a0 2024-07-27T06:45:57.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fa17fa640 1 -- 172.21.15.29:0/2071986566 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5fa80aba40 con 0x55aab01fc0a0 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fc60e2740 10 client.15237 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fc60e2740 10 client.15237 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fc60e2740 20 client.15237 populate_metadata read hostname 'smithi029' 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55aab0224f60 0x55aab0658fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fc60e2740 4 client.15237 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.776+0000 7f5fc60e2740 1 -- 172.21.15.29:0/2071986566 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "status"}) -- 0x55aab053e330 con 0x55aab0224f60 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.777+0000 7f5fbd8fa640 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55aab0224f60 0x55aab0658fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.777+0000 7f5fa17fa640 10 client.15237 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.777+0000 7f5fbd8fa640 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55aab0224f60 0x55aab0658fd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfe2b24278c260bd server_cookie=39868484f6b31bc0 in_seq=0 out_seq=0 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fa17fa640 1 -- 172.21.15.29:0/2071986566 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55aab053e330 con 0x55aab0224f60 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fa17fa640 10 client.15237 handle_command_reply: tid=0 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fc60e2740 1 client.15237 shutdown 2024-07-27T06:45:57.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fc60e2740 1 -- 172.21.15.29:0/2071986566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 msgr2=0x55aab05f1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:57.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 0x55aab05f1240 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80a9820 tx=0x7f5fa80a9b80 comp rx=0 tx=0).stop 2024-07-27T06:45:57.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fc60e2740 1 -- 172.21.15.29:0/2071986566 shutdown_connections 2024-07-27T06:45:57.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55aab0204280 0x55aab05d2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55aab01fc0a0 0x55aab05f1240 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55aab0224f60 0x55aab0658fd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fc60e2740 1 --2- 172.21.15.29:0/2071986566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aab01e38e0 0x55aab05f0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fa17fa640 0 client.15237 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:45:57.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fc60e2740 1 -- 172.21.15.29:0/2071986566 >> 172.21.15.29:0/2071986566 conn(0x55aab0219350 msgr2=0x55aab05d0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:57.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fc60e2740 1 -- 172.21.15.29:0/2071986566 shutdown_connections 2024-07-27T06:45:57.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fc60e2740 1 -- 172.21.15.29:0/2071986566 wait complete. 2024-07-27T06:45:57.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.778+0000 7f5fc60e2740 20 client.15237 trim_cache size 0 max 16384 2024-07-27T06:45:57.787 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:45:57.787 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:45:57.787 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-07-27T06:45:57.787 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24670, 2024-07-27T06:45:57.787 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:45:57.787 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:45:57.787 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:45:57.787 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 161.97860910700001, 2024-07-27T06:45:57.787 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-07-27T06:45:57.788 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:45:57.788 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 79, 2024-07-27T06:45:57.788 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 191.03891101100001 2024-07-27T06:45:57.788 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:45:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.779+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/2988966841 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5fa40b3c20 msgr2=0x7f5fa40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.779+0000 7f5fbfb5e640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5fa40b3c20 0x7f5fa40b6130 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5fb002f500 tx=0x7f5fb002f310 comp rx=0 tx=0).stop 2024-07-27T06:45:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.779+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/2988966841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb8107dd0 msgr2=0x7f5fb819a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:45:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.779+0000 7f5fbfb5e640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb8107dd0 0x7f5fb819a440 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800d770 tx=0x7f5fa800dc30 comp rx=0 tx=0).stop 2024-07-27T06:45:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.779+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/2988966841 shutdown_connections 2024-07-27T06:45:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.779+0000 7f5fbfb5e640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5fa40b3c20 0x7f5fa40b6130 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.779+0000 7f5fbfb5e640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fb8108720 0x7f5fb819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.779+0000 7f5fbfb5e640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb8107dd0 0x7f5fb819a440 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.779+0000 7f5fbfb5e640 1 --2- 172.21.15.29:0/2988966841 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fb8102490 0x7f5fb8199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:45:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.779+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/2988966841 >> 172.21.15.29:0/2988966841 conn(0x7f5fb8079160 msgr2=0x7f5fb8103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:45:57.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.779+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/2988966841 shutdown_connections 2024-07-27T06:45:57.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:45:57.779+0000 7f5fbfb5e640 1 -- 172.21.15.29:0/2988966841 wait complete. 2024-07-27T06:45:58.514 INFO:tasks.cephadm:Setup complete, yielding 2024-07-27T06:45:58.515 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-27T06:45:58.524 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-07-27T06:45:58.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- bash -c 'ceph orch status' 2024-07-27T06:45:58.791 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:45:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:45:59 smithi029 ceph-mon[32610]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:45:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:45:59 smithi077 ceph-mon[37153]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:45:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:45:59 smithi047 ceph-mon[36835]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-07-27T06:46:00.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.587+0000 7f021e29a640 1 -- 172.21.15.29:0/1661683305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02181029d0 msgr2=0x7f021810f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:00.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.587+0000 7f021e29a640 1 --2- 172.21.15.29:0/1661683305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02181029d0 0x7f021810f6f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0200009900 tx=0x7f020002a480 comp rx=0 tx=0).stop 2024-07-27T06:46:00.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.587+0000 7f021e29a640 1 -- 172.21.15.29:0/1661683305 shutdown_connections 2024-07-27T06:46:00.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.587+0000 7f021e29a640 1 --2- 172.21.15.29:0/1661683305 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f021810fc30 0x7f0218112050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:00.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.587+0000 7f021e29a640 1 --2- 172.21.15.29:0/1661683305 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02181029d0 0x7f021810f6f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:00.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.587+0000 7f021e29a640 1 --2- 172.21.15.29:0/1661683305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0218102080 0x7f0218102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:00.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.587+0000 7f021e29a640 1 -- 172.21.15.29:0/1661683305 >> 172.21.15.29:0/1661683305 conn(0x7f0218079190 msgr2=0x7f02180795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:00.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.587+0000 7f021e29a640 1 -- 172.21.15.29:0/1661683305 shutdown_connections 2024-07-27T06:46:00.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.587+0000 7f021e29a640 1 -- 172.21.15.29:0/1661683305 wait complete. 2024-07-27T06:46:00.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.588+0000 7f021e29a640 1 Processor -- start 2024-07-27T06:46:00.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.588+0000 7f021e29a640 1 -- start start 2024-07-27T06:46:00.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.588+0000 7f021e29a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0218102080 0x7f021819c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:00.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.588+0000 7f021e29a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02181029d0 0x7f021819c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:00.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.588+0000 7f021e29a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021810fc30 0x7f02181a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:00.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.588+0000 7f021e29a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0218115690 con 0x7f021810fc30 2024-07-27T06:46:00.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.588+0000 7f021e29a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f0218115510 con 0x7f02181029d0 2024-07-27T06:46:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.588+0000 7f021e29a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0218115810 con 0x7f0218102080 2024-07-27T06:46:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.588+0000 7f021c837640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021810fc30 0x7f02181a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.588+0000 7f021c837640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021810fc30 0x7f02181a1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38968/0 (socket says 172.21.15.29:38968) 2024-07-27T06:46:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.588+0000 7f021c837640 1 -- 172.21.15.29:0/559357836 learned_addr learned my addr 172.21.15.29:0/559357836 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.588+0000 7f0217fff640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0218102080 0x7f021819c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.589+0000 7f020ffff640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02181029d0 0x7f021819c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.589+0000 7f021c837640 1 -- 172.21.15.29:0/559357836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0218102080 msgr2=0x7f021819c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.589+0000 7f021c837640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0218102080 0x7f021819c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.589+0000 7f021c837640 1 -- 172.21.15.29:0/559357836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02181029d0 msgr2=0x7f021819c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.589+0000 7f021c837640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02181029d0 0x7f021819c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.589+0000 7f021c837640 1 -- 172.21.15.29:0/559357836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02181a2090 con 0x7f021810fc30 2024-07-27T06:46:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.589+0000 7f021c837640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021810fc30 0x7f02181a1960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f021000cad0 tx=0x7f021000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.589+0000 7f0215ffb640 1 -- 172.21.15.29:0/559357836 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021000ae20 con 0x7f021810fc30 2024-07-27T06:46:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.589+0000 7f0215ffb640 1 -- 172.21.15.29:0/559357836 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0210005020 con 0x7f021810fc30 2024-07-27T06:46:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.589+0000 7f0215ffb640 1 -- 172.21.15.29:0/559357836 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021000f6b0 con 0x7f021810fc30 2024-07-27T06:46:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.589+0000 7f021e29a640 1 -- 172.21.15.29:0/559357836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02181a2380 con 0x7f021810fc30 2024-07-27T06:46:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.589+0000 7f021e29a640 1 -- 172.21.15.29:0/559357836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02181a8ae0 con 0x7f021810fc30 2024-07-27T06:46:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.590+0000 7f021e29a640 1 -- 172.21.15.29:0/559357836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f021806bc20 con 0x7f021810fc30 2024-07-27T06:46:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.592+0000 7f0215ffb640 1 -- 172.21.15.29:0/559357836 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f02100051d0 con 0x7f021810fc30 2024-07-27T06:46:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.592+0000 7f0215ffb640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f02040b3ca0 0x7f02040b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.592+0000 7f0217fff640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f02040b3ca0 0x7f02040b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.593+0000 7f0215ffb640 1 -- 172.21.15.29:0/559357836 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f02100e4080 con 0x7f021810fc30 2024-07-27T06:46:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.593+0000 7f0217fff640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f02040b3ca0 0x7f02040b61b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f02080058c0 tx=0x7f020800a250 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.595+0000 7f0215ffb640 1 -- 172.21.15.29:0/559357836 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0210014030 con 0x7f021810fc30 2024-07-27T06:46:00.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.795+0000 7f021e29a640 1 -- 172.21.15.29:0/559357836 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f0218109ec0 con 0x7f02040b3ca0 2024-07-27T06:46:00.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.796+0000 7f0215ffb640 1 -- 172.21.15.29:0/559357836 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f0218109ec0 con 0x7f02040b3ca0 2024-07-27T06:46:00.797 INFO:teuthology.orchestra.run.smithi029.stdout:Backend: cephadm 2024-07-27T06:46:00.797 INFO:teuthology.orchestra.run.smithi029.stdout:Available: Yes 2024-07-27T06:46:00.797 INFO:teuthology.orchestra.run.smithi029.stdout:Paused: No 2024-07-27T06:46:00.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.798+0000 7f021e29a640 1 -- 172.21.15.29:0/559357836 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f02040b3ca0 msgr2=0x7f02040b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:00.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.798+0000 7f021e29a640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f02040b3ca0 0x7f02040b61b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f02080058c0 tx=0x7f020800a250 comp rx=0 tx=0).stop 2024-07-27T06:46:00.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.798+0000 7f021e29a640 1 -- 172.21.15.29:0/559357836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021810fc30 msgr2=0x7f02181a1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:00.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.798+0000 7f021e29a640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021810fc30 0x7f02181a1960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f021000cad0 tx=0x7f021000cf90 comp rx=0 tx=0).stop 2024-07-27T06:46:00.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.798+0000 7f021e29a640 1 -- 172.21.15.29:0/559357836 shutdown_connections 2024-07-27T06:46:00.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.798+0000 7f021e29a640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f021810fc30 0x7f02181a1960 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:00.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.798+0000 7f021e29a640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02181029d0 0x7f021819c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:00.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.798+0000 7f021e29a640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f02040b3ca0 0x7f02040b61b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:00.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.798+0000 7f021e29a640 1 --2- 172.21.15.29:0/559357836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0218102080 0x7f021819c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:00.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.798+0000 7f021e29a640 1 -- 172.21.15.29:0/559357836 >> 172.21.15.29:0/559357836 conn(0x7f0218079190 msgr2=0x7f02181097b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:00.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.798+0000 7f021e29a640 1 -- 172.21.15.29:0/559357836 shutdown_connections 2024-07-27T06:46:00.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:00.798+0000 7f021e29a640 1 -- 172.21.15.29:0/559357836 wait complete. 2024-07-27T06:46:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:01 smithi029 ceph-mon[32610]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:01 smithi077 ceph-mon[37153]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:01.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- bash -c 'ceph orch ps' 2024-07-27T06:46:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:01 smithi047 ceph-mon[36835]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:02.347 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:02 smithi029 ceph-mon[32610]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:02 smithi077 ceph-mon[37153]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:02 smithi047 ceph-mon[36835]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:03.339 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:03 smithi029 ceph-mon[32610]: pgmap v368: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:03 smithi077 ceph-mon[37153]: pgmap v368: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:03 smithi047 ceph-mon[36835]: pgmap v368: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:03.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.720+0000 7f96daafe640 1 -- 172.21.15.29:0/1800548064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96d4108740 msgr2=0x7f96d410ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:03.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.720+0000 7f96daafe640 1 --2- 172.21.15.29:0/1800548064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96d4108740 0x7f96d410ef70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009ca0 tx=0x7f96cc02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:46:03.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.720+0000 7f96daafe640 1 -- 172.21.15.29:0/1800548064 shutdown_connections 2024-07-27T06:46:03.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.720+0000 7f96daafe640 1 --2- 172.21.15.29:0/1800548064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96d4108740 0x7f96d410ef70 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:03.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.720+0000 7f96daafe640 1 --2- 172.21.15.29:0/1800548064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96d4107df0 0x7f96d4108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:03.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.720+0000 7f96daafe640 1 --2- 172.21.15.29:0/1800548064 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96d41024b0 0x7f96d41028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:03.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.720+0000 7f96daafe640 1 -- 172.21.15.29:0/1800548064 >> 172.21.15.29:0/1800548064 conn(0x7f96d4079170 msgr2=0x7f96d4079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:03.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.721+0000 7f96daafe640 1 -- 172.21.15.29:0/1800548064 shutdown_connections 2024-07-27T06:46:03.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.721+0000 7f96daafe640 1 -- 172.21.15.29:0/1800548064 wait complete. 2024-07-27T06:46:03.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.721+0000 7f96daafe640 1 Processor -- start 2024-07-27T06:46:03.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.721+0000 7f96daafe640 1 -- start start 2024-07-27T06:46:03.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96daafe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96d41024b0 0x7f96d4199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:03.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96daafe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96d4107df0 0x7f96d419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:03.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96daafe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96d4108740 0x7f96d419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:03.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96daafe640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41134d0 con 0x7f96d4108740 2024-07-27T06:46:03.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96daafe640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f96d4113350 con 0x7f96d4107df0 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96daafe640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f96d4113650 con 0x7f96d41024b0 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d909b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96d4108740 0x7f96d419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d909b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96d4108740 0x7f96d419f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38990/0 (socket says 172.21.15.29:38990) 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d909b640 1 -- 172.21.15.29:0/1009983220 learned_addr learned my addr 172.21.15.29:0/1009983220 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d889a640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96d41024b0 0x7f96d4199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d3fff640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96d4107df0 0x7f96d419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d909b640 1 -- 172.21.15.29:0/1009983220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96d41024b0 msgr2=0x7f96d4199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d909b640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96d41024b0 0x7f96d4199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d909b640 1 -- 172.21.15.29:0/1009983220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96d4107df0 msgr2=0x7f96d419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d909b640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96d4107df0 0x7f96d419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d909b640 1 -- 172.21.15.29:0/1009983220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d419fdd0 con 0x7f96d4108740 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d3fff640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96d4107df0 0x7f96d419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:46:03.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d909b640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96d4108740 0x7f96d419f6a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f96cc005cc0 tx=0x7f96cc02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:03.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.722+0000 7f96d1ffb640 1 -- 172.21.15.29:0/1009983220 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc003bf0 con 0x7f96d4108740 2024-07-27T06:46:03.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.723+0000 7f96d1ffb640 1 -- 172.21.15.29:0/1009983220 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96cc003d90 con 0x7f96d4108740 2024-07-27T06:46:03.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.723+0000 7f96daafe640 1 -- 172.21.15.29:0/1009983220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d41a0060 con 0x7f96d4108740 2024-07-27T06:46:03.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.723+0000 7f96d1ffb640 1 -- 172.21.15.29:0/1009983220 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc02fbc0 con 0x7f96d4108740 2024-07-27T06:46:03.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.723+0000 7f96daafe640 1 -- 172.21.15.29:0/1009983220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d41a6820 con 0x7f96d4108740 2024-07-27T06:46:03.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.724+0000 7f96daafe640 1 -- 172.21.15.29:0/1009983220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d4103730 con 0x7f96d4108740 2024-07-27T06:46:03.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.725+0000 7f96d1ffb640 1 -- 172.21.15.29:0/1009983220 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f96cc02fd60 con 0x7f96d4108740 2024-07-27T06:46:03.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.726+0000 7f96d1ffb640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f96c40b3c20 0x7f96c40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:03.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.726+0000 7f96d889a640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f96c40b3c20 0x7f96c40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:03.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.726+0000 7f96d1ffb640 1 -- 172.21.15.29:0/1009983220 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f96cc105660 con 0x7f96d4108740 2024-07-27T06:46:03.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.727+0000 7f96d889a640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f96c40b3c20 0x7f96c40b6130 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f96c8005d00 tx=0x7f96c800b410 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:03.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.729+0000 7f96d1ffb640 1 -- 172.21.15.29:0/1009983220 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f96cc0ccb40 con 0x7f96d4108740 2024-07-27T06:46:03.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.919+0000 7f96daafe640 1 -- 172.21.15.29:0/1009983220 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f96d410a5a0 con 0x7f96c40b3c20 2024-07-27T06:46:03.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.934+0000 7f96d1ffb640 1 -- 172.21.15.29:0/1009983220 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f96c800bcc0 con 0x7f96c40b3c20 2024-07-27T06:46:03.934 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mds.a smithi029 running (3m) 39s ago 3m 88.0M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 5c7d8d8a910a 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mds.b smithi047 running (3m) 55s ago 3m 86.3M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 6f166d060dc5 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mds.c smithi077 running (3m) 55s ago 3m 90.2M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 9bc9bead529b 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mds.d smithi029 running (3m) 39s ago 3m 82.2M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa d25557f5b03d 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mds.e smithi047 running (3m) 55s ago 3m 89.4M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 818f7d59409c 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mds.f smithi077 running (3m) 55s ago 3m 88.6M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa e312408b0360 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mds.g smithi029 running (3m) 39s ago 3m 88.0M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa af74b58927c8 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mds.h smithi047 running (3m) 55s ago 3m 87.7M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa d8f4ee956bad 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mds.i smithi077 running (3m) 55s ago 3m 94.5M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 12d1fea81523 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mds.j smithi029 running (3m) 39s ago 3m 82.0M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa b33ce37cc3a0 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mds.k smithi047 running (3m) 55s ago 3m 87.1M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa d6773413f40b 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mds.l smithi077 running (3m) 55s ago 3m 89.9M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 3ba94332af19 2024-07-27T06:46:03.935 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi029 *:9283,8765 running (12m) 39s ago 11m 584M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 43ecc3bd4dd9 2024-07-27T06:46:03.936 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi047 *:8443,8765 running (9m) 55s ago 9m 525M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa ef83de724406 2024-07-27T06:46:03.936 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.z smithi077 *:8443,8765 running (9m) 55s ago 9m 522M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 67c05b33d676 2024-07-27T06:46:03.936 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (12m) 39s ago 12m 120M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 11b5d831c49c 2024-07-27T06:46:03.936 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi047 running (10m) 55s ago 10m 108M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 16df0c60cacf 2024-07-27T06:46:03.936 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi077 running (10m) 55s ago 10m 109M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa cee05e56bdd0 2024-07-27T06:46:03.936 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (8m) 39s ago 8m 117M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 1b35a8750dac 2024-07-27T06:46:03.936 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi047 running (8m) 55s ago 8m 121M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 452ae39b8300 2024-07-27T06:46:03.936 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi077 running (7m) 55s ago 7m 112M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 498d78a4dcd3 2024-07-27T06:46:03.936 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (7m) 39s ago 7m 112M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 46b05d84b47a 2024-07-27T06:46:03.936 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi047 running (7m) 55s ago 7m 113M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 8d1851bc6716 2024-07-27T06:46:03.936 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi077 running (6m) 55s ago 6m 113M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 15abebcb10d7 2024-07-27T06:46:03.936 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi029 running (6m) 39s ago 6m 113M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa dcc6d39b70ea 2024-07-27T06:46:03.937 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi047 running (5m) 55s ago 5m 122M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 307288e550c3 2024-07-27T06:46:03.937 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8 smithi077 running (5m) 55s ago 5m 111M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa af333800843b 2024-07-27T06:46:03.937 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9 smithi029 running (4m) 39s ago 4m 120M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa e8a166e6b26e 2024-07-27T06:46:03.937 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10 smithi047 running (4m) 55s ago 4m 114M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 24d93f1b9887 2024-07-27T06:46:03.937 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11 smithi077 running (4m) 55s ago 4m 111M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 4b0c7f422434 2024-07-27T06:46:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.936+0000 7f96daafe640 1 -- 172.21.15.29:0/1009983220 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f96c40b3c20 msgr2=0x7f96c40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.936+0000 7f96daafe640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f96c40b3c20 0x7f96c40b6130 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f96c8005d00 tx=0x7f96c800b410 comp rx=0 tx=0).stop 2024-07-27T06:46:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.936+0000 7f96daafe640 1 -- 172.21.15.29:0/1009983220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96d4108740 msgr2=0x7f96d419f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.936+0000 7f96daafe640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96d4108740 0x7f96d419f6a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f96cc005cc0 tx=0x7f96cc02f380 comp rx=0 tx=0).stop 2024-07-27T06:46:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.936+0000 7f96daafe640 1 -- 172.21.15.29:0/1009983220 shutdown_connections 2024-07-27T06:46:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.936+0000 7f96daafe640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96d4108740 0x7f96d419f6a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.936+0000 7f96daafe640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96d4107df0 0x7f96d419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.936+0000 7f96daafe640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f96c40b3c20 0x7f96c40b6130 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.936+0000 7f96daafe640 1 --2- 172.21.15.29:0/1009983220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96d41024b0 0x7f96d4199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.936+0000 7f96daafe640 1 -- 172.21.15.29:0/1009983220 >> 172.21.15.29:0/1009983220 conn(0x7f96d4079170 msgr2=0x7f96d4104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:03.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.936+0000 7f96daafe640 1 -- 172.21.15.29:0/1009983220 shutdown_connections 2024-07-27T06:46:03.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:03.936+0000 7f96daafe640 1 -- 172.21.15.29:0/1009983220 wait complete. 2024-07-27T06:46:04.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- bash -c 'ceph orch ls' 2024-07-27T06:46:05.049 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:05.440 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:05 smithi029 ceph-mon[32610]: pgmap v369: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:05.440 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:05 smithi029 ceph-mon[32610]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:05 smithi077 ceph-mon[37153]: pgmap v369: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:05 smithi077 ceph-mon[37153]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:05 smithi047 ceph-mon[36835]: pgmap v369: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:05 smithi047 ceph-mon[36835]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:06.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.766+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1450673900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102970 msgr2=0x7fdd3c10f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:06.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.766+0000 7fdd43ef3640 1 --2- 172.21.15.29:0/1450673900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102970 0x7fdd3c10f690 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009900 tx=0x7fdd2c02a480 comp rx=0 tx=0).stop 2024-07-27T06:46:06.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.766+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1450673900 shutdown_connections 2024-07-27T06:46:06.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.766+0000 7fdd43ef3640 1 --2- 172.21.15.29:0/1450673900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd3c10fbd0 0x7fdd3c111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:06.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.766+0000 7fdd43ef3640 1 --2- 172.21.15.29:0/1450673900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102970 0x7fdd3c10f690 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.766+0000 7fdd43ef3640 1 --2- 172.21.15.29:0/1450673900 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd3c102020 0x7fdd3c102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.766+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1450673900 >> 172.21.15.29:0/1450673900 conn(0x7fdd3c079170 msgr2=0x7fdd3c079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.766+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1450673900 shutdown_connections 2024-07-27T06:46:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.766+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1450673900 wait complete. 2024-07-27T06:46:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.766+0000 7fdd43ef3640 1 Processor -- start 2024-07-27T06:46:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd43ef3640 1 -- start start 2024-07-27T06:46:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd43ef3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102020 0x7fdd3c19c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd43ef3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd3c102970 0x7fdd3c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd43ef3640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd3c10fbd0 0x7fdd3c1a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd43ef3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3c115690 con 0x7fdd3c102020 2024-07-27T06:46:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd43ef3640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3c115510 con 0x7fdd3c10fbd0 2024-07-27T06:46:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd43ef3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3c115810 con 0x7fdd3c102970 2024-07-27T06:46:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd41c8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102020 0x7fdd3c19c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd41c8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102020 0x7fdd3c19c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39006/0 (socket says 172.21.15.29:39006) 2024-07-27T06:46:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd41c8f640 1 -- 172.21.15.29:0/1558648130 learned_addr learned my addr 172.21.15.29:0/1558648130 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd4148e640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd3c102970 0x7fdd3c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd42490640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd3c10fbd0 0x7fdd3c1a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd41c8f640 1 -- 172.21.15.29:0/1558648130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd3c102970 msgr2=0x7fdd3c19c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd41c8f640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd3c102970 0x7fdd3c19c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd41c8f640 1 -- 172.21.15.29:0/1558648130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd3c10fbd0 msgr2=0x7fdd3c1a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd41c8f640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd3c10fbd0 0x7fdd3c1a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd41c8f640 1 -- 172.21.15.29:0/1558648130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3c1a2030 con 0x7fdd3c102020 2024-07-27T06:46:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.767+0000 7fdd42490640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd3c10fbd0 0x7fdd3c1a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:46:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.768+0000 7fdd41c8f640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102020 0x7fdd3c19c0c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800d950 tx=0x7fdd3800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.768+0000 7fdd32ffd640 1 -- 172.21.15.29:0/1558648130 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38017030 con 0x7fdd3c102020 2024-07-27T06:46:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.768+0000 7fdd32ffd640 1 -- 172.21.15.29:0/1558648130 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd38004020 con 0x7fdd3c102020 2024-07-27T06:46:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.768+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1558648130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd3c1a2320 con 0x7fdd3c102020 2024-07-27T06:46:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.768+0000 7fdd32ffd640 1 -- 172.21.15.29:0/1558648130 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd380047a0 con 0x7fdd3c102020 2024-07-27T06:46:06.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.768+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1558648130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd3c1a8a80 con 0x7fdd3c102020 2024-07-27T06:46:06.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.769+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1558648130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd3c06bc70 con 0x7fdd3c102020 2024-07-27T06:46:06.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.771+0000 7fdd32ffd640 1 -- 172.21.15.29:0/1558648130 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fdd38009a30 con 0x7fdd3c102020 2024-07-27T06:46:06.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.772+0000 7fdd32ffd640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdd280b3c20 0x7fdd280b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:06.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.772+0000 7fdd4148e640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdd280b3c20 0x7fdd280b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:06.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.772+0000 7fdd32ffd640 1 -- 172.21.15.29:0/1558648130 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fdd380e3f90 con 0x7fdd3c102020 2024-07-27T06:46:06.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.773+0000 7fdd4148e640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdd280b3c20 0x7fdd280b6130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c031440 tx=0x7fdd2c005d50 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:06.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.775+0000 7fdd32ffd640 1 -- 172.21.15.29:0/1558648130 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdd380ab4d0 con 0x7fdd3c102020 2024-07-27T06:46:06.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.967+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1558648130 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fdd3c109e80 con 0x7fdd280b3c20 2024-07-27T06:46:06.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.972+0000 7fdd32ffd640 1 -- 172.21.15.29:0/1558648130 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fdd3c109e80 con 0x7fdd280b3c20 2024-07-27T06:46:06.973 INFO:teuthology.orchestra.run.smithi029.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-07-27T06:46:06.973 INFO:teuthology.orchestra.run.smithi029.stdout:mds.all 12/12 58s ago 3m smithi029=a;smithi029=d;smithi029=g;smithi029=j;smithi047=b;smithi047=e;smithi047=h;smithi047=k;smithi077=c;smithi077=f;smithi077=i;smithi077=l;count:12 2024-07-27T06:46:06.973 INFO:teuthology.orchestra.run.smithi029.stdout:mgr 3/3 58s ago 9m smithi029=x;smithi047=y;smithi077=z;count:3 2024-07-27T06:46:06.973 INFO:teuthology.orchestra.run.smithi029.stdout:mon 3/3 58s ago 10m smithi029:172.21.15.29=a;smithi047:172.21.15.47=b;smithi077:172.21.15.77=c;count:3 2024-07-27T06:46:06.973 INFO:teuthology.orchestra.run.smithi029.stdout:osd 12 58s ago - 2024-07-27T06:46:06.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.974+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1558648130 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdd280b3c20 msgr2=0x7fdd280b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.974+0000 7fdd43ef3640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdd280b3c20 0x7fdd280b6130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c031440 tx=0x7fdd2c005d50 comp rx=0 tx=0).stop 2024-07-27T06:46:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.974+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1558648130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102020 msgr2=0x7fdd3c19c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.974+0000 7fdd43ef3640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102020 0x7fdd3c19c0c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdd3800d950 tx=0x7fdd3800de10 comp rx=0 tx=0).stop 2024-07-27T06:46:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.975+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1558648130 shutdown_connections 2024-07-27T06:46:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.975+0000 7fdd43ef3640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd3c10fbd0 0x7fdd3c1a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.975+0000 7fdd43ef3640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fdd280b3c20 0x7fdd280b6130 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.975+0000 7fdd43ef3640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd3c102970 0x7fdd3c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.975+0000 7fdd43ef3640 1 --2- 172.21.15.29:0/1558648130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd3c102020 0x7fdd3c19c0c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.975+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1558648130 >> 172.21.15.29:0/1558648130 conn(0x7fdd3c079170 msgr2=0x7fdd3c109770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.975+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1558648130 shutdown_connections 2024-07-27T06:46:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:06.975+0000 7fdd43ef3640 1 -- 172.21.15.29:0/1558648130 wait complete. 2024-07-27T06:46:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:07 smithi077 ceph-mon[37153]: pgmap v370: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:07 smithi047 ceph-mon[36835]: pgmap v370: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:07 smithi029 ceph-mon[32610]: pgmap v370: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:07.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- bash -c 'ceph orch host ls' 2024-07-27T06:46:08.087 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:08 smithi029 ceph-mon[32610]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:08 smithi077 ceph-mon[37153]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:08 smithi047 ceph-mon[36835]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:09.596 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:09 smithi029 ceph-mon[32610]: pgmap v371: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:09.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:09 smithi047 ceph-mon[36835]: pgmap v371: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:09 smithi077 ceph-mon[37153]: pgmap v371: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:09.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.951+0000 7f750f73f640 1 -- 172.21.15.29:0/1938831059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7508103590 msgr2=0x7f75081039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:09.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.951+0000 7f750f73f640 1 --2- 172.21.15.29:0/1938831059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7508103590 0x7f75081039a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7504009900 tx=0x7f750402a410 comp rx=0 tx=0).stop 2024-07-27T06:46:09.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.951+0000 7f750f73f640 1 -- 172.21.15.29:0/1938831059 shutdown_connections 2024-07-27T06:46:09.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.951+0000 7f750f73f640 1 --2- 172.21.15.29:0/1938831059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7508109820 0x7f7508110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:09.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.951+0000 7f750f73f640 1 --2- 172.21.15.29:0/1938831059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7508108ed0 0x7f75081092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:09.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.951+0000 7f750f73f640 1 --2- 172.21.15.29:0/1938831059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7508103590 0x7f75081039a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:09.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.951+0000 7f750f73f640 1 -- 172.21.15.29:0/1938831059 >> 172.21.15.29:0/1938831059 conn(0x7f750807b280 msgr2=0x7f750807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:09.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.951+0000 7f750f73f640 1 -- 172.21.15.29:0/1938831059 shutdown_connections 2024-07-27T06:46:09.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.951+0000 7f750f73f640 1 -- 172.21.15.29:0/1938831059 wait complete. 2024-07-27T06:46:09.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.952+0000 7f750f73f640 1 Processor -- start 2024-07-27T06:46:09.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.952+0000 7f750f73f640 1 -- start start 2024-07-27T06:46:09.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.952+0000 7f750f73f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7508103590 0x7f7508075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:09.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.952+0000 7f750f73f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7508108ed0 0x7f7508075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:09.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.952+0000 7f750f73f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7508109820 0x7f7508075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:09.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.952+0000 7f750f73f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f75081134f0 con 0x7f7508108ed0 2024-07-27T06:46:09.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.952+0000 7f750f73f640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7508113370 con 0x7f7508109820 2024-07-27T06:46:09.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.952+0000 7f750f73f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7508113670 con 0x7f7508103590 2024-07-27T06:46:09.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750ccda640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7508108ed0 0x7f7508075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:09.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750ccda640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7508108ed0 0x7f7508075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56008/0 (socket says 172.21.15.29:56008) 2024-07-27T06:46:09.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750ccda640 1 -- 172.21.15.29:0/3711270991 learned_addr learned my addr 172.21.15.29:0/3711270991 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:09.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750dcdc640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7508109820 0x7f7508075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:09.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750d4db640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7508103590 0x7f7508075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:09.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750ccda640 1 -- 172.21.15.29:0/3711270991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7508103590 msgr2=0x7f7508075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:09.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750ccda640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7508103590 0x7f7508075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:09.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750ccda640 1 -- 172.21.15.29:0/3711270991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7508109820 msgr2=0x7f7508075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750ccda640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7508109820 0x7f7508075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750ccda640 1 -- 172.21.15.29:0/3711270991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75081a8240 con 0x7f7508108ed0 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750dcdc640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7508109820 0x7f7508075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750d4db640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7508103590 0x7f7508075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750ccda640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7508108ed0 0x7f7508075680 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f74fc00d750 tx=0x7f74fc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f74fa7fc640 1 -- 172.21.15.29:0/3711270991 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc017070 con 0x7f7508108ed0 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750f73f640 1 -- 172.21.15.29:0/3711270991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75081a8530 con 0x7f7508108ed0 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f74fa7fc640 1 -- 172.21.15.29:0/3711270991 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74fc0054d0 con 0x7f7508108ed0 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f74fa7fc640 1 -- 172.21.15.29:0/3711270991 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc004c30 con 0x7f7508108ed0 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.953+0000 7f750f73f640 1 -- 172.21.15.29:0/3711270991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75081a8a20 con 0x7f7508108ed0 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.954+0000 7f750f73f640 1 -- 172.21.15.29:0/3711270991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75081039a0 con 0x7f7508108ed0 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.957+0000 7f74fa7fc640 1 -- 172.21.15.29:0/3711270991 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f74fc004dd0 con 0x7f7508108ed0 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.958+0000 7f74fa7fc640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f74f40b3c20 0x7f74f40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.958+0000 7f750d4db640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f74f40b3c20 0x7f74f40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.958+0000 7f74fa7fc640 1 -- 172.21.15.29:0/3711270991 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f74fc0e3c60 con 0x7f7508108ed0 2024-07-27T06:46:09.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.958+0000 7f750d4db640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f74f40b3c20 0x7f74f40b6130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7504000c00 tx=0x7f7504009680 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:09.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:09.961+0000 7f74fa7fc640 1 -- 172.21.15.29:0/3711270991 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f74fc0ab140 con 0x7f7508108ed0 2024-07-27T06:46:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.149+0000 7f750f73f640 1 -- 172.21.15.29:0/3711270991 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f7508063230 con 0x7f74f40b3c20 2024-07-27T06:46:10.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.151+0000 7f74fa7fc640 1 -- 172.21.15.29:0/3711270991 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+187 (secure 0 0 0) 0x7f7508063230 con 0x7f74f40b3c20 2024-07-27T06:46:10.151 INFO:teuthology.orchestra.run.smithi029.stdout:HOST ADDR LABELS STATUS 2024-07-27T06:46:10.152 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 172.21.15.29 2024-07-27T06:46:10.152 INFO:teuthology.orchestra.run.smithi029.stdout:smithi047 172.21.15.47 2024-07-27T06:46:10.152 INFO:teuthology.orchestra.run.smithi029.stdout:smithi077 172.21.15.77 2024-07-27T06:46:10.152 INFO:teuthology.orchestra.run.smithi029.stdout:3 hosts in cluster 2024-07-27T06:46:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.153+0000 7f750f73f640 1 -- 172.21.15.29:0/3711270991 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f74f40b3c20 msgr2=0x7f74f40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.153+0000 7f750f73f640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f74f40b3c20 0x7f74f40b6130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7504000c00 tx=0x7f7504009680 comp rx=0 tx=0).stop 2024-07-27T06:46:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.153+0000 7f750f73f640 1 -- 172.21.15.29:0/3711270991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7508108ed0 msgr2=0x7f7508075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.153+0000 7f750f73f640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7508108ed0 0x7f7508075680 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f74fc00d750 tx=0x7f74fc00dc10 comp rx=0 tx=0).stop 2024-07-27T06:46:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.153+0000 7f750f73f640 1 -- 172.21.15.29:0/3711270991 shutdown_connections 2024-07-27T06:46:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.153+0000 7f750f73f640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7508109820 0x7f7508075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.153+0000 7f750f73f640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7508108ed0 0x7f7508075680 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.153+0000 7f750f73f640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f74f40b3c20 0x7f74f40b6130 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:10.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.153+0000 7f750f73f640 1 --2- 172.21.15.29:0/3711270991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7508103590 0x7f7508075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:10.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.153+0000 7f750f73f640 1 -- 172.21.15.29:0/3711270991 >> 172.21.15.29:0/3711270991 conn(0x7f750807b280 msgr2=0x7f7508107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:10.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.153+0000 7f750f73f640 1 -- 172.21.15.29:0/3711270991 shutdown_connections 2024-07-27T06:46:10.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:10.153+0000 7f750f73f640 1 -- 172.21.15.29:0/3711270991 wait complete. 2024-07-27T06:46:10.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- bash -c 'ceph orch device ls' 2024-07-27T06:46:11.262 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:11 smithi077 ceph-mon[37153]: pgmap v372: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:11 smithi077 ceph-mon[37153]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:11.610 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:11 smithi029 ceph-mon[32610]: pgmap v372: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:11.610 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:11 smithi029 ceph-mon[32610]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:11 smithi047 ceph-mon[36835]: pgmap v372: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:11 smithi047 ceph-mon[36835]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:13.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.090+0000 7f8f5fe55640 1 -- 172.21.15.29:0/2129147233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f58102490 msgr2=0x7f8f581028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:13.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.090+0000 7f8f5fe55640 1 --2- 172.21.15.29:0/2129147233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f58102490 0x7f8f581028a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009ca0 tx=0x7f8f5002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:46:13.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.090+0000 7f8f5fe55640 1 -- 172.21.15.29:0/2129147233 shutdown_connections 2024-07-27T06:46:13.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.090+0000 7f8f5fe55640 1 --2- 172.21.15.29:0/2129147233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f58108720 0x7f8f5810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.090+0000 7f8f5fe55640 1 --2- 172.21.15.29:0/2129147233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f58107dd0 0x7f8f581081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.090+0000 7f8f5fe55640 1 --2- 172.21.15.29:0/2129147233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f58102490 0x7f8f581028a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.090+0000 7f8f5fe55640 1 -- 172.21.15.29:0/2129147233 >> 172.21.15.29:0/2129147233 conn(0x7f8f58079160 msgr2=0x7f8f58079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.090+0000 7f8f5fe55640 1 -- 172.21.15.29:0/2129147233 shutdown_connections 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.090+0000 7f8f5fe55640 1 -- 172.21.15.29:0/2129147233 wait complete. 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5fe55640 1 Processor -- start 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5fe55640 1 -- start start 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5fe55640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f58102490 0x7f8f58103c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5fe55640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f58107dd0 0x7f8f581041c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5fe55640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f58108720 0x7f8f58104700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5fe55640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8f581123a0 con 0x7f8f58108720 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5fe55640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f8f58112220 con 0x7f8f58107dd0 2024-07-27T06:46:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5fe55640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8f58112520 con 0x7f8f58102490 2024-07-27T06:46:13.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5e3f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f58108720 0x7f8f58104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:13.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5e3f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f58108720 0x7f8f58104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56014/0 (socket says 172.21.15.29:56014) 2024-07-27T06:46:13.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5e3f2640 1 -- 172.21.15.29:0/3607707977 learned_addr learned my addr 172.21.15.29:0/3607707977 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:13.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5dbf1640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f58102490 0x7f8f58103c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:13.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.091+0000 7f8f5d3f0640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f58107dd0 0x7f8f581041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:13.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.092+0000 7f8f5e3f2640 1 -- 172.21.15.29:0/3607707977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f58102490 msgr2=0x7f8f58103c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:13.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.092+0000 7f8f5e3f2640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f58102490 0x7f8f58103c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:13.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.092+0000 7f8f5e3f2640 1 -- 172.21.15.29:0/3607707977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f58107dd0 msgr2=0x7f8f581041c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:13.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.092+0000 7f8f5e3f2640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f58107dd0 0x7f8f581041c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.092+0000 7f8f5e3f2640 1 -- 172.21.15.29:0/3607707977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f581a8140 con 0x7f8f58108720 2024-07-27T06:46:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.092+0000 7f8f5e3f2640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f58108720 0x7f8f58104700 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800e970 tx=0x7f8f4800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.092+0000 7f8f56ffd640 1 -- 172.21.15.29:0/3607707977 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4800ccc0 con 0x7f8f58108720 2024-07-27T06:46:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.092+0000 7f8f56ffd640 1 -- 172.21.15.29:0/3607707977 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f4800ce60 con 0x7f8f58108720 2024-07-27T06:46:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.092+0000 7f8f5fe55640 1 -- 172.21.15.29:0/3607707977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f581a8430 con 0x7f8f58108720 2024-07-27T06:46:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.092+0000 7f8f56ffd640 1 -- 172.21.15.29:0/3607707977 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f48010430 con 0x7f8f58108720 2024-07-27T06:46:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.092+0000 7f8f5fe55640 1 -- 172.21.15.29:0/3607707977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f581a8970 con 0x7f8f58108720 2024-07-27T06:46:13.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.093+0000 7f8f5fe55640 1 -- 172.21.15.29:0/3607707977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f581028a0 con 0x7f8f58108720 2024-07-27T06:46:13.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.095+0000 7f8f56ffd640 1 -- 172.21.15.29:0/3607707977 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f8f48004f70 con 0x7f8f58108720 2024-07-27T06:46:13.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.095+0000 7f8f56ffd640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8f400b3c20 0x7f8f400b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:13.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.096+0000 7f8f56ffd640 1 -- 172.21.15.29:0/3607707977 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f8f480e4320 con 0x7f8f58108720 2024-07-27T06:46:13.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.096+0000 7f8f5dbf1640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8f400b3c20 0x7f8f400b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:13.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.096+0000 7f8f5dbf1640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8f400b3c20 0x7f8f400b6130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8f50005e30 tx=0x7f8f50009360 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:13.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.099+0000 7f8f56ffd640 1 -- 172.21.15.29:0/3607707977 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8f48055600 con 0x7f8f58108720 2024-07-27T06:46:13.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.289+0000 7f8f5fe55640 1 -- 172.21.15.29:0/3607707977 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f8f5806a8e0 con 0x7f8f400b3c20 2024-07-27T06:46:13.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.291+0000 7f8f56ffd640 1 -- 172.21.15.29:0/3607707977 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f8f5806a8e0 con 0x7f8f400b3c20 2024-07-27T06:46:13.293 INFO:teuthology.orchestra.run.smithi029.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-07-27T06:46:13.293 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009B480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-07-27T06:46:13.293 INFO:teuthology.orchestra.run.smithi029.stdout:smithi047 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75130022480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-07-27T06:46:13.293 INFO:teuthology.orchestra.run.smithi029.stdout:smithi077 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-07-27T06:46:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.293+0000 7f8f5fe55640 1 -- 172.21.15.29:0/3607707977 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8f400b3c20 msgr2=0x7f8f400b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.293+0000 7f8f5fe55640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8f400b3c20 0x7f8f400b6130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8f50005e30 tx=0x7f8f50009360 comp rx=0 tx=0).stop 2024-07-27T06:46:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.293+0000 7f8f5fe55640 1 -- 172.21.15.29:0/3607707977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f58108720 msgr2=0x7f8f58104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.293+0000 7f8f5fe55640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f58108720 0x7f8f58104700 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800e970 tx=0x7f8f4800ed70 comp rx=0 tx=0).stop 2024-07-27T06:46:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.293+0000 7f8f5fe55640 1 -- 172.21.15.29:0/3607707977 shutdown_connections 2024-07-27T06:46:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.293+0000 7f8f5fe55640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8f58108720 0x7f8f58104700 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.293+0000 7f8f5fe55640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8f58107dd0 0x7f8f581041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.293+0000 7f8f5fe55640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8f400b3c20 0x7f8f400b6130 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.293+0000 7f8f5fe55640 1 --2- 172.21.15.29:0/3607707977 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f58102490 0x7f8f58103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.293+0000 7f8f5fe55640 1 -- 172.21.15.29:0/3607707977 >> 172.21.15.29:0/3607707977 conn(0x7f8f58079160 msgr2=0x7f8f581014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:13.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.293+0000 7f8f5fe55640 1 -- 172.21.15.29:0/3607707977 shutdown_connections 2024-07-27T06:46:13.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:13.293+0000 7f8f5fe55640 1 -- 172.21.15.29:0/3607707977 wait complete. 2024-07-27T06:46:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:13 smithi077 ceph-mon[37153]: pgmap v373: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:13 smithi047 ceph-mon[36835]: pgmap v373: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:13.696 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:13 smithi029 ceph-mon[32610]: pgmap v373: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:14.169 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-27T06:46:14.177 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-07-27T06:46:14.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- bash -c 'ceph fs dump' 2024-07-27T06:46:14.456 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:14.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:14 smithi029 ceph-mon[32610]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:14 smithi047 ceph-mon[36835]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:14 smithi077 ceph-mon[37153]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:46:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:15 smithi047 ceph-mon[36835]: pgmap v374: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:15 smithi029 ceph-mon[32610]: pgmap v374: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:15 smithi077 ceph-mon[37153]: pgmap v374: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.314+0000 7f1b9945e640 1 -- 172.21.15.29:0/1492724920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109820 msgr2=0x7f1b94110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.314+0000 7f1b9945e640 1 --2- 172.21.15.29:0/1492724920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109820 0x7f1b94110050 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009ca0 tx=0x7f1b8c02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:46:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.314+0000 7f1b9945e640 1 -- 172.21.15.29:0/1492724920 shutdown_connections 2024-07-27T06:46:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.314+0000 7f1b9945e640 1 --2- 172.21.15.29:0/1492724920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94109820 0x7f1b94110050 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.314+0000 7f1b9945e640 1 --2- 172.21.15.29:0/1492724920 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b94108ed0 0x7f1b941092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.314+0000 7f1b9945e640 1 --2- 172.21.15.29:0/1492724920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b94103590 0x7f1b941039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.314+0000 7f1b9945e640 1 -- 172.21.15.29:0/1492724920 >> 172.21.15.29:0/1492724920 conn(0x7f1b9407b280 msgr2=0x7f1b9407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.314+0000 7f1b9945e640 1 -- 172.21.15.29:0/1492724920 shutdown_connections 2024-07-27T06:46:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.314+0000 7f1b9945e640 1 -- 172.21.15.29:0/1492724920 wait complete. 2024-07-27T06:46:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.315+0000 7f1b9945e640 1 Processor -- start 2024-07-27T06:46:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.315+0000 7f1b9945e640 1 -- start start 2024-07-27T06:46:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.315+0000 7f1b9945e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b94103590 0x7f1b94075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.315+0000 7f1b9945e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94108ed0 0x7f1b94075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.315+0000 7f1b9945e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b94109820 0x7f1b94075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.315+0000 7f1b9945e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1b941134f0 con 0x7f1b94108ed0 2024-07-27T06:46:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.315+0000 7f1b9945e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94113370 con 0x7f1b94103590 2024-07-27T06:46:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.315+0000 7f1b9945e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1b94113670 con 0x7f1b94109820 2024-07-27T06:46:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b927fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94108ed0 0x7f1b94075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b927fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94108ed0 0x7f1b94075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56036/0 (socket says 172.21.15.29:56036) 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b927fc640 1 -- 172.21.15.29:0/1698821418 learned_addr learned my addr 172.21.15.29:0/1698821418 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b937fe640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b94109820 0x7f1b94075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b92ffd640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b94103590 0x7f1b94075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b927fc640 1 -- 172.21.15.29:0/1698821418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b94109820 msgr2=0x7f1b94075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b927fc640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b94109820 0x7f1b94075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b927fc640 1 -- 172.21.15.29:0/1698821418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b94103590 msgr2=0x7f1b94075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b927fc640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b94103590 0x7f1b94075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b927fc640 1 -- 172.21.15.29:0/1698821418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b941a80b0 con 0x7f1b94108ed0 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b927fc640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94108ed0 0x7f1b94075680 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800c8c0 tx=0x7f1b8800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b7bfff640 1 -- 172.21.15.29:0/1698821418 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b88007bb0 con 0x7f1b94108ed0 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b9945e640 1 -- 172.21.15.29:0/1698821418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b941a83a0 con 0x7f1b94108ed0 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b7bfff640 1 -- 172.21.15.29:0/1698821418 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b88007d50 con 0x7f1b94108ed0 2024-07-27T06:46:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b7bfff640 1 -- 172.21.15.29:0/1698821418 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b880187c0 con 0x7f1b94108ed0 2024-07-27T06:46:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.316+0000 7f1b9945e640 1 -- 172.21.15.29:0/1698821418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b941a8930 con 0x7f1b94108ed0 2024-07-27T06:46:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.318+0000 7f1b9945e640 1 -- 172.21.15.29:0/1698821418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b941039a0 con 0x7f1b94108ed0 2024-07-27T06:46:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.319+0000 7f1b7bfff640 1 -- 172.21.15.29:0/1698821418 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f1b8800fa20 con 0x7f1b94108ed0 2024-07-27T06:46:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.320+0000 7f1b7bfff640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1b7c0b3c20 0x7f1b7c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.320+0000 7f1b92ffd640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1b7c0b3c20 0x7f1b7c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.320+0000 7f1b7bfff640 1 -- 172.21.15.29:0/1698821418 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f1b880e41e0 con 0x7f1b94108ed0 2024-07-27T06:46:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.320+0000 7f1b92ffd640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1b7c0b3c20 0x7f1b7c0b6130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1b84005fd0 tx=0x7f1b84005850 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.323+0000 7f1b7bfff640 1 -- 172.21.15.29:0/1698821418 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1b88014030 con 0x7f1b94108ed0 2024-07-27T06:46:16.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.546+0000 7f1b9945e640 1 -- 172.21.15.29:0/1698821418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f1b94076440 con 0x7f1b94108ed0 2024-07-27T06:46:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.546+0000 7f1b7bfff640 1 -- 172.21.15.29:0/1698821418 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3118 (secure 0 0 0) 0x7f1b880ab7b0 con 0x7f1b94108ed0 2024-07-27T06:46:16.548 INFO:teuthology.orchestra.run.smithi029.stdout:e28 2024-07-27T06:46:16.548 INFO:teuthology.orchestra.run.smithi029.stdout:btime 2024-07-27T06:43:37:875457+0000 2024-07-27T06:46:16.548 INFO:teuthology.orchestra.run.smithi029.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-07-27T06:46:16.548 INFO:teuthology.orchestra.run.smithi029.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout:legacy client fscid: 1 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout:Filesystem 'cephfs' (1) 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout:fs_name cephfs 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 28 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-07-27T06:43:14.801207+0000 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-07-27T06:43:37.017169+0000 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout:tableserver 0 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout:root 0 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout:session_timeout 300 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout:session_autoclose 300 2024-07-27T06:46:16.549 INFO:teuthology.orchestra.run.smithi029.stdout:max_file_size 1099511627776 2024-07-27T06:46:16.550 INFO:teuthology.orchestra.run.smithi029.stdout:max_xattr_size 65536 2024-07-27T06:46:16.550 INFO:teuthology.orchestra.run.smithi029.stdout:required_client_features {} 2024-07-27T06:46:16.550 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure 0 2024-07-27T06:46:16.550 INFO:teuthology.orchestra.run.smithi029.stdout:last_failure_osd_epoch 0 2024-07-27T06:46:16.550 INFO:teuthology.orchestra.run.smithi029.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-27T06:46:16.550 INFO:teuthology.orchestra.run.smithi029.stdout:max_mds 5 2024-07-27T06:46:16.550 INFO:teuthology.orchestra.run.smithi029.stdout:in 0,1,2,3,4 2024-07-27T06:46:16.550 INFO:teuthology.orchestra.run.smithi029.stdout:up {0=24670,1=24658,2=24646,3=24617,4=24613} 2024-07-27T06:46:16.550 INFO:teuthology.orchestra.run.smithi029.stdout:failed 2024-07-27T06:46:16.550 INFO:teuthology.orchestra.run.smithi029.stdout:damaged 2024-07-27T06:46:16.550 INFO:teuthology.orchestra.run.smithi029.stdout:stopped 2024-07-27T06:46:16.551 INFO:teuthology.orchestra.run.smithi029.stdout:data_pools [3] 2024-07-27T06:46:16.551 INFO:teuthology.orchestra.run.smithi029.stdout:metadata_pool 2 2024-07-27T06:46:16.551 INFO:teuthology.orchestra.run.smithi029.stdout:inline_data disabled 2024-07-27T06:46:16.551 INFO:teuthology.orchestra.run.smithi029.stdout:balancer 2024-07-27T06:46:16.551 INFO:teuthology.orchestra.run.smithi029.stdout:bal_rank_mask -1 2024-07-27T06:46:16.551 INFO:teuthology.orchestra.run.smithi029.stdout:standby_count_wanted 0 2024-07-27T06:46:16.551 INFO:teuthology.orchestra.run.smithi029.stdout:qdb_cluster leader: 24670 members: 24613,24617,24646,24658,24670 2024-07-27T06:46:16.551 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.i{0:24670} state up:active seq 9 addr [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] compat {c=[1],r=[1],i=[1fff]}] 2024-07-27T06:46:16.551 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.h{0:24584} state up:standby-replay seq 1 addr [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] compat {c=[1],r=[1],i=[1fff]}] 2024-07-27T06:46:16.551 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c{1:24658} state up:active seq 14 addr [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] compat {c=[1],r=[1],i=[1fff]}] 2024-07-27T06:46:16.551 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.e{1:24566} state up:standby-replay seq 1 addr [v2:172.21.15.47:6834/2900591116,v1:172.21.15.47:6835/2900591116] compat {c=[1],r=[1],i=[1fff]}] 2024-07-27T06:46:16.551 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.f{2:24646} state up:active seq 14 addr [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] compat {c=[1],r=[1],i=[1fff]}] 2024-07-27T06:46:16.552 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.k{2:24542} state up:standby-replay seq 1 addr [v2:172.21.15.47:6832/2402830639,v1:172.21.15.47:6833/2402830639] compat {c=[1],r=[1],i=[1fff]}] 2024-07-27T06:46:16.552 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b{3:24617} state up:active seq 10 addr [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] compat {c=[1],r=[1],i=[1fff]}] 2024-07-27T06:46:16.552 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a{3:14793} state up:standby-replay seq 1 addr [v2:172.21.15.29:6840/2982668187,v1:172.21.15.29:6841/2982668187] compat {c=[1],r=[1],i=[1fff]}] 2024-07-27T06:46:16.552 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.l{4:24613} state up:active seq 19 addr [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] compat {c=[1],r=[1],i=[1fff]}] 2024-07-27T06:46:16.552 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.g{4:14781} state up:standby-replay seq 1 addr [v2:172.21.15.29:6838/1639930501,v1:172.21.15.29:6839/1639930501] compat {c=[1],r=[1],i=[1fff]}] 2024-07-27T06:46:16.552 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:16.552 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:16.552 INFO:teuthology.orchestra.run.smithi029.stdout:Standby daemons: 2024-07-27T06:46:16.552 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:16.552 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.j{-1:14733} state up:standby seq 1 addr [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] compat {c=[1],r=[1],i=[1fff]}] 2024-07-27T06:46:16.552 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.d{-1:14739} state up:standby seq 1 addr [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] compat {c=[1],r=[1],i=[1fff]}] 2024-07-27T06:46:16.553 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-07-27T06:46:16.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.549+0000 7f1b9945e640 1 -- 172.21.15.29:0/1698821418 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1b7c0b3c20 msgr2=0x7f1b7c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:16.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.549+0000 7f1b9945e640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1b7c0b3c20 0x7f1b7c0b6130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1b84005fd0 tx=0x7f1b84005850 comp rx=0 tx=0).stop 2024-07-27T06:46:16.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.549+0000 7f1b9945e640 1 -- 172.21.15.29:0/1698821418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94108ed0 msgr2=0x7f1b94075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:16.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.549+0000 7f1b9945e640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94108ed0 0x7f1b94075680 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800c8c0 tx=0x7f1b8800cd80 comp rx=0 tx=0).stop 2024-07-27T06:46:16.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.549+0000 7f1b9945e640 1 -- 172.21.15.29:0/1698821418 shutdown_connections 2024-07-27T06:46:16.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.549+0000 7f1b9945e640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1b7c0b3c20 0x7f1b7c0b6130 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:16.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.549+0000 7f1b9945e640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b94109820 0x7f1b94075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:16.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.549+0000 7f1b9945e640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b94108ed0 0x7f1b94075680 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:16.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.549+0000 7f1b9945e640 1 --2- 172.21.15.29:0/1698821418 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b94103590 0x7f1b94075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:16.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.550+0000 7f1b9945e640 1 -- 172.21.15.29:0/1698821418 >> 172.21.15.29:0/1698821418 conn(0x7f1b9407b280 msgr2=0x7f1b94104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:16.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.550+0000 7f1b9945e640 1 -- 172.21.15.29:0/1698821418 shutdown_connections 2024-07-27T06:46:16.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:16.550+0000 7f1b9945e640 1 -- 172.21.15.29:0/1698821418 wait complete. 2024-07-27T06:46:17.285 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- bash -c 'ceph osd dump' 2024-07-27T06:46:17.561 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:17.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:17 smithi029 ceph-mon[32610]: pgmap v375: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:17.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:17 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1698821418' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-27T06:46:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:17 smithi047 ceph-mon[36835]: pgmap v375: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:17 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1698821418' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-27T06:46:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:17 smithi077 ceph-mon[37153]: pgmap v375: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:17 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1698821418' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-27T06:46:19.439 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:19 smithi029 ceph-mon[32610]: pgmap v376: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.437+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/1387436605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc4107dd0 msgr2=0x7fbbc41081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.437+0000 7fbbcc5d2640 1 --2- 172.21.15.29:0/1387436605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc4107dd0 0x7fbbc41081e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009900 tx=0x7fbbb402a410 comp rx=0 tx=0).stop 2024-07-27T06:46:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.437+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/1387436605 shutdown_connections 2024-07-27T06:46:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.437+0000 7fbbcc5d2640 1 --2- 172.21.15.29:0/1387436605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc4108720 0x7fbbc410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.437+0000 7fbbcc5d2640 1 --2- 172.21.15.29:0/1387436605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc4107dd0 0x7fbbc41081e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.437+0000 7fbbcc5d2640 1 --2- 172.21.15.29:0/1387436605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbc4102490 0x7fbbc41028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.437+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/1387436605 >> 172.21.15.29:0/1387436605 conn(0x7fbbc4079160 msgr2=0x7fbbc4079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.437+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/1387436605 shutdown_connections 2024-07-27T06:46:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.437+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/1387436605 wait complete. 2024-07-27T06:46:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.438+0000 7fbbcc5d2640 1 Processor -- start 2024-07-27T06:46:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.438+0000 7fbbcc5d2640 1 -- start start 2024-07-27T06:46:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.438+0000 7fbbcc5d2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc4102490 0x7fbbc4199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.438+0000 7fbbcc5d2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc4107dd0 0x7fbbc419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.438+0000 7fbbcc5d2640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbc4108720 0x7fbbc419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.438+0000 7fbbcc5d2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc41134a0 con 0x7fbbc4107dd0 2024-07-27T06:46:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.438+0000 7fbbcc5d2640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc4113320 con 0x7fbbc4108720 2024-07-27T06:46:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.438+0000 7fbbcc5d2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc4113620 con 0x7fbbc4102490 2024-07-27T06:46:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.438+0000 7fbbc9b6d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc4107dd0 0x7fbbc419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.438+0000 7fbbc9b6d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc4107dd0 0x7fbbc419a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50662/0 (socket says 172.21.15.29:50662) 2024-07-27T06:46:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.438+0000 7fbbc9b6d640 1 -- 172.21.15.29:0/837851119 learned_addr learned my addr 172.21.15.29:0/837851119 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.438+0000 7fbbca36e640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc4102490 0x7fbbc4199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.439+0000 7fbbcab6f640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbc4108720 0x7fbbc419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.439+0000 7fbbc9b6d640 1 -- 172.21.15.29:0/837851119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc4102490 msgr2=0x7fbbc4199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.439+0000 7fbbc9b6d640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc4102490 0x7fbbc4199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.439+0000 7fbbc9b6d640 1 -- 172.21.15.29:0/837851119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbc4108720 msgr2=0x7fbbc419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.439+0000 7fbbc9b6d640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbc4108720 0x7fbbc419f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.439+0000 7fbbc9b6d640 1 -- 172.21.15.29:0/837851119 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc419fe80 con 0x7fbbc4107dd0 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.439+0000 7fbbc9b6d640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc4107dd0 0x7fbbc419a4b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4006010 tx=0x7fbbb4004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.439+0000 7fbbbb7fe640 1 -- 172.21.15.29:0/837851119 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb403f030 con 0x7fbbc4107dd0 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.439+0000 7fbbbb7fe640 1 -- 172.21.15.29:0/837851119 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbb4003aa0 con 0x7fbbc4107dd0 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.439+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/837851119 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc41a0110 con 0x7fbbc4107dd0 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.439+0000 7fbbbb7fe640 1 -- 172.21.15.29:0/837851119 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4035070 con 0x7fbbc4107dd0 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.439+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/837851119 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc41a68d0 con 0x7fbbc4107dd0 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.440+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/837851119 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbc4103710 con 0x7fbbc4107dd0 2024-07-27T06:46:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.442+0000 7fbbbb7fe640 1 -- 172.21.15.29:0/837851119 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fbbb4042030 con 0x7fbbc4107dd0 2024-07-27T06:46:19.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.443+0000 7fbbbb7fe640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbbb00b3c20 0x7fbbb00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:19.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.443+0000 7fbbca36e640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbbb00b3c20 0x7fbbb00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:19.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.443+0000 7fbbbb7fe640 1 -- 172.21.15.29:0/837851119 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fbbb41045f0 con 0x7fbbc4107dd0 2024-07-27T06:46:19.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.443+0000 7fbbca36e640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbbb00b3c20 0x7fbbb00b6130 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000bbe0 tx=0x7fbbc000b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:19.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.446+0000 7fbbbb7fe640 1 -- 172.21.15.29:0/837851119 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbbb40cbb30 con 0x7fbbc4107dd0 2024-07-27T06:46:19.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.629+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/837851119 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fbbc410a510 con 0x7fbbc4107dd0 2024-07-27T06:46:19.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.630+0000 7fbbbb7fe640 1 -- 172.21.15.29:0/837851119 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v83) ==== 56+0+5705 (secure 0 0 0) 0x7fbbb40cb430 con 0x7fbbc4107dd0 2024-07-27T06:46:19.632 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 83 2024-07-27T06:46:19.632 INFO:teuthology.orchestra.run.smithi029.stdout:fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 2024-07-27T06:46:19.632 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-07-27T06:33:57.535327+0000 2024-07-27T06:46:19.632 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-07-27T06:45:09.477077+0000 2024-07-27T06:46:19.632 INFO:teuthology.orchestra.run.smithi029.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-07-27T06:46:19.632 INFO:teuthology.orchestra.run.smithi029.stdout:crush_version 26 2024-07-27T06:46:19.633 INFO:teuthology.orchestra.run.smithi029.stdout:full_ratio 0.95 2024-07-27T06:46:19.633 INFO:teuthology.orchestra.run.smithi029.stdout:backfillfull_ratio 0.9 2024-07-27T06:46:19.633 INFO:teuthology.orchestra.run.smithi029.stdout:nearfull_ratio 0.85 2024-07-27T06:46:19.633 INFO:teuthology.orchestra.run.smithi029.stdout:require_min_compat_client luminous 2024-07-27T06:46:19.633 INFO:teuthology.orchestra.run.smithi029.stdout:min_compat_client luminous 2024-07-27T06:46:19.633 INFO:teuthology.orchestra.run.smithi029.stdout:require_osd_release squid 2024-07-27T06:46:19.633 INFO:teuthology.orchestra.run.smithi029.stdout:stretch_mode_enabled false 2024-07-27T06:46:19.633 INFO:teuthology.orchestra.run.smithi029.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-07-27T06:46:19.633 INFO:teuthology.orchestra.run.smithi029.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-07-27T06:46:19.633 INFO:teuthology.orchestra.run.smithi029.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-07-27T06:46:19.633 INFO:teuthology.orchestra.run.smithi029.stdout:max_osd 12 2024-07-27T06:46:19.633 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 up in weight 1 up_from 9 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6802/386121641,v1:172.21.15.29:6803/386121641] [v2:172.21.15.29:6804/386121641,v1:172.21.15.29:6805/386121641] exists,up 06833d24-2e1a-4199-9338-7d6883d1bf9a 2024-07-27T06:46:19.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 up in weight 1 up_from 14 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.47:6800/2795953749,v1:172.21.15.47:6801/2795953749] [v2:172.21.15.47:6802/2795953749,v1:172.21.15.47:6803/2795953749] exists,up 906aba19-80dc-41b5-953c-af6dca604b05 2024-07-27T06:46:19.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 up in weight 1 up_from 19 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6800/2202565078,v1:172.21.15.77:6801/2202565078] [v2:172.21.15.77:6802/2202565078,v1:172.21.15.77:6803/2202565078] exists,up 194ae5a5-aebf-4cff-9b99-585a3073a269 2024-07-27T06:46:19.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 up in weight 1 up_from 26 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6810/3757476312,v1:172.21.15.29:6811/3757476312] [v2:172.21.15.29:6812/3757476312,v1:172.21.15.29:6813/3757476312] exists,up 3c22ce9c-963e-4f72-86e5-360e781526db 2024-07-27T06:46:19.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 up in weight 1 up_from 32 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.47:6808/1379363720,v1:172.21.15.47:6809/1379363720] [v2:172.21.15.47:6810/1379363720,v1:172.21.15.47:6811/1379363720] exists,up cd4b585f-e7a0-4d82-873d-3a9269ebac3d 2024-07-27T06:46:19.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 up in weight 1 up_from 37 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6808/1338438780,v1:172.21.15.77:6809/1338438780] [v2:172.21.15.77:6810/1338438780,v1:172.21.15.77:6811/1338438780] exists,up 40a3eb14-dfaa-4523-8c28-d74c19293c8a 2024-07-27T06:46:19.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 up in weight 1 up_from 43 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6818/2679402148,v1:172.21.15.29:6819/2679402148] [v2:172.21.15.29:6820/2679402148,v1:172.21.15.29:6821/2679402148] exists,up 91e85be2-10d1-4209-aa33-17da8f8bd8bd 2024-07-27T06:46:19.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 up in weight 1 up_from 48 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.47:6816/2439195589,v1:172.21.15.47:6817/2439195589] [v2:172.21.15.47:6818/2439195589,v1:172.21.15.47:6819/2439195589] exists,up c655b450-0cc4-45d4-95ea-93a57fe8e4ff 2024-07-27T06:46:19.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8 up in weight 1 up_from 54 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6816/1422290905,v1:172.21.15.77:6817/1422290905] [v2:172.21.15.77:6818/1422290905,v1:172.21.15.77:6819/1422290905] exists,up a7a95b51-f32c-433e-a197-feb6d47b6c71 2024-07-27T06:46:19.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9 up in weight 1 up_from 60 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.29:6826/1307316384,v1:172.21.15.29:6827/1307316384] [v2:172.21.15.29:6828/1307316384,v1:172.21.15.29:6829/1307316384] exists,up f352242b-0d2e-41b3-9c83-436cf432b0df 2024-07-27T06:46:19.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10 up in weight 1 up_from 65 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.47:6824/2133726850,v1:172.21.15.47:6825/2133726850] [v2:172.21.15.47:6826/2133726850,v1:172.21.15.47:6827/2133726850] exists,up 4148c2df-c3e9-4a47-88f7-36cf3eab8d99 2024-07-27T06:46:19.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11 up in weight 1 up_from 71 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.77:6824/3228299327,v1:172.21.15.77:6825/3228299327] [v2:172.21.15.77:6826/3228299327,v1:172.21.15.77:6827/3228299327] exists,up 5e8b1da3-5f65-4fe9-8722-b35921cbd8c9 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.a [2,11] 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.b [2,1] 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.c [2,4] 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 2.1d [8,11] 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.0 [4,7,5,9] 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.1b [4,6] 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.1d [4,8,5,9] 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.21 [4,6] 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.22 [3,9] 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 3.2a [3,8] 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/2378279024 expires 2024-07-28T06:36:07.527503+0000 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6801/3219967326 expires 2024-07-28T06:34:34.443023+0000 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6800/3219967326 expires 2024-07-28T06:34:34.443023+0000 2024-07-27T06:46:19.635 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/3527133965 expires 2024-07-28T06:36:07.527503+0000 2024-07-27T06:46:19.636 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/2262661620 expires 2024-07-28T06:34:34.443023+0000 2024-07-27T06:46:19.636 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6800/2720999695 expires 2024-07-28T06:34:58.250198+0000 2024-07-27T06:46:19.636 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/3156863587 expires 2024-07-28T06:36:07.527503+0000 2024-07-27T06:46:19.636 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/2424241521 expires 2024-07-28T06:34:34.443023+0000 2024-07-27T06:46:19.636 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/3881857930 expires 2024-07-28T06:34:58.250198+0000 2024-07-27T06:46:19.636 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/398508875 expires 2024-07-28T06:34:34.443023+0000 2024-07-27T06:46:19.636 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6801/2720999695 expires 2024-07-28T06:34:58.250198+0000 2024-07-27T06:46:19.636 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6800/2384940058 expires 2024-07-28T06:36:07.527503+0000 2024-07-27T06:46:19.636 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/2679856281 expires 2024-07-28T06:34:58.250198+0000 2024-07-27T06:46:19.636 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6801/2384940058 expires 2024-07-28T06:36:07.527503+0000 2024-07-27T06:46:19.636 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/2708687265 expires 2024-07-28T06:34:58.250198+0000 2024-07-27T06:46:19.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.632+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/837851119 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbbb00b3c20 msgr2=0x7fbbb00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:19.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.632+0000 7fbbcc5d2640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbbb00b3c20 0x7fbbb00b6130 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000bbe0 tx=0x7fbbc000b390 comp rx=0 tx=0).stop 2024-07-27T06:46:19.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.632+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/837851119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc4107dd0 msgr2=0x7fbbc419a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:19.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.632+0000 7fbbcc5d2640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc4107dd0 0x7fbbc419a4b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4006010 tx=0x7fbbb4004060 comp rx=0 tx=0).stop 2024-07-27T06:46:19.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.632+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/837851119 shutdown_connections 2024-07-27T06:46:19.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.632+0000 7fbbcc5d2640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbbc4108720 0x7fbbc419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:19.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.632+0000 7fbbcc5d2640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc4107dd0 0x7fbbc419a4b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:19.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.632+0000 7fbbcc5d2640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbbb00b3c20 0x7fbbb00b6130 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:19.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.632+0000 7fbbcc5d2640 1 --2- 172.21.15.29:0/837851119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc4102490 0x7fbbc4199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:19.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.632+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/837851119 >> 172.21.15.29:0/837851119 conn(0x7fbbc4079160 msgr2=0x7fbbc4103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:19.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.633+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/837851119 shutdown_connections 2024-07-27T06:46:19.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:19.633+0000 7fbbcc5d2640 1 -- 172.21.15.29:0/837851119 wait complete. 2024-07-27T06:46:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:19 smithi047 ceph-mon[36835]: pgmap v376: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:19 smithi077 ceph-mon[37153]: pgmap v376: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:20.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:20 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/837851119' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-27T06:46:20.472 INFO:teuthology.run_tasks:Running task fs.ready... 2024-07-27T06:46:20.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:46:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:20 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/837851119' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-27T06:46:20.759 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:20 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/837851119' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-27T06:46:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:21 smithi029 ceph-mon[32610]: pgmap v377: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:21 smithi047 ceph-mon[36835]: pgmap v377: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:21 smithi077 ceph-mon[37153]: pgmap v377: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:22.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.456+0000 7f64043d4640 1 -- 172.21.15.29:0/3354853733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63fc108ed0 msgr2=0x7f63fc1092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:22.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.456+0000 7f64043d4640 1 --2- 172.21.15.29:0/3354853733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63fc108ed0 0x7f63fc1092e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009900 tx=0x7f63ec02a410 comp rx=0 tx=0).stop 2024-07-27T06:46:22.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.456+0000 7f64043d4640 1 -- 172.21.15.29:0/3354853733 shutdown_connections 2024-07-27T06:46:22.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.456+0000 7f64043d4640 1 --2- 172.21.15.29:0/3354853733 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63fc109820 0x7f63fc110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:22.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.456+0000 7f64043d4640 1 --2- 172.21.15.29:0/3354853733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63fc108ed0 0x7f63fc1092e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:22.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.456+0000 7f64043d4640 1 --2- 172.21.15.29:0/3354853733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc103590 0x7f63fc1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:22.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.456+0000 7f64043d4640 1 -- 172.21.15.29:0/3354853733 >> 172.21.15.29:0/3354853733 conn(0x7f63fc07b280 msgr2=0x7f63fc07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:22.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.456+0000 7f64043d4640 1 -- 172.21.15.29:0/3354853733 shutdown_connections 2024-07-27T06:46:22.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.456+0000 7f64043d4640 1 -- 172.21.15.29:0/3354853733 wait complete. 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.457+0000 7f64043d4640 1 Processor -- start 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.457+0000 7f64043d4640 1 -- start start 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.457+0000 7f64043d4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc103590 0x7f63fc199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.457+0000 7f64043d4640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63fc108ed0 0x7f63fc19a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.457+0000 7f64043d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63fc109820 0x7f63fc19f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.457+0000 7f64043d4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1134b0 con 0x7f63fc109820 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.457+0000 7f64043d4640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc113330 con 0x7f63fc108ed0 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.457+0000 7f64043d4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc113630 con 0x7f63fc103590 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.457+0000 7f6402971640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63fc109820 0x7f63fc19f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.457+0000 7f6402971640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63fc109820 0x7f63fc19f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50680/0 (socket says 172.21.15.29:50680) 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.457+0000 7f6402971640 1 -- 172.21.15.29:0/2267119428 learned_addr learned my addr 172.21.15.29:0/2267119428 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f640196f640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63fc108ed0 0x7f63fc19a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:22.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f6402170640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc103590 0x7f63fc199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f6402971640 1 -- 172.21.15.29:0/2267119428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc103590 msgr2=0x7f63fc199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f6402971640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc103590 0x7f63fc199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f6402971640 1 -- 172.21.15.29:0/2267119428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63fc108ed0 msgr2=0x7f63fc19a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f6402971640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63fc108ed0 0x7f63fc19a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f6402971640 1 -- 172.21.15.29:0/2267119428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63fc19feb0 con 0x7f63fc109820 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f6402971640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63fc109820 0x7f63fc19f780 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f63f400eaa0 tx=0x7f63f400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f63f37fe640 1 -- 172.21.15.29:0/2267119428 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f400cdb0 con 0x7f63fc109820 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f63f37fe640 1 -- 172.21.15.29:0/2267119428 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63f4005020 con 0x7f63fc109820 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f64043d4640 1 -- 172.21.15.29:0/2267119428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63fc1a01a0 con 0x7f63fc109820 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f63f37fe640 1 -- 172.21.15.29:0/2267119428 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f40116d0 con 0x7f63fc109820 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.458+0000 7f64043d4640 1 -- 172.21.15.29:0/2267119428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63fc1a6900 con 0x7f63fc109820 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.460+0000 7f64043d4640 1 -- 172.21.15.29:0/2267119428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63fc1039a0 con 0x7f63fc109820 2024-07-27T06:46:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.461+0000 7f63f37fe640 1 -- 172.21.15.29:0/2267119428 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f63f40051d0 con 0x7f63fc109820 2024-07-27T06:46:22.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.462+0000 7f63f37fe640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63e80b3ca0 0x7f63e80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:22.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.462+0000 7f6402170640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63e80b3ca0 0x7f63e80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:22.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.462+0000 7f63f37fe640 1 -- 172.21.15.29:0/2267119428 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f63f40e3df0 con 0x7f63fc109820 2024-07-27T06:46:22.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.462+0000 7f6402170640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63e80b3ca0 0x7f63e80b61b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f63f8005fd0 tx=0x7f63f800b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:22.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.466+0000 7f63f37fe640 1 -- 172.21.15.29:0/2267119428 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f63f4016030 con 0x7f63fc109820 2024-07-27T06:46:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.686+0000 7f64043d4640 1 -- 172.21.15.29:0/2267119428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63fc10b660 con 0x7f63fc109820 2024-07-27T06:46:22.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.687+0000 7f63f37fe640 1 -- 172.21.15.29:0/2267119428 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11957 (secure 0 0 0) 0x7f63fc10b660 con 0x7f63fc109820 2024-07-27T06:46:22.689 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:22.689 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"btime":"2024-07-27T06:43:37:875457+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:43:37.017169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24613,24617,24646,24658,24670]},"id":1}]} 2024-07-27T06:46:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-07-27T06:46:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.689+0000 7f64043d4640 1 -- 172.21.15.29:0/2267119428 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63e80b3ca0 msgr2=0x7f63e80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.689+0000 7f64043d4640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63e80b3ca0 0x7f63e80b61b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f63f8005fd0 tx=0x7f63f800b390 comp rx=0 tx=0).stop 2024-07-27T06:46:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.689+0000 7f64043d4640 1 -- 172.21.15.29:0/2267119428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63fc109820 msgr2=0x7f63fc19f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.689+0000 7f64043d4640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63fc109820 0x7f63fc19f780 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f63f400eaa0 tx=0x7f63f400ef60 comp rx=0 tx=0).stop 2024-07-27T06:46:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.689+0000 7f64043d4640 1 -- 172.21.15.29:0/2267119428 shutdown_connections 2024-07-27T06:46:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.689+0000 7f64043d4640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63fc109820 0x7f63fc19f780 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.689+0000 7f64043d4640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63fc108ed0 0x7f63fc19a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.689+0000 7f64043d4640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63e80b3ca0 0x7f63e80b61b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.689+0000 7f64043d4640 1 --2- 172.21.15.29:0/2267119428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63fc103590 0x7f63fc199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.689+0000 7f64043d4640 1 -- 172.21.15.29:0/2267119428 >> 172.21.15.29:0/2267119428 conn(0x7f63fc07b280 msgr2=0x7f63fc104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.689+0000 7f64043d4640 1 -- 172.21.15.29:0/2267119428 shutdown_connections 2024-07-27T06:46:22.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:22.689+0000 7f64043d4640 1 -- 172.21.15.29:0/2267119428 wait complete. 2024-07-27T06:46:23.373 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:46:23.640 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:23 smithi047 ceph-mon[36835]: pgmap v378: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:23 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2267119428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:46:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:23 smithi029 ceph-mon[32610]: pgmap v378: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:23 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2267119428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:46:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:23 smithi077 ceph-mon[37153]: pgmap v378: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:23 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2267119428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:46:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.415+0000 7f596e723640 1 -- 172.21.15.29:0/3493841366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f596810fbc0 msgr2=0x7f5968111fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.415+0000 7f596e723640 1 --2- 172.21.15.29:0/3493841366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f596810fbc0 0x7f5968111fe0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5960009ca0 tx=0x7f596002a950 comp rx=0 tx=0).stop 2024-07-27T06:46:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.415+0000 7f596e723640 1 -- 172.21.15.29:0/3493841366 shutdown_connections 2024-07-27T06:46:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.415+0000 7f596e723640 1 --2- 172.21.15.29:0/3493841366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f596810fbc0 0x7f5968111fe0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.415+0000 7f596e723640 1 --2- 172.21.15.29:0/3493841366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5968102960 0x7f596810f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.415+0000 7f596e723640 1 --2- 172.21.15.29:0/3493841366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5968102010 0x7f5968102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.415+0000 7f596e723640 1 -- 172.21.15.29:0/3493841366 >> 172.21.15.29:0/3493841366 conn(0x7f5968079160 msgr2=0x7f5968079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:25.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.415+0000 7f596e723640 1 -- 172.21.15.29:0/3493841366 shutdown_connections 2024-07-27T06:46:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.415+0000 7f596e723640 1 -- 172.21.15.29:0/3493841366 wait complete. 2024-07-27T06:46:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f596e723640 1 Processor -- start 2024-07-27T06:46:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f596e723640 1 -- start start 2024-07-27T06:46:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f596e723640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5968102010 0x7f596819c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f596e723640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5968102960 0x7f596819c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f596e723640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f596810fbc0 0x7f59681a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f596e723640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5968115670 con 0x7f596810fbc0 2024-07-27T06:46:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f596e723640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f59681154f0 con 0x7f5968102960 2024-07-27T06:46:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f596e723640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f59681157f0 con 0x7f5968102010 2024-07-27T06:46:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f596ccc0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f596810fbc0 0x7f59681a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:25.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f596ccc0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f596810fbc0 0x7f59681a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50696/0 (socket says 172.21.15.29:50696) 2024-07-27T06:46:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f596ccc0640 1 -- 172.21.15.29:0/3695143285 learned_addr learned my addr 172.21.15.29:0/3695143285 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f595ffff640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5968102960 0x7f596819c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.416+0000 7f5967fff640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5968102010 0x7f596819c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.417+0000 7f596ccc0640 1 -- 172.21.15.29:0/3695143285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5968102010 msgr2=0x7f596819c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.417+0000 7f596ccc0640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5968102010 0x7f596819c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.417+0000 7f596ccc0640 1 -- 172.21.15.29:0/3695143285 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5968102960 msgr2=0x7f596819c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.417+0000 7f596ccc0640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5968102960 0x7f596819c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.417+0000 7f596ccc0640 1 -- 172.21.15.29:0/3695143285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59681a1f70 con 0x7f596810fbc0 2024-07-27T06:46:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.417+0000 7f596ccc0640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f596810fbc0 0x7f59681a1840 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5960004af0 tx=0x7f596002f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.417+0000 7f5965ffb640 1 -- 172.21.15.29:0/3695143285 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5960034040 con 0x7f596810fbc0 2024-07-27T06:46:25.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.417+0000 7f5965ffb640 1 -- 172.21.15.29:0/3695143285 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f596002fe80 con 0x7f596810fbc0 2024-07-27T06:46:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.417+0000 7f596e723640 1 -- 172.21.15.29:0/3695143285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59681a2200 con 0x7f596810fbc0 2024-07-27T06:46:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.417+0000 7f5965ffb640 1 -- 172.21.15.29:0/3695143285 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59600046a0 con 0x7f596810fbc0 2024-07-27T06:46:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.417+0000 7f596e723640 1 -- 172.21.15.29:0/3695143285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59681a89c0 con 0x7f596810fbc0 2024-07-27T06:46:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.418+0000 7f596e723640 1 -- 172.21.15.29:0/3695143285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f596806bc10 con 0x7f596810fbc0 2024-07-27T06:46:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.421+0000 7f5965ffb640 1 -- 172.21.15.29:0/3695143285 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f5960037020 con 0x7f596810fbc0 2024-07-27T06:46:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.421+0000 7f5965ffb640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f59540b3c20 0x7f59540b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.422+0000 7f5967fff640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f59540b3c20 0x7f59540b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.422+0000 7f5965ffb640 1 -- 172.21.15.29:0/3695143285 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f5960104c30 con 0x7f596810fbc0 2024-07-27T06:46:25.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.422+0000 7f5967fff640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f59540b3c20 0x7f59540b6130 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5958009bd0 tx=0x7f5958005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:25.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.425+0000 7f5965ffb640 1 -- 172.21.15.29:0/3695143285 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f59600cc170 con 0x7f596810fbc0 2024-07-27T06:46:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:25 smithi029 ceph-mon[32610]: pgmap v379: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:46:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.646+0000 7f596e723640 1 -- 172.21.15.29:0/3695143285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5968102420 con 0x7f596810fbc0 2024-07-27T06:46:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.647+0000 7f5965ffb640 1 -- 172.21.15.29:0/3695143285 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11957 (secure 0 0 0) 0x7f5968102420 con 0x7f596810fbc0 2024-07-27T06:46:25.648 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:25.649 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"btime":"2024-07-27T06:43:37:875457+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:43:37.017169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24613,24617,24646,24658,24670]},"id":1}]} 2024-07-27T06:46:25.649 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-07-27T06:46:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.650+0000 7f596e723640 1 -- 172.21.15.29:0/3695143285 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f59540b3c20 msgr2=0x7f59540b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.650+0000 7f596e723640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f59540b3c20 0x7f59540b6130 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5958009bd0 tx=0x7f5958005f90 comp rx=0 tx=0).stop 2024-07-27T06:46:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.650+0000 7f596e723640 1 -- 172.21.15.29:0/3695143285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f596810fbc0 msgr2=0x7f59681a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.650+0000 7f596e723640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f596810fbc0 0x7f59681a1840 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5960004af0 tx=0x7f596002f5d0 comp rx=0 tx=0).stop 2024-07-27T06:46:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.650+0000 7f596e723640 1 -- 172.21.15.29:0/3695143285 shutdown_connections 2024-07-27T06:46:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.650+0000 7f596e723640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f596810fbc0 0x7f59681a1840 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.650+0000 7f596e723640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5968102960 0x7f596819c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.650+0000 7f596e723640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f59540b3c20 0x7f59540b6130 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.650+0000 7f596e723640 1 --2- 172.21.15.29:0/3695143285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5968102010 0x7f596819c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.650+0000 7f596e723640 1 -- 172.21.15.29:0/3695143285 >> 172.21.15.29:0/3695143285 conn(0x7f5968079160 msgr2=0x7f59681096f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.650+0000 7f596e723640 1 -- 172.21.15.29:0/3695143285 shutdown_connections 2024-07-27T06:46:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:25.650+0000 7f596e723640 1 -- 172.21.15.29:0/3695143285 wait complete. 2024-07-27T06:46:25.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:25 smithi047 ceph-mon[36835]: pgmap v379: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:25.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:46:25.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:25.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:25.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:25.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:25 smithi077 ceph-mon[37153]: pgmap v379: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:46:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:26.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd dump --format=json 2024-07-27T06:46:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:26 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3695143285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:46:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:46:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:46:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:26 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3695143285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:46:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:46:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:46:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:26 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3695143285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:46:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:46:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:46:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:46:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:27 smithi047 ceph-mon[36835]: pgmap v380: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:27 smithi029 ceph-mon[32610]: pgmap v380: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:27 smithi077 ceph-mon[37153]: pgmap v380: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.372+0000 7f70aeacb640 1 -- 172.21.15.29:0/2910271000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a8103590 msgr2=0x7f70a81039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.372+0000 7f70aeacb640 1 --2- 172.21.15.29:0/2910271000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a8103590 0x7f70a81039a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f709c009900 tx=0x7f709c02a410 comp rx=0 tx=0).stop 2024-07-27T06:46:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.373+0000 7f70aeacb640 1 -- 172.21.15.29:0/2910271000 shutdown_connections 2024-07-27T06:46:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.373+0000 7f70aeacb640 1 --2- 172.21.15.29:0/2910271000 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70a8109820 0x7f70a8110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.373+0000 7f70aeacb640 1 --2- 172.21.15.29:0/2910271000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70a8108ed0 0x7f70a81092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.373+0000 7f70aeacb640 1 --2- 172.21.15.29:0/2910271000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a8103590 0x7f70a81039a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.373+0000 7f70aeacb640 1 -- 172.21.15.29:0/2910271000 >> 172.21.15.29:0/2910271000 conn(0x7f70a807b280 msgr2=0x7f70a807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.373+0000 7f70aeacb640 1 -- 172.21.15.29:0/2910271000 shutdown_connections 2024-07-27T06:46:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.373+0000 7f70aeacb640 1 -- 172.21.15.29:0/2910271000 wait complete. 2024-07-27T06:46:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70aeacb640 1 Processor -- start 2024-07-27T06:46:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70aeacb640 1 -- start start 2024-07-27T06:46:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70aeacb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70a8103590 0x7f70a8075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70aeacb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a8108ed0 0x7f70a8075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70aeacb640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70a8109820 0x7f70a8075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70aeacb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70a81134d0 con 0x7f70a8108ed0 2024-07-27T06:46:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70aeacb640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f70a8113350 con 0x7f70a8109820 2024-07-27T06:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70aeacb640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f70a8113650 con 0x7f70a8103590 2024-07-27T06:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a8108ed0 0x7f70a8075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70ac867640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70a8103590 0x7f70a8075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a8108ed0 0x7f70a8075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50716/0 (socket says 172.21.15.29:50716) 2024-07-27T06:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70a7fff640 1 -- 172.21.15.29:0/158502172 learned_addr learned my addr 172.21.15.29:0/158502172 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.374+0000 7f70ad068640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70a8109820 0x7f70a8075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.375+0000 7f70a7fff640 1 -- 172.21.15.29:0/158502172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70a8103590 msgr2=0x7f70a8075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.375+0000 7f70a7fff640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70a8103590 0x7f70a8075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.375+0000 7f70a7fff640 1 -- 172.21.15.29:0/158502172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70a8109820 msgr2=0x7f70a8075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.375+0000 7f70a7fff640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70a8109820 0x7f70a8075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.375+0000 7f70a7fff640 1 -- 172.21.15.29:0/158502172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a81a8180 con 0x7f70a8108ed0 2024-07-27T06:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.375+0000 7f70a7fff640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a8108ed0 0x7f70a8075680 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f709400c970 tx=0x7f709400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.375+0000 7f70a5ffb640 1 -- 172.21.15.29:0/158502172 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094007bb0 con 0x7f70a8108ed0 2024-07-27T06:46:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.375+0000 7f70aeacb640 1 -- 172.21.15.29:0/158502172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a81a8470 con 0x7f70a8108ed0 2024-07-27T06:46:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.375+0000 7f70a5ffb640 1 -- 172.21.15.29:0/158502172 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7094007d50 con 0x7f70a8108ed0 2024-07-27T06:46:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.375+0000 7f70a5ffb640 1 -- 172.21.15.29:0/158502172 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094004e10 con 0x7f70a8108ed0 2024-07-27T06:46:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.375+0000 7f70aeacb640 1 -- 172.21.15.29:0/158502172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a81a89b0 con 0x7f70a8108ed0 2024-07-27T06:46:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.377+0000 7f70aeacb640 1 -- 172.21.15.29:0/158502172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a81039a0 con 0x7f70a8108ed0 2024-07-27T06:46:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.378+0000 7f70a5ffb640 1 -- 172.21.15.29:0/158502172 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f7094005000 con 0x7f70a8108ed0 2024-07-27T06:46:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.379+0000 7f70a5ffb640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f70980b3c20 0x7f70980b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.379+0000 7f70ac867640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f70980b3c20 0x7f70980b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.379+0000 7f70a5ffb640 1 -- 172.21.15.29:0/158502172 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f70940e3f70 con 0x7f70a8108ed0 2024-07-27T06:46:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.379+0000 7f70ac867640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f70980b3c20 0x7f70980b6130 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f709c000c00 tx=0x7f709c031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.382+0000 7f70a5ffb640 1 -- 172.21.15.29:0/158502172 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7094014030 con 0x7f70a8108ed0 2024-07-27T06:46:28.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.566+0000 7f70aeacb640 1 -- 172.21.15.29:0/158502172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70a810b010 con 0x7f70a8108ed0 2024-07-27T06:46:28.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.568+0000 7f70a5ffb640 1 -- 172.21.15.29:0/158502172 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+23026 (secure 0 0 0) 0x7f70940ab450 con 0x7f70a8108ed0 2024-07-27T06:46:28.569 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:28.570 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","created":"2024-07-27T06:33:57.535327+0000","modified":"2024-07-27T06:45:09.477077+0000","last_up_change":"2024-07-27T06:42:05.757156+0000","last_in_change":"2024-07-27T06:41:46.423732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T06:38:21.606527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-27T06:43:07.755490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-27T06:43:11.658694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06833d24-2e1a-4199-9338-7d6883d1bf9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6803","nonce":386121641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6805","nonce":386121641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6809","nonce":386121641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6807","nonce":386121641}]},"public_addr":"172.21.15.29:6803/386121641","cluster_addr":"172.21.15.29:6805/386121641","heartbeat_back_addr":"172.21.15.29:6809/386121641","heartbeat_front_addr":"172.21.15.29:6807/386121641","state":["exists","up"]},{"osd":1,"uuid":"906aba19-80dc-41b5-953c-af6dca604b05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6801","nonce":2795953749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6803","nonce":2795953749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6807","nonce":2795953749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6805","nonce":2795953749}]},"public_addr":"172.21.15.47:6801/2795953749","cluster_addr":"172.21.15.47:6803/2795953749","heartbeat_back_addr":"172.21.15.47:6807/2795953749","heartbeat_front_addr":"172.21.15.47:6805/2795953749","state":["exists","up"]},{"osd":2,"uuid":"194ae5a5-aebf-4cff-9b99-585a3073a269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6801","nonce":2202565078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6803","nonce":2202565078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6807","nonce":2202565078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6805","nonce":2202565078}]},"public_addr":"172.21.15.77:6801/2202565078","cluster_addr":"172.21.15.77:6803/2202565078","heartbeat_back_addr":"172.21.15.77:6807/2202565078","heartbeat_front_addr":"172.21.15.77:6805/2202565078","state":["exists","up"]},{"osd":3,"uuid":"3c22ce9c-963e-4f72-86e5-360e781526db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6811","nonce":3757476312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6813","nonce":3757476312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6817","nonce":3757476312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6815","nonce":3757476312}]},"public_addr":"172.21.15.29:6811/3757476312","cluster_addr":"172.21.15.29:6813/3757476312","heartbeat_back_addr":"172.21.15.29:6817/3757476312","heartbeat_front_addr":"172.21.15.29:6815/3757476312","state":["exists","up"]},{"osd":4,"uuid":"cd4b585f-e7a0-4d82-873d-3a9269ebac3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6809","nonce":1379363720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6811","nonce":1379363720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6815","nonce":1379363720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6813","nonce":1379363720}]},"public_addr":"172.21.15.47:6809/1379363720","cluster_addr":"172.21.15.47:6811/1379363720","heartbeat_back_addr":"172.21.15.47:6815/1379363720","heartbeat_front_addr":"172.21.15.47:6813/1379363720","state":["exists","up"]},{"osd":5,"uuid":"40a3eb14-dfaa-4523-8c28-d74c19293c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6809","nonce":1338438780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6811","nonce":1338438780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6815","nonce":1338438780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6813","nonce":1338438780}]},"public_addr":"172.21.15.77:6809/1338438780","cluster_addr":"172.21.15.77:6811/1338438780","heartbeat_back_addr":"172.21.15.77:6815/1338438780","heartbeat_front_addr":"172.21.15.77:6813/1338438780","state":["exists","up"]},{"osd":6,"uuid":"91e85be2-10d1-4209-aa33-17da8f8bd8bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6819","nonce":2679402148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6821","nonce":2679402148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6825","nonce":2679402148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6823","nonce":2679402148}]},"public_addr":"172.21.15.29:6819/2679402148","cluster_addr":"172.21.15.29:6821/2679402148","heartbeat_back_addr":"172.21.15.29:6825/2679402148","heartbeat_front_addr":"172.21.15.29:6823/2679402148","state":["exists","up"]},{"osd":7,"uuid":"c655b450-0cc4-45d4-95ea-93a57fe8e4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6817","nonce":2439195589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6819","nonce":2439195589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6823","nonce":2439195589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6821","nonce":2439195589}]},"public_addr":"172.21.15.47:6817/2439195589","cluster_addr":"172.21.15.47:6819/2439195589","heartbeat_back_addr":"172.21.15.47:6823/2439195589","heartbeat_front_addr":"172.21.15.47:6821/2439195589","state":["exists","up"]},{"osd":8,"uuid":"a7a95b51-f32c-433e-a197-feb6d47b6c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6817","nonce":1422290905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6819","nonce":1422290905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6823","nonce":1422290905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6821","nonce":1422290905}]},"public_addr":"172.21.15.77:6817/1422290905","cluster_addr":"172.21.15.77:6819/1422290905","heartbeat_back_addr":"172.21.15.77:6823/1422290905","heartbeat_front_addr":"172.21.15.77:6821/1422290905","state":["exists","up"]},{"osd":9,"uuid":"f352242b-0d2e-41b3-9c83-436cf432b0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6827","nonce":1307316384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6829","nonce":1307316384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6833","nonce":1307316384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6831","nonce":1307316384}]},"public_addr":"172.21.15.29:6827/1307316384","cluster_addr":"172.21.15.29:6829/1307316384","heartbeat_back_addr":"172.21.15.29:6833/1307316384","heartbeat_front_addr":"172.21.15.29:6831/1307316384","state":["exists","up"]},{"osd":10,"uuid":"4148c2df-c3e9-4a47-88f7-36cf3eab8d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6825","nonce":2133726850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6827","nonce":2133726850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6831","nonce":2133726850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6829","nonce":2133726850}]},"public_addr":"172.21.15.47:6825/2133726850","cluster_addr":"172.21.15.47:6827/2133726850","heartbeat_back_addr":"172.21.15.47:6831/2133726850","heartbeat_front_addr":"172.21.15.47:6829/2133726850","state":["exists","up"]},{"osd":11,"uuid":"5e8b1da3-5f65-4fe9-8722-b35921cbd8c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6825","nonce":3228299327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6827","nonce":3228299327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6831","nonce":3228299327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6829","nonce":3228299327}]},"public_addr":"172.21.15.77:6825/3228299327","cluster_addr":"172.21.15.77:6827/3228299327","heartbeat_back_addr":"172.21.15.77:6831/3228299327","heartbeat_front_addr":"172.21.15.77:6829/3228299327","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:32.525206+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:56.489640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:18.887845+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:43.727678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:08.755553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:32.923461+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:59.298439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:23.944848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:48.883050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:14.566310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:39.557809+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:42:03.813903+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7},{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2378279024":"2024-07-28T06:36:07.527503+0000","172.21.15.29:6801/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3527133965":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2262661620":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/3156863587":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2424241521":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3881857930":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/398508875":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6801/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6800/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2679856281":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6801/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2708687265":"2024-07-28T06:34:58.250198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T06:46:28.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.570+0000 7f70aeacb640 1 -- 172.21.15.29:0/158502172 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f70980b3c20 msgr2=0x7f70980b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.570+0000 7f70aeacb640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f70980b3c20 0x7f70980b6130 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f709c000c00 tx=0x7f709c031400 comp rx=0 tx=0).stop 2024-07-27T06:46:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.570+0000 7f70aeacb640 1 -- 172.21.15.29:0/158502172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a8108ed0 msgr2=0x7f70a8075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.570+0000 7f70aeacb640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a8108ed0 0x7f70a8075680 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f709400c970 tx=0x7f709400ce30 comp rx=0 tx=0).stop 2024-07-27T06:46:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.570+0000 7f70aeacb640 1 -- 172.21.15.29:0/158502172 shutdown_connections 2024-07-27T06:46:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.570+0000 7f70aeacb640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70a8109820 0x7f70a8075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.570+0000 7f70aeacb640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a8108ed0 0x7f70a8075680 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.570+0000 7f70aeacb640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f70980b3c20 0x7f70980b6130 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.570+0000 7f70aeacb640 1 --2- 172.21.15.29:0/158502172 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70a8103590 0x7f70a8075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.570+0000 7f70aeacb640 1 -- 172.21.15.29:0/158502172 >> 172.21.15.29:0/158502172 conn(0x7f70a807b280 msgr2=0x7f70a8106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.570+0000 7f70aeacb640 1 -- 172.21.15.29:0/158502172 shutdown_connections 2024-07-27T06:46:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:28.570+0000 7f70aeacb640 1 -- 172.21.15.29:0/158502172 wait complete. 2024-07-27T06:46:29.276 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-27T06:43:14.801207+0000', 'modified': '2024-07-27T06:43:37.017169+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24670, 'mds_1': 24658, 'mds_2': 24646, 'mds_3': 24617, 'mds_4': 24613}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14781': {'gid': 14781, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6839/1639930501', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6838', 'nonce': 1639930501}, {'type': 'v1', 'addr': '172.21.15.29:6839', 'nonce': 1639930501}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14793': {'gid': 14793, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6841/2982668187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6840', 'nonce': 2982668187}, {'type': 'v1', 'addr': '172.21.15.29:6841', 'nonce': 2982668187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24542': {'gid': 24542, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6833/2402830639', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6832', 'nonce': 2402830639}, {'type': 'v1', 'addr': '172.21.15.47:6833', 'nonce': 2402830639}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24566': {'gid': 24566, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6835/2900591116', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6834', 'nonce': 2900591116}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 2900591116}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24584': {'gid': 24584, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6837/2573307312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6836', 'nonce': 2573307312}, {'type': 'v1', 'addr': '172.21.15.47:6837', 'nonce': 2573307312}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24613': {'gid': 24613, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.77:6833/1085922080', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6832', 'nonce': 1085922080}, {'type': 'v1', 'addr': '172.21.15.77:6833', 'nonce': 1085922080}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24617': {'gid': 24617, 'name': 'b', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.47:6839/670107266', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6838', 'nonce': 670107266}, {'type': 'v1', 'addr': '172.21.15.47:6839', 'nonce': 670107266}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24646': {'gid': 24646, 'name': 'f', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.77:6835/439792826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 439792826}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 439792826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24658': {'gid': 24658, 'name': 'c', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.77:6837/2959628937', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2959628937}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2959628937}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24670': {'gid': 24670, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.77:6839/269899139', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 269899139}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 269899139}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24670, 'qdb_cluster': [24613, 24617, 24646, 24658, 24670]} 2024-07-27T06:46:29.276 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-27T06:46:29.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.l status 2024-07-27T06:46:29.547 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:29.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:29 smithi029 ceph-mon[32610]: pgmap v381: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:29.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:29 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/158502172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:46:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:29 smithi047 ceph-mon[36835]: pgmap v381: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:29 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/158502172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:46:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:29 smithi077 ceph-mon[37153]: pgmap v381: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:29 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/158502172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:46:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.489+0000 7fae6e228640 1 -- 172.21.15.29:0/794487216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae681035e0 msgr2=0x7fae681039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.489+0000 7fae6e228640 1 --2- 172.21.15.29:0/794487216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae681035e0 0x7fae681039f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fae58009900 tx=0x7fae5802a410 comp rx=0 tx=0).stop 2024-07-27T06:46:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.489+0000 7fae6e228640 1 -- 172.21.15.29:0/794487216 shutdown_connections 2024-07-27T06:46:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.489+0000 7fae6e228640 1 --2- 172.21.15.29:0/794487216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae68109870 0x7fae681100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.489+0000 7fae6e228640 1 --2- 172.21.15.29:0/794487216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae68108f20 0x7fae68109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.489+0000 7fae6e228640 1 --2- 172.21.15.29:0/794487216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae681035e0 0x7fae681039f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.489+0000 7fae6e228640 1 -- 172.21.15.29:0/794487216 >> 172.21.15.29:0/794487216 conn(0x7fae680ff030 msgr2=0x7fae68101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.489+0000 7fae6e228640 1 -- 172.21.15.29:0/794487216 shutdown_connections 2024-07-27T06:46:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.489+0000 7fae6e228640 1 -- 172.21.15.29:0/794487216 wait complete. 2024-07-27T06:46:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.489+0000 7fae6e228640 1 Processor -- start 2024-07-27T06:46:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae6e228640 1 -- start start 2024-07-27T06:46:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae6e228640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae681035e0 0x7fae68199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae6e228640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae68108f20 0x7fae6819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae6e228640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae68109870 0x7fae6819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae6e228640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fae68113540 con 0x7fae68109870 2024-07-27T06:46:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae6e228640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fae681133c0 con 0x7fae68108f20 2024-07-27T06:46:31.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae6e228640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fae681136c0 con 0x7fae681035e0 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae67fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae68109870 0x7fae6819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae67fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae68109870 0x7fae6819f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43386/0 (socket says 172.21.15.29:43386) 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae67fff640 1 -- 172.21.15.29:0/507144859 learned_addr learned my addr 172.21.15.29:0/507144859 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae5effd640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae68108f20 0x7fae6819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae677fe640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae681035e0 0x7fae68199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae67fff640 1 -- 172.21.15.29:0/507144859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae681035e0 msgr2=0x7fae68199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.490+0000 7fae67fff640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae681035e0 0x7fae68199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.491+0000 7fae67fff640 1 -- 172.21.15.29:0/507144859 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae68108f20 msgr2=0x7fae6819a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.491+0000 7fae67fff640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae68108f20 0x7fae6819a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.491+0000 7fae67fff640 1 -- 172.21.15.29:0/507144859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6819fe80 con 0x7fae68109870 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.491+0000 7fae67fff640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae68109870 0x7fae6819f750 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fae6000eaa0 tx=0x7fae6000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.491+0000 7fae657fa640 1 -- 172.21.15.29:0/507144859 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6000cdb0 con 0x7fae68109870 2024-07-27T06:46:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.491+0000 7fae657fa640 1 -- 172.21.15.29:0/507144859 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae60005020 con 0x7fae68109870 2024-07-27T06:46:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.491+0000 7fae6e228640 1 -- 172.21.15.29:0/507144859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae681a0170 con 0x7fae68109870 2024-07-27T06:46:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.491+0000 7fae657fa640 1 -- 172.21.15.29:0/507144859 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae600116d0 con 0x7fae68109870 2024-07-27T06:46:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.491+0000 7fae6e228640 1 -- 172.21.15.29:0/507144859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae681a68d0 con 0x7fae68109870 2024-07-27T06:46:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae657fa640 1 -- 172.21.15.29:0/507144859 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fae600051d0 con 0x7fae68109870 2024-07-27T06:46:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae747ac740 1 Processor -- start 2024-07-27T06:46:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae747ac740 1 -- start start 2024-07-27T06:46:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae747ac740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e84bf60 0x55741ebf13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae747ac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 0x55741ebf18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae747ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e82b280 0x55741ebf1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae747ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55741eb67940 con 0x55741e82b280 2024-07-27T06:46:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae747ac740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55741eb56e40 con 0x55741e84bf60 2024-07-27T06:46:31.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae747ac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55741eb5ae80 con 0x55741e8230a0 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae5effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e82b280 0x55741ebf1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae5effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e82b280 0x55741ebf1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43402/0 (socket says 172.21.15.29:43402) 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae5effd640 1 -- 172.21.15.29:0/3361458559 learned_addr learned my addr 172.21.15.29:0/3361458559 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae677fe640 1 --2- 172.21.15.29:0/3361458559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 0x55741ebf18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.494+0000 7fae5effd640 1 --2- 172.21.15.29:0/3361458559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e84bf60 0x55741ebf13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.495+0000 7fae657fa640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fae540b3ca0 0x7fae540b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.495+0000 7fae5effd640 1 -- 172.21.15.29:0/3361458559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 msgr2=0x55741ebf18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.495+0000 7fae5effd640 1 --2- 172.21.15.29:0/3361458559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 0x55741ebf18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.495+0000 7fae5effd640 1 -- 172.21.15.29:0/3361458559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e84bf60 msgr2=0x55741ebf13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.495+0000 7fae5effd640 1 --2- 172.21.15.29:0/3361458559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e84bf60 0x55741ebf13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.495+0000 7fae5effd640 1 -- 172.21.15.29:0/3361458559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55741eafda60 con 0x55741e82b280 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.495+0000 7fae67fff640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fae540b3ca0 0x7fae540b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.495+0000 7fae5effd640 1 --2- 172.21.15.29:0/3361458559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e82b280 0x55741ebf1e20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fae5000bf80 tx=0x7fae50005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c7d51c5bd140f3d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:31.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.495+0000 7fae657fa640 1 -- 172.21.15.29:0/507144859 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fae600e3e20 con 0x7fae68109870 2024-07-27T06:46:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.495+0000 7fae5dffb640 1 -- 172.21.15.29:0/3361458559 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae50004220 con 0x55741e82b280 2024-07-27T06:46:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.495+0000 7fae5dffb640 1 -- 172.21.15.29:0/3361458559 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae500043c0 con 0x55741e82b280 2024-07-27T06:46:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.495+0000 7fae5dffb640 1 -- 172.21.15.29:0/3361458559 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae50004e40 con 0x55741e82b280 2024-07-27T06:46:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.496+0000 7fae747ac740 1 -- 172.21.15.29:0/3361458559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e82b280 msgr2=0x55741ebf1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.496+0000 7fae747ac740 1 --2- 172.21.15.29:0/3361458559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e82b280 0x55741ebf1e20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fae5000bf80 tx=0x7fae50005c90 comp rx=0 tx=0).stop 2024-07-27T06:46:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.496+0000 7fae67fff640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fae540b3ca0 0x7fae540b61b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fae600aaa10 tx=0x7fae600da940 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.496+0000 7fae747ac740 1 -- 172.21.15.29:0/3361458559 shutdown_connections 2024-07-27T06:46:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.496+0000 7fae747ac740 1 --2- 172.21.15.29:0/3361458559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e82b280 0x55741ebf1e20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.496+0000 7fae747ac740 1 --2- 172.21.15.29:0/3361458559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 0x55741ebf18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.496+0000 7fae747ac740 1 --2- 172.21.15.29:0/3361458559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e84bf60 0x55741ebf13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.496+0000 7fae747ac740 1 -- 172.21.15.29:0/3361458559 >> 172.21.15.29:0/3361458559 conn(0x55741e840350 msgr2=0x55741ebef290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:31.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.496+0000 7fae747ac740 1 -- 172.21.15.29:0/3361458559 shutdown_connections 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.496+0000 7fae747ac740 1 -- 172.21.15.29:0/3361458559 wait complete. 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.496+0000 7fae747ac740 1 -- start start 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae747ac740 1 Processor -- start 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae747ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e8230a0 0x55741ebfcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae747ac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e82b280 0x55741ebfd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae747ac740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e84bf60 0x55741ec08ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae747ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55741e9ba5f0 con 0x55741e8230a0 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae747ac740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55741eaf98b0 con 0x55741e84bf60 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae747ac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55741ebffb80 con 0x55741e82b280 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae5effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e8230a0 0x55741ebfcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae747ac740 10 client.0 _ll_register_callbacks cb 0x55741eb037f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae5effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e8230a0 0x55741ebfcfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43408/0 (socket says 172.21.15.29:43408) 2024-07-27T06:46:31.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae5effd640 1 -- 172.21.15.29:0/3212690995 learned_addr learned my addr 172.21.15.29:0/3212690995 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae677fe640 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e82b280 0x55741ebfd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae5effd640 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e84bf60 0x55741ec08ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae5effd640 1 -- 172.21.15.29:0/3212690995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e82b280 msgr2=0x55741ebfd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae5effd640 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e82b280 0x55741ebfd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae5effd640 1 -- 172.21.15.29:0/3212690995 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e84bf60 msgr2=0x55741ec08ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae5effd640 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e84bf60 0x55741ec08ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.497+0000 7fae5effd640 1 -- 172.21.15.29:0/3212690995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55741e8b7540 con 0x55741e8230a0 2024-07-27T06:46:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae5effd640 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e8230a0 0x55741ebfcfe0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fae5000b5f0 tx=0x7fae50004ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f05feeb492ef685 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:46:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae46ffd640 1 -- 172.21.15.29:0/3212690995 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae50005380 con 0x55741e8230a0 2024-07-27T06:46:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae46ffd640 1 -- 172.21.15.29:0/3212690995 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fae50005520 con 0x55741e8230a0 2024-07-27T06:46:31.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae747ac740 1 -- 172.21.15.29:0/3212690995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae500043c0 con 0x55741e8230a0 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae46ffd640 1 -- 172.21.15.29:0/3212690995 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fae50004800 con 0x55741e8230a0 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae747ac740 10 client.15309 fetch_fsmap learned FSMap version 28 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae747ac740 1 -- 172.21.15.29:0/3212690995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55741ec0bc50 con 0x55741e8230a0 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae46ffd640 1 -- 172.21.15.29:0/3212690995 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae500043c0 con 0x55741e8230a0 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae46ffd640 1 -- 172.21.15.29:0/3212690995 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fae5001c070 con 0x55741e8230a0 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae747ac740 10 client.15309 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae747ac740 10 client.15309 resolve_mds: resolved name 'l' to daemon mds.l 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae747ac740 20 client.15309 populate_metadata read hostname 'smithi029' 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae747ac740 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55741e80a8e0 0x55741ec0f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae747ac740 4 client.15309 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.498+0000 7fae747ac740 1 -- 172.21.15.29:0/3212690995 --> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55741eb69890 con 0x55741e80a8e0 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.499+0000 7fae677fe640 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55741e80a8e0 0x55741ec0f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.499+0000 7fae46ffd640 10 client.15309 ms_handle_connect on v2:172.21.15.77:6832/1085922080 2024-07-27T06:46:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.499+0000 7fae677fe640 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55741e80a8e0 0x55741ec0f3d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=93e54c78166be5ef server_cookie=3513d79aff115d8c in_seq=0 out_seq=0 2024-07-27T06:46:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae46ffd640 1 -- 172.21.15.29:0/3212690995 <== mds.4 v2:172.21.15.77:6832/1085922080 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55741eb69890 con 0x55741e80a8e0 2024-07-27T06:46:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae46ffd640 10 client.15309 handle_command_reply: tid=0 2024-07-27T06:46:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae747ac740 1 client.15309 shutdown 2024-07-27T06:46:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae747ac740 1 -- 172.21.15.29:0/3212690995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e8230a0 msgr2=0x55741ebfcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae747ac740 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e8230a0 0x55741ebfcfe0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fae5000b5f0 tx=0x7fae50004ae0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae747ac740 1 -- 172.21.15.29:0/3212690995 shutdown_connections 2024-07-27T06:46:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae747ac740 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55741e80a8e0 0x55741ec0f3d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae747ac740 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e84bf60 0x55741ec08ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae747ac740 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e82b280 0x55741ebfd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae46ffd640 0 client.15309 ms_handle_reset on v2:172.21.15.77:6832/1085922080 2024-07-27T06:46:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae747ac740 1 --2- 172.21.15.29:0/3212690995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e8230a0 0x55741ebfcfe0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae747ac740 1 -- 172.21.15.29:0/3212690995 >> 172.21.15.29:0/3212690995 conn(0x55741e840350 msgr2=0x55741ebf4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:31.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae747ac740 1 -- 172.21.15.29:0/3212690995 shutdown_connections 2024-07-27T06:46:31.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.500+0000 7fae747ac740 1 -- 172.21.15.29:0/3212690995 wait complete. 2024-07-27T06:46:31.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.501+0000 7fae747ac740 20 client.15309 trim_cache size 0 max 16384 2024-07-27T06:46:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.518+0000 7fae747ac740 1 Processor -- start 2024-07-27T06:46:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae747ac740 1 -- start start 2024-07-27T06:46:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae747ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 0x55741ec0df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae747ac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 0x55741ec08e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae747ac740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e82b280 0x55741ebfb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae747ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55741ec0d710 con 0x55741e80a8e0 2024-07-27T06:46:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae747ac740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55741ec0dde0 con 0x55741e82b280 2024-07-27T06:46:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae747ac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55741ec0e4b0 con 0x55741e8230a0 2024-07-27T06:46:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae5effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 0x55741ec0df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae5effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 0x55741ec0df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43416/0 (socket says 172.21.15.29:43416) 2024-07-27T06:46:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae5effd640 1 -- 172.21.15.29:0/2586995270 learned_addr learned my addr 172.21.15.29:0/2586995270 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae677fe640 1 --2- 172.21.15.29:0/2586995270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 0x55741ec08e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae5effd640 1 -- 172.21.15.29:0/2586995270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 msgr2=0x55741ec08e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae5effd640 1 --2- 172.21.15.29:0/2586995270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 0x55741ec08e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae5effd640 1 -- 172.21.15.29:0/2586995270 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e82b280 msgr2=0x55741ebfb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae5effd640 1 --2- 172.21.15.29:0/2586995270 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e82b280 0x55741ebfb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae5effd640 1 -- 172.21.15.29:0/2586995270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55741ebf4200 con 0x55741e80a8e0 2024-07-27T06:46:31.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.519+0000 7fae5effd640 1 --2- 172.21.15.29:0/2586995270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 0x55741ec0df60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fae50004690 tx=0x7fae5000bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1c3d869da6a002 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:31.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.520+0000 7fae5d7fa640 1 -- 172.21.15.29:0/2586995270 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5002a950 con 0x55741e80a8e0 2024-07-27T06:46:31.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.520+0000 7fae5d7fa640 1 -- 172.21.15.29:0/2586995270 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae50005630 con 0x55741e80a8e0 2024-07-27T06:46:31.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.520+0000 7fae5d7fa640 1 -- 172.21.15.29:0/2586995270 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae50010a10 con 0x55741e80a8e0 2024-07-27T06:46:31.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.520+0000 7fae747ac740 1 -- 172.21.15.29:0/2586995270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 msgr2=0x55741ec0df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:31.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.520+0000 7fae747ac740 1 --2- 172.21.15.29:0/2586995270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 0x55741ec0df60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fae50004690 tx=0x7fae5000bfb0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.520+0000 7fae747ac740 1 -- 172.21.15.29:0/2586995270 shutdown_connections 2024-07-27T06:46:31.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.520+0000 7fae747ac740 1 --2- 172.21.15.29:0/2586995270 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e82b280 0x55741ebfb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.521+0000 7fae747ac740 1 --2- 172.21.15.29:0/2586995270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 0x55741ec08e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.521+0000 7fae747ac740 1 --2- 172.21.15.29:0/2586995270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 0x55741ec0df60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.521+0000 7fae747ac740 1 -- 172.21.15.29:0/2586995270 >> 172.21.15.29:0/2586995270 conn(0x55741e840350 msgr2=0x55741ebf7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.521+0000 7fae747ac740 1 -- 172.21.15.29:0/2586995270 shutdown_connections 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.521+0000 7fae747ac740 1 -- 172.21.15.29:0/2586995270 wait complete. 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.521+0000 7fae747ac740 1 -- start start 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.521+0000 7fae747ac740 1 Processor -- start 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.521+0000 7fae747ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 0x55741ec10960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae747ac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 0x55741ec10f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae747ac740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e82b280 0x55741ec15d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae747ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55741ec0f250 con 0x55741e80a8e0 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae747ac740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55741ec0eb80 con 0x55741e82b280 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae747ac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55741ec0f920 con 0x55741e8230a0 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae747ac740 10 client.0 _ll_register_callbacks cb 0x7fae500046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae5effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 0x55741ec10960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae5effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 0x55741ec10960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43420/0 (socket says 172.21.15.29:43420) 2024-07-27T06:46:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae5effd640 1 -- 172.21.15.29:0/2659789529 learned_addr learned my addr 172.21.15.29:0/2659789529 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae677fe640 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 0x55741ec10f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae5effd640 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e82b280 0x55741ec15d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae5effd640 1 -- 172.21.15.29:0/2659789529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 msgr2=0x55741ec10f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae5effd640 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 0x55741ec10f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae5effd640 1 -- 172.21.15.29:0/2659789529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e82b280 msgr2=0x55741ec15d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae5effd640 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e82b280 0x55741ec15d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae5effd640 1 -- 172.21.15.29:0/2659789529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55741ec163b0 con 0x55741e80a8e0 2024-07-27T06:46:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae5effd640 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 0x55741ec10960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fae500049a0 tx=0x7fae500057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8453278ecd49adf7 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:46:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.522+0000 7fae46ffd640 1 -- 172.21.15.29:0/2659789529 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae50005180 con 0x55741e80a8e0 2024-07-27T06:46:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae46ffd640 1 -- 172.21.15.29:0/2659789529 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fae50005320 con 0x55741e80a8e0 2024-07-27T06:46:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae747ac740 1 -- 172.21.15.29:0/2659789529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae50005630 con 0x55741e80a8e0 2024-07-27T06:46:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae46ffd640 1 -- 172.21.15.29:0/2659789529 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fae50010790 con 0x55741e80a8e0 2024-07-27T06:46:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae747ac740 10 client.15315 fetch_fsmap learned FSMap version 28 2024-07-27T06:46:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae747ac740 1 -- 172.21.15.29:0/2659789529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55741ec176e0 con 0x55741e80a8e0 2024-07-27T06:46:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae46ffd640 1 -- 172.21.15.29:0/2659789529 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae50005630 con 0x55741e80a8e0 2024-07-27T06:46:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae46ffd640 1 -- 172.21.15.29:0/2659789529 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fae5001c050 con 0x55741e80a8e0 2024-07-27T06:46:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae747ac740 10 client.15315 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:46:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae747ac740 10 client.15315 resolve_mds: resolved name 'l' to daemon mds.l 2024-07-27T06:46:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae747ac740 20 client.15315 populate_metadata read hostname 'smithi029' 2024-07-27T06:46:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae747ac740 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55741e84bf60 0x55741ec7ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae747ac740 4 client.15315 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:46:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.523+0000 7fae747ac740 1 -- 172.21.15.29:0/2659789529 --> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] -- command(tid 0: {"prefix": "status"}) -- 0x55741eb65330 con 0x55741e84bf60 2024-07-27T06:46:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.524+0000 7fae677fe640 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55741e84bf60 0x55741ec7ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.524+0000 7fae677fe640 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55741e84bf60 0x55741ec7ad50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3b8bffbeda576a97 server_cookie=345729723b684b96 in_seq=0 out_seq=0 2024-07-27T06:46:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.524+0000 7fae46ffd640 10 client.15315 ms_handle_connect on v2:172.21.15.77:6832/1085922080 2024-07-27T06:46:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.524+0000 7fae46ffd640 1 -- 172.21.15.29:0/2659789529 <== mds.4 v2:172.21.15.77:6832/1085922080 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55741eb65330 con 0x55741e84bf60 2024-07-27T06:46:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.524+0000 7fae46ffd640 10 client.15315 handle_command_reply: tid=0 2024-07-27T06:46:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.524+0000 7fae747ac740 1 client.15315 shutdown 2024-07-27T06:46:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.524+0000 7fae747ac740 1 -- 172.21.15.29:0/2659789529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 msgr2=0x55741ec10960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.524+0000 7fae747ac740 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 0x55741ec10960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fae500049a0 tx=0x7fae500057e0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.524+0000 7fae747ac740 1 -- 172.21.15.29:0/2659789529 shutdown_connections 2024-07-27T06:46:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.524+0000 7fae747ac740 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55741e84bf60 0x55741ec7ad50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae747ac740 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55741e82b280 0x55741ec15d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae747ac740 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55741e8230a0 0x55741ec10f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae747ac740 1 --2- 172.21.15.29:0/2659789529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55741e80a8e0 0x55741ec10960 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae46ffd640 0 client.15315 ms_handle_reset on v2:172.21.15.77:6832/1085922080 2024-07-27T06:46:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae747ac740 1 -- 172.21.15.29:0/2659789529 >> 172.21.15.29:0/2659789529 conn(0x55741e840350 msgr2=0x55741ebfb620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae747ac740 1 -- 172.21.15.29:0/2659789529 shutdown_connections 2024-07-27T06:46:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae747ac740 1 -- 172.21.15.29:0/2659789529 wait complete. 2024-07-27T06:46:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae747ac740 20 client.15315 trim_cache size 0 max 16384 2024-07-27T06:46:31.533 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:46:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:46:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 4, 2024-07-27T06:46:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24613, 2024-07-27T06:46:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:46:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:46:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:46:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 178.66335769599999, 2024-07-27T06:46:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-07-27T06:46:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:46:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 79, 2024-07-27T06:46:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 249.643126634 2024-07-27T06:46:31.534 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:46:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae6e228640 1 -- 172.21.15.29:0/507144859 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fae540b3ca0 msgr2=0x7fae540b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae6e228640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fae540b3ca0 0x7fae540b61b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fae600aaa10 tx=0x7fae600da940 comp rx=0 tx=0).stop 2024-07-27T06:46:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae6e228640 1 -- 172.21.15.29:0/507144859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae68109870 msgr2=0x7fae6819f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae6e228640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae68109870 0x7fae6819f750 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fae6000eaa0 tx=0x7fae6000ef60 comp rx=0 tx=0).stop 2024-07-27T06:46:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae6e228640 1 -- 172.21.15.29:0/507144859 shutdown_connections 2024-07-27T06:46:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae6e228640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae68109870 0x7fae6819f750 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae6e228640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae68108f20 0x7fae6819a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae6e228640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fae540b3ca0 0x7fae540b61b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae6e228640 1 --2- 172.21.15.29:0/507144859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae681035e0 0x7fae68199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae6e228640 1 -- 172.21.15.29:0/507144859 >> 172.21.15.29:0/507144859 conn(0x7fae680ff030 msgr2=0x7fae68101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae6e228640 1 -- 172.21.15.29:0/507144859 shutdown_connections 2024-07-27T06:46:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:31.525+0000 7fae6e228640 1 -- 172.21.15.29:0/507144859 wait complete. 2024-07-27T06:46:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:31 smithi047 ceph-mon[36835]: pgmap v382: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:31 smithi029 ceph-mon[32610]: pgmap v382: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:31 smithi077 ceph-mon[37153]: pgmap v382: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:32.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.b status 2024-07-27T06:46:32.500 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:33.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:33 smithi029 ceph-mon[32610]: pgmap v383: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:33 smithi047 ceph-mon[36835]: pgmap v383: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:33 smithi077 ceph-mon[37153]: pgmap v383: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:34.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.163+0000 7f252653d640 1 -- 172.21.15.29:0/280342511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520108ed0 msgr2=0x7f25201092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:34.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.163+0000 7f252653d640 1 --2- 172.21.15.29:0/280342511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520108ed0 0x7f25201092e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2514009900 tx=0x7f251402a410 comp rx=0 tx=0).stop 2024-07-27T06:46:34.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.163+0000 7f252653d640 1 -- 172.21.15.29:0/280342511 shutdown_connections 2024-07-27T06:46:34.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.163+0000 7f252653d640 1 --2- 172.21.15.29:0/280342511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2520109820 0x7f2520110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.163+0000 7f252653d640 1 --2- 172.21.15.29:0/280342511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520108ed0 0x7f25201092e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.163+0000 7f252653d640 1 --2- 172.21.15.29:0/280342511 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2520103590 0x7f25201039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.163+0000 7f252653d640 1 -- 172.21.15.29:0/280342511 >> 172.21.15.29:0/280342511 conn(0x7f252007b280 msgr2=0x7f252007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:34.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.163+0000 7f252653d640 1 -- 172.21.15.29:0/280342511 shutdown_connections 2024-07-27T06:46:34.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.163+0000 7f252653d640 1 -- 172.21.15.29:0/280342511 wait complete. 2024-07-27T06:46:34.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.163+0000 7f252653d640 1 Processor -- start 2024-07-27T06:46:34.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f252653d640 1 -- start start 2024-07-27T06:46:34.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f252653d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520103590 0x7f2520199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f252653d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2520108ed0 0x7f252019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f252653d640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2520109820 0x7f252019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f252653d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f25201134b0 con 0x7f2520103590 2024-07-27T06:46:34.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f252653d640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f2520113330 con 0x7f2520109820 2024-07-27T06:46:34.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f252653d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2520113630 con 0x7f2520108ed0 2024-07-27T06:46:34.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f251ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520103590 0x7f2520199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f251ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520103590 0x7f2520199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43438/0 (socket says 172.21.15.29:43438) 2024-07-27T06:46:34.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f251ffff640 1 -- 172.21.15.29:0/3373896605 learned_addr learned my addr 172.21.15.29:0/3373896605 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:34.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f251f7fe640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2520108ed0 0x7f252019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f2524ada640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2520109820 0x7f252019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f251ffff640 1 -- 172.21.15.29:0/3373896605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2520108ed0 msgr2=0x7f252019a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f251ffff640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2520108ed0 0x7f252019a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f251ffff640 1 -- 172.21.15.29:0/3373896605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2520109820 msgr2=0x7f252019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f251ffff640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2520109820 0x7f252019f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.164+0000 7f251ffff640 1 -- 172.21.15.29:0/3373896605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252019fe50 con 0x7f2520103590 2024-07-27T06:46:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.165+0000 7f251ffff640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520103590 0x7f2520199ee0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f251000d950 tx=0x7f251000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.165+0000 7f251d7fa640 1 -- 172.21.15.29:0/3373896605 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2510017070 con 0x7f2520103590 2024-07-27T06:46:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.165+0000 7f251d7fa640 1 -- 172.21.15.29:0/3373896605 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2510009c30 con 0x7f2520103590 2024-07-27T06:46:34.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.165+0000 7f252653d640 1 -- 172.21.15.29:0/3373896605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25201a0140 con 0x7f2520103590 2024-07-27T06:46:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.165+0000 7f251d7fa640 1 -- 172.21.15.29:0/3373896605 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251000f620 con 0x7f2520103590 2024-07-27T06:46:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.165+0000 7f252653d640 1 -- 172.21.15.29:0/3373896605 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25201a68a0 con 0x7f2520103590 2024-07-27T06:46:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.167+0000 7f252cac1740 1 Processor -- start 2024-07-27T06:46:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.167+0000 7f252cac1740 1 -- start start 2024-07-27T06:46:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f251d7fa640 1 -- 172.21.15.29:0/3373896605 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f2510010460 con 0x7f2520103590 2024-07-27T06:46:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f252cac1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a839df60 0x5622a8743380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f252cac1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a83750a0 0x5622a87438c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f252cac1740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a837d280 0x5622a8743e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f252cac1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5622a86b9920 con 0x5622a83750a0 2024-07-27T06:46:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f252cac1740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5622a86a8e20 con 0x5622a837d280 2024-07-27T06:46:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f252cac1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622a86ace60 con 0x5622a839df60 2024-07-27T06:46:34.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f251ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a83750a0 0x5622a87438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f251ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a83750a0 0x5622a87438c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43446/0 (socket says 172.21.15.29:43446) 2024-07-27T06:46:34.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f251ffff640 1 -- 172.21.15.29:0/2721795362 learned_addr learned my addr 172.21.15.29:0/2721795362 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:34.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f251f7fe640 1 --2- 172.21.15.29:0/2721795362 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a837d280 0x5622a8743e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f2524ada640 1 --2- 172.21.15.29:0/2721795362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a839df60 0x5622a8743380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f251ffff640 1 -- 172.21.15.29:0/2721795362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a839df60 msgr2=0x5622a8743380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:34.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f251ffff640 1 --2- 172.21.15.29:0/2721795362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a839df60 0x5622a8743380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f251ffff640 1 -- 172.21.15.29:0/2721795362 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a837d280 msgr2=0x5622a8743e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:34.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f251ffff640 1 --2- 172.21.15.29:0/2721795362 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a837d280 0x5622a8743e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.168+0000 7f251ffff640 1 -- 172.21.15.29:0/2721795362 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622a864fa40 con 0x5622a83750a0 2024-07-27T06:46:34.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.169+0000 7f2524ada640 1 --2- 172.21.15.29:0/2721795362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a839df60 0x5622a8743380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:34.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.169+0000 7f251f7fe640 1 --2- 172.21.15.29:0/2721795362 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a837d280 0x5622a8743e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:34.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.169+0000 7f251d7fa640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f25080b3c20 0x7f25080b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.169+0000 7f251ffff640 1 --2- 172.21.15.29:0/2721795362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a83750a0 0x5622a87438c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f25100e7b90 tx=0x7f25100ec500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d67cf545f4372b server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:34.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.169+0000 7f2524ada640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f25080b3c20 0x7f25080b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.169+0000 7f251d7fa640 1 -- 172.21.15.29:0/3373896605 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f25100e3ab0 con 0x7f2520103590 2024-07-27T06:46:34.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.169+0000 7f25067fc640 1 -- 172.21.15.29:0/2721795362 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25100a6750 con 0x5622a83750a0 2024-07-27T06:46:34.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.169+0000 7f25067fc640 1 -- 172.21.15.29:0/2721795362 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25100aa3a0 con 0x5622a83750a0 2024-07-27T06:46:34.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.169+0000 7f25067fc640 1 -- 172.21.15.29:0/2721795362 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25100aa000 con 0x5622a83750a0 2024-07-27T06:46:34.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.170+0000 7f252cac1740 1 -- 172.21.15.29:0/2721795362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a83750a0 msgr2=0x5622a87438c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:34.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.170+0000 7f2524ada640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f25080b3c20 0x7f25080b6130 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2518005fd0 tx=0x7f25180094a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:34.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.170+0000 7f252cac1740 1 --2- 172.21.15.29:0/2721795362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a83750a0 0x5622a87438c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f25100e7b90 tx=0x7f25100ec500 comp rx=0 tx=0).stop 2024-07-27T06:46:34.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.170+0000 7f252cac1740 1 -- 172.21.15.29:0/2721795362 shutdown_connections 2024-07-27T06:46:34.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.171+0000 7f252cac1740 1 --2- 172.21.15.29:0/2721795362 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a837d280 0x5622a8743e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.171+0000 7f252cac1740 1 --2- 172.21.15.29:0/2721795362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a83750a0 0x5622a87438c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.171+0000 7f252cac1740 1 --2- 172.21.15.29:0/2721795362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a839df60 0x5622a8743380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.171+0000 7f252cac1740 1 -- 172.21.15.29:0/2721795362 >> 172.21.15.29:0/2721795362 conn(0x5622a8392350 msgr2=0x5622a8741270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:34.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.171+0000 7f252cac1740 1 -- 172.21.15.29:0/2721795362 shutdown_connections 2024-07-27T06:46:34.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.171+0000 7f252cac1740 1 -- 172.21.15.29:0/2721795362 wait complete. 2024-07-27T06:46:34.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.171+0000 7f252cac1740 1 -- start start 2024-07-27T06:46:34.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.171+0000 7f252cac1740 1 Processor -- start 2024-07-27T06:46:34.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f252cac1740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a83750a0 0x5622a874efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f252cac1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 0x5622a874f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f252cac1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a839df60 0x5622a875adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f252cac1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5622a850c5f0 con 0x5622a837d280 2024-07-27T06:46:34.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f252cac1740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5622a864b890 con 0x5622a83750a0 2024-07-27T06:46:34.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f252cac1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622a8751aa0 con 0x5622a839df60 2024-07-27T06:46:34.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f252cac1740 10 client.0 _ll_register_callbacks cb 0x5622a86557d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:46:34.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f251f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 0x5622a874f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f251ffff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a83750a0 0x5622a874efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f2524ada640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a839df60 0x5622a875adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f251f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 0x5622a874f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43456/0 (socket says 172.21.15.29:43456) 2024-07-27T06:46:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f251f7fe640 1 -- 172.21.15.29:0/4025420716 learned_addr learned my addr 172.21.15.29:0/4025420716 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f2524ada640 1 -- 172.21.15.29:0/4025420716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a83750a0 msgr2=0x5622a874efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f2524ada640 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a83750a0 0x5622a874efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f2524ada640 1 -- 172.21.15.29:0/4025420716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 msgr2=0x5622a874f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f2524ada640 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 0x5622a874f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f2524ada640 1 -- 172.21.15.29:0/4025420716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622a8409540 con 0x5622a839df60 2024-07-27T06:46:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f251f7fe640 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 0x5622a874f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:46:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.172+0000 7f251ffff640 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a83750a0 0x5622a874efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:46:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.173+0000 7f2524ada640 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a839df60 0x5622a875adf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2518011c40 tx=0x7f2518004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92b7a545e2aafa46 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.173+0000 7f24ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:46:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.173+0000 7f24ff7fe640 1 -- 172.21.15.29:0/4025420716 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2518004d20 con 0x5622a839df60 2024-07-27T06:46:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.173+0000 7f252cac1740 1 -- 172.21.15.29:0/4025420716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25100aa3a0 con 0x5622a839df60 2024-07-27T06:46:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.173+0000 7f24ff7fe640 1 -- 172.21.15.29:0/4025420716 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f251800f8e0 con 0x5622a839df60 2024-07-27T06:46:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.173+0000 7f24ff7fe640 1 -- 172.21.15.29:0/4025420716 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f251800fae0 con 0x5622a839df60 2024-07-27T06:46:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.173+0000 7f252cac1740 10 client.25108 fetch_fsmap learned FSMap version 28 2024-07-27T06:46:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.173+0000 7f252cac1740 1 -- 172.21.15.29:0/4025420716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5622a875dc10 con 0x5622a839df60 2024-07-27T06:46:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.173+0000 7f24ff7fe640 1 -- 172.21.15.29:0/4025420716 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25100aa3a0 con 0x5622a839df60 2024-07-27T06:46:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.174+0000 7f24ff7fe640 1 -- 172.21.15.29:0/4025420716 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2518030470 con 0x5622a839df60 2024-07-27T06:46:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.174+0000 7f252cac1740 10 client.25108 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:46:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.174+0000 7f252cac1740 10 client.25108 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-27T06:46:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.174+0000 7f252cac1740 20 client.25108 populate_metadata read hostname 'smithi029' 2024-07-27T06:46:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.174+0000 7f252cac1740 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5622a835c8e0 0x5622a8761620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.174+0000 7f252cac1740 4 client.25108 mds_command: new command op to 24617 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.174+0000 7f252cac1740 1 -- 172.21.15.29:0/4025420716 --> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622a86bb870 con 0x5622a835c8e0 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.174+0000 7f251ffff640 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5622a835c8e0 0x5622a8761620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.175+0000 7f251ffff640 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5622a835c8e0 0x5622a8761620 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b3e22ed63d5c1336 server_cookie=30b7893c00499e9 in_seq=0 out_seq=0 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.175+0000 7f24ff7fe640 10 client.25108 ms_handle_connect on v2:172.21.15.47:6838/670107266 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.175+0000 7f24ff7fe640 1 -- 172.21.15.29:0/4025420716 <== mds.3 v2:172.21.15.47:6838/670107266 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5622a86bb870 con 0x5622a835c8e0 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.175+0000 7f24ff7fe640 10 client.25108 handle_command_reply: tid=0 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.175+0000 7f252cac1740 1 client.25108 shutdown 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.176+0000 7f252cac1740 1 -- 172.21.15.29:0/4025420716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a839df60 msgr2=0x5622a875adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.176+0000 7f252cac1740 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a839df60 0x5622a875adf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2518011c40 tx=0x7f2518004a40 comp rx=0 tx=0).stop 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.176+0000 7f252cac1740 1 -- 172.21.15.29:0/4025420716 shutdown_connections 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.176+0000 7f252cac1740 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5622a835c8e0 0x5622a8761620 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.176+0000 7f252cac1740 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a839df60 0x5622a875adf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.176+0000 7f252cac1740 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 0x5622a874f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.176+0000 7f252cac1740 1 --2- 172.21.15.29:0/4025420716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a83750a0 0x5622a874efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.176+0000 7f252cac1740 1 -- 172.21.15.29:0/4025420716 >> 172.21.15.29:0/4025420716 conn(0x5622a8392350 msgr2=0x5622a87443b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:34.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.176+0000 7f24ff7fe640 0 client.25108 ms_handle_reset on v2:172.21.15.47:6838/670107266 2024-07-27T06:46:34.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.176+0000 7f252cac1740 1 -- 172.21.15.29:0/4025420716 shutdown_connections 2024-07-27T06:46:34.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.176+0000 7f252cac1740 1 -- 172.21.15.29:0/4025420716 wait complete. 2024-07-27T06:46:34.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.176+0000 7f252cac1740 20 client.25108 trim_cache size 0 max 16384 2024-07-27T06:46:34.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.194+0000 7f252cac1740 1 Processor -- start 2024-07-27T06:46:34.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.194+0000 7f252cac1740 1 -- start start 2024-07-27T06:46:34.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.194+0000 7f252cac1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a83750a0 0x5622a8749f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.194+0000 7f252cac1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 0x5622a874a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.194+0000 7f252cac1740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a839df60 0x5622a874d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.194+0000 7f252cac1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5622a875f960 con 0x5622a837d280 2024-07-27T06:46:34.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.194+0000 7f252cac1740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5622a8760030 con 0x5622a839df60 2024-07-27T06:46:34.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.194+0000 7f252cac1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622a8760700 con 0x5622a83750a0 2024-07-27T06:46:34.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f251f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 0x5622a874a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f251f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 0x5622a874a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43460/0 (socket says 172.21.15.29:43460) 2024-07-27T06:46:34.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f251f7fe640 1 -- 172.21.15.29:0/3757369496 learned_addr learned my addr 172.21.15.29:0/3757369496 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:34.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f251ffff640 1 --2- 172.21.15.29:0/3757369496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a83750a0 0x5622a8749f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f2524ada640 1 --2- 172.21.15.29:0/3757369496 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a839df60 0x5622a874d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f251f7fe640 1 -- 172.21.15.29:0/3757369496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a83750a0 msgr2=0x5622a8749f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:34.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f251f7fe640 1 --2- 172.21.15.29:0/3757369496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a83750a0 0x5622a8749f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f251f7fe640 1 -- 172.21.15.29:0/3757369496 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a839df60 msgr2=0x5622a874d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:34.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f251f7fe640 1 --2- 172.21.15.29:0/3757369496 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a839df60 0x5622a874d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f251f7fe640 1 -- 172.21.15.29:0/3757369496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622a874d2d0 con 0x5622a837d280 2024-07-27T06:46:34.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f251f7fe640 1 --2- 172.21.15.29:0/3757369496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 0x5622a874a450 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2514006010 tx=0x7f25140315c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec76903f061c861 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:34.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f2505ffb640 1 -- 172.21.15.29:0/3757369496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514035040 con 0x5622a837d280 2024-07-27T06:46:34.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f2505ffb640 1 -- 172.21.15.29:0/3757369496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25140318b0 con 0x5622a837d280 2024-07-27T06:46:34.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.195+0000 7f2505ffb640 1 -- 172.21.15.29:0/3757369496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514004920 con 0x5622a837d280 2024-07-27T06:46:34.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.196+0000 7f252cac1740 1 -- 172.21.15.29:0/3757369496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 msgr2=0x5622a874a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:34.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.196+0000 7f252cac1740 1 --2- 172.21.15.29:0/3757369496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 0x5622a874a450 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2514006010 tx=0x7f25140315c0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.196+0000 7f252cac1740 1 -- 172.21.15.29:0/3757369496 shutdown_connections 2024-07-27T06:46:34.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.196+0000 7f252cac1740 1 --2- 172.21.15.29:0/3757369496 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a839df60 0x5622a874d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.196+0000 7f252cac1740 1 --2- 172.21.15.29:0/3757369496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a837d280 0x5622a874a450 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.196+0000 7f252cac1740 1 --2- 172.21.15.29:0/3757369496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a83750a0 0x5622a8749f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.196+0000 7f252cac1740 1 -- 172.21.15.29:0/3757369496 >> 172.21.15.29:0/3757369496 conn(0x5622a8392350 msgr2=0x5622a87491f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.196+0000 7f252cac1740 1 -- 172.21.15.29:0/3757369496 shutdown_connections 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.196+0000 7f252cac1740 1 -- 172.21.15.29:0/3757369496 wait complete. 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f252cac1740 1 -- start start 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f252cac1740 1 Processor -- start 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f252cac1740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a83750a0 0x5622a87629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f252cac1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a837d280 0x5622a8762f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f252cac1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a839df60 0x5622a8767d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f252cac1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5622a87614a0 con 0x5622a839df60 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f252cac1740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5622a8760dd0 con 0x5622a83750a0 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f252cac1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622a8761b70 con 0x5622a837d280 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f252cac1740 10 client.0 _ll_register_callbacks cb 0x7f2518012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:46:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f2524ada640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a839df60 0x5622a8767d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f2524ada640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a839df60 0x5622a8767d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43474/0 (socket says 172.21.15.29:43474) 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f2524ada640 1 -- 172.21.15.29:0/2832984200 learned_addr learned my addr 172.21.15.29:0/2832984200 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.197+0000 7f251ffff640 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a83750a0 0x5622a87629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f251f7fe640 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a837d280 0x5622a8762f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f2524ada640 1 -- 172.21.15.29:0/2832984200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a837d280 msgr2=0x5622a8762f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f2524ada640 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a837d280 0x5622a8762f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f2524ada640 1 -- 172.21.15.29:0/2832984200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a83750a0 msgr2=0x5622a87629c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f2524ada640 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a83750a0 0x5622a87629c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f2524ada640 1 -- 172.21.15.29:0/2832984200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622a87683d0 con 0x5622a839df60 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f251ffff640 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a83750a0 0x5622a87629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f2524ada640 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a839df60 0x5622a8767d60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f251800a010 tx=0x7f251800ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65eff775432c74d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f24ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:46:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f24ff7fe640 1 -- 172.21.15.29:0/2832984200 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2518036040 con 0x5622a839df60 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f24ff7fe640 1 -- 172.21.15.29:0/2832984200 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f2518030470 con 0x5622a839df60 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f252cac1740 1 -- 172.21.15.29:0/2832984200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25140318b0 con 0x5622a839df60 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f24ff7fe640 1 -- 172.21.15.29:0/2832984200 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2518030670 con 0x5622a839df60 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f252cac1740 10 client.15333 fetch_fsmap learned FSMap version 28 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.198+0000 7f252cac1740 1 -- 172.21.15.29:0/2832984200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5622a8769790 con 0x5622a839df60 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.199+0000 7f24ff7fe640 1 -- 172.21.15.29:0/2832984200 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25140318b0 con 0x5622a839df60 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.199+0000 7f24ff7fe640 1 -- 172.21.15.29:0/2832984200 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2518005670 con 0x5622a839df60 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.199+0000 7f252cac1740 10 client.15333 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.199+0000 7f252cac1740 10 client.15333 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.199+0000 7f252cac1740 20 client.15333 populate_metadata read hostname 'smithi029' 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.199+0000 7f252cac1740 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5622a8331c90 0x5622a87cd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.199+0000 7f252cac1740 4 client.15333 mds_command: new command op to 24617 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:46:34.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.199+0000 7f252cac1740 1 -- 172.21.15.29:0/2832984200 --> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] -- command(tid 0: {"prefix": "status"}) -- 0x5622a86b7310 con 0x5622a8331c90 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.199+0000 7f251ffff640 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5622a8331c90 0x5622a87cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f24ff7fe640 10 client.15333 ms_handle_connect on v2:172.21.15.47:6838/670107266 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f251ffff640 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5622a8331c90 0x5622a87cd280 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cb1cc7fdf0ac2a41 server_cookie=3dd59e642fcdf947 in_seq=0 out_seq=0 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f24ff7fe640 1 -- 172.21.15.29:0/2832984200 <== mds.3 v2:172.21.15.47:6838/670107266 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5622a86b7310 con 0x5622a8331c90 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f24ff7fe640 10 client.15333 handle_command_reply: tid=0 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f252cac1740 1 client.15333 shutdown 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f252cac1740 1 -- 172.21.15.29:0/2832984200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a839df60 msgr2=0x5622a8767d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f252cac1740 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a839df60 0x5622a8767d60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f251800a010 tx=0x7f251800ff30 comp rx=0 tx=0).stop 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f252cac1740 1 -- 172.21.15.29:0/2832984200 shutdown_connections 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f252cac1740 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5622a8331c90 0x5622a87cd280 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f252cac1740 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622a839df60 0x5622a8767d60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f252cac1740 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622a837d280 0x5622a8762f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f252cac1740 1 --2- 172.21.15.29:0/2832984200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622a83750a0 0x5622a87629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f24ff7fe640 0 client.15333 ms_handle_reset on v2:172.21.15.47:6838/670107266 2024-07-27T06:46:34.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f252cac1740 1 -- 172.21.15.29:0/2832984200 >> 172.21.15.29:0/2832984200 conn(0x5622a8392350 msgr2=0x5622a874d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:34.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f252cac1740 1 -- 172.21.15.29:0/2832984200 shutdown_connections 2024-07-27T06:46:34.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f252cac1740 1 -- 172.21.15.29:0/2832984200 wait complete. 2024-07-27T06:46:34.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.200+0000 7f252cac1740 20 client.15333 trim_cache size 0 max 16384 2024-07-27T06:46:34.208 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:46:34.208 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:46:34.208 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 3, 2024-07-27T06:46:34.208 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24617, 2024-07-27T06:46:34.208 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:46:34.209 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:46:34.209 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:46:34.209 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 182.34767547499999, 2024-07-27T06:46:34.209 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-07-27T06:46:34.209 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:46:34.209 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 79, 2024-07-27T06:46:34.209 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 217.49430945099999 2024-07-27T06:46:34.209 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:46:34.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.201+0000 7f252653d640 1 -- 172.21.15.29:0/3373896605 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f25080b3c20 msgr2=0x7f25080b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:34.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.201+0000 7f252653d640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f25080b3c20 0x7f25080b6130 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2518005fd0 tx=0x7f25180094a0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.201+0000 7f252653d640 1 -- 172.21.15.29:0/3373896605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520103590 msgr2=0x7f2520199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:34.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.201+0000 7f252653d640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520103590 0x7f2520199ee0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f251000d950 tx=0x7f251000de10 comp rx=0 tx=0).stop 2024-07-27T06:46:34.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.201+0000 7f252653d640 1 -- 172.21.15.29:0/3373896605 shutdown_connections 2024-07-27T06:46:34.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.201+0000 7f252653d640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2520109820 0x7f252019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.201+0000 7f252653d640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f25080b3c20 0x7f25080b6130 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.201+0000 7f252653d640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2520108ed0 0x7f252019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.201+0000 7f252653d640 1 --2- 172.21.15.29:0/3373896605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2520103590 0x7f2520199ee0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:34.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.201+0000 7f252653d640 1 -- 172.21.15.29:0/3373896605 >> 172.21.15.29:0/3373896605 conn(0x7f252007b280 msgr2=0x7f2520106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:34.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.201+0000 7f252653d640 1 -- 172.21.15.29:0/3373896605 shutdown_connections 2024-07-27T06:46:34.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:34.201+0000 7f252653d640 1 -- 172.21.15.29:0/3373896605 wait complete. 2024-07-27T06:46:34.977 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.f status 2024-07-27T06:46:35.251 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:35.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:35 smithi029 ceph-mon[32610]: pgmap v384: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:35 smithi047 ceph-mon[36835]: pgmap v384: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:35 smithi077 ceph-mon[37153]: pgmap v384: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:36.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.913+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1663141724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107df0 msgr2=0x7fddcc108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.913+0000 7fddd2f1c640 1 --2- 172.21.15.29:0/1663141724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107df0 0x7fddcc108200 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fddc0009900 tx=0x7fddc002a410 comp rx=0 tx=0).stop 2024-07-27T06:46:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.913+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1663141724 shutdown_connections 2024-07-27T06:46:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.913+0000 7fddd2f1c640 1 --2- 172.21.15.29:0/1663141724 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddcc108740 0x7fddcc10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.913+0000 7fddd2f1c640 1 --2- 172.21.15.29:0/1663141724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc107df0 0x7fddcc108200 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.913+0000 7fddd2f1c640 1 --2- 172.21.15.29:0/1663141724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddcc1024b0 0x7fddcc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.913+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1663141724 >> 172.21.15.29:0/1663141724 conn(0x7fddcc079170 msgr2=0x7fddcc079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.913+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1663141724 shutdown_connections 2024-07-27T06:46:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.913+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1663141724 wait complete. 2024-07-27T06:46:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd2f1c640 1 Processor -- start 2024-07-27T06:46:36.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd2f1c640 1 -- start start 2024-07-27T06:46:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd2f1c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc1024b0 0x7fddcc198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd2f1c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddcc107df0 0x7fddcc199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd2f1c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddcc108740 0x7fddcc199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd2f1c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc112370 con 0x7fddcc1024b0 2024-07-27T06:46:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd2f1c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1121f0 con 0x7fddcc107df0 2024-07-27T06:46:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd2f1c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1124f0 con 0x7fddcc108740 2024-07-27T06:46:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd0cb8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc1024b0 0x7fddcc198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd0cb8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc1024b0 0x7fddcc198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43484/0 (socket says 172.21.15.29:43484) 2024-07-27T06:46:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd0cb8640 1 -- 172.21.15.29:0/1948550396 learned_addr learned my addr 172.21.15.29:0/1948550396 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd14b9640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddcc108740 0x7fddcc199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddcbfff640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddcc107df0 0x7fddcc199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd0cb8640 1 -- 172.21.15.29:0/1948550396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddcc108740 msgr2=0x7fddcc199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd0cb8640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddcc108740 0x7fddcc199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.914+0000 7fddd0cb8640 1 -- 172.21.15.29:0/1948550396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddcc107df0 msgr2=0x7fddcc199010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.915+0000 7fddd0cb8640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddcc107df0 0x7fddcc199010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.915+0000 7fddd0cb8640 1 -- 172.21.15.29:0/1948550396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddcc1a8150 con 0x7fddcc1024b0 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.915+0000 7fddcbfff640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddcc107df0 0x7fddcc199010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.915+0000 7fddd0cb8640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc1024b0 0x7fddcc198ad0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fddbc00c920 tx=0x7fddbc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.915+0000 7fddc9ffb640 1 -- 172.21.15.29:0/1948550396 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc007c70 con 0x7fddcc1024b0 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.915+0000 7fddc9ffb640 1 -- 172.21.15.29:0/1948550396 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddbc007e10 con 0x7fddcc1024b0 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.915+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1948550396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddcc1a83e0 con 0x7fddcc1024b0 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.915+0000 7fddc9ffb640 1 -- 172.21.15.29:0/1948550396 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc018820 con 0x7fddcc1024b0 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.915+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1948550396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddcc1a8940 con 0x7fddcc1024b0 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.918+0000 7fddc9ffb640 1 -- 172.21.15.29:0/1948550396 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fddbc00f450 con 0x7fddcc1024b0 2024-07-27T06:46:36.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.918+0000 7fddd94a0740 1 Processor -- start 2024-07-27T06:46:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.918+0000 7fddd94a0740 1 -- start start 2024-07-27T06:46:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.918+0000 7fddd94a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 0x55cba1186380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.918+0000 7fddd94a0740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0db80a0 0x55cba11868c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.918+0000 7fddd94a0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0dc0280 0x55cba1186e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.918+0000 7fddd94a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cba10fc920 con 0x55cba0de0f60 2024-07-27T06:46:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.918+0000 7fddd94a0740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55cba10ebe20 con 0x55cba0db80a0 2024-07-27T06:46:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.918+0000 7fddd94a0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cba10efe60 con 0x55cba0dc0280 2024-07-27T06:46:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddd14b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 0x55cba1186380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddd14b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 0x55cba1186380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43500/0 (socket says 172.21.15.29:43500) 2024-07-27T06:46:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddd14b9640 1 -- 172.21.15.29:0/2756094895 learned_addr learned my addr 172.21.15.29:0/2756094895 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddd0cb8640 1 --2- 172.21.15.29:0/2756094895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0db80a0 0x55cba11868c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddcbfff640 1 --2- 172.21.15.29:0/2756094895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0dc0280 0x55cba1186e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddd14b9640 1 -- 172.21.15.29:0/2756094895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0dc0280 msgr2=0x55cba1186e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddd14b9640 1 --2- 172.21.15.29:0/2756094895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0dc0280 0x55cba1186e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddd14b9640 1 -- 172.21.15.29:0/2756094895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0db80a0 msgr2=0x55cba11868c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddd14b9640 1 --2- 172.21.15.29:0/2756094895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0db80a0 0x55cba11868c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddd14b9640 1 -- 172.21.15.29:0/2756094895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cba1092a40 con 0x55cba0de0f60 2024-07-27T06:46:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddd0cb8640 1 --2- 172.21.15.29:0/2756094895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0db80a0 0x55cba11868c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddc9ffb640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fddb40b3ca0 0x7fddb40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddd0cb8640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fddb40b3ca0 0x7fddb40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddd14b9640 1 --2- 172.21.15.29:0/2756094895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 0x55cba1186380 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fddc4010370 tx=0x7fddc4010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4f3ab1c5f01807b server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddc9ffb640 1 -- 172.21.15.29:0/1948550396 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fddbc0e3e50 con 0x7fddcc1024b0 2024-07-27T06:46:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.919+0000 7fddb2ffd640 1 -- 172.21.15.29:0/2756094895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc4016070 con 0x55cba0de0f60 2024-07-27T06:46:36.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.920+0000 7fddb2ffd640 1 -- 172.21.15.29:0/2756094895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddc4004d20 con 0x55cba0de0f60 2024-07-27T06:46:36.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.920+0000 7fddb2ffd640 1 -- 172.21.15.29:0/2756094895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc4005350 con 0x55cba0de0f60 2024-07-27T06:46:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.920+0000 7fddd94a0740 1 -- 172.21.15.29:0/2756094895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 msgr2=0x55cba1186380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.920+0000 7fddd94a0740 1 --2- 172.21.15.29:0/2756094895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 0x55cba1186380 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fddc4010370 tx=0x7fddc4010830 comp rx=0 tx=0).stop 2024-07-27T06:46:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.920+0000 7fddd0cb8640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fddb40b3ca0 0x7fddb40b61b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55cba1185000 tx=0x7fddbc0e7660 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.920+0000 7fddd94a0740 1 -- 172.21.15.29:0/2756094895 shutdown_connections 2024-07-27T06:46:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.921+0000 7fddd94a0740 1 --2- 172.21.15.29:0/2756094895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0dc0280 0x55cba1186e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.921+0000 7fddd94a0740 1 --2- 172.21.15.29:0/2756094895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0db80a0 0x55cba11868c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.921+0000 7fddd94a0740 1 --2- 172.21.15.29:0/2756094895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 0x55cba1186380 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.921+0000 7fddd94a0740 1 -- 172.21.15.29:0/2756094895 >> 172.21.15.29:0/2756094895 conn(0x55cba0dd5350 msgr2=0x55cba1184270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.921+0000 7fddd94a0740 1 -- 172.21.15.29:0/2756094895 shutdown_connections 2024-07-27T06:46:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.921+0000 7fddd94a0740 1 -- 172.21.15.29:0/2756094895 wait complete. 2024-07-27T06:46:36.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.921+0000 7fddd94a0740 1 -- start start 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.921+0000 7fddd94a0740 1 Processor -- start 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd94a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 0x55cba1191f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd94a0740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0dc0280 0x55cba11924a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd94a0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0de0f60 0x55cba119de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd94a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cba0f4f5f0 con 0x55cba0db80a0 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd94a0740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55cba108e890 con 0x55cba0dc0280 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd94a0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cba1194b60 con 0x55cba0de0f60 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd94a0740 10 client.0 _ll_register_callbacks cb 0x55cba10987d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd14b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 0x55cba1191f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd14b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 0x55cba1191f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43510/0 (socket says 172.21.15.29:43510) 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd14b9640 1 -- 172.21.15.29:0/3967098132 learned_addr learned my addr 172.21.15.29:0/3967098132 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd14b9640 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0de0f60 0x55cba119de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddcbfff640 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0dc0280 0x55cba11924a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd14b9640 1 -- 172.21.15.29:0/3967098132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0de0f60 msgr2=0x55cba119de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd14b9640 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0de0f60 0x55cba119de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd14b9640 1 -- 172.21.15.29:0/3967098132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0dc0280 msgr2=0x55cba11924a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd14b9640 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0dc0280 0x55cba11924a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd14b9640 1 -- 172.21.15.29:0/3967098132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cba0e4c540 con 0x55cba0db80a0 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddcbfff640 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0dc0280 0x55cba11924a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd14b9640 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 0x55cba1191f60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fddc4004ff0 tx=0x7fddc4010fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f513cbd7f9220f server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddabfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddabfff640 1 -- 172.21.15.29:0/3967098132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc4016070 con 0x55cba0db80a0 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddabfff640 1 -- 172.21.15.29:0/3967098132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fddc4022040 con 0x55cba0db80a0 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.922+0000 7fddd94a0740 1 -- 172.21.15.29:0/3967098132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddc4004d20 con 0x55cba0db80a0 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.923+0000 7fddabfff640 1 -- 172.21.15.29:0/3967098132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fddc40128b0 con 0x55cba0db80a0 2024-07-27T06:46:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.923+0000 7fddd94a0740 10 client.15345 fetch_fsmap learned FSMap version 28 2024-07-27T06:46:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.923+0000 7fddd94a0740 1 -- 172.21.15.29:0/3967098132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cba11a0b40 con 0x55cba0db80a0 2024-07-27T06:46:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.923+0000 7fddabfff640 1 -- 172.21.15.29:0/3967098132 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddc4004d20 con 0x55cba0db80a0 2024-07-27T06:46:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.923+0000 7fddabfff640 1 -- 172.21.15.29:0/3967098132 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fddc400f040 con 0x55cba0db80a0 2024-07-27T06:46:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.923+0000 7fddd94a0740 10 client.15345 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:46:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.923+0000 7fddd94a0740 10 client.15345 resolve_mds: resolved name 'f' to daemon mds.f 2024-07-27T06:46:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.923+0000 7fddd94a0740 20 client.15345 populate_metadata read hostname 'smithi029' 2024-07-27T06:46:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.923+0000 7fddd94a0740 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55cba0d9f8e0 0x55cba11a4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.923+0000 7fddd94a0740 4 client.15345 mds_command: new command op to 24646 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:46:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.923+0000 7fddd94a0740 1 -- 172.21.15.29:0/3967098132 --> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cba10fe870 con 0x55cba0d9f8e0 2024-07-27T06:46:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.923+0000 7fddcbfff640 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55cba0d9f8e0 0x55cba11a4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.924+0000 7fddabfff640 10 client.15345 ms_handle_connect on v2:172.21.15.77:6834/439792826 2024-07-27T06:46:36.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.924+0000 7fddcbfff640 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55cba0d9f8e0 0x55cba11a4290 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a170ca78e238e31e server_cookie=4a9b67a396f5f2e3 in_seq=0 out_seq=0 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddabfff640 1 -- 172.21.15.29:0/3967098132 <== mds.2 v2:172.21.15.77:6834/439792826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cba10fe870 con 0x55cba0d9f8e0 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddabfff640 10 client.15345 handle_command_reply: tid=0 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddd94a0740 1 client.15345 shutdown 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddd94a0740 1 -- 172.21.15.29:0/3967098132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 msgr2=0x55cba1191f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddd94a0740 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 0x55cba1191f60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fddc4004ff0 tx=0x7fddc4010fa0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddd94a0740 1 -- 172.21.15.29:0/3967098132 shutdown_connections 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddd94a0740 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55cba0d9f8e0 0x55cba11a4290 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddd94a0740 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0de0f60 0x55cba119de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddd94a0740 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0dc0280 0x55cba11924a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddd94a0740 1 --2- 172.21.15.29:0/3967098132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 0x55cba1191f60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddd94a0740 1 -- 172.21.15.29:0/3967098132 >> 172.21.15.29:0/3967098132 conn(0x55cba0dd5350 msgr2=0x55cba1189560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddabfff640 0 client.15345 ms_handle_reset on v2:172.21.15.77:6834/439792826 2024-07-27T06:46:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddd94a0740 1 -- 172.21.15.29:0/3967098132 shutdown_connections 2024-07-27T06:46:36.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddd94a0740 1 -- 172.21.15.29:0/3967098132 wait complete. 2024-07-27T06:46:36.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.925+0000 7fddd94a0740 20 client.15345 trim_cache size 0 max 16384 2024-07-27T06:46:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.943+0000 7fddd94a0740 1 Processor -- start 2024-07-27T06:46:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.943+0000 7fddd94a0740 1 -- start start 2024-07-27T06:46:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.943+0000 7fddd94a0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0db80a0 0x55cba119ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.943+0000 7fddd94a0740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0dc0280 0x55cba1191250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.943+0000 7fddd94a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 0x55cba118cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd94a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cba11a25d0 con 0x55cba0de0f60 2024-07-27T06:46:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd94a0740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55cba11a2ca0 con 0x55cba0dc0280 2024-07-27T06:46:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd94a0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cba11a3370 con 0x55cba0db80a0 2024-07-27T06:46:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd14b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 0x55cba118cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd14b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 0x55cba118cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43524/0 (socket says 172.21.15.29:43524) 2024-07-27T06:46:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd14b9640 1 -- 172.21.15.29:0/1727309641 learned_addr learned my addr 172.21.15.29:0/1727309641 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd14b9640 1 --2- 172.21.15.29:0/1727309641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0db80a0 0x55cba119ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddcbfff640 1 --2- 172.21.15.29:0/1727309641 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0dc0280 0x55cba1191250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd14b9640 1 -- 172.21.15.29:0/1727309641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0db80a0 msgr2=0x55cba119ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd14b9640 1 --2- 172.21.15.29:0/1727309641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0db80a0 0x55cba119ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd14b9640 1 -- 172.21.15.29:0/1727309641 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0dc0280 msgr2=0x55cba1191250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd14b9640 1 --2- 172.21.15.29:0/1727309641 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0dc0280 0x55cba1191250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd14b9640 1 -- 172.21.15.29:0/1727309641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cba1192af0 con 0x55cba0de0f60 2024-07-27T06:46:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddcbfff640 1 --2- 172.21.15.29:0/1727309641 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0dc0280 0x55cba1191250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:46:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddd14b9640 1 --2- 172.21.15.29:0/1727309641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 0x55cba118cea0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009ac0 tx=0x7fddc4010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5588d1abc0b5e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddb27fc640 1 -- 172.21.15.29:0/1727309641 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc4016040 con 0x55cba0de0f60 2024-07-27T06:46:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddb27fc640 1 -- 172.21.15.29:0/1727309641 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddc4029d30 con 0x55cba0de0f60 2024-07-27T06:46:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.944+0000 7fddb27fc640 1 -- 172.21.15.29:0/1727309641 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc4012a50 con 0x55cba0de0f60 2024-07-27T06:46:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.945+0000 7fddd94a0740 1 -- 172.21.15.29:0/1727309641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 msgr2=0x55cba118cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.945+0000 7fddd94a0740 1 --2- 172.21.15.29:0/1727309641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 0x55cba118cea0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009ac0 tx=0x7fddc4010830 comp rx=0 tx=0).stop 2024-07-27T06:46:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.945+0000 7fddd94a0740 1 -- 172.21.15.29:0/1727309641 shutdown_connections 2024-07-27T06:46:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.945+0000 7fddd94a0740 1 --2- 172.21.15.29:0/1727309641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0de0f60 0x55cba118cea0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.945+0000 7fddd94a0740 1 --2- 172.21.15.29:0/1727309641 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0dc0280 0x55cba1191250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.945+0000 7fddd94a0740 1 --2- 172.21.15.29:0/1727309641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0db80a0 0x55cba119ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.945+0000 7fddd94a0740 1 -- 172.21.15.29:0/1727309641 >> 172.21.15.29:0/1727309641 conn(0x55cba0dd5350 msgr2=0x55cba1191f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.945+0000 7fddd94a0740 1 -- 172.21.15.29:0/1727309641 shutdown_connections 2024-07-27T06:46:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.945+0000 7fddd94a0740 1 -- 172.21.15.29:0/1727309641 wait complete. 2024-07-27T06:46:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.946+0000 7fddd94a0740 1 -- start start 2024-07-27T06:46:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.946+0000 7fddd94a0740 1 Processor -- start 2024-07-27T06:46:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.946+0000 7fddd94a0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 0x55cba11a5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.946+0000 7fddd94a0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0dc0280 0x55cba11a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.946+0000 7fddd94a0740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0de0f60 0x55cba11ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.946+0000 7fddd94a0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55cba11a4110 con 0x55cba0db80a0 2024-07-27T06:46:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.946+0000 7fddd94a0740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55cba11a3a40 con 0x55cba0de0f60 2024-07-27T06:46:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.946+0000 7fddd94a0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cba11a47e0 con 0x55cba0dc0280 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.946+0000 7fddd94a0740 10 client.0 _ll_register_callbacks cb 0x7fddc4004c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddd14b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 0x55cba11a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddd14b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 0x55cba11a5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43540/0 (socket says 172.21.15.29:43540) 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddd14b9640 1 -- 172.21.15.29:0/2807150970 learned_addr learned my addr 172.21.15.29:0/2807150970 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddcbfff640 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0dc0280 0x55cba11a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddd14b9640 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0de0f60 0x55cba11ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddd14b9640 1 -- 172.21.15.29:0/2807150970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0dc0280 msgr2=0x55cba11a5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddd14b9640 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0dc0280 0x55cba11a5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddd14b9640 1 -- 172.21.15.29:0/2807150970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0de0f60 msgr2=0x55cba11ac4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddd14b9640 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0de0f60 0x55cba11ac4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddd14b9640 1 -- 172.21.15.29:0/2807150970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cba11acb00 con 0x55cba0db80a0 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddd14b9640 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 0x55cba11a5690 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fddc40098b0 tx=0x7fddc4009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9286ddfe5e97c8e9 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddabfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddabfff640 1 -- 172.21.15.29:0/2807150970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc4016040 con 0x55cba0db80a0 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddd94a0740 1 -- 172.21.15.29:0/2807150970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddc4029d30 con 0x55cba0db80a0 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.947+0000 7fddabfff640 1 -- 172.21.15.29:0/2807150970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fddc4029a90 con 0x55cba0db80a0 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.948+0000 7fddabfff640 1 -- 172.21.15.29:0/2807150970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fddc4012d00 con 0x55cba0db80a0 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.948+0000 7fddd94a0740 10 client.15351 fetch_fsmap learned FSMap version 28 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.948+0000 7fddd94a0740 1 -- 172.21.15.29:0/2807150970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cba120bea0 con 0x55cba0db80a0 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.948+0000 7fddabfff640 1 -- 172.21.15.29:0/2807150970 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddc4029d30 con 0x55cba0db80a0 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.948+0000 7fddabfff640 1 -- 172.21.15.29:0/2807150970 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fddc4031410 con 0x55cba0db80a0 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.948+0000 7fddd94a0740 10 client.15351 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.948+0000 7fddd94a0740 10 client.15351 resolve_mds: resolved name 'f' to daemon mds.f 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.948+0000 7fddd94a0740 20 client.15351 populate_metadata read hostname 'smithi029' 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.948+0000 7fddd94a0740 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55cba0d74c90 0x55cba120f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.948+0000 7fddd94a0740 4 client.15351 mds_command: new command op to 24646 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:46:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.948+0000 7fddd94a0740 1 -- 172.21.15.29:0/2807150970 --> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] -- command(tid 0: {"prefix": "status"}) -- 0x55cba10fa310 con 0x55cba0d74c90 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.948+0000 7fddcbfff640 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55cba0d74c90 0x55cba120f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddcbfff640 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55cba0d74c90 0x55cba120f470 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2cd087c829c34914 server_cookie=ee6efa07b609e360 in_seq=0 out_seq=0 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddabfff640 10 client.15351 ms_handle_connect on v2:172.21.15.77:6834/439792826 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddabfff640 1 -- 172.21.15.29:0/2807150970 <== mds.2 v2:172.21.15.77:6834/439792826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55cba10fa310 con 0x55cba0d74c90 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddabfff640 10 client.15351 handle_command_reply: tid=0 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddd94a0740 1 client.15351 shutdown 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddd94a0740 1 -- 172.21.15.29:0/2807150970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 msgr2=0x55cba11a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddd94a0740 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 0x55cba11a5690 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fddc40098b0 tx=0x7fddc4009800 comp rx=0 tx=0).stop 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddd94a0740 1 -- 172.21.15.29:0/2807150970 shutdown_connections 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddd94a0740 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55cba0d74c90 0x55cba120f470 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddd94a0740 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55cba0de0f60 0x55cba11ac4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddd94a0740 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cba0dc0280 0x55cba11a5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddd94a0740 1 --2- 172.21.15.29:0/2807150970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55cba0db80a0 0x55cba11a5690 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddd94a0740 1 -- 172.21.15.29:0/2807150970 >> 172.21.15.29:0/2807150970 conn(0x55cba0dd5350 msgr2=0x55cba1190540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.949+0000 7fddabfff640 0 client.15351 ms_handle_reset on v2:172.21.15.77:6834/439792826 2024-07-27T06:46:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.950+0000 7fddd94a0740 1 -- 172.21.15.29:0/2807150970 shutdown_connections 2024-07-27T06:46:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.950+0000 7fddd94a0740 1 -- 172.21.15.29:0/2807150970 wait complete. 2024-07-27T06:46:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.950+0000 7fddd94a0740 20 client.15351 trim_cache size 0 max 16384 2024-07-27T06:46:36.957 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:46:36.957 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:46:36.957 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 2, 2024-07-27T06:46:36.957 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24646, 2024-07-27T06:46:36.958 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:46:36.958 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:46:36.958 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:46:36.958 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 186.107201425, 2024-07-27T06:46:36.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-07-27T06:46:36.958 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:46:36.958 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 79, 2024-07-27T06:46:36.958 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 237.87963429600001 2024-07-27T06:46:36.958 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:46:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.951+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1948550396 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fddb40b3ca0 msgr2=0x7fddb40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.951+0000 7fddd2f1c640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fddb40b3ca0 0x7fddb40b61b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55cba1185000 tx=0x7fddbc0e7660 comp rx=0 tx=0).stop 2024-07-27T06:46:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.951+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1948550396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc1024b0 msgr2=0x7fddcc198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.951+0000 7fddd2f1c640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc1024b0 0x7fddcc198ad0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fddbc00c920 tx=0x7fddbc00cde0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.951+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1948550396 shutdown_connections 2024-07-27T06:46:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.951+0000 7fddd2f1c640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fddb40b3ca0 0x7fddb40b61b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.951+0000 7fddd2f1c640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fddcc108740 0x7fddcc199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.951+0000 7fddd2f1c640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddcc107df0 0x7fddcc199010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.951+0000 7fddd2f1c640 1 --2- 172.21.15.29:0/1948550396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddcc1024b0 0x7fddcc198ad0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.951+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1948550396 >> 172.21.15.29:0/1948550396 conn(0x7fddcc079170 msgr2=0x7fddcc104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.951+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1948550396 shutdown_connections 2024-07-27T06:46:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:36.951+0000 7fddd2f1c640 1 -- 172.21.15.29:0/1948550396 wait complete. 2024-07-27T06:46:37.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:37 smithi029 ceph-mon[32610]: pgmap v385: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:37.665 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.c status 2024-07-27T06:46:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:37 smithi047 ceph-mon[36835]: pgmap v385: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:37 smithi077 ceph-mon[37153]: pgmap v385: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:37.937 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:39.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.627+0000 7fa2f1119640 1 -- 172.21.15.29:0/2730521985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec075800 msgr2=0x7fa2ec075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:39.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.627+0000 7fa2f1119640 1 --2- 172.21.15.29:0/2730521985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec075800 0x7fa2ec075c10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009900 tx=0x7fa2e002a410 comp rx=0 tx=0).stop 2024-07-27T06:46:39.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.627+0000 7fa2f1119640 1 -- 172.21.15.29:0/2730521985 shutdown_connections 2024-07-27T06:46:39.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.627+0000 7fa2f1119640 1 --2- 172.21.15.29:0/2730521985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2ec113e70 0x7fa2ec116340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.627+0000 7fa2f1119640 1 --2- 172.21.15.29:0/2730521985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec075800 0x7fa2ec075c10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.627+0000 7fa2f1119640 1 --2- 172.21.15.29:0/2730521985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2ec076db0 0x7fa2ec0751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.627+0000 7fa2f1119640 1 -- 172.21.15.29:0/2730521985 >> 172.21.15.29:0/2730521985 conn(0x7fa2ec0ff010 msgr2=0x7fa2ec101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:39.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.627+0000 7fa2f1119640 1 -- 172.21.15.29:0/2730521985 shutdown_connections 2024-07-27T06:46:39.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.627+0000 7fa2f1119640 1 -- 172.21.15.29:0/2730521985 wait complete. 2024-07-27T06:46:39.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.628+0000 7fa2f1119640 1 Processor -- start 2024-07-27T06:46:39.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.628+0000 7fa2f1119640 1 -- start start 2024-07-27T06:46:39.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2f1119640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2ec075800 0x7fa2ec1a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2f1119640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec076db0 0x7fa2ec1a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2f1119640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2ec113e70 0x7fa2ec1a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2f1119640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec119ab0 con 0x7fa2ec076db0 2024-07-27T06:46:39.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2f1119640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec119930 con 0x7fa2ec075800 2024-07-27T06:46:39.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2f1119640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec119c30 con 0x7fa2ec113e70 2024-07-27T06:46:39.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2ea59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec076db0 0x7fa2ec1a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2ea59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec076db0 0x7fa2ec1a0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57990/0 (socket says 172.21.15.29:57990) 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2ea59c640 1 -- 172.21.15.29:0/2082480861 learned_addr learned my addr 172.21.15.29:0/2082480861 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2ead9d640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2ec075800 0x7fa2ec1a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2eb59e640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2ec113e70 0x7fa2ec1a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2ea59c640 1 -- 172.21.15.29:0/2082480861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2ec113e70 msgr2=0x7fa2ec1a5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2ec113e70 0x7fa2ec1a5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2ea59c640 1 -- 172.21.15.29:0/2082480861 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2ec075800 msgr2=0x7fa2ec1a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2ec075800 0x7fa2ec1a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2ea59c640 1 -- 172.21.15.29:0/2082480861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ec1a6450 con 0x7fa2ec076db0 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2ead9d640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2ec075800 0x7fa2ec1a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.629+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec076db0 0x7fa2ec1a0a80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0032fd0 tx=0x7fa2e0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.630+0000 7fa2d3fff640 1 -- 172.21.15.29:0/2082480861 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e003f030 con 0x7fa2ec076db0 2024-07-27T06:46:39.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.630+0000 7fa2f1119640 1 -- 172.21.15.29:0/2082480861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ec1a66e0 con 0x7fa2ec076db0 2024-07-27T06:46:39.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.630+0000 7fa2d3fff640 1 -- 172.21.15.29:0/2082480861 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2e0004490 con 0x7fa2ec076db0 2024-07-27T06:46:39.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.630+0000 7fa2d3fff640 1 -- 172.21.15.29:0/2082480861 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0035070 con 0x7fa2ec076db0 2024-07-27T06:46:39.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.630+0000 7fa2f1119640 1 -- 172.21.15.29:0/2082480861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2ec1acea0 con 0x7fa2ec076db0 2024-07-27T06:46:39.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.632+0000 7fa2f769d740 1 Processor -- start 2024-07-27T06:46:39.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.632+0000 7fa2f769d740 1 -- start start 2024-07-27T06:46:39.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2d3fff640 1 -- 172.21.15.29:0/2082480861 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fa2e0042030 con 0x7fa2ec076db0 2024-07-27T06:46:39.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2f769d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d054f60 0x559b0d3fa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2f769d740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d3fa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2f769d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 0x559b0d3fae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2f769d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559b0d370940 con 0x559b0d034280 2024-07-27T06:46:39.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2f769d740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x559b0d35fe40 con 0x559b0d02c0a0 2024-07-27T06:46:39.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2f769d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559b0d363e80 con 0x559b0d054f60 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2ea59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 0x559b0d3fae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2ea59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 0x559b0d3fae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57992/0 (socket says 172.21.15.29:57992) 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2ea59c640 1 -- 172.21.15.29:0/4263268912 learned_addr learned my addr 172.21.15.29:0/4263268912 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2eb59e640 1 --2- 172.21.15.29:0/4263268912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d054f60 0x559b0d3fa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2ead9d640 1 --2- 172.21.15.29:0/4263268912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d3fa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2ea59c640 1 -- 172.21.15.29:0/4263268912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d054f60 msgr2=0x559b0d3fa3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/4263268912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d054f60 0x559b0d3fa3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2ea59c640 1 -- 172.21.15.29:0/4263268912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 msgr2=0x559b0d3fa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/4263268912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d3fa8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.633+0000 7fa2ea59c640 1 -- 172.21.15.29:0/4263268912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b0d306a60 con 0x559b0d034280 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.634+0000 7fa2d3fff640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa2d40b3c20 0x7fa2d40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.634+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/4263268912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 0x559b0d3fae20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa2e010c540 tx=0x7fa2e010c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0cfb2c364a26d4 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:39.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.634+0000 7fa2eb59e640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa2d40b3c20 0x7fa2d40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.634+0000 7fa2d17fa640 1 -- 172.21.15.29:0/4263268912 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e00c7050 con 0x559b0d034280 2024-07-27T06:46:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.634+0000 7fa2d3fff640 1 -- 172.21.15.29:0/2082480861 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fa2e0104350 con 0x7fa2ec076db0 2024-07-27T06:46:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.634+0000 7fa2d17fa640 1 -- 172.21.15.29:0/4263268912 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2e00caca0 con 0x559b0d034280 2024-07-27T06:46:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.634+0000 7fa2d17fa640 1 -- 172.21.15.29:0/4263268912 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e00ca900 con 0x559b0d034280 2024-07-27T06:46:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.635+0000 7fa2f769d740 1 -- 172.21.15.29:0/4263268912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 msgr2=0x559b0d3fae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.635+0000 7fa2eb59e640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa2d40b3c20 0x7fa2d40b6130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4005fd0 tx=0x7fa2e4009740 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.635+0000 7fa2f769d740 1 --2- 172.21.15.29:0/4263268912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 0x559b0d3fae20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa2e010c540 tx=0x7fa2e010c630 comp rx=0 tx=0).stop 2024-07-27T06:46:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.635+0000 7fa2f769d740 1 -- 172.21.15.29:0/4263268912 shutdown_connections 2024-07-27T06:46:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.635+0000 7fa2f769d740 1 --2- 172.21.15.29:0/4263268912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 0x559b0d3fae20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.635+0000 7fa2f769d740 1 --2- 172.21.15.29:0/4263268912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d3fa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.635+0000 7fa2f769d740 1 --2- 172.21.15.29:0/4263268912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d054f60 0x559b0d3fa3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.635+0000 7fa2f769d740 1 -- 172.21.15.29:0/4263268912 >> 172.21.15.29:0/4263268912 conn(0x559b0d049350 msgr2=0x559b0d3f8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.635+0000 7fa2f769d740 1 -- 172.21.15.29:0/4263268912 shutdown_connections 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.635+0000 7fa2f769d740 1 -- 172.21.15.29:0/4263268912 wait complete. 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.635+0000 7fa2f769d740 1 -- start start 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2f769d740 1 Processor -- start 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2f769d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d02c0a0 0x559b0d405f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2f769d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d034280 0x559b0d4064a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2f769d740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d054f60 0x559b0d411ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2f769d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559b0d1c35f0 con 0x559b0d02c0a0 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2f769d740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x559b0d3028b0 con 0x559b0d054f60 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2f769d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559b0d408940 con 0x559b0d034280 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2f769d740 10 client.0 _ll_register_callbacks cb 0x559b0d30c7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2ead9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d02c0a0 0x559b0d405f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2ead9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d02c0a0 0x559b0d405f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58006/0 (socket says 172.21.15.29:58006) 2024-07-27T06:46:39.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2ead9d640 1 -- 172.21.15.29:0/3412379001 learned_addr learned my addr 172.21.15.29:0/3412379001 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d034280 0x559b0d4064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2eb59e640 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d054f60 0x559b0d411ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2ead9d640 1 -- 172.21.15.29:0/3412379001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d034280 msgr2=0x559b0d4064a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2ead9d640 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d034280 0x559b0d4064a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2ead9d640 1 -- 172.21.15.29:0/3412379001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d054f60 msgr2=0x559b0d411ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2ead9d640 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d054f60 0x559b0d411ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2ead9d640 1 -- 172.21.15.29:0/3412379001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b0d0c0540 con 0x559b0d02c0a0 2024-07-27T06:46:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2eb59e640 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d054f60 0x559b0d411ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:46:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.636+0000 7fa2ead9d640 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d02c0a0 0x559b0d405f60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00bfb0 tx=0x7fa2dc0128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ccd84b3de7f313 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:46:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2ceff5640 1 -- 172.21.15.29:0/3412379001 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc0035f0 con 0x559b0d02c0a0 2024-07-27T06:46:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2ceff5640 1 -- 172.21.15.29:0/3412379001 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fa2dc003790 con 0x559b0d02c0a0 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2f769d740 1 -- 172.21.15.29:0/3412379001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2e00caca0 con 0x559b0d02c0a0 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2ceff5640 1 -- 172.21.15.29:0/3412379001 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa2dc003990 con 0x559b0d02c0a0 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2f769d740 10 client.15363 fetch_fsmap learned FSMap version 28 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2f769d740 1 -- 172.21.15.29:0/3412379001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559b0d414930 con 0x559b0d02c0a0 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2ceff5640 1 -- 172.21.15.29:0/3412379001 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2e00caca0 con 0x559b0d02c0a0 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2ceff5640 1 -- 172.21.15.29:0/3412379001 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa2dc019070 con 0x559b0d02c0a0 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2f769d740 10 client.15363 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2f769d740 10 client.15363 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2f769d740 20 client.15363 populate_metadata read hostname 'smithi029' 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2f769d740 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x559b0d0138e0 0x559b0d418460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2f769d740 4 client.15363 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.637+0000 7fa2f769d740 1 -- 172.21.15.29:0/3412379001 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b0d372890 con 0x559b0d0138e0 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.638+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x559b0d0138e0 0x559b0d418460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.638+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x559b0d0138e0 0x559b0d418460 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1257cdf33bc5775b server_cookie=826b53259e948aa9 in_seq=0 out_seq=0 2024-07-27T06:46:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.638+0000 7fa2ceff5640 10 client.15363 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:46:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2ceff5640 1 -- 172.21.15.29:0/3412379001 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559b0d372890 con 0x559b0d0138e0 2024-07-27T06:46:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2ceff5640 10 client.15363 handle_command_reply: tid=0 2024-07-27T06:46:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2f769d740 1 client.15363 shutdown 2024-07-27T06:46:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2f769d740 1 -- 172.21.15.29:0/3412379001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d02c0a0 msgr2=0x559b0d405f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2f769d740 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d02c0a0 0x559b0d405f60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00bfb0 tx=0x7fa2dc0128e0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2f769d740 1 -- 172.21.15.29:0/3412379001 shutdown_connections 2024-07-27T06:46:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2f769d740 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x559b0d0138e0 0x559b0d418460 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2f769d740 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d054f60 0x559b0d411ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2f769d740 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d034280 0x559b0d4064a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2ceff5640 0 client.15363 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:46:39.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2f769d740 1 --2- 172.21.15.29:0/3412379001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d02c0a0 0x559b0d405f60 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2f769d740 1 -- 172.21.15.29:0/3412379001 >> 172.21.15.29:0/3412379001 conn(0x559b0d049350 msgr2=0x559b0d3fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:39.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2f769d740 1 -- 172.21.15.29:0/3412379001 shutdown_connections 2024-07-27T06:46:39.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2f769d740 1 -- 172.21.15.29:0/3412379001 wait complete. 2024-07-27T06:46:39.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.639+0000 7fa2f769d740 20 client.15363 trim_cache size 0 max 16384 2024-07-27T06:46:39.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.657+0000 7fa2f769d740 1 Processor -- start 2024-07-27T06:46:39.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.657+0000 7fa2f769d740 1 -- start start 2024-07-27T06:46:39.659 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:39 smithi029 ceph-mon[32610]: pgmap v386: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2f769d740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d405250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2f769d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 0x559b0d411990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2f769d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d054f60 0x559b0d4045e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2f769d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559b0d4167a0 con 0x559b0d034280 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2f769d740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x559b0d416e70 con 0x559b0d02c0a0 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2f769d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559b0d417540 con 0x559b0d054f60 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2ea59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 0x559b0d411990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2ea59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 0x559b0d411990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58014/0 (socket says 172.21.15.29:58014) 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2ea59c640 1 -- 172.21.15.29:0/3033289425 learned_addr learned my addr 172.21.15.29:0/3033289425 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2eb59e640 1 --2- 172.21.15.29:0/3033289425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d054f60 0x559b0d4045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2ead9d640 1 --2- 172.21.15.29:0/3033289425 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d405250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2ea59c640 1 -- 172.21.15.29:0/3033289425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d054f60 msgr2=0x559b0d4045e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/3033289425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d054f60 0x559b0d4045e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2ea59c640 1 -- 172.21.15.29:0/3033289425 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 msgr2=0x559b0d405250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/3033289425 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d405250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2ea59c640 1 -- 172.21.15.29:0/3033289425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b0d4015f0 con 0x559b0d034280 2024-07-27T06:46:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.658+0000 7fa2ead9d640 1 --2- 172.21.15.29:0/3033289425 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d405250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.659+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/3033289425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 0x559b0d411990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0003fd0 tx=0x7fa2e002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b6df20530c9251 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.659+0000 7fa2d0ff9640 1 -- 172.21.15.29:0/3033289425 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0049040 con 0x559b0d034280 2024-07-27T06:46:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.659+0000 7fa2d0ff9640 1 -- 172.21.15.29:0/3033289425 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2e00caca0 con 0x559b0d034280 2024-07-27T06:46:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.659+0000 7fa2d0ff9640 1 -- 172.21.15.29:0/3033289425 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e00caae0 con 0x559b0d034280 2024-07-27T06:46:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.660+0000 7fa2f769d740 1 -- 172.21.15.29:0/3033289425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 msgr2=0x559b0d411990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.660+0000 7fa2f769d740 1 --2- 172.21.15.29:0/3033289425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 0x559b0d411990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0003fd0 tx=0x7fa2e002a410 comp rx=0 tx=0).stop 2024-07-27T06:46:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.660+0000 7fa2f769d740 1 -- 172.21.15.29:0/3033289425 shutdown_connections 2024-07-27T06:46:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.660+0000 7fa2f769d740 1 --2- 172.21.15.29:0/3033289425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d054f60 0x559b0d4045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.660+0000 7fa2f769d740 1 --2- 172.21.15.29:0/3033289425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d034280 0x559b0d411990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.660+0000 7fa2f769d740 1 --2- 172.21.15.29:0/3033289425 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d405250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.660+0000 7fa2f769d740 1 -- 172.21.15.29:0/3033289425 >> 172.21.15.29:0/3033289425 conn(0x559b0d049350 msgr2=0x559b0d405f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.660+0000 7fa2f769d740 1 -- 172.21.15.29:0/3033289425 shutdown_connections 2024-07-27T06:46:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.660+0000 7fa2f769d740 1 -- 172.21.15.29:0/3033289425 wait complete. 2024-07-27T06:46:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.660+0000 7fa2f769d740 1 -- start start 2024-07-27T06:46:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.660+0000 7fa2f769d740 1 Processor -- start 2024-07-27T06:46:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2f769d740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d420ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2f769d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d034280 0x559b0d41ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2f769d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d054f60 0x559b0d41f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2f769d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559b0d4182e0 con 0x559b0d054f60 2024-07-27T06:46:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2f769d740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x559b0d417c10 con 0x559b0d02c0a0 2024-07-27T06:46:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2f769d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559b0d4189b0 con 0x559b0d034280 2024-07-27T06:46:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2f769d740 10 client.0 _ll_register_callbacks cb 0x7fa2dc00bbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2eb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d054f60 0x559b0d41f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2eb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d054f60 0x559b0d41f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58018/0 (socket says 172.21.15.29:58018) 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2eb59e640 1 -- 172.21.15.29:0/2144637852 learned_addr learned my addr 172.21.15.29:0/2144637852 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2ea59c640 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d034280 0x559b0d41ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2ead9d640 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d420ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2eb59e640 1 -- 172.21.15.29:0/2144637852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d034280 msgr2=0x559b0d41ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2eb59e640 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d034280 0x559b0d41ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2eb59e640 1 -- 172.21.15.29:0/2144637852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 msgr2=0x559b0d420ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2eb59e640 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d420ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.661+0000 7fa2eb59e640 1 -- 172.21.15.29:0/2144637852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b0d419660 con 0x559b0d054f60 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.662+0000 7fa2eb59e640 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d054f60 0x559b0d41f430 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa2e401db40 tx=0x7fa2e4014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1889a966897ac5f3 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.662+0000 7fa2ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:46:39.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.662+0000 7fa2ceff5640 1 -- 172.21.15.29:0/2144637852 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4004b20 con 0x559b0d054f60 2024-07-27T06:46:39.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.662+0000 7fa2ceff5640 1 -- 172.21.15.29:0/2144637852 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fa2e4004cc0 con 0x559b0d054f60 2024-07-27T06:46:39.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.662+0000 7fa2f769d740 1 -- 172.21.15.29:0/2144637852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2e00caca0 con 0x559b0d054f60 2024-07-27T06:46:39.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.662+0000 7fa2ceff5640 1 -- 172.21.15.29:0/2144637852 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa2e4035050 con 0x559b0d054f60 2024-07-27T06:46:39.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.662+0000 7fa2f769d740 10 client.15369 fetch_fsmap learned FSMap version 28 2024-07-27T06:46:39.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.662+0000 7fa2f769d740 1 -- 172.21.15.29:0/2144637852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559b0d41ff80 con 0x559b0d054f60 2024-07-27T06:46:39.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.662+0000 7fa2ceff5640 1 -- 172.21.15.29:0/2144637852 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2e401ecc0 con 0x559b0d054f60 2024-07-27T06:46:39.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.662+0000 7fa2ceff5640 1 -- 172.21.15.29:0/2144637852 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa2e400f040 con 0x559b0d054f60 2024-07-27T06:46:39.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.662+0000 7fa2f769d740 10 client.15369 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:46:39.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.663+0000 7fa2f769d740 10 client.15369 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-27T06:46:39.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.663+0000 7fa2f769d740 20 client.15369 populate_metadata read hostname 'smithi029' 2024-07-27T06:46:39.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.663+0000 7fa2f769d740 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x559b0cfe8c90 0x559b0d4842a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:39.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.663+0000 7fa2f769d740 4 client.15369 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:46:39.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.663+0000 7fa2f769d740 1 -- 172.21.15.29:0/2144637852 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "status"}) -- 0x7fa2e00caca0 con 0x559b0cfe8c90 2024-07-27T06:46:39.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.663+0000 7fa2ead9d640 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x559b0cfe8c90 0x559b0d4842a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:39.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.663+0000 7fa2ceff5640 10 client.15369 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:46:39.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.663+0000 7fa2ead9d640 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x559b0cfe8c90 0x559b0d4842a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fc9065dde5e1e160 server_cookie=6e9e9a3722c4729c in_seq=0 out_seq=0 2024-07-27T06:46:39.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2ceff5640 1 -- 172.21.15.29:0/2144637852 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x7fa2e00caca0 con 0x559b0cfe8c90 2024-07-27T06:46:39.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2ceff5640 10 client.15369 handle_command_reply: tid=0 2024-07-27T06:46:39.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2f769d740 1 client.15369 shutdown 2024-07-27T06:46:39.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2f769d740 1 -- 172.21.15.29:0/2144637852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d054f60 msgr2=0x559b0d41f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:39.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2f769d740 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d054f60 0x559b0d41f430 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa2e401db40 tx=0x7fa2e4014040 comp rx=0 tx=0).stop 2024-07-27T06:46:39.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2f769d740 1 -- 172.21.15.29:0/2144637852 shutdown_connections 2024-07-27T06:46:39.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2f769d740 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x559b0cfe8c90 0x559b0d4842a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2f769d740 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559b0d054f60 0x559b0d41f430 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2ceff5640 0 client.15369 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:46:39.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2f769d740 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559b0d034280 0x559b0d41ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2f769d740 1 --2- 172.21.15.29:0/2144637852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559b0d02c0a0 0x559b0d420ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2f769d740 1 -- 172.21.15.29:0/2144637852 >> 172.21.15.29:0/2144637852 conn(0x559b0d049350 msgr2=0x559b0d4041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:39.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2f769d740 1 -- 172.21.15.29:0/2144637852 shutdown_connections 2024-07-27T06:46:39.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2f769d740 1 -- 172.21.15.29:0/2144637852 wait complete. 2024-07-27T06:46:39.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.664+0000 7fa2f769d740 20 client.15369 trim_cache size 0 max 16384 2024-07-27T06:46:39.676 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:46:39.677 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:46:39.677 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 1, 2024-07-27T06:46:39.677 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24658, 2024-07-27T06:46:39.677 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:46:39.677 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:46:39.678 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:46:39.678 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 189.82817217499999, 2024-07-27T06:46:39.678 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-07-27T06:46:39.678 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:46:39.678 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 79, 2024-07-27T06:46:39.678 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 238.038923138 2024-07-27T06:46:39.679 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:46:39.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.665+0000 7fa2f1119640 1 -- 172.21.15.29:0/2082480861 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa2d40b3c20 msgr2=0x7fa2d40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:39.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.665+0000 7fa2f1119640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa2d40b3c20 0x7fa2d40b6130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4005fd0 tx=0x7fa2e4009740 comp rx=0 tx=0).stop 2024-07-27T06:46:39.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.665+0000 7fa2f1119640 1 -- 172.21.15.29:0/2082480861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec076db0 msgr2=0x7fa2ec1a0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:39.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.665+0000 7fa2f1119640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec076db0 0x7fa2ec1a0a80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0032fd0 tx=0x7fa2e0004270 comp rx=0 tx=0).stop 2024-07-27T06:46:39.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.665+0000 7fa2f1119640 1 -- 172.21.15.29:0/2082480861 shutdown_connections 2024-07-27T06:46:39.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.665+0000 7fa2f1119640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa2d40b3c20 0x7fa2d40b6130 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.665+0000 7fa2f1119640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2ec113e70 0x7fa2ec1a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.665+0000 7fa2f1119640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2ec076db0 0x7fa2ec1a0a80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.665+0000 7fa2f1119640 1 --2- 172.21.15.29:0/2082480861 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa2ec075800 0x7fa2ec1a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:39.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.665+0000 7fa2f1119640 1 -- 172.21.15.29:0/2082480861 >> 172.21.15.29:0/2082480861 conn(0x7fa2ec0ff010 msgr2=0x7fa2ec10fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:39.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.665+0000 7fa2f1119640 1 -- 172.21.15.29:0/2082480861 shutdown_connections 2024-07-27T06:46:39.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:39.665+0000 7fa2f1119640 1 -- 172.21.15.29:0/2082480861 wait complete. 2024-07-27T06:46:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:39 smithi047 ceph-mon[36835]: pgmap v386: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:39 smithi077 ceph-mon[37153]: pgmap v386: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:40.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.i status 2024-07-27T06:46:40.655 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:41.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:41 smithi077 ceph-mon[37153]: pgmap v387: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:41.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:41 smithi047 ceph-mon[36835]: pgmap v387: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:41 smithi029 ceph-mon[32610]: pgmap v387: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:46:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.389+0000 7feae70f0640 1 -- 172.21.15.29:0/2875655740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae0103590 msgr2=0x7feae01039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.389+0000 7feae70f0640 1 --2- 172.21.15.29:0/2875655740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae0103590 0x7feae01039a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fead0009900 tx=0x7fead002a410 comp rx=0 tx=0).stop 2024-07-27T06:46:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.389+0000 7feae70f0640 1 -- 172.21.15.29:0/2875655740 shutdown_connections 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.389+0000 7feae70f0640 1 --2- 172.21.15.29:0/2875655740 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feae0109820 0x7feae0110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.389+0000 7feae70f0640 1 --2- 172.21.15.29:0/2875655740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feae0108ed0 0x7feae01092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.389+0000 7feae70f0640 1 --2- 172.21.15.29:0/2875655740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae0103590 0x7feae01039a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.389+0000 7feae70f0640 1 -- 172.21.15.29:0/2875655740 >> 172.21.15.29:0/2875655740 conn(0x7feae007b280 msgr2=0x7feae007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.389+0000 7feae70f0640 1 -- 172.21.15.29:0/2875655740 shutdown_connections 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.389+0000 7feae70f0640 1 -- 172.21.15.29:0/2875655740 wait complete. 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.390+0000 7feae70f0640 1 Processor -- start 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.390+0000 7feae70f0640 1 -- start start 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.390+0000 7feae70f0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feae0103590 0x7feae0199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.390+0000 7feae70f0640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feae0108ed0 0x7feae019a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.390+0000 7feae70f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae0109820 0x7feae019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.390+0000 7feae70f0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feae01134f0 con 0x7feae0109820 2024-07-27T06:46:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.390+0000 7feae70f0640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7feae0113370 con 0x7feae0108ed0 2024-07-27T06:46:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.390+0000 7feae70f0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feae0113670 con 0x7feae0103590 2024-07-27T06:46:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.390+0000 7feae568d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae0109820 0x7feae019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.390+0000 7feae568d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae0109820 0x7feae019f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58042/0 (socket says 172.21.15.29:58042) 2024-07-27T06:46:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.390+0000 7feae568d640 1 -- 172.21.15.29:0/4128055012 learned_addr learned my addr 172.21.15.29:0/4128055012 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.390+0000 7feae4e8c640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feae0103590 0x7feae0199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.391+0000 7feadffff640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feae0108ed0 0x7feae019a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.391+0000 7feae568d640 1 -- 172.21.15.29:0/4128055012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feae0103590 msgr2=0x7feae0199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.391+0000 7feae568d640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feae0103590 0x7feae0199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.391+0000 7feae568d640 1 -- 172.21.15.29:0/4128055012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feae0108ed0 msgr2=0x7feae019a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.391+0000 7feae568d640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feae0108ed0 0x7feae019a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.391+0000 7feae568d640 1 -- 172.21.15.29:0/4128055012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae019fe10 con 0x7feae0109820 2024-07-27T06:46:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.391+0000 7feae568d640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae0109820 0x7feae019f6e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fead800eaa0 tx=0x7fead800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.391+0000 7feaddffb640 1 -- 172.21.15.29:0/4128055012 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead800cdb0 con 0x7feae0109820 2024-07-27T06:46:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.391+0000 7feaddffb640 1 -- 172.21.15.29:0/4128055012 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fead8005020 con 0x7feae0109820 2024-07-27T06:46:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.391+0000 7feaddffb640 1 -- 172.21.15.29:0/4128055012 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead80116d0 con 0x7feae0109820 2024-07-27T06:46:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.391+0000 7feae70f0640 1 -- 172.21.15.29:0/4128055012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae01a0100 con 0x7feae0109820 2024-07-27T06:46:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.391+0000 7feae70f0640 1 -- 172.21.15.29:0/4128055012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae01a6860 con 0x7feae0109820 2024-07-27T06:46:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.394+0000 7feaed674740 1 Processor -- start 2024-07-27T06:46:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.394+0000 7feaddffb640 1 -- 172.21.15.29:0/4128055012 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7fead80051d0 con 0x7feae0109820 2024-07-27T06:46:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.394+0000 7feaed674740 1 -- start start 2024-07-27T06:46:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feaed674740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c383a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feaed674740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a786a0a0 0x55e3a7c388e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feaed674740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 0x55e3a7c38e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feaed674740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7bae940 con 0x55e3a7872280 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feaed674740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7b9de40 con 0x55e3a7892f60 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feaed674740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7ba1e80 con 0x55e3a786a0a0 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feadffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 0x55e3a7c38e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feadffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 0x55e3a7c38e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58054/0 (socket says 172.21.15.29:58054) 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feadffff640 1 -- 172.21.15.29:0/1930790201 learned_addr learned my addr 172.21.15.29:0/1930790201 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feadffff640 1 --2- 172.21.15.29:0/1930790201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feae4e8c640 1 --2- 172.21.15.29:0/1930790201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a786a0a0 0x55e3a7c388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feaddffb640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7feac80b3ca0 0x7feac80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feadffff640 1 -- 172.21.15.29:0/1930790201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a786a0a0 msgr2=0x55e3a7c388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feadffff640 1 --2- 172.21.15.29:0/1930790201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a786a0a0 0x55e3a7c388e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feadffff640 1 -- 172.21.15.29:0/1930790201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 msgr2=0x55e3a7c383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feae568d640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7feac80b3ca0 0x7feac80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feadffff640 1 --2- 172.21.15.29:0/1930790201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c383a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.395+0000 7feadffff640 1 -- 172.21.15.29:0/1930790201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3a7b44a60 con 0x55e3a7872280 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.396+0000 7feaddffb640 1 -- 172.21.15.29:0/4128055012 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fead80e3f80 con 0x7feae0109820 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.396+0000 7feadffff640 1 --2- 172.21.15.29:0/1930790201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 0x55e3a7c38e20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fead4014960 tx=0x7fead400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383151c87b1bfa65 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.396+0000 7feac6ffd640 1 -- 172.21.15.29:0/1930790201 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead40142d0 con 0x55e3a7872280 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.396+0000 7feac6ffd640 1 -- 172.21.15.29:0/1930790201 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fead40054d0 con 0x55e3a7872280 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.396+0000 7feac6ffd640 1 -- 172.21.15.29:0/1930790201 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead4004da0 con 0x55e3a7872280 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.397+0000 7feaed674740 1 -- 172.21.15.29:0/1930790201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 msgr2=0x55e3a7c38e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.397+0000 7feae568d640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7feac80b3ca0 0x7feac80b61b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fead80aaac0 tx=0x7fead80741e0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.397+0000 7feaed674740 1 --2- 172.21.15.29:0/1930790201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 0x55e3a7c38e20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fead4014960 tx=0x7fead400c490 comp rx=0 tx=0).stop 2024-07-27T06:46:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.397+0000 7feaed674740 1 -- 172.21.15.29:0/1930790201 shutdown_connections 2024-07-27T06:46:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.397+0000 7feaed674740 1 --2- 172.21.15.29:0/1930790201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 0x55e3a7c38e20 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.397+0000 7feaed674740 1 --2- 172.21.15.29:0/1930790201 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a786a0a0 0x55e3a7c388e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.397+0000 7feaed674740 1 --2- 172.21.15.29:0/1930790201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c383a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.397+0000 7feaed674740 1 -- 172.21.15.29:0/1930790201 >> 172.21.15.29:0/1930790201 conn(0x55e3a7887350 msgr2=0x55e3a7c36290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.397+0000 7feaed674740 1 -- 172.21.15.29:0/1930790201 shutdown_connections 2024-07-27T06:46:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.397+0000 7feaed674740 1 -- 172.21.15.29:0/1930790201 wait complete. 2024-07-27T06:46:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.397+0000 7feaed674740 1 -- start start 2024-07-27T06:46:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.397+0000 7feaed674740 1 Processor -- start 2024-07-27T06:46:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feaed674740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 0x55e3a7c482e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feaed674740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a7872280 0x55e3a7c48820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feaed674740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c541e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feaed674740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7a015f0 con 0x55e3a786a0a0 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feaed674740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7b408b0 con 0x55e3a7892f60 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feaed674740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7c4ae80 con 0x55e3a7872280 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feaed674740 10 client.0 _ll_register_callbacks cb 0x55e3a7b4a7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feadffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 0x55e3a7c482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feae4e8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a7872280 0x55e3a7c48820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feae4e8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a7872280 0x55e3a7c48820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:47152/0 (socket says 172.21.15.29:47152) 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feae4e8c640 1 -- 172.21.15.29:0/728305034 learned_addr learned my addr 172.21.15.29:0/728305034 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feadffff640 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 0x55e3a7c482e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58064/0 (socket says 172.21.15.29:58064) 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.398+0000 7feadffff640 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feadffff640 1 -- 172.21.15.29:0/728305034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a7872280 msgr2=0x55e3a7c48820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feadffff640 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a7872280 0x55e3a7c48820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feadffff640 1 -- 172.21.15.29:0/728305034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 msgr2=0x55e3a7c482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feadffff640 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 0x55e3a7c482e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feadffff640 1 -- 172.21.15.29:0/728305034 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3a78fe540 con 0x55e3a7892f60 2024-07-27T06:46:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feadffff640 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c541e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fead4004a80 tx=0x7fead4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4360063f9d2b312 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feabffff640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-07-27T06:46:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feabffff640 1 -- 172.21.15.29:0/728305034 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead4005230 con 0x55e3a7892f60 2024-07-27T06:46:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feabffff640 1 -- 172.21.15.29:0/728305034 <== mon.2 v2:172.21.15.47:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fead4004b20 con 0x55e3a7892f60 2024-07-27T06:46:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feaed674740 1 -- 172.21.15.29:0/728305034 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fead40054d0 con 0x55e3a7892f60 2024-07-27T06:46:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feabffff640 1 -- 172.21.15.29:0/728305034 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fead4004d20 con 0x55e3a7892f60 2024-07-27T06:46:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feaed674740 10 client.25049 fetch_fsmap learned FSMap version 28 2024-07-27T06:46:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.399+0000 7feaed674740 1 -- 172.21.15.29:0/728305034 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e3a7c56e50 con 0x55e3a7892f60 2024-07-27T06:46:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.400+0000 7feabffff640 1 -- 172.21.15.29:0/728305034 <== mon.2 v2:172.21.15.47:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fead40054d0 con 0x55e3a7892f60 2024-07-27T06:46:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.400+0000 7feabffff640 1 -- 172.21.15.29:0/728305034 <== mon.2 v2:172.21.15.47:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fead4012450 con 0x55e3a7892f60 2024-07-27T06:46:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.400+0000 7feaed674740 10 client.25049 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:46:42.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.400+0000 7feaed674740 10 client.25049 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-27T06:46:42.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.400+0000 7feaed674740 20 client.25049 populate_metadata read hostname 'smithi029' 2024-07-27T06:46:42.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.400+0000 7feaed674740 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3a78518e0 0x55e3a7c5a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.400+0000 7feaed674740 4 client.25049 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:46:42.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.400+0000 7feaed674740 1 -- 172.21.15.29:0/728305034 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3a7bb0890 con 0x55e3a78518e0 2024-07-27T06:46:42.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.400+0000 7feae4e8c640 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3a78518e0 0x55e3a7c5a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.401+0000 7feae4e8c640 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3a78518e0 0x55e3a7c5a600 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fde8687152a41be4 server_cookie=4b51ccf821897b8b in_seq=0 out_seq=0 2024-07-27T06:46:42.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.401+0000 7feabffff640 10 client.25049 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:46:42.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feabffff640 1 -- 172.21.15.29:0/728305034 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e3a7bb0890 con 0x55e3a78518e0 2024-07-27T06:46:42.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feabffff640 10 client.25049 handle_command_reply: tid=0 2024-07-27T06:46:42.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feaed674740 1 client.25049 shutdown 2024-07-27T06:46:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feaed674740 1 -- 172.21.15.29:0/728305034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 msgr2=0x55e3a7c541e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feaed674740 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c541e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fead4004a80 tx=0x7fead4005850 comp rx=0 tx=0).stop 2024-07-27T06:46:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feaed674740 1 -- 172.21.15.29:0/728305034 shutdown_connections 2024-07-27T06:46:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feaed674740 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c541e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feaed674740 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3a78518e0 0x55e3a7c5a600 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feaed674740 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a7872280 0x55e3a7c48820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feaed674740 1 --2- 172.21.15.29:0/728305034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 0x55e3a7c482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feaed674740 1 -- 172.21.15.29:0/728305034 >> 172.21.15.29:0/728305034 conn(0x55e3a7887350 msgr2=0x55e3a7c34850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feabffff640 0 client.25049 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:46:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feaed674740 1 -- 172.21.15.29:0/728305034 shutdown_connections 2024-07-27T06:46:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feaed674740 1 -- 172.21.15.29:0/728305034 wait complete. 2024-07-27T06:46:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.402+0000 7feaed674740 20 client.25049 trim_cache size 0 max 16384 2024-07-27T06:46:42.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.420+0000 7feaed674740 1 Processor -- start 2024-07-27T06:46:42.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.420+0000 7feaed674740 1 -- start start 2024-07-27T06:46:42.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feaed674740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a786a0a0 0x55e3a7c59190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feaed674740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 0x55e3a7c475d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feaed674740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c54150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feaed674740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7c58940 con 0x55e3a7872280 2024-07-27T06:46:42.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feaed674740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7c59010 con 0x55e3a7892f60 2024-07-27T06:46:42.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feaed674740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7c596e0 con 0x55e3a786a0a0 2024-07-27T06:46:42.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feae4e8c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 0x55e3a7c475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feae4e8c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 0x55e3a7c475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58076/0 (socket says 172.21.15.29:58076) 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feae4e8c640 1 -- 172.21.15.29:0/2825619057 learned_addr learned my addr 172.21.15.29:0/2825619057 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feadffff640 1 --2- 172.21.15.29:0/2825619057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c54150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feae4e8c640 1 -- 172.21.15.29:0/2825619057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a786a0a0 msgr2=0x55e3a7c59190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feae4e8c640 1 --2- 172.21.15.29:0/2825619057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a786a0a0 0x55e3a7c59190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feae4e8c640 1 -- 172.21.15.29:0/2825619057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 msgr2=0x55e3a7c54150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feae4e8c640 1 --2- 172.21.15.29:0/2825619057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c54150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feae4e8c640 1 -- 172.21.15.29:0/2825619057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3a7c42d10 con 0x55e3a7872280 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feadffff640 1 --2- 172.21.15.29:0/2825619057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c54150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feae4e8c640 1 --2- 172.21.15.29:0/2825619057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 0x55e3a7c475d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fead0006010 tx=0x7fead0005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311550df188c331b server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.421+0000 7feac67fc640 1 -- 172.21.15.29:0/2825619057 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead004a070 con 0x55e3a7872280 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.422+0000 7feac67fc640 1 -- 172.21.15.29:0/2825619057 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fead00054f0 con 0x55e3a7872280 2024-07-27T06:46:42.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.422+0000 7feac67fc640 1 -- 172.21.15.29:0/2825619057 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0035040 con 0x55e3a7872280 2024-07-27T06:46:42.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.422+0000 7feaed674740 1 -- 172.21.15.29:0/2825619057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 msgr2=0x55e3a7c475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:42.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.422+0000 7feaed674740 1 --2- 172.21.15.29:0/2825619057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 0x55e3a7c475d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fead0006010 tx=0x7fead0005a20 comp rx=0 tx=0).stop 2024-07-27T06:46:42.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.422+0000 7feaed674740 1 -- 172.21.15.29:0/2825619057 shutdown_connections 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.422+0000 7feaed674740 1 --2- 172.21.15.29:0/2825619057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c54150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.422+0000 7feaed674740 1 --2- 172.21.15.29:0/2825619057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a7872280 0x55e3a7c475d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.422+0000 7feaed674740 1 --2- 172.21.15.29:0/2825619057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a786a0a0 0x55e3a7c59190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.422+0000 7feaed674740 1 -- 172.21.15.29:0/2825619057 >> 172.21.15.29:0/2825619057 conn(0x55e3a7887350 msgr2=0x55e3a7c43220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.422+0000 7feaed674740 1 -- 172.21.15.29:0/2825619057 shutdown_connections 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.423+0000 7feaed674740 1 -- 172.21.15.29:0/2825619057 wait complete. 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.423+0000 7feaed674740 1 -- start start 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.423+0000 7feaed674740 1 Processor -- start 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.423+0000 7feaed674740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 0x55e3a7c5ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.423+0000 7feaed674740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a7872280 0x55e3a7c5bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.423+0000 7feaed674740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c60af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.423+0000 7feaed674740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7c5a480 con 0x55e3a786a0a0 2024-07-27T06:46:42.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.423+0000 7feaed674740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7c59db0 con 0x55e3a7892f60 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.423+0000 7feaed674740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7c5ab50 con 0x55e3a7872280 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.423+0000 7feaed674740 10 client.0 _ll_register_callbacks cb 0x7fead40140d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feadffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 0x55e3a7c5ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feadffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 0x55e3a7c5ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58082/0 (socket says 172.21.15.29:58082) 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feadffff640 1 -- 172.21.15.29:0/974314016 learned_addr learned my addr 172.21.15.29:0/974314016 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feae4e8c640 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a7872280 0x55e3a7c5bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feadffff640 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c60af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feadffff640 1 -- 172.21.15.29:0/974314016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a7872280 msgr2=0x55e3a7c5bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feadffff640 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a7872280 0x55e3a7c5bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feadffff640 1 -- 172.21.15.29:0/974314016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 msgr2=0x55e3a7c60af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feadffff640 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c60af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feadffff640 1 -- 172.21.15.29:0/974314016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3a7c61160 con 0x55e3a786a0a0 2024-07-27T06:46:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feadffff640 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 0x55e3a7c5ba00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fead4004a80 tx=0x7fead4005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3376e95d39cb059e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feabffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feabffff640 1 -- 172.21.15.29:0/974314016 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead40145e0 con 0x55e3a786a0a0 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feabffff640 1 -- 172.21.15.29:0/974314016 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fead400c870 con 0x55e3a786a0a0 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.424+0000 7feaed674740 1 -- 172.21.15.29:0/974314016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fead00054f0 con 0x55e3a786a0a0 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.425+0000 7feabffff640 1 -- 172.21.15.29:0/974314016 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fead4005230 con 0x55e3a786a0a0 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.425+0000 7feaed674740 10 client.15387 fetch_fsmap learned FSMap version 28 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.425+0000 7feaed674740 1 -- 172.21.15.29:0/974314016 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e3a7c62490 con 0x55e3a786a0a0 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.425+0000 7feabffff640 1 -- 172.21.15.29:0/974314016 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fead00054f0 con 0x55e3a786a0a0 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.425+0000 7feabffff640 1 -- 172.21.15.29:0/974314016 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fead400fc00 con 0x55e3a786a0a0 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.425+0000 7feaed674740 10 client.15387 fetch_fsmap finished waiting for FSMap version 28 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.425+0000 7feaed674740 10 client.15387 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.425+0000 7feaed674740 20 client.15387 populate_metadata read hostname 'smithi029' 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.425+0000 7feaed674740 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3a7826c90 0x55e3a7cc5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:42.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.425+0000 7feaed674740 4 client.15387 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.425+0000 7feaed674740 1 -- 172.21.15.29:0/974314016 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "status"}) -- 0x55e3a7bac330 con 0x55e3a7826c90 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.425+0000 7feae4e8c640 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3a7826c90 0x55e3a7cc5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feae4e8c640 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3a7826c90 0x55e3a7cc5df0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cb0448b1e08a8e9 server_cookie=850b8320d96fdce8 in_seq=0 out_seq=0 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feabffff640 10 client.15387 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feabffff640 1 -- 172.21.15.29:0/974314016 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55e3a7bac330 con 0x55e3a7826c90 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feabffff640 10 client.15387 handle_command_reply: tid=0 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feaed674740 1 client.15387 shutdown 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feaed674740 1 -- 172.21.15.29:0/974314016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 msgr2=0x55e3a7c5ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feaed674740 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 0x55e3a7c5ba00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fead4004a80 tx=0x7fead4005970 comp rx=0 tx=0).stop 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feaed674740 1 -- 172.21.15.29:0/974314016 shutdown_connections 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feaed674740 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3a7892f60 0x55e3a7c60af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feaed674740 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3a7826c90 0x55e3a7cc5df0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feaed674740 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3a7872280 0x55e3a7c5bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feaed674740 1 --2- 172.21.15.29:0/974314016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3a786a0a0 0x55e3a7c5ba00 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feabffff640 0 client.15387 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:46:42.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.426+0000 7feaed674740 1 -- 172.21.15.29:0/974314016 >> 172.21.15.29:0/974314016 conn(0x55e3a7887350 msgr2=0x55e3a7c46560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:42.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feaed674740 1 -- 172.21.15.29:0/974314016 shutdown_connections 2024-07-27T06:46:42.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feaed674740 1 -- 172.21.15.29:0/974314016 wait complete. 2024-07-27T06:46:42.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feaed674740 20 client.15387 trim_cache size 0 max 16384 2024-07-27T06:46:42.433 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:46:42.433 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:46:42.433 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-07-27T06:46:42.433 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24670, 2024-07-27T06:46:42.433 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:46:42.434 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:46:42.434 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:46:42.434 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 206.628230368, 2024-07-27T06:46:42.434 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 28, 2024-07-27T06:46:42.434 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:46:42.434 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 79, 2024-07-27T06:46:42.434 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 235.68852962400001 2024-07-27T06:46:42.434 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:46:42.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feae70f0640 1 -- 172.21.15.29:0/4128055012 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7feac80b3ca0 msgr2=0x7feac80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:42.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feae70f0640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7feac80b3ca0 0x7feac80b61b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fead80aaac0 tx=0x7fead80741e0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feae70f0640 1 -- 172.21.15.29:0/4128055012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae0109820 msgr2=0x7feae019f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:42.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feae70f0640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae0109820 0x7feae019f6e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fead800eaa0 tx=0x7fead800ef60 comp rx=0 tx=0).stop 2024-07-27T06:46:42.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feae70f0640 1 -- 172.21.15.29:0/4128055012 shutdown_connections 2024-07-27T06:46:42.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feae70f0640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae0109820 0x7feae019f6e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feae70f0640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feae0108ed0 0x7feae019a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feae70f0640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7feac80b3ca0 0x7feac80b61b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feae70f0640 1 --2- 172.21.15.29:0/4128055012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feae0103590 0x7feae0199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:42.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feae70f0640 1 -- 172.21.15.29:0/4128055012 >> 172.21.15.29:0/4128055012 conn(0x7feae007b280 msgr2=0x7feae0106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:42.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.427+0000 7feae70f0640 1 -- 172.21.15.29:0/4128055012 shutdown_connections 2024-07-27T06:46:42.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:42.428+0000 7feae70f0640 1 -- 172.21.15.29:0/4128055012 wait complete. 2024-07-27T06:46:43.153 INFO:teuthology.run_tasks:Running task sequential... 2024-07-27T06:46:43.163 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-07-27T06:46:43.163 INFO:teuthology.task.sequential:In sequential, running task print... 2024-07-27T06:46:43.164 INFO:teuthology.task.print:Enabling mgr modules 2024-07-27T06:46:43.164 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-07-27T06:46:43.165 INFO:teuthology.task.exec:Executing custom commands... 2024-07-27T06:46:43.165 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-07-27T06:46:43.165 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-07-27T06:46:43.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.280+0000 7f79fa62c640 1 -- 172.21.15.29:0/4014274357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f410d7a0 msgr2=0x7f79f410fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:43.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.281+0000 7f79fa62c640 1 --2- 172.21.15.29:0/4014274357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f410d7a0 0x7f79f410fc90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009ca0 tx=0x7f79ec02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:46:43.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.281+0000 7f79fa62c640 1 -- 172.21.15.29:0/4014274357 shutdown_connections 2024-07-27T06:46:43.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.281+0000 7f79fa62c640 1 --2- 172.21.15.29:0/4014274357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f410d7a0 0x7f79f410fc90 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:43.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.281+0000 7f79fa62c640 1 --2- 172.21.15.29:0/4014274357 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79f4104c90 0x7f79f410d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.281+0000 7f79fa62c640 1 --2- 172.21.15.29:0/4014274357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79f4104340 0x7f79f4104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.281+0000 7f79fa62c640 1 -- 172.21.15.29:0/4014274357 >> 172.21.15.29:0/4014274357 conn(0x7f79f40fddf0 msgr2=0x7f79f4100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.281+0000 7f79fa62c640 1 -- 172.21.15.29:0/4014274357 shutdown_connections 2024-07-27T06:46:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.281+0000 7f79fa62c640 1 -- 172.21.15.29:0/4014274357 wait complete. 2024-07-27T06:46:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.281+0000 7f79fa62c640 1 Processor -- start 2024-07-27T06:46:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.281+0000 7f79fa62c640 1 -- start start 2024-07-27T06:46:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79fa62c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79f4104340 0x7f79f4199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79fa62c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79f4104c90 0x7f79f419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79fa62c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f410d7a0 0x7f79f419f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79fa62c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f79f4113420 con 0x7f79f410d7a0 2024-07-27T06:46:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79fa62c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f79f41132a0 con 0x7f79f4104340 2024-07-27T06:46:43.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79fa62c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f79f41135a0 con 0x7f79f4104c90 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79f8bc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f410d7a0 0x7f79f419f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79f8bc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f410d7a0 0x7f79f419f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58096/0 (socket says 172.21.15.29:58096) 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79f8bc9640 1 -- 172.21.15.29:0/3363313749 learned_addr learned my addr 172.21.15.29:0/3363313749 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79ebfff640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79f4104c90 0x7f79f419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79f3fff640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79f4104340 0x7f79f4199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79f8bc9640 1 -- 172.21.15.29:0/3363313749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79f4104c90 msgr2=0x7f79f419a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79f8bc9640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79f4104c90 0x7f79f419a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79f8bc9640 1 -- 172.21.15.29:0/3363313749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79f4104340 msgr2=0x7f79f4199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79f8bc9640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79f4104340 0x7f79f4199de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79f8bc9640 1 -- 172.21.15.29:0/3363313749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f419fd10 con 0x7f79f410d7a0 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.282+0000 7f79f8bc9640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f410d7a0 0x7f79f419f610 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f79ec005cc0 tx=0x7f79ec02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.283+0000 7f79f1ffb640 1 -- 172.21.15.29:0/3363313749 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec003bf0 con 0x7f79f410d7a0 2024-07-27T06:46:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.283+0000 7f79f1ffb640 1 -- 172.21.15.29:0/3363313749 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79ec003d90 con 0x7f79f410d7a0 2024-07-27T06:46:43.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.283+0000 7f79fa62c640 1 -- 172.21.15.29:0/3363313749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f419ffa0 con 0x7f79f410d7a0 2024-07-27T06:46:43.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.283+0000 7f79f1ffb640 1 -- 172.21.15.29:0/3363313749 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec02fbc0 con 0x7f79f410d7a0 2024-07-27T06:46:43.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.283+0000 7f79fa62c640 1 -- 172.21.15.29:0/3363313749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f41a6790 con 0x7f79f410d7a0 2024-07-27T06:46:43.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.284+0000 7f79fa62c640 1 -- 172.21.15.29:0/3363313749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f4104750 con 0x7f79f410d7a0 2024-07-27T06:46:43.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.285+0000 7f79f1ffb640 1 -- 172.21.15.29:0/3363313749 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) ==== 151076+0+0 (secure 0 0 0) 0x7f79ec02fd60 con 0x7f79f410d7a0 2024-07-27T06:46:43.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.286+0000 7f79f1ffb640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f79e00b3c20 0x7f79e00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:43.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.286+0000 7f79f3fff640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f79e00b3c20 0x7f79e00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:43.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.286+0000 7f79f1ffb640 1 -- 172.21.15.29:0/3363313749 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f79ec1054b0 con 0x7f79f410d7a0 2024-07-27T06:46:43.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.286+0000 7f79f3fff640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f79e00b3c20 0x7f79e00b6130 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f79e40058c0 tx=0x7f79e4005c60 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:43.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.290+0000 7f79f1ffb640 1 -- 172.21.15.29:0/3363313749 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f79ec0cc9f0 con 0x7f79f410d7a0 2024-07-27T06:46:43.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:43.535+0000 7f79fa62c640 1 -- 172.21.15.29:0/3363313749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f79f402e0c0 con 0x7f79f410d7a0 2024-07-27T06:46:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:43 smithi047 ceph-mon[36835]: pgmap v388: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:43 smithi029 ceph-mon[32610]: pgmap v388: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:43 smithi077 ceph-mon[37153]: pgmap v388: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:46:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.413+0000 7f79f1ffb640 1 -- 172.21.15.29:0/3363313749 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 module 'snap_schedule' is already enabled v23) ==== 133+0+0 (secure 0 0 0) 0x7f79ec0098b0 con 0x7f79f410d7a0 2024-07-27T06:46:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:module 'snap_schedule' is already enabled 2024-07-27T06:46:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.416+0000 7f79fa62c640 1 -- 172.21.15.29:0/3363313749 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f79e00b3c20 msgr2=0x7f79e00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.416+0000 7f79fa62c640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f79e00b3c20 0x7f79e00b6130 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f79e40058c0 tx=0x7f79e4005c60 comp rx=0 tx=0).stop 2024-07-27T06:46:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.416+0000 7f79fa62c640 1 -- 172.21.15.29:0/3363313749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f410d7a0 msgr2=0x7f79f419f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.416+0000 7f79fa62c640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f410d7a0 0x7f79f419f610 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f79ec005cc0 tx=0x7f79ec02f380 comp rx=0 tx=0).stop 2024-07-27T06:46:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.416+0000 7f79fa62c640 1 -- 172.21.15.29:0/3363313749 shutdown_connections 2024-07-27T06:46:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.416+0000 7f79fa62c640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79f410d7a0 0x7f79f419f610 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.416+0000 7f79fa62c640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f79e00b3c20 0x7f79e00b6130 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.416+0000 7f79fa62c640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f79f4104c90 0x7f79f419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.416+0000 7f79fa62c640 1 --2- 172.21.15.29:0/3363313749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79f4104340 0x7f79f4199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.416+0000 7f79fa62c640 1 -- 172.21.15.29:0/3363313749 >> 172.21.15.29:0/3363313749 conn(0x7f79f40fddf0 msgr2=0x7f79f4105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.416+0000 7f79fa62c640 1 -- 172.21.15.29:0/3363313749 shutdown_connections 2024-07-27T06:46:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:44.416+0000 7f79fa62c640 1 -- 172.21.15.29:0/3363313749 wait complete. 2024-07-27T06:46:44.437 INFO:teuthology.run_tasks:Running task kclient... 2024-07-27T06:46:44.465 INFO:tasks.kclient:Mounting kernel clients... 2024-07-27T06:46:44.465 INFO:tasks.kclient:config is {'client.0': {}} 2024-07-27T06:46:44.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-27T06:46:44.466 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-07-27T06:46:44.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-27T06:46:44.484 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2024-07-27T06:46:44.553 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2024-07-27T06:46:44.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-27T06:46:44.555 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-07-27T06:46:44.556 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-27T06:46:44.556 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-27T06:46:44.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs ls 2024-07-27T06:46:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:44 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3363313749' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-27T06:46:44.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:44 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3363313749' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-27T06:46:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:44 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3363313749' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-27T06:46:44.862 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:45 smithi047 ceph-mon[36835]: pgmap v389: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:45 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3363313749' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-27T06:46:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:45 smithi047 ceph-mon[36835]: mgrmap e23: x(active, since 10m), standbys: y, z 2024-07-27T06:46:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:45 smithi029 ceph-mon[32610]: pgmap v389: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:45 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3363313749' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-27T06:46:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:45 smithi029 ceph-mon[32610]: mgrmap e23: x(active, since 10m), standbys: y, z 2024-07-27T06:46:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:45 smithi077 ceph-mon[37153]: pgmap v389: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-27T06:46:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:45 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3363313749' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-27T06:46:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:45 smithi077 ceph-mon[37153]: mgrmap e23: x(active, since 10m), standbys: y, z 2024-07-27T06:46:46.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.517+0000 7f805a59c640 1 -- 172.21.15.29:0/262542738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8054102510 msgr2=0x7f8054102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:46.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.517+0000 7f805a59c640 1 --2- 172.21.15.29:0/262542738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8054102510 0x7f8054102920 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8044009900 tx=0x7f804402a410 comp rx=0 tx=0).stop 2024-07-27T06:46:46.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.517+0000 7f805a59c640 1 -- 172.21.15.29:0/262542738 shutdown_connections 2024-07-27T06:46:46.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.517+0000 7f805a59c640 1 --2- 172.21.15.29:0/262542738 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8054108750 0x7f805410ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.517+0000 7f805a59c640 1 --2- 172.21.15.29:0/262542738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8054107e00 0x7f8054108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.517+0000 7f805a59c640 1 --2- 172.21.15.29:0/262542738 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8054102510 0x7f8054102920 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.517+0000 7f805a59c640 1 -- 172.21.15.29:0/262542738 >> 172.21.15.29:0/262542738 conn(0x7f8054079190 msgr2=0x7f80540795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.517+0000 7f805a59c640 1 -- 172.21.15.29:0/262542738 shutdown_connections 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.517+0000 7f805a59c640 1 -- 172.21.15.29:0/262542738 wait complete. 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f805a59c640 1 Processor -- start 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f805a59c640 1 -- start start 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f805a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8054102510 0x7f805419ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f805a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8054107e00 0x7f805419b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f805a59c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8054108750 0x7f805419b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f805a59c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f80541123d0 con 0x7f8054107e00 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f805a59c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f8054112250 con 0x7f8054108750 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f805a59c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8054112550 con 0x7f8054102510 2024-07-27T06:46:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f80537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8054107e00 0x7f805419b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f80537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8054107e00 0x7f805419b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58100/0 (socket says 172.21.15.29:58100) 2024-07-27T06:46:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f80537fe640 1 -- 172.21.15.29:0/1785889504 learned_addr learned my addr 172.21.15.29:0/1785889504 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f8053fff640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8054102510 0x7f805419ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.518+0000 7f8058b39640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8054108750 0x7f805419b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.519+0000 7f80537fe640 1 -- 172.21.15.29:0/1785889504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8054102510 msgr2=0x7f805419ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.519+0000 7f80537fe640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8054102510 0x7f805419ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.519+0000 7f80537fe640 1 -- 172.21.15.29:0/1785889504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8054108750 msgr2=0x7f805419b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.519+0000 7f80537fe640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8054108750 0x7f805419b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.519+0000 7f80537fe640 1 -- 172.21.15.29:0/1785889504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80541a5fd0 con 0x7f8054107e00 2024-07-27T06:46:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.519+0000 7f80537fe640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8054107e00 0x7f805419b280 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f804800e9f0 tx=0x7f804800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.519+0000 7f80517fa640 1 -- 172.21.15.29:0/1785889504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804800ccc0 con 0x7f8054107e00 2024-07-27T06:46:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.519+0000 7f805a59c640 1 -- 172.21.15.29:0/1785889504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80541a62c0 con 0x7f8054107e00 2024-07-27T06:46:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.519+0000 7f80517fa640 1 -- 172.21.15.29:0/1785889504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80480054d0 con 0x7f8054107e00 2024-07-27T06:46:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.519+0000 7f80517fa640 1 -- 172.21.15.29:0/1785889504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8048010430 con 0x7f8054107e00 2024-07-27T06:46:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.519+0000 7f805a59c640 1 -- 172.21.15.29:0/1785889504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80541a6800 con 0x7f8054107e00 2024-07-27T06:46:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.520+0000 7f805a59c640 1 -- 172.21.15.29:0/1785889504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8054103740 con 0x7f8054107e00 2024-07-27T06:46:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.523+0000 7f80517fa640 1 -- 172.21.15.29:0/1785889504 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f80480046c0 con 0x7f8054107e00 2024-07-27T06:46:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.523+0000 7f80517fa640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f803c0b3c50 0x7f803c0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.523+0000 7f8053fff640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f803c0b3c50 0x7f803c0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.524+0000 7f80517fa640 1 -- 172.21.15.29:0/1785889504 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f80480e4290 con 0x7f8054107e00 2024-07-27T06:46:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.524+0000 7f8053fff640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f803c0b3c50 0x7f803c0b6160 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8044000c00 tx=0x7f8044031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.526+0000 7f80517fa640 1 -- 172.21.15.29:0/1785889504 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8048055600 con 0x7f8054107e00 2024-07-27T06:46:46.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.747+0000 7f805a59c640 1 -- 172.21.15.29:0/1785889504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8054102920 con 0x7f8054107e00 2024-07-27T06:46:46.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.747+0000 7f80517fa640 1 -- 172.21.15.29:0/1785889504 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+73 (secure 0 0 0) 0x7f80480ab7d0 con 0x7f8054107e00 2024-07-27T06:46:46.749 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-27T06:46:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.749+0000 7f805a59c640 1 -- 172.21.15.29:0/1785889504 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f803c0b3c50 msgr2=0x7f803c0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:46.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.749+0000 7f805a59c640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f803c0b3c50 0x7f803c0b6160 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8044000c00 tx=0x7f8044031400 comp rx=0 tx=0).stop 2024-07-27T06:46:46.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.749+0000 7f805a59c640 1 -- 172.21.15.29:0/1785889504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8054107e00 msgr2=0x7f805419b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:46.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.749+0000 7f805a59c640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8054107e00 0x7f805419b280 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f804800e9f0 tx=0x7f804800eeb0 comp rx=0 tx=0).stop 2024-07-27T06:46:46.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.749+0000 7f805a59c640 1 -- 172.21.15.29:0/1785889504 shutdown_connections 2024-07-27T06:46:46.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.749+0000 7f805a59c640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8054108750 0x7f805419b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:46.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.749+0000 7f805a59c640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8054107e00 0x7f805419b280 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:46.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.749+0000 7f805a59c640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f803c0b3c50 0x7f803c0b6160 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:46.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.749+0000 7f805a59c640 1 --2- 172.21.15.29:0/1785889504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8054102510 0x7f805419ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:46.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.749+0000 7f805a59c640 1 -- 172.21.15.29:0/1785889504 >> 172.21.15.29:0/1785889504 conn(0x7f8054079190 msgr2=0x7f8054100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:46.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.749+0000 7f805a59c640 1 -- 172.21.15.29:0/1785889504 shutdown_connections 2024-07-27T06:46:46.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:46.749+0000 7f805a59c640 1 -- 172.21.15.29:0/1785889504 wait complete. 2024-07-27T06:46:47.426 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-07-27T06:46:47.426 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-27T06:46:47.427 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2024-07-27T06:46:47.427 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-27T06:46:47.427 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-27T06:46:47.427 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-07-27T06:46:47.427 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-27T06:46:47.427 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-27T06:46:47.427 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-27T06:46:47.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-27T06:46:47.427 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2024-07-27T06:46:47.446 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-27T06:46:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-27T06:46:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2024-07-27T06:46:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-07-27T06:46:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2024-07-27T06:46:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-07-27T06:46:47.447 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-27T06:46:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2024-07-27T06:46:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2024-07-27T06:46:47.448 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-27T06:46:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2024-07-27T06:46:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2024-07-27T06:46:47.448 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-27T06:46:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2024-07-27T06:46:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2024-07-27T06:46:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-07-27T06:46:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41941sec preferred_lft 41941sec 2024-07-27T06:46:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2024-07-27T06:46:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-07-27T06:46:47.450 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-27T06:46:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2024-07-27T06:46:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2024-07-27T06:46:47.451 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-27T06:46:47.451 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-27T06:46:47.451 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-07-27T06:46:47.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2024-07-27T06:46:47.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2024-07-27T06:46:47.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2024-07-27T06:46:47.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-27T06:46:47.452 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-07-27T06:46:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-27T06:46:47.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:47 smithi029 ceph-mon[32610]: pgmap v390: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:47.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:47 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1785889504' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-27T06:46:47.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-27T06:46:47.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-27T06:46:47.660 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-27T06:46:47.687 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-07-27T06:46:47.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-27T06:46:47.689 DEBUG:teuthology.orchestra.run.smithi029:> route 2024-07-27T06:46:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:47 smithi047 ceph-mon[36835]: pgmap v390: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:47 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1785889504' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-27T06:46:47.767 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2024-07-27T06:46:47.767 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-27T06:46:47.767 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-07-27T06:46:47.767 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-07-27T06:46:47.767 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-27T06:46:47.767 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-27T06:46:47.768 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-07-27T06:46:47.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-07-27T06:46:47.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-07-27T06:46:47.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-07-27T06:46:47.768 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-07-27T06:46:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:47 smithi077 ceph-mon[37153]: pgmap v390: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:46:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:47 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1785889504' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-27T06:46:47.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-27T06:46:48.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-27T06:46:48.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-27T06:46:48.073 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-07-27T06:46:48.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-27T06:46:48.089 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2024-07-27T06:46:48.145 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-27T06:46:48.146 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-07-27T06:46:48.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-27T06:46:48.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-07-27T06:46:48.146 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-07-27T06:46:48.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-27T06:46:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-27T06:46:48.268 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-07-27T06:46:48.268 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-27T06:46:48.268 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-07-27T06:46:48.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-07-27T06:46:48.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-27T06:46:48.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-07-27T06:46:48.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-07-27T06:46:48.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-07-27T06:46:48.269 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-07-27T06:46:48.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-27T06:46:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-27T06:46:48.399 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-27T06:46:48.399 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-07-27T06:46:48.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2024-07-27T06:46:48.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2024-07-27T06:46:48.400 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-07-27T06:46:48.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-27T06:46:48.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-27T06:46:48.483 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-27T06:46:48.484 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-27T06:46:48.501 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-27T06:46:48.502 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-27T06:46:48.502 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-27T06:46:48.558 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-07-27T06:46:48.558 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-07-27T06:46:48.558 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/617f3b02-55d2-4590-8cf9-6ef1a11d6f17 2024-07-27T06:46:48.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-27T06:46:48.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/617f3b02-55d2-4590-8cf9-6ef1a11d6f17 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-07-27T06:46:48.676 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-07-27T06:46:48.677 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-07-27T06:46:48.810 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-07-27T06:46:48.810 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-07-27T06:46:48.810 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.47:6789,172.21.15.77:6789" 2024-07-27T06:46:48.810 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with new device syntax: 0@0a8a0a4c-4be2-11ef-bcc7-c7b262605968.cephfs=/volumes/qa/sv_1/617f3b02-55d2-4590-8cf9-6ef1a11d6f17 2024-07-27T06:46:48.810 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync,name=0,key=0,mon_addr=172.21.15.29:6789/172.21.15.47:6789/172.21.15.77:6789" will pass to kernel 2024-07-27T06:46:48.811 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-27T06:46:48.812 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2024-07-27T06:46:48.812 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-27T06:46:48.812 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2024-07-27T06:46:48.812 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-27T06:46:48.813 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-27T06:46:48.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-27T06:46:48.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-27T06:46:48.842 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-27T06:46:48.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-27T06:46:48.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-27T06:46:48.942 INFO:teuthology.orchestra.run.smithi029.stdout:client15402 2024-07-27T06:46:48.942 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:46:48.943 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-27T06:46:48.960 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-27T06:46:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-27T06:46:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-27T06:46:49.061 INFO:teuthology.orchestra.run.smithi029.stdout:client15402 2024-07-27T06:46:49.062 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-27T06:46:49.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-27T06:46:49.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-27T06:46:49.121 INFO:teuthology.orchestra.run.smithi029.stdout:client15402 2024-07-27T06:46:49.122 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968.client15402/status') 2024-07-27T06:46:49.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-27T06:46:49.182 INFO:teuthology.orchestra.run.smithi029.stdout:instance: client.15402 (3)192.168.0.1:0/2451673514 2024-07-27T06:46:49.182 INFO:teuthology.orchestra.run.smithi029.stdout:blocklisted: false 2024-07-27T06:46:49.183 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records in 2024-07-27T06:46:49.183 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records out 2024-07-27T06:46:49.183 INFO:teuthology.orchestra.run.smithi029.stderr:70 bytes copied, 2.4132e-05 s, 2.9 MB/s 2024-07-27T06:46:49.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-27T06:46:49.190 INFO:teuthology.run_tasks:Running task exec... 2024-07-27T06:46:49.199 INFO:teuthology.task.exec:Executing custom commands... 2024-07-27T06:46:49.200 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-07-27T06:46:49.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-07-27T06:46:49.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.315+0000 7fd3635ec640 1 -- 172.21.15.29:0/3873734289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35c104cd0 msgr2=0x7fd35c10d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:49.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.315+0000 7fd3635ec640 1 --2- 172.21.15.29:0/3873734289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35c104cd0 0x7fd35c10d210 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd34c009900 tx=0x7fd34c02a410 comp rx=0 tx=0).stop 2024-07-27T06:46:49.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.315+0000 7fd3635ec640 1 -- 172.21.15.29:0/3873734289 shutdown_connections 2024-07-27T06:46:49.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.315+0000 7fd3635ec640 1 --2- 172.21.15.29:0/3873734289 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd35c10d7e0 0x7fd35c10fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:49.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.315+0000 7fd3635ec640 1 --2- 172.21.15.29:0/3873734289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35c104cd0 0x7fd35c10d210 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:49.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.315+0000 7fd3635ec640 1 --2- 172.21.15.29:0/3873734289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd35c104380 0x7fd35c104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:49.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.315+0000 7fd3635ec640 1 -- 172.21.15.29:0/3873734289 >> 172.21.15.29:0/3873734289 conn(0x7fd35c07a140 msgr2=0x7fd35c07a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:49.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.315+0000 7fd3635ec640 1 -- 172.21.15.29:0/3873734289 shutdown_connections 2024-07-27T06:46:49.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.315+0000 7fd3635ec640 1 -- 172.21.15.29:0/3873734289 wait complete. 2024-07-27T06:46:49.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.316+0000 7fd3635ec640 1 Processor -- start 2024-07-27T06:46:49.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.316+0000 7fd3635ec640 1 -- start start 2024-07-27T06:46:49.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.316+0000 7fd3635ec640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd35c104380 0x7fd35c1020b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:49.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.316+0000 7fd3635ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35c104cd0 0x7fd35c1025f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:49.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.316+0000 7fd3635ec640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd35c10d7e0 0x7fd35c102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:49.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.316+0000 7fd3635ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c113420 con 0x7fd35c104cd0 2024-07-27T06:46:49.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.316+0000 7fd3635ec640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c1132a0 con 0x7fd35c104380 2024-07-27T06:46:49.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.316+0000 7fd3635ec640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c1135a0 con 0x7fd35c10d7e0 2024-07-27T06:46:49.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd360b87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35c104cd0 0x7fd35c1025f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:49.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd360b87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35c104cd0 0x7fd35c1025f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53196/0 (socket says 172.21.15.29:53196) 2024-07-27T06:46:49.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd360b87640 1 -- 172.21.15.29:0/2681252963 learned_addr learned my addr 172.21.15.29:0/2681252963 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:49.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd361388640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd35c104380 0x7fd35c1020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:49.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd361b89640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd35c10d7e0 0x7fd35c102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:49.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd360b87640 1 -- 172.21.15.29:0/2681252963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd35c10d7e0 msgr2=0x7fd35c102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:49.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd360b87640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd35c10d7e0 0x7fd35c102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:49.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd360b87640 1 -- 172.21.15.29:0/2681252963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd35c104380 msgr2=0x7fd35c1020b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:49.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd360b87640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd35c104380 0x7fd35c1020b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:49.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd360b87640 1 -- 172.21.15.29:0/2681252963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35c1a8110 con 0x7fd35c104cd0 2024-07-27T06:46:49.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd361b89640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd35c10d7e0 0x7fd35c102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:49.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd360b87640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35c104cd0 0x7fd35c1025f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd34c006010 tx=0x7fd34c004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:49.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd3527fc640 1 -- 172.21.15.29:0/2681252963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c03f030 con 0x7fd35c104cd0 2024-07-27T06:46:49.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd3527fc640 1 -- 172.21.15.29:0/2681252963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd34c003aa0 con 0x7fd35c104cd0 2024-07-27T06:46:49.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd3635ec640 1 -- 172.21.15.29:0/2681252963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35c1a83a0 con 0x7fd35c104cd0 2024-07-27T06:46:49.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd3527fc640 1 -- 172.21.15.29:0/2681252963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c035070 con 0x7fd35c104cd0 2024-07-27T06:46:49.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.317+0000 7fd3635ec640 1 -- 172.21.15.29:0/2681252963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd35c1a8890 con 0x7fd35c104cd0 2024-07-27T06:46:49.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.318+0000 7fd3635ec640 1 -- 172.21.15.29:0/2681252963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd35c104790 con 0x7fd35c104cd0 2024-07-27T06:46:49.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.321+0000 7fd3527fc640 1 -- 172.21.15.29:0/2681252963 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7fd34c042030 con 0x7fd35c104cd0 2024-07-27T06:46:49.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.322+0000 7fd3527fc640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3480b3c50 0x7fd3480b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:49.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.322+0000 7fd361388640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3480b3c50 0x7fd3480b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:49.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.322+0000 7fd3527fc640 1 -- 172.21.15.29:0/2681252963 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd34c1046c0 con 0x7fd35c104cd0 2024-07-27T06:46:49.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.322+0000 7fd361388640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3480b3c50 0x7fd3480b6160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd35c103b00 tx=0x7fd35800b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:49.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.325+0000 7fd3527fc640 1 -- 172.21.15.29:0/2681252963 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd34c0cbc00 con 0x7fd35c104cd0 2024-07-27T06:46:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:49 smithi029 ceph-mon[32610]: pgmap v391: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:49.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:49.559+0000 7fd3635ec640 1 -- 172.21.15.29:0/2681252963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fd35c1a8c10 con 0x7fd35c104cd0 2024-07-27T06:46:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:49 smithi047 ceph-mon[36835]: pgmap v391: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:49 smithi077 ceph-mon[37153]: pgmap v391: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-27T06:46:50.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.426+0000 7fd3527fc640 1 -- 172.21.15.29:0/2681252963 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v29) ==== 117+0+0 (secure 0 0 0) 0x7fd34c0cb500 con 0x7fd35c104cd0 2024-07-27T06:46:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.429+0000 7fd3635ec640 1 -- 172.21.15.29:0/2681252963 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3480b3c50 msgr2=0x7fd3480b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.429+0000 7fd3635ec640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3480b3c50 0x7fd3480b6160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd35c103b00 tx=0x7fd35800b390 comp rx=0 tx=0).stop 2024-07-27T06:46:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.429+0000 7fd3635ec640 1 -- 172.21.15.29:0/2681252963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35c104cd0 msgr2=0x7fd35c1025f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.429+0000 7fd3635ec640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35c104cd0 0x7fd35c1025f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd34c006010 tx=0x7fd34c004060 comp rx=0 tx=0).stop 2024-07-27T06:46:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.429+0000 7fd3635ec640 1 -- 172.21.15.29:0/2681252963 shutdown_connections 2024-07-27T06:46:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.429+0000 7fd3635ec640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3480b3c50 0x7fd3480b6160 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.429+0000 7fd3635ec640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd35c10d7e0 0x7fd35c102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.429+0000 7fd3635ec640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd35c104cd0 0x7fd35c1025f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.429+0000 7fd3635ec640 1 --2- 172.21.15.29:0/2681252963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd35c104380 0x7fd35c1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.429+0000 7fd3635ec640 1 -- 172.21.15.29:0/2681252963 >> 172.21.15.29:0/2681252963 conn(0x7fd35c07a140 msgr2=0x7fd35c1072f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.429+0000 7fd3635ec640 1 -- 172.21.15.29:0/2681252963 shutdown_connections 2024-07-27T06:46:50.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.429+0000 7fd3635ec640 1 -- 172.21.15.29:0/2681252963 wait complete. 2024-07-27T06:46:50.445 INFO:teuthology.run_tasks:Running task check-counter... 2024-07-27T06:46:50.457 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-07-27T06:46:50.467 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-27T06:46:50.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.590+0000 7f75eb90e640 1 -- 172.21.15.29:0/2823106932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75e4107d30 msgr2=0x7f75e4108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:50.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.590+0000 7f75eb90e640 1 --2- 172.21.15.29:0/2823106932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75e4107d30 0x7f75e4108140 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f75d4009900 tx=0x7f75d402a3f0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.590+0000 7f75eb90e640 1 -- 172.21.15.29:0/2823106932 shutdown_connections 2024-07-27T06:46:50.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.590+0000 7f75eb90e640 1 --2- 172.21.15.29:0/2823106932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e4108680 0x7f75e410eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.590+0000 7f75eb90e640 1 --2- 172.21.15.29:0/2823106932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75e4107d30 0x7f75e4108140 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.590+0000 7f75eb90e640 1 --2- 172.21.15.29:0/2823106932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e4102400 0x7f75e4102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.590+0000 7f75eb90e640 1 -- 172.21.15.29:0/2823106932 >> 172.21.15.29:0/2823106932 conn(0x7f75e40fdde0 msgr2=0x7f75e4100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:50.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.590+0000 7f75eb90e640 1 -- 172.21.15.29:0/2823106932 shutdown_connections 2024-07-27T06:46:50.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.590+0000 7f75eb90e640 1 -- 172.21.15.29:0/2823106932 wait complete. 2024-07-27T06:46:50.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.591+0000 7f75eb90e640 1 Processor -- start 2024-07-27T06:46:50.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.591+0000 7f75eb90e640 1 -- start start 2024-07-27T06:46:50.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.591+0000 7f75eb90e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75e4102400 0x7f75e4198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:50.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.591+0000 7f75eb90e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e4107d30 0x7f75e41992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:50.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.591+0000 7f75eb90e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e4108680 0x7f75e419e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:50.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.591+0000 7f75eb90e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f75e4112340 con 0x7f75e4102400 2024-07-27T06:46:50.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.591+0000 7f75eb90e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f75e41121c0 con 0x7f75e4108680 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.591+0000 7f75eb90e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f75e41124c0 con 0x7f75e4107d30 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.591+0000 7f75e96aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75e4102400 0x7f75e4198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.591+0000 7f75e96aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75e4102400 0x7f75e4198d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53220/0 (socket says 172.21.15.29:53220) 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.591+0000 7f75e96aa640 1 -- 172.21.15.29:0/1408007329 learned_addr learned my addr 172.21.15.29:0/1408007329 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75e8ea9640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e4107d30 0x7f75e41992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75e9eab640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e4108680 0x7f75e419e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75e96aa640 1 -- 172.21.15.29:0/1408007329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e4107d30 msgr2=0x7f75e41992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75e96aa640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e4107d30 0x7f75e41992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75e96aa640 1 -- 172.21.15.29:0/1408007329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e4108680 msgr2=0x7f75e419e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75e96aa640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e4108680 0x7f75e419e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75e96aa640 1 -- 172.21.15.29:0/1408007329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e419ebe0 con 0x7f75e4102400 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75e8ea9640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e4107d30 0x7f75e41992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:50.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75e96aa640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75e4102400 0x7f75e4198d40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f75e000d950 tx=0x7f75e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:50.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75da7fc640 1 -- 172.21.15.29:0/1408007329 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e0017030 con 0x7f75e4102400 2024-07-27T06:46:50.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75eb90e640 1 -- 172.21.15.29:0/1408007329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75e419eed0 con 0x7f75e4102400 2024-07-27T06:46:50.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75da7fc640 1 -- 172.21.15.29:0/1408007329 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75e0004020 con 0x7f75e4102400 2024-07-27T06:46:50.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75da7fc640 1 -- 172.21.15.29:0/1408007329 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e00047a0 con 0x7f75e4102400 2024-07-27T06:46:50.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.592+0000 7f75eb90e640 1 -- 172.21.15.29:0/1408007329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e41a56f0 con 0x7f75e4102400 2024-07-27T06:46:50.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.593+0000 7f75eb90e640 1 -- 172.21.15.29:0/1408007329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75e4102810 con 0x7f75e4102400 2024-07-27T06:46:50.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.595+0000 7f75da7fc640 1 -- 172.21.15.29:0/1408007329 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f75e0009a30 con 0x7f75e4102400 2024-07-27T06:46:50.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.595+0000 7f75da7fc640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f75d00b3c20 0x7f75d00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:50.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.596+0000 7f75e8ea9640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f75d00b3c20 0x7f75d00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:50.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.596+0000 7f75da7fc640 1 -- 172.21.15.29:0/1408007329 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f75e00e3f40 con 0x7f75e4102400 2024-07-27T06:46:50.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.596+0000 7f75e8ea9640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f75d00b3c20 0x7f75d00b6130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f75d402a9e0 tx=0x7f75d4031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:50.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.598+0000 7f75da7fc640 1 -- 172.21.15.29:0/1408007329 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f75e00ab480 con 0x7f75e4102400 2024-07-27T06:46:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:50 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2681252963' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-27T06:46:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:50 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2681252963' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-27T06:46:50.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.795+0000 7f75eb90e640 1 -- 172.21.15.29:0/1408007329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75e402e0c0 con 0x7f75e4102400 2024-07-27T06:46:50.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.796+0000 7f75da7fc640 1 -- 172.21.15.29:0/1408007329 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+23026 (secure 0 0 0) 0x7f75e00aad80 con 0x7f75e4102400 2024-07-27T06:46:50.798 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:50.798 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","created":"2024-07-27T06:33:57.535327+0000","modified":"2024-07-27T06:45:09.477077+0000","last_up_change":"2024-07-27T06:42:05.757156+0000","last_in_change":"2024-07-27T06:41:46.423732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T06:38:21.606527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-27T06:43:07.755490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-27T06:43:11.658694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06833d24-2e1a-4199-9338-7d6883d1bf9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6803","nonce":386121641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6805","nonce":386121641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6809","nonce":386121641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6807","nonce":386121641}]},"public_addr":"172.21.15.29:6803/386121641","cluster_addr":"172.21.15.29:6805/386121641","heartbeat_back_addr":"172.21.15.29:6809/386121641","heartbeat_front_addr":"172.21.15.29:6807/386121641","state":["exists","up"]},{"osd":1,"uuid":"906aba19-80dc-41b5-953c-af6dca604b05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6801","nonce":2795953749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6803","nonce":2795953749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6807","nonce":2795953749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6805","nonce":2795953749}]},"public_addr":"172.21.15.47:6801/2795953749","cluster_addr":"172.21.15.47:6803/2795953749","heartbeat_back_addr":"172.21.15.47:6807/2795953749","heartbeat_front_addr":"172.21.15.47:6805/2795953749","state":["exists","up"]},{"osd":2,"uuid":"194ae5a5-aebf-4cff-9b99-585a3073a269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6801","nonce":2202565078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6803","nonce":2202565078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6807","nonce":2202565078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6805","nonce":2202565078}]},"public_addr":"172.21.15.77:6801/2202565078","cluster_addr":"172.21.15.77:6803/2202565078","heartbeat_back_addr":"172.21.15.77:6807/2202565078","heartbeat_front_addr":"172.21.15.77:6805/2202565078","state":["exists","up"]},{"osd":3,"uuid":"3c22ce9c-963e-4f72-86e5-360e781526db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6811","nonce":3757476312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6813","nonce":3757476312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6817","nonce":3757476312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6815","nonce":3757476312}]},"public_addr":"172.21.15.29:6811/3757476312","cluster_addr":"172.21.15.29:6813/3757476312","heartbeat_back_addr":"172.21.15.29:6817/3757476312","heartbeat_front_addr":"172.21.15.29:6815/3757476312","state":["exists","up"]},{"osd":4,"uuid":"cd4b585f-e7a0-4d82-873d-3a9269ebac3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6809","nonce":1379363720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6811","nonce":1379363720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6815","nonce":1379363720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6813","nonce":1379363720}]},"public_addr":"172.21.15.47:6809/1379363720","cluster_addr":"172.21.15.47:6811/1379363720","heartbeat_back_addr":"172.21.15.47:6815/1379363720","heartbeat_front_addr":"172.21.15.47:6813/1379363720","state":["exists","up"]},{"osd":5,"uuid":"40a3eb14-dfaa-4523-8c28-d74c19293c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6809","nonce":1338438780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6811","nonce":1338438780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6815","nonce":1338438780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6813","nonce":1338438780}]},"public_addr":"172.21.15.77:6809/1338438780","cluster_addr":"172.21.15.77:6811/1338438780","heartbeat_back_addr":"172.21.15.77:6815/1338438780","heartbeat_front_addr":"172.21.15.77:6813/1338438780","state":["exists","up"]},{"osd":6,"uuid":"91e85be2-10d1-4209-aa33-17da8f8bd8bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6819","nonce":2679402148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6821","nonce":2679402148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6825","nonce":2679402148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6823","nonce":2679402148}]},"public_addr":"172.21.15.29:6819/2679402148","cluster_addr":"172.21.15.29:6821/2679402148","heartbeat_back_addr":"172.21.15.29:6825/2679402148","heartbeat_front_addr":"172.21.15.29:6823/2679402148","state":["exists","up"]},{"osd":7,"uuid":"c655b450-0cc4-45d4-95ea-93a57fe8e4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6817","nonce":2439195589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6819","nonce":2439195589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6823","nonce":2439195589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6821","nonce":2439195589}]},"public_addr":"172.21.15.47:6817/2439195589","cluster_addr":"172.21.15.47:6819/2439195589","heartbeat_back_addr":"172.21.15.47:6823/2439195589","heartbeat_front_addr":"172.21.15.47:6821/2439195589","state":["exists","up"]},{"osd":8,"uuid":"a7a95b51-f32c-433e-a197-feb6d47b6c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6817","nonce":1422290905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6819","nonce":1422290905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6823","nonce":1422290905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6821","nonce":1422290905}]},"public_addr":"172.21.15.77:6817/1422290905","cluster_addr":"172.21.15.77:6819/1422290905","heartbeat_back_addr":"172.21.15.77:6823/1422290905","heartbeat_front_addr":"172.21.15.77:6821/1422290905","state":["exists","up"]},{"osd":9,"uuid":"f352242b-0d2e-41b3-9c83-436cf432b0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6827","nonce":1307316384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6829","nonce":1307316384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6833","nonce":1307316384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6831","nonce":1307316384}]},"public_addr":"172.21.15.29:6827/1307316384","cluster_addr":"172.21.15.29:6829/1307316384","heartbeat_back_addr":"172.21.15.29:6833/1307316384","heartbeat_front_addr":"172.21.15.29:6831/1307316384","state":["exists","up"]},{"osd":10,"uuid":"4148c2df-c3e9-4a47-88f7-36cf3eab8d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6825","nonce":2133726850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6827","nonce":2133726850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6831","nonce":2133726850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6829","nonce":2133726850}]},"public_addr":"172.21.15.47:6825/2133726850","cluster_addr":"172.21.15.47:6827/2133726850","heartbeat_back_addr":"172.21.15.47:6831/2133726850","heartbeat_front_addr":"172.21.15.47:6829/2133726850","state":["exists","up"]},{"osd":11,"uuid":"5e8b1da3-5f65-4fe9-8722-b35921cbd8c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6825","nonce":3228299327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6827","nonce":3228299327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6831","nonce":3228299327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6829","nonce":3228299327}]},"public_addr":"172.21.15.77:6825/3228299327","cluster_addr":"172.21.15.77:6827/3228299327","heartbeat_back_addr":"172.21.15.77:6831/3228299327","heartbeat_front_addr":"172.21.15.77:6829/3228299327","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:32.525206+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:56.489640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:18.887845+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:43.727678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:08.755553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:32.923461+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:59.298439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:23.944848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:48.883050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:14.566310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:39.557809+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:42:03.813903+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7},{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2378279024":"2024-07-28T06:36:07.527503+0000","172.21.15.29:6801/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3527133965":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2262661620":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/3156863587":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2424241521":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3881857930":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/398508875":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6801/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6800/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2679856281":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6801/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2708687265":"2024-07-28T06:34:58.250198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T06:46:50.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.798+0000 7f75eb90e640 1 -- 172.21.15.29:0/1408007329 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f75d00b3c20 msgr2=0x7f75d00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:50.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.798+0000 7f75eb90e640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f75d00b3c20 0x7f75d00b6130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f75d402a9e0 tx=0x7f75d4031400 comp rx=0 tx=0).stop 2024-07-27T06:46:50.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.798+0000 7f75eb90e640 1 -- 172.21.15.29:0/1408007329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75e4102400 msgr2=0x7f75e4198d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:50.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.798+0000 7f75eb90e640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75e4102400 0x7f75e4198d40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f75e000d950 tx=0x7f75e000de10 comp rx=0 tx=0).stop 2024-07-27T06:46:50.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.798+0000 7f75eb90e640 1 -- 172.21.15.29:0/1408007329 shutdown_connections 2024-07-27T06:46:50.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.798+0000 7f75eb90e640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e4108680 0x7f75e419e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.798+0000 7f75eb90e640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f75d00b3c20 0x7f75d00b6130 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.798+0000 7f75eb90e640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75e4107d30 0x7f75e41992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.798+0000 7f75eb90e640 1 --2- 172.21.15.29:0/1408007329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75e4102400 0x7f75e4198d40 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.798+0000 7f75eb90e640 1 -- 172.21.15.29:0/1408007329 >> 172.21.15.29:0/1408007329 conn(0x7f75e40fdde0 msgr2=0x7f75e41052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:50.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.798+0000 7f75eb90e640 1 -- 172.21.15.29:0/1408007329 shutdown_connections 2024-07-27T06:46:50.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.798+0000 7f75eb90e640 1 -- 172.21.15.29:0/1408007329 wait complete. 2024-07-27T06:46:50.812 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-27T06:38:21.606527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-07-27T06:43:07.755490+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-07-27T06:43:11.658694+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-27T06:46:50.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-07-27T06:46:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:50 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2681252963' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-27T06:46:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.929+0000 7f7e960ec640 1 -- 172.21.15.29:0/1807754134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e90104350 msgr2=0x7f7e90104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.929+0000 7f7e960ec640 1 --2- 172.21.15.29:0/1807754134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e90104350 0x7f7e90104760 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009900 tx=0x7f7e8002a410 comp rx=0 tx=0).stop 2024-07-27T06:46:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.929+0000 7f7e960ec640 1 -- 172.21.15.29:0/1807754134 shutdown_connections 2024-07-27T06:46:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.929+0000 7f7e960ec640 1 --2- 172.21.15.29:0/1807754134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e9010d7b0 0x7f7e9010fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.929+0000 7f7e960ec640 1 --2- 172.21.15.29:0/1807754134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e90104ca0 0x7f7e9010d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.929+0000 7f7e960ec640 1 --2- 172.21.15.29:0/1807754134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e90104350 0x7f7e90104760 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.929+0000 7f7e960ec640 1 -- 172.21.15.29:0/1807754134 >> 172.21.15.29:0/1807754134 conn(0x7f7e900fde20 msgr2=0x7f7e90100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.929+0000 7f7e960ec640 1 -- 172.21.15.29:0/1807754134 shutdown_connections 2024-07-27T06:46:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.929+0000 7f7e960ec640 1 -- 172.21.15.29:0/1807754134 wait complete. 2024-07-27T06:46:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.930+0000 7f7e960ec640 1 Processor -- start 2024-07-27T06:46:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.930+0000 7f7e960ec640 1 -- start start 2024-07-27T06:46:50.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.930+0000 7f7e960ec640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e90104350 0x7f7e90199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:50.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.930+0000 7f7e960ec640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e90104ca0 0x7f7e9019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:50.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.930+0000 7f7e960ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e9010d7b0 0x7f7e9019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:50.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.930+0000 7f7e960ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7e90113440 con 0x7f7e9010d7b0 2024-07-27T06:46:50.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.930+0000 7f7e960ec640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7e901132c0 con 0x7f7e90104350 2024-07-27T06:46:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.930+0000 7f7e960ec640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7e901135c0 con 0x7f7e90104ca0 2024-07-27T06:46:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e9010d7b0 0x7f7e9019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e9010d7b0 0x7f7e9019f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53242/0 (socket says 172.21.15.29:53242) 2024-07-27T06:46:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8ffff640 1 -- 172.21.15.29:0/3395678500 learned_addr learned my addr 172.21.15.29:0/3395678500 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8f7fe640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e90104350 0x7f7e90199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e86ffd640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e90104ca0 0x7f7e9019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8ffff640 1 -- 172.21.15.29:0/3395678500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e90104ca0 msgr2=0x7f7e9019a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8ffff640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e90104ca0 0x7f7e9019a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8ffff640 1 -- 172.21.15.29:0/3395678500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e90104350 msgr2=0x7f7e90199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8ffff640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e90104350 0x7f7e90199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8ffff640 1 -- 172.21.15.29:0/3395678500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9019fd90 con 0x7f7e9010d7b0 2024-07-27T06:46:50.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e86ffd640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e90104ca0 0x7f7e9019a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:46:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8f7fe640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e90104350 0x7f7e90199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:46:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8ffff640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e9010d7b0 0x7f7e9019f690 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800eb10 tx=0x7f7e8800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8d7fa640 1 -- 172.21.15.29:0/3395678500 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e88004b20 con 0x7f7e9010d7b0 2024-07-27T06:46:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8d7fa640 1 -- 172.21.15.29:0/3395678500 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e88004cc0 con 0x7f7e9010d7b0 2024-07-27T06:46:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e960ec640 1 -- 172.21.15.29:0/3395678500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e901a0080 con 0x7f7e9010d7b0 2024-07-27T06:46:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e8d7fa640 1 -- 172.21.15.29:0/3395678500 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e88010670 con 0x7f7e9010d7b0 2024-07-27T06:46:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.931+0000 7f7e960ec640 1 -- 172.21.15.29:0/3395678500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e901a6810 con 0x7f7e9010d7b0 2024-07-27T06:46:50.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.932+0000 7f7e960ec640 1 -- 172.21.15.29:0/3395678500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e9006bb40 con 0x7f7e9010d7b0 2024-07-27T06:46:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.935+0000 7f7e8d7fa640 1 -- 172.21.15.29:0/3395678500 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f7e88005000 con 0x7f7e9010d7b0 2024-07-27T06:46:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.935+0000 7f7e8d7fa640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7e7c0b3c20 0x7f7e7c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.935+0000 7f7e8f7fe640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7e7c0b3c20 0x7f7e7c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.935+0000 7f7e8d7fa640 1 -- 172.21.15.29:0/3395678500 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7e880e46b0 con 0x7f7e9010d7b0 2024-07-27T06:46:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.935+0000 7f7e8f7fe640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7e7c0b3c20 0x7f7e7c0b6130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7e800096c0 tx=0x7f7e80031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:50.938+0000 7f7e8d7fa640 1 -- 172.21.15.29:0/3395678500 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7e88055600 con 0x7f7e9010d7b0 2024-07-27T06:46:51.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.130+0000 7f7e960ec640 1 -- 172.21.15.29:0/3395678500 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7e9019af80 con 0x7f7e9010d7b0 2024-07-27T06:46:51.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.131+0000 7f7e8d7fa640 1 -- 172.21.15.29:0/3395678500 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f7e880abb90 con 0x7f7e9010d7b0 2024-07-27T06:46:51.132 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-07-27T06:46:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.134+0000 7f7e960ec640 1 -- 172.21.15.29:0/3395678500 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7e7c0b3c20 msgr2=0x7f7e7c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.134+0000 7f7e960ec640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7e7c0b3c20 0x7f7e7c0b6130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7e800096c0 tx=0x7f7e80031400 comp rx=0 tx=0).stop 2024-07-27T06:46:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.134+0000 7f7e960ec640 1 -- 172.21.15.29:0/3395678500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e9010d7b0 msgr2=0x7f7e9019f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.134+0000 7f7e960ec640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e9010d7b0 0x7f7e9019f690 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800eb10 tx=0x7f7e8800c2d0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.134+0000 7f7e960ec640 1 -- 172.21.15.29:0/3395678500 shutdown_connections 2024-07-27T06:46:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.134+0000 7f7e960ec640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7e9010d7b0 0x7f7e9019f690 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.134+0000 7f7e960ec640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7e7c0b3c20 0x7f7e7c0b6130 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.134+0000 7f7e960ec640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e90104ca0 0x7f7e9019a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.134+0000 7f7e960ec640 1 --2- 172.21.15.29:0/3395678500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7e90104350 0x7f7e90199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.134+0000 7f7e960ec640 1 -- 172.21.15.29:0/3395678500 >> 172.21.15.29:0/3395678500 conn(0x7f7e900fde20 msgr2=0x7f7e90107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.134+0000 7f7e960ec640 1 -- 172.21.15.29:0/3395678500 shutdown_connections 2024-07-27T06:46:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.134+0000 7f7e960ec640 1 -- 172.21.15.29:0/3395678500 wait complete. 2024-07-27T06:46:51.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-07-27T06:46:51.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.264+0000 7f329c88c640 1 -- 172.21.15.29:0/742858453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329806b790 msgr2=0x7f329806bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:51.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.264+0000 7f329c88c640 1 --2- 172.21.15.29:0/742858453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329806b790 0x7f329806bba0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3288009900 tx=0x7f328802a410 comp rx=0 tx=0).stop 2024-07-27T06:46:51.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.264+0000 7f329c88c640 1 -- 172.21.15.29:0/742858453 shutdown_connections 2024-07-27T06:46:51.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.264+0000 7f329c88c640 1 --2- 172.21.15.29:0/742858453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3298109780 0x7f329810ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.264+0000 7f329c88c640 1 --2- 172.21.15.29:0/742858453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3298108e30 0x7f3298109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.264+0000 7f329c88c640 1 --2- 172.21.15.29:0/742858453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f329806b790 0x7f329806bba0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.264+0000 7f329c88c640 1 -- 172.21.15.29:0/742858453 >> 172.21.15.29:0/742858453 conn(0x7f3298078fa0 msgr2=0x7f32980793b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:51.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.264+0000 7f329c88c640 1 -- 172.21.15.29:0/742858453 shutdown_connections 2024-07-27T06:46:51.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.264+0000 7f329c88c640 1 -- 172.21.15.29:0/742858453 wait complete. 2024-07-27T06:46:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.265+0000 7f329c88c640 1 Processor -- start 2024-07-27T06:46:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.265+0000 7f329c88c640 1 -- start start 2024-07-27T06:46:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f329c88c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329806b790 0x7f3298199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f329c88c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3298108e30 0x7f329819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f329c88c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3298109780 0x7f329819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f329c88c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f32981133f0 con 0x7f3298108e30 2024-07-27T06:46:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f329c88c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3298113270 con 0x7f329806b790 2024-07-27T06:46:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f329c88c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3298113570 con 0x7f3298109780 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f3295d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3298108e30 0x7f329819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f3295d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3298108e30 0x7f329819a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53258/0 (socket says 172.21.15.29:53258) 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f3295d9b640 1 -- 172.21.15.29:0/2460789574 learned_addr learned my addr 172.21.15.29:0/2460789574 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f329659c640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329806b790 0x7f3298199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f3295d9b640 1 -- 172.21.15.29:0/2460789574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3298109780 msgr2=0x7f329819f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f3295d9b640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3298109780 0x7f329819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f3295d9b640 1 -- 172.21.15.29:0/2460789574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329806b790 msgr2=0x7f3298199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f3295d9b640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329806b790 0x7f3298199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f3295d9b640 1 -- 172.21.15.29:0/2460789574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329819fd40 con 0x7f3298108e30 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f3295d9b640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3298108e30 0x7f329819a3b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f328c00e9f0 tx=0x7f328c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f32877fe640 1 -- 172.21.15.29:0/2460789574 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c00ccc0 con 0x7f3298108e30 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f32877fe640 1 -- 172.21.15.29:0/2460789574 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f328c0054d0 con 0x7f3298108e30 2024-07-27T06:46:51.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f329c88c640 1 -- 172.21.15.29:0/2460789574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32981a0030 con 0x7f3298108e30 2024-07-27T06:46:51.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f32877fe640 1 -- 172.21.15.29:0/2460789574 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c010430 con 0x7f3298108e30 2024-07-27T06:46:51.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.266+0000 7f329c88c640 1 -- 172.21.15.29:0/2460789574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32981a67c0 con 0x7f3298108e30 2024-07-27T06:46:51.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.267+0000 7f329c88c640 1 -- 172.21.15.29:0/2460789574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f329806bba0 con 0x7f3298108e30 2024-07-27T06:46:51.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.269+0000 7f32877fe640 1 -- 172.21.15.29:0/2460789574 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f328c0046c0 con 0x7f3298108e30 2024-07-27T06:46:51.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.270+0000 7f32877fe640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f32800b3ca0 0x7f32800b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:51.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.270+0000 7f32877fe640 1 -- 172.21.15.29:0/2460789574 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f328c0e4360 con 0x7f3298108e30 2024-07-27T06:46:51.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.270+0000 7f329659c640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f32800b3ca0 0x7f32800b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:51.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.270+0000 7f329659c640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f32800b3ca0 0x7f32800b61b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3288031440 tx=0x7f3288009560 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:51.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.273+0000 7f32877fe640 1 -- 172.21.15.29:0/2460789574 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f328c055600 con 0x7f3298108e30 2024-07-27T06:46:51.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.471+0000 7f329c88c640 1 -- 172.21.15.29:0/2460789574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f329802e0c0 con 0x7f3298108e30 2024-07-27T06:46:51.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.471+0000 7f32877fe640 1 -- 172.21.15.29:0/2460789574 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) ==== 104+0+11 (secure 0 0 0) 0x7f328c0ab840 con 0x7f3298108e30 2024-07-27T06:46:51.472 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-07-27T06:46:51.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.473+0000 7f329c88c640 1 -- 172.21.15.29:0/2460789574 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f32800b3ca0 msgr2=0x7f32800b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:51.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.473+0000 7f329c88c640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f32800b3ca0 0x7f32800b61b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3288031440 tx=0x7f3288009560 comp rx=0 tx=0).stop 2024-07-27T06:46:51.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.473+0000 7f329c88c640 1 -- 172.21.15.29:0/2460789574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3298108e30 msgr2=0x7f329819a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:51.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.473+0000 7f329c88c640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3298108e30 0x7f329819a3b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f328c00e9f0 tx=0x7f328c00eeb0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.473+0000 7f329c88c640 1 -- 172.21.15.29:0/2460789574 shutdown_connections 2024-07-27T06:46:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.473+0000 7f329c88c640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f32800b3ca0 0x7f32800b61b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.473+0000 7f329c88c640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3298109780 0x7f329819f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.473+0000 7f329c88c640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3298108e30 0x7f329819a3b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.473+0000 7f329c88c640 1 --2- 172.21.15.29:0/2460789574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f329806b790 0x7f3298199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.473+0000 7f329c88c640 1 -- 172.21.15.29:0/2460789574 >> 172.21.15.29:0/2460789574 conn(0x7f3298078fa0 msgr2=0x7f3298106e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.474+0000 7f329c88c640 1 -- 172.21.15.29:0/2460789574 shutdown_connections 2024-07-27T06:46:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.474+0000 7f329c88c640 1 -- 172.21.15.29:0/2460789574 wait complete. 2024-07-27T06:46:51.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-07-27T06:46:51.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.603+0000 7f57c74b7640 1 -- 172.21.15.29:0/547089687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57c0109780 msgr2=0x7f57c010ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.603+0000 7f57c74b7640 1 --2- 172.21.15.29:0/547089687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57c0109780 0x7f57c010ff90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009ca0 tx=0x7f57b802a8c0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.603+0000 7f57c74b7640 1 -- 172.21.15.29:0/547089687 shutdown_connections 2024-07-27T06:46:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.603+0000 7f57c74b7640 1 --2- 172.21.15.29:0/547089687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57c0109780 0x7f57c010ff90 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.603+0000 7f57c74b7640 1 --2- 172.21.15.29:0/547089687 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57c0108e30 0x7f57c0109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.603+0000 7f57c74b7640 1 --2- 172.21.15.29:0/547089687 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57c006b790 0x7f57c006bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.603+0000 7f57c74b7640 1 -- 172.21.15.29:0/547089687 >> 172.21.15.29:0/547089687 conn(0x7f57c0078fa0 msgr2=0x7f57c00793b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.604+0000 7f57c74b7640 1 -- 172.21.15.29:0/547089687 shutdown_connections 2024-07-27T06:46:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.604+0000 7f57c74b7640 1 -- 172.21.15.29:0/547089687 wait complete. 2024-07-27T06:46:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.604+0000 7f57c74b7640 1 Processor -- start 2024-07-27T06:46:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.604+0000 7f57c74b7640 1 -- start start 2024-07-27T06:46:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c74b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57c006b790 0x7f57c0199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c74b7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57c0108e30 0x7f57c019a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c74b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57c0109780 0x7f57c019f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c74b7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f57c01133d0 con 0x7f57c0109780 2024-07-27T06:46:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c74b7640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f57c0113250 con 0x7f57c0108e30 2024-07-27T06:46:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c74b7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f57c0113550 con 0x7f57c006b790 2024-07-27T06:46:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c5a54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57c0109780 0x7f57c019f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c5a54640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57c0109780 0x7f57c019f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53278/0 (socket says 172.21.15.29:53278) 2024-07-27T06:46:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c5a54640 1 -- 172.21.15.29:0/842555962 learned_addr learned my addr 172.21.15.29:0/842555962 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c4a52640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57c0108e30 0x7f57c019a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:51.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c5253640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57c006b790 0x7f57c0199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c5a54640 1 -- 172.21.15.29:0/842555962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57c006b790 msgr2=0x7f57c0199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c5a54640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57c006b790 0x7f57c0199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c5a54640 1 -- 172.21.15.29:0/842555962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57c0108e30 msgr2=0x7f57c019a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c5a54640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57c0108e30 0x7f57c019a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c5a54640 1 -- 172.21.15.29:0/842555962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c019fd20 con 0x7f57c0109780 2024-07-27T06:46:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c4a52640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57c0108e30 0x7f57c019a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c5a54640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57c0109780 0x7f57c019f620 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f57b8005cc0 tx=0x7f57b8031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57b67fc640 1 -- 172.21.15.29:0/842555962 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b8003b70 con 0x7f57c0109780 2024-07-27T06:46:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57b67fc640 1 -- 172.21.15.29:0/842555962 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57b80318f0 con 0x7f57c0109780 2024-07-27T06:46:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.605+0000 7f57c74b7640 1 -- 172.21.15.29:0/842555962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c019ffb0 con 0x7f57c0109780 2024-07-27T06:46:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.606+0000 7f57b67fc640 1 -- 172.21.15.29:0/842555962 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b802f950 con 0x7f57c0109780 2024-07-27T06:46:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.606+0000 7f57c74b7640 1 -- 172.21.15.29:0/842555962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c01a67a0 con 0x7f57c0109780 2024-07-27T06:46:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.607+0000 7f57c74b7640 1 -- 172.21.15.29:0/842555962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c006bba0 con 0x7f57c0109780 2024-07-27T06:46:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.608+0000 7f57b67fc640 1 -- 172.21.15.29:0/842555962 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f57b802fb10 con 0x7f57c0109780 2024-07-27T06:46:51.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.609+0000 7f57b67fc640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f57ac0b3ca0 0x7f57ac0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:51.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.610+0000 7f57c5253640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f57ac0b3ca0 0x7f57ac0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:51.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.610+0000 7f57b67fc640 1 -- 172.21.15.29:0/842555962 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f57b8105330 con 0x7f57c0109780 2024-07-27T06:46:51.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.610+0000 7f57c5253640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f57ac0b3ca0 0x7f57ac0b61b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009b60 tx=0x7f57bc005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:51.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.613+0000 7f57b67fc640 1 -- 172.21.15.29:0/842555962 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f57b80cc870 con 0x7f57c0109780 2024-07-27T06:46:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:51 smithi047 ceph-mon[36835]: pgmap v392: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:46:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:51 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2681252963' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-27T06:46:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:51 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:46:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:51 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1408007329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:46:51.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:51 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3395678500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-27T06:46:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:51 smithi029 ceph-mon[32610]: pgmap v392: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:46:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:51 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2681252963' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-27T06:46:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:51 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:46:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:51 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1408007329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:46:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:51 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3395678500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-27T06:46:51.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.807+0000 7f57c74b7640 1 -- 172.21.15.29:0/842555962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f57c002e0c0 con 0x7f57c0109780 2024-07-27T06:46:51.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.807+0000 7f57b67fc640 1 -- 172.21.15.29:0/842555962 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) ==== 100+0+11 (secure 0 0 0) 0x7f57b80cc170 con 0x7f57c0109780 2024-07-27T06:46:51.808 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-07-27T06:46:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.809+0000 7f57c74b7640 1 -- 172.21.15.29:0/842555962 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f57ac0b3ca0 msgr2=0x7f57ac0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.810+0000 7f57c74b7640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f57ac0b3ca0 0x7f57ac0b61b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009b60 tx=0x7f57bc005f90 comp rx=0 tx=0).stop 2024-07-27T06:46:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.810+0000 7f57c74b7640 1 -- 172.21.15.29:0/842555962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57c0109780 msgr2=0x7f57c019f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.810+0000 7f57c74b7640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57c0109780 0x7f57c019f620 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f57b8005cc0 tx=0x7f57b8031700 comp rx=0 tx=0).stop 2024-07-27T06:46:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.810+0000 7f57c74b7640 1 -- 172.21.15.29:0/842555962 shutdown_connections 2024-07-27T06:46:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.810+0000 7f57c74b7640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57c0109780 0x7f57c019f620 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.810+0000 7f57c74b7640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57c0108e30 0x7f57c019a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.810+0000 7f57c74b7640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f57ac0b3ca0 0x7f57ac0b61b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.810+0000 7f57c74b7640 1 --2- 172.21.15.29:0/842555962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57c006b790 0x7f57c0199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.810+0000 7f57c74b7640 1 -- 172.21.15.29:0/842555962 >> 172.21.15.29:0/842555962 conn(0x7f57c0078fa0 msgr2=0x7f57c0105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.810+0000 7f57c74b7640 1 -- 172.21.15.29:0/842555962 shutdown_connections 2024-07-27T06:46:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:51.810+0000 7f57c74b7640 1 -- 172.21.15.29:0/842555962 wait complete. 2024-07-27T06:46:51.822 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-07-27T06:46:51.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:46:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:51 smithi077 ceph-mon[37153]: pgmap v392: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T06:46:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:51 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2681252963' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-27T06:46:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:51 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:46:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:51 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1408007329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:46:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:51 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3395678500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-27T06:46:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:52 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2460789574' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-27T06:46:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:52 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/842555962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-27T06:46:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:52 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2460789574' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-27T06:46:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:52 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/842555962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-27T06:46:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:52 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2460789574' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-27T06:46:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:52 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/842555962' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-27T06:46:53.530 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:53 smithi029 ceph-mon[32610]: pgmap v393: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:46:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:53 smithi047 ceph-mon[36835]: pgmap v393: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:46:53.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.770+0000 7fedf259a640 1 -- 172.21.15.29:0/1985183923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedec109820 msgr2=0x7fedec110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.770+0000 7fedf259a640 1 --2- 172.21.15.29:0/1985183923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedec109820 0x7fedec110050 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fede4009ca0 tx=0x7fede402a8e0 comp rx=0 tx=0).stop 2024-07-27T06:46:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.770+0000 7fedf259a640 1 -- 172.21.15.29:0/1985183923 shutdown_connections 2024-07-27T06:46:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.770+0000 7fedf259a640 1 --2- 172.21.15.29:0/1985183923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedec109820 0x7fedec110050 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.770+0000 7fedf259a640 1 --2- 172.21.15.29:0/1985183923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedec108ed0 0x7fedec1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.770+0000 7fedf259a640 1 --2- 172.21.15.29:0/1985183923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedec103590 0x7fedec1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.770+0000 7fedf259a640 1 -- 172.21.15.29:0/1985183923 >> 172.21.15.29:0/1985183923 conn(0x7fedec07b280 msgr2=0x7fedec07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.770+0000 7fedf259a640 1 -- 172.21.15.29:0/1985183923 shutdown_connections 2024-07-27T06:46:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.770+0000 7fedf259a640 1 -- 172.21.15.29:0/1985183923 wait complete. 2024-07-27T06:46:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedf259a640 1 Processor -- start 2024-07-27T06:46:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedf259a640 1 -- start start 2024-07-27T06:46:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedf259a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedec103590 0x7fedec199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedf259a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedec108ed0 0x7fedec19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedf259a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedec109820 0x7fedec19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedf259a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fedec113540 con 0x7fedec103590 2024-07-27T06:46:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedf259a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fedec1133c0 con 0x7fedec109820 2024-07-27T06:46:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedf259a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fedec1136c0 con 0x7fedec108ed0 2024-07-27T06:46:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedec103590 0x7fedec199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedec103590 0x7fedec199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53292/0 (socket says 172.21.15.29:53292) 2024-07-27T06:46:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedebfff640 1 -- 172.21.15.29:0/3099154967 learned_addr learned my addr 172.21.15.29:0/3099154967 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedeb7fe640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedec108ed0 0x7fedec19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.771+0000 7fedf0b37640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedec109820 0x7fedec19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.772+0000 7fedebfff640 1 -- 172.21.15.29:0/3099154967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedec108ed0 msgr2=0x7fedec19a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.772+0000 7fedebfff640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedec108ed0 0x7fedec19a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.772+0000 7fedebfff640 1 -- 172.21.15.29:0/3099154967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedec109820 msgr2=0x7fedec19f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.772+0000 7fedebfff640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedec109820 0x7fedec19f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.772+0000 7fedebfff640 1 -- 172.21.15.29:0/3099154967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedec19fe40 con 0x7fedec103590 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.772+0000 7fedeb7fe640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedec108ed0 0x7fedec19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.772+0000 7fedebfff640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedec103590 0x7fedec199ed0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7feddc00d770 tx=0x7feddc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.772+0000 7fede97fa640 1 -- 172.21.15.29:0/3099154967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc017070 con 0x7fedec103590 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.772+0000 7fedf259a640 1 -- 172.21.15.29:0/3099154967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedec1a0130 con 0x7fedec103590 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.772+0000 7fede97fa640 1 -- 172.21.15.29:0/3099154967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feddc0054d0 con 0x7fedec103590 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.772+0000 7fede97fa640 1 -- 172.21.15.29:0/3099154967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc004c30 con 0x7fedec103590 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.772+0000 7fedf259a640 1 -- 172.21.15.29:0/3099154967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedec1a6890 con 0x7fedec103590 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.773+0000 7fedf259a640 1 -- 172.21.15.29:0/3099154967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedec1039a0 con 0x7fedec103590 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.775+0000 7fede97fa640 1 -- 172.21.15.29:0/3099154967 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7feddc004dd0 con 0x7fedec103590 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.776+0000 7fede97fa640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fedd40b3c20 0x7fedd40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.776+0000 7fedeb7fe640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fedd40b3c20 0x7fedd40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.776+0000 7fede97fa640 1 -- 172.21.15.29:0/3099154967 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7feddc0e3cc0 con 0x7fedec103590 2024-07-27T06:46:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.776+0000 7fedeb7fe640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fedd40b3c20 0x7fedd40b6130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fedec107dd0 tx=0x7fede0008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.779+0000 7fede97fa640 1 -- 172.21.15.29:0/3099154967 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7feddc0ab200 con 0x7fedec103590 2024-07-27T06:46:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:53 smithi077 ceph-mon[37153]: pgmap v393: 97 pgs: 97 active+clean; 644 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:46:54.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:53.999+0000 7fedf259a640 1 -- 172.21.15.29:0/3099154967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedec02e0c0 con 0x7fedec103590 2024-07-27T06:46:54.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.000+0000 7fede97fa640 1 -- 172.21.15.29:0/3099154967 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11957 (secure 0 0 0) 0x7feddc0aab00 con 0x7fedec103590 2024-07-27T06:46:54.001 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:54.001 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-07-27T06:46:50:421100+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:46:49.561737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24670,24658,24646,24617,24613]},"id":1}]} 2024-07-27T06:46:54.001 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-07-27T06:46:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.002+0000 7fedf259a640 1 -- 172.21.15.29:0/3099154967 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fedd40b3c20 msgr2=0x7fedd40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.002+0000 7fedf259a640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fedd40b3c20 0x7fedd40b6130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fedec107dd0 tx=0x7fede0008040 comp rx=0 tx=0).stop 2024-07-27T06:46:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.002+0000 7fedf259a640 1 -- 172.21.15.29:0/3099154967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedec103590 msgr2=0x7fedec199ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.002+0000 7fedf259a640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedec103590 0x7fedec199ed0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7feddc00d770 tx=0x7feddc00dc30 comp rx=0 tx=0).stop 2024-07-27T06:46:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.002+0000 7fedf259a640 1 -- 172.21.15.29:0/3099154967 shutdown_connections 2024-07-27T06:46:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.002+0000 7fedf259a640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedec109820 0x7fedec19f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.002+0000 7fedf259a640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fedd40b3c20 0x7fedd40b6130 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.002+0000 7fedf259a640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fedec108ed0 0x7fedec19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.002+0000 7fedf259a640 1 --2- 172.21.15.29:0/3099154967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedec103590 0x7fedec199ed0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.002+0000 7fedf259a640 1 -- 172.21.15.29:0/3099154967 >> 172.21.15.29:0/3099154967 conn(0x7fedec07b280 msgr2=0x7fedec104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:54.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.002+0000 7fedf259a640 1 -- 172.21.15.29:0/3099154967 shutdown_connections 2024-07-27T06:46:54.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.002+0000 7fedf259a640 1 -- 172.21.15.29:0/3099154967 wait complete. 2024-07-27T06:46:54.612 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:54 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3099154967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:46:54.644 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-07-27T06:46:54.644 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-07-27T06:46:54.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-27T06:46:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:54 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3099154967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:46:54.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.765+0000 7fae9e63d640 1 -- 172.21.15.29:0/3230177012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae98108680 msgr2=0x7fae9810eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:54.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.765+0000 7fae9e63d640 1 --2- 172.21.15.29:0/3230177012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae98108680 0x7fae9810eeb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fae90009ca0 tx=0x7fae9002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.765+0000 7fae9e63d640 1 -- 172.21.15.29:0/3230177012 shutdown_connections 2024-07-27T06:46:54.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.765+0000 7fae9e63d640 1 --2- 172.21.15.29:0/3230177012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae98108680 0x7fae9810eeb0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.765+0000 7fae9e63d640 1 --2- 172.21.15.29:0/3230177012 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae98107d30 0x7fae98108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.765+0000 7fae9e63d640 1 --2- 172.21.15.29:0/3230177012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae98102ad0 0x7fae98102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.765+0000 7fae9e63d640 1 -- 172.21.15.29:0/3230177012 >> 172.21.15.29:0/3230177012 conn(0x7fae980796f0 msgr2=0x7fae98079b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:54.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.765+0000 7fae9e63d640 1 -- 172.21.15.29:0/3230177012 shutdown_connections 2024-07-27T06:46:54.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.765+0000 7fae9e63d640 1 -- 172.21.15.29:0/3230177012 wait complete. 2024-07-27T06:46:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae9e63d640 1 Processor -- start 2024-07-27T06:46:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae9e63d640 1 -- start start 2024-07-27T06:46:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae9e63d640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae98102ad0 0x7fae9819ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae9e63d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae98107d30 0x7fae9819b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae9e63d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae98108680 0x7fae9819b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae9e63d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fae98112330 con 0x7fae98108680 2024-07-27T06:46:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae9e63d640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fae981121b0 con 0x7fae98102ad0 2024-07-27T06:46:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae9e63d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fae981124b0 con 0x7fae98107d30 2024-07-27T06:46:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae9cbda640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae98108680 0x7fae9819b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae9cbda640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae98108680 0x7fae9819b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53308/0 (socket says 172.21.15.29:53308) 2024-07-27T06:46:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae9cbda640 1 -- 172.21.15.29:0/785158200 learned_addr learned my addr 172.21.15.29:0/785158200 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae97fff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae98102ad0 0x7fae9819ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.766+0000 7fae977fe640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae98107d30 0x7fae9819b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.767+0000 7fae9cbda640 1 -- 172.21.15.29:0/785158200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae98107d30 msgr2=0x7fae9819b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.767+0000 7fae9cbda640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae98107d30 0x7fae9819b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.767+0000 7fae9cbda640 1 -- 172.21.15.29:0/785158200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae98102ad0 msgr2=0x7fae9819ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.767+0000 7fae9cbda640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae98102ad0 0x7fae9819ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.767+0000 7fae9cbda640 1 -- 172.21.15.29:0/785158200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae981a5ee0 con 0x7fae98108680 2024-07-27T06:46:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.767+0000 7fae9cbda640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae98108680 0x7fae9819b6a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fae90005cc0 tx=0x7fae9002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.767+0000 7fae957fa640 1 -- 172.21.15.29:0/785158200 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90003bf0 con 0x7fae98108680 2024-07-27T06:46:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.767+0000 7fae957fa640 1 -- 172.21.15.29:0/785158200 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae90003d90 con 0x7fae98108680 2024-07-27T06:46:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.767+0000 7fae9e63d640 1 -- 172.21.15.29:0/785158200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae981a6170 con 0x7fae98108680 2024-07-27T06:46:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.767+0000 7fae957fa640 1 -- 172.21.15.29:0/785158200 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9002fbc0 con 0x7fae98108680 2024-07-27T06:46:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.767+0000 7fae9e63d640 1 -- 172.21.15.29:0/785158200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae981a6710 con 0x7fae98108680 2024-07-27T06:46:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.768+0000 7fae9e63d640 1 -- 172.21.15.29:0/785158200 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae98102ee0 con 0x7fae98108680 2024-07-27T06:46:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.771+0000 7fae957fa640 1 -- 172.21.15.29:0/785158200 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7fae9002fd60 con 0x7fae98108680 2024-07-27T06:46:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.771+0000 7fae957fa640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fae800b3ca0 0x7fae800b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.771+0000 7fae97fff640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fae800b3ca0 0x7fae800b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.771+0000 7fae957fa640 1 -- 172.21.15.29:0/785158200 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fae90105540 con 0x7fae98108680 2024-07-27T06:46:54.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.772+0000 7fae97fff640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fae800b3ca0 0x7fae800b61b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fae981022a0 tx=0x7fae8800b410 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:54.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.774+0000 7fae957fa640 1 -- 172.21.15.29:0/785158200 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fae900cca80 con 0x7fae98108680 2024-07-27T06:46:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:54 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3099154967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:46:54.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.964+0000 7fae9e63d640 1 -- 172.21.15.29:0/785158200 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fae98063230 con 0x7fae800b3ca0 2024-07-27T06:46:54.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.969+0000 7fae957fa640 1 -- 172.21.15.29:0/785158200 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259157 (secure 0 0 0) 0x7fae98063230 con 0x7fae800b3ca0 2024-07-27T06:46:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:54.976 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-07-27T06:46:54.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.972+0000 7fae9e63d640 1 -- 172.21.15.29:0/785158200 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fae800b3ca0 msgr2=0x7fae800b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:54.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.972+0000 7fae9e63d640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fae800b3ca0 0x7fae800b61b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fae981022a0 tx=0x7fae8800b410 comp rx=0 tx=0).stop 2024-07-27T06:46:54.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.972+0000 7fae9e63d640 1 -- 172.21.15.29:0/785158200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae98108680 msgr2=0x7fae9819b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:54.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.972+0000 7fae9e63d640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae98108680 0x7fae9819b6a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fae90005cc0 tx=0x7fae9002f380 comp rx=0 tx=0).stop 2024-07-27T06:46:54.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.972+0000 7fae9e63d640 1 -- 172.21.15.29:0/785158200 shutdown_connections 2024-07-27T06:46:54.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.972+0000 7fae9e63d640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae98108680 0x7fae9819b6a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.972+0000 7fae9e63d640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fae800b3ca0 0x7fae800b61b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.972+0000 7fae9e63d640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae98107d30 0x7fae9819b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.972+0000 7fae9e63d640 1 --2- 172.21.15.29:0/785158200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae98102ad0 0x7fae9819ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:54.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.972+0000 7fae9e63d640 1 -- 172.21.15.29:0/785158200 >> 172.21.15.29:0/785158200 conn(0x7fae980796f0 msgr2=0x7fae98105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:54.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.972+0000 7fae9e63d640 1 -- 172.21.15.29:0/785158200 shutdown_connections 2024-07-27T06:46:54.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:54.972+0000 7fae9e63d640 1 -- 172.21.15.29:0/785158200 wait complete. 2024-07-27T06:46:54.985 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-07-27T06:46:53.682409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665205,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2071,"num_read_kb":1165,"num_write":389,"num_write_kb":2309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2953401,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":821,"ondisk_log_size":821,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360476,"kb_used_data":11280,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581348,"statfs":{"total":1151940427776,"available":1151571300352,"internally_reserved":0,"allocated":11550720,"data_stored":7501220,"data_compressed":247335,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5989,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":132,"num_read_kb":77,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003514"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483739+0000","last_change":"2024-07-27T06:43:12.802353+0000","last_active":"2024-07-27T06:45:09.483739+0000","last_peered":"2024-07-27T06:45:09.483739+0000","last_clean":"2024-07-27T06:45:09.483739+0000","last_became_active":"2024-07-27T06:43:12.802028+0000","last_became_peered":"2024-07-27T06:43:12.802028+0000","last_unstale":"2024-07-27T06:45:09.483739+0000","last_undegraded":"2024-07-27T06:45:09.483739+0000","last_fullsized":"2024-07-27T06:45:09.483739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:22:24.292567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862408+0000","last_change":"2024-07-27T06:43:12.806269+0000","last_active":"2024-07-27T06:45:09.862408+0000","last_peered":"2024-07-27T06:45:09.862408+0000","last_clean":"2024-07-27T06:45:09.862408+0000","last_became_active":"2024-07-27T06:43:12.803232+0000","last_became_peered":"2024-07-27T06:43:12.803232+0000","last_unstale":"2024-07-27T06:45:09.862408+0000","last_undegraded":"2024-07-27T06:45:09.862408+0000","last_fullsized":"2024-07-27T06:45:09.862408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:57:53.673850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862259+0000","last_change":"2024-07-27T06:43:12.805542+0000","last_active":"2024-07-27T06:45:09.862259+0000","last_peered":"2024-07-27T06:45:09.862259+0000","last_clean":"2024-07-27T06:45:09.862259+0000","last_became_active":"2024-07-27T06:43:12.804951+0000","last_became_peered":"2024-07-27T06:43:12.804951+0000","last_unstale":"2024-07-27T06:45:09.862259+0000","last_undegraded":"2024-07-27T06:45:09.862259+0000","last_fullsized":"2024-07-27T06:45:09.862259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:06:40.469695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862158+0000","last_change":"2024-07-27T06:43:12.796950+0000","last_active":"2024-07-27T06:45:09.862158+0000","last_peered":"2024-07-27T06:45:09.862158+0000","last_clean":"2024-07-27T06:45:09.862158+0000","last_became_active":"2024-07-27T06:43:12.796549+0000","last_became_peered":"2024-07-27T06:43:12.796549+0000","last_unstale":"2024-07-27T06:45:09.862158+0000","last_undegraded":"2024-07-27T06:45:09.862158+0000","last_fullsized":"2024-07-27T06:45:09.862158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:30:36.391046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486015+0000","last_change":"2024-07-27T06:43:12.799838+0000","last_active":"2024-07-27T06:45:09.486015+0000","last_peered":"2024-07-27T06:45:09.486015+0000","last_clean":"2024-07-27T06:45:09.486015+0000","last_became_active":"2024-07-27T06:43:12.799462+0000","last_became_peered":"2024-07-27T06:43:12.799462+0000","last_unstale":"2024-07-27T06:45:09.486015+0000","last_undegraded":"2024-07-27T06:45:09.486015+0000","last_fullsized":"2024-07-27T06:45:09.486015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:45:19.371259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481631+0000","last_change":"2024-07-27T06:43:12.797187+0000","last_active":"2024-07-27T06:45:09.481631+0000","last_peered":"2024-07-27T06:45:09.481631+0000","last_clean":"2024-07-27T06:45:09.481631+0000","last_became_active":"2024-07-27T06:43:12.796633+0000","last_became_peered":"2024-07-27T06:43:12.796633+0000","last_unstale":"2024-07-27T06:45:09.481631+0000","last_undegraded":"2024-07-27T06:45:09.481631+0000","last_fullsized":"2024-07-27T06:45:09.481631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:22:20.882899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488423+0000","last_change":"2024-07-27T06:43:12.793786+0000","last_active":"2024-07-27T06:45:09.488423+0000","last_peered":"2024-07-27T06:45:09.488423+0000","last_clean":"2024-07-27T06:45:09.488423+0000","last_became_active":"2024-07-27T06:43:12.793413+0000","last_became_peered":"2024-07-27T06:43:12.793413+0000","last_unstale":"2024-07-27T06:45:09.488423+0000","last_undegraded":"2024-07-27T06:45:09.488423+0000","last_fullsized":"2024-07-27T06:45:09.488423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:14.015373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485385+0000","last_change":"2024-07-27T06:43:12.791304+0000","last_active":"2024-07-27T06:45:09.485385+0000","last_peered":"2024-07-27T06:45:09.485385+0000","last_clean":"2024-07-27T06:45:09.485385+0000","last_became_active":"2024-07-27T06:43:12.790801+0000","last_became_peered":"2024-07-27T06:43:12.790801+0000","last_unstale":"2024-07-27T06:45:09.485385+0000","last_undegraded":"2024-07-27T06:45:09.485385+0000","last_fullsized":"2024-07-27T06:45:09.485385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:12:37.606748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822529+0000","last_change":"2024-07-27T06:43:12.797778+0000","last_active":"2024-07-27T06:45:09.822529+0000","last_peered":"2024-07-27T06:45:09.822529+0000","last_clean":"2024-07-27T06:45:09.822529+0000","last_became_active":"2024-07-27T06:43:12.796784+0000","last_became_peered":"2024-07-27T06:43:12.796784+0000","last_unstale":"2024-07-27T06:45:09.822529+0000","last_undegraded":"2024-07-27T06:45:09.822529+0000","last_fullsized":"2024-07-27T06:45:09.822529+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:52:22.275282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.482109+0000","last_change":"2024-07-27T06:43:12.798970+0000","last_active":"2024-07-27T06:45:09.482109+0000","last_peered":"2024-07-27T06:45:09.482109+0000","last_clean":"2024-07-27T06:45:09.482109+0000","last_became_active":"2024-07-27T06:43:12.798549+0000","last_became_peered":"2024-07-27T06:43:12.798549+0000","last_unstale":"2024-07-27T06:45:09.482109+0000","last_undegraded":"2024-07-27T06:45:09.482109+0000","last_fullsized":"2024-07-27T06:45:09.482109+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:58:32.464429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488604+0000","last_change":"2024-07-27T06:43:12.809405+0000","last_active":"2024-07-27T06:45:09.488604+0000","last_peered":"2024-07-27T06:45:09.488604+0000","last_clean":"2024-07-27T06:45:09.488604+0000","last_became_active":"2024-07-27T06:43:12.809030+0000","last_became_peered":"2024-07-27T06:43:12.809030+0000","last_unstale":"2024-07-27T06:45:09.488604+0000","last_undegraded":"2024-07-27T06:45:09.488604+0000","last_fullsized":"2024-07-27T06:45:09.488604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:46:35.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862193+0000","last_change":"2024-07-27T06:43:12.802647+0000","last_active":"2024-07-27T06:45:09.862193+0000","last_peered":"2024-07-27T06:45:09.862193+0000","last_clean":"2024-07-27T06:45:09.862193+0000","last_became_active":"2024-07-27T06:43:12.802247+0000","last_became_peered":"2024-07-27T06:43:12.802247+0000","last_unstale":"2024-07-27T06:45:09.862193+0000","last_undegraded":"2024-07-27T06:45:09.862193+0000","last_fullsized":"2024-07-27T06:45:09.862193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:44:24.564852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481628+0000","last_change":"2024-07-27T06:43:12.809140+0000","last_active":"2024-07-27T06:45:09.481628+0000","last_peered":"2024-07-27T06:45:09.481628+0000","last_clean":"2024-07-27T06:45:09.481628+0000","last_became_active":"2024-07-27T06:43:12.808692+0000","last_became_peered":"2024-07-27T06:43:12.808692+0000","last_unstale":"2024-07-27T06:45:09.481628+0000","last_undegraded":"2024-07-27T06:45:09.481628+0000","last_fullsized":"2024-07-27T06:45:09.481628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:37:45.326234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862226+0000","last_change":"2024-07-27T06:43:12.802838+0000","last_active":"2024-07-27T06:45:09.862226+0000","last_peered":"2024-07-27T06:45:09.862226+0000","last_clean":"2024-07-27T06:45:09.862226+0000","last_became_active":"2024-07-27T06:43:12.802503+0000","last_became_peered":"2024-07-27T06:43:12.802503+0000","last_unstale":"2024-07-27T06:45:09.862226+0000","last_undegraded":"2024-07-27T06:45:09.862226+0000","last_fullsized":"2024-07-27T06:45:09.862226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:41:03.428162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862170+0000","last_change":"2024-07-27T06:43:12.801626+0000","last_active":"2024-07-27T06:45:09.862170+0000","last_peered":"2024-07-27T06:45:09.862170+0000","last_clean":"2024-07-27T06:45:09.862170+0000","last_became_active":"2024-07-27T06:43:12.801266+0000","last_became_peered":"2024-07-27T06:43:12.801266+0000","last_unstale":"2024-07-27T06:45:09.862170+0000","last_undegraded":"2024-07-27T06:45:09.862170+0000","last_fullsized":"2024-07-27T06:45:09.862170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:54:31.934223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.821975+0000","last_change":"2024-07-27T06:43:12.804543+0000","last_active":"2024-07-27T06:45:09.821975+0000","last_peered":"2024-07-27T06:45:09.821975+0000","last_clean":"2024-07-27T06:45:09.821975+0000","last_became_active":"2024-07-27T06:43:12.804173+0000","last_became_peered":"2024-07-27T06:43:12.804173+0000","last_unstale":"2024-07-27T06:45:09.821975+0000","last_undegraded":"2024-07-27T06:45:09.821975+0000","last_fullsized":"2024-07-27T06:45:09.821975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:18:25.337333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483797+0000","last_change":"2024-07-27T06:43:12.798604+0000","last_active":"2024-07-27T06:45:09.483797+0000","last_peered":"2024-07-27T06:45:09.483797+0000","last_clean":"2024-07-27T06:45:09.483797+0000","last_became_active":"2024-07-27T06:43:12.798228+0000","last_became_peered":"2024-07-27T06:43:12.798228+0000","last_unstale":"2024-07-27T06:45:09.483797+0000","last_undegraded":"2024-07-27T06:45:09.483797+0000","last_fullsized":"2024-07-27T06:45:09.483797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:04:35.851399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633341+0000","last_change":"2024-07-27T06:43:12.800452+0000","last_active":"2024-07-27T06:45:09.633341+0000","last_peered":"2024-07-27T06:45:09.633341+0000","last_clean":"2024-07-27T06:45:09.633341+0000","last_became_active":"2024-07-27T06:43:12.800084+0000","last_became_peered":"2024-07-27T06:43:12.800084+0000","last_unstale":"2024-07-27T06:45:09.633341+0000","last_undegraded":"2024-07-27T06:45:09.633341+0000","last_fullsized":"2024-07-27T06:45:09.633341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:54:27.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485773+0000","last_change":"2024-07-27T06:43:12.800761+0000","last_active":"2024-07-27T06:45:09.485773+0000","last_peered":"2024-07-27T06:45:09.485773+0000","last_clean":"2024-07-27T06:45:09.485773+0000","last_became_active":"2024-07-27T06:43:12.800414+0000","last_became_peered":"2024-07-27T06:43:12.800414+0000","last_unstale":"2024-07-27T06:45:09.485773+0000","last_undegraded":"2024-07-27T06:45:09.485773+0000","last_fullsized":"2024-07-27T06:45:09.485773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:02:47.247522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483375+0000","last_change":"2024-07-27T06:43:12.795470+0000","last_active":"2024-07-27T06:45:09.483375+0000","last_peered":"2024-07-27T06:45:09.483375+0000","last_clean":"2024-07-27T06:45:09.483375+0000","last_became_active":"2024-07-27T06:43:12.795136+0000","last_became_peered":"2024-07-27T06:43:12.795136+0000","last_unstale":"2024-07-27T06:45:09.483375+0000","last_undegraded":"2024-07-27T06:45:09.483375+0000","last_fullsized":"2024-07-27T06:45:09.483375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:01:22.178699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481748+0000","last_change":"2024-07-27T06:43:12.803897+0000","last_active":"2024-07-27T06:45:09.481748+0000","last_peered":"2024-07-27T06:45:09.481748+0000","last_clean":"2024-07-27T06:45:09.481748+0000","last_became_active":"2024-07-27T06:43:12.803546+0000","last_became_peered":"2024-07-27T06:43:12.803546+0000","last_unstale":"2024-07-27T06:45:09.481748+0000","last_undegraded":"2024-07-27T06:45:09.481748+0000","last_fullsized":"2024-07-27T06:45:09.481748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:13:50.816831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822201+0000","last_change":"2024-07-27T06:44:09.238514+0000","last_active":"2024-07-27T06:45:09.822201+0000","last_peered":"2024-07-27T06:45:09.822201+0000","last_clean":"2024-07-27T06:45:09.822201+0000","last_became_active":"2024-07-27T06:44:09.237903+0000","last_became_peered":"2024-07-27T06:44:09.237903+0000","last_unstale":"2024-07-27T06:45:09.822201+0000","last_undegraded":"2024-07-27T06:45:09.822201+0000","last_fullsized":"2024-07-27T06:45:09.822201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:20:59.887491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485428+0000","last_change":"2024-07-27T06:43:12.788570+0000","last_active":"2024-07-27T06:45:09.485428+0000","last_peered":"2024-07-27T06:45:09.485428+0000","last_clean":"2024-07-27T06:45:09.485428+0000","last_became_active":"2024-07-27T06:43:12.788199+0000","last_became_peered":"2024-07-27T06:43:12.788199+0000","last_unstale":"2024-07-27T06:45:09.485428+0000","last_undegraded":"2024-07-27T06:45:09.485428+0000","last_fullsized":"2024-07-27T06:45:09.485428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:44:26.751094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862750+0000","last_change":"2024-07-27T06:43:12.792866+0000","last_active":"2024-07-27T06:45:09.862750+0000","last_peered":"2024-07-27T06:45:09.862750+0000","last_clean":"2024-07-27T06:45:09.862750+0000","last_became_active":"2024-07-27T06:43:12.792389+0000","last_became_peered":"2024-07-27T06:43:12.792389+0000","last_unstale":"2024-07-27T06:45:09.862750+0000","last_undegraded":"2024-07-27T06:45:09.862750+0000","last_fullsized":"2024-07-27T06:45:09.862750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:10:37.367642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485830+0000","last_change":"2024-07-27T06:43:12.797002+0000","last_active":"2024-07-27T06:45:09.485830+0000","last_peered":"2024-07-27T06:45:09.485830+0000","last_clean":"2024-07-27T06:45:09.485830+0000","last_became_active":"2024-07-27T06:43:12.796651+0000","last_became_peered":"2024-07-27T06:43:12.796651+0000","last_unstale":"2024-07-27T06:45:09.485830+0000","last_undegraded":"2024-07-27T06:45:09.485830+0000","last_fullsized":"2024-07-27T06:45:09.485830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:28:06.934042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485867+0000","last_change":"2024-07-27T06:43:12.798549+0000","last_active":"2024-07-27T06:45:09.485867+0000","last_peered":"2024-07-27T06:45:09.485867+0000","last_clean":"2024-07-27T06:45:09.485867+0000","last_became_active":"2024-07-27T06:43:12.798201+0000","last_became_peered":"2024-07-27T06:43:12.798201+0000","last_unstale":"2024-07-27T06:45:09.485867+0000","last_undegraded":"2024-07-27T06:45:09.485867+0000","last_fullsized":"2024-07-27T06:45:09.485867+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:59:47.525536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483656+0000","last_change":"2024-07-27T06:43:12.791053+0000","last_active":"2024-07-27T06:45:09.483656+0000","last_peered":"2024-07-27T06:45:09.483656+0000","last_clean":"2024-07-27T06:45:09.483656+0000","last_became_active":"2024-07-27T06:43:12.790718+0000","last_became_peered":"2024-07-27T06:43:12.790718+0000","last_unstale":"2024-07-27T06:45:09.483656+0000","last_undegraded":"2024-07-27T06:45:09.483656+0000","last_fullsized":"2024-07-27T06:45:09.483656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:09.455399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481811+0000","last_change":"2024-07-27T06:43:12.804180+0000","last_active":"2024-07-27T06:45:09.481811+0000","last_peered":"2024-07-27T06:45:09.481811+0000","last_clean":"2024-07-27T06:45:09.481811+0000","last_became_active":"2024-07-27T06:43:12.803881+0000","last_became_peered":"2024-07-27T06:43:12.803881+0000","last_unstale":"2024-07-27T06:45:09.481811+0000","last_undegraded":"2024-07-27T06:45:09.481811+0000","last_fullsized":"2024-07-27T06:45:09.481811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:01:05.009233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862727+0000","last_change":"2024-07-27T06:43:12.809150+0000","last_active":"2024-07-27T06:45:09.862727+0000","last_peered":"2024-07-27T06:45:09.862727+0000","last_clean":"2024-07-27T06:45:09.862727+0000","last_became_active":"2024-07-27T06:43:12.808428+0000","last_became_peered":"2024-07-27T06:43:12.808428+0000","last_unstale":"2024-07-27T06:45:09.862727+0000","last_undegraded":"2024-07-27T06:45:09.862727+0000","last_fullsized":"2024-07-27T06:45:09.862727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:07:13.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486325+0000","last_change":"2024-07-27T06:44:09.233785+0000","last_active":"2024-07-27T06:45:09.486325+0000","last_peered":"2024-07-27T06:45:09.486325+0000","last_clean":"2024-07-27T06:45:09.486325+0000","last_became_active":"2024-07-27T06:44:09.233184+0000","last_became_peered":"2024-07-27T06:44:09.233184+0000","last_unstale":"2024-07-27T06:45:09.486325+0000","last_undegraded":"2024-07-27T06:45:09.486325+0000","last_fullsized":"2024-07-27T06:45:09.486325+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:12:48.556383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486129+0000","last_change":"2024-07-27T06:44:09.320182+0000","last_active":"2024-07-27T06:45:09.486129+0000","last_peered":"2024-07-27T06:45:09.486129+0000","last_clean":"2024-07-27T06:45:09.486129+0000","last_became_active":"2024-07-27T06:44:09.319661+0000","last_became_peered":"2024-07-27T06:44:09.319661+0000","last_unstale":"2024-07-27T06:45:09.486129+0000","last_undegraded":"2024-07-27T06:45:09.486129+0000","last_fullsized":"2024-07-27T06:45:09.486129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:52:11.006680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483342+0000","last_change":"2024-07-27T06:43:12.787866+0000","last_active":"2024-07-27T06:45:09.483342+0000","last_peered":"2024-07-27T06:45:09.483342+0000","last_clean":"2024-07-27T06:45:09.483342+0000","last_became_active":"2024-07-27T06:43:12.787513+0000","last_became_peered":"2024-07-27T06:43:12.787513+0000","last_unstale":"2024-07-27T06:45:09.483342+0000","last_undegraded":"2024-07-27T06:45:09.483342+0000","last_fullsized":"2024-07-27T06:45:09.483342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:48:57.876588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485381+0000","last_change":"2024-07-27T06:43:12.791158+0000","last_active":"2024-07-27T06:45:09.485381+0000","last_peered":"2024-07-27T06:45:09.485381+0000","last_clean":"2024-07-27T06:45:09.485381+0000","last_became_active":"2024-07-27T06:43:12.790771+0000","last_became_peered":"2024-07-27T06:43:12.790771+0000","last_unstale":"2024-07-27T06:45:09.485381+0000","last_undegraded":"2024-07-27T06:45:09.485381+0000","last_fullsized":"2024-07-27T06:45:09.485381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:10:18.739265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'7","reported_seq":239,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:48.192811+0000","last_change":"2024-07-27T06:43:09.769281+0000","last_active":"2024-07-27T06:46:48.192811+0000","last_peered":"2024-07-27T06:46:48.192811+0000","last_clean":"2024-07-27T06:46:48.192811+0000","last_became_active":"2024-07-27T06:43:09.768864+0000","last_became_peered":"2024-07-27T06:43:09.768864+0000","last_unstale":"2024-07-27T06:46:48.192811+0000","last_undegraded":"2024-07-27T06:46:48.192811+0000","last_fullsized":"2024-07-27T06:46:48.192811+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:55:14.191180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":200,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862960+0000","last_change":"2024-07-27T06:44:09.319930+0000","last_active":"2024-07-27T06:45:09.862960+0000","last_peered":"2024-07-27T06:45:09.862960+0000","last_clean":"2024-07-27T06:45:09.862960+0000","last_became_active":"2024-07-27T06:44:09.319535+0000","last_became_peered":"2024-07-27T06:44:09.319535+0000","last_unstale":"2024-07-27T06:45:09.862960+0000","last_undegraded":"2024-07-27T06:45:09.862960+0000","last_fullsized":"2024-07-27T06:45:09.862960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:39:52.887860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481674+0000","last_change":"2024-07-27T06:43:09.772227+0000","last_active":"2024-07-27T06:45:09.481674+0000","last_peered":"2024-07-27T06:45:09.481674+0000","last_clean":"2024-07-27T06:45:09.481674+0000","last_became_active":"2024-07-27T06:43:09.771812+0000","last_became_peered":"2024-07-27T06:43:09.771812+0000","last_unstale":"2024-07-27T06:45:09.481674+0000","last_undegraded":"2024-07-27T06:45:09.481674+0000","last_fullsized":"2024-07-27T06:45:09.481674+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:27:28.054590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481030+0000","last_change":"2024-07-27T06:43:09.766490+0000","last_active":"2024-07-27T06:45:09.481030+0000","last_peered":"2024-07-27T06:45:09.481030+0000","last_clean":"2024-07-27T06:45:09.481030+0000","last_became_active":"2024-07-27T06:43:09.765973+0000","last_became_peered":"2024-07-27T06:43:09.765973+0000","last_unstale":"2024-07-27T06:45:09.481030+0000","last_undegraded":"2024-07-27T06:45:09.481030+0000","last_fullsized":"2024-07-27T06:45:09.481030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:00:02.913080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485458+0000","last_change":"2024-07-27T06:43:12.805234+0000","last_active":"2024-07-27T06:45:09.485458+0000","last_peered":"2024-07-27T06:45:09.485458+0000","last_clean":"2024-07-27T06:45:09.485458+0000","last_became_active":"2024-07-27T06:43:12.804908+0000","last_became_peered":"2024-07-27T06:43:12.804908+0000","last_unstale":"2024-07-27T06:45:09.485458+0000","last_undegraded":"2024-07-27T06:45:09.485458+0000","last_fullsized":"2024-07-27T06:45:09.485458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:09:34.988755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862248+0000","last_change":"2024-07-27T06:43:09.774201+0000","last_active":"2024-07-27T06:45:09.862248+0000","last_peered":"2024-07-27T06:45:09.862248+0000","last_clean":"2024-07-27T06:45:09.862248+0000","last_became_active":"2024-07-27T06:43:09.773817+0000","last_became_peered":"2024-07-27T06:43:09.773817+0000","last_unstale":"2024-07-27T06:45:09.862248+0000","last_undegraded":"2024-07-27T06:45:09.862248+0000","last_fullsized":"2024-07-27T06:45:09.862248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:06.166251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633110+0000","last_change":"2024-07-27T06:43:12.800174+0000","last_active":"2024-07-27T06:45:09.633110+0000","last_peered":"2024-07-27T06:45:09.633110+0000","last_clean":"2024-07-27T06:45:09.633110+0000","last_became_active":"2024-07-27T06:43:12.799769+0000","last_became_peered":"2024-07-27T06:43:12.799769+0000","last_unstale":"2024-07-27T06:45:09.633110+0000","last_undegraded":"2024-07-27T06:45:09.633110+0000","last_fullsized":"2024-07-27T06:45:09.633110+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:43:06.890372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":435,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:51.560476+0000","last_change":"2024-07-27T06:43:09.776612+0000","last_active":"2024-07-27T06:46:51.560476+0000","last_peered":"2024-07-27T06:46:51.560476+0000","last_clean":"2024-07-27T06:46:51.560476+0000","last_became_active":"2024-07-27T06:43:09.775761+0000","last_became_peered":"2024-07-27T06:43:09.775761+0000","last_unstale":"2024-07-27T06:46:51.560476+0000","last_undegraded":"2024-07-27T06:46:51.560476+0000","last_fullsized":"2024-07-27T06:46:51.560476+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:42:52.132357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":397,"num_read_kb":397,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822478+0000","last_change":"2024-07-27T06:43:12.797200+0000","last_active":"2024-07-27T06:45:09.822478+0000","last_peered":"2024-07-27T06:45:09.822478+0000","last_clean":"2024-07-27T06:45:09.822478+0000","last_became_active":"2024-07-27T06:43:12.796288+0000","last_became_peered":"2024-07-27T06:43:12.796288+0000","last_unstale":"2024-07-27T06:45:09.822478+0000","last_undegraded":"2024-07-27T06:45:09.822478+0000","last_fullsized":"2024-07-27T06:45:09.822478+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:31:27.603883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488738+0000","last_change":"2024-07-27T06:43:09.772658+0000","last_active":"2024-07-27T06:45:09.488738+0000","last_peered":"2024-07-27T06:45:09.488738+0000","last_clean":"2024-07-27T06:45:09.488738+0000","last_became_active":"2024-07-27T06:43:09.772277+0000","last_became_peered":"2024-07-27T06:43:09.772277+0000","last_unstale":"2024-07-27T06:45:09.488738+0000","last_undegraded":"2024-07-27T06:45:09.488738+0000","last_fullsized":"2024-07-27T06:45:09.488738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:09:16.854718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862551+0000","last_change":"2024-07-27T06:43:12.809643+0000","last_active":"2024-07-27T06:45:09.862551+0000","last_peered":"2024-07-27T06:45:09.862551+0000","last_clean":"2024-07-27T06:45:09.862551+0000","last_became_active":"2024-07-27T06:43:12.809020+0000","last_became_peered":"2024-07-27T06:43:12.809020+0000","last_unstale":"2024-07-27T06:45:09.862551+0000","last_undegraded":"2024-07-27T06:45:09.862551+0000","last_fullsized":"2024-07-27T06:45:09.862551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:08:03.062914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.863022+0000","last_change":"2024-07-27T06:43:10.534042+0000","last_active":"2024-07-27T06:45:09.863022+0000","last_peered":"2024-07-27T06:45:09.863022+0000","last_clean":"2024-07-27T06:45:09.863022+0000","last_became_active":"2024-07-27T06:43:10.533615+0000","last_became_peered":"2024-07-27T06:43:10.533615+0000","last_unstale":"2024-07-27T06:45:09.863022+0000","last_undegraded":"2024-07-27T06:45:09.863022+0000","last_fullsized":"2024-07-27T06:45:09.863022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:09:32.235633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822222+0000","last_change":"2024-07-27T06:43:12.798764+0000","last_active":"2024-07-27T06:45:09.822222+0000","last_peered":"2024-07-27T06:45:09.822222+0000","last_clean":"2024-07-27T06:45:09.822222+0000","last_became_active":"2024-07-27T06:43:12.797500+0000","last_became_peered":"2024-07-27T06:43:12.797500+0000","last_unstale":"2024-07-27T06:45:09.822222+0000","last_undegraded":"2024-07-27T06:45:09.822222+0000","last_fullsized":"2024-07-27T06:45:09.822222+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:12:09.368711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"83'37","reported_seq":479,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:52.199549+0000","last_change":"2024-07-27T06:43:09.775954+0000","last_active":"2024-07-27T06:46:52.199549+0000","last_peered":"2024-07-27T06:46:52.199549+0000","last_clean":"2024-07-27T06:46:52.199549+0000","last_became_active":"2024-07-27T06:43:09.775498+0000","last_became_peered":"2024-07-27T06:43:09.775498+0000","last_unstale":"2024-07-27T06:46:52.199549+0000","last_undegraded":"2024-07-27T06:46:52.199549+0000","last_fullsized":"2024-07-27T06:46:52.199549+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:08:46.965525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66277,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":410,"num_read_kb":270,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486559+0000","last_change":"2024-07-27T06:43:12.797220+0000","last_active":"2024-07-27T06:45:09.486559+0000","last_peered":"2024-07-27T06:45:09.486559+0000","last_clean":"2024-07-27T06:45:09.486559+0000","last_became_active":"2024-07-27T06:43:12.796889+0000","last_became_peered":"2024-07-27T06:43:12.796889+0000","last_unstale":"2024-07-27T06:45:09.486559+0000","last_undegraded":"2024-07-27T06:45:09.486559+0000","last_fullsized":"2024-07-27T06:45:09.486559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:24:36.275738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485662+0000","last_change":"2024-07-27T06:43:09.776600+0000","last_active":"2024-07-27T06:45:09.485662+0000","last_peered":"2024-07-27T06:45:09.485662+0000","last_clean":"2024-07-27T06:45:09.485662+0000","last_became_active":"2024-07-27T06:43:09.776163+0000","last_became_peered":"2024-07-27T06:43:09.776163+0000","last_unstale":"2024-07-27T06:45:09.485662+0000","last_undegraded":"2024-07-27T06:45:09.485662+0000","last_fullsized":"2024-07-27T06:45:09.485662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:22:15.947916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481215+0000","last_change":"2024-07-27T06:43:12.804984+0000","last_active":"2024-07-27T06:45:09.481215+0000","last_peered":"2024-07-27T06:45:09.481215+0000","last_clean":"2024-07-27T06:45:09.481215+0000","last_became_active":"2024-07-27T06:43:12.804644+0000","last_became_peered":"2024-07-27T06:43:12.804644+0000","last_unstale":"2024-07-27T06:45:09.481215+0000","last_undegraded":"2024-07-27T06:45:09.481215+0000","last_fullsized":"2024-07-27T06:45:09.481215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.355981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862246+0000","last_change":"2024-07-27T06:43:09.772489+0000","last_active":"2024-07-27T06:45:09.862246+0000","last_peered":"2024-07-27T06:45:09.862246+0000","last_clean":"2024-07-27T06:45:09.862246+0000","last_became_active":"2024-07-27T06:43:09.772069+0000","last_became_peered":"2024-07-27T06:43:09.772069+0000","last_unstale":"2024-07-27T06:45:09.862246+0000","last_undegraded":"2024-07-27T06:45:09.862246+0000","last_fullsized":"2024-07-27T06:45:09.862246+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:15:30.549971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488589+0000","last_change":"2024-07-27T06:43:12.803376+0000","last_active":"2024-07-27T06:45:09.488589+0000","last_peered":"2024-07-27T06:45:09.488589+0000","last_clean":"2024-07-27T06:45:09.488589+0000","last_became_active":"2024-07-27T06:43:12.802961+0000","last_became_peered":"2024-07-27T06:43:12.802961+0000","last_unstale":"2024-07-27T06:45:09.488589+0000","last_undegraded":"2024-07-27T06:45:09.488589+0000","last_fullsized":"2024-07-27T06:45:09.488589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:00:24.649877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":236,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:49.193981+0000","last_change":"2024-07-27T06:43:09.773182+0000","last_active":"2024-07-27T06:46:49.193981+0000","last_peered":"2024-07-27T06:46:49.193981+0000","last_clean":"2024-07-27T06:46:49.193981+0000","last_became_active":"2024-07-27T06:43:09.772126+0000","last_became_peered":"2024-07-27T06:43:09.772126+0000","last_unstale":"2024-07-27T06:46:49.193981+0000","last_undegraded":"2024-07-27T06:46:49.193981+0000","last_fullsized":"2024-07-27T06:46:49.193981+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:45:26.598799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":200,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488424+0000","last_change":"2024-07-27T06:43:12.799385+0000","last_active":"2024-07-27T06:45:09.488424+0000","last_peered":"2024-07-27T06:45:09.488424+0000","last_clean":"2024-07-27T06:45:09.488424+0000","last_became_active":"2024-07-27T06:43:12.798639+0000","last_became_peered":"2024-07-27T06:43:12.798639+0000","last_unstale":"2024-07-27T06:45:09.488424+0000","last_undegraded":"2024-07-27T06:45:09.488424+0000","last_fullsized":"2024-07-27T06:45:09.488424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:56:29.462491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485303+0000","last_change":"2024-07-27T06:43:10.535429+0000","last_active":"2024-07-27T06:45:09.485303+0000","last_peered":"2024-07-27T06:45:09.485303+0000","last_clean":"2024-07-27T06:45:09.485303+0000","last_became_active":"2024-07-27T06:43:10.534952+0000","last_became_peered":"2024-07-27T06:43:10.534952+0000","last_unstale":"2024-07-27T06:45:09.485303+0000","last_undegraded":"2024-07-27T06:45:09.485303+0000","last_fullsized":"2024-07-27T06:45:09.485303+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:54:20.018263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862706+0000","last_change":"2024-07-27T06:43:12.804266+0000","last_active":"2024-07-27T06:45:09.862706+0000","last_peered":"2024-07-27T06:45:09.862706+0000","last_clean":"2024-07-27T06:45:09.862706+0000","last_became_active":"2024-07-27T06:43:12.803888+0000","last_became_peered":"2024-07-27T06:43:12.803888+0000","last_unstale":"2024-07-27T06:45:09.862706+0000","last_undegraded":"2024-07-27T06:45:09.862706+0000","last_fullsized":"2024-07-27T06:45:09.862706+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:48:00.490592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485827+0000","last_change":"2024-07-27T06:43:09.777656+0000","last_active":"2024-07-27T06:45:09.485827+0000","last_peered":"2024-07-27T06:45:09.485827+0000","last_clean":"2024-07-27T06:45:09.485827+0000","last_became_active":"2024-07-27T06:43:09.776620+0000","last_became_peered":"2024-07-27T06:43:09.776620+0000","last_unstale":"2024-07-27T06:45:09.485827+0000","last_undegraded":"2024-07-27T06:45:09.485827+0000","last_fullsized":"2024-07-27T06:45:09.485827+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:50.203623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483544+0000","last_change":"2024-07-27T06:43:12.796179+0000","last_active":"2024-07-27T06:45:09.483544+0000","last_peered":"2024-07-27T06:45:09.483544+0000","last_clean":"2024-07-27T06:45:09.483544+0000","last_became_active":"2024-07-27T06:43:12.795701+0000","last_became_peered":"2024-07-27T06:43:12.795701+0000","last_unstale":"2024-07-27T06:45:09.483544+0000","last_undegraded":"2024-07-27T06:45:09.483544+0000","last_fullsized":"2024-07-27T06:45:09.483544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:20:12.236769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481627+0000","last_change":"2024-07-27T06:43:09.769949+0000","last_active":"2024-07-27T06:45:09.481627+0000","last_peered":"2024-07-27T06:45:09.481627+0000","last_clean":"2024-07-27T06:45:09.481627+0000","last_became_active":"2024-07-27T06:43:09.769602+0000","last_became_peered":"2024-07-27T06:43:09.769602+0000","last_unstale":"2024-07-27T06:45:09.481627+0000","last_undegraded":"2024-07-27T06:45:09.481627+0000","last_fullsized":"2024-07-27T06:45:09.481627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:28:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488869+0000","last_change":"2024-07-27T06:43:12.802230+0000","last_active":"2024-07-27T06:45:09.488869+0000","last_peered":"2024-07-27T06:45:09.488869+0000","last_clean":"2024-07-27T06:45:09.488869+0000","last_became_active":"2024-07-27T06:43:12.800944+0000","last_became_peered":"2024-07-27T06:43:12.800944+0000","last_unstale":"2024-07-27T06:45:09.488869+0000","last_undegraded":"2024-07-27T06:45:09.488869+0000","last_fullsized":"2024-07-27T06:45:09.488869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:05:54.585955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862270+0000","last_change":"2024-07-27T06:43:09.768239+0000","last_active":"2024-07-27T06:45:09.862270+0000","last_peered":"2024-07-27T06:45:09.862270+0000","last_clean":"2024-07-27T06:45:09.862270+0000","last_became_active":"2024-07-27T06:43:09.767584+0000","last_became_peered":"2024-07-27T06:43:09.767584+0000","last_unstale":"2024-07-27T06:45:09.862270+0000","last_undegraded":"2024-07-27T06:45:09.862270+0000","last_fullsized":"2024-07-27T06:45:09.862270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:35:39.552246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488571+0000","last_change":"2024-07-27T06:43:12.803779+0000","last_active":"2024-07-27T06:45:09.488571+0000","last_peered":"2024-07-27T06:45:09.488571+0000","last_clean":"2024-07-27T06:45:09.488571+0000","last_became_active":"2024-07-27T06:43:12.803359+0000","last_became_peered":"2024-07-27T06:43:12.803359+0000","last_unstale":"2024-07-27T06:45:09.488571+0000","last_undegraded":"2024-07-27T06:45:09.488571+0000","last_fullsized":"2024-07-27T06:45:09.488571+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:32:53.167039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'7","reported_seq":189,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:49.557688+0000","last_change":"2024-07-27T06:44:09.241782+0000","last_active":"2024-07-27T06:46:49.557688+0000","last_peered":"2024-07-27T06:46:49.557688+0000","last_clean":"2024-07-27T06:46:49.557688+0000","last_became_active":"2024-07-27T06:44:09.234692+0000","last_became_peered":"2024-07-27T06:44:09.234692+0000","last_unstale":"2024-07-27T06:46:49.557688+0000","last_undegraded":"2024-07-27T06:46:49.557688+0000","last_fullsized":"2024-07-27T06:46:49.557688+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:45:02.155686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":166,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1561,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488800+0000","last_change":"2024-07-27T06:43:12.805830+0000","last_active":"2024-07-27T06:45:09.488800+0000","last_peered":"2024-07-27T06:45:09.488800+0000","last_clean":"2024-07-27T06:45:09.488800+0000","last_became_active":"2024-07-27T06:43:12.805324+0000","last_became_peered":"2024-07-27T06:43:12.805324+0000","last_unstale":"2024-07-27T06:45:09.488800+0000","last_undegraded":"2024-07-27T06:45:09.488800+0000","last_fullsized":"2024-07-27T06:45:09.488800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:38:31.430676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":109,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.489084+0000","last_change":"2024-07-27T06:41:43.549132+0000","last_active":"2024-07-27T06:45:09.489084+0000","last_peered":"2024-07-27T06:45:09.489084+0000","last_clean":"2024-07-27T06:45:09.489084+0000","last_became_active":"2024-07-27T06:41:43.542635+0000","last_became_peered":"2024-07-27T06:41:43.542635+0000","last_unstale":"2024-07-27T06:45:09.489084+0000","last_undegraded":"2024-07-27T06:45:09.489084+0000","last_fullsized":"2024-07-27T06:45:09.489084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_clean_scrub_stamp":"2024-07-27T06:38:21.639907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:10:06.417387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.861700+0000","last_change":"2024-07-27T06:43:09.768774+0000","last_active":"2024-07-27T06:45:09.861700+0000","last_peered":"2024-07-27T06:45:09.861700+0000","last_clean":"2024-07-27T06:45:09.861700+0000","last_became_active":"2024-07-27T06:43:09.768345+0000","last_became_peered":"2024-07-27T06:43:09.768345+0000","last_unstale":"2024-07-27T06:45:09.861700+0000","last_undegraded":"2024-07-27T06:45:09.861700+0000","last_fullsized":"2024-07-27T06:45:09.861700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:40:49.411746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485670+0000","last_change":"2024-07-27T06:43:12.797817+0000","last_active":"2024-07-27T06:45:09.485670+0000","last_peered":"2024-07-27T06:45:09.485670+0000","last_clean":"2024-07-27T06:45:09.485670+0000","last_became_active":"2024-07-27T06:43:12.797483+0000","last_became_peered":"2024-07-27T06:43:12.797483+0000","last_unstale":"2024-07-27T06:45:09.485670+0000","last_undegraded":"2024-07-27T06:45:09.485670+0000","last_fullsized":"2024-07-27T06:45:09.485670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:24:44.539505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488887+0000","last_change":"2024-07-27T06:43:09.776880+0000","last_active":"2024-07-27T06:45:09.488887+0000","last_peered":"2024-07-27T06:45:09.488887+0000","last_clean":"2024-07-27T06:45:09.488887+0000","last_became_active":"2024-07-27T06:43:09.776486+0000","last_became_peered":"2024-07-27T06:43:09.776486+0000","last_unstale":"2024-07-27T06:45:09.488887+0000","last_undegraded":"2024-07-27T06:45:09.488887+0000","last_fullsized":"2024-07-27T06:45:09.488887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:55:25.436405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.480961+0000","last_change":"2024-07-27T06:43:12.799755+0000","last_active":"2024-07-27T06:45:09.480961+0000","last_peered":"2024-07-27T06:45:09.480961+0000","last_clean":"2024-07-27T06:45:09.480961+0000","last_became_active":"2024-07-27T06:43:12.799412+0000","last_became_peered":"2024-07-27T06:43:12.799412+0000","last_unstale":"2024-07-27T06:45:09.480961+0000","last_undegraded":"2024-07-27T06:45:09.480961+0000","last_fullsized":"2024-07-27T06:45:09.480961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:45:50.545431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":235,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:49.193450+0000","last_change":"2024-07-27T06:43:10.533039+0000","last_active":"2024-07-27T06:46:49.193450+0000","last_peered":"2024-07-27T06:46:49.193450+0000","last_clean":"2024-07-27T06:46:49.193450+0000","last_became_active":"2024-07-27T06:43:10.532583+0000","last_became_peered":"2024-07-27T06:43:10.532583+0000","last_unstale":"2024-07-27T06:46:49.193450+0000","last_undegraded":"2024-07-27T06:46:49.193450+0000","last_fullsized":"2024-07-27T06:46:49.193450+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:04:51.954463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":198,"num_read_kb":198,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633096+0000","last_change":"2024-07-27T06:44:09.238218+0000","last_active":"2024-07-27T06:45:09.633096+0000","last_peered":"2024-07-27T06:45:09.633096+0000","last_clean":"2024-07-27T06:45:09.633096+0000","last_became_active":"2024-07-27T06:44:09.237841+0000","last_became_peered":"2024-07-27T06:44:09.237841+0000","last_unstale":"2024-07-27T06:45:09.633096+0000","last_undegraded":"2024-07-27T06:45:09.633096+0000","last_fullsized":"2024-07-27T06:45:09.633096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:19.101854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"83'8","reported_seq":242,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:50.237432+0000","last_change":"2024-07-27T06:43:09.776214+0000","last_active":"2024-07-27T06:46:50.237432+0000","last_peered":"2024-07-27T06:46:50.237432+0000","last_clean":"2024-07-27T06:46:50.237432+0000","last_became_active":"2024-07-27T06:43:09.775719+0000","last_became_peered":"2024-07-27T06:43:09.775719+0000","last_unstale":"2024-07-27T06:46:50.237432+0000","last_undegraded":"2024-07-27T06:46:50.237432+0000","last_fullsized":"2024-07-27T06:46:50.237432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:57:20.381599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":202,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481735+0000","last_change":"2024-07-27T06:43:12.794909+0000","last_active":"2024-07-27T06:45:09.481735+0000","last_peered":"2024-07-27T06:45:09.481735+0000","last_clean":"2024-07-27T06:45:09.481735+0000","last_became_active":"2024-07-27T06:43:12.794497+0000","last_became_peered":"2024-07-27T06:43:12.794497+0000","last_unstale":"2024-07-27T06:45:09.481735+0000","last_undegraded":"2024-07-27T06:45:09.481735+0000","last_fullsized":"2024-07-27T06:45:09.481735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:42:49.760126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633255+0000","last_change":"2024-07-27T06:43:10.534240+0000","last_active":"2024-07-27T06:45:09.633255+0000","last_peered":"2024-07-27T06:45:09.633255+0000","last_clean":"2024-07-27T06:45:09.633255+0000","last_became_active":"2024-07-27T06:43:10.533596+0000","last_became_peered":"2024-07-27T06:43:10.533596+0000","last_unstale":"2024-07-27T06:45:09.633255+0000","last_undegraded":"2024-07-27T06:45:09.633255+0000","last_fullsized":"2024-07-27T06:45:09.633255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:16.098839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481991+0000","last_change":"2024-07-27T06:43:12.805248+0000","last_active":"2024-07-27T06:45:09.481991+0000","last_peered":"2024-07-27T06:45:09.481991+0000","last_clean":"2024-07-27T06:45:09.481991+0000","last_became_active":"2024-07-27T06:43:12.803693+0000","last_became_peered":"2024-07-27T06:43:12.803693+0000","last_unstale":"2024-07-27T06:45:09.481991+0000","last_undegraded":"2024-07-27T06:45:09.481991+0000","last_fullsized":"2024-07-27T06:45:09.481991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:17.466465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486692+0000","last_change":"2024-07-27T06:43:10.533701+0000","last_active":"2024-07-27T06:45:09.486692+0000","last_peered":"2024-07-27T06:45:09.486692+0000","last_clean":"2024-07-27T06:45:09.486692+0000","last_became_active":"2024-07-27T06:43:10.533109+0000","last_became_peered":"2024-07-27T06:43:10.533109+0000","last_unstale":"2024-07-27T06:45:09.486692+0000","last_undegraded":"2024-07-27T06:45:09.486692+0000","last_fullsized":"2024-07-27T06:45:09.486692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:32:52.906691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481918+0000","last_change":"2024-07-27T06:43:12.798479+0000","last_active":"2024-07-27T06:45:09.481918+0000","last_peered":"2024-07-27T06:45:09.481918+0000","last_clean":"2024-07-27T06:45:09.481918+0000","last_became_active":"2024-07-27T06:43:12.798124+0000","last_became_peered":"2024-07-27T06:43:12.798124+0000","last_unstale":"2024-07-27T06:45:09.481918+0000","last_undegraded":"2024-07-27T06:45:09.481918+0000","last_fullsized":"2024-07-27T06:45:09.481918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:14:55.267213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"83'17","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:22.215894+0000","last_change":"2024-07-27T06:43:09.770122+0000","last_active":"2024-07-27T06:45:22.215894+0000","last_peered":"2024-07-27T06:45:22.215894+0000","last_clean":"2024-07-27T06:45:22.215894+0000","last_became_active":"2024-07-27T06:43:09.769784+0000","last_became_peered":"2024-07-27T06:43:09.769784+0000","last_unstale":"2024-07-27T06:45:22.215894+0000","last_undegraded":"2024-07-27T06:45:22.215894+0000","last_fullsized":"2024-07-27T06:45:22.215894+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:47:00.398135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485891+0000","last_change":"2024-07-27T06:43:12.799603+0000","last_active":"2024-07-27T06:45:09.485891+0000","last_peered":"2024-07-27T06:45:09.485891+0000","last_clean":"2024-07-27T06:45:09.485891+0000","last_became_active":"2024-07-27T06:43:12.799189+0000","last_became_peered":"2024-07-27T06:43:12.799189+0000","last_unstale":"2024-07-27T06:45:09.485891+0000","last_undegraded":"2024-07-27T06:45:09.485891+0000","last_fullsized":"2024-07-27T06:45:09.485891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:32:30.603258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862220+0000","last_change":"2024-07-27T06:43:10.004726+0000","last_active":"2024-07-27T06:45:09.862220+0000","last_peered":"2024-07-27T06:45:09.862220+0000","last_clean":"2024-07-27T06:45:09.862220+0000","last_became_active":"2024-07-27T06:43:10.004367+0000","last_became_peered":"2024-07-27T06:43:10.004367+0000","last_unstale":"2024-07-27T06:45:09.862220+0000","last_undegraded":"2024-07-27T06:45:09.862220+0000","last_fullsized":"2024-07-27T06:45:09.862220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:22:39.942391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862427+0000","last_change":"2024-07-27T06:43:12.796511+0000","last_active":"2024-07-27T06:45:09.862427+0000","last_peered":"2024-07-27T06:45:09.862427+0000","last_clean":"2024-07-27T06:45:09.862427+0000","last_became_active":"2024-07-27T06:43:12.796152+0000","last_became_peered":"2024-07-27T06:43:12.796152+0000","last_unstale":"2024-07-27T06:45:09.862427+0000","last_undegraded":"2024-07-27T06:45:09.862427+0000","last_fullsized":"2024-07-27T06:45:09.862427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:03:02.508990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822077+0000","last_change":"2024-07-27T06:43:10.533656+0000","last_active":"2024-07-27T06:45:09.822077+0000","last_peered":"2024-07-27T06:45:09.822077+0000","last_clean":"2024-07-27T06:45:09.822077+0000","last_became_active":"2024-07-27T06:43:10.533295+0000","last_became_peered":"2024-07-27T06:43:10.533295+0000","last_unstale":"2024-07-27T06:45:09.822077+0000","last_undegraded":"2024-07-27T06:45:09.822077+0000","last_fullsized":"2024-07-27T06:45:09.822077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:11:00.032511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485237+0000","last_change":"2024-07-27T06:43:12.795106+0000","last_active":"2024-07-27T06:45:09.485237+0000","last_peered":"2024-07-27T06:45:09.485237+0000","last_clean":"2024-07-27T06:45:09.485237+0000","last_became_active":"2024-07-27T06:43:12.794729+0000","last_became_peered":"2024-07-27T06:43:12.794729+0000","last_unstale":"2024-07-27T06:45:09.485237+0000","last_undegraded":"2024-07-27T06:45:09.485237+0000","last_fullsized":"2024-07-27T06:45:09.485237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:19:16.542363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485673+0000","last_change":"2024-07-27T06:43:09.770016+0000","last_active":"2024-07-27T06:45:09.485673+0000","last_peered":"2024-07-27T06:45:09.485673+0000","last_clean":"2024-07-27T06:45:09.485673+0000","last_became_active":"2024-07-27T06:43:09.769672+0000","last_became_peered":"2024-07-27T06:43:09.769672+0000","last_unstale":"2024-07-27T06:45:09.485673+0000","last_undegraded":"2024-07-27T06:45:09.485673+0000","last_fullsized":"2024-07-27T06:45:09.485673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:30:55.737494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483659+0000","last_change":"2024-07-27T06:43:12.793690+0000","last_active":"2024-07-27T06:45:09.483659+0000","last_peered":"2024-07-27T06:45:09.483659+0000","last_clean":"2024-07-27T06:45:09.483659+0000","last_became_active":"2024-07-27T06:43:12.793352+0000","last_became_peered":"2024-07-27T06:43:12.793352+0000","last_unstale":"2024-07-27T06:45:09.483659+0000","last_undegraded":"2024-07-27T06:45:09.483659+0000","last_fullsized":"2024-07-27T06:45:09.483659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:22:47.606990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.491196+0000","last_change":"2024-07-27T06:45:09.490738+0000","last_active":"2024-07-27T06:45:09.491196+0000","last_peered":"2024-07-27T06:45:09.491196+0000","last_clean":"2024-07-27T06:45:09.491196+0000","last_became_active":"2024-07-27T06:45:09.487147+0000","last_became_peered":"2024-07-27T06:45:09.487147+0000","last_unstale":"2024-07-27T06:45:09.491196+0000","last_undegraded":"2024-07-27T06:45:09.491196+0000","last_fullsized":"2024-07-27T06:45:09.491196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:46.732987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485570+0000","last_change":"2024-07-27T06:43:12.792648+0000","last_active":"2024-07-27T06:45:09.485570+0000","last_peered":"2024-07-27T06:45:09.485570+0000","last_clean":"2024-07-27T06:45:09.485570+0000","last_became_active":"2024-07-27T06:43:12.792338+0000","last_became_peered":"2024-07-27T06:43:12.792338+0000","last_unstale":"2024-07-27T06:45:09.485570+0000","last_undegraded":"2024-07-27T06:45:09.485570+0000","last_fullsized":"2024-07-27T06:45:09.485570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:36:52.605957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.827632+0000","last_change":"2024-07-27T06:45:09.827140+0000","last_active":"2024-07-27T06:45:09.827632+0000","last_peered":"2024-07-27T06:45:09.827632+0000","last_clean":"2024-07-27T06:45:09.827632+0000","last_became_active":"2024-07-27T06:45:09.823119+0000","last_became_peered":"2024-07-27T06:45:09.823119+0000","last_unstale":"2024-07-27T06:45:09.827632+0000","last_undegraded":"2024-07-27T06:45:09.827632+0000","last_fullsized":"2024-07-27T06:45:09.827632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:32:19.682193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488730+0000","last_change":"2024-07-27T06:43:12.809347+0000","last_active":"2024-07-27T06:45:09.488730+0000","last_peered":"2024-07-27T06:45:09.488730+0000","last_clean":"2024-07-27T06:45:09.488730+0000","last_became_active":"2024-07-27T06:43:12.808980+0000","last_became_peered":"2024-07-27T06:43:12.808980+0000","last_unstale":"2024-07-27T06:45:09.488730+0000","last_undegraded":"2024-07-27T06:45:09.488730+0000","last_fullsized":"2024-07-27T06:45:09.488730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:05:14.132641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633284+0000","last_change":"2024-07-27T06:44:09.237908+0000","last_active":"2024-07-27T06:45:09.633284+0000","last_peered":"2024-07-27T06:45:09.633284+0000","last_clean":"2024-07-27T06:45:09.633284+0000","last_became_active":"2024-07-27T06:44:09.237503+0000","last_became_peered":"2024-07-27T06:44:09.237503+0000","last_unstale":"2024-07-27T06:45:09.633284+0000","last_undegraded":"2024-07-27T06:45:09.633284+0000","last_fullsized":"2024-07-27T06:45:09.633284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:29.654244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486391+0000","last_change":"2024-07-27T06:43:09.776000+0000","last_active":"2024-07-27T06:45:09.486391+0000","last_peered":"2024-07-27T06:45:09.486391+0000","last_clean":"2024-07-27T06:45:09.486391+0000","last_became_active":"2024-07-27T06:43:09.775522+0000","last_became_peered":"2024-07-27T06:43:09.775522+0000","last_unstale":"2024-07-27T06:45:09.486391+0000","last_undegraded":"2024-07-27T06:45:09.486391+0000","last_fullsized":"2024-07-27T06:45:09.486391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:39:42.650177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822084+0000","last_change":"2024-07-27T06:43:12.807062+0000","last_active":"2024-07-27T06:45:09.822084+0000","last_peered":"2024-07-27T06:45:09.822084+0000","last_clean":"2024-07-27T06:45:09.822084+0000","last_became_active":"2024-07-27T06:43:12.805816+0000","last_became_peered":"2024-07-27T06:43:12.805816+0000","last_unstale":"2024-07-27T06:45:09.822084+0000","last_undegraded":"2024-07-27T06:45:09.822084+0000","last_fullsized":"2024-07-27T06:45:09.822084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:44:16.261663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483465+0000","last_change":"2024-07-27T06:44:09.250070+0000","last_active":"2024-07-27T06:45:09.483465+0000","last_peered":"2024-07-27T06:45:09.483465+0000","last_clean":"2024-07-27T06:45:09.483465+0000","last_became_active":"2024-07-27T06:44:09.237290+0000","last_became_peered":"2024-07-27T06:44:09.237290+0000","last_unstale":"2024-07-27T06:45:09.483465+0000","last_undegraded":"2024-07-27T06:45:09.483465+0000","last_fullsized":"2024-07-27T06:45:09.483465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:51.590487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862423+0000","last_change":"2024-07-27T06:43:12.809135+0000","last_active":"2024-07-27T06:45:09.862423+0000","last_peered":"2024-07-27T06:45:09.862423+0000","last_clean":"2024-07-27T06:45:09.862423+0000","last_became_active":"2024-07-27T06:43:12.806731+0000","last_became_peered":"2024-07-27T06:43:12.806731+0000","last_unstale":"2024-07-27T06:45:09.862423+0000","last_undegraded":"2024-07-27T06:45:09.862423+0000","last_fullsized":"2024-07-27T06:45:09.862423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:47:50.397517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486050+0000","last_change":"2024-07-27T06:43:10.533608+0000","last_active":"2024-07-27T06:45:09.486050+0000","last_peered":"2024-07-27T06:45:09.486050+0000","last_clean":"2024-07-27T06:45:09.486050+0000","last_became_active":"2024-07-27T06:43:10.532663+0000","last_became_peered":"2024-07-27T06:43:10.532663+0000","last_unstale":"2024-07-27T06:45:09.486050+0000","last_undegraded":"2024-07-27T06:45:09.486050+0000","last_fullsized":"2024-07-27T06:45:09.486050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:05:33.155427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485883+0000","last_change":"2024-07-27T06:43:12.789890+0000","last_active":"2024-07-27T06:45:09.485883+0000","last_peered":"2024-07-27T06:45:09.485883+0000","last_clean":"2024-07-27T06:45:09.485883+0000","last_became_active":"2024-07-27T06:43:12.789561+0000","last_became_peered":"2024-07-27T06:43:12.789561+0000","last_unstale":"2024-07-27T06:45:09.485883+0000","last_undegraded":"2024-07-27T06:45:09.485883+0000","last_fullsized":"2024-07-27T06:45:09.485883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:35:49.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"83'15","reported_seq":246,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:50.236769+0000","last_change":"2024-07-27T06:43:09.773471+0000","last_active":"2024-07-27T06:46:50.236769+0000","last_peered":"2024-07-27T06:46:50.236769+0000","last_clean":"2024-07-27T06:46:50.236769+0000","last_became_active":"2024-07-27T06:43:09.773213+0000","last_became_peered":"2024-07-27T06:43:09.773213+0000","last_unstale":"2024-07-27T06:46:50.236769+0000","last_undegraded":"2024-07-27T06:46:50.236769+0000","last_fullsized":"2024-07-27T06:46:50.236769+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:54:38.770198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":199,"num_read_kb":199,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74603,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1990,"num_read_kb":1096,"num_write":264,"num_write_kb":157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1561,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":213063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283116,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678076,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":461919,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874304,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1113320,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037831,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":461317,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234059,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":461695,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430287,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":1055194,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593813,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":461874,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790042,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":1054032,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953567,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":929792,"data_stored":462597,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":460859,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378730,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":522433,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542254,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":983040,"data_stored":523075,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705779,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":462905,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":68013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T06:46:54.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-27T06:46:55.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.140+0000 7f8dc7b20640 1 -- 172.21.15.29:0/4117021770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dc0107e00 msgr2=0x7f8dc010e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:55.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.140+0000 7f8dc7b20640 1 --2- 172.21.15.29:0/4117021770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dc0107e00 0x7f8dc010e630 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009ca0 tx=0x7f8db802a8c0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.140+0000 7f8dc7b20640 1 -- 172.21.15.29:0/4117021770 shutdown_connections 2024-07-27T06:46:55.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.140+0000 7f8dc7b20640 1 --2- 172.21.15.29:0/4117021770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dc0107e00 0x7f8dc010e630 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.140+0000 7f8dc7b20640 1 --2- 172.21.15.29:0/4117021770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc01074b0 0x7f8dc01078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.140+0000 7f8dc7b20640 1 --2- 172.21.15.29:0/4117021770 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dc0101b80 0x7f8dc0101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.140+0000 7f8dc7b20640 1 -- 172.21.15.29:0/4117021770 >> 172.21.15.29:0/4117021770 conn(0x7f8dc0079750 msgr2=0x7f8dc0079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:55.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.140+0000 7f8dc7b20640 1 -- 172.21.15.29:0/4117021770 shutdown_connections 2024-07-27T06:46:55.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.140+0000 7f8dc7b20640 1 -- 172.21.15.29:0/4117021770 wait complete. 2024-07-27T06:46:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc7b20640 1 Processor -- start 2024-07-27T06:46:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc7b20640 1 -- start start 2024-07-27T06:46:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc7b20640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc0101b80 0x7f8dc0199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc7b20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dc01074b0 0x7f8dc019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc7b20640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dc0107e00 0x7f8dc019f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc7b20640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc0112340 con 0x7f8dc01074b0 2024-07-27T06:46:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc7b20640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc01121c0 con 0x7f8dc0107e00 2024-07-27T06:46:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc7b20640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc01124c0 con 0x7f8dc0101b80 2024-07-27T06:46:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc50bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dc01074b0 0x7f8dc019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc50bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dc01074b0 0x7f8dc019a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53328/0 (socket says 172.21.15.29:53328) 2024-07-27T06:46:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc50bb640 1 -- 172.21.15.29:0/3684177277 learned_addr learned my addr 172.21.15.29:0/3684177277 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc58bc640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc0101b80 0x7f8dc0199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc60bd640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dc0107e00 0x7f8dc019f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc50bb640 1 -- 172.21.15.29:0/3684177277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc0101b80 msgr2=0x7f8dc0199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc50bb640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc0101b80 0x7f8dc0199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc50bb640 1 -- 172.21.15.29:0/3684177277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dc0107e00 msgr2=0x7f8dc019f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc50bb640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dc0107e00 0x7f8dc019f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc50bb640 1 -- 172.21.15.29:0/3684177277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc019fdc0 con 0x7f8dc01074b0 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.141+0000 7f8dc60bd640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dc0107e00 0x7f8dc019f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.142+0000 7f8dc50bb640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dc01074b0 0x7f8dc019a430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8db000e9c0 tx=0x7f8db000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.142+0000 7f8db6ffd640 1 -- 172.21.15.29:0/3684177277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db000ccc0 con 0x7f8dc01074b0 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.142+0000 7f8dc7b20640 1 -- 172.21.15.29:0/3684177277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc01a00b0 con 0x7f8dc01074b0 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.142+0000 7f8db6ffd640 1 -- 172.21.15.29:0/3684177277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8db00054d0 con 0x7f8dc01074b0 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.142+0000 7f8db6ffd640 1 -- 172.21.15.29:0/3684177277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0010430 con 0x7f8dc01074b0 2024-07-27T06:46:55.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.142+0000 7f8dc7b20640 1 -- 172.21.15.29:0/3684177277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dc01a6840 con 0x7f8dc01074b0 2024-07-27T06:46:55.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.143+0000 7f8dc7b20640 1 -- 172.21.15.29:0/3684177277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dc0102df0 con 0x7f8dc01074b0 2024-07-27T06:46:55.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.145+0000 7f8db6ffd640 1 -- 172.21.15.29:0/3684177277 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f8db0005000 con 0x7f8dc01074b0 2024-07-27T06:46:55.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.145+0000 7f8db6ffd640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8dac0b3c20 0x7f8dac0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:55.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.145+0000 7f8dc58bc640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8dac0b3c20 0x7f8dac0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:55.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.146+0000 7f8db6ffd640 1 -- 172.21.15.29:0/3684177277 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f8db00e42d0 con 0x7f8dc01074b0 2024-07-27T06:46:55.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.146+0000 7f8dc58bc640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8dac0b3c20 0x7f8dac0b6130 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc009bd0 tx=0x7f8dbc005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:55.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.148+0000 7f8db6ffd640 1 -- 172.21.15.29:0/3684177277 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8db0055600 con 0x7f8dc01074b0 2024-07-27T06:46:55.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.340+0000 7f8dc7b20640 1 -- 172.21.15.29:0/3684177277 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8dc01052a0 con 0x7f8dac0b3c20 2024-07-27T06:46:55.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.345+0000 7f8db6ffd640 1 -- 172.21.15.29:0/3684177277 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259157 (secure 0 0 0) 0x7f8dc01052a0 con 0x7f8dac0b3c20 2024-07-27T06:46:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:55.350 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-07-27T06:46:55.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.348+0000 7f8dc7b20640 1 -- 172.21.15.29:0/3684177277 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8dac0b3c20 msgr2=0x7f8dac0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:55.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.348+0000 7f8dc7b20640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8dac0b3c20 0x7f8dac0b6130 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc009bd0 tx=0x7f8dbc005f90 comp rx=0 tx=0).stop 2024-07-27T06:46:55.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.348+0000 7f8dc7b20640 1 -- 172.21.15.29:0/3684177277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dc01074b0 msgr2=0x7f8dc019a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:55.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.348+0000 7f8dc7b20640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dc01074b0 0x7f8dc019a430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8db000e9c0 tx=0x7f8db000ee80 comp rx=0 tx=0).stop 2024-07-27T06:46:55.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.348+0000 7f8dc7b20640 1 -- 172.21.15.29:0/3684177277 shutdown_connections 2024-07-27T06:46:55.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.348+0000 7f8dc7b20640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8dc0107e00 0x7f8dc019f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.348+0000 7f8dc7b20640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8dc01074b0 0x7f8dc019a430 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.348+0000 7f8dc7b20640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f8dac0b3c20 0x7f8dac0b6130 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.348+0000 7f8dc7b20640 1 --2- 172.21.15.29:0/3684177277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8dc0101b80 0x7f8dc0199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.348+0000 7f8dc7b20640 1 -- 172.21.15.29:0/3684177277 >> 172.21.15.29:0/3684177277 conn(0x7f8dc0079750 msgr2=0x7f8dc0100a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:55.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.348+0000 7f8dc7b20640 1 -- 172.21.15.29:0/3684177277 shutdown_connections 2024-07-27T06:46:55.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.348+0000 7f8dc7b20640 1 -- 172.21.15.29:0/3684177277 wait complete. 2024-07-27T06:46:55.360 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-07-27T06:46:53.682409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665205,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2071,"num_read_kb":1165,"num_write":389,"num_write_kb":2309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2953401,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":821,"ondisk_log_size":821,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360476,"kb_used_data":11280,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581348,"statfs":{"total":1151940427776,"available":1151571300352,"internally_reserved":0,"allocated":11550720,"data_stored":7501220,"data_compressed":247335,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5989,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":132,"num_read_kb":77,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003514"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483739+0000","last_change":"2024-07-27T06:43:12.802353+0000","last_active":"2024-07-27T06:45:09.483739+0000","last_peered":"2024-07-27T06:45:09.483739+0000","last_clean":"2024-07-27T06:45:09.483739+0000","last_became_active":"2024-07-27T06:43:12.802028+0000","last_became_peered":"2024-07-27T06:43:12.802028+0000","last_unstale":"2024-07-27T06:45:09.483739+0000","last_undegraded":"2024-07-27T06:45:09.483739+0000","last_fullsized":"2024-07-27T06:45:09.483739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:22:24.292567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862408+0000","last_change":"2024-07-27T06:43:12.806269+0000","last_active":"2024-07-27T06:45:09.862408+0000","last_peered":"2024-07-27T06:45:09.862408+0000","last_clean":"2024-07-27T06:45:09.862408+0000","last_became_active":"2024-07-27T06:43:12.803232+0000","last_became_peered":"2024-07-27T06:43:12.803232+0000","last_unstale":"2024-07-27T06:45:09.862408+0000","last_undegraded":"2024-07-27T06:45:09.862408+0000","last_fullsized":"2024-07-27T06:45:09.862408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:57:53.673850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862259+0000","last_change":"2024-07-27T06:43:12.805542+0000","last_active":"2024-07-27T06:45:09.862259+0000","last_peered":"2024-07-27T06:45:09.862259+0000","last_clean":"2024-07-27T06:45:09.862259+0000","last_became_active":"2024-07-27T06:43:12.804951+0000","last_became_peered":"2024-07-27T06:43:12.804951+0000","last_unstale":"2024-07-27T06:45:09.862259+0000","last_undegraded":"2024-07-27T06:45:09.862259+0000","last_fullsized":"2024-07-27T06:45:09.862259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:06:40.469695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862158+0000","last_change":"2024-07-27T06:43:12.796950+0000","last_active":"2024-07-27T06:45:09.862158+0000","last_peered":"2024-07-27T06:45:09.862158+0000","last_clean":"2024-07-27T06:45:09.862158+0000","last_became_active":"2024-07-27T06:43:12.796549+0000","last_became_peered":"2024-07-27T06:43:12.796549+0000","last_unstale":"2024-07-27T06:45:09.862158+0000","last_undegraded":"2024-07-27T06:45:09.862158+0000","last_fullsized":"2024-07-27T06:45:09.862158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:30:36.391046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486015+0000","last_change":"2024-07-27T06:43:12.799838+0000","last_active":"2024-07-27T06:45:09.486015+0000","last_peered":"2024-07-27T06:45:09.486015+0000","last_clean":"2024-07-27T06:45:09.486015+0000","last_became_active":"2024-07-27T06:43:12.799462+0000","last_became_peered":"2024-07-27T06:43:12.799462+0000","last_unstale":"2024-07-27T06:45:09.486015+0000","last_undegraded":"2024-07-27T06:45:09.486015+0000","last_fullsized":"2024-07-27T06:45:09.486015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:45:19.371259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481631+0000","last_change":"2024-07-27T06:43:12.797187+0000","last_active":"2024-07-27T06:45:09.481631+0000","last_peered":"2024-07-27T06:45:09.481631+0000","last_clean":"2024-07-27T06:45:09.481631+0000","last_became_active":"2024-07-27T06:43:12.796633+0000","last_became_peered":"2024-07-27T06:43:12.796633+0000","last_unstale":"2024-07-27T06:45:09.481631+0000","last_undegraded":"2024-07-27T06:45:09.481631+0000","last_fullsized":"2024-07-27T06:45:09.481631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:22:20.882899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488423+0000","last_change":"2024-07-27T06:43:12.793786+0000","last_active":"2024-07-27T06:45:09.488423+0000","last_peered":"2024-07-27T06:45:09.488423+0000","last_clean":"2024-07-27T06:45:09.488423+0000","last_became_active":"2024-07-27T06:43:12.793413+0000","last_became_peered":"2024-07-27T06:43:12.793413+0000","last_unstale":"2024-07-27T06:45:09.488423+0000","last_undegraded":"2024-07-27T06:45:09.488423+0000","last_fullsized":"2024-07-27T06:45:09.488423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:14.015373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485385+0000","last_change":"2024-07-27T06:43:12.791304+0000","last_active":"2024-07-27T06:45:09.485385+0000","last_peered":"2024-07-27T06:45:09.485385+0000","last_clean":"2024-07-27T06:45:09.485385+0000","last_became_active":"2024-07-27T06:43:12.790801+0000","last_became_peered":"2024-07-27T06:43:12.790801+0000","last_unstale":"2024-07-27T06:45:09.485385+0000","last_undegraded":"2024-07-27T06:45:09.485385+0000","last_fullsized":"2024-07-27T06:45:09.485385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:12:37.606748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822529+0000","last_change":"2024-07-27T06:43:12.797778+0000","last_active":"2024-07-27T06:45:09.822529+0000","last_peered":"2024-07-27T06:45:09.822529+0000","last_clean":"2024-07-27T06:45:09.822529+0000","last_became_active":"2024-07-27T06:43:12.796784+0000","last_became_peered":"2024-07-27T06:43:12.796784+0000","last_unstale":"2024-07-27T06:45:09.822529+0000","last_undegraded":"2024-07-27T06:45:09.822529+0000","last_fullsized":"2024-07-27T06:45:09.822529+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:52:22.275282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.482109+0000","last_change":"2024-07-27T06:43:12.798970+0000","last_active":"2024-07-27T06:45:09.482109+0000","last_peered":"2024-07-27T06:45:09.482109+0000","last_clean":"2024-07-27T06:45:09.482109+0000","last_became_active":"2024-07-27T06:43:12.798549+0000","last_became_peered":"2024-07-27T06:43:12.798549+0000","last_unstale":"2024-07-27T06:45:09.482109+0000","last_undegraded":"2024-07-27T06:45:09.482109+0000","last_fullsized":"2024-07-27T06:45:09.482109+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:58:32.464429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488604+0000","last_change":"2024-07-27T06:43:12.809405+0000","last_active":"2024-07-27T06:45:09.488604+0000","last_peered":"2024-07-27T06:45:09.488604+0000","last_clean":"2024-07-27T06:45:09.488604+0000","last_became_active":"2024-07-27T06:43:12.809030+0000","last_became_peered":"2024-07-27T06:43:12.809030+0000","last_unstale":"2024-07-27T06:45:09.488604+0000","last_undegraded":"2024-07-27T06:45:09.488604+0000","last_fullsized":"2024-07-27T06:45:09.488604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:46:35.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862193+0000","last_change":"2024-07-27T06:43:12.802647+0000","last_active":"2024-07-27T06:45:09.862193+0000","last_peered":"2024-07-27T06:45:09.862193+0000","last_clean":"2024-07-27T06:45:09.862193+0000","last_became_active":"2024-07-27T06:43:12.802247+0000","last_became_peered":"2024-07-27T06:43:12.802247+0000","last_unstale":"2024-07-27T06:45:09.862193+0000","last_undegraded":"2024-07-27T06:45:09.862193+0000","last_fullsized":"2024-07-27T06:45:09.862193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:44:24.564852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481628+0000","last_change":"2024-07-27T06:43:12.809140+0000","last_active":"2024-07-27T06:45:09.481628+0000","last_peered":"2024-07-27T06:45:09.481628+0000","last_clean":"2024-07-27T06:45:09.481628+0000","last_became_active":"2024-07-27T06:43:12.808692+0000","last_became_peered":"2024-07-27T06:43:12.808692+0000","last_unstale":"2024-07-27T06:45:09.481628+0000","last_undegraded":"2024-07-27T06:45:09.481628+0000","last_fullsized":"2024-07-27T06:45:09.481628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:37:45.326234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862226+0000","last_change":"2024-07-27T06:43:12.802838+0000","last_active":"2024-07-27T06:45:09.862226+0000","last_peered":"2024-07-27T06:45:09.862226+0000","last_clean":"2024-07-27T06:45:09.862226+0000","last_became_active":"2024-07-27T06:43:12.802503+0000","last_became_peered":"2024-07-27T06:43:12.802503+0000","last_unstale":"2024-07-27T06:45:09.862226+0000","last_undegraded":"2024-07-27T06:45:09.862226+0000","last_fullsized":"2024-07-27T06:45:09.862226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:41:03.428162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862170+0000","last_change":"2024-07-27T06:43:12.801626+0000","last_active":"2024-07-27T06:45:09.862170+0000","last_peered":"2024-07-27T06:45:09.862170+0000","last_clean":"2024-07-27T06:45:09.862170+0000","last_became_active":"2024-07-27T06:43:12.801266+0000","last_became_peered":"2024-07-27T06:43:12.801266+0000","last_unstale":"2024-07-27T06:45:09.862170+0000","last_undegraded":"2024-07-27T06:45:09.862170+0000","last_fullsized":"2024-07-27T06:45:09.862170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:54:31.934223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.821975+0000","last_change":"2024-07-27T06:43:12.804543+0000","last_active":"2024-07-27T06:45:09.821975+0000","last_peered":"2024-07-27T06:45:09.821975+0000","last_clean":"2024-07-27T06:45:09.821975+0000","last_became_active":"2024-07-27T06:43:12.804173+0000","last_became_peered":"2024-07-27T06:43:12.804173+0000","last_unstale":"2024-07-27T06:45:09.821975+0000","last_undegraded":"2024-07-27T06:45:09.821975+0000","last_fullsized":"2024-07-27T06:45:09.821975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:18:25.337333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483797+0000","last_change":"2024-07-27T06:43:12.798604+0000","last_active":"2024-07-27T06:45:09.483797+0000","last_peered":"2024-07-27T06:45:09.483797+0000","last_clean":"2024-07-27T06:45:09.483797+0000","last_became_active":"2024-07-27T06:43:12.798228+0000","last_became_peered":"2024-07-27T06:43:12.798228+0000","last_unstale":"2024-07-27T06:45:09.483797+0000","last_undegraded":"2024-07-27T06:45:09.483797+0000","last_fullsized":"2024-07-27T06:45:09.483797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:04:35.851399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633341+0000","last_change":"2024-07-27T06:43:12.800452+0000","last_active":"2024-07-27T06:45:09.633341+0000","last_peered":"2024-07-27T06:45:09.633341+0000","last_clean":"2024-07-27T06:45:09.633341+0000","last_became_active":"2024-07-27T06:43:12.800084+0000","last_became_peered":"2024-07-27T06:43:12.800084+0000","last_unstale":"2024-07-27T06:45:09.633341+0000","last_undegraded":"2024-07-27T06:45:09.633341+0000","last_fullsized":"2024-07-27T06:45:09.633341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:54:27.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485773+0000","last_change":"2024-07-27T06:43:12.800761+0000","last_active":"2024-07-27T06:45:09.485773+0000","last_peered":"2024-07-27T06:45:09.485773+0000","last_clean":"2024-07-27T06:45:09.485773+0000","last_became_active":"2024-07-27T06:43:12.800414+0000","last_became_peered":"2024-07-27T06:43:12.800414+0000","last_unstale":"2024-07-27T06:45:09.485773+0000","last_undegraded":"2024-07-27T06:45:09.485773+0000","last_fullsized":"2024-07-27T06:45:09.485773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:02:47.247522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483375+0000","last_change":"2024-07-27T06:43:12.795470+0000","last_active":"2024-07-27T06:45:09.483375+0000","last_peered":"2024-07-27T06:45:09.483375+0000","last_clean":"2024-07-27T06:45:09.483375+0000","last_became_active":"2024-07-27T06:43:12.795136+0000","last_became_peered":"2024-07-27T06:43:12.795136+0000","last_unstale":"2024-07-27T06:45:09.483375+0000","last_undegraded":"2024-07-27T06:45:09.483375+0000","last_fullsized":"2024-07-27T06:45:09.483375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:01:22.178699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481748+0000","last_change":"2024-07-27T06:43:12.803897+0000","last_active":"2024-07-27T06:45:09.481748+0000","last_peered":"2024-07-27T06:45:09.481748+0000","last_clean":"2024-07-27T06:45:09.481748+0000","last_became_active":"2024-07-27T06:43:12.803546+0000","last_became_peered":"2024-07-27T06:43:12.803546+0000","last_unstale":"2024-07-27T06:45:09.481748+0000","last_undegraded":"2024-07-27T06:45:09.481748+0000","last_fullsized":"2024-07-27T06:45:09.481748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:13:50.816831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822201+0000","last_change":"2024-07-27T06:44:09.238514+0000","last_active":"2024-07-27T06:45:09.822201+0000","last_peered":"2024-07-27T06:45:09.822201+0000","last_clean":"2024-07-27T06:45:09.822201+0000","last_became_active":"2024-07-27T06:44:09.237903+0000","last_became_peered":"2024-07-27T06:44:09.237903+0000","last_unstale":"2024-07-27T06:45:09.822201+0000","last_undegraded":"2024-07-27T06:45:09.822201+0000","last_fullsized":"2024-07-27T06:45:09.822201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:20:59.887491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485428+0000","last_change":"2024-07-27T06:43:12.788570+0000","last_active":"2024-07-27T06:45:09.485428+0000","last_peered":"2024-07-27T06:45:09.485428+0000","last_clean":"2024-07-27T06:45:09.485428+0000","last_became_active":"2024-07-27T06:43:12.788199+0000","last_became_peered":"2024-07-27T06:43:12.788199+0000","last_unstale":"2024-07-27T06:45:09.485428+0000","last_undegraded":"2024-07-27T06:45:09.485428+0000","last_fullsized":"2024-07-27T06:45:09.485428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:44:26.751094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862750+0000","last_change":"2024-07-27T06:43:12.792866+0000","last_active":"2024-07-27T06:45:09.862750+0000","last_peered":"2024-07-27T06:45:09.862750+0000","last_clean":"2024-07-27T06:45:09.862750+0000","last_became_active":"2024-07-27T06:43:12.792389+0000","last_became_peered":"2024-07-27T06:43:12.792389+0000","last_unstale":"2024-07-27T06:45:09.862750+0000","last_undegraded":"2024-07-27T06:45:09.862750+0000","last_fullsized":"2024-07-27T06:45:09.862750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:10:37.367642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485830+0000","last_change":"2024-07-27T06:43:12.797002+0000","last_active":"2024-07-27T06:45:09.485830+0000","last_peered":"2024-07-27T06:45:09.485830+0000","last_clean":"2024-07-27T06:45:09.485830+0000","last_became_active":"2024-07-27T06:43:12.796651+0000","last_became_peered":"2024-07-27T06:43:12.796651+0000","last_unstale":"2024-07-27T06:45:09.485830+0000","last_undegraded":"2024-07-27T06:45:09.485830+0000","last_fullsized":"2024-07-27T06:45:09.485830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:28:06.934042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485867+0000","last_change":"2024-07-27T06:43:12.798549+0000","last_active":"2024-07-27T06:45:09.485867+0000","last_peered":"2024-07-27T06:45:09.485867+0000","last_clean":"2024-07-27T06:45:09.485867+0000","last_became_active":"2024-07-27T06:43:12.798201+0000","last_became_peered":"2024-07-27T06:43:12.798201+0000","last_unstale":"2024-07-27T06:45:09.485867+0000","last_undegraded":"2024-07-27T06:45:09.485867+0000","last_fullsized":"2024-07-27T06:45:09.485867+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:59:47.525536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483656+0000","last_change":"2024-07-27T06:43:12.791053+0000","last_active":"2024-07-27T06:45:09.483656+0000","last_peered":"2024-07-27T06:45:09.483656+0000","last_clean":"2024-07-27T06:45:09.483656+0000","last_became_active":"2024-07-27T06:43:12.790718+0000","last_became_peered":"2024-07-27T06:43:12.790718+0000","last_unstale":"2024-07-27T06:45:09.483656+0000","last_undegraded":"2024-07-27T06:45:09.483656+0000","last_fullsized":"2024-07-27T06:45:09.483656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:09.455399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481811+0000","last_change":"2024-07-27T06:43:12.804180+0000","last_active":"2024-07-27T06:45:09.481811+0000","last_peered":"2024-07-27T06:45:09.481811+0000","last_clean":"2024-07-27T06:45:09.481811+0000","last_became_active":"2024-07-27T06:43:12.803881+0000","last_became_peered":"2024-07-27T06:43:12.803881+0000","last_unstale":"2024-07-27T06:45:09.481811+0000","last_undegraded":"2024-07-27T06:45:09.481811+0000","last_fullsized":"2024-07-27T06:45:09.481811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:01:05.009233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862727+0000","last_change":"2024-07-27T06:43:12.809150+0000","last_active":"2024-07-27T06:45:09.862727+0000","last_peered":"2024-07-27T06:45:09.862727+0000","last_clean":"2024-07-27T06:45:09.862727+0000","last_became_active":"2024-07-27T06:43:12.808428+0000","last_became_peered":"2024-07-27T06:43:12.808428+0000","last_unstale":"2024-07-27T06:45:09.862727+0000","last_undegraded":"2024-07-27T06:45:09.862727+0000","last_fullsized":"2024-07-27T06:45:09.862727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:07:13.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486325+0000","last_change":"2024-07-27T06:44:09.233785+0000","last_active":"2024-07-27T06:45:09.486325+0000","last_peered":"2024-07-27T06:45:09.486325+0000","last_clean":"2024-07-27T06:45:09.486325+0000","last_became_active":"2024-07-27T06:44:09.233184+0000","last_became_peered":"2024-07-27T06:44:09.233184+0000","last_unstale":"2024-07-27T06:45:09.486325+0000","last_undegraded":"2024-07-27T06:45:09.486325+0000","last_fullsized":"2024-07-27T06:45:09.486325+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:12:48.556383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486129+0000","last_change":"2024-07-27T06:44:09.320182+0000","last_active":"2024-07-27T06:45:09.486129+0000","last_peered":"2024-07-27T06:45:09.486129+0000","last_clean":"2024-07-27T06:45:09.486129+0000","last_became_active":"2024-07-27T06:44:09.319661+0000","last_became_peered":"2024-07-27T06:44:09.319661+0000","last_unstale":"2024-07-27T06:45:09.486129+0000","last_undegraded":"2024-07-27T06:45:09.486129+0000","last_fullsized":"2024-07-27T06:45:09.486129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:52:11.006680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483342+0000","last_change":"2024-07-27T06:43:12.787866+0000","last_active":"2024-07-27T06:45:09.483342+0000","last_peered":"2024-07-27T06:45:09.483342+0000","last_clean":"2024-07-27T06:45:09.483342+0000","last_became_active":"2024-07-27T06:43:12.787513+0000","last_became_peered":"2024-07-27T06:43:12.787513+0000","last_unstale":"2024-07-27T06:45:09.483342+0000","last_undegraded":"2024-07-27T06:45:09.483342+0000","last_fullsized":"2024-07-27T06:45:09.483342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:48:57.876588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485381+0000","last_change":"2024-07-27T06:43:12.791158+0000","last_active":"2024-07-27T06:45:09.485381+0000","last_peered":"2024-07-27T06:45:09.485381+0000","last_clean":"2024-07-27T06:45:09.485381+0000","last_became_active":"2024-07-27T06:43:12.790771+0000","last_became_peered":"2024-07-27T06:43:12.790771+0000","last_unstale":"2024-07-27T06:45:09.485381+0000","last_undegraded":"2024-07-27T06:45:09.485381+0000","last_fullsized":"2024-07-27T06:45:09.485381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:10:18.739265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'7","reported_seq":239,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:48.192811+0000","last_change":"2024-07-27T06:43:09.769281+0000","last_active":"2024-07-27T06:46:48.192811+0000","last_peered":"2024-07-27T06:46:48.192811+0000","last_clean":"2024-07-27T06:46:48.192811+0000","last_became_active":"2024-07-27T06:43:09.768864+0000","last_became_peered":"2024-07-27T06:43:09.768864+0000","last_unstale":"2024-07-27T06:46:48.192811+0000","last_undegraded":"2024-07-27T06:46:48.192811+0000","last_fullsized":"2024-07-27T06:46:48.192811+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:55:14.191180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":200,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862960+0000","last_change":"2024-07-27T06:44:09.319930+0000","last_active":"2024-07-27T06:45:09.862960+0000","last_peered":"2024-07-27T06:45:09.862960+0000","last_clean":"2024-07-27T06:45:09.862960+0000","last_became_active":"2024-07-27T06:44:09.319535+0000","last_became_peered":"2024-07-27T06:44:09.319535+0000","last_unstale":"2024-07-27T06:45:09.862960+0000","last_undegraded":"2024-07-27T06:45:09.862960+0000","last_fullsized":"2024-07-27T06:45:09.862960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:39:52.887860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481674+0000","last_change":"2024-07-27T06:43:09.772227+0000","last_active":"2024-07-27T06:45:09.481674+0000","last_peered":"2024-07-27T06:45:09.481674+0000","last_clean":"2024-07-27T06:45:09.481674+0000","last_became_active":"2024-07-27T06:43:09.771812+0000","last_became_peered":"2024-07-27T06:43:09.771812+0000","last_unstale":"2024-07-27T06:45:09.481674+0000","last_undegraded":"2024-07-27T06:45:09.481674+0000","last_fullsized":"2024-07-27T06:45:09.481674+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:27:28.054590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481030+0000","last_change":"2024-07-27T06:43:09.766490+0000","last_active":"2024-07-27T06:45:09.481030+0000","last_peered":"2024-07-27T06:45:09.481030+0000","last_clean":"2024-07-27T06:45:09.481030+0000","last_became_active":"2024-07-27T06:43:09.765973+0000","last_became_peered":"2024-07-27T06:43:09.765973+0000","last_unstale":"2024-07-27T06:45:09.481030+0000","last_undegraded":"2024-07-27T06:45:09.481030+0000","last_fullsized":"2024-07-27T06:45:09.481030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:00:02.913080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485458+0000","last_change":"2024-07-27T06:43:12.805234+0000","last_active":"2024-07-27T06:45:09.485458+0000","last_peered":"2024-07-27T06:45:09.485458+0000","last_clean":"2024-07-27T06:45:09.485458+0000","last_became_active":"2024-07-27T06:43:12.804908+0000","last_became_peered":"2024-07-27T06:43:12.804908+0000","last_unstale":"2024-07-27T06:45:09.485458+0000","last_undegraded":"2024-07-27T06:45:09.485458+0000","last_fullsized":"2024-07-27T06:45:09.485458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:09:34.988755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862248+0000","last_change":"2024-07-27T06:43:09.774201+0000","last_active":"2024-07-27T06:45:09.862248+0000","last_peered":"2024-07-27T06:45:09.862248+0000","last_clean":"2024-07-27T06:45:09.862248+0000","last_became_active":"2024-07-27T06:43:09.773817+0000","last_became_peered":"2024-07-27T06:43:09.773817+0000","last_unstale":"2024-07-27T06:45:09.862248+0000","last_undegraded":"2024-07-27T06:45:09.862248+0000","last_fullsized":"2024-07-27T06:45:09.862248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:06.166251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633110+0000","last_change":"2024-07-27T06:43:12.800174+0000","last_active":"2024-07-27T06:45:09.633110+0000","last_peered":"2024-07-27T06:45:09.633110+0000","last_clean":"2024-07-27T06:45:09.633110+0000","last_became_active":"2024-07-27T06:43:12.799769+0000","last_became_peered":"2024-07-27T06:43:12.799769+0000","last_unstale":"2024-07-27T06:45:09.633110+0000","last_undegraded":"2024-07-27T06:45:09.633110+0000","last_fullsized":"2024-07-27T06:45:09.633110+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:43:06.890372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":435,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:51.560476+0000","last_change":"2024-07-27T06:43:09.776612+0000","last_active":"2024-07-27T06:46:51.560476+0000","last_peered":"2024-07-27T06:46:51.560476+0000","last_clean":"2024-07-27T06:46:51.560476+0000","last_became_active":"2024-07-27T06:43:09.775761+0000","last_became_peered":"2024-07-27T06:43:09.775761+0000","last_unstale":"2024-07-27T06:46:51.560476+0000","last_undegraded":"2024-07-27T06:46:51.560476+0000","last_fullsized":"2024-07-27T06:46:51.560476+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:42:52.132357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":397,"num_read_kb":397,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822478+0000","last_change":"2024-07-27T06:43:12.797200+0000","last_active":"2024-07-27T06:45:09.822478+0000","last_peered":"2024-07-27T06:45:09.822478+0000","last_clean":"2024-07-27T06:45:09.822478+0000","last_became_active":"2024-07-27T06:43:12.796288+0000","last_became_peered":"2024-07-27T06:43:12.796288+0000","last_unstale":"2024-07-27T06:45:09.822478+0000","last_undegraded":"2024-07-27T06:45:09.822478+0000","last_fullsized":"2024-07-27T06:45:09.822478+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:31:27.603883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488738+0000","last_change":"2024-07-27T06:43:09.772658+0000","last_active":"2024-07-27T06:45:09.488738+0000","last_peered":"2024-07-27T06:45:09.488738+0000","last_clean":"2024-07-27T06:45:09.488738+0000","last_became_active":"2024-07-27T06:43:09.772277+0000","last_became_peered":"2024-07-27T06:43:09.772277+0000","last_unstale":"2024-07-27T06:45:09.488738+0000","last_undegraded":"2024-07-27T06:45:09.488738+0000","last_fullsized":"2024-07-27T06:45:09.488738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:09:16.854718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862551+0000","last_change":"2024-07-27T06:43:12.809643+0000","last_active":"2024-07-27T06:45:09.862551+0000","last_peered":"2024-07-27T06:45:09.862551+0000","last_clean":"2024-07-27T06:45:09.862551+0000","last_became_active":"2024-07-27T06:43:12.809020+0000","last_became_peered":"2024-07-27T06:43:12.809020+0000","last_unstale":"2024-07-27T06:45:09.862551+0000","last_undegraded":"2024-07-27T06:45:09.862551+0000","last_fullsized":"2024-07-27T06:45:09.862551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:08:03.062914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.863022+0000","last_change":"2024-07-27T06:43:10.534042+0000","last_active":"2024-07-27T06:45:09.863022+0000","last_peered":"2024-07-27T06:45:09.863022+0000","last_clean":"2024-07-27T06:45:09.863022+0000","last_became_active":"2024-07-27T06:43:10.533615+0000","last_became_peered":"2024-07-27T06:43:10.533615+0000","last_unstale":"2024-07-27T06:45:09.863022+0000","last_undegraded":"2024-07-27T06:45:09.863022+0000","last_fullsized":"2024-07-27T06:45:09.863022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:09:32.235633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822222+0000","last_change":"2024-07-27T06:43:12.798764+0000","last_active":"2024-07-27T06:45:09.822222+0000","last_peered":"2024-07-27T06:45:09.822222+0000","last_clean":"2024-07-27T06:45:09.822222+0000","last_became_active":"2024-07-27T06:43:12.797500+0000","last_became_peered":"2024-07-27T06:43:12.797500+0000","last_unstale":"2024-07-27T06:45:09.822222+0000","last_undegraded":"2024-07-27T06:45:09.822222+0000","last_fullsized":"2024-07-27T06:45:09.822222+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:12:09.368711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"83'37","reported_seq":479,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:52.199549+0000","last_change":"2024-07-27T06:43:09.775954+0000","last_active":"2024-07-27T06:46:52.199549+0000","last_peered":"2024-07-27T06:46:52.199549+0000","last_clean":"2024-07-27T06:46:52.199549+0000","last_became_active":"2024-07-27T06:43:09.775498+0000","last_became_peered":"2024-07-27T06:43:09.775498+0000","last_unstale":"2024-07-27T06:46:52.199549+0000","last_undegraded":"2024-07-27T06:46:52.199549+0000","last_fullsized":"2024-07-27T06:46:52.199549+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:08:46.965525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66277,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":410,"num_read_kb":270,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486559+0000","last_change":"2024-07-27T06:43:12.797220+0000","last_active":"2024-07-27T06:45:09.486559+0000","last_peered":"2024-07-27T06:45:09.486559+0000","last_clean":"2024-07-27T06:45:09.486559+0000","last_became_active":"2024-07-27T06:43:12.796889+0000","last_became_peered":"2024-07-27T06:43:12.796889+0000","last_unstale":"2024-07-27T06:45:09.486559+0000","last_undegraded":"2024-07-27T06:45:09.486559+0000","last_fullsized":"2024-07-27T06:45:09.486559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:24:36.275738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485662+0000","last_change":"2024-07-27T06:43:09.776600+0000","last_active":"2024-07-27T06:45:09.485662+0000","last_peered":"2024-07-27T06:45:09.485662+0000","last_clean":"2024-07-27T06:45:09.485662+0000","last_became_active":"2024-07-27T06:43:09.776163+0000","last_became_peered":"2024-07-27T06:43:09.776163+0000","last_unstale":"2024-07-27T06:45:09.485662+0000","last_undegraded":"2024-07-27T06:45:09.485662+0000","last_fullsized":"2024-07-27T06:45:09.485662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:22:15.947916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481215+0000","last_change":"2024-07-27T06:43:12.804984+0000","last_active":"2024-07-27T06:45:09.481215+0000","last_peered":"2024-07-27T06:45:09.481215+0000","last_clean":"2024-07-27T06:45:09.481215+0000","last_became_active":"2024-07-27T06:43:12.804644+0000","last_became_peered":"2024-07-27T06:43:12.804644+0000","last_unstale":"2024-07-27T06:45:09.481215+0000","last_undegraded":"2024-07-27T06:45:09.481215+0000","last_fullsized":"2024-07-27T06:45:09.481215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.355981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862246+0000","last_change":"2024-07-27T06:43:09.772489+0000","last_active":"2024-07-27T06:45:09.862246+0000","last_peered":"2024-07-27T06:45:09.862246+0000","last_clean":"2024-07-27T06:45:09.862246+0000","last_became_active":"2024-07-27T06:43:09.772069+0000","last_became_peered":"2024-07-27T06:43:09.772069+0000","last_unstale":"2024-07-27T06:45:09.862246+0000","last_undegraded":"2024-07-27T06:45:09.862246+0000","last_fullsized":"2024-07-27T06:45:09.862246+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:15:30.549971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488589+0000","last_change":"2024-07-27T06:43:12.803376+0000","last_active":"2024-07-27T06:45:09.488589+0000","last_peered":"2024-07-27T06:45:09.488589+0000","last_clean":"2024-07-27T06:45:09.488589+0000","last_became_active":"2024-07-27T06:43:12.802961+0000","last_became_peered":"2024-07-27T06:43:12.802961+0000","last_unstale":"2024-07-27T06:45:09.488589+0000","last_undegraded":"2024-07-27T06:45:09.488589+0000","last_fullsized":"2024-07-27T06:45:09.488589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:00:24.649877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":236,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:49.193981+0000","last_change":"2024-07-27T06:43:09.773182+0000","last_active":"2024-07-27T06:46:49.193981+0000","last_peered":"2024-07-27T06:46:49.193981+0000","last_clean":"2024-07-27T06:46:49.193981+0000","last_became_active":"2024-07-27T06:43:09.772126+0000","last_became_peered":"2024-07-27T06:43:09.772126+0000","last_unstale":"2024-07-27T06:46:49.193981+0000","last_undegraded":"2024-07-27T06:46:49.193981+0000","last_fullsized":"2024-07-27T06:46:49.193981+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:45:26.598799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":200,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488424+0000","last_change":"2024-07-27T06:43:12.799385+0000","last_active":"2024-07-27T06:45:09.488424+0000","last_peered":"2024-07-27T06:45:09.488424+0000","last_clean":"2024-07-27T06:45:09.488424+0000","last_became_active":"2024-07-27T06:43:12.798639+0000","last_became_peered":"2024-07-27T06:43:12.798639+0000","last_unstale":"2024-07-27T06:45:09.488424+0000","last_undegraded":"2024-07-27T06:45:09.488424+0000","last_fullsized":"2024-07-27T06:45:09.488424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:56:29.462491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485303+0000","last_change":"2024-07-27T06:43:10.535429+0000","last_active":"2024-07-27T06:45:09.485303+0000","last_peered":"2024-07-27T06:45:09.485303+0000","last_clean":"2024-07-27T06:45:09.485303+0000","last_became_active":"2024-07-27T06:43:10.534952+0000","last_became_peered":"2024-07-27T06:43:10.534952+0000","last_unstale":"2024-07-27T06:45:09.485303+0000","last_undegraded":"2024-07-27T06:45:09.485303+0000","last_fullsized":"2024-07-27T06:45:09.485303+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:54:20.018263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862706+0000","last_change":"2024-07-27T06:43:12.804266+0000","last_active":"2024-07-27T06:45:09.862706+0000","last_peered":"2024-07-27T06:45:09.862706+0000","last_clean":"2024-07-27T06:45:09.862706+0000","last_became_active":"2024-07-27T06:43:12.803888+0000","last_became_peered":"2024-07-27T06:43:12.803888+0000","last_unstale":"2024-07-27T06:45:09.862706+0000","last_undegraded":"2024-07-27T06:45:09.862706+0000","last_fullsized":"2024-07-27T06:45:09.862706+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:48:00.490592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485827+0000","last_change":"2024-07-27T06:43:09.777656+0000","last_active":"2024-07-27T06:45:09.485827+0000","last_peered":"2024-07-27T06:45:09.485827+0000","last_clean":"2024-07-27T06:45:09.485827+0000","last_became_active":"2024-07-27T06:43:09.776620+0000","last_became_peered":"2024-07-27T06:43:09.776620+0000","last_unstale":"2024-07-27T06:45:09.485827+0000","last_undegraded":"2024-07-27T06:45:09.485827+0000","last_fullsized":"2024-07-27T06:45:09.485827+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:50.203623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483544+0000","last_change":"2024-07-27T06:43:12.796179+0000","last_active":"2024-07-27T06:45:09.483544+0000","last_peered":"2024-07-27T06:45:09.483544+0000","last_clean":"2024-07-27T06:45:09.483544+0000","last_became_active":"2024-07-27T06:43:12.795701+0000","last_became_peered":"2024-07-27T06:43:12.795701+0000","last_unstale":"2024-07-27T06:45:09.483544+0000","last_undegraded":"2024-07-27T06:45:09.483544+0000","last_fullsized":"2024-07-27T06:45:09.483544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:20:12.236769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481627+0000","last_change":"2024-07-27T06:43:09.769949+0000","last_active":"2024-07-27T06:45:09.481627+0000","last_peered":"2024-07-27T06:45:09.481627+0000","last_clean":"2024-07-27T06:45:09.481627+0000","last_became_active":"2024-07-27T06:43:09.769602+0000","last_became_peered":"2024-07-27T06:43:09.769602+0000","last_unstale":"2024-07-27T06:45:09.481627+0000","last_undegraded":"2024-07-27T06:45:09.481627+0000","last_fullsized":"2024-07-27T06:45:09.481627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:28:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488869+0000","last_change":"2024-07-27T06:43:12.802230+0000","last_active":"2024-07-27T06:45:09.488869+0000","last_peered":"2024-07-27T06:45:09.488869+0000","last_clean":"2024-07-27T06:45:09.488869+0000","last_became_active":"2024-07-27T06:43:12.800944+0000","last_became_peered":"2024-07-27T06:43:12.800944+0000","last_unstale":"2024-07-27T06:45:09.488869+0000","last_undegraded":"2024-07-27T06:45:09.488869+0000","last_fullsized":"2024-07-27T06:45:09.488869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:05:54.585955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862270+0000","last_change":"2024-07-27T06:43:09.768239+0000","last_active":"2024-07-27T06:45:09.862270+0000","last_peered":"2024-07-27T06:45:09.862270+0000","last_clean":"2024-07-27T06:45:09.862270+0000","last_became_active":"2024-07-27T06:43:09.767584+0000","last_became_peered":"2024-07-27T06:43:09.767584+0000","last_unstale":"2024-07-27T06:45:09.862270+0000","last_undegraded":"2024-07-27T06:45:09.862270+0000","last_fullsized":"2024-07-27T06:45:09.862270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:35:39.552246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488571+0000","last_change":"2024-07-27T06:43:12.803779+0000","last_active":"2024-07-27T06:45:09.488571+0000","last_peered":"2024-07-27T06:45:09.488571+0000","last_clean":"2024-07-27T06:45:09.488571+0000","last_became_active":"2024-07-27T06:43:12.803359+0000","last_became_peered":"2024-07-27T06:43:12.803359+0000","last_unstale":"2024-07-27T06:45:09.488571+0000","last_undegraded":"2024-07-27T06:45:09.488571+0000","last_fullsized":"2024-07-27T06:45:09.488571+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:32:53.167039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'7","reported_seq":189,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:49.557688+0000","last_change":"2024-07-27T06:44:09.241782+0000","last_active":"2024-07-27T06:46:49.557688+0000","last_peered":"2024-07-27T06:46:49.557688+0000","last_clean":"2024-07-27T06:46:49.557688+0000","last_became_active":"2024-07-27T06:44:09.234692+0000","last_became_peered":"2024-07-27T06:44:09.234692+0000","last_unstale":"2024-07-27T06:46:49.557688+0000","last_undegraded":"2024-07-27T06:46:49.557688+0000","last_fullsized":"2024-07-27T06:46:49.557688+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:45:02.155686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":166,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1561,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488800+0000","last_change":"2024-07-27T06:43:12.805830+0000","last_active":"2024-07-27T06:45:09.488800+0000","last_peered":"2024-07-27T06:45:09.488800+0000","last_clean":"2024-07-27T06:45:09.488800+0000","last_became_active":"2024-07-27T06:43:12.805324+0000","last_became_peered":"2024-07-27T06:43:12.805324+0000","last_unstale":"2024-07-27T06:45:09.488800+0000","last_undegraded":"2024-07-27T06:45:09.488800+0000","last_fullsized":"2024-07-27T06:45:09.488800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:38:31.430676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":109,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.489084+0000","last_change":"2024-07-27T06:41:43.549132+0000","last_active":"2024-07-27T06:45:09.489084+0000","last_peered":"2024-07-27T06:45:09.489084+0000","last_clean":"2024-07-27T06:45:09.489084+0000","last_became_active":"2024-07-27T06:41:43.542635+0000","last_became_peered":"2024-07-27T06:41:43.542635+0000","last_unstale":"2024-07-27T06:45:09.489084+0000","last_undegraded":"2024-07-27T06:45:09.489084+0000","last_fullsized":"2024-07-27T06:45:09.489084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_clean_scrub_stamp":"2024-07-27T06:38:21.639907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:10:06.417387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.861700+0000","last_change":"2024-07-27T06:43:09.768774+0000","last_active":"2024-07-27T06:45:09.861700+0000","last_peered":"2024-07-27T06:45:09.861700+0000","last_clean":"2024-07-27T06:45:09.861700+0000","last_became_active":"2024-07-27T06:43:09.768345+0000","last_became_peered":"2024-07-27T06:43:09.768345+0000","last_unstale":"2024-07-27T06:45:09.861700+0000","last_undegraded":"2024-07-27T06:45:09.861700+0000","last_fullsized":"2024-07-27T06:45:09.861700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:40:49.411746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485670+0000","last_change":"2024-07-27T06:43:12.797817+0000","last_active":"2024-07-27T06:45:09.485670+0000","last_peered":"2024-07-27T06:45:09.485670+0000","last_clean":"2024-07-27T06:45:09.485670+0000","last_became_active":"2024-07-27T06:43:12.797483+0000","last_became_peered":"2024-07-27T06:43:12.797483+0000","last_unstale":"2024-07-27T06:45:09.485670+0000","last_undegraded":"2024-07-27T06:45:09.485670+0000","last_fullsized":"2024-07-27T06:45:09.485670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:24:44.539505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488887+0000","last_change":"2024-07-27T06:43:09.776880+0000","last_active":"2024-07-27T06:45:09.488887+0000","last_peered":"2024-07-27T06:45:09.488887+0000","last_clean":"2024-07-27T06:45:09.488887+0000","last_became_active":"2024-07-27T06:43:09.776486+0000","last_became_peered":"2024-07-27T06:43:09.776486+0000","last_unstale":"2024-07-27T06:45:09.488887+0000","last_undegraded":"2024-07-27T06:45:09.488887+0000","last_fullsized":"2024-07-27T06:45:09.488887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:55:25.436405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.480961+0000","last_change":"2024-07-27T06:43:12.799755+0000","last_active":"2024-07-27T06:45:09.480961+0000","last_peered":"2024-07-27T06:45:09.480961+0000","last_clean":"2024-07-27T06:45:09.480961+0000","last_became_active":"2024-07-27T06:43:12.799412+0000","last_became_peered":"2024-07-27T06:43:12.799412+0000","last_unstale":"2024-07-27T06:45:09.480961+0000","last_undegraded":"2024-07-27T06:45:09.480961+0000","last_fullsized":"2024-07-27T06:45:09.480961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:45:50.545431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":235,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:49.193450+0000","last_change":"2024-07-27T06:43:10.533039+0000","last_active":"2024-07-27T06:46:49.193450+0000","last_peered":"2024-07-27T06:46:49.193450+0000","last_clean":"2024-07-27T06:46:49.193450+0000","last_became_active":"2024-07-27T06:43:10.532583+0000","last_became_peered":"2024-07-27T06:43:10.532583+0000","last_unstale":"2024-07-27T06:46:49.193450+0000","last_undegraded":"2024-07-27T06:46:49.193450+0000","last_fullsized":"2024-07-27T06:46:49.193450+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:04:51.954463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":198,"num_read_kb":198,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633096+0000","last_change":"2024-07-27T06:44:09.238218+0000","last_active":"2024-07-27T06:45:09.633096+0000","last_peered":"2024-07-27T06:45:09.633096+0000","last_clean":"2024-07-27T06:45:09.633096+0000","last_became_active":"2024-07-27T06:44:09.237841+0000","last_became_peered":"2024-07-27T06:44:09.237841+0000","last_unstale":"2024-07-27T06:45:09.633096+0000","last_undegraded":"2024-07-27T06:45:09.633096+0000","last_fullsized":"2024-07-27T06:45:09.633096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:19.101854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"83'8","reported_seq":242,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:50.237432+0000","last_change":"2024-07-27T06:43:09.776214+0000","last_active":"2024-07-27T06:46:50.237432+0000","last_peered":"2024-07-27T06:46:50.237432+0000","last_clean":"2024-07-27T06:46:50.237432+0000","last_became_active":"2024-07-27T06:43:09.775719+0000","last_became_peered":"2024-07-27T06:43:09.775719+0000","last_unstale":"2024-07-27T06:46:50.237432+0000","last_undegraded":"2024-07-27T06:46:50.237432+0000","last_fullsized":"2024-07-27T06:46:50.237432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:57:20.381599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":202,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481735+0000","last_change":"2024-07-27T06:43:12.794909+0000","last_active":"2024-07-27T06:45:09.481735+0000","last_peered":"2024-07-27T06:45:09.481735+0000","last_clean":"2024-07-27T06:45:09.481735+0000","last_became_active":"2024-07-27T06:43:12.794497+0000","last_became_peered":"2024-07-27T06:43:12.794497+0000","last_unstale":"2024-07-27T06:45:09.481735+0000","last_undegraded":"2024-07-27T06:45:09.481735+0000","last_fullsized":"2024-07-27T06:45:09.481735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:42:49.760126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633255+0000","last_change":"2024-07-27T06:43:10.534240+0000","last_active":"2024-07-27T06:45:09.633255+0000","last_peered":"2024-07-27T06:45:09.633255+0000","last_clean":"2024-07-27T06:45:09.633255+0000","last_became_active":"2024-07-27T06:43:10.533596+0000","last_became_peered":"2024-07-27T06:43:10.533596+0000","last_unstale":"2024-07-27T06:45:09.633255+0000","last_undegraded":"2024-07-27T06:45:09.633255+0000","last_fullsized":"2024-07-27T06:45:09.633255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:16.098839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481991+0000","last_change":"2024-07-27T06:43:12.805248+0000","last_active":"2024-07-27T06:45:09.481991+0000","last_peered":"2024-07-27T06:45:09.481991+0000","last_clean":"2024-07-27T06:45:09.481991+0000","last_became_active":"2024-07-27T06:43:12.803693+0000","last_became_peered":"2024-07-27T06:43:12.803693+0000","last_unstale":"2024-07-27T06:45:09.481991+0000","last_undegraded":"2024-07-27T06:45:09.481991+0000","last_fullsized":"2024-07-27T06:45:09.481991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:17.466465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486692+0000","last_change":"2024-07-27T06:43:10.533701+0000","last_active":"2024-07-27T06:45:09.486692+0000","last_peered":"2024-07-27T06:45:09.486692+0000","last_clean":"2024-07-27T06:45:09.486692+0000","last_became_active":"2024-07-27T06:43:10.533109+0000","last_became_peered":"2024-07-27T06:43:10.533109+0000","last_unstale":"2024-07-27T06:45:09.486692+0000","last_undegraded":"2024-07-27T06:45:09.486692+0000","last_fullsized":"2024-07-27T06:45:09.486692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:32:52.906691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481918+0000","last_change":"2024-07-27T06:43:12.798479+0000","last_active":"2024-07-27T06:45:09.481918+0000","last_peered":"2024-07-27T06:45:09.481918+0000","last_clean":"2024-07-27T06:45:09.481918+0000","last_became_active":"2024-07-27T06:43:12.798124+0000","last_became_peered":"2024-07-27T06:43:12.798124+0000","last_unstale":"2024-07-27T06:45:09.481918+0000","last_undegraded":"2024-07-27T06:45:09.481918+0000","last_fullsized":"2024-07-27T06:45:09.481918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:14:55.267213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"83'17","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:22.215894+0000","last_change":"2024-07-27T06:43:09.770122+0000","last_active":"2024-07-27T06:45:22.215894+0000","last_peered":"2024-07-27T06:45:22.215894+0000","last_clean":"2024-07-27T06:45:22.215894+0000","last_became_active":"2024-07-27T06:43:09.769784+0000","last_became_peered":"2024-07-27T06:43:09.769784+0000","last_unstale":"2024-07-27T06:45:22.215894+0000","last_undegraded":"2024-07-27T06:45:22.215894+0000","last_fullsized":"2024-07-27T06:45:22.215894+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:47:00.398135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485891+0000","last_change":"2024-07-27T06:43:12.799603+0000","last_active":"2024-07-27T06:45:09.485891+0000","last_peered":"2024-07-27T06:45:09.485891+0000","last_clean":"2024-07-27T06:45:09.485891+0000","last_became_active":"2024-07-27T06:43:12.799189+0000","last_became_peered":"2024-07-27T06:43:12.799189+0000","last_unstale":"2024-07-27T06:45:09.485891+0000","last_undegraded":"2024-07-27T06:45:09.485891+0000","last_fullsized":"2024-07-27T06:45:09.485891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:32:30.603258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862220+0000","last_change":"2024-07-27T06:43:10.004726+0000","last_active":"2024-07-27T06:45:09.862220+0000","last_peered":"2024-07-27T06:45:09.862220+0000","last_clean":"2024-07-27T06:45:09.862220+0000","last_became_active":"2024-07-27T06:43:10.004367+0000","last_became_peered":"2024-07-27T06:43:10.004367+0000","last_unstale":"2024-07-27T06:45:09.862220+0000","last_undegraded":"2024-07-27T06:45:09.862220+0000","last_fullsized":"2024-07-27T06:45:09.862220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:22:39.942391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862427+0000","last_change":"2024-07-27T06:43:12.796511+0000","last_active":"2024-07-27T06:45:09.862427+0000","last_peered":"2024-07-27T06:45:09.862427+0000","last_clean":"2024-07-27T06:45:09.862427+0000","last_became_active":"2024-07-27T06:43:12.796152+0000","last_became_peered":"2024-07-27T06:43:12.796152+0000","last_unstale":"2024-07-27T06:45:09.862427+0000","last_undegraded":"2024-07-27T06:45:09.862427+0000","last_fullsized":"2024-07-27T06:45:09.862427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:03:02.508990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822077+0000","last_change":"2024-07-27T06:43:10.533656+0000","last_active":"2024-07-27T06:45:09.822077+0000","last_peered":"2024-07-27T06:45:09.822077+0000","last_clean":"2024-07-27T06:45:09.822077+0000","last_became_active":"2024-07-27T06:43:10.533295+0000","last_became_peered":"2024-07-27T06:43:10.533295+0000","last_unstale":"2024-07-27T06:45:09.822077+0000","last_undegraded":"2024-07-27T06:45:09.822077+0000","last_fullsized":"2024-07-27T06:45:09.822077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:11:00.032511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485237+0000","last_change":"2024-07-27T06:43:12.795106+0000","last_active":"2024-07-27T06:45:09.485237+0000","last_peered":"2024-07-27T06:45:09.485237+0000","last_clean":"2024-07-27T06:45:09.485237+0000","last_became_active":"2024-07-27T06:43:12.794729+0000","last_became_peered":"2024-07-27T06:43:12.794729+0000","last_unstale":"2024-07-27T06:45:09.485237+0000","last_undegraded":"2024-07-27T06:45:09.485237+0000","last_fullsized":"2024-07-27T06:45:09.485237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:19:16.542363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485673+0000","last_change":"2024-07-27T06:43:09.770016+0000","last_active":"2024-07-27T06:45:09.485673+0000","last_peered":"2024-07-27T06:45:09.485673+0000","last_clean":"2024-07-27T06:45:09.485673+0000","last_became_active":"2024-07-27T06:43:09.769672+0000","last_became_peered":"2024-07-27T06:43:09.769672+0000","last_unstale":"2024-07-27T06:45:09.485673+0000","last_undegraded":"2024-07-27T06:45:09.485673+0000","last_fullsized":"2024-07-27T06:45:09.485673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:30:55.737494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483659+0000","last_change":"2024-07-27T06:43:12.793690+0000","last_active":"2024-07-27T06:45:09.483659+0000","last_peered":"2024-07-27T06:45:09.483659+0000","last_clean":"2024-07-27T06:45:09.483659+0000","last_became_active":"2024-07-27T06:43:12.793352+0000","last_became_peered":"2024-07-27T06:43:12.793352+0000","last_unstale":"2024-07-27T06:45:09.483659+0000","last_undegraded":"2024-07-27T06:45:09.483659+0000","last_fullsized":"2024-07-27T06:45:09.483659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:22:47.606990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.491196+0000","last_change":"2024-07-27T06:45:09.490738+0000","last_active":"2024-07-27T06:45:09.491196+0000","last_peered":"2024-07-27T06:45:09.491196+0000","last_clean":"2024-07-27T06:45:09.491196+0000","last_became_active":"2024-07-27T06:45:09.487147+0000","last_became_peered":"2024-07-27T06:45:09.487147+0000","last_unstale":"2024-07-27T06:45:09.491196+0000","last_undegraded":"2024-07-27T06:45:09.491196+0000","last_fullsized":"2024-07-27T06:45:09.491196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:46.732987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485570+0000","last_change":"2024-07-27T06:43:12.792648+0000","last_active":"2024-07-27T06:45:09.485570+0000","last_peered":"2024-07-27T06:45:09.485570+0000","last_clean":"2024-07-27T06:45:09.485570+0000","last_became_active":"2024-07-27T06:43:12.792338+0000","last_became_peered":"2024-07-27T06:43:12.792338+0000","last_unstale":"2024-07-27T06:45:09.485570+0000","last_undegraded":"2024-07-27T06:45:09.485570+0000","last_fullsized":"2024-07-27T06:45:09.485570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:36:52.605957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.827632+0000","last_change":"2024-07-27T06:45:09.827140+0000","last_active":"2024-07-27T06:45:09.827632+0000","last_peered":"2024-07-27T06:45:09.827632+0000","last_clean":"2024-07-27T06:45:09.827632+0000","last_became_active":"2024-07-27T06:45:09.823119+0000","last_became_peered":"2024-07-27T06:45:09.823119+0000","last_unstale":"2024-07-27T06:45:09.827632+0000","last_undegraded":"2024-07-27T06:45:09.827632+0000","last_fullsized":"2024-07-27T06:45:09.827632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:32:19.682193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488730+0000","last_change":"2024-07-27T06:43:12.809347+0000","last_active":"2024-07-27T06:45:09.488730+0000","last_peered":"2024-07-27T06:45:09.488730+0000","last_clean":"2024-07-27T06:45:09.488730+0000","last_became_active":"2024-07-27T06:43:12.808980+0000","last_became_peered":"2024-07-27T06:43:12.808980+0000","last_unstale":"2024-07-27T06:45:09.488730+0000","last_undegraded":"2024-07-27T06:45:09.488730+0000","last_fullsized":"2024-07-27T06:45:09.488730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:05:14.132641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633284+0000","last_change":"2024-07-27T06:44:09.237908+0000","last_active":"2024-07-27T06:45:09.633284+0000","last_peered":"2024-07-27T06:45:09.633284+0000","last_clean":"2024-07-27T06:45:09.633284+0000","last_became_active":"2024-07-27T06:44:09.237503+0000","last_became_peered":"2024-07-27T06:44:09.237503+0000","last_unstale":"2024-07-27T06:45:09.633284+0000","last_undegraded":"2024-07-27T06:45:09.633284+0000","last_fullsized":"2024-07-27T06:45:09.633284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:29.654244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486391+0000","last_change":"2024-07-27T06:43:09.776000+0000","last_active":"2024-07-27T06:45:09.486391+0000","last_peered":"2024-07-27T06:45:09.486391+0000","last_clean":"2024-07-27T06:45:09.486391+0000","last_became_active":"2024-07-27T06:43:09.775522+0000","last_became_peered":"2024-07-27T06:43:09.775522+0000","last_unstale":"2024-07-27T06:45:09.486391+0000","last_undegraded":"2024-07-27T06:45:09.486391+0000","last_fullsized":"2024-07-27T06:45:09.486391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:39:42.650177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822084+0000","last_change":"2024-07-27T06:43:12.807062+0000","last_active":"2024-07-27T06:45:09.822084+0000","last_peered":"2024-07-27T06:45:09.822084+0000","last_clean":"2024-07-27T06:45:09.822084+0000","last_became_active":"2024-07-27T06:43:12.805816+0000","last_became_peered":"2024-07-27T06:43:12.805816+0000","last_unstale":"2024-07-27T06:45:09.822084+0000","last_undegraded":"2024-07-27T06:45:09.822084+0000","last_fullsized":"2024-07-27T06:45:09.822084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:44:16.261663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483465+0000","last_change":"2024-07-27T06:44:09.250070+0000","last_active":"2024-07-27T06:45:09.483465+0000","last_peered":"2024-07-27T06:45:09.483465+0000","last_clean":"2024-07-27T06:45:09.483465+0000","last_became_active":"2024-07-27T06:44:09.237290+0000","last_became_peered":"2024-07-27T06:44:09.237290+0000","last_unstale":"2024-07-27T06:45:09.483465+0000","last_undegraded":"2024-07-27T06:45:09.483465+0000","last_fullsized":"2024-07-27T06:45:09.483465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:51.590487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862423+0000","last_change":"2024-07-27T06:43:12.809135+0000","last_active":"2024-07-27T06:45:09.862423+0000","last_peered":"2024-07-27T06:45:09.862423+0000","last_clean":"2024-07-27T06:45:09.862423+0000","last_became_active":"2024-07-27T06:43:12.806731+0000","last_became_peered":"2024-07-27T06:43:12.806731+0000","last_unstale":"2024-07-27T06:45:09.862423+0000","last_undegraded":"2024-07-27T06:45:09.862423+0000","last_fullsized":"2024-07-27T06:45:09.862423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:47:50.397517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486050+0000","last_change":"2024-07-27T06:43:10.533608+0000","last_active":"2024-07-27T06:45:09.486050+0000","last_peered":"2024-07-27T06:45:09.486050+0000","last_clean":"2024-07-27T06:45:09.486050+0000","last_became_active":"2024-07-27T06:43:10.532663+0000","last_became_peered":"2024-07-27T06:43:10.532663+0000","last_unstale":"2024-07-27T06:45:09.486050+0000","last_undegraded":"2024-07-27T06:45:09.486050+0000","last_fullsized":"2024-07-27T06:45:09.486050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:05:33.155427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485883+0000","last_change":"2024-07-27T06:43:12.789890+0000","last_active":"2024-07-27T06:45:09.485883+0000","last_peered":"2024-07-27T06:45:09.485883+0000","last_clean":"2024-07-27T06:45:09.485883+0000","last_became_active":"2024-07-27T06:43:12.789561+0000","last_became_peered":"2024-07-27T06:43:12.789561+0000","last_unstale":"2024-07-27T06:45:09.485883+0000","last_undegraded":"2024-07-27T06:45:09.485883+0000","last_fullsized":"2024-07-27T06:45:09.485883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:35:49.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"83'15","reported_seq":246,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:50.236769+0000","last_change":"2024-07-27T06:43:09.773471+0000","last_active":"2024-07-27T06:46:50.236769+0000","last_peered":"2024-07-27T06:46:50.236769+0000","last_clean":"2024-07-27T06:46:50.236769+0000","last_became_active":"2024-07-27T06:43:09.773213+0000","last_became_peered":"2024-07-27T06:43:09.773213+0000","last_unstale":"2024-07-27T06:46:50.236769+0000","last_undegraded":"2024-07-27T06:46:50.236769+0000","last_fullsized":"2024-07-27T06:46:50.236769+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:54:38.770198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":199,"num_read_kb":199,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74603,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1990,"num_read_kb":1096,"num_write":264,"num_write_kb":157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1561,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":213063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283116,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678076,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":461919,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874304,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1113320,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037831,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":461317,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234059,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":461695,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430287,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":1055194,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593813,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":461874,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790042,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":1054032,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953567,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":929792,"data_stored":462597,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":460859,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378730,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":522433,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542254,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":983040,"data_stored":523075,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705779,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":462905,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":68013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T06:46:55.365 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-07-27T06:46:55.365 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-27T06:46:55.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.515+0000 7ffa40dbe640 1 -- 172.21.15.29:0/3933619615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c108e40 msgr2=0x7ffa3c109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:55.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.515+0000 7ffa40dbe640 1 --2- 172.21.15.29:0/3933619615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c108e40 0x7ffa3c109250 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009900 tx=0x7ffa3002a410 comp rx=0 tx=0).stop 2024-07-27T06:46:55.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.515+0000 7ffa40dbe640 1 -- 172.21.15.29:0/3933619615 shutdown_connections 2024-07-27T06:46:55.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.515+0000 7ffa40dbe640 1 --2- 172.21.15.29:0/3933619615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c109790 0x7ffa3c10ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.515+0000 7ffa40dbe640 1 --2- 172.21.15.29:0/3933619615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c108e40 0x7ffa3c109250 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.515+0000 7ffa40dbe640 1 --2- 172.21.15.29:0/3933619615 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa3c101a70 0x7ffa3c101e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.515+0000 7ffa40dbe640 1 -- 172.21.15.29:0/3933619615 >> 172.21.15.29:0/3933619615 conn(0x7ffa3c0798a0 msgr2=0x7ffa3c079cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:55.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.515+0000 7ffa40dbe640 1 -- 172.21.15.29:0/3933619615 shutdown_connections 2024-07-27T06:46:55.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.516+0000 7ffa40dbe640 1 -- 172.21.15.29:0/3933619615 wait complete. 2024-07-27T06:46:55.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.516+0000 7ffa40dbe640 1 Processor -- start 2024-07-27T06:46:55.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.516+0000 7ffa40dbe640 1 -- start start 2024-07-27T06:46:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa40dbe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa3c101a70 0x7ffa3c199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa40dbe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c108e40 0x7ffa3c19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa40dbe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c109790 0x7ffa3c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa40dbe640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c113420 con 0x7ffa3c109790 2024-07-27T06:46:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa40dbe640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c1132a0 con 0x7ffa3c101a70 2024-07-27T06:46:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa40dbe640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c1135a0 con 0x7ffa3c108e40 2024-07-27T06:46:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa3ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c109790 0x7ffa3c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa3ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c109790 0x7ffa3c19f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53342/0 (socket says 172.21.15.29:53342) 2024-07-27T06:46:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa3ad9d640 1 -- 172.21.15.29:0/634667969 learned_addr learned my addr 172.21.15.29:0/634667969 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa3a59c640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa3c101a70 0x7ffa3c199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa39d9b640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c108e40 0x7ffa3c19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa3ad9d640 1 -- 172.21.15.29:0/634667969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c108e40 msgr2=0x7ffa3c19a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa3ad9d640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c108e40 0x7ffa3c19a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa3ad9d640 1 -- 172.21.15.29:0/634667969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa3c101a70 msgr2=0x7ffa3c199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa3ad9d640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa3c101a70 0x7ffa3c199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.517+0000 7ffa3ad9d640 1 -- 172.21.15.29:0/634667969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c19fd70 con 0x7ffa3c109790 2024-07-27T06:46:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.518+0000 7ffa3ad9d640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c109790 0x7ffa3c19f670 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400cad0 tx=0x7ffa3400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.518+0000 7ffa2b7fe640 1 -- 172.21.15.29:0/634667969 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3400ae20 con 0x7ffa3c109790 2024-07-27T06:46:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.518+0000 7ffa40dbe640 1 -- 172.21.15.29:0/634667969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa3c1a0060 con 0x7ffa3c109790 2024-07-27T06:46:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.518+0000 7ffa2b7fe640 1 -- 172.21.15.29:0/634667969 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa34005020 con 0x7ffa3c109790 2024-07-27T06:46:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.518+0000 7ffa2b7fe640 1 -- 172.21.15.29:0/634667969 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3400f6b0 con 0x7ffa3c109790 2024-07-27T06:46:55.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.518+0000 7ffa40dbe640 1 -- 172.21.15.29:0/634667969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa3c1a67f0 con 0x7ffa3c109790 2024-07-27T06:46:55.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.519+0000 7ffa40dbe640 1 -- 172.21.15.29:0/634667969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa3c101e80 con 0x7ffa3c109790 2024-07-27T06:46:55.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.520+0000 7ffa2b7fe640 1 -- 172.21.15.29:0/634667969 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7ffa340051d0 con 0x7ffa3c109790 2024-07-27T06:46:55.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.521+0000 7ffa2b7fe640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ffa240b3c20 0x7ffa240b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:55.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.522+0000 7ffa3a59c640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ffa240b3c20 0x7ffa240b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:55.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.522+0000 7ffa2b7fe640 1 -- 172.21.15.29:0/634667969 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7ffa340e4100 con 0x7ffa3c109790 2024-07-27T06:46:55.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.522+0000 7ffa3a59c640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ffa240b3c20 0x7ffa240b6130 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c0058c0 tx=0x7ffa2c00a2d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:55.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.525+0000 7ffa2b7fe640 1 -- 172.21.15.29:0/634667969 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ffa34014030 con 0x7ffa3c109790 2024-07-27T06:46:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:55 smithi047 ceph-mon[36835]: pgmap v394: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-27T06:46:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:55 smithi029 ceph-mon[32610]: pgmap v394: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-27T06:46:55.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.715+0000 7ffa40dbe640 1 -- 172.21.15.29:0/634667969 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffa3c063230 con 0x7ffa240b3c20 2024-07-27T06:46:55.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.720+0000 7ffa2b7fe640 1 -- 172.21.15.29:0/634667969 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259157 (secure 0 0 0) 0x7ffa3c063230 con 0x7ffa240b3c20 2024-07-27T06:46:55.723 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:55.724 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-07-27T06:46:55.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.723+0000 7ffa40dbe640 1 -- 172.21.15.29:0/634667969 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ffa240b3c20 msgr2=0x7ffa240b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:55.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.723+0000 7ffa40dbe640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ffa240b3c20 0x7ffa240b6130 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c0058c0 tx=0x7ffa2c00a2d0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.723+0000 7ffa40dbe640 1 -- 172.21.15.29:0/634667969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c109790 msgr2=0x7ffa3c19f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:55.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.723+0000 7ffa40dbe640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c109790 0x7ffa3c19f670 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400cad0 tx=0x7ffa3400cf90 comp rx=0 tx=0).stop 2024-07-27T06:46:55.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.723+0000 7ffa40dbe640 1 -- 172.21.15.29:0/634667969 shutdown_connections 2024-07-27T06:46:55.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.723+0000 7ffa40dbe640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c109790 0x7ffa3c19f670 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.723+0000 7ffa40dbe640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ffa240b3c20 0x7ffa240b6130 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.723+0000 7ffa40dbe640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c108e40 0x7ffa3c19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.723+0000 7ffa40dbe640 1 --2- 172.21.15.29:0/634667969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa3c101a70 0x7ffa3c199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:55.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.723+0000 7ffa40dbe640 1 -- 172.21.15.29:0/634667969 >> 172.21.15.29:0/634667969 conn(0x7ffa3c0798a0 msgr2=0x7ffa3c106ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:55.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.723+0000 7ffa40dbe640 1 -- 172.21.15.29:0/634667969 shutdown_connections 2024-07-27T06:46:55.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:55.723+0000 7ffa40dbe640 1 -- 172.21.15.29:0/634667969 wait complete. 2024-07-27T06:46:55.736 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-07-27T06:46:55.682864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665205,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2086,"num_read_kb":1165,"num_write":389,"num_write_kb":2309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2953401,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":821,"ondisk_log_size":821,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360480,"kb_used_data":11288,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581344,"statfs":{"total":1151940427776,"available":1151571296256,"internally_reserved":0,"allocated":11558912,"data_stored":7507209,"data_compressed":247335,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5989,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":57,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003308"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483739+0000","last_change":"2024-07-27T06:43:12.802353+0000","last_active":"2024-07-27T06:45:09.483739+0000","last_peered":"2024-07-27T06:45:09.483739+0000","last_clean":"2024-07-27T06:45:09.483739+0000","last_became_active":"2024-07-27T06:43:12.802028+0000","last_became_peered":"2024-07-27T06:43:12.802028+0000","last_unstale":"2024-07-27T06:45:09.483739+0000","last_undegraded":"2024-07-27T06:45:09.483739+0000","last_fullsized":"2024-07-27T06:45:09.483739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:22:24.292567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862408+0000","last_change":"2024-07-27T06:43:12.806269+0000","last_active":"2024-07-27T06:45:09.862408+0000","last_peered":"2024-07-27T06:45:09.862408+0000","last_clean":"2024-07-27T06:45:09.862408+0000","last_became_active":"2024-07-27T06:43:12.803232+0000","last_became_peered":"2024-07-27T06:43:12.803232+0000","last_unstale":"2024-07-27T06:45:09.862408+0000","last_undegraded":"2024-07-27T06:45:09.862408+0000","last_fullsized":"2024-07-27T06:45:09.862408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:57:53.673850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862259+0000","last_change":"2024-07-27T06:43:12.805542+0000","last_active":"2024-07-27T06:45:09.862259+0000","last_peered":"2024-07-27T06:45:09.862259+0000","last_clean":"2024-07-27T06:45:09.862259+0000","last_became_active":"2024-07-27T06:43:12.804951+0000","last_became_peered":"2024-07-27T06:43:12.804951+0000","last_unstale":"2024-07-27T06:45:09.862259+0000","last_undegraded":"2024-07-27T06:45:09.862259+0000","last_fullsized":"2024-07-27T06:45:09.862259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:06:40.469695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862158+0000","last_change":"2024-07-27T06:43:12.796950+0000","last_active":"2024-07-27T06:45:09.862158+0000","last_peered":"2024-07-27T06:45:09.862158+0000","last_clean":"2024-07-27T06:45:09.862158+0000","last_became_active":"2024-07-27T06:43:12.796549+0000","last_became_peered":"2024-07-27T06:43:12.796549+0000","last_unstale":"2024-07-27T06:45:09.862158+0000","last_undegraded":"2024-07-27T06:45:09.862158+0000","last_fullsized":"2024-07-27T06:45:09.862158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:30:36.391046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486015+0000","last_change":"2024-07-27T06:43:12.799838+0000","last_active":"2024-07-27T06:45:09.486015+0000","last_peered":"2024-07-27T06:45:09.486015+0000","last_clean":"2024-07-27T06:45:09.486015+0000","last_became_active":"2024-07-27T06:43:12.799462+0000","last_became_peered":"2024-07-27T06:43:12.799462+0000","last_unstale":"2024-07-27T06:45:09.486015+0000","last_undegraded":"2024-07-27T06:45:09.486015+0000","last_fullsized":"2024-07-27T06:45:09.486015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:45:19.371259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481631+0000","last_change":"2024-07-27T06:43:12.797187+0000","last_active":"2024-07-27T06:45:09.481631+0000","last_peered":"2024-07-27T06:45:09.481631+0000","last_clean":"2024-07-27T06:45:09.481631+0000","last_became_active":"2024-07-27T06:43:12.796633+0000","last_became_peered":"2024-07-27T06:43:12.796633+0000","last_unstale":"2024-07-27T06:45:09.481631+0000","last_undegraded":"2024-07-27T06:45:09.481631+0000","last_fullsized":"2024-07-27T06:45:09.481631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:22:20.882899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488423+0000","last_change":"2024-07-27T06:43:12.793786+0000","last_active":"2024-07-27T06:45:09.488423+0000","last_peered":"2024-07-27T06:45:09.488423+0000","last_clean":"2024-07-27T06:45:09.488423+0000","last_became_active":"2024-07-27T06:43:12.793413+0000","last_became_peered":"2024-07-27T06:43:12.793413+0000","last_unstale":"2024-07-27T06:45:09.488423+0000","last_undegraded":"2024-07-27T06:45:09.488423+0000","last_fullsized":"2024-07-27T06:45:09.488423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:14.015373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485385+0000","last_change":"2024-07-27T06:43:12.791304+0000","last_active":"2024-07-27T06:45:09.485385+0000","last_peered":"2024-07-27T06:45:09.485385+0000","last_clean":"2024-07-27T06:45:09.485385+0000","last_became_active":"2024-07-27T06:43:12.790801+0000","last_became_peered":"2024-07-27T06:43:12.790801+0000","last_unstale":"2024-07-27T06:45:09.485385+0000","last_undegraded":"2024-07-27T06:45:09.485385+0000","last_fullsized":"2024-07-27T06:45:09.485385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:12:37.606748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822529+0000","last_change":"2024-07-27T06:43:12.797778+0000","last_active":"2024-07-27T06:45:09.822529+0000","last_peered":"2024-07-27T06:45:09.822529+0000","last_clean":"2024-07-27T06:45:09.822529+0000","last_became_active":"2024-07-27T06:43:12.796784+0000","last_became_peered":"2024-07-27T06:43:12.796784+0000","last_unstale":"2024-07-27T06:45:09.822529+0000","last_undegraded":"2024-07-27T06:45:09.822529+0000","last_fullsized":"2024-07-27T06:45:09.822529+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:52:22.275282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.482109+0000","last_change":"2024-07-27T06:43:12.798970+0000","last_active":"2024-07-27T06:45:09.482109+0000","last_peered":"2024-07-27T06:45:09.482109+0000","last_clean":"2024-07-27T06:45:09.482109+0000","last_became_active":"2024-07-27T06:43:12.798549+0000","last_became_peered":"2024-07-27T06:43:12.798549+0000","last_unstale":"2024-07-27T06:45:09.482109+0000","last_undegraded":"2024-07-27T06:45:09.482109+0000","last_fullsized":"2024-07-27T06:45:09.482109+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:58:32.464429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488604+0000","last_change":"2024-07-27T06:43:12.809405+0000","last_active":"2024-07-27T06:45:09.488604+0000","last_peered":"2024-07-27T06:45:09.488604+0000","last_clean":"2024-07-27T06:45:09.488604+0000","last_became_active":"2024-07-27T06:43:12.809030+0000","last_became_peered":"2024-07-27T06:43:12.809030+0000","last_unstale":"2024-07-27T06:45:09.488604+0000","last_undegraded":"2024-07-27T06:45:09.488604+0000","last_fullsized":"2024-07-27T06:45:09.488604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:46:35.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862193+0000","last_change":"2024-07-27T06:43:12.802647+0000","last_active":"2024-07-27T06:45:09.862193+0000","last_peered":"2024-07-27T06:45:09.862193+0000","last_clean":"2024-07-27T06:45:09.862193+0000","last_became_active":"2024-07-27T06:43:12.802247+0000","last_became_peered":"2024-07-27T06:43:12.802247+0000","last_unstale":"2024-07-27T06:45:09.862193+0000","last_undegraded":"2024-07-27T06:45:09.862193+0000","last_fullsized":"2024-07-27T06:45:09.862193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:44:24.564852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481628+0000","last_change":"2024-07-27T06:43:12.809140+0000","last_active":"2024-07-27T06:45:09.481628+0000","last_peered":"2024-07-27T06:45:09.481628+0000","last_clean":"2024-07-27T06:45:09.481628+0000","last_became_active":"2024-07-27T06:43:12.808692+0000","last_became_peered":"2024-07-27T06:43:12.808692+0000","last_unstale":"2024-07-27T06:45:09.481628+0000","last_undegraded":"2024-07-27T06:45:09.481628+0000","last_fullsized":"2024-07-27T06:45:09.481628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:37:45.326234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862226+0000","last_change":"2024-07-27T06:43:12.802838+0000","last_active":"2024-07-27T06:45:09.862226+0000","last_peered":"2024-07-27T06:45:09.862226+0000","last_clean":"2024-07-27T06:45:09.862226+0000","last_became_active":"2024-07-27T06:43:12.802503+0000","last_became_peered":"2024-07-27T06:43:12.802503+0000","last_unstale":"2024-07-27T06:45:09.862226+0000","last_undegraded":"2024-07-27T06:45:09.862226+0000","last_fullsized":"2024-07-27T06:45:09.862226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:41:03.428162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862170+0000","last_change":"2024-07-27T06:43:12.801626+0000","last_active":"2024-07-27T06:45:09.862170+0000","last_peered":"2024-07-27T06:45:09.862170+0000","last_clean":"2024-07-27T06:45:09.862170+0000","last_became_active":"2024-07-27T06:43:12.801266+0000","last_became_peered":"2024-07-27T06:43:12.801266+0000","last_unstale":"2024-07-27T06:45:09.862170+0000","last_undegraded":"2024-07-27T06:45:09.862170+0000","last_fullsized":"2024-07-27T06:45:09.862170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:54:31.934223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.821975+0000","last_change":"2024-07-27T06:43:12.804543+0000","last_active":"2024-07-27T06:45:09.821975+0000","last_peered":"2024-07-27T06:45:09.821975+0000","last_clean":"2024-07-27T06:45:09.821975+0000","last_became_active":"2024-07-27T06:43:12.804173+0000","last_became_peered":"2024-07-27T06:43:12.804173+0000","last_unstale":"2024-07-27T06:45:09.821975+0000","last_undegraded":"2024-07-27T06:45:09.821975+0000","last_fullsized":"2024-07-27T06:45:09.821975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:18:25.337333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483797+0000","last_change":"2024-07-27T06:43:12.798604+0000","last_active":"2024-07-27T06:45:09.483797+0000","last_peered":"2024-07-27T06:45:09.483797+0000","last_clean":"2024-07-27T06:45:09.483797+0000","last_became_active":"2024-07-27T06:43:12.798228+0000","last_became_peered":"2024-07-27T06:43:12.798228+0000","last_unstale":"2024-07-27T06:45:09.483797+0000","last_undegraded":"2024-07-27T06:45:09.483797+0000","last_fullsized":"2024-07-27T06:45:09.483797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:04:35.851399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633341+0000","last_change":"2024-07-27T06:43:12.800452+0000","last_active":"2024-07-27T06:45:09.633341+0000","last_peered":"2024-07-27T06:45:09.633341+0000","last_clean":"2024-07-27T06:45:09.633341+0000","last_became_active":"2024-07-27T06:43:12.800084+0000","last_became_peered":"2024-07-27T06:43:12.800084+0000","last_unstale":"2024-07-27T06:45:09.633341+0000","last_undegraded":"2024-07-27T06:45:09.633341+0000","last_fullsized":"2024-07-27T06:45:09.633341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:54:27.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485773+0000","last_change":"2024-07-27T06:43:12.800761+0000","last_active":"2024-07-27T06:45:09.485773+0000","last_peered":"2024-07-27T06:45:09.485773+0000","last_clean":"2024-07-27T06:45:09.485773+0000","last_became_active":"2024-07-27T06:43:12.800414+0000","last_became_peered":"2024-07-27T06:43:12.800414+0000","last_unstale":"2024-07-27T06:45:09.485773+0000","last_undegraded":"2024-07-27T06:45:09.485773+0000","last_fullsized":"2024-07-27T06:45:09.485773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:02:47.247522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483375+0000","last_change":"2024-07-27T06:43:12.795470+0000","last_active":"2024-07-27T06:45:09.483375+0000","last_peered":"2024-07-27T06:45:09.483375+0000","last_clean":"2024-07-27T06:45:09.483375+0000","last_became_active":"2024-07-27T06:43:12.795136+0000","last_became_peered":"2024-07-27T06:43:12.795136+0000","last_unstale":"2024-07-27T06:45:09.483375+0000","last_undegraded":"2024-07-27T06:45:09.483375+0000","last_fullsized":"2024-07-27T06:45:09.483375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:01:22.178699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481748+0000","last_change":"2024-07-27T06:43:12.803897+0000","last_active":"2024-07-27T06:45:09.481748+0000","last_peered":"2024-07-27T06:45:09.481748+0000","last_clean":"2024-07-27T06:45:09.481748+0000","last_became_active":"2024-07-27T06:43:12.803546+0000","last_became_peered":"2024-07-27T06:43:12.803546+0000","last_unstale":"2024-07-27T06:45:09.481748+0000","last_undegraded":"2024-07-27T06:45:09.481748+0000","last_fullsized":"2024-07-27T06:45:09.481748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:13:50.816831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822201+0000","last_change":"2024-07-27T06:44:09.238514+0000","last_active":"2024-07-27T06:45:09.822201+0000","last_peered":"2024-07-27T06:45:09.822201+0000","last_clean":"2024-07-27T06:45:09.822201+0000","last_became_active":"2024-07-27T06:44:09.237903+0000","last_became_peered":"2024-07-27T06:44:09.237903+0000","last_unstale":"2024-07-27T06:45:09.822201+0000","last_undegraded":"2024-07-27T06:45:09.822201+0000","last_fullsized":"2024-07-27T06:45:09.822201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:20:59.887491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485428+0000","last_change":"2024-07-27T06:43:12.788570+0000","last_active":"2024-07-27T06:45:09.485428+0000","last_peered":"2024-07-27T06:45:09.485428+0000","last_clean":"2024-07-27T06:45:09.485428+0000","last_became_active":"2024-07-27T06:43:12.788199+0000","last_became_peered":"2024-07-27T06:43:12.788199+0000","last_unstale":"2024-07-27T06:45:09.485428+0000","last_undegraded":"2024-07-27T06:45:09.485428+0000","last_fullsized":"2024-07-27T06:45:09.485428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:44:26.751094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862750+0000","last_change":"2024-07-27T06:43:12.792866+0000","last_active":"2024-07-27T06:45:09.862750+0000","last_peered":"2024-07-27T06:45:09.862750+0000","last_clean":"2024-07-27T06:45:09.862750+0000","last_became_active":"2024-07-27T06:43:12.792389+0000","last_became_peered":"2024-07-27T06:43:12.792389+0000","last_unstale":"2024-07-27T06:45:09.862750+0000","last_undegraded":"2024-07-27T06:45:09.862750+0000","last_fullsized":"2024-07-27T06:45:09.862750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:10:37.367642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485830+0000","last_change":"2024-07-27T06:43:12.797002+0000","last_active":"2024-07-27T06:45:09.485830+0000","last_peered":"2024-07-27T06:45:09.485830+0000","last_clean":"2024-07-27T06:45:09.485830+0000","last_became_active":"2024-07-27T06:43:12.796651+0000","last_became_peered":"2024-07-27T06:43:12.796651+0000","last_unstale":"2024-07-27T06:45:09.485830+0000","last_undegraded":"2024-07-27T06:45:09.485830+0000","last_fullsized":"2024-07-27T06:45:09.485830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:28:06.934042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485867+0000","last_change":"2024-07-27T06:43:12.798549+0000","last_active":"2024-07-27T06:45:09.485867+0000","last_peered":"2024-07-27T06:45:09.485867+0000","last_clean":"2024-07-27T06:45:09.485867+0000","last_became_active":"2024-07-27T06:43:12.798201+0000","last_became_peered":"2024-07-27T06:43:12.798201+0000","last_unstale":"2024-07-27T06:45:09.485867+0000","last_undegraded":"2024-07-27T06:45:09.485867+0000","last_fullsized":"2024-07-27T06:45:09.485867+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:59:47.525536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483656+0000","last_change":"2024-07-27T06:43:12.791053+0000","last_active":"2024-07-27T06:45:09.483656+0000","last_peered":"2024-07-27T06:45:09.483656+0000","last_clean":"2024-07-27T06:45:09.483656+0000","last_became_active":"2024-07-27T06:43:12.790718+0000","last_became_peered":"2024-07-27T06:43:12.790718+0000","last_unstale":"2024-07-27T06:45:09.483656+0000","last_undegraded":"2024-07-27T06:45:09.483656+0000","last_fullsized":"2024-07-27T06:45:09.483656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:09.455399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481811+0000","last_change":"2024-07-27T06:43:12.804180+0000","last_active":"2024-07-27T06:45:09.481811+0000","last_peered":"2024-07-27T06:45:09.481811+0000","last_clean":"2024-07-27T06:45:09.481811+0000","last_became_active":"2024-07-27T06:43:12.803881+0000","last_became_peered":"2024-07-27T06:43:12.803881+0000","last_unstale":"2024-07-27T06:45:09.481811+0000","last_undegraded":"2024-07-27T06:45:09.481811+0000","last_fullsized":"2024-07-27T06:45:09.481811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:01:05.009233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862727+0000","last_change":"2024-07-27T06:43:12.809150+0000","last_active":"2024-07-27T06:45:09.862727+0000","last_peered":"2024-07-27T06:45:09.862727+0000","last_clean":"2024-07-27T06:45:09.862727+0000","last_became_active":"2024-07-27T06:43:12.808428+0000","last_became_peered":"2024-07-27T06:43:12.808428+0000","last_unstale":"2024-07-27T06:45:09.862727+0000","last_undegraded":"2024-07-27T06:45:09.862727+0000","last_fullsized":"2024-07-27T06:45:09.862727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:07:13.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486325+0000","last_change":"2024-07-27T06:44:09.233785+0000","last_active":"2024-07-27T06:45:09.486325+0000","last_peered":"2024-07-27T06:45:09.486325+0000","last_clean":"2024-07-27T06:45:09.486325+0000","last_became_active":"2024-07-27T06:44:09.233184+0000","last_became_peered":"2024-07-27T06:44:09.233184+0000","last_unstale":"2024-07-27T06:45:09.486325+0000","last_undegraded":"2024-07-27T06:45:09.486325+0000","last_fullsized":"2024-07-27T06:45:09.486325+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:12:48.556383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486129+0000","last_change":"2024-07-27T06:44:09.320182+0000","last_active":"2024-07-27T06:45:09.486129+0000","last_peered":"2024-07-27T06:45:09.486129+0000","last_clean":"2024-07-27T06:45:09.486129+0000","last_became_active":"2024-07-27T06:44:09.319661+0000","last_became_peered":"2024-07-27T06:44:09.319661+0000","last_unstale":"2024-07-27T06:45:09.486129+0000","last_undegraded":"2024-07-27T06:45:09.486129+0000","last_fullsized":"2024-07-27T06:45:09.486129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:52:11.006680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483342+0000","last_change":"2024-07-27T06:43:12.787866+0000","last_active":"2024-07-27T06:45:09.483342+0000","last_peered":"2024-07-27T06:45:09.483342+0000","last_clean":"2024-07-27T06:45:09.483342+0000","last_became_active":"2024-07-27T06:43:12.787513+0000","last_became_peered":"2024-07-27T06:43:12.787513+0000","last_unstale":"2024-07-27T06:45:09.483342+0000","last_undegraded":"2024-07-27T06:45:09.483342+0000","last_fullsized":"2024-07-27T06:45:09.483342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:48:57.876588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485381+0000","last_change":"2024-07-27T06:43:12.791158+0000","last_active":"2024-07-27T06:45:09.485381+0000","last_peered":"2024-07-27T06:45:09.485381+0000","last_clean":"2024-07-27T06:45:09.485381+0000","last_became_active":"2024-07-27T06:43:12.790771+0000","last_became_peered":"2024-07-27T06:43:12.790771+0000","last_unstale":"2024-07-27T06:45:09.485381+0000","last_undegraded":"2024-07-27T06:45:09.485381+0000","last_fullsized":"2024-07-27T06:45:09.485381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:10:18.739265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'7","reported_seq":244,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:53.201763+0000","last_change":"2024-07-27T06:43:09.769281+0000","last_active":"2024-07-27T06:46:53.201763+0000","last_peered":"2024-07-27T06:46:53.201763+0000","last_clean":"2024-07-27T06:46:53.201763+0000","last_became_active":"2024-07-27T06:43:09.768864+0000","last_became_peered":"2024-07-27T06:43:09.768864+0000","last_unstale":"2024-07-27T06:46:53.201763+0000","last_undegraded":"2024-07-27T06:46:53.201763+0000","last_fullsized":"2024-07-27T06:46:53.201763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:55:14.191180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":205,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862960+0000","last_change":"2024-07-27T06:44:09.319930+0000","last_active":"2024-07-27T06:45:09.862960+0000","last_peered":"2024-07-27T06:45:09.862960+0000","last_clean":"2024-07-27T06:45:09.862960+0000","last_became_active":"2024-07-27T06:44:09.319535+0000","last_became_peered":"2024-07-27T06:44:09.319535+0000","last_unstale":"2024-07-27T06:45:09.862960+0000","last_undegraded":"2024-07-27T06:45:09.862960+0000","last_fullsized":"2024-07-27T06:45:09.862960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:39:52.887860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481674+0000","last_change":"2024-07-27T06:43:09.772227+0000","last_active":"2024-07-27T06:45:09.481674+0000","last_peered":"2024-07-27T06:45:09.481674+0000","last_clean":"2024-07-27T06:45:09.481674+0000","last_became_active":"2024-07-27T06:43:09.771812+0000","last_became_peered":"2024-07-27T06:43:09.771812+0000","last_unstale":"2024-07-27T06:45:09.481674+0000","last_undegraded":"2024-07-27T06:45:09.481674+0000","last_fullsized":"2024-07-27T06:45:09.481674+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:27:28.054590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481030+0000","last_change":"2024-07-27T06:43:09.766490+0000","last_active":"2024-07-27T06:45:09.481030+0000","last_peered":"2024-07-27T06:45:09.481030+0000","last_clean":"2024-07-27T06:45:09.481030+0000","last_became_active":"2024-07-27T06:43:09.765973+0000","last_became_peered":"2024-07-27T06:43:09.765973+0000","last_unstale":"2024-07-27T06:45:09.481030+0000","last_undegraded":"2024-07-27T06:45:09.481030+0000","last_fullsized":"2024-07-27T06:45:09.481030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:00:02.913080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485458+0000","last_change":"2024-07-27T06:43:12.805234+0000","last_active":"2024-07-27T06:45:09.485458+0000","last_peered":"2024-07-27T06:45:09.485458+0000","last_clean":"2024-07-27T06:45:09.485458+0000","last_became_active":"2024-07-27T06:43:12.804908+0000","last_became_peered":"2024-07-27T06:43:12.804908+0000","last_unstale":"2024-07-27T06:45:09.485458+0000","last_undegraded":"2024-07-27T06:45:09.485458+0000","last_fullsized":"2024-07-27T06:45:09.485458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:09:34.988755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862248+0000","last_change":"2024-07-27T06:43:09.774201+0000","last_active":"2024-07-27T06:45:09.862248+0000","last_peered":"2024-07-27T06:45:09.862248+0000","last_clean":"2024-07-27T06:45:09.862248+0000","last_became_active":"2024-07-27T06:43:09.773817+0000","last_became_peered":"2024-07-27T06:43:09.773817+0000","last_unstale":"2024-07-27T06:45:09.862248+0000","last_undegraded":"2024-07-27T06:45:09.862248+0000","last_fullsized":"2024-07-27T06:45:09.862248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:06.166251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633110+0000","last_change":"2024-07-27T06:43:12.800174+0000","last_active":"2024-07-27T06:45:09.633110+0000","last_peered":"2024-07-27T06:45:09.633110+0000","last_clean":"2024-07-27T06:45:09.633110+0000","last_became_active":"2024-07-27T06:43:12.799769+0000","last_became_peered":"2024-07-27T06:43:12.799769+0000","last_unstale":"2024-07-27T06:45:09.633110+0000","last_undegraded":"2024-07-27T06:45:09.633110+0000","last_fullsized":"2024-07-27T06:45:09.633110+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:43:06.890372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":435,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:51.560476+0000","last_change":"2024-07-27T06:43:09.776612+0000","last_active":"2024-07-27T06:46:51.560476+0000","last_peered":"2024-07-27T06:46:51.560476+0000","last_clean":"2024-07-27T06:46:51.560476+0000","last_became_active":"2024-07-27T06:43:09.775761+0000","last_became_peered":"2024-07-27T06:43:09.775761+0000","last_unstale":"2024-07-27T06:46:51.560476+0000","last_undegraded":"2024-07-27T06:46:51.560476+0000","last_fullsized":"2024-07-27T06:46:51.560476+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:42:52.132357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":397,"num_read_kb":397,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822478+0000","last_change":"2024-07-27T06:43:12.797200+0000","last_active":"2024-07-27T06:45:09.822478+0000","last_peered":"2024-07-27T06:45:09.822478+0000","last_clean":"2024-07-27T06:45:09.822478+0000","last_became_active":"2024-07-27T06:43:12.796288+0000","last_became_peered":"2024-07-27T06:43:12.796288+0000","last_unstale":"2024-07-27T06:45:09.822478+0000","last_undegraded":"2024-07-27T06:45:09.822478+0000","last_fullsized":"2024-07-27T06:45:09.822478+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:31:27.603883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488738+0000","last_change":"2024-07-27T06:43:09.772658+0000","last_active":"2024-07-27T06:45:09.488738+0000","last_peered":"2024-07-27T06:45:09.488738+0000","last_clean":"2024-07-27T06:45:09.488738+0000","last_became_active":"2024-07-27T06:43:09.772277+0000","last_became_peered":"2024-07-27T06:43:09.772277+0000","last_unstale":"2024-07-27T06:45:09.488738+0000","last_undegraded":"2024-07-27T06:45:09.488738+0000","last_fullsized":"2024-07-27T06:45:09.488738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:09:16.854718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862551+0000","last_change":"2024-07-27T06:43:12.809643+0000","last_active":"2024-07-27T06:45:09.862551+0000","last_peered":"2024-07-27T06:45:09.862551+0000","last_clean":"2024-07-27T06:45:09.862551+0000","last_became_active":"2024-07-27T06:43:12.809020+0000","last_became_peered":"2024-07-27T06:43:12.809020+0000","last_unstale":"2024-07-27T06:45:09.862551+0000","last_undegraded":"2024-07-27T06:45:09.862551+0000","last_fullsized":"2024-07-27T06:45:09.862551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:08:03.062914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.863022+0000","last_change":"2024-07-27T06:43:10.534042+0000","last_active":"2024-07-27T06:45:09.863022+0000","last_peered":"2024-07-27T06:45:09.863022+0000","last_clean":"2024-07-27T06:45:09.863022+0000","last_became_active":"2024-07-27T06:43:10.533615+0000","last_became_peered":"2024-07-27T06:43:10.533615+0000","last_unstale":"2024-07-27T06:45:09.863022+0000","last_undegraded":"2024-07-27T06:45:09.863022+0000","last_fullsized":"2024-07-27T06:45:09.863022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:09:32.235633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822222+0000","last_change":"2024-07-27T06:43:12.798764+0000","last_active":"2024-07-27T06:45:09.822222+0000","last_peered":"2024-07-27T06:45:09.822222+0000","last_clean":"2024-07-27T06:45:09.822222+0000","last_became_active":"2024-07-27T06:43:12.797500+0000","last_became_peered":"2024-07-27T06:43:12.797500+0000","last_unstale":"2024-07-27T06:45:09.822222+0000","last_undegraded":"2024-07-27T06:45:09.822222+0000","last_fullsized":"2024-07-27T06:45:09.822222+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:12:09.368711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"83'37","reported_seq":479,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:52.199549+0000","last_change":"2024-07-27T06:43:09.775954+0000","last_active":"2024-07-27T06:46:52.199549+0000","last_peered":"2024-07-27T06:46:52.199549+0000","last_clean":"2024-07-27T06:46:52.199549+0000","last_became_active":"2024-07-27T06:43:09.775498+0000","last_became_peered":"2024-07-27T06:43:09.775498+0000","last_unstale":"2024-07-27T06:46:52.199549+0000","last_undegraded":"2024-07-27T06:46:52.199549+0000","last_fullsized":"2024-07-27T06:46:52.199549+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:08:46.965525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66277,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":410,"num_read_kb":270,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486559+0000","last_change":"2024-07-27T06:43:12.797220+0000","last_active":"2024-07-27T06:45:09.486559+0000","last_peered":"2024-07-27T06:45:09.486559+0000","last_clean":"2024-07-27T06:45:09.486559+0000","last_became_active":"2024-07-27T06:43:12.796889+0000","last_became_peered":"2024-07-27T06:43:12.796889+0000","last_unstale":"2024-07-27T06:45:09.486559+0000","last_undegraded":"2024-07-27T06:45:09.486559+0000","last_fullsized":"2024-07-27T06:45:09.486559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:24:36.275738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485662+0000","last_change":"2024-07-27T06:43:09.776600+0000","last_active":"2024-07-27T06:45:09.485662+0000","last_peered":"2024-07-27T06:45:09.485662+0000","last_clean":"2024-07-27T06:45:09.485662+0000","last_became_active":"2024-07-27T06:43:09.776163+0000","last_became_peered":"2024-07-27T06:43:09.776163+0000","last_unstale":"2024-07-27T06:45:09.485662+0000","last_undegraded":"2024-07-27T06:45:09.485662+0000","last_fullsized":"2024-07-27T06:45:09.485662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:22:15.947916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481215+0000","last_change":"2024-07-27T06:43:12.804984+0000","last_active":"2024-07-27T06:45:09.481215+0000","last_peered":"2024-07-27T06:45:09.481215+0000","last_clean":"2024-07-27T06:45:09.481215+0000","last_became_active":"2024-07-27T06:43:12.804644+0000","last_became_peered":"2024-07-27T06:43:12.804644+0000","last_unstale":"2024-07-27T06:45:09.481215+0000","last_undegraded":"2024-07-27T06:45:09.481215+0000","last_fullsized":"2024-07-27T06:45:09.481215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.355981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862246+0000","last_change":"2024-07-27T06:43:09.772489+0000","last_active":"2024-07-27T06:45:09.862246+0000","last_peered":"2024-07-27T06:45:09.862246+0000","last_clean":"2024-07-27T06:45:09.862246+0000","last_became_active":"2024-07-27T06:43:09.772069+0000","last_became_peered":"2024-07-27T06:43:09.772069+0000","last_unstale":"2024-07-27T06:45:09.862246+0000","last_undegraded":"2024-07-27T06:45:09.862246+0000","last_fullsized":"2024-07-27T06:45:09.862246+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:15:30.549971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488589+0000","last_change":"2024-07-27T06:43:12.803376+0000","last_active":"2024-07-27T06:45:09.488589+0000","last_peered":"2024-07-27T06:45:09.488589+0000","last_clean":"2024-07-27T06:45:09.488589+0000","last_became_active":"2024-07-27T06:43:12.802961+0000","last_became_peered":"2024-07-27T06:43:12.802961+0000","last_unstale":"2024-07-27T06:45:09.488589+0000","last_undegraded":"2024-07-27T06:45:09.488589+0000","last_fullsized":"2024-07-27T06:45:09.488589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:00:24.649877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":241,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:54.202267+0000","last_change":"2024-07-27T06:43:09.773182+0000","last_active":"2024-07-27T06:46:54.202267+0000","last_peered":"2024-07-27T06:46:54.202267+0000","last_clean":"2024-07-27T06:46:54.202267+0000","last_became_active":"2024-07-27T06:43:09.772126+0000","last_became_peered":"2024-07-27T06:43:09.772126+0000","last_unstale":"2024-07-27T06:46:54.202267+0000","last_undegraded":"2024-07-27T06:46:54.202267+0000","last_fullsized":"2024-07-27T06:46:54.202267+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:45:26.598799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":205,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488424+0000","last_change":"2024-07-27T06:43:12.799385+0000","last_active":"2024-07-27T06:45:09.488424+0000","last_peered":"2024-07-27T06:45:09.488424+0000","last_clean":"2024-07-27T06:45:09.488424+0000","last_became_active":"2024-07-27T06:43:12.798639+0000","last_became_peered":"2024-07-27T06:43:12.798639+0000","last_unstale":"2024-07-27T06:45:09.488424+0000","last_undegraded":"2024-07-27T06:45:09.488424+0000","last_fullsized":"2024-07-27T06:45:09.488424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:56:29.462491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485303+0000","last_change":"2024-07-27T06:43:10.535429+0000","last_active":"2024-07-27T06:45:09.485303+0000","last_peered":"2024-07-27T06:45:09.485303+0000","last_clean":"2024-07-27T06:45:09.485303+0000","last_became_active":"2024-07-27T06:43:10.534952+0000","last_became_peered":"2024-07-27T06:43:10.534952+0000","last_unstale":"2024-07-27T06:45:09.485303+0000","last_undegraded":"2024-07-27T06:45:09.485303+0000","last_fullsized":"2024-07-27T06:45:09.485303+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:54:20.018263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862706+0000","last_change":"2024-07-27T06:43:12.804266+0000","last_active":"2024-07-27T06:45:09.862706+0000","last_peered":"2024-07-27T06:45:09.862706+0000","last_clean":"2024-07-27T06:45:09.862706+0000","last_became_active":"2024-07-27T06:43:12.803888+0000","last_became_peered":"2024-07-27T06:43:12.803888+0000","last_unstale":"2024-07-27T06:45:09.862706+0000","last_undegraded":"2024-07-27T06:45:09.862706+0000","last_fullsized":"2024-07-27T06:45:09.862706+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:48:00.490592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485827+0000","last_change":"2024-07-27T06:43:09.777656+0000","last_active":"2024-07-27T06:45:09.485827+0000","last_peered":"2024-07-27T06:45:09.485827+0000","last_clean":"2024-07-27T06:45:09.485827+0000","last_became_active":"2024-07-27T06:43:09.776620+0000","last_became_peered":"2024-07-27T06:43:09.776620+0000","last_unstale":"2024-07-27T06:45:09.485827+0000","last_undegraded":"2024-07-27T06:45:09.485827+0000","last_fullsized":"2024-07-27T06:45:09.485827+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:50.203623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483544+0000","last_change":"2024-07-27T06:43:12.796179+0000","last_active":"2024-07-27T06:45:09.483544+0000","last_peered":"2024-07-27T06:45:09.483544+0000","last_clean":"2024-07-27T06:45:09.483544+0000","last_became_active":"2024-07-27T06:43:12.795701+0000","last_became_peered":"2024-07-27T06:43:12.795701+0000","last_unstale":"2024-07-27T06:45:09.483544+0000","last_undegraded":"2024-07-27T06:45:09.483544+0000","last_fullsized":"2024-07-27T06:45:09.483544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:20:12.236769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481627+0000","last_change":"2024-07-27T06:43:09.769949+0000","last_active":"2024-07-27T06:45:09.481627+0000","last_peered":"2024-07-27T06:45:09.481627+0000","last_clean":"2024-07-27T06:45:09.481627+0000","last_became_active":"2024-07-27T06:43:09.769602+0000","last_became_peered":"2024-07-27T06:43:09.769602+0000","last_unstale":"2024-07-27T06:45:09.481627+0000","last_undegraded":"2024-07-27T06:45:09.481627+0000","last_fullsized":"2024-07-27T06:45:09.481627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:28:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488869+0000","last_change":"2024-07-27T06:43:12.802230+0000","last_active":"2024-07-27T06:45:09.488869+0000","last_peered":"2024-07-27T06:45:09.488869+0000","last_clean":"2024-07-27T06:45:09.488869+0000","last_became_active":"2024-07-27T06:43:12.800944+0000","last_became_peered":"2024-07-27T06:43:12.800944+0000","last_unstale":"2024-07-27T06:45:09.488869+0000","last_undegraded":"2024-07-27T06:45:09.488869+0000","last_fullsized":"2024-07-27T06:45:09.488869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:05:54.585955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862270+0000","last_change":"2024-07-27T06:43:09.768239+0000","last_active":"2024-07-27T06:45:09.862270+0000","last_peered":"2024-07-27T06:45:09.862270+0000","last_clean":"2024-07-27T06:45:09.862270+0000","last_became_active":"2024-07-27T06:43:09.767584+0000","last_became_peered":"2024-07-27T06:43:09.767584+0000","last_unstale":"2024-07-27T06:45:09.862270+0000","last_undegraded":"2024-07-27T06:45:09.862270+0000","last_fullsized":"2024-07-27T06:45:09.862270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:35:39.552246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488571+0000","last_change":"2024-07-27T06:43:12.803779+0000","last_active":"2024-07-27T06:45:09.488571+0000","last_peered":"2024-07-27T06:45:09.488571+0000","last_clean":"2024-07-27T06:45:09.488571+0000","last_became_active":"2024-07-27T06:43:12.803359+0000","last_became_peered":"2024-07-27T06:43:12.803359+0000","last_unstale":"2024-07-27T06:45:09.488571+0000","last_undegraded":"2024-07-27T06:45:09.488571+0000","last_fullsized":"2024-07-27T06:45:09.488571+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:32:53.167039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'7","reported_seq":194,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:54.569730+0000","last_change":"2024-07-27T06:44:09.241782+0000","last_active":"2024-07-27T06:46:54.569730+0000","last_peered":"2024-07-27T06:46:54.569730+0000","last_clean":"2024-07-27T06:46:54.569730+0000","last_became_active":"2024-07-27T06:44:09.234692+0000","last_became_peered":"2024-07-27T06:44:09.234692+0000","last_unstale":"2024-07-27T06:46:54.569730+0000","last_undegraded":"2024-07-27T06:46:54.569730+0000","last_fullsized":"2024-07-27T06:46:54.569730+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:45:02.155686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":171,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1561,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488800+0000","last_change":"2024-07-27T06:43:12.805830+0000","last_active":"2024-07-27T06:45:09.488800+0000","last_peered":"2024-07-27T06:45:09.488800+0000","last_clean":"2024-07-27T06:45:09.488800+0000","last_became_active":"2024-07-27T06:43:12.805324+0000","last_became_peered":"2024-07-27T06:43:12.805324+0000","last_unstale":"2024-07-27T06:45:09.488800+0000","last_undegraded":"2024-07-27T06:45:09.488800+0000","last_fullsized":"2024-07-27T06:45:09.488800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:38:31.430676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":109,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.489084+0000","last_change":"2024-07-27T06:41:43.549132+0000","last_active":"2024-07-27T06:45:09.489084+0000","last_peered":"2024-07-27T06:45:09.489084+0000","last_clean":"2024-07-27T06:45:09.489084+0000","last_became_active":"2024-07-27T06:41:43.542635+0000","last_became_peered":"2024-07-27T06:41:43.542635+0000","last_unstale":"2024-07-27T06:45:09.489084+0000","last_undegraded":"2024-07-27T06:45:09.489084+0000","last_fullsized":"2024-07-27T06:45:09.489084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_clean_scrub_stamp":"2024-07-27T06:38:21.639907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:10:06.417387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.861700+0000","last_change":"2024-07-27T06:43:09.768774+0000","last_active":"2024-07-27T06:45:09.861700+0000","last_peered":"2024-07-27T06:45:09.861700+0000","last_clean":"2024-07-27T06:45:09.861700+0000","last_became_active":"2024-07-27T06:43:09.768345+0000","last_became_peered":"2024-07-27T06:43:09.768345+0000","last_unstale":"2024-07-27T06:45:09.861700+0000","last_undegraded":"2024-07-27T06:45:09.861700+0000","last_fullsized":"2024-07-27T06:45:09.861700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:40:49.411746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485670+0000","last_change":"2024-07-27T06:43:12.797817+0000","last_active":"2024-07-27T06:45:09.485670+0000","last_peered":"2024-07-27T06:45:09.485670+0000","last_clean":"2024-07-27T06:45:09.485670+0000","last_became_active":"2024-07-27T06:43:12.797483+0000","last_became_peered":"2024-07-27T06:43:12.797483+0000","last_unstale":"2024-07-27T06:45:09.485670+0000","last_undegraded":"2024-07-27T06:45:09.485670+0000","last_fullsized":"2024-07-27T06:45:09.485670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:24:44.539505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488887+0000","last_change":"2024-07-27T06:43:09.776880+0000","last_active":"2024-07-27T06:45:09.488887+0000","last_peered":"2024-07-27T06:45:09.488887+0000","last_clean":"2024-07-27T06:45:09.488887+0000","last_became_active":"2024-07-27T06:43:09.776486+0000","last_became_peered":"2024-07-27T06:43:09.776486+0000","last_unstale":"2024-07-27T06:45:09.488887+0000","last_undegraded":"2024-07-27T06:45:09.488887+0000","last_fullsized":"2024-07-27T06:45:09.488887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:55:25.436405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.480961+0000","last_change":"2024-07-27T06:43:12.799755+0000","last_active":"2024-07-27T06:45:09.480961+0000","last_peered":"2024-07-27T06:45:09.480961+0000","last_clean":"2024-07-27T06:45:09.480961+0000","last_became_active":"2024-07-27T06:43:12.799412+0000","last_became_peered":"2024-07-27T06:43:12.799412+0000","last_unstale":"2024-07-27T06:45:09.480961+0000","last_undegraded":"2024-07-27T06:45:09.480961+0000","last_fullsized":"2024-07-27T06:45:09.480961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:45:50.545431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":235,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:49.193450+0000","last_change":"2024-07-27T06:43:10.533039+0000","last_active":"2024-07-27T06:46:49.193450+0000","last_peered":"2024-07-27T06:46:49.193450+0000","last_clean":"2024-07-27T06:46:49.193450+0000","last_became_active":"2024-07-27T06:43:10.532583+0000","last_became_peered":"2024-07-27T06:43:10.532583+0000","last_unstale":"2024-07-27T06:46:49.193450+0000","last_undegraded":"2024-07-27T06:46:49.193450+0000","last_fullsized":"2024-07-27T06:46:49.193450+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:04:51.954463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":198,"num_read_kb":198,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633096+0000","last_change":"2024-07-27T06:44:09.238218+0000","last_active":"2024-07-27T06:45:09.633096+0000","last_peered":"2024-07-27T06:45:09.633096+0000","last_clean":"2024-07-27T06:45:09.633096+0000","last_became_active":"2024-07-27T06:44:09.237841+0000","last_became_peered":"2024-07-27T06:44:09.237841+0000","last_unstale":"2024-07-27T06:45:09.633096+0000","last_undegraded":"2024-07-27T06:45:09.633096+0000","last_fullsized":"2024-07-27T06:45:09.633096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:19.101854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"83'8","reported_seq":242,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:50.237432+0000","last_change":"2024-07-27T06:43:09.776214+0000","last_active":"2024-07-27T06:46:50.237432+0000","last_peered":"2024-07-27T06:46:50.237432+0000","last_clean":"2024-07-27T06:46:50.237432+0000","last_became_active":"2024-07-27T06:43:09.775719+0000","last_became_peered":"2024-07-27T06:43:09.775719+0000","last_unstale":"2024-07-27T06:46:50.237432+0000","last_undegraded":"2024-07-27T06:46:50.237432+0000","last_fullsized":"2024-07-27T06:46:50.237432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:57:20.381599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":202,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481735+0000","last_change":"2024-07-27T06:43:12.794909+0000","last_active":"2024-07-27T06:45:09.481735+0000","last_peered":"2024-07-27T06:45:09.481735+0000","last_clean":"2024-07-27T06:45:09.481735+0000","last_became_active":"2024-07-27T06:43:12.794497+0000","last_became_peered":"2024-07-27T06:43:12.794497+0000","last_unstale":"2024-07-27T06:45:09.481735+0000","last_undegraded":"2024-07-27T06:45:09.481735+0000","last_fullsized":"2024-07-27T06:45:09.481735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:42:49.760126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633255+0000","last_change":"2024-07-27T06:43:10.534240+0000","last_active":"2024-07-27T06:45:09.633255+0000","last_peered":"2024-07-27T06:45:09.633255+0000","last_clean":"2024-07-27T06:45:09.633255+0000","last_became_active":"2024-07-27T06:43:10.533596+0000","last_became_peered":"2024-07-27T06:43:10.533596+0000","last_unstale":"2024-07-27T06:45:09.633255+0000","last_undegraded":"2024-07-27T06:45:09.633255+0000","last_fullsized":"2024-07-27T06:45:09.633255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:16.098839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481991+0000","last_change":"2024-07-27T06:43:12.805248+0000","last_active":"2024-07-27T06:45:09.481991+0000","last_peered":"2024-07-27T06:45:09.481991+0000","last_clean":"2024-07-27T06:45:09.481991+0000","last_became_active":"2024-07-27T06:43:12.803693+0000","last_became_peered":"2024-07-27T06:43:12.803693+0000","last_unstale":"2024-07-27T06:45:09.481991+0000","last_undegraded":"2024-07-27T06:45:09.481991+0000","last_fullsized":"2024-07-27T06:45:09.481991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:17.466465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486692+0000","last_change":"2024-07-27T06:43:10.533701+0000","last_active":"2024-07-27T06:45:09.486692+0000","last_peered":"2024-07-27T06:45:09.486692+0000","last_clean":"2024-07-27T06:45:09.486692+0000","last_became_active":"2024-07-27T06:43:10.533109+0000","last_became_peered":"2024-07-27T06:43:10.533109+0000","last_unstale":"2024-07-27T06:45:09.486692+0000","last_undegraded":"2024-07-27T06:45:09.486692+0000","last_fullsized":"2024-07-27T06:45:09.486692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:32:52.906691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481918+0000","last_change":"2024-07-27T06:43:12.798479+0000","last_active":"2024-07-27T06:45:09.481918+0000","last_peered":"2024-07-27T06:45:09.481918+0000","last_clean":"2024-07-27T06:45:09.481918+0000","last_became_active":"2024-07-27T06:43:12.798124+0000","last_became_peered":"2024-07-27T06:43:12.798124+0000","last_unstale":"2024-07-27T06:45:09.481918+0000","last_undegraded":"2024-07-27T06:45:09.481918+0000","last_fullsized":"2024-07-27T06:45:09.481918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:14:55.267213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"83'17","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:22.215894+0000","last_change":"2024-07-27T06:43:09.770122+0000","last_active":"2024-07-27T06:45:22.215894+0000","last_peered":"2024-07-27T06:45:22.215894+0000","last_clean":"2024-07-27T06:45:22.215894+0000","last_became_active":"2024-07-27T06:43:09.769784+0000","last_became_peered":"2024-07-27T06:43:09.769784+0000","last_unstale":"2024-07-27T06:45:22.215894+0000","last_undegraded":"2024-07-27T06:45:22.215894+0000","last_fullsized":"2024-07-27T06:45:22.215894+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:47:00.398135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485891+0000","last_change":"2024-07-27T06:43:12.799603+0000","last_active":"2024-07-27T06:45:09.485891+0000","last_peered":"2024-07-27T06:45:09.485891+0000","last_clean":"2024-07-27T06:45:09.485891+0000","last_became_active":"2024-07-27T06:43:12.799189+0000","last_became_peered":"2024-07-27T06:43:12.799189+0000","last_unstale":"2024-07-27T06:45:09.485891+0000","last_undegraded":"2024-07-27T06:45:09.485891+0000","last_fullsized":"2024-07-27T06:45:09.485891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:32:30.603258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862220+0000","last_change":"2024-07-27T06:43:10.004726+0000","last_active":"2024-07-27T06:45:09.862220+0000","last_peered":"2024-07-27T06:45:09.862220+0000","last_clean":"2024-07-27T06:45:09.862220+0000","last_became_active":"2024-07-27T06:43:10.004367+0000","last_became_peered":"2024-07-27T06:43:10.004367+0000","last_unstale":"2024-07-27T06:45:09.862220+0000","last_undegraded":"2024-07-27T06:45:09.862220+0000","last_fullsized":"2024-07-27T06:45:09.862220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:22:39.942391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862427+0000","last_change":"2024-07-27T06:43:12.796511+0000","last_active":"2024-07-27T06:45:09.862427+0000","last_peered":"2024-07-27T06:45:09.862427+0000","last_clean":"2024-07-27T06:45:09.862427+0000","last_became_active":"2024-07-27T06:43:12.796152+0000","last_became_peered":"2024-07-27T06:43:12.796152+0000","last_unstale":"2024-07-27T06:45:09.862427+0000","last_undegraded":"2024-07-27T06:45:09.862427+0000","last_fullsized":"2024-07-27T06:45:09.862427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:03:02.508990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822077+0000","last_change":"2024-07-27T06:43:10.533656+0000","last_active":"2024-07-27T06:45:09.822077+0000","last_peered":"2024-07-27T06:45:09.822077+0000","last_clean":"2024-07-27T06:45:09.822077+0000","last_became_active":"2024-07-27T06:43:10.533295+0000","last_became_peered":"2024-07-27T06:43:10.533295+0000","last_unstale":"2024-07-27T06:45:09.822077+0000","last_undegraded":"2024-07-27T06:45:09.822077+0000","last_fullsized":"2024-07-27T06:45:09.822077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:11:00.032511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485237+0000","last_change":"2024-07-27T06:43:12.795106+0000","last_active":"2024-07-27T06:45:09.485237+0000","last_peered":"2024-07-27T06:45:09.485237+0000","last_clean":"2024-07-27T06:45:09.485237+0000","last_became_active":"2024-07-27T06:43:12.794729+0000","last_became_peered":"2024-07-27T06:43:12.794729+0000","last_unstale":"2024-07-27T06:45:09.485237+0000","last_undegraded":"2024-07-27T06:45:09.485237+0000","last_fullsized":"2024-07-27T06:45:09.485237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:19:16.542363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485673+0000","last_change":"2024-07-27T06:43:09.770016+0000","last_active":"2024-07-27T06:45:09.485673+0000","last_peered":"2024-07-27T06:45:09.485673+0000","last_clean":"2024-07-27T06:45:09.485673+0000","last_became_active":"2024-07-27T06:43:09.769672+0000","last_became_peered":"2024-07-27T06:43:09.769672+0000","last_unstale":"2024-07-27T06:45:09.485673+0000","last_undegraded":"2024-07-27T06:45:09.485673+0000","last_fullsized":"2024-07-27T06:45:09.485673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:30:55.737494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483659+0000","last_change":"2024-07-27T06:43:12.793690+0000","last_active":"2024-07-27T06:45:09.483659+0000","last_peered":"2024-07-27T06:45:09.483659+0000","last_clean":"2024-07-27T06:45:09.483659+0000","last_became_active":"2024-07-27T06:43:12.793352+0000","last_became_peered":"2024-07-27T06:43:12.793352+0000","last_unstale":"2024-07-27T06:45:09.483659+0000","last_undegraded":"2024-07-27T06:45:09.483659+0000","last_fullsized":"2024-07-27T06:45:09.483659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:22:47.606990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.491196+0000","last_change":"2024-07-27T06:45:09.490738+0000","last_active":"2024-07-27T06:45:09.491196+0000","last_peered":"2024-07-27T06:45:09.491196+0000","last_clean":"2024-07-27T06:45:09.491196+0000","last_became_active":"2024-07-27T06:45:09.487147+0000","last_became_peered":"2024-07-27T06:45:09.487147+0000","last_unstale":"2024-07-27T06:45:09.491196+0000","last_undegraded":"2024-07-27T06:45:09.491196+0000","last_fullsized":"2024-07-27T06:45:09.491196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:46.732987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485570+0000","last_change":"2024-07-27T06:43:12.792648+0000","last_active":"2024-07-27T06:45:09.485570+0000","last_peered":"2024-07-27T06:45:09.485570+0000","last_clean":"2024-07-27T06:45:09.485570+0000","last_became_active":"2024-07-27T06:43:12.792338+0000","last_became_peered":"2024-07-27T06:43:12.792338+0000","last_unstale":"2024-07-27T06:45:09.485570+0000","last_undegraded":"2024-07-27T06:45:09.485570+0000","last_fullsized":"2024-07-27T06:45:09.485570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:36:52.605957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.827632+0000","last_change":"2024-07-27T06:45:09.827140+0000","last_active":"2024-07-27T06:45:09.827632+0000","last_peered":"2024-07-27T06:45:09.827632+0000","last_clean":"2024-07-27T06:45:09.827632+0000","last_became_active":"2024-07-27T06:45:09.823119+0000","last_became_peered":"2024-07-27T06:45:09.823119+0000","last_unstale":"2024-07-27T06:45:09.827632+0000","last_undegraded":"2024-07-27T06:45:09.827632+0000","last_fullsized":"2024-07-27T06:45:09.827632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:32:19.682193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488730+0000","last_change":"2024-07-27T06:43:12.809347+0000","last_active":"2024-07-27T06:45:09.488730+0000","last_peered":"2024-07-27T06:45:09.488730+0000","last_clean":"2024-07-27T06:45:09.488730+0000","last_became_active":"2024-07-27T06:43:12.808980+0000","last_became_peered":"2024-07-27T06:43:12.808980+0000","last_unstale":"2024-07-27T06:45:09.488730+0000","last_undegraded":"2024-07-27T06:45:09.488730+0000","last_fullsized":"2024-07-27T06:45:09.488730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:05:14.132641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633284+0000","last_change":"2024-07-27T06:44:09.237908+0000","last_active":"2024-07-27T06:45:09.633284+0000","last_peered":"2024-07-27T06:45:09.633284+0000","last_clean":"2024-07-27T06:45:09.633284+0000","last_became_active":"2024-07-27T06:44:09.237503+0000","last_became_peered":"2024-07-27T06:44:09.237503+0000","last_unstale":"2024-07-27T06:45:09.633284+0000","last_undegraded":"2024-07-27T06:45:09.633284+0000","last_fullsized":"2024-07-27T06:45:09.633284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:29.654244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486391+0000","last_change":"2024-07-27T06:43:09.776000+0000","last_active":"2024-07-27T06:45:09.486391+0000","last_peered":"2024-07-27T06:45:09.486391+0000","last_clean":"2024-07-27T06:45:09.486391+0000","last_became_active":"2024-07-27T06:43:09.775522+0000","last_became_peered":"2024-07-27T06:43:09.775522+0000","last_unstale":"2024-07-27T06:45:09.486391+0000","last_undegraded":"2024-07-27T06:45:09.486391+0000","last_fullsized":"2024-07-27T06:45:09.486391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:39:42.650177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822084+0000","last_change":"2024-07-27T06:43:12.807062+0000","last_active":"2024-07-27T06:45:09.822084+0000","last_peered":"2024-07-27T06:45:09.822084+0000","last_clean":"2024-07-27T06:45:09.822084+0000","last_became_active":"2024-07-27T06:43:12.805816+0000","last_became_peered":"2024-07-27T06:43:12.805816+0000","last_unstale":"2024-07-27T06:45:09.822084+0000","last_undegraded":"2024-07-27T06:45:09.822084+0000","last_fullsized":"2024-07-27T06:45:09.822084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:44:16.261663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483465+0000","last_change":"2024-07-27T06:44:09.250070+0000","last_active":"2024-07-27T06:45:09.483465+0000","last_peered":"2024-07-27T06:45:09.483465+0000","last_clean":"2024-07-27T06:45:09.483465+0000","last_became_active":"2024-07-27T06:44:09.237290+0000","last_became_peered":"2024-07-27T06:44:09.237290+0000","last_unstale":"2024-07-27T06:45:09.483465+0000","last_undegraded":"2024-07-27T06:45:09.483465+0000","last_fullsized":"2024-07-27T06:45:09.483465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:51.590487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862423+0000","last_change":"2024-07-27T06:43:12.809135+0000","last_active":"2024-07-27T06:45:09.862423+0000","last_peered":"2024-07-27T06:45:09.862423+0000","last_clean":"2024-07-27T06:45:09.862423+0000","last_became_active":"2024-07-27T06:43:12.806731+0000","last_became_peered":"2024-07-27T06:43:12.806731+0000","last_unstale":"2024-07-27T06:45:09.862423+0000","last_undegraded":"2024-07-27T06:45:09.862423+0000","last_fullsized":"2024-07-27T06:45:09.862423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:47:50.397517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486050+0000","last_change":"2024-07-27T06:43:10.533608+0000","last_active":"2024-07-27T06:45:09.486050+0000","last_peered":"2024-07-27T06:45:09.486050+0000","last_clean":"2024-07-27T06:45:09.486050+0000","last_became_active":"2024-07-27T06:43:10.532663+0000","last_became_peered":"2024-07-27T06:43:10.532663+0000","last_unstale":"2024-07-27T06:45:09.486050+0000","last_undegraded":"2024-07-27T06:45:09.486050+0000","last_fullsized":"2024-07-27T06:45:09.486050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:05:33.155427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485883+0000","last_change":"2024-07-27T06:43:12.789890+0000","last_active":"2024-07-27T06:45:09.485883+0000","last_peered":"2024-07-27T06:45:09.485883+0000","last_clean":"2024-07-27T06:45:09.485883+0000","last_became_active":"2024-07-27T06:43:12.789561+0000","last_became_peered":"2024-07-27T06:43:12.789561+0000","last_unstale":"2024-07-27T06:45:09.485883+0000","last_undegraded":"2024-07-27T06:45:09.485883+0000","last_fullsized":"2024-07-27T06:45:09.485883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:35:49.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"83'15","reported_seq":246,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:50.236769+0000","last_change":"2024-07-27T06:43:09.773471+0000","last_active":"2024-07-27T06:46:50.236769+0000","last_peered":"2024-07-27T06:46:50.236769+0000","last_clean":"2024-07-27T06:46:50.236769+0000","last_became_active":"2024-07-27T06:43:09.773213+0000","last_became_peered":"2024-07-27T06:43:09.773213+0000","last_unstale":"2024-07-27T06:46:50.236769+0000","last_undegraded":"2024-07-27T06:46:50.236769+0000","last_fullsized":"2024-07-27T06:46:50.236769+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:54:38.770198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":199,"num_read_kb":199,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74603,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2005,"num_read_kb":1096,"num_write":264,"num_write_kb":157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1561,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":218436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283116,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678076,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":461919,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874304,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1113320,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037831,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":461317,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234060,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":461695,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430287,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":1055194,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593813,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":461874,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790043,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":1054032,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953568,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":929792,"data_stored":462597,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":460859,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378730,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":522433,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542255,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":529064,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705780,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":462905,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":68013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T06:46:55.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:46:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:55 smithi077 ceph-mon[37153]: pgmap v394: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-27T06:46:56.041 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:56 smithi029 ceph-mon[32610]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:46:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:56 smithi029 ceph-mon[32610]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:46:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:56 smithi077 ceph-mon[37153]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:46:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:56 smithi077 ceph-mon[37153]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:46:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:56 smithi047 ceph-mon[36835]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:46:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:56 smithi047 ceph-mon[36835]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:46:57.704 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:57 smithi029 ceph-mon[32610]: pgmap v395: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-27T06:46:57.704 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:57 smithi029 ceph-mon[32610]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:46:57.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.736+0000 7f5fc9a96640 1 -- 172.21.15.29:0/3459495903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc4103590 msgr2=0x7f5fc41039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.736+0000 7f5fc9a96640 1 --2- 172.21.15.29:0/3459495903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc4103590 0x7f5fc41039a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4009900 tx=0x7f5fb402a410 comp rx=0 tx=0).stop 2024-07-27T06:46:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.736+0000 7f5fc9a96640 1 -- 172.21.15.29:0/3459495903 shutdown_connections 2024-07-27T06:46:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.736+0000 7f5fc9a96640 1 --2- 172.21.15.29:0/3459495903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fc4109820 0x7f5fc4110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.736+0000 7f5fc9a96640 1 --2- 172.21.15.29:0/3459495903 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fc4108ed0 0x7f5fc41092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.736+0000 7f5fc9a96640 1 --2- 172.21.15.29:0/3459495903 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc4103590 0x7f5fc41039a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.736+0000 7f5fc9a96640 1 -- 172.21.15.29:0/3459495903 >> 172.21.15.29:0/3459495903 conn(0x7f5fc407b280 msgr2=0x7f5fc407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.736+0000 7f5fc9a96640 1 -- 172.21.15.29:0/3459495903 shutdown_connections 2024-07-27T06:46:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.736+0000 7f5fc9a96640 1 -- 172.21.15.29:0/3459495903 wait complete. 2024-07-27T06:46:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.736+0000 7f5fc9a96640 1 Processor -- start 2024-07-27T06:46:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc9a96640 1 -- start start 2024-07-27T06:46:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc9a96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fc4103590 0x7f5fc4199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc9a96640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc4108ed0 0x7f5fc419a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc9a96640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fc4109820 0x7f5fc419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc9a96640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc41134f0 con 0x7f5fc4108ed0 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc9a96640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc4113370 con 0x7f5fc4109820 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc9a96640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc4113670 con 0x7f5fc4103590 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc4108ed0 0x7f5fc419a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc4108ed0 0x7f5fc419a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53366/0 (socket says 172.21.15.29:53366) 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc2ffd640 1 -- 172.21.15.29:0/2329230743 learned_addr learned my addr 172.21.15.29:0/2329230743 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc3fff640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fc4109820 0x7f5fc419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc37fe640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fc4103590 0x7f5fc4199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc2ffd640 1 -- 172.21.15.29:0/2329230743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fc4103590 msgr2=0x7f5fc4199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc2ffd640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fc4103590 0x7f5fc4199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:57.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc2ffd640 1 -- 172.21.15.29:0/2329230743 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fc4109820 msgr2=0x7f5fc419f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc2ffd640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fc4109820 0x7f5fc419f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc2ffd640 1 -- 172.21.15.29:0/2329230743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc419fe50 con 0x7f5fc4108ed0 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.737+0000 7f5fc37fe640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fc4103590 0x7f5fc4199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.738+0000 7f5fc2ffd640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc4108ed0 0x7f5fc419a540 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00c970 tx=0x7f5fac00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.738+0000 7f5fc0ff9640 1 -- 172.21.15.29:0/2329230743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac004220 con 0x7f5fc4108ed0 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.738+0000 7f5fc9a96640 1 -- 172.21.15.29:0/2329230743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc41a0140 con 0x7f5fc4108ed0 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.738+0000 7f5fc0ff9640 1 -- 172.21.15.29:0/2329230743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fac0043c0 con 0x7f5fc4108ed0 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.738+0000 7f5fc0ff9640 1 -- 172.21.15.29:0/2329230743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac004a00 con 0x7f5fc4108ed0 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.738+0000 7f5fc9a96640 1 -- 172.21.15.29:0/2329230743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc41a6960 con 0x7f5fc4108ed0 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.739+0000 7f5fc9a96640 1 -- 172.21.15.29:0/2329230743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fc41039a0 con 0x7f5fc4108ed0 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.741+0000 7f5fc0ff9640 1 -- 172.21.15.29:0/2329230743 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f5fac0079b0 con 0x7f5fc4108ed0 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.741+0000 7f5fc0ff9640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5fb00b3c50 0x7f5fb00b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:46:57.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.742+0000 7f5fc0ff9640 1 -- 172.21.15.29:0/2329230743 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f5fac0e3e70 con 0x7f5fc4108ed0 2024-07-27T06:46:57.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.742+0000 7f5fc37fe640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5fb00b3c50 0x7f5fb00b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:46:57.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.742+0000 7f5fc37fe640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5fb00b3c50 0x7f5fb00b6160 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4031440 tx=0x7f5fb4005d50 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:46:57.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.745+0000 7f5fc0ff9640 1 -- 172.21.15.29:0/2329230743 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5fac014030 con 0x7f5fc4108ed0 2024-07-27T06:46:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:57 smithi077 ceph-mon[37153]: pgmap v395: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-27T06:46:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:57 smithi077 ceph-mon[37153]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:46:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:57 smithi047 ceph-mon[36835]: pgmap v395: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-27T06:46:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:57 smithi047 ceph-mon[36835]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:46:57.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.964+0000 7f5fc9a96640 1 -- 172.21.15.29:0/2329230743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fc410b600 con 0x7f5fc4108ed0 2024-07-27T06:46:57.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.965+0000 7f5fc0ff9640 1 -- 172.21.15.29:0/2329230743 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11957 (secure 0 0 0) 0x7f5fac0cd070 con 0x7f5fc4108ed0 2024-07-27T06:46:57.966 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:46:57.966 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-07-27T06:46:50:421100+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:46:49.561737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24670,24658,24646,24617,24613]},"id":1}]} 2024-07-27T06:46:57.967 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-07-27T06:46:57.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.967+0000 7f5fc9a96640 1 -- 172.21.15.29:0/2329230743 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5fb00b3c50 msgr2=0x7f5fb00b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:57.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.967+0000 7f5fc9a96640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5fb00b3c50 0x7f5fb00b6160 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4031440 tx=0x7f5fb4005d50 comp rx=0 tx=0).stop 2024-07-27T06:46:57.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.967+0000 7f5fc9a96640 1 -- 172.21.15.29:0/2329230743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc4108ed0 msgr2=0x7f5fc419a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:46:57.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.967+0000 7f5fc9a96640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc4108ed0 0x7f5fc419a540 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00c970 tx=0x7f5fac00ce30 comp rx=0 tx=0).stop 2024-07-27T06:46:57.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.967+0000 7f5fc9a96640 1 -- 172.21.15.29:0/2329230743 shutdown_connections 2024-07-27T06:46:57.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.967+0000 7f5fc9a96640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5fc4109820 0x7f5fc419f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:57.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.967+0000 7f5fc9a96640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fc4108ed0 0x7f5fc419a540 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.967+0000 7f5fc9a96640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5fb00b3c50 0x7f5fb00b6160 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.967+0000 7f5fc9a96640 1 --2- 172.21.15.29:0/2329230743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fc4103590 0x7f5fc4199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:46:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.967+0000 7f5fc9a96640 1 -- 172.21.15.29:0/2329230743 >> 172.21.15.29:0/2329230743 conn(0x7f5fc407b280 msgr2=0x7f5fc4107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:46:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.968+0000 7f5fc9a96640 1 -- 172.21.15.29:0/2329230743 shutdown_connections 2024-07-27T06:46:57.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:46:57.968+0000 7f5fc9a96640 1 -- 172.21.15.29:0/2329230743 wait complete. 2024-07-27T06:46:58.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:58 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2329230743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:46:58.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd dump --format=json 2024-07-27T06:46:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:58 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2329230743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:46:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:58 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2329230743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:46:58.969 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:46:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:46:59 smithi029 ceph-mon[32610]: pgmap v396: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-27T06:46:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:46:59 smithi077 ceph-mon[37153]: pgmap v396: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-27T06:46:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:46:59 smithi047 ceph-mon[36835]: pgmap v396: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-27T06:47:00.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.751+0000 7f3dd4dce640 1 -- 172.21.15.29:0/512976331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd01024b0 msgr2=0x7f3dd01028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:00.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.751+0000 7f3dd4dce640 1 --2- 172.21.15.29:0/512976331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd01024b0 0x7f3dd01028c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0009900 tx=0x7f3dc002a3f0 comp rx=0 tx=0).stop 2024-07-27T06:47:00.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.751+0000 7f3dd4dce640 1 -- 172.21.15.29:0/512976331 shutdown_connections 2024-07-27T06:47:00.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.751+0000 7f3dd4dce640 1 --2- 172.21.15.29:0/512976331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3dd0108740 0x7f3dd010ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:00.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.751+0000 7f3dd4dce640 1 --2- 172.21.15.29:0/512976331 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3dd0107df0 0x7f3dd0108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.751+0000 7f3dd4dce640 1 --2- 172.21.15.29:0/512976331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd01024b0 0x7f3dd01028c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.751+0000 7f3dd4dce640 1 -- 172.21.15.29:0/512976331 >> 172.21.15.29:0/512976331 conn(0x7f3dd0079170 msgr2=0x7f3dd0079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.751+0000 7f3dd4dce640 1 -- 172.21.15.29:0/512976331 shutdown_connections 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.751+0000 7f3dd4dce640 1 -- 172.21.15.29:0/512976331 wait complete. 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.752+0000 7f3dd4dce640 1 Processor -- start 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.752+0000 7f3dd4dce640 1 -- start start 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dd4dce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3dd01024b0 0x7f3dd0199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dd4dce640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3dd0107df0 0x7f3dd019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dd4dce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd0108740 0x7f3dd019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dd4dce640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd01123a0 con 0x7f3dd0108740 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dd4dce640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd0112220 con 0x7f3dd0107df0 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dd4dce640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd0112520 con 0x7f3dd01024b0 2024-07-27T06:47:00.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd0108740 0x7f3dd019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd0108740 0x7f3dd019f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36574/0 (socket says 172.21.15.29:36574) 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dced9d640 1 -- 172.21.15.29:0/232090788 learned_addr learned my addr 172.21.15.29:0/232090788 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dce59c640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3dd01024b0 0x7f3dd0199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dc5d9b640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3dd0107df0 0x7f3dd019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dced9d640 1 -- 172.21.15.29:0/232090788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3dd01024b0 msgr2=0x7f3dd0199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dced9d640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3dd01024b0 0x7f3dd0199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dced9d640 1 -- 172.21.15.29:0/232090788 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3dd0107df0 msgr2=0x7f3dd019a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dced9d640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3dd0107df0 0x7f3dd019a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dced9d640 1 -- 172.21.15.29:0/232090788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd019fdf0 con 0x7f3dd0108740 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dc5d9b640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3dd0107df0 0x7f3dd019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dce59c640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3dd01024b0 0x7f3dd0199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.753+0000 7f3dced9d640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd0108740 0x7f3dd019f780 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009fd0 tx=0x7f3dc800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.754+0000 7f3dc7fff640 1 -- 172.21.15.29:0/232090788 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8017070 con 0x7f3dd0108740 2024-07-27T06:47:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.754+0000 7f3dd4dce640 1 -- 172.21.15.29:0/232090788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dd01a00e0 con 0x7f3dd0108740 2024-07-27T06:47:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.754+0000 7f3dc7fff640 1 -- 172.21.15.29:0/232090788 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dc8003e20 con 0x7f3dd0108740 2024-07-27T06:47:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.754+0000 7f3dd4dce640 1 -- 172.21.15.29:0/232090788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dd01a6900 con 0x7f3dd0108740 2024-07-27T06:47:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.754+0000 7f3dc7fff640 1 -- 172.21.15.29:0/232090788 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8010460 con 0x7f3dd0108740 2024-07-27T06:47:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.755+0000 7f3dd4dce640 1 -- 172.21.15.29:0/232090788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dd0103730 con 0x7f3dd0108740 2024-07-27T06:47:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.757+0000 7f3dc7fff640 1 -- 172.21.15.29:0/232090788 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f3dc8003970 con 0x7f3dd0108740 2024-07-27T06:47:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.757+0000 7f3dc7fff640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3dbc0b3c50 0x7f3dbc0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.758+0000 7f3dce59c640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3dbc0b3c50 0x7f3dbc0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.758+0000 7f3dc7fff640 1 -- 172.21.15.29:0/232090788 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f3dc80e3f10 con 0x7f3dd0108740 2024-07-27T06:47:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.758+0000 7f3dce59c640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3dbc0b3c50 0x7f3dbc0b6160 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0031440 tx=0x7f3dc0009680 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:00.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.760+0000 7f3dc7fff640 1 -- 172.21.15.29:0/232090788 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3dc80ab450 con 0x7f3dd0108740 2024-07-27T06:47:00.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.945+0000 7f3dd4dce640 1 -- 172.21.15.29:0/232090788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3dd019b070 con 0x7f3dd0108740 2024-07-27T06:47:00.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.947+0000 7f3dc7fff640 1 -- 172.21.15.29:0/232090788 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+23026 (secure 0 0 0) 0x7f3dc80aad50 con 0x7f3dd0108740 2024-07-27T06:47:00.948 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:47:00.948 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","created":"2024-07-27T06:33:57.535327+0000","modified":"2024-07-27T06:45:09.477077+0000","last_up_change":"2024-07-27T06:42:05.757156+0000","last_in_change":"2024-07-27T06:41:46.423732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T06:38:21.606527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-27T06:43:07.755490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-27T06:43:11.658694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06833d24-2e1a-4199-9338-7d6883d1bf9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6803","nonce":386121641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6805","nonce":386121641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6809","nonce":386121641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6807","nonce":386121641}]},"public_addr":"172.21.15.29:6803/386121641","cluster_addr":"172.21.15.29:6805/386121641","heartbeat_back_addr":"172.21.15.29:6809/386121641","heartbeat_front_addr":"172.21.15.29:6807/386121641","state":["exists","up"]},{"osd":1,"uuid":"906aba19-80dc-41b5-953c-af6dca604b05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6801","nonce":2795953749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6803","nonce":2795953749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6807","nonce":2795953749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6805","nonce":2795953749}]},"public_addr":"172.21.15.47:6801/2795953749","cluster_addr":"172.21.15.47:6803/2795953749","heartbeat_back_addr":"172.21.15.47:6807/2795953749","heartbeat_front_addr":"172.21.15.47:6805/2795953749","state":["exists","up"]},{"osd":2,"uuid":"194ae5a5-aebf-4cff-9b99-585a3073a269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6801","nonce":2202565078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6803","nonce":2202565078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6807","nonce":2202565078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6805","nonce":2202565078}]},"public_addr":"172.21.15.77:6801/2202565078","cluster_addr":"172.21.15.77:6803/2202565078","heartbeat_back_addr":"172.21.15.77:6807/2202565078","heartbeat_front_addr":"172.21.15.77:6805/2202565078","state":["exists","up"]},{"osd":3,"uuid":"3c22ce9c-963e-4f72-86e5-360e781526db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6811","nonce":3757476312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6813","nonce":3757476312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6817","nonce":3757476312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6815","nonce":3757476312}]},"public_addr":"172.21.15.29:6811/3757476312","cluster_addr":"172.21.15.29:6813/3757476312","heartbeat_back_addr":"172.21.15.29:6817/3757476312","heartbeat_front_addr":"172.21.15.29:6815/3757476312","state":["exists","up"]},{"osd":4,"uuid":"cd4b585f-e7a0-4d82-873d-3a9269ebac3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6809","nonce":1379363720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6811","nonce":1379363720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6815","nonce":1379363720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6813","nonce":1379363720}]},"public_addr":"172.21.15.47:6809/1379363720","cluster_addr":"172.21.15.47:6811/1379363720","heartbeat_back_addr":"172.21.15.47:6815/1379363720","heartbeat_front_addr":"172.21.15.47:6813/1379363720","state":["exists","up"]},{"osd":5,"uuid":"40a3eb14-dfaa-4523-8c28-d74c19293c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6809","nonce":1338438780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6811","nonce":1338438780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6815","nonce":1338438780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6813","nonce":1338438780}]},"public_addr":"172.21.15.77:6809/1338438780","cluster_addr":"172.21.15.77:6811/1338438780","heartbeat_back_addr":"172.21.15.77:6815/1338438780","heartbeat_front_addr":"172.21.15.77:6813/1338438780","state":["exists","up"]},{"osd":6,"uuid":"91e85be2-10d1-4209-aa33-17da8f8bd8bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6819","nonce":2679402148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6821","nonce":2679402148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6825","nonce":2679402148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6823","nonce":2679402148}]},"public_addr":"172.21.15.29:6819/2679402148","cluster_addr":"172.21.15.29:6821/2679402148","heartbeat_back_addr":"172.21.15.29:6825/2679402148","heartbeat_front_addr":"172.21.15.29:6823/2679402148","state":["exists","up"]},{"osd":7,"uuid":"c655b450-0cc4-45d4-95ea-93a57fe8e4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6817","nonce":2439195589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6819","nonce":2439195589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6823","nonce":2439195589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6821","nonce":2439195589}]},"public_addr":"172.21.15.47:6817/2439195589","cluster_addr":"172.21.15.47:6819/2439195589","heartbeat_back_addr":"172.21.15.47:6823/2439195589","heartbeat_front_addr":"172.21.15.47:6821/2439195589","state":["exists","up"]},{"osd":8,"uuid":"a7a95b51-f32c-433e-a197-feb6d47b6c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6817","nonce":1422290905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6819","nonce":1422290905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6823","nonce":1422290905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6821","nonce":1422290905}]},"public_addr":"172.21.15.77:6817/1422290905","cluster_addr":"172.21.15.77:6819/1422290905","heartbeat_back_addr":"172.21.15.77:6823/1422290905","heartbeat_front_addr":"172.21.15.77:6821/1422290905","state":["exists","up"]},{"osd":9,"uuid":"f352242b-0d2e-41b3-9c83-436cf432b0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6827","nonce":1307316384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6829","nonce":1307316384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6833","nonce":1307316384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6831","nonce":1307316384}]},"public_addr":"172.21.15.29:6827/1307316384","cluster_addr":"172.21.15.29:6829/1307316384","heartbeat_back_addr":"172.21.15.29:6833/1307316384","heartbeat_front_addr":"172.21.15.29:6831/1307316384","state":["exists","up"]},{"osd":10,"uuid":"4148c2df-c3e9-4a47-88f7-36cf3eab8d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6825","nonce":2133726850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6827","nonce":2133726850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6831","nonce":2133726850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6829","nonce":2133726850}]},"public_addr":"172.21.15.47:6825/2133726850","cluster_addr":"172.21.15.47:6827/2133726850","heartbeat_back_addr":"172.21.15.47:6831/2133726850","heartbeat_front_addr":"172.21.15.47:6829/2133726850","state":["exists","up"]},{"osd":11,"uuid":"5e8b1da3-5f65-4fe9-8722-b35921cbd8c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6825","nonce":3228299327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6827","nonce":3228299327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6831","nonce":3228299327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6829","nonce":3228299327}]},"public_addr":"172.21.15.77:6825/3228299327","cluster_addr":"172.21.15.77:6827/3228299327","heartbeat_back_addr":"172.21.15.77:6831/3228299327","heartbeat_front_addr":"172.21.15.77:6829/3228299327","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:32.525206+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:56.489640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:18.887845+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:43.727678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:08.755553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:32.923461+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:59.298439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:23.944848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:48.883050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:14.566310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:39.557809+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:42:03.813903+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7},{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2378279024":"2024-07-28T06:36:07.527503+0000","172.21.15.29:6801/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3527133965":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2262661620":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/3156863587":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2424241521":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3881857930":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/398508875":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6801/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6800/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2679856281":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6801/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2708687265":"2024-07-28T06:34:58.250198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T06:47:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.949+0000 7f3dd4dce640 1 -- 172.21.15.29:0/232090788 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3dbc0b3c50 msgr2=0x7f3dbc0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.949+0000 7f3dd4dce640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3dbc0b3c50 0x7f3dbc0b6160 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0031440 tx=0x7f3dc0009680 comp rx=0 tx=0).stop 2024-07-27T06:47:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.949+0000 7f3dd4dce640 1 -- 172.21.15.29:0/232090788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd0108740 msgr2=0x7f3dd019f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.949+0000 7f3dd4dce640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd0108740 0x7f3dd019f780 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009fd0 tx=0x7f3dc800b260 comp rx=0 tx=0).stop 2024-07-27T06:47:00.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.949+0000 7f3dd4dce640 1 -- 172.21.15.29:0/232090788 shutdown_connections 2024-07-27T06:47:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.949+0000 7f3dd4dce640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd0108740 0x7f3dd019f780 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.949+0000 7f3dd4dce640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3dd0107df0 0x7f3dd019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.949+0000 7f3dd4dce640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3dbc0b3c50 0x7f3dbc0b6160 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.949+0000 7f3dd4dce640 1 --2- 172.21.15.29:0/232090788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3dd01024b0 0x7f3dd0199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.949+0000 7f3dd4dce640 1 -- 172.21.15.29:0/232090788 >> 172.21.15.29:0/232090788 conn(0x7f3dd0079170 msgr2=0x7f3dd01062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.949+0000 7f3dd4dce640 1 -- 172.21.15.29:0/232090788 shutdown_connections 2024-07-27T06:47:00.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:00.950+0000 7f3dd4dce640 1 -- 172.21.15.29:0/232090788 wait complete. 2024-07-27T06:47:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:01 smithi029 ceph-mon[32610]: pgmap v397: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-27T06:47:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:01 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/232090788' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:47:01.726 DEBUG:tasks.fwd_scrub:Yielding 2024-07-27T06:47:01.727 INFO:teuthology.run_tasks:Running task exec... 2024-07-27T06:47:01.738 INFO:teuthology.task.exec:Executing custom commands... 2024-07-27T06:47:01.738 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-07-27T06:47:01.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-07-27T06:47:01.739 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-07-27T06:47:01.739 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-27T06:47:01.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-27T06:47:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:01 smithi077 ceph-mon[37153]: pgmap v397: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-27T06:47:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:01 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/232090788' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:47:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.861+0000 7f9d13ca8640 1 -- 172.21.15.29:0/1756570640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d0c107e60 msgr2=0x7f9d0c108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.861+0000 7f9d13ca8640 1 --2- 172.21.15.29:0/1756570640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d0c107e60 0x7f9d0c108270 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009900 tx=0x7f9cfc02a410 comp rx=0 tx=0).stop 2024-07-27T06:47:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.861+0000 7f9d13ca8640 1 -- 172.21.15.29:0/1756570640 shutdown_connections 2024-07-27T06:47:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.861+0000 7f9d13ca8640 1 --2- 172.21.15.29:0/1756570640 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d0c1087b0 0x7f9d0c10efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.861+0000 7f9d13ca8640 1 --2- 172.21.15.29:0/1756570640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d0c107e60 0x7f9d0c108270 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.861+0000 7f9d13ca8640 1 --2- 172.21.15.29:0/1756570640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d0c102b60 0x7f9d0c102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.861+0000 7f9d13ca8640 1 -- 172.21.15.29:0/1756570640 >> 172.21.15.29:0/1756570640 conn(0x7f9d0c079880 msgr2=0x7f9d0c079c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.861+0000 7f9d13ca8640 1 -- 172.21.15.29:0/1756570640 shutdown_connections 2024-07-27T06:47:01.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.861+0000 7f9d13ca8640 1 -- 172.21.15.29:0/1756570640 wait complete. 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.861+0000 7f9d13ca8640 1 Processor -- start 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.861+0000 7f9d13ca8640 1 -- start start 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d13ca8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d0c102b60 0x7f9d0c0fe6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d13ca8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d0c107e60 0x7f9d0c0fec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d13ca8640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d0c1087b0 0x7f9d0c0ff150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d13ca8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c112440 con 0x7f9d0c102b60 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d13ca8640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c1122c0 con 0x7f9d0c1087b0 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d13ca8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c1125c0 con 0x7f9d0c107e60 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d11a44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d0c102b60 0x7f9d0c0fe6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d11a44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d0c102b60 0x7f9d0c0fe6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36588/0 (socket says 172.21.15.29:36588) 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d11a44640 1 -- 172.21.15.29:0/2279459163 learned_addr learned my addr 172.21.15.29:0/2279459163 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d11243640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d0c107e60 0x7f9d0c0fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:01.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d11a44640 1 -- 172.21.15.29:0/2279459163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d0c107e60 msgr2=0x7f9d0c0fec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d11a44640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d0c107e60 0x7f9d0c0fec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d11a44640 1 -- 172.21.15.29:0/2279459163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d0c1087b0 msgr2=0x7f9d0c0ff150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d12245640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d0c1087b0 0x7f9d0c0ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d11a44640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d0c1087b0 0x7f9d0c0ff150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d11a44640 1 -- 172.21.15.29:0/2279459163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0c1a82e0 con 0x7f9d0c102b60 2024-07-27T06:47:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.862+0000 7f9d11a44640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d0c102b60 0x7f9d0c0fe6d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9d0800e9f0 tx=0x7f9d0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.863+0000 7f9d02ffd640 1 -- 172.21.15.29:0/2279459163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0800cd80 con 0x7f9d0c102b60 2024-07-27T06:47:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.863+0000 7f9d02ffd640 1 -- 172.21.15.29:0/2279459163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d080054d0 con 0x7f9d0c102b60 2024-07-27T06:47:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.863+0000 7f9d13ca8640 1 -- 172.21.15.29:0/2279459163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0c1a85d0 con 0x7f9d0c102b60 2024-07-27T06:47:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.863+0000 7f9d02ffd640 1 -- 172.21.15.29:0/2279459163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d08010690 con 0x7f9d0c102b60 2024-07-27T06:47:01.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.863+0000 7f9d13ca8640 1 -- 172.21.15.29:0/2279459163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d0c1a8af0 con 0x7f9d0c102b60 2024-07-27T06:47:01.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.864+0000 7f9d13ca8640 1 -- 172.21.15.29:0/2279459163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d0c102f70 con 0x7f9d0c102b60 2024-07-27T06:47:01.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.867+0000 7f9d02ffd640 1 -- 172.21.15.29:0/2279459163 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f9d08004e10 con 0x7f9d0c102b60 2024-07-27T06:47:01.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.867+0000 7f9d02ffd640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9cf80b3cd0 0x7f9cf80b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:01.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.867+0000 7f9d02ffd640 1 -- 172.21.15.29:0/2279459163 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f9d080e43c0 con 0x7f9d0c102b60 2024-07-27T06:47:01.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.867+0000 7f9d11243640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9cf80b3cd0 0x7f9cf80b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:01.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.868+0000 7f9d11243640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9cf80b3cd0 0x7f9cf80b61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009230 tx=0x7f9cfc031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:01.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:01.870+0000 7f9d02ffd640 1 -- 172.21.15.29:0/2279459163 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9d08055600 con 0x7f9d0c102b60 2024-07-27T06:47:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:01 smithi047 ceph-mon[36835]: pgmap v397: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-27T06:47:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:01 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/232090788' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:47:02.016 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:02.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.096+0000 7f9d13ca8640 1 -- 172.21.15.29:0/2279459163 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f9d0c063230 con 0x7f9cf80b3cd0 2024-07-27T06:47:02.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.257+0000 7f9d02ffd640 1 -- 172.21.15.29:0/2279459163 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9d0c063230 con 0x7f9cf80b3cd0 2024-07-27T06:47:02.258 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path / 2024-07-27T06:47:02.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.259+0000 7f9d13ca8640 1 -- 172.21.15.29:0/2279459163 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9cf80b3cd0 msgr2=0x7f9cf80b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:02.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.259+0000 7f9d13ca8640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9cf80b3cd0 0x7f9cf80b61e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009230 tx=0x7f9cfc031400 comp rx=0 tx=0).stop 2024-07-27T06:47:02.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.259+0000 7f9d13ca8640 1 -- 172.21.15.29:0/2279459163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d0c102b60 msgr2=0x7f9d0c0fe6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:02.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.259+0000 7f9d13ca8640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d0c102b60 0x7f9d0c0fe6d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9d0800e9f0 tx=0x7f9d0800edf0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.259+0000 7f9d13ca8640 1 -- 172.21.15.29:0/2279459163 shutdown_connections 2024-07-27T06:47:02.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.259+0000 7f9d13ca8640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9d0c1087b0 0x7f9d0c0ff150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.259+0000 7f9d13ca8640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9cf80b3cd0 0x7f9cf80b61e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.259+0000 7f9d13ca8640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d0c107e60 0x7f9d0c0fec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.259+0000 7f9d13ca8640 1 --2- 172.21.15.29:0/2279459163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d0c102b60 0x7f9d0c0fe6d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.259+0000 7f9d13ca8640 1 -- 172.21.15.29:0/2279459163 >> 172.21.15.29:0/2279459163 conn(0x7f9d0c079880 msgr2=0x7f9d0c101b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:02.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.259+0000 7f9d13ca8640 1 -- 172.21.15.29:0/2279459163 shutdown_connections 2024-07-27T06:47:02.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.259+0000 7f9d13ca8640 1 -- 172.21.15.29:0/2279459163 wait complete. 2024-07-27T06:47:02.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-07-27T06:47:02.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.391+0000 7f2b30ba1640 1 -- 172.21.15.29:0/2321378807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b2c102220 msgr2=0x7f2b2c102630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:02.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.391+0000 7f2b30ba1640 1 --2- 172.21.15.29:0/2321378807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b2c102220 0x7f2b2c102630 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c009900 tx=0x7f2b1c02a410 comp rx=0 tx=0).stop 2024-07-27T06:47:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.391+0000 7f2b30ba1640 1 -- 172.21.15.29:0/2321378807 shutdown_connections 2024-07-27T06:47:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.391+0000 7f2b30ba1640 1 --2- 172.21.15.29:0/2321378807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b2c1084a0 0x7f2b2c10ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.391+0000 7f2b30ba1640 1 --2- 172.21.15.29:0/2321378807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2c107b50 0x7f2b2c107f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.391+0000 7f2b30ba1640 1 --2- 172.21.15.29:0/2321378807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b2c102220 0x7f2b2c102630 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.391+0000 7f2b30ba1640 1 -- 172.21.15.29:0/2321378807 >> 172.21.15.29:0/2321378807 conn(0x7f2b2c0fdc40 msgr2=0x7f2b2c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.392+0000 7f2b30ba1640 1 -- 172.21.15.29:0/2321378807 shutdown_connections 2024-07-27T06:47:02.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.392+0000 7f2b30ba1640 1 -- 172.21.15.29:0/2321378807 wait complete. 2024-07-27T06:47:02.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.392+0000 7f2b30ba1640 1 Processor -- start 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b30ba1640 1 -- start start 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b30ba1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2c102220 0x7f2b2c19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b30ba1640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b2c107b50 0x7f2b2c19b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b30ba1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b2c1084a0 0x7f2b2c19b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b30ba1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c112170 con 0x7f2b2c1084a0 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b30ba1640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c111ff0 con 0x7f2b2c107b50 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b30ba1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c1122f0 con 0x7f2b2c102220 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b2bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b2c1084a0 0x7f2b2c19b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b2bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b2c1084a0 0x7f2b2c19b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36620/0 (socket says 172.21.15.29:36620) 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b2bfff640 1 -- 172.21.15.29:0/141469839 learned_addr learned my addr 172.21.15.29:0/141469839 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b2b7fe640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2c102220 0x7f2b2c19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b2affd640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b2c107b50 0x7f2b2c19b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:02.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b2bfff640 1 -- 172.21.15.29:0/141469839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2c102220 msgr2=0x7f2b2c19aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b2bfff640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2c102220 0x7f2b2c19aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b2bfff640 1 -- 172.21.15.29:0/141469839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b2c107b50 msgr2=0x7f2b2c19b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b2bfff640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b2c107b50 0x7f2b2c19b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.393+0000 7f2b2bfff640 1 -- 172.21.15.29:0/141469839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b2c1a5d80 con 0x7f2b2c1084a0 2024-07-27T06:47:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.394+0000 7f2b2affd640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b2c107b50 0x7f2b2c19b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.394+0000 7f2b2bfff640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b2c1084a0 0x7f2b2c19b540 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009fd0 tx=0x7f2b2400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.394+0000 7f2b28ff9640 1 -- 172.21.15.29:0/141469839 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b24017070 con 0x7f2b2c1084a0 2024-07-27T06:47:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.394+0000 7f2b28ff9640 1 -- 172.21.15.29:0/141469839 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b24003e20 con 0x7f2b2c1084a0 2024-07-27T06:47:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.394+0000 7f2b28ff9640 1 -- 172.21.15.29:0/141469839 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b24010460 con 0x7f2b2c1084a0 2024-07-27T06:47:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.394+0000 7f2b30ba1640 1 -- 172.21.15.29:0/141469839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b2c1a6070 con 0x7f2b2c1084a0 2024-07-27T06:47:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.394+0000 7f2b30ba1640 1 -- 172.21.15.29:0/141469839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b2c1a6570 con 0x7f2b2c1084a0 2024-07-27T06:47:02.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.395+0000 7f2b30ba1640 1 -- 172.21.15.29:0/141469839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b2c103490 con 0x7f2b2c1084a0 2024-07-27T06:47:02.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.397+0000 7f2b28ff9640 1 -- 172.21.15.29:0/141469839 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f2b24003970 con 0x7f2b2c1084a0 2024-07-27T06:47:02.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.397+0000 7f2b28ff9640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2b140b3c10 0x7f2b140b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:02.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.398+0000 7f2b28ff9640 1 -- 172.21.15.29:0/141469839 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f2b240e3af0 con 0x7f2b2c1084a0 2024-07-27T06:47:02.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.398+0000 7f2b2b7fe640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2b140b3c10 0x7f2b140b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:02.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.398+0000 7f2b2b7fe640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2b140b3c10 0x7f2b140b6120 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c000c00 tx=0x7f2b1c031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:02.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.401+0000 7f2b28ff9640 1 -- 172.21.15.29:0/141469839 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2b240ab030 con 0x7f2b2c1084a0 2024-07-27T06:47:02.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.623+0000 7f2b30ba1640 1 -- 172.21.15.29:0/141469839 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f2b2c0632d0 con 0x7f2b140b3c10 2024-07-27T06:47:02.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.686+0000 7f2b28ff9640 1 -- 172.21.15.29:0/141469839 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f2b2c0632d0 con 0x7f2b140b3c10 2024-07-27T06:47:02.687 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path / 2024-07-27T06:47:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.688+0000 7f2b30ba1640 1 -- 172.21.15.29:0/141469839 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2b140b3c10 msgr2=0x7f2b140b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.688+0000 7f2b30ba1640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2b140b3c10 0x7f2b140b6120 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c000c00 tx=0x7f2b1c031400 comp rx=0 tx=0).stop 2024-07-27T06:47:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.688+0000 7f2b30ba1640 1 -- 172.21.15.29:0/141469839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b2c1084a0 msgr2=0x7f2b2c19b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.688+0000 7f2b30ba1640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b2c1084a0 0x7f2b2c19b540 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009fd0 tx=0x7f2b2400b260 comp rx=0 tx=0).stop 2024-07-27T06:47:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.688+0000 7f2b30ba1640 1 -- 172.21.15.29:0/141469839 shutdown_connections 2024-07-27T06:47:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.688+0000 7f2b30ba1640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b2c1084a0 0x7f2b2c19b540 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.688+0000 7f2b30ba1640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b2c107b50 0x7f2b2c19b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.688+0000 7f2b30ba1640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2b140b3c10 0x7f2b140b6120 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.688+0000 7f2b30ba1640 1 --2- 172.21.15.29:0/141469839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2c102220 0x7f2b2c19aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.688+0000 7f2b30ba1640 1 -- 172.21.15.29:0/141469839 >> 172.21.15.29:0/141469839 conn(0x7f2b2c0fdc40 msgr2=0x7f2b2c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.689+0000 7f2b30ba1640 1 -- 172.21.15.29:0/141469839 shutdown_connections 2024-07-27T06:47:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.689+0000 7f2b30ba1640 1 -- 172.21.15.29:0/141469839 wait complete. 2024-07-27T06:47:02.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-07-27T06:47:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.816+0000 7fd711fba640 1 -- 172.21.15.29:0/338355272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70c108670 msgr2=0x7fd70c10eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.816+0000 7fd711fba640 1 --2- 172.21.15.29:0/338355272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70c108670 0x7fd70c10eea0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd704009ca0 tx=0x7fd70402a8c0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.816+0000 7fd711fba640 1 -- 172.21.15.29:0/338355272 shutdown_connections 2024-07-27T06:47:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.816+0000 7fd711fba640 1 --2- 172.21.15.29:0/338355272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70c108670 0x7fd70c10eea0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.816+0000 7fd711fba640 1 --2- 172.21.15.29:0/338355272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd70c107d20 0x7fd70c108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.816+0000 7fd711fba640 1 --2- 172.21.15.29:0/338355272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd70c1023f0 0x7fd70c102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.816+0000 7fd711fba640 1 -- 172.21.15.29:0/338355272 >> 172.21.15.29:0/338355272 conn(0x7fd70c0fddf0 msgr2=0x7fd70c100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.816+0000 7fd711fba640 1 -- 172.21.15.29:0/338355272 shutdown_connections 2024-07-27T06:47:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.816+0000 7fd711fba640 1 -- 172.21.15.29:0/338355272 wait complete. 2024-07-27T06:47:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.817+0000 7fd711fba640 1 Processor -- start 2024-07-27T06:47:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.817+0000 7fd711fba640 1 -- start start 2024-07-27T06:47:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.817+0000 7fd711fba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70c1023f0 0x7fd70c198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.817+0000 7fd711fba640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd70c107d20 0x7fd70c1992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.817+0000 7fd711fba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd70c108670 0x7fd70c19e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.817+0000 7fd711fba640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c112370 con 0x7fd70c1023f0 2024-07-27T06:47:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.817+0000 7fd711fba640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c1121f0 con 0x7fd70c107d20 2024-07-27T06:47:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.817+0000 7fd711fba640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c1124f0 con 0x7fd70c108670 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.817+0000 7fd70b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70c1023f0 0x7fd70c198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.817+0000 7fd70b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70c1023f0 0x7fd70c198d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36632/0 (socket says 172.21.15.29:36632) 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.817+0000 7fd70b7fe640 1 -- 172.21.15.29:0/3438784916 learned_addr learned my addr 172.21.15.29:0/3438784916 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd70affd640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd70c107d20 0x7fd70c1992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd70bfff640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd70c108670 0x7fd70c19e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd70b7fe640 1 -- 172.21.15.29:0/3438784916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd70c108670 msgr2=0x7fd70c19e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd70b7fe640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd70c108670 0x7fd70c19e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd70b7fe640 1 -- 172.21.15.29:0/3438784916 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd70c107d20 msgr2=0x7fd70c1992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd70b7fe640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd70c107d20 0x7fd70c1992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd70b7fe640 1 -- 172.21.15.29:0/3438784916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70c19ec40 con 0x7fd70c1023f0 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd70bfff640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd70c108670 0x7fd70c19e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd70b7fe640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70c1023f0 0x7fd70c198d10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00d950 tx=0x7fd6fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd708ff9640 1 -- 172.21.15.29:0/3438784916 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc017070 con 0x7fd70c1023f0 2024-07-27T06:47:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd711fba640 1 -- 172.21.15.29:0/3438784916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70c19ef30 con 0x7fd70c1023f0 2024-07-27T06:47:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd708ff9640 1 -- 172.21.15.29:0/3438784916 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6fc009c30 con 0x7fd70c1023f0 2024-07-27T06:47:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd708ff9640 1 -- 172.21.15.29:0/3438784916 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc00f620 con 0x7fd70c1023f0 2024-07-27T06:47:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.818+0000 7fd711fba640 1 -- 172.21.15.29:0/3438784916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd70c1a56c0 con 0x7fd70c1023f0 2024-07-27T06:47:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.819+0000 7fd711fba640 1 -- 172.21.15.29:0/3438784916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd70c102800 con 0x7fd70c1023f0 2024-07-27T06:47:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.821+0000 7fd708ff9640 1 -- 172.21.15.29:0/3438784916 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7fd6fc010460 con 0x7fd70c1023f0 2024-07-27T06:47:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.822+0000 7fd708ff9640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd6f80b3c20 0x7fd6f80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.822+0000 7fd70affd640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd6f80b3c20 0x7fd6f80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.822+0000 7fd708ff9640 1 -- 172.21.15.29:0/3438784916 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd6fc0e39a0 con 0x7fd70c1023f0 2024-07-27T06:47:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.822+0000 7fd70affd640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd6f80b3c20 0x7fd6f80b6130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd70c101640 tx=0x7fd6f400b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:02.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:02.826+0000 7fd708ff9640 1 -- 172.21.15.29:0/3438784916 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd6fc0aaee0 con 0x7fd70c1023f0 2024-07-27T06:47:03.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.045+0000 7fd711fba640 1 -- 172.21.15.29:0/3438784916 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fd70c063230 con 0x7fd6f80b3c20 2024-07-27T06:47:03.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.052+0000 7fd708ff9640 1 -- 172.21.15.29:0/3438784916 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fd70c063230 con 0x7fd6f80b3c20 2024-07-27T06:47:03.053 INFO:teuthology.orchestra.run.smithi029.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-07-27T00:00:00", "created": "2024-07-27T06:47:02", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-07-27T06:47:03.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.054+0000 7fd711fba640 1 -- 172.21.15.29:0/3438784916 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd6f80b3c20 msgr2=0x7fd6f80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.054+0000 7fd711fba640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd6f80b3c20 0x7fd6f80b6130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd70c101640 tx=0x7fd6f400b390 comp rx=0 tx=0).stop 2024-07-27T06:47:03.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.054+0000 7fd711fba640 1 -- 172.21.15.29:0/3438784916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70c1023f0 msgr2=0x7fd70c198d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.054+0000 7fd711fba640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70c1023f0 0x7fd70c198d10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00d950 tx=0x7fd6fc00de10 comp rx=0 tx=0).stop 2024-07-27T06:47:03.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.054+0000 7fd711fba640 1 -- 172.21.15.29:0/3438784916 shutdown_connections 2024-07-27T06:47:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.054+0000 7fd711fba640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd6f80b3c20 0x7fd6f80b6130 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.054+0000 7fd711fba640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd70c108670 0x7fd70c19e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.054+0000 7fd711fba640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd70c107d20 0x7fd70c1992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.054+0000 7fd711fba640 1 --2- 172.21.15.29:0/3438784916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd70c1023f0 0x7fd70c198d10 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.054+0000 7fd711fba640 1 -- 172.21.15.29:0/3438784916 >> 172.21.15.29:0/3438784916 conn(0x7fd70c0fddf0 msgr2=0x7fd70c1072f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.054+0000 7fd711fba640 1 -- 172.21.15.29:0/3438784916 shutdown_connections 2024-07-27T06:47:03.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.054+0000 7fd711fba640 1 -- 172.21.15.29:0/3438784916 wait complete. 2024-07-27T06:47:03.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-07-27T06:47:03.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.185+0000 7f6725e68640 1 -- 172.21.15.29:0/3002043478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720104340 msgr2=0x7f6720104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.185+0000 7f6725e68640 1 --2- 172.21.15.29:0/3002043478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720104340 0x7f6720104750 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6710009900 tx=0x7f671002a410 comp rx=0 tx=0).stop 2024-07-27T06:47:03.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.185+0000 7f6725e68640 1 -- 172.21.15.29:0/3002043478 shutdown_connections 2024-07-27T06:47:03.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.185+0000 7f6725e68640 1 --2- 172.21.15.29:0/3002043478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f672010d7a0 0x7f672010fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.185+0000 7f6725e68640 1 --2- 172.21.15.29:0/3002043478 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6720104c90 0x7f672010d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.185+0000 7f6725e68640 1 --2- 172.21.15.29:0/3002043478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720104340 0x7f6720104750 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.185+0000 7f6725e68640 1 -- 172.21.15.29:0/3002043478 >> 172.21.15.29:0/3002043478 conn(0x7f67200fddf0 msgr2=0x7f6720100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:03.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.185+0000 7f6725e68640 1 -- 172.21.15.29:0/3002043478 shutdown_connections 2024-07-27T06:47:03.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.185+0000 7f6725e68640 1 -- 172.21.15.29:0/3002043478 wait complete. 2024-07-27T06:47:03.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.186+0000 7f6725e68640 1 Processor -- start 2024-07-27T06:47:03.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.186+0000 7f6725e68640 1 -- start start 2024-07-27T06:47:03.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f6725e68640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720104340 0x7f6720199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f6725e68640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6720104c90 0x7f672019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f6725e68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f672010d7a0 0x7f672019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f6725e68640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6720113430 con 0x7f6720104340 2024-07-27T06:47:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f6725e68640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f67201132b0 con 0x7f6720104c90 2024-07-27T06:47:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f6725e68640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f67201135b0 con 0x7f672010d7a0 2024-07-27T06:47:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f671f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720104340 0x7f6720199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f671f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720104340 0x7f6720199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36638/0 (socket says 172.21.15.29:36638) 2024-07-27T06:47:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f671f7fe640 1 -- 172.21.15.29:0/1845147236 learned_addr learned my addr 172.21.15.29:0/1845147236 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f671ffff640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f672010d7a0 0x7f672019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f671f7fe640 1 -- 172.21.15.29:0/1845147236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f672010d7a0 msgr2=0x7f672019f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f671f7fe640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f672010d7a0 0x7f672019f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f671effd640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6720104c90 0x7f672019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f671f7fe640 1 -- 172.21.15.29:0/1845147236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6720104c90 msgr2=0x7f672019a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f671f7fe640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6720104c90 0x7f672019a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f671f7fe640 1 -- 172.21.15.29:0/1845147236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672019fd80 con 0x7f6720104340 2024-07-27T06:47:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.187+0000 7f671effd640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6720104c90 0x7f672019a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.188+0000 7f671f7fe640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720104340 0x7f6720199e50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6710005940 tx=0x7f67100042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.188+0000 7f671cff9640 1 -- 172.21.15.29:0/1845147236 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671003f030 con 0x7f6720104340 2024-07-27T06:47:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.188+0000 7f671cff9640 1 -- 172.21.15.29:0/1845147236 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6710003cb0 con 0x7f6720104340 2024-07-27T06:47:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.188+0000 7f6725e68640 1 -- 172.21.15.29:0/1845147236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67201a0010 con 0x7f6720104340 2024-07-27T06:47:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.188+0000 7f671cff9640 1 -- 172.21.15.29:0/1845147236 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6710035070 con 0x7f6720104340 2024-07-27T06:47:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.188+0000 7f6725e68640 1 -- 172.21.15.29:0/1845147236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67201a6800 con 0x7f6720104340 2024-07-27T06:47:03.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.189+0000 7f6725e68640 1 -- 172.21.15.29:0/1845147236 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6720104750 con 0x7f6720104340 2024-07-27T06:47:03.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.190+0000 7f671cff9640 1 -- 172.21.15.29:0/1845147236 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f6710042030 con 0x7f6720104340 2024-07-27T06:47:03.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.192+0000 7f671cff9640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f67080b3c20 0x7f67080b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.192+0000 7f671effd640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f67080b3c20 0x7f67080b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.192+0000 7f671cff9640 1 -- 172.21.15.29:0/1845147236 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f67101046c0 con 0x7f6720104340 2024-07-27T06:47:03.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.192+0000 7f671effd640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f67080b3c20 0x7f67080b6130 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6714005d00 tx=0x7f6714005ed0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:03.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.195+0000 7f671cff9640 1 -- 172.21.15.29:0/1845147236 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f67100cbc00 con 0x7f6720104340 2024-07-27T06:47:03.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.429+0000 7f6725e68640 1 -- 172.21.15.29:0/1845147236 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f67201079a0 con 0x7f67080b3c20 2024-07-27T06:47:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.436+0000 7f671cff9640 1 -- 172.21.15.29:0/1845147236 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f67201079a0 con 0x7f67080b3c20 2024-07-27T06:47:03.437 INFO:teuthology.orchestra.run.smithi029.stdout:/ 1m 6m3h 2024-07-27T06:47:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.439+0000 7f6725e68640 1 -- 172.21.15.29:0/1845147236 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f67080b3c20 msgr2=0x7f67080b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.439+0000 7f6725e68640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f67080b3c20 0x7f67080b6130 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6714005d00 tx=0x7f6714005ed0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.439+0000 7f6725e68640 1 -- 172.21.15.29:0/1845147236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720104340 msgr2=0x7f6720199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.439+0000 7f6725e68640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720104340 0x7f6720199e50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6710005940 tx=0x7f67100042f0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.439+0000 7f6725e68640 1 -- 172.21.15.29:0/1845147236 shutdown_connections 2024-07-27T06:47:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.439+0000 7f6725e68640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f67080b3c20 0x7f67080b6130 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.439+0000 7f6725e68640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f672010d7a0 0x7f672019f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.439+0000 7f6725e68640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6720104c90 0x7f672019a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.439+0000 7f6725e68640 1 --2- 172.21.15.29:0/1845147236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6720104340 0x7f6720199e50 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.439+0000 7f6725e68640 1 -- 172.21.15.29:0/1845147236 >> 172.21.15.29:0/1845147236 conn(0x7f67200fddf0 msgr2=0x7f6720107290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:03.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.439+0000 7f6725e68640 1 -- 172.21.15.29:0/1845147236 shutdown_connections 2024-07-27T06:47:03.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.439+0000 7f6725e68640 1 -- 172.21.15.29:0/1845147236 wait complete. 2024-07-27T06:47:03.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-07-27T06:47:03.486 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-07-27T06:47:03.497 INFO:teuthology.run_tasks:Running task background_exec... 2024-07-27T06:47:03.516 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-07-27T06:47:03.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-07-27T06:47:03.529 INFO:teuthology.run_tasks:Running task quiescer... 2024-07-27T06:47:03.543 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-27T06:47:03.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph pg dump --format=json 2024-07-27T06:47:03.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:03 smithi029 ceph-mon[32610]: pgmap v398: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-27T06:47:03.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:03 smithi029 ceph-mon[32610]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:47:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.745+0000 7f9300980640 1 -- 172.21.15.29:0/2248949674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92fc101a80 msgr2=0x7f92fc101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.745+0000 7f9300980640 1 --2- 172.21.15.29:0/2248949674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92fc101a80 0x7f92fc101e90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f92ec009900 tx=0x7f92ec02a410 comp rx=0 tx=0).stop 2024-07-27T06:47:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.746+0000 7f9300980640 1 -- 172.21.15.29:0/2248949674 shutdown_connections 2024-07-27T06:47:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.746+0000 7f9300980640 1 --2- 172.21.15.29:0/2248949674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92fc106620 0x7f92fc110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.746+0000 7f9300980640 1 --2- 172.21.15.29:0/2248949674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92fc105cd0 0x7f92fc1060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.746+0000 7f9300980640 1 --2- 172.21.15.29:0/2248949674 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92fc101a80 0x7f92fc101e90 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.746+0000 7f9300980640 1 -- 172.21.15.29:0/2248949674 >> 172.21.15.29:0/2248949674 conn(0x7f92fc077690 msgr2=0x7f92fc077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.746+0000 7f9300980640 1 -- 172.21.15.29:0/2248949674 shutdown_connections 2024-07-27T06:47:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.746+0000 7f9300980640 1 -- 172.21.15.29:0/2248949674 wait complete. 2024-07-27T06:47:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.747+0000 7f9300980640 1 Processor -- start 2024-07-27T06:47:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.747+0000 7f9300980640 1 -- start start 2024-07-27T06:47:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.747+0000 7f9300980640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92fc101a80 0x7f92fc199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.747+0000 7f9300980640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92fc105cd0 0x7f92fc19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.747+0000 7f9300980640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92fc106620 0x7f92fc19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.747+0000 7f9300980640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc113530 con 0x7f92fc106620 2024-07-27T06:47:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.747+0000 7f9300980640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1133b0 con 0x7f92fc105cd0 2024-07-27T06:47:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.747+0000 7f9300980640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1136b0 con 0x7f92fc101a80 2024-07-27T06:47:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.747+0000 7f92fad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92fc106620 0x7f92fc19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.748+0000 7f92fad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92fc106620 0x7f92fc19f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36658/0 (socket says 172.21.15.29:36658) 2024-07-27T06:47:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.748+0000 7f92fad9d640 1 -- 172.21.15.29:0/3102492038 learned_addr learned my addr 172.21.15.29:0/3102492038 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.748+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92fc105cd0 0x7f92fc19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.748+0000 7f92fa59c640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92fc101a80 0x7f92fc199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.748+0000 7f92fad9d640 1 -- 172.21.15.29:0/3102492038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92fc101a80 msgr2=0x7f92fc199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.748+0000 7f92fad9d640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92fc101a80 0x7f92fc199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.748+0000 7f92fad9d640 1 -- 172.21.15.29:0/3102492038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92fc105cd0 msgr2=0x7f92fc19a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.748+0000 7f92fad9d640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92fc105cd0 0x7f92fc19a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.748+0000 7f92fad9d640 1 -- 172.21.15.29:0/3102492038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92fc19fe70 con 0x7f92fc106620 2024-07-27T06:47:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.748+0000 7f92fad9d640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92fc106620 0x7f92fc19f740 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f92f4009fd0 tx=0x7f92f400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.748+0000 7f92eb7fe640 1 -- 172.21.15.29:0/3102492038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f4017070 con 0x7f92fc106620 2024-07-27T06:47:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.748+0000 7f92eb7fe640 1 -- 172.21.15.29:0/3102492038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92f4003e20 con 0x7f92fc106620 2024-07-27T06:47:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.749+0000 7f92eb7fe640 1 -- 172.21.15.29:0/3102492038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f4010460 con 0x7f92fc106620 2024-07-27T06:47:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.749+0000 7f9300980640 1 -- 172.21.15.29:0/3102492038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92fc1a0160 con 0x7f92fc106620 2024-07-27T06:47:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.749+0000 7f9300980640 1 -- 172.21.15.29:0/3102492038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92fc1a68c0 con 0x7f92fc106620 2024-07-27T06:47:03.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f9306f04740 1 Processor -- start 2024-07-27T06:47:03.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f92eb7fe640 1 -- 172.21.15.29:0/3102492038 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 23) ==== 151076+0+0 (secure 0 0 0) 0x7f92f4003970 con 0x7f92fc106620 2024-07-27T06:47:03.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f9306f04740 1 -- start start 2024-07-27T06:47:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f9306f04740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1cc350 0x556f1b5713b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f9306f04740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1a3490 0x556f1b5718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f9306f04740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 0x556f1b571e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f9306f04740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556f1b4592f0 con 0x556f1b1cc350 2024-07-27T06:47:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f9306f04740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x556f1b4e7a80 con 0x556f1b1ab670 2024-07-27T06:47:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f9306f04740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556f1b4d6f80 con 0x556f1b1a3490 2024-07-27T06:47:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f92f9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1cc350 0x556f1b5713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f92f9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1cc350 0x556f1b5713b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36668/0 (socket says 172.21.15.29:36668) 2024-07-27T06:47:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f92f9d9b640 1 -- 172.21.15.29:0/4149139760 learned_addr learned my addr 172.21.15.29:0/4149139760 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.752+0000 7f92fa59c640 1 --2- 172.21.15.29:0/4149139760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1a3490 0x556f1b5718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.753+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/4149139760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 0x556f1b571e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.753+0000 7f92eb7fe640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f92e40b3c60 0x7f92e40b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.753+0000 7f92f9d9b640 1 -- 172.21.15.29:0/4149139760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1a3490 msgr2=0x556f1b5718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.753+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/4149139760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1a3490 0x556f1b5718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.753+0000 7f92f9d9b640 1 -- 172.21.15.29:0/4149139760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 msgr2=0x556f1b571e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.753+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/4149139760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 0x556f1b571e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.753+0000 7f92f9d9b640 1 -- 172.21.15.29:0/4149139760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f1b237930 con 0x556f1b1cc350 2024-07-27T06:47:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.753+0000 7f92fad9d640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f92e40b3c60 0x7f92e40b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.753+0000 7f92eb7fe640 1 -- 172.21.15.29:0/3102492038 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f92f40e3a50 con 0x7f92fc106620 2024-07-27T06:47:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.753+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/4149139760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1cc350 0x556f1b5713b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f92f00058a0 tx=0x7f92f0005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff0293040806dfe server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.753+0000 7f92e8ff9640 1 -- 172.21.15.29:0/4149139760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f00120d0 con 0x556f1b1cc350 2024-07-27T06:47:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.753+0000 7f92e8ff9640 1 -- 172.21.15.29:0/4149139760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92f00054d0 con 0x556f1b1cc350 2024-07-27T06:47:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.754+0000 7f92e8ff9640 1 -- 172.21.15.29:0/4149139760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f0004ad0 con 0x556f1b1cc350 2024-07-27T06:47:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.754+0000 7f9306f04740 1 -- 172.21.15.29:0/4149139760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1cc350 msgr2=0x556f1b5713b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.754+0000 7f9306f04740 1 --2- 172.21.15.29:0/4149139760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1cc350 0x556f1b5713b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f92f00058a0 tx=0x7f92f0005c90 comp rx=0 tx=0).stop 2024-07-27T06:47:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.754+0000 7f92fad9d640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f92e40b3c60 0x7f92e40b6170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f92f400dfd0 tx=0x7f92f40cd040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.755+0000 7f9306f04740 1 -- 172.21.15.29:0/4149139760 shutdown_connections 2024-07-27T06:47:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.755+0000 7f9306f04740 1 --2- 172.21.15.29:0/4149139760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 0x556f1b571e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.755+0000 7f9306f04740 1 --2- 172.21.15.29:0/4149139760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1a3490 0x556f1b5718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.755+0000 7f9306f04740 1 --2- 172.21.15.29:0/4149139760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1cc350 0x556f1b5713b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.755+0000 7f9306f04740 1 -- 172.21.15.29:0/4149139760 >> 172.21.15.29:0/4149139760 conn(0x556f1b1c0740 msgr2=0x556f1b56f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.755+0000 7f9306f04740 1 -- 172.21.15.29:0/4149139760 shutdown_connections 2024-07-27T06:47:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.755+0000 7f9306f04740 1 -- 172.21.15.29:0/4149139760 wait complete. 2024-07-27T06:47:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.755+0000 7f9306f04740 1 -- start start 2024-07-27T06:47:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.755+0000 7f9306f04740 1 Processor -- start 2024-07-27T06:47:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f9306f04740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1a3490 0x556f1b5812b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f9306f04740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1ab670 0x556f1b5817f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f9306f04740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1cc350 0x556f1b58d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f9306f04740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556f1b4794f0 con 0x556f1b1ab670 2024-07-27T06:47:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f9306f04740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x556f1b4dafc0 con 0x556f1b1cc350 2024-07-27T06:47:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f9306f04740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556f1b33a890 con 0x556f1b1a3490 2024-07-27T06:47:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f9306f04740 10 client.0 _ll_register_callbacks cb 0x556f1b483500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f92fa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1ab670 0x556f1b5817f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f92fa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1ab670 0x556f1b5817f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36678/0 (socket says 172.21.15.29:36678) 2024-07-27T06:47:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f92fa59c640 1 -- 172.21.15.29:0/2404664086 learned_addr learned my addr 172.21.15.29:0/2404664086 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1cc350 0x556f1b58d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1a3490 0x556f1b5812b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f92fa59c640 1 -- 172.21.15.29:0/2404664086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1a3490 msgr2=0x556f1b5812b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f92fa59c640 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1a3490 0x556f1b5812b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f92fa59c640 1 -- 172.21.15.29:0/2404664086 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1cc350 msgr2=0x556f1b58d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f92fa59c640 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1cc350 0x556f1b58d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.756+0000 7f92fa59c640 1 -- 172.21.15.29:0/2404664086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f1b488c50 con 0x556f1b1ab670 2024-07-27T06:47:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.757+0000 7f92fa59c640 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1ab670 0x556f1b5817f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f92ec000c00 tx=0x7f92ec005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25d0cfa8c8daa3d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.757+0000 7f92e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.757+0000 7f92e1ffb640 1 -- 172.21.15.29:0/2404664086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec035040 con 0x556f1b1ab670 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.757+0000 7f9306f04740 1 -- 172.21.15.29:0/2404664086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92f00054d0 con 0x556f1b1ab670 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.757+0000 7f92e1ffb640 1 -- 172.21.15.29:0/2404664086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f92ec0046c0 con 0x556f1b1ab670 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.757+0000 7f92e1ffb640 1 -- 172.21.15.29:0/2404664086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f92ec005620 con 0x556f1b1ab670 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.757+0000 7f9306f04740 10 client.15510 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.757+0000 7f9306f04740 1 -- 172.21.15.29:0/2404664086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f1b58fd40 con 0x556f1b1ab670 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.757+0000 7f92e1ffb640 1 -- 172.21.15.29:0/2404664086 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92f00054d0 con 0x556f1b1ab670 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.758+0000 7f92e1ffb640 1 -- 172.21.15.29:0/2404664086 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f92ec044020 con 0x556f1b1ab670 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.758+0000 7f9306f04740 10 client.15510 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.758+0000 7f9306f04740 10 client.15510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.758+0000 7f9306f04740 20 client.15510 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.758+0000 7f9306f04740 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556f1b18acd0 0x556f1b593260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.758+0000 7f9306f04740 4 client.15510 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:03.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.758+0000 7f9306f04740 1 -- 172.21.15.29:0/2404664086 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f1b4e99d0 con 0x556f1b18acd0 2024-07-27T06:47:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.758+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556f1b18acd0 0x556f1b593260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.759+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556f1b18acd0 0x556f1b593260 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bef7cdbe01ec83ff server_cookie=adee321baa814e0c in_seq=0 out_seq=0 2024-07-27T06:47:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.759+0000 7f92e1ffb640 10 client.15510 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.759+0000 7f92e1ffb640 1 -- 172.21.15.29:0/2404664086 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556f1b4e99d0 con 0x556f1b18acd0 2024-07-27T06:47:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.759+0000 7f92e1ffb640 10 client.15510 handle_command_reply: tid=0 2024-07-27T06:47:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.759+0000 7f9306f04740 1 client.15510 shutdown 2024-07-27T06:47:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.760+0000 7f9306f04740 1 -- 172.21.15.29:0/2404664086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1ab670 msgr2=0x556f1b5817f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.760+0000 7f9306f04740 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1ab670 0x556f1b5817f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f92ec000c00 tx=0x7f92ec005060 comp rx=0 tx=0).stop 2024-07-27T06:47:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.760+0000 7f9306f04740 1 -- 172.21.15.29:0/2404664086 shutdown_connections 2024-07-27T06:47:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.760+0000 7f9306f04740 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1cc350 0x556f1b58d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.760+0000 7f9306f04740 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1ab670 0x556f1b5817f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.760+0000 7f9306f04740 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556f1b18acd0 0x556f1b593260 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.760+0000 7f9306f04740 1 --2- 172.21.15.29:0/2404664086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1a3490 0x556f1b5812b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.760+0000 7f92e1ffb640 0 client.15510 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.760+0000 7f9306f04740 1 -- 172.21.15.29:0/2404664086 >> 172.21.15.29:0/2404664086 conn(0x556f1b1c0740 msgr2=0x556f1b574500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.760+0000 7f9306f04740 1 -- 172.21.15.29:0/2404664086 shutdown_connections 2024-07-27T06:47:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.760+0000 7f9306f04740 1 -- 172.21.15.29:0/2404664086 wait complete. 2024-07-27T06:47:03.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.760+0000 7f9306f04740 20 client.15510 trim_cache size 0 max 16384 2024-07-27T06:47:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.784+0000 7f9306f04740 1 Processor -- start 2024-07-27T06:47:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.784+0000 7f9306f04740 1 -- start start 2024-07-27T06:47:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.784+0000 7f9306f04740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 0x556f1b582ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.784+0000 7f9306f04740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 0x556f1b58cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.784+0000 7f9306f04740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1cc350 0x556f1b58d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.784+0000 7f9306f04740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1acf00 con 0x556f1b1a3490 2024-07-27T06:47:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.784+0000 7f9306f04740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1aea40 con 0x556f1b1ab670 2024-07-27T06:47:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.784+0000 7f9306f04740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1ad5d0 con 0x556f1b1cc350 2024-07-27T06:47:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.784+0000 7f92f9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 0x556f1b582ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.784+0000 7f92f9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 0x556f1b582ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36686/0 (socket says 172.21.15.29:36686) 2024-07-27T06:47:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.784+0000 7f92f9d9b640 1 -- 172.21.15.29:0/2541799949 learned_addr learned my addr 172.21.15.29:0/2541799949 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.784+0000 7f92fa59c640 1 --2- 172.21.15.29:0/2541799949 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 0x556f1b58cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.785+0000 7f92f9d9b640 1 -- 172.21.15.29:0/2541799949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1cc350 msgr2=0x556f1b58d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.785+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/2541799949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1cc350 0x556f1b58d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.785+0000 7f92f9d9b640 1 -- 172.21.15.29:0/2541799949 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 msgr2=0x556f1b58cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.785+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/2541799949 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 0x556f1b58cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.785+0000 7f92f9d9b640 1 -- 172.21.15.29:0/2541799949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f1b57c8d0 con 0x556f1b1a3490 2024-07-27T06:47:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.785+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/2541799949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 0x556f1b582ce0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f92f000bb80 tx=0x7f92f0005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2c6291b148b5ec server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.785+0000 7f92e3fff640 1 -- 172.21.15.29:0/2541799949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f0033070 con 0x556f1b1a3490 2024-07-27T06:47:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.785+0000 7f92e3fff640 1 -- 172.21.15.29:0/2541799949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92f00044f0 con 0x556f1b1a3490 2024-07-27T06:47:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.785+0000 7f92e3fff640 1 -- 172.21.15.29:0/2541799949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f000eba0 con 0x556f1b1a3490 2024-07-27T06:47:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.786+0000 7f9306f04740 1 -- 172.21.15.29:0/2541799949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 msgr2=0x556f1b582ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.786+0000 7f9306f04740 1 --2- 172.21.15.29:0/2541799949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 0x556f1b582ce0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f92f000bb80 tx=0x7f92f0005c90 comp rx=0 tx=0).stop 2024-07-27T06:47:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.786+0000 7f9306f04740 1 -- 172.21.15.29:0/2541799949 shutdown_connections 2024-07-27T06:47:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.786+0000 7f9306f04740 1 --2- 172.21.15.29:0/2541799949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1cc350 0x556f1b58d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.786+0000 7f9306f04740 1 --2- 172.21.15.29:0/2541799949 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 0x556f1b58cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.786+0000 7f9306f04740 1 --2- 172.21.15.29:0/2541799949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 0x556f1b582ce0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.786+0000 7f9306f04740 1 -- 172.21.15.29:0/2541799949 >> 172.21.15.29:0/2541799949 conn(0x556f1b1c0740 msgr2=0x556f1b5825d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.786+0000 7f9306f04740 1 -- 172.21.15.29:0/2541799949 shutdown_connections 2024-07-27T06:47:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.786+0000 7f9306f04740 1 -- 172.21.15.29:0/2541799949 wait complete. 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.786+0000 7f9306f04740 1 -- start start 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.787+0000 7f9306f04740 1 Processor -- start 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.787+0000 7f9306f04740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 0x556f1b5893e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.787+0000 7f9306f04740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 0x556f1b584890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.787+0000 7f92f9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 0x556f1b5893e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.787+0000 7f92f9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 0x556f1b5893e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36694/0 (socket says 172.21.15.29:36694) 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.787+0000 7f9306f04740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1cc350 0x556f1b584dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.787+0000 7f9306f04740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1ae370 con 0x556f1b1a3490 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.787+0000 7f92f9d9b640 1 -- 172.21.15.29:0/330179824 learned_addr learned my addr 172.21.15.29:0/330179824 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.787+0000 7f9306f04740 1 -- 172.21.15.29:0/330179824 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1adca0 con 0x556f1b1ab670 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.787+0000 7f9306f04740 1 -- 172.21.15.29:0/330179824 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1aaee0 con 0x556f1b1cc350 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.787+0000 7f9306f04740 10 client.0 _ll_register_callbacks cb 0x7f92ec004320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.787+0000 7f92fa59c640 1 --2- 172.21.15.29:0/330179824 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 0x556f1b584890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f92f9d9b640 1 -- 172.21.15.29:0/330179824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1cc350 msgr2=0x556f1b584dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-27T06:47:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/330179824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1cc350 0x556f1b584dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f92f9d9b640 1 -- 172.21.15.29:0/330179824 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 msgr2=0x556f1b584890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/330179824 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 0x556f1b584890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f92f9d9b640 1 -- 172.21.15.29:0/330179824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f1b585310 con 0x556f1b1a3490 2024-07-27T06:47:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f92f9d9b640 1 --2- 172.21.15.29:0/330179824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 0x556f1b5893e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f92f0014010 tx=0x7f92f0012de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da9e99065a4d220 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f92e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f92e1ffb640 1 -- 172.21.15.29:0/330179824 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f003e060 con 0x556f1b1a3490 2024-07-27T06:47:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f92e1ffb640 1 -- 172.21.15.29:0/330179824 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f92f000b690 con 0x556f1b1a3490 2024-07-27T06:47:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f9306f04740 1 -- 172.21.15.29:0/330179824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92f00044f0 con 0x556f1b1a3490 2024-07-27T06:47:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f92e1ffb640 1 -- 172.21.15.29:0/330179824 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f92f0033070 con 0x556f1b1a3490 2024-07-27T06:47:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f9306f04740 10 client.15516 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.788+0000 7f9306f04740 1 -- 172.21.15.29:0/330179824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f1b5fb620 con 0x556f1b1a3490 2024-07-27T06:47:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.789+0000 7f92e1ffb640 1 -- 172.21.15.29:0/330179824 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92f00044f0 con 0x556f1b1a3490 2024-07-27T06:47:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.789+0000 7f92e1ffb640 1 -- 172.21.15.29:0/330179824 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f92f00109d0 con 0x556f1b1a3490 2024-07-27T06:47:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.789+0000 7f9306f04740 10 client.15516 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.789+0000 7f9306f04740 10 client.15516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.789+0000 7f9306f04740 20 client.15516 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.789+0000 7f9306f04740 1 --2- 172.21.15.29:0/330179824 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556f1b160080 0x556f1b5fdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.789+0000 7f9306f04740 4 client.15516 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-27T06:47:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.789+0000 7f9306f04740 1 -- 172.21.15.29:0/330179824 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556f1b4e5470 con 0x556f1b160080 2024-07-27T06:47:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.789+0000 7f92fa59c640 1 --2- 172.21.15.29:0/330179824 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556f1b160080 0x556f1b5fdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.790+0000 7f92e1ffb640 10 client.15516 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:03.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.790+0000 7f92fa59c640 1 --2- 172.21.15.29:0/330179824 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556f1b160080 0x556f1b5fdc20 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4481fbf6518303fa server_cookie=e08661afaad8c3ac in_seq=0 out_seq=0 2024-07-27T06:47:03.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f92e1ffb640 1 -- 172.21.15.29:0/330179824 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556f1b4e5470 con 0x556f1b160080 2024-07-27T06:47:03.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f92e1ffb640 10 client.15516 handle_command_reply: tid=0 2024-07-27T06:47:03.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f9306f04740 1 client.15516 shutdown 2024-07-27T06:47:03.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f9306f04740 1 -- 172.21.15.29:0/330179824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 msgr2=0x556f1b5893e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f9306f04740 1 --2- 172.21.15.29:0/330179824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 0x556f1b5893e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f92f0014010 tx=0x7f92f0012de0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f9306f04740 1 -- 172.21.15.29:0/330179824 shutdown_connections 2024-07-27T06:47:03.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f9306f04740 1 --2- 172.21.15.29:0/330179824 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556f1b160080 0x556f1b5fdc20 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f9306f04740 1 --2- 172.21.15.29:0/330179824 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556f1b1cc350 0x556f1b584dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f9306f04740 1 --2- 172.21.15.29:0/330179824 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556f1b1ab670 0x556f1b584890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f9306f04740 1 --2- 172.21.15.29:0/330179824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556f1b1a3490 0x556f1b5893e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f9306f04740 1 -- 172.21.15.29:0/330179824 >> 172.21.15.29:0/330179824 conn(0x556f1b1c0740 msgr2=0x556f1b56f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f92e1ffb640 0 client.15516 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.791+0000 7f9306f04740 1 -- 172.21.15.29:0/330179824 shutdown_connections 2024-07-27T06:47:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9306f04740 1 -- 172.21.15.29:0/330179824 wait complete. 2024-07-27T06:47:03.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9306f04740 20 client.15516 trim_cache size 0 max 16384 2024-07-27T06:47:03.797 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:47:03.797 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-07-27T06:47:03.797 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "ac591078-3819-4ade-9981-0f2249333d08", 2024-07-27T06:47:03.798 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-07-27T06:47:03.798 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:47:03.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9300980640 1 -- 172.21.15.29:0/3102492038 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f92e40b3c60 msgr2=0x7f92e40b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9300980640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f92e40b3c60 0x7f92e40b6170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f92f400dfd0 tx=0x7f92f40cd040 comp rx=0 tx=0).stop 2024-07-27T06:47:03.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9300980640 1 -- 172.21.15.29:0/3102492038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92fc106620 msgr2=0x7f92fc19f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:03.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9300980640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92fc106620 0x7f92fc19f740 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f92f4009fd0 tx=0x7f92f400b260 comp rx=0 tx=0).stop 2024-07-27T06:47:03.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9300980640 1 -- 172.21.15.29:0/3102492038 shutdown_connections 2024-07-27T06:47:03.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9300980640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92fc106620 0x7f92fc19f740 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9300980640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92fc105cd0 0x7f92fc19a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9300980640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f92e40b3c60 0x7f92e40b6170 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9300980640 1 --2- 172.21.15.29:0/3102492038 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92fc101a80 0x7f92fc199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:03.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9300980640 1 -- 172.21.15.29:0/3102492038 >> 172.21.15.29:0/3102492038 conn(0x7f92fc077690 msgr2=0x7f92fc106ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:03.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9300980640 1 -- 172.21.15.29:0/3102492038 shutdown_connections 2024-07-27T06:47:03.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:03.792+0000 7f9300980640 1 -- 172.21.15.29:0/3102492038 wait complete. 2024-07-27T06:47:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:03 smithi077 ceph-mon[37153]: pgmap v398: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-27T06:47:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:03 smithi077 ceph-mon[37153]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:47:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:03 smithi047 ceph-mon[36835]: pgmap v398: 97 pgs: 97 active+clean; 650 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-27T06:47:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:03 smithi047 ceph-mon[36835]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:47:04.055 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:04.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-27T06:47:04.725 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:04 smithi029 ceph-mon[32610]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:47:04.725 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:04 smithi029 ceph-mon[32610]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:47:04.726 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:04 smithi029 ceph-mon[32610]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:47:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:04 smithi077 ceph-mon[37153]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:47:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:04 smithi077 ceph-mon[37153]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:47:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:04 smithi077 ceph-mon[37153]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:47:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:04 smithi047 ceph-mon[36835]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:47:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:04 smithi047 ceph-mon[36835]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:47:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:04 smithi047 ceph-mon[36835]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T06:47:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:05 smithi029 ceph-mon[32610]: pgmap v399: 97 pgs: 97 active+clean; 1.0 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 92 KiB/s wr, 26 op/s 2024-07-27T06:47:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:05 smithi029 ceph-mon[32610]: scrub queued for path: / 2024-07-27T06:47:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:05 smithi029 ceph-mon[32610]: scrub summary: idle+waiting paths [/] 2024-07-27T06:47:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:05 smithi029 ceph-mon[32610]: scrub summary: active paths [/] 2024-07-27T06:47:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:05 smithi029 ceph-mon[32610]: scrub summary: idle+waiting paths [/] 2024-07-27T06:47:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:05 smithi029 ceph-mon[32610]: mgrmap e24: x(active, since 10m), standbys: y, z 2024-07-27T06:47:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:05 smithi077 ceph-mon[37153]: pgmap v399: 97 pgs: 97 active+clean; 1.0 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 92 KiB/s wr, 26 op/s 2024-07-27T06:47:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:05 smithi077 ceph-mon[37153]: scrub queued for path: / 2024-07-27T06:47:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:05 smithi077 ceph-mon[37153]: scrub summary: idle+waiting paths [/] 2024-07-27T06:47:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:05 smithi077 ceph-mon[37153]: scrub summary: active paths [/] 2024-07-27T06:47:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:05 smithi077 ceph-mon[37153]: scrub summary: idle+waiting paths [/] 2024-07-27T06:47:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:05 smithi077 ceph-mon[37153]: mgrmap e24: x(active, since 10m), standbys: y, z 2024-07-27T06:47:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:05 smithi047 ceph-mon[36835]: pgmap v399: 97 pgs: 97 active+clean; 1.0 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 92 KiB/s wr, 26 op/s 2024-07-27T06:47:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:05 smithi047 ceph-mon[36835]: scrub queued for path: / 2024-07-27T06:47:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:05 smithi047 ceph-mon[36835]: scrub summary: idle+waiting paths [/] 2024-07-27T06:47:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:05 smithi047 ceph-mon[36835]: scrub summary: active paths [/] 2024-07-27T06:47:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:05 smithi047 ceph-mon[36835]: scrub summary: idle+waiting paths [/] 2024-07-27T06:47:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:05 smithi047 ceph-mon[36835]: mgrmap e24: x(active, since 10m), standbys: y, z 2024-07-27T06:47:05.968 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:07.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.220+0000 7fe0bcdba640 1 -- 172.21.15.29:0/888029929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0b8102510 msgr2=0x7fe0b8102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:07.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.220+0000 7fe0bcdba640 1 --2- 172.21.15.29:0/888029929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0b8102510 0x7fe0b8102920 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009900 tx=0x7fe0a802a410 comp rx=0 tx=0).stop 2024-07-27T06:47:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.221+0000 7fe0bcdba640 1 -- 172.21.15.29:0/888029929 shutdown_connections 2024-07-27T06:47:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.221+0000 7fe0bcdba640 1 --2- 172.21.15.29:0/888029929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0b8108750 0x7fe0b810ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.221+0000 7fe0bcdba640 1 --2- 172.21.15.29:0/888029929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b8107e00 0x7fe0b8108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.221+0000 7fe0bcdba640 1 --2- 172.21.15.29:0/888029929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0b8102510 0x7fe0b8102920 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:07.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.221+0000 7fe0bcdba640 1 -- 172.21.15.29:0/888029929 >> 172.21.15.29:0/888029929 conn(0x7fe0b8079190 msgr2=0x7fe0b80795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.221+0000 7fe0bcdba640 1 -- 172.21.15.29:0/888029929 shutdown_connections 2024-07-27T06:47:07.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.221+0000 7fe0bcdba640 1 -- 172.21.15.29:0/888029929 wait complete. 2024-07-27T06:47:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.222+0000 7fe0bcdba640 1 Processor -- start 2024-07-27T06:47:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.222+0000 7fe0bcdba640 1 -- start start 2024-07-27T06:47:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.222+0000 7fe0bcdba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0b8102510 0x7fe0b819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.222+0000 7fe0bcdba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b8107e00 0x7fe0b819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0bcdba640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0b8108750 0x7fe0b819b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0bcdba640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81123d0 con 0x7fe0b8102510 2024-07-27T06:47:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0bcdba640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b8112250 con 0x7fe0b8108750 2024-07-27T06:47:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0bcdba640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b8112550 con 0x7fe0b8107e00 2024-07-27T06:47:07.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0b659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0b8102510 0x7fe0b819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:07.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0b659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0b8102510 0x7fe0b819ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36716/0 (socket says 172.21.15.29:36716) 2024-07-27T06:47:07.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0b659c640 1 -- 172.21.15.29:0/1810008475 learned_addr learned my addr 172.21.15.29:0/1810008475 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:07.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0b6d9d640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0b8108750 0x7fe0b819b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:07.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0b5d9b640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b8107e00 0x7fe0b819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:07.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0b659c640 1 -- 172.21.15.29:0/1810008475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b8107e00 msgr2=0x7fe0b819b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:07.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0b659c640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b8107e00 0x7fe0b819b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:07.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0b659c640 1 -- 172.21.15.29:0/1810008475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0b8108750 msgr2=0x7fe0b819b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:07.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0b659c640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0b8108750 0x7fe0b819b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:07.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0b659c640 1 -- 172.21.15.29:0/1810008475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b81a5fd0 con 0x7fe0b8102510 2024-07-27T06:47:07.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0b659c640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0b8102510 0x7fe0b819ad40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8005940 tx=0x7fe0a80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:07.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.223+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1810008475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a803f030 con 0x7fe0b8102510 2024-07-27T06:47:07.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.224+0000 7fe0bcdba640 1 -- 172.21.15.29:0/1810008475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b8198b30 con 0x7fe0b8102510 2024-07-27T06:47:07.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.224+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1810008475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0a80044c0 con 0x7fe0b8102510 2024-07-27T06:47:07.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.224+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1810008475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8035070 con 0x7fe0b8102510 2024-07-27T06:47:07.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.224+0000 7fe0bcdba640 1 -- 172.21.15.29:0/1810008475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b8199010 con 0x7fe0b8102510 2024-07-27T06:47:07.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.225+0000 7fe0bcdba640 1 -- 172.21.15.29:0/1810008475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0b8103740 con 0x7fe0b8102510 2024-07-27T06:47:07.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.226+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1810008475 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7fe0a8042030 con 0x7fe0b8102510 2024-07-27T06:47:07.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.227+0000 7fe0af7fe640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe0a40b3ce0 0x7fe0a40b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:07.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.227+0000 7fe0b5d9b640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe0a40b3ce0 0x7fe0a40b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:07.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.227+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1810008475 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fe0a8104590 con 0x7fe0b8102510 2024-07-27T06:47:07.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.228+0000 7fe0b5d9b640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe0a40b3ce0 0x7fe0a40b61f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe0b81006f0 tx=0x7fe0a0005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:07.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.231+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1810008475 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe0a80cb910 con 0x7fe0b8102510 2024-07-27T06:47:07.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.414+0000 7fe0bcdba640 1 -- 172.21.15.29:0/1810008475 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe0b8063230 con 0x7fe0a40b3ce0 2024-07-27T06:47:07.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.419+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1810008475 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259296 (secure 0 0 0) 0x7fe0b8063230 con 0x7fe0a40b3ce0 2024-07-27T06:47:07.422 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:47:07.423 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-07-27T06:47:07.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.423+0000 7fe0bcdba640 1 -- 172.21.15.29:0/1810008475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe0a40b3ce0 msgr2=0x7fe0a40b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:07.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.423+0000 7fe0bcdba640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe0a40b3ce0 0x7fe0a40b61f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe0b81006f0 tx=0x7fe0a0005f90 comp rx=0 tx=0).stop 2024-07-27T06:47:07.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.423+0000 7fe0bcdba640 1 -- 172.21.15.29:0/1810008475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0b8102510 msgr2=0x7fe0b819ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.423+0000 7fe0bcdba640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0b8102510 0x7fe0b819ad40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8005940 tx=0x7fe0a80040c0 comp rx=0 tx=0).stop 2024-07-27T06:47:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.423+0000 7fe0bcdba640 1 -- 172.21.15.29:0/1810008475 shutdown_connections 2024-07-27T06:47:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.423+0000 7fe0bcdba640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0b8108750 0x7fe0b819b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.423+0000 7fe0bcdba640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe0a40b3ce0 0x7fe0a40b61f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.423+0000 7fe0bcdba640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b8107e00 0x7fe0b819b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.423+0000 7fe0bcdba640 1 --2- 172.21.15.29:0/1810008475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0b8102510 0x7fe0b819ad40 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.423+0000 7fe0bcdba640 1 -- 172.21.15.29:0/1810008475 >> 172.21.15.29:0/1810008475 conn(0x7fe0b8079190 msgr2=0x7fe0b8100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.423+0000 7fe0bcdba640 1 -- 172.21.15.29:0/1810008475 shutdown_connections 2024-07-27T06:47:07.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:07.423+0000 7fe0bcdba640 1 -- 172.21.15.29:0/1810008475 wait complete. 2024-07-27T06:47:07.693 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:07 smithi029 ceph-mon[32610]: pgmap v400: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 163 KiB/s wr, 32 op/s 2024-07-27T06:47:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:07 smithi077 ceph-mon[37153]: pgmap v400: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 163 KiB/s wr, 32 op/s 2024-07-27T06:47:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:07 smithi047 ceph-mon[36835]: pgmap v400: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 163 KiB/s wr, 32 op/s 2024-07-27T06:47:08.570 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-07-27T06:47:05.685593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1452197,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2320,"num_read_kb":1334,"num_write":531,"num_write_kb":4269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2953401,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":922,"ondisk_log_size":922,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360648,"kb_used_data":11456,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581176,"statfs":{"total":1151940427776,"available":1151571124224,"internally_reserved":0,"allocated":11730944,"data_stored":9091939,"data_compressed":322095,"data_compressed_allocated":430080,"data_compressed_original":3686400,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":786992,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":249,"num_read_kb":169,"num_write":142,"num_write_kb":1960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003183"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483739+0000","last_change":"2024-07-27T06:43:12.802353+0000","last_active":"2024-07-27T06:45:09.483739+0000","last_peered":"2024-07-27T06:45:09.483739+0000","last_clean":"2024-07-27T06:45:09.483739+0000","last_became_active":"2024-07-27T06:43:12.802028+0000","last_became_peered":"2024-07-27T06:43:12.802028+0000","last_unstale":"2024-07-27T06:45:09.483739+0000","last_undegraded":"2024-07-27T06:45:09.483739+0000","last_fullsized":"2024-07-27T06:45:09.483739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:22:24.292567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862408+0000","last_change":"2024-07-27T06:43:12.806269+0000","last_active":"2024-07-27T06:45:09.862408+0000","last_peered":"2024-07-27T06:45:09.862408+0000","last_clean":"2024-07-27T06:45:09.862408+0000","last_became_active":"2024-07-27T06:43:12.803232+0000","last_became_peered":"2024-07-27T06:43:12.803232+0000","last_unstale":"2024-07-27T06:45:09.862408+0000","last_undegraded":"2024-07-27T06:45:09.862408+0000","last_fullsized":"2024-07-27T06:45:09.862408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:57:53.673850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862259+0000","last_change":"2024-07-27T06:43:12.805542+0000","last_active":"2024-07-27T06:45:09.862259+0000","last_peered":"2024-07-27T06:45:09.862259+0000","last_clean":"2024-07-27T06:45:09.862259+0000","last_became_active":"2024-07-27T06:43:12.804951+0000","last_became_peered":"2024-07-27T06:43:12.804951+0000","last_unstale":"2024-07-27T06:45:09.862259+0000","last_undegraded":"2024-07-27T06:45:09.862259+0000","last_fullsized":"2024-07-27T06:45:09.862259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:06:40.469695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862158+0000","last_change":"2024-07-27T06:43:12.796950+0000","last_active":"2024-07-27T06:45:09.862158+0000","last_peered":"2024-07-27T06:45:09.862158+0000","last_clean":"2024-07-27T06:45:09.862158+0000","last_became_active":"2024-07-27T06:43:12.796549+0000","last_became_peered":"2024-07-27T06:43:12.796549+0000","last_unstale":"2024-07-27T06:45:09.862158+0000","last_undegraded":"2024-07-27T06:45:09.862158+0000","last_fullsized":"2024-07-27T06:45:09.862158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:30:36.391046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486015+0000","last_change":"2024-07-27T06:43:12.799838+0000","last_active":"2024-07-27T06:45:09.486015+0000","last_peered":"2024-07-27T06:45:09.486015+0000","last_clean":"2024-07-27T06:45:09.486015+0000","last_became_active":"2024-07-27T06:43:12.799462+0000","last_became_peered":"2024-07-27T06:43:12.799462+0000","last_unstale":"2024-07-27T06:45:09.486015+0000","last_undegraded":"2024-07-27T06:45:09.486015+0000","last_fullsized":"2024-07-27T06:45:09.486015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:45:19.371259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481631+0000","last_change":"2024-07-27T06:43:12.797187+0000","last_active":"2024-07-27T06:45:09.481631+0000","last_peered":"2024-07-27T06:45:09.481631+0000","last_clean":"2024-07-27T06:45:09.481631+0000","last_became_active":"2024-07-27T06:43:12.796633+0000","last_became_peered":"2024-07-27T06:43:12.796633+0000","last_unstale":"2024-07-27T06:45:09.481631+0000","last_undegraded":"2024-07-27T06:45:09.481631+0000","last_fullsized":"2024-07-27T06:45:09.481631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:22:20.882899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488423+0000","last_change":"2024-07-27T06:43:12.793786+0000","last_active":"2024-07-27T06:45:09.488423+0000","last_peered":"2024-07-27T06:45:09.488423+0000","last_clean":"2024-07-27T06:45:09.488423+0000","last_became_active":"2024-07-27T06:43:12.793413+0000","last_became_peered":"2024-07-27T06:43:12.793413+0000","last_unstale":"2024-07-27T06:45:09.488423+0000","last_undegraded":"2024-07-27T06:45:09.488423+0000","last_fullsized":"2024-07-27T06:45:09.488423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:14.015373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485385+0000","last_change":"2024-07-27T06:43:12.791304+0000","last_active":"2024-07-27T06:45:09.485385+0000","last_peered":"2024-07-27T06:45:09.485385+0000","last_clean":"2024-07-27T06:45:09.485385+0000","last_became_active":"2024-07-27T06:43:12.790801+0000","last_became_peered":"2024-07-27T06:43:12.790801+0000","last_unstale":"2024-07-27T06:45:09.485385+0000","last_undegraded":"2024-07-27T06:45:09.485385+0000","last_fullsized":"2024-07-27T06:45:09.485385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:12:37.606748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822529+0000","last_change":"2024-07-27T06:43:12.797778+0000","last_active":"2024-07-27T06:45:09.822529+0000","last_peered":"2024-07-27T06:45:09.822529+0000","last_clean":"2024-07-27T06:45:09.822529+0000","last_became_active":"2024-07-27T06:43:12.796784+0000","last_became_peered":"2024-07-27T06:43:12.796784+0000","last_unstale":"2024-07-27T06:45:09.822529+0000","last_undegraded":"2024-07-27T06:45:09.822529+0000","last_fullsized":"2024-07-27T06:45:09.822529+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:52:22.275282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.482109+0000","last_change":"2024-07-27T06:43:12.798970+0000","last_active":"2024-07-27T06:45:09.482109+0000","last_peered":"2024-07-27T06:45:09.482109+0000","last_clean":"2024-07-27T06:45:09.482109+0000","last_became_active":"2024-07-27T06:43:12.798549+0000","last_became_peered":"2024-07-27T06:43:12.798549+0000","last_unstale":"2024-07-27T06:45:09.482109+0000","last_undegraded":"2024-07-27T06:45:09.482109+0000","last_fullsized":"2024-07-27T06:45:09.482109+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:58:32.464429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488604+0000","last_change":"2024-07-27T06:43:12.809405+0000","last_active":"2024-07-27T06:45:09.488604+0000","last_peered":"2024-07-27T06:45:09.488604+0000","last_clean":"2024-07-27T06:45:09.488604+0000","last_became_active":"2024-07-27T06:43:12.809030+0000","last_became_peered":"2024-07-27T06:43:12.809030+0000","last_unstale":"2024-07-27T06:45:09.488604+0000","last_undegraded":"2024-07-27T06:45:09.488604+0000","last_fullsized":"2024-07-27T06:45:09.488604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:46:35.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862193+0000","last_change":"2024-07-27T06:43:12.802647+0000","last_active":"2024-07-27T06:45:09.862193+0000","last_peered":"2024-07-27T06:45:09.862193+0000","last_clean":"2024-07-27T06:45:09.862193+0000","last_became_active":"2024-07-27T06:43:12.802247+0000","last_became_peered":"2024-07-27T06:43:12.802247+0000","last_unstale":"2024-07-27T06:45:09.862193+0000","last_undegraded":"2024-07-27T06:45:09.862193+0000","last_fullsized":"2024-07-27T06:45:09.862193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:44:24.564852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481628+0000","last_change":"2024-07-27T06:43:12.809140+0000","last_active":"2024-07-27T06:45:09.481628+0000","last_peered":"2024-07-27T06:45:09.481628+0000","last_clean":"2024-07-27T06:45:09.481628+0000","last_became_active":"2024-07-27T06:43:12.808692+0000","last_became_peered":"2024-07-27T06:43:12.808692+0000","last_unstale":"2024-07-27T06:45:09.481628+0000","last_undegraded":"2024-07-27T06:45:09.481628+0000","last_fullsized":"2024-07-27T06:45:09.481628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:37:45.326234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862226+0000","last_change":"2024-07-27T06:43:12.802838+0000","last_active":"2024-07-27T06:45:09.862226+0000","last_peered":"2024-07-27T06:45:09.862226+0000","last_clean":"2024-07-27T06:45:09.862226+0000","last_became_active":"2024-07-27T06:43:12.802503+0000","last_became_peered":"2024-07-27T06:43:12.802503+0000","last_unstale":"2024-07-27T06:45:09.862226+0000","last_undegraded":"2024-07-27T06:45:09.862226+0000","last_fullsized":"2024-07-27T06:45:09.862226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:41:03.428162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862170+0000","last_change":"2024-07-27T06:43:12.801626+0000","last_active":"2024-07-27T06:45:09.862170+0000","last_peered":"2024-07-27T06:45:09.862170+0000","last_clean":"2024-07-27T06:45:09.862170+0000","last_became_active":"2024-07-27T06:43:12.801266+0000","last_became_peered":"2024-07-27T06:43:12.801266+0000","last_unstale":"2024-07-27T06:45:09.862170+0000","last_undegraded":"2024-07-27T06:45:09.862170+0000","last_fullsized":"2024-07-27T06:45:09.862170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:54:31.934223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.821975+0000","last_change":"2024-07-27T06:43:12.804543+0000","last_active":"2024-07-27T06:45:09.821975+0000","last_peered":"2024-07-27T06:45:09.821975+0000","last_clean":"2024-07-27T06:45:09.821975+0000","last_became_active":"2024-07-27T06:43:12.804173+0000","last_became_peered":"2024-07-27T06:43:12.804173+0000","last_unstale":"2024-07-27T06:45:09.821975+0000","last_undegraded":"2024-07-27T06:45:09.821975+0000","last_fullsized":"2024-07-27T06:45:09.821975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:18:25.337333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483797+0000","last_change":"2024-07-27T06:43:12.798604+0000","last_active":"2024-07-27T06:45:09.483797+0000","last_peered":"2024-07-27T06:45:09.483797+0000","last_clean":"2024-07-27T06:45:09.483797+0000","last_became_active":"2024-07-27T06:43:12.798228+0000","last_became_peered":"2024-07-27T06:43:12.798228+0000","last_unstale":"2024-07-27T06:45:09.483797+0000","last_undegraded":"2024-07-27T06:45:09.483797+0000","last_fullsized":"2024-07-27T06:45:09.483797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:04:35.851399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633341+0000","last_change":"2024-07-27T06:43:12.800452+0000","last_active":"2024-07-27T06:45:09.633341+0000","last_peered":"2024-07-27T06:45:09.633341+0000","last_clean":"2024-07-27T06:45:09.633341+0000","last_became_active":"2024-07-27T06:43:12.800084+0000","last_became_peered":"2024-07-27T06:43:12.800084+0000","last_unstale":"2024-07-27T06:45:09.633341+0000","last_undegraded":"2024-07-27T06:45:09.633341+0000","last_fullsized":"2024-07-27T06:45:09.633341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:54:27.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485773+0000","last_change":"2024-07-27T06:43:12.800761+0000","last_active":"2024-07-27T06:45:09.485773+0000","last_peered":"2024-07-27T06:45:09.485773+0000","last_clean":"2024-07-27T06:45:09.485773+0000","last_became_active":"2024-07-27T06:43:12.800414+0000","last_became_peered":"2024-07-27T06:43:12.800414+0000","last_unstale":"2024-07-27T06:45:09.485773+0000","last_undegraded":"2024-07-27T06:45:09.485773+0000","last_fullsized":"2024-07-27T06:45:09.485773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:02:47.247522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483375+0000","last_change":"2024-07-27T06:43:12.795470+0000","last_active":"2024-07-27T06:45:09.483375+0000","last_peered":"2024-07-27T06:45:09.483375+0000","last_clean":"2024-07-27T06:45:09.483375+0000","last_became_active":"2024-07-27T06:43:12.795136+0000","last_became_peered":"2024-07-27T06:43:12.795136+0000","last_unstale":"2024-07-27T06:45:09.483375+0000","last_undegraded":"2024-07-27T06:45:09.483375+0000","last_fullsized":"2024-07-27T06:45:09.483375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:01:22.178699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481748+0000","last_change":"2024-07-27T06:43:12.803897+0000","last_active":"2024-07-27T06:45:09.481748+0000","last_peered":"2024-07-27T06:45:09.481748+0000","last_clean":"2024-07-27T06:45:09.481748+0000","last_became_active":"2024-07-27T06:43:12.803546+0000","last_became_peered":"2024-07-27T06:43:12.803546+0000","last_unstale":"2024-07-27T06:45:09.481748+0000","last_undegraded":"2024-07-27T06:45:09.481748+0000","last_fullsized":"2024-07-27T06:45:09.481748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:13:50.816831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822201+0000","last_change":"2024-07-27T06:44:09.238514+0000","last_active":"2024-07-27T06:45:09.822201+0000","last_peered":"2024-07-27T06:45:09.822201+0000","last_clean":"2024-07-27T06:45:09.822201+0000","last_became_active":"2024-07-27T06:44:09.237903+0000","last_became_peered":"2024-07-27T06:44:09.237903+0000","last_unstale":"2024-07-27T06:45:09.822201+0000","last_undegraded":"2024-07-27T06:45:09.822201+0000","last_fullsized":"2024-07-27T06:45:09.822201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:20:59.887491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485428+0000","last_change":"2024-07-27T06:43:12.788570+0000","last_active":"2024-07-27T06:45:09.485428+0000","last_peered":"2024-07-27T06:45:09.485428+0000","last_clean":"2024-07-27T06:45:09.485428+0000","last_became_active":"2024-07-27T06:43:12.788199+0000","last_became_peered":"2024-07-27T06:43:12.788199+0000","last_unstale":"2024-07-27T06:45:09.485428+0000","last_undegraded":"2024-07-27T06:45:09.485428+0000","last_fullsized":"2024-07-27T06:45:09.485428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:44:26.751094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862750+0000","last_change":"2024-07-27T06:43:12.792866+0000","last_active":"2024-07-27T06:45:09.862750+0000","last_peered":"2024-07-27T06:45:09.862750+0000","last_clean":"2024-07-27T06:45:09.862750+0000","last_became_active":"2024-07-27T06:43:12.792389+0000","last_became_peered":"2024-07-27T06:43:12.792389+0000","last_unstale":"2024-07-27T06:45:09.862750+0000","last_undegraded":"2024-07-27T06:45:09.862750+0000","last_fullsized":"2024-07-27T06:45:09.862750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:10:37.367642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485830+0000","last_change":"2024-07-27T06:43:12.797002+0000","last_active":"2024-07-27T06:45:09.485830+0000","last_peered":"2024-07-27T06:45:09.485830+0000","last_clean":"2024-07-27T06:45:09.485830+0000","last_became_active":"2024-07-27T06:43:12.796651+0000","last_became_peered":"2024-07-27T06:43:12.796651+0000","last_unstale":"2024-07-27T06:45:09.485830+0000","last_undegraded":"2024-07-27T06:45:09.485830+0000","last_fullsized":"2024-07-27T06:45:09.485830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:28:06.934042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485867+0000","last_change":"2024-07-27T06:43:12.798549+0000","last_active":"2024-07-27T06:45:09.485867+0000","last_peered":"2024-07-27T06:45:09.485867+0000","last_clean":"2024-07-27T06:45:09.485867+0000","last_became_active":"2024-07-27T06:43:12.798201+0000","last_became_peered":"2024-07-27T06:43:12.798201+0000","last_unstale":"2024-07-27T06:45:09.485867+0000","last_undegraded":"2024-07-27T06:45:09.485867+0000","last_fullsized":"2024-07-27T06:45:09.485867+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:59:47.525536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483656+0000","last_change":"2024-07-27T06:43:12.791053+0000","last_active":"2024-07-27T06:45:09.483656+0000","last_peered":"2024-07-27T06:45:09.483656+0000","last_clean":"2024-07-27T06:45:09.483656+0000","last_became_active":"2024-07-27T06:43:12.790718+0000","last_became_peered":"2024-07-27T06:43:12.790718+0000","last_unstale":"2024-07-27T06:45:09.483656+0000","last_undegraded":"2024-07-27T06:45:09.483656+0000","last_fullsized":"2024-07-27T06:45:09.483656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:09.455399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481811+0000","last_change":"2024-07-27T06:43:12.804180+0000","last_active":"2024-07-27T06:45:09.481811+0000","last_peered":"2024-07-27T06:45:09.481811+0000","last_clean":"2024-07-27T06:45:09.481811+0000","last_became_active":"2024-07-27T06:43:12.803881+0000","last_became_peered":"2024-07-27T06:43:12.803881+0000","last_unstale":"2024-07-27T06:45:09.481811+0000","last_undegraded":"2024-07-27T06:45:09.481811+0000","last_fullsized":"2024-07-27T06:45:09.481811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:01:05.009233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862727+0000","last_change":"2024-07-27T06:43:12.809150+0000","last_active":"2024-07-27T06:45:09.862727+0000","last_peered":"2024-07-27T06:45:09.862727+0000","last_clean":"2024-07-27T06:45:09.862727+0000","last_became_active":"2024-07-27T06:43:12.808428+0000","last_became_peered":"2024-07-27T06:43:12.808428+0000","last_unstale":"2024-07-27T06:45:09.862727+0000","last_undegraded":"2024-07-27T06:45:09.862727+0000","last_fullsized":"2024-07-27T06:45:09.862727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:07:13.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486325+0000","last_change":"2024-07-27T06:44:09.233785+0000","last_active":"2024-07-27T06:45:09.486325+0000","last_peered":"2024-07-27T06:45:09.486325+0000","last_clean":"2024-07-27T06:45:09.486325+0000","last_became_active":"2024-07-27T06:44:09.233184+0000","last_became_peered":"2024-07-27T06:44:09.233184+0000","last_unstale":"2024-07-27T06:45:09.486325+0000","last_undegraded":"2024-07-27T06:45:09.486325+0000","last_fullsized":"2024-07-27T06:45:09.486325+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:12:48.556383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486129+0000","last_change":"2024-07-27T06:44:09.320182+0000","last_active":"2024-07-27T06:45:09.486129+0000","last_peered":"2024-07-27T06:45:09.486129+0000","last_clean":"2024-07-27T06:45:09.486129+0000","last_became_active":"2024-07-27T06:44:09.319661+0000","last_became_peered":"2024-07-27T06:44:09.319661+0000","last_unstale":"2024-07-27T06:45:09.486129+0000","last_undegraded":"2024-07-27T06:45:09.486129+0000","last_fullsized":"2024-07-27T06:45:09.486129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:52:11.006680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483342+0000","last_change":"2024-07-27T06:43:12.787866+0000","last_active":"2024-07-27T06:45:09.483342+0000","last_peered":"2024-07-27T06:45:09.483342+0000","last_clean":"2024-07-27T06:45:09.483342+0000","last_became_active":"2024-07-27T06:43:12.787513+0000","last_became_peered":"2024-07-27T06:43:12.787513+0000","last_unstale":"2024-07-27T06:45:09.483342+0000","last_undegraded":"2024-07-27T06:45:09.483342+0000","last_fullsized":"2024-07-27T06:45:09.483342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:48:57.876588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485381+0000","last_change":"2024-07-27T06:43:12.791158+0000","last_active":"2024-07-27T06:45:09.485381+0000","last_peered":"2024-07-27T06:45:09.485381+0000","last_clean":"2024-07-27T06:45:09.485381+0000","last_became_active":"2024-07-27T06:43:12.790771+0000","last_became_peered":"2024-07-27T06:43:12.790771+0000","last_unstale":"2024-07-27T06:45:09.485381+0000","last_undegraded":"2024-07-27T06:45:09.485381+0000","last_fullsized":"2024-07-27T06:45:09.485381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:10:18.739265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'7","reported_seq":254,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:03.217126+0000","last_change":"2024-07-27T06:43:09.769281+0000","last_active":"2024-07-27T06:47:03.217126+0000","last_peered":"2024-07-27T06:47:03.217126+0000","last_clean":"2024-07-27T06:47:03.217126+0000","last_became_active":"2024-07-27T06:43:09.768864+0000","last_became_peered":"2024-07-27T06:43:09.768864+0000","last_unstale":"2024-07-27T06:47:03.217126+0000","last_undegraded":"2024-07-27T06:47:03.217126+0000","last_fullsized":"2024-07-27T06:47:03.217126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:55:14.191180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":215,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862960+0000","last_change":"2024-07-27T06:44:09.319930+0000","last_active":"2024-07-27T06:45:09.862960+0000","last_peered":"2024-07-27T06:45:09.862960+0000","last_clean":"2024-07-27T06:45:09.862960+0000","last_became_active":"2024-07-27T06:44:09.319535+0000","last_became_peered":"2024-07-27T06:44:09.319535+0000","last_unstale":"2024-07-27T06:45:09.862960+0000","last_undegraded":"2024-07-27T06:45:09.862960+0000","last_fullsized":"2024-07-27T06:45:09.862960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:39:52.887860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481674+0000","last_change":"2024-07-27T06:43:09.772227+0000","last_active":"2024-07-27T06:45:09.481674+0000","last_peered":"2024-07-27T06:45:09.481674+0000","last_clean":"2024-07-27T06:45:09.481674+0000","last_became_active":"2024-07-27T06:43:09.771812+0000","last_became_peered":"2024-07-27T06:43:09.771812+0000","last_unstale":"2024-07-27T06:45:09.481674+0000","last_undegraded":"2024-07-27T06:45:09.481674+0000","last_fullsized":"2024-07-27T06:45:09.481674+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:27:28.054590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481030+0000","last_change":"2024-07-27T06:43:09.766490+0000","last_active":"2024-07-27T06:45:09.481030+0000","last_peered":"2024-07-27T06:45:09.481030+0000","last_clean":"2024-07-27T06:45:09.481030+0000","last_became_active":"2024-07-27T06:43:09.765973+0000","last_became_peered":"2024-07-27T06:43:09.765973+0000","last_unstale":"2024-07-27T06:45:09.481030+0000","last_undegraded":"2024-07-27T06:45:09.481030+0000","last_fullsized":"2024-07-27T06:45:09.481030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:00:02.913080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485458+0000","last_change":"2024-07-27T06:43:12.805234+0000","last_active":"2024-07-27T06:45:09.485458+0000","last_peered":"2024-07-27T06:45:09.485458+0000","last_clean":"2024-07-27T06:45:09.485458+0000","last_became_active":"2024-07-27T06:43:12.804908+0000","last_became_peered":"2024-07-27T06:43:12.804908+0000","last_unstale":"2024-07-27T06:45:09.485458+0000","last_undegraded":"2024-07-27T06:45:09.485458+0000","last_fullsized":"2024-07-27T06:45:09.485458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:09:34.988755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862248+0000","last_change":"2024-07-27T06:43:09.774201+0000","last_active":"2024-07-27T06:45:09.862248+0000","last_peered":"2024-07-27T06:45:09.862248+0000","last_clean":"2024-07-27T06:45:09.862248+0000","last_became_active":"2024-07-27T06:43:09.773817+0000","last_became_peered":"2024-07-27T06:43:09.773817+0000","last_unstale":"2024-07-27T06:45:09.862248+0000","last_undegraded":"2024-07-27T06:45:09.862248+0000","last_fullsized":"2024-07-27T06:45:09.862248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:06.166251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633110+0000","last_change":"2024-07-27T06:43:12.800174+0000","last_active":"2024-07-27T06:45:09.633110+0000","last_peered":"2024-07-27T06:45:09.633110+0000","last_clean":"2024-07-27T06:45:09.633110+0000","last_became_active":"2024-07-27T06:43:12.799769+0000","last_became_peered":"2024-07-27T06:43:12.799769+0000","last_unstale":"2024-07-27T06:45:09.633110+0000","last_undegraded":"2024-07-27T06:45:09.633110+0000","last_fullsized":"2024-07-27T06:45:09.633110+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:43:06.890372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":455,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:01.581775+0000","last_change":"2024-07-27T06:43:09.776612+0000","last_active":"2024-07-27T06:47:01.581775+0000","last_peered":"2024-07-27T06:47:01.581775+0000","last_clean":"2024-07-27T06:47:01.581775+0000","last_became_active":"2024-07-27T06:43:09.775761+0000","last_became_peered":"2024-07-27T06:43:09.775761+0000","last_unstale":"2024-07-27T06:47:01.581775+0000","last_undegraded":"2024-07-27T06:47:01.581775+0000","last_fullsized":"2024-07-27T06:47:01.581775+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:42:52.132357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":417,"num_read_kb":417,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822478+0000","last_change":"2024-07-27T06:43:12.797200+0000","last_active":"2024-07-27T06:45:09.822478+0000","last_peered":"2024-07-27T06:45:09.822478+0000","last_clean":"2024-07-27T06:45:09.822478+0000","last_became_active":"2024-07-27T06:43:12.796288+0000","last_became_peered":"2024-07-27T06:43:12.796288+0000","last_unstale":"2024-07-27T06:45:09.822478+0000","last_undegraded":"2024-07-27T06:45:09.822478+0000","last_fullsized":"2024-07-27T06:45:09.822478+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:31:27.603883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488738+0000","last_change":"2024-07-27T06:43:09.772658+0000","last_active":"2024-07-27T06:45:09.488738+0000","last_peered":"2024-07-27T06:45:09.488738+0000","last_clean":"2024-07-27T06:45:09.488738+0000","last_became_active":"2024-07-27T06:43:09.772277+0000","last_became_peered":"2024-07-27T06:43:09.772277+0000","last_unstale":"2024-07-27T06:45:09.488738+0000","last_undegraded":"2024-07-27T06:45:09.488738+0000","last_fullsized":"2024-07-27T06:45:09.488738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:09:16.854718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862551+0000","last_change":"2024-07-27T06:43:12.809643+0000","last_active":"2024-07-27T06:45:09.862551+0000","last_peered":"2024-07-27T06:45:09.862551+0000","last_clean":"2024-07-27T06:45:09.862551+0000","last_became_active":"2024-07-27T06:43:12.809020+0000","last_became_peered":"2024-07-27T06:43:12.809020+0000","last_unstale":"2024-07-27T06:45:09.862551+0000","last_undegraded":"2024-07-27T06:45:09.862551+0000","last_fullsized":"2024-07-27T06:45:09.862551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:08:03.062914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.863022+0000","last_change":"2024-07-27T06:43:10.534042+0000","last_active":"2024-07-27T06:45:09.863022+0000","last_peered":"2024-07-27T06:45:09.863022+0000","last_clean":"2024-07-27T06:45:09.863022+0000","last_became_active":"2024-07-27T06:43:10.533615+0000","last_became_peered":"2024-07-27T06:43:10.533615+0000","last_unstale":"2024-07-27T06:45:09.863022+0000","last_undegraded":"2024-07-27T06:45:09.863022+0000","last_fullsized":"2024-07-27T06:45:09.863022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:09:32.235633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822222+0000","last_change":"2024-07-27T06:43:12.798764+0000","last_active":"2024-07-27T06:45:09.822222+0000","last_peered":"2024-07-27T06:45:09.822222+0000","last_clean":"2024-07-27T06:45:09.822222+0000","last_became_active":"2024-07-27T06:43:12.797500+0000","last_became_peered":"2024-07-27T06:43:12.797500+0000","last_unstale":"2024-07-27T06:45:09.822222+0000","last_undegraded":"2024-07-27T06:45:09.822222+0000","last_fullsized":"2024-07-27T06:45:09.822222+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:12:09.368711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"83'37","reported_seq":499,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:02.214898+0000","last_change":"2024-07-27T06:43:09.775954+0000","last_active":"2024-07-27T06:47:02.214898+0000","last_peered":"2024-07-27T06:47:02.214898+0000","last_clean":"2024-07-27T06:47:02.214898+0000","last_became_active":"2024-07-27T06:43:09.775498+0000","last_became_peered":"2024-07-27T06:43:09.775498+0000","last_unstale":"2024-07-27T06:47:02.214898+0000","last_undegraded":"2024-07-27T06:47:02.214898+0000","last_fullsized":"2024-07-27T06:47:02.214898+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:08:46.965525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66277,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":430,"num_read_kb":280,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486559+0000","last_change":"2024-07-27T06:43:12.797220+0000","last_active":"2024-07-27T06:45:09.486559+0000","last_peered":"2024-07-27T06:45:09.486559+0000","last_clean":"2024-07-27T06:45:09.486559+0000","last_became_active":"2024-07-27T06:43:12.796889+0000","last_became_peered":"2024-07-27T06:43:12.796889+0000","last_unstale":"2024-07-27T06:45:09.486559+0000","last_undegraded":"2024-07-27T06:45:09.486559+0000","last_fullsized":"2024-07-27T06:45:09.486559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:24:36.275738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485662+0000","last_change":"2024-07-27T06:43:09.776600+0000","last_active":"2024-07-27T06:45:09.485662+0000","last_peered":"2024-07-27T06:45:09.485662+0000","last_clean":"2024-07-27T06:45:09.485662+0000","last_became_active":"2024-07-27T06:43:09.776163+0000","last_became_peered":"2024-07-27T06:43:09.776163+0000","last_unstale":"2024-07-27T06:45:09.485662+0000","last_undegraded":"2024-07-27T06:45:09.485662+0000","last_fullsized":"2024-07-27T06:45:09.485662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:22:15.947916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481215+0000","last_change":"2024-07-27T06:43:12.804984+0000","last_active":"2024-07-27T06:45:09.481215+0000","last_peered":"2024-07-27T06:45:09.481215+0000","last_clean":"2024-07-27T06:45:09.481215+0000","last_became_active":"2024-07-27T06:43:12.804644+0000","last_became_peered":"2024-07-27T06:43:12.804644+0000","last_unstale":"2024-07-27T06:45:09.481215+0000","last_undegraded":"2024-07-27T06:45:09.481215+0000","last_fullsized":"2024-07-27T06:45:09.481215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.355981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862246+0000","last_change":"2024-07-27T06:43:09.772489+0000","last_active":"2024-07-27T06:45:09.862246+0000","last_peered":"2024-07-27T06:45:09.862246+0000","last_clean":"2024-07-27T06:45:09.862246+0000","last_became_active":"2024-07-27T06:43:09.772069+0000","last_became_peered":"2024-07-27T06:43:09.772069+0000","last_unstale":"2024-07-27T06:45:09.862246+0000","last_undegraded":"2024-07-27T06:45:09.862246+0000","last_fullsized":"2024-07-27T06:45:09.862246+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:15:30.549971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488589+0000","last_change":"2024-07-27T06:43:12.803376+0000","last_active":"2024-07-27T06:45:09.488589+0000","last_peered":"2024-07-27T06:45:09.488589+0000","last_clean":"2024-07-27T06:45:09.488589+0000","last_became_active":"2024-07-27T06:43:12.802961+0000","last_became_peered":"2024-07-27T06:43:12.802961+0000","last_unstale":"2024-07-27T06:45:09.488589+0000","last_undegraded":"2024-07-27T06:45:09.488589+0000","last_fullsized":"2024-07-27T06:45:09.488589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:00:24.649877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":251,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:04.218431+0000","last_change":"2024-07-27T06:43:09.773182+0000","last_active":"2024-07-27T06:47:04.218431+0000","last_peered":"2024-07-27T06:47:04.218431+0000","last_clean":"2024-07-27T06:47:04.218431+0000","last_became_active":"2024-07-27T06:43:09.772126+0000","last_became_peered":"2024-07-27T06:43:09.772126+0000","last_unstale":"2024-07-27T06:47:04.218431+0000","last_undegraded":"2024-07-27T06:47:04.218431+0000","last_fullsized":"2024-07-27T06:47:04.218431+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:45:26.598799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":215,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488424+0000","last_change":"2024-07-27T06:43:12.799385+0000","last_active":"2024-07-27T06:45:09.488424+0000","last_peered":"2024-07-27T06:45:09.488424+0000","last_clean":"2024-07-27T06:45:09.488424+0000","last_became_active":"2024-07-27T06:43:12.798639+0000","last_became_peered":"2024-07-27T06:43:12.798639+0000","last_unstale":"2024-07-27T06:45:09.488424+0000","last_undegraded":"2024-07-27T06:45:09.488424+0000","last_fullsized":"2024-07-27T06:45:09.488424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:56:29.462491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485303+0000","last_change":"2024-07-27T06:43:10.535429+0000","last_active":"2024-07-27T06:45:09.485303+0000","last_peered":"2024-07-27T06:45:09.485303+0000","last_clean":"2024-07-27T06:45:09.485303+0000","last_became_active":"2024-07-27T06:43:10.534952+0000","last_became_peered":"2024-07-27T06:43:10.534952+0000","last_unstale":"2024-07-27T06:45:09.485303+0000","last_undegraded":"2024-07-27T06:45:09.485303+0000","last_fullsized":"2024-07-27T06:45:09.485303+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:54:20.018263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862706+0000","last_change":"2024-07-27T06:43:12.804266+0000","last_active":"2024-07-27T06:45:09.862706+0000","last_peered":"2024-07-27T06:45:09.862706+0000","last_clean":"2024-07-27T06:45:09.862706+0000","last_became_active":"2024-07-27T06:43:12.803888+0000","last_became_peered":"2024-07-27T06:43:12.803888+0000","last_unstale":"2024-07-27T06:45:09.862706+0000","last_undegraded":"2024-07-27T06:45:09.862706+0000","last_fullsized":"2024-07-27T06:45:09.862706+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:48:00.490592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485827+0000","last_change":"2024-07-27T06:43:09.777656+0000","last_active":"2024-07-27T06:45:09.485827+0000","last_peered":"2024-07-27T06:45:09.485827+0000","last_clean":"2024-07-27T06:45:09.485827+0000","last_became_active":"2024-07-27T06:43:09.776620+0000","last_became_peered":"2024-07-27T06:43:09.776620+0000","last_unstale":"2024-07-27T06:45:09.485827+0000","last_undegraded":"2024-07-27T06:45:09.485827+0000","last_fullsized":"2024-07-27T06:45:09.485827+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:50.203623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483544+0000","last_change":"2024-07-27T06:43:12.796179+0000","last_active":"2024-07-27T06:45:09.483544+0000","last_peered":"2024-07-27T06:45:09.483544+0000","last_clean":"2024-07-27T06:45:09.483544+0000","last_became_active":"2024-07-27T06:43:12.795701+0000","last_became_peered":"2024-07-27T06:43:12.795701+0000","last_unstale":"2024-07-27T06:45:09.483544+0000","last_undegraded":"2024-07-27T06:45:09.483544+0000","last_fullsized":"2024-07-27T06:45:09.483544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:20:12.236769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481627+0000","last_change":"2024-07-27T06:43:09.769949+0000","last_active":"2024-07-27T06:45:09.481627+0000","last_peered":"2024-07-27T06:45:09.481627+0000","last_clean":"2024-07-27T06:45:09.481627+0000","last_became_active":"2024-07-27T06:43:09.769602+0000","last_became_peered":"2024-07-27T06:43:09.769602+0000","last_unstale":"2024-07-27T06:45:09.481627+0000","last_undegraded":"2024-07-27T06:45:09.481627+0000","last_fullsized":"2024-07-27T06:45:09.481627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:28:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488869+0000","last_change":"2024-07-27T06:43:12.802230+0000","last_active":"2024-07-27T06:45:09.488869+0000","last_peered":"2024-07-27T06:45:09.488869+0000","last_clean":"2024-07-27T06:45:09.488869+0000","last_became_active":"2024-07-27T06:43:12.800944+0000","last_became_peered":"2024-07-27T06:43:12.800944+0000","last_unstale":"2024-07-27T06:45:09.488869+0000","last_undegraded":"2024-07-27T06:45:09.488869+0000","last_fullsized":"2024-07-27T06:45:09.488869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:05:54.585955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'62","reported_seq":124,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:03.049948+0000","last_change":"2024-07-27T06:43:09.768239+0000","last_active":"2024-07-27T06:47:03.049948+0000","last_peered":"2024-07-27T06:47:03.049948+0000","last_clean":"2024-07-27T06:47:03.049948+0000","last_became_active":"2024-07-27T06:43:09.767584+0000","last_became_peered":"2024-07-27T06:43:09.767584+0000","last_unstale":"2024-07-27T06:47:03.049948+0000","last_undegraded":"2024-07-27T06:47:03.049948+0000","last_fullsized":"2024-07-27T06:47:03.049948+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:35:39.552246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393846,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":32,"num_write":71,"num_write_kb":873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488571+0000","last_change":"2024-07-27T06:43:12.803779+0000","last_active":"2024-07-27T06:45:09.488571+0000","last_peered":"2024-07-27T06:45:09.488571+0000","last_clean":"2024-07-27T06:45:09.488571+0000","last_became_active":"2024-07-27T06:43:12.803359+0000","last_became_peered":"2024-07-27T06:43:12.803359+0000","last_unstale":"2024-07-27T06:45:09.488571+0000","last_undegraded":"2024-07-27T06:45:09.488571+0000","last_fullsized":"2024-07-27T06:45:09.488571+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:32:53.167039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'7","reported_seq":204,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:04.587818+0000","last_change":"2024-07-27T06:44:09.241782+0000","last_active":"2024-07-27T06:47:04.587818+0000","last_peered":"2024-07-27T06:47:04.587818+0000","last_clean":"2024-07-27T06:47:04.587818+0000","last_became_active":"2024-07-27T06:44:09.234692+0000","last_became_peered":"2024-07-27T06:44:09.234692+0000","last_unstale":"2024-07-27T06:47:04.587818+0000","last_undegraded":"2024-07-27T06:47:04.587818+0000","last_fullsized":"2024-07-27T06:47:04.587818+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:45:02.155686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":181,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1561,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488800+0000","last_change":"2024-07-27T06:43:12.805830+0000","last_active":"2024-07-27T06:45:09.488800+0000","last_peered":"2024-07-27T06:45:09.488800+0000","last_clean":"2024-07-27T06:45:09.488800+0000","last_became_active":"2024-07-27T06:43:12.805324+0000","last_became_peered":"2024-07-27T06:43:12.805324+0000","last_unstale":"2024-07-27T06:45:09.488800+0000","last_undegraded":"2024-07-27T06:45:09.488800+0000","last_fullsized":"2024-07-27T06:45:09.488800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:38:31.430676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":109,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.489084+0000","last_change":"2024-07-27T06:41:43.549132+0000","last_active":"2024-07-27T06:45:09.489084+0000","last_peered":"2024-07-27T06:45:09.489084+0000","last_clean":"2024-07-27T06:45:09.489084+0000","last_became_active":"2024-07-27T06:41:43.542635+0000","last_became_peered":"2024-07-27T06:41:43.542635+0000","last_unstale":"2024-07-27T06:45:09.489084+0000","last_undegraded":"2024-07-27T06:45:09.489084+0000","last_fullsized":"2024-07-27T06:45:09.489084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_clean_scrub_stamp":"2024-07-27T06:38:21.639907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:10:06.417387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.861700+0000","last_change":"2024-07-27T06:43:09.768774+0000","last_active":"2024-07-27T06:45:09.861700+0000","last_peered":"2024-07-27T06:45:09.861700+0000","last_clean":"2024-07-27T06:45:09.861700+0000","last_became_active":"2024-07-27T06:43:09.768345+0000","last_became_peered":"2024-07-27T06:43:09.768345+0000","last_unstale":"2024-07-27T06:45:09.861700+0000","last_undegraded":"2024-07-27T06:45:09.861700+0000","last_fullsized":"2024-07-27T06:45:09.861700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:40:49.411746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485670+0000","last_change":"2024-07-27T06:43:12.797817+0000","last_active":"2024-07-27T06:45:09.485670+0000","last_peered":"2024-07-27T06:45:09.485670+0000","last_clean":"2024-07-27T06:45:09.485670+0000","last_became_active":"2024-07-27T06:43:12.797483+0000","last_became_peered":"2024-07-27T06:43:12.797483+0000","last_unstale":"2024-07-27T06:45:09.485670+0000","last_undegraded":"2024-07-27T06:45:09.485670+0000","last_fullsized":"2024-07-27T06:45:09.485670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:24:44.539505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488887+0000","last_change":"2024-07-27T06:43:09.776880+0000","last_active":"2024-07-27T06:45:09.488887+0000","last_peered":"2024-07-27T06:45:09.488887+0000","last_clean":"2024-07-27T06:45:09.488887+0000","last_became_active":"2024-07-27T06:43:09.776486+0000","last_became_peered":"2024-07-27T06:43:09.776486+0000","last_unstale":"2024-07-27T06:45:09.488887+0000","last_undegraded":"2024-07-27T06:45:09.488887+0000","last_fullsized":"2024-07-27T06:45:09.488887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:55:25.436405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.480961+0000","last_change":"2024-07-27T06:43:12.799755+0000","last_active":"2024-07-27T06:45:09.480961+0000","last_peered":"2024-07-27T06:45:09.480961+0000","last_clean":"2024-07-27T06:45:09.480961+0000","last_became_active":"2024-07-27T06:43:12.799412+0000","last_became_peered":"2024-07-27T06:43:12.799412+0000","last_unstale":"2024-07-27T06:45:09.480961+0000","last_undegraded":"2024-07-27T06:45:09.480961+0000","last_fullsized":"2024-07-27T06:45:09.480961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:45:50.545431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":245,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:46:59.210015+0000","last_change":"2024-07-27T06:43:10.533039+0000","last_active":"2024-07-27T06:46:59.210015+0000","last_peered":"2024-07-27T06:46:59.210015+0000","last_clean":"2024-07-27T06:46:59.210015+0000","last_became_active":"2024-07-27T06:43:10.532583+0000","last_became_peered":"2024-07-27T06:43:10.532583+0000","last_unstale":"2024-07-27T06:46:59.210015+0000","last_undegraded":"2024-07-27T06:46:59.210015+0000","last_fullsized":"2024-07-27T06:46:59.210015+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:04:51.954463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":208,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633096+0000","last_change":"2024-07-27T06:44:09.238218+0000","last_active":"2024-07-27T06:45:09.633096+0000","last_peered":"2024-07-27T06:45:09.633096+0000","last_clean":"2024-07-27T06:45:09.633096+0000","last_became_active":"2024-07-27T06:44:09.237841+0000","last_became_peered":"2024-07-27T06:44:09.237841+0000","last_unstale":"2024-07-27T06:45:09.633096+0000","last_undegraded":"2024-07-27T06:45:09.633096+0000","last_fullsized":"2024-07-27T06:45:09.633096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:19.101854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"83'8","reported_seq":252,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:00.252579+0000","last_change":"2024-07-27T06:43:09.776214+0000","last_active":"2024-07-27T06:47:00.252579+0000","last_peered":"2024-07-27T06:47:00.252579+0000","last_clean":"2024-07-27T06:47:00.252579+0000","last_became_active":"2024-07-27T06:43:09.775719+0000","last_became_peered":"2024-07-27T06:43:09.775719+0000","last_unstale":"2024-07-27T06:47:00.252579+0000","last_undegraded":"2024-07-27T06:47:00.252579+0000","last_fullsized":"2024-07-27T06:47:00.252579+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:57:20.381599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481735+0000","last_change":"2024-07-27T06:43:12.794909+0000","last_active":"2024-07-27T06:45:09.481735+0000","last_peered":"2024-07-27T06:45:09.481735+0000","last_clean":"2024-07-27T06:45:09.481735+0000","last_became_active":"2024-07-27T06:43:12.794497+0000","last_became_peered":"2024-07-27T06:43:12.794497+0000","last_unstale":"2024-07-27T06:45:09.481735+0000","last_undegraded":"2024-07-27T06:45:09.481735+0000","last_fullsized":"2024-07-27T06:45:09.481735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:42:49.760126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633255+0000","last_change":"2024-07-27T06:43:10.534240+0000","last_active":"2024-07-27T06:45:09.633255+0000","last_peered":"2024-07-27T06:45:09.633255+0000","last_clean":"2024-07-27T06:45:09.633255+0000","last_became_active":"2024-07-27T06:43:10.533596+0000","last_became_peered":"2024-07-27T06:43:10.533596+0000","last_unstale":"2024-07-27T06:45:09.633255+0000","last_undegraded":"2024-07-27T06:45:09.633255+0000","last_fullsized":"2024-07-27T06:45:09.633255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:16.098839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481991+0000","last_change":"2024-07-27T06:43:12.805248+0000","last_active":"2024-07-27T06:45:09.481991+0000","last_peered":"2024-07-27T06:45:09.481991+0000","last_clean":"2024-07-27T06:45:09.481991+0000","last_became_active":"2024-07-27T06:43:12.803693+0000","last_became_peered":"2024-07-27T06:43:12.803693+0000","last_unstale":"2024-07-27T06:45:09.481991+0000","last_undegraded":"2024-07-27T06:45:09.481991+0000","last_fullsized":"2024-07-27T06:45:09.481991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:17.466465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486692+0000","last_change":"2024-07-27T06:43:10.533701+0000","last_active":"2024-07-27T06:45:09.486692+0000","last_peered":"2024-07-27T06:45:09.486692+0000","last_clean":"2024-07-27T06:45:09.486692+0000","last_became_active":"2024-07-27T06:43:10.533109+0000","last_became_peered":"2024-07-27T06:43:10.533109+0000","last_unstale":"2024-07-27T06:45:09.486692+0000","last_undegraded":"2024-07-27T06:45:09.486692+0000","last_fullsized":"2024-07-27T06:45:09.486692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:32:52.906691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481918+0000","last_change":"2024-07-27T06:43:12.798479+0000","last_active":"2024-07-27T06:45:09.481918+0000","last_peered":"2024-07-27T06:45:09.481918+0000","last_clean":"2024-07-27T06:45:09.481918+0000","last_became_active":"2024-07-27T06:43:12.798124+0000","last_became_peered":"2024-07-27T06:43:12.798124+0000","last_unstale":"2024-07-27T06:45:09.481918+0000","last_undegraded":"2024-07-27T06:45:09.481918+0000","last_fullsized":"2024-07-27T06:45:09.481918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:14:55.267213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"83'59","reported_seq":120,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:02.222792+0000","last_change":"2024-07-27T06:43:09.770122+0000","last_active":"2024-07-27T06:47:02.222792+0000","last_peered":"2024-07-27T06:47:02.222792+0000","last_clean":"2024-07-27T06:47:02.222792+0000","last_became_active":"2024-07-27T06:43:09.769784+0000","last_became_peered":"2024-07-27T06:43:09.769784+0000","last_unstale":"2024-07-27T06:47:02.222792+0000","last_undegraded":"2024-07-27T06:47:02.222792+0000","last_fullsized":"2024-07-27T06:47:02.222792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:47:00.398135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":103,"num_write_kb":1094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485891+0000","last_change":"2024-07-27T06:43:12.799603+0000","last_active":"2024-07-27T06:45:09.485891+0000","last_peered":"2024-07-27T06:45:09.485891+0000","last_clean":"2024-07-27T06:45:09.485891+0000","last_became_active":"2024-07-27T06:43:12.799189+0000","last_became_peered":"2024-07-27T06:43:12.799189+0000","last_unstale":"2024-07-27T06:45:09.485891+0000","last_undegraded":"2024-07-27T06:45:09.485891+0000","last_fullsized":"2024-07-27T06:45:09.485891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:32:30.603258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862220+0000","last_change":"2024-07-27T06:43:10.004726+0000","last_active":"2024-07-27T06:45:09.862220+0000","last_peered":"2024-07-27T06:45:09.862220+0000","last_clean":"2024-07-27T06:45:09.862220+0000","last_became_active":"2024-07-27T06:43:10.004367+0000","last_became_peered":"2024-07-27T06:43:10.004367+0000","last_unstale":"2024-07-27T06:45:09.862220+0000","last_undegraded":"2024-07-27T06:45:09.862220+0000","last_fullsized":"2024-07-27T06:45:09.862220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:22:39.942391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862427+0000","last_change":"2024-07-27T06:43:12.796511+0000","last_active":"2024-07-27T06:45:09.862427+0000","last_peered":"2024-07-27T06:45:09.862427+0000","last_clean":"2024-07-27T06:45:09.862427+0000","last_became_active":"2024-07-27T06:43:12.796152+0000","last_became_peered":"2024-07-27T06:43:12.796152+0000","last_unstale":"2024-07-27T06:45:09.862427+0000","last_undegraded":"2024-07-27T06:45:09.862427+0000","last_fullsized":"2024-07-27T06:45:09.862427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:03:02.508990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822077+0000","last_change":"2024-07-27T06:43:10.533656+0000","last_active":"2024-07-27T06:45:09.822077+0000","last_peered":"2024-07-27T06:45:09.822077+0000","last_clean":"2024-07-27T06:45:09.822077+0000","last_became_active":"2024-07-27T06:43:10.533295+0000","last_became_peered":"2024-07-27T06:43:10.533295+0000","last_unstale":"2024-07-27T06:45:09.822077+0000","last_undegraded":"2024-07-27T06:45:09.822077+0000","last_fullsized":"2024-07-27T06:45:09.822077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:11:00.032511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485237+0000","last_change":"2024-07-27T06:43:12.795106+0000","last_active":"2024-07-27T06:45:09.485237+0000","last_peered":"2024-07-27T06:45:09.485237+0000","last_clean":"2024-07-27T06:45:09.485237+0000","last_became_active":"2024-07-27T06:43:12.794729+0000","last_became_peered":"2024-07-27T06:43:12.794729+0000","last_unstale":"2024-07-27T06:45:09.485237+0000","last_undegraded":"2024-07-27T06:45:09.485237+0000","last_fullsized":"2024-07-27T06:45:09.485237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:19:16.542363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485673+0000","last_change":"2024-07-27T06:43:09.770016+0000","last_active":"2024-07-27T06:45:09.485673+0000","last_peered":"2024-07-27T06:45:09.485673+0000","last_clean":"2024-07-27T06:45:09.485673+0000","last_became_active":"2024-07-27T06:43:09.769672+0000","last_became_peered":"2024-07-27T06:43:09.769672+0000","last_unstale":"2024-07-27T06:45:09.485673+0000","last_undegraded":"2024-07-27T06:45:09.485673+0000","last_fullsized":"2024-07-27T06:45:09.485673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:30:55.737494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483659+0000","last_change":"2024-07-27T06:43:12.793690+0000","last_active":"2024-07-27T06:45:09.483659+0000","last_peered":"2024-07-27T06:45:09.483659+0000","last_clean":"2024-07-27T06:45:09.483659+0000","last_became_active":"2024-07-27T06:43:12.793352+0000","last_became_peered":"2024-07-27T06:43:12.793352+0000","last_unstale":"2024-07-27T06:45:09.483659+0000","last_undegraded":"2024-07-27T06:45:09.483659+0000","last_fullsized":"2024-07-27T06:45:09.483659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:22:47.606990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.491196+0000","last_change":"2024-07-27T06:45:09.490738+0000","last_active":"2024-07-27T06:45:09.491196+0000","last_peered":"2024-07-27T06:45:09.491196+0000","last_clean":"2024-07-27T06:45:09.491196+0000","last_became_active":"2024-07-27T06:45:09.487147+0000","last_became_peered":"2024-07-27T06:45:09.487147+0000","last_unstale":"2024-07-27T06:45:09.491196+0000","last_undegraded":"2024-07-27T06:45:09.491196+0000","last_fullsized":"2024-07-27T06:45:09.491196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:46.732987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485570+0000","last_change":"2024-07-27T06:43:12.792648+0000","last_active":"2024-07-27T06:45:09.485570+0000","last_peered":"2024-07-27T06:45:09.485570+0000","last_clean":"2024-07-27T06:45:09.485570+0000","last_became_active":"2024-07-27T06:43:12.792338+0000","last_became_peered":"2024-07-27T06:43:12.792338+0000","last_unstale":"2024-07-27T06:45:09.485570+0000","last_undegraded":"2024-07-27T06:45:09.485570+0000","last_fullsized":"2024-07-27T06:45:09.485570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:36:52.605957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.827632+0000","last_change":"2024-07-27T06:45:09.827140+0000","last_active":"2024-07-27T06:45:09.827632+0000","last_peered":"2024-07-27T06:45:09.827632+0000","last_clean":"2024-07-27T06:45:09.827632+0000","last_became_active":"2024-07-27T06:45:09.823119+0000","last_became_peered":"2024-07-27T06:45:09.823119+0000","last_unstale":"2024-07-27T06:45:09.827632+0000","last_undegraded":"2024-07-27T06:45:09.827632+0000","last_fullsized":"2024-07-27T06:45:09.827632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:32:19.682193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488730+0000","last_change":"2024-07-27T06:43:12.809347+0000","last_active":"2024-07-27T06:45:09.488730+0000","last_peered":"2024-07-27T06:45:09.488730+0000","last_clean":"2024-07-27T06:45:09.488730+0000","last_became_active":"2024-07-27T06:43:12.808980+0000","last_became_peered":"2024-07-27T06:43:12.808980+0000","last_unstale":"2024-07-27T06:45:09.488730+0000","last_undegraded":"2024-07-27T06:45:09.488730+0000","last_fullsized":"2024-07-27T06:45:09.488730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:05:14.132641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633284+0000","last_change":"2024-07-27T06:44:09.237908+0000","last_active":"2024-07-27T06:45:09.633284+0000","last_peered":"2024-07-27T06:45:09.633284+0000","last_clean":"2024-07-27T06:45:09.633284+0000","last_became_active":"2024-07-27T06:44:09.237503+0000","last_became_peered":"2024-07-27T06:44:09.237503+0000","last_unstale":"2024-07-27T06:45:09.633284+0000","last_undegraded":"2024-07-27T06:45:09.633284+0000","last_fullsized":"2024-07-27T06:45:09.633284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:29.654244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486391+0000","last_change":"2024-07-27T06:43:09.776000+0000","last_active":"2024-07-27T06:45:09.486391+0000","last_peered":"2024-07-27T06:45:09.486391+0000","last_clean":"2024-07-27T06:45:09.486391+0000","last_became_active":"2024-07-27T06:43:09.775522+0000","last_became_peered":"2024-07-27T06:43:09.775522+0000","last_unstale":"2024-07-27T06:45:09.486391+0000","last_undegraded":"2024-07-27T06:45:09.486391+0000","last_fullsized":"2024-07-27T06:45:09.486391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:39:42.650177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822084+0000","last_change":"2024-07-27T06:43:12.807062+0000","last_active":"2024-07-27T06:45:09.822084+0000","last_peered":"2024-07-27T06:45:09.822084+0000","last_clean":"2024-07-27T06:45:09.822084+0000","last_became_active":"2024-07-27T06:43:12.805816+0000","last_became_peered":"2024-07-27T06:43:12.805816+0000","last_unstale":"2024-07-27T06:45:09.822084+0000","last_undegraded":"2024-07-27T06:45:09.822084+0000","last_fullsized":"2024-07-27T06:45:09.822084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:44:16.261663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483465+0000","last_change":"2024-07-27T06:44:09.250070+0000","last_active":"2024-07-27T06:45:09.483465+0000","last_peered":"2024-07-27T06:45:09.483465+0000","last_clean":"2024-07-27T06:45:09.483465+0000","last_became_active":"2024-07-27T06:44:09.237290+0000","last_became_peered":"2024-07-27T06:44:09.237290+0000","last_unstale":"2024-07-27T06:45:09.483465+0000","last_undegraded":"2024-07-27T06:45:09.483465+0000","last_fullsized":"2024-07-27T06:45:09.483465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:51.590487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862423+0000","last_change":"2024-07-27T06:43:12.809135+0000","last_active":"2024-07-27T06:45:09.862423+0000","last_peered":"2024-07-27T06:45:09.862423+0000","last_clean":"2024-07-27T06:45:09.862423+0000","last_became_active":"2024-07-27T06:43:12.806731+0000","last_became_peered":"2024-07-27T06:43:12.806731+0000","last_unstale":"2024-07-27T06:45:09.862423+0000","last_undegraded":"2024-07-27T06:45:09.862423+0000","last_fullsized":"2024-07-27T06:45:09.862423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:47:50.397517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486050+0000","last_change":"2024-07-27T06:43:10.533608+0000","last_active":"2024-07-27T06:45:09.486050+0000","last_peered":"2024-07-27T06:45:09.486050+0000","last_clean":"2024-07-27T06:45:09.486050+0000","last_became_active":"2024-07-27T06:43:10.532663+0000","last_became_peered":"2024-07-27T06:43:10.532663+0000","last_unstale":"2024-07-27T06:45:09.486050+0000","last_undegraded":"2024-07-27T06:45:09.486050+0000","last_fullsized":"2024-07-27T06:45:09.486050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:05:33.155427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485883+0000","last_change":"2024-07-27T06:43:12.789890+0000","last_active":"2024-07-27T06:45:09.485883+0000","last_peered":"2024-07-27T06:45:09.485883+0000","last_clean":"2024-07-27T06:45:09.485883+0000","last_became_active":"2024-07-27T06:43:12.789561+0000","last_became_peered":"2024-07-27T06:43:12.789561+0000","last_unstale":"2024-07-27T06:45:09.485883+0000","last_undegraded":"2024-07-27T06:45:09.485883+0000","last_fullsized":"2024-07-27T06:45:09.485883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:35:49.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"83'15","reported_seq":256,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:00.251798+0000","last_change":"2024-07-27T06:43:09.773471+0000","last_active":"2024-07-27T06:47:00.251798+0000","last_peered":"2024-07-27T06:47:00.251798+0000","last_clean":"2024-07-27T06:47:00.251798+0000","last_became_active":"2024-07-27T06:43:09.773213+0000","last_became_peered":"2024-07-27T06:43:09.773213+0000","last_unstale":"2024-07-27T06:47:00.251798+0000","last_undegraded":"2024-07-27T06:47:00.251798+0000","last_fullsized":"2024-07-27T06:47:00.251798+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:54:38.770198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":209,"num_read_kb":209,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":861595,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2239,"num_read_kb":1265,"num_write":406,"num_write_kb":2117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1561,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":823296,"data_stored":2584785,"data_compressed":111761,"data_compressed_allocated":147456,"data_compressed_original":2285568,"omap_allocated":0,"internal_metadata":0},"log_size":328,"ondisk_log_size":328,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283116,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678078,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":461919,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874306,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1118693,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037833,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":461317,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234062,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":854911,"data_compressed":30901,"data_compressed_allocated":40960,"data_compressed_original":425984,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430289,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":1055194,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593815,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":461874,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790045,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":1447808,"data_compressed":65700,"data_compressed_allocated":86016,"data_compressed_original":991232,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953570,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":929792,"data_stored":462597,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":1247851,"data_compressed":48992,"data_compressed_allocated":65536,"data_compressed_original":794624,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378732,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":527806,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542257,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":529064,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705782,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":462905,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":67611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":787422,"data_compressed":37380,"data_compressed_allocated":49152,"data_compressed_original":761856,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":397128,"data_compressed":18091,"data_compressed_allocated":24576,"data_compressed_original":368640,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19272,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":461789,"data_compressed":17729,"data_compressed_allocated":24576,"data_compressed_original":368640,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T06:47:08.575 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph pg dump --format=json 2024-07-27T06:47:08.841 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:08 smithi029 ceph-mon[32610]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:47:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:08 smithi077 ceph-mon[37153]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:47:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:08 smithi047 ceph-mon[36835]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:47:09.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.118+0000 7f949e83c640 1 -- 172.21.15.29:0/375995412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9498109840 msgr2=0x7f9498110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.118+0000 7f949e83c640 1 --2- 172.21.15.29:0/375995412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9498109840 0x7f9498110070 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9490009ca0 tx=0x7f949002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.118+0000 7f949e83c640 1 -- 172.21.15.29:0/375995412 shutdown_connections 2024-07-27T06:47:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.118+0000 7f949e83c640 1 --2- 172.21.15.29:0/375995412 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9498109840 0x7f9498110070 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.118+0000 7f949e83c640 1 --2- 172.21.15.29:0/375995412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9498108ef0 0x7f9498109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.118+0000 7f949e83c640 1 --2- 172.21.15.29:0/375995412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94981035b0 0x7f94981039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.118+0000 7f949e83c640 1 -- 172.21.15.29:0/375995412 >> 172.21.15.29:0/375995412 conn(0x7f94980ff000 msgr2=0x7f9498101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.118+0000 7f949e83c640 1 -- 172.21.15.29:0/375995412 shutdown_connections 2024-07-27T06:47:09.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.118+0000 7f949e83c640 1 -- 172.21.15.29:0/375995412 wait complete. 2024-07-27T06:47:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.119+0000 7f949e83c640 1 Processor -- start 2024-07-27T06:47:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.119+0000 7f949e83c640 1 -- start start 2024-07-27T06:47:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.119+0000 7f949e83c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94981035b0 0x7f9498199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.119+0000 7f949e83c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9498108ef0 0x7f949819a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.119+0000 7f949e83c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9498109840 0x7f949819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.119+0000 7f949e83c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9498113510 con 0x7f94981035b0 2024-07-27T06:47:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.119+0000 7f949e83c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f9498113390 con 0x7f9498109840 2024-07-27T06:47:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.119+0000 7f949e83c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9498113690 con 0x7f9498108ef0 2024-07-27T06:47:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.119+0000 7f9497fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94981035b0 0x7f9498199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.119+0000 7f9497fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94981035b0 0x7f9498199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43292/0 (socket says 172.21.15.29:43292) 2024-07-27T06:47:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.119+0000 7f9497fff640 1 -- 172.21.15.29:0/3543536255 learned_addr learned my addr 172.21.15.29:0/3543536255 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f949cdd9640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9498109840 0x7f949819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f94977fe640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9498108ef0 0x7f949819a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f9497fff640 1 -- 172.21.15.29:0/3543536255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9498108ef0 msgr2=0x7f949819a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f9497fff640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9498108ef0 0x7f949819a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f9497fff640 1 -- 172.21.15.29:0/3543536255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9498109840 msgr2=0x7f949819f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f9497fff640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9498109840 0x7f949819f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f9497fff640 1 -- 172.21.15.29:0/3543536255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949819fe30 con 0x7f94981035b0 2024-07-27T06:47:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f949cdd9640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9498109840 0x7f949819f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f9497fff640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94981035b0 0x7f9498199ec0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f948800d950 tx=0x7f948800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f94957fa640 1 -- 172.21.15.29:0/3543536255 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9488017070 con 0x7f94981035b0 2024-07-27T06:47:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f94957fa640 1 -- 172.21.15.29:0/3543536255 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9488009c30 con 0x7f94981035b0 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f949e83c640 1 -- 172.21.15.29:0/3543536255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94981a0120 con 0x7f94981035b0 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f94957fa640 1 -- 172.21.15.29:0/3543536255 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948800f620 con 0x7f94981035b0 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.120+0000 7f949e83c640 1 -- 172.21.15.29:0/3543536255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94981a6880 con 0x7f94981035b0 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.123+0000 7f94957fa640 1 -- 172.21.15.29:0/3543536255 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f948800f7c0 con 0x7f94981035b0 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.123+0000 7f94a4dc0740 1 Processor -- start 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.123+0000 7f94a4dc0740 1 -- start start 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.123+0000 7f94a4dc0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519099070 0x55c51943e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.123+0000 7f94a4dc0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c5190701b0 0x55c51943ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.123+0000 7f94a4dc0740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519078390 0x55c51943f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.123+0000 7f94a4dc0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c5193b4b90 con 0x55c519099070 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.123+0000 7f94a4dc0740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55c5193a4090 con 0x55c519078390 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.123+0000 7f94a4dc0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c5193a80d0 con 0x55c5190701b0 2024-07-27T06:47:09.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f949cdd9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519099070 0x55c51943e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f949cdd9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519099070 0x55c51943e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43298/0 (socket says 172.21.15.29:43298) 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f949cdd9640 1 -- 172.21.15.29:0/2883875683 learned_addr learned my addr 172.21.15.29:0/2883875683 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f9497fff640 1 --2- 172.21.15.29:0/2883875683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c5190701b0 0x55c51943ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f94977fe640 1 --2- 172.21.15.29:0/2883875683 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519078390 0x55c51943f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f94957fa640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f94800b3cb0 0x7f94800b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f949cdd9640 1 -- 172.21.15.29:0/2883875683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c5190701b0 msgr2=0x55c51943ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f949cdd9640 1 --2- 172.21.15.29:0/2883875683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c5190701b0 0x55c51943ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f949cdd9640 1 -- 172.21.15.29:0/2883875683 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519078390 msgr2=0x55c51943f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f949cdd9640 1 --2- 172.21.15.29:0/2883875683 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519078390 0x55c51943f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f949cdd9640 1 -- 172.21.15.29:0/2883875683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c51934ab70 con 0x55c519099070 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f94957fa640 1 -- 172.21.15.29:0/3543536255 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f94880e3c70 con 0x7f94981035b0 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f949cdd9640 1 --2- 172.21.15.29:0/2883875683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519099070 0x55c51943e670 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9490009830 tx=0x7f9490009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dde74e678212503 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f949cdd9640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f94800b3cb0 0x7f94800b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.124+0000 7f947e7fc640 1 -- 172.21.15.29:0/2883875683 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949004a040 con 0x55c519099070 2024-07-27T06:47:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.125+0000 7f947e7fc640 1 -- 172.21.15.29:0/2883875683 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9490036040 con 0x55c519099070 2024-07-27T06:47:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.125+0000 7f947e7fc640 1 -- 172.21.15.29:0/2883875683 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9490004400 con 0x55c519099070 2024-07-27T06:47:09.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.126+0000 7f94a4dc0740 1 -- 172.21.15.29:0/2883875683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519099070 msgr2=0x55c51943e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:09.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.126+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/2883875683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519099070 0x55c51943e670 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9490009830 tx=0x7f9490009860 comp rx=0 tx=0).stop 2024-07-27T06:47:09.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.126+0000 7f949cdd9640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f94800b3cb0 0x7f94800b61c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9490045000 tx=0x7f9490004660 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:09.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.126+0000 7f94a4dc0740 1 -- 172.21.15.29:0/2883875683 shutdown_connections 2024-07-27T06:47:09.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.126+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/2883875683 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519078390 0x55c51943f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.126+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/2883875683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c5190701b0 0x55c51943ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.126+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/2883875683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519099070 0x55c51943e670 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.126+0000 7f94a4dc0740 1 -- 172.21.15.29:0/2883875683 >> 172.21.15.29:0/2883875683 conn(0x55c51908d460 msgr2=0x55c51943c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:09.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.126+0000 7f94a4dc0740 1 -- 172.21.15.29:0/2883875683 shutdown_connections 2024-07-27T06:47:09.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.126+0000 7f94a4dc0740 1 -- 172.21.15.29:0/2883875683 wait complete. 2024-07-27T06:47:09.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.127+0000 7f94a4dc0740 1 -- start start 2024-07-27T06:47:09.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.127+0000 7f94a4dc0740 1 Processor -- start 2024-07-27T06:47:09.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.127+0000 7f94a4dc0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 0x55c51944ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.127+0000 7f94a4dc0740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519078390 0x55c51944f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.127+0000 7f94a4dc0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c519099070 0x55c519454320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.127+0000 7f94a4dc0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c519207700 con 0x55c5190701b0 2024-07-27T06:47:09.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.127+0000 7f94a4dc0740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55c5193469c0 con 0x55c519078390 2024-07-27T06:47:09.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.127+0000 7f94a4dc0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c51944cdf0 con 0x55c519099070 2024-07-27T06:47:09.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f94a4dc0740 10 client.0 _ll_register_callbacks cb 0x55c5193614d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:09.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f9497fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 0x55c51944ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f9497fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 0x55c51944ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43302/0 (socket says 172.21.15.29:43302) 2024-07-27T06:47:09.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f9497fff640 1 -- 172.21.15.29:0/766266278 learned_addr learned my addr 172.21.15.29:0/766266278 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:09.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f949cdd9640 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c519099070 0x55c519454320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f94977fe640 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519078390 0x55c51944f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f9497fff640 1 -- 172.21.15.29:0/766266278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c519099070 msgr2=0x55c519454320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f9497fff640 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c519099070 0x55c519454320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f9497fff640 1 -- 172.21.15.29:0/766266278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519078390 msgr2=0x55c51944f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f9497fff640 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519078390 0x55c51944f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f9497fff640 1 -- 172.21.15.29:0/766266278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c519104650 con 0x55c5190701b0 2024-07-27T06:47:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f9497fff640 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 0x55c51944ed20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f94880058c0 tx=0x7f94880aa290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c40c6aa906be45 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.128+0000 7f94777fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.129+0000 7f94777fe640 1 -- 172.21.15.29:0/766266278 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9488020070 con 0x55c5190701b0 2024-07-27T06:47:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.129+0000 7f94777fe640 1 -- 172.21.15.29:0/766266278 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f94880920f0 con 0x55c5190701b0 2024-07-27T06:47:09.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.129+0000 7f94a4dc0740 1 -- 172.21.15.29:0/766266278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9490036040 con 0x55c5190701b0 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.129+0000 7f94777fe640 1 -- 172.21.15.29:0/766266278 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9488009de0 con 0x55c5190701b0 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.129+0000 7f94a4dc0740 10 client.15534 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.129+0000 7f94a4dc0740 1 -- 172.21.15.29:0/766266278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c51945d270 con 0x55c5190701b0 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.129+0000 7f94777fe640 1 -- 172.21.15.29:0/766266278 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9490036040 con 0x55c5190701b0 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.129+0000 7f94777fe640 1 -- 172.21.15.29:0/766266278 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f94880b3120 con 0x55c5190701b0 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.130+0000 7f94a4dc0740 10 client.15534 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.130+0000 7f94a4dc0740 10 client.15534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.130+0000 7f94a4dc0740 20 client.15534 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.130+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55c5190579f0 0x55c519460780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.130+0000 7f94a4dc0740 4 client.15534 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.130+0000 7f94a4dc0740 1 -- 172.21.15.29:0/766266278 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5193b6ae0 con 0x55c5190579f0 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.130+0000 7f94977fe640 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55c5190579f0 0x55c519460780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.130+0000 7f94777fe640 10 client.15534 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.131+0000 7f94977fe640 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55c5190579f0 0x55c519460780 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6356d8efa81ef1df server_cookie=fd9866ff4ac65a9e in_seq=0 out_seq=0 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.131+0000 7f94777fe640 1 -- 172.21.15.29:0/766266278 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c5193b6ae0 con 0x55c5190579f0 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.131+0000 7f94777fe640 10 client.15534 handle_command_reply: tid=0 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.131+0000 7f94a4dc0740 1 client.15534 shutdown 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.132+0000 7f94a4dc0740 1 -- 172.21.15.29:0/766266278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 msgr2=0x55c51944ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.132+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 0x55c51944ed20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f94880058c0 tx=0x7f94880aa290 comp rx=0 tx=0).stop 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.132+0000 7f94a4dc0740 1 -- 172.21.15.29:0/766266278 shutdown_connections 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.132+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55c5190579f0 0x55c519460780 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.132+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c519099070 0x55c519454320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.132+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519078390 0x55c51944f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.132+0000 7f94777fe640 0 client.15534 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.132+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/766266278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 0x55c51944ed20 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.132+0000 7f94a4dc0740 1 -- 172.21.15.29:0/766266278 >> 172.21.15.29:0/766266278 conn(0x55c51908d460 msgr2=0x55c51943ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:09.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.132+0000 7f94a4dc0740 1 -- 172.21.15.29:0/766266278 shutdown_connections 2024-07-27T06:47:09.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.132+0000 7f94a4dc0740 1 -- 172.21.15.29:0/766266278 wait complete. 2024-07-27T06:47:09.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.132+0000 7f94a4dc0740 20 client.15534 trim_cache size 0 max 16384 2024-07-27T06:47:09.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.150+0000 7f94a4dc0740 1 Processor -- start 2024-07-27T06:47:09.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.150+0000 7f94a4dc0740 1 -- start start 2024-07-27T06:47:09.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.150+0000 7f94a4dc0740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c5190579f0 0x55c51943c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.150+0000 7f94a4dc0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 0x55c519450040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.150+0000 7f94a4dc0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c519078390 0x55c519454100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.150+0000 7f94a4dc0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9498074f30 con 0x55c5190701b0 2024-07-27T06:47:09.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.150+0000 7f94a4dc0740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f9498076a70 con 0x55c5190579f0 2024-07-27T06:47:09.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.150+0000 7f94a4dc0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9498075600 con 0x55c519078390 2024-07-27T06:47:09.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.151+0000 7f94977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 0x55c519450040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.151+0000 7f94977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 0x55c519450040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43318/0 (socket says 172.21.15.29:43318) 2024-07-27T06:47:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.151+0000 7f94977fe640 1 -- 172.21.15.29:0/2929529026 learned_addr learned my addr 172.21.15.29:0/2929529026 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.151+0000 7f9497fff640 1 --2- 172.21.15.29:0/2929529026 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c5190579f0 0x55c51943c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.151+0000 7f949cdd9640 1 --2- 172.21.15.29:0/2929529026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c519078390 0x55c519454100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.151+0000 7f94977fe640 1 -- 172.21.15.29:0/2929529026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c519078390 msgr2=0x55c519454100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.151+0000 7f94977fe640 1 --2- 172.21.15.29:0/2929529026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c519078390 0x55c519454100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.151+0000 7f94977fe640 1 -- 172.21.15.29:0/2929529026 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c5190579f0 msgr2=0x55c51943c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.151+0000 7f94977fe640 1 --2- 172.21.15.29:0/2929529026 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c5190579f0 0x55c51943c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.151+0000 7f94977fe640 1 -- 172.21.15.29:0/2929529026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c519450ef0 con 0x55c5190701b0 2024-07-27T06:47:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.151+0000 7f94977fe640 1 --2- 172.21.15.29:0/2929529026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 0x55c519450040 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f948c000c00 tx=0x7f948c014900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb5dcf8e6a9215b server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.152+0000 7f947dffb640 1 -- 172.21.15.29:0/2929529026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c014190 con 0x55c5190701b0 2024-07-27T06:47:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.152+0000 7f947dffb640 1 -- 172.21.15.29:0/2929529026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f948c012900 con 0x55c5190701b0 2024-07-27T06:47:09.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.152+0000 7f947dffb640 1 -- 172.21.15.29:0/2929529026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c026840 con 0x55c5190701b0 2024-07-27T06:47:09.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.153+0000 7f94a4dc0740 1 -- 172.21.15.29:0/2929529026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 msgr2=0x55c519450040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.153+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/2929529026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 0x55c519450040 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f948c000c00 tx=0x7f948c014900 comp rx=0 tx=0).stop 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.153+0000 7f94a4dc0740 1 -- 172.21.15.29:0/2929529026 shutdown_connections 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.153+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/2929529026 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c519078390 0x55c519454100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.153+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/2929529026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c5190701b0 0x55c519450040 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.153+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/2929529026 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c5190579f0 0x55c51943c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.153+0000 7f94a4dc0740 1 -- 172.21.15.29:0/2929529026 >> 172.21.15.29:0/2929529026 conn(0x55c51908d460 msgr2=0x55c51943f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.153+0000 7f94a4dc0740 1 -- 172.21.15.29:0/2929529026 shutdown_connections 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.153+0000 7f94a4dc0740 1 -- 172.21.15.29:0/2929529026 wait complete. 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.154+0000 7f94a4dc0740 1 -- start start 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.154+0000 7f94a4dc0740 1 Processor -- start 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.154+0000 7f94a4dc0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c5190579f0 0x55c519459000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.154+0000 7f94a4dc0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519078390 0x55c519459540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.154+0000 7f94a4dc0740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519099070 0x55c519457fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.154+0000 7f94a4dc0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f94980763a0 con 0x55c519078390 2024-07-27T06:47:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.154+0000 7f94a4dc0740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f9498075cd0 con 0x55c519099070 2024-07-27T06:47:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.154+0000 7f94a4dc0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9498072f10 con 0x55c5190579f0 2024-07-27T06:47:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.154+0000 7f94a4dc0740 10 client.0 _ll_register_callbacks cb 0x7f948800df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f94977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519078390 0x55c519459540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f94977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519078390 0x55c519459540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43326/0 (socket says 172.21.15.29:43326) 2024-07-27T06:47:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f94977fe640 1 -- 172.21.15.29:0/652010206 learned_addr learned my addr 172.21.15.29:0/652010206 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f949cdd9640 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519099070 0x55c519457fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f9497fff640 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c5190579f0 0x55c519459000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f94977fe640 1 -- 172.21.15.29:0/652010206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c5190579f0 msgr2=0x55c519459000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f94977fe640 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c5190579f0 0x55c519459000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f94977fe640 1 -- 172.21.15.29:0/652010206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519099070 msgr2=0x55c519457fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:09.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f94977fe640 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519099070 0x55c519457fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f94977fe640 1 -- 172.21.15.29:0/652010206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c519459de0 con 0x55c519078390 2024-07-27T06:47:09.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f949cdd9640 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519099070 0x55c519457fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f94977fe640 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519078390 0x55c519459540 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f948c014040 tx=0x7f948c00bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9be57ccb85f49 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.155+0000 7f94777fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.156+0000 7f94777fe640 1 -- 172.21.15.29:0/652010206 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c012660 con 0x55c519078390 2024-07-27T06:47:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.156+0000 7f94a4dc0740 1 -- 172.21.15.29:0/652010206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f948c012900 con 0x55c519078390 2024-07-27T06:47:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.156+0000 7f94777fe640 1 -- 172.21.15.29:0/652010206 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f948c012ab0 con 0x55c519078390 2024-07-27T06:47:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.156+0000 7f94777fe640 1 -- 172.21.15.29:0/652010206 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f948c012cb0 con 0x55c519078390 2024-07-27T06:47:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.156+0000 7f94a4dc0740 10 client.15540 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.156+0000 7f94a4dc0740 1 -- 172.21.15.29:0/652010206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c5194c9450 con 0x55c519078390 2024-07-27T06:47:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.156+0000 7f94777fe640 1 -- 172.21.15.29:0/652010206 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f948c012900 con 0x55c519078390 2024-07-27T06:47:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.156+0000 7f94777fe640 1 -- 172.21.15.29:0/652010206 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f948c037030 con 0x55c519078390 2024-07-27T06:47:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.156+0000 7f94a4dc0740 10 client.15540 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.156+0000 7f94a4dc0740 10 client.15540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.156+0000 7f94a4dc0740 20 client.15540 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.156+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55c51902cda0 0x55c5194cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.157+0000 7f94a4dc0740 4 client.15540 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-27T06:47:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.157+0000 7f94a4dc0740 1 -- 172.21.15.29:0/652010206 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c5193b2580 con 0x55c51902cda0 2024-07-27T06:47:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.157+0000 7f9497fff640 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55c51902cda0 0x55c5194cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:09.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.157+0000 7f9497fff640 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55c51902cda0 0x55c5194cbc00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2655202ea4508a4 server_cookie=5f97e2d55afc3ade in_seq=0 out_seq=0 2024-07-27T06:47:09.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.157+0000 7f94777fe640 10 client.15540 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:09.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94777fe640 1 -- 172.21.15.29:0/652010206 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c5193b2580 con 0x55c51902cda0 2024-07-27T06:47:09.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94777fe640 10 client.15540 handle_command_reply: tid=0 2024-07-27T06:47:09.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94a4dc0740 1 client.15540 shutdown 2024-07-27T06:47:09.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94a4dc0740 1 -- 172.21.15.29:0/652010206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519078390 msgr2=0x55c519459540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:09.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519078390 0x55c519459540 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f948c014040 tx=0x7f948c00bf30 comp rx=0 tx=0).stop 2024-07-27T06:47:09.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94a4dc0740 1 -- 172.21.15.29:0/652010206 shutdown_connections 2024-07-27T06:47:09.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55c519099070 0x55c519457fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c519078390 0x55c519459540 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55c51902cda0 0x55c5194cbc00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94a4dc0740 1 --2- 172.21.15.29:0/652010206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c5190579f0 0x55c519459000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94777fe640 0 client.15540 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:09.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94a4dc0740 1 -- 172.21.15.29:0/652010206 >> 172.21.15.29:0/652010206 conn(0x55c51908d460 msgr2=0x55c51943c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:09.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94a4dc0740 1 -- 172.21.15.29:0/652010206 shutdown_connections 2024-07-27T06:47:09.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94a4dc0740 1 -- 172.21.15.29:0/652010206 wait complete. 2024-07-27T06:47:09.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.158+0000 7f94a4dc0740 20 client.15540 trim_cache size 0 max 16384 2024-07-27T06:47:09.163 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:47:09.163 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-27T06:47:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-07-27T06:47:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: "ac591078-3819-4ade-9981-0f2249333d08": { 2024-07-27T06:47:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-07-27T06:47:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "ac591078-3819-4ade-9981-0f2249333d08", 2024-07-27T06:47:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-07-27T06:47:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:47:09.164 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:47:09.164 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:47:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.159+0000 7f949e83c640 1 -- 172.21.15.29:0/3543536255 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f94800b3cb0 msgr2=0x7f94800b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.159+0000 7f949e83c640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f94800b3cb0 0x7f94800b61c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9490045000 tx=0x7f9490004660 comp rx=0 tx=0).stop 2024-07-27T06:47:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.159+0000 7f949e83c640 1 -- 172.21.15.29:0/3543536255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94981035b0 msgr2=0x7f9498199ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.159+0000 7f949e83c640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94981035b0 0x7f9498199ec0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f948800d950 tx=0x7f948800de10 comp rx=0 tx=0).stop 2024-07-27T06:47:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.159+0000 7f949e83c640 1 -- 172.21.15.29:0/3543536255 shutdown_connections 2024-07-27T06:47:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.159+0000 7f949e83c640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9498109840 0x7f949819f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.159+0000 7f949e83c640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f94800b3cb0 0x7f94800b61c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.159+0000 7f949e83c640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9498108ef0 0x7f949819a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.159+0000 7f949e83c640 1 --2- 172.21.15.29:0/3543536255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94981035b0 0x7f9498199ec0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.159+0000 7f949e83c640 1 -- 172.21.15.29:0/3543536255 >> 172.21.15.29:0/3543536255 conn(0x7f94980ff000 msgr2=0x7f9498101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:09.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.159+0000 7f949e83c640 1 -- 172.21.15.29:0/3543536255 shutdown_connections 2024-07-27T06:47:09.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:09.159+0000 7f949e83c640 1 -- 172.21.15.29:0/3543536255 wait complete. 2024-07-27T06:47:09.169 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:09.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:09 smithi047 ceph-mon[36835]: pgmap v401: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 174 KiB/s wr, 57 op/s 2024-07-27T06:47:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:09 smithi077 ceph-mon[37153]: pgmap v401: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 174 KiB/s wr, 57 op/s 2024-07-27T06:47:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:09 smithi029 ceph-mon[32610]: pgmap v401: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 174 KiB/s wr, 57 op/s 2024-07-27T06:47:10.101 INFO:tasks.cephfs.filesystem:scrub status for tag:ac591078-3819-4ade-9981-0f2249333d08 - {'path': '/', 'tag': 'ac591078-3819-4ade-9981-0f2249333d08', 'options': 'recursive,force'} 2024-07-27T06:47:11.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:11 smithi029 ceph-mon[32610]: pgmap v402: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 174 KiB/s wr, 54 op/s 2024-07-27T06:47:11.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.653+0000 7f5944425640 1 -- 172.21.15.29:0/508540677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593c103590 msgr2=0x7f593c1039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:11.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.653+0000 7f5944425640 1 --2- 172.21.15.29:0/508540677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593c103590 0x7f593c1039a0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5938009900 tx=0x7f593802a410 comp rx=0 tx=0).stop 2024-07-27T06:47:11.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.653+0000 7f5944425640 1 -- 172.21.15.29:0/508540677 shutdown_connections 2024-07-27T06:47:11.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.653+0000 7f5944425640 1 --2- 172.21.15.29:0/508540677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c109820 0x7f593c110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:11.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.654+0000 7f5944425640 1 --2- 172.21.15.29:0/508540677 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f593c108ed0 0x7f593c1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:11.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.654+0000 7f5944425640 1 --2- 172.21.15.29:0/508540677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593c103590 0x7f593c1039a0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:11.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.654+0000 7f5944425640 1 -- 172.21.15.29:0/508540677 >> 172.21.15.29:0/508540677 conn(0x7f593c07b280 msgr2=0x7f593c07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:11.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.654+0000 7f5944425640 1 -- 172.21.15.29:0/508540677 shutdown_connections 2024-07-27T06:47:11.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.654+0000 7f5944425640 1 -- 172.21.15.29:0/508540677 wait complete. 2024-07-27T06:47:11.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.654+0000 7f5944425640 1 Processor -- start 2024-07-27T06:47:11.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f5944425640 1 -- start start 2024-07-27T06:47:11.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f5944425640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593c103590 0x7f593c075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f5944425640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f593c108ed0 0x7f593c075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f5944425640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c109820 0x7f593c075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f5944425640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f593c1134f0 con 0x7f593c103590 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f5944425640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f593c113370 con 0x7f593c109820 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f5944425640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f593c113670 con 0x7f593c108ed0 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f59421c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593c103590 0x7f593c075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f59421c1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593c103590 0x7f593c075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43350/0 (socket says 172.21.15.29:43350) 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f59421c1640 1 -- 172.21.15.29:0/1141322690 learned_addr learned my addr 172.21.15.29:0/1141322690 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f59419c0640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f593c108ed0 0x7f593c075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f59429c2640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c109820 0x7f593c075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f59421c1640 1 -- 172.21.15.29:0/1141322690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f593c108ed0 msgr2=0x7f593c075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f59421c1640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f593c108ed0 0x7f593c075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:11.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f59421c1640 1 -- 172.21.15.29:0/1141322690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c109820 msgr2=0x7f593c075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f59421c1640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c109820 0x7f593c075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.655+0000 7f59421c1640 1 -- 172.21.15.29:0/1141322690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593c1a8260 con 0x7f593c103590 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.656+0000 7f59421c1640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593c103590 0x7f593c075140 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5938005940 tx=0x7f59380042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.656+0000 7f59337fe640 1 -- 172.21.15.29:0/1141322690 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593803f030 con 0x7f593c103590 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.656+0000 7f59337fe640 1 -- 172.21.15.29:0/1141322690 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5938003cb0 con 0x7f593c103590 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.656+0000 7f5944425640 1 -- 172.21.15.29:0/1141322690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f593c1a84f0 con 0x7f593c103590 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.656+0000 7f59337fe640 1 -- 172.21.15.29:0/1141322690 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5938035070 con 0x7f593c103590 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.656+0000 7f5944425640 1 -- 172.21.15.29:0/1141322690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f593c1a8a20 con 0x7f593c103590 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.657+0000 7f5944425640 1 -- 172.21.15.29:0/1141322690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f593c1039a0 con 0x7f593c103590 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.659+0000 7f59337fe640 1 -- 172.21.15.29:0/1141322690 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f5938042030 con 0x7f593c103590 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.659+0000 7f59337fe640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f59280b3d30 0x7f59280b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.660+0000 7f59337fe640 1 -- 172.21.15.29:0/1141322690 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f59381049f0 con 0x7f593c103590 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.660+0000 7f59419c0640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f59280b3d30 0x7f59280b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.660+0000 7f59419c0640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f59280b3d30 0x7f59280b6240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f593c105f60 tx=0x7f592c00b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:11.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.664+0000 7f59337fe640 1 -- 172.21.15.29:0/1141322690 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f59380cbdd0 con 0x7f593c103590 2024-07-27T06:47:11.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.844+0000 7f5944425640 1 -- 172.21.15.29:0/1141322690 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f593c063230 con 0x7f59280b3d30 2024-07-27T06:47:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.849+0000 7f59337fe640 1 -- 172.21.15.29:0/1141322690 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259301 (secure 0 0 0) 0x7f593c063230 con 0x7f59280b3d30 2024-07-27T06:47:11.852 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:47:11.853 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-07-27T06:47:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.852+0000 7f5944425640 1 -- 172.21.15.29:0/1141322690 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f59280b3d30 msgr2=0x7f59280b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.852+0000 7f5944425640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f59280b3d30 0x7f59280b6240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f593c105f60 tx=0x7f592c00b390 comp rx=0 tx=0).stop 2024-07-27T06:47:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.852+0000 7f5944425640 1 -- 172.21.15.29:0/1141322690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593c103590 msgr2=0x7f593c075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.852+0000 7f5944425640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593c103590 0x7f593c075140 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5938005940 tx=0x7f59380042f0 comp rx=0 tx=0).stop 2024-07-27T06:47:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.852+0000 7f5944425640 1 -- 172.21.15.29:0/1141322690 shutdown_connections 2024-07-27T06:47:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.852+0000 7f5944425640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f593c109820 0x7f593c075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:11.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.852+0000 7f5944425640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f59280b3d30 0x7f59280b6240 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:11.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.852+0000 7f5944425640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f593c108ed0 0x7f593c075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:11.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.852+0000 7f5944425640 1 --2- 172.21.15.29:0/1141322690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f593c103590 0x7f593c075140 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:11.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.852+0000 7f5944425640 1 -- 172.21.15.29:0/1141322690 >> 172.21.15.29:0/1141322690 conn(0x7f593c07b280 msgr2=0x7f593c107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:11.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.852+0000 7f5944425640 1 -- 172.21.15.29:0/1141322690 shutdown_connections 2024-07-27T06:47:11.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:11.852+0000 7f5944425640 1 -- 172.21.15.29:0/1141322690 wait complete. 2024-07-27T06:47:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:11 smithi077 ceph-mon[37153]: pgmap v402: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 174 KiB/s wr, 54 op/s 2024-07-27T06:47:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:11 smithi047 ceph-mon[36835]: pgmap v402: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 174 KiB/s wr, 54 op/s 2024-07-27T06:47:12.758 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-07-27T06:47:11.687088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1452197,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2577,"num_read_kb":1546,"num_write":613,"num_write_kb":4397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2953401,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":964,"ondisk_log_size":964,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360728,"kb_used_data":11532,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581096,"statfs":{"total":1151940427776,"available":1151571042304,"internally_reserved":0,"allocated":11808768,"data_stored":9878931,"data_compressed":359475,"data_compressed_allocated":479232,"data_compressed_original":4448256,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":786992,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":456,"num_read_kb":356,"num_write":224,"num_write_kb":2088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003033"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483739+0000","last_change":"2024-07-27T06:43:12.802353+0000","last_active":"2024-07-27T06:45:09.483739+0000","last_peered":"2024-07-27T06:45:09.483739+0000","last_clean":"2024-07-27T06:45:09.483739+0000","last_became_active":"2024-07-27T06:43:12.802028+0000","last_became_peered":"2024-07-27T06:43:12.802028+0000","last_unstale":"2024-07-27T06:45:09.483739+0000","last_undegraded":"2024-07-27T06:45:09.483739+0000","last_fullsized":"2024-07-27T06:45:09.483739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:22:24.292567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862408+0000","last_change":"2024-07-27T06:43:12.806269+0000","last_active":"2024-07-27T06:45:09.862408+0000","last_peered":"2024-07-27T06:45:09.862408+0000","last_clean":"2024-07-27T06:45:09.862408+0000","last_became_active":"2024-07-27T06:43:12.803232+0000","last_became_peered":"2024-07-27T06:43:12.803232+0000","last_unstale":"2024-07-27T06:45:09.862408+0000","last_undegraded":"2024-07-27T06:45:09.862408+0000","last_fullsized":"2024-07-27T06:45:09.862408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:57:53.673850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862259+0000","last_change":"2024-07-27T06:43:12.805542+0000","last_active":"2024-07-27T06:45:09.862259+0000","last_peered":"2024-07-27T06:45:09.862259+0000","last_clean":"2024-07-27T06:45:09.862259+0000","last_became_active":"2024-07-27T06:43:12.804951+0000","last_became_peered":"2024-07-27T06:43:12.804951+0000","last_unstale":"2024-07-27T06:45:09.862259+0000","last_undegraded":"2024-07-27T06:45:09.862259+0000","last_fullsized":"2024-07-27T06:45:09.862259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:06:40.469695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862158+0000","last_change":"2024-07-27T06:43:12.796950+0000","last_active":"2024-07-27T06:45:09.862158+0000","last_peered":"2024-07-27T06:45:09.862158+0000","last_clean":"2024-07-27T06:45:09.862158+0000","last_became_active":"2024-07-27T06:43:12.796549+0000","last_became_peered":"2024-07-27T06:43:12.796549+0000","last_unstale":"2024-07-27T06:45:09.862158+0000","last_undegraded":"2024-07-27T06:45:09.862158+0000","last_fullsized":"2024-07-27T06:45:09.862158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:30:36.391046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486015+0000","last_change":"2024-07-27T06:43:12.799838+0000","last_active":"2024-07-27T06:45:09.486015+0000","last_peered":"2024-07-27T06:45:09.486015+0000","last_clean":"2024-07-27T06:45:09.486015+0000","last_became_active":"2024-07-27T06:43:12.799462+0000","last_became_peered":"2024-07-27T06:43:12.799462+0000","last_unstale":"2024-07-27T06:45:09.486015+0000","last_undegraded":"2024-07-27T06:45:09.486015+0000","last_fullsized":"2024-07-27T06:45:09.486015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:45:19.371259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481631+0000","last_change":"2024-07-27T06:43:12.797187+0000","last_active":"2024-07-27T06:45:09.481631+0000","last_peered":"2024-07-27T06:45:09.481631+0000","last_clean":"2024-07-27T06:45:09.481631+0000","last_became_active":"2024-07-27T06:43:12.796633+0000","last_became_peered":"2024-07-27T06:43:12.796633+0000","last_unstale":"2024-07-27T06:45:09.481631+0000","last_undegraded":"2024-07-27T06:45:09.481631+0000","last_fullsized":"2024-07-27T06:45:09.481631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:22:20.882899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488423+0000","last_change":"2024-07-27T06:43:12.793786+0000","last_active":"2024-07-27T06:45:09.488423+0000","last_peered":"2024-07-27T06:45:09.488423+0000","last_clean":"2024-07-27T06:45:09.488423+0000","last_became_active":"2024-07-27T06:43:12.793413+0000","last_became_peered":"2024-07-27T06:43:12.793413+0000","last_unstale":"2024-07-27T06:45:09.488423+0000","last_undegraded":"2024-07-27T06:45:09.488423+0000","last_fullsized":"2024-07-27T06:45:09.488423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:14.015373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485385+0000","last_change":"2024-07-27T06:43:12.791304+0000","last_active":"2024-07-27T06:45:09.485385+0000","last_peered":"2024-07-27T06:45:09.485385+0000","last_clean":"2024-07-27T06:45:09.485385+0000","last_became_active":"2024-07-27T06:43:12.790801+0000","last_became_peered":"2024-07-27T06:43:12.790801+0000","last_unstale":"2024-07-27T06:45:09.485385+0000","last_undegraded":"2024-07-27T06:45:09.485385+0000","last_fullsized":"2024-07-27T06:45:09.485385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:12:37.606748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822529+0000","last_change":"2024-07-27T06:43:12.797778+0000","last_active":"2024-07-27T06:45:09.822529+0000","last_peered":"2024-07-27T06:45:09.822529+0000","last_clean":"2024-07-27T06:45:09.822529+0000","last_became_active":"2024-07-27T06:43:12.796784+0000","last_became_peered":"2024-07-27T06:43:12.796784+0000","last_unstale":"2024-07-27T06:45:09.822529+0000","last_undegraded":"2024-07-27T06:45:09.822529+0000","last_fullsized":"2024-07-27T06:45:09.822529+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:52:22.275282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.482109+0000","last_change":"2024-07-27T06:43:12.798970+0000","last_active":"2024-07-27T06:45:09.482109+0000","last_peered":"2024-07-27T06:45:09.482109+0000","last_clean":"2024-07-27T06:45:09.482109+0000","last_became_active":"2024-07-27T06:43:12.798549+0000","last_became_peered":"2024-07-27T06:43:12.798549+0000","last_unstale":"2024-07-27T06:45:09.482109+0000","last_undegraded":"2024-07-27T06:45:09.482109+0000","last_fullsized":"2024-07-27T06:45:09.482109+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:58:32.464429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488604+0000","last_change":"2024-07-27T06:43:12.809405+0000","last_active":"2024-07-27T06:45:09.488604+0000","last_peered":"2024-07-27T06:45:09.488604+0000","last_clean":"2024-07-27T06:45:09.488604+0000","last_became_active":"2024-07-27T06:43:12.809030+0000","last_became_peered":"2024-07-27T06:43:12.809030+0000","last_unstale":"2024-07-27T06:45:09.488604+0000","last_undegraded":"2024-07-27T06:45:09.488604+0000","last_fullsized":"2024-07-27T06:45:09.488604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:46:35.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862193+0000","last_change":"2024-07-27T06:43:12.802647+0000","last_active":"2024-07-27T06:45:09.862193+0000","last_peered":"2024-07-27T06:45:09.862193+0000","last_clean":"2024-07-27T06:45:09.862193+0000","last_became_active":"2024-07-27T06:43:12.802247+0000","last_became_peered":"2024-07-27T06:43:12.802247+0000","last_unstale":"2024-07-27T06:45:09.862193+0000","last_undegraded":"2024-07-27T06:45:09.862193+0000","last_fullsized":"2024-07-27T06:45:09.862193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:44:24.564852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481628+0000","last_change":"2024-07-27T06:43:12.809140+0000","last_active":"2024-07-27T06:45:09.481628+0000","last_peered":"2024-07-27T06:45:09.481628+0000","last_clean":"2024-07-27T06:45:09.481628+0000","last_became_active":"2024-07-27T06:43:12.808692+0000","last_became_peered":"2024-07-27T06:43:12.808692+0000","last_unstale":"2024-07-27T06:45:09.481628+0000","last_undegraded":"2024-07-27T06:45:09.481628+0000","last_fullsized":"2024-07-27T06:45:09.481628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:37:45.326234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862226+0000","last_change":"2024-07-27T06:43:12.802838+0000","last_active":"2024-07-27T06:45:09.862226+0000","last_peered":"2024-07-27T06:45:09.862226+0000","last_clean":"2024-07-27T06:45:09.862226+0000","last_became_active":"2024-07-27T06:43:12.802503+0000","last_became_peered":"2024-07-27T06:43:12.802503+0000","last_unstale":"2024-07-27T06:45:09.862226+0000","last_undegraded":"2024-07-27T06:45:09.862226+0000","last_fullsized":"2024-07-27T06:45:09.862226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:41:03.428162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862170+0000","last_change":"2024-07-27T06:43:12.801626+0000","last_active":"2024-07-27T06:45:09.862170+0000","last_peered":"2024-07-27T06:45:09.862170+0000","last_clean":"2024-07-27T06:45:09.862170+0000","last_became_active":"2024-07-27T06:43:12.801266+0000","last_became_peered":"2024-07-27T06:43:12.801266+0000","last_unstale":"2024-07-27T06:45:09.862170+0000","last_undegraded":"2024-07-27T06:45:09.862170+0000","last_fullsized":"2024-07-27T06:45:09.862170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:54:31.934223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.821975+0000","last_change":"2024-07-27T06:43:12.804543+0000","last_active":"2024-07-27T06:45:09.821975+0000","last_peered":"2024-07-27T06:45:09.821975+0000","last_clean":"2024-07-27T06:45:09.821975+0000","last_became_active":"2024-07-27T06:43:12.804173+0000","last_became_peered":"2024-07-27T06:43:12.804173+0000","last_unstale":"2024-07-27T06:45:09.821975+0000","last_undegraded":"2024-07-27T06:45:09.821975+0000","last_fullsized":"2024-07-27T06:45:09.821975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:18:25.337333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483797+0000","last_change":"2024-07-27T06:43:12.798604+0000","last_active":"2024-07-27T06:45:09.483797+0000","last_peered":"2024-07-27T06:45:09.483797+0000","last_clean":"2024-07-27T06:45:09.483797+0000","last_became_active":"2024-07-27T06:43:12.798228+0000","last_became_peered":"2024-07-27T06:43:12.798228+0000","last_unstale":"2024-07-27T06:45:09.483797+0000","last_undegraded":"2024-07-27T06:45:09.483797+0000","last_fullsized":"2024-07-27T06:45:09.483797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:04:35.851399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633341+0000","last_change":"2024-07-27T06:43:12.800452+0000","last_active":"2024-07-27T06:45:09.633341+0000","last_peered":"2024-07-27T06:45:09.633341+0000","last_clean":"2024-07-27T06:45:09.633341+0000","last_became_active":"2024-07-27T06:43:12.800084+0000","last_became_peered":"2024-07-27T06:43:12.800084+0000","last_unstale":"2024-07-27T06:45:09.633341+0000","last_undegraded":"2024-07-27T06:45:09.633341+0000","last_fullsized":"2024-07-27T06:45:09.633341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:54:27.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485773+0000","last_change":"2024-07-27T06:43:12.800761+0000","last_active":"2024-07-27T06:45:09.485773+0000","last_peered":"2024-07-27T06:45:09.485773+0000","last_clean":"2024-07-27T06:45:09.485773+0000","last_became_active":"2024-07-27T06:43:12.800414+0000","last_became_peered":"2024-07-27T06:43:12.800414+0000","last_unstale":"2024-07-27T06:45:09.485773+0000","last_undegraded":"2024-07-27T06:45:09.485773+0000","last_fullsized":"2024-07-27T06:45:09.485773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:02:47.247522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483375+0000","last_change":"2024-07-27T06:43:12.795470+0000","last_active":"2024-07-27T06:45:09.483375+0000","last_peered":"2024-07-27T06:45:09.483375+0000","last_clean":"2024-07-27T06:45:09.483375+0000","last_became_active":"2024-07-27T06:43:12.795136+0000","last_became_peered":"2024-07-27T06:43:12.795136+0000","last_unstale":"2024-07-27T06:45:09.483375+0000","last_undegraded":"2024-07-27T06:45:09.483375+0000","last_fullsized":"2024-07-27T06:45:09.483375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:01:22.178699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481748+0000","last_change":"2024-07-27T06:43:12.803897+0000","last_active":"2024-07-27T06:45:09.481748+0000","last_peered":"2024-07-27T06:45:09.481748+0000","last_clean":"2024-07-27T06:45:09.481748+0000","last_became_active":"2024-07-27T06:43:12.803546+0000","last_became_peered":"2024-07-27T06:43:12.803546+0000","last_unstale":"2024-07-27T06:45:09.481748+0000","last_undegraded":"2024-07-27T06:45:09.481748+0000","last_fullsized":"2024-07-27T06:45:09.481748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:13:50.816831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822201+0000","last_change":"2024-07-27T06:44:09.238514+0000","last_active":"2024-07-27T06:45:09.822201+0000","last_peered":"2024-07-27T06:45:09.822201+0000","last_clean":"2024-07-27T06:45:09.822201+0000","last_became_active":"2024-07-27T06:44:09.237903+0000","last_became_peered":"2024-07-27T06:44:09.237903+0000","last_unstale":"2024-07-27T06:45:09.822201+0000","last_undegraded":"2024-07-27T06:45:09.822201+0000","last_fullsized":"2024-07-27T06:45:09.822201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:20:59.887491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485428+0000","last_change":"2024-07-27T06:43:12.788570+0000","last_active":"2024-07-27T06:45:09.485428+0000","last_peered":"2024-07-27T06:45:09.485428+0000","last_clean":"2024-07-27T06:45:09.485428+0000","last_became_active":"2024-07-27T06:43:12.788199+0000","last_became_peered":"2024-07-27T06:43:12.788199+0000","last_unstale":"2024-07-27T06:45:09.485428+0000","last_undegraded":"2024-07-27T06:45:09.485428+0000","last_fullsized":"2024-07-27T06:45:09.485428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:44:26.751094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862750+0000","last_change":"2024-07-27T06:43:12.792866+0000","last_active":"2024-07-27T06:45:09.862750+0000","last_peered":"2024-07-27T06:45:09.862750+0000","last_clean":"2024-07-27T06:45:09.862750+0000","last_became_active":"2024-07-27T06:43:12.792389+0000","last_became_peered":"2024-07-27T06:43:12.792389+0000","last_unstale":"2024-07-27T06:45:09.862750+0000","last_undegraded":"2024-07-27T06:45:09.862750+0000","last_fullsized":"2024-07-27T06:45:09.862750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:10:37.367642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485830+0000","last_change":"2024-07-27T06:43:12.797002+0000","last_active":"2024-07-27T06:45:09.485830+0000","last_peered":"2024-07-27T06:45:09.485830+0000","last_clean":"2024-07-27T06:45:09.485830+0000","last_became_active":"2024-07-27T06:43:12.796651+0000","last_became_peered":"2024-07-27T06:43:12.796651+0000","last_unstale":"2024-07-27T06:45:09.485830+0000","last_undegraded":"2024-07-27T06:45:09.485830+0000","last_fullsized":"2024-07-27T06:45:09.485830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:28:06.934042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485867+0000","last_change":"2024-07-27T06:43:12.798549+0000","last_active":"2024-07-27T06:45:09.485867+0000","last_peered":"2024-07-27T06:45:09.485867+0000","last_clean":"2024-07-27T06:45:09.485867+0000","last_became_active":"2024-07-27T06:43:12.798201+0000","last_became_peered":"2024-07-27T06:43:12.798201+0000","last_unstale":"2024-07-27T06:45:09.485867+0000","last_undegraded":"2024-07-27T06:45:09.485867+0000","last_fullsized":"2024-07-27T06:45:09.485867+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:59:47.525536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483656+0000","last_change":"2024-07-27T06:43:12.791053+0000","last_active":"2024-07-27T06:45:09.483656+0000","last_peered":"2024-07-27T06:45:09.483656+0000","last_clean":"2024-07-27T06:45:09.483656+0000","last_became_active":"2024-07-27T06:43:12.790718+0000","last_became_peered":"2024-07-27T06:43:12.790718+0000","last_unstale":"2024-07-27T06:45:09.483656+0000","last_undegraded":"2024-07-27T06:45:09.483656+0000","last_fullsized":"2024-07-27T06:45:09.483656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:09.455399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481811+0000","last_change":"2024-07-27T06:43:12.804180+0000","last_active":"2024-07-27T06:45:09.481811+0000","last_peered":"2024-07-27T06:45:09.481811+0000","last_clean":"2024-07-27T06:45:09.481811+0000","last_became_active":"2024-07-27T06:43:12.803881+0000","last_became_peered":"2024-07-27T06:43:12.803881+0000","last_unstale":"2024-07-27T06:45:09.481811+0000","last_undegraded":"2024-07-27T06:45:09.481811+0000","last_fullsized":"2024-07-27T06:45:09.481811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:01:05.009233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862727+0000","last_change":"2024-07-27T06:43:12.809150+0000","last_active":"2024-07-27T06:45:09.862727+0000","last_peered":"2024-07-27T06:45:09.862727+0000","last_clean":"2024-07-27T06:45:09.862727+0000","last_became_active":"2024-07-27T06:43:12.808428+0000","last_became_peered":"2024-07-27T06:43:12.808428+0000","last_unstale":"2024-07-27T06:45:09.862727+0000","last_undegraded":"2024-07-27T06:45:09.862727+0000","last_fullsized":"2024-07-27T06:45:09.862727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:07:13.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486325+0000","last_change":"2024-07-27T06:44:09.233785+0000","last_active":"2024-07-27T06:45:09.486325+0000","last_peered":"2024-07-27T06:45:09.486325+0000","last_clean":"2024-07-27T06:45:09.486325+0000","last_became_active":"2024-07-27T06:44:09.233184+0000","last_became_peered":"2024-07-27T06:44:09.233184+0000","last_unstale":"2024-07-27T06:45:09.486325+0000","last_undegraded":"2024-07-27T06:45:09.486325+0000","last_fullsized":"2024-07-27T06:45:09.486325+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:12:48.556383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486129+0000","last_change":"2024-07-27T06:44:09.320182+0000","last_active":"2024-07-27T06:45:09.486129+0000","last_peered":"2024-07-27T06:45:09.486129+0000","last_clean":"2024-07-27T06:45:09.486129+0000","last_became_active":"2024-07-27T06:44:09.319661+0000","last_became_peered":"2024-07-27T06:44:09.319661+0000","last_unstale":"2024-07-27T06:45:09.486129+0000","last_undegraded":"2024-07-27T06:45:09.486129+0000","last_fullsized":"2024-07-27T06:45:09.486129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:52:11.006680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483342+0000","last_change":"2024-07-27T06:43:12.787866+0000","last_active":"2024-07-27T06:45:09.483342+0000","last_peered":"2024-07-27T06:45:09.483342+0000","last_clean":"2024-07-27T06:45:09.483342+0000","last_became_active":"2024-07-27T06:43:12.787513+0000","last_became_peered":"2024-07-27T06:43:12.787513+0000","last_unstale":"2024-07-27T06:45:09.483342+0000","last_undegraded":"2024-07-27T06:45:09.483342+0000","last_fullsized":"2024-07-27T06:45:09.483342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:48:57.876588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485381+0000","last_change":"2024-07-27T06:43:12.791158+0000","last_active":"2024-07-27T06:45:09.485381+0000","last_peered":"2024-07-27T06:45:09.485381+0000","last_clean":"2024-07-27T06:45:09.485381+0000","last_became_active":"2024-07-27T06:43:12.790771+0000","last_became_peered":"2024-07-27T06:43:12.790771+0000","last_unstale":"2024-07-27T06:45:09.485381+0000","last_undegraded":"2024-07-27T06:45:09.485381+0000","last_fullsized":"2024-07-27T06:45:09.485381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:10:18.739265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'7","reported_seq":259,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:08.225697+0000","last_change":"2024-07-27T06:43:09.769281+0000","last_active":"2024-07-27T06:47:08.225697+0000","last_peered":"2024-07-27T06:47:08.225697+0000","last_clean":"2024-07-27T06:47:08.225697+0000","last_became_active":"2024-07-27T06:43:09.768864+0000","last_became_peered":"2024-07-27T06:43:09.768864+0000","last_unstale":"2024-07-27T06:47:08.225697+0000","last_undegraded":"2024-07-27T06:47:08.225697+0000","last_fullsized":"2024-07-27T06:47:08.225697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:55:14.191180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":220,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862960+0000","last_change":"2024-07-27T06:44:09.319930+0000","last_active":"2024-07-27T06:45:09.862960+0000","last_peered":"2024-07-27T06:45:09.862960+0000","last_clean":"2024-07-27T06:45:09.862960+0000","last_became_active":"2024-07-27T06:44:09.319535+0000","last_became_peered":"2024-07-27T06:44:09.319535+0000","last_unstale":"2024-07-27T06:45:09.862960+0000","last_undegraded":"2024-07-27T06:45:09.862960+0000","last_fullsized":"2024-07-27T06:45:09.862960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:39:52.887860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481674+0000","last_change":"2024-07-27T06:43:09.772227+0000","last_active":"2024-07-27T06:45:09.481674+0000","last_peered":"2024-07-27T06:45:09.481674+0000","last_clean":"2024-07-27T06:45:09.481674+0000","last_became_active":"2024-07-27T06:43:09.771812+0000","last_became_peered":"2024-07-27T06:43:09.771812+0000","last_unstale":"2024-07-27T06:45:09.481674+0000","last_undegraded":"2024-07-27T06:45:09.481674+0000","last_fullsized":"2024-07-27T06:45:09.481674+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:27:28.054590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481030+0000","last_change":"2024-07-27T06:43:09.766490+0000","last_active":"2024-07-27T06:45:09.481030+0000","last_peered":"2024-07-27T06:45:09.481030+0000","last_clean":"2024-07-27T06:45:09.481030+0000","last_became_active":"2024-07-27T06:43:09.765973+0000","last_became_peered":"2024-07-27T06:43:09.765973+0000","last_unstale":"2024-07-27T06:45:09.481030+0000","last_undegraded":"2024-07-27T06:45:09.481030+0000","last_fullsized":"2024-07-27T06:45:09.481030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:00:02.913080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485458+0000","last_change":"2024-07-27T06:43:12.805234+0000","last_active":"2024-07-27T06:45:09.485458+0000","last_peered":"2024-07-27T06:45:09.485458+0000","last_clean":"2024-07-27T06:45:09.485458+0000","last_became_active":"2024-07-27T06:43:12.804908+0000","last_became_peered":"2024-07-27T06:43:12.804908+0000","last_unstale":"2024-07-27T06:45:09.485458+0000","last_undegraded":"2024-07-27T06:45:09.485458+0000","last_fullsized":"2024-07-27T06:45:09.485458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:09:34.988755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862248+0000","last_change":"2024-07-27T06:43:09.774201+0000","last_active":"2024-07-27T06:45:09.862248+0000","last_peered":"2024-07-27T06:45:09.862248+0000","last_clean":"2024-07-27T06:45:09.862248+0000","last_became_active":"2024-07-27T06:43:09.773817+0000","last_became_peered":"2024-07-27T06:43:09.773817+0000","last_unstale":"2024-07-27T06:45:09.862248+0000","last_undegraded":"2024-07-27T06:45:09.862248+0000","last_fullsized":"2024-07-27T06:45:09.862248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:06.166251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633110+0000","last_change":"2024-07-27T06:43:12.800174+0000","last_active":"2024-07-27T06:45:09.633110+0000","last_peered":"2024-07-27T06:45:09.633110+0000","last_clean":"2024-07-27T06:45:09.633110+0000","last_became_active":"2024-07-27T06:43:12.799769+0000","last_became_peered":"2024-07-27T06:43:12.799769+0000","last_unstale":"2024-07-27T06:45:09.633110+0000","last_undegraded":"2024-07-27T06:45:09.633110+0000","last_fullsized":"2024-07-27T06:45:09.633110+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:43:06.890372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":465,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:06.591075+0000","last_change":"2024-07-27T06:43:09.776612+0000","last_active":"2024-07-27T06:47:06.591075+0000","last_peered":"2024-07-27T06:47:06.591075+0000","last_clean":"2024-07-27T06:47:06.591075+0000","last_became_active":"2024-07-27T06:43:09.775761+0000","last_became_peered":"2024-07-27T06:43:09.775761+0000","last_unstale":"2024-07-27T06:47:06.591075+0000","last_undegraded":"2024-07-27T06:47:06.591075+0000","last_fullsized":"2024-07-27T06:47:06.591075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:42:52.132357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":427,"num_read_kb":427,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822478+0000","last_change":"2024-07-27T06:43:12.797200+0000","last_active":"2024-07-27T06:45:09.822478+0000","last_peered":"2024-07-27T06:45:09.822478+0000","last_clean":"2024-07-27T06:45:09.822478+0000","last_became_active":"2024-07-27T06:43:12.796288+0000","last_became_peered":"2024-07-27T06:43:12.796288+0000","last_unstale":"2024-07-27T06:45:09.822478+0000","last_undegraded":"2024-07-27T06:45:09.822478+0000","last_fullsized":"2024-07-27T06:45:09.822478+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:31:27.603883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488738+0000","last_change":"2024-07-27T06:43:09.772658+0000","last_active":"2024-07-27T06:45:09.488738+0000","last_peered":"2024-07-27T06:45:09.488738+0000","last_clean":"2024-07-27T06:45:09.488738+0000","last_became_active":"2024-07-27T06:43:09.772277+0000","last_became_peered":"2024-07-27T06:43:09.772277+0000","last_unstale":"2024-07-27T06:45:09.488738+0000","last_undegraded":"2024-07-27T06:45:09.488738+0000","last_fullsized":"2024-07-27T06:45:09.488738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:09:16.854718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862551+0000","last_change":"2024-07-27T06:43:12.809643+0000","last_active":"2024-07-27T06:45:09.862551+0000","last_peered":"2024-07-27T06:45:09.862551+0000","last_clean":"2024-07-27T06:45:09.862551+0000","last_became_active":"2024-07-27T06:43:12.809020+0000","last_became_peered":"2024-07-27T06:43:12.809020+0000","last_unstale":"2024-07-27T06:45:09.862551+0000","last_undegraded":"2024-07-27T06:45:09.862551+0000","last_fullsized":"2024-07-27T06:45:09.862551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:08:03.062914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.863022+0000","last_change":"2024-07-27T06:43:10.534042+0000","last_active":"2024-07-27T06:45:09.863022+0000","last_peered":"2024-07-27T06:45:09.863022+0000","last_clean":"2024-07-27T06:45:09.863022+0000","last_became_active":"2024-07-27T06:43:10.533615+0000","last_became_peered":"2024-07-27T06:43:10.533615+0000","last_unstale":"2024-07-27T06:45:09.863022+0000","last_undegraded":"2024-07-27T06:45:09.863022+0000","last_fullsized":"2024-07-27T06:45:09.863022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:09:32.235633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822222+0000","last_change":"2024-07-27T06:43:12.798764+0000","last_active":"2024-07-27T06:45:09.822222+0000","last_peered":"2024-07-27T06:45:09.822222+0000","last_clean":"2024-07-27T06:45:09.822222+0000","last_became_active":"2024-07-27T06:43:12.797500+0000","last_became_peered":"2024-07-27T06:43:12.797500+0000","last_unstale":"2024-07-27T06:45:09.822222+0000","last_undegraded":"2024-07-27T06:45:09.822222+0000","last_fullsized":"2024-07-27T06:45:09.822222+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:12:09.368711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"83'37","reported_seq":510,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:07.223137+0000","last_change":"2024-07-27T06:43:09.775954+0000","last_active":"2024-07-27T06:47:07.223137+0000","last_peered":"2024-07-27T06:47:07.223137+0000","last_clean":"2024-07-27T06:47:07.223137+0000","last_became_active":"2024-07-27T06:43:09.775498+0000","last_became_peered":"2024-07-27T06:43:09.775498+0000","last_unstale":"2024-07-27T06:47:07.223137+0000","last_undegraded":"2024-07-27T06:47:07.223137+0000","last_fullsized":"2024-07-27T06:47:07.223137+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:08:46.965525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66277,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":441,"num_read_kb":286,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486559+0000","last_change":"2024-07-27T06:43:12.797220+0000","last_active":"2024-07-27T06:45:09.486559+0000","last_peered":"2024-07-27T06:45:09.486559+0000","last_clean":"2024-07-27T06:45:09.486559+0000","last_became_active":"2024-07-27T06:43:12.796889+0000","last_became_peered":"2024-07-27T06:43:12.796889+0000","last_unstale":"2024-07-27T06:45:09.486559+0000","last_undegraded":"2024-07-27T06:45:09.486559+0000","last_fullsized":"2024-07-27T06:45:09.486559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:24:36.275738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485662+0000","last_change":"2024-07-27T06:43:09.776600+0000","last_active":"2024-07-27T06:45:09.485662+0000","last_peered":"2024-07-27T06:45:09.485662+0000","last_clean":"2024-07-27T06:45:09.485662+0000","last_became_active":"2024-07-27T06:43:09.776163+0000","last_became_peered":"2024-07-27T06:43:09.776163+0000","last_unstale":"2024-07-27T06:45:09.485662+0000","last_undegraded":"2024-07-27T06:45:09.485662+0000","last_fullsized":"2024-07-27T06:45:09.485662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:22:15.947916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481215+0000","last_change":"2024-07-27T06:43:12.804984+0000","last_active":"2024-07-27T06:45:09.481215+0000","last_peered":"2024-07-27T06:45:09.481215+0000","last_clean":"2024-07-27T06:45:09.481215+0000","last_became_active":"2024-07-27T06:43:12.804644+0000","last_became_peered":"2024-07-27T06:43:12.804644+0000","last_unstale":"2024-07-27T06:45:09.481215+0000","last_undegraded":"2024-07-27T06:45:09.481215+0000","last_fullsized":"2024-07-27T06:45:09.481215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.355981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862246+0000","last_change":"2024-07-27T06:43:09.772489+0000","last_active":"2024-07-27T06:45:09.862246+0000","last_peered":"2024-07-27T06:45:09.862246+0000","last_clean":"2024-07-27T06:45:09.862246+0000","last_became_active":"2024-07-27T06:43:09.772069+0000","last_became_peered":"2024-07-27T06:43:09.772069+0000","last_unstale":"2024-07-27T06:45:09.862246+0000","last_undegraded":"2024-07-27T06:45:09.862246+0000","last_fullsized":"2024-07-27T06:45:09.862246+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:15:30.549971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488589+0000","last_change":"2024-07-27T06:43:12.803376+0000","last_active":"2024-07-27T06:45:09.488589+0000","last_peered":"2024-07-27T06:45:09.488589+0000","last_clean":"2024-07-27T06:45:09.488589+0000","last_became_active":"2024-07-27T06:43:12.802961+0000","last_became_peered":"2024-07-27T06:43:12.802961+0000","last_unstale":"2024-07-27T06:45:09.488589+0000","last_undegraded":"2024-07-27T06:45:09.488589+0000","last_fullsized":"2024-07-27T06:45:09.488589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:00:24.649877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":256,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:09.226417+0000","last_change":"2024-07-27T06:43:09.773182+0000","last_active":"2024-07-27T06:47:09.226417+0000","last_peered":"2024-07-27T06:47:09.226417+0000","last_clean":"2024-07-27T06:47:09.226417+0000","last_became_active":"2024-07-27T06:43:09.772126+0000","last_became_peered":"2024-07-27T06:43:09.772126+0000","last_unstale":"2024-07-27T06:47:09.226417+0000","last_undegraded":"2024-07-27T06:47:09.226417+0000","last_fullsized":"2024-07-27T06:47:09.226417+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:45:26.598799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":220,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488424+0000","last_change":"2024-07-27T06:43:12.799385+0000","last_active":"2024-07-27T06:45:09.488424+0000","last_peered":"2024-07-27T06:45:09.488424+0000","last_clean":"2024-07-27T06:45:09.488424+0000","last_became_active":"2024-07-27T06:43:12.798639+0000","last_became_peered":"2024-07-27T06:43:12.798639+0000","last_unstale":"2024-07-27T06:45:09.488424+0000","last_undegraded":"2024-07-27T06:45:09.488424+0000","last_fullsized":"2024-07-27T06:45:09.488424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:56:29.462491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485303+0000","last_change":"2024-07-27T06:43:10.535429+0000","last_active":"2024-07-27T06:45:09.485303+0000","last_peered":"2024-07-27T06:45:09.485303+0000","last_clean":"2024-07-27T06:45:09.485303+0000","last_became_active":"2024-07-27T06:43:10.534952+0000","last_became_peered":"2024-07-27T06:43:10.534952+0000","last_unstale":"2024-07-27T06:45:09.485303+0000","last_undegraded":"2024-07-27T06:45:09.485303+0000","last_fullsized":"2024-07-27T06:45:09.485303+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:54:20.018263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862706+0000","last_change":"2024-07-27T06:43:12.804266+0000","last_active":"2024-07-27T06:45:09.862706+0000","last_peered":"2024-07-27T06:45:09.862706+0000","last_clean":"2024-07-27T06:45:09.862706+0000","last_became_active":"2024-07-27T06:43:12.803888+0000","last_became_peered":"2024-07-27T06:43:12.803888+0000","last_unstale":"2024-07-27T06:45:09.862706+0000","last_undegraded":"2024-07-27T06:45:09.862706+0000","last_fullsized":"2024-07-27T06:45:09.862706+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:48:00.490592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485827+0000","last_change":"2024-07-27T06:43:09.777656+0000","last_active":"2024-07-27T06:45:09.485827+0000","last_peered":"2024-07-27T06:45:09.485827+0000","last_clean":"2024-07-27T06:45:09.485827+0000","last_became_active":"2024-07-27T06:43:09.776620+0000","last_became_peered":"2024-07-27T06:43:09.776620+0000","last_unstale":"2024-07-27T06:45:09.485827+0000","last_undegraded":"2024-07-27T06:45:09.485827+0000","last_fullsized":"2024-07-27T06:45:09.485827+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:50.203623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483544+0000","last_change":"2024-07-27T06:43:12.796179+0000","last_active":"2024-07-27T06:45:09.483544+0000","last_peered":"2024-07-27T06:45:09.483544+0000","last_clean":"2024-07-27T06:45:09.483544+0000","last_became_active":"2024-07-27T06:43:12.795701+0000","last_became_peered":"2024-07-27T06:43:12.795701+0000","last_unstale":"2024-07-27T06:45:09.483544+0000","last_undegraded":"2024-07-27T06:45:09.483544+0000","last_fullsized":"2024-07-27T06:45:09.483544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:20:12.236769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481627+0000","last_change":"2024-07-27T06:43:09.769949+0000","last_active":"2024-07-27T06:45:09.481627+0000","last_peered":"2024-07-27T06:45:09.481627+0000","last_clean":"2024-07-27T06:45:09.481627+0000","last_became_active":"2024-07-27T06:43:09.769602+0000","last_became_peered":"2024-07-27T06:43:09.769602+0000","last_unstale":"2024-07-27T06:45:09.481627+0000","last_undegraded":"2024-07-27T06:45:09.481627+0000","last_fullsized":"2024-07-27T06:45:09.481627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:28:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488869+0000","last_change":"2024-07-27T06:43:12.802230+0000","last_active":"2024-07-27T06:45:09.488869+0000","last_peered":"2024-07-27T06:45:09.488869+0000","last_clean":"2024-07-27T06:45:09.488869+0000","last_became_active":"2024-07-27T06:43:12.800944+0000","last_became_peered":"2024-07-27T06:43:12.800944+0000","last_unstale":"2024-07-27T06:45:09.488869+0000","last_undegraded":"2024-07-27T06:45:09.488869+0000","last_fullsized":"2024-07-27T06:45:09.488869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:05:54.585955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'62","reported_seq":124,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:03.049948+0000","last_change":"2024-07-27T06:43:09.768239+0000","last_active":"2024-07-27T06:47:03.049948+0000","last_peered":"2024-07-27T06:47:03.049948+0000","last_clean":"2024-07-27T06:47:03.049948+0000","last_became_active":"2024-07-27T06:43:09.767584+0000","last_became_peered":"2024-07-27T06:43:09.767584+0000","last_unstale":"2024-07-27T06:47:03.049948+0000","last_undegraded":"2024-07-27T06:47:03.049948+0000","last_fullsized":"2024-07-27T06:47:03.049948+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:35:39.552246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393846,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":32,"num_write":71,"num_write_kb":873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488571+0000","last_change":"2024-07-27T06:43:12.803779+0000","last_active":"2024-07-27T06:45:09.488571+0000","last_peered":"2024-07-27T06:45:09.488571+0000","last_clean":"2024-07-27T06:45:09.488571+0000","last_became_active":"2024-07-27T06:43:12.803359+0000","last_became_peered":"2024-07-27T06:43:12.803359+0000","last_unstale":"2024-07-27T06:45:09.488571+0000","last_undegraded":"2024-07-27T06:45:09.488571+0000","last_fullsized":"2024-07-27T06:45:09.488571+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:32:53.167039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'7","reported_seq":209,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:09.597024+0000","last_change":"2024-07-27T06:44:09.241782+0000","last_active":"2024-07-27T06:47:09.597024+0000","last_peered":"2024-07-27T06:47:09.597024+0000","last_clean":"2024-07-27T06:47:09.597024+0000","last_became_active":"2024-07-27T06:44:09.234692+0000","last_became_peered":"2024-07-27T06:44:09.234692+0000","last_unstale":"2024-07-27T06:47:09.597024+0000","last_undegraded":"2024-07-27T06:47:09.597024+0000","last_fullsized":"2024-07-27T06:47:09.597024+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:45:02.155686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":186,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1561,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488800+0000","last_change":"2024-07-27T06:43:12.805830+0000","last_active":"2024-07-27T06:45:09.488800+0000","last_peered":"2024-07-27T06:45:09.488800+0000","last_clean":"2024-07-27T06:45:09.488800+0000","last_became_active":"2024-07-27T06:43:12.805324+0000","last_became_peered":"2024-07-27T06:43:12.805324+0000","last_unstale":"2024-07-27T06:45:09.488800+0000","last_undegraded":"2024-07-27T06:45:09.488800+0000","last_fullsized":"2024-07-27T06:45:09.488800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:38:31.430676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":109,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.489084+0000","last_change":"2024-07-27T06:41:43.549132+0000","last_active":"2024-07-27T06:45:09.489084+0000","last_peered":"2024-07-27T06:45:09.489084+0000","last_clean":"2024-07-27T06:45:09.489084+0000","last_became_active":"2024-07-27T06:41:43.542635+0000","last_became_peered":"2024-07-27T06:41:43.542635+0000","last_unstale":"2024-07-27T06:45:09.489084+0000","last_undegraded":"2024-07-27T06:45:09.489084+0000","last_fullsized":"2024-07-27T06:45:09.489084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_clean_scrub_stamp":"2024-07-27T06:38:21.639907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:10:06.417387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.861700+0000","last_change":"2024-07-27T06:43:09.768774+0000","last_active":"2024-07-27T06:45:09.861700+0000","last_peered":"2024-07-27T06:45:09.861700+0000","last_clean":"2024-07-27T06:45:09.861700+0000","last_became_active":"2024-07-27T06:43:09.768345+0000","last_became_peered":"2024-07-27T06:43:09.768345+0000","last_unstale":"2024-07-27T06:45:09.861700+0000","last_undegraded":"2024-07-27T06:45:09.861700+0000","last_fullsized":"2024-07-27T06:45:09.861700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:40:49.411746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485670+0000","last_change":"2024-07-27T06:43:12.797817+0000","last_active":"2024-07-27T06:45:09.485670+0000","last_peered":"2024-07-27T06:45:09.485670+0000","last_clean":"2024-07-27T06:45:09.485670+0000","last_became_active":"2024-07-27T06:43:12.797483+0000","last_became_peered":"2024-07-27T06:43:12.797483+0000","last_unstale":"2024-07-27T06:45:09.485670+0000","last_undegraded":"2024-07-27T06:45:09.485670+0000","last_fullsized":"2024-07-27T06:45:09.485670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:24:44.539505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488887+0000","last_change":"2024-07-27T06:43:09.776880+0000","last_active":"2024-07-27T06:45:09.488887+0000","last_peered":"2024-07-27T06:45:09.488887+0000","last_clean":"2024-07-27T06:45:09.488887+0000","last_became_active":"2024-07-27T06:43:09.776486+0000","last_became_peered":"2024-07-27T06:43:09.776486+0000","last_unstale":"2024-07-27T06:45:09.488887+0000","last_undegraded":"2024-07-27T06:45:09.488887+0000","last_fullsized":"2024-07-27T06:45:09.488887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:55:25.436405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.480961+0000","last_change":"2024-07-27T06:43:12.799755+0000","last_active":"2024-07-27T06:45:09.480961+0000","last_peered":"2024-07-27T06:45:09.480961+0000","last_clean":"2024-07-27T06:45:09.480961+0000","last_became_active":"2024-07-27T06:43:12.799412+0000","last_became_peered":"2024-07-27T06:43:12.799412+0000","last_unstale":"2024-07-27T06:45:09.480961+0000","last_undegraded":"2024-07-27T06:45:09.480961+0000","last_fullsized":"2024-07-27T06:45:09.480961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:45:50.545431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":255,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:09.225886+0000","last_change":"2024-07-27T06:43:10.533039+0000","last_active":"2024-07-27T06:47:09.225886+0000","last_peered":"2024-07-27T06:47:09.225886+0000","last_clean":"2024-07-27T06:47:09.225886+0000","last_became_active":"2024-07-27T06:43:10.532583+0000","last_became_peered":"2024-07-27T06:43:10.532583+0000","last_unstale":"2024-07-27T06:47:09.225886+0000","last_undegraded":"2024-07-27T06:47:09.225886+0000","last_fullsized":"2024-07-27T06:47:09.225886+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:04:51.954463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":218,"num_read_kb":218,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633096+0000","last_change":"2024-07-27T06:44:09.238218+0000","last_active":"2024-07-27T06:45:09.633096+0000","last_peered":"2024-07-27T06:45:09.633096+0000","last_clean":"2024-07-27T06:45:09.633096+0000","last_became_active":"2024-07-27T06:44:09.237841+0000","last_became_peered":"2024-07-27T06:44:09.237841+0000","last_unstale":"2024-07-27T06:45:09.633096+0000","last_undegraded":"2024-07-27T06:45:09.633096+0000","last_fullsized":"2024-07-27T06:45:09.633096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:19.101854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"83'8","reported_seq":257,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:05.259817+0000","last_change":"2024-07-27T06:43:09.776214+0000","last_active":"2024-07-27T06:47:05.259817+0000","last_peered":"2024-07-27T06:47:05.259817+0000","last_clean":"2024-07-27T06:47:05.259817+0000","last_became_active":"2024-07-27T06:43:09.775719+0000","last_became_peered":"2024-07-27T06:43:09.775719+0000","last_unstale":"2024-07-27T06:47:05.259817+0000","last_undegraded":"2024-07-27T06:47:05.259817+0000","last_fullsized":"2024-07-27T06:47:05.259817+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:57:20.381599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":217,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481735+0000","last_change":"2024-07-27T06:43:12.794909+0000","last_active":"2024-07-27T06:45:09.481735+0000","last_peered":"2024-07-27T06:45:09.481735+0000","last_clean":"2024-07-27T06:45:09.481735+0000","last_became_active":"2024-07-27T06:43:12.794497+0000","last_became_peered":"2024-07-27T06:43:12.794497+0000","last_unstale":"2024-07-27T06:45:09.481735+0000","last_undegraded":"2024-07-27T06:45:09.481735+0000","last_fullsized":"2024-07-27T06:45:09.481735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:42:49.760126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633255+0000","last_change":"2024-07-27T06:43:10.534240+0000","last_active":"2024-07-27T06:45:09.633255+0000","last_peered":"2024-07-27T06:45:09.633255+0000","last_clean":"2024-07-27T06:45:09.633255+0000","last_became_active":"2024-07-27T06:43:10.533596+0000","last_became_peered":"2024-07-27T06:43:10.533596+0000","last_unstale":"2024-07-27T06:45:09.633255+0000","last_undegraded":"2024-07-27T06:45:09.633255+0000","last_fullsized":"2024-07-27T06:45:09.633255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:16.098839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481991+0000","last_change":"2024-07-27T06:43:12.805248+0000","last_active":"2024-07-27T06:45:09.481991+0000","last_peered":"2024-07-27T06:45:09.481991+0000","last_clean":"2024-07-27T06:45:09.481991+0000","last_became_active":"2024-07-27T06:43:12.803693+0000","last_became_peered":"2024-07-27T06:43:12.803693+0000","last_unstale":"2024-07-27T06:45:09.481991+0000","last_undegraded":"2024-07-27T06:45:09.481991+0000","last_fullsized":"2024-07-27T06:45:09.481991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:17.466465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486692+0000","last_change":"2024-07-27T06:43:10.533701+0000","last_active":"2024-07-27T06:45:09.486692+0000","last_peered":"2024-07-27T06:45:09.486692+0000","last_clean":"2024-07-27T06:45:09.486692+0000","last_became_active":"2024-07-27T06:43:10.533109+0000","last_became_peered":"2024-07-27T06:43:10.533109+0000","last_unstale":"2024-07-27T06:45:09.486692+0000","last_undegraded":"2024-07-27T06:45:09.486692+0000","last_fullsized":"2024-07-27T06:45:09.486692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:32:52.906691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481918+0000","last_change":"2024-07-27T06:43:12.798479+0000","last_active":"2024-07-27T06:45:09.481918+0000","last_peered":"2024-07-27T06:45:09.481918+0000","last_clean":"2024-07-27T06:45:09.481918+0000","last_became_active":"2024-07-27T06:43:12.798124+0000","last_became_peered":"2024-07-27T06:43:12.798124+0000","last_unstale":"2024-07-27T06:45:09.481918+0000","last_undegraded":"2024-07-27T06:45:09.481918+0000","last_fullsized":"2024-07-27T06:45:09.481918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:14:55.267213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"83'101","reported_seq":222,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:03.434736+0000","last_change":"2024-07-27T06:43:09.770122+0000","last_active":"2024-07-27T06:47:03.434736+0000","last_peered":"2024-07-27T06:47:03.434736+0000","last_clean":"2024-07-27T06:47:03.434736+0000","last_became_active":"2024-07-27T06:43:09.769784+0000","last_became_peered":"2024-07-27T06:43:09.769784+0000","last_unstale":"2024-07-27T06:47:03.434736+0000","last_undegraded":"2024-07-27T06:47:03.434736+0000","last_fullsized":"2024-07-27T06:47:03.434736+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:47:00.398135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":302,"num_read_kb":268,"num_write":185,"num_write_kb":1222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485891+0000","last_change":"2024-07-27T06:43:12.799603+0000","last_active":"2024-07-27T06:45:09.485891+0000","last_peered":"2024-07-27T06:45:09.485891+0000","last_clean":"2024-07-27T06:45:09.485891+0000","last_became_active":"2024-07-27T06:43:12.799189+0000","last_became_peered":"2024-07-27T06:43:12.799189+0000","last_unstale":"2024-07-27T06:45:09.485891+0000","last_undegraded":"2024-07-27T06:45:09.485891+0000","last_fullsized":"2024-07-27T06:45:09.485891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:32:30.603258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862220+0000","last_change":"2024-07-27T06:43:10.004726+0000","last_active":"2024-07-27T06:45:09.862220+0000","last_peered":"2024-07-27T06:45:09.862220+0000","last_clean":"2024-07-27T06:45:09.862220+0000","last_became_active":"2024-07-27T06:43:10.004367+0000","last_became_peered":"2024-07-27T06:43:10.004367+0000","last_unstale":"2024-07-27T06:45:09.862220+0000","last_undegraded":"2024-07-27T06:45:09.862220+0000","last_fullsized":"2024-07-27T06:45:09.862220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:22:39.942391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862427+0000","last_change":"2024-07-27T06:43:12.796511+0000","last_active":"2024-07-27T06:45:09.862427+0000","last_peered":"2024-07-27T06:45:09.862427+0000","last_clean":"2024-07-27T06:45:09.862427+0000","last_became_active":"2024-07-27T06:43:12.796152+0000","last_became_peered":"2024-07-27T06:43:12.796152+0000","last_unstale":"2024-07-27T06:45:09.862427+0000","last_undegraded":"2024-07-27T06:45:09.862427+0000","last_fullsized":"2024-07-27T06:45:09.862427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:03:02.508990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822077+0000","last_change":"2024-07-27T06:43:10.533656+0000","last_active":"2024-07-27T06:45:09.822077+0000","last_peered":"2024-07-27T06:45:09.822077+0000","last_clean":"2024-07-27T06:45:09.822077+0000","last_became_active":"2024-07-27T06:43:10.533295+0000","last_became_peered":"2024-07-27T06:43:10.533295+0000","last_unstale":"2024-07-27T06:45:09.822077+0000","last_undegraded":"2024-07-27T06:45:09.822077+0000","last_fullsized":"2024-07-27T06:45:09.822077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:11:00.032511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485237+0000","last_change":"2024-07-27T06:43:12.795106+0000","last_active":"2024-07-27T06:45:09.485237+0000","last_peered":"2024-07-27T06:45:09.485237+0000","last_clean":"2024-07-27T06:45:09.485237+0000","last_became_active":"2024-07-27T06:43:12.794729+0000","last_became_peered":"2024-07-27T06:43:12.794729+0000","last_unstale":"2024-07-27T06:45:09.485237+0000","last_undegraded":"2024-07-27T06:45:09.485237+0000","last_fullsized":"2024-07-27T06:45:09.485237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:19:16.542363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485673+0000","last_change":"2024-07-27T06:43:09.770016+0000","last_active":"2024-07-27T06:45:09.485673+0000","last_peered":"2024-07-27T06:45:09.485673+0000","last_clean":"2024-07-27T06:45:09.485673+0000","last_became_active":"2024-07-27T06:43:09.769672+0000","last_became_peered":"2024-07-27T06:43:09.769672+0000","last_unstale":"2024-07-27T06:45:09.485673+0000","last_undegraded":"2024-07-27T06:45:09.485673+0000","last_fullsized":"2024-07-27T06:45:09.485673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:30:55.737494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483659+0000","last_change":"2024-07-27T06:43:12.793690+0000","last_active":"2024-07-27T06:45:09.483659+0000","last_peered":"2024-07-27T06:45:09.483659+0000","last_clean":"2024-07-27T06:45:09.483659+0000","last_became_active":"2024-07-27T06:43:12.793352+0000","last_became_peered":"2024-07-27T06:43:12.793352+0000","last_unstale":"2024-07-27T06:45:09.483659+0000","last_undegraded":"2024-07-27T06:45:09.483659+0000","last_fullsized":"2024-07-27T06:45:09.483659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:22:47.606990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.491196+0000","last_change":"2024-07-27T06:45:09.490738+0000","last_active":"2024-07-27T06:45:09.491196+0000","last_peered":"2024-07-27T06:45:09.491196+0000","last_clean":"2024-07-27T06:45:09.491196+0000","last_became_active":"2024-07-27T06:45:09.487147+0000","last_became_peered":"2024-07-27T06:45:09.487147+0000","last_unstale":"2024-07-27T06:45:09.491196+0000","last_undegraded":"2024-07-27T06:45:09.491196+0000","last_fullsized":"2024-07-27T06:45:09.491196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:46.732987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485570+0000","last_change":"2024-07-27T06:43:12.792648+0000","last_active":"2024-07-27T06:45:09.485570+0000","last_peered":"2024-07-27T06:45:09.485570+0000","last_clean":"2024-07-27T06:45:09.485570+0000","last_became_active":"2024-07-27T06:43:12.792338+0000","last_became_peered":"2024-07-27T06:43:12.792338+0000","last_unstale":"2024-07-27T06:45:09.485570+0000","last_undegraded":"2024-07-27T06:45:09.485570+0000","last_fullsized":"2024-07-27T06:45:09.485570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:36:52.605957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.827632+0000","last_change":"2024-07-27T06:45:09.827140+0000","last_active":"2024-07-27T06:45:09.827632+0000","last_peered":"2024-07-27T06:45:09.827632+0000","last_clean":"2024-07-27T06:45:09.827632+0000","last_became_active":"2024-07-27T06:45:09.823119+0000","last_became_peered":"2024-07-27T06:45:09.823119+0000","last_unstale":"2024-07-27T06:45:09.827632+0000","last_undegraded":"2024-07-27T06:45:09.827632+0000","last_fullsized":"2024-07-27T06:45:09.827632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:32:19.682193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488730+0000","last_change":"2024-07-27T06:43:12.809347+0000","last_active":"2024-07-27T06:45:09.488730+0000","last_peered":"2024-07-27T06:45:09.488730+0000","last_clean":"2024-07-27T06:45:09.488730+0000","last_became_active":"2024-07-27T06:43:12.808980+0000","last_became_peered":"2024-07-27T06:43:12.808980+0000","last_unstale":"2024-07-27T06:45:09.488730+0000","last_undegraded":"2024-07-27T06:45:09.488730+0000","last_fullsized":"2024-07-27T06:45:09.488730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:05:14.132641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633284+0000","last_change":"2024-07-27T06:44:09.237908+0000","last_active":"2024-07-27T06:45:09.633284+0000","last_peered":"2024-07-27T06:45:09.633284+0000","last_clean":"2024-07-27T06:45:09.633284+0000","last_became_active":"2024-07-27T06:44:09.237503+0000","last_became_peered":"2024-07-27T06:44:09.237503+0000","last_unstale":"2024-07-27T06:45:09.633284+0000","last_undegraded":"2024-07-27T06:45:09.633284+0000","last_fullsized":"2024-07-27T06:45:09.633284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:29.654244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486391+0000","last_change":"2024-07-27T06:43:09.776000+0000","last_active":"2024-07-27T06:45:09.486391+0000","last_peered":"2024-07-27T06:45:09.486391+0000","last_clean":"2024-07-27T06:45:09.486391+0000","last_became_active":"2024-07-27T06:43:09.775522+0000","last_became_peered":"2024-07-27T06:43:09.775522+0000","last_unstale":"2024-07-27T06:45:09.486391+0000","last_undegraded":"2024-07-27T06:45:09.486391+0000","last_fullsized":"2024-07-27T06:45:09.486391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:39:42.650177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822084+0000","last_change":"2024-07-27T06:43:12.807062+0000","last_active":"2024-07-27T06:45:09.822084+0000","last_peered":"2024-07-27T06:45:09.822084+0000","last_clean":"2024-07-27T06:45:09.822084+0000","last_became_active":"2024-07-27T06:43:12.805816+0000","last_became_peered":"2024-07-27T06:43:12.805816+0000","last_unstale":"2024-07-27T06:45:09.822084+0000","last_undegraded":"2024-07-27T06:45:09.822084+0000","last_fullsized":"2024-07-27T06:45:09.822084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:44:16.261663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483465+0000","last_change":"2024-07-27T06:44:09.250070+0000","last_active":"2024-07-27T06:45:09.483465+0000","last_peered":"2024-07-27T06:45:09.483465+0000","last_clean":"2024-07-27T06:45:09.483465+0000","last_became_active":"2024-07-27T06:44:09.237290+0000","last_became_peered":"2024-07-27T06:44:09.237290+0000","last_unstale":"2024-07-27T06:45:09.483465+0000","last_undegraded":"2024-07-27T06:45:09.483465+0000","last_fullsized":"2024-07-27T06:45:09.483465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:51.590487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862423+0000","last_change":"2024-07-27T06:43:12.809135+0000","last_active":"2024-07-27T06:45:09.862423+0000","last_peered":"2024-07-27T06:45:09.862423+0000","last_clean":"2024-07-27T06:45:09.862423+0000","last_became_active":"2024-07-27T06:43:12.806731+0000","last_became_peered":"2024-07-27T06:43:12.806731+0000","last_unstale":"2024-07-27T06:45:09.862423+0000","last_undegraded":"2024-07-27T06:45:09.862423+0000","last_fullsized":"2024-07-27T06:45:09.862423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:47:50.397517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:10.601326+0000","last_change":"2024-07-27T06:43:10.533608+0000","last_active":"2024-07-27T06:45:10.601326+0000","last_peered":"2024-07-27T06:45:10.601326+0000","last_clean":"2024-07-27T06:45:10.601326+0000","last_became_active":"2024-07-27T06:43:10.532663+0000","last_became_peered":"2024-07-27T06:43:10.532663+0000","last_unstale":"2024-07-27T06:45:10.601326+0000","last_undegraded":"2024-07-27T06:45:10.601326+0000","last_fullsized":"2024-07-27T06:45:10.601326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:05:33.155427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485883+0000","last_change":"2024-07-27T06:43:12.789890+0000","last_active":"2024-07-27T06:45:09.485883+0000","last_peered":"2024-07-27T06:45:09.485883+0000","last_clean":"2024-07-27T06:45:09.485883+0000","last_became_active":"2024-07-27T06:43:12.789561+0000","last_became_peered":"2024-07-27T06:43:12.789561+0000","last_unstale":"2024-07-27T06:45:09.485883+0000","last_undegraded":"2024-07-27T06:45:09.485883+0000","last_fullsized":"2024-07-27T06:45:09.485883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:35:49.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"83'15","reported_seq":262,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:05.259130+0000","last_change":"2024-07-27T06:43:09.773471+0000","last_active":"2024-07-27T06:47:05.259130+0000","last_peered":"2024-07-27T06:47:05.259130+0000","last_clean":"2024-07-27T06:47:05.259130+0000","last_became_active":"2024-07-27T06:43:09.773213+0000","last_became_peered":"2024-07-27T06:43:09.773213+0000","last_unstale":"2024-07-27T06:47:05.259130+0000","last_undegraded":"2024-07-27T06:47:05.259130+0000","last_fullsized":"2024-07-27T06:47:05.259130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:54:38.770198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":215,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":861595,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2496,"num_read_kb":1477,"num_write":488,"num_write_kb":2245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1561,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":823296,"data_stored":2584785,"data_compressed":112140,"data_compressed_allocated":147456,"data_compressed_original":2285568,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283116,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678080,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":461919,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874307,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":1512469,"data_compressed":65700,"data_compressed_allocated":86016,"data_compressed_original":991232,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037834,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":854533,"data_compressed":30901,"data_compressed_allocated":40960,"data_compressed_original":425984,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234063,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":854911,"data_compressed":30901,"data_compressed_allocated":40960,"data_compressed_original":425984,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430290,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":1055194,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593817,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":461874,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790046,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":1447808,"data_compressed":65700,"data_compressed_allocated":86016,"data_compressed_original":991232,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953571,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":929792,"data_stored":462597,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":1247851,"data_compressed":48992,"data_compressed_allocated":65536,"data_compressed_original":794624,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378734,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":527806,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542258,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":529064,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705783,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":462905,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":67611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":787422,"data_compressed":37380,"data_compressed_allocated":49152,"data_compressed_original":761856,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":397128,"data_compressed":18091,"data_compressed_allocated":24576,"data_compressed_original":368640,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":461789,"data_compressed":18091,"data_compressed_allocated":24576,"data_compressed_original":368640,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T06:47:12.761 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-27T06:47:12.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph pg dump --format=json 2024-07-27T06:47:13.036 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:13 smithi029 ceph-mon[32610]: pgmap v403: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 174 KiB/s wr, 56 op/s 2024-07-27T06:47:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:13 smithi029 ceph-mon[32610]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:47:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:13 smithi077 ceph-mon[37153]: pgmap v403: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 174 KiB/s wr, 56 op/s 2024-07-27T06:47:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:13 smithi077 ceph-mon[37153]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:47:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:13 smithi047 ceph-mon[36835]: pgmap v403: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 174 KiB/s wr, 56 op/s 2024-07-27T06:47:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:13 smithi047 ceph-mon[36835]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:47:14.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.734+0000 7f7892b34640 1 -- 172.21.15.29:0/4185734144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f788c1024b0 msgr2=0x7f788c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:14.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.734+0000 7f7892b34640 1 --2- 172.21.15.29:0/4185734144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f788c1024b0 0x7f788c1028c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f787c009900 tx=0x7f787c02a410 comp rx=0 tx=0).stop 2024-07-27T06:47:14.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.735+0000 7f7892b34640 1 -- 172.21.15.29:0/4185734144 shutdown_connections 2024-07-27T06:47:14.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.735+0000 7f7892b34640 1 --2- 172.21.15.29:0/4185734144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f788c108740 0x7f788c10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:14.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.735+0000 7f7892b34640 1 --2- 172.21.15.29:0/4185734144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f788c107df0 0x7f788c108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:14.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.735+0000 7f7892b34640 1 --2- 172.21.15.29:0/4185734144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f788c1024b0 0x7f788c1028c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:14.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.735+0000 7f7892b34640 1 -- 172.21.15.29:0/4185734144 >> 172.21.15.29:0/4185734144 conn(0x7f788c079170 msgr2=0x7f788c079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:14.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.735+0000 7f7892b34640 1 -- 172.21.15.29:0/4185734144 shutdown_connections 2024-07-27T06:47:14.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.735+0000 7f7892b34640 1 -- 172.21.15.29:0/4185734144 wait complete. 2024-07-27T06:47:14.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f7892b34640 1 Processor -- start 2024-07-27T06:47:14.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f7892b34640 1 -- start start 2024-07-27T06:47:14.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f7892b34640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f788c1024b0 0x7f788c19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:14.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f7892b34640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f788c107df0 0x7f788c19b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:14.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f7892b34640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f788c108740 0x7f788c19b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:14.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f7892b34640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f788c1123c0 con 0x7f788c1024b0 2024-07-27T06:47:14.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f7892b34640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f788c112240 con 0x7f788c107df0 2024-07-27T06:47:14.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f7892b34640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f788c112540 con 0x7f788c108740 2024-07-27T06:47:14.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f78908d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f788c1024b0 0x7f788c19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:14.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f78908d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f788c1024b0 0x7f788c19ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43364/0 (socket says 172.21.15.29:43364) 2024-07-27T06:47:14.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f78908d0640 1 -- 172.21.15.29:0/977167407 learned_addr learned my addr 172.21.15.29:0/977167407 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:14.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f788bfff640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f788c107df0 0x7f788c19b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:14.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.736+0000 7f78910d1640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f788c108740 0x7f788c19b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:14.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.737+0000 7f78908d0640 1 -- 172.21.15.29:0/977167407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f788c108740 msgr2=0x7f788c19b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:14.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.737+0000 7f78908d0640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f788c108740 0x7f788c19b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:14.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.737+0000 7f78908d0640 1 -- 172.21.15.29:0/977167407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f788c107df0 msgr2=0x7f788c19b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:14.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.737+0000 7f78908d0640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f788c107df0 0x7f788c19b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:14.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.737+0000 7f78908d0640 1 -- 172.21.15.29:0/977167407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788c1a5f30 con 0x7f788c1024b0 2024-07-27T06:47:14.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.737+0000 7f78910d1640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f788c108740 0x7f788c19b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:14.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.737+0000 7f78908d0640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f788c1024b0 0x7f788c19ad30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f787c005940 tx=0x7f787c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:14.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.737+0000 7f7889ffb640 1 -- 172.21.15.29:0/977167407 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c03f030 con 0x7f788c1024b0 2024-07-27T06:47:14.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.737+0000 7f7889ffb640 1 -- 172.21.15.29:0/977167407 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f787c003cb0 con 0x7f788c1024b0 2024-07-27T06:47:14.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.737+0000 7f7892b34640 1 -- 172.21.15.29:0/977167407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788c1a61c0 con 0x7f788c1024b0 2024-07-27T06:47:14.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.737+0000 7f7889ffb640 1 -- 172.21.15.29:0/977167407 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c035070 con 0x7f788c1024b0 2024-07-27T06:47:14.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.737+0000 7f7892b34640 1 -- 172.21.15.29:0/977167407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f788c1a66a0 con 0x7f788c1024b0 2024-07-27T06:47:14.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.738+0000 7f7892b34640 1 -- 172.21.15.29:0/977167407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f788c103730 con 0x7f788c1024b0 2024-07-27T06:47:14.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.741+0000 7f7889ffb640 1 -- 172.21.15.29:0/977167407 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f787c042030 con 0x7f788c1024b0 2024-07-27T06:47:14.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.741+0000 7f7889ffb640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78780b3ce0 0x7f78780b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:14.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.742+0000 7f7889ffb640 1 -- 172.21.15.29:0/977167407 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f787c1046a0 con 0x7f788c1024b0 2024-07-27T06:47:14.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.742+0000 7f788bfff640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78780b3ce0 0x7f78780b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:14.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.742+0000 7f788bfff640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78780b3ce0 0x7f78780b61f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f788c100630 tx=0x7f787400a2d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:14.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.745+0000 7f7889ffb640 1 -- 172.21.15.29:0/977167407 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f787c0cba80 con 0x7f788c1024b0 2024-07-27T06:47:14.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.928+0000 7f7892b34640 1 -- 172.21.15.29:0/977167407 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f788c063230 con 0x7f78780b3ce0 2024-07-27T06:47:14.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.932+0000 7f7889ffb640 1 -- 172.21.15.29:0/977167407 <== mgr.14208 v2:172.21.15.29:6800/2980069896 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259301 (secure 0 0 0) 0x7f788c063230 con 0x7f78780b3ce0 2024-07-27T06:47:14.936 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:47:14.937 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-07-27T06:47:14.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.936+0000 7f7892b34640 1 -- 172.21.15.29:0/977167407 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78780b3ce0 msgr2=0x7f78780b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:14.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.936+0000 7f7892b34640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78780b3ce0 0x7f78780b61f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f788c100630 tx=0x7f787400a2d0 comp rx=0 tx=0).stop 2024-07-27T06:47:14.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.936+0000 7f7892b34640 1 -- 172.21.15.29:0/977167407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f788c1024b0 msgr2=0x7f788c19ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:14.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.936+0000 7f7892b34640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f788c1024b0 0x7f788c19ad30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f787c005940 tx=0x7f787c0042f0 comp rx=0 tx=0).stop 2024-07-27T06:47:14.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.936+0000 7f7892b34640 1 -- 172.21.15.29:0/977167407 shutdown_connections 2024-07-27T06:47:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.936+0000 7f7892b34640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78780b3ce0 0x7f78780b61f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.936+0000 7f7892b34640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f788c108740 0x7f788c19b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.936+0000 7f7892b34640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f788c107df0 0x7f788c19b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.936+0000 7f7892b34640 1 --2- 172.21.15.29:0/977167407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f788c1024b0 0x7f788c19ad30 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.936+0000 7f7892b34640 1 -- 172.21.15.29:0/977167407 >> 172.21.15.29:0/977167407 conn(0x7f788c079170 msgr2=0x7f788c100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.936+0000 7f7892b34640 1 -- 172.21.15.29:0/977167407 shutdown_connections 2024-07-27T06:47:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:14.936+0000 7f7892b34640 1 -- 172.21.15.29:0/977167407 wait complete. 2024-07-27T06:47:15.761 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:15 smithi029 ceph-mon[32610]: pgmap v404: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 174 KiB/s wr, 57 op/s 2024-07-27T06:47:15.762 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-07-27T06:47:13.687639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1452197,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2607,"num_read_kb":1566,"num_write":613,"num_write_kb":4397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2953401,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":964,"ondisk_log_size":964,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360728,"kb_used_data":11532,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581096,"statfs":{"total":1151940427776,"available":1151571042304,"internally_reserved":0,"allocated":11808768,"data_stored":9878931,"data_compressed":359475,"data_compressed_allocated":479232,"data_compressed_original":4448256,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":786992,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":466,"num_read_kb":371,"num_write":224,"num_write_kb":2088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003161"},"pg_stats":[{"pgid":"3.3f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483739+0000","last_change":"2024-07-27T06:43:12.802353+0000","last_active":"2024-07-27T06:45:09.483739+0000","last_peered":"2024-07-27T06:45:09.483739+0000","last_clean":"2024-07-27T06:45:09.483739+0000","last_became_active":"2024-07-27T06:43:12.802028+0000","last_became_peered":"2024-07-27T06:43:12.802028+0000","last_unstale":"2024-07-27T06:45:09.483739+0000","last_undegraded":"2024-07-27T06:45:09.483739+0000","last_fullsized":"2024-07-27T06:45:09.483739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:22:24.292567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862408+0000","last_change":"2024-07-27T06:43:12.806269+0000","last_active":"2024-07-27T06:45:09.862408+0000","last_peered":"2024-07-27T06:45:09.862408+0000","last_clean":"2024-07-27T06:45:09.862408+0000","last_became_active":"2024-07-27T06:43:12.803232+0000","last_became_peered":"2024-07-27T06:43:12.803232+0000","last_unstale":"2024-07-27T06:45:09.862408+0000","last_undegraded":"2024-07-27T06:45:09.862408+0000","last_fullsized":"2024-07-27T06:45:09.862408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:57:53.673850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862259+0000","last_change":"2024-07-27T06:43:12.805542+0000","last_active":"2024-07-27T06:45:09.862259+0000","last_peered":"2024-07-27T06:45:09.862259+0000","last_clean":"2024-07-27T06:45:09.862259+0000","last_became_active":"2024-07-27T06:43:12.804951+0000","last_became_peered":"2024-07-27T06:43:12.804951+0000","last_unstale":"2024-07-27T06:45:09.862259+0000","last_undegraded":"2024-07-27T06:45:09.862259+0000","last_fullsized":"2024-07-27T06:45:09.862259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:06:40.469695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862158+0000","last_change":"2024-07-27T06:43:12.796950+0000","last_active":"2024-07-27T06:45:09.862158+0000","last_peered":"2024-07-27T06:45:09.862158+0000","last_clean":"2024-07-27T06:45:09.862158+0000","last_became_active":"2024-07-27T06:43:12.796549+0000","last_became_peered":"2024-07-27T06:43:12.796549+0000","last_unstale":"2024-07-27T06:45:09.862158+0000","last_undegraded":"2024-07-27T06:45:09.862158+0000","last_fullsized":"2024-07-27T06:45:09.862158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:30:36.391046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486015+0000","last_change":"2024-07-27T06:43:12.799838+0000","last_active":"2024-07-27T06:45:09.486015+0000","last_peered":"2024-07-27T06:45:09.486015+0000","last_clean":"2024-07-27T06:45:09.486015+0000","last_became_active":"2024-07-27T06:43:12.799462+0000","last_became_peered":"2024-07-27T06:43:12.799462+0000","last_unstale":"2024-07-27T06:45:09.486015+0000","last_undegraded":"2024-07-27T06:45:09.486015+0000","last_fullsized":"2024-07-27T06:45:09.486015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:45:19.371259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481631+0000","last_change":"2024-07-27T06:43:12.797187+0000","last_active":"2024-07-27T06:45:09.481631+0000","last_peered":"2024-07-27T06:45:09.481631+0000","last_clean":"2024-07-27T06:45:09.481631+0000","last_became_active":"2024-07-27T06:43:12.796633+0000","last_became_peered":"2024-07-27T06:43:12.796633+0000","last_unstale":"2024-07-27T06:45:09.481631+0000","last_undegraded":"2024-07-27T06:45:09.481631+0000","last_fullsized":"2024-07-27T06:45:09.481631+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:22:20.882899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488423+0000","last_change":"2024-07-27T06:43:12.793786+0000","last_active":"2024-07-27T06:45:09.488423+0000","last_peered":"2024-07-27T06:45:09.488423+0000","last_clean":"2024-07-27T06:45:09.488423+0000","last_became_active":"2024-07-27T06:43:12.793413+0000","last_became_peered":"2024-07-27T06:43:12.793413+0000","last_unstale":"2024-07-27T06:45:09.488423+0000","last_undegraded":"2024-07-27T06:45:09.488423+0000","last_fullsized":"2024-07-27T06:45:09.488423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:05:14.015373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485385+0000","last_change":"2024-07-27T06:43:12.791304+0000","last_active":"2024-07-27T06:45:09.485385+0000","last_peered":"2024-07-27T06:45:09.485385+0000","last_clean":"2024-07-27T06:45:09.485385+0000","last_became_active":"2024-07-27T06:43:12.790801+0000","last_became_peered":"2024-07-27T06:43:12.790801+0000","last_unstale":"2024-07-27T06:45:09.485385+0000","last_undegraded":"2024-07-27T06:45:09.485385+0000","last_fullsized":"2024-07-27T06:45:09.485385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:12:37.606748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822529+0000","last_change":"2024-07-27T06:43:12.797778+0000","last_active":"2024-07-27T06:45:09.822529+0000","last_peered":"2024-07-27T06:45:09.822529+0000","last_clean":"2024-07-27T06:45:09.822529+0000","last_became_active":"2024-07-27T06:43:12.796784+0000","last_became_peered":"2024-07-27T06:43:12.796784+0000","last_unstale":"2024-07-27T06:45:09.822529+0000","last_undegraded":"2024-07-27T06:45:09.822529+0000","last_fullsized":"2024-07-27T06:45:09.822529+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:52:22.275282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"81'16","reported_seq":42,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.482109+0000","last_change":"2024-07-27T06:43:12.798970+0000","last_active":"2024-07-27T06:45:09.482109+0000","last_peered":"2024-07-27T06:45:09.482109+0000","last_clean":"2024-07-27T06:45:09.482109+0000","last_became_active":"2024-07-27T06:43:12.798549+0000","last_became_peered":"2024-07-27T06:43:12.798549+0000","last_unstale":"2024-07-27T06:45:09.482109+0000","last_undegraded":"2024-07-27T06:45:09.482109+0000","last_fullsized":"2024-07-27T06:45:09.482109+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:58:32.464429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488604+0000","last_change":"2024-07-27T06:43:12.809405+0000","last_active":"2024-07-27T06:45:09.488604+0000","last_peered":"2024-07-27T06:45:09.488604+0000","last_clean":"2024-07-27T06:45:09.488604+0000","last_became_active":"2024-07-27T06:43:12.809030+0000","last_became_peered":"2024-07-27T06:43:12.809030+0000","last_unstale":"2024-07-27T06:45:09.488604+0000","last_undegraded":"2024-07-27T06:45:09.488604+0000","last_fullsized":"2024-07-27T06:45:09.488604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:46:35.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862193+0000","last_change":"2024-07-27T06:43:12.802647+0000","last_active":"2024-07-27T06:45:09.862193+0000","last_peered":"2024-07-27T06:45:09.862193+0000","last_clean":"2024-07-27T06:45:09.862193+0000","last_became_active":"2024-07-27T06:43:12.802247+0000","last_became_peered":"2024-07-27T06:43:12.802247+0000","last_unstale":"2024-07-27T06:45:09.862193+0000","last_undegraded":"2024-07-27T06:45:09.862193+0000","last_fullsized":"2024-07-27T06:45:09.862193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:44:24.564852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481628+0000","last_change":"2024-07-27T06:43:12.809140+0000","last_active":"2024-07-27T06:45:09.481628+0000","last_peered":"2024-07-27T06:45:09.481628+0000","last_clean":"2024-07-27T06:45:09.481628+0000","last_became_active":"2024-07-27T06:43:12.808692+0000","last_became_peered":"2024-07-27T06:43:12.808692+0000","last_unstale":"2024-07-27T06:45:09.481628+0000","last_undegraded":"2024-07-27T06:45:09.481628+0000","last_fullsized":"2024-07-27T06:45:09.481628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:37:45.326234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862226+0000","last_change":"2024-07-27T06:43:12.802838+0000","last_active":"2024-07-27T06:45:09.862226+0000","last_peered":"2024-07-27T06:45:09.862226+0000","last_clean":"2024-07-27T06:45:09.862226+0000","last_became_active":"2024-07-27T06:43:12.802503+0000","last_became_peered":"2024-07-27T06:43:12.802503+0000","last_unstale":"2024-07-27T06:45:09.862226+0000","last_undegraded":"2024-07-27T06:45:09.862226+0000","last_fullsized":"2024-07-27T06:45:09.862226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:41:03.428162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862170+0000","last_change":"2024-07-27T06:43:12.801626+0000","last_active":"2024-07-27T06:45:09.862170+0000","last_peered":"2024-07-27T06:45:09.862170+0000","last_clean":"2024-07-27T06:45:09.862170+0000","last_became_active":"2024-07-27T06:43:12.801266+0000","last_became_peered":"2024-07-27T06:43:12.801266+0000","last_unstale":"2024-07-27T06:45:09.862170+0000","last_undegraded":"2024-07-27T06:45:09.862170+0000","last_fullsized":"2024-07-27T06:45:09.862170+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:54:31.934223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.821975+0000","last_change":"2024-07-27T06:43:12.804543+0000","last_active":"2024-07-27T06:45:09.821975+0000","last_peered":"2024-07-27T06:45:09.821975+0000","last_clean":"2024-07-27T06:45:09.821975+0000","last_became_active":"2024-07-27T06:43:12.804173+0000","last_became_peered":"2024-07-27T06:43:12.804173+0000","last_unstale":"2024-07-27T06:45:09.821975+0000","last_undegraded":"2024-07-27T06:45:09.821975+0000","last_fullsized":"2024-07-27T06:45:09.821975+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:18:25.337333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483797+0000","last_change":"2024-07-27T06:43:12.798604+0000","last_active":"2024-07-27T06:45:09.483797+0000","last_peered":"2024-07-27T06:45:09.483797+0000","last_clean":"2024-07-27T06:45:09.483797+0000","last_became_active":"2024-07-27T06:43:12.798228+0000","last_became_peered":"2024-07-27T06:43:12.798228+0000","last_unstale":"2024-07-27T06:45:09.483797+0000","last_undegraded":"2024-07-27T06:45:09.483797+0000","last_fullsized":"2024-07-27T06:45:09.483797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:04:35.851399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633341+0000","last_change":"2024-07-27T06:43:12.800452+0000","last_active":"2024-07-27T06:45:09.633341+0000","last_peered":"2024-07-27T06:45:09.633341+0000","last_clean":"2024-07-27T06:45:09.633341+0000","last_became_active":"2024-07-27T06:43:12.800084+0000","last_became_peered":"2024-07-27T06:43:12.800084+0000","last_unstale":"2024-07-27T06:45:09.633341+0000","last_undegraded":"2024-07-27T06:45:09.633341+0000","last_fullsized":"2024-07-27T06:45:09.633341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:54:27.575926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485773+0000","last_change":"2024-07-27T06:43:12.800761+0000","last_active":"2024-07-27T06:45:09.485773+0000","last_peered":"2024-07-27T06:45:09.485773+0000","last_clean":"2024-07-27T06:45:09.485773+0000","last_became_active":"2024-07-27T06:43:12.800414+0000","last_became_peered":"2024-07-27T06:43:12.800414+0000","last_unstale":"2024-07-27T06:45:09.485773+0000","last_undegraded":"2024-07-27T06:45:09.485773+0000","last_fullsized":"2024-07-27T06:45:09.485773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:02:47.247522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483375+0000","last_change":"2024-07-27T06:43:12.795470+0000","last_active":"2024-07-27T06:45:09.483375+0000","last_peered":"2024-07-27T06:45:09.483375+0000","last_clean":"2024-07-27T06:45:09.483375+0000","last_became_active":"2024-07-27T06:43:12.795136+0000","last_became_peered":"2024-07-27T06:43:12.795136+0000","last_unstale":"2024-07-27T06:45:09.483375+0000","last_undegraded":"2024-07-27T06:45:09.483375+0000","last_fullsized":"2024-07-27T06:45:09.483375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:01:22.178699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481748+0000","last_change":"2024-07-27T06:43:12.803897+0000","last_active":"2024-07-27T06:45:09.481748+0000","last_peered":"2024-07-27T06:45:09.481748+0000","last_clean":"2024-07-27T06:45:09.481748+0000","last_became_active":"2024-07-27T06:43:12.803546+0000","last_became_peered":"2024-07-27T06:43:12.803546+0000","last_unstale":"2024-07-27T06:45:09.481748+0000","last_undegraded":"2024-07-27T06:45:09.481748+0000","last_fullsized":"2024-07-27T06:45:09.481748+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:13:50.816831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"81'10","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822201+0000","last_change":"2024-07-27T06:44:09.238514+0000","last_active":"2024-07-27T06:45:09.822201+0000","last_peered":"2024-07-27T06:45:09.822201+0000","last_clean":"2024-07-27T06:45:09.822201+0000","last_became_active":"2024-07-27T06:44:09.237903+0000","last_became_peered":"2024-07-27T06:44:09.237903+0000","last_unstale":"2024-07-27T06:45:09.822201+0000","last_undegraded":"2024-07-27T06:45:09.822201+0000","last_fullsized":"2024-07-27T06:45:09.822201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:20:59.887491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485428+0000","last_change":"2024-07-27T06:43:12.788570+0000","last_active":"2024-07-27T06:45:09.485428+0000","last_peered":"2024-07-27T06:45:09.485428+0000","last_clean":"2024-07-27T06:45:09.485428+0000","last_became_active":"2024-07-27T06:43:12.788199+0000","last_became_peered":"2024-07-27T06:43:12.788199+0000","last_unstale":"2024-07-27T06:45:09.485428+0000","last_undegraded":"2024-07-27T06:45:09.485428+0000","last_fullsized":"2024-07-27T06:45:09.485428+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:44:26.751094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862750+0000","last_change":"2024-07-27T06:43:12.792866+0000","last_active":"2024-07-27T06:45:09.862750+0000","last_peered":"2024-07-27T06:45:09.862750+0000","last_clean":"2024-07-27T06:45:09.862750+0000","last_became_active":"2024-07-27T06:43:12.792389+0000","last_became_peered":"2024-07-27T06:43:12.792389+0000","last_unstale":"2024-07-27T06:45:09.862750+0000","last_undegraded":"2024-07-27T06:45:09.862750+0000","last_fullsized":"2024-07-27T06:45:09.862750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:10:37.367642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485830+0000","last_change":"2024-07-27T06:43:12.797002+0000","last_active":"2024-07-27T06:45:09.485830+0000","last_peered":"2024-07-27T06:45:09.485830+0000","last_clean":"2024-07-27T06:45:09.485830+0000","last_became_active":"2024-07-27T06:43:12.796651+0000","last_became_peered":"2024-07-27T06:43:12.796651+0000","last_unstale":"2024-07-27T06:45:09.485830+0000","last_undegraded":"2024-07-27T06:45:09.485830+0000","last_fullsized":"2024-07-27T06:45:09.485830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:28:06.934042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485867+0000","last_change":"2024-07-27T06:43:12.798549+0000","last_active":"2024-07-27T06:45:09.485867+0000","last_peered":"2024-07-27T06:45:09.485867+0000","last_clean":"2024-07-27T06:45:09.485867+0000","last_became_active":"2024-07-27T06:43:12.798201+0000","last_became_peered":"2024-07-27T06:43:12.798201+0000","last_unstale":"2024-07-27T06:45:09.485867+0000","last_undegraded":"2024-07-27T06:45:09.485867+0000","last_fullsized":"2024-07-27T06:45:09.485867+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:59:47.525536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483656+0000","last_change":"2024-07-27T06:43:12.791053+0000","last_active":"2024-07-27T06:45:09.483656+0000","last_peered":"2024-07-27T06:45:09.483656+0000","last_clean":"2024-07-27T06:45:09.483656+0000","last_became_active":"2024-07-27T06:43:12.790718+0000","last_became_peered":"2024-07-27T06:43:12.790718+0000","last_unstale":"2024-07-27T06:45:09.483656+0000","last_undegraded":"2024-07-27T06:45:09.483656+0000","last_fullsized":"2024-07-27T06:45:09.483656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:09.455399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"81'4","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481811+0000","last_change":"2024-07-27T06:43:12.804180+0000","last_active":"2024-07-27T06:45:09.481811+0000","last_peered":"2024-07-27T06:45:09.481811+0000","last_clean":"2024-07-27T06:45:09.481811+0000","last_became_active":"2024-07-27T06:43:12.803881+0000","last_became_peered":"2024-07-27T06:43:12.803881+0000","last_unstale":"2024-07-27T06:45:09.481811+0000","last_undegraded":"2024-07-27T06:45:09.481811+0000","last_fullsized":"2024-07-27T06:45:09.481811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:01:05.009233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862727+0000","last_change":"2024-07-27T06:43:12.809150+0000","last_active":"2024-07-27T06:45:09.862727+0000","last_peered":"2024-07-27T06:45:09.862727+0000","last_clean":"2024-07-27T06:45:09.862727+0000","last_became_active":"2024-07-27T06:43:12.808428+0000","last_became_peered":"2024-07-27T06:43:12.808428+0000","last_unstale":"2024-07-27T06:45:09.862727+0000","last_undegraded":"2024-07-27T06:45:09.862727+0000","last_fullsized":"2024-07-27T06:45:09.862727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:07:13.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486325+0000","last_change":"2024-07-27T06:44:09.233785+0000","last_active":"2024-07-27T06:45:09.486325+0000","last_peered":"2024-07-27T06:45:09.486325+0000","last_clean":"2024-07-27T06:45:09.486325+0000","last_became_active":"2024-07-27T06:44:09.233184+0000","last_became_peered":"2024-07-27T06:44:09.233184+0000","last_unstale":"2024-07-27T06:45:09.486325+0000","last_undegraded":"2024-07-27T06:45:09.486325+0000","last_fullsized":"2024-07-27T06:45:09.486325+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:12:48.556383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"81'12","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486129+0000","last_change":"2024-07-27T06:44:09.320182+0000","last_active":"2024-07-27T06:45:09.486129+0000","last_peered":"2024-07-27T06:45:09.486129+0000","last_clean":"2024-07-27T06:45:09.486129+0000","last_became_active":"2024-07-27T06:44:09.319661+0000","last_became_peered":"2024-07-27T06:44:09.319661+0000","last_unstale":"2024-07-27T06:45:09.486129+0000","last_undegraded":"2024-07-27T06:45:09.486129+0000","last_fullsized":"2024-07-27T06:45:09.486129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:52:11.006680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483342+0000","last_change":"2024-07-27T06:43:12.787866+0000","last_active":"2024-07-27T06:45:09.483342+0000","last_peered":"2024-07-27T06:45:09.483342+0000","last_clean":"2024-07-27T06:45:09.483342+0000","last_became_active":"2024-07-27T06:43:12.787513+0000","last_became_peered":"2024-07-27T06:43:12.787513+0000","last_unstale":"2024-07-27T06:45:09.483342+0000","last_undegraded":"2024-07-27T06:45:09.483342+0000","last_fullsized":"2024-07-27T06:45:09.483342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:48:57.876588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485381+0000","last_change":"2024-07-27T06:43:12.791158+0000","last_active":"2024-07-27T06:45:09.485381+0000","last_peered":"2024-07-27T06:45:09.485381+0000","last_clean":"2024-07-27T06:45:09.485381+0000","last_became_active":"2024-07-27T06:43:12.790771+0000","last_became_peered":"2024-07-27T06:43:12.790771+0000","last_unstale":"2024-07-27T06:45:09.485381+0000","last_undegraded":"2024-07-27T06:45:09.485381+0000","last_fullsized":"2024-07-27T06:45:09.485381+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:10:18.739265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"81'7","reported_seq":259,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:08.225697+0000","last_change":"2024-07-27T06:43:09.769281+0000","last_active":"2024-07-27T06:47:08.225697+0000","last_peered":"2024-07-27T06:47:08.225697+0000","last_clean":"2024-07-27T06:47:08.225697+0000","last_became_active":"2024-07-27T06:43:09.768864+0000","last_became_peered":"2024-07-27T06:43:09.768864+0000","last_unstale":"2024-07-27T06:47:08.225697+0000","last_undegraded":"2024-07-27T06:47:08.225697+0000","last_fullsized":"2024-07-27T06:47:08.225697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:55:14.191180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":220,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"81'8","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862960+0000","last_change":"2024-07-27T06:44:09.319930+0000","last_active":"2024-07-27T06:45:09.862960+0000","last_peered":"2024-07-27T06:45:09.862960+0000","last_clean":"2024-07-27T06:45:09.862960+0000","last_became_active":"2024-07-27T06:44:09.319535+0000","last_became_peered":"2024-07-27T06:44:09.319535+0000","last_unstale":"2024-07-27T06:45:09.862960+0000","last_undegraded":"2024-07-27T06:45:09.862960+0000","last_fullsized":"2024-07-27T06:45:09.862960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:39:52.887860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481674+0000","last_change":"2024-07-27T06:43:09.772227+0000","last_active":"2024-07-27T06:45:09.481674+0000","last_peered":"2024-07-27T06:45:09.481674+0000","last_clean":"2024-07-27T06:45:09.481674+0000","last_became_active":"2024-07-27T06:43:09.771812+0000","last_became_peered":"2024-07-27T06:43:09.771812+0000","last_unstale":"2024-07-27T06:45:09.481674+0000","last_undegraded":"2024-07-27T06:45:09.481674+0000","last_fullsized":"2024-07-27T06:45:09.481674+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:27:28.054590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481030+0000","last_change":"2024-07-27T06:43:09.766490+0000","last_active":"2024-07-27T06:45:09.481030+0000","last_peered":"2024-07-27T06:45:09.481030+0000","last_clean":"2024-07-27T06:45:09.481030+0000","last_became_active":"2024-07-27T06:43:09.765973+0000","last_became_peered":"2024-07-27T06:43:09.765973+0000","last_unstale":"2024-07-27T06:45:09.481030+0000","last_undegraded":"2024-07-27T06:45:09.481030+0000","last_fullsized":"2024-07-27T06:45:09.481030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:00:02.913080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485458+0000","last_change":"2024-07-27T06:43:12.805234+0000","last_active":"2024-07-27T06:45:09.485458+0000","last_peered":"2024-07-27T06:45:09.485458+0000","last_clean":"2024-07-27T06:45:09.485458+0000","last_became_active":"2024-07-27T06:43:12.804908+0000","last_became_peered":"2024-07-27T06:43:12.804908+0000","last_unstale":"2024-07-27T06:45:09.485458+0000","last_undegraded":"2024-07-27T06:45:09.485458+0000","last_fullsized":"2024-07-27T06:45:09.485458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:09:34.988755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862248+0000","last_change":"2024-07-27T06:43:09.774201+0000","last_active":"2024-07-27T06:45:09.862248+0000","last_peered":"2024-07-27T06:45:09.862248+0000","last_clean":"2024-07-27T06:45:09.862248+0000","last_became_active":"2024-07-27T06:43:09.773817+0000","last_became_peered":"2024-07-27T06:43:09.773817+0000","last_unstale":"2024-07-27T06:45:09.862248+0000","last_undegraded":"2024-07-27T06:45:09.862248+0000","last_fullsized":"2024-07-27T06:45:09.862248+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:06.166251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633110+0000","last_change":"2024-07-27T06:43:12.800174+0000","last_active":"2024-07-27T06:45:09.633110+0000","last_peered":"2024-07-27T06:45:09.633110+0000","last_clean":"2024-07-27T06:45:09.633110+0000","last_became_active":"2024-07-27T06:43:12.799769+0000","last_became_peered":"2024-07-27T06:43:12.799769+0000","last_unstale":"2024-07-27T06:45:09.633110+0000","last_undegraded":"2024-07-27T06:45:09.633110+0000","last_fullsized":"2024-07-27T06:45:09.633110+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:43:06.890372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":475,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:11.600411+0000","last_change":"2024-07-27T06:43:09.776612+0000","last_active":"2024-07-27T06:47:11.600411+0000","last_peered":"2024-07-27T06:47:11.600411+0000","last_clean":"2024-07-27T06:47:11.600411+0000","last_became_active":"2024-07-27T06:43:09.775761+0000","last_became_peered":"2024-07-27T06:43:09.775761+0000","last_unstale":"2024-07-27T06:47:11.600411+0000","last_undegraded":"2024-07-27T06:47:11.600411+0000","last_fullsized":"2024-07-27T06:47:11.600411+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:42:52.132357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":437,"num_read_kb":437,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"81'10","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822478+0000","last_change":"2024-07-27T06:43:12.797200+0000","last_active":"2024-07-27T06:45:09.822478+0000","last_peered":"2024-07-27T06:45:09.822478+0000","last_clean":"2024-07-27T06:45:09.822478+0000","last_became_active":"2024-07-27T06:43:12.796288+0000","last_became_peered":"2024-07-27T06:43:12.796288+0000","last_unstale":"2024-07-27T06:45:09.822478+0000","last_undegraded":"2024-07-27T06:45:09.822478+0000","last_fullsized":"2024-07-27T06:45:09.822478+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:31:27.603883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488738+0000","last_change":"2024-07-27T06:43:09.772658+0000","last_active":"2024-07-27T06:45:09.488738+0000","last_peered":"2024-07-27T06:45:09.488738+0000","last_clean":"2024-07-27T06:45:09.488738+0000","last_became_active":"2024-07-27T06:43:09.772277+0000","last_became_peered":"2024-07-27T06:43:09.772277+0000","last_unstale":"2024-07-27T06:45:09.488738+0000","last_undegraded":"2024-07-27T06:45:09.488738+0000","last_fullsized":"2024-07-27T06:45:09.488738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:09:16.854718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862551+0000","last_change":"2024-07-27T06:43:12.809643+0000","last_active":"2024-07-27T06:45:09.862551+0000","last_peered":"2024-07-27T06:45:09.862551+0000","last_clean":"2024-07-27T06:45:09.862551+0000","last_became_active":"2024-07-27T06:43:12.809020+0000","last_became_peered":"2024-07-27T06:43:12.809020+0000","last_unstale":"2024-07-27T06:45:09.862551+0000","last_undegraded":"2024-07-27T06:45:09.862551+0000","last_fullsized":"2024-07-27T06:45:09.862551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:08:03.062914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.863022+0000","last_change":"2024-07-27T06:43:10.534042+0000","last_active":"2024-07-27T06:45:09.863022+0000","last_peered":"2024-07-27T06:45:09.863022+0000","last_clean":"2024-07-27T06:45:09.863022+0000","last_became_active":"2024-07-27T06:43:10.533615+0000","last_became_peered":"2024-07-27T06:43:10.533615+0000","last_unstale":"2024-07-27T06:45:09.863022+0000","last_undegraded":"2024-07-27T06:45:09.863022+0000","last_fullsized":"2024-07-27T06:45:09.863022+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:09:32.235633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"81'11","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822222+0000","last_change":"2024-07-27T06:43:12.798764+0000","last_active":"2024-07-27T06:45:09.822222+0000","last_peered":"2024-07-27T06:45:09.822222+0000","last_clean":"2024-07-27T06:45:09.822222+0000","last_became_active":"2024-07-27T06:43:12.797500+0000","last_became_peered":"2024-07-27T06:43:12.797500+0000","last_unstale":"2024-07-27T06:45:09.822222+0000","last_undegraded":"2024-07-27T06:45:09.822222+0000","last_fullsized":"2024-07-27T06:45:09.822222+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:12:09.368711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"83'37","reported_seq":520,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:12.231112+0000","last_change":"2024-07-27T06:43:09.775954+0000","last_active":"2024-07-27T06:47:12.231112+0000","last_peered":"2024-07-27T06:47:12.231112+0000","last_clean":"2024-07-27T06:47:12.231112+0000","last_became_active":"2024-07-27T06:43:09.775498+0000","last_became_peered":"2024-07-27T06:43:09.775498+0000","last_unstale":"2024-07-27T06:47:12.231112+0000","last_undegraded":"2024-07-27T06:47:12.231112+0000","last_fullsized":"2024-07-27T06:47:12.231112+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:08:46.965525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66277,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":451,"num_read_kb":291,"num_write":39,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486559+0000","last_change":"2024-07-27T06:43:12.797220+0000","last_active":"2024-07-27T06:45:09.486559+0000","last_peered":"2024-07-27T06:45:09.486559+0000","last_clean":"2024-07-27T06:45:09.486559+0000","last_became_active":"2024-07-27T06:43:12.796889+0000","last_became_peered":"2024-07-27T06:43:12.796889+0000","last_unstale":"2024-07-27T06:45:09.486559+0000","last_undegraded":"2024-07-27T06:45:09.486559+0000","last_fullsized":"2024-07-27T06:45:09.486559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:24:36.275738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485662+0000","last_change":"2024-07-27T06:43:09.776600+0000","last_active":"2024-07-27T06:45:09.485662+0000","last_peered":"2024-07-27T06:45:09.485662+0000","last_clean":"2024-07-27T06:45:09.485662+0000","last_became_active":"2024-07-27T06:43:09.776163+0000","last_became_peered":"2024-07-27T06:43:09.776163+0000","last_unstale":"2024-07-27T06:45:09.485662+0000","last_undegraded":"2024-07-27T06:45:09.485662+0000","last_fullsized":"2024-07-27T06:45:09.485662+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:22:15.947916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481215+0000","last_change":"2024-07-27T06:43:12.804984+0000","last_active":"2024-07-27T06:45:09.481215+0000","last_peered":"2024-07-27T06:45:09.481215+0000","last_clean":"2024-07-27T06:45:09.481215+0000","last_became_active":"2024-07-27T06:43:12.804644+0000","last_became_peered":"2024-07-27T06:43:12.804644+0000","last_unstale":"2024-07-27T06:45:09.481215+0000","last_undegraded":"2024-07-27T06:45:09.481215+0000","last_fullsized":"2024-07-27T06:45:09.481215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:29:11.355981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862246+0000","last_change":"2024-07-27T06:43:09.772489+0000","last_active":"2024-07-27T06:45:09.862246+0000","last_peered":"2024-07-27T06:45:09.862246+0000","last_clean":"2024-07-27T06:45:09.862246+0000","last_became_active":"2024-07-27T06:43:09.772069+0000","last_became_peered":"2024-07-27T06:43:09.772069+0000","last_unstale":"2024-07-27T06:45:09.862246+0000","last_undegraded":"2024-07-27T06:45:09.862246+0000","last_fullsized":"2024-07-27T06:45:09.862246+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:15:30.549971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"81'15","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488589+0000","last_change":"2024-07-27T06:43:12.803376+0000","last_active":"2024-07-27T06:45:09.488589+0000","last_peered":"2024-07-27T06:45:09.488589+0000","last_clean":"2024-07-27T06:45:09.488589+0000","last_became_active":"2024-07-27T06:43:12.802961+0000","last_became_peered":"2024-07-27T06:43:12.802961+0000","last_unstale":"2024-07-27T06:45:09.488589+0000","last_undegraded":"2024-07-27T06:45:09.488589+0000","last_fullsized":"2024-07-27T06:45:09.488589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:00:24.649877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":256,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:09.226417+0000","last_change":"2024-07-27T06:43:09.773182+0000","last_active":"2024-07-27T06:47:09.226417+0000","last_peered":"2024-07-27T06:47:09.226417+0000","last_clean":"2024-07-27T06:47:09.226417+0000","last_became_active":"2024-07-27T06:43:09.772126+0000","last_became_peered":"2024-07-27T06:43:09.772126+0000","last_unstale":"2024-07-27T06:47:09.226417+0000","last_undegraded":"2024-07-27T06:47:09.226417+0000","last_fullsized":"2024-07-27T06:47:09.226417+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T17:45:26.598799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":220,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488424+0000","last_change":"2024-07-27T06:43:12.799385+0000","last_active":"2024-07-27T06:45:09.488424+0000","last_peered":"2024-07-27T06:45:09.488424+0000","last_clean":"2024-07-27T06:45:09.488424+0000","last_became_active":"2024-07-27T06:43:12.798639+0000","last_became_peered":"2024-07-27T06:43:12.798639+0000","last_unstale":"2024-07-27T06:45:09.488424+0000","last_undegraded":"2024-07-27T06:45:09.488424+0000","last_fullsized":"2024-07-27T06:45:09.488424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:56:29.462491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485303+0000","last_change":"2024-07-27T06:43:10.535429+0000","last_active":"2024-07-27T06:45:09.485303+0000","last_peered":"2024-07-27T06:45:09.485303+0000","last_clean":"2024-07-27T06:45:09.485303+0000","last_became_active":"2024-07-27T06:43:10.534952+0000","last_became_peered":"2024-07-27T06:43:10.534952+0000","last_unstale":"2024-07-27T06:45:09.485303+0000","last_undegraded":"2024-07-27T06:45:09.485303+0000","last_fullsized":"2024-07-27T06:45:09.485303+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:54:20.018263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862706+0000","last_change":"2024-07-27T06:43:12.804266+0000","last_active":"2024-07-27T06:45:09.862706+0000","last_peered":"2024-07-27T06:45:09.862706+0000","last_clean":"2024-07-27T06:45:09.862706+0000","last_became_active":"2024-07-27T06:43:12.803888+0000","last_became_peered":"2024-07-27T06:43:12.803888+0000","last_unstale":"2024-07-27T06:45:09.862706+0000","last_undegraded":"2024-07-27T06:45:09.862706+0000","last_fullsized":"2024-07-27T06:45:09.862706+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:48:00.490592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485827+0000","last_change":"2024-07-27T06:43:09.777656+0000","last_active":"2024-07-27T06:45:09.485827+0000","last_peered":"2024-07-27T06:45:09.485827+0000","last_clean":"2024-07-27T06:45:09.485827+0000","last_became_active":"2024-07-27T06:43:09.776620+0000","last_became_peered":"2024-07-27T06:43:09.776620+0000","last_unstale":"2024-07-27T06:45:09.485827+0000","last_undegraded":"2024-07-27T06:45:09.485827+0000","last_fullsized":"2024-07-27T06:45:09.485827+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:11:50.203623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483544+0000","last_change":"2024-07-27T06:43:12.796179+0000","last_active":"2024-07-27T06:45:09.483544+0000","last_peered":"2024-07-27T06:45:09.483544+0000","last_clean":"2024-07-27T06:45:09.483544+0000","last_became_active":"2024-07-27T06:43:12.795701+0000","last_became_peered":"2024-07-27T06:43:12.795701+0000","last_unstale":"2024-07-27T06:45:09.483544+0000","last_undegraded":"2024-07-27T06:45:09.483544+0000","last_fullsized":"2024-07-27T06:45:09.483544+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:20:12.236769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481627+0000","last_change":"2024-07-27T06:43:09.769949+0000","last_active":"2024-07-27T06:45:09.481627+0000","last_peered":"2024-07-27T06:45:09.481627+0000","last_clean":"2024-07-27T06:45:09.481627+0000","last_became_active":"2024-07-27T06:43:09.769602+0000","last_became_peered":"2024-07-27T06:43:09.769602+0000","last_unstale":"2024-07-27T06:45:09.481627+0000","last_undegraded":"2024-07-27T06:45:09.481627+0000","last_fullsized":"2024-07-27T06:45:09.481627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:28:02.883423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488869+0000","last_change":"2024-07-27T06:43:12.802230+0000","last_active":"2024-07-27T06:45:09.488869+0000","last_peered":"2024-07-27T06:45:09.488869+0000","last_clean":"2024-07-27T06:45:09.488869+0000","last_became_active":"2024-07-27T06:43:12.800944+0000","last_became_peered":"2024-07-27T06:43:12.800944+0000","last_unstale":"2024-07-27T06:45:09.488869+0000","last_undegraded":"2024-07-27T06:45:09.488869+0000","last_fullsized":"2024-07-27T06:45:09.488869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:05:54.585955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"83'62","reported_seq":124,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:03.049948+0000","last_change":"2024-07-27T06:43:09.768239+0000","last_active":"2024-07-27T06:47:03.049948+0000","last_peered":"2024-07-27T06:47:03.049948+0000","last_clean":"2024-07-27T06:47:03.049948+0000","last_became_active":"2024-07-27T06:43:09.767584+0000","last_became_peered":"2024-07-27T06:43:09.767584+0000","last_unstale":"2024-07-27T06:47:03.049948+0000","last_undegraded":"2024-07-27T06:47:03.049948+0000","last_fullsized":"2024-07-27T06:47:03.049948+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:35:39.552246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393846,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":32,"num_write":71,"num_write_kb":873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488571+0000","last_change":"2024-07-27T06:43:12.803779+0000","last_active":"2024-07-27T06:45:09.488571+0000","last_peered":"2024-07-27T06:45:09.488571+0000","last_clean":"2024-07-27T06:45:09.488571+0000","last_became_active":"2024-07-27T06:43:12.803359+0000","last_became_peered":"2024-07-27T06:43:12.803359+0000","last_unstale":"2024-07-27T06:45:09.488571+0000","last_undegraded":"2024-07-27T06:45:09.488571+0000","last_fullsized":"2024-07-27T06:45:09.488571+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:32:53.167039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'7","reported_seq":209,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:09.597024+0000","last_change":"2024-07-27T06:44:09.241782+0000","last_active":"2024-07-27T06:47:09.597024+0000","last_peered":"2024-07-27T06:47:09.597024+0000","last_clean":"2024-07-27T06:47:09.597024+0000","last_became_active":"2024-07-27T06:44:09.234692+0000","last_became_peered":"2024-07-27T06:44:09.234692+0000","last_unstale":"2024-07-27T06:47:09.597024+0000","last_undegraded":"2024-07-27T06:47:09.597024+0000","last_fullsized":"2024-07-27T06:47:09.597024+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:45:02.155686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":186,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1561,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488800+0000","last_change":"2024-07-27T06:43:12.805830+0000","last_active":"2024-07-27T06:45:09.488800+0000","last_peered":"2024-07-27T06:45:09.488800+0000","last_clean":"2024-07-27T06:45:09.488800+0000","last_became_active":"2024-07-27T06:43:12.805324+0000","last_became_peered":"2024-07-27T06:43:12.805324+0000","last_unstale":"2024-07-27T06:45:09.488800+0000","last_undegraded":"2024-07-27T06:45:09.488800+0000","last_fullsized":"2024-07-27T06:45:09.488800+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:38:31.430676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":109,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.489084+0000","last_change":"2024-07-27T06:41:43.549132+0000","last_active":"2024-07-27T06:45:09.489084+0000","last_peered":"2024-07-27T06:45:09.489084+0000","last_clean":"2024-07-27T06:45:09.489084+0000","last_became_active":"2024-07-27T06:41:43.542635+0000","last_became_peered":"2024-07-27T06:41:43.542635+0000","last_unstale":"2024-07-27T06:45:09.489084+0000","last_undegraded":"2024-07-27T06:45:09.489084+0000","last_fullsized":"2024-07-27T06:45:09.489084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:38:21.639907+0000","last_clean_scrub_stamp":"2024-07-27T06:38:21.639907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:10:06.417387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.861700+0000","last_change":"2024-07-27T06:43:09.768774+0000","last_active":"2024-07-27T06:45:09.861700+0000","last_peered":"2024-07-27T06:45:09.861700+0000","last_clean":"2024-07-27T06:45:09.861700+0000","last_became_active":"2024-07-27T06:43:09.768345+0000","last_became_peered":"2024-07-27T06:43:09.768345+0000","last_unstale":"2024-07-27T06:45:09.861700+0000","last_undegraded":"2024-07-27T06:45:09.861700+0000","last_fullsized":"2024-07-27T06:45:09.861700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:40:49.411746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485670+0000","last_change":"2024-07-27T06:43:12.797817+0000","last_active":"2024-07-27T06:45:09.485670+0000","last_peered":"2024-07-27T06:45:09.485670+0000","last_clean":"2024-07-27T06:45:09.485670+0000","last_became_active":"2024-07-27T06:43:12.797483+0000","last_became_peered":"2024-07-27T06:43:12.797483+0000","last_unstale":"2024-07-27T06:45:09.485670+0000","last_undegraded":"2024-07-27T06:45:09.485670+0000","last_fullsized":"2024-07-27T06:45:09.485670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:24:44.539505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:11.996718+0000","last_change":"2024-07-27T06:43:09.776880+0000","last_active":"2024-07-27T06:45:11.996718+0000","last_peered":"2024-07-27T06:45:11.996718+0000","last_clean":"2024-07-27T06:45:11.996718+0000","last_became_active":"2024-07-27T06:43:09.776486+0000","last_became_peered":"2024-07-27T06:43:09.776486+0000","last_unstale":"2024-07-27T06:45:11.996718+0000","last_undegraded":"2024-07-27T06:45:11.996718+0000","last_fullsized":"2024-07-27T06:45:11.996718+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:55:25.436405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.480961+0000","last_change":"2024-07-27T06:43:12.799755+0000","last_active":"2024-07-27T06:45:09.480961+0000","last_peered":"2024-07-27T06:45:09.480961+0000","last_clean":"2024-07-27T06:45:09.480961+0000","last_became_active":"2024-07-27T06:43:12.799412+0000","last_became_peered":"2024-07-27T06:43:12.799412+0000","last_unstale":"2024-07-27T06:45:09.480961+0000","last_undegraded":"2024-07-27T06:45:09.480961+0000","last_fullsized":"2024-07-27T06:45:09.480961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:45:50.545431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":255,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:09.225886+0000","last_change":"2024-07-27T06:43:10.533039+0000","last_active":"2024-07-27T06:47:09.225886+0000","last_peered":"2024-07-27T06:47:09.225886+0000","last_clean":"2024-07-27T06:47:09.225886+0000","last_became_active":"2024-07-27T06:43:10.532583+0000","last_became_peered":"2024-07-27T06:43:10.532583+0000","last_unstale":"2024-07-27T06:47:09.225886+0000","last_undegraded":"2024-07-27T06:47:09.225886+0000","last_fullsized":"2024-07-27T06:47:09.225886+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:04:51.954463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":218,"num_read_kb":218,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"81'8","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633096+0000","last_change":"2024-07-27T06:44:09.238218+0000","last_active":"2024-07-27T06:45:09.633096+0000","last_peered":"2024-07-27T06:45:09.633096+0000","last_clean":"2024-07-27T06:45:09.633096+0000","last_became_active":"2024-07-27T06:44:09.237841+0000","last_became_peered":"2024-07-27T06:44:09.237841+0000","last_unstale":"2024-07-27T06:45:09.633096+0000","last_undegraded":"2024-07-27T06:45:09.633096+0000","last_fullsized":"2024-07-27T06:45:09.633096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:19.101854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.2","version":"83'8","reported_seq":262,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:10.266911+0000","last_change":"2024-07-27T06:43:09.776214+0000","last_active":"2024-07-27T06:47:10.266911+0000","last_peered":"2024-07-27T06:47:10.266911+0000","last_clean":"2024-07-27T06:47:10.266911+0000","last_became_active":"2024-07-27T06:43:09.775719+0000","last_became_peered":"2024-07-27T06:43:09.775719+0000","last_unstale":"2024-07-27T06:47:10.266911+0000","last_undegraded":"2024-07-27T06:47:10.266911+0000","last_fullsized":"2024-07-27T06:47:10.266911+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T06:57:20.381599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1428,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":222,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481735+0000","last_change":"2024-07-27T06:43:12.794909+0000","last_active":"2024-07-27T06:45:09.481735+0000","last_peered":"2024-07-27T06:45:09.481735+0000","last_clean":"2024-07-27T06:45:09.481735+0000","last_became_active":"2024-07-27T06:43:12.794497+0000","last_became_peered":"2024-07-27T06:43:12.794497+0000","last_unstale":"2024-07-27T06:45:09.481735+0000","last_undegraded":"2024-07-27T06:45:09.481735+0000","last_fullsized":"2024-07-27T06:45:09.481735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:42:49.760126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633255+0000","last_change":"2024-07-27T06:43:10.534240+0000","last_active":"2024-07-27T06:45:09.633255+0000","last_peered":"2024-07-27T06:45:09.633255+0000","last_clean":"2024-07-27T06:45:09.633255+0000","last_became_active":"2024-07-27T06:43:10.533596+0000","last_became_peered":"2024-07-27T06:43:10.533596+0000","last_unstale":"2024-07-27T06:45:09.633255+0000","last_undegraded":"2024-07-27T06:45:09.633255+0000","last_fullsized":"2024-07-27T06:45:09.633255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:40:16.098839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481991+0000","last_change":"2024-07-27T06:43:12.805248+0000","last_active":"2024-07-27T06:45:09.481991+0000","last_peered":"2024-07-27T06:45:09.481991+0000","last_clean":"2024-07-27T06:45:09.481991+0000","last_became_active":"2024-07-27T06:43:12.803693+0000","last_became_peered":"2024-07-27T06:43:12.803693+0000","last_unstale":"2024-07-27T06:45:09.481991+0000","last_undegraded":"2024-07-27T06:45:09.481991+0000","last_fullsized":"2024-07-27T06:45:09.481991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T16:58:17.466465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486692+0000","last_change":"2024-07-27T06:43:10.533701+0000","last_active":"2024-07-27T06:45:09.486692+0000","last_peered":"2024-07-27T06:45:09.486692+0000","last_clean":"2024-07-27T06:45:09.486692+0000","last_became_active":"2024-07-27T06:43:10.533109+0000","last_became_peered":"2024-07-27T06:43:10.533109+0000","last_unstale":"2024-07-27T06:45:09.486692+0000","last_undegraded":"2024-07-27T06:45:09.486692+0000","last_fullsized":"2024-07-27T06:45:09.486692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:32:52.906691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.481918+0000","last_change":"2024-07-27T06:43:12.798479+0000","last_active":"2024-07-27T06:45:09.481918+0000","last_peered":"2024-07-27T06:45:09.481918+0000","last_clean":"2024-07-27T06:45:09.481918+0000","last_became_active":"2024-07-27T06:43:12.798124+0000","last_became_peered":"2024-07-27T06:43:12.798124+0000","last_unstale":"2024-07-27T06:45:09.481918+0000","last_undegraded":"2024-07-27T06:45:09.481918+0000","last_fullsized":"2024-07-27T06:45:09.481918+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:14:55.267213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"83'101","reported_seq":222,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:03.434736+0000","last_change":"2024-07-27T06:43:09.770122+0000","last_active":"2024-07-27T06:47:03.434736+0000","last_peered":"2024-07-27T06:47:03.434736+0000","last_clean":"2024-07-27T06:47:03.434736+0000","last_became_active":"2024-07-27T06:43:09.769784+0000","last_became_peered":"2024-07-27T06:43:09.769784+0000","last_unstale":"2024-07-27T06:47:03.434736+0000","last_undegraded":"2024-07-27T06:47:03.434736+0000","last_fullsized":"2024-07-27T06:47:03.434736+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:47:00.398135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":302,"num_read_kb":268,"num_write":185,"num_write_kb":1222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485891+0000","last_change":"2024-07-27T06:43:12.799603+0000","last_active":"2024-07-27T06:45:09.485891+0000","last_peered":"2024-07-27T06:45:09.485891+0000","last_clean":"2024-07-27T06:45:09.485891+0000","last_became_active":"2024-07-27T06:43:12.799189+0000","last_became_peered":"2024-07-27T06:43:12.799189+0000","last_unstale":"2024-07-27T06:45:09.485891+0000","last_undegraded":"2024-07-27T06:45:09.485891+0000","last_fullsized":"2024-07-27T06:45:09.485891+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:32:30.603258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862220+0000","last_change":"2024-07-27T06:43:10.004726+0000","last_active":"2024-07-27T06:45:09.862220+0000","last_peered":"2024-07-27T06:45:09.862220+0000","last_clean":"2024-07-27T06:45:09.862220+0000","last_became_active":"2024-07-27T06:43:10.004367+0000","last_became_peered":"2024-07-27T06:43:10.004367+0000","last_unstale":"2024-07-27T06:45:09.862220+0000","last_undegraded":"2024-07-27T06:45:09.862220+0000","last_fullsized":"2024-07-27T06:45:09.862220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T11:22:39.942391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"81'7","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862427+0000","last_change":"2024-07-27T06:43:12.796511+0000","last_active":"2024-07-27T06:45:09.862427+0000","last_peered":"2024-07-27T06:45:09.862427+0000","last_clean":"2024-07-27T06:45:09.862427+0000","last_became_active":"2024-07-27T06:43:12.796152+0000","last_became_peered":"2024-07-27T06:43:12.796152+0000","last_unstale":"2024-07-27T06:45:09.862427+0000","last_undegraded":"2024-07-27T06:45:09.862427+0000","last_fullsized":"2024-07-27T06:45:09.862427+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:03:02.508990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822077+0000","last_change":"2024-07-27T06:43:10.533656+0000","last_active":"2024-07-27T06:45:09.822077+0000","last_peered":"2024-07-27T06:45:09.822077+0000","last_clean":"2024-07-27T06:45:09.822077+0000","last_became_active":"2024-07-27T06:43:10.533295+0000","last_became_peered":"2024-07-27T06:43:10.533295+0000","last_unstale":"2024-07-27T06:45:09.822077+0000","last_undegraded":"2024-07-27T06:45:09.822077+0000","last_fullsized":"2024-07-27T06:45:09.822077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:11:00.032511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"81'2","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485237+0000","last_change":"2024-07-27T06:43:12.795106+0000","last_active":"2024-07-27T06:45:09.485237+0000","last_peered":"2024-07-27T06:45:09.485237+0000","last_clean":"2024-07-27T06:45:09.485237+0000","last_became_active":"2024-07-27T06:43:12.794729+0000","last_became_peered":"2024-07-27T06:43:12.794729+0000","last_unstale":"2024-07-27T06:45:09.485237+0000","last_undegraded":"2024-07-27T06:45:09.485237+0000","last_fullsized":"2024-07-27T06:45:09.485237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:19:16.542363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485673+0000","last_change":"2024-07-27T06:43:09.770016+0000","last_active":"2024-07-27T06:45:09.485673+0000","last_peered":"2024-07-27T06:45:09.485673+0000","last_clean":"2024-07-27T06:45:09.485673+0000","last_became_active":"2024-07-27T06:43:09.769672+0000","last_became_peered":"2024-07-27T06:43:09.769672+0000","last_unstale":"2024-07-27T06:45:09.485673+0000","last_undegraded":"2024-07-27T06:45:09.485673+0000","last_fullsized":"2024-07-27T06:45:09.485673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:30:55.737494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483659+0000","last_change":"2024-07-27T06:43:12.793690+0000","last_active":"2024-07-27T06:45:09.483659+0000","last_peered":"2024-07-27T06:45:09.483659+0000","last_clean":"2024-07-27T06:45:09.483659+0000","last_became_active":"2024-07-27T06:43:12.793352+0000","last_became_peered":"2024-07-27T06:43:12.793352+0000","last_unstale":"2024-07-27T06:45:09.483659+0000","last_undegraded":"2024-07-27T06:45:09.483659+0000","last_fullsized":"2024-07-27T06:45:09.483659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T07:22:47.606990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.491196+0000","last_change":"2024-07-27T06:45:09.490738+0000","last_active":"2024-07-27T06:45:09.491196+0000","last_peered":"2024-07-27T06:45:09.491196+0000","last_clean":"2024-07-27T06:45:09.491196+0000","last_became_active":"2024-07-27T06:45:09.487147+0000","last_became_peered":"2024-07-27T06:45:09.487147+0000","last_unstale":"2024-07-27T06:45:09.491196+0000","last_undegraded":"2024-07-27T06:45:09.491196+0000","last_fullsized":"2024-07-27T06:45:09.491196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:38:46.732987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"81'8","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485570+0000","last_change":"2024-07-27T06:43:12.792648+0000","last_active":"2024-07-27T06:45:09.485570+0000","last_peered":"2024-07-27T06:45:09.485570+0000","last_clean":"2024-07-27T06:45:09.485570+0000","last_became_active":"2024-07-27T06:43:12.792338+0000","last_became_peered":"2024-07-27T06:43:12.792338+0000","last_unstale":"2024-07-27T06:45:09.485570+0000","last_undegraded":"2024-07-27T06:45:09.485570+0000","last_fullsized":"2024-07-27T06:45:09.485570+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:36:52.605957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.827632+0000","last_change":"2024-07-27T06:45:09.827140+0000","last_active":"2024-07-27T06:45:09.827632+0000","last_peered":"2024-07-27T06:45:09.827632+0000","last_clean":"2024-07-27T06:45:09.827632+0000","last_became_active":"2024-07-27T06:45:09.823119+0000","last_became_peered":"2024-07-27T06:45:09.823119+0000","last_unstale":"2024-07-27T06:45:09.827632+0000","last_undegraded":"2024-07-27T06:45:09.827632+0000","last_fullsized":"2024-07-27T06:45:09.827632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T13:32:19.682193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.488730+0000","last_change":"2024-07-27T06:43:12.809347+0000","last_active":"2024-07-27T06:45:09.488730+0000","last_peered":"2024-07-27T06:45:09.488730+0000","last_clean":"2024-07-27T06:45:09.488730+0000","last_became_active":"2024-07-27T06:43:12.808980+0000","last_became_peered":"2024-07-27T06:43:12.808980+0000","last_unstale":"2024-07-27T06:45:09.488730+0000","last_undegraded":"2024-07-27T06:45:09.488730+0000","last_fullsized":"2024-07-27T06:45:09.488730+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:05:14.132641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"81'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.633284+0000","last_change":"2024-07-27T06:44:09.237908+0000","last_active":"2024-07-27T06:45:09.633284+0000","last_peered":"2024-07-27T06:45:09.633284+0000","last_clean":"2024-07-27T06:45:09.633284+0000","last_became_active":"2024-07-27T06:44:09.237503+0000","last_became_peered":"2024-07-27T06:44:09.237503+0000","last_unstale":"2024-07-27T06:45:09.633284+0000","last_undegraded":"2024-07-27T06:45:09.633284+0000","last_fullsized":"2024-07-27T06:45:09.633284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T12:50:29.654244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.486391+0000","last_change":"2024-07-27T06:43:09.776000+0000","last_active":"2024-07-27T06:45:09.486391+0000","last_peered":"2024-07-27T06:45:09.486391+0000","last_clean":"2024-07-27T06:45:09.486391+0000","last_became_active":"2024-07-27T06:43:09.775522+0000","last_became_peered":"2024-07-27T06:43:09.775522+0000","last_unstale":"2024-07-27T06:45:09.486391+0000","last_undegraded":"2024-07-27T06:45:09.486391+0000","last_fullsized":"2024-07-27T06:45:09.486391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T08:39:42.650177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"81'9","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.822084+0000","last_change":"2024-07-27T06:43:12.807062+0000","last_active":"2024-07-27T06:45:09.822084+0000","last_peered":"2024-07-27T06:45:09.822084+0000","last_clean":"2024-07-27T06:45:09.822084+0000","last_became_active":"2024-07-27T06:43:12.805816+0000","last_became_peered":"2024-07-27T06:43:12.805816+0000","last_unstale":"2024-07-27T06:45:09.822084+0000","last_undegraded":"2024-07-27T06:45:09.822084+0000","last_fullsized":"2024-07-27T06:45:09.822084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:44:16.261663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.483465+0000","last_change":"2024-07-27T06:44:09.250070+0000","last_active":"2024-07-27T06:45:09.483465+0000","last_peered":"2024-07-27T06:45:09.483465+0000","last_clean":"2024-07-27T06:45:09.483465+0000","last_became_active":"2024-07-27T06:44:09.237290+0000","last_became_peered":"2024-07-27T06:44:09.237290+0000","last_unstale":"2024-07-27T06:45:09.483465+0000","last_undegraded":"2024-07-27T06:45:09.483465+0000","last_fullsized":"2024-07-27T06:45:09.483465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T15:43:51.590487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1f","version":"81'6","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.862423+0000","last_change":"2024-07-27T06:43:12.809135+0000","last_active":"2024-07-27T06:45:09.862423+0000","last_peered":"2024-07-27T06:45:09.862423+0000","last_clean":"2024-07-27T06:45:09.862423+0000","last_became_active":"2024-07-27T06:43:12.806731+0000","last_became_peered":"2024-07-27T06:43:12.806731+0000","last_unstale":"2024-07-27T06:45:09.862423+0000","last_undegraded":"2024-07-27T06:45:09.862423+0000","last_fullsized":"2024-07-27T06:45:09.862423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:47:50.397517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:10.601326+0000","last_change":"2024-07-27T06:43:10.533608+0000","last_active":"2024-07-27T06:45:10.601326+0000","last_peered":"2024-07-27T06:45:10.601326+0000","last_clean":"2024-07-27T06:45:10.601326+0000","last_became_active":"2024-07-27T06:43:10.532663+0000","last_became_peered":"2024-07-27T06:43:10.532663+0000","last_unstale":"2024-07-27T06:45:10.601326+0000","last_undegraded":"2024-07-27T06:45:10.601326+0000","last_fullsized":"2024-07-27T06:45:10.601326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T09:05:33.155427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"81'5","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:45:09.485883+0000","last_change":"2024-07-27T06:43:12.789890+0000","last_active":"2024-07-27T06:45:09.485883+0000","last_peered":"2024-07-27T06:45:09.485883+0000","last_clean":"2024-07-27T06:45:09.485883+0000","last_became_active":"2024-07-27T06:43:12.789561+0000","last_became_peered":"2024-07-27T06:43:12.789561+0000","last_unstale":"2024-07-27T06:45:09.485883+0000","last_undegraded":"2024-07-27T06:45:09.485883+0000","last_fullsized":"2024-07-27T06:45:09.485883+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:11.766526+0000","last_clean_scrub_stamp":"2024-07-27T06:43:11.766526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T14:35:49.454287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"83'15","reported_seq":267,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-07-27T06:47:10.266314+0000","last_change":"2024-07-27T06:43:09.773471+0000","last_active":"2024-07-27T06:47:10.266314+0000","last_peered":"2024-07-27T06:47:10.266314+0000","last_clean":"2024-07-27T06:47:10.266314+0000","last_became_active":"2024-07-27T06:43:09.773213+0000","last_became_peered":"2024-07-27T06:43:09.773213+0000","last_unstale":"2024-07-27T06:47:10.266314+0000","last_undegraded":"2024-07-27T06:47:10.266314+0000","last_fullsized":"2024-07-27T06:47:10.266314+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-27T06:43:08.745542+0000","last_clean_scrub_stamp":"2024-07-27T06:43:08.745542+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T10:54:38.770198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":220,"num_read_kb":220,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":861595,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":2526,"num_read_kb":1497,"num_write":488,"num_write_kb":2245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1561,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":823296,"data_stored":2584785,"data_compressed":112140,"data_compressed_allocated":147456,"data_compressed_original":2285568,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283116,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678080,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":461919,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874308,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":1512469,"data_compressed":65700,"data_compressed_allocated":86016,"data_compressed_original":991232,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037835,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":854533,"data_compressed":30901,"data_compressed_allocated":40960,"data_compressed_original":425984,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234063,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":854911,"data_compressed":30901,"data_compressed_allocated":40960,"data_compressed_original":425984,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430291,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":1055194,"data_compressed":47609,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593817,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":461874,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790046,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":1447808,"data_compressed":65700,"data_compressed_allocated":86016,"data_compressed_original":991232,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953571,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":929792,"data_stored":462597,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":1247851,"data_compressed":48992,"data_compressed_allocated":65536,"data_compressed_original":794624,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378734,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":527806,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542258,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":529064,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705783,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":462905,"data_compressed":11612,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34377,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35997,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":67611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":787422,"data_compressed":37380,"data_compressed_allocated":49152,"data_compressed_original":761856,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":397128,"data_compressed":18091,"data_compressed_allocated":24576,"data_compressed_original":368640,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":461789,"data_compressed":18091,"data_compressed_allocated":24576,"data_compressed_original":368640,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-27T06:47:15.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:47:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:15 smithi077 ceph-mon[37153]: pgmap v404: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 174 KiB/s wr, 57 op/s 2024-07-27T06:47:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:15 smithi047 ceph-mon[36835]: pgmap v404: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 174 KiB/s wr, 57 op/s 2024-07-27T06:47:16.040 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.679+0000 7f050ece3640 1 -- 172.21.15.29:0/2889055153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0508108670 msgr2=0x7f050810eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.679+0000 7f050ece3640 1 --2- 172.21.15.29:0/2889055153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0508108670 0x7f050810eea0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0500009ca0 tx=0x7f050002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.679+0000 7f050ece3640 1 -- 172.21.15.29:0/2889055153 shutdown_connections 2024-07-27T06:47:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.679+0000 7f050ece3640 1 --2- 172.21.15.29:0/2889055153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0508108670 0x7f050810eea0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.679+0000 7f050ece3640 1 --2- 172.21.15.29:0/2889055153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0508107d20 0x7f0508108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.679+0000 7f050ece3640 1 --2- 172.21.15.29:0/2889055153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05081023f0 0x7f0508102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.679+0000 7f050ece3640 1 -- 172.21.15.29:0/2889055153 >> 172.21.15.29:0/2889055153 conn(0x7f05080fddf0 msgr2=0x7f0508100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.679+0000 7f050ece3640 1 -- 172.21.15.29:0/2889055153 shutdown_connections 2024-07-27T06:47:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.679+0000 7f050ece3640 1 -- 172.21.15.29:0/2889055153 wait complete. 2024-07-27T06:47:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.680+0000 7f050ece3640 1 Processor -- start 2024-07-27T06:47:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.680+0000 7f050ece3640 1 -- start start 2024-07-27T06:47:16.694 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.680+0000 7f050ece3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05081023f0 0x7f0508199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.680+0000 7f050ece3640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0508107d20 0x7f050819a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ece3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0508108670 0x7f050819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ece3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0508112340 con 0x7f05081023f0 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ece3640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f05081121c0 con 0x7f0508107d20 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ece3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f05081124c0 con 0x7f0508108670 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ca7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05081023f0 0x7f0508199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ca7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05081023f0 0x7f0508199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43382/0 (socket says 172.21.15.29:43382) 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ca7f640 1 -- 172.21.15.29:0/928891400 learned_addr learned my addr 172.21.15.29:0/928891400 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f0507fff640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0508107d20 0x7f050819a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050d280640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0508108670 0x7f050819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ca7f640 1 -- 172.21.15.29:0/928891400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0508108670 msgr2=0x7f050819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ca7f640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0508108670 0x7f050819f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ca7f640 1 -- 172.21.15.29:0/928891400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0508107d20 msgr2=0x7f050819a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ca7f640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0508107d20 0x7f050819a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ca7f640 1 -- 172.21.15.29:0/928891400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050819fcf0 con 0x7f05081023f0 2024-07-27T06:47:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050d280640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0508108670 0x7f050819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ca7f640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05081023f0 0x7f0508199e50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00e9f0 tx=0x7f04fc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f0505ffb640 1 -- 172.21.15.29:0/928891400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc00cd80 con 0x7f05081023f0 2024-07-27T06:47:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f0505ffb640 1 -- 172.21.15.29:0/928891400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04fc0054d0 con 0x7f05081023f0 2024-07-27T06:47:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f050ece3640 1 -- 172.21.15.29:0/928891400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050819ffe0 con 0x7f05081023f0 2024-07-27T06:47:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.681+0000 7f0505ffb640 1 -- 172.21.15.29:0/928891400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc010690 con 0x7f05081023f0 2024-07-27T06:47:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.682+0000 7f050ece3640 1 -- 172.21.15.29:0/928891400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05081a6800 con 0x7f05081023f0 2024-07-27T06:47:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.684+0000 7f0505ffb640 1 -- 172.21.15.29:0/928891400 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f04fc004e90 con 0x7f05081023f0 2024-07-27T06:47:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.684+0000 7f0515bf4740 1 Processor -- start 2024-07-27T06:47:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.684+0000 7f0515bf4740 1 -- start start 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.684+0000 7f0515bf4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 0x55d77a5d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.684+0000 7f0515bf4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.684+0000 7f0515bf4740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a20f4d0 0x55d77a5d3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.684+0000 7f0515bf4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d77a5483b0 con 0x55d77a2301b0 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.684+0000 7f0515bf4740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55d77a5378b0 con 0x55d77a20f4d0 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.684+0000 7f0515bf4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d77a53b8f0 con 0x55d77a2072f0 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f050d280640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 0x55d77a5d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f050d280640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 0x55d77a5d2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43388/0 (socket says 172.21.15.29:43388) 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f050d280640 1 -- 172.21.15.29:0/859956255 learned_addr learned my addr 172.21.15.29:0/859956255 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f050ca7f640 1 --2- 172.21.15.29:0/859956255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f0507fff640 1 --2- 172.21.15.29:0/859956255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a20f4d0 0x55d77a5d3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f050d280640 1 -- 172.21.15.29:0/859956255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 msgr2=0x55d77a5d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f050d280640 1 --2- 172.21.15.29:0/859956255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f050d280640 1 -- 172.21.15.29:0/859956255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a20f4d0 msgr2=0x55d77a5d3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f050d280640 1 --2- 172.21.15.29:0/859956255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a20f4d0 0x55d77a5d3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f050d280640 1 -- 172.21.15.29:0/859956255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d77a4de9a0 con 0x55d77a2301b0 2024-07-27T06:47:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f0505ffb640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f04f80b3cb0 0x7f04f80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f050ca7f640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f04f80b3cb0 0x7f04f80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f050d280640 1 --2- 172.21.15.29:0/859956255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 0x55d77a5d2ce0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0500006010 tx=0x7f0500009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972d912a84771c66 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.685+0000 7f0505ffb640 1 -- 172.21.15.29:0/928891400 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f04fc0e4430 con 0x7f05081023f0 2024-07-27T06:47:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.686+0000 7f04f2ffd640 1 -- 172.21.15.29:0/859956255 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0500031cf0 con 0x55d77a2301b0 2024-07-27T06:47:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.686+0000 7f04f2ffd640 1 -- 172.21.15.29:0/859956255 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0500036070 con 0x55d77a2301b0 2024-07-27T06:47:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.686+0000 7f04f2ffd640 1 -- 172.21.15.29:0/859956255 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05000042a0 con 0x55d77a2301b0 2024-07-27T06:47:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.686+0000 7f0515bf4740 1 -- 172.21.15.29:0/859956255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 msgr2=0x55d77a5d2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.686+0000 7f050ca7f640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f04f80b3cb0 0x7f04f80b61c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x55d77a5d1960 tx=0x7f04fc004a90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.686+0000 7f0515bf4740 1 --2- 172.21.15.29:0/859956255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 0x55d77a5d2ce0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0500006010 tx=0x7f0500009800 comp rx=0 tx=0).stop 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.687+0000 7f0515bf4740 1 -- 172.21.15.29:0/859956255 shutdown_connections 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.687+0000 7f0515bf4740 1 --2- 172.21.15.29:0/859956255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a20f4d0 0x55d77a5d3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.687+0000 7f0515bf4740 1 --2- 172.21.15.29:0/859956255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.687+0000 7f0515bf4740 1 --2- 172.21.15.29:0/859956255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 0x55d77a5d2ce0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.687+0000 7f0515bf4740 1 -- 172.21.15.29:0/859956255 >> 172.21.15.29:0/859956255 conn(0x55d77a224300 msgr2=0x55d77a5d0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.687+0000 7f0515bf4740 1 -- 172.21.15.29:0/859956255 shutdown_connections 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.687+0000 7f0515bf4740 1 -- 172.21.15.29:0/859956255 wait complete. 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.687+0000 7f0515bf4740 1 -- start start 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.687+0000 7f0515bf4740 1 Processor -- start 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f0515bf4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5de7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f0515bf4740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a20f4d0 0x55d77a5ded20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f0515bf4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 0x55d77a5ea720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f0515bf4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d77a471ab0 con 0x55d77a2301b0 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f0515bf4740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55d77a4bfcf0 con 0x55d77a20f4d0 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f0515bf4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d77a5e1400 con 0x55d77a2072f0 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f0515bf4740 10 client.0 _ll_register_callbacks cb 0x55d77a5319f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f050d280640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 0x55d77a5ea720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f050d280640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 0x55d77a5ea720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43398/0 (socket says 172.21.15.29:43398) 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f050d280640 1 -- 172.21.15.29:0/1446036523 learned_addr learned my addr 172.21.15.29:0/1446036523 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f0507fff640 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a20f4d0 0x55d77a5ded20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f050d280640 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5de7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f050d280640 1 -- 172.21.15.29:0/1446036523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 msgr2=0x55d77a5de7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f050d280640 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5de7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f050d280640 1 -- 172.21.15.29:0/1446036523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a20f4d0 msgr2=0x55d77a5ded20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f050d280640 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a20f4d0 0x55d77a5ded20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f050d280640 1 -- 172.21.15.29:0/1446036523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d77a29b790 con 0x55d77a2301b0 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f050d280640 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 0x55d77a5ea720 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0500003ca0 tx=0x7f0500003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1217ca81e78163ca server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f04e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.688+0000 7f04e7fff640 1 -- 172.21.15.29:0/1446036523 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0500049040 con 0x55d77a2301b0 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f04e7fff640 1 -- 172.21.15.29:0/1446036523 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f0500031450 con 0x55d77a2301b0 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f0515bf4740 1 -- 172.21.15.29:0/1446036523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0500036070 con 0x55d77a2301b0 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f04e7fff640 1 -- 172.21.15.29:0/1446036523 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0500031650 con 0x55d77a2301b0 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f0515bf4740 10 client.15564 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f0515bf4740 1 -- 172.21.15.29:0/1446036523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d77a314620 con 0x55d77a2301b0 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f04e7fff640 1 -- 172.21.15.29:0/1446036523 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0500036070 con 0x55d77a2301b0 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f04e7fff640 1 -- 172.21.15.29:0/1446036523 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f050002f820 con 0x55d77a2301b0 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f0515bf4740 10 client.15564 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f0515bf4740 10 client.15564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f0515bf4740 20 client.15564 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f0515bf4740 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d77a1eed80 0x55d77a5f0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f0515bf4740 4 client.15564 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.689+0000 7f0515bf4740 1 -- 172.21.15.29:0/1446036523 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d77a54a300 con 0x55d77a1eed80 2024-07-27T06:47:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.690+0000 7f0507fff640 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d77a1eed80 0x55d77a5f0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.690+0000 7f0507fff640 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d77a1eed80 0x55d77a5f0520 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47b5744994f689e6 server_cookie=146fa9098bc70fdc in_seq=0 out_seq=0 2024-07-27T06:47:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.690+0000 7f04e7fff640 10 client.15564 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f04e7fff640 1 -- 172.21.15.29:0/1446036523 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d77a54a300 con 0x55d77a1eed80 2024-07-27T06:47:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f04e7fff640 10 client.15564 handle_command_reply: tid=0 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f0515bf4740 1 client.15564 shutdown 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f0515bf4740 1 -- 172.21.15.29:0/1446036523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 msgr2=0x55d77a5ea720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f0515bf4740 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 0x55d77a5ea720 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0500003ca0 tx=0x7f0500003da0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f0515bf4740 1 -- 172.21.15.29:0/1446036523 shutdown_connections 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f0515bf4740 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2301b0 0x55d77a5ea720 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f0515bf4740 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a20f4d0 0x55d77a5ded20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f0515bf4740 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d77a1eed80 0x55d77a5f0520 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f0515bf4740 1 --2- 172.21.15.29:0/1446036523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5de7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f04e7fff640 0 client.15564 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f0515bf4740 1 -- 172.21.15.29:0/1446036523 >> 172.21.15.29:0/1446036523 conn(0x55d77a224300 msgr2=0x55d77a5d3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f0515bf4740 1 -- 172.21.15.29:0/1446036523 shutdown_connections 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f0515bf4740 1 -- 172.21.15.29:0/1446036523 wait complete. 2024-07-27T06:47:16.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.691+0000 7f0515bf4740 20 client.15564 trim_cache size 0 max 16384 2024-07-27T06:47:16.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f0515bf4740 1 Processor -- start 2024-07-27T06:47:16.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f0515bf4740 1 -- start start 2024-07-27T06:47:16.709 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f0515bf4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2072f0 0x55d77a5e04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f0515bf4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a20f4d0 0x55d77a5d8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f0515bf4740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a2301b0 0x55d77a5f3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f0515bf4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f05081aeff0 con 0x55d77a2072f0 2024-07-27T06:47:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f0515bf4740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f05081b0b30 con 0x55d77a2301b0 2024-07-27T06:47:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f0515bf4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f05081af6c0 con 0x55d77a20f4d0 2024-07-27T06:47:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f050d280640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2072f0 0x55d77a5e04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f050d280640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2072f0 0x55d77a5e04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43404/0 (socket says 172.21.15.29:43404) 2024-07-27T06:47:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f050d280640 1 -- 172.21.15.29:0/1144868083 learned_addr learned my addr 172.21.15.29:0/1144868083 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f0507fff640 1 --2- 172.21.15.29:0/1144868083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a20f4d0 0x55d77a5d8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f050d280640 1 -- 172.21.15.29:0/1144868083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a20f4d0 msgr2=0x55d77a5d8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f050d280640 1 --2- 172.21.15.29:0/1144868083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a20f4d0 0x55d77a5d8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f050d280640 1 -- 172.21.15.29:0/1144868083 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a2301b0 msgr2=0x55d77a5f3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:16.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f050d280640 1 --2- 172.21.15.29:0/1144868083 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a2301b0 0x55d77a5f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f050d280640 1 -- 172.21.15.29:0/1144868083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d77a5d5af0 con 0x55d77a2072f0 2024-07-27T06:47:16.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.707+0000 7f050d280640 1 --2- 172.21.15.29:0/1144868083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2072f0 0x55d77a5e04b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0500009ca0 tx=0x7f050002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee0e8343bd614255 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:16.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.708+0000 7f04f27fc640 1 -- 172.21.15.29:0/1144868083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050002feb0 con 0x55d77a2072f0 2024-07-27T06:47:16.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.708+0000 7f04f27fc640 1 -- 172.21.15.29:0/1144868083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f050005e070 con 0x55d77a2072f0 2024-07-27T06:47:16.711 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.708+0000 7f04f27fc640 1 -- 172.21.15.29:0/1144868083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0500030c50 con 0x55d77a2072f0 2024-07-27T06:47:16.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.708+0000 7f0515bf4740 1 -- 172.21.15.29:0/1144868083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2072f0 msgr2=0x55d77a5e04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:16.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.708+0000 7f0515bf4740 1 --2- 172.21.15.29:0/1144868083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2072f0 0x55d77a5e04b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0500009ca0 tx=0x7f050002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.708+0000 7f0515bf4740 1 -- 172.21.15.29:0/1144868083 shutdown_connections 2024-07-27T06:47:16.713 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.708+0000 7f0515bf4740 1 --2- 172.21.15.29:0/1144868083 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a2301b0 0x55d77a5f3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.708+0000 7f0515bf4740 1 --2- 172.21.15.29:0/1144868083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a20f4d0 0x55d77a5d8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.708+0000 7f0515bf4740 1 --2- 172.21.15.29:0/1144868083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a2072f0 0x55d77a5e04b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.708+0000 7f0515bf4740 1 -- 172.21.15.29:0/1144868083 >> 172.21.15.29:0/1144868083 conn(0x55d77a224300 msgr2=0x55d77a5dce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.708+0000 7f0515bf4740 1 -- 172.21.15.29:0/1144868083 shutdown_connections 2024-07-27T06:47:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.708+0000 7f0515bf4740 1 -- 172.21.15.29:0/1144868083 wait complete. 2024-07-27T06:47:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.709+0000 7f0515bf4740 1 -- start start 2024-07-27T06:47:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.709+0000 7f0515bf4740 1 Processor -- start 2024-07-27T06:47:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.709+0000 7f0515bf4740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.709+0000 7f0515bf4740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a20f4d0 0x55d77a5f2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.709+0000 7f0515bf4740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a2301b0 0x55d77a5f2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.709+0000 7f0515bf4740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f05081b0460 con 0x55d77a20f4d0 2024-07-27T06:47:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.709+0000 7f0515bf4740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f05081afd90 con 0x55d77a2301b0 2024-07-27T06:47:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.709+0000 7f0515bf4740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f05081acfd0 con 0x55d77a2072f0 2024-07-27T06:47:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.709+0000 7f0515bf4740 10 client.0 _ll_register_callbacks cb 0x7f04f400c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.709+0000 7f0507fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a20f4d0 0x55d77a5f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.709+0000 7f0507fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a20f4d0 0x55d77a5f2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43414/0 (socket says 172.21.15.29:43414) 2024-07-27T06:47:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.709+0000 7f0507fff640 1 -- 172.21.15.29:0/413161067 learned_addr learned my addr 172.21.15.29:0/413161067 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f050d280640 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a2301b0 0x55d77a5f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f050d280640 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f0507fff640 1 -- 172.21.15.29:0/413161067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 msgr2=0x55d77a5f3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f0507fff640 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5f3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f0507fff640 1 -- 172.21.15.29:0/413161067 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a2301b0 msgr2=0x55d77a5f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f0507fff640 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a2301b0 0x55d77a5f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f0507fff640 1 -- 172.21.15.29:0/413161067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d77a5f2dd0 con 0x55d77a20f4d0 2024-07-27T06:47:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f050d280640 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5f3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f0507fff640 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a20f4d0 0x55d77a5f2100 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x55d77a548470 tx=0x7f04f40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5643ae7c91db8492 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f04e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:16.716 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:16 smithi029 ceph-mon[32610]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:47:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f04e7fff640 1 -- 172.21.15.29:0/413161067 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4005220 con 0x55d77a20f4d0 2024-07-27T06:47:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f04e7fff640 1 -- 172.21.15.29:0/413161067 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f04f40053c0 con 0x55d77a20f4d0 2024-07-27T06:47:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f0515bf4740 1 -- 172.21.15.29:0/413161067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f050005e070 con 0x55d77a20f4d0 2024-07-27T06:47:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f04e7fff640 1 -- 172.21.15.29:0/413161067 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f04f402b050 con 0x55d77a20f4d0 2024-07-27T06:47:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f0515bf4740 10 client.15570 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.710+0000 7f0515bf4740 1 -- 172.21.15.29:0/413161067 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d77a657a60 con 0x55d77a20f4d0 2024-07-27T06:47:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.711+0000 7f04e7fff640 1 -- 172.21.15.29:0/413161067 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f050005e070 con 0x55d77a20f4d0 2024-07-27T06:47:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.711+0000 7f04e7fff640 1 -- 172.21.15.29:0/413161067 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f04f4004830 con 0x55d77a20f4d0 2024-07-27T06:47:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.711+0000 7f0515bf4740 10 client.15570 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.711+0000 7f0515bf4740 10 client.15570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.711+0000 7f0515bf4740 20 client.15570 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.711+0000 7f0515bf4740 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d77a1c41c0 0x55d77a65a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.711+0000 7f0515bf4740 4 client.15570 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-27T06:47:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.711+0000 7f0515bf4740 1 -- 172.21.15.29:0/413161067 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d77a545da0 con 0x55d77a1c41c0 2024-07-27T06:47:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.711+0000 7f050d280640 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d77a1c41c0 0x55d77a65a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.711+0000 7f04e7fff640 10 client.15570 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.711+0000 7f050d280640 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d77a1c41c0 0x55d77a65a2a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7791d18c2e633b02 server_cookie=c0573ee63e6b3644 in_seq=0 out_seq=0 2024-07-27T06:47:16.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.725+0000 7f04e7fff640 1 -- 172.21.15.29:0/413161067 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d77a545da0 con 0x55d77a1c41c0 2024-07-27T06:47:16.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.725+0000 7f04e7fff640 10 client.15570 handle_command_reply: tid=0 2024-07-27T06:47:16.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.725+0000 7f0515bf4740 1 client.15570 shutdown 2024-07-27T06:47:16.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.726+0000 7f0515bf4740 1 -- 172.21.15.29:0/413161067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a20f4d0 msgr2=0x55d77a5f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:16.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.726+0000 7f0515bf4740 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a20f4d0 0x55d77a5f2100 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x55d77a548470 tx=0x7f04f40039c0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.726+0000 7f0515bf4740 1 -- 172.21.15.29:0/413161067 shutdown_connections 2024-07-27T06:47:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.726+0000 7f0515bf4740 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d77a2301b0 0x55d77a5f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.726+0000 7f0515bf4740 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d77a20f4d0 0x55d77a5f2100 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.726+0000 7f0515bf4740 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d77a1c41c0 0x55d77a65a2a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.726+0000 7f0515bf4740 1 --2- 172.21.15.29:0/413161067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d77a2072f0 0x55d77a5f3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.726+0000 7f04e7fff640 0 client.15570 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.726+0000 7f0515bf4740 1 -- 172.21.15.29:0/413161067 >> 172.21.15.29:0/413161067 conn(0x55d77a224300 msgr2=0x55d77a5e0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:16.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.726+0000 7f0515bf4740 1 -- 172.21.15.29:0/413161067 shutdown_connections 2024-07-27T06:47:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.726+0000 7f0515bf4740 1 -- 172.21.15.29:0/413161067 wait complete. 2024-07-27T06:47:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.726+0000 7f0515bf4740 20 client.15570 trim_cache size 0 max 16384 2024-07-27T06:47:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-07-27T06:47:16.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-07-27T06:47:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-07-27T06:47:16.731 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-07-27T06:47:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.727+0000 7f050ece3640 1 -- 172.21.15.29:0/928891400 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f04f80b3cb0 msgr2=0x7f04f80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.727+0000 7f050ece3640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f04f80b3cb0 0x7f04f80b61c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x55d77a5d1960 tx=0x7f04fc004a90 comp rx=0 tx=0).stop 2024-07-27T06:47:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.727+0000 7f050ece3640 1 -- 172.21.15.29:0/928891400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05081023f0 msgr2=0x7f0508199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.727+0000 7f050ece3640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05081023f0 0x7f0508199e50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00e9f0 tx=0x7f04fc00edf0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.727+0000 7f050ece3640 1 -- 172.21.15.29:0/928891400 shutdown_connections 2024-07-27T06:47:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.727+0000 7f050ece3640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f04f80b3cb0 0x7f04f80b61c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.727+0000 7f050ece3640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0508108670 0x7f050819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.727+0000 7f050ece3640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0508107d20 0x7f050819a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.727+0000 7f050ece3640 1 --2- 172.21.15.29:0/928891400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05081023f0 0x7f0508199e50 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:16.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.727+0000 7f050ece3640 1 -- 172.21.15.29:0/928891400 >> 172.21.15.29:0/928891400 conn(0x7f05080fddf0 msgr2=0x7f050806ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:16.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.727+0000 7f050ece3640 1 -- 172.21.15.29:0/928891400 shutdown_connections 2024-07-27T06:47:16.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:16.727+0000 7f050ece3640 1 -- 172.21.15.29:0/928891400 wait complete. 2024-07-27T06:47:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:16 smithi077 ceph-mon[37153]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:47:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:16 smithi047 ceph-mon[36835]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-27T06:47:17.629 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:17 smithi029 ceph-mon[32610]: pgmap v405: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 83 KiB/s wr, 41 op/s 2024-07-27T06:47:17.629 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:17 smithi029 ceph-mon[32610]: scrub summary: idle 2024-07-27T06:47:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:17 smithi077 ceph-mon[37153]: pgmap v405: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 83 KiB/s wr, 41 op/s 2024-07-27T06:47:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:17 smithi077 ceph-mon[37153]: scrub summary: idle 2024-07-27T06:47:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.887+0000 7f1140b37640 1 -- 172.21.15.29:0/3476244151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f113c075ad0 msgr2=0x7f113c10f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.887+0000 7f1140b37640 1 --2- 172.21.15.29:0/3476244151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f113c075ad0 0x7f113c10f640 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1130009900 tx=0x7f113002a410 comp rx=0 tx=0).stop 2024-07-27T06:47:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.887+0000 7f1140b37640 1 -- 172.21.15.29:0/3476244151 shutdown_connections 2024-07-27T06:47:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.887+0000 7f1140b37640 1 --2- 172.21.15.29:0/3476244151 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f113c10fb80 0x7f113c111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.887+0000 7f1140b37640 1 --2- 172.21.15.29:0/3476244151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f113c075ad0 0x7f113c10f640 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.887+0000 7f1140b37640 1 --2- 172.21.15.29:0/3476244151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c075180 0x7f113c075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.887+0000 7f1140b37640 1 -- 172.21.15.29:0/3476244151 >> 172.21.15.29:0/3476244151 conn(0x7f113c0ff030 msgr2=0x7f113c101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.887+0000 7f1140b37640 1 -- 172.21.15.29:0/3476244151 shutdown_connections 2024-07-27T06:47:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.887+0000 7f1140b37640 1 -- 172.21.15.29:0/3476244151 wait complete. 2024-07-27T06:47:17.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.888+0000 7f1140b37640 1 Processor -- start 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.888+0000 7f1140b37640 1 -- start start 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f1140b37640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f113c075180 0x7f113c19c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f1140b37640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c075ad0 0x7f113c19c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f1140b37640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f113c10fb80 0x7f113c1a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f1140b37640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f113c1156a0 con 0x7f113c075180 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f1140b37640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f113c115520 con 0x7f113c10fb80 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f1140b37640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f113c115820 con 0x7f113c075ad0 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f113a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f113c075180 0x7f113c19c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f113a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f113c075180 0x7f113c19c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43440/0 (socket says 172.21.15.29:43440) 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f113a59c640 1 -- 172.21.15.29:0/2553146570 learned_addr learned my addr 172.21.15.29:0/2553146570 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f1139d9b640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c075ad0 0x7f113c19c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f113ad9d640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f113c10fb80 0x7f113c1a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:17.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f113a59c640 1 -- 172.21.15.29:0/2553146570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c075ad0 msgr2=0x7f113c19c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f113a59c640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c075ad0 0x7f113c19c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f113a59c640 1 -- 172.21.15.29:0/2553146570 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f113c10fb80 msgr2=0x7f113c1a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f113a59c640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f113c10fb80 0x7f113c1a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f113a59c640 1 -- 172.21.15.29:0/2553146570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113c1a20a0 con 0x7f113c075180 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f113ad9d640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f113c10fb80 0x7f113c1a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f1139d9b640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c075ad0 0x7f113c19c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.889+0000 7f113a59c640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f113c075180 0x7f113c19c130 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f112c00d950 tx=0x7f112c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.890+0000 7f112b7fe640 1 -- 172.21.15.29:0/2553146570 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c017070 con 0x7f113c075180 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.890+0000 7f1140b37640 1 -- 172.21.15.29:0/2553146570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f113c1a2390 con 0x7f113c075180 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.890+0000 7f112b7fe640 1 -- 172.21.15.29:0/2553146570 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f112c009c30 con 0x7f113c075180 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.890+0000 7f112b7fe640 1 -- 172.21.15.29:0/2553146570 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c00f620 con 0x7f113c075180 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.890+0000 7f1140b37640 1 -- 172.21.15.29:0/2553146570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f113c1a8af0 con 0x7f113c075180 2024-07-27T06:47:17.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.891+0000 7f1140b37640 1 -- 172.21.15.29:0/2553146570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f113c075590 con 0x7f113c075180 2024-07-27T06:47:17.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.892+0000 7f112b7fe640 1 -- 172.21.15.29:0/2553146570 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f112c005030 con 0x7f113c075180 2024-07-27T06:47:17.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.893+0000 7f112b7fe640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f11240b3cb0 0x7f11240b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:17.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.893+0000 7f1139d9b640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f11240b3cb0 0x7f11240b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:17.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.893+0000 7f112b7fe640 1 -- 172.21.15.29:0/2553146570 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f112c0e3bd0 con 0x7f113c075180 2024-07-27T06:47:17.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.894+0000 7f1139d9b640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f11240b3cb0 0x7f11240b61c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1130009230 tx=0x7f1130000bc0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:17.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:17.896+0000 7f112b7fe640 1 -- 172.21.15.29:0/2553146570 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f112c0aaf50 con 0x7f113c075180 2024-07-27T06:47:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:17 smithi047 ceph-mon[36835]: pgmap v405: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 83 KiB/s wr, 41 op/s 2024-07-27T06:47:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:17 smithi047 ceph-mon[36835]: scrub summary: idle 2024-07-27T06:47:18.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.117+0000 7f1140b37640 1 -- 172.21.15.29:0/2553146570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f113c19d260 con 0x7f113c075180 2024-07-27T06:47:18.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.117+0000 7f112b7fe640 1 -- 172.21.15.29:0/2553146570 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11957 (secure 0 0 0) 0x7f112c009de0 con 0x7f113c075180 2024-07-27T06:47:18.119 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:47:18.119 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-07-27T06:46:50:421100+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:46:49.561737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24670,24658,24646,24617,24613]},"id":1}]} 2024-07-27T06:47:18.119 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-07-27T06:47:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.120+0000 7f1140b37640 1 -- 172.21.15.29:0/2553146570 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f11240b3cb0 msgr2=0x7f11240b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.120+0000 7f1140b37640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f11240b3cb0 0x7f11240b61c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1130009230 tx=0x7f1130000bc0 comp rx=0 tx=0).stop 2024-07-27T06:47:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.120+0000 7f1140b37640 1 -- 172.21.15.29:0/2553146570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f113c075180 msgr2=0x7f113c19c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.120+0000 7f1140b37640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f113c075180 0x7f113c19c130 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f112c00d950 tx=0x7f112c00de10 comp rx=0 tx=0).stop 2024-07-27T06:47:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.120+0000 7f1140b37640 1 -- 172.21.15.29:0/2553146570 shutdown_connections 2024-07-27T06:47:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.120+0000 7f1140b37640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f113c10fb80 0x7f113c1a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.120+0000 7f1140b37640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f11240b3cb0 0x7f11240b61c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.120+0000 7f1140b37640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f113c075ad0 0x7f113c19c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.120+0000 7f1140b37640 1 --2- 172.21.15.29:0/2553146570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f113c075180 0x7f113c19c130 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.120+0000 7f1140b37640 1 -- 172.21.15.29:0/2553146570 >> 172.21.15.29:0/2553146570 conn(0x7f113c0ff030 msgr2=0x7f113c109700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:18.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.120+0000 7f1140b37640 1 -- 172.21.15.29:0/2553146570 shutdown_connections 2024-07-27T06:47:18.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:18.120+0000 7f1140b37640 1 -- 172.21.15.29:0/2553146570 wait complete. 2024-07-27T06:47:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:18 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2553146570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:47:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:18 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2553146570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:47:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:18 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2553146570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:47:18.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:47:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:19.686 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:19 smithi029 ceph-mon[32610]: pgmap v406: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 KiB/s wr, 37 op/s 2024-07-27T06:47:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:19 smithi077 ceph-mon[37153]: pgmap v406: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 KiB/s wr, 37 op/s 2024-07-27T06:47:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:19 smithi047 ceph-mon[36835]: pgmap v406: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 KiB/s wr, 37 op/s 2024-07-27T06:47:21.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.094+0000 7f9bfd771640 1 -- 172.21.15.29:0/3648013049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8102490 msgr2=0x7f9bf81028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:21.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.094+0000 7f9bfd771640 1 --2- 172.21.15.29:0/3648013049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8102490 0x7f9bf81028a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9be0009900 tx=0x7f9be002a410 comp rx=0 tx=0).stop 2024-07-27T06:47:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.094+0000 7f9bfd771640 1 -- 172.21.15.29:0/3648013049 shutdown_connections 2024-07-27T06:47:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.094+0000 7f9bfd771640 1 --2- 172.21.15.29:0/3648013049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bf8108720 0x7f9bf810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.094+0000 7f9bfd771640 1 --2- 172.21.15.29:0/3648013049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf8107dd0 0x7f9bf81081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.094+0000 7f9bfd771640 1 --2- 172.21.15.29:0/3648013049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8102490 0x7f9bf81028a0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.094+0000 7f9bfd771640 1 -- 172.21.15.29:0/3648013049 >> 172.21.15.29:0/3648013049 conn(0x7f9bf8079160 msgr2=0x7f9bf8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.094+0000 7f9bfd771640 1 -- 172.21.15.29:0/3648013049 shutdown_connections 2024-07-27T06:47:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.094+0000 7f9bfd771640 1 -- 172.21.15.29:0/3648013049 wait complete. 2024-07-27T06:47:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.095+0000 7f9bfd771640 1 Processor -- start 2024-07-27T06:47:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.095+0000 7f9bfd771640 1 -- start start 2024-07-27T06:47:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bfd771640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bf8102490 0x7f9bf819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bfd771640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf8107dd0 0x7f9bf819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:21.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bfd771640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8108720 0x7f9bf819b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bfd771640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf81123a0 con 0x7f9bf8108720 2024-07-27T06:47:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bfd771640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf8112220 con 0x7f9bf8102490 2024-07-27T06:47:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bfd771640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf8112520 con 0x7f9bf8107dd0 2024-07-27T06:47:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bf77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8108720 0x7f9bf819b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bf77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8108720 0x7f9bf819b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42374/0 (socket says 172.21.15.29:42374) 2024-07-27T06:47:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bf77fe640 1 -- 172.21.15.29:0/1363648929 learned_addr learned my addr 172.21.15.29:0/1363648929 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bf67fc640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf8107dd0 0x7f9bf819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bf6ffd640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bf8102490 0x7f9bf819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bf77fe640 1 -- 172.21.15.29:0/1363648929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf8107dd0 msgr2=0x7f9bf819b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bf77fe640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf8107dd0 0x7f9bf819b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bf77fe640 1 -- 172.21.15.29:0/1363648929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bf8102490 msgr2=0x7f9bf819ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:21.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bf77fe640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bf8102490 0x7f9bf819ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bf77fe640 1 -- 172.21.15.29:0/1363648929 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf81a6010 con 0x7f9bf8108720 2024-07-27T06:47:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bf67fc640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf8107dd0 0x7f9bf819b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9bf77fe640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8108720 0x7f9bf819b7d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000cad0 tx=0x7f9bf000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.096+0000 7f9beffff640 1 -- 172.21.15.29:0/1363648929 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf000ae20 con 0x7f9bf8108720 2024-07-27T06:47:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.097+0000 7f9beffff640 1 -- 172.21.15.29:0/1363648929 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bf0005020 con 0x7f9bf8108720 2024-07-27T06:47:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.097+0000 7f9bfd771640 1 -- 172.21.15.29:0/1363648929 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf81a6300 con 0x7f9bf8108720 2024-07-27T06:47:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.097+0000 7f9beffff640 1 -- 172.21.15.29:0/1363648929 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf000f6b0 con 0x7f9bf8108720 2024-07-27T06:47:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.097+0000 7f9bfd771640 1 -- 172.21.15.29:0/1363648929 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf81a67f0 con 0x7f9bf8108720 2024-07-27T06:47:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.098+0000 7f9bfd771640 1 -- 172.21.15.29:0/1363648929 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bf81028a0 con 0x7f9bf8108720 2024-07-27T06:47:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.099+0000 7f9beffff640 1 -- 172.21.15.29:0/1363648929 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f9bf00051d0 con 0x7f9bf8108720 2024-07-27T06:47:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.100+0000 7f9beffff640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9be40b3cb0 0x7f9be40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.101+0000 7f9bf6ffd640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9be40b3cb0 0x7f9be40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:21.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.101+0000 7f9beffff640 1 -- 172.21.15.29:0/1363648929 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f9bf00e43c0 con 0x7f9bf8108720 2024-07-27T06:47:21.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.101+0000 7f9bf6ffd640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9be40b3cb0 0x7f9be40b61c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9be0000c00 tx=0x7f9be0031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:21.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.104+0000 7f9beffff640 1 -- 172.21.15.29:0/1363648929 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9bf0014030 con 0x7f9bf8108720 2024-07-27T06:47:21.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.325+0000 7f9bfd771640 1 -- 172.21.15.29:0/1363648929 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9bf806acc0 con 0x7f9bf8108720 2024-07-27T06:47:21.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.326+0000 7f9beffff640 1 -- 172.21.15.29:0/1363648929 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11957 (secure 0 0 0) 0x7f9bf00ab7a0 con 0x7f9bf8108720 2024-07-27T06:47:21.328 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:47:21.328 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-07-27T06:46:50:421100+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:46:49.561737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24670,24658,24646,24617,24613]},"id":1}]} 2024-07-27T06:47:21.328 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-07-27T06:47:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.328+0000 7f9bfd771640 1 -- 172.21.15.29:0/1363648929 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9be40b3cb0 msgr2=0x7f9be40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.328+0000 7f9bfd771640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9be40b3cb0 0x7f9be40b61c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9be0000c00 tx=0x7f9be0031400 comp rx=0 tx=0).stop 2024-07-27T06:47:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.328+0000 7f9bfd771640 1 -- 172.21.15.29:0/1363648929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8108720 msgr2=0x7f9bf819b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.328+0000 7f9bfd771640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8108720 0x7f9bf819b7d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000cad0 tx=0x7f9bf000cf90 comp rx=0 tx=0).stop 2024-07-27T06:47:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.328+0000 7f9bfd771640 1 -- 172.21.15.29:0/1363648929 shutdown_connections 2024-07-27T06:47:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.328+0000 7f9bfd771640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf8108720 0x7f9bf819b7d0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.328+0000 7f9bfd771640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f9be40b3cb0 0x7f9be40b61c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.328+0000 7f9bfd771640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bf8107dd0 0x7f9bf819b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.328+0000 7f9bfd771640 1 --2- 172.21.15.29:0/1363648929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9bf8102490 0x7f9bf819ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.328+0000 7f9bfd771640 1 -- 172.21.15.29:0/1363648929 >> 172.21.15.29:0/1363648929 conn(0x7f9bf8079160 msgr2=0x7f9bf8100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.328+0000 7f9bfd771640 1 -- 172.21.15.29:0/1363648929 shutdown_connections 2024-07-27T06:47:21.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:21.328+0000 7f9bfd771640 1 -- 172.21.15.29:0/1363648929 wait complete. 2024-07-27T06:47:21.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:21 smithi029 ceph-mon[32610]: pgmap v407: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-27T06:47:21.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:21 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1363648929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:47:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:21 smithi077 ceph-mon[37153]: pgmap v407: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-27T06:47:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:21 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1363648929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:47:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:21 smithi047 ceph-mon[36835]: pgmap v407: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-27T06:47:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:21 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1363648929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:47:22.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph osd dump --format=json 2024-07-27T06:47:22.462 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:23 smithi077 ceph-mon[37153]: pgmap v408: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.9 KiB/s wr, 14 op/s 2024-07-27T06:47:23.868 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:23 smithi029 ceph-mon[32610]: pgmap v408: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.9 KiB/s wr, 14 op/s 2024-07-27T06:47:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:23 smithi047 ceph-mon[36835]: pgmap v408: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.9 KiB/s wr, 14 op/s 2024-07-27T06:47:24.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.444+0000 7efd8437a640 1 -- 172.21.15.29:0/2722757780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd7c102490 msgr2=0x7efd7c1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:24.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.444+0000 7efd8437a640 1 --2- 172.21.15.29:0/2722757780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd7c102490 0x7efd7c1028a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7efd78009900 tx=0x7efd7802a410 comp rx=0 tx=0).stop 2024-07-27T06:47:24.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.444+0000 7efd8437a640 1 -- 172.21.15.29:0/2722757780 shutdown_connections 2024-07-27T06:47:24.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.444+0000 7efd8437a640 1 --2- 172.21.15.29:0/2722757780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd7c108720 0x7efd7c10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.444+0000 7efd8437a640 1 --2- 172.21.15.29:0/2722757780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd7c107dd0 0x7efd7c1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.444+0000 7efd8437a640 1 --2- 172.21.15.29:0/2722757780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd7c102490 0x7efd7c1028a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.444+0000 7efd8437a640 1 -- 172.21.15.29:0/2722757780 >> 172.21.15.29:0/2722757780 conn(0x7efd7c079160 msgr2=0x7efd7c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.444+0000 7efd8437a640 1 -- 172.21.15.29:0/2722757780 shutdown_connections 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.444+0000 7efd8437a640 1 -- 172.21.15.29:0/2722757780 wait complete. 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.445+0000 7efd8437a640 1 Processor -- start 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.445+0000 7efd8437a640 1 -- start start 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.445+0000 7efd8437a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd7c102490 0x7efd7c10f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.445+0000 7efd8437a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd7c107dd0 0x7efd7c10ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.445+0000 7efd8437a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd7c108720 0x7efd7c110460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.445+0000 7efd8437a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c1134b0 con 0x7efd7c102490 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.445+0000 7efd8437a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c113330 con 0x7efd7c107dd0 2024-07-27T06:47:24.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.445+0000 7efd8437a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efd7c113630 con 0x7efd7c108720 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.445+0000 7efd82116640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd7c102490 0x7efd7c10f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.445+0000 7efd82116640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd7c102490 0x7efd7c10f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42396/0 (socket says 172.21.15.29:42396) 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.445+0000 7efd82116640 1 -- 172.21.15.29:0/2227926007 learned_addr learned my addr 172.21.15.29:0/2227926007 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd82917640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd7c108720 0x7efd7c110460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd81915640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd7c107dd0 0x7efd7c10ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd82116640 1 -- 172.21.15.29:0/2227926007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd7c108720 msgr2=0x7efd7c110460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd82116640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd7c108720 0x7efd7c110460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd82116640 1 -- 172.21.15.29:0/2227926007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd7c107dd0 msgr2=0x7efd7c10ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd82116640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd7c107dd0 0x7efd7c10ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd82116640 1 -- 172.21.15.29:0/2227926007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd7c1a8070 con 0x7efd7c102490 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd81915640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd7c107dd0 0x7efd7c10ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd82116640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd7c102490 0x7efd7c10f9e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7efd78005940 tx=0x7efd780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd737fe640 1 -- 172.21.15.29:0/2227926007 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7803f030 con 0x7efd7c102490 2024-07-27T06:47:24.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd737fe640 1 -- 172.21.15.29:0/2227926007 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd78003cb0 con 0x7efd7c102490 2024-07-27T06:47:24.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd737fe640 1 -- 172.21.15.29:0/2227926007 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd78035070 con 0x7efd7c102490 2024-07-27T06:47:24.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd8437a640 1 -- 172.21.15.29:0/2227926007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd7c1a8300 con 0x7efd7c102490 2024-07-27T06:47:24.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.446+0000 7efd8437a640 1 -- 172.21.15.29:0/2227926007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd7c1a87e0 con 0x7efd7c102490 2024-07-27T06:47:24.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.447+0000 7efd8437a640 1 -- 172.21.15.29:0/2227926007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd7c1028a0 con 0x7efd7c102490 2024-07-27T06:47:24.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.449+0000 7efd737fe640 1 -- 172.21.15.29:0/2227926007 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7efd78042030 con 0x7efd7c102490 2024-07-27T06:47:24.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.450+0000 7efd737fe640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efd680b3cb0 0x7efd680b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:24.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.450+0000 7efd81915640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efd680b3cb0 0x7efd680b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:24.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.450+0000 7efd737fe640 1 -- 172.21.15.29:0/2227926007 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7efd78104a60 con 0x7efd7c102490 2024-07-27T06:47:24.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.451+0000 7efd81915640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efd680b3cb0 0x7efd680b61c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efd7c1006f0 tx=0x7efd6c00a5c0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:24.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.453+0000 7efd737fe640 1 -- 172.21.15.29:0/2227926007 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7efd780cbe40 con 0x7efd7c102490 2024-07-27T06:47:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.639+0000 7efd8437a640 1 -- 172.21.15.29:0/2227926007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd7c110ce0 con 0x7efd7c102490 2024-07-27T06:47:24.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.641+0000 7efd737fe640 1 -- 172.21.15.29:0/2227926007 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+23026 (secure 0 0 0) 0x7efd780cb740 con 0x7efd7c102490 2024-07-27T06:47:24.643 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:47:24.643 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"fsid":"0a8a0a4c-4be2-11ef-bcc7-c7b262605968","created":"2024-07-27T06:33:57.535327+0000","modified":"2024-07-27T06:45:09.477077+0000","last_up_change":"2024-07-27T06:42:05.757156+0000","last_in_change":"2024-07-27T06:41:46.423732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-27T06:38:21.606527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-27T06:43:07.755490+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-27T06:43:11.658694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"06833d24-2e1a-4199-9338-7d6883d1bf9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6803","nonce":386121641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6805","nonce":386121641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6809","nonce":386121641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":386121641},{"type":"v1","addr":"172.21.15.29:6807","nonce":386121641}]},"public_addr":"172.21.15.29:6803/386121641","cluster_addr":"172.21.15.29:6805/386121641","heartbeat_back_addr":"172.21.15.29:6809/386121641","heartbeat_front_addr":"172.21.15.29:6807/386121641","state":["exists","up"]},{"osd":1,"uuid":"906aba19-80dc-41b5-953c-af6dca604b05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6801","nonce":2795953749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6803","nonce":2795953749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6807","nonce":2795953749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":2795953749},{"type":"v1","addr":"172.21.15.47:6805","nonce":2795953749}]},"public_addr":"172.21.15.47:6801/2795953749","cluster_addr":"172.21.15.47:6803/2795953749","heartbeat_back_addr":"172.21.15.47:6807/2795953749","heartbeat_front_addr":"172.21.15.47:6805/2795953749","state":["exists","up"]},{"osd":2,"uuid":"194ae5a5-aebf-4cff-9b99-585a3073a269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6801","nonce":2202565078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6803","nonce":2202565078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6807","nonce":2202565078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":2202565078},{"type":"v1","addr":"172.21.15.77:6805","nonce":2202565078}]},"public_addr":"172.21.15.77:6801/2202565078","cluster_addr":"172.21.15.77:6803/2202565078","heartbeat_back_addr":"172.21.15.77:6807/2202565078","heartbeat_front_addr":"172.21.15.77:6805/2202565078","state":["exists","up"]},{"osd":3,"uuid":"3c22ce9c-963e-4f72-86e5-360e781526db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6811","nonce":3757476312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6813","nonce":3757476312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6817","nonce":3757476312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3757476312},{"type":"v1","addr":"172.21.15.29:6815","nonce":3757476312}]},"public_addr":"172.21.15.29:6811/3757476312","cluster_addr":"172.21.15.29:6813/3757476312","heartbeat_back_addr":"172.21.15.29:6817/3757476312","heartbeat_front_addr":"172.21.15.29:6815/3757476312","state":["exists","up"]},{"osd":4,"uuid":"cd4b585f-e7a0-4d82-873d-3a9269ebac3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6809","nonce":1379363720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6811","nonce":1379363720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6815","nonce":1379363720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":1379363720},{"type":"v1","addr":"172.21.15.47:6813","nonce":1379363720}]},"public_addr":"172.21.15.47:6809/1379363720","cluster_addr":"172.21.15.47:6811/1379363720","heartbeat_back_addr":"172.21.15.47:6815/1379363720","heartbeat_front_addr":"172.21.15.47:6813/1379363720","state":["exists","up"]},{"osd":5,"uuid":"40a3eb14-dfaa-4523-8c28-d74c19293c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6809","nonce":1338438780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6811","nonce":1338438780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6815","nonce":1338438780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1338438780},{"type":"v1","addr":"172.21.15.77:6813","nonce":1338438780}]},"public_addr":"172.21.15.77:6809/1338438780","cluster_addr":"172.21.15.77:6811/1338438780","heartbeat_back_addr":"172.21.15.77:6815/1338438780","heartbeat_front_addr":"172.21.15.77:6813/1338438780","state":["exists","up"]},{"osd":6,"uuid":"91e85be2-10d1-4209-aa33-17da8f8bd8bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6819","nonce":2679402148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6821","nonce":2679402148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6825","nonce":2679402148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":2679402148},{"type":"v1","addr":"172.21.15.29:6823","nonce":2679402148}]},"public_addr":"172.21.15.29:6819/2679402148","cluster_addr":"172.21.15.29:6821/2679402148","heartbeat_back_addr":"172.21.15.29:6825/2679402148","heartbeat_front_addr":"172.21.15.29:6823/2679402148","state":["exists","up"]},{"osd":7,"uuid":"c655b450-0cc4-45d4-95ea-93a57fe8e4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6817","nonce":2439195589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6819","nonce":2439195589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6823","nonce":2439195589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":2439195589},{"type":"v1","addr":"172.21.15.47:6821","nonce":2439195589}]},"public_addr":"172.21.15.47:6817/2439195589","cluster_addr":"172.21.15.47:6819/2439195589","heartbeat_back_addr":"172.21.15.47:6823/2439195589","heartbeat_front_addr":"172.21.15.47:6821/2439195589","state":["exists","up"]},{"osd":8,"uuid":"a7a95b51-f32c-433e-a197-feb6d47b6c71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6817","nonce":1422290905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6819","nonce":1422290905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6823","nonce":1422290905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":1422290905},{"type":"v1","addr":"172.21.15.77:6821","nonce":1422290905}]},"public_addr":"172.21.15.77:6817/1422290905","cluster_addr":"172.21.15.77:6819/1422290905","heartbeat_back_addr":"172.21.15.77:6823/1422290905","heartbeat_front_addr":"172.21.15.77:6821/1422290905","state":["exists","up"]},{"osd":9,"uuid":"f352242b-0d2e-41b3-9c83-436cf432b0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6827","nonce":1307316384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6829","nonce":1307316384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6833","nonce":1307316384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":1307316384},{"type":"v1","addr":"172.21.15.29:6831","nonce":1307316384}]},"public_addr":"172.21.15.29:6827/1307316384","cluster_addr":"172.21.15.29:6829/1307316384","heartbeat_back_addr":"172.21.15.29:6833/1307316384","heartbeat_front_addr":"172.21.15.29:6831/1307316384","state":["exists","up"]},{"osd":10,"uuid":"4148c2df-c3e9-4a47-88f7-36cf3eab8d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6825","nonce":2133726850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6827","nonce":2133726850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6831","nonce":2133726850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":2133726850},{"type":"v1","addr":"172.21.15.47:6829","nonce":2133726850}]},"public_addr":"172.21.15.47:6825/2133726850","cluster_addr":"172.21.15.47:6827/2133726850","heartbeat_back_addr":"172.21.15.47:6831/2133726850","heartbeat_front_addr":"172.21.15.47:6829/2133726850","state":["exists","up"]},{"osd":11,"uuid":"5e8b1da3-5f65-4fe9-8722-b35921cbd8c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6825","nonce":3228299327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6827","nonce":3228299327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6831","nonce":3228299327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3228299327},{"type":"v1","addr":"172.21.15.77:6829","nonce":3228299327}]},"public_addr":"172.21.15.77:6825/3228299327","cluster_addr":"172.21.15.77:6827/3228299327","heartbeat_back_addr":"172.21.15.77:6831/3228299327","heartbeat_front_addr":"172.21.15.77:6829/3228299327","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:32.525206+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:37:56.489640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:18.887845+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:38:43.727678+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:08.755553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:32.923461+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:39:59.298439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:23.944848+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:40:48.883050+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:14.566310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:41:39.557809+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-27T06:42:03.813903+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7},{"from":5,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2378279024":"2024-07-28T06:36:07.527503+0000","172.21.15.29:6801/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/3219967326":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3527133965":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2262661620":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6800/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/3156863587":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2424241521":"2024-07-28T06:34:34.443023+0000","172.21.15.29:0/3881857930":"2024-07-28T06:34:58.250198+0000","172.21.15.29:0/398508875":"2024-07-28T06:34:34.443023+0000","172.21.15.29:6801/2720999695":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6800/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2679856281":"2024-07-28T06:34:58.250198+0000","172.21.15.29:6801/2384940058":"2024-07-28T06:36:07.527503+0000","172.21.15.29:0/2708687265":"2024-07-28T06:34:58.250198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-27T06:47:24.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.643+0000 7efd8437a640 1 -- 172.21.15.29:0/2227926007 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efd680b3cb0 msgr2=0x7efd680b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:24.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.643+0000 7efd8437a640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efd680b3cb0 0x7efd680b61c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efd7c1006f0 tx=0x7efd6c00a5c0 comp rx=0 tx=0).stop 2024-07-27T06:47:24.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.643+0000 7efd8437a640 1 -- 172.21.15.29:0/2227926007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd7c102490 msgr2=0x7efd7c10f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:24.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.643+0000 7efd8437a640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd7c102490 0x7efd7c10f9e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7efd78005940 tx=0x7efd780042f0 comp rx=0 tx=0).stop 2024-07-27T06:47:24.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.643+0000 7efd8437a640 1 -- 172.21.15.29:0/2227926007 shutdown_connections 2024-07-27T06:47:24.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.643+0000 7efd8437a640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efd680b3cb0 0x7efd680b61c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:24.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.643+0000 7efd8437a640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd7c108720 0x7efd7c110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:24.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.643+0000 7efd8437a640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd7c107dd0 0x7efd7c10ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:24.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.643+0000 7efd8437a640 1 --2- 172.21.15.29:0/2227926007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd7c102490 0x7efd7c10f9e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:24.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.643+0000 7efd8437a640 1 -- 172.21.15.29:0/2227926007 >> 172.21.15.29:0/2227926007 conn(0x7efd7c079160 msgr2=0x7efd7c100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:24.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.643+0000 7efd8437a640 1 -- 172.21.15.29:0/2227926007 shutdown_connections 2024-07-27T06:47:24.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:24.643+0000 7efd8437a640 1 -- 172.21.15.29:0/2227926007 wait complete. 2024-07-27T06:47:25.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-07-27T06:47:25.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:25 smithi029 ceph-mon[32610]: pgmap v409: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.9 KiB/s wr, 15 op/s 2024-07-27T06:47:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:25 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2227926007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:47:25.717 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:25 smithi077 ceph-mon[37153]: pgmap v409: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.9 KiB/s wr, 15 op/s 2024-07-27T06:47:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:25 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2227926007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:47:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:25 smithi047 ceph-mon[36835]: pgmap v409: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.9 KiB/s wr, 15 op/s 2024-07-27T06:47:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:25 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2227926007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-27T06:47:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:26 smithi077 ceph-mon[37153]: pgmap v410: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 13 op/s 2024-07-27T06:47:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:47:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:47:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:47:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:47:26.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:26 smithi029 ceph-mon[32610]: pgmap v410: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 13 op/s 2024-07-27T06:47:26.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:47:26.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:47:26.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:47:26.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:47:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:26 smithi047 ceph-mon[36835]: pgmap v410: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 KiB/s wr, 13 op/s 2024-07-27T06:47:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:47:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:47:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:47:26.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:47:28.258 INFO:teuthology.orchestra.run.smithi029.stdout:{"run_dir":"/var/run/ceph"} 2024-07-27T06:47:28.259 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-07-27T06:47:28.259 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 5532 2024-07-27T06:47:28.259 DEBUG:tasks.quiescer:Yielding 2024-07-27T06:47:28.259 INFO:teuthology.run_tasks:Running task workunit... 2024-07-27T06:47:28.274 INFO:tasks.workunit:Pulling workunits from ref e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:47:28.274 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-07-27T06:47:28.274 INFO:tasks.workunit:timeout=3h 2024-07-27T06:47:28.274 INFO:tasks.workunit:cleanup=True 2024-07-27T06:47:28.275 DEBUG:teuthology.orchestra.run.smithi029:> stat -- /home/ubuntu/cephtest/mnt.0 2024-07-27T06:47:28.275 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:47:28.294 INFO:teuthology.orchestra.run.smithi029.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-07-27T06:47:28.294 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-07-27T06:47:28.294 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-07-27T06:47:28.294 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-27T06:47:28.294 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-07-27T06:47:28.295 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-07-27 06:43:50.376576263 +0000 2024-07-27T06:47:28.295 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-07-27 06:43:50.376576263 +0000 2024-07-27T06:47:28.295 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-07-27 06:46:48.837385683 +0000 2024-07-27T06:47:28.295 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-07-27 06:43:50.376576263 +0000 2024-07-27T06:47:28.295 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-07-27T06:47:28.295 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-07-27T06:47:28.370 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-27T06:47:28.426 INFO:tasks.workunit.client.0.smithi029.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-07-27T06:47:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:28.934 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:28 smithi029 ceph-mon[32610]: pgmap v411: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 15 op/s 2024-07-27T06:47:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:28 smithi077 ceph-mon[37153]: pgmap v411: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 15 op/s 2024-07-27T06:47:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:28 smithi047 ceph-mon[36835]: pgmap v411: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 15 op/s 2024-07-27T06:47:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.845+0000 7f05b10e3640 1 -- 172.21.15.29:0/3051149645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05ac104370 msgr2=0x7f05ac104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.845+0000 7f05b10e3640 1 --2- 172.21.15.29:0/3051149645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05ac104370 0x7f05ac104780 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f059c009900 tx=0x7f059c02a410 comp rx=0 tx=0).stop 2024-07-27T06:47:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.845+0000 7f05b10e3640 1 -- 172.21.15.29:0/3051149645 shutdown_connections 2024-07-27T06:47:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.845+0000 7f05b10e3640 1 --2- 172.21.15.29:0/3051149645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05ac10d7d0 0x7f05ac10fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.845+0000 7f05b10e3640 1 --2- 172.21.15.29:0/3051149645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05ac104cc0 0x7f05ac10d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.845+0000 7f05b10e3640 1 --2- 172.21.15.29:0/3051149645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05ac104370 0x7f05ac104780 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.845+0000 7f05b10e3640 1 -- 172.21.15.29:0/3051149645 >> 172.21.15.29:0/3051149645 conn(0x7f05ac0fdde0 msgr2=0x7f05ac100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.845+0000 7f05b10e3640 1 -- 172.21.15.29:0/3051149645 shutdown_connections 2024-07-27T06:47:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.845+0000 7f05b10e3640 1 -- 172.21.15.29:0/3051149645 wait complete. 2024-07-27T06:47:29.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.846+0000 7f05b10e3640 1 Processor -- start 2024-07-27T06:47:29.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.846+0000 7f05b10e3640 1 -- start start 2024-07-27T06:47:29.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05b10e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05ac104370 0x7f05ac199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05b10e3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05ac104cc0 0x7f05ac19a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05b10e3640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05ac10d7d0 0x7f05ac19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05b10e3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac113460 con 0x7f05ac104370 2024-07-27T06:47:29.850 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05b10e3640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac1132e0 con 0x7f05ac10d7d0 2024-07-27T06:47:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05b10e3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac1135e0 con 0x7f05ac104cc0 2024-07-27T06:47:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05ac104370 0x7f05ac199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05ac104370 0x7f05ac199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45554/0 (socket says 172.21.15.29:45554) 2024-07-27T06:47:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05aad9d640 1 -- 172.21.15.29:0/657693229 learned_addr learned my addr 172.21.15.29:0/657693229 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05aa59c640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05ac104cc0 0x7f05ac19a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05ab59e640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05ac10d7d0 0x7f05ac19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05aad9d640 1 -- 172.21.15.29:0/657693229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05ac104cc0 msgr2=0x7f05ac19a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05aad9d640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05ac104cc0 0x7f05ac19a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05aad9d640 1 -- 172.21.15.29:0/657693229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05ac10d7d0 msgr2=0x7f05ac19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05aad9d640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05ac10d7d0 0x7f05ac19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.847+0000 7f05aad9d640 1 -- 172.21.15.29:0/657693229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ac19fe70 con 0x7f05ac104370 2024-07-27T06:47:29.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.848+0000 7f05aad9d640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05ac104370 0x7f05ac199f40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f059c005940 tx=0x7f059c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:29.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.848+0000 7f0593fff640 1 -- 172.21.15.29:0/657693229 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c03f030 con 0x7f05ac104370 2024-07-27T06:47:29.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.848+0000 7f0593fff640 1 -- 172.21.15.29:0/657693229 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f059c0044c0 con 0x7f05ac104370 2024-07-27T06:47:29.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.848+0000 7f05b10e3640 1 -- 172.21.15.29:0/657693229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05ac1a0100 con 0x7f05ac104370 2024-07-27T06:47:29.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.848+0000 7f0593fff640 1 -- 172.21.15.29:0/657693229 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c035070 con 0x7f05ac104370 2024-07-27T06:47:29.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.848+0000 7f05b10e3640 1 -- 172.21.15.29:0/657693229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05ac1a68f0 con 0x7f05ac104370 2024-07-27T06:47:29.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.851+0000 7f05b7ff5740 1 Processor -- start 2024-07-27T06:47:29.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.851+0000 7f05b7ff5740 1 -- start start 2024-07-27T06:47:29.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.851+0000 7f0593fff640 1 -- 172.21.15.29:0/657693229 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f059c042030 con 0x7f05ac104370 2024-07-27T06:47:29.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.851+0000 7f05b7ff5740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c4277fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.851+0000 7f05b7ff5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42780220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.851+0000 7f05b7ff5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c42780760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.851+0000 7f05b7ff5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561c426f53b0 con 0x561c423b42f0 2024-07-27T06:47:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.851+0000 7f05b7ff5740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x561c426e48b0 con 0x561c423dd1b0 2024-07-27T06:47:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.851+0000 7f05b7ff5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561c426e88f0 con 0x561c423bc4d0 2024-07-27T06:47:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.851+0000 7f05aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42780220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f05ab59e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c4277fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f05aa59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c42780760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f05aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42780220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45568/0 (socket says 172.21.15.29:45568) 2024-07-27T06:47:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f05aad9d640 1 -- 172.21.15.29:0/3223427695 learned_addr learned my addr 172.21.15.29:0/3223427695 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f05aad9d640 1 -- 172.21.15.29:0/3223427695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 msgr2=0x561c42780760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f05aad9d640 1 --2- 172.21.15.29:0/3223427695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c42780760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f05aad9d640 1 -- 172.21.15.29:0/3223427695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 msgr2=0x561c4277fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f05aad9d640 1 --2- 172.21.15.29:0/3223427695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c4277fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f05aad9d640 1 -- 172.21.15.29:0/3223427695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c4268b9a0 con 0x561c423b42f0 2024-07-27T06:47:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f0593fff640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f05940b3cb0 0x7f05940b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f05aad9d640 1 --2- 172.21.15.29:0/3223427695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42780220 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f059c101b20 tx=0x7f059c10c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8010512a5e42430b server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f05ab59e640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f05940b3cb0 0x7f05940b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f0593fff640 1 -- 172.21.15.29:0/657693229 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f059c104310 con 0x7f05ac104370 2024-07-27T06:47:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.852+0000 7f05917fa640 1 -- 172.21.15.29:0/3223427695 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c0c6eb0 con 0x561c423b42f0 2024-07-27T06:47:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.853+0000 7f05917fa640 1 -- 172.21.15.29:0/3223427695 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f059c0cab00 con 0x561c423b42f0 2024-07-27T06:47:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.853+0000 7f05917fa640 1 -- 172.21.15.29:0/3223427695 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c0ca760 con 0x561c423b42f0 2024-07-27T06:47:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.854+0000 7f05ab59e640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f05940b3cb0 0x7f05940b61c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f05a4005fd0 tx=0x7f05a4009500 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.854+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3223427695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 msgr2=0x561c42780220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.854+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3223427695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42780220 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f059c101b20 tx=0x7f059c10c640 comp rx=0 tx=0).stop 2024-07-27T06:47:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.854+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3223427695 shutdown_connections 2024-07-27T06:47:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.854+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3223427695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c42780760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.854+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3223427695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42780220 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.854+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3223427695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c4277fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.854+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3223427695 >> 172.21.15.29:0/3223427695 conn(0x561c423d1300 msgr2=0x561c4277dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:29.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.854+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3223427695 shutdown_connections 2024-07-27T06:47:29.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.854+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3223427695 wait complete. 2024-07-27T06:47:29.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.855+0000 7f05b7ff5740 1 -- start start 2024-07-27T06:47:29.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.855+0000 7f05b7ff5740 1 Processor -- start 2024-07-27T06:47:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.855+0000 7f05b7ff5740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423b42f0 0x561c4278fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.855+0000 7f05b7ff5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423bc4d0 0x561c427900b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.855+0000 7f05b7ff5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423dd1b0 0x561c4279b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05b7ff5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561c4261eab0 con 0x561c423bc4d0 2024-07-27T06:47:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05b7ff5740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x561c4266ccf0 con 0x561c423b42f0 2024-07-27T06:47:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05b7ff5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561c42792600 con 0x561c423dd1b0 2024-07-27T06:47:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05b7ff5740 10 client.0 _ll_register_callbacks cb 0x561c426de9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05aa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423bc4d0 0x561c427900b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05aa59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423bc4d0 0x561c427900b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45570/0 (socket says 172.21.15.29:45570) 2024-07-27T06:47:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05aa59c640 1 -- 172.21.15.29:0/3726644386 learned_addr learned my addr 172.21.15.29:0/3726644386 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05ab59e640 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423dd1b0 0x561c4279b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05aad9d640 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423b42f0 0x561c4278fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05aa59c640 1 -- 172.21.15.29:0/3726644386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423dd1b0 msgr2=0x561c4279b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05aa59c640 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423dd1b0 0x561c4279b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05aa59c640 1 -- 172.21.15.29:0/3726644386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423b42f0 msgr2=0x561c4278fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05aa59c640 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423b42f0 0x561c4278fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05aa59c640 1 -- 172.21.15.29:0/3726644386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c42448790 con 0x561c423bc4d0 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f05aa59c640 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423bc4d0 0x561c427900b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f05a00126e0 tx=0x7f05a0012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2293c2389b19b499 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.856+0000 7f058eff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f058eff5640 1 -- 172.21.15.29:0/3726644386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a0012190 con 0x561c423bc4d0 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f058eff5640 1 -- 172.21.15.29:0/3726644386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f05a0004020 con 0x561c423bc4d0 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3726644386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f059c0cab00 con 0x561c423bc4d0 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f058eff5640 1 -- 172.21.15.29:0/3726644386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f05a0004220 con 0x561c423bc4d0 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f05b7ff5740 10 client.15600 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3726644386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561c424c1620 con 0x561c423bc4d0 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f058eff5640 1 -- 172.21.15.29:0/3726644386 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f059c0cab00 con 0x561c423bc4d0 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f058eff5640 1 -- 172.21.15.29:0/3726644386 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f05a0003890 con 0x561c423bc4d0 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f05b7ff5740 10 client.15600 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f05b7ff5740 10 client.15600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f05b7ff5740 20 client.15600 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x561c4239bd80 0x561c427a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f05b7ff5740 4 client.15600 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.857+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3726644386 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c426f7300 con 0x561c4239bd80 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.858+0000 7f05aad9d640 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x561c4239bd80 0x561c427a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.858+0000 7f05aad9d640 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x561c4239bd80 0x561c427a19a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ca9e2b5f81f6c22 server_cookie=f426e14178856e02 in_seq=0 out_seq=0 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.858+0000 7f058eff5640 10 client.15600 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f058eff5640 1 -- 172.21.15.29:0/3726644386 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561c426f7300 con 0x561c4239bd80 2024-07-27T06:47:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f058eff5640 10 client.15600 handle_command_reply: tid=0 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f05b7ff5740 1 client.15600 shutdown 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3726644386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423bc4d0 msgr2=0x561c427900b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423bc4d0 0x561c427900b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f05a00126e0 tx=0x7f05a0012ba0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3726644386 shutdown_connections 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x561c4239bd80 0x561c427a19a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423dd1b0 0x561c4279b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423bc4d0 0x561c427900b0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f058eff5640 0 client.15600 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3726644386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423b42f0 0x561c4278fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3726644386 >> 172.21.15.29:0/3726644386 conn(0x561c423d1300 msgr2=0x561c4277c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3726644386 shutdown_connections 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3726644386 wait complete. 2024-07-27T06:47:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.859+0000 7f05b7ff5740 20 client.15600 trim_cache size 0 max 16384 2024-07-27T06:47:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05b7ff5740 1 Processor -- start 2024-07-27T06:47:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05b7ff5740 1 -- start start 2024-07-27T06:47:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05b7ff5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42781420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05b7ff5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c4278f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05b7ff5740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c4279b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05b7ff5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac1acf70 con 0x561c423b42f0 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05b7ff5740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac1aeab0 con 0x561c423dd1b0 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05b7ff5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac1ad640 con 0x561c423bc4d0 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42781420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05aa59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c4278f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42781420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45578/0 (socket says 172.21.15.29:45578) 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05aad9d640 1 -- 172.21.15.29:0/3700169754 learned_addr learned my addr 172.21.15.29:0/3700169754 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.880+0000 7f05ab59e640 1 --2- 172.21.15.29:0/3700169754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c4279b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.881+0000 7f05aa59c640 1 -- 172.21.15.29:0/3700169754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 msgr2=0x561c4279b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.881+0000 7f05aa59c640 1 --2- 172.21.15.29:0/3700169754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c4279b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.881+0000 7f05aa59c640 1 -- 172.21.15.29:0/3700169754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 msgr2=0x561c42781420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.881+0000 7f05aa59c640 1 --2- 172.21.15.29:0/3700169754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42781420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.881+0000 7f05aa59c640 1 -- 172.21.15.29:0/3700169754 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c42777d60 con 0x561c423bc4d0 2024-07-27T06:47:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.881+0000 7f05ab59e640 1 --2- 172.21.15.29:0/3700169754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c4279b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.881+0000 7f05aa59c640 1 --2- 172.21.15.29:0/3700169754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c4278f1d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009de0 tx=0x7f05a00056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=265760174b6e3646 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.881+0000 7f0590ff9640 1 -- 172.21.15.29:0/3700169754 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a0029070 con 0x561c423bc4d0 2024-07-27T06:47:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.881+0000 7f0590ff9640 1 -- 172.21.15.29:0/3700169754 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05a0003aa0 con 0x561c423bc4d0 2024-07-27T06:47:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.881+0000 7f0590ff9640 1 -- 172.21.15.29:0/3700169754 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a000fb20 con 0x561c423bc4d0 2024-07-27T06:47:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.882+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3700169754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 msgr2=0x561c4278f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.882+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3700169754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c4278f1d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009de0 tx=0x7f05a00056d0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.883+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3700169754 shutdown_connections 2024-07-27T06:47:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.883+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3700169754 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c4279b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.883+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3700169754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c4278f1d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.883+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/3700169754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42781420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.883+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3700169754 >> 172.21.15.29:0/3700169754 conn(0x561c423d1300 msgr2=0x561c42780d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.883+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3700169754 shutdown_connections 2024-07-27T06:47:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.883+0000 7f05b7ff5740 1 -- 172.21.15.29:0/3700169754 wait complete. 2024-07-27T06:47:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.883+0000 7f05b7ff5740 1 -- start start 2024-07-27T06:47:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.883+0000 7f05b7ff5740 1 Processor -- start 2024-07-27T06:47:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05b7ff5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42797ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05b7ff5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c42798070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05b7ff5740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c427948f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05b7ff5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac1ae3e0 con 0x561c423b42f0 2024-07-27T06:47:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05b7ff5740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac1add10 con 0x561c423dd1b0 2024-07-27T06:47:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05b7ff5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac1aaf50 con 0x561c423bc4d0 2024-07-27T06:47:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05b7ff5740 10 client.0 _ll_register_callbacks cb 0x7f05a0009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42797ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42797ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45588/0 (socket says 172.21.15.29:45588) 2024-07-27T06:47:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05aad9d640 1 -- 172.21.15.29:0/1794927600 learned_addr learned my addr 172.21.15.29:0/1794927600 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05ab59e640 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c427948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05aa59c640 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c42798070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05aad9d640 1 -- 172.21.15.29:0/1794927600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 msgr2=0x561c42798070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05aad9d640 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c42798070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05aad9d640 1 -- 172.21.15.29:0/1794927600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 msgr2=0x561c427948f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05aad9d640 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c427948f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05aad9d640 1 -- 172.21.15.29:0/1794927600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c42795080 con 0x561c423b42f0 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05ab59e640 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c427948f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f05aad9d640 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42797ab0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f059c107fd0 tx=0x7f059c03b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a800541dbb26228b server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.884+0000 7f058eff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.885+0000 7f058eff5640 1 -- 172.21.15.29:0/1794927600 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c049040 con 0x561c423b42f0 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.885+0000 7f058eff5640 1 -- 172.21.15.29:0/1794927600 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f059c101e00 con 0x561c423b42f0 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.885+0000 7f05b7ff5740 1 -- 172.21.15.29:0/1794927600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561c426f2da0 con 0x561c423b42f0 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.885+0000 7f058eff5640 1 -- 172.21.15.29:0/1794927600 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f059c0ca940 con 0x561c423b42f0 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.885+0000 7f05b7ff5740 10 client.15606 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.885+0000 7f05b7ff5740 1 -- 172.21.15.29:0/1794927600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561c42808bc0 con 0x561c423b42f0 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.885+0000 7f058eff5640 1 -- 172.21.15.29:0/1794927600 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561c426f2da0 con 0x561c423b42f0 2024-07-27T06:47:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.885+0000 7f058eff5640 1 -- 172.21.15.29:0/1794927600 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f059c107cb0 con 0x561c423b42f0 2024-07-27T06:47:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.885+0000 7f05b7ff5740 10 client.15606 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.885+0000 7f05b7ff5740 10 client.15606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.885+0000 7f05b7ff5740 20 client.15606 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.886+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x561c423711c0 0x561c4280b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.886+0000 7f05b7ff5740 4 client.15606 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-27T06:47:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.886+0000 7f05b7ff5740 1 -- 172.21.15.29:0/1794927600 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561c426f04e0 con 0x561c423711c0 2024-07-27T06:47:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.886+0000 7f05aa59c640 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x561c423711c0 0x561c4280b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.886+0000 7f058eff5640 10 client.15606 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.886+0000 7f05aa59c640 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x561c423711c0 0x561c4280b190 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c917faef6663703 server_cookie=368d2b6c8207ce1e in_seq=0 out_seq=0 2024-07-27T06:47:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f058eff5640 1 -- 172.21.15.29:0/1794927600 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561c426f04e0 con 0x561c423711c0 2024-07-27T06:47:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f058eff5640 10 client.15606 handle_command_reply: tid=0 2024-07-27T06:47:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f05b7ff5740 1 client.15606 shutdown 2024-07-27T06:47:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f05b7ff5740 1 -- 172.21.15.29:0/1794927600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 msgr2=0x561c42797ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42797ab0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f059c107fd0 tx=0x7f059c03b4f0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f05b7ff5740 1 -- 172.21.15.29:0/1794927600 shutdown_connections 2024-07-27T06:47:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561c423dd1b0 0x561c427948f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x561c423711c0 0x561c4280b190 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561c423bc4d0 0x561c42798070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f05b7ff5740 1 --2- 172.21.15.29:0/1794927600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561c423b42f0 0x561c42797ab0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f058eff5640 0 client.15606 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f05b7ff5740 1 -- 172.21.15.29:0/1794927600 >> 172.21.15.29:0/1794927600 conn(0x561c423d1300 msgr2=0x561c427816d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f05b7ff5740 1 -- 172.21.15.29:0/1794927600 shutdown_connections 2024-07-27T06:47:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f05b7ff5740 1 -- 172.21.15.29:0/1794927600 wait complete. 2024-07-27T06:47:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.896+0000 7f05b7ff5740 20 client.15606 trim_cache size 0 max 16384 2024-07-27T06:47:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-07-27T06:47:29.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-07-27T06:47:29.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-07-27T06:47:29.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-07-27T06:47:29.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.897+0000 7f05b10e3640 1 -- 172.21.15.29:0/657693229 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f05940b3cb0 msgr2=0x7f05940b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.897+0000 7f05b10e3640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f05940b3cb0 0x7f05940b61c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f05a4005fd0 tx=0x7f05a4009500 comp rx=0 tx=0).stop 2024-07-27T06:47:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.897+0000 7f05b10e3640 1 -- 172.21.15.29:0/657693229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05ac104370 msgr2=0x7f05ac199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.897+0000 7f05b10e3640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05ac104370 0x7f05ac199f40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f059c005940 tx=0x7f059c0040c0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.897+0000 7f05b10e3640 1 -- 172.21.15.29:0/657693229 shutdown_connections 2024-07-27T06:47:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.897+0000 7f05b10e3640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05ac10d7d0 0x7f05ac19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.897+0000 7f05b10e3640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f05940b3cb0 0x7f05940b61c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.897+0000 7f05b10e3640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05ac104cc0 0x7f05ac19a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.897+0000 7f05b10e3640 1 --2- 172.21.15.29:0/657693229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05ac104370 0x7f05ac199f40 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.897+0000 7f05b10e3640 1 -- 172.21.15.29:0/657693229 >> 172.21.15.29:0/657693229 conn(0x7f05ac0fdde0 msgr2=0x7f05ac1072c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.897+0000 7f05b10e3640 1 -- 172.21.15.29:0/657693229 shutdown_connections 2024-07-27T06:47:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:29.897+0000 7f05b10e3640 1 -- 172.21.15.29:0/657693229 wait complete. 2024-07-27T06:47:30.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.407+0000 7f62e4789640 1 -- 172.21.15.29:0/3105580239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc102510 msgr2=0x7f62dc102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:30.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.407+0000 7f62e4789640 1 --2- 172.21.15.29:0/3105580239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc102510 0x7f62dc102920 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009900 tx=0x7f62d802a3f0 comp rx=0 tx=0).stop 2024-07-27T06:47:30.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.407+0000 7f62e4789640 1 -- 172.21.15.29:0/3105580239 shutdown_connections 2024-07-27T06:47:30.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.407+0000 7f62e4789640 1 --2- 172.21.15.29:0/3105580239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62dc1087a0 0x7f62dc10efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:30.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.407+0000 7f62e4789640 1 --2- 172.21.15.29:0/3105580239 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62dc107e50 0x7f62dc108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.407+0000 7f62e4789640 1 --2- 172.21.15.29:0/3105580239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc102510 0x7f62dc102920 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.407+0000 7f62e4789640 1 -- 172.21.15.29:0/3105580239 >> 172.21.15.29:0/3105580239 conn(0x7f62dc0791a0 msgr2=0x7f62dc0795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.407+0000 7f62e4789640 1 -- 172.21.15.29:0/3105580239 shutdown_connections 2024-07-27T06:47:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.407+0000 7f62e4789640 1 -- 172.21.15.29:0/3105580239 wait complete. 2024-07-27T06:47:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.408+0000 7f62e4789640 1 Processor -- start 2024-07-27T06:47:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.408+0000 7f62e4789640 1 -- start start 2024-07-27T06:47:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.408+0000 7f62e4789640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc102510 0x7f62dc19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.408+0000 7f62e4789640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62dc107e50 0x7f62dc19b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.408+0000 7f62e4789640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62dc1087a0 0x7f62dc19b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.408+0000 7f62e4789640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc112450 con 0x7f62dc102510 2024-07-27T06:47:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.408+0000 7f62e4789640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc1122d0 con 0x7f62dc1087a0 2024-07-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.408+0000 7f62e4789640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc1125d0 con 0x7f62dc107e50 2024-07-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.408+0000 7f62e2525640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc102510 0x7f62dc19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.408+0000 7f62e2525640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc102510 0x7f62dc19ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45598/0 (socket says 172.21.15.29:45598) 2024-07-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.408+0000 7f62e2525640 1 -- 172.21.15.29:0/1724184650 learned_addr learned my addr 172.21.15.29:0/1724184650 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.408+0000 7f62e2d26640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62dc1087a0 0x7f62dc19b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62e1d24640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62dc107e50 0x7f62dc19b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62e2525640 1 -- 172.21.15.29:0/1724184650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62dc107e50 msgr2=0x7f62dc19b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62e2525640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62dc107e50 0x7f62dc19b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62e2525640 1 -- 172.21.15.29:0/1724184650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62dc1087a0 msgr2=0x7f62dc19b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62e2525640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62dc1087a0 0x7f62dc19b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62e2525640 1 -- 172.21.15.29:0/1724184650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62dc1a5fd0 con 0x7f62dc102510 2024-07-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62e1d24640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62dc107e50 0x7f62dc19b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62e2525640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc102510 0x7f62dc19ada0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f62d8005940 tx=0x7f62d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62d37fe640 1 -- 172.21.15.29:0/1724184650 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d803f030 con 0x7f62dc102510 2024-07-27T06:47:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62d37fe640 1 -- 172.21.15.29:0/1724184650 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62d8003cb0 con 0x7f62dc102510 2024-07-27T06:47:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62e4789640 1 -- 172.21.15.29:0/1724184650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62dc1a6260 con 0x7f62dc102510 2024-07-27T06:47:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62d37fe640 1 -- 172.21.15.29:0/1724184650 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8035070 con 0x7f62dc102510 2024-07-27T06:47:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.409+0000 7f62e4789640 1 -- 172.21.15.29:0/1724184650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62dc1a6790 con 0x7f62dc102510 2024-07-27T06:47:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.410+0000 7f62e4789640 1 -- 172.21.15.29:0/1724184650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62dc103790 con 0x7f62dc102510 2024-07-27T06:47:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.412+0000 7f62d37fe640 1 -- 172.21.15.29:0/1724184650 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f62d8042030 con 0x7f62dc102510 2024-07-27T06:47:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.413+0000 7f62d37fe640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f62c80b3d30 0x7f62c80b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.413+0000 7f62e1d24640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f62c80b3d30 0x7f62c80b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.413+0000 7f62d37fe640 1 -- 172.21.15.29:0/1724184650 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f62d8104a50 con 0x7f62dc102510 2024-07-27T06:47:30.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.413+0000 7f62e1d24640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f62c80b3d30 0x7f62c80b6240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0058c0 tx=0x7f62cc005850 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:30.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.416+0000 7f62d37fe640 1 -- 172.21.15.29:0/1724184650 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f62d80cbdd0 con 0x7f62dc102510 2024-07-27T06:47:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.634+0000 7f62e4789640 1 -- 172.21.15.29:0/1724184650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62dc102920 con 0x7f62dc102510 2024-07-27T06:47:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.634+0000 7f62d37fe640 1 -- 172.21.15.29:0/1724184650 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11957 (secure 0 0 0) 0x7f62d80d40a0 con 0x7f62dc102510 2024-07-27T06:47:30.636 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:47:30.637 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-07-27T06:46:50:421100+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:46:49.561737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24670,24658,24646,24617,24613]},"id":1}]} 2024-07-27T06:47:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-07-27T06:47:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.637+0000 7f62e4789640 1 -- 172.21.15.29:0/1724184650 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f62c80b3d30 msgr2=0x7f62c80b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.637+0000 7f62e4789640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f62c80b3d30 0x7f62c80b6240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0058c0 tx=0x7f62cc005850 comp rx=0 tx=0).stop 2024-07-27T06:47:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.637+0000 7f62e4789640 1 -- 172.21.15.29:0/1724184650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc102510 msgr2=0x7f62dc19ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.637+0000 7f62e4789640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc102510 0x7f62dc19ada0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f62d8005940 tx=0x7f62d80042f0 comp rx=0 tx=0).stop 2024-07-27T06:47:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.637+0000 7f62e4789640 1 -- 172.21.15.29:0/1724184650 shutdown_connections 2024-07-27T06:47:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.637+0000 7f62e4789640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62dc1087a0 0x7f62dc19b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.637+0000 7f62e4789640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f62c80b3d30 0x7f62c80b6240 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.637+0000 7f62e4789640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62dc107e50 0x7f62dc19b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.637+0000 7f62e4789640 1 --2- 172.21.15.29:0/1724184650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62dc102510 0x7f62dc19ada0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.637+0000 7f62e4789640 1 -- 172.21.15.29:0/1724184650 >> 172.21.15.29:0/1724184650 conn(0x7f62dc0791a0 msgr2=0x7f62dc105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.637+0000 7f62e4789640 1 -- 172.21.15.29:0/1724184650 shutdown_connections 2024-07-27T06:47:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:30.637+0000 7f62e4789640 1 -- 172.21.15.29:0/1724184650 wait complete. 2024-07-27T06:47:31.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:30 smithi029 ceph-mon[32610]: pgmap v412: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-07-27T06:47:31.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:30 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1724184650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:47:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:30 smithi077 ceph-mon[37153]: pgmap v412: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-07-27T06:47:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:30 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1724184650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:47:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:30 smithi047 ceph-mon[36835]: pgmap v412: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-07-27T06:47:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:30 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1724184650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:47:31.568 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-27T06:43:14.801207+0000', 'modified': '2024-07-27T06:46:49.561737+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24670, 'mds_1': 24658, 'mds_2': 24646, 'mds_3': 24617, 'mds_4': 24613}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14781': {'gid': 14781, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6839/1639930501', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6838', 'nonce': 1639930501}, {'type': 'v1', 'addr': '172.21.15.29:6839', 'nonce': 1639930501}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14793': {'gid': 14793, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.29:6841/2982668187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6840', 'nonce': 2982668187}, {'type': 'v1', 'addr': '172.21.15.29:6841', 'nonce': 2982668187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24542': {'gid': 24542, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6833/2402830639', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6832', 'nonce': 2402830639}, {'type': 'v1', 'addr': '172.21.15.47:6833', 'nonce': 2402830639}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24566': {'gid': 24566, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6835/2900591116', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6834', 'nonce': 2900591116}, {'type': 'v1', 'addr': '172.21.15.47:6835', 'nonce': 2900591116}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24584': {'gid': 24584, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.47:6837/2573307312', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6836', 'nonce': 2573307312}, {'type': 'v1', 'addr': '172.21.15.47:6837', 'nonce': 2573307312}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24613': {'gid': 24613, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.77:6833/1085922080', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6832', 'nonce': 1085922080}, {'type': 'v1', 'addr': '172.21.15.77:6833', 'nonce': 1085922080}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24617': {'gid': 24617, 'name': 'b', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.47:6839/670107266', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.47:6838', 'nonce': 670107266}, {'type': 'v1', 'addr': '172.21.15.47:6839', 'nonce': 670107266}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24646': {'gid': 24646, 'name': 'f', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.77:6835/439792826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6834', 'nonce': 439792826}, {'type': 'v1', 'addr': '172.21.15.77:6835', 'nonce': 439792826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24658': {'gid': 24658, 'name': 'c', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.77:6837/2959628937', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 2959628937}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 2959628937}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24670': {'gid': 24670, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.77:6839/269899139', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6838', 'nonce': 269899139}, {'type': 'v1', 'addr': '172.21.15.77:6839', 'nonce': 269899139}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24670, 'qdb_cluster': [24670, 24658, 24646, 24617, 24613]} 2024-07-27T06:47:31.568 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-27T06:47:31.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.l status 2024-07-27T06:47:31.879 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:33.019 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:32 smithi029 ceph-mon[32610]: pgmap v413: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-07-27T06:47:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:32 smithi077 ceph-mon[37153]: pgmap v413: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-07-27T06:47:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:32 smithi047 ceph-mon[36835]: pgmap v413: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-07-27T06:47:33.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.673+0000 7fd922b71640 1 -- 172.21.15.29:0/2930950337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91c109830 msgr2=0x7fd91c110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.673+0000 7fd922b71640 1 --2- 172.21.15.29:0/2930950337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91c109830 0x7fd91c110060 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd914009ca0 tx=0x7fd91402a8c0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.673+0000 7fd922b71640 1 -- 172.21.15.29:0/2930950337 shutdown_connections 2024-07-27T06:47:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.673+0000 7fd922b71640 1 --2- 172.21.15.29:0/2930950337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91c109830 0x7fd91c110060 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.673+0000 7fd922b71640 1 --2- 172.21.15.29:0/2930950337 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd91c108ee0 0x7fd91c1092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.673+0000 7fd922b71640 1 --2- 172.21.15.29:0/2930950337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c1035f0 0x7fd91c103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.673+0000 7fd922b71640 1 -- 172.21.15.29:0/2930950337 >> 172.21.15.29:0/2930950337 conn(0x7fd91c07b340 msgr2=0x7fd91c07b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.673+0000 7fd922b71640 1 -- 172.21.15.29:0/2930950337 shutdown_connections 2024-07-27T06:47:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.673+0000 7fd922b71640 1 -- 172.21.15.29:0/2930950337 wait complete. 2024-07-27T06:47:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.673+0000 7fd922b71640 1 Processor -- start 2024-07-27T06:47:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd922b71640 1 -- start start 2024-07-27T06:47:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd922b71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c1035f0 0x7fd91c075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd922b71640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd91c108ee0 0x7fd91c075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd922b71640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91c109830 0x7fd91c075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd922b71640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c1134e0 con 0x7fd91c109830 2024-07-27T06:47:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd922b71640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c113360 con 0x7fd91c108ee0 2024-07-27T06:47:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd922b71640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c113660 con 0x7fd91c1035f0 2024-07-27T06:47:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd92110e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91c109830 0x7fd91c075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd92110e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91c109830 0x7fd91c075bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45610/0 (socket says 172.21.15.29:45610) 2024-07-27T06:47:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd92110e640 1 -- 172.21.15.29:0/987220247 learned_addr learned my addr 172.21.15.29:0/987220247 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd92090d640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c1035f0 0x7fd91c075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd91bfff640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd91c108ee0 0x7fd91c075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd92110e640 1 -- 172.21.15.29:0/987220247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c1035f0 msgr2=0x7fd91c075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd92110e640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c1035f0 0x7fd91c075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd92110e640 1 -- 172.21.15.29:0/987220247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd91c108ee0 msgr2=0x7fd91c075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd92110e640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd91c108ee0 0x7fd91c075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.674+0000 7fd92110e640 1 -- 172.21.15.29:0/987220247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd91c1a80e0 con 0x7fd91c109830 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.675+0000 7fd92110e640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91c109830 0x7fd91c075bc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd914005cc0 tx=0x7fd91402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.675+0000 7fd919ffb640 1 -- 172.21.15.29:0/987220247 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd914003bf0 con 0x7fd91c109830 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.675+0000 7fd922b71640 1 -- 172.21.15.29:0/987220247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd91c1a8370 con 0x7fd91c109830 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.675+0000 7fd919ffb640 1 -- 172.21.15.29:0/987220247 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd914003d90 con 0x7fd91c109830 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.675+0000 7fd919ffb640 1 -- 172.21.15.29:0/987220247 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91402fb20 con 0x7fd91c109830 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.675+0000 7fd922b71640 1 -- 172.21.15.29:0/987220247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd91c1a8900 con 0x7fd91c109830 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd9290f5740 1 Processor -- start 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd919ffb640 1 -- 172.21.15.29:0/987220247 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7fd914004580 con 0x7fd91c109830 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd9290f5740 1 -- start start 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd9290f5740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422b1af60 0x55f422ec0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd9290f5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 0x55f422ec08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd9290f5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422afa280 0x55f422ec0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd9290f5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f422e36920 con 0x55f422af20a0 2024-07-27T06:47:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd9290f5740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55f422e25e20 con 0x55f422b1af60 2024-07-27T06:47:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd9290f5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f422e29e60 con 0x55f422afa280 2024-07-27T06:47:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd92090d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 0x55f422ec08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd92090d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 0x55f422ec08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45618/0 (socket says 172.21.15.29:45618) 2024-07-27T06:47:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd92090d640 1 -- 172.21.15.29:0/3527913071 learned_addr learned my addr 172.21.15.29:0/3527913071 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.678+0000 7fd91bfff640 1 --2- 172.21.15.29:0/3527913071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422b1af60 0x55f422ec0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd91bfff640 1 --2- 172.21.15.29:0/3527913071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422afa280 0x55f422ec0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd92090d640 1 -- 172.21.15.29:0/3527913071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422afa280 msgr2=0x55f422ec0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd92090d640 1 --2- 172.21.15.29:0/3527913071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422afa280 0x55f422ec0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd92090d640 1 -- 172.21.15.29:0/3527913071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422b1af60 msgr2=0x55f422ec0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd92090d640 1 --2- 172.21.15.29:0/3527913071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422b1af60 0x55f422ec0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd92090d640 1 -- 172.21.15.29:0/3527913071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f422dcca40 con 0x55f422af20a0 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd91bfff640 1 --2- 172.21.15.29:0/3527913071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422afa280 0x55f422ec0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd919ffb640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd90c0b3cb0 0x7fd90c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd92090d640 1 --2- 172.21.15.29:0/3527913071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 0x55f422ec08c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd91000ac80 tx=0x7fd91000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd5ef622d5802e0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd91bfff640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd90c0b3cb0 0x7fd90c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd919ffb640 1 -- 172.21.15.29:0/987220247 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd914105300 con 0x7fd91c109830 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd906ffd640 1 -- 172.21.15.29:0/3527913071 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91001d070 con 0x55f422af20a0 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd906ffd640 1 -- 172.21.15.29:0/3527913071 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd910018af0 con 0x55f422af20a0 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.679+0000 7fd906ffd640 1 -- 172.21.15.29:0/3527913071 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910010ae0 con 0x55f422af20a0 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.680+0000 7fd9290f5740 1 -- 172.21.15.29:0/3527913071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 msgr2=0x55f422ec08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.680+0000 7fd91bfff640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd90c0b3cb0 0x7fd90c0b61c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd91c108a00 tx=0x7fd908014040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.680+0000 7fd9290f5740 1 --2- 172.21.15.29:0/3527913071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 0x55f422ec08c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd91000ac80 tx=0x7fd91000f790 comp rx=0 tx=0).stop 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.680+0000 7fd9290f5740 1 -- 172.21.15.29:0/3527913071 shutdown_connections 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.680+0000 7fd9290f5740 1 --2- 172.21.15.29:0/3527913071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422afa280 0x55f422ec0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.680+0000 7fd9290f5740 1 --2- 172.21.15.29:0/3527913071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 0x55f422ec08c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.680+0000 7fd9290f5740 1 --2- 172.21.15.29:0/3527913071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422b1af60 0x55f422ec0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.680+0000 7fd9290f5740 1 -- 172.21.15.29:0/3527913071 >> 172.21.15.29:0/3527913071 conn(0x55f422b0f350 msgr2=0x55f422ebe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.680+0000 7fd9290f5740 1 -- 172.21.15.29:0/3527913071 shutdown_connections 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.680+0000 7fd9290f5740 1 -- 172.21.15.29:0/3527913071 wait complete. 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.680+0000 7fd9290f5740 1 -- start start 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd9290f5740 1 Processor -- start 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd9290f5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 0x55f422ecbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd9290f5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422afa280 0x55f422ecc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd9290f5740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422b1af60 0x55f422ed7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd9290f5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f422c895f0 con 0x55f422af20a0 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd9290f5740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55f422dc8890 con 0x55f422b1af60 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd9290f5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f422eceb60 con 0x55f422afa280 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd9290f5740 10 client.0 _ll_register_callbacks cb 0x55f422dd27d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd92090d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 0x55f422ecbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd92090d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 0x55f422ecbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45634/0 (socket says 172.21.15.29:45634) 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd92090d640 1 -- 172.21.15.29:0/2957870090 learned_addr learned my addr 172.21.15.29:0/2957870090 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd91bfff640 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422afa280 0x55f422ecc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.681+0000 7fd92110e640 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422b1af60 0x55f422ed7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd92090d640 1 -- 172.21.15.29:0/2957870090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422afa280 msgr2=0x55f422ecc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd92090d640 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422afa280 0x55f422ecc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd92090d640 1 -- 172.21.15.29:0/2957870090 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422b1af60 msgr2=0x55f422ed7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd92090d640 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422b1af60 0x55f422ed7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd92090d640 1 -- 172.21.15.29:0/2957870090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f422b86540 con 0x55f422af20a0 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd92110e640 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422b1af60 0x55f422ed7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd91bfff640 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422afa280 0x55f422ecc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd92090d640 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 0x55f422ecbf60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd91000a670 tx=0x7fd910010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5589b3b632b1d0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd8fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd8fbfff640 1 -- 172.21.15.29:0/2957870090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910027020 con 0x55f422af20a0 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd8fbfff640 1 -- 172.21.15.29:0/2957870090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd910018880 con 0x55f422af20a0 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd9290f5740 1 -- 172.21.15.29:0/2957870090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd910018af0 con 0x55f422af20a0 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd8fbfff640 1 -- 172.21.15.29:0/2957870090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd910032050 con 0x55f422af20a0 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd9290f5740 10 client.15624 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd9290f5740 1 -- 172.21.15.29:0/2957870090 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f422edaab0 con 0x55f422af20a0 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.682+0000 7fd8fbfff640 1 -- 172.21.15.29:0/2957870090 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd910018af0 con 0x55f422af20a0 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.683+0000 7fd8fbfff640 1 -- 172.21.15.29:0/2957870090 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd9100046c0 con 0x55f422af20a0 2024-07-27T06:47:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.683+0000 7fd9290f5740 10 client.15624 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.683+0000 7fd9290f5740 10 client.15624 resolve_mds: resolved name 'l' to daemon mds.l 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.683+0000 7fd9290f5740 20 client.15624 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.683+0000 7fd9290f5740 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55f422ad98e0 0x55f422ede150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.683+0000 7fd9290f5740 4 client.15624 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.683+0000 7fd9290f5740 1 -- 172.21.15.29:0/2957870090 --> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f422e38870 con 0x55f422ad98e0 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.683+0000 7fd91bfff640 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55f422ad98e0 0x55f422ede150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.683+0000 7fd91bfff640 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55f422ad98e0 0x55f422ede150 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=89c22660abc4d23e server_cookie=c69613463341c06c in_seq=0 out_seq=0 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.683+0000 7fd8fbfff640 10 client.15624 ms_handle_connect on v2:172.21.15.77:6832/1085922080 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.684+0000 7fd8fbfff640 1 -- 172.21.15.29:0/2957870090 <== mds.4 v2:172.21.15.77:6832/1085922080 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f422e38870 con 0x55f422ad98e0 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.684+0000 7fd8fbfff640 10 client.15624 handle_command_reply: tid=0 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.684+0000 7fd9290f5740 1 client.15624 shutdown 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.684+0000 7fd9290f5740 1 -- 172.21.15.29:0/2957870090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 msgr2=0x55f422ecbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.684+0000 7fd9290f5740 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 0x55f422ecbf60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd91000a670 tx=0x7fd910010660 comp rx=0 tx=0).stop 2024-07-27T06:47:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.684+0000 7fd9290f5740 1 -- 172.21.15.29:0/2957870090 shutdown_connections 2024-07-27T06:47:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.684+0000 7fd9290f5740 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55f422ad98e0 0x55f422ede150 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.684+0000 7fd9290f5740 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422b1af60 0x55f422ed7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.684+0000 7fd9290f5740 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422afa280 0x55f422ecc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.684+0000 7fd9290f5740 1 --2- 172.21.15.29:0/2957870090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422af20a0 0x55f422ecbf60 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.684+0000 7fd8fbfff640 0 client.15624 ms_handle_reset on v2:172.21.15.77:6832/1085922080 2024-07-27T06:47:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.684+0000 7fd9290f5740 1 -- 172.21.15.29:0/2957870090 >> 172.21.15.29:0/2957870090 conn(0x55f422b0f350 msgr2=0x55f422ec3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.685+0000 7fd9290f5740 1 -- 172.21.15.29:0/2957870090 shutdown_connections 2024-07-27T06:47:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.685+0000 7fd9290f5740 1 -- 172.21.15.29:0/2957870090 wait complete. 2024-07-27T06:47:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.685+0000 7fd9290f5740 20 client.15624 trim_cache size 0 max 16384 2024-07-27T06:47:33.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.703+0000 7fd9290f5740 1 Processor -- start 2024-07-27T06:47:33.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.703+0000 7fd9290f5740 1 -- start start 2024-07-27T06:47:33.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.703+0000 7fd9290f5740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422af20a0 0x55f422edcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd9290f5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422afa280 0x55f422ec7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd9290f5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422b1af60 0x55f422ed7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd9290f5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f422edc490 con 0x55f422afa280 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd9290f5740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55f422edcb60 con 0x55f422af20a0 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd9290f5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f422edd230 con 0x55f422b1af60 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd91bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422afa280 0x55f422ec7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd91bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422afa280 0x55f422ec7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45638/0 (socket says 172.21.15.29:45638) 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd91bfff640 1 -- 172.21.15.29:0/115627791 learned_addr learned my addr 172.21.15.29:0/115627791 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd92090d640 1 --2- 172.21.15.29:0/115627791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422af20a0 0x55f422edcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd92110e640 1 --2- 172.21.15.29:0/115627791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422b1af60 0x55f422ed7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd91bfff640 1 -- 172.21.15.29:0/115627791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422b1af60 msgr2=0x55f422ed7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd91bfff640 1 --2- 172.21.15.29:0/115627791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422b1af60 0x55f422ed7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd91bfff640 1 -- 172.21.15.29:0/115627791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422af20a0 msgr2=0x55f422edcce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd91bfff640 1 --2- 172.21.15.29:0/115627791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422af20a0 0x55f422edcce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd91bfff640 1 -- 172.21.15.29:0/115627791 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f422ecd020 con 0x55f422afa280 2024-07-27T06:47:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd92110e640 1 --2- 172.21.15.29:0/115627791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422b1af60 0x55f422ed7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd92090d640 1 --2- 172.21.15.29:0/115627791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422af20a0 0x55f422edcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd91bfff640 1 --2- 172.21.15.29:0/115627791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422afa280 0x55f422ec7510 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd90800b620 tx=0x7fd9080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e0ff0e7391e6b4 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd9067fc640 1 -- 172.21.15.29:0/115627791 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90801ed50 con 0x55f422afa280 2024-07-27T06:47:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd9067fc640 1 -- 172.21.15.29:0/115627791 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd908005020 con 0x55f422afa280 2024-07-27T06:47:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.704+0000 7fd9067fc640 1 -- 172.21.15.29:0/115627791 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd908004890 con 0x55f422afa280 2024-07-27T06:47:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.705+0000 7fd9290f5740 1 -- 172.21.15.29:0/115627791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422afa280 msgr2=0x55f422ec7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.705+0000 7fd9290f5740 1 --2- 172.21.15.29:0/115627791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422afa280 0x55f422ec7510 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd90800b620 tx=0x7fd9080046c0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.705+0000 7fd9290f5740 1 -- 172.21.15.29:0/115627791 shutdown_connections 2024-07-27T06:47:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.705+0000 7fd9290f5740 1 --2- 172.21.15.29:0/115627791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422b1af60 0x55f422ed7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.705+0000 7fd9290f5740 1 --2- 172.21.15.29:0/115627791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422afa280 0x55f422ec7510 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.705+0000 7fd9290f5740 1 --2- 172.21.15.29:0/115627791 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422af20a0 0x55f422edcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.705+0000 7fd9290f5740 1 -- 172.21.15.29:0/115627791 >> 172.21.15.29:0/115627791 conn(0x55f422b0f350 msgr2=0x55f422ec3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.705+0000 7fd9290f5740 1 -- 172.21.15.29:0/115627791 shutdown_connections 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.705+0000 7fd9290f5740 1 -- 172.21.15.29:0/115627791 wait complete. 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.706+0000 7fd9290f5740 1 -- start start 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.706+0000 7fd9290f5740 1 Processor -- start 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.706+0000 7fd9290f5740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422af20a0 0x55f422edf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.706+0000 7fd9290f5740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422afa280 0x55f422edfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.706+0000 7fd9290f5740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422b1af60 0x55f422ee4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.706+0000 7fd9290f5740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f422eddfd0 con 0x55f422b1af60 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.706+0000 7fd9290f5740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55f422edd900 con 0x55f422afa280 2024-07-27T06:47:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.706+0000 7fd9290f5740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f422ede6a0 con 0x55f422af20a0 2024-07-27T06:47:33.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.706+0000 7fd9290f5740 10 client.0 _ll_register_callbacks cb 0x7fd91000a8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:33.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd92110e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422b1af60 0x55f422ee4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd92110e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422b1af60 0x55f422ee4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45646/0 (socket says 172.21.15.29:45646) 2024-07-27T06:47:33.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd92110e640 1 -- 172.21.15.29:0/1636477728 learned_addr learned my addr 172.21.15.29:0/1636477728 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:33.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd91bfff640 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422afa280 0x55f422edfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd92090d640 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422af20a0 0x55f422edf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd92110e640 1 -- 172.21.15.29:0/1636477728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422af20a0 msgr2=0x55f422edf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:33.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd92110e640 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422af20a0 0x55f422edf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd92110e640 1 -- 172.21.15.29:0/1636477728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422afa280 msgr2=0x55f422edfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:33.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd92110e640 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422afa280 0x55f422edfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd92110e640 1 -- 172.21.15.29:0/1636477728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f422ee5190 con 0x55f422b1af60 2024-07-27T06:47:33.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd92110e640 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422b1af60 0x55f422ee4b20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd914006010 tx=0x7fd9140cb920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19426107c4d77fe0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd8fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd8fbfff640 1 -- 172.21.15.29:0/1636477728 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9140e5070 con 0x55f422b1af60 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd8fbfff640 1 -- 172.21.15.29:0/1636477728 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd9140cc350 con 0x55f422b1af60 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd9290f5740 1 -- 172.21.15.29:0/1636477728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd908005020 con 0x55f422b1af60 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd8fbfff640 1 -- 172.21.15.29:0/1636477728 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd91402fed0 con 0x55f422b1af60 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd9290f5740 10 client.15630 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.707+0000 7fd9290f5740 1 -- 172.21.15.29:0/1636477728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f422ee6430 con 0x55f422b1af60 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.708+0000 7fd8fbfff640 1 -- 172.21.15.29:0/1636477728 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd908005020 con 0x55f422b1af60 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.708+0000 7fd8fbfff640 1 -- 172.21.15.29:0/1636477728 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd9140d4d10 con 0x55f422b1af60 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.708+0000 7fd9290f5740 10 client.15630 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.708+0000 7fd9290f5740 10 client.15630 resolve_mds: resolved name 'l' to daemon mds.l 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.708+0000 7fd9290f5740 20 client.15630 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:33.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.708+0000 7fd9290f5740 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55f422aaec90 0x55f422f49ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.708+0000 7fd9290f5740 4 client.15630 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.708+0000 7fd9290f5740 1 -- 172.21.15.29:0/1636477728 --> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] -- command(tid 0: {"prefix": "status"}) -- 0x55f422e34310 con 0x55f422aaec90 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.708+0000 7fd92090d640 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55f422aaec90 0x55f422f49ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd8fbfff640 10 client.15630 ms_handle_connect on v2:172.21.15.77:6832/1085922080 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd92090d640 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55f422aaec90 0x55f422f49ed0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=85995e36198d3231 server_cookie=aad743c109e81538 in_seq=0 out_seq=0 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd8fbfff640 1 -- 172.21.15.29:0/1636477728 <== mds.4 v2:172.21.15.77:6832/1085922080 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f422e34310 con 0x55f422aaec90 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd8fbfff640 10 client.15630 handle_command_reply: tid=0 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd9290f5740 1 client.15630 shutdown 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd9290f5740 1 -- 172.21.15.29:0/1636477728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422b1af60 msgr2=0x55f422ee4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd9290f5740 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422b1af60 0x55f422ee4b20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd914006010 tx=0x7fd9140cb920 comp rx=0 tx=0).stop 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd9290f5740 1 -- 172.21.15.29:0/1636477728 shutdown_connections 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd9290f5740 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55f422aaec90 0x55f422f49ed0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd9290f5740 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f422b1af60 0x55f422ee4b20 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd9290f5740 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f422afa280 0x55f422edfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd8fbfff640 0 client.15630 ms_handle_reset on v2:172.21.15.77:6832/1085922080 2024-07-27T06:47:33.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd9290f5740 1 --2- 172.21.15.29:0/1636477728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f422af20a0 0x55f422edf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd9290f5740 1 -- 172.21.15.29:0/1636477728 >> 172.21.15.29:0/1636477728 conn(0x55f422b0f350 msgr2=0x55f422ec3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:33.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd9290f5740 1 -- 172.21.15.29:0/1636477728 shutdown_connections 2024-07-27T06:47:33.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd9290f5740 1 -- 172.21.15.29:0/1636477728 wait complete. 2024-07-27T06:47:33.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.709+0000 7fd9290f5740 20 client.15630 trim_cache size 0 max 16384 2024-07-27T06:47:33.723 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:47:33.723 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:47:33.724 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 4, 2024-07-27T06:47:33.724 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24613, 2024-07-27T06:47:33.724 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:47:33.724 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:47:33.724 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:47:33.724 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 240.847901791, 2024-07-27T06:47:33.724 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 29, 2024-07-27T06:47:33.724 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:47:33.724 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 83, 2024-07-27T06:47:33.724 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 311.82767044399998 2024-07-27T06:47:33.724 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:47:33.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.710+0000 7fd922b71640 1 -- 172.21.15.29:0/987220247 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd90c0b3cb0 msgr2=0x7fd90c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:33.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.710+0000 7fd922b71640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd90c0b3cb0 0x7fd90c0b61c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd91c108a00 tx=0x7fd908014040 comp rx=0 tx=0).stop 2024-07-27T06:47:33.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.710+0000 7fd922b71640 1 -- 172.21.15.29:0/987220247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91c109830 msgr2=0x7fd91c075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:33.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.710+0000 7fd922b71640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91c109830 0x7fd91c075bc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd914005cc0 tx=0x7fd91402f380 comp rx=0 tx=0).stop 2024-07-27T06:47:33.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.710+0000 7fd922b71640 1 -- 172.21.15.29:0/987220247 shutdown_connections 2024-07-27T06:47:33.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.710+0000 7fd922b71640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd91c109830 0x7fd91c075bc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.710+0000 7fd922b71640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd91c108ee0 0x7fd91c075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.710+0000 7fd922b71640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd90c0b3cb0 0x7fd90c0b61c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.710+0000 7fd922b71640 1 --2- 172.21.15.29:0/987220247 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd91c1035f0 0x7fd91c075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:33.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.710+0000 7fd922b71640 1 -- 172.21.15.29:0/987220247 >> 172.21.15.29:0/987220247 conn(0x7fd91c07b340 msgr2=0x7fd91c106f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:33.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.710+0000 7fd922b71640 1 -- 172.21.15.29:0/987220247 shutdown_connections 2024-07-27T06:47:33.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:33.710+0000 7fd922b71640 1 -- 172.21.15.29:0/987220247 wait complete. 2024-07-27T06:47:34.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.b status 2024-07-27T06:47:34.814 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:35.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:34 smithi029 ceph-mon[32610]: pgmap v414: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-07-27T06:47:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:34 smithi077 ceph-mon[37153]: pgmap v414: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-07-27T06:47:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:34 smithi047 ceph-mon[36835]: pgmap v414: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-07-27T06:47:36.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.564+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/1384305020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d01035a0 msgr2=0x7fe8d01039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.564+0000 7fe8d6e6f640 1 --2- 172.21.15.29:0/1384305020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d01035a0 0x7fe8d01039b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8009900 tx=0x7fe8b802a410 comp rx=0 tx=0).stop 2024-07-27T06:47:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.564+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/1384305020 shutdown_connections 2024-07-27T06:47:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.564+0000 7fe8d6e6f640 1 --2- 172.21.15.29:0/1384305020 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8d0109830 0x7fe8d0110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.565+0000 7fe8d6e6f640 1 --2- 172.21.15.29:0/1384305020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d0108ee0 0x7fe8d01092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.565+0000 7fe8d6e6f640 1 --2- 172.21.15.29:0/1384305020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d01035a0 0x7fe8d01039b0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.565+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/1384305020 >> 172.21.15.29:0/1384305020 conn(0x7fe8d00ff010 msgr2=0x7fe8d0101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.565+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/1384305020 shutdown_connections 2024-07-27T06:47:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.565+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/1384305020 wait complete. 2024-07-27T06:47:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.565+0000 7fe8d6e6f640 1 Processor -- start 2024-07-27T06:47:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.565+0000 7fe8d6e6f640 1 -- start start 2024-07-27T06:47:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d6e6f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8d01035a0 0x7fe8d0075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d6e6f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d0108ee0 0x7fe8d0075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d6e6f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d0109830 0x7fe8d0075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d6e6f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d01134e0 con 0x7fe8d0109830 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d6e6f640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d0113360 con 0x7fe8d01035a0 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d6e6f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d0113660 con 0x7fe8d0108ee0 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d540c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d0109830 0x7fe8d0075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d540c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d0109830 0x7fe8d0075bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45660/0 (socket says 172.21.15.29:45660) 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d540c640 1 -- 172.21.15.29:0/829972256 learned_addr learned my addr 172.21.15.29:0/829972256 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d4c0b640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8d01035a0 0x7fe8d0075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8cffff640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d0108ee0 0x7fe8d0075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d540c640 1 -- 172.21.15.29:0/829972256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d0108ee0 msgr2=0x7fe8d0075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d540c640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d0108ee0 0x7fe8d0075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d540c640 1 -- 172.21.15.29:0/829972256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8d01035a0 msgr2=0x7fe8d0075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d540c640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8d01035a0 0x7fe8d0075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.566+0000 7fe8d540c640 1 -- 172.21.15.29:0/829972256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d01a8210 con 0x7fe8d0109830 2024-07-27T06:47:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.567+0000 7fe8d540c640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d0109830 0x7fe8d0075bc0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800d750 tx=0x7fe8c800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.567+0000 7fe8cdffb640 1 -- 172.21.15.29:0/829972256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8017070 con 0x7fe8d0109830 2024-07-27T06:47:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.567+0000 7fe8cdffb640 1 -- 172.21.15.29:0/829972256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8c80054d0 con 0x7fe8d0109830 2024-07-27T06:47:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.567+0000 7fe8cdffb640 1 -- 172.21.15.29:0/829972256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8004c60 con 0x7fe8d0109830 2024-07-27T06:47:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.567+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/829972256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8d01a8500 con 0x7fe8d0109830 2024-07-27T06:47:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.567+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/829972256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8d01a89f0 con 0x7fe8d0109830 2024-07-27T06:47:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.570+0000 7fe8cdffb640 1 -- 172.21.15.29:0/829972256 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7fe8c8009bd0 con 0x7fe8d0109830 2024-07-27T06:47:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.570+0000 7fe8dd3f3740 1 Processor -- start 2024-07-27T06:47:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.570+0000 7fe8dd3f3740 1 -- start start 2024-07-27T06:47:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.570+0000 7fe8dd3f3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54e8f60 0x55d0b588e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.570+0000 7fe8dd3f3740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54c00a0 0x55d0b588e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.570+0000 7fe8dd3f3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 0x55d0b588ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.570+0000 7fe8dd3f3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d0b5804940 con 0x55d0b54c8280 2024-07-27T06:47:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.570+0000 7fe8dd3f3740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55d0b57f3e40 con 0x55d0b54c00a0 2024-07-27T06:47:36.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.570+0000 7fe8dd3f3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d0b57f7e80 con 0x55d0b54e8f60 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 0x55d0b588ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 0x55d0b588ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45674/0 (socket says 172.21.15.29:45674) 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8cffff640 1 -- 172.21.15.29:0/2142033215 learned_addr learned my addr 172.21.15.29:0/2142033215 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8d4c0b640 1 --2- 172.21.15.29:0/2142033215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54c00a0 0x55d0b588e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8cffff640 1 --2- 172.21.15.29:0/2142033215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54e8f60 0x55d0b588e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8cffff640 1 -- 172.21.15.29:0/2142033215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54e8f60 msgr2=0x55d0b588e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8cffff640 1 --2- 172.21.15.29:0/2142033215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54e8f60 0x55d0b588e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8cffff640 1 -- 172.21.15.29:0/2142033215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54c00a0 msgr2=0x55d0b588e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8cffff640 1 --2- 172.21.15.29:0/2142033215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54c00a0 0x55d0b588e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8cdffb640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe8bc0b3d30 0x7fe8bc0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8cffff640 1 -- 172.21.15.29:0/2142033215 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0b579aa60 con 0x55d0b54c8280 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8d540c640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe8bc0b3d30 0x7fe8bc0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8cffff640 1 --2- 172.21.15.29:0/2142033215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 0x55d0b588ee20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0009fb0 tx=0x7fe8c00098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eeec6c6e0bf14f8 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.571+0000 7fe8cdffb640 1 -- 172.21.15.29:0/829972256 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fe8c80e38a0 con 0x7fe8d0109830 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.572+0000 7fe8c6ffd640 1 -- 172.21.15.29:0/2142033215 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c0010af0 con 0x55d0b54c8280 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.572+0000 7fe8c6ffd640 1 -- 172.21.15.29:0/2142033215 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8c0010c90 con 0x55d0b54c8280 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.572+0000 7fe8c6ffd640 1 -- 172.21.15.29:0/2142033215 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c000c390 con 0x55d0b54c8280 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.573+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2142033215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 msgr2=0x55d0b588ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.573+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/2142033215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 0x55d0b588ee20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0009fb0 tx=0x7fe8c00098f0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.573+0000 7fe8d540c640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe8bc0b3d30 0x7fe8bc0b6240 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80aa510 tx=0x7fe8c8077cd0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.573+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2142033215 shutdown_connections 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.573+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/2142033215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 0x55d0b588ee20 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.573+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/2142033215 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54c00a0 0x55d0b588e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.573+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/2142033215 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54e8f60 0x55d0b588e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.573+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2142033215 >> 172.21.15.29:0/2142033215 conn(0x55d0b54dd350 msgr2=0x55d0b588c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:36.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.573+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2142033215 shutdown_connections 2024-07-27T06:47:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.573+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2142033215 wait complete. 2024-07-27T06:47:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.573+0000 7fe8dd3f3740 1 -- start start 2024-07-27T06:47:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.573+0000 7fe8dd3f3740 1 Processor -- start 2024-07-27T06:47:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8dd3f3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 0x55d0b5899fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8dd3f3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54c8280 0x55d0b589a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8dd3f3740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54e8f60 0x55d0b58a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8dd3f3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d0b56575f0 con 0x55d0b54c00a0 2024-07-27T06:47:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8dd3f3740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55d0b57968b0 con 0x55d0b54e8f60 2024-07-27T06:47:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8dd3f3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d0b589cb80 con 0x55d0b54c8280 2024-07-27T06:47:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8dd3f3740 10 client.0 _ll_register_callbacks cb 0x55d0b57a07f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 0x55d0b5899fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 0x55d0b5899fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45680/0 (socket says 172.21.15.29:45680) 2024-07-27T06:47:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8cffff640 1 -- 172.21.15.29:0/2707717758 learned_addr learned my addr 172.21.15.29:0/2707717758 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8d4c0b640 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54c8280 0x55d0b589a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8cffff640 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54e8f60 0x55d0b58a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8cffff640 1 -- 172.21.15.29:0/2707717758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54c8280 msgr2=0x55d0b589a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8cffff640 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54c8280 0x55d0b589a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8cffff640 1 -- 172.21.15.29:0/2707717758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54e8f60 msgr2=0x55d0b58a5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8cffff640 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54e8f60 0x55d0b58a5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.574+0000 7fe8cffff640 1 -- 172.21.15.29:0/2707717758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0b5554540 con 0x55d0b54c00a0 2024-07-27T06:47:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.575+0000 7fe8cffff640 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 0x55d0b5899fb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0002230 tx=0x7fe8c001dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef51600dd1b0b5aa server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.575+0000 7fe8affff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.575+0000 7fe8affff640 1 -- 172.21.15.29:0/2707717758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c0004c40 con 0x55d0b54c00a0 2024-07-27T06:47:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.575+0000 7fe8affff640 1 -- 172.21.15.29:0/2707717758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fe8c0004de0 con 0x55d0b54c00a0 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.575+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2707717758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8c0010c90 con 0x55d0b54c00a0 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.575+0000 7fe8affff640 1 -- 172.21.15.29:0/2707717758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe8c0035050 con 0x55d0b54c00a0 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.575+0000 7fe8dd3f3740 10 client.15642 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.575+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2707717758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d0b58a8be0 con 0x55d0b54c00a0 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.575+0000 7fe8affff640 1 -- 172.21.15.29:0/2707717758 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8c0010c90 con 0x55d0b54c00a0 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.576+0000 7fe8affff640 1 -- 172.21.15.29:0/2707717758 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe8c002c070 con 0x55d0b54c00a0 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.576+0000 7fe8dd3f3740 10 client.15642 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.576+0000 7fe8dd3f3740 10 client.15642 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.576+0000 7fe8dd3f3740 20 client.15642 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.576+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55d0b54a78e0 0x55d0b58ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.576+0000 7fe8dd3f3740 4 client.15642 mds_command: new command op to 24617 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.576+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2707717758 --> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0b5806890 con 0x55d0b54a78e0 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.576+0000 7fe8d4c0b640 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55d0b54a78e0 0x55d0b58ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.577+0000 7fe8d4c0b640 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55d0b54a78e0 0x55d0b58ac390 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=beb20d4a3c47c7d5 server_cookie=15b138a57537683e in_seq=0 out_seq=0 2024-07-27T06:47:36.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.577+0000 7fe8affff640 10 client.15642 ms_handle_connect on v2:172.21.15.47:6838/670107266 2024-07-27T06:47:36.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.577+0000 7fe8affff640 1 -- 172.21.15.29:0/2707717758 <== mds.3 v2:172.21.15.47:6838/670107266 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d0b5806890 con 0x55d0b54a78e0 2024-07-27T06:47:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.577+0000 7fe8affff640 10 client.15642 handle_command_reply: tid=0 2024-07-27T06:47:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.577+0000 7fe8dd3f3740 1 client.15642 shutdown 2024-07-27T06:47:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.578+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2707717758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 msgr2=0x55d0b5899fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.578+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 0x55d0b5899fb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0002230 tx=0x7fe8c001dbe0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.578+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2707717758 shutdown_connections 2024-07-27T06:47:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.578+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55d0b54a78e0 0x55d0b58ac390 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.578+0000 7fe8affff640 0 client.15642 ms_handle_reset on v2:172.21.15.47:6838/670107266 2024-07-27T06:47:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.578+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54e8f60 0x55d0b58a5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.578+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54c8280 0x55d0b589a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.578+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/2707717758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 0x55d0b5899fb0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.578+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2707717758 >> 172.21.15.29:0/2707717758 conn(0x55d0b54dd350 msgr2=0x55d0b5891530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:36.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.578+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2707717758 shutdown_connections 2024-07-27T06:47:36.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.578+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/2707717758 wait complete. 2024-07-27T06:47:36.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.578+0000 7fe8dd3f3740 20 client.15642 trim_cache size 0 max 16384 2024-07-27T06:47:36.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.596+0000 7fe8dd3f3740 1 Processor -- start 2024-07-27T06:47:36.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.596+0000 7fe8dd3f3740 1 -- start start 2024-07-27T06:47:36.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.596+0000 7fe8dd3f3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54a78e0 0x55d0b58ac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.596+0000 7fe8dd3f3740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54c00a0 0x55d0b588feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.596+0000 7fe8dd3f3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 0x55d0b58a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.596+0000 7fe8dd3f3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d0b58aa6d0 con 0x55d0b54c8280 2024-07-27T06:47:36.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.596+0000 7fe8dd3f3740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55d0b58aada0 con 0x55d0b54c00a0 2024-07-27T06:47:36.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.596+0000 7fe8dd3f3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d0b58ab470 con 0x55d0b54a78e0 2024-07-27T06:47:36.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.596+0000 7fe8cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 0x55d0b58a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.596+0000 7fe8cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 0x55d0b58a5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45690/0 (socket says 172.21.15.29:45690) 2024-07-27T06:47:36.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.596+0000 7fe8cffff640 1 -- 172.21.15.29:0/935427484 learned_addr learned my addr 172.21.15.29:0/935427484 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.597+0000 7fe8d4c0b640 1 --2- 172.21.15.29:0/935427484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54c00a0 0x55d0b588feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.597+0000 7fe8cffff640 1 --2- 172.21.15.29:0/935427484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54a78e0 0x55d0b58ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.597+0000 7fe8cffff640 1 -- 172.21.15.29:0/935427484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54a78e0 msgr2=0x55d0b58ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.597+0000 7fe8cffff640 1 --2- 172.21.15.29:0/935427484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54a78e0 0x55d0b58ac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.597+0000 7fe8cffff640 1 -- 172.21.15.29:0/935427484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54c00a0 msgr2=0x55d0b588feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.597+0000 7fe8cffff640 1 --2- 172.21.15.29:0/935427484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54c00a0 0x55d0b588feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.597+0000 7fe8cffff640 1 -- 172.21.15.29:0/935427484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0b5891200 con 0x55d0b54c8280 2024-07-27T06:47:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.597+0000 7fe8d4c0b640 1 --2- 172.21.15.29:0/935427484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54c00a0 0x55d0b588feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.597+0000 7fe8cffff640 1 --2- 172.21.15.29:0/935427484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 0x55d0b58a5de0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0005790 tx=0x7fe8c0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3f481cf8320bf5 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.597+0000 7fe8c67fc640 1 -- 172.21.15.29:0/935427484 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c0005810 con 0x55d0b54c8280 2024-07-27T06:47:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.597+0000 7fe8c67fc640 1 -- 172.21.15.29:0/935427484 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8c0023070 con 0x55d0b54c8280 2024-07-27T06:47:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.597+0000 7fe8c67fc640 1 -- 172.21.15.29:0/935427484 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c00052d0 con 0x55d0b54c8280 2024-07-27T06:47:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.598+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/935427484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 msgr2=0x55d0b58a5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.598+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/935427484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 0x55d0b58a5de0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0005790 tx=0x7fe8c0009f80 comp rx=0 tx=0).stop 2024-07-27T06:47:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.598+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/935427484 shutdown_connections 2024-07-27T06:47:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.598+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/935427484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c8280 0x55d0b58a5de0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.598+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/935427484 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54c00a0 0x55d0b588feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.598+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/935427484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54a78e0 0x55d0b58ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.598+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/935427484 >> 172.21.15.29:0/935427484 conn(0x55d0b54dd350 msgr2=0x55d0b5894ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.598+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/935427484 shutdown_connections 2024-07-27T06:47:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.598+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/935427484 wait complete. 2024-07-27T06:47:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.598+0000 7fe8dd3f3740 1 -- start start 2024-07-27T06:47:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.598+0000 7fe8dd3f3740 1 Processor -- start 2024-07-27T06:47:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8dd3f3740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54a78e0 0x55d0b58a5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8dd3f3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 0x55d0b58a5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8dd3f3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54c8280 0x55d0b58b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8dd3f3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d0b58ac210 con 0x55d0b54c00a0 2024-07-27T06:47:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8dd3f3740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55d0b58abb40 con 0x55d0b54a78e0 2024-07-27T06:47:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8dd3f3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d0b58ac8e0 con 0x55d0b54c8280 2024-07-27T06:47:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8dd3f3740 10 client.0 _ll_register_callbacks cb 0x7fe8c0010ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8d4c0b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 0x55d0b58a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8d4c0b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 0x55d0b58a5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45692/0 (socket says 172.21.15.29:45692) 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8d4c0b640 1 -- 172.21.15.29:0/3564570921 learned_addr learned my addr 172.21.15.29:0/3564570921 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8cffff640 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54a78e0 0x55d0b58a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8cffff640 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54c8280 0x55d0b58b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8d4c0b640 1 -- 172.21.15.29:0/3564570921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54c8280 msgr2=0x55d0b58b2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8d4c0b640 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54c8280 0x55d0b58b2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8d4c0b640 1 -- 172.21.15.29:0/3564570921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54a78e0 msgr2=0x55d0b58a5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8d4c0b640 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54a78e0 0x55d0b58a5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8d4c0b640 1 -- 172.21.15.29:0/3564570921 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0b58b3530 con 0x55d0b54c00a0 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.599+0000 7fe8cffff640 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54a78e0 0x55d0b58a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.600+0000 7fe8d4c0b640 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 0x55d0b58a5640 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe8b802a450 tx=0x7fe8b80041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ece50c1baf761cd server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.600+0000 7fe8affff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.600+0000 7fe8affff640 1 -- 172.21.15.29:0/3564570921 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b8048020 con 0x55d0b54c00a0 2024-07-27T06:47:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.600+0000 7fe8affff640 1 -- 172.21.15.29:0/3564570921 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fe8b8005240 con 0x55d0b54c00a0 2024-07-27T06:47:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.600+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/3564570921 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8c0023070 con 0x55d0b54c00a0 2024-07-27T06:47:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.600+0000 7fe8affff640 1 -- 172.21.15.29:0/3564570921 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe8b8053050 con 0x55d0b54c00a0 2024-07-27T06:47:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.600+0000 7fe8dd3f3740 10 client.15648 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.600+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/3564570921 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d0b58b4950 con 0x55d0b54c00a0 2024-07-27T06:47:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.600+0000 7fe8affff640 1 -- 172.21.15.29:0/3564570921 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8c0023070 con 0x55d0b54c00a0 2024-07-27T06:47:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.600+0000 7fe8affff640 1 -- 172.21.15.29:0/3564570921 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe8b80035f0 con 0x55d0b54c00a0 2024-07-27T06:47:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.601+0000 7fe8dd3f3740 10 client.15648 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.601+0000 7fe8dd3f3740 10 client.15648 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-27T06:47:36.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.601+0000 7fe8dd3f3740 20 client.15648 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.601+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55d0b54e8f60 0x55d0b5918410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.601+0000 7fe8dd3f3740 4 client.15648 mds_command: new command op to 24617 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:47:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.601+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/3564570921 --> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] -- command(tid 0: {"prefix": "status"}) -- 0x55d0b5802330 con 0x55d0b54e8f60 2024-07-27T06:47:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.601+0000 7fe8cffff640 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55d0b54e8f60 0x55d0b5918410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.601+0000 7fe8cffff640 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55d0b54e8f60 0x55d0b5918410 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f50392fb2742bb03 server_cookie=445697da10bdbfb6 in_seq=0 out_seq=0 2024-07-27T06:47:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.601+0000 7fe8affff640 10 client.15648 ms_handle_connect on v2:172.21.15.47:6838/670107266 2024-07-27T06:47:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.601+0000 7fe8affff640 1 -- 172.21.15.29:0/3564570921 <== mds.3 v2:172.21.15.47:6838/670107266 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d0b5802330 con 0x55d0b54e8f60 2024-07-27T06:47:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8affff640 10 client.15648 handle_command_reply: tid=0 2024-07-27T06:47:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8dd3f3740 1 client.15648 shutdown 2024-07-27T06:47:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/3564570921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 msgr2=0x55d0b58a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 0x55d0b58a5640 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe8b802a450 tx=0x7fe8b80041f0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/3564570921 shutdown_connections 2024-07-27T06:47:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x55d0b54e8f60 0x55d0b5918410 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d0b54c8280 0x55d0b58b2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d0b54c00a0 0x55d0b58a5640 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8dd3f3740 1 --2- 172.21.15.29:0/3564570921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d0b54a78e0 0x55d0b58a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8affff640 0 client.15648 ms_handle_reset on v2:172.21.15.47:6838/670107266 2024-07-27T06:47:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/3564570921 >> 172.21.15.29:0/3564570921 conn(0x55d0b54dd350 msgr2=0x55d0b58914a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/3564570921 shutdown_connections 2024-07-27T06:47:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8dd3f3740 1 -- 172.21.15.29:0/3564570921 wait complete. 2024-07-27T06:47:36.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.602+0000 7fe8dd3f3740 20 client.15648 trim_cache size 0 max 16384 2024-07-27T06:47:36.609 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:47:36.610 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:47:36.610 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 3, 2024-07-27T06:47:36.610 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24617, 2024-07-27T06:47:36.610 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:47:36.610 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:47:36.610 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:47:36.610 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 244.74940873599999, 2024-07-27T06:47:36.610 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 29, 2024-07-27T06:47:36.610 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:47:36.610 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 83, 2024-07-27T06:47:36.610 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 279.89604017300002 2024-07-27T06:47:36.610 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:47:36.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.603+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/829972256 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe8bc0b3d30 msgr2=0x7fe8bc0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:36.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.603+0000 7fe8d6e6f640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe8bc0b3d30 0x7fe8bc0b6240 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80aa510 tx=0x7fe8c8077cd0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.603+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/829972256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d0109830 msgr2=0x7fe8d0075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:36.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.603+0000 7fe8d6e6f640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d0109830 0x7fe8d0075bc0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800d750 tx=0x7fe8c800dc10 comp rx=0 tx=0).stop 2024-07-27T06:47:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.603+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/829972256 shutdown_connections 2024-07-27T06:47:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.603+0000 7fe8d6e6f640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8d0109830 0x7fe8d0075bc0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.603+0000 7fe8d6e6f640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe8bc0b3d30 0x7fe8bc0b6240 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.603+0000 7fe8d6e6f640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8d0108ee0 0x7fe8d0075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.603+0000 7fe8d6e6f640 1 --2- 172.21.15.29:0/829972256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe8d01035a0 0x7fe8d0075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.603+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/829972256 >> 172.21.15.29:0/829972256 conn(0x7fe8d00ff010 msgr2=0x7fe8d0101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.603+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/829972256 shutdown_connections 2024-07-27T06:47:36.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:36.603+0000 7fe8d6e6f640 1 -- 172.21.15.29:0/829972256 wait complete. 2024-07-27T06:47:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:36 smithi029 ceph-mon[32610]: pgmap v415: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-07-27T06:47:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:36 smithi077 ceph-mon[37153]: pgmap v415: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-07-27T06:47:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:36 smithi047 ceph-mon[36835]: pgmap v415: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-07-27T06:47:37.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.f status 2024-07-27T06:47:37.596 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:39.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:38 smithi029 ceph-mon[32610]: pgmap v416: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.8 KiB/s wr, 14 op/s 2024-07-27T06:47:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:38 smithi077 ceph-mon[37153]: pgmap v416: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.8 KiB/s wr, 14 op/s 2024-07-27T06:47:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:38 smithi047 ceph-mon[36835]: pgmap v416: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.8 KiB/s wr, 14 op/s 2024-07-27T06:47:39.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.425+0000 7ff37895a640 1 -- 172.21.15.29:0/1723688675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff374102490 msgr2=0x7ff3741028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:39.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.425+0000 7ff37895a640 1 --2- 172.21.15.29:0/1723688675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff374102490 0x7ff3741028a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff364009900 tx=0x7ff36402a410 comp rx=0 tx=0).stop 2024-07-27T06:47:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.425+0000 7ff37895a640 1 -- 172.21.15.29:0/1723688675 shutdown_connections 2024-07-27T06:47:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.425+0000 7ff37895a640 1 --2- 172.21.15.29:0/1723688675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff374108720 0x7ff37410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.425+0000 7ff37895a640 1 --2- 172.21.15.29:0/1723688675 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff374107dd0 0x7ff3741081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.425+0000 7ff37895a640 1 --2- 172.21.15.29:0/1723688675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff374102490 0x7ff3741028a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.425+0000 7ff37895a640 1 -- 172.21.15.29:0/1723688675 >> 172.21.15.29:0/1723688675 conn(0x7ff374079160 msgr2=0x7ff374079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.425+0000 7ff37895a640 1 -- 172.21.15.29:0/1723688675 shutdown_connections 2024-07-27T06:47:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.425+0000 7ff37895a640 1 -- 172.21.15.29:0/1723688675 wait complete. 2024-07-27T06:47:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.426+0000 7ff37895a640 1 Processor -- start 2024-07-27T06:47:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.426+0000 7ff37895a640 1 -- start start 2024-07-27T06:47:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.426+0000 7ff37895a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff374102490 0x7ff37419c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.426+0000 7ff37895a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff374107dd0 0x7ff37419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.426+0000 7ff37895a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff374108720 0x7ff3741a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.426+0000 7ff37895a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff374112380 con 0x7ff374107dd0 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.426+0000 7ff37895a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7ff374112200 con 0x7ff374102490 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.426+0000 7ff37895a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff374112500 con 0x7ff374108720 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.426+0000 7ff371d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff374107dd0 0x7ff37419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.426+0000 7ff371d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff374107dd0 0x7ff37419c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50902/0 (socket says 172.21.15.29:50902) 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.426+0000 7ff371d9b640 1 -- 172.21.15.29:0/699320265 learned_addr learned my addr 172.21.15.29:0/699320265 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff37259c640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff374102490 0x7ff37419c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff372d9d640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff374108720 0x7ff3741a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff371d9b640 1 -- 172.21.15.29:0/699320265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff374108720 msgr2=0x7ff3741a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff371d9b640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff374108720 0x7ff3741a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff371d9b640 1 -- 172.21.15.29:0/699320265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff374102490 msgr2=0x7ff37419c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff371d9b640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff374102490 0x7ff37419c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff371d9b640 1 -- 172.21.15.29:0/699320265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3741a2080 con 0x7ff374107dd0 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff371d9b640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff374107dd0 0x7ff37419c6b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff36800ccb0 tx=0x7ff36800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff3637fe640 1 -- 172.21.15.29:0/699320265 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff368004b20 con 0x7ff374107dd0 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff37895a640 1 -- 172.21.15.29:0/699320265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3741a2370 con 0x7ff374107dd0 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff3637fe640 1 -- 172.21.15.29:0/699320265 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff368004cc0 con 0x7ff374107dd0 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff3637fe640 1 -- 172.21.15.29:0/699320265 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36800f690 con 0x7ff374107dd0 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.427+0000 7ff37895a640 1 -- 172.21.15.29:0/699320265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3741a8ad0 con 0x7ff374107dd0 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.430+0000 7ff37eede740 1 Processor -- start 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.430+0000 7ff37eede740 1 -- start start 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.430+0000 7ff3637fe640 1 -- 172.21.15.29:0/699320265 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7ff36800f8f0 con 0x7ff374107dd0 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff37eede740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 0x561ae92ea380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff37eede740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae92ea8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff37eede740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae92eae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff37eede740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561ae9260920 con 0x561ae8f44f60 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff37eede740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x561ae924fe20 con 0x561ae8f1c0a0 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff37eede740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ae9253e60 con 0x561ae8f24280 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff372d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 0x561ae92ea380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff372d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 0x561ae92ea380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50918/0 (socket says 172.21.15.29:50918) 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff372d9d640 1 -- 172.21.15.29:0/2625301564 learned_addr learned my addr 172.21.15.29:0/2625301564 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff37259c640 1 --2- 172.21.15.29:0/2625301564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae92ea8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff371d9b640 1 --2- 172.21.15.29:0/2625301564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae92eae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff372d9d640 1 -- 172.21.15.29:0/2625301564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 msgr2=0x561ae92eae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff372d9d640 1 --2- 172.21.15.29:0/2625301564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae92eae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff372d9d640 1 -- 172.21.15.29:0/2625301564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 msgr2=0x561ae92ea8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff372d9d640 1 --2- 172.21.15.29:0/2625301564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae92ea8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff372d9d640 1 -- 172.21.15.29:0/2625301564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ae91f6a40 con 0x561ae8f44f60 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff371d9b640 1 --2- 172.21.15.29:0/2625301564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae92eae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.431+0000 7ff3637fe640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff35c0b3cb0 0x7ff35c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.432+0000 7ff372d9d640 1 --2- 172.21.15.29:0/2625301564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 0x561ae92ea380 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00ab00 tx=0x7ff36c00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c34eae89b833c11 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.432+0000 7ff37259c640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff35c0b3cb0 0x7ff35c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.432+0000 7ff3637fe640 1 -- 172.21.15.29:0/699320265 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7ff3680e4280 con 0x7ff374107dd0 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.432+0000 7ff360ff9640 1 -- 172.21.15.29:0/2625301564 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c01d070 con 0x561ae8f44f60 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.432+0000 7ff360ff9640 1 -- 172.21.15.29:0/2625301564 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff36c0188f0 con 0x561ae8f44f60 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.432+0000 7ff360ff9640 1 -- 172.21.15.29:0/2625301564 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c0109d0 con 0x561ae8f44f60 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.433+0000 7ff37eede740 1 -- 172.21.15.29:0/2625301564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 msgr2=0x561ae92ea380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.433+0000 7ff37eede740 1 --2- 172.21.15.29:0/2625301564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 0x561ae92ea380 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00ab00 tx=0x7ff36c00f3c0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.433+0000 7ff37259c640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff35c0b3cb0 0x7ff35c0b61c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff364031440 tx=0x7ff364009650 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.433+0000 7ff37eede740 1 -- 172.21.15.29:0/2625301564 shutdown_connections 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.433+0000 7ff37eede740 1 --2- 172.21.15.29:0/2625301564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae92eae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.433+0000 7ff37eede740 1 --2- 172.21.15.29:0/2625301564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae92ea8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.433+0000 7ff37eede740 1 --2- 172.21.15.29:0/2625301564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 0x561ae92ea380 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.433+0000 7ff37eede740 1 -- 172.21.15.29:0/2625301564 >> 172.21.15.29:0/2625301564 conn(0x561ae8f39350 msgr2=0x561ae92e8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.433+0000 7ff37eede740 1 -- 172.21.15.29:0/2625301564 shutdown_connections 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.433+0000 7ff37eede740 1 -- 172.21.15.29:0/2625301564 wait complete. 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.433+0000 7ff37eede740 1 -- start start 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.433+0000 7ff37eede740 1 Processor -- start 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff37eede740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae92f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff37eede740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae92f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff37eede740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 0x561ae9301e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff37eede740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561ae90b35f0 con 0x561ae8f44f60 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff37eede740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x561ae91f2890 con 0x561ae8f1c0a0 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff37eede740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ae92f8b60 con 0x561ae8f24280 2024-07-27T06:47:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff37eede740 10 client.0 _ll_register_callbacks cb 0x561ae91fc7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff372d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 0x561ae9301e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff372d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 0x561ae9301e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50920/0 (socket says 172.21.15.29:50920) 2024-07-27T06:47:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff372d9d640 1 -- 172.21.15.29:0/1795526946 learned_addr learned my addr 172.21.15.29:0/1795526946 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff371d9b640 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae92f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff37259c640 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae92f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff372d9d640 1 -- 172.21.15.29:0/1795526946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 msgr2=0x561ae92f64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff372d9d640 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae92f64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff372d9d640 1 -- 172.21.15.29:0/1795526946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 msgr2=0x561ae92f5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff372d9d640 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae92f5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff372d9d640 1 -- 172.21.15.29:0/1795526946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ae8fb0540 con 0x561ae8f44f60 2024-07-27T06:47:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff37259c640 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae92f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff372d9d640 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 0x561ae9301e50 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00a6d0 tx=0x7ff36c010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50dd19d08423538 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.434+0000 7ff359ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff359ffb640 1 -- 172.21.15.29:0/1795526946 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c01d050 con 0x561ae8f44f60 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff359ffb640 1 -- 172.21.15.29:0/1795526946 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7ff36c005030 con 0x561ae8f44f60 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff37eede740 1 -- 172.21.15.29:0/1795526946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff36c0188f0 con 0x561ae8f44f60 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff359ffb640 1 -- 172.21.15.29:0/1795526946 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff36c031050 con 0x561ae8f44f60 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff37eede740 10 client.15660 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff37eede740 1 -- 172.21.15.29:0/1795526946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561ae9304b40 con 0x561ae8f44f60 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff359ffb640 1 -- 172.21.15.29:0/1795526946 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff36c0188f0 con 0x561ae8f44f60 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff359ffb640 1 -- 172.21.15.29:0/1795526946 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff36c02c030 con 0x561ae8f44f60 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff37eede740 10 client.15660 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff37eede740 10 client.15660 resolve_mds: resolved name 'f' to daemon mds.f 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff37eede740 20 client.15660 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff37eede740 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x561ae8f038e0 0x561ae9308230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff37eede740 4 client.15660 mds_command: new command op to 24646 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.435+0000 7ff37eede740 1 -- 172.21.15.29:0/1795526946 --> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ae9262870 con 0x561ae8f038e0 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.436+0000 7ff37259c640 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x561ae8f038e0 0x561ae9308230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.436+0000 7ff359ffb640 10 client.15660 ms_handle_connect on v2:172.21.15.77:6834/439792826 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.436+0000 7ff37259c640 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x561ae8f038e0 0x561ae9308230 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bacdaaa7bdca195e server_cookie=b11fef590ab87d9a in_seq=0 out_seq=0 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff359ffb640 1 -- 172.21.15.29:0/1795526946 <== mds.2 v2:172.21.15.77:6834/439792826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561ae9262870 con 0x561ae8f038e0 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff359ffb640 10 client.15660 handle_command_reply: tid=0 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff37eede740 1 client.15660 shutdown 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff37eede740 1 -- 172.21.15.29:0/1795526946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 msgr2=0x561ae9301e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff37eede740 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 0x561ae9301e50 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00a6d0 tx=0x7ff36c010430 comp rx=0 tx=0).stop 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff37eede740 1 -- 172.21.15.29:0/1795526946 shutdown_connections 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff37eede740 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x561ae8f038e0 0x561ae9308230 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff37eede740 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f44f60 0x561ae9301e50 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff359ffb640 0 client.15660 ms_handle_reset on v2:172.21.15.77:6834/439792826 2024-07-27T06:47:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff37eede740 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae92f64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff37eede740 1 --2- 172.21.15.29:0/1795526946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae92f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff37eede740 1 -- 172.21.15.29:0/1795526946 >> 172.21.15.29:0/1795526946 conn(0x561ae8f39350 msgr2=0x561ae92ed560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff37eede740 1 -- 172.21.15.29:0/1795526946 shutdown_connections 2024-07-27T06:47:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.437+0000 7ff37eede740 1 -- 172.21.15.29:0/1795526946 wait complete. 2024-07-27T06:47:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.438+0000 7ff37eede740 20 client.15660 trim_cache size 0 max 16384 2024-07-27T06:47:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff37eede740 1 Processor -- start 2024-07-27T06:47:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff37eede740 1 -- start start 2024-07-27T06:47:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff37eede740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f038e0 0x561ae9307490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff37eede740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae9301dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff37eede740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f24280 0x561ae92e7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff37eede740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561ae9306570 con 0x561ae8f24280 2024-07-27T06:47:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff37eede740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x561ae9306c40 con 0x561ae8f1c0a0 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff37eede740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ae9307310 con 0x561ae8f038e0 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff372d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f24280 0x561ae92e7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff372d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f24280 0x561ae92e7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50934/0 (socket says 172.21.15.29:50934) 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff372d9d640 1 -- 172.21.15.29:0/1689186386 learned_addr learned my addr 172.21.15.29:0/1689186386 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff371d9b640 1 --2- 172.21.15.29:0/1689186386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae9301dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff37259c640 1 --2- 172.21.15.29:0/1689186386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f038e0 0x561ae9307490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff372d9d640 1 -- 172.21.15.29:0/1689186386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f038e0 msgr2=0x561ae9307490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff372d9d640 1 --2- 172.21.15.29:0/1689186386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f038e0 0x561ae9307490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff372d9d640 1 -- 172.21.15.29:0/1689186386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 msgr2=0x561ae9301dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff372d9d640 1 --2- 172.21.15.29:0/1689186386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae9301dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.456+0000 7ff372d9d640 1 -- 172.21.15.29:0/1689186386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ae92ed1e0 con 0x561ae8f24280 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.457+0000 7ff37259c640 1 --2- 172.21.15.29:0/1689186386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f038e0 0x561ae9307490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.457+0000 7ff372d9d640 1 --2- 172.21.15.29:0/1689186386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f24280 0x561ae92e7840 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00f940 tx=0x7ff36c00a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dd51fbd0efbd3 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.457+0000 7ff35bfff640 1 -- 172.21.15.29:0/1689186386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c03b070 con 0x561ae8f24280 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.457+0000 7ff35bfff640 1 -- 172.21.15.29:0/1689186386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff36c01e040 con 0x561ae8f24280 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.457+0000 7ff35bfff640 1 -- 172.21.15.29:0/1689186386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c01d050 con 0x561ae8f24280 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.458+0000 7ff37eede740 1 -- 172.21.15.29:0/1689186386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f24280 msgr2=0x561ae92e7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.458+0000 7ff37eede740 1 --2- 172.21.15.29:0/1689186386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f24280 0x561ae92e7840 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00f940 tx=0x7ff36c00a4c0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.458+0000 7ff37eede740 1 -- 172.21.15.29:0/1689186386 shutdown_connections 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.458+0000 7ff37eede740 1 --2- 172.21.15.29:0/1689186386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f24280 0x561ae92e7840 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.458+0000 7ff37eede740 1 --2- 172.21.15.29:0/1689186386 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae9301dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.458+0000 7ff37eede740 1 --2- 172.21.15.29:0/1689186386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f038e0 0x561ae9307490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.458+0000 7ff37eede740 1 -- 172.21.15.29:0/1689186386 >> 172.21.15.29:0/1689186386 conn(0x561ae8f39350 msgr2=0x561ae92f45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.458+0000 7ff37eede740 1 -- 172.21.15.29:0/1689186386 shutdown_connections 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.458+0000 7ff37eede740 1 -- 172.21.15.29:0/1689186386 wait complete. 2024-07-27T06:47:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.458+0000 7ff37eede740 1 -- start start 2024-07-27T06:47:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.458+0000 7ff37eede740 1 Processor -- start 2024-07-27T06:47:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37eede740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f038e0 0x561ae930eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37eede740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae930f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37eede740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae9309860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37eede740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561ae93080b0 con 0x561ae8f038e0 2024-07-27T06:47:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37eede740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x561ae93079e0 con 0x561ae8f1c0a0 2024-07-27T06:47:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37eede740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ae9308780 con 0x561ae8f24280 2024-07-27T06:47:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37eede740 10 client.0 _ll_register_callbacks cb 0x7ff36c00a950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f038e0 0x561ae930eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f038e0 0x561ae930eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50942/0 (socket says 172.21.15.29:50942) 2024-07-27T06:47:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37259c640 1 -- 172.21.15.29:0/4096028356 learned_addr learned my addr 172.21.15.29:0/4096028356 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff372d9d640 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae9309860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff371d9b640 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae930f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37259c640 1 -- 172.21.15.29:0/4096028356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 msgr2=0x561ae9309860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37259c640 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae9309860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37259c640 1 -- 172.21.15.29:0/4096028356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 msgr2=0x561ae930f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37259c640 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae930f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37259c640 1 -- 172.21.15.29:0/4096028356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ae930fca0 con 0x561ae8f038e0 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff37259c640 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f038e0 0x561ae930eee0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff364009900 tx=0x7ff364009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45bbd3dac5b55ac9 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.459+0000 7ff359ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff359ffb640 1 -- 172.21.15.29:0/4096028356 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff364054030 con 0x561ae8f038e0 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff37eede740 1 -- 172.21.15.29:0/4096028356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff36c01e040 con 0x561ae8f038e0 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff359ffb640 1 -- 172.21.15.29:0/4096028356 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7ff3640035f0 con 0x561ae8f038e0 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff359ffb640 1 -- 172.21.15.29:0/4096028356 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff364035040 con 0x561ae8f038e0 2024-07-27T06:47:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff37eede740 10 client.15666 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff37eede740 1 -- 172.21.15.29:0/4096028356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561ae9310480 con 0x561ae8f038e0 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff359ffb640 1 -- 172.21.15.29:0/4096028356 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff36c01e040 con 0x561ae8f038e0 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff359ffb640 1 -- 172.21.15.29:0/4096028356 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff3640628b0 con 0x561ae8f038e0 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff37eede740 10 client.15666 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff37eede740 10 client.15666 resolve_mds: resolved name 'f' to daemon mds.f 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff37eede740 20 client.15666 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff37eede740 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x561ae8f44f60 0x561ae93742d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff37eede740 4 client.15666 mds_command: new command op to 24646 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.460+0000 7ff37eede740 1 -- 172.21.15.29:0/4096028356 --> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] -- command(tid 0: {"prefix": "status"}) -- 0x561ae925e310 con 0x561ae8f44f60 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.461+0000 7ff371d9b640 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x561ae8f44f60 0x561ae93742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.461+0000 7ff359ffb640 10 client.15666 ms_handle_connect on v2:172.21.15.77:6834/439792826 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.461+0000 7ff371d9b640 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x561ae8f44f60 0x561ae93742d0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cc533728254acba3 server_cookie=e97dcb85dd862860 in_seq=0 out_seq=0 2024-07-27T06:47:39.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.461+0000 7ff359ffb640 1 -- 172.21.15.29:0/4096028356 <== mds.2 v2:172.21.15.77:6834/439792826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561ae925e310 con 0x561ae8f44f60 2024-07-27T06:47:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.461+0000 7ff359ffb640 10 client.15666 handle_command_reply: tid=0 2024-07-27T06:47:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.461+0000 7ff37eede740 1 client.15666 shutdown 2024-07-27T06:47:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37eede740 1 -- 172.21.15.29:0/4096028356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f038e0 msgr2=0x561ae930eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37eede740 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f038e0 0x561ae930eee0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff364009900 tx=0x7ff364009260 comp rx=0 tx=0).stop 2024-07-27T06:47:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37eede740 1 -- 172.21.15.29:0/4096028356 shutdown_connections 2024-07-27T06:47:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37eede740 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x561ae8f44f60 0x561ae93742d0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37eede740 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ae8f24280 0x561ae9309860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37eede740 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ae8f1c0a0 0x561ae930f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37eede740 1 --2- 172.21.15.29:0/4096028356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ae8f038e0 0x561ae930eee0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff359ffb640 0 client.15666 ms_handle_reset on v2:172.21.15.77:6834/439792826 2024-07-27T06:47:39.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37eede740 1 -- 172.21.15.29:0/4096028356 >> 172.21.15.29:0/4096028356 conn(0x561ae8f39350 msgr2=0x561ae92f5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:39.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37eede740 1 -- 172.21.15.29:0/4096028356 shutdown_connections 2024-07-27T06:47:39.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37eede740 1 -- 172.21.15.29:0/4096028356 wait complete. 2024-07-27T06:47:39.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37eede740 20 client.15666 trim_cache size 0 max 16384 2024-07-27T06:47:39.473 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:47:39.473 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:47:39.473 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 2, 2024-07-27T06:47:39.473 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24646, 2024-07-27T06:47:39.473 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:47:39.473 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:47:39.474 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:47:39.474 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 248.61938264200001, 2024-07-27T06:47:39.474 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 29, 2024-07-27T06:47:39.474 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:47:39.474 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 83, 2024-07-27T06:47:39.474 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 300.39181327599999 2024-07-27T06:47:39.474 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:47:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37895a640 1 -- 172.21.15.29:0/699320265 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff35c0b3cb0 msgr2=0x7ff35c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37895a640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff35c0b3cb0 0x7ff35c0b61c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff364031440 tx=0x7ff364009650 comp rx=0 tx=0).stop 2024-07-27T06:47:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37895a640 1 -- 172.21.15.29:0/699320265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff374107dd0 msgr2=0x7ff37419c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37895a640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff374107dd0 0x7ff37419c6b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff36800ccb0 tx=0x7ff36800a620 comp rx=0 tx=0).stop 2024-07-27T06:47:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37895a640 1 -- 172.21.15.29:0/699320265 shutdown_connections 2024-07-27T06:47:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37895a640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff35c0b3cb0 0x7ff35c0b61c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37895a640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff374108720 0x7ff3741a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37895a640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff374107dd0 0x7ff37419c6b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37895a640 1 --2- 172.21.15.29:0/699320265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff374102490 0x7ff37419c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.462+0000 7ff37895a640 1 -- 172.21.15.29:0/699320265 >> 172.21.15.29:0/699320265 conn(0x7ff374079160 msgr2=0x7ff37406a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:39.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.463+0000 7ff37895a640 1 -- 172.21.15.29:0/699320265 shutdown_connections 2024-07-27T06:47:39.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:39.463+0000 7ff37895a640 1 -- 172.21.15.29:0/699320265 wait complete. 2024-07-27T06:47:40.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-27T06:47:40.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.c status 2024-07-27T06:47:40.376 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:40 smithi077 ceph-mon[37153]: pgmap v417: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-07-27T06:47:41.128 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:40 smithi047 ceph-mon[36835]: pgmap v417: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-07-27T06:47:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:40 smithi029 ceph-mon[32610]: pgmap v417: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-07-27T06:47:43.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.004+0000 7fcccc206640 1 -- 172.21.15.29:0/411547705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc4102420 msgr2=0x7fccc4102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.004+0000 7fcccc206640 1 --2- 172.21.15.29:0/411547705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc4102420 0x7fccc4102830 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009ca0 tx=0x7fccc002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.004+0000 7fcccc206640 1 -- 172.21.15.29:0/411547705 shutdown_connections 2024-07-27T06:47:43.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.004+0000 7fcccc206640 1 --2- 172.21.15.29:0/411547705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc41086a0 0x7fccc410eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.004+0000 7fcccc206640 1 --2- 172.21.15.29:0/411547705 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccc4107d50 0x7fccc4108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.004+0000 7fcccc206640 1 --2- 172.21.15.29:0/411547705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc4102420 0x7fccc4102830 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.006 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.004+0000 7fcccc206640 1 -- 172.21.15.29:0/411547705 >> 172.21.15.29:0/411547705 conn(0x7fccc40fdde0 msgr2=0x7fccc4100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.004+0000 7fcccc206640 1 -- 172.21.15.29:0/411547705 shutdown_connections 2024-07-27T06:47:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.004+0000 7fcccc206640 1 -- 172.21.15.29:0/411547705 wait complete. 2024-07-27T06:47:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.004+0000 7fcccc206640 1 Processor -- start 2024-07-27T06:47:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.004+0000 7fcccc206640 1 -- start start 2024-07-27T06:47:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fcccc206640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc4102420 0x7fccc419acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fcccc206640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccc4107d50 0x7fccc419b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fcccc206640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc41086a0 0x7fccc419b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fcccc206640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41123c0 con 0x7fccc4102420 2024-07-27T06:47:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fcccc206640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fccc4112240 con 0x7fccc4107d50 2024-07-27T06:47:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fcccc206640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fccc4112540 con 0x7fccc41086a0 2024-07-27T06:47:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fccc9fa2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc4102420 0x7fccc419acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fccc9fa2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc4102420 0x7fccc419acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50950/0 (socket says 172.21.15.29:50950) 2024-07-27T06:47:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fccc9fa2640 1 -- 172.21.15.29:0/3032853148 learned_addr learned my addr 172.21.15.29:0/3032853148 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fccca7a3640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc41086a0 0x7fccc419b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fccc97a1640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccc4107d50 0x7fccc419b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fccc9fa2640 1 -- 172.21.15.29:0/3032853148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc41086a0 msgr2=0x7fccc419b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fccc9fa2640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc41086a0 0x7fccc419b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fccc9fa2640 1 -- 172.21.15.29:0/3032853148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccc4107d50 msgr2=0x7fccc419b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fccc9fa2640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccc4107d50 0x7fccc419b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fccc9fa2640 1 -- 172.21.15.29:0/3032853148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc41a5f70 con 0x7fccc4102420 2024-07-27T06:47:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.005+0000 7fccc9fa2640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc4102420 0x7fccc419acb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fccc0005e30 tx=0x7fccc0031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.006+0000 7fccb6ffd640 1 -- 172.21.15.29:0/3032853148 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0003b70 con 0x7fccc4102420 2024-07-27T06:47:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.006+0000 7fccb6ffd640 1 -- 172.21.15.29:0/3032853148 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccc0003d10 con 0x7fccc4102420 2024-07-27T06:47:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.006+0000 7fccb6ffd640 1 -- 172.21.15.29:0/3032853148 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc002fa50 con 0x7fccc4102420 2024-07-27T06:47:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.006+0000 7fcccc206640 1 -- 172.21.15.29:0/3032853148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc41a6200 con 0x7fccc4102420 2024-07-27T06:47:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.006+0000 7fcccc206640 1 -- 172.21.15.29:0/3032853148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc41a67a0 con 0x7fccc4102420 2024-07-27T06:47:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.008+0000 7fccb6ffd640 1 -- 172.21.15.29:0/3032853148 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7fccc002fbf0 con 0x7fccc4102420 2024-07-27T06:47:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.008+0000 7fccd31f8740 1 Processor -- start 2024-07-27T06:47:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccd31f8740 1 -- start start 2024-07-27T06:47:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccd31f8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d620e1b0 0x5622d65b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccd31f8740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d61e52f0 0x5622d65b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccd31f8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61ed4d0 0x5622d65b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccd31f8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5622d65263b0 con 0x5622d620e1b0 2024-07-27T06:47:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccd31f8740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5622d65158b0 con 0x5622d61e52f0 2024-07-27T06:47:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccd31f8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622d65198f0 con 0x5622d61ed4d0 2024-07-27T06:47:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccca7a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d620e1b0 0x5622d65b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccca7a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d620e1b0 0x5622d65b50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50964/0 (socket says 172.21.15.29:50964) 2024-07-27T06:47:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccca7a3640 1 -- 172.21.15.29:0/1331741805 learned_addr learned my addr 172.21.15.29:0/1331741805 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccc9fa2640 1 --2- 172.21.15.29:0/1331741805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d61e52f0 0x5622d65b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccc97a1640 1 --2- 172.21.15.29:0/1331741805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61ed4d0 0x5622d65b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.009+0000 7fccb6ffd640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fccb00b3d30 0x7fccb00b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.010+0000 7fccca7a3640 1 -- 172.21.15.29:0/1331741805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61ed4d0 msgr2=0x5622d65b5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.010+0000 7fccca7a3640 1 --2- 172.21.15.29:0/1331741805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61ed4d0 0x5622d65b5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.010+0000 7fccca7a3640 1 -- 172.21.15.29:0/1331741805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d61e52f0 msgr2=0x5622d65b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.010+0000 7fccca7a3640 1 --2- 172.21.15.29:0/1331741805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d61e52f0 0x5622d65b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.010+0000 7fccca7a3640 1 -- 172.21.15.29:0/1331741805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622d64bc9a0 con 0x5622d620e1b0 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.010+0000 7fccc9fa2640 1 --2- 172.21.15.29:0/1331741805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d61e52f0 0x5622d65b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.010+0000 7fccb6ffd640 1 -- 172.21.15.29:0/3032853148 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fccc0105360 con 0x7fccc4102420 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.010+0000 7fccca7a3640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fccb00b3d30 0x7fccb00b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.010+0000 7fccca7a3640 1 --2- 172.21.15.29:0/1331741805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d620e1b0 0x5622d65b50f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00b880 tx=0x7fccbc00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b615c0b9c3ebabbb server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.010+0000 7fccaffff640 1 -- 172.21.15.29:0/1331741805 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc024070 con 0x5622d620e1b0 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.010+0000 7fccaffff640 1 -- 172.21.15.29:0/1331741805 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccbc0047a0 con 0x5622d620e1b0 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.010+0000 7fccaffff640 1 -- 172.21.15.29:0/1331741805 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc011680 con 0x5622d620e1b0 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.011+0000 7fccd31f8740 1 -- 172.21.15.29:0/1331741805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d620e1b0 msgr2=0x5622d65b50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.011+0000 7fccd31f8740 1 --2- 172.21.15.29:0/1331741805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d620e1b0 0x5622d65b50f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00b880 tx=0x7fccbc00bd40 comp rx=0 tx=0).stop 2024-07-27T06:47:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.011+0000 7fccca7a3640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fccb00b3d30 0x7fccb00b6240 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fccbc021000 tx=0x7fccbc01e000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.011+0000 7fccd31f8740 1 -- 172.21.15.29:0/1331741805 shutdown_connections 2024-07-27T06:47:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.011+0000 7fccd31f8740 1 --2- 172.21.15.29:0/1331741805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61ed4d0 0x5622d65b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.011+0000 7fccd31f8740 1 --2- 172.21.15.29:0/1331741805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d61e52f0 0x5622d65b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.011+0000 7fccd31f8740 1 --2- 172.21.15.29:0/1331741805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d620e1b0 0x5622d65b50f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.011+0000 7fccd31f8740 1 -- 172.21.15.29:0/1331741805 >> 172.21.15.29:0/1331741805 conn(0x5622d6202300 msgr2=0x5622d65b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.011+0000 7fccd31f8740 1 -- 172.21.15.29:0/1331741805 shutdown_connections 2024-07-27T06:47:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.011+0000 7fccd31f8740 1 -- 172.21.15.29:0/1331741805 wait complete. 2024-07-27T06:47:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccd31f8740 1 -- start start 2024-07-27T06:47:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccd31f8740 1 Processor -- start 2024-07-27T06:47:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccd31f8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 0x5622d65c0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccd31f8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccd31f8740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccd31f8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5622d644fab0 con 0x5622d61ed4d0 2024-07-27T06:47:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccd31f8740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5622d649dcf0 con 0x5622d620e1b0 2024-07-27T06:47:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccd31f8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622d65c3870 con 0x5622d61e52f0 2024-07-27T06:47:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccd31f8740 10 client.0 _ll_register_callbacks cb 0x5622d650f9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccc97a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccc97a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50966/0 (socket says 172.21.15.29:50966) 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccc97a1640 1 -- 172.21.15.29:0/673322632 learned_addr learned my addr 172.21.15.29:0/673322632 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.012+0000 7fccc9fa2640 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 0x5622d65c0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccca7a3640 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccc97a1640 1 -- 172.21.15.29:0/673322632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 msgr2=0x5622d65c0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccc97a1640 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 0x5622d65c0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccc97a1640 1 -- 172.21.15.29:0/673322632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 msgr2=0x5622d65ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccc97a1640 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccc97a1640 1 -- 172.21.15.29:0/673322632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622d6279790 con 0x5622d61ed4d0 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccc97a1640 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c1120 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fccb8012700 tx=0x7fccb8012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3acd117bef732f7 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccad7fa640 1 -- 172.21.15.29:0/673322632 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb8004320 con 0x5622d61ed4d0 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccad7fa640 1 -- 172.21.15.29:0/673322632 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fccb80119a0 con 0x5622d61ed4d0 2024-07-27T06:47:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccd31f8740 1 -- 172.21.15.29:0/673322632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccbc0047a0 con 0x5622d61ed4d0 2024-07-27T06:47:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccad7fa640 1 -- 172.21.15.29:0/673322632 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fccb80044c0 con 0x5622d61ed4d0 2024-07-27T06:47:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccd31f8740 10 client.15678 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccd31f8740 1 -- 172.21.15.29:0/673322632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5622d62f2620 con 0x5622d61ed4d0 2024-07-27T06:47:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.013+0000 7fccad7fa640 1 -- 172.21.15.29:0/673322632 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccbc0047a0 con 0x5622d61ed4d0 2024-07-27T06:47:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.014+0000 7fccad7fa640 1 -- 172.21.15.29:0/673322632 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fccb800f440 con 0x5622d61ed4d0 2024-07-27T06:47:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.014+0000 7fccd31f8740 10 client.15678 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.014+0000 7fccd31f8740 10 client.15678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.014+0000 7fccd31f8740 20 client.15678 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.014+0000 7fccd31f8740 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5622d61ccd80 0x5622d65d2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.014+0000 7fccd31f8740 4 client.15678 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.014+0000 7fccd31f8740 1 -- 172.21.15.29:0/673322632 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622d6528300 con 0x5622d61ccd80 2024-07-27T06:47:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.014+0000 7fccc9fa2640 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5622d61ccd80 0x5622d65d2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.014+0000 7fccc9fa2640 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5622d61ccd80 0x5622d65d2bb0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ea4825e2a4582e8 server_cookie=1b5609e16ded9d60 in_seq=0 out_seq=0 2024-07-27T06:47:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.014+0000 7fccad7fa640 10 client.15678 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.015+0000 7fccad7fa640 1 -- 172.21.15.29:0/673322632 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5622d6528300 con 0x5622d61ccd80 2024-07-27T06:47:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.015+0000 7fccad7fa640 10 client.15678 handle_command_reply: tid=0 2024-07-27T06:47:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.015+0000 7fccd31f8740 1 client.15678 shutdown 2024-07-27T06:47:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.015+0000 7fccd31f8740 1 -- 172.21.15.29:0/673322632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 msgr2=0x5622d65c1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.015+0000 7fccd31f8740 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c1120 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fccb8012700 tx=0x7fccb8012bc0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.016+0000 7fccd31f8740 1 -- 172.21.15.29:0/673322632 shutdown_connections 2024-07-27T06:47:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.016+0000 7fccd31f8740 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.016+0000 7fccd31f8740 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c1120 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.016+0000 7fccd31f8740 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5622d61ccd80 0x5622d65d2bb0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.016+0000 7fccd31f8740 1 --2- 172.21.15.29:0/673322632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 0x5622d65c0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.016+0000 7fccd31f8740 1 -- 172.21.15.29:0/673322632 >> 172.21.15.29:0/673322632 conn(0x5622d6202300 msgr2=0x5622d65ad2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.016+0000 7fccad7fa640 0 client.15678 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.016+0000 7fccd31f8740 1 -- 172.21.15.29:0/673322632 shutdown_connections 2024-07-27T06:47:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.016+0000 7fccd31f8740 1 -- 172.21.15.29:0/673322632 wait complete. 2024-07-27T06:47:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.016+0000 7fccd31f8740 20 client.15678 trim_cache size 0 max 16384 2024-07-27T06:47:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.031+0000 7fccd31f8740 1 Processor -- start 2024-07-27T06:47:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.031+0000 7fccd31f8740 1 -- start start 2024-07-27T06:47:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.031+0000 7fccd31f8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 0x5622d65b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.031+0000 7fccd31f8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.031+0000 7fccd31f8740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65c0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.031+0000 7fccd31f8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41acf20 con 0x5622d61ed4d0 2024-07-27T06:47:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.031+0000 7fccd31f8740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41aea60 con 0x5622d620e1b0 2024-07-27T06:47:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.031+0000 7fccd31f8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41ad5f0 con 0x5622d61e52f0 2024-07-27T06:47:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.031+0000 7fccc97a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.031+0000 7fccc97a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50976/0 (socket says 172.21.15.29:50976) 2024-07-27T06:47:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.031+0000 7fccc97a1640 1 -- 172.21.15.29:0/3731284132 learned_addr learned my addr 172.21.15.29:0/3731284132 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.031+0000 7fccc9fa2640 1 --2- 172.21.15.29:0/3731284132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 0x5622d65b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.032+0000 7fccca7a3640 1 --2- 172.21.15.29:0/3731284132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65c0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.032+0000 7fccc97a1640 1 -- 172.21.15.29:0/3731284132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 msgr2=0x5622d65b6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.032+0000 7fccc97a1640 1 --2- 172.21.15.29:0/3731284132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 0x5622d65b6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.032+0000 7fccc97a1640 1 -- 172.21.15.29:0/3731284132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 msgr2=0x5622d65c0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.032+0000 7fccc97a1640 1 --2- 172.21.15.29:0/3731284132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65c0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.032+0000 7fccc97a1640 1 -- 172.21.15.29:0/3731284132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622d65d2f20 con 0x5622d61ed4d0 2024-07-27T06:47:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.032+0000 7fccc97a1640 1 --2- 172.21.15.29:0/3731284132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c2600 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fccb8005c20 tx=0x7fccb8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20520139ce556b92 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.032+0000 7fccaf7fe640 1 -- 172.21.15.29:0/3731284132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb8029070 con 0x5622d61ed4d0 2024-07-27T06:47:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.032+0000 7fccaf7fe640 1 -- 172.21.15.29:0/3731284132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccb802a040 con 0x5622d61ed4d0 2024-07-27T06:47:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.032+0000 7fccaf7fe640 1 -- 172.21.15.29:0/3731284132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb80052c0 con 0x5622d61ed4d0 2024-07-27T06:47:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.033+0000 7fccd31f8740 1 -- 172.21.15.29:0/3731284132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 msgr2=0x5622d65c2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.033+0000 7fccd31f8740 1 --2- 172.21.15.29:0/3731284132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c2600 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fccb8005c20 tx=0x7fccb8011890 comp rx=0 tx=0).stop 2024-07-27T06:47:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.033+0000 7fccd31f8740 1 -- 172.21.15.29:0/3731284132 shutdown_connections 2024-07-27T06:47:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.033+0000 7fccd31f8740 1 --2- 172.21.15.29:0/3731284132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65c0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.033+0000 7fccd31f8740 1 --2- 172.21.15.29:0/3731284132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c2600 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.033+0000 7fccd31f8740 1 --2- 172.21.15.29:0/3731284132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 0x5622d65b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.033+0000 7fccd31f8740 1 -- 172.21.15.29:0/3731284132 >> 172.21.15.29:0/3731284132 conn(0x5622d6202300 msgr2=0x5622d65c1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.033+0000 7fccd31f8740 1 -- 172.21.15.29:0/3731284132 shutdown_connections 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.033+0000 7fccd31f8740 1 -- 172.21.15.29:0/3731284132 wait complete. 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.033+0000 7fccd31f8740 1 -- start start 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.033+0000 7fccd31f8740 1 Processor -- start 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccd31f8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 0x5622d65c9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccd31f8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccd31f8740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65c6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccd31f8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41ae390 con 0x5622d61ed4d0 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccd31f8740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41adcc0 con 0x5622d620e1b0 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccd31f8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fccc41aaf00 con 0x5622d61e52f0 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccd31f8740 10 client.0 _ll_register_callbacks cb 0x7fccb80097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccc97a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccc97a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50984/0 (socket says 172.21.15.29:50984) 2024-07-27T06:47:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccc97a1640 1 -- 172.21.15.29:0/965114267 learned_addr learned my addr 172.21.15.29:0/965114267 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccc9fa2640 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 0x5622d65c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccca7a3640 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccc97a1640 1 -- 172.21.15.29:0/965114267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 msgr2=0x5622d65c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccc97a1640 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 0x5622d65c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccc97a1640 1 -- 172.21.15.29:0/965114267 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 msgr2=0x5622d65c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccc97a1640 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccc97a1640 1 -- 172.21.15.29:0/965114267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622d65c66f0 con 0x5622d61ed4d0 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccca7a3640 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccc97a1640 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c5ac0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fccb800ffe0 tx=0x7fccb800f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9adb4942fa14e24 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.034+0000 7fccad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccad7fa640 1 -- 172.21.15.29:0/965114267 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb8029040 con 0x5622d61ed4d0 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccad7fa640 1 -- 172.21.15.29:0/965114267 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fccb8011bb0 con 0x5622d61ed4d0 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccd31f8740 1 -- 172.21.15.29:0/965114267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccb802a040 con 0x5622d61ed4d0 2024-07-27T06:47:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccad7fa640 1 -- 172.21.15.29:0/965114267 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fccb8011db0 con 0x5622d61ed4d0 2024-07-27T06:47:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccd31f8740 10 client.15684 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccd31f8740 1 -- 172.21.15.29:0/965114267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5622d663a1e0 con 0x5622d61ed4d0 2024-07-27T06:47:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccad7fa640 1 -- 172.21.15.29:0/965114267 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccb802a040 con 0x5622d61ed4d0 2024-07-27T06:47:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccad7fa640 1 -- 172.21.15.29:0/965114267 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fccb8004b20 con 0x5622d61ed4d0 2024-07-27T06:47:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccd31f8740 10 client.15684 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccd31f8740 10 client.15684 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccd31f8740 20 client.15684 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccd31f8740 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5622d61a21c0 0x5622d663c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccd31f8740 4 client.15684 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-27T06:47:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.035+0000 7fccd31f8740 1 -- 172.21.15.29:0/965114267 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5622d6523da0 con 0x5622d61a21c0 2024-07-27T06:47:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.036+0000 7fccc9fa2640 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5622d61a21c0 0x5622d663c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.036+0000 7fccad7fa640 10 client.15684 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:43.043 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.036+0000 7fccc9fa2640 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5622d61a21c0 0x5622d663c880 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0c0137fc182936f server_cookie=9fc7f9e284afd40f in_seq=0 out_seq=0 2024-07-27T06:47:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccad7fa640 1 -- 172.21.15.29:0/965114267 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5622d6523da0 con 0x5622d61a21c0 2024-07-27T06:47:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccad7fa640 10 client.15684 handle_command_reply: tid=0 2024-07-27T06:47:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccd31f8740 1 client.15684 shutdown 2024-07-27T06:47:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccd31f8740 1 -- 172.21.15.29:0/965114267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 msgr2=0x5622d65c5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccd31f8740 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c5ac0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fccb800ffe0 tx=0x7fccb800f670 comp rx=0 tx=0).stop 2024-07-27T06:47:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccd31f8740 1 -- 172.21.15.29:0/965114267 shutdown_connections 2024-07-27T06:47:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccd31f8740 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5622d620e1b0 0x5622d65c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccd31f8740 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5622d61ed4d0 0x5622d65c5ac0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccd31f8740 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5622d61a21c0 0x5622d663c880 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccd31f8740 1 --2- 172.21.15.29:0/965114267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d61e52f0 0x5622d65c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccd31f8740 1 -- 172.21.15.29:0/965114267 >> 172.21.15.29:0/965114267 conn(0x5622d6202300 msgr2=0x5622d65b68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccad7fa640 0 client.15684 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccd31f8740 1 -- 172.21.15.29:0/965114267 shutdown_connections 2024-07-27T06:47:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccd31f8740 1 -- 172.21.15.29:0/965114267 wait complete. 2024-07-27T06:47:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.043+0000 7fccd31f8740 20 client.15684 trim_cache size 0 max 16384 2024-07-27T06:47:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-07-27T06:47:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-07-27T06:47:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-07-27T06:47:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-07-27T06:47:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.044+0000 7fcccc206640 1 -- 172.21.15.29:0/3032853148 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fccb00b3d30 msgr2=0x7fccb00b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.044+0000 7fcccc206640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fccb00b3d30 0x7fccb00b6240 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fccbc021000 tx=0x7fccbc01e000 comp rx=0 tx=0).stop 2024-07-27T06:47:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.044+0000 7fcccc206640 1 -- 172.21.15.29:0/3032853148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc4102420 msgr2=0x7fccc419acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.044+0000 7fcccc206640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc4102420 0x7fccc419acb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fccc0005e30 tx=0x7fccc0031840 comp rx=0 tx=0).stop 2024-07-27T06:47:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.044+0000 7fcccc206640 1 -- 172.21.15.29:0/3032853148 shutdown_connections 2024-07-27T06:47:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.044+0000 7fcccc206640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fccb00b3d30 0x7fccb00b6240 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.044+0000 7fcccc206640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccc41086a0 0x7fccc419b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.044+0000 7fcccc206640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccc4107d50 0x7fccc419b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.044+0000 7fcccc206640 1 --2- 172.21.15.29:0/3032853148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc4102420 0x7fccc419acb0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.044+0000 7fcccc206640 1 -- 172.21.15.29:0/3032853148 >> 172.21.15.29:0/3032853148 conn(0x7fccc40fdde0 msgr2=0x7fccc4100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.044+0000 7fcccc206640 1 -- 172.21.15.29:0/3032853148 shutdown_connections 2024-07-27T06:47:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:43.044+0000 7fcccc206640 1 -- 172.21.15.29:0/3032853148 wait complete. 2024-07-27T06:47:43.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:42 smithi029 ceph-mon[32610]: pgmap v418: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-07-27T06:47:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:42 smithi077 ceph-mon[37153]: pgmap v418: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-07-27T06:47:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:42 smithi047 ceph-mon[36835]: pgmap v418: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-07-27T06:47:43.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.735+0000 7f78d513a640 1 -- 172.21.15.29:0/3429108759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0108720 msgr2=0x7f78d010ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.735+0000 7f78d513a640 1 --2- 172.21.15.29:0/3429108759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0108720 0x7f78d010ef50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f78c8009ca0 tx=0x7f78c802a8c0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.735+0000 7f78d513a640 1 -- 172.21.15.29:0/3429108759 shutdown_connections 2024-07-27T06:47:43.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.735+0000 7f78d513a640 1 --2- 172.21.15.29:0/3429108759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0108720 0x7f78d010ef50 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.735+0000 7f78d513a640 1 --2- 172.21.15.29:0/3429108759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78d0107dd0 0x7f78d01081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.735+0000 7f78d513a640 1 --2- 172.21.15.29:0/3429108759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78d0102490 0x7f78d01028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.735+0000 7f78d513a640 1 -- 172.21.15.29:0/3429108759 >> 172.21.15.29:0/3429108759 conn(0x7f78d0079160 msgr2=0x7f78d0079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:43.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.735+0000 7f78d513a640 1 -- 172.21.15.29:0/3429108759 shutdown_connections 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.735+0000 7f78d513a640 1 -- 172.21.15.29:0/3429108759 wait complete. 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.736+0000 7f78d513a640 1 Processor -- start 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.736+0000 7f78d513a640 1 -- start start 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.736+0000 7f78d513a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0102490 0x7f78d0198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.736+0000 7f78d513a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78d0107dd0 0x7f78d0199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.736+0000 7f78d513a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78d0108720 0x7f78d019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.736+0000 7f78d513a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f78d01123a0 con 0x7f78d0102490 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.736+0000 7f78d513a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f78d0112220 con 0x7f78d0107dd0 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.736+0000 7f78d513a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f78d0112520 con 0x7f78d0108720 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78ced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0102490 0x7f78d0198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78ced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0102490 0x7f78d0198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51052/0 (socket says 172.21.15.29:51052) 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78ced9d640 1 -- 172.21.15.29:0/803904066 learned_addr learned my addr 172.21.15.29:0/803904066 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78ce59c640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78d0107dd0 0x7f78d0199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78cf59e640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78d0108720 0x7f78d019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78ced9d640 1 -- 172.21.15.29:0/803904066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78d0108720 msgr2=0x7f78d019e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78ced9d640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78d0108720 0x7f78d019e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78ced9d640 1 -- 172.21.15.29:0/803904066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78d0107dd0 msgr2=0x7f78d0199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78ced9d640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78d0107dd0 0x7f78d0199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78ced9d640 1 -- 172.21.15.29:0/803904066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d019ed40 con 0x7f78d0102490 2024-07-27T06:47:43.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78ced9d640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0102490 0x7f78d0198dd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f78c000e9f0 tx=0x7f78c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:43.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78b7fff640 1 -- 172.21.15.29:0/803904066 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c000cd80 con 0x7f78d0102490 2024-07-27T06:47:43.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78d513a640 1 -- 172.21.15.29:0/803904066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d019f030 con 0x7f78d0102490 2024-07-27T06:47:43.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78b7fff640 1 -- 172.21.15.29:0/803904066 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78c00108e0 con 0x7f78d0102490 2024-07-27T06:47:43.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78b7fff640 1 -- 172.21.15.29:0/803904066 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0018980 con 0x7f78d0102490 2024-07-27T06:47:43.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.737+0000 7f78d513a640 1 -- 172.21.15.29:0/803904066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d006a890 con 0x7f78d0102490 2024-07-27T06:47:43.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.740+0000 7f78b7fff640 1 -- 172.21.15.29:0/803904066 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f78c0010430 con 0x7f78d0102490 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.741+0000 7f78db6be740 1 Processor -- start 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.741+0000 7f78db6be740 1 -- start start 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.741+0000 7f78b7fff640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78b80b3cb0 0x7f78b80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.741+0000 7f78b7fff640 1 -- 172.21.15.29:0/803904066 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f78c00e4200 con 0x7f78d0102490 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.741+0000 7f78cf59e640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78b80b3cb0 0x7f78b80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.741+0000 7f78db6be740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 0x56446ddda380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.741+0000 7f78db6be740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da0c0a0 0x56446ddda8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.741+0000 7f78db6be740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da14280 0x56446dddae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.741+0000 7f78db6be740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56446dd50920 con 0x56446da34f60 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78db6be740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x56446dd3fe20 con 0x56446da0c0a0 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78db6be740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56446dd43e60 con 0x56446da14280 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78ced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 0x56446ddda380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78ced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 0x56446ddda380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51054/0 (socket says 172.21.15.29:51054) 2024-07-27T06:47:43.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78ced9d640 1 -- 172.21.15.29:0/3557319475 learned_addr learned my addr 172.21.15.29:0/3557319475 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78ce59c640 1 --2- 172.21.15.29:0/3557319475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da0c0a0 0x56446ddda8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78ced9d640 1 -- 172.21.15.29:0/3557319475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da14280 msgr2=0x56446dddae00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78ced9d640 1 --2- 172.21.15.29:0/3557319475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da14280 0x56446dddae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78ced9d640 1 -- 172.21.15.29:0/3557319475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da0c0a0 msgr2=0x56446ddda8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78ced9d640 1 --2- 172.21.15.29:0/3557319475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da0c0a0 0x56446ddda8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78ced9d640 1 -- 172.21.15.29:0/3557319475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56446dce6a40 con 0x56446da34f60 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78ce59c640 1 --2- 172.21.15.29:0/3557319475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da0c0a0 0x56446ddda8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78cf59e640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78b80b3cb0 0x7f78b80b61c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f78c8006010 tx=0x7f78c80093e0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78ced9d640 1 --2- 172.21.15.29:0/3557319475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 0x56446ddda380 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f78c00a7010 tx=0x7f78c00aaf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5add547106319f77 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78b57fa640 1 -- 172.21.15.29:0/3557319475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c00b3f90 con 0x56446da34f60 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78b57fa640 1 -- 172.21.15.29:0/3557319475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78c0010a90 con 0x56446da34f60 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.742+0000 7f78b57fa640 1 -- 172.21.15.29:0/3557319475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c00a6900 con 0x56446da34f60 2024-07-27T06:47:43.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.743+0000 7f78db6be740 1 -- 172.21.15.29:0/3557319475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 msgr2=0x56446ddda380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.743+0000 7f78db6be740 1 --2- 172.21.15.29:0/3557319475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 0x56446ddda380 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f78c00a7010 tx=0x7f78c00aaf90 comp rx=0 tx=0).stop 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.743+0000 7f78db6be740 1 -- 172.21.15.29:0/3557319475 shutdown_connections 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.743+0000 7f78db6be740 1 --2- 172.21.15.29:0/3557319475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da14280 0x56446dddae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.743+0000 7f78db6be740 1 --2- 172.21.15.29:0/3557319475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da0c0a0 0x56446ddda8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.743+0000 7f78db6be740 1 --2- 172.21.15.29:0/3557319475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 0x56446ddda380 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.743+0000 7f78db6be740 1 -- 172.21.15.29:0/3557319475 >> 172.21.15.29:0/3557319475 conn(0x56446da29350 msgr2=0x56446ddd8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.743+0000 7f78db6be740 1 -- 172.21.15.29:0/3557319475 shutdown_connections 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.744+0000 7f78db6be740 1 -- 172.21.15.29:0/3557319475 wait complete. 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.744+0000 7f78db6be740 1 -- start start 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.744+0000 7f78db6be740 1 Processor -- start 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78db6be740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da0c0a0 0x56446dde5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78db6be740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da14280 0x56446dde6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78db6be740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da34f60 0x56446ddf1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78db6be740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56446dba35f0 con 0x56446da0c0a0 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78db6be740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x56446dce2890 con 0x56446da34f60 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78db6be740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56446dde8b60 con 0x56446da14280 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78ced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da0c0a0 0x56446dde5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78ced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da0c0a0 0x56446dde5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51068/0 (socket says 172.21.15.29:51068) 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78ced9d640 1 -- 172.21.15.29:0/4227012957 learned_addr learned my addr 172.21.15.29:0/4227012957 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78db6be740 10 client.0 _ll_register_callbacks cb 0x56446dcec7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78ce59c640 1 --2- 172.21.15.29:0/4227012957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da14280 0x56446dde6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78ced9d640 1 -- 172.21.15.29:0/4227012957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da14280 msgr2=0x56446dde6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78ced9d640 1 --2- 172.21.15.29:0/4227012957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da14280 0x56446dde6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78ced9d640 1 -- 172.21.15.29:0/4227012957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da34f60 msgr2=0x56446ddf1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78ced9d640 1 --2- 172.21.15.29:0/4227012957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da34f60 0x56446ddf1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78ced9d640 1 -- 172.21.15.29:0/4227012957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56446daa0540 con 0x56446da0c0a0 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78ced9d640 1 --2- 172.21.15.29:0/4227012957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da0c0a0 0x56446dde5f90 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f78c000c6a0 tx=0x7f78c000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196b2fff328bfa15 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.745+0000 7f78b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.746+0000 7f78b2ff5640 1 -- 172.21.15.29:0/4227012957 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c00c9030 con 0x56446da0c0a0 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.746+0000 7f78b2ff5640 1 -- 172.21.15.29:0/4227012957 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f78c00a7580 con 0x56446da0c0a0 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.746+0000 7f78db6be740 1 -- 172.21.15.29:0/4227012957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78c0010a90 con 0x56446da0c0a0 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.746+0000 7f78b2ff5640 1 -- 172.21.15.29:0/4227012957 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f78c00a6e50 con 0x56446da0c0a0 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.746+0000 7f78db6be740 10 client.15714 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.746+0000 7f78db6be740 1 -- 172.21.15.29:0/4227012957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56446ddf4b10 con 0x56446da0c0a0 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.746+0000 7f78b2ff5640 1 -- 172.21.15.29:0/4227012957 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78c0010a90 con 0x56446da0c0a0 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.746+0000 7f78b2ff5640 1 -- 172.21.15.29:0/4227012957 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f78c0018ce0 con 0x56446da0c0a0 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.746+0000 7f78db6be740 10 client.15714 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.746+0000 7f78db6be740 10 client.15714 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-27T06:47:43.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.746+0000 7f78db6be740 20 client.15714 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.746+0000 7f78db6be740 1 --2- 172.21.15.29:0/4227012957 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56446d9f38e0 0x56446ddf80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.747+0000 7f78db6be740 4 client.15714 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.747+0000 7f78db6be740 1 -- 172.21.15.29:0/4227012957 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56446dd52870 con 0x56446d9f38e0 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.747+0000 7f78ce59c640 1 --2- 172.21.15.29:0/4227012957 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56446d9f38e0 0x56446ddf80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.747+0000 7f78ce59c640 1 --2- 172.21.15.29:0/4227012957 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56446d9f38e0 0x56446ddf80e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8595d191478e1b7b server_cookie=4288f28ecfb65389 in_seq=0 out_seq=0 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.747+0000 7f78b2ff5640 10 client.15714 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.748+0000 7f78b2ff5640 1 -- 172.21.15.29:0/4227012957 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56446dd52870 con 0x56446d9f38e0 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.748+0000 7f78b2ff5640 10 client.15714 handle_command_reply: tid=0 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.748+0000 7f78db6be740 1 client.15714 shutdown 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.748+0000 7f78db6be740 1 -- 172.21.15.29:0/4227012957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da0c0a0 msgr2=0x56446dde5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.748+0000 7f78db6be740 1 --2- 172.21.15.29:0/4227012957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da0c0a0 0x56446dde5f90 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f78c000c6a0 tx=0x7f78c000cf20 comp rx=0 tx=0).stop 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.749+0000 7f78db6be740 1 -- 172.21.15.29:0/4227012957 shutdown_connections 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.749+0000 7f78db6be740 1 --2- 172.21.15.29:0/4227012957 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56446d9f38e0 0x56446ddf80e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.749+0000 7f78db6be740 1 --2- 172.21.15.29:0/4227012957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da34f60 0x56446ddf1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.749+0000 7f78b2ff5640 0 client.15714 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.749+0000 7f78db6be740 1 --2- 172.21.15.29:0/4227012957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da14280 0x56446dde6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.749+0000 7f78db6be740 1 --2- 172.21.15.29:0/4227012957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da0c0a0 0x56446dde5f90 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.749+0000 7f78db6be740 1 -- 172.21.15.29:0/4227012957 >> 172.21.15.29:0/4227012957 conn(0x56446da29350 msgr2=0x56446dddd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.749+0000 7f78db6be740 1 -- 172.21.15.29:0/4227012957 shutdown_connections 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.749+0000 7f78db6be740 1 -- 172.21.15.29:0/4227012957 wait complete. 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.749+0000 7f78db6be740 20 client.15714 trim_cache size 0 max 16384 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78db6be740 1 Processor -- start 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78db6be740 1 -- start start 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78db6be740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da0c0a0 0x56446dddf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78db6be740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da14280 0x56446dddf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78db6be740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da34f60 0x56446ddf1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78db6be740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56446ddf6420 con 0x56446da14280 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78db6be740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x56446ddf6af0 con 0x56446da34f60 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78db6be740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56446ddf71c0 con 0x56446da0c0a0 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78ce59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da14280 0x56446dddf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78ce59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da14280 0x56446dddf9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51076/0 (socket says 172.21.15.29:51076) 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78ce59c640 1 -- 172.21.15.29:0/2688053731 learned_addr learned my addr 172.21.15.29:0/2688053731 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78cf59e640 1 --2- 172.21.15.29:0/2688053731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da34f60 0x56446ddf1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.768+0000 7f78ced9d640 1 --2- 172.21.15.29:0/2688053731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da0c0a0 0x56446dddf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.769+0000 7f78ce59c640 1 -- 172.21.15.29:0/2688053731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da0c0a0 msgr2=0x56446dddf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.769+0000 7f78ce59c640 1 --2- 172.21.15.29:0/2688053731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da0c0a0 0x56446dddf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.769+0000 7f78ce59c640 1 -- 172.21.15.29:0/2688053731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da34f60 msgr2=0x56446ddf1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.769+0000 7f78ce59c640 1 --2- 172.21.15.29:0/2688053731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da34f60 0x56446ddf1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.769+0000 7f78ce59c640 1 -- 172.21.15.29:0/2688053731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56446ddd24b0 con 0x56446da14280 2024-07-27T06:47:43.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.769+0000 7f78ce59c640 1 --2- 172.21.15.29:0/2688053731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da14280 0x56446dddf9d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f78d0079760 tx=0x7f78c4004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68a5b17b09eb76f server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.769+0000 7f78b4ff9640 1 -- 172.21.15.29:0/2688053731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c4003d80 con 0x56446da14280 2024-07-27T06:47:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.769+0000 7f78b4ff9640 1 -- 172.21.15.29:0/2688053731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78c401e8d0 con 0x56446da14280 2024-07-27T06:47:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.769+0000 7f78b4ff9640 1 -- 172.21.15.29:0/2688053731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c4012af0 con 0x56446da14280 2024-07-27T06:47:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.770+0000 7f78db6be740 1 -- 172.21.15.29:0/2688053731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da14280 msgr2=0x56446dddf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.770+0000 7f78db6be740 1 --2- 172.21.15.29:0/2688053731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da14280 0x56446dddf9d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f78d0079760 tx=0x7f78c4004410 comp rx=0 tx=0).stop 2024-07-27T06:47:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.770+0000 7f78db6be740 1 -- 172.21.15.29:0/2688053731 shutdown_connections 2024-07-27T06:47:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.770+0000 7f78db6be740 1 --2- 172.21.15.29:0/2688053731 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da34f60 0x56446ddf1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.770+0000 7f78db6be740 1 --2- 172.21.15.29:0/2688053731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da14280 0x56446dddf9d0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.770+0000 7f78db6be740 1 --2- 172.21.15.29:0/2688053731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da0c0a0 0x56446dddf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.770+0000 7f78db6be740 1 -- 172.21.15.29:0/2688053731 >> 172.21.15.29:0/2688053731 conn(0x56446da29350 msgr2=0x56446dde5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:43.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.770+0000 7f78db6be740 1 -- 172.21.15.29:0/2688053731 shutdown_connections 2024-07-27T06:47:43.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.770+0000 7f78db6be740 1 -- 172.21.15.29:0/2688053731 wait complete. 2024-07-27T06:47:43.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78db6be740 1 -- start start 2024-07-27T06:47:43.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78db6be740 1 Processor -- start 2024-07-27T06:47:43.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78db6be740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da0c0a0 0x56446ddf94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78db6be740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da14280 0x56446ddf9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78db6be740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 0x56446ddfeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78db6be740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56446ddf7f60 con 0x56446da34f60 2024-07-27T06:47:43.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78db6be740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x56446ddf7890 con 0x56446da14280 2024-07-27T06:47:43.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78db6be740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56446ddf8630 con 0x56446da0c0a0 2024-07-27T06:47:43.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78db6be740 10 client.0 _ll_register_callbacks cb 0x7f78c00af360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:43.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78cf59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 0x56446ddfeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78cf59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 0x56446ddfeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51092/0 (socket says 172.21.15.29:51092) 2024-07-27T06:47:43.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78cf59e640 1 -- 172.21.15.29:0/4104574986 learned_addr learned my addr 172.21.15.29:0/4104574986 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:43.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78ce59c640 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da14280 0x56446ddf9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.771+0000 7f78ced9d640 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da0c0a0 0x56446ddf94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78cf59e640 1 -- 172.21.15.29:0/4104574986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da0c0a0 msgr2=0x56446ddf94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78cf59e640 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da0c0a0 0x56446ddf94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78cf59e640 1 -- 172.21.15.29:0/4104574986 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da14280 msgr2=0x56446ddf9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:43.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78cf59e640 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da14280 0x56446ddf9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78cf59e640 1 -- 172.21.15.29:0/4104574986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56446ddff110 con 0x56446da34f60 2024-07-27T06:47:43.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78cf59e640 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 0x56446ddfeaa0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f78c8031700 tx=0x7f78c80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4deb933e1272a08 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:43.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:43.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78b2ff5640 1 -- 172.21.15.29:0/4104574986 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c8004040 con 0x56446da34f60 2024-07-27T06:47:43.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78b2ff5640 1 -- 172.21.15.29:0/4104574986 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f78c80041e0 con 0x56446da34f60 2024-07-27T06:47:43.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78db6be740 1 -- 172.21.15.29:0/4104574986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78c401e8d0 con 0x56446da34f60 2024-07-27T06:47:43.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78b2ff5640 1 -- 172.21.15.29:0/4104574986 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f78c80043e0 con 0x56446da34f60 2024-07-27T06:47:43.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78db6be740 10 client.15720 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:43.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.772+0000 7f78db6be740 1 -- 172.21.15.29:0/4104574986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56446de004d0 con 0x56446da34f60 2024-07-27T06:47:43.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.773+0000 7f78b2ff5640 1 -- 172.21.15.29:0/4104574986 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78c401e8d0 con 0x56446da34f60 2024-07-27T06:47:43.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.773+0000 7f78b2ff5640 1 -- 172.21.15.29:0/4104574986 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f78c8043040 con 0x56446da34f60 2024-07-27T06:47:43.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.773+0000 7f78db6be740 10 client.15720 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:43.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.773+0000 7f78db6be740 10 client.15720 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-27T06:47:43.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.773+0000 7f78db6be740 20 client.15720 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:43.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.773+0000 7f78db6be740 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56446d9c8c90 0x56446de64010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:43.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.773+0000 7f78db6be740 4 client.15720 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:47:43.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.773+0000 7f78db6be740 1 -- 172.21.15.29:0/4104574986 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "status"}) -- 0x56446dd4e310 con 0x56446d9c8c90 2024-07-27T06:47:43.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.773+0000 7f78ced9d640 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56446d9c8c90 0x56446de64010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:43.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78b2ff5640 10 client.15720 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:47:43.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78ced9d640 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56446d9c8c90 0x56446de64010 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e5e40512a285accb server_cookie=e20b918742d3e006 in_seq=0 out_seq=0 2024-07-27T06:47:43.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78b2ff5640 1 -- 172.21.15.29:0/4104574986 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56446dd4e310 con 0x56446d9c8c90 2024-07-27T06:47:43.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78b2ff5640 10 client.15720 handle_command_reply: tid=0 2024-07-27T06:47:43.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78db6be740 1 client.15720 shutdown 2024-07-27T06:47:43.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78db6be740 1 -- 172.21.15.29:0/4104574986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 msgr2=0x56446ddfeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78db6be740 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 0x56446ddfeaa0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f78c8031700 tx=0x7f78c80095c0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78db6be740 1 -- 172.21.15.29:0/4104574986 shutdown_connections 2024-07-27T06:47:43.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78db6be740 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x56446d9c8c90 0x56446de64010 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.808 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:47:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:47:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 1, 2024-07-27T06:47:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24658, 2024-07-27T06:47:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:47:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:47:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:47:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 253.93865063000001, 2024-07-27T06:47:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 29, 2024-07-27T06:47:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:47:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 83, 2024-07-27T06:47:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 302.14940225800001 2024-07-27T06:47:43.810 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:47:43.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78db6be740 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56446da34f60 0x56446ddfeaa0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78db6be740 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56446da14280 0x56446ddf9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78db6be740 1 --2- 172.21.15.29:0/4104574986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56446da0c0a0 0x56446ddf94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78b2ff5640 0 client.15720 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:47:43.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78db6be740 1 -- 172.21.15.29:0/4104574986 >> 172.21.15.29:0/4104574986 conn(0x56446da29350 msgr2=0x56446dddf490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:43.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78db6be740 1 -- 172.21.15.29:0/4104574986 shutdown_connections 2024-07-27T06:47:43.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.774+0000 7f78db6be740 1 -- 172.21.15.29:0/4104574986 wait complete. 2024-07-27T06:47:43.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78db6be740 20 client.15720 trim_cache size 0 max 16384 2024-07-27T06:47:43.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78d513a640 1 -- 172.21.15.29:0/803904066 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78b80b3cb0 msgr2=0x7f78b80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78d513a640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78b80b3cb0 0x7f78b80b61c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f78c8006010 tx=0x7f78c80093e0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78d513a640 1 -- 172.21.15.29:0/803904066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0102490 msgr2=0x7f78d0198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:43.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78d513a640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0102490 0x7f78d0198dd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f78c000e9f0 tx=0x7f78c000eeb0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78d513a640 1 -- 172.21.15.29:0/803904066 shutdown_connections 2024-07-27T06:47:43.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78d513a640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f78b80b3cb0 0x7f78b80b61c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78d513a640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78d0108720 0x7f78d019e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78d513a640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78d0107dd0 0x7f78d0199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78d513a640 1 --2- 172.21.15.29:0/803904066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d0102490 0x7f78d0198dd0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:43.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78d513a640 1 -- 172.21.15.29:0/803904066 >> 172.21.15.29:0/803904066 conn(0x7f78d0079160 msgr2=0x7f78d0104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:43.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78d513a640 1 -- 172.21.15.29:0/803904066 shutdown_connections 2024-07-27T06:47:43.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.775+0000 7f78d513a640 1 -- 172.21.15.29:0/803904066 wait complete. 2024-07-27T06:47:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.609+0000 7f7c9167d640 1 -- 172.21.15.29:0/3843892054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c1029d0 msgr2=0x7f7c8c10f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.609+0000 7f7c9167d640 1 --2- 172.21.15.29:0/3843892054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c1029d0 0x7f7c8c10f6f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009900 tx=0x7f7c7402a480 comp rx=0 tx=0).stop 2024-07-27T06:47:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.609+0000 7f7c9167d640 1 -- 172.21.15.29:0/3843892054 shutdown_connections 2024-07-27T06:47:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.609+0000 7f7c9167d640 1 --2- 172.21.15.29:0/3843892054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c10fc30 0x7f7c8c112050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.609+0000 7f7c9167d640 1 --2- 172.21.15.29:0/3843892054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c1029d0 0x7f7c8c10f6f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.609+0000 7f7c9167d640 1 --2- 172.21.15.29:0/3843892054 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c8c102080 0x7f7c8c102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.609+0000 7f7c9167d640 1 -- 172.21.15.29:0/3843892054 >> 172.21.15.29:0/3843892054 conn(0x7f7c8c0791a0 msgr2=0x7f7c8c0795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.609+0000 7f7c9167d640 1 -- 172.21.15.29:0/3843892054 shutdown_connections 2024-07-27T06:47:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.609+0000 7f7c9167d640 1 -- 172.21.15.29:0/3843892054 wait complete. 2024-07-27T06:47:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.610+0000 7f7c9167d640 1 Processor -- start 2024-07-27T06:47:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.610+0000 7f7c9167d640 1 -- start start 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c9167d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c102080 0x7f7c8c19c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c9167d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c1029d0 0x7f7c8c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c9167d640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c8c10fc30 0x7f7c8c1a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c9167d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c115710 con 0x7f7c8c102080 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c9167d640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c115590 con 0x7f7c8c10fc30 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c9167d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c115890 con 0x7f7c8c1029d0 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c8affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c102080 0x7f7c8c19c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c8affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c102080 0x7f7c8c19c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51006/0 (socket says 172.21.15.29:51006) 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c8affd640 1 -- 172.21.15.29:0/1313225396 learned_addr learned my addr 172.21.15.29:0/1313225396 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c827fc640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c1029d0 0x7f7c8c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c8c10fc30 0x7f7c8c1a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c8affd640 1 -- 172.21.15.29:0/1313225396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c1029d0 msgr2=0x7f7c8c19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c8affd640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c1029d0 0x7f7c8c19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c8affd640 1 -- 172.21.15.29:0/1313225396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c8c10fc30 msgr2=0x7f7c8c1a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c8affd640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c8c10fc30 0x7f7c8c1a1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c8affd640 1 -- 172.21.15.29:0/1313225396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8c1a2210 con 0x7f7c8c102080 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.611+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c8c10fc30 0x7f7c8c1a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.612+0000 7f7c8affd640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c102080 0x7f7c8c19c2a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c00c890 tx=0x7f7c7c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.612+0000 7f7c88ff9640 1 -- 172.21.15.29:0/1313225396 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c004220 con 0x7f7c8c102080 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.612+0000 7f7c9167d640 1 -- 172.21.15.29:0/1313225396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c8c1a2500 con 0x7f7c8c102080 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.612+0000 7f7c88ff9640 1 -- 172.21.15.29:0/1313225396 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c7c0043c0 con 0x7f7c8c102080 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.612+0000 7f7c88ff9640 1 -- 172.21.15.29:0/1313225396 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c004a00 con 0x7f7c8c102080 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.612+0000 7f7c9167d640 1 -- 172.21.15.29:0/1313225396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c8c1a8c60 con 0x7f7c8c102080 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.615+0000 7f7c97c01740 1 Processor -- start 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.615+0000 7f7c88ff9640 1 -- 172.21.15.29:0/1313225396 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f7c7c004ba0 con 0x7f7c8c102080 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.615+0000 7f7c97c01740 1 -- start start 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.615+0000 7f7c97c01740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc376550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.615+0000 7f7c97c01740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc376a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.615+0000 7f7c97c01740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc376fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.615+0000 7f7c97c01740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5650dc2ecb90 con 0x5650dbfd1070 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.615+0000 7f7c97c01740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5650dc2dc090 con 0x5650dbfb0390 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.615+0000 7f7c97c01740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5650dc2e00d0 con 0x5650dbfa81b0 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.615+0000 7f7c8b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc376550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.615+0000 7f7c8b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc376550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51012/0 (socket says 172.21.15.29:51012) 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.615+0000 7f7c8b7fe640 1 -- 172.21.15.29:0/139203718 learned_addr learned my addr 172.21.15.29:0/139203718 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c827fc640 1 --2- 172.21.15.29:0/139203718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc376fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c8affd640 1 --2- 172.21.15.29:0/139203718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc376a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c8b7fe640 1 -- 172.21.15.29:0/139203718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 msgr2=0x5650dc376a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/139203718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc376a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c8b7fe640 1 -- 172.21.15.29:0/139203718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 msgr2=0x5650dc376fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/139203718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc376fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c8b7fe640 1 -- 172.21.15.29:0/139203718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650dc282b70 con 0x5650dbfd1070 2024-07-27T06:47:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/139203718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc376550 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009f30 tx=0x7f7c84010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=101e402e2396697b server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c88ff9640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7c780b3d30 0x7f7c780b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c8affd640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7c780b3d30 0x7f7c780b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c817fa640 1 -- 172.21.15.29:0/139203718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84016070 con 0x5650dbfd1070 2024-07-27T06:47:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c817fa640 1 -- 172.21.15.29:0/139203718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c8401bb10 con 0x5650dbfd1070 2024-07-27T06:47:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.616+0000 7f7c817fa640 1 -- 172.21.15.29:0/139203718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84012c30 con 0x5650dbfd1070 2024-07-27T06:47:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.617+0000 7f7c88ff9640 1 -- 172.21.15.29:0/1313225396 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7c7c0e3c30 con 0x7f7c8c102080 2024-07-27T06:47:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.617+0000 7f7c97c01740 1 -- 172.21.15.29:0/139203718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 msgr2=0x5650dc376550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.617+0000 7f7c97c01740 1 --2- 172.21.15.29:0/139203718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc376550 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009f30 tx=0x7f7c84010800 comp rx=0 tx=0).stop 2024-07-27T06:47:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.617+0000 7f7c8affd640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7c780b3d30 0x7f7c780b6240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5650dc3751d0 tx=0x7f7c7c000bc0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 -- 172.21.15.29:0/139203718 shutdown_connections 2024-07-27T06:47:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 --2- 172.21.15.29:0/139203718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc376fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 --2- 172.21.15.29:0/139203718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc376a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 --2- 172.21.15.29:0/139203718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc376550 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 -- 172.21.15.29:0/139203718 >> 172.21.15.29:0/139203718 conn(0x5650dbfc5460 msgr2=0x5650dc374440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 -- 172.21.15.29:0/139203718 shutdown_connections 2024-07-27T06:47:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 -- 172.21.15.29:0/139203718 wait complete. 2024-07-27T06:47:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 -- start start 2024-07-27T06:47:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 Processor -- start 2024-07-27T06:47:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc382180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc3826c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc38e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5650dc13f700 con 0x5650dbfd1070 2024-07-27T06:47:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5650dc27e9c0 con 0x5650dbfb0390 2024-07-27T06:47:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.618+0000 7f7c97c01740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5650dc384d30 con 0x5650dbfa81b0 2024-07-27T06:47:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c97c01740 10 client.0 _ll_register_callbacks cb 0x5650dc2994d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c8b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc38e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c8b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc38e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51018/0 (socket says 172.21.15.29:51018) 2024-07-27T06:47:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c8b7fe640 1 -- 172.21.15.29:0/1361683576 learned_addr learned my addr 172.21.15.29:0/1361683576 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c827fc640 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc3826c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc382180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c8b7fe640 1 -- 172.21.15.29:0/1361683576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 msgr2=0x5650dc382180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc382180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c8b7fe640 1 -- 172.21.15.29:0/1361683576 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 msgr2=0x5650dc3826c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc3826c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c8b7fe640 1 -- 172.21.15.29:0/1361683576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650dc03c650 con 0x5650dbfd1070 2024-07-27T06:47:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c827fc640 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc3826c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc38e050 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7c84012570 tx=0x7f7c84012810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c74ceefffab0d67 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c727fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c727fc640 1 -- 172.21.15.29:0/1361683576 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84016040 con 0x5650dbfd1070 2024-07-27T06:47:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c727fc640 1 -- 172.21.15.29:0/1361683576 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7c84013c70 con 0x5650dbfd1070 2024-07-27T06:47:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.619+0000 7f7c97c01740 1 -- 172.21.15.29:0/1361683576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c8401bb10 con 0x5650dbfd1070 2024-07-27T06:47:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.620+0000 7f7c727fc640 1 -- 172.21.15.29:0/1361683576 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7c84013e70 con 0x5650dbfd1070 2024-07-27T06:47:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.620+0000 7f7c97c01740 10 client.15696 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.620+0000 7f7c97c01740 1 -- 172.21.15.29:0/1361683576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5650dc390d80 con 0x5650dbfd1070 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.620+0000 7f7c727fc640 1 -- 172.21.15.29:0/1361683576 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c8401bb10 con 0x5650dbfd1070 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.620+0000 7f7c727fc640 1 -- 172.21.15.29:0/1361683576 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7c8401dce0 con 0x5650dbfd1070 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.620+0000 7f7c97c01740 10 client.15696 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.620+0000 7f7c97c01740 10 client.15696 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.620+0000 7f7c97c01740 20 client.15696 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.620+0000 7f7c97c01740 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5650dbf8f9f0 0x5650dc394140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.620+0000 7f7c97c01740 4 client.15696 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.620+0000 7f7c97c01740 1 -- 172.21.15.29:0/1361683576 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650dc2eeae0 con 0x5650dbf8f9f0 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.620+0000 7f7c827fc640 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5650dbf8f9f0 0x5650dc394140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.621+0000 7f7c827fc640 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5650dbf8f9f0 0x5650dc394140 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10dba3ce44505823 server_cookie=98ea75762de40027 in_seq=0 out_seq=0 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.621+0000 7f7c727fc640 10 client.15696 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.621+0000 7f7c727fc640 1 -- 172.21.15.29:0/1361683576 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5650dc2eeae0 con 0x5650dbf8f9f0 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c727fc640 10 client.15696 handle_command_reply: tid=0 2024-07-27T06:47:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c97c01740 1 client.15696 shutdown 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c97c01740 1 -- 172.21.15.29:0/1361683576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 msgr2=0x5650dc38e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c97c01740 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc38e050 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7c84012570 tx=0x7f7c84012810 comp rx=0 tx=0).stop 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c97c01740 1 -- 172.21.15.29:0/1361683576 shutdown_connections 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c97c01740 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc38e050 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c97c01740 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc3826c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c97c01740 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5650dbf8f9f0 0x5650dc394140 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c97c01740 1 --2- 172.21.15.29:0/1361683576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc382180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c97c01740 1 -- 172.21.15.29:0/1361683576 >> 172.21.15.29:0/1361683576 conn(0x5650dbfc5460 msgr2=0x5650dc372c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c727fc640 0 client.15696 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c97c01740 1 -- 172.21.15.29:0/1361683576 shutdown_connections 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c97c01740 1 -- 172.21.15.29:0/1361683576 wait complete. 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.622+0000 7f7c97c01740 20 client.15696 trim_cache size 0 max 16384 2024-07-27T06:47:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.641+0000 7f7c97c01740 1 Processor -- start 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.641+0000 7f7c97c01740 1 -- start start 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.642+0000 7f7c97c01740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc383bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.642+0000 7f7c97c01740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfb0390 0x5650dc38dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.642+0000 7f7c97c01740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfd1070 0x5650dc38e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.642+0000 7f7c97c01740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c1b13d0 con 0x5650dbfb0390 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.642+0000 7f7c97c01740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c1b2f10 con 0x5650dbfd1070 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.642+0000 7f7c97c01740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c1b1aa0 con 0x5650dbfa81b0 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.642+0000 7f7c827fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfb0390 0x5650dc38dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.642+0000 7f7c827fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfb0390 0x5650dc38dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51030/0 (socket says 172.21.15.29:51030) 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.642+0000 7f7c827fc640 1 -- 172.21.15.29:0/1369091877 learned_addr learned my addr 172.21.15.29:0/1369091877 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.642+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/1369091877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc383bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.642+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/1369091877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfd1070 0x5650dc38e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.643+0000 7f7c827fc640 1 -- 172.21.15.29:0/1369091877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 msgr2=0x5650dc383bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.643+0000 7f7c827fc640 1 --2- 172.21.15.29:0/1369091877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc383bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.643+0000 7f7c827fc640 1 -- 172.21.15.29:0/1369091877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfd1070 msgr2=0x5650dc38e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.643+0000 7f7c827fc640 1 --2- 172.21.15.29:0/1369091877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfd1070 0x5650dc38e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.643+0000 7f7c827fc640 1 -- 172.21.15.29:0/1369091877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650dc38ea40 con 0x5650dbfb0390 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.643+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/1369091877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc383bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.643+0000 7f7c827fc640 1 --2- 172.21.15.29:0/1369091877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfb0390 0x5650dc38dfc0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7c74031440 tx=0x7f7c74005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a132a1e98554de server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.643+0000 7f7c80ff9640 1 -- 172.21.15.29:0/1369091877 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c74035040 con 0x5650dbfb0390 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.643+0000 7f7c80ff9640 1 -- 172.21.15.29:0/1369091877 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c7404a070 con 0x5650dbfb0390 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.643+0000 7f7c80ff9640 1 -- 172.21.15.29:0/1369091877 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c740049b0 con 0x5650dbfb0390 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.644+0000 7f7c97c01740 1 -- 172.21.15.29:0/1369091877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfb0390 msgr2=0x5650dc38dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.644+0000 7f7c97c01740 1 --2- 172.21.15.29:0/1369091877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfb0390 0x5650dc38dfc0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7c74031440 tx=0x7f7c74005e90 comp rx=0 tx=0).stop 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.644+0000 7f7c97c01740 1 -- 172.21.15.29:0/1369091877 shutdown_connections 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.644+0000 7f7c97c01740 1 --2- 172.21.15.29:0/1369091877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfd1070 0x5650dc38e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.644+0000 7f7c97c01740 1 --2- 172.21.15.29:0/1369091877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfb0390 0x5650dc38dfc0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.644+0000 7f7c97c01740 1 --2- 172.21.15.29:0/1369091877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc383bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.644+0000 7f7c97c01740 1 -- 172.21.15.29:0/1369091877 >> 172.21.15.29:0/1369091877 conn(0x5650dbfc5460 msgr2=0x5650dc3834a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.644+0000 7f7c97c01740 1 -- 172.21.15.29:0/1369091877 shutdown_connections 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.644+0000 7f7c97c01740 1 -- 172.21.15.29:0/1369091877 wait complete. 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c97c01740 1 -- start start 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c97c01740 1 Processor -- start 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c97c01740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc394470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c97c01740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc394a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c97c01740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc3fa680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c97c01740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c1b2840 con 0x5650dbfd1070 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c97c01740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c1b2170 con 0x5650dbfb0390 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c97c01740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c1af3b0 con 0x5650dbfa81b0 2024-07-27T06:47:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c97c01740 10 client.0 _ll_register_callbacks cb 0x7f7c74009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:44.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c8b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc3fa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c8b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc3fa680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51034/0 (socket says 172.21.15.29:51034) 2024-07-27T06:47:44.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c8b7fe640 1 -- 172.21.15.29:0/2603376760 learned_addr learned my addr 172.21.15.29:0/2603376760 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:44.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c827fc640 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc394a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.645+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc394470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c8b7fe640 1 -- 172.21.15.29:0/2603376760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 msgr2=0x5650dc394470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:44.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc394470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c8b7fe640 1 -- 172.21.15.29:0/2603376760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 msgr2=0x5650dc394a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:44.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc394a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c8b7fe640 1 -- 172.21.15.29:0/2603376760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650dc3fabc0 con 0x5650dbfd1070 2024-07-27T06:47:44.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c8b7fe640 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc3fa680 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x5650dc384b30 tx=0x7f7c84012810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5173f763e31f090e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c727fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c727fc640 1 -- 172.21.15.29:0/2603376760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84016050 con 0x5650dbfd1070 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c727fc640 1 -- 172.21.15.29:0/2603376760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7c84004a00 con 0x5650dbfd1070 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c97c01740 1 -- 172.21.15.29:0/2603376760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c7404a070 con 0x5650dbfd1070 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c727fc640 1 -- 172.21.15.29:0/2603376760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7c84044050 con 0x5650dbfd1070 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c97c01740 10 client.15702 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.646+0000 7f7c97c01740 1 -- 172.21.15.29:0/2603376760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5650dc3fbcc0 con 0x5650dbfd1070 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.647+0000 7f7c727fc640 1 -- 172.21.15.29:0/2603376760 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c7404a070 con 0x5650dbfd1070 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.647+0000 7f7c727fc640 1 -- 172.21.15.29:0/2603376760 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7c8400f040 con 0x5650dbfd1070 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.647+0000 7f7c97c01740 10 client.15702 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.647+0000 7f7c97c01740 10 client.15702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.647+0000 7f7c97c01740 20 client.15702 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:44.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.647+0000 7f7c97c01740 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5650dbf64da0 0x5650dc3fe3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.647+0000 7f7c97c01740 4 client.15702 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.647+0000 7f7c97c01740 1 -- 172.21.15.29:0/2603376760 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5650dc2ea580 con 0x5650dbf64da0 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.647+0000 7f7c827fc640 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5650dbf64da0 0x5650dc3fe3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c727fc640 10 client.15702 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c827fc640 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5650dbf64da0 0x5650dc3fe3c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=925784e616507e23 server_cookie=76dbd15557ab0561 in_seq=0 out_seq=0 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c727fc640 1 -- 172.21.15.29:0/2603376760 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5650dc2ea580 con 0x5650dbf64da0 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c727fc640 10 client.15702 handle_command_reply: tid=0 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c97c01740 1 client.15702 shutdown 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c97c01740 1 -- 172.21.15.29:0/2603376760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 msgr2=0x5650dc3fa680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c97c01740 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc3fa680 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x5650dc384b30 tx=0x7f7c84012810 comp rx=0 tx=0).stop 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c97c01740 1 -- 172.21.15.29:0/2603376760 shutdown_connections 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c97c01740 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5650dbfd1070 0x5650dc3fa680 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c97c01740 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5650dbfb0390 0x5650dc394a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c97c01740 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5650dbf64da0 0x5650dc3fe3c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c97c01740 1 --2- 172.21.15.29:0/2603376760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5650dbfa81b0 0x5650dc394470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.648+0000 7f7c97c01740 1 -- 172.21.15.29:0/2603376760 >> 172.21.15.29:0/2603376760 conn(0x5650dbfc5460 msgr2=0x5650dc377580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.649+0000 7f7c727fc640 0 client.15702 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.649+0000 7f7c97c01740 1 -- 172.21.15.29:0/2603376760 shutdown_connections 2024-07-27T06:47:44.258 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:47:44.258 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-07-27T06:47:44.258 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-07-27T06:47:44.258 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:47:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.649+0000 7f7c97c01740 1 -- 172.21.15.29:0/2603376760 wait complete. 2024-07-27T06:47:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.649+0000 7f7c97c01740 20 client.15702 trim_cache size 0 max 16384 2024-07-27T06:47:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.649+0000 7f7c9167d640 1 -- 172.21.15.29:0/1313225396 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7c780b3d30 msgr2=0x7f7c780b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.649+0000 7f7c9167d640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7c780b3d30 0x7f7c780b6240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5650dc3751d0 tx=0x7f7c7c000bc0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.649+0000 7f7c9167d640 1 -- 172.21.15.29:0/1313225396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c102080 msgr2=0x7f7c8c19c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.649+0000 7f7c9167d640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c102080 0x7f7c8c19c2a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c00c890 tx=0x7f7c7c00cd50 comp rx=0 tx=0).stop 2024-07-27T06:47:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.649+0000 7f7c9167d640 1 -- 172.21.15.29:0/1313225396 shutdown_connections 2024-07-27T06:47:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.650+0000 7f7c9167d640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c8c10fc30 0x7f7c8c1a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.650+0000 7f7c9167d640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7c780b3d30 0x7f7c780b6240 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.650+0000 7f7c9167d640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c1029d0 0x7f7c8c19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.650+0000 7f7c9167d640 1 --2- 172.21.15.29:0/1313225396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c8c102080 0x7f7c8c19c2a0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:44.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.650+0000 7f7c9167d640 1 -- 172.21.15.29:0/1313225396 >> 172.21.15.29:0/1313225396 conn(0x7f7c8c0791a0 msgr2=0x7f7c8c109860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:44.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.650+0000 7f7c9167d640 1 -- 172.21.15.29:0/1313225396 shutdown_connections 2024-07-27T06:47:44.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:43.650+0000 7f7c9167d640 1 -- 172.21.15.29:0/1313225396 wait complete. 2024-07-27T06:47:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:44 smithi029 ceph-mon[32610]: pgmap v419: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-07-27T06:47:45.005 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-27T06:47:45.006 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:47:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:44 smithi077 ceph-mon[37153]: pgmap v419: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-07-27T06:47:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:44 smithi047 ceph-mon[36835]: pgmap v419: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-07-27T06:47:45.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.i status 2024-07-27T06:47:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:46 smithi029 ceph-mon[32610]: pgmap v420: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-07-27T06:47:47.034 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:46 smithi077 ceph-mon[37153]: pgmap v420: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-07-27T06:47:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:46 smithi047 ceph-mon[36835]: pgmap v420: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-07-27T06:47:47.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.280+0000 7f7c4fc56640 1 -- 172.21.15.29:0/4048103001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c48102490 msgr2=0x7f7c481028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:47.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.280+0000 7f7c4fc56640 1 --2- 172.21.15.29:0/4048103001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c48102490 0x7f7c481028a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009900 tx=0x7f7c4402a410 comp rx=0 tx=0).stop 2024-07-27T06:47:47.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.280+0000 7f7c4fc56640 1 -- 172.21.15.29:0/4048103001 shutdown_connections 2024-07-27T06:47:47.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.280+0000 7f7c4fc56640 1 --2- 172.21.15.29:0/4048103001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c48108720 0x7f7c4810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:47.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.280+0000 7f7c4fc56640 1 --2- 172.21.15.29:0/4048103001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48107dd0 0x7f7c481081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:47.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.280+0000 7f7c4fc56640 1 --2- 172.21.15.29:0/4048103001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c48102490 0x7f7c481028a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:47.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.280+0000 7f7c4fc56640 1 -- 172.21.15.29:0/4048103001 >> 172.21.15.29:0/4048103001 conn(0x7f7c48079160 msgr2=0x7f7c48079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:47.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.280+0000 7f7c4fc56640 1 -- 172.21.15.29:0/4048103001 shutdown_connections 2024-07-27T06:47:47.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.280+0000 7f7c4fc56640 1 -- 172.21.15.29:0/4048103001 wait complete. 2024-07-27T06:47:47.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.280+0000 7f7c4fc56640 1 Processor -- start 2024-07-27T06:47:47.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4fc56640 1 -- start start 2024-07-27T06:47:47.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4fc56640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c48102490 0x7f7c48103c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:47.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4fc56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48107dd0 0x7f7c481041c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:47.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4fc56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c48108720 0x7f7c48104700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:47.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4fc56640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7c481123a0 con 0x7f7c48108720 2024-07-27T06:47:47.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4fc56640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48112220 con 0x7f7c48102490 2024-07-27T06:47:47.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4fc56640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48112520 con 0x7f7c48107dd0 2024-07-27T06:47:47.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4e1f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c48108720 0x7f7c48104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:47.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4e1f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c48108720 0x7f7c48104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51116/0 (socket says 172.21.15.29:51116) 2024-07-27T06:47:47.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4e1f3640 1 -- 172.21.15.29:0/2276437162 learned_addr learned my addr 172.21.15.29:0/2276437162 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:47.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4d1f1640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48107dd0 0x7f7c481041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:47.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4d9f2640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c48102490 0x7f7c48103c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:47.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4e1f3640 1 -- 172.21.15.29:0/2276437162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48107dd0 msgr2=0x7f7c481041c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:47.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4e1f3640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48107dd0 0x7f7c481041c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:47.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4e1f3640 1 -- 172.21.15.29:0/2276437162 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c48102490 msgr2=0x7f7c48103c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:47.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4e1f3640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c48102490 0x7f7c48103c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:47.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.281+0000 7f7c4e1f3640 1 -- 172.21.15.29:0/2276437162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c481a80d0 con 0x7f7c48108720 2024-07-27T06:47:47.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.282+0000 7f7c4e1f3640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c48108720 0x7f7c48104700 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000cce0 tx=0x7f7c4000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:47.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.282+0000 7f7c3effd640 1 -- 172.21.15.29:0/2276437162 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4000f450 con 0x7f7c48108720 2024-07-27T06:47:47.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.282+0000 7f7c3effd640 1 -- 172.21.15.29:0/2276437162 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c4000faa0 con 0x7f7c48108720 2024-07-27T06:47:47.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.282+0000 7f7c3effd640 1 -- 172.21.15.29:0/2276437162 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40018b70 con 0x7f7c48108720 2024-07-27T06:47:47.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.282+0000 7f7c4fc56640 1 -- 172.21.15.29:0/2276437162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c481a83c0 con 0x7f7c48108720 2024-07-27T06:47:47.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.282+0000 7f7c4fc56640 1 -- 172.21.15.29:0/2276437162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c481a88b0 con 0x7f7c48108720 2024-07-27T06:47:47.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.284+0000 7f7c4fc56640 1 -- 172.21.15.29:0/2276437162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c481028a0 con 0x7f7c48108720 2024-07-27T06:47:47.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.285+0000 7f7c3effd640 1 -- 172.21.15.29:0/2276437162 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f7c40018d10 con 0x7f7c48108720 2024-07-27T06:47:47.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.286+0000 7f7c3effd640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7c340b3d30 0x7f7c340b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:47.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.286+0000 7f7c4d9f2640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7c340b3d30 0x7f7c340b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:47.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.286+0000 7f7c3effd640 1 -- 172.21.15.29:0/2276437162 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7c400e43a0 con 0x7f7c48108720 2024-07-27T06:47:47.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.287+0000 7f7c4d9f2640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7c340b3d30 0x7f7c340b6240 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7c44000c00 tx=0x7f7c44031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:47.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.295+0000 7f7c3effd640 1 -- 172.21.15.29:0/2276437162 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7c40014030 con 0x7f7c48108720 2024-07-27T06:47:47.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.547+0000 7f7c4fc56640 1 -- 172.21.15.29:0/2276437162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c481a8d30 con 0x7f7c48108720 2024-07-27T06:47:47.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.548+0000 7f7c3effd640 1 -- 172.21.15.29:0/2276437162 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11957 (secure 0 0 0) 0x7f7c400ab780 con 0x7f7c48108720 2024-07-27T06:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:47:47.550 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-07-27T06:46:50:421100+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:46:49.561737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24670,24658,24646,24617,24613]},"id":1}]} 2024-07-27T06:47:47.550 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-07-27T06:47:47.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.551+0000 7f7c4fc56640 1 -- 172.21.15.29:0/2276437162 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7c340b3d30 msgr2=0x7f7c340b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:47.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.551+0000 7f7c4fc56640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7c340b3d30 0x7f7c340b6240 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7c44000c00 tx=0x7f7c44031400 comp rx=0 tx=0).stop 2024-07-27T06:47:47.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.551+0000 7f7c4fc56640 1 -- 172.21.15.29:0/2276437162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c48108720 msgr2=0x7f7c48104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:47.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.551+0000 7f7c4fc56640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c48108720 0x7f7c48104700 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000cce0 tx=0x7f7c4000a620 comp rx=0 tx=0).stop 2024-07-27T06:47:47.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.551+0000 7f7c4fc56640 1 -- 172.21.15.29:0/2276437162 shutdown_connections 2024-07-27T06:47:47.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.551+0000 7f7c4fc56640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c48108720 0x7f7c48104700 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:47.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.551+0000 7f7c4fc56640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7c340b3d30 0x7f7c340b6240 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:47.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.551+0000 7f7c4fc56640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c48107dd0 0x7f7c481041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:47.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.551+0000 7f7c4fc56640 1 --2- 172.21.15.29:0/2276437162 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7c48102490 0x7f7c48103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:47.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.551+0000 7f7c4fc56640 1 -- 172.21.15.29:0/2276437162 >> 172.21.15.29:0/2276437162 conn(0x7f7c48079160 msgr2=0x7f7c48101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:47.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.551+0000 7f7c4fc56640 1 -- 172.21.15.29:0/2276437162 shutdown_connections 2024-07-27T06:47:47.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:47.551+0000 7f7c4fc56640 1 -- 172.21.15.29:0/2276437162 wait complete. 2024-07-27T06:47:47.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:47 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2276437162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:47:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:47 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2276437162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:47:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:47 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2276437162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:47:48.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-27T06:47:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:48 smithi077 ceph-mon[37153]: pgmap v421: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 426 B/s wr, 12 op/s 2024-07-27T06:47:49.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:48 smithi029 ceph-mon[32610]: pgmap v421: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 426 B/s wr, 12 op/s 2024-07-27T06:47:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:48 smithi047 ceph-mon[36835]: pgmap v421: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 426 B/s wr, 12 op/s 2024-07-27T06:47:49.626 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:50.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.357+0000 7f7aec9a5640 1 -- 172.21.15.29:0/3394933709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae8107dd0 msgr2=0x7f7ae81081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.357+0000 7f7aec9a5640 1 --2- 172.21.15.29:0/3394933709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae8107dd0 0x7f7ae81081e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009900 tx=0x7f7adc02a410 comp rx=0 tx=0).stop 2024-07-27T06:47:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.358+0000 7f7aec9a5640 1 -- 172.21.15.29:0/3394933709 shutdown_connections 2024-07-27T06:47:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.358+0000 7f7aec9a5640 1 --2- 172.21.15.29:0/3394933709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ae8108720 0x7f7ae810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.358+0000 7f7aec9a5640 1 --2- 172.21.15.29:0/3394933709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae8107dd0 0x7f7ae81081e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.358+0000 7f7aec9a5640 1 --2- 172.21.15.29:0/3394933709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae8102490 0x7f7ae81028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.358+0000 7f7aec9a5640 1 -- 172.21.15.29:0/3394933709 >> 172.21.15.29:0/3394933709 conn(0x7f7ae8079160 msgr2=0x7f7ae8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.358+0000 7f7aec9a5640 1 -- 172.21.15.29:0/3394933709 shutdown_connections 2024-07-27T06:47:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.358+0000 7f7aec9a5640 1 -- 172.21.15.29:0/3394933709 wait complete. 2024-07-27T06:47:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.358+0000 7f7aec9a5640 1 Processor -- start 2024-07-27T06:47:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.359+0000 7f7aec9a5640 1 -- start start 2024-07-27T06:47:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.359+0000 7f7aec9a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae8102490 0x7f7ae819c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.359+0000 7f7aec9a5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ae8107dd0 0x7f7ae819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.359+0000 7f7aec9a5640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae8108720 0x7f7ae81a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.359+0000 7f7aec9a5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae8112350 con 0x7f7ae8102490 2024-07-27T06:47:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.359+0000 7f7aec9a5640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae81121d0 con 0x7f7ae8108720 2024-07-27T06:47:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.359+0000 7f7aec9a5640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae81124d0 con 0x7f7ae8107dd0 2024-07-27T06:47:50.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.359+0000 7f7ae659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae8102490 0x7f7ae819c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.359+0000 7f7ae659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae8102490 0x7f7ae819c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46156/0 (socket says 172.21.15.29:46156) 2024-07-27T06:47:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.359+0000 7f7ae659c640 1 -- 172.21.15.29:0/4289175299 learned_addr learned my addr 172.21.15.29:0/4289175299 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.359+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ae8107dd0 0x7f7ae819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.359+0000 7f7ae6d9d640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae8108720 0x7f7ae81a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.360+0000 7f7ae659c640 1 -- 172.21.15.29:0/4289175299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ae8107dd0 msgr2=0x7f7ae819c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.360+0000 7f7ae659c640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ae8107dd0 0x7f7ae819c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.360+0000 7f7ae659c640 1 -- 172.21.15.29:0/4289175299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae8108720 msgr2=0x7f7ae81a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.360+0000 7f7ae659c640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae8108720 0x7f7ae81a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.360+0000 7f7ae659c640 1 -- 172.21.15.29:0/4289175299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae81a1fc0 con 0x7f7ae8102490 2024-07-27T06:47:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.360+0000 7f7ae659c640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae8102490 0x7f7ae819c050 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800e9f0 tx=0x7f7ad800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.360+0000 7f7ad77fe640 1 -- 172.21.15.29:0/4289175299 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad800cd80 con 0x7f7ae8102490 2024-07-27T06:47:50.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.360+0000 7f7aec9a5640 1 -- 172.21.15.29:0/4289175299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae81a22b0 con 0x7f7ae8102490 2024-07-27T06:47:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.360+0000 7f7ad77fe640 1 -- 172.21.15.29:0/4289175299 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ad80108e0 con 0x7f7ae8102490 2024-07-27T06:47:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.360+0000 7f7ad77fe640 1 -- 172.21.15.29:0/4289175299 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8018980 con 0x7f7ae8102490 2024-07-27T06:47:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.360+0000 7f7aec9a5640 1 -- 172.21.15.29:0/4289175299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae81a8a10 con 0x7f7ae8102490 2024-07-27T06:47:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.363+0000 7f7af2f29740 1 Processor -- start 2024-07-27T06:47:50.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.363+0000 7f7af2f29740 1 -- start start 2024-07-27T06:47:50.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.363+0000 7f7af2f29740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7cef60 0x563e0fb743a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.363+0000 7f7ad77fe640 1 -- 172.21.15.29:0/4289175299 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f7ad8010430 con 0x7f7ae8102490 2024-07-27T06:47:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.363+0000 7f7af2f29740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb748e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.363+0000 7f7af2f29740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb74e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.363+0000 7f7af2f29740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563e0faea940 con 0x563e0f7a60a0 2024-07-27T06:47:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.363+0000 7f7af2f29740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x563e0fad9e40 con 0x563e0f7cef60 2024-07-27T06:47:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.363+0000 7f7af2f29740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e0fadde80 con 0x563e0f7ae280 2024-07-27T06:47:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.363+0000 7f7ae659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb748e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.363+0000 7f7ae659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb748e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46162/0 (socket says 172.21.15.29:46162) 2024-07-27T06:47:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.363+0000 7f7ae659c640 1 -- 172.21.15.29:0/858917694 learned_addr learned my addr 172.21.15.29:0/858917694 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.364+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/858917694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb74e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.364+0000 7f7ae6d9d640 1 --2- 172.21.15.29:0/858917694 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7cef60 0x563e0fb743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.364+0000 7f7ae5d9b640 1 -- 172.21.15.29:0/858917694 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7cef60 msgr2=0x563e0fb743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:50.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.364+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/858917694 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7cef60 0x563e0fb743a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.364+0000 7f7ae5d9b640 1 -- 172.21.15.29:0/858917694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 msgr2=0x563e0fb748e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:50.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.364+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/858917694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb748e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.364+0000 7f7ae5d9b640 1 -- 172.21.15.29:0/858917694 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e0fa80a60 con 0x563e0f7ae280 2024-07-27T06:47:50.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.364+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/858917694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb74e20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7adc006010 tx=0x7f7adc032bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1d2c8e603bf2c33 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:50.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.365+0000 7f7ad4ff9640 1 -- 172.21.15.29:0/858917694 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc040070 con 0x563e0f7ae280 2024-07-27T06:47:50.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.365+0000 7f7ad77fe640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7ad00b3cb0 0x7f7ad00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.365+0000 7f7ad4ff9640 1 -- 172.21.15.29:0/858917694 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7adc004030 con 0x563e0f7ae280 2024-07-27T06:47:50.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.365+0000 7f7ad4ff9640 1 -- 172.21.15.29:0/858917694 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc035040 con 0x563e0f7ae280 2024-07-27T06:47:50.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.365+0000 7f7ae6d9d640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7ad00b3cb0 0x7f7ad00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.365+0000 7f7ad77fe640 1 -- 172.21.15.29:0/4289175299 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7ad80e41f0 con 0x7f7ae8102490 2024-07-27T06:47:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.366+0000 7f7af2f29740 1 -- 172.21.15.29:0/858917694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 msgr2=0x563e0fb74e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.366+0000 7f7ae6d9d640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7ad00b3cb0 0x7f7ad00b61c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0005fd0 tx=0x7f7ae0009600 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.366+0000 7f7af2f29740 1 --2- 172.21.15.29:0/858917694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb74e20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7adc006010 tx=0x7f7adc032bd0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.366+0000 7f7af2f29740 1 -- 172.21.15.29:0/858917694 shutdown_connections 2024-07-27T06:47:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.366+0000 7f7af2f29740 1 --2- 172.21.15.29:0/858917694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb74e20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.366+0000 7f7af2f29740 1 --2- 172.21.15.29:0/858917694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.366+0000 7f7af2f29740 1 --2- 172.21.15.29:0/858917694 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7cef60 0x563e0fb743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.366+0000 7f7af2f29740 1 -- 172.21.15.29:0/858917694 >> 172.21.15.29:0/858917694 conn(0x563e0f7c3350 msgr2=0x563e0fb72290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:50.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.366+0000 7f7af2f29740 1 -- 172.21.15.29:0/858917694 shutdown_connections 2024-07-27T06:47:50.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.366+0000 7f7af2f29740 1 -- 172.21.15.29:0/858917694 wait complete. 2024-07-27T06:47:50.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.367+0000 7f7af2f29740 1 -- start start 2024-07-27T06:47:50.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.367+0000 7f7af2f29740 1 Processor -- start 2024-07-27T06:47:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.367+0000 7f7af2f29740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7a60a0 0x563e0fb7ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.367+0000 7f7af2f29740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7ae280 0x563e0fb804c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.367+0000 7f7af2f29740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7cef60 0x563e0fb8be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.367+0000 7f7af2f29740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563e0f93d5f0 con 0x563e0f7ae280 2024-07-27T06:47:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.367+0000 7f7af2f29740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x563e0fa7c8b0 con 0x563e0f7cef60 2024-07-27T06:47:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.367+0000 7f7af2f29740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e0fb82b20 con 0x563e0f7a60a0 2024-07-27T06:47:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.367+0000 7f7af2f29740 10 client.0 _ll_register_callbacks cb 0x563e0fa867f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.367+0000 7f7ae5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7ae280 0x563e0fb804c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.367+0000 7f7ae5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7ae280 0x563e0fb804c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46174/0 (socket says 172.21.15.29:46174) 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.367+0000 7f7ae5d9b640 1 -- 172.21.15.29:0/3392605682 learned_addr learned my addr 172.21.15.29:0/3392605682 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7ae659c640 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7a60a0 0x563e0fb7ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7ae6d9d640 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7cef60 0x563e0fb8be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7ae5d9b640 1 -- 172.21.15.29:0/3392605682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7a60a0 msgr2=0x563e0fb7ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7a60a0 0x563e0fb7ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7ae5d9b640 1 -- 172.21.15.29:0/3392605682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7cef60 msgr2=0x563e0fb8be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7cef60 0x563e0fb8be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7ae5d9b640 1 -- 172.21.15.29:0/3392605682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e0f83a540 con 0x563e0f7ae280 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7ae6d9d640 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7cef60 0x563e0fb8be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7ae280 0x563e0fb804c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7adc000c00 tx=0x7f7adc005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3386e0fcea203ef server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7acdffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7acdffb640 1 -- 172.21.15.29:0/3392605682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc04c070 con 0x563e0f7ae280 2024-07-27T06:47:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7af2f29740 1 -- 172.21.15.29:0/3392605682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7adc004030 con 0x563e0f7ae280 2024-07-27T06:47:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.368+0000 7f7acdffb640 1 -- 172.21.15.29:0/3392605682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7adc0054c0 con 0x563e0f7ae280 2024-07-27T06:47:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.369+0000 7f7acdffb640 1 -- 172.21.15.29:0/3392605682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7adc058050 con 0x563e0f7ae280 2024-07-27T06:47:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.369+0000 7f7af2f29740 10 client.15738 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.369+0000 7f7af2f29740 1 -- 172.21.15.29:0/3392605682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563e0fb8eb00 con 0x563e0f7ae280 2024-07-27T06:47:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.369+0000 7f7acdffb640 1 -- 172.21.15.29:0/3392605682 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7adc004030 con 0x563e0f7ae280 2024-07-27T06:47:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.369+0000 7f7acdffb640 1 -- 172.21.15.29:0/3392605682 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7adc003c10 con 0x563e0f7ae280 2024-07-27T06:47:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.369+0000 7f7af2f29740 10 client.15738 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.369+0000 7f7af2f29740 10 client.15738 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-27T06:47:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.369+0000 7f7af2f29740 20 client.15738 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.369+0000 7f7af2f29740 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x563e0f78d8e0 0x563e0fb926f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.369+0000 7f7af2f29740 4 client.15738 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.369+0000 7f7af2f29740 1 -- 172.21.15.29:0/3392605682 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e0faec890 con 0x563e0f78d8e0 2024-07-27T06:47:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.370+0000 7f7ae659c640 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x563e0f78d8e0 0x563e0fb926f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.370+0000 7f7ae659c640 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x563e0f78d8e0 0x563e0fb926f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8edc0d9b6694df88 server_cookie=30b2ff5fac6c655b in_seq=0 out_seq=0 2024-07-27T06:47:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.370+0000 7f7acdffb640 10 client.15738 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7acdffb640 1 -- 172.21.15.29:0/3392605682 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563e0faec890 con 0x563e0f78d8e0 2024-07-27T06:47:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7acdffb640 10 client.15738 handle_command_reply: tid=0 2024-07-27T06:47:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7af2f29740 1 client.15738 shutdown 2024-07-27T06:47:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7af2f29740 1 -- 172.21.15.29:0/3392605682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7ae280 msgr2=0x563e0fb804c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7af2f29740 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7ae280 0x563e0fb804c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7adc000c00 tx=0x7f7adc005ea0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7af2f29740 1 -- 172.21.15.29:0/3392605682 shutdown_connections 2024-07-27T06:47:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7af2f29740 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7cef60 0x563e0fb8be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7af2f29740 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7ae280 0x563e0fb804c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7af2f29740 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x563e0f78d8e0 0x563e0fb926f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7af2f29740 1 --2- 172.21.15.29:0/3392605682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7a60a0 0x563e0fb7ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7af2f29740 1 -- 172.21.15.29:0/3392605682 >> 172.21.15.29:0/3392605682 conn(0x563e0f7c3350 msgr2=0x563e0fb77580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7acdffb640 0 client.15738 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.371+0000 7f7af2f29740 1 -- 172.21.15.29:0/3392605682 shutdown_connections 2024-07-27T06:47:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.372+0000 7f7af2f29740 1 -- 172.21.15.29:0/3392605682 wait complete. 2024-07-27T06:47:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.372+0000 7f7af2f29740 20 client.15738 trim_cache size 0 max 16384 2024-07-27T06:47:50.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.390+0000 7f7af2f29740 1 Processor -- start 2024-07-27T06:47:50.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.390+0000 7f7af2f29740 1 -- start start 2024-07-27T06:47:50.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.390+0000 7f7af2f29740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7c3350 0x563e0fb7f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.390+0000 7f7af2f29740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb8bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.390+0000 7f7af2f29740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb8c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.390+0000 7f7af2f29740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563e0fb90a30 con 0x563e0f7a60a0 2024-07-27T06:47:50.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.390+0000 7f7af2f29740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x563e0fb91100 con 0x563e0f7c3350 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.390+0000 7f7af2f29740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e0fb917d0 con 0x563e0f7ae280 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7ae5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb8bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7ae5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb8bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46184/0 (socket says 172.21.15.29:46184) 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7ae5d9b640 1 -- 172.21.15.29:0/2397991690 learned_addr learned my addr 172.21.15.29:0/2397991690 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7ae659c640 1 --2- 172.21.15.29:0/2397991690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7c3350 0x563e0fb7f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7ae6d9d640 1 --2- 172.21.15.29:0/2397991690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb8c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7ae5d9b640 1 -- 172.21.15.29:0/2397991690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 msgr2=0x563e0fb8c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/2397991690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb8c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7ae5d9b640 1 -- 172.21.15.29:0/2397991690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7c3350 msgr2=0x563e0fb7f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/2397991690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7c3350 0x563e0fb7f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7ae5d9b640 1 -- 172.21.15.29:0/2397991690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e0fb80920 con 0x563e0f7a60a0 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7ae659c640 1 --2- 172.21.15.29:0/2397991690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7c3350 0x563e0fb7f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:50.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/2397991690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb8bd80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0095d0 tx=0x7f7adc0045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e420b87ab08a06 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:50.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.391+0000 7f7acffff640 1 -- 172.21.15.29:0/2397991690 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc04c070 con 0x563e0f7a60a0 2024-07-27T06:47:50.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.392+0000 7f7acffff640 1 -- 172.21.15.29:0/2397991690 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7adc0035f0 con 0x563e0f7a60a0 2024-07-27T06:47:50.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.392+0000 7f7acffff640 1 -- 172.21.15.29:0/2397991690 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc000e60 con 0x563e0f7a60a0 2024-07-27T06:47:50.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.392+0000 7f7af2f29740 1 -- 172.21.15.29:0/2397991690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 msgr2=0x563e0fb8bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:50.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.392+0000 7f7af2f29740 1 --2- 172.21.15.29:0/2397991690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb8bd80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0095d0 tx=0x7f7adc0045b0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.392+0000 7f7af2f29740 1 -- 172.21.15.29:0/2397991690 shutdown_connections 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.392+0000 7f7af2f29740 1 --2- 172.21.15.29:0/2397991690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb8c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.392+0000 7f7af2f29740 1 --2- 172.21.15.29:0/2397991690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb8bd80 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.392+0000 7f7af2f29740 1 --2- 172.21.15.29:0/2397991690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7c3350 0x563e0fb7f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.392+0000 7f7af2f29740 1 -- 172.21.15.29:0/2397991690 >> 172.21.15.29:0/2397991690 conn(0x563e0fb722d0 msgr2=0x563e0fb8fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.393+0000 7f7af2f29740 1 -- 172.21.15.29:0/2397991690 shutdown_connections 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.393+0000 7f7af2f29740 1 -- 172.21.15.29:0/2397991690 wait complete. 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.393+0000 7f7af2f29740 1 -- start start 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.393+0000 7f7af2f29740 1 Processor -- start 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.393+0000 7f7af2f29740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb9b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.393+0000 7f7af2f29740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb98ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.393+0000 7f7af2f29740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7c3350 0x563e0fb99580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.393+0000 7f7af2f29740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x563e0fb92570 con 0x563e0f7a60a0 2024-07-27T06:47:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.393+0000 7f7af2f29740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x563e0fb91ea0 con 0x563e0f7c3350 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.393+0000 7f7af2f29740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e0fb92c40 con 0x563e0f7ae280 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7ae659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb9b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7af2f29740 10 client.0 _ll_register_callbacks cb 0x7f7adc032b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7ae659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb9b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46186/0 (socket says 172.21.15.29:46186) 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7ae659c640 1 -- 172.21.15.29:0/2693427647 learned_addr learned my addr 172.21.15.29:0/2693427647 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb98ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7ae6d9d640 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7c3350 0x563e0fb99580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7ae659c640 1 -- 172.21.15.29:0/2693427647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 msgr2=0x563e0fb98ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7ae659c640 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb98ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7ae659c640 1 -- 172.21.15.29:0/2693427647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7c3350 msgr2=0x563e0fb99580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7ae659c640 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7c3350 0x563e0fb99580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7ae659c640 1 -- 172.21.15.29:0/2693427647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e0fb938f0 con 0x563e0f7a60a0 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7ae659c640 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb9b1f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7ad80e1b10 tx=0x7f7ad800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50694bb3205b794 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7acdffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.394+0000 7f7acdffb640 1 -- 172.21.15.29:0/2693427647 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8010720 con 0x563e0f7a60a0 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7acdffb640 1 -- 172.21.15.29:0/2693427647 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7ad80a7570 con 0x563e0f7a60a0 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7af2f29740 1 -- 172.21.15.29:0/2693427647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7adc0035f0 con 0x563e0f7a60a0 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7acdffb640 1 -- 172.21.15.29:0/2693427647 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7ad80be050 con 0x563e0f7a60a0 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7af2f29740 10 client.15744 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7af2f29740 1 -- 172.21.15.29:0/2693427647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563e0fb9a070 con 0x563e0f7a60a0 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7acdffb640 1 -- 172.21.15.29:0/2693427647 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7adc0035f0 con 0x563e0f7a60a0 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7acdffb640 1 -- 172.21.15.29:0/2693427647 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7ad80af070 con 0x563e0f7a60a0 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7af2f29740 10 client.15744 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7af2f29740 10 client.15744 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7af2f29740 20 client.15744 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7af2f29740 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x563e0f7cef60 0x563e0fc06c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7af2f29740 4 client.15744 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-27T06:47:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.395+0000 7f7af2f29740 1 -- 172.21.15.29:0/2693427647 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "status"}) -- 0x563e0fae8330 con 0x563e0f7cef60 2024-07-27T06:47:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.396+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x563e0f7cef60 0x563e0fc06c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.396+0000 7f7ae5d9b640 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x563e0f7cef60 0x563e0fc06c30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcf48437be5ab5cb server_cookie=ee16effa0f72129f in_seq=0 out_seq=0 2024-07-27T06:47:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.396+0000 7f7acdffb640 10 client.15744 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.396+0000 7f7acdffb640 1 -- 172.21.15.29:0/2693427647 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563e0fae8330 con 0x563e0f7cef60 2024-07-27T06:47:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.396+0000 7f7acdffb640 10 client.15744 handle_command_reply: tid=0 2024-07-27T06:47:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.396+0000 7f7af2f29740 1 client.15744 shutdown 2024-07-27T06:47:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.396+0000 7f7af2f29740 1 -- 172.21.15.29:0/2693427647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 msgr2=0x563e0fb9b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.396+0000 7f7af2f29740 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb9b1f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7ad80e1b10 tx=0x7f7ad800cf20 comp rx=0 tx=0).stop 2024-07-27T06:47:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7af2f29740 1 -- 172.21.15.29:0/2693427647 shutdown_connections 2024-07-27T06:47:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7af2f29740 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563e0f7c3350 0x563e0fb99580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7af2f29740 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x563e0f7cef60 0x563e0fc06c30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7af2f29740 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e0f7ae280 0x563e0fb98ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7af2f29740 1 --2- 172.21.15.29:0/2693427647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563e0f7a60a0 0x563e0fb9b1f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7af2f29740 1 -- 172.21.15.29:0/2693427647 >> 172.21.15.29:0/2693427647 conn(0x563e0fb722d0 msgr2=0x563e0fb7aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:50.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7acdffb640 0 client.15744 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:50.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7af2f29740 1 -- 172.21.15.29:0/2693427647 shutdown_connections 2024-07-27T06:47:50.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7af2f29740 1 -- 172.21.15.29:0/2693427647 wait complete. 2024-07-27T06:47:50.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7af2f29740 20 client.15744 trim_cache size 0 max 16384 2024-07-27T06:47:50.403 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:47:50.403 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "0a8a0a4c-4be2-11ef-bcc7-c7b262605968", 2024-07-27T06:47:50.404 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-07-27T06:47:50.404 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 24670, 2024-07-27T06:47:50.404 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-07-27T06:47:50.404 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-07-27T06:47:50.404 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-07-27T06:47:50.404 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 274.59829338899999, 2024-07-27T06:47:50.404 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 29, 2024-07-27T06:47:50.404 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 83, 2024-07-27T06:47:50.404 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 83, 2024-07-27T06:47:50.404 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 303.65859363300001 2024-07-27T06:47:50.404 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:47:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7aec9a5640 1 -- 172.21.15.29:0/4289175299 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7ad00b3cb0 msgr2=0x7f7ad00b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7aec9a5640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7ad00b3cb0 0x7f7ad00b61c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0005fd0 tx=0x7f7ae0009600 comp rx=0 tx=0).stop 2024-07-27T06:47:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7aec9a5640 1 -- 172.21.15.29:0/4289175299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae8102490 msgr2=0x7f7ae819c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7aec9a5640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae8102490 0x7f7ae819c050 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800e9f0 tx=0x7f7ad800eeb0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7aec9a5640 1 -- 172.21.15.29:0/4289175299 shutdown_connections 2024-07-27T06:47:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7aec9a5640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae8108720 0x7f7ae81a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7aec9a5640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7ad00b3cb0 0x7f7ad00b61c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7aec9a5640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ae8107dd0 0x7f7ae819c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.397+0000 7f7aec9a5640 1 --2- 172.21.15.29:0/4289175299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ae8102490 0x7f7ae819c050 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:50.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.398+0000 7f7aec9a5640 1 -- 172.21.15.29:0/4289175299 >> 172.21.15.29:0/4289175299 conn(0x7f7ae8079160 msgr2=0x7f7ae8103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:50.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.398+0000 7f7aec9a5640 1 -- 172.21.15.29:0/4289175299 shutdown_connections 2024-07-27T06:47:50.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:50.398+0000 7f7aec9a5640 1 -- 172.21.15.29:0/4289175299 wait complete. 2024-07-27T06:47:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:50 smithi029 ceph-mon[32610]: pgmap v422: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-27T06:47:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:50 smithi077 ceph-mon[37153]: pgmap v422: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-27T06:47:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:50 smithi047 ceph-mon[36835]: pgmap v422: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-27T06:47:51.731 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-07-27T06:47:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.913+0000 7f176b62c640 1 -- 172.21.15.29:0/3777831690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1764104330 msgr2=0x7f1764104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:52.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.913+0000 7f176b62c640 1 --2- 172.21.15.29:0/3777831690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1764104330 0x7f1764104740 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1760009900 tx=0x7f176002a410 comp rx=0 tx=0).stop 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.913+0000 7f176b62c640 1 -- 172.21.15.29:0/3777831690 shutdown_connections 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.913+0000 7f176b62c640 1 --2- 172.21.15.29:0/3777831690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f176410d8e0 0x7f176410fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.913+0000 7f176b62c640 1 --2- 172.21.15.29:0/3777831690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1764104c80 0x7f176410d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.913+0000 7f176b62c640 1 --2- 172.21.15.29:0/3777831690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1764104330 0x7f1764104740 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.913+0000 7f176b62c640 1 -- 172.21.15.29:0/3777831690 >> 172.21.15.29:0/3777831690 conn(0x7f17640791c0 msgr2=0x7f17640795d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.913+0000 7f176b62c640 1 -- 172.21.15.29:0/3777831690 shutdown_connections 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.913+0000 7f176b62c640 1 -- 172.21.15.29:0/3777831690 wait complete. 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f176b62c640 1 Processor -- start 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f176b62c640 1 -- start start 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f176b62c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1764104330 0x7f1764102240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f176b62c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1764104c80 0x7f1764102780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f176b62c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f176410d8e0 0x7f1764102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f176b62c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17641134c0 con 0x7f176410d8e0 2024-07-27T06:47:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f176b62c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f1764113340 con 0x7f1764104330 2024-07-27T06:47:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f176b62c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1764113640 con 0x7f1764104c80 2024-07-27T06:47:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f1769bc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f176410d8e0 0x7f1764102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f1769bc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f176410d8e0 0x7f1764102cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46210/0 (socket says 172.21.15.29:46210) 2024-07-27T06:47:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f17693c8640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1764104330 0x7f1764102240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f1769bc9640 1 -- 172.21.15.29:0/2119517085 learned_addr learned my addr 172.21.15.29:0/2119517085 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.914+0000 7f1768bc7640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1764104c80 0x7f1764102780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.915+0000 7f1768bc7640 1 -- 172.21.15.29:0/2119517085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1764104330 msgr2=0x7f1764102240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.915+0000 7f1768bc7640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1764104330 0x7f1764102240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.915+0000 7f1768bc7640 1 -- 172.21.15.29:0/2119517085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f176410d8e0 msgr2=0x7f1764102cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:52.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.915+0000 7f1768bc7640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f176410d8e0 0x7f1764102cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.915+0000 7f1768bc7640 1 -- 172.21.15.29:0/2119517085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17641a8120 con 0x7f1764104c80 2024-07-27T06:47:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.915+0000 7f1768bc7640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1764104c80 0x7f1764102780 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f175400e9c0 tx=0x7f175400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.915+0000 7f175a7fc640 1 -- 172.21.15.29:0/2119517085 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175400ccc0 con 0x7f1764104c80 2024-07-27T06:47:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.915+0000 7f176b62c640 1 -- 172.21.15.29:0/2119517085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17641a8410 con 0x7f1764104c80 2024-07-27T06:47:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.915+0000 7f175a7fc640 1 -- 172.21.15.29:0/2119517085 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17540108e0 con 0x7f1764104c80 2024-07-27T06:47:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.915+0000 7f175a7fc640 1 -- 172.21.15.29:0/2119517085 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17540187a0 con 0x7f1764104c80 2024-07-27T06:47:52.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.916+0000 7f176b62c640 1 -- 172.21.15.29:0/2119517085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17641a8950 con 0x7f1764104c80 2024-07-27T06:47:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.918+0000 7f175a7fc640 1 -- 172.21.15.29:0/2119517085 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f1754010430 con 0x7f1764104c80 2024-07-27T06:47:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.919+0000 7f1771bb0740 1 Processor -- start 2024-07-27T06:47:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.919+0000 7f1771bb0740 1 -- start start 2024-07-27T06:47:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.919+0000 7f175a7fc640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17500b3cb0 0x7f17500b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f1769bc9640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17500b3cb0 0x7f17500b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f175a7fc640 1 -- 172.21.15.29:0/2119517085 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f17540e42f0 con 0x7f1764104c80 2024-07-27T06:47:52.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f1771bb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce0806070 0x558ce0bab570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f1771bb0740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 0x558ce0babab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f1771bb0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07e5390 0x558ce0babff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f1769bc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07e5390 0x558ce0babff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f1769bc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07e5390 0x558ce0babff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46222/0 (socket says 172.21.15.29:46222) 2024-07-27T06:47:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f1769bc9640 1 -- 172.21.15.29:0/3769450392 learned_addr learned my addr 172.21.15.29:0/3769450392 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f1771bb0740 1 -- 172.21.15.29:0/3769450392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558ce0b21bb0 con 0x558ce07e5390 2024-07-27T06:47:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f1771bb0740 1 -- 172.21.15.29:0/3769450392 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x558ce0b110b0 con 0x558ce07dd1b0 2024-07-27T06:47:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f17693c8640 1 --2- 172.21.15.29:0/3769450392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce0806070 0x558ce0bab570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f1771bb0740 1 -- 172.21.15.29:0/3769450392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558ce0b150f0 con 0x558ce0806070 2024-07-27T06:47:52.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f1768bc7640 1 --2- 172.21.15.29:0/3769450392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 0x558ce0babab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f17693c8640 1 -- 172.21.15.29:0/3769450392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 msgr2=0x558ce0babab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:52.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f17693c8640 1 --2- 172.21.15.29:0/3769450392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 0x558ce0babab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f17693c8640 1 -- 172.21.15.29:0/3769450392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07e5390 msgr2=0x558ce0babff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:52.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.920+0000 7f17693c8640 1 --2- 172.21.15.29:0/3769450392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07e5390 0x558ce0babff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.921+0000 7f17693c8640 1 -- 172.21.15.29:0/3769450392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ce0ab7b90 con 0x558ce0806070 2024-07-27T06:47:52.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.921+0000 7f1769bc9640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17500b3cb0 0x7f17500b61c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f175c013850 tx=0x7f175c013c50 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:52.924 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:52 smithi029 ceph-mon[32610]: pgmap v423: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-27T06:47:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.921+0000 7f17693c8640 1 --2- 172.21.15.29:0/3769450392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce0806070 0x558ce0bab570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1764103c90 tx=0x7f1760032c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=204117e3a20af6a5 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.921+0000 7f174f7fe640 1 -- 172.21.15.29:0/3769450392 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176003f070 con 0x558ce0806070 2024-07-27T06:47:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.921+0000 7f174f7fe640 1 -- 172.21.15.29:0/3769450392 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1760004030 con 0x558ce0806070 2024-07-27T06:47:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.921+0000 7f174f7fe640 1 -- 172.21.15.29:0/3769450392 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1760035070 con 0x558ce0806070 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.922+0000 7f1771bb0740 1 -- 172.21.15.29:0/3769450392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce0806070 msgr2=0x558ce0bab570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.922+0000 7f1771bb0740 1 --2- 172.21.15.29:0/3769450392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce0806070 0x558ce0bab570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1764103c90 tx=0x7f1760032c30 comp rx=0 tx=0).stop 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.923+0000 7f1771bb0740 1 -- 172.21.15.29:0/3769450392 shutdown_connections 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.923+0000 7f1771bb0740 1 --2- 172.21.15.29:0/3769450392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07e5390 0x558ce0babff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.923+0000 7f1771bb0740 1 --2- 172.21.15.29:0/3769450392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 0x558ce0babab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.923+0000 7f1771bb0740 1 --2- 172.21.15.29:0/3769450392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce0806070 0x558ce0bab570 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.923+0000 7f1771bb0740 1 -- 172.21.15.29:0/3769450392 >> 172.21.15.29:0/3769450392 conn(0x558ce07fa460 msgr2=0x558ce0ba9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.923+0000 7f1771bb0740 1 -- 172.21.15.29:0/3769450392 shutdown_connections 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.923+0000 7f1771bb0740 1 -- 172.21.15.29:0/3769450392 wait complete. 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.923+0000 7f1771bb0740 1 -- start start 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.923+0000 7f1771bb0740 1 Processor -- start 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.924+0000 7f1771bb0740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 0x558ce0bb71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.924+0000 7f1771bb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0bb7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.924+0000 7f1771bb0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 0x558ce0bc2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.924+0000 7f1769bc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 0x558ce0bc2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.924+0000 7f1769bc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 0x558ce0bc2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46224/0 (socket says 172.21.15.29:46224) 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.924+0000 7f1769bc9640 1 -- 172.21.15.29:0/2012396300 learned_addr learned my addr 172.21.15.29:0/2012396300 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.924+0000 7f1771bb0740 1 -- 172.21.15.29:0/2012396300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558ce0974700 con 0x558ce0806070 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.924+0000 7f1771bb0740 1 -- 172.21.15.29:0/2012396300 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x558ce0ab39e0 con 0x558ce07dd1b0 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.924+0000 7f1771bb0740 1 -- 172.21.15.29:0/2012396300 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558ce0bb9cf0 con 0x558ce07e5390 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.924+0000 7f1768bc7640 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0bb7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.925+0000 7f17693c8640 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 0x558ce0bb71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.925+0000 7f1768bc7640 1 -- 172.21.15.29:0/2012396300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 msgr2=0x558ce0bb71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.925+0000 7f1768bc7640 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 0x558ce0bb71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.925+0000 7f1768bc7640 1 -- 172.21.15.29:0/2012396300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 msgr2=0x558ce0bc2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.925+0000 7f1768bc7640 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 0x558ce0bc2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.925+0000 7f1768bc7640 1 -- 172.21.15.29:0/2012396300 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ce0871650 con 0x558ce07e5390 2024-07-27T06:47:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.925+0000 7f1771bb0740 10 client.0 _ll_register_callbacks cb 0x558ce0ace4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.925+0000 7f1769bc9640 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 0x558ce0bc2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:47:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.925+0000 7f1771bb0740 1 -- 172.21.15.29:0/2012396300 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1760004030 con 0x558ce07e5390 2024-07-27T06:47:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.925+0000 7f1768bc7640 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0bb7740 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17540e1b60 tx=0x7f17540ab790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a629c14ae5fdb6aa server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.925+0000 7f174cff9640 10 client.25471 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:47:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.926+0000 7f174cff9640 1 -- 172.21.15.29:0/2012396300 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17540c5060 con 0x558ce07e5390 2024-07-27T06:47:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.926+0000 7f174cff9640 1 -- 172.21.15.29:0/2012396300 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f1754010a90 con 0x558ce07e5390 2024-07-27T06:47:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.926+0000 7f174cff9640 1 -- 172.21.15.29:0/2012396300 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f17540b4080 con 0x558ce07e5390 2024-07-27T06:47:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.926+0000 7f1771bb0740 10 client.25471 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:52.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.926+0000 7f1771bb0740 1 -- 172.21.15.29:0/2012396300 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=29}) -- 0x558ce0bc5c40 con 0x558ce07e5390 2024-07-27T06:47:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.927+0000 7f174cff9640 1 -- 172.21.15.29:0/2012396300 <== mon.1 v2:172.21.15.77:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f17540aa230 con 0x558ce07e5390 2024-07-27T06:47:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.927+0000 7f1771bb0740 10 client.25471 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.927+0000 7f1771bb0740 10 client.25471 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-07-27T06:47:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.927+0000 7f1771bb0740 20 client.25471 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.927+0000 7f1771bb0740 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x558ce07c49f0 0x558ce0bc9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.927+0000 7f1771bb0740 4 client.25471 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.927+0000 7f1771bb0740 1 -- 172.21.15.29:0/2012396300 --> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ce0b23b00 con 0x558ce07c49f0 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.927+0000 7f17693c8640 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x558ce07c49f0 0x558ce0bc9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.927+0000 7f17693c8640 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x558ce07c49f0 0x558ce0bc9080 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=82b8b039cb92aec0 server_cookie=c4ea67415e9b7643 in_seq=0 out_seq=0 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.927+0000 7f174cff9640 10 client.25471 ms_handle_connect on v2:172.21.15.77:6832/1085922080 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.928+0000 7f174cff9640 1 -- 172.21.15.29:0/2012396300 <== mds.4 v2:172.21.15.77:6832/1085922080 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558ce0b23b00 con 0x558ce07c49f0 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.928+0000 7f174cff9640 10 client.25471 handle_command_reply: tid=0 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.928+0000 7f1771bb0740 1 client.25471 shutdown 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.928+0000 7f1771bb0740 1 -- 172.21.15.29:0/2012396300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 msgr2=0x558ce0bb7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.928+0000 7f1771bb0740 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0bb7740 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17540e1b60 tx=0x7f17540ab790 comp rx=0 tx=0).stop 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.928+0000 7f1771bb0740 1 -- 172.21.15.29:0/2012396300 shutdown_connections 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.928+0000 7f1771bb0740 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x558ce07c49f0 0x558ce0bc9080 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.928+0000 7f1771bb0740 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 0x558ce0bc2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.928+0000 7f1771bb0740 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0bb7740 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.928+0000 7f1771bb0740 1 --2- 172.21.15.29:0/2012396300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 0x558ce0bb71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.928+0000 7f1771bb0740 1 -- 172.21.15.29:0/2012396300 >> 172.21.15.29:0/2012396300 conn(0x558ce07fa460 msgr2=0x558ce0ba7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.929+0000 7f174cff9640 0 client.25471 ms_handle_reset on v2:172.21.15.77:6832/1085922080 2024-07-27T06:47:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.929+0000 7f1771bb0740 1 -- 172.21.15.29:0/2012396300 shutdown_connections 2024-07-27T06:47:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.929+0000 7f1771bb0740 1 -- 172.21.15.29:0/2012396300 wait complete. 2024-07-27T06:47:52.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.929+0000 7f1771bb0740 20 client.25471 trim_cache size 0 max 16384 2024-07-27T06:47:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.942+0000 7f1771bb0740 1 Processor -- start 2024-07-27T06:47:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1771bb0740 1 -- start start 2024-07-27T06:47:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1771bb0740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 0x558ce0bc2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1771bb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0ba7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1771bb0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 0x558ce0ba8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1771bb0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17641b11f0 con 0x558ce0806070 2024-07-27T06:47:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1771bb0740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f17641b2d30 con 0x558ce07dd1b0 2024-07-27T06:47:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1771bb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f17641b18c0 con 0x558ce07e5390 2024-07-27T06:47:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1769bc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 0x558ce0ba8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1769bc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 0x558ce0ba8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46232/0 (socket says 172.21.15.29:46232) 2024-07-27T06:47:52.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1769bc9640 1 -- 172.21.15.29:0/288583376 learned_addr learned my addr 172.21.15.29:0/288583376 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:52.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1768bc7640 1 --2- 172.21.15.29:0/288583376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0ba7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f17693c8640 1 --2- 172.21.15.29:0/288583376 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 0x558ce0bc2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1769bc9640 1 -- 172.21.15.29:0/288583376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 msgr2=0x558ce0ba7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:52.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1769bc9640 1 --2- 172.21.15.29:0/288583376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0ba7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1769bc9640 1 -- 172.21.15.29:0/288583376 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 msgr2=0x558ce0bc2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:52.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1769bc9640 1 --2- 172.21.15.29:0/288583376 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 0x558ce0bc2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.943+0000 7f1769bc9640 1 -- 172.21.15.29:0/288583376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ce07c4c20 con 0x558ce0806070 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.944+0000 7f1769bc9640 1 --2- 172.21.15.29:0/288583376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 0x558ce0ba8180 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f175c011fb0 tx=0x7f175c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d3f98c89fbbc53 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.944+0000 7f174effd640 1 -- 172.21.15.29:0/288583376 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c02d070 con 0x558ce0806070 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.944+0000 7f174effd640 1 -- 172.21.15.29:0/288583376 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f175c01e940 con 0x558ce0806070 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.944+0000 7f174effd640 1 -- 172.21.15.29:0/288583376 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c0288b0 con 0x558ce0806070 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.945+0000 7f1771bb0740 1 -- 172.21.15.29:0/288583376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 msgr2=0x558ce0ba8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.945+0000 7f1771bb0740 1 --2- 172.21.15.29:0/288583376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 0x558ce0ba8180 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f175c011fb0 tx=0x7f175c01d000 comp rx=0 tx=0).stop 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.945+0000 7f1771bb0740 1 -- 172.21.15.29:0/288583376 shutdown_connections 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.945+0000 7f1771bb0740 1 --2- 172.21.15.29:0/288583376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce0806070 0x558ce0ba8180 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.945+0000 7f1771bb0740 1 --2- 172.21.15.29:0/288583376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0ba7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.945+0000 7f1771bb0740 1 --2- 172.21.15.29:0/288583376 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce07dd1b0 0x558ce0bc2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.945+0000 7f1771bb0740 1 -- 172.21.15.29:0/288583376 >> 172.21.15.29:0/288583376 conn(0x558ce07fa460 msgr2=0x558ce0bb5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.945+0000 7f1771bb0740 1 -- 172.21.15.29:0/288583376 shutdown_connections 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.945+0000 7f1771bb0740 1 -- 172.21.15.29:0/288583376 wait complete. 2024-07-27T06:47:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.946+0000 7f1771bb0740 1 -- start start 2024-07-27T06:47:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.946+0000 7f1771bb0740 1 Processor -- start 2024-07-27T06:47:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f1771bb0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07dd1b0 0x558ce0bba710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f1771bb0740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0bbacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f1771bb0740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce0806070 0x558ce0c37fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f1771bb0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17641b2660 con 0x558ce07dd1b0 2024-07-27T06:47:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f1771bb0740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f17641b1f90 con 0x558ce0806070 2024-07-27T06:47:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f1771bb0740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f17641af1d0 con 0x558ce07e5390 2024-07-27T06:47:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f17693c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07dd1b0 0x558ce0bba710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f1771bb0740 10 client.0 _ll_register_callbacks cb 0x7f17540a6e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f17693c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07dd1b0 0x558ce0bba710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46242/0 (socket says 172.21.15.29:46242) 2024-07-27T06:47:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f17693c8640 1 -- 172.21.15.29:0/2103978273 learned_addr learned my addr 172.21.15.29:0/2103978273 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f1768bc7640 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0bbacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f1769bc9640 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce0806070 0x558ce0c37fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f17693c8640 1 -- 172.21.15.29:0/2103978273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 msgr2=0x558ce0bbacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f17693c8640 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0bbacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f17693c8640 1 -- 172.21.15.29:0/2103978273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce0806070 msgr2=0x558ce0c37fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f17693c8640 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce0806070 0x558ce0c37fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f17693c8640 1 -- 172.21.15.29:0/2103978273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ce0c38650 con 0x558ce07dd1b0 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f17693c8640 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07dd1b0 0x558ce0bba710 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1760000c00 tx=0x7f1760004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906b0848042f9c7e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.947+0000 7f174cff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.948+0000 7f174cff9640 1 -- 172.21.15.29:0/2103978273 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176003f040 con 0x558ce07dd1b0 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.948+0000 7f174cff9640 1 -- 172.21.15.29:0/2103978273 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f17600314a0 con 0x558ce07dd1b0 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.948+0000 7f1771bb0740 1 -- 172.21.15.29:0/2103978273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f175c01e940 con 0x558ce07dd1b0 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.948+0000 7f174cff9640 1 -- 172.21.15.29:0/2103978273 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1760035040 con 0x558ce07dd1b0 2024-07-27T06:47:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.948+0000 7f1771bb0740 10 client.15762 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.948+0000 7f1771bb0740 1 -- 172.21.15.29:0/2103978273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558ce0c398b0 con 0x558ce07dd1b0 2024-07-27T06:47:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.948+0000 7f174cff9640 1 -- 172.21.15.29:0/2103978273 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f175c01e940 con 0x558ce07dd1b0 2024-07-27T06:47:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.948+0000 7f174cff9640 1 -- 172.21.15.29:0/2103978273 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1760055070 con 0x558ce07dd1b0 2024-07-27T06:47:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.949+0000 7f1771bb0740 10 client.15762 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.949+0000 7f1771bb0740 10 client.15762 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-07-27T06:47:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.949+0000 7f1771bb0740 20 client.15762 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.949+0000 7f1771bb0740 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x558ce0799da0 0x558ce0c3bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.949+0000 7f1771bb0740 4 client.15762 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:47:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.949+0000 7f1771bb0740 1 -- 172.21.15.29:0/2103978273 --> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558ce0b1f5a0 con 0x558ce0799da0 2024-07-27T06:47:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.949+0000 7f1768bc7640 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x558ce0799da0 0x558ce0c3bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.949+0000 7f1768bc7640 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x558ce0799da0 0x558ce0c3bba0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d612b63174a76ae2 server_cookie=fe8f0d34c92299c in_seq=0 out_seq=0 2024-07-27T06:47:52.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.949+0000 7f174cff9640 10 client.15762 ms_handle_connect on v2:172.21.15.77:6832/1085922080 2024-07-27T06:47:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f174cff9640 1 -- 172.21.15.29:0/2103978273 <== mds.4 v2:172.21.15.77:6832/1085922080 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558ce0b1f5a0 con 0x558ce0799da0 2024-07-27T06:47:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f174cff9640 10 client.15762 handle_command_reply: tid=0 2024-07-27T06:47:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f1771bb0740 1 client.15762 shutdown 2024-07-27T06:47:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f1771bb0740 1 -- 172.21.15.29:0/2103978273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07dd1b0 msgr2=0x558ce0bba710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f1771bb0740 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07dd1b0 0x558ce0bba710 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1760000c00 tx=0x7f1760004510 comp rx=0 tx=0).stop 2024-07-27T06:47:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f1771bb0740 1 -- 172.21.15.29:0/2103978273 shutdown_connections 2024-07-27T06:47:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f1771bb0740 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x558ce0799da0 0x558ce0c3bba0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f1771bb0740 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558ce0806070 0x558ce0c37fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f1771bb0740 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ce07e5390 0x558ce0bbacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f1771bb0740 1 --2- 172.21.15.29:0/2103978273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558ce07dd1b0 0x558ce0bba710 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f174cff9640 0 client.15762 ms_handle_reset on v2:172.21.15.77:6832/1085922080 2024-07-27T06:47:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f1771bb0740 1 -- 172.21.15.29:0/2103978273 >> 172.21.15.29:0/2103978273 conn(0x558ce07fa460 msgr2=0x558ce0ba7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f1771bb0740 1 -- 172.21.15.29:0/2103978273 shutdown_connections 2024-07-27T06:47:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f1771bb0740 1 -- 172.21.15.29:0/2103978273 wait complete. 2024-07-27T06:47:52.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.950+0000 7f1771bb0740 20 client.15762 trim_cache size 0 max 16384 2024-07-27T06:47:52.955 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:47:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.951+0000 7f176b62c640 1 -- 172.21.15.29:0/2119517085 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17500b3cb0 msgr2=0x7f17500b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:52.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.951+0000 7f176b62c640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17500b3cb0 0x7f17500b61c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f175c013850 tx=0x7f175c013c50 comp rx=0 tx=0).stop 2024-07-27T06:47:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.951+0000 7f176b62c640 1 -- 172.21.15.29:0/2119517085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1764104c80 msgr2=0x7f1764102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.951+0000 7f176b62c640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1764104c80 0x7f1764102780 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f175400e9c0 tx=0x7f175400ee80 comp rx=0 tx=0).stop 2024-07-27T06:47:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.951+0000 7f176b62c640 1 -- 172.21.15.29:0/2119517085 shutdown_connections 2024-07-27T06:47:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.951+0000 7f176b62c640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f176410d8e0 0x7f1764102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.951+0000 7f176b62c640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17500b3cb0 0x7f17500b61c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.951+0000 7f176b62c640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1764104c80 0x7f1764102780 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.951+0000 7f176b62c640 1 --2- 172.21.15.29:0/2119517085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1764104330 0x7f1764102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.951+0000 7f176b62c640 1 -- 172.21.15.29:0/2119517085 >> 172.21.15.29:0/2119517085 conn(0x7f17640791c0 msgr2=0x7f17640ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.951+0000 7f176b62c640 1 -- 172.21.15.29:0/2119517085 shutdown_connections 2024-07-27T06:47:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:52.951+0000 7f176b62c640 1 -- 172.21.15.29:0/2119517085 wait complete. 2024-07-27T06:47:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:52 smithi077 ceph-mon[37153]: pgmap v423: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-27T06:47:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:52 smithi047 ceph-mon[36835]: pgmap v423: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-27T06:47:53.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-27T06:47:54.073 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:54 smithi077 ceph-mon[37153]: pgmap v424: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-27T06:47:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:54 smithi047 ceph-mon[36835]: pgmap v424: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-27T06:47:55.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:54 smithi029 ceph-mon[32610]: pgmap v424: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-27T06:47:56.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.100+0000 7fd05e545640 1 -- 172.21.15.29:0/580910551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd058107dd0 msgr2=0x7fd0581081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.100+0000 7fd05e545640 1 --2- 172.21.15.29:0/580910551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd058107dd0 0x7fd0581081e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd04c009980 tx=0x7fd04c02e820 comp rx=0 tx=0).stop 2024-07-27T06:47:56.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.100+0000 7fd05e545640 1 -- 172.21.15.29:0/580910551 shutdown_connections 2024-07-27T06:47:56.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.100+0000 7fd05e545640 1 --2- 172.21.15.29:0/580910551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd058108720 0x7fd05810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.100+0000 7fd05e545640 1 --2- 172.21.15.29:0/580910551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd058107dd0 0x7fd0581081e0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.100+0000 7fd05e545640 1 --2- 172.21.15.29:0/580910551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd058102490 0x7fd0581028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.100+0000 7fd05e545640 1 -- 172.21.15.29:0/580910551 >> 172.21.15.29:0/580910551 conn(0x7fd058079160 msgr2=0x7fd058079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:56.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.101+0000 7fd05e545640 1 -- 172.21.15.29:0/580910551 shutdown_connections 2024-07-27T06:47:56.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.101+0000 7fd05e545640 1 -- 172.21.15.29:0/580910551 wait complete. 2024-07-27T06:47:56.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.102+0000 7fd05e545640 1 Processor -- start 2024-07-27T06:47:56.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.102+0000 7fd05e545640 1 -- start start 2024-07-27T06:47:56.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.102+0000 7fd05e545640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd058102490 0x7fd058197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.102+0000 7fd05e545640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd058107dd0 0x7fd058198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.102+0000 7fd05e545640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd058108720 0x7fd05819d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.102+0000 7fd05e545640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd058111290 con 0x7fd058107dd0 2024-07-27T06:47:56.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.102+0000 7fd05e545640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd058111110 con 0x7fd058102490 2024-07-27T06:47:56.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.102+0000 7fd05e545640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd058111410 con 0x7fd058108720 2024-07-27T06:47:56.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.102+0000 7fd0577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd058107dd0 0x7fd058198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.102+0000 7fd0577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd058107dd0 0x7fd058198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46258/0 (socket says 172.21.15.29:46258) 2024-07-27T06:47:56.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.102+0000 7fd0577fe640 1 -- 172.21.15.29:0/1692288671 learned_addr learned my addr 172.21.15.29:0/1692288671 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:56.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.102+0000 7fd057fff640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd058102490 0x7fd058197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.103+0000 7fd05cae2640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd058108720 0x7fd05819d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.103+0000 7fd057fff640 1 -- 172.21.15.29:0/1692288671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd058108720 msgr2=0x7fd05819d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.103+0000 7fd057fff640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd058108720 0x7fd05819d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.103+0000 7fd057fff640 1 -- 172.21.15.29:0/1692288671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd058107dd0 msgr2=0x7fd058198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.103+0000 7fd057fff640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd058107dd0 0x7fd058198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.103+0000 7fd057fff640 1 -- 172.21.15.29:0/1692288671 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05819dd10 con 0x7fd058102490 2024-07-27T06:47:56.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.103+0000 7fd0577fe640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd058107dd0 0x7fd058198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:47:56.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.103+0000 7fd057fff640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd058102490 0x7fd058197da0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd04800e980 tx=0x7fd04800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:56.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.104+0000 7fd0557fa640 1 -- 172.21.15.29:0/1692288671 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048004230 con 0x7fd058102490 2024-07-27T06:47:56.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.104+0000 7fd05e545640 1 -- 172.21.15.29:0/1692288671 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd05819e000 con 0x7fd058102490 2024-07-27T06:47:56.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.104+0000 7fd05e545640 1 -- 172.21.15.29:0/1692288671 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0581a4760 con 0x7fd058102490 2024-07-27T06:47:56.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.104+0000 7fd0557fa640 1 -- 172.21.15.29:0/1692288671 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd04800cd60 con 0x7fd058102490 2024-07-27T06:47:56.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.104+0000 7fd0557fa640 1 -- 172.21.15.29:0/1692288671 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048003c60 con 0x7fd058102490 2024-07-27T06:47:56.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.108+0000 7fd064ac9740 1 Processor -- start 2024-07-27T06:47:56.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.108+0000 7fd064ac9740 1 -- start start 2024-07-27T06:47:56.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.109+0000 7fd064ac9740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c222070 0x56242c5c7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.109+0000 7fd064ac9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c1f91b0 0x56242c5c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.109+0000 7fd064ac9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c201390 0x56242c5c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.109+0000 7fd064ac9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56242c53dbb0 con 0x56242c201390 2024-07-27T06:47:56.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.109+0000 7fd064ac9740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x56242c52d0b0 con 0x56242c222070 2024-07-27T06:47:56.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.109+0000 7fd064ac9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56242c5310f0 con 0x56242c1f91b0 2024-07-27T06:47:56.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.109+0000 7fd0557fa640 1 -- 172.21.15.29:0/1692288671 <== mon.2 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7fd0480047c0 con 0x7fd058102490 2024-07-27T06:47:56.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.109+0000 7fd057fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c1f91b0 0x56242c5c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.110+0000 7fd057fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c1f91b0 0x56242c5c7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:46984/0 (socket says 172.21.15.29:46984) 2024-07-27T06:47:56.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.110+0000 7fd057fff640 1 -- 172.21.15.29:0/3188872985 learned_addr learned my addr 172.21.15.29:0/3188872985 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:56.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.110+0000 7fd05cae2640 1 --2- 172.21.15.29:0/3188872985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c222070 0x56242c5c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.110+0000 7fd0577fe640 1 --2- 172.21.15.29:0/3188872985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c201390 0x56242c5c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.110+0000 7fd05cae2640 1 -- 172.21.15.29:0/3188872985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c1f91b0 msgr2=0x56242c5c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.110+0000 7fd05cae2640 1 --2- 172.21.15.29:0/3188872985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c1f91b0 0x56242c5c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.110+0000 7fd05cae2640 1 -- 172.21.15.29:0/3188872985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c201390 msgr2=0x56242c5c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.110+0000 7fd05cae2640 1 --2- 172.21.15.29:0/3188872985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c201390 0x56242c5c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.110+0000 7fd05cae2640 1 -- 172.21.15.29:0/3188872985 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56242c4d3b90 con 0x56242c222070 2024-07-27T06:47:56.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.110+0000 7fd0577fe640 1 --2- 172.21.15.29:0/3188872985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c201390 0x56242c5c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:56.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.110+0000 7fd05cae2640 1 --2- 172.21.15.29:0/3188872985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c222070 0x56242c5c7570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd05000bf50 tx=0x7fd05000d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=889b12c1d76ec819 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:56.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.111+0000 7fd03e7fc640 1 -- 172.21.15.29:0/3188872985 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05001b070 con 0x56242c222070 2024-07-27T06:47:56.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.111+0000 7fd03e7fc640 1 -- 172.21.15.29:0/3188872985 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd050003e20 con 0x56242c222070 2024-07-27T06:47:56.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.111+0000 7fd03e7fc640 1 -- 172.21.15.29:0/3188872985 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050016620 con 0x56242c222070 2024-07-27T06:47:56.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.111+0000 7fd0557fa640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd0400c4dc0 0x7fd0400c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.112+0000 7fd057fff640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd0400c4dc0 0x7fd0400c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.112+0000 7fd0557fa640 1 -- 172.21.15.29:0/1692288671 <== mon.2 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd0480e3f80 con 0x7fd058102490 2024-07-27T06:47:56.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.112+0000 7fd064ac9740 1 -- 172.21.15.29:0/3188872985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c222070 msgr2=0x56242c5c7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.112+0000 7fd064ac9740 1 --2- 172.21.15.29:0/3188872985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c222070 0x56242c5c7570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd05000bf50 tx=0x7fd05000d040 comp rx=0 tx=0).stop 2024-07-27T06:47:56.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.112+0000 7fd064ac9740 1 -- 172.21.15.29:0/3188872985 shutdown_connections 2024-07-27T06:47:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.112+0000 7fd064ac9740 1 --2- 172.21.15.29:0/3188872985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c201390 0x56242c5c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.112+0000 7fd064ac9740 1 --2- 172.21.15.29:0/3188872985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c1f91b0 0x56242c5c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.112+0000 7fd064ac9740 1 --2- 172.21.15.29:0/3188872985 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c222070 0x56242c5c7570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.112+0000 7fd064ac9740 1 -- 172.21.15.29:0/3188872985 >> 172.21.15.29:0/3188872985 conn(0x56242c216460 msgr2=0x56242c5c5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.112+0000 7fd057fff640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd0400c4dc0 0x7fd0400c72d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd0480aaa10 tx=0x7fd0480e7b70 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.112+0000 7fd064ac9740 1 -- 172.21.15.29:0/3188872985 shutdown_connections 2024-07-27T06:47:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.112+0000 7fd064ac9740 1 -- 172.21.15.29:0/3188872985 wait complete. 2024-07-27T06:47:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.113+0000 7fd064ac9740 1 -- start start 2024-07-27T06:47:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.113+0000 7fd064ac9740 1 Processor -- start 2024-07-27T06:47:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.113+0000 7fd064ac9740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c1f91b0 0x56242c5d3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.113+0000 7fd064ac9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c201390 0x56242c5d36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.113+0000 7fd064ac9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5df0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.114+0000 7fd05cae2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5df0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.114+0000 7fd05cae2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5df0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46276/0 (socket says 172.21.15.29:46276) 2024-07-27T06:47:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.114+0000 7fd05cae2640 1 -- 172.21.15.29:0/168461563 learned_addr learned my addr 172.21.15.29:0/168461563 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.114+0000 7fd0577fe640 1 --2- 172.21.15.29:0/168461563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c201390 0x56242c5d36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.114+0000 7fd064ac9740 1 -- 172.21.15.29:0/168461563 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56242c390700 con 0x56242c222070 2024-07-27T06:47:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.114+0000 7fd064ac9740 1 -- 172.21.15.29:0/168461563 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x56242c4cf9e0 con 0x56242c1f91b0 2024-07-27T06:47:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.114+0000 7fd064ac9740 1 -- 172.21.15.29:0/168461563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56242c5d5d50 con 0x56242c201390 2024-07-27T06:47:56.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.114+0000 7fd05cae2640 1 -- 172.21.15.29:0/168461563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c201390 msgr2=0x56242c5d36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.114+0000 7fd05cae2640 1 --2- 172.21.15.29:0/168461563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c201390 0x56242c5d36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.114+0000 7fd05cae2640 1 -- 172.21.15.29:0/168461563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c1f91b0 msgr2=0x56242c5d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.114+0000 7fd05cae2640 1 --2- 172.21.15.29:0/168461563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c1f91b0 0x56242c5d3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.114+0000 7fd05cae2640 1 -- 172.21.15.29:0/168461563 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56242c28d650 con 0x56242c222070 2024-07-27T06:47:56.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.115+0000 7fd05cae2640 1 --2- 172.21.15.29:0/168461563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5df0a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd05000afd0 tx=0x7fd05000bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a9a8be8c67fc2d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:56.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.115+0000 7fd0377fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:56.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.115+0000 7fd064ac9740 10 client.0 _ll_register_callbacks cb 0x56242c4ea4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:56.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.115+0000 7fd064ac9740 1 -- 172.21.15.29:0/168461563 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd050003e20 con 0x56242c222070 2024-07-27T06:47:56.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.115+0000 7fd0377fe640 1 -- 172.21.15.29:0/168461563 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05001b030 con 0x56242c222070 2024-07-27T06:47:56.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.115+0000 7fd0377fe640 1 -- 172.21.15.29:0/168461563 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd050004c40 con 0x56242c222070 2024-07-27T06:47:56.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.115+0000 7fd0377fe640 1 -- 172.21.15.29:0/168461563 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd050004e60 con 0x56242c222070 2024-07-27T06:47:56.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.115+0000 7fd064ac9740 10 client.15774 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:56.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.115+0000 7fd064ac9740 1 -- 172.21.15.29:0/168461563 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=29}) -- 0x56242c5e1b90 con 0x56242c222070 2024-07-27T06:47:56.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.116+0000 7fd0377fe640 1 -- 172.21.15.29:0/168461563 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd05004d030 con 0x56242c222070 2024-07-27T06:47:56.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.116+0000 7fd064ac9740 10 client.15774 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.116+0000 7fd064ac9740 10 client.15774 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.116+0000 7fd064ac9740 20 client.15774 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.116+0000 7fd064ac9740 1 --2- 172.21.15.29:0/168461563 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x56242c1e09f0 0x56242c5e4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.116+0000 7fd064ac9740 4 client.15774 mds_command: new command op to 24617 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.116+0000 7fd064ac9740 1 -- 172.21.15.29:0/168461563 --> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56242c53fb00 con 0x56242c1e09f0 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.116+0000 7fd0577fe640 1 --2- 172.21.15.29:0/168461563 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x56242c1e09f0 0x56242c5e4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.117+0000 7fd0377fe640 10 client.15774 ms_handle_connect on v2:172.21.15.47:6838/670107266 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.117+0000 7fd0577fe640 1 --2- 172.21.15.29:0/168461563 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x56242c1e09f0 0x56242c5e4ef0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5a1fc61ccf917e6d server_cookie=dd558b6fc11811e6 in_seq=0 out_seq=0 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.117+0000 7fd0377fe640 1 -- 172.21.15.29:0/168461563 <== mds.3 v2:172.21.15.47:6838/670107266 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56242c53fb00 con 0x56242c1e09f0 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.117+0000 7fd0377fe640 10 client.15774 handle_command_reply: tid=0 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.117+0000 7fd064ac9740 1 client.15774 shutdown 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.118+0000 7fd064ac9740 1 -- 172.21.15.29:0/168461563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 msgr2=0x56242c5df0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.118+0000 7fd064ac9740 1 --2- 172.21.15.29:0/168461563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5df0a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd05000afd0 tx=0x7fd05000bca0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.118+0000 7fd064ac9740 1 -- 172.21.15.29:0/168461563 shutdown_connections 2024-07-27T06:47:56.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.118+0000 7fd064ac9740 1 --2- 172.21.15.29:0/168461563 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x56242c1e09f0 0x56242c5e4ef0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.118+0000 7fd0377fe640 0 client.15774 ms_handle_reset on v2:172.21.15.47:6838/670107266 2024-07-27T06:47:56.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.118+0000 7fd064ac9740 1 --2- 172.21.15.29:0/168461563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5df0a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.118+0000 7fd064ac9740 1 --2- 172.21.15.29:0/168461563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c201390 0x56242c5d36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.118+0000 7fd064ac9740 1 --2- 172.21.15.29:0/168461563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c1f91b0 0x56242c5d3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.118+0000 7fd064ac9740 1 -- 172.21.15.29:0/168461563 >> 172.21.15.29:0/168461563 conn(0x56242c216460 msgr2=0x56242c5c3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:56.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.118+0000 7fd064ac9740 1 -- 172.21.15.29:0/168461563 shutdown_connections 2024-07-27T06:47:56.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.118+0000 7fd064ac9740 1 -- 172.21.15.29:0/168461563 wait complete. 2024-07-27T06:47:56.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.118+0000 7fd064ac9740 20 client.15774 trim_cache size 0 max 16384 2024-07-27T06:47:56.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.132+0000 7fd064ac9740 1 Processor -- start 2024-07-27T06:47:56.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.132+0000 7fd064ac9740 1 -- start start 2024-07-27T06:47:56.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.132+0000 7fd064ac9740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c1f91b0 0x56242c5c4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.132+0000 7fd064ac9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c201390 0x56242c5dee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.132+0000 7fd064ac9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5ea5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.132+0000 7fd064ac9740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581ad110 con 0x56242c222070 2024-07-27T06:47:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.132+0000 7fd064ac9740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581aec50 con 0x56242c1f91b0 2024-07-27T06:47:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.132+0000 7fd064ac9740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581ad7e0 con 0x56242c201390 2024-07-27T06:47:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.133+0000 7fd05cae2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5ea5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.133+0000 7fd05cae2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5ea5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46286/0 (socket says 172.21.15.29:46286) 2024-07-27T06:47:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.133+0000 7fd05cae2640 1 -- 172.21.15.29:0/2264439811 learned_addr learned my addr 172.21.15.29:0/2264439811 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.133+0000 7fd05cae2640 1 --2- 172.21.15.29:0/2264439811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c1f91b0 0x56242c5c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.133+0000 7fd0577fe640 1 --2- 172.21.15.29:0/2264439811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c201390 0x56242c5dee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.133+0000 7fd0577fe640 1 -- 172.21.15.29:0/2264439811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c1f91b0 msgr2=0x56242c5c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.133+0000 7fd0577fe640 1 --2- 172.21.15.29:0/2264439811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c1f91b0 0x56242c5c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.133+0000 7fd0577fe640 1 -- 172.21.15.29:0/2264439811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 msgr2=0x56242c5ea5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.133+0000 7fd0577fe640 1 --2- 172.21.15.29:0/2264439811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5ea5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.133+0000 7fd0577fe640 1 -- 172.21.15.29:0/2264439811 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56242c5df960 con 0x56242c201390 2024-07-27T06:47:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.134+0000 7fd0577fe640 1 --2- 172.21.15.29:0/2264439811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c201390 0x56242c5dee90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd04c009430 tx=0x7fd04c036860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53fef5143f1db22f server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.134+0000 7fd03dffb640 1 -- 172.21.15.29:0/2264439811 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c004040 con 0x56242c201390 2024-07-27T06:47:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.134+0000 7fd03dffb640 1 -- 172.21.15.29:0/2264439811 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd04c0104f0 con 0x56242c201390 2024-07-27T06:47:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.134+0000 7fd03dffb640 1 -- 172.21.15.29:0/2264439811 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c005320 con 0x56242c201390 2024-07-27T06:47:56.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.135+0000 7fd064ac9740 1 -- 172.21.15.29:0/2264439811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c201390 msgr2=0x56242c5dee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.135+0000 7fd064ac9740 1 --2- 172.21.15.29:0/2264439811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c201390 0x56242c5dee90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd04c009430 tx=0x7fd04c036860 comp rx=0 tx=0).stop 2024-07-27T06:47:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.135+0000 7fd064ac9740 1 -- 172.21.15.29:0/2264439811 shutdown_connections 2024-07-27T06:47:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.135+0000 7fd064ac9740 1 --2- 172.21.15.29:0/2264439811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5ea5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.135+0000 7fd064ac9740 1 --2- 172.21.15.29:0/2264439811 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c201390 0x56242c5dee90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.135+0000 7fd064ac9740 1 --2- 172.21.15.29:0/2264439811 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c1f91b0 0x56242c5c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.135+0000 7fd064ac9740 1 -- 172.21.15.29:0/2264439811 >> 172.21.15.29:0/2264439811 conn(0x56242c216460 msgr2=0x56242c5c3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.135+0000 7fd064ac9740 1 -- 172.21.15.29:0/2264439811 shutdown_connections 2024-07-27T06:47:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.135+0000 7fd064ac9740 1 -- 172.21.15.29:0/2264439811 wait complete. 2024-07-27T06:47:56.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.136+0000 7fd064ac9740 1 -- start start 2024-07-27T06:47:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.136+0000 7fd064ac9740 1 Processor -- start 2024-07-27T06:47:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.137+0000 7fd064ac9740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c1f91b0 0x56242c5db7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.137+0000 7fd064ac9740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c201390 0x56242c5d6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.137+0000 7fd064ac9740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5d7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.137+0000 7fd0577fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c201390 0x56242c5d6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd0577fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c201390 0x56242c5d6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.29:57736/0 (socket says 172.21.15.29:57736) 2024-07-27T06:47:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd05cae2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5d7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd05cae2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5d7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46290/0 (socket says 172.21.15.29:46290) 2024-07-27T06:47:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd0577fe640 1 -- 172.21.15.29:0/962680884 learned_addr learned my addr 172.21.15.29:0/962680884 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd064ac9740 1 -- 172.21.15.29:0/962680884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581ae580 con 0x56242c222070 2024-07-27T06:47:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd064ac9740 1 -- 172.21.15.29:0/962680884 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581adeb0 con 0x56242c201390 2024-07-27T06:47:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd064ac9740 1 -- 172.21.15.29:0/962680884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581ab0f0 con 0x56242c1f91b0 2024-07-27T06:47:56.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd0577fe640 1 -- 172.21.15.29:0/962680884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c1f91b0 msgr2=0x56242c5db7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd0577fe640 1 --2- 172.21.15.29:0/962680884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c1f91b0 0x56242c5db7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd0577fe640 1 -- 172.21.15.29:0/962680884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 msgr2=0x56242c5d7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd0577fe640 1 --2- 172.21.15.29:0/962680884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5d7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd0577fe640 1 -- 172.21.15.29:0/962680884 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56242c5d79e0 con 0x56242c201390 2024-07-27T06:47:56.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.138+0000 7fd05cae2640 1 --2- 172.21.15.29:0/962680884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5d7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:47:56.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.139+0000 7fd064ac9740 10 client.0 _ll_register_callbacks cb 0x7fd050019070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:56.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.139+0000 7fd064ac9740 1 -- 172.21.15.29:0/962680884 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd04c0104f0 con 0x56242c201390 2024-07-27T06:47:56.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.139+0000 7fd0577fe640 1 --2- 172.21.15.29:0/962680884 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c201390 0x56242c5d6c50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd04c000c00 tx=0x7fd04c005020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92bb102b235b0f7b server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:56.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.139+0000 7fd0377fe640 10 client.25355 ms_handle_connect on v2:172.21.15.47:3300/0 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.139+0000 7fd0377fe640 1 -- 172.21.15.29:0/962680884 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c010300 con 0x56242c201390 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.139+0000 7fd0377fe640 1 -- 172.21.15.29:0/962680884 <== mon.2 v2:172.21.15.47:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd04c0052a0 con 0x56242c201390 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.139+0000 7fd0377fe640 1 -- 172.21.15.29:0/962680884 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd04c0054e0 con 0x56242c201390 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.139+0000 7fd064ac9740 10 client.25355 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.139+0000 7fd064ac9740 1 -- 172.21.15.29:0/962680884 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({fsmap=29}) -- 0x56242c64d450 con 0x56242c201390 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.140+0000 7fd0377fe640 1 -- 172.21.15.29:0/962680884 <== mon.2 v2:172.21.15.47:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd04c004040 con 0x56242c201390 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.140+0000 7fd064ac9740 10 client.25355 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.140+0000 7fd064ac9740 10 client.25355 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.140+0000 7fd064ac9740 20 client.25355 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.140+0000 7fd064ac9740 1 --2- 172.21.15.29:0/962680884 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x56242c1b5da0 0x56242c64fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.140+0000 7fd064ac9740 4 client.25355 mds_command: new command op to 24617 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.140+0000 7fd064ac9740 1 -- 172.21.15.29:0/962680884 --> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56242c53b5a0 con 0x56242c1b5da0 2024-07-27T06:47:56.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.140+0000 7fd05cae2640 1 --2- 172.21.15.29:0/962680884 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x56242c1b5da0 0x56242c64fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd0377fe640 10 client.25355 ms_handle_connect on v2:172.21.15.47:6838/670107266 2024-07-27T06:47:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd05cae2640 1 --2- 172.21.15.29:0/962680884 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x56242c1b5da0 0x56242c64fa60 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=54535d2788c58103 server_cookie=f1695f4eb4408f3 in_seq=0 out_seq=0 2024-07-27T06:47:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd0377fe640 1 -- 172.21.15.29:0/962680884 <== mds.3 v2:172.21.15.47:6838/670107266 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56242c53b5a0 con 0x56242c1b5da0 2024-07-27T06:47:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd0377fe640 10 client.25355 handle_command_reply: tid=0 2024-07-27T06:47:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd064ac9740 1 client.25355 shutdown 2024-07-27T06:47:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd064ac9740 1 -- 172.21.15.29:0/962680884 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c201390 msgr2=0x56242c5d6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd064ac9740 1 --2- 172.21.15.29:0/962680884 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c201390 0x56242c5d6c50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd04c000c00 tx=0x7fd04c005020 comp rx=0 tx=0).stop 2024-07-27T06:47:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd064ac9740 1 -- 172.21.15.29:0/962680884 shutdown_connections 2024-07-27T06:47:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd064ac9740 1 --2- 172.21.15.29:0/962680884 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x56242c1b5da0 0x56242c64fa60 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd064ac9740 1 --2- 172.21.15.29:0/962680884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56242c222070 0x56242c5d7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd064ac9740 1 --2- 172.21.15.29:0/962680884 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56242c201390 0x56242c5d6c50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd064ac9740 1 --2- 172.21.15.29:0/962680884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56242c1f91b0 0x56242c5db7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.141+0000 7fd064ac9740 1 -- 172.21.15.29:0/962680884 >> 172.21.15.29:0/962680884 conn(0x56242c216460 msgr2=0x56242c5cc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:56.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.142+0000 7fd0377fe640 0 client.25355 ms_handle_reset on v2:172.21.15.47:6838/670107266 2024-07-27T06:47:56.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.142+0000 7fd064ac9740 1 -- 172.21.15.29:0/962680884 shutdown_connections 2024-07-27T06:47:56.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.142+0000 7fd064ac9740 1 -- 172.21.15.29:0/962680884 wait complete. 2024-07-27T06:47:56.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.142+0000 7fd064ac9740 20 client.25355 trim_cache size 0 max 16384 2024-07-27T06:47:56.145 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:47:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.143+0000 7fd05e545640 1 -- 172.21.15.29:0/1692288671 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd0400c4dc0 msgr2=0x7fd0400c72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.143+0000 7fd05e545640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd0400c4dc0 0x7fd0400c72d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd0480aaa10 tx=0x7fd0480e7b70 comp rx=0 tx=0).stop 2024-07-27T06:47:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.143+0000 7fd05e545640 1 -- 172.21.15.29:0/1692288671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd058102490 msgr2=0x7fd058197da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.143+0000 7fd05e545640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd058102490 0x7fd058197da0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd04800e980 tx=0x7fd04800ee40 comp rx=0 tx=0).stop 2024-07-27T06:47:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.143+0000 7fd05e545640 1 -- 172.21.15.29:0/1692288671 shutdown_connections 2024-07-27T06:47:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.143+0000 7fd05e545640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd0400c4dc0 0x7fd0400c72d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.143+0000 7fd05e545640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd058108720 0x7fd05819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.143+0000 7fd05e545640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd058107dd0 0x7fd058198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.143+0000 7fd05e545640 1 --2- 172.21.15.29:0/1692288671 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd058102490 0x7fd058197da0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.143+0000 7fd05e545640 1 -- 172.21.15.29:0/1692288671 >> 172.21.15.29:0/1692288671 conn(0x7fd058079160 msgr2=0x7fd058103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:56.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.143+0000 7fd05e545640 1 -- 172.21.15.29:0/1692288671 shutdown_connections 2024-07-27T06:47:56.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:56.144+0000 7fd05e545640 1 -- 172.21.15.29:0/1692288671 wait complete. 2024-07-27T06:47:56.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.181+0000 7f3d194a9640 1 -- 172.21.15.29:0/4171047145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d14102a80 msgr2=0x7f3d14106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.181+0000 7f3d194a9640 1 --2- 172.21.15.29:0/4171047145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d14102a80 0x7f3d14106fa0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009900 tx=0x7f3d0402a3f0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.181+0000 7f3d194a9640 1 -- 172.21.15.29:0/4171047145 shutdown_connections 2024-07-27T06:47:56.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.181+0000 7f3d194a9640 1 --2- 172.21.15.29:0/4171047145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d141075f0 0x7f3d14107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.181+0000 7f3d194a9640 1 --2- 172.21.15.29:0/4171047145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d14102a80 0x7f3d14106fa0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.181+0000 7f3d194a9640 1 --2- 172.21.15.29:0/4171047145 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d14102130 0x7f3d14102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.181+0000 7f3d194a9640 1 -- 172.21.15.29:0/4171047145 >> 172.21.15.29:0/4171047145 conn(0x7f3d140fddf0 msgr2=0x7f3d14100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:56.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.181+0000 7f3d194a9640 1 -- 172.21.15.29:0/4171047145 shutdown_connections 2024-07-27T06:47:56.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.181+0000 7f3d194a9640 1 -- 172.21.15.29:0/4171047145 wait complete. 2024-07-27T06:47:56.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.182+0000 7f3d194a9640 1 Processor -- start 2024-07-27T06:47:56.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.182+0000 7f3d194a9640 1 -- start start 2024-07-27T06:47:56.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.182+0000 7f3d194a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d14102130 0x7f3d141a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.182+0000 7f3d194a9640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d14102a80 0x7f3d141a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.184 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d194a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d141075f0 0x7f3d141a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d194a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3d14119a80 con 0x7f3d141075f0 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d194a9640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3d14119900 con 0x7f3d14102a80 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d194a9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3d14119c00 con 0x7f3d14102130 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d12ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d14102130 0x7f3d141a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d141075f0 0x7f3d141a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d12ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d14102130 0x7f3d141a0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:47020/0 (socket says 172.21.15.29:47020) 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d12ffd640 1 -- 172.21.15.29:0/1238539573 learned_addr learned my addr 172.21.15.29:0/1238539573 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d127fc640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d14102a80 0x7f3d141a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d12ffd640 1 -- 172.21.15.29:0/1238539573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d14102a80 msgr2=0x7f3d141a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d12ffd640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d14102a80 0x7f3d141a0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d12ffd640 1 -- 172.21.15.29:0/1238539573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d141075f0 msgr2=0x7f3d141a5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d12ffd640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d141075f0 0x7f3d141a5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.185 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d12ffd640 1 -- 172.21.15.29:0/1238539573 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d141a6450 con 0x7f3d14102130 2024-07-27T06:47:56.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.183+0000 7f3d127fc640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d14102a80 0x7f3d141a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:56.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.184+0000 7f3d12ffd640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d14102130 0x7f3d141a0520 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00e9f0 tx=0x7f3cfc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:56.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.184+0000 7f3d0bfff640 1 -- 172.21.15.29:0/1238539573 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc00ccc0 con 0x7f3d14102130 2024-07-27T06:47:56.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.184+0000 7f3d194a9640 1 -- 172.21.15.29:0/1238539573 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d141a6740 con 0x7f3d14102130 2024-07-27T06:47:56.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.184+0000 7f3d194a9640 1 -- 172.21.15.29:0/1238539573 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d141aced0 con 0x7f3d14102130 2024-07-27T06:47:56.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.184+0000 7f3d0bfff640 1 -- 172.21.15.29:0/1238539573 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cfc0108e0 con 0x7f3d14102130 2024-07-27T06:47:56.186 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.184+0000 7f3d0bfff640 1 -- 172.21.15.29:0/1238539573 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc018880 con 0x7f3d14102130 2024-07-27T06:47:56.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.187+0000 7f3d203c1740 1 Processor -- start 2024-07-27T06:47:56.187 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.187+0000 7f3d203c1740 1 -- start start 2024-07-27T06:47:56.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.187+0000 7f3d203c1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcfa1b0 0x558f2e09c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.187+0000 7f3d203c1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcd12f0 0x558f2e09cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.187+0000 7f3d203c1740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd94d0 0x558f2e09d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.187+0000 7f3d203c1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558f2e0123b0 con 0x558f2dcfa1b0 2024-07-27T06:47:56.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.187+0000 7f3d203c1740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x558f2e0018b0 con 0x558f2dcd94d0 2024-07-27T06:47:56.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.187+0000 7f3d203c1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558f2e0058f0 con 0x558f2dcd12f0 2024-07-27T06:47:56.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.187+0000 7f3d137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcfa1b0 0x558f2e09c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.187+0000 7f3d137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcfa1b0 0x558f2e09c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46322/0 (socket says 172.21.15.29:46322) 2024-07-27T06:47:56.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.187+0000 7f3d137fe640 1 -- 172.21.15.29:0/1483869667 learned_addr learned my addr 172.21.15.29:0/1483869667 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:56.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.188+0000 7f3d127fc640 1 --2- 172.21.15.29:0/1483869667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd94d0 0x558f2e09d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.188+0000 7f3d137fe640 1 -- 172.21.15.29:0/1483869667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcd12f0 msgr2=0x558f2e09cbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T06:47:56.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.188+0000 7f3d137fe640 1 --2- 172.21.15.29:0/1483869667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcd12f0 0x558f2e09cbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.188+0000 7f3d137fe640 1 -- 172.21.15.29:0/1483869667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd94d0 msgr2=0x558f2e09d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.188+0000 7f3d137fe640 1 --2- 172.21.15.29:0/1483869667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd94d0 0x558f2e09d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.188+0000 7f3d137fe640 1 -- 172.21.15.29:0/1483869667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f2dfa89a0 con 0x558f2dcfa1b0 2024-07-27T06:47:56.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.188+0000 7f3d0bfff640 1 -- 172.21.15.29:0/1238539573 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7f3cfc010430 con 0x7f3d14102130 2024-07-27T06:47:56.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.188+0000 7f3d137fe640 1 --2- 172.21.15.29:0/1483869667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcfa1b0 0x558f2e09c6b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c009f40 tx=0x7f3d0c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5ef238d367feea server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:56.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.188+0000 7f3d097fa640 1 -- 172.21.15.29:0/1483869667 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c004d20 con 0x558f2dcfa1b0 2024-07-27T06:47:56.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.188+0000 7f3d097fa640 1 -- 172.21.15.29:0/1483869667 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d0c004050 con 0x558f2dcfa1b0 2024-07-27T06:47:56.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.188+0000 7f3d097fa640 1 -- 172.21.15.29:0/1483869667 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c005280 con 0x558f2dcfa1b0 2024-07-27T06:47:56.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.189+0000 7f3d0bfff640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3d000b3d60 0x7f3d000b6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.189+0000 7f3d0bfff640 1 -- 172.21.15.29:0/1238539573 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f3cfc0e41b0 con 0x7f3d14102130 2024-07-27T06:47:56.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.189+0000 7f3d12ffd640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3d000b3d60 0x7f3d000b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.190+0000 7f3d203c1740 1 -- 172.21.15.29:0/1483869667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcfa1b0 msgr2=0x558f2e09c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.190+0000 7f3d203c1740 1 --2- 172.21.15.29:0/1483869667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcfa1b0 0x558f2e09c6b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c009f40 tx=0x7f3d0c0118d0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.190+0000 7f3d203c1740 1 -- 172.21.15.29:0/1483869667 shutdown_connections 2024-07-27T06:47:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.190+0000 7f3d203c1740 1 --2- 172.21.15.29:0/1483869667 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd94d0 0x558f2e09d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.190+0000 7f3d203c1740 1 --2- 172.21.15.29:0/1483869667 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcd12f0 0x558f2e09cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.190+0000 7f3d203c1740 1 --2- 172.21.15.29:0/1483869667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcfa1b0 0x558f2e09c6b0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.190+0000 7f3d203c1740 1 -- 172.21.15.29:0/1483869667 >> 172.21.15.29:0/1483869667 conn(0x558f2dcee300 msgr2=0x558f2e09a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.190+0000 7f3d203c1740 1 -- 172.21.15.29:0/1483869667 shutdown_connections 2024-07-27T06:47:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.190+0000 7f3d203c1740 1 -- 172.21.15.29:0/1483869667 wait complete. 2024-07-27T06:47:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.190+0000 7f3d12ffd640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3d000b3d60 0x7f3d000b6270 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x558f2e09b330 tx=0x7f3cfc018420 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.190+0000 7f3d203c1740 1 -- start start 2024-07-27T06:47:56.192 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.191+0000 7f3d203c1740 1 Processor -- start 2024-07-27T06:47:56.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.191+0000 7f3d203c1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 0x558f2e0a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.191+0000 7f3d203c1740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd94d0 0x558f2e0a86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.191+0000 7f3d203c1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcfa1b0 0x558f2e0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.191+0000 7f3d203c1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558f2df3bab0 con 0x558f2dcd12f0 2024-07-27T06:47:56.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.191+0000 7f3d203c1740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x558f2df89cf0 con 0x558f2dcd94d0 2024-07-27T06:47:56.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.191+0000 7f3d203c1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558f2e0aae30 con 0x558f2dcfa1b0 2024-07-27T06:47:56.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.191+0000 7f3d137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 0x558f2e0a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.191+0000 7f3d137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 0x558f2e0a8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46334/0 (socket says 172.21.15.29:46334) 2024-07-27T06:47:56.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.191+0000 7f3d137fe640 1 -- 172.21.15.29:0/537938554 learned_addr learned my addr 172.21.15.29:0/537938554 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:56.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3d203c1740 10 client.0 _ll_register_callbacks cb 0x558f2dffb3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:56.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3d127fc640 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd94d0 0x558f2e0a86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3d137fe640 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcfa1b0 0x558f2e0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3d137fe640 1 -- 172.21.15.29:0/537938554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcfa1b0 msgr2=0x558f2e0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3d137fe640 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcfa1b0 0x558f2e0b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3d137fe640 1 -- 172.21.15.29:0/537938554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd94d0 msgr2=0x558f2e0a86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3d137fe640 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd94d0 0x558f2e0a86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3d137fe640 1 -- 172.21.15.29:0/537938554 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f2dd65790 con 0x558f2dcd12f0 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3d137fe640 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 0x558f2e0a8190 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c0048f0 tx=0x7f3d0c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d822e8b259cc1d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3cfa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3cfa7fc640 1 -- 172.21.15.29:0/537938554 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c003c30 con 0x558f2dcd12f0 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3cfa7fc640 1 -- 172.21.15.29:0/537938554 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f3d0c003dd0 con 0x558f2dcd12f0 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.192+0000 7f3d203c1740 1 -- 172.21.15.29:0/537938554 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d0c004050 con 0x558f2dcd12f0 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.193+0000 7f3cfa7fc640 1 -- 172.21.15.29:0/537938554 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3d0c036050 con 0x558f2dcd12f0 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.193+0000 7f3d203c1740 10 client.15792 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.193+0000 7f3d203c1740 1 -- 172.21.15.29:0/537938554 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558f2ddde620 con 0x558f2dcd12f0 2024-07-27T06:47:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.193+0000 7f3cfa7fc640 1 -- 172.21.15.29:0/537938554 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d0c004050 con 0x558f2dcd12f0 2024-07-27T06:47:56.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.193+0000 7f3cfa7fc640 1 -- 172.21.15.29:0/537938554 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3d0c028480 con 0x558f2dcd12f0 2024-07-27T06:47:56.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.193+0000 7f3d203c1740 10 client.15792 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:56.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.193+0000 7f3d203c1740 10 client.15792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:56.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.193+0000 7f3d203c1740 20 client.15792 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:56.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.193+0000 7f3d203c1740 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x558f2dcb8d80 0x558f2e0ba020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.193+0000 7f3d203c1740 4 client.15792 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:56.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.193+0000 7f3d203c1740 1 -- 172.21.15.29:0/537938554 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f2e014300 con 0x558f2dcb8d80 2024-07-27T06:47:56.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.194+0000 7f3d127fc640 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x558f2dcb8d80 0x558f2e0ba020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.194+0000 7f3cfa7fc640 10 client.15792 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:56.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.194+0000 7f3d127fc640 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x558f2dcb8d80 0x558f2e0ba020 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d5a364c85de2d8e server_cookie=4e3c73df8cc633f4 in_seq=0 out_seq=0 2024-07-27T06:47:56.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3cfa7fc640 1 -- 172.21.15.29:0/537938554 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558f2e014300 con 0x558f2dcb8d80 2024-07-27T06:47:56.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3cfa7fc640 10 client.15792 handle_command_reply: tid=0 2024-07-27T06:47:56.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3d203c1740 1 client.15792 shutdown 2024-07-27T06:47:56.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3d203c1740 1 -- 172.21.15.29:0/537938554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 msgr2=0x558f2e0a8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3d203c1740 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 0x558f2e0a8190 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c0048f0 tx=0x7f3d0c005bd0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3d203c1740 1 -- 172.21.15.29:0/537938554 shutdown_connections 2024-07-27T06:47:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3d203c1740 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x558f2dcb8d80 0x558f2e0ba020 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3cfa7fc640 0 client.15792 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3d203c1740 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcfa1b0 0x558f2e0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3d203c1740 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd94d0 0x558f2e0a86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3d203c1740 1 --2- 172.21.15.29:0/537938554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 0x558f2e0a8190 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3d203c1740 1 -- 172.21.15.29:0/537938554 >> 172.21.15.29:0/537938554 conn(0x558f2dcee300 msgr2=0x558f2e09d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3d203c1740 1 -- 172.21.15.29:0/537938554 shutdown_connections 2024-07-27T06:47:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3d203c1740 1 -- 172.21.15.29:0/537938554 wait complete. 2024-07-27T06:47:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.195+0000 7f3d203c1740 20 client.15792 trim_cache size 0 max 16384 2024-07-27T06:47:56.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.211+0000 7f3d203c1740 1 Processor -- start 2024-07-27T06:47:56.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.211+0000 7f3d203c1740 1 -- start start 2024-07-27T06:47:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.211+0000 7f3d203c1740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcb8d80 0x558f2e0a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.211+0000 7f3d203c1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 0x558f2e0a9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.211+0000 7f3d203c1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcd94d0 0x558f2e0b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.211+0000 7f3d203c1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf400e7b0 con 0x558f2dcd12f0 2024-07-27T06:47:56.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.211+0000 7f3d203c1740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf40102f0 con 0x558f2dcb8d80 2024-07-27T06:47:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.211+0000 7f3d203c1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf400ee80 con 0x558f2dcd94d0 2024-07-27T06:47:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.211+0000 7f3d127fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 0x558f2e0a9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d127fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 0x558f2e0a9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46336/0 (socket says 172.21.15.29:46336) 2024-07-27T06:47:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d127fc640 1 -- 172.21.15.29:0/1799217152 learned_addr learned my addr 172.21.15.29:0/1799217152 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d137fe640 1 --2- 172.21.15.29:0/1799217152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcb8d80 0x558f2e0a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d137fe640 1 --2- 172.21.15.29:0/1799217152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcd94d0 0x558f2e0b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d127fc640 1 -- 172.21.15.29:0/1799217152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcd94d0 msgr2=0x558f2e0b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d127fc640 1 --2- 172.21.15.29:0/1799217152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcd94d0 0x558f2e0b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d127fc640 1 -- 172.21.15.29:0/1799217152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcb8d80 msgr2=0x558f2e0a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d127fc640 1 --2- 172.21.15.29:0/1799217152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcb8d80 0x558f2e0a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.214 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d127fc640 1 -- 172.21.15.29:0/1799217152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f2e0a9240 con 0x558f2dcd12f0 2024-07-27T06:47:56.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d127fc640 1 --2- 172.21.15.29:0/1799217152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 0x558f2e0a9f70 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3d04005ab0 tx=0x7f3d04000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed80269f3ad24cb6 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:56.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d08ff9640 1 -- 172.21.15.29:0/1799217152 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d04035040 con 0x558f2dcd12f0 2024-07-27T06:47:56.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d08ff9640 1 -- 172.21.15.29:0/1799217152 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d0404a070 con 0x558f2dcd12f0 2024-07-27T06:47:56.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.212+0000 7f3d08ff9640 1 -- 172.21.15.29:0/1799217152 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d040035f0 con 0x558f2dcd12f0 2024-07-27T06:47:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.214+0000 7f3d203c1740 1 -- 172.21.15.29:0/1799217152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 msgr2=0x558f2e0a9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.214+0000 7f3d203c1740 1 --2- 172.21.15.29:0/1799217152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 0x558f2e0a9f70 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3d04005ab0 tx=0x7f3d04000bc0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.214+0000 7f3d203c1740 1 -- 172.21.15.29:0/1799217152 shutdown_connections 2024-07-27T06:47:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.214+0000 7f3d203c1740 1 --2- 172.21.15.29:0/1799217152 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcd94d0 0x558f2e0b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.214+0000 7f3d203c1740 1 --2- 172.21.15.29:0/1799217152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd12f0 0x558f2e0a9f70 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.214+0000 7f3d203c1740 1 --2- 172.21.15.29:0/1799217152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcb8d80 0x558f2e0a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.214+0000 7f3d203c1740 1 -- 172.21.15.29:0/1799217152 >> 172.21.15.29:0/1799217152 conn(0x558f2dcee300 msgr2=0x558f2e0a2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.214+0000 7f3d203c1740 1 -- 172.21.15.29:0/1799217152 shutdown_connections 2024-07-27T06:47:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.214+0000 7f3d203c1740 1 -- 172.21.15.29:0/1799217152 wait complete. 2024-07-27T06:47:56.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.214+0000 7f3d203c1740 1 -- start start 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.214+0000 7f3d203c1740 1 Processor -- start 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.215+0000 7f3d203c1740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcb8d80 0x558f2e0b0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.215+0000 7f3d203c1740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd12f0 0x558f2e0aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.215+0000 7f3d203c1740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd94d0 0x558f2e0af170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.215+0000 7f3d203c1740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf400fc20 con 0x558f2dcd94d0 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.215+0000 7f3d203c1740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf400f550 con 0x558f2dcd12f0 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.215+0000 7f3d203c1740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf400c790 con 0x558f2dcb8d80 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.215+0000 7f3d137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd94d0 0x558f2e0af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.215+0000 7f3d203c1740 10 client.0 _ll_register_callbacks cb 0x7f3cfc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.215+0000 7f3d137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd94d0 0x558f2e0af170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46342/0 (socket says 172.21.15.29:46342) 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.215+0000 7f3d137fe640 1 -- 172.21.15.29:0/2175231034 learned_addr learned my addr 172.21.15.29:0/2175231034 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.215+0000 7f3d127fc640 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd12f0 0x558f2e0aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.220 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3d137fe640 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcb8d80 0x558f2e0b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3d137fe640 1 -- 172.21.15.29:0/2175231034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcb8d80 msgr2=0x558f2e0b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3d137fe640 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcb8d80 0x558f2e0b0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3d137fe640 1 -- 172.21.15.29:0/2175231034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd12f0 msgr2=0x558f2e0aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3d137fe640 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd12f0 0x558f2e0aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3d137fe640 1 -- 172.21.15.29:0/2175231034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f2e0af8b0 con 0x558f2dcd94d0 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3d137fe640 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd94d0 0x558f2e0af170 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c009d50 tx=0x7f3d0c004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b951f2da8425fa server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3cfa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3cfa7fc640 1 -- 172.21.15.29:0/2175231034 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c028c40 con 0x558f2dcd94d0 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3cfa7fc640 1 -- 172.21.15.29:0/2175231034 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f3d0c003a30 con 0x558f2dcd94d0 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3d203c1740 1 -- 172.21.15.29:0/2175231034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d0404a070 con 0x558f2dcd94d0 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3cfa7fc640 1 -- 172.21.15.29:0/2175231034 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3d0c003c30 con 0x558f2dcd94d0 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3d203c1740 10 client.15798 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.216+0000 7f3d203c1740 1 -- 172.21.15.29:0/2175231034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558f2e1215e0 con 0x558f2dcd94d0 2024-07-27T06:47:56.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.217+0000 7f3cfa7fc640 1 -- 172.21.15.29:0/2175231034 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d0404a070 con 0x558f2dcd94d0 2024-07-27T06:47:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.217+0000 7f3cfa7fc640 1 -- 172.21.15.29:0/2175231034 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3d0c026900 con 0x558f2dcd94d0 2024-07-27T06:47:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.217+0000 7f3d203c1740 10 client.15798 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.217+0000 7f3d203c1740 10 client.15798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:47:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.217+0000 7f3d203c1740 20 client.15798 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.217+0000 7f3d203c1740 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x558f2dcfa1b0 0x558f2e123c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.217+0000 7f3d203c1740 4 client.15798 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-27T06:47:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.217+0000 7f3d203c1740 1 -- 172.21.15.29:0/2175231034 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558f2e00fda0 con 0x558f2dcfa1b0 2024-07-27T06:47:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.217+0000 7f3d127fc640 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x558f2dcfa1b0 0x558f2e123c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.218+0000 7f3cfa7fc640 10 client.15798 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:56.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.218+0000 7f3d127fc640 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x558f2dcfa1b0 0x558f2e123c30 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e205eb265d3fa4 server_cookie=9069d71de7aee859 in_seq=0 out_seq=0 2024-07-27T06:47:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.222+0000 7f3cfa7fc640 1 -- 172.21.15.29:0/2175231034 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558f2e00fda0 con 0x558f2dcfa1b0 2024-07-27T06:47:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.222+0000 7f3cfa7fc640 10 client.15798 handle_command_reply: tid=0 2024-07-27T06:47:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.222+0000 7f3d203c1740 1 client.15798 shutdown 2024-07-27T06:47:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.222+0000 7f3d203c1740 1 -- 172.21.15.29:0/2175231034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd94d0 msgr2=0x558f2e0af170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.222+0000 7f3d203c1740 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd94d0 0x558f2e0af170 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c009d50 tx=0x7f3d0c004280 comp rx=0 tx=0).stop 2024-07-27T06:47:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.222+0000 7f3d203c1740 1 -- 172.21.15.29:0/2175231034 shutdown_connections 2024-07-27T06:47:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.222+0000 7f3d203c1740 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558f2dcd94d0 0x558f2e0af170 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.222+0000 7f3d203c1740 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558f2dcd12f0 0x558f2e0aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.222+0000 7f3d203c1740 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x558f2dcfa1b0 0x558f2e123c30 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.222+0000 7f3d203c1740 1 --2- 172.21.15.29:0/2175231034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558f2dcb8d80 0x558f2e0b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.222+0000 7f3cfa7fc640 0 client.15798 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:47:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.222+0000 7f3d203c1740 1 -- 172.21.15.29:0/2175231034 >> 172.21.15.29:0/2175231034 conn(0x558f2dcee300 msgr2=0x558f2e0be730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d203c1740 1 -- 172.21.15.29:0/2175231034 shutdown_connections 2024-07-27T06:47:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d203c1740 1 -- 172.21.15.29:0/2175231034 wait complete. 2024-07-27T06:47:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d203c1740 20 client.15798 trim_cache size 0 max 16384 2024-07-27T06:47:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-07-27T06:47:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-07-27T06:47:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-07-27T06:47:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-07-27T06:47:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d194a9640 1 -- 172.21.15.29:0/1238539573 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3d000b3d60 msgr2=0x7f3d000b6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d194a9640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3d000b3d60 0x7f3d000b6270 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x558f2e09b330 tx=0x7f3cfc018420 comp rx=0 tx=0).stop 2024-07-27T06:47:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d194a9640 1 -- 172.21.15.29:0/1238539573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d14102130 msgr2=0x7f3d141a0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d194a9640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d14102130 0x7f3d141a0520 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00e9f0 tx=0x7f3cfc00edf0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d194a9640 1 -- 172.21.15.29:0/1238539573 shutdown_connections 2024-07-27T06:47:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d194a9640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d141075f0 0x7f3d141a5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d194a9640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d14102a80 0x7f3d141a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d194a9640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3d000b3d60 0x7f3d000b6270 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d194a9640 1 --2- 172.21.15.29:0/1238539573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d14102130 0x7f3d141a0520 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.223+0000 7f3d194a9640 1 -- 172.21.15.29:0/1238539573 >> 172.21.15.29:0/1238539573 conn(0x7f3d140fddf0 msgr2=0x7f3d14104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.224+0000 7f3d194a9640 1 -- 172.21.15.29:0/1238539573 shutdown_connections 2024-07-27T06:47:56.227 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:47:56.224+0000 7f3d194a9640 1 -- 172.21.15.29:0/1238539573 wait complete. 2024-07-27T06:47:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:56 smithi029 ceph-mon[32610]: pgmap v425: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-27T06:47:57.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-27T06:47:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:56 smithi077 ceph-mon[37153]: pgmap v425: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-27T06:47:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:56 smithi047 ceph-mon[36835]: pgmap v425: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-27T06:47:57.426 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:47:58.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:47:58 smithi029 ceph-mon[32610]: pgmap v426: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 12 op/s 2024-07-27T06:47:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:47:58 smithi077 ceph-mon[37153]: pgmap v426: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 12 op/s 2024-07-27T06:47:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:47:58 smithi047 ceph-mon[36835]: pgmap v426: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 12 op/s 2024-07-27T06:47:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.275+0000 7fb656226640 1 -- 172.21.15.29:0/2658371373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb650108720 msgr2=0x7fb65010ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.275+0000 7fb656226640 1 --2- 172.21.15.29:0/2658371373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb650108720 0x7fb65010ef50 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb648009ca0 tx=0x7fb64802a8c0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.275+0000 7fb656226640 1 -- 172.21.15.29:0/2658371373 shutdown_connections 2024-07-27T06:47:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.275+0000 7fb656226640 1 --2- 172.21.15.29:0/2658371373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb650108720 0x7fb65010ef50 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.275+0000 7fb656226640 1 --2- 172.21.15.29:0/2658371373 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb650107dd0 0x7fb6501081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.275+0000 7fb656226640 1 --2- 172.21.15.29:0/2658371373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb650102490 0x7fb6501028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.275+0000 7fb656226640 1 -- 172.21.15.29:0/2658371373 >> 172.21.15.29:0/2658371373 conn(0x7fb650079160 msgr2=0x7fb650079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.275+0000 7fb656226640 1 -- 172.21.15.29:0/2658371373 shutdown_connections 2024-07-27T06:47:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.275+0000 7fb656226640 1 -- 172.21.15.29:0/2658371373 wait complete. 2024-07-27T06:47:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.276+0000 7fb656226640 1 Processor -- start 2024-07-27T06:47:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.276+0000 7fb656226640 1 -- start start 2024-07-27T06:47:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.276+0000 7fb656226640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb650102490 0x7fb650199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.276+0000 7fb656226640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb650107dd0 0x7fb65019a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb656226640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb650108720 0x7fb65019f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb656226640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb6501134b0 con 0x7fb650108720 2024-07-27T06:47:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb656226640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fb650113330 con 0x7fb650102490 2024-07-27T06:47:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb656226640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb650113630 con 0x7fb650107dd0 2024-07-27T06:47:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb64ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb650108720 0x7fb65019f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb64ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb650108720 0x7fb65019f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49824/0 (socket says 172.21.15.29:49824) 2024-07-27T06:47:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb64ffff640 1 -- 172.21.15.29:0/3265207224 learned_addr learned my addr 172.21.15.29:0/3265207224 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb650102490 0x7fb650199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb64effd640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb650107dd0 0x7fb65019a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb64ffff640 1 -- 172.21.15.29:0/3265207224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb650107dd0 msgr2=0x7fb65019a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb64ffff640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb650107dd0 0x7fb65019a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb64ffff640 1 -- 172.21.15.29:0/3265207224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb650102490 msgr2=0x7fb650199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb64ffff640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb650102490 0x7fb650199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb64ffff640 1 -- 172.21.15.29:0/3265207224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65019fd70 con 0x7fb650108720 2024-07-27T06:47:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.277+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb650102490 0x7fb650199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.278+0000 7fb64ffff640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb650108720 0x7fb65019f640 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb6480098d0 tx=0x7fb6480317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.278+0000 7fb64cff9640 1 -- 172.21.15.29:0/3265207224 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648003b70 con 0x7fb650108720 2024-07-27T06:47:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.278+0000 7fb64cff9640 1 -- 172.21.15.29:0/3265207224 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb648003d10 con 0x7fb650108720 2024-07-27T06:47:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.278+0000 7fb64cff9640 1 -- 172.21.15.29:0/3265207224 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64802fa20 con 0x7fb650108720 2024-07-27T06:47:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.278+0000 7fb656226640 1 -- 172.21.15.29:0/3265207224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6501a0000 con 0x7fb650108720 2024-07-27T06:47:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.278+0000 7fb656226640 1 -- 172.21.15.29:0/3265207224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb65010f8c0 con 0x7fb650108720 2024-07-27T06:47:59.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.281+0000 7fb65c7aa740 1 Processor -- start 2024-07-27T06:47:59.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.281+0000 7fb64cff9640 1 -- 172.21.15.29:0/3265207224 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) ==== 151133+0+0 (secure 0 0 0) 0x7fb64802fbc0 con 0x7fb650108720 2024-07-27T06:47:59.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.282+0000 7fb65c7aa740 1 -- start start 2024-07-27T06:47:59.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.282+0000 7fb65c7aa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d534070 0x55ca8d8d9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.282+0000 7fb65c7aa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8d9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.282+0000 7fb65c7aa740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d513390 0x55ca8d8d9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.282+0000 7fb65c7aa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ca8d84fbb0 con 0x55ca8d534070 2024-07-27T06:47:59.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.282+0000 7fb65c7aa740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55ca8d83f0b0 con 0x55ca8d513390 2024-07-27T06:47:59.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.282+0000 7fb65c7aa740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ca8d8430f0 con 0x55ca8d50b1b0 2024-07-27T06:47:59.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.282+0000 7fb64effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d534070 0x55ca8d8d9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.282+0000 7fb64effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d534070 0x55ca8d8d9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49826/0 (socket says 172.21.15.29:49826) 2024-07-27T06:47:59.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.282+0000 7fb64effd640 1 -- 172.21.15.29:0/300932043 learned_addr learned my addr 172.21.15.29:0/300932043 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:59.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.282+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/300932043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8d9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.283+0000 7fb64cff9640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb6380b3cb0 0x7fb6380b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.283+0000 7fb64f7fe640 1 -- 172.21.15.29:0/300932043 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d513390 msgr2=0x55ca8d8d9ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T06:47:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.283+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/300932043 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d513390 0x55ca8d8d9ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.283+0000 7fb64f7fe640 1 -- 172.21.15.29:0/300932043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d534070 msgr2=0x55ca8d8d9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.283+0000 7fb64cff9640 1 -- 172.21.15.29:0/3265207224 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fb648105560 con 0x7fb650108720 2024-07-27T06:47:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.283+0000 7fb64ffff640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb6380b3cb0 0x7fb6380b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.283+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/300932043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d534070 0x55ca8d8d9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.283+0000 7fb64f7fe640 1 -- 172.21.15.29:0/300932043 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca8d7e5b90 con 0x55ca8d50b1b0 2024-07-27T06:47:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.283+0000 7fb64effd640 1 --2- 172.21.15.29:0/300932043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d534070 0x55ca8d8d9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:47:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.284+0000 7fb64ffff640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb6380b3cb0 0x7fb6380b61c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb648003fd0 tx=0x7fb648009430 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.284+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/300932043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8d9ab0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb640009c30 tx=0x7fb6400117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31b119fffd803a45 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:59.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.284+0000 7fb635ffb640 1 -- 172.21.15.29:0/300932043 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640005020 con 0x55ca8d50b1b0 2024-07-27T06:47:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.284+0000 7fb635ffb640 1 -- 172.21.15.29:0/300932043 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6400047a0 con 0x55ca8d50b1b0 2024-07-27T06:47:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.284+0000 7fb635ffb640 1 -- 172.21.15.29:0/300932043 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6400126b0 con 0x55ca8d50b1b0 2024-07-27T06:47:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.285+0000 7fb65c7aa740 1 -- 172.21.15.29:0/300932043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 msgr2=0x55ca8d8d9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:59.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.285+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/300932043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8d9ab0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb640009c30 tx=0x7fb6400117b0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.285+0000 7fb65c7aa740 1 -- 172.21.15.29:0/300932043 shutdown_connections 2024-07-27T06:47:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.285+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/300932043 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d513390 0x55ca8d8d9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.285+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/300932043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8d9ab0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.285+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/300932043 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d534070 0x55ca8d8d9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.285+0000 7fb65c7aa740 1 -- 172.21.15.29:0/300932043 >> 172.21.15.29:0/300932043 conn(0x55ca8d528460 msgr2=0x55ca8d8d7460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.285+0000 7fb65c7aa740 1 -- 172.21.15.29:0/300932043 shutdown_connections 2024-07-27T06:47:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.285+0000 7fb65c7aa740 1 -- 172.21.15.29:0/300932043 wait complete. 2024-07-27T06:47:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.286+0000 7fb65c7aa740 1 -- start start 2024-07-27T06:47:59.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.286+0000 7fb65c7aa740 1 Processor -- start 2024-07-27T06:47:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.286+0000 7fb65c7aa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8e5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.286+0000 7fb65c7aa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 0x55ca8d8e56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.286+0000 7fb65c7aa740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d534070 0x55ca8d8f1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.286+0000 7fb65c7aa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ca8d6a2700 con 0x55ca8d513390 2024-07-27T06:47:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.286+0000 7fb65c7aa740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55ca8d7e19e0 con 0x55ca8d534070 2024-07-27T06:47:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.286+0000 7fb65c7aa740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ca8d8e7d50 con 0x55ca8d50b1b0 2024-07-27T06:47:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb65c7aa740 10 client.0 _ll_register_callbacks cb 0x55ca8d7fc4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb64f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 0x55ca8d8e56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb64f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 0x55ca8d8e56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49828/0 (socket says 172.21.15.29:49828) 2024-07-27T06:47:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb64f7fe640 1 -- 172.21.15.29:0/4016304975 learned_addr learned my addr 172.21.15.29:0/4016304975 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb64effd640 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8e5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb64effd640 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d534070 0x55ca8d8f1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb64f7fe640 1 -- 172.21.15.29:0/4016304975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 msgr2=0x55ca8d8e5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:59.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8e5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb64f7fe640 1 -- 172.21.15.29:0/4016304975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d534070 msgr2=0x55ca8d8f1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d534070 0x55ca8d8f1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb64f7fe640 1 -- 172.21.15.29:0/4016304975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca8d59f650 con 0x55ca8d513390 2024-07-27T06:47:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb64effd640 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d534070 0x55ca8d8f1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:47:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 0x55ca8d8e56b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb640009570 tx=0x7fb640004b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c32908be6c9046 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.287+0000 7fb62effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.288+0000 7fb62effd640 1 -- 172.21.15.29:0/4016304975 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640028020 con 0x55ca8d513390 2024-07-27T06:47:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.288+0000 7fb65c7aa740 1 -- 172.21.15.29:0/4016304975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6400047a0 con 0x55ca8d513390 2024-07-27T06:47:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.288+0000 7fb62effd640 1 -- 172.21.15.29:0/4016304975 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fb6400050f0 con 0x55ca8d513390 2024-07-27T06:47:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.288+0000 7fb62effd640 1 -- 172.21.15.29:0/4016304975 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb640033050 con 0x55ca8d513390 2024-07-27T06:47:59.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.288+0000 7fb65c7aa740 10 client.15810 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:59.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.288+0000 7fb65c7aa740 1 -- 172.21.15.29:0/4016304975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ca8d8f3d90 con 0x55ca8d513390 2024-07-27T06:47:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.288+0000 7fb62effd640 1 -- 172.21.15.29:0/4016304975 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6400047a0 con 0x55ca8d513390 2024-07-27T06:47:59.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.288+0000 7fb62effd640 1 -- 172.21.15.29:0/4016304975 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb640012450 con 0x55ca8d513390 2024-07-27T06:47:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.289+0000 7fb65c7aa740 10 client.15810 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.289+0000 7fb65c7aa740 10 client.15810 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-07-27T06:47:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.289+0000 7fb65c7aa740 20 client.15810 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.289+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55ca8d4f29f0 0x55ca8d8f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.289+0000 7fb65c7aa740 4 client.15810 mds_command: new command op to 24646 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:47:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.289+0000 7fb65c7aa740 1 -- 172.21.15.29:0/4016304975 --> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca8d851b00 con 0x55ca8d4f29f0 2024-07-27T06:47:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.289+0000 7fb64effd640 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55ca8d4f29f0 0x55ca8d8f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.290+0000 7fb64effd640 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55ca8d4f29f0 0x55ca8d8f71a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bdde2a8791feb343 server_cookie=7189290c01a8be2c in_seq=0 out_seq=0 2024-07-27T06:47:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.290+0000 7fb62effd640 10 client.15810 ms_handle_connect on v2:172.21.15.77:6834/439792826 2024-07-27T06:47:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.290+0000 7fb62effd640 1 -- 172.21.15.29:0/4016304975 <== mds.2 v2:172.21.15.77:6834/439792826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ca8d851b00 con 0x55ca8d4f29f0 2024-07-27T06:47:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.290+0000 7fb62effd640 10 client.15810 handle_command_reply: tid=0 2024-07-27T06:47:59.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb65c7aa740 1 client.15810 shutdown 2024-07-27T06:47:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb65c7aa740 1 -- 172.21.15.29:0/4016304975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 msgr2=0x55ca8d8e56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 0x55ca8d8e56b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fb640009570 tx=0x7fb640004b30 comp rx=0 tx=0).stop 2024-07-27T06:47:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb65c7aa740 1 -- 172.21.15.29:0/4016304975 shutdown_connections 2024-07-27T06:47:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55ca8d4f29f0 0x55ca8d8f71a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb62effd640 0 client.15810 ms_handle_reset on v2:172.21.15.77:6834/439792826 2024-07-27T06:47:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d534070 0x55ca8d8f1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 0x55ca8d8e56b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/4016304975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8e5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb65c7aa740 1 -- 172.21.15.29:0/4016304975 >> 172.21.15.29:0/4016304975 conn(0x55ca8d528460 msgr2=0x55ca8d8d5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb65c7aa740 1 -- 172.21.15.29:0/4016304975 shutdown_connections 2024-07-27T06:47:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb65c7aa740 1 -- 172.21.15.29:0/4016304975 wait complete. 2024-07-27T06:47:59.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.291+0000 7fb65c7aa740 20 client.15810 trim_cache size 0 max 16384 2024-07-27T06:47:59.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.317+0000 7fb65c7aa740 1 Processor -- start 2024-07-27T06:47:59.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.317+0000 7fb65c7aa740 1 -- start start 2024-07-27T06:47:59.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.317+0000 7fb65c7aa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8d7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.317+0000 7fb65c7aa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 0x55ca8d8f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.317+0000 7fb65c7aa740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d534070 0x55ca8d8f14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.317+0000 7fb65c7aa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb6501ace80 con 0x55ca8d513390 2024-07-27T06:47:59.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.317+0000 7fb65c7aa740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fb6501ae9c0 con 0x55ca8d534070 2024-07-27T06:47:59.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.317+0000 7fb65c7aa740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb6501ad550 con 0x55ca8d50b1b0 2024-07-27T06:47:59.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.318+0000 7fb64f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 0x55ca8d8f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.318+0000 7fb64f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 0x55ca8d8f0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49830/0 (socket says 172.21.15.29:49830) 2024-07-27T06:47:59.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.318+0000 7fb64f7fe640 1 -- 172.21.15.29:0/3019642699 learned_addr learned my addr 172.21.15.29:0/3019642699 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:59.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.318+0000 7fb64effd640 1 --2- 172.21.15.29:0/3019642699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d534070 0x55ca8d8f14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.318+0000 7fb64effd640 1 --2- 172.21.15.29:0/3019642699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8d7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.318+0000 7fb64f7fe640 1 -- 172.21.15.29:0/3019642699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 msgr2=0x55ca8d8d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:59.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.318+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/3019642699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8d7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.318+0000 7fb64f7fe640 1 -- 172.21.15.29:0/3019642699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d534070 msgr2=0x55ca8d8f14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:59.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.318+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/3019642699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d534070 0x55ca8d8f14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.318+0000 7fb64f7fe640 1 -- 172.21.15.29:0/3019642699 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca8d8f1a30 con 0x55ca8d513390 2024-07-27T06:47:59.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.319+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/3019642699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 0x55ca8d8f0fb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb640001eb0 tx=0x7fb6400092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddbae8f7bd41b39e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:59.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.319+0000 7fb6357fa640 1 -- 172.21.15.29:0/3019642699 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640028070 con 0x55ca8d513390 2024-07-27T06:47:59.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.319+0000 7fb6357fa640 1 -- 172.21.15.29:0/3019642699 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb640004ba0 con 0x55ca8d513390 2024-07-27T06:47:59.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.319+0000 7fb6357fa640 1 -- 172.21.15.29:0/3019642699 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640005350 con 0x55ca8d513390 2024-07-27T06:47:59.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.320+0000 7fb65c7aa740 1 -- 172.21.15.29:0/3019642699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 msgr2=0x55ca8d8f0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:59.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.320+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/3019642699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 0x55ca8d8f0fb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb640001eb0 tx=0x7fb6400092a0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.320+0000 7fb65c7aa740 1 -- 172.21.15.29:0/3019642699 shutdown_connections 2024-07-27T06:47:59.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.320+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/3019642699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d534070 0x55ca8d8f14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.320+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/3019642699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d513390 0x55ca8d8f0fb0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.320+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/3019642699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8d7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.320+0000 7fb65c7aa740 1 -- 172.21.15.29:0/3019642699 >> 172.21.15.29:0/3019642699 conn(0x55ca8d528460 msgr2=0x55ca8d8d6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:59.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.320+0000 7fb65c7aa740 1 -- 172.21.15.29:0/3019642699 shutdown_connections 2024-07-27T06:47:59.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.320+0000 7fb65c7aa740 1 -- 172.21.15.29:0/3019642699 wait complete. 2024-07-27T06:47:59.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.321+0000 7fb65c7aa740 1 -- start start 2024-07-27T06:47:59.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.321+0000 7fb65c7aa740 1 Processor -- start 2024-07-27T06:47:59.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb65c7aa740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8fa900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb65c7aa740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d513390 0x55ca8d8f7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb65c7aa740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d534070 0x55ca8d8f8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb65c7aa740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb6501ae2f0 con 0x55ca8d50b1b0 2024-07-27T06:47:59.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb65c7aa740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fb6501adc20 con 0x55ca8d513390 2024-07-27T06:47:59.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb65c7aa740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb6501aae60 con 0x55ca8d534070 2024-07-27T06:47:59.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb64effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8fa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb65c7aa740 10 client.0 _ll_register_callbacks cb 0x7fb64400fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:47:59.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb64effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8fa900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49832/0 (socket says 172.21.15.29:49832) 2024-07-27T06:47:59.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb64effd640 1 -- 172.21.15.29:0/1282181636 learned_addr learned my addr 172.21.15.29:0/1282181636 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:47:59.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d513390 0x55ca8d8f7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb64effd640 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d534070 0x55ca8d8f8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb64effd640 1 -- 172.21.15.29:0/1282181636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d534070 msgr2=0x55ca8d8f8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb64effd640 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d534070 0x55ca8d8f8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb64effd640 1 -- 172.21.15.29:0/1282181636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d513390 msgr2=0x55ca8d8f7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb64effd640 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d513390 0x55ca8d8f7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.322+0000 7fb64effd640 1 -- 172.21.15.29:0/1282181636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca8d8f8b00 con 0x55ca8d50b1b0 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.323+0000 7fb64effd640 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8fa900 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55ca8d6a2800 tx=0x7fb6400098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4e408d4fa82399 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.323+0000 7fb62effd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.323+0000 7fb62effd640 1 -- 172.21.15.29:0/1282181636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb644005020 con 0x55ca8d50b1b0 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.323+0000 7fb62effd640 1 -- 172.21.15.29:0/1282181636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fb6440051c0 con 0x55ca8d50b1b0 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.323+0000 7fb65c7aa740 1 -- 172.21.15.29:0/1282181636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb640004ba0 con 0x55ca8d50b1b0 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.323+0000 7fb62effd640 1 -- 172.21.15.29:0/1282181636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb6440053c0 con 0x55ca8d50b1b0 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.323+0000 7fb65c7aa740 10 client.15816 fetch_fsmap learned FSMap version 29 2024-07-27T06:47:59.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.323+0000 7fb65c7aa740 1 -- 172.21.15.29:0/1282181636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ca8d95f730 con 0x55ca8d50b1b0 2024-07-27T06:47:59.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.324+0000 7fb62effd640 1 -- 172.21.15.29:0/1282181636 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb640004ba0 con 0x55ca8d50b1b0 2024-07-27T06:47:59.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.324+0000 7fb62effd640 1 -- 172.21.15.29:0/1282181636 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb640009d20 con 0x55ca8d50b1b0 2024-07-27T06:47:59.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.324+0000 7fb65c7aa740 10 client.15816 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:47:59.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.324+0000 7fb65c7aa740 10 client.15816 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-07-27T06:47:59.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.324+0000 7fb65c7aa740 20 client.15816 populate_metadata read hostname 'smithi029' 2024-07-27T06:47:59.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.324+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55ca8d4c7da0 0x55ca8d961ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:47:59.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.324+0000 7fb65c7aa740 4 client.15816 mds_command: new command op to 24646 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:47:59.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.324+0000 7fb65c7aa740 1 -- 172.21.15.29:0/1282181636 --> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ca8d84d5a0 con 0x55ca8d4c7da0 2024-07-27T06:47:59.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.324+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55ca8d4c7da0 0x55ca8d961ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:47:59.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb62effd640 10 client.15816 ms_handle_connect on v2:172.21.15.77:6834/439792826 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb64f7fe640 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55ca8d4c7da0 0x55ca8d961ed0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=107b66446900efdc server_cookie=6643adf131944bce in_seq=0 out_seq=0 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb62effd640 1 -- 172.21.15.29:0/1282181636 <== mds.2 v2:172.21.15.77:6834/439792826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ca8d84d5a0 con 0x55ca8d4c7da0 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb62effd640 10 client.15816 handle_command_reply: tid=0 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb65c7aa740 1 client.15816 shutdown 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb65c7aa740 1 -- 172.21.15.29:0/1282181636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d50b1b0 msgr2=0x55ca8d8fa900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8fa900 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55ca8d6a2800 tx=0x7fb6400098c0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb65c7aa740 1 -- 172.21.15.29:0/1282181636 shutdown_connections 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x55ca8d4c7da0 0x55ca8d961ed0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ca8d534070 0x55ca8d8f8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb62effd640 0 client.15816 ms_handle_reset on v2:172.21.15.77:6834/439792826 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ca8d513390 0x55ca8d8f7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb65c7aa740 1 --2- 172.21.15.29:0/1282181636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ca8d50b1b0 0x55ca8d8fa900 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.325+0000 7fb65c7aa740 1 -- 172.21.15.29:0/1282181636 >> 172.21.15.29:0/1282181636 conn(0x55ca8d528460 msgr2=0x55ca8d8da5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:59.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.326+0000 7fb65c7aa740 1 -- 172.21.15.29:0/1282181636 shutdown_connections 2024-07-27T06:47:59.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.326+0000 7fb65c7aa740 1 -- 172.21.15.29:0/1282181636 wait complete. 2024-07-27T06:47:59.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.326+0000 7fb65c7aa740 20 client.15816 trim_cache size 0 max 16384 2024-07-27T06:47:59.330 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:47:59.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.327+0000 7fb656226640 1 -- 172.21.15.29:0/3265207224 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb6380b3cb0 msgr2=0x7fb6380b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:59.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.327+0000 7fb656226640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb6380b3cb0 0x7fb6380b61c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb648003fd0 tx=0x7fb648009430 comp rx=0 tx=0).stop 2024-07-27T06:47:59.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.327+0000 7fb656226640 1 -- 172.21.15.29:0/3265207224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb650108720 msgr2=0x7fb65019f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:47:59.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.327+0000 7fb656226640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb650108720 0x7fb65019f640 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb6480098d0 tx=0x7fb6480317b0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.327+0000 7fb656226640 1 -- 172.21.15.29:0/3265207224 shutdown_connections 2024-07-27T06:47:59.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.327+0000 7fb656226640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb650108720 0x7fb65019f640 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.327+0000 7fb656226640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb6380b3cb0 0x7fb6380b61c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.327+0000 7fb656226640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb650107dd0 0x7fb65019a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.327+0000 7fb656226640 1 --2- 172.21.15.29:0/3265207224 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb650102490 0x7fb650199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:47:59.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.327+0000 7fb656226640 1 -- 172.21.15.29:0/3265207224 >> 172.21.15.29:0/3265207224 conn(0x7fb650079160 msgr2=0x7fb650104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:47:59.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.327+0000 7fb656226640 1 -- 172.21.15.29:0/3265207224 shutdown_connections 2024-07-27T06:47:59.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:47:59.327+0000 7fb656226640 1 -- 172.21.15.29:0/3265207224 wait complete. 2024-07-27T06:48:00.182 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-27T06:48:00.483 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:00.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:00 smithi029 ceph-mon[32610]: pgmap v427: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-07-27T06:48:00.885 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:48:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:48:00.699+0000 7fdc3af1d640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-27T06:48:00.886 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:48:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:48:00.699+0000 7fdc3af1d640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-27T06:48:00.886 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:48:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:48:00.699+0000 7fdc3af1d640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-27T06:48:00.886 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:48:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:48:00.699+0000 7fdc3af1d640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-27T06:48:00.886 INFO:journalctl@ceph.mgr.x.smithi029.stdout:Jul 27 06:48:00 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mgr-x[32837]: 2024-07-27T06:48:00.699+0000 7fdc3af1d640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-27T06:48:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:00 smithi077 ceph-mon[37153]: pgmap v427: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-07-27T06:48:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:00 smithi047 ceph-mon[36835]: pgmap v427: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-07-27T06:48:02.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.495+0000 7f7d261d5640 1 -- 172.21.15.29:0/2124415524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d20102490 msgr2=0x7f7d201028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:02.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.495+0000 7f7d261d5640 1 --2- 172.21.15.29:0/2124415524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d20102490 0x7f7d201028a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7d10009900 tx=0x7f7d1002a410 comp rx=0 tx=0).stop 2024-07-27T06:48:02.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.496+0000 7f7d261d5640 1 -- 172.21.15.29:0/2124415524 shutdown_connections 2024-07-27T06:48:02.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.496+0000 7f7d261d5640 1 --2- 172.21.15.29:0/2124415524 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d20108720 0x7f7d2010ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.496+0000 7f7d261d5640 1 --2- 172.21.15.29:0/2124415524 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d20107dd0 0x7f7d201081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.496+0000 7f7d261d5640 1 --2- 172.21.15.29:0/2124415524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d20102490 0x7f7d201028a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.496+0000 7f7d261d5640 1 -- 172.21.15.29:0/2124415524 >> 172.21.15.29:0/2124415524 conn(0x7f7d20079160 msgr2=0x7f7d20079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:02.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.496+0000 7f7d261d5640 1 -- 172.21.15.29:0/2124415524 shutdown_connections 2024-07-27T06:48:02.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.496+0000 7f7d261d5640 1 -- 172.21.15.29:0/2124415524 wait complete. 2024-07-27T06:48:02.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.497+0000 7f7d261d5640 1 Processor -- start 2024-07-27T06:48:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.497+0000 7f7d261d5640 1 -- start start 2024-07-27T06:48:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.498+0000 7f7d261d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d20102490 0x7f7d201145e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.498+0000 7f7d261d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d20107dd0 0x7f7d20112bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.498+0000 7f7d1f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d20102490 0x7f7d201145e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.498+0000 7f7d1f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d20102490 0x7f7d201145e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:42028/0 (socket says 172.21.15.29:42028) 2024-07-27T06:48:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.498+0000 7f7d1effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d20107dd0 0x7f7d20112bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.498+0000 7f7d1effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d20107dd0 0x7f7d20112bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49874/0 (socket says 172.21.15.29:49874) 2024-07-27T06:48:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.498+0000 7f7d1f7fe640 1 -- 172.21.15.29:0/1375253801 learned_addr learned my addr 172.21.15.29:0/1375253801 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.498+0000 7f7d261d5640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d20108720 0x7f7d20113110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.498+0000 7f7d261d5640 1 -- 172.21.15.29:0/1375253801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201123a0 con 0x7f7d20107dd0 2024-07-27T06:48:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.498+0000 7f7d261d5640 1 -- 172.21.15.29:0/1375253801 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7d20112220 con 0x7f7d20108720 2024-07-27T06:48:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.498+0000 7f7d261d5640 1 -- 172.21.15.29:0/1375253801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7d20112520 con 0x7f7d20102490 2024-07-27T06:48:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.499+0000 7f7d1ffff640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d20108720 0x7f7d20113110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.499+0000 7f7d1effd640 1 -- 172.21.15.29:0/1375253801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d20102490 msgr2=0x7f7d201145e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.499+0000 7f7d1effd640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d20102490 0x7f7d201145e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.499+0000 7f7d1effd640 1 -- 172.21.15.29:0/1375253801 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d20108720 msgr2=0x7f7d20113110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.499+0000 7f7d1effd640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d20108720 0x7f7d20113110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.499+0000 7f7d1effd640 1 -- 172.21.15.29:0/1375253801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d20113890 con 0x7f7d20107dd0 2024-07-27T06:48:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.499+0000 7f7d1effd640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d20107dd0 0x7f7d20112bd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7d14005fd0 tx=0x7f7d1400de70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.500+0000 7f7d1cff9640 1 -- 172.21.15.29:0/1375253801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d14004230 con 0x7f7d20107dd0 2024-07-27T06:48:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.500+0000 7f7d261d5640 1 -- 172.21.15.29:0/1375253801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d201ac8a0 con 0x7f7d20107dd0 2024-07-27T06:48:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.500+0000 7f7d1cff9640 1 -- 172.21.15.29:0/1375253801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d14009d60 con 0x7f7d20107dd0 2024-07-27T06:48:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.500+0000 7f7d1cff9640 1 -- 172.21.15.29:0/1375253801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d14003e60 con 0x7f7d20107dd0 2024-07-27T06:48:02.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.500+0000 7f7d261d5640 1 -- 172.21.15.29:0/1375253801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d201acdd0 con 0x7f7d20107dd0 2024-07-27T06:48:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.504+0000 7f7d1cff9640 1 -- 172.21.15.29:0/1375253801 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f7d140098b0 con 0x7f7d20107dd0 2024-07-27T06:48:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.504+0000 7f7d2c759740 1 Processor -- start 2024-07-27T06:48:02.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.504+0000 7f7d2c759740 1 -- start start 2024-07-27T06:48:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.504+0000 7f7d2c759740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c589070 0x55774c92e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.504+0000 7f7d2c759740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 0x55774c92ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.504+0000 7f7d2c759740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c92f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.504+0000 7f7d2c759740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55774c8a4bb0 con 0x55774c5601b0 2024-07-27T06:48:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.504+0000 7f7d2c759740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55774c8940b0 con 0x55774c568390 2024-07-27T06:48:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.504+0000 7f7d2c759740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55774c8980f0 con 0x55774c589070 2024-07-27T06:48:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.504+0000 7f7d1cff9640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7d080b3dc0 0x7f7d080b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.505+0000 7f7d1f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 0x55774c92ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.505+0000 7f7d1ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c589070 0x55774c92e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.505+0000 7f7d1f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 0x55774c92ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49890/0 (socket says 172.21.15.29:49890) 2024-07-27T06:48:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.505+0000 7f7d1f7fe640 1 -- 172.21.15.29:0/1283108220 learned_addr learned my addr 172.21.15.29:0/1283108220 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.505+0000 7f7d1ffff640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7d080b3dc0 0x7f7d080b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.505+0000 7f7d1cff9640 1 -- 172.21.15.29:0/1375253801 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7d140e3890 con 0x7f7d20107dd0 2024-07-27T06:48:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.505+0000 7f7d1f7fe640 1 -- 172.21.15.29:0/1283108220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c589070 msgr2=0x55774c92e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.505+0000 7f7d1f7fe640 1 --2- 172.21.15.29:0/1283108220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c589070 0x55774c92e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.505+0000 7f7d1f7fe640 1 -- 172.21.15.29:0/1283108220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 msgr2=0x55774c92f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.505+0000 7f7d1f7fe640 1 --2- 172.21.15.29:0/1283108220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c92f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.505+0000 7f7d1f7fe640 1 -- 172.21.15.29:0/1283108220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55774c83ab90 con 0x55774c5601b0 2024-07-27T06:48:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.505+0000 7f7d1f7fe640 1 --2- 172.21.15.29:0/1283108220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 0x55774c92ebd0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7d10000c00 tx=0x7f7d10031c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e7cf349e09eeaf8 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.506+0000 7f7d05ffb640 1 -- 172.21.15.29:0/1283108220 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10040070 con 0x55774c5601b0 2024-07-27T06:48:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.506+0000 7f7d05ffb640 1 -- 172.21.15.29:0/1283108220 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d100316a0 con 0x55774c5601b0 2024-07-27T06:48:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.506+0000 7f7d05ffb640 1 -- 172.21.15.29:0/1283108220 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10035040 con 0x55774c5601b0 2024-07-27T06:48:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.506+0000 7f7d1ffff640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7d080b3dc0 0x7f7d080b62d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7d18005fd0 tx=0x7f7d18009600 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:02.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.507+0000 7f7d2c759740 1 -- 172.21.15.29:0/1283108220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 msgr2=0x55774c92ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.507+0000 7f7d2c759740 1 --2- 172.21.15.29:0/1283108220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 0x55774c92ebd0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7d10000c00 tx=0x7f7d10031c40 comp rx=0 tx=0).stop 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.507+0000 7f7d2c759740 1 -- 172.21.15.29:0/1283108220 shutdown_connections 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.507+0000 7f7d2c759740 1 --2- 172.21.15.29:0/1283108220 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c92f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.507+0000 7f7d2c759740 1 --2- 172.21.15.29:0/1283108220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 0x55774c92ebd0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.507+0000 7f7d2c759740 1 --2- 172.21.15.29:0/1283108220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c589070 0x55774c92e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.507+0000 7f7d2c759740 1 -- 172.21.15.29:0/1283108220 >> 172.21.15.29:0/1283108220 conn(0x55774c57d460 msgr2=0x55774c92c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.507+0000 7f7d2c759740 1 -- 172.21.15.29:0/1283108220 shutdown_connections 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.507+0000 7f7d2c759740 1 -- 172.21.15.29:0/1283108220 wait complete. 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.508+0000 7f7d2c759740 1 -- start start 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.508+0000 7f7d2c759740 1 Processor -- start 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.509+0000 7f7d2c759740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 0x55774c93a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.509+0000 7f7d1f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 0x55774c93a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.509+0000 7f7d1f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 0x55774c93a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49902/0 (socket says 172.21.15.29:49902) 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.509+0000 7f7d1f7fe640 1 -- 172.21.15.29:0/3115061718 learned_addr learned my addr 172.21.15.29:0/3115061718 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.509+0000 7f7d2c759740 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c93a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.509+0000 7f7d2c759740 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c589070 0x55774c946100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.509+0000 7f7d2c759740 1 -- 172.21.15.29:0/3115061718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55774c6f7700 con 0x55774c5601b0 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.509+0000 7f7d2c759740 1 -- 172.21.15.29:0/3115061718 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55774c8369e0 con 0x55774c568390 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.509+0000 7f7d2c759740 1 -- 172.21.15.29:0/3115061718 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55774c93ce10 con 0x55774c589070 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.509+0000 7f7d1effd640 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c93a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7d1ffff640 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c589070 0x55774c946100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7d1f7fe640 1 -- 172.21.15.29:0/3115061718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c589070 msgr2=0x55774c946100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7d1f7fe640 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c589070 0x55774c946100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7d1f7fe640 1 -- 172.21.15.29:0/3115061718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 msgr2=0x55774c93a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7d1f7fe640 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c93a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7d1f7fe640 1 -- 172.21.15.29:0/3115061718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55774c5f4650 con 0x55774c5601b0 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7d1ffff640 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c589070 0x55774c946100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7d1f7fe640 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 0x55774c93a2d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f7d10005360 tx=0x7f7d100050a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c978b0a6ffdb80 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7cfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7d2c759740 10 client.0 _ll_register_callbacks cb 0x55774c8514f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7cfeffd640 1 -- 172.21.15.29:0/3115061718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10040030 con 0x55774c5601b0 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7cfeffd640 1 -- 172.21.15.29:0/3115061718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7d10004050 con 0x55774c5601b0 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.510+0000 7f7d2c759740 1 -- 172.21.15.29:0/3115061718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d100316a0 con 0x55774c5601b0 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.511+0000 7f7cfeffd640 1 -- 172.21.15.29:0/3115061718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7d10056050 con 0x55774c5601b0 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.511+0000 7f7d2c759740 10 client.15834 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.511+0000 7f7d2c759740 1 -- 172.21.15.29:0/3115061718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55774c948be0 con 0x55774c5601b0 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.511+0000 7f7cfeffd640 1 -- 172.21.15.29:0/3115061718 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7d10004250 con 0x55774c5601b0 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.511+0000 7f7d2c759740 10 client.15834 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.511+0000 7f7d2c759740 10 client.15834 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-07-27T06:48:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.511+0000 7f7d2c759740 20 client.15834 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.511+0000 7f7d2c759740 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x55774c5479f0 0x55774c94bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.511+0000 7f7d2c759740 4 client.15834 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.511+0000 7f7d2c759740 1 -- 172.21.15.29:0/3115061718 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55774c8a6b00 con 0x55774c5479f0 2024-07-27T06:48:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.512+0000 7f7d1effd640 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x55774c5479f0 0x55774c94bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.512+0000 7f7d1effd640 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x55774c5479f0 0x55774c94bf40 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ffb3aad5da5ebc48 server_cookie=353dcaab0292d057 in_seq=0 out_seq=0 2024-07-27T06:48:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.512+0000 7f7cfeffd640 10 client.15834 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:48:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.513+0000 7f7cfeffd640 1 -- 172.21.15.29:0/3115061718 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55774c8a6b00 con 0x55774c5479f0 2024-07-27T06:48:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.513+0000 7f7cfeffd640 10 client.15834 handle_command_reply: tid=0 2024-07-27T06:48:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.513+0000 7f7d2c759740 1 client.15834 shutdown 2024-07-27T06:48:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.513+0000 7f7d2c759740 1 -- 172.21.15.29:0/3115061718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 msgr2=0x55774c93a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.513+0000 7f7d2c759740 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 0x55774c93a2d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f7d10005360 tx=0x7f7d100050a0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.513+0000 7f7d2c759740 1 -- 172.21.15.29:0/3115061718 shutdown_connections 2024-07-27T06:48:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.513+0000 7f7d2c759740 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x55774c5479f0 0x55774c94bf40 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.513+0000 7f7cfeffd640 0 client.15834 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:48:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.513+0000 7f7d2c759740 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c589070 0x55774c946100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.513+0000 7f7d2c759740 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c93a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.513+0000 7f7d2c759740 1 --2- 172.21.15.29:0/3115061718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c5601b0 0x55774c93a2d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.513+0000 7f7d2c759740 1 -- 172.21.15.29:0/3115061718 >> 172.21.15.29:0/3115061718 conn(0x55774c57d460 msgr2=0x55774c931820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.514+0000 7f7d2c759740 1 -- 172.21.15.29:0/3115061718 shutdown_connections 2024-07-27T06:48:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.514+0000 7f7d2c759740 1 -- 172.21.15.29:0/3115061718 wait complete. 2024-07-27T06:48:02.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.514+0000 7f7d2c759740 20 client.15834 trim_cache size 0 max 16384 2024-07-27T06:48:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.540+0000 7f7d2c759740 1 Processor -- start 2024-07-27T06:48:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.540+0000 7f7d2c759740 1 -- start start 2024-07-27T06:48:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.540+0000 7f7d2c759740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c5479f0 0x55774c946070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.540+0000 7f7d2c759740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c92c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.540+0000 7f7d2c759740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 0x55774c933720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.540+0000 7f7d2c759740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201b5760 con 0x55774c589070 2024-07-27T06:48:02.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.540+0000 7f7d2c759740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201b72a0 con 0x55774c568390 2024-07-27T06:48:02.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.540+0000 7f7d2c759740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201b5e30 con 0x55774c5479f0 2024-07-27T06:48:02.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.540+0000 7f7d1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 0x55774c933720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.540+0000 7f7d1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 0x55774c933720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49910/0 (socket says 172.21.15.29:49910) 2024-07-27T06:48:02.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.540+0000 7f7d1ffff640 1 -- 172.21.15.29:0/3239112356 learned_addr learned my addr 172.21.15.29:0/3239112356 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.540+0000 7f7d1f7fe640 1 --2- 172.21.15.29:0/3239112356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c5479f0 0x55774c946070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.541+0000 7f7d1ffff640 1 -- 172.21.15.29:0/3239112356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c5479f0 msgr2=0x55774c946070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.541+0000 7f7d1ffff640 1 --2- 172.21.15.29:0/3239112356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c5479f0 0x55774c946070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.541+0000 7f7d1ffff640 1 -- 172.21.15.29:0/3239112356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 msgr2=0x55774c92c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.541+0000 7f7d1effd640 1 --2- 172.21.15.29:0/3239112356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c92c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.541+0000 7f7d1ffff640 1 --2- 172.21.15.29:0/3239112356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c92c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.541+0000 7f7d1ffff640 1 -- 172.21.15.29:0/3239112356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55774c93b390 con 0x55774c589070 2024-07-27T06:48:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.541+0000 7f7d1ffff640 1 --2- 172.21.15.29:0/3239112356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 0x55774c933720 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7d18012fb0 tx=0x7f7d18009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c8ab9e6bd3c574c server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.541+0000 7f7d057fa640 1 -- 172.21.15.29:0/3239112356 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1802c070 con 0x55774c589070 2024-07-27T06:48:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.542+0000 7f7d057fa640 1 -- 172.21.15.29:0/3239112356 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d1801eb30 con 0x55774c589070 2024-07-27T06:48:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.542+0000 7f7d057fa640 1 -- 172.21.15.29:0/3239112356 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18027be0 con 0x55774c589070 2024-07-27T06:48:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.543+0000 7f7d2c759740 1 -- 172.21.15.29:0/3239112356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 msgr2=0x55774c933720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.543+0000 7f7d2c759740 1 --2- 172.21.15.29:0/3239112356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 0x55774c933720 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7d18012fb0 tx=0x7f7d18009960 comp rx=0 tx=0).stop 2024-07-27T06:48:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.543+0000 7f7d2c759740 1 -- 172.21.15.29:0/3239112356 shutdown_connections 2024-07-27T06:48:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.543+0000 7f7d2c759740 1 --2- 172.21.15.29:0/3239112356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 0x55774c933720 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.543+0000 7f7d2c759740 1 --2- 172.21.15.29:0/3239112356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c92c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.543+0000 7f7d2c759740 1 --2- 172.21.15.29:0/3239112356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c5479f0 0x55774c946070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.543+0000 7f7d2c759740 1 -- 172.21.15.29:0/3239112356 >> 172.21.15.29:0/3239112356 conn(0x55774c57d460 msgr2=0x55774c92bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.543+0000 7f7d2c759740 1 -- 172.21.15.29:0/3239112356 shutdown_connections 2024-07-27T06:48:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.543+0000 7f7d2c759740 1 -- 172.21.15.29:0/3239112356 wait complete. 2024-07-27T06:48:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.543+0000 7f7d2c759740 1 -- start start 2024-07-27T06:48:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.544+0000 7f7d2c759740 1 Processor -- start 2024-07-27T06:48:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.544+0000 7f7d2c759740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c5479f0 0x55774c94b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.544+0000 7f7d2c759740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c94b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.544+0000 7f7d2c759740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 0x55774c9b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.544+0000 7f7d1f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c5479f0 0x55774c94b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.544+0000 7f7d1f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c5479f0 0x55774c94b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:42072/0 (socket says 172.21.15.29:42072) 2024-07-27T06:48:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.544+0000 7f7d1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 0x55774c9b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 0x55774c9b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49920/0 (socket says 172.21.15.29:49920) 2024-07-27T06:48:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d1f7fe640 1 -- 172.21.15.29:0/149539218 learned_addr learned my addr 172.21.15.29:0/149539218 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d2c759740 1 -- 172.21.15.29:0/149539218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201b6bd0 con 0x55774c589070 2024-07-27T06:48:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d2c759740 1 -- 172.21.15.29:0/149539218 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201b6500 con 0x55774c568390 2024-07-27T06:48:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d2c759740 1 -- 172.21.15.29:0/149539218 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201b3740 con 0x55774c5479f0 2024-07-27T06:48:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d1effd640 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c94b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d2c759740 10 client.0 _ll_register_callbacks cb 0x7f7d10031b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d1ffff640 1 -- 172.21.15.29:0/149539218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c5479f0 msgr2=0x55774c94b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d1ffff640 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c5479f0 0x55774c94b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d1ffff640 1 -- 172.21.15.29:0/149539218 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 msgr2=0x55774c94b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d1ffff640 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c94b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d1ffff640 1 -- 172.21.15.29:0/149539218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55774c9b39c0 con 0x55774c589070 2024-07-27T06:48:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d1f7fe640 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c5479f0 0x55774c94b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.546+0000 7f7d1ffff640 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 0x55774c9b3480 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800afd0 tx=0x7f7d1801e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6af55dde7b6a84 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.546+0000 7f7cfeffd640 10 client.15840 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.545+0000 7f7d2c759740 1 -- 172.21.15.29:0/149539218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d1801eb30 con 0x55774c589070 2024-07-27T06:48:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.546+0000 7f7cfeffd640 1 -- 172.21.15.29:0/149539218 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1802c030 con 0x55774c589070 2024-07-27T06:48:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.546+0000 7f7cfeffd640 1 -- 172.21.15.29:0/149539218 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f7d18004c50 con 0x55774c589070 2024-07-27T06:48:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.546+0000 7f7cfeffd640 1 -- 172.21.15.29:0/149539218 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7d18012d00 con 0x55774c589070 2024-07-27T06:48:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.546+0000 7f7d2c759740 10 client.15840 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:02.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.546+0000 7f7d2c759740 1 -- 172.21.15.29:0/149539218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55774c9b49e0 con 0x55774c589070 2024-07-27T06:48:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.547+0000 7f7cfeffd640 1 -- 172.21.15.29:0/149539218 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7d1801ece0 con 0x55774c589070 2024-07-27T06:48:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.547+0000 7f7d2c759740 10 client.15840 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.547+0000 7f7d2c759740 10 client.15840 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-07-27T06:48:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.547+0000 7f7d2c759740 20 client.15840 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.547+0000 7f7d2c759740 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x55774c51cda0 0x55774c9b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.547+0000 7f7d2c759740 4 client.15840 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:48:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.547+0000 7f7d2c759740 1 -- 172.21.15.29:0/149539218 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f7d1801eb30 con 0x55774c51cda0 2024-07-27T06:48:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.547+0000 7f7d1f7fe640 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x55774c51cda0 0x55774c9b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.548+0000 7f7d1f7fe640 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x55774c51cda0 0x55774c9b6f10 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2101215e1e676453 server_cookie=6ebb1a074068d4d9 in_seq=0 out_seq=0 2024-07-27T06:48:02.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.548+0000 7f7cfeffd640 10 client.15840 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.548+0000 7f7cfeffd640 1 -- 172.21.15.29:0/149539218 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f7d1801eb30 con 0x55774c51cda0 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.548+0000 7f7cfeffd640 10 client.15840 handle_command_reply: tid=0 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.548+0000 7f7d2c759740 1 client.15840 shutdown 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.548+0000 7f7d2c759740 1 -- 172.21.15.29:0/149539218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 msgr2=0x55774c9b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.548+0000 7f7d2c759740 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 0x55774c9b3480 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800afd0 tx=0x7f7d1801e930 comp rx=0 tx=0).stop 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.549+0000 7f7d2c759740 1 -- 172.21.15.29:0/149539218 shutdown_connections 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.549+0000 7f7d2c759740 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x55774c51cda0 0x55774c9b6f10 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.549+0000 7f7cfeffd640 0 client.15840 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.549+0000 7f7d2c759740 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55774c589070 0x55774c9b3480 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.549+0000 7f7d2c759740 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55774c568390 0x55774c94b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.549+0000 7f7d2c759740 1 --2- 172.21.15.29:0/149539218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55774c5479f0 0x55774c94b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.549+0000 7f7d2c759740 1 -- 172.21.15.29:0/149539218 >> 172.21.15.29:0/149539218 conn(0x55774c57d460 msgr2=0x55774c93b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:02.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.549+0000 7f7d2c759740 1 -- 172.21.15.29:0/149539218 shutdown_connections 2024-07-27T06:48:02.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.549+0000 7f7d2c759740 1 -- 172.21.15.29:0/149539218 wait complete. 2024-07-27T06:48:02.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.549+0000 7f7d2c759740 20 client.15840 trim_cache size 0 max 16384 2024-07-27T06:48:02.554 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:48:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.550+0000 7f7d261d5640 1 -- 172.21.15.29:0/1375253801 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7d080b3dc0 msgr2=0x7f7d080b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.550+0000 7f7d261d5640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7d080b3dc0 0x7f7d080b62d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7d18005fd0 tx=0x7f7d18009600 comp rx=0 tx=0).stop 2024-07-27T06:48:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.550+0000 7f7d261d5640 1 -- 172.21.15.29:0/1375253801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d20107dd0 msgr2=0x7f7d20112bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.550+0000 7f7d261d5640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d20107dd0 0x7f7d20112bd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7d14005fd0 tx=0x7f7d1400de70 comp rx=0 tx=0).stop 2024-07-27T06:48:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.551+0000 7f7d261d5640 1 -- 172.21.15.29:0/1375253801 shutdown_connections 2024-07-27T06:48:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.551+0000 7f7d261d5640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7d20108720 0x7f7d20113110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.551+0000 7f7d261d5640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d20107dd0 0x7f7d20112bd0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.551+0000 7f7d261d5640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f7d080b3dc0 0x7f7d080b62d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.551+0000 7f7d261d5640 1 --2- 172.21.15.29:0/1375253801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d20102490 0x7f7d201145e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.551+0000 7f7d261d5640 1 -- 172.21.15.29:0/1375253801 >> 172.21.15.29:0/1375253801 conn(0x7f7d20079160 msgr2=0x7f7d200ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:02.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.551+0000 7f7d261d5640 1 -- 172.21.15.29:0/1375253801 shutdown_connections 2024-07-27T06:48:02.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:02.551+0000 7f7d261d5640 1 -- 172.21.15.29:0/1375253801 wait complete. 2024-07-27T06:48:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:02 smithi077 ceph-mon[37153]: pgmap v428: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-27T06:48:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:02 smithi077 ceph-mon[37153]: mgrmap e25: x(active, since 11m), standbys: y, z 2024-07-27T06:48:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:02 smithi047 ceph-mon[36835]: pgmap v428: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-27T06:48:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:02 smithi047 ceph-mon[36835]: mgrmap e25: x(active, since 11m), standbys: y, z 2024-07-27T06:48:03.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:02 smithi029 ceph-mon[32610]: pgmap v428: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-27T06:48:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:02 smithi029 ceph-mon[32610]: mgrmap e25: x(active, since 11m), standbys: y, z 2024-07-27T06:48:03.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-27T06:48:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:04 smithi077 ceph-mon[37153]: pgmap v429: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 25 op/s 2024-07-27T06:48:05.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:04 smithi029 ceph-mon[32610]: pgmap v429: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 25 op/s 2024-07-27T06:48:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:04 smithi047 ceph-mon[36835]: pgmap v429: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 25 op/s 2024-07-27T06:48:05.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.850+0000 7f81d6d0e640 1 -- 172.21.15.29:0/1701213592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d01042b0 msgr2=0x7f81d01046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:05.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.850+0000 7f81d6d0e640 1 --2- 172.21.15.29:0/1701213592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d01042b0 0x7f81d01046c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009900 tx=0x7f81c402a410 comp rx=0 tx=0).stop 2024-07-27T06:48:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.850+0000 7f81d6d0e640 1 -- 172.21.15.29:0/1701213592 shutdown_connections 2024-07-27T06:48:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.850+0000 7f81d6d0e640 1 --2- 172.21.15.29:0/1701213592 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81d010d860 0x7f81d010fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.850+0000 7f81d6d0e640 1 --2- 172.21.15.29:0/1701213592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81d0104ce0 0x7f81d010d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.850+0000 7f81d6d0e640 1 --2- 172.21.15.29:0/1701213592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d01042b0 0x7f81d01046c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.850+0000 7f81d6d0e640 1 -- 172.21.15.29:0/1701213592 >> 172.21.15.29:0/1701213592 conn(0x7f81d0079160 msgr2=0x7f81d0079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.850+0000 7f81d6d0e640 1 -- 172.21.15.29:0/1701213592 shutdown_connections 2024-07-27T06:48:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.850+0000 7f81d6d0e640 1 -- 172.21.15.29:0/1701213592 wait complete. 2024-07-27T06:48:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.851+0000 7f81d6d0e640 1 Processor -- start 2024-07-27T06:48:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.851+0000 7f81d6d0e640 1 -- start start 2024-07-27T06:48:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.852+0000 7f81d6d0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81d01042b0 0x7f81d0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.852+0000 7f81d6d0e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d0104ce0 0x7f81d0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.852+0000 7f81d6d0e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81d010d860 0x7f81d019d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.852+0000 7f81d6d0e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f81d01112e0 con 0x7f81d0104ce0 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.852+0000 7f81d6d0e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f81d0111160 con 0x7f81d010d860 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.852+0000 7f81d6d0e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f81d0111460 con 0x7f81d01042b0 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.852+0000 7f81cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d0104ce0 0x7f81d0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.852+0000 7f81cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d0104ce0 0x7f81d0198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49930/0 (socket says 172.21.15.29:49930) 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.852+0000 7f81cffff640 1 -- 172.21.15.29:0/83566836 learned_addr learned my addr 172.21.15.29:0/83566836 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.852+0000 7f81d4aaa640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81d01042b0 0x7f81d0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.852+0000 7f81d52ab640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81d010d860 0x7f81d019d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.853+0000 7f81d4aaa640 1 -- 172.21.15.29:0/83566836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81d010d860 msgr2=0x7f81d019d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.853+0000 7f81d4aaa640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81d010d860 0x7f81d019d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.853+0000 7f81d4aaa640 1 -- 172.21.15.29:0/83566836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d0104ce0 msgr2=0x7f81d0198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.853+0000 7f81d4aaa640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d0104ce0 0x7f81d0198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.853+0000 7f81d4aaa640 1 -- 172.21.15.29:0/83566836 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d019da30 con 0x7f81d01042b0 2024-07-27T06:48:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.853+0000 7f81d4aaa640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81d01042b0 0x7f81d0197ac0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009850 tx=0x7f81c4030650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.853+0000 7f81cdffb640 1 -- 172.21.15.29:0/83566836 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c403f070 con 0x7f81d01042b0 2024-07-27T06:48:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.854+0000 7f81cdffb640 1 -- 172.21.15.29:0/83566836 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81c4004460 con 0x7f81d01042b0 2024-07-27T06:48:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.854+0000 7f81cdffb640 1 -- 172.21.15.29:0/83566836 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c4035030 con 0x7f81d01042b0 2024-07-27T06:48:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.854+0000 7f81d6d0e640 1 -- 172.21.15.29:0/83566836 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d019dcc0 con 0x7f81d01042b0 2024-07-27T06:48:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.854+0000 7f81d6d0e640 1 -- 172.21.15.29:0/83566836 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d019e060 con 0x7f81d01042b0 2024-07-27T06:48:05.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.858+0000 7f81cdffb640 1 -- 172.21.15.29:0/83566836 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f81c4042070 con 0x7f81d01042b0 2024-07-27T06:48:05.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.858+0000 7f81dd292740 1 Processor -- start 2024-07-27T06:48:05.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.858+0000 7f81dd292740 1 -- start start 2024-07-27T06:48:05.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81dd292740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987ce070 0x55bb98b73570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81dd292740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987a51b0 0x55bb98b73ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81dd292740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b73ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81dd292740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bb98ae9bb0 con 0x55bb987ad390 2024-07-27T06:48:05.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81dd292740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55bb98ad90b0 con 0x55bb987ce070 2024-07-27T06:48:05.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81dd292740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bb98add0f0 con 0x55bb987a51b0 2024-07-27T06:48:05.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b73ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b73ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49942/0 (socket says 172.21.15.29:49942) 2024-07-27T06:48:05.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81cffff640 1 -- 172.21.15.29:0/43048514 learned_addr learned my addr 172.21.15.29:0/43048514 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:05.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81d52ab640 1 --2- 172.21.15.29:0/43048514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987ce070 0x55bb98b73570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81d4aaa640 1 --2- 172.21.15.29:0/43048514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987a51b0 0x55bb98b73ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81cffff640 1 -- 172.21.15.29:0/43048514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987a51b0 msgr2=0x55bb98b73ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:05.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81cffff640 1 --2- 172.21.15.29:0/43048514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987a51b0 0x55bb98b73ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81cffff640 1 -- 172.21.15.29:0/43048514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987ce070 msgr2=0x55bb98b73570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:05.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81cffff640 1 --2- 172.21.15.29:0/43048514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987ce070 0x55bb98b73570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.859+0000 7f81cffff640 1 -- 172.21.15.29:0/43048514 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb98a7fb90 con 0x55bb987ad390 2024-07-27T06:48:05.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.860+0000 7f81cffff640 1 --2- 172.21.15.29:0/43048514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b73ff0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009d00 tx=0x7f81bc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c557edbc8a1588e7 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.860+0000 7f81baffd640 1 -- 172.21.15.29:0/43048514 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc005220 con 0x55bb987ad390 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.860+0000 7f81baffd640 1 -- 172.21.15.29:0/43048514 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81bc004c50 con 0x55bb987ad390 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.860+0000 7f81baffd640 1 -- 172.21.15.29:0/43048514 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc019810 con 0x55bb987ad390 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.860+0000 7f81cdffb640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f81c00bc5d0 0x7f81c00beae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.860+0000 7f81d52ab640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f81c00bc5d0 0x7f81c00beae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.861+0000 7f81cdffb640 1 -- 172.21.15.29:0/83566836 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f81c41046d0 con 0x7f81d01042b0 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.861+0000 7f81dd292740 1 -- 172.21.15.29:0/43048514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 msgr2=0x55bb98b73ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.861+0000 7f81d52ab640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f81c00bc5d0 0x7f81c00beae0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f81c8005fd0 tx=0x7f81c8011040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.861+0000 7f81dd292740 1 --2- 172.21.15.29:0/43048514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b73ff0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009d00 tx=0x7f81bc011810 comp rx=0 tx=0).stop 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.861+0000 7f81dd292740 1 -- 172.21.15.29:0/43048514 shutdown_connections 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.861+0000 7f81dd292740 1 --2- 172.21.15.29:0/43048514 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b73ff0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.861+0000 7f81dd292740 1 --2- 172.21.15.29:0/43048514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987a51b0 0x55bb98b73ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.861+0000 7f81dd292740 1 --2- 172.21.15.29:0/43048514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987ce070 0x55bb98b73570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.861+0000 7f81dd292740 1 -- 172.21.15.29:0/43048514 >> 172.21.15.29:0/43048514 conn(0x55bb987c2460 msgr2=0x55bb98b71460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:05.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.861+0000 7f81dd292740 1 -- 172.21.15.29:0/43048514 shutdown_connections 2024-07-27T06:48:05.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.862+0000 7f81dd292740 1 -- 172.21.15.29:0/43048514 wait complete. 2024-07-27T06:48:05.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.862+0000 7f81dd292740 1 -- start start 2024-07-27T06:48:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.862+0000 7f81dd292740 1 Processor -- start 2024-07-27T06:48:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.863+0000 7f81dd292740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987a51b0 0x55bb98b7f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.863+0000 7f81dd292740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b7f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.863+0000 7f81cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b7f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.863+0000 7f81d4aaa640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987a51b0 0x55bb98b7f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.863+0000 7f81d4aaa640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987a51b0 0x55bb98b7f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.29:40394/0 (socket says 172.21.15.29:40394) 2024-07-27T06:48:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.863+0000 7f81cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b7f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49958/0 (socket says 172.21.15.29:49958) 2024-07-27T06:48:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.863+0000 7f81d4aaa640 1 -- 172.21.15.29:0/3152759851 learned_addr learned my addr 172.21.15.29:0/3152759851 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.863+0000 7f81dd292740 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987ce070 0x55bb98b9c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.863+0000 7f81dd292740 1 -- 172.21.15.29:0/3152759851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bb9893c700 con 0x55bb987ad390 2024-07-27T06:48:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.863+0000 7f81dd292740 1 -- 172.21.15.29:0/3152759851 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55bb98a7b9e0 con 0x55bb987a51b0 2024-07-27T06:48:05.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.863+0000 7f81dd292740 1 -- 172.21.15.29:0/3152759851 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bb98b81d50 con 0x55bb987ce070 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.864+0000 7f81d52ab640 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987ce070 0x55bb98b9c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.864+0000 7f81d4aaa640 1 -- 172.21.15.29:0/3152759851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987ce070 msgr2=0x55bb98b9c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.864+0000 7f81d4aaa640 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987ce070 0x55bb98b9c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.864+0000 7f81d4aaa640 1 -- 172.21.15.29:0/3152759851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 msgr2=0x55bb98b7f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.864+0000 7f81d4aaa640 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b7f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.864+0000 7f81d4aaa640 1 -- 172.21.15.29:0/3152759851 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb98839650 con 0x55bb987a51b0 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.864+0000 7f81cffff640 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b7f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.864+0000 7f81d4aaa640 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987a51b0 0x55bb98b7f0f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f81c402a930 tx=0x7f81c40c7320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7087eb5014b5b0cc server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.864+0000 7f81dd292740 10 client.0 _ll_register_callbacks cb 0x55bb98a964f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.864+0000 7f81dd292740 1 -- 172.21.15.29:0/3152759851 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81bc004c50 con 0x55bb987a51b0 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.866+0000 7f81affff640 10 client.25415 ms_handle_connect on v2:172.21.15.47:3300/0 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.866+0000 7f81affff640 1 -- 172.21.15.29:0/3152759851 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c4049070 con 0x55bb987a51b0 2024-07-27T06:48:05.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.866+0000 7f81affff640 1 -- 172.21.15.29:0/3152759851 <== mon.2 v2:172.21.15.47:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f81c40cb600 con 0x55bb987a51b0 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.866+0000 7f81affff640 1 -- 172.21.15.29:0/3152759851 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f81c4030e10 con 0x55bb987a51b0 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.866+0000 7f81dd292740 10 client.25415 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.866+0000 7f81dd292740 1 -- 172.21.15.29:0/3152759851 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55bb98baf8e0 con 0x55bb987a51b0 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.866+0000 7f81affff640 1 -- 172.21.15.29:0/3152759851 <== mon.2 v2:172.21.15.47:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f81c40d3f10 con 0x55bb987a51b0 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.866+0000 7f81dd292740 10 client.25415 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.867+0000 7f81dd292740 10 client.25415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.867+0000 7f81dd292740 20 client.25415 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.867+0000 7f81dd292740 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb9878c9f0 0x55bb98bb2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.867+0000 7f81dd292740 4 client.25415 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.867+0000 7f81dd292740 1 -- 172.21.15.29:0/3152759851 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f81bc004c50 con 0x55bb9878c9f0 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.867+0000 7f81cffff640 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb9878c9f0 0x55bb98bb2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.867+0000 7f81affff640 10 client.25415 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.867+0000 7f81cffff640 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb9878c9f0 0x55bb98bb2f70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4e75396aaa6f3c1 server_cookie=b73958ea4f167c6b in_seq=0 out_seq=0 2024-07-27T06:48:05.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.868+0000 7f81affff640 1 -- 172.21.15.29:0/3152759851 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f81bc004c50 con 0x55bb9878c9f0 2024-07-27T06:48:05.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.868+0000 7f81affff640 10 client.25415 handle_command_reply: tid=0 2024-07-27T06:48:05.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.868+0000 7f81dd292740 1 client.25415 shutdown 2024-07-27T06:48:05.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.868+0000 7f81dd292740 1 -- 172.21.15.29:0/3152759851 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987a51b0 msgr2=0x55bb98b7f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:05.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.868+0000 7f81dd292740 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987a51b0 0x55bb98b7f0f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f81c402a930 tx=0x7f81c40c7320 comp rx=0 tx=0).stop 2024-07-27T06:48:05.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.869+0000 7f81dd292740 1 -- 172.21.15.29:0/3152759851 shutdown_connections 2024-07-27T06:48:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.869+0000 7f81dd292740 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb9878c9f0 0x55bb98bb2f70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.869+0000 7f81dd292740 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987ce070 0x55bb98b9c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.869+0000 7f81dd292740 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b7f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.869+0000 7f81affff640 0 client.25415 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.869+0000 7f81dd292740 1 --2- 172.21.15.29:0/3152759851 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987a51b0 0x55bb98b7f0f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.869+0000 7f81dd292740 1 -- 172.21.15.29:0/3152759851 >> 172.21.15.29:0/3152759851 conn(0x55bb987c2460 msgr2=0x55bb98b6fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.869+0000 7f81dd292740 1 -- 172.21.15.29:0/3152759851 shutdown_connections 2024-07-27T06:48:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.869+0000 7f81dd292740 1 -- 172.21.15.29:0/3152759851 wait complete. 2024-07-27T06:48:05.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.869+0000 7f81dd292740 20 client.25415 trim_cache size 0 max 16384 2024-07-27T06:48:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.895+0000 7f81dd292740 1 Processor -- start 2024-07-27T06:48:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.895+0000 7f81dd292740 1 -- start start 2024-07-27T06:48:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81dd292740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb9878c9f0 0x55bb98b7f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81dd292740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987a51b0 0x55bb98b702b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81dd292740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b9bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81dd292740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f81d01a47b0 con 0x55bb987ad390 2024-07-27T06:48:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81dd292740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f81d01a62f0 con 0x55bb9878c9f0 2024-07-27T06:48:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81dd292740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f81d01a4e80 con 0x55bb987a51b0 2024-07-27T06:48:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81d52ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b9bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81d4aaa640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb9878c9f0 0x55bb98b7f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81cffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987a51b0 0x55bb98b702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81d52ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b9bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49970/0 (socket says 172.21.15.29:49970) 2024-07-27T06:48:05.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81d52ab640 1 -- 172.21.15.29:0/1614057265 learned_addr learned my addr 172.21.15.29:0/1614057265 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:05.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81cffff640 1 -- 172.21.15.29:0/1614057265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb9878c9f0 msgr2=0x55bb98b7f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:05.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81cffff640 1 --2- 172.21.15.29:0/1614057265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb9878c9f0 0x55bb98b7f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.896+0000 7f81cffff640 1 -- 172.21.15.29:0/1614057265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 msgr2=0x55bb98b9bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:05.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.897+0000 7f81cffff640 1 --2- 172.21.15.29:0/1614057265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b9bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.897+0000 7f81cffff640 1 -- 172.21.15.29:0/1614057265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb98b816e0 con 0x55bb987a51b0 2024-07-27T06:48:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.897+0000 7f81d52ab640 1 --2- 172.21.15.29:0/1614057265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b9bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.897+0000 7f81cffff640 1 --2- 172.21.15.29:0/1614057265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987a51b0 0x55bb98b702b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81bc01b040 tx=0x7f81bc009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95262c0625923044 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.897+0000 7f81ba7fc640 1 -- 172.21.15.29:0/1614057265 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc004450 con 0x55bb987a51b0 2024-07-27T06:48:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.897+0000 7f81ba7fc640 1 -- 172.21.15.29:0/1614057265 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81bc003aa0 con 0x55bb987a51b0 2024-07-27T06:48:05.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.897+0000 7f81ba7fc640 1 -- 172.21.15.29:0/1614057265 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc02b6d0 con 0x55bb987a51b0 2024-07-27T06:48:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.898+0000 7f81dd292740 1 -- 172.21.15.29:0/1614057265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987a51b0 msgr2=0x55bb98b702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.898+0000 7f81dd292740 1 --2- 172.21.15.29:0/1614057265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987a51b0 0x55bb98b702b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81bc01b040 tx=0x7f81bc009310 comp rx=0 tx=0).stop 2024-07-27T06:48:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.899+0000 7f81dd292740 1 -- 172.21.15.29:0/1614057265 shutdown_connections 2024-07-27T06:48:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.899+0000 7f81dd292740 1 --2- 172.21.15.29:0/1614057265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987ad390 0x55bb98b9bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.899+0000 7f81dd292740 1 --2- 172.21.15.29:0/1614057265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb987a51b0 0x55bb98b702b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.899+0000 7f81dd292740 1 --2- 172.21.15.29:0/1614057265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb9878c9f0 0x55bb98b7f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.899+0000 7f81dd292740 1 -- 172.21.15.29:0/1614057265 >> 172.21.15.29:0/1614057265 conn(0x55bb987c2460 msgr2=0x55bb98b7f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:05.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.899+0000 7f81dd292740 1 -- 172.21.15.29:0/1614057265 shutdown_connections 2024-07-27T06:48:05.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.899+0000 7f81dd292740 1 -- 172.21.15.29:0/1614057265 wait complete. 2024-07-27T06:48:05.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.899+0000 7f81dd292740 1 -- start start 2024-07-27T06:48:05.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.899+0000 7f81dd292740 1 Processor -- start 2024-07-27T06:48:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.900+0000 7f81dd292740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb9878c9f0 0x55bb98b987f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.900+0000 7f81dd292740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987a51b0 0x55bb98b93cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.900+0000 7f81cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987a51b0 0x55bb98b93cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.900+0000 7f81cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987a51b0 0x55bb98b93cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49976/0 (socket says 172.21.15.29:49976) 2024-07-27T06:48:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.900+0000 7f81dd292740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987ad390 0x55bb98b94340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.900+0000 7f81d4aaa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb9878c9f0 0x55bb98b987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.900+0000 7f81cffff640 1 -- 172.21.15.29:0/12785701 learned_addr learned my addr 172.21.15.29:0/12785701 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.901+0000 7f81dd292740 1 -- 172.21.15.29:0/12785701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f81d01a5c20 con 0x55bb987a51b0 2024-07-27T06:48:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.901+0000 7f81dd292740 1 -- 172.21.15.29:0/12785701 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f81d01a5550 con 0x55bb987ad390 2024-07-27T06:48:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.901+0000 7f81dd292740 1 -- 172.21.15.29:0/12785701 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f81d00732a0 con 0x55bb9878c9f0 2024-07-27T06:48:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.901+0000 7f81d52ab640 1 --2- 172.21.15.29:0/12785701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987ad390 0x55bb98b94340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.901+0000 7f81cffff640 1 -- 172.21.15.29:0/12785701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb9878c9f0 msgr2=0x55bb98b987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.901+0000 7f81cffff640 1 --2- 172.21.15.29:0/12785701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb9878c9f0 0x55bb98b987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.901+0000 7f81cffff640 1 -- 172.21.15.29:0/12785701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987ad390 msgr2=0x55bb98b94340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.901+0000 7f81cffff640 1 --2- 172.21.15.29:0/12785701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987ad390 0x55bb98b94340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.901+0000 7f81cffff640 1 -- 172.21.15.29:0/12785701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb98b94ab0 con 0x55bb987a51b0 2024-07-27T06:48:05.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.901+0000 7f81dd292740 10 client.0 _ll_register_callbacks cb 0x7f81c4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.901+0000 7f81cffff640 1 --2- 172.21.15.29:0/12785701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987a51b0 0x55bb98b93cf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009250 tx=0x7f81bc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a33d4c4d8dd0ef2 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.902+0000 7f81affff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.902+0000 7f81affff640 1 -- 172.21.15.29:0/12785701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc003c50 con 0x55bb987a51b0 2024-07-27T06:48:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.902+0000 7f81affff640 1 -- 172.21.15.29:0/12785701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f81bc003df0 con 0x55bb987a51b0 2024-07-27T06:48:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.902+0000 7f81dd292740 1 -- 172.21.15.29:0/12785701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81bc003aa0 con 0x55bb987a51b0 2024-07-27T06:48:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.902+0000 7f81affff640 1 -- 172.21.15.29:0/12785701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f81bc003ff0 con 0x55bb987a51b0 2024-07-27T06:48:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.902+0000 7f81dd292740 10 client.15858 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:05.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.902+0000 7f81dd292740 1 -- 172.21.15.29:0/12785701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55bb98c1b3e0 con 0x55bb987a51b0 2024-07-27T06:48:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.903+0000 7f81affff640 1 -- 172.21.15.29:0/12785701 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f81bc02b2e0 con 0x55bb987a51b0 2024-07-27T06:48:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.903+0000 7f81dd292740 10 client.15858 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.903+0000 7f81dd292740 10 client.15858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.903+0000 7f81dd292740 20 client.15858 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.903+0000 7f81dd292740 1 --2- 172.21.15.29:0/12785701 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb987ce070 0x55bb98c1d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:05.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.903+0000 7f81dd292740 4 client.15858 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.903+0000 7f81dd292740 1 -- 172.21.15.29:0/12785701 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f81bc003aa0 con 0x55bb987ce070 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.903+0000 7f81d4aaa640 1 --2- 172.21.15.29:0/12785701 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb987ce070 0x55bb98c1d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.904+0000 7f81affff640 10 client.15858 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.904+0000 7f81d4aaa640 1 --2- 172.21.15.29:0/12785701 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb987ce070 0x55bb98c1d7d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5377afc85fc2daf server_cookie=644c0e44694a9ac7 in_seq=0 out_seq=0 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.904+0000 7f81affff640 1 -- 172.21.15.29:0/12785701 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f81bc003aa0 con 0x55bb987ce070 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.904+0000 7f81affff640 10 client.15858 handle_command_reply: tid=0 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.904+0000 7f81dd292740 1 client.15858 shutdown 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.904+0000 7f81dd292740 1 -- 172.21.15.29:0/12785701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987a51b0 msgr2=0x55bb98b93cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.904+0000 7f81dd292740 1 --2- 172.21.15.29:0/12785701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987a51b0 0x55bb98b93cf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009250 tx=0x7f81bc011810 comp rx=0 tx=0).stop 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.905+0000 7f81dd292740 1 -- 172.21.15.29:0/12785701 shutdown_connections 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.905+0000 7f81dd292740 1 --2- 172.21.15.29:0/12785701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb987ad390 0x55bb98b94340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.905+0000 7f81dd292740 1 --2- 172.21.15.29:0/12785701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb987a51b0 0x55bb98b93cf0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.905+0000 7f81dd292740 1 --2- 172.21.15.29:0/12785701 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb987ce070 0x55bb98c1d7d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.905+0000 7f81dd292740 1 --2- 172.21.15.29:0/12785701 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb9878c9f0 0x55bb98b987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.905+0000 7f81dd292740 1 -- 172.21.15.29:0/12785701 >> 172.21.15.29:0/12785701 conn(0x55bb987c2460 msgr2=0x55bb98b80250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:05.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.905+0000 7f81affff640 0 client.15858 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:05.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.905+0000 7f81dd292740 1 -- 172.21.15.29:0/12785701 shutdown_connections 2024-07-27T06:48:05.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.905+0000 7f81dd292740 1 -- 172.21.15.29:0/12785701 wait complete. 2024-07-27T06:48:05.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.905+0000 7f81dd292740 20 client.15858 trim_cache size 0 max 16384 2024-07-27T06:48:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:48:05.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.906+0000 7f81d6d0e640 1 -- 172.21.15.29:0/83566836 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f81c00bc5d0 msgr2=0x7f81c00beae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:05.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.906+0000 7f81d6d0e640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f81c00bc5d0 0x7f81c00beae0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f81c8005fd0 tx=0x7f81c8011040 comp rx=0 tx=0).stop 2024-07-27T06:48:05.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.906+0000 7f81d6d0e640 1 -- 172.21.15.29:0/83566836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81d01042b0 msgr2=0x7f81d0197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:05.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.906+0000 7f81d6d0e640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81d01042b0 0x7f81d0197ac0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009850 tx=0x7f81c4030650 comp rx=0 tx=0).stop 2024-07-27T06:48:05.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.906+0000 7f81d6d0e640 1 -- 172.21.15.29:0/83566836 shutdown_connections 2024-07-27T06:48:05.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.906+0000 7f81d6d0e640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f81d010d860 0x7f81d019d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.906+0000 7f81d6d0e640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81d0104ce0 0x7f81d0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.906+0000 7f81d6d0e640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f81c00bc5d0 0x7f81c00beae0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.906+0000 7f81d6d0e640 1 --2- 172.21.15.29:0/83566836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81d01042b0 0x7f81d0197ac0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:05.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.906+0000 7f81d6d0e640 1 -- 172.21.15.29:0/83566836 >> 172.21.15.29:0/83566836 conn(0x7f81d0079160 msgr2=0x7f81d0105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:05.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.906+0000 7f81d6d0e640 1 -- 172.21.15.29:0/83566836 shutdown_connections 2024-07-27T06:48:05.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:05.906+0000 7f81d6d0e640 1 -- 172.21.15.29:0/83566836 wait complete. 2024-07-27T06:48:06.733 INFO:tasks.quiescer.fs.[cephfs]:Total duration (45.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-07-27T06:48:06.733 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.55 2024-07-27T06:48:06.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:48:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:06 smithi077 ceph-mon[37153]: pgmap v430: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-07-27T06:48:07.133 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:06 smithi047 ceph-mon[36835]: pgmap v430: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-07-27T06:48:07.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:06 smithi029 ceph-mon[32610]: pgmap v430: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-07-27T06:48:07.776 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-27T06:48:07.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-27T06:48:08.650 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:08.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:08 smithi029 ceph-mon[32610]: pgmap v431: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-07-27T06:48:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:08 smithi077 ceph-mon[37153]: pgmap v431: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-07-27T06:48:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:08 smithi047 ceph-mon[36835]: pgmap v431: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-07-27T06:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.384+0000 7f692e606640 1 -- 172.21.15.29:0/140948393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69281075e0 msgr2=0x7f69281079f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.384+0000 7f692e606640 1 --2- 172.21.15.29:0/140948393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69281075e0 0x7f69281079f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f691000aa10 tx=0x7f691002b720 comp rx=0 tx=0).stop 2024-07-27T06:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.385+0000 7f692e606640 1 -- 172.21.15.29:0/140948393 shutdown_connections 2024-07-27T06:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.385+0000 7f692e606640 1 --2- 172.21.15.29:0/140948393 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6928107f30 0x7f692810e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.385+0000 7f692e606640 1 --2- 172.21.15.29:0/140948393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69281075e0 0x7f69281079f0 secure :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f691000aa10 tx=0x7f691002b720 comp rx=0 tx=0).stop 2024-07-27T06:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.385+0000 7f692e606640 1 --2- 172.21.15.29:0/140948393 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6928101bb0 0x7f6928101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.385+0000 7f692e606640 1 -- 172.21.15.29:0/140948393 >> 172.21.15.29:0/140948393 conn(0x7f6928079880 msgr2=0x7f6928079c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:09.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.385+0000 7f692e606640 1 -- 172.21.15.29:0/140948393 shutdown_connections 2024-07-27T06:48:09.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.385+0000 7f692e606640 1 -- 172.21.15.29:0/140948393 wait complete. 2024-07-27T06:48:09.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.386+0000 7f692e606640 1 Processor -- start 2024-07-27T06:48:09.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.386+0000 7f692e606640 1 -- start start 2024-07-27T06:48:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.387+0000 7f692e606640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6928101bb0 0x7f6928197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.387+0000 7f692e606640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6928107f30 0x7f69281984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.387+0000 7f692e606640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f692819d690 0x7f692819daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.387+0000 7f692e606640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f69281113c0 con 0x7f692819d690 2024-07-27T06:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.387+0000 7f6927fff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6928101bb0 0x7f6928197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.387+0000 7f692cba3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f692819d690 0x7f692819daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.387+0000 7f6927fff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6928101bb0 0x7f6928197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.29:55964/0 (socket says 172.21.15.29:55964) 2024-07-27T06:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.387+0000 7f692cba3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f692819d690 0x7f692819daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51330/0 (socket says 172.21.15.29:51330) 2024-07-27T06:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.387+0000 7f6927fff640 1 -- 172.21.15.29:0/3423043032 learned_addr learned my addr 172.21.15.29:0/3423043032 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.387+0000 7f692e606640 1 -- 172.21.15.29:0/3423043032 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f6928111240 con 0x7f6928101bb0 2024-07-27T06:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.387+0000 7f691ffff640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6928107f30 0x7f69281984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.387+0000 7f692e606640 1 -- 172.21.15.29:0/3423043032 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6928111540 con 0x7f6928107f30 2024-07-27T06:48:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.388+0000 7f6927fff640 1 -- 172.21.15.29:0/3423043032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6928107f30 msgr2=0x7f69281984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.388+0000 7f6927fff640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6928107f30 0x7f69281984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.388+0000 7f6927fff640 1 -- 172.21.15.29:0/3423043032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f692819d690 msgr2=0x7f692819daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.388+0000 7f6927fff640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f692819d690 0x7f692819daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.388+0000 7f6927fff640 1 -- 172.21.15.29:0/3423043032 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692819e270 con 0x7f6928101bb0 2024-07-27T06:48:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.388+0000 7f6927fff640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6928101bb0 0x7f6928197f20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f691800eb00 tx=0x7f691800ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.389+0000 7f6925ffb640 1 -- 172.21.15.29:0/3423043032 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691800bb70 con 0x7f6928101bb0 2024-07-27T06:48:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.389+0000 7f6925ffb640 1 -- 172.21.15.29:0/3423043032 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f691800bd10 con 0x7f6928101bb0 2024-07-27T06:48:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.389+0000 7f692e606640 1 -- 172.21.15.29:0/3423043032 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69281a4810 con 0x7f6928101bb0 2024-07-27T06:48:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.389+0000 7f692e606640 1 -- 172.21.15.29:0/3423043032 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69281a4c10 con 0x7f6928101bb0 2024-07-27T06:48:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.389+0000 7f6925ffb640 1 -- 172.21.15.29:0/3423043032 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6918017410 con 0x7f6928101bb0 2024-07-27T06:48:09.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f69355fc740 1 Processor -- start 2024-07-27T06:48:09.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f69355fc740 1 -- start start 2024-07-27T06:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f6925ffb640 1 -- 172.21.15.29:0/3423043032 <== mon.2 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f69180175b0 con 0x7f6928101bb0 2024-07-27T06:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f69355fc740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93fa1b0 0x5587b979cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f69355fc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d12f0 0x5587b979d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f69355fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93d94d0 0x5587b979d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f69355fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5587b97123b0 con 0x5587b93d94d0 2024-07-27T06:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f69355fc740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5587b97018b0 con 0x5587b93fa1b0 2024-07-27T06:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f69355fc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5587b97058f0 con 0x5587b93d12f0 2024-07-27T06:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f691ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93d94d0 0x5587b979d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f691ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93d94d0 0x5587b979d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51344/0 (socket says 172.21.15.29:51344) 2024-07-27T06:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f691ffff640 1 -- 172.21.15.29:0/649711672 learned_addr learned my addr 172.21.15.29:0/649711672 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.393+0000 7f692cba3640 1 --2- 172.21.15.29:0/649711672 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93fa1b0 0x5587b979cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.395 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.394+0000 7f6927fff640 1 --2- 172.21.15.29:0/649711672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d12f0 0x5587b979d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.394+0000 7f691ffff640 1 -- 172.21.15.29:0/649711672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d12f0 msgr2=0x5587b979d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.394+0000 7f691ffff640 1 --2- 172.21.15.29:0/649711672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d12f0 0x5587b979d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.394+0000 7f691ffff640 1 -- 172.21.15.29:0/649711672 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93fa1b0 msgr2=0x5587b979cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.394+0000 7f691ffff640 1 --2- 172.21.15.29:0/649711672 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93fa1b0 0x5587b979cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.394+0000 7f691ffff640 1 -- 172.21.15.29:0/649711672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587b96a89a0 con 0x5587b93d94d0 2024-07-27T06:48:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.394+0000 7f691ffff640 1 --2- 172.21.15.29:0/649711672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93d94d0 0x5587b979d760 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f691002bd10 tx=0x7f6910004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e38887a6a2ee608 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.395+0000 7f691e7fc640 1 -- 172.21.15.29:0/649711672 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910041020 con 0x5587b93d94d0 2024-07-27T06:48:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.395+0000 7f691e7fc640 1 -- 172.21.15.29:0/649711672 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6910004430 con 0x5587b93d94d0 2024-07-27T06:48:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.395+0000 7f691e7fc640 1 -- 172.21.15.29:0/649711672 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910036040 con 0x5587b93d94d0 2024-07-27T06:48:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.396+0000 7f69355fc740 1 -- 172.21.15.29:0/649711672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93d94d0 msgr2=0x5587b979d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.396+0000 7f69355fc740 1 --2- 172.21.15.29:0/649711672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93d94d0 0x5587b979d760 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f691002bd10 tx=0x7f6910004040 comp rx=0 tx=0).stop 2024-07-27T06:48:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.396+0000 7f69355fc740 1 -- 172.21.15.29:0/649711672 shutdown_connections 2024-07-27T06:48:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.396+0000 7f6925ffb640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f69140b3e40 0x7f69140b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.397+0000 7f69355fc740 1 --2- 172.21.15.29:0/649711672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93d94d0 0x5587b979d760 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.397+0000 7f692cba3640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f69140b3e40 0x7f69140b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.397+0000 7f6925ffb640 1 -- 172.21.15.29:0/3423043032 <== mon.2 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f69180e3570 con 0x7f6928101bb0 2024-07-27T06:48:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.397+0000 7f69355fc740 1 --2- 172.21.15.29:0/649711672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d12f0 0x5587b979d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.397+0000 7f69355fc740 1 --2- 172.21.15.29:0/649711672 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93fa1b0 0x5587b979cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.397+0000 7f69355fc740 1 -- 172.21.15.29:0/649711672 >> 172.21.15.29:0/649711672 conn(0x5587b93ee300 msgr2=0x5587b979abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.397+0000 7f692cba3640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f69140b3e40 0x7f69140b6350 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6920005fd0 tx=0x7f6920009500 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.397+0000 7f69355fc740 1 -- 172.21.15.29:0/649711672 shutdown_connections 2024-07-27T06:48:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.397+0000 7f69355fc740 1 -- 172.21.15.29:0/649711672 wait complete. 2024-07-27T06:48:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.398+0000 7f69355fc740 1 -- start start 2024-07-27T06:48:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.398+0000 7f69355fc740 1 Processor -- start 2024-07-27T06:48:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f69355fc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d12f0 0x5587b97a8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f69355fc740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d94d0 0x5587b97b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f69355fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93fa1b0 0x5587b97a9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f6927fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d12f0 0x5587b97a8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f6927fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d12f0 0x5587b97a8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:57980/0 (socket says 172.21.15.29:57980) 2024-07-27T06:48:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f692cba3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93fa1b0 0x5587b97a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f692cba3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93fa1b0 0x5587b97a9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51354/0 (socket says 172.21.15.29:51354) 2024-07-27T06:48:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f691ffff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d94d0 0x5587b97b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f691ffff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d94d0 0x5587b97b35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.29:55980/0 (socket says 172.21.15.29:55980) 2024-07-27T06:48:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f69355fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5587b963bab0 con 0x5587b93fa1b0 2024-07-27T06:48:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f69355fc740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5587b9689cf0 con 0x5587b93d94d0 2024-07-27T06:48:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f69355fc740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5587b97a73c0 con 0x5587b93d12f0 2024-07-27T06:48:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.399+0000 7f6927fff640 1 -- 172.21.15.29:0/1073373385 learned_addr learned my addr 172.21.15.29:0/1073373385 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:09.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.400+0000 7f6927fff640 1 -- 172.21.15.29:0/1073373385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d94d0 msgr2=0x5587b97b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:09.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.400+0000 7f69355fc740 10 client.0 _ll_register_callbacks cb 0x5587b96fb9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:09.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.400+0000 7f6927fff640 1 --2- 172.21.15.29:0/1073373385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d94d0 0x5587b97b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.403 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.400+0000 7f6927fff640 1 -- 172.21.15.29:0/1073373385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93fa1b0 msgr2=0x5587b97a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.400+0000 7f6927fff640 1 --2- 172.21.15.29:0/1073373385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93fa1b0 0x5587b97a9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.400+0000 7f6927fff640 1 -- 172.21.15.29:0/1073373385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587b9465790 con 0x5587b93d12f0 2024-07-27T06:48:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.400+0000 7f692cba3640 1 --2- 172.21.15.29:0/1073373385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93fa1b0 0x5587b97a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.400+0000 7f69355fc740 1 -- 172.21.15.29:0/1073373385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6910004430 con 0x5587b93d12f0 2024-07-27T06:48:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.400+0000 7f6927fff640 1 --2- 172.21.15.29:0/1073373385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d12f0 0x5587b97a8e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f69180a95a0 tx=0x7f69180abc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=684a6185b569344f server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.400+0000 7f69077fe640 10 client.25579 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:48:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.400+0000 7f69077fe640 1 -- 172.21.15.29:0/1073373385 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69180aca30 con 0x5587b93d12f0 2024-07-27T06:48:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.400+0000 7f69077fe640 1 -- 172.21.15.29:0/1073373385 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f69180046c0 con 0x5587b93d12f0 2024-07-27T06:48:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.401+0000 7f69077fe640 1 -- 172.21.15.29:0/1073373385 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f69180ac330 con 0x5587b93d12f0 2024-07-27T06:48:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.401+0000 7f69355fc740 10 client.25579 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.401+0000 7f69355fc740 1 -- 172.21.15.29:0/1073373385 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5587b94de620 con 0x5587b93d12f0 2024-07-27T06:48:09.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.401+0000 7f69077fe640 1 -- 172.21.15.29:0/1073373385 <== mon.1 v2:172.21.15.77:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f69180a85c0 con 0x5587b93d12f0 2024-07-27T06:48:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.401+0000 7f69355fc740 10 client.25579 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.401+0000 7f69355fc740 10 client.25579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.401+0000 7f69355fc740 20 client.25579 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.401+0000 7f69355fc740 1 --2- 172.21.15.29:0/1073373385 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5587b93b8d80 0x5587b97cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.401+0000 7f69355fc740 4 client.25579 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.401+0000 7f69355fc740 1 -- 172.21.15.29:0/1073373385 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587b9714300 con 0x5587b93b8d80 2024-07-27T06:48:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.402+0000 7f691ffff640 1 --2- 172.21.15.29:0/1073373385 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5587b93b8d80 0x5587b97cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.402+0000 7f69077fe640 10 client.25579 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.402+0000 7f691ffff640 1 --2- 172.21.15.29:0/1073373385 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5587b93b8d80 0x5587b97cb5d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f69a93b46247dec4 server_cookie=e3deea3cf0537ab in_seq=0 out_seq=0 2024-07-27T06:48:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69077fe640 1 -- 172.21.15.29:0/1073373385 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5587b9714300 con 0x5587b93b8d80 2024-07-27T06:48:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69077fe640 10 client.25579 handle_command_reply: tid=0 2024-07-27T06:48:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69355fc740 1 client.25579 shutdown 2024-07-27T06:48:09.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69355fc740 1 -- 172.21.15.29:0/1073373385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d12f0 msgr2=0x5587b97a8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69355fc740 1 --2- 172.21.15.29:0/1073373385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d12f0 0x5587b97a8e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f69180a95a0 tx=0x7f69180abc80 comp rx=0 tx=0).stop 2024-07-27T06:48:09.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69355fc740 1 -- 172.21.15.29:0/1073373385 shutdown_connections 2024-07-27T06:48:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69355fc740 1 --2- 172.21.15.29:0/1073373385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93fa1b0 0x5587b97a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69355fc740 1 --2- 172.21.15.29:0/1073373385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d94d0 0x5587b97b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69355fc740 1 --2- 172.21.15.29:0/1073373385 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5587b93b8d80 0x5587b97cb5d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69355fc740 1 --2- 172.21.15.29:0/1073373385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d12f0 0x5587b97a8e30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69077fe640 0 client.25579 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69355fc740 1 -- 172.21.15.29:0/1073373385 >> 172.21.15.29:0/1073373385 conn(0x5587b93ee300 msgr2=0x5587b979e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.403+0000 7f69355fc740 1 -- 172.21.15.29:0/1073373385 shutdown_connections 2024-07-27T06:48:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.404+0000 7f69355fc740 1 -- 172.21.15.29:0/1073373385 wait complete. 2024-07-27T06:48:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.404+0000 7f69355fc740 20 client.25579 trim_cache size 0 max 16384 2024-07-27T06:48:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.423+0000 7fd3ee39a640 1 -- 172.21.15.29:0/1333563362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3e810f500 msgr2=0x7fd3e810f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.423+0000 7fd3ee39a640 1 --2- 172.21.15.29:0/1333563362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3e810f500 0x7fd3e810f910 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009900 tx=0x7fd3dc02a3f0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.423+0000 7fd3ee39a640 1 -- 172.21.15.29:0/1333563362 shutdown_connections 2024-07-27T06:48:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.423+0000 7fd3ee39a640 1 --2- 172.21.15.29:0/1333563362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e810fe50 0x7fd3e8116680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.423+0000 7fd3ee39a640 1 --2- 172.21.15.29:0/1333563362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3e810f500 0x7fd3e810f910 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.423+0000 7fd3ee39a640 1 --2- 172.21.15.29:0/1333563362 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3e8109bc0 0x7fd3e8109fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.423+0000 7fd3ee39a640 1 -- 172.21.15.29:0/1333563362 >> 172.21.15.29:0/1333563362 conn(0x7fd3e8077b20 msgr2=0x7fd3e8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:09.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.423+0000 7fd3ee39a640 1 -- 172.21.15.29:0/1333563362 shutdown_connections 2024-07-27T06:48:09.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.423+0000 7fd3ee39a640 1 -- 172.21.15.29:0/1333563362 wait complete. 2024-07-27T06:48:09.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.425+0000 7fd3ee39a640 1 Processor -- start 2024-07-27T06:48:09.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.425+0000 7fd3ee39a640 1 -- start start 2024-07-27T06:48:09.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.426+0000 7fd3ee39a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3e8109bc0 0x7fd3e81a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.426+0000 7fd3ee39a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e810f500 0x7fd3e81a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.426+0000 7fd3ee39a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3e810fe50 0x7fd3e81a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.426+0000 7fd3ee39a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e8119b60 con 0x7fd3e8109bc0 2024-07-27T06:48:09.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.426+0000 7fd3ee39a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e81199e0 con 0x7fd3e810fe50 2024-07-27T06:48:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.426+0000 7fd3ee39a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e8119ce0 con 0x7fd3e810f500 2024-07-27T06:48:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.426+0000 7fd3e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3e8109bc0 0x7fd3e81a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.427+0000 7fd3e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3e8109bc0 0x7fd3e81a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51366/0 (socket says 172.21.15.29:51366) 2024-07-27T06:48:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.427+0000 7fd3e7fff640 1 -- 172.21.15.29:0/500422157 learned_addr learned my addr 172.21.15.29:0/500422157 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.427+0000 7fd3e77fe640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e810f500 0x7fd3e81a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.427+0000 7fd3ec937640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3e810fe50 0x7fd3e81a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.427+0000 7fd3ec937640 1 -- 172.21.15.29:0/500422157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e810f500 msgr2=0x7fd3e81a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.427+0000 7fd3ec937640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e810f500 0x7fd3e81a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.427+0000 7fd3ec937640 1 -- 172.21.15.29:0/500422157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3e8109bc0 msgr2=0x7fd3e81a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.427+0000 7fd3ec937640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3e8109bc0 0x7fd3e81a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.427+0000 7fd3ec937640 1 -- 172.21.15.29:0/500422157 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e81a6440 con 0x7fd3e810fe50 2024-07-27T06:48:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.427+0000 7fd3e7fff640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3e8109bc0 0x7fd3e81a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.427+0000 7fd3ec937640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3e810fe50 0x7fd3e81a5d10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0009fd0 tx=0x7fd3e000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.428+0000 7fd3e57fa640 1 -- 172.21.15.29:0/500422157 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e000ce20 con 0x7fd3e810fe50 2024-07-27T06:48:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.428+0000 7fd3ee39a640 1 -- 172.21.15.29:0/500422157 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e81a6730 con 0x7fd3e810fe50 2024-07-27T06:48:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.428+0000 7fd3ee39a640 1 -- 172.21.15.29:0/500422157 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e81ace90 con 0x7fd3e810fe50 2024-07-27T06:48:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.429+0000 7fd3e57fa640 1 -- 172.21.15.29:0/500422157 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3e0005020 con 0x7fd3e810fe50 2024-07-27T06:48:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.429+0000 7fd3e57fa640 1 -- 172.21.15.29:0/500422157 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0010690 con 0x7fd3e810fe50 2024-07-27T06:48:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.430+0000 7fd3ee39a640 1 -- 172.21.15.29:0/500422157 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3d40052c0 con 0x7fd3e810fe50 2024-07-27T06:48:09.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.433+0000 7fd3e57fa640 1 -- 172.21.15.29:0/500422157 <== mon.2 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fd3e0004a40 con 0x7fd3e810fe50 2024-07-27T06:48:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.434+0000 7fd3e57fa640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3d00b3dc0 0x7fd3d00b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.434+0000 7fd3e57fa640 1 -- 172.21.15.29:0/500422157 <== mon.2 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd3e00e4b10 con 0x7fd3e810fe50 2024-07-27T06:48:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.434+0000 7fd3e7fff640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3d00b3dc0 0x7fd3d00b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.435+0000 7fd3e7fff640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3d00b3dc0 0x7fd3d00b62d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8011550 tx=0x7fd3d8005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.436+0000 7f69355fc740 1 Processor -- start 2024-07-27T06:48:09.436 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.436+0000 7f69355fc740 1 -- start start 2024-07-27T06:48:09.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.436+0000 7fd3e57fa640 1 -- 172.21.15.29:0/500422157 <== mon.2 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd3e00abdd0 con 0x7fd3e810fe50 2024-07-27T06:48:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.436+0000 7f69355fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 0x5587b979a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.437 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.436+0000 7f69355fc740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d12f0 0x5587b979a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.437+0000 7f69355fc740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d94d0 0x5587b97a3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.437+0000 7f69355fc740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f69281ad400 con 0x5587b93b8d80 2024-07-27T06:48:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.437+0000 7f6927fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 0x5587b979a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.437+0000 7f6927fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 0x5587b979a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51368/0 (socket says 172.21.15.29:51368) 2024-07-27T06:48:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.437+0000 7f6927fff640 1 -- 172.21.15.29:0/3929946481 learned_addr learned my addr 172.21.15.29:0/3929946481 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:09.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.437+0000 7f692cba3640 1 --2- 172.21.15.29:0/3929946481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d94d0 0x5587b97a3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.437+0000 7f691ffff640 1 --2- 172.21.15.29:0/3929946481 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d12f0 0x5587b979a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.437+0000 7f69355fc740 1 -- 172.21.15.29:0/3929946481 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f69281aef40 con 0x5587b93d12f0 2024-07-27T06:48:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.437+0000 7f69355fc740 1 -- 172.21.15.29:0/3929946481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f69281adad0 con 0x5587b93d94d0 2024-07-27T06:48:09.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.438+0000 7f6927fff640 1 -- 172.21.15.29:0/3929946481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d94d0 msgr2=0x5587b97a3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.438+0000 7f6927fff640 1 --2- 172.21.15.29:0/3929946481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d94d0 0x5587b97a3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.438+0000 7f6927fff640 1 -- 172.21.15.29:0/3929946481 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d12f0 msgr2=0x5587b979a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.438+0000 7f6927fff640 1 --2- 172.21.15.29:0/3929946481 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d12f0 0x5587b979a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.438+0000 7f6927fff640 1 -- 172.21.15.29:0/3929946481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587b979db20 con 0x5587b93b8d80 2024-07-27T06:48:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.438+0000 7f6927fff640 1 --2- 172.21.15.29:0/3929946481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 0x5587b979a2c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f69180b0950 tx=0x7f691800d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912fdd4f761807f3 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.438+0000 7f691dffb640 1 -- 172.21.15.29:0/3929946481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69180aca30 con 0x5587b93b8d80 2024-07-27T06:48:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.439+0000 7f691dffb640 1 -- 172.21.15.29:0/3929946481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6910004430 con 0x5587b93b8d80 2024-07-27T06:48:09.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.439+0000 7f691dffb640 1 -- 172.21.15.29:0/3929946481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69180ab590 con 0x5587b93b8d80 2024-07-27T06:48:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.440+0000 7f69355fc740 1 -- 172.21.15.29:0/3929946481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 msgr2=0x5587b979a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.440+0000 7f69355fc740 1 --2- 172.21.15.29:0/3929946481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 0x5587b979a2c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f69180b0950 tx=0x7f691800d000 comp rx=0 tx=0).stop 2024-07-27T06:48:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.440+0000 7f69355fc740 1 -- 172.21.15.29:0/3929946481 shutdown_connections 2024-07-27T06:48:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.440+0000 7f69355fc740 1 --2- 172.21.15.29:0/3929946481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d94d0 0x5587b97a3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.440+0000 7f69355fc740 1 --2- 172.21.15.29:0/3929946481 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d12f0 0x5587b979a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.440+0000 7f69355fc740 1 --2- 172.21.15.29:0/3929946481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 0x5587b979a2c0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.440+0000 7f69355fc740 1 -- 172.21.15.29:0/3929946481 >> 172.21.15.29:0/3929946481 conn(0x5587b93ee300 msgr2=0x5587b97a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.440+0000 7f69355fc740 1 -- 172.21.15.29:0/3929946481 shutdown_connections 2024-07-27T06:48:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.440+0000 7f69355fc740 1 -- 172.21.15.29:0/3929946481 wait complete. 2024-07-27T06:48:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.441+0000 7f69355fc740 1 -- start start 2024-07-27T06:48:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.441+0000 7f69355fc740 1 Processor -- start 2024-07-27T06:48:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.441+0000 7f69355fc740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 0x5587b97d9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.442+0000 7f6927fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 0x5587b97d9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.442+0000 7f6927fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 0x5587b97d9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51376/0 (socket says 172.21.15.29:51376) 2024-07-27T06:48:09.445 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.442+0000 7f6927fff640 1 -- 172.21.15.29:0/2019492445 learned_addr learned my addr 172.21.15.29:0/2019492445 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.442+0000 7f69355fc740 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d12f0 0x5587b97da1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.442+0000 7f69355fc740 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d94d0 0x5587b9831bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.442+0000 7f69355fc740 1 -- 172.21.15.29:0/2019492445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f69281ae870 con 0x5587b93b8d80 2024-07-27T06:48:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.442+0000 7f69355fc740 1 -- 172.21.15.29:0/2019492445 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f69281ae1a0 con 0x5587b93d12f0 2024-07-27T06:48:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.442+0000 7f69355fc740 1 -- 172.21.15.29:0/2019492445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f69281ab3e0 con 0x5587b93d94d0 2024-07-27T06:48:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.442+0000 7f692cba3640 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d94d0 0x5587b9831bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.442+0000 7f691ffff640 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d12f0 0x5587b97da1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f692cba3640 1 -- 172.21.15.29:0/2019492445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d12f0 msgr2=0x5587b97da1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f692cba3640 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d12f0 0x5587b97da1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f692cba3640 1 -- 172.21.15.29:0/2019492445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 msgr2=0x5587b97d9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f692cba3640 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 0x5587b97d9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f692cba3640 1 -- 172.21.15.29:0/2019492445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587b97daa60 con 0x5587b93d94d0 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f691ffff640 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d12f0 0x5587b97da1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f6927fff640 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 0x5587b97d9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f69355fc740 10 client.0 _ll_register_callbacks cb 0x7f69180a81f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f692cba3640 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d94d0 0x5587b9831bc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6920012ea0 tx=0x7f6920004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=976176caa1f9638 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f69355fc740 1 -- 172.21.15.29:0/2019492445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6910004430 con 0x5587b93d94d0 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f69077fe640 10 client.25591 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f69077fe640 1 -- 172.21.15.29:0/2019492445 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692002f060 con 0x5587b93d94d0 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f69077fe640 1 -- 172.21.15.29:0/2019492445 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f692002a400 con 0x5587b93d94d0 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f69077fe640 1 -- 172.21.15.29:0/2019492445 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f692002a620 con 0x5587b93d94d0 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f69355fc740 10 client.25591 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.443+0000 7f69355fc740 1 -- 172.21.15.29:0/2019492445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5587b9833070 con 0x5587b93d94d0 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.444+0000 7f69077fe640 1 -- 172.21.15.29:0/2019492445 <== mon.1 v2:172.21.15.77:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f692002a7c0 con 0x5587b93d94d0 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.444+0000 7f69355fc740 10 client.25591 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:09.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.444+0000 7f69355fc740 10 client.25591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.444+0000 7f69355fc740 20 client.25591 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.444+0000 7f69355fc740 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5587b93fa1b0 0x5587b98356b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.444+0000 7f69355fc740 4 client.25591 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-27T06:48:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.444+0000 7f69355fc740 1 -- 172.21.15.29:0/2019492445 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f6910004430 con 0x5587b93fa1b0 2024-07-27T06:48:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.444+0000 7f6927fff640 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5587b93fa1b0 0x5587b98356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.445+0000 7f6927fff640 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5587b93fa1b0 0x5587b98356b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d399d5a262cd5203 server_cookie=281db5efccb0d22d in_seq=0 out_seq=0 2024-07-27T06:48:09.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.445+0000 7f69077fe640 10 client.25591 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.454+0000 7f69077fe640 1 -- 172.21.15.29:0/2019492445 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f6910004430 con 0x5587b93fa1b0 2024-07-27T06:48:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.454+0000 7f69077fe640 10 client.25591 handle_command_reply: tid=0 2024-07-27T06:48:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.454+0000 7f69355fc740 1 client.25591 shutdown 2024-07-27T06:48:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.455+0000 7f69355fc740 1 -- 172.21.15.29:0/2019492445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d94d0 msgr2=0x5587b9831bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.455+0000 7f69355fc740 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d94d0 0x5587b9831bc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6920012ea0 tx=0x7f6920004a40 comp rx=0 tx=0).stop 2024-07-27T06:48:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.455+0000 7f69355fc740 1 -- 172.21.15.29:0/2019492445 shutdown_connections 2024-07-27T06:48:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.455+0000 7f69355fc740 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5587b93fa1b0 0x5587b98356b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.455+0000 7f69355fc740 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5587b93d94d0 0x5587b9831bc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.455+0000 7f69077fe640 0 client.25591 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.455+0000 7f69355fc740 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5587b93d12f0 0x5587b97da1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.455+0000 7f69355fc740 1 --2- 172.21.15.29:0/2019492445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5587b93b8d80 0x5587b97d9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.455+0000 7f69355fc740 1 -- 172.21.15.29:0/2019492445 >> 172.21.15.29:0/2019492445 conn(0x5587b93ee300 msgr2=0x5587b97a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.455+0000 7f69355fc740 1 -- 172.21.15.29:0/2019492445 shutdown_connections 2024-07-27T06:48:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.455+0000 7f69355fc740 1 -- 172.21.15.29:0/2019492445 wait complete. 2024-07-27T06:48:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.455+0000 7f69355fc740 20 client.25591 trim_cache size 0 max 16384 2024-07-27T06:48:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-07-27T06:48:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-07-27T06:48:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-07-27T06:48:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-07-27T06:48:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.456+0000 7f692e606640 1 -- 172.21.15.29:0/3423043032 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f69140b3e40 msgr2=0x7f69140b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.456+0000 7f692e606640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f69140b3e40 0x7f69140b6350 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6920005fd0 tx=0x7f6920009500 comp rx=0 tx=0).stop 2024-07-27T06:48:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.456+0000 7f692e606640 1 -- 172.21.15.29:0/3423043032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6928101bb0 msgr2=0x7f6928197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.456+0000 7f692e606640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6928101bb0 0x7f6928197f20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f691800eb00 tx=0x7f691800ef00 comp rx=0 tx=0).stop 2024-07-27T06:48:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.456+0000 7f692e606640 1 -- 172.21.15.29:0/3423043032 shutdown_connections 2024-07-27T06:48:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.456+0000 7f692e606640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f692819d690 0x7f692819daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.456+0000 7f692e606640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f69140b3e40 0x7f69140b6350 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.456+0000 7f692e606640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6928107f30 0x7f69281984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.456+0000 7f692e606640 1 --2- 172.21.15.29:0/3423043032 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6928101bb0 0x7f6928197f20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.456+0000 7f692e606640 1 -- 172.21.15.29:0/3423043032 >> 172.21.15.29:0/3423043032 conn(0x7f6928079880 msgr2=0x7f6928103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.456+0000 7f692e606640 1 -- 172.21.15.29:0/3423043032 shutdown_connections 2024-07-27T06:48:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:09.457+0000 7f692e606640 1 -- 172.21.15.29:0/3423043032 wait complete. 2024-07-27T06:48:09.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.756+0000 7fd3ee39a640 1 -- 172.21.15.29:0/500422157 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3d40055b0 con 0x7fd3e810fe50 2024-07-27T06:48:09.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.757+0000 7fd3e57fa640 1 -- 172.21.15.29:0/500422157 <== mon.2 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11957 (secure 0 0 0) 0x7fd3e00ab6d0 con 0x7fd3e810fe50 2024-07-27T06:48:09.758 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-07-27T06:48:09.759 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:48:09.759 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-07-27T06:46:50:421100+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:46:49.561737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24670,24658,24646,24617,24613]},"id":1}]} 2024-07-27T06:48:09.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.760+0000 7fd3ee39a640 1 -- 172.21.15.29:0/500422157 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3d00b3dc0 msgr2=0x7fd3d00b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.760+0000 7fd3ee39a640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3d00b3dc0 0x7fd3d00b62d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8011550 tx=0x7fd3d8005f90 comp rx=0 tx=0).stop 2024-07-27T06:48:09.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.760+0000 7fd3ee39a640 1 -- 172.21.15.29:0/500422157 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3e810fe50 msgr2=0x7fd3e81a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:09.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.760+0000 7fd3ee39a640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3e810fe50 0x7fd3e81a5d10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0009fd0 tx=0x7fd3e000ef10 comp rx=0 tx=0).stop 2024-07-27T06:48:09.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.760+0000 7fd3ee39a640 1 -- 172.21.15.29:0/500422157 shutdown_connections 2024-07-27T06:48:09.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.760+0000 7fd3ee39a640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3e810fe50 0x7fd3e81a5d10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.760+0000 7fd3ee39a640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3d00b3dc0 0x7fd3d00b62d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.760+0000 7fd3ee39a640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e810f500 0x7fd3e81a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.760+0000 7fd3ee39a640 1 --2- 172.21.15.29:0/500422157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3e8109bc0 0x7fd3e81a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:09.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.760+0000 7fd3ee39a640 1 -- 172.21.15.29:0/500422157 >> 172.21.15.29:0/500422157 conn(0x7fd3e8077b20 msgr2=0x7fd3e810d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:09.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.760+0000 7fd3ee39a640 1 -- 172.21.15.29:0/500422157 shutdown_connections 2024-07-27T06:48:09.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:09.760+0000 7fd3ee39a640 1 -- 172.21.15.29:0/500422157 wait complete. 2024-07-27T06:48:09.901 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:09 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/500422157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:48:10.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:09 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/500422157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:48:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:09 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/500422157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:48:10.684 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24670 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0' 2024-07-27T06:48:10.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- timeout 105 ceph tell mds.24670 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0 2024-07-27T06:48:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:10 smithi029 ceph-mon[32610]: pgmap v432: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-07-27T06:48:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:10 smithi077 ceph-mon[37153]: pgmap v432: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-07-27T06:48:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:10 smithi047 ceph-mon[36835]: pgmap v432: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-07-27T06:48:11.208 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.810+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/611498718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfdc107eb0 msgr2=0x7fcfdc10e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.810+0000 7fcfe3cd3640 1 --2- 172.21.15.29:0/611498718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfdc107eb0 0x7fcfdc10e6e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400adb0 tx=0x7fcfd402b7a0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.811+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/611498718 shutdown_connections 2024-07-27T06:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.811+0000 7fcfe3cd3640 1 --2- 172.21.15.29:0/611498718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfdc107eb0 0x7fcfdc10e6e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.811+0000 7fcfe3cd3640 1 --2- 172.21.15.29:0/611498718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcfdc107560 0x7fcfdc107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.811+0000 7fcfe3cd3640 1 --2- 172.21.15.29:0/611498718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfdc101c20 0x7fcfdc102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.811+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/611498718 >> 172.21.15.29:0/611498718 conn(0x7fcfdc077690 msgr2=0x7fcfdc077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.811+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/611498718 shutdown_connections 2024-07-27T06:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.812+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/611498718 wait complete. 2024-07-27T06:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.813+0000 7fcfe3cd3640 1 Processor -- start 2024-07-27T06:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.813+0000 7fcfe3cd3640 1 -- start start 2024-07-27T06:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.813+0000 7fcfe3cd3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfdc101c20 0x7fcfdc199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.814+0000 7fcfe1a6f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfdc101c20 0x7fcfdc199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.814+0000 7fcfe1a6f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfdc101c20 0x7fcfdc199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51390/0 (socket says 172.21.15.29:51390) 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.814+0000 7fcfe1a6f640 1 -- 172.21.15.29:0/649845375 learned_addr learned my addr 172.21.15.29:0/649845375 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.814+0000 7fcfe3cd3640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcfdc107560 0x7fcfdc19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.814+0000 7fcfe3cd3640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfdc107eb0 0x7fcfdc19f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.814+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/649845375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc1112f0 con 0x7fcfdc101c20 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.814+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/649845375 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc111170 con 0x7fcfdc107560 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.815+0000 7fcfe126e640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcfdc107560 0x7fcfdc19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.815+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/649845375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc111470 con 0x7fcfdc107eb0 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.815+0000 7fcfe2270640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfdc107eb0 0x7fcfdc19f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.815+0000 7fcfe1a6f640 1 -- 172.21.15.29:0/649845375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfdc107eb0 msgr2=0x7fcfdc19f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.815+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfdc107eb0 0x7fcfdc19f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.815+0000 7fcfe1a6f640 1 -- 172.21.15.29:0/649845375 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcfdc107560 msgr2=0x7fcfdc19a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.815+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcfdc107560 0x7fcfdc19a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.816+0000 7fcfe1a6f640 1 -- 172.21.15.29:0/649845375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc19fdf0 con 0x7fcfdc101c20 2024-07-27T06:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.816+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfdc101c20 0x7fcfdc199e80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800ed60 tx=0x7fcfd800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.817+0000 7fcfd2ffd640 1 -- 172.21.15.29:0/649845375 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8009e30 con 0x7fcfdc101c20 2024-07-27T06:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.817+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/649845375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfdc1a00e0 con 0x7fcfdc101c20 2024-07-27T06:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.817+0000 7fcfd2ffd640 1 -- 172.21.15.29:0/649845375 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfd80054d0 con 0x7fcfdc101c20 2024-07-27T06:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.817+0000 7fcfd2ffd640 1 -- 172.21.15.29:0/649845375 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8009e30 con 0x7fcfdc101c20 2024-07-27T06:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.817+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/649845375 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfdc0fe6f0 con 0x7fcfdc101c20 2024-07-27T06:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.821+0000 7fcfd2ffd640 1 -- 172.21.15.29:0/649845375 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fcfd80046c0 con 0x7fcfdc101c20 2024-07-27T06:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.823+0000 7fcfd2ffd640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcfc80b4140 0x7fcfc80b6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.823+0000 7fcfd2ffd640 1 -- 172.21.15.29:0/649845375 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fcfd800cdb0 con 0x7fcfdc101c20 2024-07-27T06:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.823+0000 7fcfe126e640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcfc80b4140 0x7fcfc80b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.824+0000 7fcfe126e640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcfc80b4140 0x7fcfc80b6650 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc105740 tx=0x7fcfcc005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.825+0000 7fcfea257740 1 Processor -- start 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.825+0000 7fcfea257740 1 -- start start 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.825+0000 7fcfea257740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488236350 0x5604885db3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.825+0000 7fcfea257740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604885db8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.825+0000 7fcfea257740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560488215670 0x5604885dbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfea257740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5604884c32f0 con 0x56048820d490 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfea257740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x560488551a80 con 0x560488215670 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfea257740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560488540f80 con 0x560488236350 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfe126e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604885db8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfe1a6f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488236350 0x5604885db3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfe126e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604885db8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51404/0 (socket says 172.21.15.29:51404) 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfe126e640 1 -- 172.21.15.29:0/3232120492 learned_addr learned my addr 172.21.15.29:0/3232120492 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfe2270640 1 --2- 172.21.15.29:0/3232120492 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560488215670 0x5604885dbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfe126e640 1 -- 172.21.15.29:0/3232120492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488236350 msgr2=0x5604885db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfe126e640 1 --2- 172.21.15.29:0/3232120492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488236350 0x5604885db3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfe126e640 1 -- 172.21.15.29:0/3232120492 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560488215670 msgr2=0x5604885dbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfe126e640 1 --2- 172.21.15.29:0/3232120492 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560488215670 0x5604885dbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfe126e640 1 -- 172.21.15.29:0/3232120492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604882a1930 con 0x56048820d490 2024-07-27T06:48:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/3232120492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488236350 0x5604885db3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.826+0000 7fcfe126e640 1 --2- 172.21.15.29:0/3232120492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604885db8f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00e600 tx=0x7fcfcc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b122a333ed8fb38 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.827+0000 7fcfc7fff640 1 -- 172.21.15.29:0/3232120492 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc004020 con 0x56048820d490 2024-07-27T06:48:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.827+0000 7fcfc7fff640 1 -- 172.21.15.29:0/3232120492 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfcc0041c0 con 0x56048820d490 2024-07-27T06:48:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.827+0000 7fcfc7fff640 1 -- 172.21.15.29:0/3232120492 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc01c650 con 0x56048820d490 2024-07-27T06:48:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.828+0000 7fcfea257740 1 -- 172.21.15.29:0/3232120492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 msgr2=0x5604885db8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.828+0000 7fcfea257740 1 --2- 172.21.15.29:0/3232120492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604885db8f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00e600 tx=0x7fcfcc014000 comp rx=0 tx=0).stop 2024-07-27T06:48:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.828+0000 7fcfea257740 1 -- 172.21.15.29:0/3232120492 shutdown_connections 2024-07-27T06:48:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.828+0000 7fcfea257740 1 --2- 172.21.15.29:0/3232120492 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560488215670 0x5604885dbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.828+0000 7fcfea257740 1 --2- 172.21.15.29:0/3232120492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604885db8f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.828+0000 7fcfea257740 1 --2- 172.21.15.29:0/3232120492 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488236350 0x5604885db3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.828+0000 7fcfea257740 1 -- 172.21.15.29:0/3232120492 >> 172.21.15.29:0/3232120492 conn(0x56048822a740 msgr2=0x5604885d92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.828+0000 7fcfea257740 1 -- 172.21.15.29:0/3232120492 shutdown_connections 2024-07-27T06:48:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.828+0000 7fcfea257740 1 -- 172.21.15.29:0/3232120492 wait complete. 2024-07-27T06:48:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.829+0000 7fcfea257740 1 -- start start 2024-07-27T06:48:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.829+0000 7fcfea257740 1 Processor -- start 2024-07-27T06:48:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.830+0000 7fcfea257740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56048820d490 0x5604885efda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.830+0000 7fcfea257740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560488215670 0x5604885f0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.830+0000 7fcfe126e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560488215670 0x5604885f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.830+0000 7fcfe1a6f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56048820d490 0x5604885efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.830+0000 7fcfe126e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560488215670 0x5604885f0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51408/0 (socket says 172.21.15.29:51408) 2024-07-27T06:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.830+0000 7fcfe1a6f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56048820d490 0x5604885efda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.29:56060/0 (socket says 172.21.15.29:56060) 2024-07-27T06:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.830+0000 7fcfe126e640 1 -- 172.21.15.29:0/3047646033 learned_addr learned my addr 172.21.15.29:0/3047646033 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.830+0000 7fcfea257740 1 --2- 172.21.15.29:0/3047646033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488236350 0x5604885f53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.831+0000 7fcfea257740 1 -- 172.21.15.29:0/3047646033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5604884e34f0 con 0x560488215670 2024-07-27T06:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.831+0000 7fcfea257740 1 -- 172.21.15.29:0/3047646033 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x560488544fc0 con 0x56048820d490 2024-07-27T06:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.831+0000 7fcfea257740 1 -- 172.21.15.29:0/3047646033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5604883a4890 con 0x560488236350 2024-07-27T06:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.831+0000 7fcfea257740 10 client.0 _ll_register_callbacks cb 0x5604884ed500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.831+0000 7fcfe126e640 1 -- 172.21.15.29:0/3047646033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488236350 msgr2=0x5604885f53a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.831+0000 7fcfe126e640 1 --2- 172.21.15.29:0/3047646033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488236350 0x5604885f53a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.831+0000 7fcfe126e640 1 -- 172.21.15.29:0/3047646033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56048820d490 msgr2=0x5604885efda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.831+0000 7fcfe126e640 1 --2- 172.21.15.29:0/3047646033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56048820d490 0x5604885efda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.831+0000 7fcfe126e640 1 -- 172.21.15.29:0/3047646033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604884f2c50 con 0x560488215670 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.831+0000 7fcfe126e640 1 --2- 172.21.15.29:0/3047646033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560488215670 0x5604885f0380 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc003db0 tx=0x7fcfcc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab1959816e2a49e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.831+0000 7fcfc57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.832+0000 7fcfea257740 1 -- 172.21.15.29:0/3047646033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfcc0041c0 con 0x560488215670 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.832+0000 7fcfc57fa640 1 -- 172.21.15.29:0/3047646033 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc01c610 con 0x560488215670 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.832+0000 7fcfc57fa640 1 -- 172.21.15.29:0/3047646033 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fcfcc01c950 con 0x560488215670 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.832+0000 7fcfc57fa640 1 -- 172.21.15.29:0/3047646033 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcfcc01cb90 con 0x560488215670 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.832+0000 7fcfea257740 10 client.15894 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.832+0000 7fcfea257740 1 -- 172.21.15.29:0/3047646033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5604885fe210 con 0x560488215670 2024-07-27T06:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.833+0000 7fcfc57fa640 1 -- 172.21.15.29:0/3047646033 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fcfcc037030 con 0x560488215670 2024-07-27T06:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.833+0000 7fcfea257740 10 client.15894 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.833+0000 7fcfea257740 10 client.15894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.833+0000 7fcfea257740 20 client.15894 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.833+0000 7fcfea257740 1 --2- 172.21.15.29:0/3047646033 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5604881f4cd0 0x560488601ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.833+0000 7fcfea257740 4 client.15894 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.833+0000 7fcfea257740 1 -- 172.21.15.29:0/3047646033 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcfcc0041c0 con 0x5604881f4cd0 2024-07-27T06:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.833+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/3047646033 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5604881f4cd0 0x560488601ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.833+0000 7fcfc57fa640 10 client.15894 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.833+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/3047646033 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5604881f4cd0 0x560488601ab0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f23f5c5c72ab503 server_cookie=94a827f2850f07a0 in_seq=0 out_seq=0 2024-07-27T06:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.834+0000 7fcfc57fa640 1 -- 172.21.15.29:0/3047646033 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fcfcc0041c0 con 0x5604881f4cd0 2024-07-27T06:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.834+0000 7fcfc57fa640 10 client.15894 handle_command_reply: tid=0 2024-07-27T06:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.834+0000 7fcfea257740 1 client.15894 shutdown 2024-07-27T06:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.834+0000 7fcfea257740 1 -- 172.21.15.29:0/3047646033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560488215670 msgr2=0x5604885f0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.834+0000 7fcfea257740 1 --2- 172.21.15.29:0/3047646033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560488215670 0x5604885f0380 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc003db0 tx=0x7fcfcc0040c0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.834+0000 7fcfea257740 1 -- 172.21.15.29:0/3047646033 shutdown_connections 2024-07-27T06:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.835+0000 7fcfea257740 1 --2- 172.21.15.29:0/3047646033 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5604881f4cd0 0x560488601ab0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.835+0000 7fcfea257740 1 --2- 172.21.15.29:0/3047646033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488236350 0x5604885f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.835+0000 7fcfc57fa640 0 client.15894 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.835+0000 7fcfea257740 1 --2- 172.21.15.29:0/3047646033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560488215670 0x5604885f0380 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.835+0000 7fcfea257740 1 --2- 172.21.15.29:0/3047646033 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56048820d490 0x5604885efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.835+0000 7fcfea257740 1 -- 172.21.15.29:0/3047646033 >> 172.21.15.29:0/3047646033 conn(0x56048822a740 msgr2=0x5604885de540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.835+0000 7fcfea257740 1 -- 172.21.15.29:0/3047646033 shutdown_connections 2024-07-27T06:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.835+0000 7fcfea257740 1 -- 172.21.15.29:0/3047646033 wait complete. 2024-07-27T06:48:11.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.835+0000 7fcfea257740 20 client.15894 trim_cache size 0 max 16384 2024-07-27T06:48:11.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.862+0000 7fcfea257740 1 Processor -- start 2024-07-27T06:48:11.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.862+0000 7fcfea257740 1 -- start start 2024-07-27T06:48:11.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.862+0000 7fcfea257740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5604881f4cd0 0x5604885f5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.862+0000 7fcfea257740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604885debc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfea257740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488215670 0x5604885e8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfea257740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc1acf90 con 0x56048820d490 2024-07-27T06:48:11.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfe126e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604885debc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfe126e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604885debc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51410/0 (socket says 172.21.15.29:51410) 2024-07-27T06:48:11.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfe126e640 1 -- 172.21.15.29:0/1361753752 learned_addr learned my addr 172.21.15.29:0/1361753752 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:11.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfe1a6f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5604881f4cd0 0x5604885f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfea257740 1 -- 172.21.15.29:0/1361753752 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc1aead0 con 0x5604881f4cd0 2024-07-27T06:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfea257740 1 -- 172.21.15.29:0/1361753752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc1ad660 con 0x560488215670 2024-07-27T06:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfe2270640 1 --2- 172.21.15.29:0/1361753752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488215670 0x5604885e8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfe1a6f640 1 -- 172.21.15.29:0/1361753752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488215670 msgr2=0x5604885e8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/1361753752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488215670 0x5604885e8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfe1a6f640 1 -- 172.21.15.29:0/1361753752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 msgr2=0x5604885debc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/1361753752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604885debc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfe1a6f640 1 -- 172.21.15.29:0/1361753752 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604885dd6a0 con 0x5604881f4cd0 2024-07-27T06:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfe2270640 1 --2- 172.21.15.29:0/1361753752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488215670 0x5604885e8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.863+0000 7fcfe126e640 1 --2- 172.21.15.29:0/1361753752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604885debc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.864+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/1361753752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5604881f4cd0 0x5604885f5980 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80e15a0 tx=0x7fcfd800cfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13c0e4636aa1452d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.864+0000 7fcfc77fe640 1 -- 172.21.15.29:0/1361753752 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd80b3450 con 0x5604881f4cd0 2024-07-27T06:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.864+0000 7fcfc77fe640 1 -- 172.21.15.29:0/1361753752 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfd80a6040 con 0x5604881f4cd0 2024-07-27T06:48:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.864+0000 7fcfc77fe640 1 -- 172.21.15.29:0/1361753752 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd80c52d0 con 0x5604881f4cd0 2024-07-27T06:48:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.865+0000 7fcfea257740 1 -- 172.21.15.29:0/1361753752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5604881f4cd0 msgr2=0x5604885f5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.865+0000 7fcfea257740 1 --2- 172.21.15.29:0/1361753752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5604881f4cd0 0x5604885f5980 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80e15a0 tx=0x7fcfd800cfb0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.865+0000 7fcfea257740 1 -- 172.21.15.29:0/1361753752 shutdown_connections 2024-07-27T06:48:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.865+0000 7fcfea257740 1 --2- 172.21.15.29:0/1361753752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488215670 0x5604885e8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.865+0000 7fcfea257740 1 --2- 172.21.15.29:0/1361753752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604885debc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.865+0000 7fcfea257740 1 --2- 172.21.15.29:0/1361753752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5604881f4cd0 0x5604885f5980 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.865+0000 7fcfea257740 1 -- 172.21.15.29:0/1361753752 >> 172.21.15.29:0/1361753752 conn(0x56048822a740 msgr2=0x5604885de4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.865+0000 7fcfea257740 1 -- 172.21.15.29:0/1361753752 shutdown_connections 2024-07-27T06:48:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.865+0000 7fcfea257740 1 -- 172.21.15.29:0/1361753752 wait complete. 2024-07-27T06:48:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.866+0000 7fcfea257740 1 -- start start 2024-07-27T06:48:11.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.866+0000 7fcfea257740 1 Processor -- start 2024-07-27T06:48:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.866+0000 7fcfea257740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5604881f4cd0 0x560488606f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.866+0000 7fcfea257740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604886074c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.866+0000 7fcfea257740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488215670 0x560488607a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.866+0000 7fcfea257740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc1ae400 con 0x56048820d490 2024-07-27T06:48:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.866+0000 7fcfea257740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc1add30 con 0x5604881f4cd0 2024-07-27T06:48:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.866+0000 7fcfea257740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc1aaf70 con 0x560488215670 2024-07-27T06:48:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.866+0000 7fcfea257740 10 client.0 _ll_register_callbacks cb 0x7fcfcc0056f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfe126e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604886074c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfe2270640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488215670 0x560488607a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfe126e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604886074c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51414/0 (socket says 172.21.15.29:51414) 2024-07-27T06:48:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfe126e640 1 -- 172.21.15.29:0/367284908 learned_addr learned my addr 172.21.15.29:0/367284908 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5604881f4cd0 0x560488606f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfe2270640 1 -- 172.21.15.29:0/367284908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5604881f4cd0 msgr2=0x560488606f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfe2270640 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5604881f4cd0 0x560488606f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfe2270640 1 -- 172.21.15.29:0/367284908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 msgr2=0x5604886074c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfe2270640 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604886074c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfe2270640 1 -- 172.21.15.29:0/367284908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604885fa9c0 con 0x560488215670 2024-07-27T06:48:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5604881f4cd0 0x560488606f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:48:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfe2270640 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488215670 0x560488607a00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4030f50 tx=0x7fcfd400a910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7870e45a95abbae server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:11.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfc57fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:48:11.939 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:48:11.940 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-07-27T06:48:11.940 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "57d65104-c53e-48c0-9954-c270f4a2f4e8", 2024-07-27T06:48:11.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-07-27T06:48:11.940 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:48:11.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.867+0000 7fcfc57fa640 1 -- 172.21.15.29:0/367284908 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4003d80 con 0x560488215670 2024-07-27T06:48:11.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfc57fa640 1 -- 172.21.15.29:0/367284908 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fcfd4032400 con 0x560488215670 2024-07-27T06:48:11.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfea257740 1 -- 172.21.15.29:0/367284908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfd80a6040 con 0x560488215670 2024-07-27T06:48:11.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfc57fa640 1 -- 172.21.15.29:0/367284908 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcfd4053050 con 0x560488215670 2024-07-27T06:48:11.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfea257740 10 client.25606 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:11.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfea257740 1 -- 172.21.15.29:0/367284908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56048866a4c0 con 0x560488215670 2024-07-27T06:48:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfc57fa640 1 -- 172.21.15.29:0/367284908 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfd404a070 con 0x560488215670 2024-07-27T06:48:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfc57fa640 1 -- 172.21.15.29:0/367284908 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fcfd4031610 con 0x560488215670 2024-07-27T06:48:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfea257740 10 client.25606 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfea257740 10 client.25606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfea257740 20 client.25606 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfea257740 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x560488236350 0x56048866cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfea257740 4 client.25606 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-27T06:48:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.868+0000 7fcfea257740 1 -- 172.21.15.29:0/367284908 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fcfd80a6040 con 0x560488236350 2024-07-27T06:48:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.869+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x560488236350 0x56048866cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.869+0000 7fcfc57fa640 10 client.25606 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.869+0000 7fcfe1a6f640 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x560488236350 0x56048866cc70 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45c143198f34c005 server_cookie=fe90969e92e2be38 in_seq=0 out_seq=0 2024-07-27T06:48:11.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.870+0000 7fcfc57fa640 1 -- 172.21.15.29:0/367284908 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fcfd80a6040 con 0x560488236350 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.870+0000 7fcfc57fa640 10 client.25606 handle_command_reply: tid=0 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.870+0000 7fcfea257740 1 client.25606 shutdown 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.870+0000 7fcfea257740 1 -- 172.21.15.29:0/367284908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488215670 msgr2=0x560488607a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.870+0000 7fcfea257740 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488215670 0x560488607a00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4030f50 tx=0x7fcfd400a910 comp rx=0 tx=0).stop 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.870+0000 7fcfea257740 1 -- 172.21.15.29:0/367284908 shutdown_connections 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.870+0000 7fcfea257740 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x560488236350 0x56048866cc70 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.870+0000 7fcfc57fa640 0 client.25606 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.870+0000 7fcfea257740 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560488215670 0x560488607a00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.870+0000 7fcfea257740 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56048820d490 0x5604886074c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.870+0000 7fcfea257740 1 --2- 172.21.15.29:0/367284908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5604881f4cd0 0x560488606f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.870+0000 7fcfea257740 1 -- 172.21.15.29:0/367284908 >> 172.21.15.29:0/367284908 conn(0x56048822a740 msgr2=0x5604885f4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.871+0000 7fcfea257740 1 -- 172.21.15.29:0/367284908 shutdown_connections 2024-07-27T06:48:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.871+0000 7fcfea257740 1 -- 172.21.15.29:0/367284908 wait complete. 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.871+0000 7fcfea257740 20 client.25606 trim_cache size 0 max 16384 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.871+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/649845375 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcfc80b4140 msgr2=0x7fcfc80b6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.871+0000 7fcfe3cd3640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcfc80b4140 0x7fcfc80b6650 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc105740 tx=0x7fcfcc005f90 comp rx=0 tx=0).stop 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.871+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/649845375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfdc101c20 msgr2=0x7fcfdc199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.871+0000 7fcfe3cd3640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfdc101c20 0x7fcfdc199e80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800ed60 tx=0x7fcfd800c6a0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.872+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/649845375 shutdown_connections 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.872+0000 7fcfe3cd3640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fcfc80b4140 0x7fcfc80b6650 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.872+0000 7fcfe3cd3640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcfdc107eb0 0x7fcfdc19f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.872+0000 7fcfe3cd3640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcfdc107560 0x7fcfdc19a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.872+0000 7fcfe3cd3640 1 --2- 172.21.15.29:0/649845375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfdc101c20 0x7fcfdc199e80 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.872+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/649845375 >> 172.21.15.29:0/649845375 conn(0x7fcfdc077690 msgr2=0x7fcfdc103730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.872+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/649845375 shutdown_connections 2024-07-27T06:48:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:11.872+0000 7fcfe3cd3640 1 -- 172.21.15.29:0/649845375 wait complete. 2024-07-27T06:48:12.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-27T06:48:13.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:12 smithi029 ceph-mon[32610]: pgmap v433: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-07-27T06:48:13.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:12 smithi029 ceph-mon[32610]: scrub queued for path: / 2024-07-27T06:48:13.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:12 smithi029 ceph-mon[32610]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:13.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:12 smithi029 ceph-mon[32610]: scrub summary: active paths [/] 2024-07-27T06:48:13.002 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:12 smithi029 ceph-mon[32610]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:12 smithi077 ceph-mon[37153]: pgmap v433: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-07-27T06:48:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:12 smithi077 ceph-mon[37153]: scrub queued for path: / 2024-07-27T06:48:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:12 smithi077 ceph-mon[37153]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:12 smithi077 ceph-mon[37153]: scrub summary: active paths [/] 2024-07-27T06:48:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:12 smithi077 ceph-mon[37153]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:12 smithi047 ceph-mon[36835]: pgmap v433: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-07-27T06:48:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:12 smithi047 ceph-mon[36835]: scrub queued for path: / 2024-07-27T06:48:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:12 smithi047 ceph-mon[36835]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:12 smithi047 ceph-mon[36835]: scrub summary: active paths [/] 2024-07-27T06:48:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:12 smithi047 ceph-mon[36835]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:13.875 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:14 smithi077 ceph-mon[37153]: pgmap v434: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-07-27T06:48:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:14 smithi047 ceph-mon[36835]: pgmap v434: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-07-27T06:48:15.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:14 smithi029 ceph-mon[32610]: pgmap v434: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-07-27T06:48:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:16 smithi077 ceph-mon[37153]: pgmap v435: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-07-27T06:48:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:16 smithi077 ceph-mon[37153]: scrub summary: idle 2024-07-27T06:48:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:16 smithi047 ceph-mon[36835]: pgmap v435: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-07-27T06:48:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:16 smithi047 ceph-mon[36835]: scrub summary: idle 2024-07-27T06:48:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:16 smithi029 ceph-mon[32610]: pgmap v435: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-07-27T06:48:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:16 smithi029 ceph-mon[32610]: scrub summary: idle 2024-07-27T06:48:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:18 smithi077 ceph-mon[37153]: pgmap v436: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1023 B/s wr, 14 op/s 2024-07-27T06:48:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:18 smithi047 ceph-mon[36835]: pgmap v436: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1023 B/s wr, 14 op/s 2024-07-27T06:48:19.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:18 smithi029 ceph-mon[32610]: pgmap v436: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1023 B/s wr, 14 op/s 2024-07-27T06:48:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:20 smithi077 ceph-mon[37153]: pgmap v437: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-27T06:48:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:20 smithi047 ceph-mon[36835]: pgmap v437: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-27T06:48:21.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:20 smithi029 ceph-mon[32610]: pgmap v437: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-27T06:48:22.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.615+0000 7f016fcc2640 1 -- 172.21.15.29:0/524052283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0168101bb0 msgr2=0x7f0168101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.617 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.615+0000 7f016fcc2640 1 --2- 172.21.15.29:0/524052283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0168101bb0 0x7f0168101fc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f016400a970 tx=0x7f016402b670 comp rx=0 tx=0).stop 2024-07-27T06:48:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.615+0000 7f016fcc2640 1 -- 172.21.15.29:0/524052283 shutdown_connections 2024-07-27T06:48:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.615+0000 7f016fcc2640 1 --2- 172.21.15.29:0/524052283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0168107f30 0x7f016810e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.615+0000 7f016fcc2640 1 --2- 172.21.15.29:0/524052283 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01681075e0 0x7f01681079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.615+0000 7f016fcc2640 1 --2- 172.21.15.29:0/524052283 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0168101bb0 0x7f0168101fc0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.615+0000 7f016fcc2640 1 -- 172.21.15.29:0/524052283 >> 172.21.15.29:0/524052283 conn(0x7f0168079880 msgr2=0x7f0168079c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.616+0000 7f016fcc2640 1 -- 172.21.15.29:0/524052283 shutdown_connections 2024-07-27T06:48:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.616+0000 7f016fcc2640 1 -- 172.21.15.29:0/524052283 wait complete. 2024-07-27T06:48:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.617+0000 7f016fcc2640 1 Processor -- start 2024-07-27T06:48:22.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.617+0000 7f016fcc2640 1 -- start start 2024-07-27T06:48:22.619 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.618+0000 7f016fcc2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0168101bb0 0x7f01681a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.618+0000 7f016da5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0168101bb0 0x7f01681a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.618+0000 7f016da5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0168101bb0 0x7f01681a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60756/0 (socket says 172.21.15.29:60756) 2024-07-27T06:48:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.618+0000 7f016da5e640 1 -- 172.21.15.29:0/3408495746 learned_addr learned my addr 172.21.15.29:0/3408495746 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.618+0000 7f016fcc2640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01681075e0 0x7f01681a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.618+0000 7f016fcc2640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0168107f30 0x7f01681a5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.619+0000 7f016fcc2640 1 -- 172.21.15.29:0/3408495746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f01681113c0 con 0x7f0168101bb0 2024-07-27T06:48:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.619+0000 7f016fcc2640 1 -- 172.21.15.29:0/3408495746 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f0168111240 con 0x7f0168107f30 2024-07-27T06:48:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.619+0000 7f016fcc2640 1 -- 172.21.15.29:0/3408495746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0168111540 con 0x7f01681075e0 2024-07-27T06:48:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.619+0000 7f016d25d640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01681075e0 0x7f01681a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.619+0000 7f016e25f640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0168107f30 0x7f01681a5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.619+0000 7f016e25f640 1 -- 172.21.15.29:0/3408495746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01681075e0 msgr2=0x7f01681a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.619+0000 7f016e25f640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01681075e0 0x7f01681a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.619+0000 7f016e25f640 1 -- 172.21.15.29:0/3408495746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0168101bb0 msgr2=0x7f01681a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.619+0000 7f016e25f640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0168101bb0 0x7f01681a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.620+0000 7f016e25f640 1 -- 172.21.15.29:0/3408495746 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01681a6580 con 0x7f0168107f30 2024-07-27T06:48:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.620+0000 7f016da5e640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0168101bb0 0x7f01681a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.620+0000 7f016e25f640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0168107f30 0x7f01681a5e80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f016000efc0 tx=0x7f016000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.620+0000 7f015effd640 1 -- 172.21.15.29:0/3408495746 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160003970 con 0x7f0168107f30 2024-07-27T06:48:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.620+0000 7f016fcc2640 1 -- 172.21.15.29:0/3408495746 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01681a6870 con 0x7f0168107f30 2024-07-27T06:48:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.620+0000 7f015effd640 1 -- 172.21.15.29:0/3408495746 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0160004020 con 0x7f0168107f30 2024-07-27T06:48:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.620+0000 7f015effd640 1 -- 172.21.15.29:0/3408495746 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016000b940 con 0x7f0168107f30 2024-07-27T06:48:22.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.620+0000 7f016fcc2640 1 -- 172.21.15.29:0/3408495746 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01681ad000 con 0x7f0168107f30 2024-07-27T06:48:22.625 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.624+0000 7f0176be8740 1 Processor -- start 2024-07-27T06:48:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.624+0000 7f015effd640 1 -- 172.21.15.29:0/3408495746 <== mon.2 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f016001d020 con 0x7f0168107f30 2024-07-27T06:48:22.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.625+0000 7f0176be8740 1 -- start start 2024-07-27T06:48:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.625+0000 7f0176be8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93e91b0 0x5575f978bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.625+0000 7f0176be8740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c02f0 0x5575f978c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.625+0000 7f0176be8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 0x5575f978c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.625+0000 7f0176be8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5575f97013b0 con 0x5575f93c84d0 2024-07-27T06:48:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.625+0000 7f0176be8740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5575f96f08b0 con 0x5575f93c02f0 2024-07-27T06:48:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.625+0000 7f0176be8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5575f96f48f0 con 0x5575f93e91b0 2024-07-27T06:48:22.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.626+0000 7f016d25d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 0x5575f978c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.626+0000 7f016da5e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c02f0 0x5575f978c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.626+0000 7f016d25d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 0x5575f978c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60762/0 (socket says 172.21.15.29:60762) 2024-07-27T06:48:22.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.626+0000 7f016d25d640 1 -- 172.21.15.29:0/531801847 learned_addr learned my addr 172.21.15.29:0/531801847 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.626+0000 7f015effd640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f01540b3dc0 0x7f01540b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.626+0000 7f015effd640 1 -- 172.21.15.29:0/3408495746 <== mon.2 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f01600e3dd0 con 0x7f0168107f30 2024-07-27T06:48:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.626+0000 7f016d25d640 1 -- 172.21.15.29:0/531801847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93e91b0 msgr2=0x5575f978bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.626+0000 7f016d25d640 1 --2- 172.21.15.29:0/531801847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93e91b0 0x5575f978bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.626+0000 7f016d25d640 1 -- 172.21.15.29:0/531801847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c02f0 msgr2=0x5575f978c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.626+0000 7f016d25d640 1 --2- 172.21.15.29:0/531801847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c02f0 0x5575f978c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.626+0000 7f016d25d640 1 -- 172.21.15.29:0/531801847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575f96979a0 con 0x5575f93c84d0 2024-07-27T06:48:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.627+0000 7f016e25f640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f01540b3dc0 0x7f01540b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.627+0000 7f016d25d640 1 --2- 172.21.15.29:0/531801847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 0x5575f978c760 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f015800bcd0 tx=0x7f01580127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c5742b7f872644 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.627+0000 7f0153fff640 1 -- 172.21.15.29:0/531801847 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158005220 con 0x5575f93c84d0 2024-07-27T06:48:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.627+0000 7f0153fff640 1 -- 172.21.15.29:0/531801847 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0158004c50 con 0x5575f93c84d0 2024-07-27T06:48:22.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.627+0000 7f0153fff640 1 -- 172.21.15.29:0/531801847 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158010870 con 0x5575f93c84d0 2024-07-27T06:48:22.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.628+0000 7f0176be8740 1 -- 172.21.15.29:0/531801847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 msgr2=0x5575f978c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.628+0000 7f016e25f640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f01540b3dc0 0x7f01540b62d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0160005fd0 tx=0x7f0160009900 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.628+0000 7f0176be8740 1 --2- 172.21.15.29:0/531801847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 0x5575f978c760 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f015800bcd0 tx=0x7f01580127c0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.628+0000 7f0176be8740 1 -- 172.21.15.29:0/531801847 shutdown_connections 2024-07-27T06:48:22.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.628+0000 7f0176be8740 1 --2- 172.21.15.29:0/531801847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 0x5575f978c760 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.628+0000 7f0176be8740 1 --2- 172.21.15.29:0/531801847 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c02f0 0x5575f978c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.628+0000 7f0176be8740 1 --2- 172.21.15.29:0/531801847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93e91b0 0x5575f978bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.628+0000 7f0176be8740 1 -- 172.21.15.29:0/531801847 >> 172.21.15.29:0/531801847 conn(0x5575f93dd300 msgr2=0x5575f9789bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.629+0000 7f0176be8740 1 -- 172.21.15.29:0/531801847 shutdown_connections 2024-07-27T06:48:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.629+0000 7f0176be8740 1 -- 172.21.15.29:0/531801847 wait complete. 2024-07-27T06:48:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.629+0000 7f0176be8740 1 -- start start 2024-07-27T06:48:22.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.629+0000 7f0176be8740 1 Processor -- start 2024-07-27T06:48:22.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.630+0000 7f0176be8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f9797770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.630+0000 7f0176be8740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c84d0 0x5575f9797cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.630+0000 7f0176be8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93e91b0 0x5575f97a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f016da5e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c84d0 0x5575f9797cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f016da5e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c84d0 0x5575f9797cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.29:55274/0 (socket says 172.21.15.29:55274) 2024-07-27T06:48:22.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f016d25d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f9797770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f016d25d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f9797770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:32968/0 (socket says 172.21.15.29:32968) 2024-07-27T06:48:22.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f016da5e640 1 -- 172.21.15.29:0/3045427673 learned_addr learned my addr 172.21.15.29:0/3045427673 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f0176be8740 1 -- 172.21.15.29:0/3045427673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5575f962aab0 con 0x5575f93e91b0 2024-07-27T06:48:22.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f0176be8740 1 -- 172.21.15.29:0/3045427673 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5575f9678cf0 con 0x5575f93c84d0 2024-07-27T06:48:22.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f0176be8740 1 -- 172.21.15.29:0/3045427673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5575f979a500 con 0x5575f93c02f0 2024-07-27T06:48:22.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f016d25d640 1 --2- 172.21.15.29:0/3045427673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93e91b0 0x5575f97a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f016d25d640 1 -- 172.21.15.29:0/3045427673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c84d0 msgr2=0x5575f9797cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f016d25d640 1 --2- 172.21.15.29:0/3045427673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c84d0 0x5575f9797cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f016d25d640 1 -- 172.21.15.29:0/3045427673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93e91b0 msgr2=0x5575f97a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f016d25d640 1 --2- 172.21.15.29:0/3045427673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93e91b0 0x5575f97a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.631+0000 7f016d25d640 1 -- 172.21.15.29:0/3045427673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575f9454790 con 0x5575f93c02f0 2024-07-27T06:48:22.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.632+0000 7f0176be8740 10 client.0 _ll_register_callbacks cb 0x5575f96ea9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:22.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.632+0000 7f0176be8740 1 -- 172.21.15.29:0/3045427673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0158004c50 con 0x5575f93c02f0 2024-07-27T06:48:22.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.632+0000 7f016d25d640 1 --2- 172.21.15.29:0/3045427673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f9797770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0158001eb0 tx=0x7f0158010450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4645617150ec4c57 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.632+0000 7f01517fa640 10 client.25615 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:48:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.632+0000 7f01517fa640 1 -- 172.21.15.29:0/3045427673 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158004020 con 0x5575f93c02f0 2024-07-27T06:48:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.632+0000 7f01517fa640 1 -- 172.21.15.29:0/3045427673 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f01580041c0 con 0x5575f93c02f0 2024-07-27T06:48:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.632+0000 7f01517fa640 1 -- 172.21.15.29:0/3045427673 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0158004400 con 0x5575f93c02f0 2024-07-27T06:48:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.632+0000 7f0176be8740 10 client.25615 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.632+0000 7f0176be8740 1 -- 172.21.15.29:0/3045427673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5575f94cd620 con 0x5575f93c02f0 2024-07-27T06:48:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.633+0000 7f01517fa640 1 -- 172.21.15.29:0/3045427673 <== mon.1 v2:172.21.15.77:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0158022070 con 0x5575f93c02f0 2024-07-27T06:48:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.633+0000 7f0176be8740 10 client.25615 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.633+0000 7f0176be8740 10 client.25615 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.633+0000 7f0176be8740 20 client.25615 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.633+0000 7f0176be8740 1 --2- 172.21.15.29:0/3045427673 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5575f93a7d80 0x5575f97a9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.633+0000 7f0176be8740 4 client.25615 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:22.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.633+0000 7f0176be8740 1 -- 172.21.15.29:0/3045427673 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575f9703300 con 0x5575f93a7d80 2024-07-27T06:48:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.634+0000 7f016da5e640 1 --2- 172.21.15.29:0/3045427673 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5575f93a7d80 0x5575f97a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.634+0000 7f01517fa640 10 client.25615 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.634+0000 7f016da5e640 1 --2- 172.21.15.29:0/3045427673 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5575f93a7d80 0x5575f97a9320 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e93261ba30962f1a server_cookie=3038475f3ccaf338 in_seq=0 out_seq=0 2024-07-27T06:48:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.635+0000 7f01517fa640 1 -- 172.21.15.29:0/3045427673 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5575f9703300 con 0x5575f93a7d80 2024-07-27T06:48:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.635+0000 7f01517fa640 10 client.25615 handle_command_reply: tid=0 2024-07-27T06:48:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.635+0000 7f0176be8740 1 client.25615 shutdown 2024-07-27T06:48:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.635+0000 7f0176be8740 1 -- 172.21.15.29:0/3045427673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 msgr2=0x5575f9797770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.635+0000 7f0176be8740 1 --2- 172.21.15.29:0/3045427673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f9797770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0158001eb0 tx=0x7f0158010450 comp rx=0 tx=0).stop 2024-07-27T06:48:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.635+0000 7f0176be8740 1 -- 172.21.15.29:0/3045427673 shutdown_connections 2024-07-27T06:48:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.635+0000 7f0176be8740 1 --2- 172.21.15.29:0/3045427673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93e91b0 0x5575f97a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.636+0000 7f0176be8740 1 --2- 172.21.15.29:0/3045427673 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c84d0 0x5575f9797cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.636+0000 7f0176be8740 1 --2- 172.21.15.29:0/3045427673 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5575f93a7d80 0x5575f97a9320 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.636+0000 7f0176be8740 1 --2- 172.21.15.29:0/3045427673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f9797770 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.636+0000 7f01517fa640 0 client.25615 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.636+0000 7f0176be8740 1 -- 172.21.15.29:0/3045427673 >> 172.21.15.29:0/3045427673 conn(0x5575f93dd300 msgr2=0x5575f978cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.636+0000 7f0176be8740 1 -- 172.21.15.29:0/3045427673 shutdown_connections 2024-07-27T06:48:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.636+0000 7f0176be8740 1 -- 172.21.15.29:0/3045427673 wait complete. 2024-07-27T06:48:22.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.636+0000 7f0176be8740 20 client.25615 trim_cache size 0 max 16384 2024-07-27T06:48:22.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.654+0000 7f0176be8740 1 Processor -- start 2024-07-27T06:48:22.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.654+0000 7f0176be8740 1 -- start start 2024-07-27T06:48:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.654+0000 7f0176be8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93a7d80 0x5575f9799440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.654+0000 7f0176be8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f9797a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.654+0000 7f0176be8740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c84d0 0x5575f97aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.654+0000 7f0176be8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f01681b5640 con 0x5575f93a7d80 2024-07-27T06:48:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.654+0000 7f0176be8740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f01681b7180 con 0x5575f93c84d0 2024-07-27T06:48:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.654+0000 7f0176be8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f01681b5d10 con 0x5575f93c02f0 2024-07-27T06:48:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.655+0000 7f016d25d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93a7d80 0x5575f9799440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.655+0000 7f016d25d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93a7d80 0x5575f9799440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60780/0 (socket says 172.21.15.29:60780) 2024-07-27T06:48:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.655+0000 7f016d25d640 1 -- 172.21.15.29:0/2906426693 learned_addr learned my addr 172.21.15.29:0/2906426693 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.655+0000 7f016d25d640 1 --2- 172.21.15.29:0/2906426693 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c84d0 0x5575f97aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.655+0000 7f016d25d640 1 -- 172.21.15.29:0/2906426693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 msgr2=0x5575f9797a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.655+0000 7f016d25d640 1 --2- 172.21.15.29:0/2906426693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f9797a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.655+0000 7f016d25d640 1 -- 172.21.15.29:0/2906426693 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c84d0 msgr2=0x5575f97aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.655+0000 7f016d25d640 1 --2- 172.21.15.29:0/2906426693 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c84d0 0x5575f97aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.655+0000 7f016d25d640 1 -- 172.21.15.29:0/2906426693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575f978e010 con 0x5575f93a7d80 2024-07-27T06:48:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.655+0000 7f016d25d640 1 --2- 172.21.15.29:0/2906426693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93a7d80 0x5575f9799440 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f01580048e0 tx=0x7f015801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7450080a9369f69 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.656+0000 7f01537fe640 1 -- 172.21.15.29:0/2906426693 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158038070 con 0x5575f93a7d80 2024-07-27T06:48:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.656+0000 7f01537fe640 1 -- 172.21.15.29:0/2906426693 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0158004c50 con 0x5575f93a7d80 2024-07-27T06:48:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.656+0000 7f01537fe640 1 -- 172.21.15.29:0/2906426693 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01580289e0 con 0x5575f93a7d80 2024-07-27T06:48:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.657+0000 7f0176be8740 1 -- 172.21.15.29:0/2906426693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93a7d80 msgr2=0x5575f9799440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.657+0000 7f0176be8740 1 --2- 172.21.15.29:0/2906426693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93a7d80 0x5575f9799440 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f01580048e0 tx=0x7f015801f000 comp rx=0 tx=0).stop 2024-07-27T06:48:22.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.657+0000 7f0176be8740 1 -- 172.21.15.29:0/2906426693 shutdown_connections 2024-07-27T06:48:22.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.657+0000 7f0176be8740 1 --2- 172.21.15.29:0/2906426693 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93c84d0 0x5575f97aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.657+0000 7f0176be8740 1 --2- 172.21.15.29:0/2906426693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f9797a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.657+0000 7f0176be8740 1 --2- 172.21.15.29:0/2906426693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93a7d80 0x5575f9799440 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.657+0000 7f0176be8740 1 -- 172.21.15.29:0/2906426693 >> 172.21.15.29:0/2906426693 conn(0x5575f93dd300 msgr2=0x5575f9790da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.657+0000 7f0176be8740 1 -- 172.21.15.29:0/2906426693 shutdown_connections 2024-07-27T06:48:22.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.657+0000 7f0176be8740 1 -- 172.21.15.29:0/2906426693 wait complete. 2024-07-27T06:48:22.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.658+0000 7f0176be8740 1 -- start start 2024-07-27T06:48:22.660 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.658+0000 7f0176be8740 1 Processor -- start 2024-07-27T06:48:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.658+0000 7f0176be8740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93a7d80 0x5575f979fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.658+0000 7f0176be8740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f979c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f0176be8740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 0x5575f979cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f0176be8740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f01681b6ab0 con 0x5575f93c84d0 2024-07-27T06:48:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f0176be8740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f01681b63e0 con 0x5575f93a7d80 2024-07-27T06:48:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f0176be8740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f01681b3620 con 0x5575f93c02f0 2024-07-27T06:48:22.662 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f0176be8740 10 client.0 _ll_register_callbacks cb 0x7f016400a7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f016d25d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 0x5575f979cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f016d25d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 0x5575f979cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60784/0 (socket says 172.21.15.29:60784) 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f016d25d640 1 -- 172.21.15.29:0/3246685059 learned_addr learned my addr 172.21.15.29:0/3246685059 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f016da5e640 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f979c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f016da5e640 1 -- 172.21.15.29:0/3246685059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93a7d80 msgr2=0x5575f979fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f016da5e640 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93a7d80 0x5575f979fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f016da5e640 1 -- 172.21.15.29:0/3246685059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 msgr2=0x5575f979cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f016da5e640 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 0x5575f979cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f016da5e640 1 -- 172.21.15.29:0/3246685059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575f979d420 con 0x5575f93c02f0 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.659+0000 7f016d25d640 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 0x5575f979cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.660+0000 7f016da5e640 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f979c750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x5575f97a9a90 tx=0x7f01640328d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89095781647ce80b server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.660+0000 7f01517fa640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:48:22.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.660+0000 7f01517fa640 1 -- 172.21.15.29:0/3246685059 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0164032d00 con 0x5575f93c02f0 2024-07-27T06:48:22.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.660+0000 7f01517fa640 1 -- 172.21.15.29:0/3246685059 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f0164005250 con 0x5575f93c02f0 2024-07-27T06:48:22.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.660+0000 7f0176be8740 1 -- 172.21.15.29:0/3246685059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0158004c50 con 0x5575f93c02f0 2024-07-27T06:48:22.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.660+0000 7f01517fa640 1 -- 172.21.15.29:0/3246685059 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0164057050 con 0x5575f93c02f0 2024-07-27T06:48:22.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.660+0000 7f0176be8740 10 client.25618 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:22.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.660+0000 7f0176be8740 1 -- 172.21.15.29:0/3246685059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5575f9810a80 con 0x5575f93c02f0 2024-07-27T06:48:22.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.661+0000 7f01517fa640 1 -- 172.21.15.29:0/3246685059 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0158004c50 con 0x5575f93c02f0 2024-07-27T06:48:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.661+0000 7f01517fa640 1 -- 172.21.15.29:0/3246685059 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f016400b040 con 0x5575f93c02f0 2024-07-27T06:48:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.661+0000 7f0176be8740 10 client.25618 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.661+0000 7f0176be8740 10 client.25618 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.661+0000 7f0176be8740 20 client.25618 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.661+0000 7f0176be8740 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5575f93e91b0 0x5575f9813290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.661+0000 7f0176be8740 4 client.25618 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-27T06:48:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.661+0000 7f0176be8740 1 -- 172.21.15.29:0/3246685059 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575f96feda0 con 0x5575f93e91b0 2024-07-27T06:48:22.665 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.662+0000 7f016d25d640 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5575f93e91b0 0x5575f9813290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.662+0000 7f016d25d640 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5575f93e91b0 0x5575f9813290 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d61449bd439cb55c server_cookie=59b86778cc42369b in_seq=0 out_seq=0 2024-07-27T06:48:22.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.662+0000 7f01517fa640 10 client.25618 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:22.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.666+0000 7f01517fa640 1 -- 172.21.15.29:0/3246685059 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5575f96feda0 con 0x5575f93e91b0 2024-07-27T06:48:22.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.666+0000 7f01517fa640 10 client.25618 handle_command_reply: tid=0 2024-07-27T06:48:22.668 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.666+0000 7f0176be8740 1 client.25618 shutdown 2024-07-27T06:48:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.667+0000 7f0176be8740 1 -- 172.21.15.29:0/3246685059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 msgr2=0x5575f979c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.667+0000 7f0176be8740 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f979c750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x5575f97a9a90 tx=0x7f01640328d0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.667+0000 7f0176be8740 1 -- 172.21.15.29:0/3246685059 shutdown_connections 2024-07-27T06:48:22.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.667+0000 7f0176be8740 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575f93c84d0 0x5575f979cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.667+0000 7f0176be8740 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5575f93e91b0 0x5575f9813290 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.667+0000 7f01517fa640 0 client.25618 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:22.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.667+0000 7f0176be8740 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575f93c02f0 0x5575f979c750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.668+0000 7f0176be8740 1 --2- 172.21.15.29:0/3246685059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5575f93a7d80 0x5575f979fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.668+0000 7f0176be8740 1 -- 172.21.15.29:0/3246685059 >> 172.21.15.29:0/3246685059 conn(0x5575f93dd300 msgr2=0x5575f9799130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.668+0000 7f0176be8740 1 -- 172.21.15.29:0/3246685059 shutdown_connections 2024-07-27T06:48:22.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.668+0000 7f0176be8740 1 -- 172.21.15.29:0/3246685059 wait complete. 2024-07-27T06:48:22.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.668+0000 7f0176be8740 20 client.25618 trim_cache size 0 max 16384 2024-07-27T06:48:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-07-27T06:48:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-07-27T06:48:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-07-27T06:48:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-07-27T06:48:22.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.669+0000 7f016fcc2640 1 -- 172.21.15.29:0/3408495746 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f01540b3dc0 msgr2=0x7f01540b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.669+0000 7f016fcc2640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f01540b3dc0 0x7f01540b62d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0160005fd0 tx=0x7f0160009900 comp rx=0 tx=0).stop 2024-07-27T06:48:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.669+0000 7f016fcc2640 1 -- 172.21.15.29:0/3408495746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0168107f30 msgr2=0x7f01681a5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.669+0000 7f016fcc2640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0168107f30 0x7f01681a5e80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f016000efc0 tx=0x7f016000c800 comp rx=0 tx=0).stop 2024-07-27T06:48:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.669+0000 7f016fcc2640 1 -- 172.21.15.29:0/3408495746 shutdown_connections 2024-07-27T06:48:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.669+0000 7f016fcc2640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0168107f30 0x7f01681a5e80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.669+0000 7f016fcc2640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f01540b3dc0 0x7f01540b62d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.669+0000 7f016fcc2640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01681075e0 0x7f01681a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.669+0000 7f016fcc2640 1 --2- 172.21.15.29:0/3408495746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0168101bb0 0x7f01681a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.669+0000 7f016fcc2640 1 -- 172.21.15.29:0/3408495746 >> 172.21.15.29:0/3408495746 conn(0x7f0168079880 msgr2=0x7f0168105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.669+0000 7f016fcc2640 1 -- 172.21.15.29:0/3408495746 shutdown_connections 2024-07-27T06:48:22.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:22.669+0000 7f016fcc2640 1 -- 172.21.15.29:0/3408495746 wait complete. 2024-07-27T06:48:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.686+0000 7f53a06e0640 1 -- 172.21.15.29:0/1931603177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5398107df0 msgr2=0x7f5398108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.686+0000 7f53a06e0640 1 --2- 172.21.15.29:0/1931603177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5398107df0 0x7f5398108200 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f538800b9f0 tx=0x7f538802c0a0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.686+0000 7f53a06e0640 1 -- 172.21.15.29:0/1931603177 shutdown_connections 2024-07-27T06:48:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.686+0000 7f53a06e0640 1 --2- 172.21.15.29:0/1931603177 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5398108740 0x7f539810ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.686+0000 7f53a06e0640 1 --2- 172.21.15.29:0/1931603177 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5398107df0 0x7f5398108200 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.686+0000 7f53a06e0640 1 --2- 172.21.15.29:0/1931603177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53981024b0 0x7f53981028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.686+0000 7f53a06e0640 1 -- 172.21.15.29:0/1931603177 >> 172.21.15.29:0/1931603177 conn(0x7f53980791f0 msgr2=0x7f5398079600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.687+0000 7f53a06e0640 1 -- 172.21.15.29:0/1931603177 shutdown_connections 2024-07-27T06:48:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.687+0000 7f53a06e0640 1 -- 172.21.15.29:0/1931603177 wait complete. 2024-07-27T06:48:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.688+0000 7f53a06e0640 1 Processor -- start 2024-07-27T06:48:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.688+0000 7f53a06e0640 1 -- start start 2024-07-27T06:48:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.689+0000 7f53a06e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53981024b0 0x7f5398197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.689+0000 7f53a06e0640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5398107df0 0x7f5398198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.689+0000 7f53a06e0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5398108740 0x7f539819d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.689+0000 7f539e47c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53981024b0 0x7f5398197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.689+0000 7f53a06e0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5398111300 con 0x7f53981024b0 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.689+0000 7f53a06e0640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f5398111180 con 0x7f5398107df0 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.689+0000 7f539e47c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53981024b0 0x7f5398197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60804/0 (socket says 172.21.15.29:60804) 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.689+0000 7f53a06e0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5398111480 con 0x7f5398108740 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.689+0000 7f539e47c640 1 -- 172.21.15.29:0/3384337526 learned_addr learned my addr 172.21.15.29:0/3384337526 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.689+0000 7f539ec7d640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5398108740 0x7f539819d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.689+0000 7f539dc7b640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5398107df0 0x7f5398198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.690+0000 7f539e47c640 1 -- 172.21.15.29:0/3384337526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5398108740 msgr2=0x7f539819d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.690+0000 7f539e47c640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5398108740 0x7f539819d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.690+0000 7f539e47c640 1 -- 172.21.15.29:0/3384337526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5398107df0 msgr2=0x7f5398198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.690+0000 7f539e47c640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5398107df0 0x7f5398198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.690+0000 7f539e47c640 1 -- 172.21.15.29:0/3384337526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539819dae0 con 0x7f53981024b0 2024-07-27T06:48:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.690+0000 7f539e47c640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53981024b0 0x7f5398197b70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f539400e9c0 tx=0x7f539400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.690+0000 7f538f7fe640 1 -- 172.21.15.29:0/3384337526 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539400cde0 con 0x7f53981024b0 2024-07-27T06:48:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.690+0000 7f538f7fe640 1 -- 172.21.15.29:0/3384337526 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53940035f0 con 0x7f53981024b0 2024-07-27T06:48:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.690+0000 7f538f7fe640 1 -- 172.21.15.29:0/3384337526 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394010690 con 0x7f53981024b0 2024-07-27T06:48:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.691+0000 7f53a06e0640 1 -- 172.21.15.29:0/3384337526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f539819ddd0 con 0x7f53981024b0 2024-07-27T06:48:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.691+0000 7f53a06e0640 1 -- 172.21.15.29:0/3384337526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53981a4530 con 0x7f53981024b0 2024-07-27T06:48:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.695+0000 7f538f7fe640 1 -- 172.21.15.29:0/3384337526 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f5394004000 con 0x7f53981024b0 2024-07-27T06:48:22.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.695+0000 7f53a6c64740 1 Processor -- start 2024-07-27T06:48:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.696+0000 7f53a6c64740 1 -- start start 2024-07-27T06:48:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.696+0000 7f53a6c64740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db77850 0x564b0df1d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.696+0000 7f53a6c64740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db4e990 0x564b0df1da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.696+0000 7f53a6c64740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db56b70 0x564b0df1df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.696+0000 7f53a6c64740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564b0de92690 con 0x564b0db77850 2024-07-27T06:48:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.696+0000 7f53a6c64740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x564b0de81b90 con 0x564b0db4e990 2024-07-27T06:48:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.696+0000 7f53a6c64740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564b0de85bd0 con 0x564b0db56b70 2024-07-27T06:48:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.696+0000 7f539ec7d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db77850 0x564b0df1d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.696+0000 7f539ec7d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db77850 0x564b0df1d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60810/0 (socket says 172.21.15.29:60810) 2024-07-27T06:48:22.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.696+0000 7f539ec7d640 1 -- 172.21.15.29:0/2529772722 learned_addr learned my addr 172.21.15.29:0/2529772722 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.697+0000 7f538f7fe640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f53840b3d40 0x7f53840b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.697+0000 7f539dc7b640 1 --2- 172.21.15.29:0/2529772722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db56b70 0x564b0df1df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.697+0000 7f539ec7d640 1 -- 172.21.15.29:0/2529772722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db56b70 msgr2=0x564b0df1df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.697+0000 7f539ec7d640 1 --2- 172.21.15.29:0/2529772722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db56b70 0x564b0df1df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.697+0000 7f539ec7d640 1 -- 172.21.15.29:0/2529772722 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db4e990 msgr2=0x564b0df1da50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-27T06:48:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.697+0000 7f539ec7d640 1 --2- 172.21.15.29:0/2529772722 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db4e990 0x564b0df1da50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.697+0000 7f539ec7d640 1 -- 172.21.15.29:0/2529772722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b0de0da10 con 0x564b0db77850 2024-07-27T06:48:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.697+0000 7f538f7fe640 1 -- 172.21.15.29:0/3384337526 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f53940e4250 con 0x7f53981024b0 2024-07-27T06:48:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.698+0000 7f539ec7d640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f53840b3d40 0x7f53840b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.698+0000 7f539ec7d640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f53840b3d40 0x7f53840b6250 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5390009fd0 tx=0x7f5390013040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.699+0000 7f539ec7d640 1 --2- 172.21.15.29:0/2529772722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db77850 0x564b0df1d510 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f5390009f80 tx=0x7f53900118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5e6720ae11784e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.699+0000 7f538cff9640 1 -- 172.21.15.29:0/2529772722 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5390021070 con 0x564b0db77850 2024-07-27T06:48:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.699+0000 7f538cff9640 1 -- 172.21.15.29:0/2529772722 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5390003e70 con 0x564b0db77850 2024-07-27T06:48:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.699+0000 7f538cff9640 1 -- 172.21.15.29:0/2529772722 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539001a6e0 con 0x564b0db77850 2024-07-27T06:48:22.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.700+0000 7f53a6c64740 1 -- 172.21.15.29:0/2529772722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db77850 msgr2=0x564b0df1d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.700+0000 7f53a6c64740 1 --2- 172.21.15.29:0/2529772722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db77850 0x564b0df1d510 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f5390009f80 tx=0x7f53900118d0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.700+0000 7f53a6c64740 1 -- 172.21.15.29:0/2529772722 shutdown_connections 2024-07-27T06:48:22.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.700+0000 7f53a6c64740 1 --2- 172.21.15.29:0/2529772722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db56b70 0x564b0df1df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.700+0000 7f53a6c64740 1 --2- 172.21.15.29:0/2529772722 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db4e990 0x564b0df1da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.701+0000 7f53a6c64740 1 --2- 172.21.15.29:0/2529772722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db77850 0x564b0df1d510 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.701+0000 7f53a6c64740 1 -- 172.21.15.29:0/2529772722 >> 172.21.15.29:0/2529772722 conn(0x564b0db6bc40 msgr2=0x564b0df1b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.701+0000 7f53a6c64740 1 -- 172.21.15.29:0/2529772722 shutdown_connections 2024-07-27T06:48:22.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.701+0000 7f53a6c64740 1 -- 172.21.15.29:0/2529772722 wait complete. 2024-07-27T06:48:22.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.702+0000 7f53a6c64740 1 -- start start 2024-07-27T06:48:22.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.702+0000 7f53a6c64740 1 Processor -- start 2024-07-27T06:48:22.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.703+0000 7f53a6c64740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db4e990 0x564b0df28d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.703+0000 7f539e47c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db4e990 0x564b0df28d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.703+0000 7f539e47c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db4e990 0x564b0df28d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60818/0 (socket says 172.21.15.29:60818) 2024-07-27T06:48:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.703+0000 7f53a6c64740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db56b70 0x564b0df292d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.703+0000 7f539e47c640 1 -- 172.21.15.29:0/757925134 learned_addr learned my addr 172.21.15.29:0/757925134 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.703+0000 7f53a6c64740 1 --2- 172.21.15.29:0/757925134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db77850 0x564b0df34f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.703+0000 7f53a6c64740 1 -- 172.21.15.29:0/757925134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564b0dce57b0 con 0x564b0db4e990 2024-07-27T06:48:22.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.703+0000 7f53a6c64740 1 -- 172.21.15.29:0/757925134 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x564b0de24210 con 0x564b0db56b70 2024-07-27T06:48:22.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.704+0000 7f53a6c64740 1 -- 172.21.15.29:0/757925134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564b0df2bc00 con 0x564b0db77850 2024-07-27T06:48:22.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.704+0000 7f53a6c64740 10 client.0 _ll_register_callbacks cb 0x564b0de052c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:22.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.704+0000 7f539e47c640 1 -- 172.21.15.29:0/757925134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db77850 msgr2=0x564b0df34f00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-27T06:48:22.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.704+0000 7f539e47c640 1 --2- 172.21.15.29:0/757925134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db77850 0x564b0df34f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.704+0000 7f539e47c640 1 -- 172.21.15.29:0/757925134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db56b70 msgr2=0x564b0df292d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-27T06:48:22.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.704+0000 7f539e47c640 1 --2- 172.21.15.29:0/757925134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db56b70 0x564b0df292d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.704+0000 7f539e47c640 1 -- 172.21.15.29:0/757925134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b0de35710 con 0x564b0db4e990 2024-07-27T06:48:22.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.704+0000 7f539e47c640 1 --2- 172.21.15.29:0/757925134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db4e990 0x564b0df28d30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f539400cfd0 tx=0x7f53940043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd9b7b929f31654c server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.704+0000 7f5381ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:22.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.705+0000 7f5381ffb640 1 -- 172.21.15.29:0/757925134 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53940c5060 con 0x564b0db4e990 2024-07-27T06:48:22.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.705+0000 7f5381ffb640 1 -- 172.21.15.29:0/757925134 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f53940a7100 con 0x564b0db4e990 2024-07-27T06:48:22.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.705+0000 7f53a6c64740 1 -- 172.21.15.29:0/757925134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564b0de945e0 con 0x564b0db4e990 2024-07-27T06:48:22.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.705+0000 7f5381ffb640 1 -- 172.21.15.29:0/757925134 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f53940ca050 con 0x564b0db4e990 2024-07-27T06:48:22.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.705+0000 7f53a6c64740 10 client.15930 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:22.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.705+0000 7f53a6c64740 1 -- 172.21.15.29:0/757925134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564b0dbf1ba0 con 0x564b0db4e990 2024-07-27T06:48:22.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.705+0000 7f5381ffb640 1 -- 172.21.15.29:0/757925134 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53940ab570 con 0x564b0db4e990 2024-07-27T06:48:22.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.705+0000 7f5381ffb640 1 -- 172.21.15.29:0/757925134 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f53940b3b20 con 0x564b0db4e990 2024-07-27T06:48:22.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.706+0000 7f53a6c64740 10 client.15930 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:22.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.706+0000 7f53a6c64740 10 client.15930 resolve_mds: validated gid 24670 aka daemon mds.i 2024-07-27T06:48:22.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.706+0000 7f53a6c64740 20 client.15930 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:22.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.706+0000 7f53a6c64740 1 --2- 172.21.15.29:0/757925134 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564b0db361d0 0x564b0df3ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.706+0000 7f53a6c64740 4 client.15930 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:22.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.706+0000 7f53a6c64740 1 -- 172.21.15.29:0/757925134 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b0de945e0 con 0x564b0db361d0 2024-07-27T06:48:22.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.706+0000 7f539dc7b640 1 --2- 172.21.15.29:0/757925134 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564b0db361d0 0x564b0df3ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.708+0000 7f539dc7b640 1 --2- 172.21.15.29:0/757925134 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564b0db361d0 0x564b0df3ae20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=891c7373b14fd212 server_cookie=4bf549a579984eda in_seq=0 out_seq=0 2024-07-27T06:48:22.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.708+0000 7f5381ffb640 10 client.15930 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:22.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.709+0000 7f5381ffb640 1 -- 172.21.15.29:0/757925134 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564b0de945e0 con 0x564b0db361d0 2024-07-27T06:48:22.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.709+0000 7f5381ffb640 10 client.15930 handle_command_reply: tid=0 2024-07-27T06:48:22.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.709+0000 7f53a6c64740 1 client.15930 shutdown 2024-07-27T06:48:22.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.709+0000 7f53a6c64740 1 -- 172.21.15.29:0/757925134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db4e990 msgr2=0x564b0df28d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.709+0000 7f53a6c64740 1 --2- 172.21.15.29:0/757925134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db4e990 0x564b0df28d30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f539400cfd0 tx=0x7f53940043b0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.709+0000 7f53a6c64740 1 -- 172.21.15.29:0/757925134 shutdown_connections 2024-07-27T06:48:22.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.709+0000 7f53a6c64740 1 --2- 172.21.15.29:0/757925134 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564b0db361d0 0x564b0df3ae20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.710+0000 7f53a6c64740 1 --2- 172.21.15.29:0/757925134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db77850 0x564b0df34f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.710+0000 7f5381ffb640 0 client.15930 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:22.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.710+0000 7f53a6c64740 1 --2- 172.21.15.29:0/757925134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db56b70 0x564b0df292d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.710+0000 7f53a6c64740 1 --2- 172.21.15.29:0/757925134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db4e990 0x564b0df28d30 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.710+0000 7f53a6c64740 1 -- 172.21.15.29:0/757925134 >> 172.21.15.29:0/757925134 conn(0x564b0db6bc40 msgr2=0x564b0df206f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.710+0000 7f53a6c64740 1 -- 172.21.15.29:0/757925134 shutdown_connections 2024-07-27T06:48:22.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.710+0000 7f53a6c64740 1 -- 172.21.15.29:0/757925134 wait complete. 2024-07-27T06:48:22.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.711+0000 7f53a6c64740 20 client.15930 trim_cache size 0 max 16384 2024-07-27T06:48:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.748+0000 7f53a6c64740 1 Processor -- start 2024-07-27T06:48:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.748+0000 7f53a6c64740 1 -- start start 2024-07-27T06:48:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.749+0000 7f53a6c64740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 0x564b0df1a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.749+0000 7f53a6c64740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db4e990 0x564b0df2a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.749+0000 7f539e47c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 0x564b0df1a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.749+0000 7f539e47c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 0x564b0df1a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60822/0 (socket says 172.21.15.29:60822) 2024-07-27T06:48:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.749+0000 7f539e47c640 1 -- 172.21.15.29:0/3175603891 learned_addr learned my addr 172.21.15.29:0/3175603891 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.749+0000 7f53a6c64740 1 --2- 172.21.15.29:0/3175603891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db56b70 0x564b0df34d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.749+0000 7f539dc7b640 1 --2- 172.21.15.29:0/3175603891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db4e990 0x564b0df2a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.750+0000 7f53a6c64740 1 -- 172.21.15.29:0/3175603891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f53981acdb0 con 0x564b0db361d0 2024-07-27T06:48:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.750+0000 7f53a6c64740 1 -- 172.21.15.29:0/3175603891 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f53981ae8f0 con 0x564b0db4e990 2024-07-27T06:48:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.750+0000 7f53a6c64740 1 -- 172.21.15.29:0/3175603891 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f53981ad480 con 0x564b0db56b70 2024-07-27T06:48:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.750+0000 7f539ec7d640 1 --2- 172.21.15.29:0/3175603891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db56b70 0x564b0df34d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.750+0000 7f539e47c640 1 -- 172.21.15.29:0/3175603891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db56b70 msgr2=0x564b0df34d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.750+0000 7f539e47c640 1 --2- 172.21.15.29:0/3175603891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db56b70 0x564b0df34d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.750+0000 7f539e47c640 1 -- 172.21.15.29:0/3175603891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db4e990 msgr2=0x564b0df2a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.750+0000 7f539e47c640 1 --2- 172.21.15.29:0/3175603891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db4e990 0x564b0df2a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.750+0000 7f539e47c640 1 -- 172.21.15.29:0/3175603891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b0df29730 con 0x564b0db361d0 2024-07-27T06:48:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.751+0000 7f539e47c640 1 --2- 172.21.15.29:0/3175603891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 0x564b0df1a3f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f53940aab70 tx=0x7f53940af340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecddfd3010f837e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.751+0000 7f5383fff640 1 -- 172.21.15.29:0/3175603891 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53940aae70 con 0x564b0db361d0 2024-07-27T06:48:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.751+0000 7f5383fff640 1 -- 172.21.15.29:0/3175603891 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53940af470 con 0x564b0db361d0 2024-07-27T06:48:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.751+0000 7f5383fff640 1 -- 172.21.15.29:0/3175603891 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53940b3a90 con 0x564b0db361d0 2024-07-27T06:48:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.753+0000 7f53a6c64740 1 -- 172.21.15.29:0/3175603891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 msgr2=0x564b0df1a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.753+0000 7f53a6c64740 1 --2- 172.21.15.29:0/3175603891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 0x564b0df1a3f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f53940aab70 tx=0x7f53940af340 comp rx=0 tx=0).stop 2024-07-27T06:48:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.753+0000 7f53a6c64740 1 -- 172.21.15.29:0/3175603891 shutdown_connections 2024-07-27T06:48:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.753+0000 7f53a6c64740 1 --2- 172.21.15.29:0/3175603891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db56b70 0x564b0df34d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.753+0000 7f53a6c64740 1 --2- 172.21.15.29:0/3175603891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db4e990 0x564b0df2a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.753+0000 7f53a6c64740 1 --2- 172.21.15.29:0/3175603891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 0x564b0df1a3f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.753+0000 7f53a6c64740 1 -- 172.21.15.29:0/3175603891 >> 172.21.15.29:0/3175603891 conn(0x564b0db6bc40 msgr2=0x564b0df2a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.753+0000 7f53a6c64740 1 -- 172.21.15.29:0/3175603891 shutdown_connections 2024-07-27T06:48:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.753+0000 7f53a6c64740 1 -- 172.21.15.29:0/3175603891 wait complete. 2024-07-27T06:48:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.754+0000 7f53a6c64740 1 -- start start 2024-07-27T06:48:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.754+0000 7f53a6c64740 1 Processor -- start 2024-07-27T06:48:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.755+0000 7f53a6c64740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 0x564b0df2d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.755+0000 7f539e47c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 0x564b0df2d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.755+0000 7f539e47c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 0x564b0df2d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60832/0 (socket says 172.21.15.29:60832) 2024-07-27T06:48:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.755+0000 7f539e47c640 1 -- 172.21.15.29:0/1528247149 learned_addr learned my addr 172.21.15.29:0/1528247149 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.755+0000 7f53a6c64740 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db4e990 0x564b0df2d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.755+0000 7f53a6c64740 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db56b70 0x564b0dfa25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.756+0000 7f53a6c64740 1 -- 172.21.15.29:0/1528247149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f53981ae220 con 0x564b0db361d0 2024-07-27T06:48:22.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.756+0000 7f53a6c64740 1 -- 172.21.15.29:0/1528247149 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f53981adb50 con 0x564b0db56b70 2024-07-27T06:48:22.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.756+0000 7f53a6c64740 1 -- 172.21.15.29:0/1528247149 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f53981aad90 con 0x564b0db4e990 2024-07-27T06:48:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.756+0000 7f539dc7b640 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db4e990 0x564b0df2d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.756+0000 7f539ec7d640 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db56b70 0x564b0dfa25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.756+0000 7f539e47c640 1 -- 172.21.15.29:0/1528247149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db4e990 msgr2=0x564b0df2d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.756+0000 7f539e47c640 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db4e990 0x564b0df2d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.756+0000 7f539e47c640 1 -- 172.21.15.29:0/1528247149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db56b70 msgr2=0x564b0dfa25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.756+0000 7f539e47c640 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db56b70 0x564b0dfa25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.756+0000 7f539e47c640 1 -- 172.21.15.29:0/1528247149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b0dfa2ae0 con 0x564b0db361d0 2024-07-27T06:48:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.757+0000 7f53a6c64740 10 client.0 _ll_register_callbacks cb 0x7f53940aa8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.757+0000 7f53a6c64740 1 -- 172.21.15.29:0/1528247149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53940af470 con 0x564b0db361d0 2024-07-27T06:48:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.757+0000 7f539e47c640 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 0x564b0df2d400 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f539400c6a0 tx=0x7f5394003e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c652ec40c2e0a810 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.757+0000 7f5381ffb640 10 client.15936 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.757+0000 7f5381ffb640 1 -- 172.21.15.29:0/1528247149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53940b5070 con 0x564b0db361d0 2024-07-27T06:48:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.758+0000 7f5381ffb640 1 -- 172.21.15.29:0/1528247149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f53940aa2e0 con 0x564b0db361d0 2024-07-27T06:48:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.758+0000 7f5381ffb640 1 -- 172.21.15.29:0/1528247149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f53940aa5e0 con 0x564b0db361d0 2024-07-27T06:48:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.758+0000 7f53a6c64740 10 client.15936 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.758+0000 7f53a6c64740 1 -- 172.21.15.29:0/1528247149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=29}) -- 0x564b0dfa2ae0 con 0x564b0db361d0 2024-07-27T06:48:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.758+0000 7f5381ffb640 1 -- 172.21.15.29:0/1528247149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f539408a070 con 0x564b0db361d0 2024-07-27T06:48:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.758+0000 7f53a6c64740 10 client.15936 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.758+0000 7f53a6c64740 10 client.15936 resolve_mds: validated gid 24670 aka daemon mds.i 2024-07-27T06:48:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.758+0000 7f53a6c64740 20 client.15936 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.758+0000 7f53a6c64740 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564b0db77850 0x564b0dfa5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.758+0000 7f53a6c64740 4 client.15936 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}] 2024-07-27T06:48:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.758+0000 7f53a6c64740 1 -- 172.21.15.29:0/1528247149 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}) -- 0x564b0de90080 con 0x564b0db77850 2024-07-27T06:48:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.759+0000 7f539dc7b640 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564b0db77850 0x564b0dfa5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.759+0000 7f5381ffb640 10 client.15936 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.759+0000 7f539dc7b640 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564b0db77850 0x564b0dfa5cb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65d7c53a5a4d6179 server_cookie=e0896f743bc64bfc in_seq=0 out_seq=0 2024-07-27T06:48:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.769+0000 7f5381ffb640 1 -- 172.21.15.29:0/1528247149 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x564b0de90080 con 0x564b0db77850 2024-07-27T06:48:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.769+0000 7f5381ffb640 10 client.15936 handle_command_reply: tid=0 2024-07-27T06:48:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.769+0000 7f53a6c64740 1 client.15936 shutdown 2024-07-27T06:48:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.770+0000 7f53a6c64740 1 -- 172.21.15.29:0/1528247149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 msgr2=0x564b0df2d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.770+0000 7f53a6c64740 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 0x564b0df2d400 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f539400c6a0 tx=0x7f5394003e50 comp rx=0 tx=0).stop 2024-07-27T06:48:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.770+0000 7f53a6c64740 1 -- 172.21.15.29:0/1528247149 shutdown_connections 2024-07-27T06:48:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.770+0000 7f53a6c64740 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564b0db56b70 0x564b0dfa25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.770+0000 7f53a6c64740 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564b0db77850 0x564b0dfa5cb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.770+0000 7f5381ffb640 0 client.15936 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.770+0000 7f53a6c64740 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564b0db4e990 0x564b0df2d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.770+0000 7f53a6c64740 1 --2- 172.21.15.29:0/1528247149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564b0db361d0 0x564b0df2d400 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.770+0000 7f53a6c64740 1 -- 172.21.15.29:0/1528247149 >> 172.21.15.29:0/1528247149 conn(0x564b0db6bc40 msgr2=0x564b0df2ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.770+0000 7f53a6c64740 1 -- 172.21.15.29:0/1528247149 shutdown_connections 2024-07-27T06:48:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.770+0000 7f53a6c64740 1 -- 172.21.15.29:0/1528247149 wait complete. 2024-07-27T06:48:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.771+0000 7f53a6c64740 20 client.15936 trim_cache size 0 max 16384 2024-07-27T06:48:22.774 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:48:22.774 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 29, 2024-07-27T06:48:22.774 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24670, 2024-07-27T06:48:22.774 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 2, 2024-07-27T06:48:22.774 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-07-27T06:48:22.774 INFO:teuthology.orchestra.run.smithi029.stdout: "7f206a77": { 2024-07-27T06:48:22.774 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 2, 2024-07-27T06:48:22.774 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-07-27T06:48:22.774 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-07-27T06:48:22.774 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 142.6, 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:22.775 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:22.776 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:22.776 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:48:22.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.773+0000 7f53a06e0640 1 -- 172.21.15.29:0/3384337526 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f53840b3d40 msgr2=0x7f53840b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.773+0000 7f53a06e0640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f53840b3d40 0x7f53840b6250 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5390009fd0 tx=0x7f5390013040 comp rx=0 tx=0).stop 2024-07-27T06:48:22.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.773+0000 7f53a06e0640 1 -- 172.21.15.29:0/3384337526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53981024b0 msgr2=0x7f5398197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.773+0000 7f53a06e0640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53981024b0 0x7f5398197b70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f539400e9c0 tx=0x7f539400ee80 comp rx=0 tx=0).stop 2024-07-27T06:48:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.774+0000 7f53a06e0640 1 -- 172.21.15.29:0/3384337526 shutdown_connections 2024-07-27T06:48:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.774+0000 7f53a06e0640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f53840b3d40 0x7f53840b6250 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.774+0000 7f53a06e0640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5398108740 0x7f539819d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.774+0000 7f53a06e0640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5398107df0 0x7f5398198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.774+0000 7f53a06e0640 1 --2- 172.21.15.29:0/3384337526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53981024b0 0x7f5398197b70 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.774+0000 7f53a06e0640 1 -- 172.21.15.29:0/3384337526 >> 172.21.15.29:0/3384337526 conn(0x7f53980791f0 msgr2=0x7f5398105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.774+0000 7f53a06e0640 1 -- 172.21.15.29:0/3384337526 shutdown_connections 2024-07-27T06:48:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.774+0000 7f53a06e0640 1 -- 172.21.15.29:0/3384337526 wait complete. 2024-07-27T06:48:22.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.919+0000 7f9180884640 1 -- 172.21.15.29:0/495149366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f917c107de0 msgr2=0x7f917c1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.919+0000 7f9180884640 1 --2- 172.21.15.29:0/495149366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f917c107de0 0x7f917c1081f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9164009900 tx=0x7f916402a3f0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.920+0000 7f9180884640 1 -- 172.21.15.29:0/495149366 shutdown_connections 2024-07-27T06:48:22.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.920+0000 7f9180884640 1 --2- 172.21.15.29:0/495149366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f917c108730 0x7f917c10ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.920+0000 7f9180884640 1 --2- 172.21.15.29:0/495149366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f917c107de0 0x7f917c1081f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.920+0000 7f9180884640 1 --2- 172.21.15.29:0/495149366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917c1024a0 0x7f917c1028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.920+0000 7f9180884640 1 -- 172.21.15.29:0/495149366 >> 172.21.15.29:0/495149366 conn(0x7f917c079160 msgr2=0x7f917c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.920+0000 7f9180884640 1 -- 172.21.15.29:0/495149366 shutdown_connections 2024-07-27T06:48:22.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.920+0000 7f9180884640 1 -- 172.21.15.29:0/495149366 wait complete. 2024-07-27T06:48:22.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.921+0000 7f9180884640 1 Processor -- start 2024-07-27T06:48:22.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.921+0000 7f9180884640 1 -- start start 2024-07-27T06:48:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.922+0000 7f9180884640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f917c1024a0 0x7f917c197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.922+0000 7f9180884640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917c107de0 0x7f917c198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.922+0000 7f9180884640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f917c108730 0x7f917c19d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.922+0000 7f9180884640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f917c1112a0 con 0x7f917c107de0 2024-07-27T06:48:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.922+0000 7f9180884640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f917c111120 con 0x7f917c108730 2024-07-27T06:48:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.922+0000 7f917a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f917c1024a0 0x7f917c197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.922+0000 7f917a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f917c1024a0 0x7f917c197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:33070/0 (socket says 172.21.15.29:33070) 2024-07-27T06:48:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.922+0000 7f917a59c640 1 -- 172.21.15.29:0/4089309208 learned_addr learned my addr 172.21.15.29:0/4089309208 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.922+0000 7f9180884640 1 -- 172.21.15.29:0/4089309208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f917c111420 con 0x7f917c1024a0 2024-07-27T06:48:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.923+0000 7f917ad9d640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f917c108730 0x7f917c19d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.923+0000 7f9173fff640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917c107de0 0x7f917c198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.923+0000 7f917a59c640 1 -- 172.21.15.29:0/4089309208 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f917c108730 msgr2=0x7f917c19d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.923+0000 7f917a59c640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f917c108730 0x7f917c19d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.923+0000 7f917a59c640 1 -- 172.21.15.29:0/4089309208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917c107de0 msgr2=0x7f917c198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.923+0000 7f917a59c640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917c107de0 0x7f917c198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.923+0000 7f917a59c640 1 -- 172.21.15.29:0/4089309208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c19dbe0 con 0x7f917c1024a0 2024-07-27T06:48:22.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.923+0000 7f917a59c640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f917c1024a0 0x7f917c197c70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f916c00ea60 tx=0x7f916c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.924+0000 7f917ad9d640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f917c108730 0x7f917c19d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:48:22.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.924+0000 7f91737fe640 1 -- 172.21.15.29:0/4089309208 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c00cde0 con 0x7f917c1024a0 2024-07-27T06:48:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.924+0000 7f91737fe640 1 -- 172.21.15.29:0/4089309208 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f916c0035f0 con 0x7f917c1024a0 2024-07-27T06:48:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.924+0000 7f91737fe640 1 -- 172.21.15.29:0/4089309208 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c010690 con 0x7f917c1024a0 2024-07-27T06:48:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.924+0000 7f9180884640 1 -- 172.21.15.29:0/4089309208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f917c19ded0 con 0x7f917c1024a0 2024-07-27T06:48:22.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.924+0000 7f9180884640 1 -- 172.21.15.29:0/4089309208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f917c1a4630 con 0x7f917c1024a0 2024-07-27T06:48:22.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.928+0000 7f9186e08740 1 Processor -- start 2024-07-27T06:48:22.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.928+0000 7f9186e08740 1 -- start start 2024-07-27T06:48:22.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.928+0000 7f91737fe640 1 -- 172.21.15.29:0/4089309208 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f916c004020 con 0x7f917c1024a0 2024-07-27T06:48:22.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.928+0000 7f9186e08740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8c3070 0x55abbec68550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.928+0000 7f9186e08740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe89a1b0 0x55abbec68a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.928+0000 7f9186e08740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8a2390 0x55abbec68fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.928+0000 7f9186e08740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55abbebdeb90 con 0x55abbe89a1b0 2024-07-27T06:48:22.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.928+0000 7f9186e08740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55abbebce090 con 0x55abbe8c3070 2024-07-27T06:48:22.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.928+0000 7f9186e08740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55abbebd20d0 con 0x55abbe8a2390 2024-07-27T06:48:22.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.929+0000 7f917a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe89a1b0 0x55abbec68a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.929+0000 7f9173fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8a2390 0x55abbec68fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.929+0000 7f9173fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8a2390 0x55abbec68fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:33084/0 (socket says 172.21.15.29:33084) 2024-07-27T06:48:22.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.929+0000 7f9173fff640 1 -- 172.21.15.29:0/3157417659 learned_addr learned my addr 172.21.15.29:0/3157417659 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.929+0000 7f917ad9d640 1 --2- 172.21.15.29:0/3157417659 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8c3070 0x55abbec68550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.930+0000 7f9173fff640 1 -- 172.21.15.29:0/3157417659 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8c3070 msgr2=0x55abbec68550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.930+0000 7f9173fff640 1 --2- 172.21.15.29:0/3157417659 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8c3070 0x55abbec68550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.930+0000 7f9173fff640 1 -- 172.21.15.29:0/3157417659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe89a1b0 msgr2=0x55abbec68a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.930+0000 7f9173fff640 1 --2- 172.21.15.29:0/3157417659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe89a1b0 0x55abbec68a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.930+0000 7f91737fe640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f91680b3dc0 0x7f91680b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.930+0000 7f9173fff640 1 -- 172.21.15.29:0/3157417659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abbeb74b70 con 0x55abbe8a2390 2024-07-27T06:48:22.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.930+0000 7f917a59c640 1 --2- 172.21.15.29:0/3157417659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe89a1b0 0x55abbec68a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:22.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.930+0000 7f917ad9d640 1 --2- 172.21.15.29:0/3157417659 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8c3070 0x55abbec68550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:48:22.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.930+0000 7f91737fe640 1 -- 172.21.15.29:0/4089309208 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f916c0e4310 con 0x7f917c1024a0 2024-07-27T06:48:22.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.930+0000 7f917ad9d640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f91680b3dc0 0x7f91680b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.931+0000 7f9173fff640 1 --2- 172.21.15.29:0/3157417659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8a2390 0x55abbec68fd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9164033000 tx=0x7f9164032b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53c11c1f07375a78 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.931+0000 7f9170ff9640 1 -- 172.21.15.29:0/3157417659 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164035040 con 0x55abbe8a2390 2024-07-27T06:48:22.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.931+0000 7f917ad9d640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f91680b3dc0 0x7f91680b62d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f917400b480 tx=0x7f9174012040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.931+0000 7f9170ff9640 1 -- 172.21.15.29:0/3157417659 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9164004030 con 0x55abbe8a2390 2024-07-27T06:48:22.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.932+0000 7f9170ff9640 1 -- 172.21.15.29:0/3157417659 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91640054f0 con 0x55abbe8a2390 2024-07-27T06:48:22.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.933+0000 7f9186e08740 1 -- 172.21.15.29:0/3157417659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8a2390 msgr2=0x55abbec68fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.933+0000 7f9186e08740 1 --2- 172.21.15.29:0/3157417659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8a2390 0x55abbec68fd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9164033000 tx=0x7f9164032b20 comp rx=0 tx=0).stop 2024-07-27T06:48:22.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.933+0000 7f9186e08740 1 -- 172.21.15.29:0/3157417659 shutdown_connections 2024-07-27T06:48:22.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.933+0000 7f9186e08740 1 --2- 172.21.15.29:0/3157417659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8a2390 0x55abbec68fd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.933+0000 7f9186e08740 1 --2- 172.21.15.29:0/3157417659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe89a1b0 0x55abbec68a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.933+0000 7f9186e08740 1 --2- 172.21.15.29:0/3157417659 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8c3070 0x55abbec68550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.933+0000 7f9186e08740 1 -- 172.21.15.29:0/3157417659 >> 172.21.15.29:0/3157417659 conn(0x55abbe8b7460 msgr2=0x55abbec66440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.933+0000 7f9186e08740 1 -- 172.21.15.29:0/3157417659 shutdown_connections 2024-07-27T06:48:22.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.933+0000 7f9186e08740 1 -- 172.21.15.29:0/3157417659 wait complete. 2024-07-27T06:48:22.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.934+0000 7f9186e08740 1 -- start start 2024-07-27T06:48:22.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.934+0000 7f9186e08740 1 Processor -- start 2024-07-27T06:48:22.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.934+0000 7f9186e08740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe89a1b0 0x55abbec7c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.934+0000 7f9186e08740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8a2390 0x55abbec7cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f9186e08740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8c3070 0x55abbec88780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f9186e08740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55abbea31700 con 0x55abbe8c3070 2024-07-27T06:48:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f9186e08740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55abbeb709c0 con 0x55abbe8a2390 2024-07-27T06:48:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f9186e08740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55abbec7f450 con 0x55abbe89a1b0 2024-07-27T06:48:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f9186e08740 10 client.0 _ll_register_callbacks cb 0x55abbeb8b4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f917ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8c3070 0x55abbec88780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f917ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8c3070 0x55abbec88780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60870/0 (socket says 172.21.15.29:60870) 2024-07-27T06:48:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f917ad9d640 1 -- 172.21.15.29:0/4197274068 learned_addr learned my addr 172.21.15.29:0/4197274068 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f9173fff640 1 --2- 172.21.15.29:0/4197274068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8a2390 0x55abbec7cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f917ad9d640 1 -- 172.21.15.29:0/4197274068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe89a1b0 msgr2=0x55abbec7c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f917ad9d640 1 --2- 172.21.15.29:0/4197274068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe89a1b0 0x55abbec7c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f917ad9d640 1 -- 172.21.15.29:0/4197274068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8a2390 msgr2=0x55abbec7cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f917ad9d640 1 --2- 172.21.15.29:0/4197274068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8a2390 0x55abbec7cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.935+0000 7f917ad9d640 1 -- 172.21.15.29:0/4197274068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abbe92e650 con 0x55abbe8c3070 2024-07-27T06:48:22.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.936+0000 7f917ad9d640 1 --2- 172.21.15.29:0/4197274068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8c3070 0x55abbec88780 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f917401c760 tx=0x7f9174004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc801c7e0bc4726 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.936+0000 7f9161ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:22.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.936+0000 7f9161ffb640 1 -- 172.21.15.29:0/4197274068 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9174019570 con 0x55abbe8c3070 2024-07-27T06:48:22.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.936+0000 7f9186e08740 1 -- 172.21.15.29:0/4197274068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9164004030 con 0x55abbe8c3070 2024-07-27T06:48:22.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.936+0000 7f9161ffb640 1 -- 172.21.15.29:0/4197274068 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f9174019710 con 0x55abbe8c3070 2024-07-27T06:48:22.939 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:22 smithi029 ceph-mon[32610]: pgmap v438: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 13 op/s 2024-07-27T06:48:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.936+0000 7f9161ffb640 1 -- 172.21.15.29:0/4197274068 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9174019910 con 0x55abbe8c3070 2024-07-27T06:48:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.937+0000 7f9186e08740 10 client.15948 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.937+0000 7f9186e08740 1 -- 172.21.15.29:0/4197274068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55abbec8b3e0 con 0x55abbe8c3070 2024-07-27T06:48:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.937+0000 7f9161ffb640 1 -- 172.21.15.29:0/4197274068 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9174025400 con 0x55abbe8c3070 2024-07-27T06:48:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.938+0000 7f9161ffb640 1 -- 172.21.15.29:0/4197274068 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f917401a3f0 con 0x55abbe8c3070 2024-07-27T06:48:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.938+0000 7f9186e08740 10 client.15948 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.938+0000 7f9186e08740 10 client.15948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.938+0000 7f9186e08740 20 client.15948 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.938+0000 7f9186e08740 1 --2- 172.21.15.29:0/4197274068 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55abbe8819f0 0x55abbec8ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.938+0000 7f9186e08740 4 client.15948 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.938+0000 7f9186e08740 1 -- 172.21.15.29:0/4197274068 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9164004030 con 0x55abbe8819f0 2024-07-27T06:48:22.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.938+0000 7f917a59c640 1 --2- 172.21.15.29:0/4197274068 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55abbe8819f0 0x55abbec8ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.939+0000 7f9161ffb640 10 client.15948 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:22.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.939+0000 7f917a59c640 1 --2- 172.21.15.29:0/4197274068 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55abbe8819f0 0x55abbec8ea20 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80801b4cb30258f7 server_cookie=d4e8e745b9af171e in_seq=0 out_seq=0 2024-07-27T06:48:22.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.940+0000 7f9161ffb640 1 -- 172.21.15.29:0/4197274068 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f9164004030 con 0x55abbe8819f0 2024-07-27T06:48:22.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.940+0000 7f9161ffb640 10 client.15948 handle_command_reply: tid=0 2024-07-27T06:48:22.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.940+0000 7f9186e08740 1 client.15948 shutdown 2024-07-27T06:48:22.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.940+0000 7f9186e08740 1 -- 172.21.15.29:0/4197274068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8c3070 msgr2=0x55abbec88780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.940+0000 7f9186e08740 1 --2- 172.21.15.29:0/4197274068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8c3070 0x55abbec88780 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f917401c760 tx=0x7f9174004ab0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.941+0000 7f9186e08740 1 -- 172.21.15.29:0/4197274068 shutdown_connections 2024-07-27T06:48:22.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.941+0000 7f9186e08740 1 --2- 172.21.15.29:0/4197274068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8c3070 0x55abbec88780 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.942+0000 7f9186e08740 1 --2- 172.21.15.29:0/4197274068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8a2390 0x55abbec7cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.942+0000 7f9186e08740 1 --2- 172.21.15.29:0/4197274068 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55abbe8819f0 0x55abbec8ea20 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.942+0000 7f9186e08740 1 --2- 172.21.15.29:0/4197274068 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe89a1b0 0x55abbec7c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.942+0000 7f9161ffb640 0 client.15948 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.942+0000 7f9186e08740 1 -- 172.21.15.29:0/4197274068 >> 172.21.15.29:0/4197274068 conn(0x55abbe8b7460 msgr2=0x55abbec64c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.942+0000 7f9186e08740 1 -- 172.21.15.29:0/4197274068 shutdown_connections 2024-07-27T06:48:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.942+0000 7f9186e08740 1 -- 172.21.15.29:0/4197274068 wait complete. 2024-07-27T06:48:22.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.942+0000 7f9186e08740 20 client.15948 trim_cache size 0 max 16384 2024-07-27T06:48:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.976+0000 7f9186e08740 1 Processor -- start 2024-07-27T06:48:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.976+0000 7f9186e08740 1 -- start start 2024-07-27T06:48:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.976+0000 7f9186e08740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8819f0 0x55abbec66470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.977+0000 7f9186e08740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe89a1b0 0x55abbec88570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.977+0000 7f917a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8819f0 0x55abbec66470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.977+0000 7f917a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8819f0 0x55abbec66470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:33108/0 (socket says 172.21.15.29:33108) 2024-07-27T06:48:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.977+0000 7f917a59c640 1 -- 172.21.15.29:0/960258543 learned_addr learned my addr 172.21.15.29:0/960258543 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.977+0000 7f9186e08740 1 --2- 172.21.15.29:0/960258543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8a2390 0x55abbec941e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.977+0000 7f9186e08740 1 -- 172.21.15.29:0/960258543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f917c1acf00 con 0x55abbe89a1b0 2024-07-27T06:48:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.977+0000 7f9173fff640 1 --2- 172.21.15.29:0/960258543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe89a1b0 0x55abbec88570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.977+0000 7f9186e08740 1 -- 172.21.15.29:0/960258543 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f917c1aea40 con 0x55abbe8a2390 2024-07-27T06:48:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.977+0000 7f9186e08740 1 -- 172.21.15.29:0/960258543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f917c1ad5d0 con 0x55abbe8819f0 2024-07-27T06:48:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.978+0000 7f917ad9d640 1 --2- 172.21.15.29:0/960258543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8a2390 0x55abbec941e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.978+0000 7f9173fff640 1 -- 172.21.15.29:0/960258543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8819f0 msgr2=0x55abbec66470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.978+0000 7f9173fff640 1 --2- 172.21.15.29:0/960258543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8819f0 0x55abbec66470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.978+0000 7f9173fff640 1 -- 172.21.15.29:0/960258543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8a2390 msgr2=0x55abbec941e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.978+0000 7f9173fff640 1 --2- 172.21.15.29:0/960258543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8a2390 0x55abbec941e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.978+0000 7f9173fff640 1 -- 172.21.15.29:0/960258543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abbec7da20 con 0x55abbe89a1b0 2024-07-27T06:48:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.978+0000 7f9173fff640 1 --2- 172.21.15.29:0/960258543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe89a1b0 0x55abbec88570 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9164042000 tx=0x7f9164005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf7aea5d3f9d983 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.979+0000 7f9163fff640 1 -- 172.21.15.29:0/960258543 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164035070 con 0x55abbe89a1b0 2024-07-27T06:48:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.979+0000 7f9163fff640 1 -- 172.21.15.29:0/960258543 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9164032d40 con 0x55abbe89a1b0 2024-07-27T06:48:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.979+0000 7f9163fff640 1 -- 172.21.15.29:0/960258543 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164005290 con 0x55abbe89a1b0 2024-07-27T06:48:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.980+0000 7f9186e08740 1 -- 172.21.15.29:0/960258543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe89a1b0 msgr2=0x55abbec88570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.980+0000 7f9186e08740 1 --2- 172.21.15.29:0/960258543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe89a1b0 0x55abbec88570 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9164042000 tx=0x7f9164005fd0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.980+0000 7f9186e08740 1 -- 172.21.15.29:0/960258543 shutdown_connections 2024-07-27T06:48:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.980+0000 7f9186e08740 1 --2- 172.21.15.29:0/960258543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8a2390 0x55abbec941e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.980+0000 7f9186e08740 1 --2- 172.21.15.29:0/960258543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe89a1b0 0x55abbec88570 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.980+0000 7f9186e08740 1 --2- 172.21.15.29:0/960258543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe8819f0 0x55abbec66470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.980+0000 7f9186e08740 1 -- 172.21.15.29:0/960258543 >> 172.21.15.29:0/960258543 conn(0x55abbe8b7460 msgr2=0x55abbec65d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.981+0000 7f9186e08740 1 -- 172.21.15.29:0/960258543 shutdown_connections 2024-07-27T06:48:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.981+0000 7f9186e08740 1 -- 172.21.15.29:0/960258543 wait complete. 2024-07-27T06:48:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.981+0000 7f9186e08740 1 -- start start 2024-07-27T06:48:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.981+0000 7f9186e08740 1 Processor -- start 2024-07-27T06:48:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.982+0000 7f9186e08740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8819f0 0x55abbec84e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.982+0000 7f9186e08740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe89a1b0 0x55abbec802f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.982+0000 7f9186e08740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8a2390 0x55abbec808b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.982+0000 7f917ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8a2390 0x55abbec808b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.982+0000 7f917a59c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8819f0 0x55abbec84e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.982+0000 7f9173fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe89a1b0 0x55abbec802f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.982+0000 7f917ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8a2390 0x55abbec808b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60886/0 (socket says 172.21.15.29:60886) 2024-07-27T06:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.982+0000 7f917ad9d640 1 -- 172.21.15.29:0/4238274318 learned_addr learned my addr 172.21.15.29:0/4238274318 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.982+0000 7f9186e08740 1 -- 172.21.15.29:0/4238274318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f917c1ae370 con 0x55abbe8a2390 2024-07-27T06:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.982+0000 7f9186e08740 1 -- 172.21.15.29:0/4238274318 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f917c1adca0 con 0x55abbe8819f0 2024-07-27T06:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.982+0000 7f9186e08740 1 -- 172.21.15.29:0/4238274318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f917c1aaee0 con 0x55abbe89a1b0 2024-07-27T06:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.983+0000 7f9173fff640 1 -- 172.21.15.29:0/4238274318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8819f0 msgr2=0x55abbec84e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.983+0000 7f9173fff640 1 --2- 172.21.15.29:0/4238274318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8819f0 0x55abbec84e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.983+0000 7f9173fff640 1 -- 172.21.15.29:0/4238274318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8a2390 msgr2=0x55abbec808b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.983+0000 7f9173fff640 1 --2- 172.21.15.29:0/4238274318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8a2390 0x55abbec808b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.983+0000 7f9173fff640 1 -- 172.21.15.29:0/4238274318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abbec81040 con 0x55abbe89a1b0 2024-07-27T06:48:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.983+0000 7f917ad9d640 1 --2- 172.21.15.29:0/4238274318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8a2390 0x55abbec808b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.983+0000 7f9186e08740 10 client.0 _ll_register_callbacks cb 0x7f917400b200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.983+0000 7f9173fff640 1 --2- 172.21.15.29:0/4238274318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe89a1b0 0x55abbec802f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x55abbec66f20 tx=0x7f91640316d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85c53d2a9953c93 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.983+0000 7f9186e08740 1 -- 172.21.15.29:0/4238274318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9164032d40 con 0x55abbe89a1b0 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.983+0000 7f9161ffb640 10 client.25648 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.983+0000 7f9161ffb640 1 -- 172.21.15.29:0/4238274318 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164035040 con 0x55abbe89a1b0 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.983+0000 7f9161ffb640 1 -- 172.21.15.29:0/4238274318 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f91640318c0 con 0x55abbe89a1b0 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.984+0000 7f9161ffb640 1 -- 172.21.15.29:0/4238274318 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9164004250 con 0x55abbe89a1b0 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.984+0000 7f9186e08740 10 client.25648 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.984+0000 7f9186e08740 1 -- 172.21.15.29:0/4238274318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55abbecf6dc0 con 0x55abbe89a1b0 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.984+0000 7f9161ffb640 1 -- 172.21.15.29:0/4238274318 <== mon.1 v2:172.21.15.77:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f91640035f0 con 0x55abbe89a1b0 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.985+0000 7f9186e08740 10 client.25648 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.985+0000 7f9186e08740 10 client.25648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.985+0000 7f9186e08740 20 client.25648 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.985+0000 7f9186e08740 1 --2- 172.21.15.29:0/4238274318 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55abbe8c3070 0x55abbecf9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.985+0000 7f9186e08740 4 client.25648 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.985+0000 7f9186e08740 1 -- 172.21.15.29:0/4238274318 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55abbebe0ae0 con 0x55abbe8c3070 2024-07-27T06:48:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.985+0000 7f917a59c640 1 --2- 172.21.15.29:0/4238274318 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55abbe8c3070 0x55abbecf9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.986+0000 7f917a59c640 1 --2- 172.21.15.29:0/4238274318 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55abbe8c3070 0x55abbecf9070 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ac50aebf18e461b server_cookie=4c4fa4865037b741 in_seq=0 out_seq=0 2024-07-27T06:48:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.986+0000 7f9161ffb640 10 client.25648 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.986+0000 7f9161ffb640 1 -- 172.21.15.29:0/4238274318 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55abbebe0ae0 con 0x55abbe8c3070 2024-07-27T06:48:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.986+0000 7f9161ffb640 10 client.25648 handle_command_reply: tid=0 2024-07-27T06:48:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.986+0000 7f9186e08740 1 client.25648 shutdown 2024-07-27T06:48:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.986+0000 7f9186e08740 1 -- 172.21.15.29:0/4238274318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe89a1b0 msgr2=0x55abbec802f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.986+0000 7f9186e08740 1 --2- 172.21.15.29:0/4238274318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe89a1b0 0x55abbec802f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x55abbec66f20 tx=0x7f91640316d0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.987+0000 7f9186e08740 1 -- 172.21.15.29:0/4238274318 shutdown_connections 2024-07-27T06:48:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.987+0000 7f9186e08740 1 --2- 172.21.15.29:0/4238274318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55abbe8a2390 0x55abbec808b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.987+0000 7f9186e08740 1 --2- 172.21.15.29:0/4238274318 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55abbe8c3070 0x55abbecf9070 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.987+0000 7f9186e08740 1 --2- 172.21.15.29:0/4238274318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55abbe89a1b0 0x55abbec802f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.987+0000 7f9186e08740 1 --2- 172.21.15.29:0/4238274318 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55abbe8819f0 0x55abbec84e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.987+0000 7f9186e08740 1 -- 172.21.15.29:0/4238274318 >> 172.21.15.29:0/4238274318 conn(0x55abbe8b7460 msgr2=0x55abbec6b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.987+0000 7f9161ffb640 0 client.25648 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.987+0000 7f9186e08740 1 -- 172.21.15.29:0/4238274318 shutdown_connections 2024-07-27T06:48:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.987+0000 7f9186e08740 1 -- 172.21.15.29:0/4238274318 wait complete. 2024-07-27T06:48:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.988+0000 7f9186e08740 20 client.25648 trim_cache size 0 max 16384 2024-07-27T06:48:22.991 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:48:22.991 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-07-27T06:48:22.991 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-07-27T06:48:22.991 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:48:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.989+0000 7f9180884640 1 -- 172.21.15.29:0/4089309208 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f91680b3dc0 msgr2=0x7f91680b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.989+0000 7f9180884640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f91680b3dc0 0x7f91680b62d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f917400b480 tx=0x7f9174012040 comp rx=0 tx=0).stop 2024-07-27T06:48:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.989+0000 7f9180884640 1 -- 172.21.15.29:0/4089309208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f917c1024a0 msgr2=0x7f917c197c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.989+0000 7f9180884640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f917c1024a0 0x7f917c197c70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f916c00ea60 tx=0x7f916c00ef20 comp rx=0 tx=0).stop 2024-07-27T06:48:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.990+0000 7f9180884640 1 -- 172.21.15.29:0/4089309208 shutdown_connections 2024-07-27T06:48:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.990+0000 7f9180884640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f917c108730 0x7f917c19d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.990+0000 7f9180884640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f917c107de0 0x7f917c198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.990+0000 7f9180884640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f91680b3dc0 0x7f91680b62d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.990+0000 7f9180884640 1 --2- 172.21.15.29:0/4089309208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f917c1024a0 0x7f917c197c70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.990+0000 7f9180884640 1 -- 172.21.15.29:0/4089309208 >> 172.21.15.29:0/4089309208 conn(0x7f917c079160 msgr2=0x7f917c103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.990+0000 7f9180884640 1 -- 172.21.15.29:0/4089309208 shutdown_connections 2024-07-27T06:48:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:22.991+0000 7f9180884640 1 -- 172.21.15.29:0/4089309208 wait complete. 2024-07-27T06:48:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:22 smithi077 ceph-mon[37153]: pgmap v438: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 13 op/s 2024-07-27T06:48:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:22 smithi047 ceph-mon[36835]: pgmap v438: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 13 op/s 2024-07-27T06:48:24.060 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-27T06:48:24.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:48:24.566 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '7f206a77', quiesce took 17.8 seconds. Will release after: 4.75 2024-07-27T06:48:24.686 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:24 smithi077 ceph-mon[37153]: pgmap v439: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 511 B/s wr, 12 op/s 2024-07-27T06:48:25.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:24 smithi029 ceph-mon[32610]: pgmap v439: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 511 B/s wr, 12 op/s 2024-07-27T06:48:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:24 smithi047 ceph-mon[36835]: pgmap v439: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 511 B/s wr, 12 op/s 2024-07-27T06:48:26.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.750+0000 7f71b0fce640 1 -- 172.21.15.29:0/1952879915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac102490 msgr2=0x7f71ac1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:26.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.750+0000 7f71b0fce640 1 --2- 172.21.15.29:0/1952879915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac102490 0x7f71ac1028a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f719c009860 tx=0x7f719c02a340 comp rx=0 tx=0).stop 2024-07-27T06:48:26.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.750+0000 7f71b0fce640 1 -- 172.21.15.29:0/1952879915 shutdown_connections 2024-07-27T06:48:26.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.750+0000 7f71b0fce640 1 --2- 172.21.15.29:0/1952879915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71ac108720 0x7f71ac10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:26.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.750+0000 7f71b0fce640 1 --2- 172.21.15.29:0/1952879915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71ac107dd0 0x7f71ac1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:26.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.750+0000 7f71b0fce640 1 --2- 172.21.15.29:0/1952879915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac102490 0x7f71ac1028a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:26.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.750+0000 7f71b0fce640 1 -- 172.21.15.29:0/1952879915 >> 172.21.15.29:0/1952879915 conn(0x7f71ac079160 msgr2=0x7f71ac079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:26.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.750+0000 7f71b0fce640 1 -- 172.21.15.29:0/1952879915 shutdown_connections 2024-07-27T06:48:26.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.750+0000 7f71b0fce640 1 -- 172.21.15.29:0/1952879915 wait complete. 2024-07-27T06:48:26.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.751+0000 7f71b0fce640 1 Processor -- start 2024-07-27T06:48:26.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.752+0000 7f71b0fce640 1 -- start start 2024-07-27T06:48:26.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.752+0000 7f71b0fce640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71ac102490 0x7f71ac199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:26.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.752+0000 7f71b0fce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac107dd0 0x7f71ac19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:26.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.752+0000 7f71b0fce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71ac108720 0x7f71ac19f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:26.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.752+0000 7f71b0fce640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac112380 con 0x7f71ac108720 2024-07-27T06:48:26.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.752+0000 7f71b0fce640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac112200 con 0x7f71ac102490 2024-07-27T06:48:26.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.752+0000 7f71b0fce640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac112500 con 0x7f71ac107dd0 2024-07-27T06:48:26.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.752+0000 7f71a9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac107dd0 0x7f71ac19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.752+0000 7f71aad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71ac108720 0x7f71ac19f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.752+0000 7f71aa59c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71ac102490 0x7f71ac199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.752+0000 7f71a9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac107dd0 0x7f71ac19a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:33134/0 (socket says 172.21.15.29:33134) 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.752+0000 7f71a9d9b640 1 -- 172.21.15.29:0/102444143 learned_addr learned my addr 172.21.15.29:0/102444143 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.753+0000 7f71a9d9b640 1 -- 172.21.15.29:0/102444143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71ac102490 msgr2=0x7f71ac199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.753+0000 7f71a9d9b640 1 --2- 172.21.15.29:0/102444143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71ac102490 0x7f71ac199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.753+0000 7f71a9d9b640 1 -- 172.21.15.29:0/102444143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71ac108720 msgr2=0x7f71ac19f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.753+0000 7f71a9d9b640 1 --2- 172.21.15.29:0/102444143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71ac108720 0x7f71ac19f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.753+0000 7f71a9d9b640 1 -- 172.21.15.29:0/102444143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71ac19fdd0 con 0x7f71ac107dd0 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.753+0000 7f71a9d9b640 1 --2- 172.21.15.29:0/102444143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac107dd0 0x7f71ac19a400 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f719400e9a0 tx=0x7f719400ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.753+0000 7f71a37fe640 1 -- 172.21.15.29:0/102444143 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719400cd80 con 0x7f71ac107dd0 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.753+0000 7f71b0fce640 1 -- 172.21.15.29:0/102444143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71ac1a00c0 con 0x7f71ac107dd0 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.753+0000 7f71a37fe640 1 -- 172.21.15.29:0/102444143 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71940035f0 con 0x7f71ac107dd0 2024-07-27T06:48:26.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.753+0000 7f71a37fe640 1 -- 172.21.15.29:0/102444143 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194010690 con 0x7f71ac107dd0 2024-07-27T06:48:26.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.753+0000 7f71b0fce640 1 -- 172.21.15.29:0/102444143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71ac108f20 con 0x7f71ac107dd0 2024-07-27T06:48:26.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.755+0000 7f71a17fa640 1 -- 172.21.15.29:0/102444143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71ac1028a0 con 0x7f71ac107dd0 2024-07-27T06:48:26.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.756+0000 7f71a37fe640 1 -- 172.21.15.29:0/102444143 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f7194004020 con 0x7f71ac107dd0 2024-07-27T06:48:26.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.757+0000 7f71a37fe640 1 --2- 172.21.15.29:0/102444143 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f71980b3dc0 0x7f71980b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:26.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.757+0000 7f71aa59c640 1 --2- 172.21.15.29:0/102444143 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f71980b3dc0 0x7f71980b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:26.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.757+0000 7f71a37fe640 1 -- 172.21.15.29:0/102444143 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f71940e4240 con 0x7f71ac107dd0 2024-07-27T06:48:26.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.757+0000 7f71aa59c640 1 --2- 172.21.15.29:0/102444143 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f71980b3dc0 0x7f71980b62d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f719c02a930 tx=0x7f719c0058d0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:26.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:26.761+0000 7f71a37fe640 1 -- 172.21.15.29:0/102444143 <== mon.1 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f71940ab500 con 0x7f71ac107dd0 2024-07-27T06:48:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:26 smithi029 ceph-mon[32610]: pgmap v440: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-07-27T06:48:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:48:27.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.027+0000 7f71a17fa640 1 -- 172.21.15.29:0/102444143 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71ac10bb80 con 0x7f71ac107dd0 2024-07-27T06:48:27.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.028+0000 7f71a37fe640 1 -- 172.21.15.29:0/102444143 <== mon.1 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11957 (secure 0 0 0) 0x7f71940aae00 con 0x7f71ac107dd0 2024-07-27T06:48:27.030 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:48:27.030 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-07-27T06:46:50:421100+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:46:49.561737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24670,24658,24646,24617,24613]},"id":1}]} 2024-07-27T06:48:27.030 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-07-27T06:48:27.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.030+0000 7f71a17fa640 1 -- 172.21.15.29:0/102444143 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f71980b3dc0 msgr2=0x7f71980b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:27.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.030+0000 7f71a17fa640 1 --2- 172.21.15.29:0/102444143 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f71980b3dc0 0x7f71980b62d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f719c02a930 tx=0x7f719c0058d0 comp rx=0 tx=0).stop 2024-07-27T06:48:27.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.030+0000 7f71a17fa640 1 -- 172.21.15.29:0/102444143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac107dd0 msgr2=0x7f71ac19a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:27.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.030+0000 7f71a17fa640 1 --2- 172.21.15.29:0/102444143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac107dd0 0x7f71ac19a400 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f719400e9a0 tx=0x7f719400ee60 comp rx=0 tx=0).stop 2024-07-27T06:48:27.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.030+0000 7f71a17fa640 1 -- 172.21.15.29:0/102444143 shutdown_connections 2024-07-27T06:48:27.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.030+0000 7f71a17fa640 1 --2- 172.21.15.29:0/102444143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71ac108720 0x7f71ac19f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:27.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.030+0000 7f71a17fa640 1 --2- 172.21.15.29:0/102444143 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f71980b3dc0 0x7f71980b62d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:27.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.030+0000 7f71a17fa640 1 --2- 172.21.15.29:0/102444143 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71ac107dd0 0x7f71ac19a400 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:27.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.030+0000 7f71a17fa640 1 --2- 172.21.15.29:0/102444143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71ac102490 0x7f71ac199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:27.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.030+0000 7f71a17fa640 1 -- 172.21.15.29:0/102444143 >> 172.21.15.29:0/102444143 conn(0x7f71ac079160 msgr2=0x7f71ac101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:27.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.030+0000 7f71a17fa640 1 -- 172.21.15.29:0/102444143 shutdown_connections 2024-07-27T06:48:27.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:27.031+0000 7f71a17fa640 1 -- 172.21.15.29:0/102444143 wait complete. 2024-07-27T06:48:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:26 smithi077 ceph-mon[37153]: pgmap v440: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-07-27T06:48:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:48:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:26 smithi047 ceph-mon[36835]: pgmap v440: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-07-27T06:48:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:48:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:48:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:48:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:48:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:27 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/102444143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:48:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:48:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:48:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:48:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:27 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/102444143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:48:28.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:48:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:48:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:48:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:27 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/102444143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:48:28.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-27T06:48:28.779 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:28 smithi047 ceph-mon[36835]: pgmap v441: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-27T06:48:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:28 smithi029 ceph-mon[32610]: pgmap v441: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-27T06:48:29.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:48:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:28 smithi077 ceph-mon[37153]: pgmap v441: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-27T06:48:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:30.100 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 84% (10781/12697) Updating files: 85% (10793/12697) Updating files: 86% (10920/12697) Updating files: 87% (11047/12697) Updating files: 88% (11174/12697) Updating files: 89% (11301/12697) Updating files: 90% (11428/12697) Updating files: 91% (11555/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-07-27T06:48:30.331 INFO:tasks.workunit.client.0.smithi029.stderr:Note: switching to 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'. 2024-07-27T06:48:30.331 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:48:30.331 INFO:tasks.workunit.client.0.smithi029.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr:state without impacting any branches by switching back to a branch. 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr: git switch -c 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr:Or undo this operation with: 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr: git switch - 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-27T06:48:30.332 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:48:30.333 INFO:tasks.workunit.client.0.smithi029.stderr:HEAD is now at e8f88bb1fe0 Merge PR #54435 into wip-vshankar-testing-20240725.121331-debug 2024-07-27T06:48:30.335 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-07-27T06:48:30.367 INFO:tasks.workunit.client.0.smithi029.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-07-27T06:48:30.369 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-27T06:48:30.370 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-07-27T06:48:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:30 smithi029 ceph-mon[32610]: pgmap v442: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-07-27T06:48:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:30 smithi077 ceph-mon[37153]: pgmap v442: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-07-27T06:48:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:30 smithi047 ceph-mon[36835]: pgmap v442: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-07-27T06:48:33.109 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:32 smithi029 ceph-mon[32610]: pgmap v443: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-27T06:48:33.189 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-07-27T06:48:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.283+0000 7f994ed96640 1 -- 172.21.15.29:0/3350912385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948108750 msgr2=0x7f994810ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.283+0000 7f994ed96640 1 --2- 172.21.15.29:0/3350912385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948108750 0x7f994810ef80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f9940009ca0 tx=0x7f994002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.284+0000 7f994ed96640 1 -- 172.21.15.29:0/3350912385 shutdown_connections 2024-07-27T06:48:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.284+0000 7f994ed96640 1 --2- 172.21.15.29:0/3350912385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948108750 0x7f994810ef80 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.284+0000 7f994ed96640 1 --2- 172.21.15.29:0/3350912385 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9948107e00 0x7f9948108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.284+0000 7f994ed96640 1 --2- 172.21.15.29:0/3350912385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99481024c0 0x7f99481028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.284+0000 7f994ed96640 1 -- 172.21.15.29:0/3350912385 >> 172.21.15.29:0/3350912385 conn(0x7f9948079180 msgr2=0x7f9948079590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.284+0000 7f994ed96640 1 -- 172.21.15.29:0/3350912385 shutdown_connections 2024-07-27T06:48:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.284+0000 7f994ed96640 1 -- 172.21.15.29:0/3350912385 wait complete. 2024-07-27T06:48:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994ed96640 1 Processor -- start 2024-07-27T06:48:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994ed96640 1 -- start start 2024-07-27T06:48:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994ed96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99481024c0 0x7f9948198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994ed96640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9948107e00 0x7f99481992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994ed96640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948108750 0x7f994819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994ed96640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9948112400 con 0x7f9948108750 2024-07-27T06:48:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994ed96640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f9948112280 con 0x7f9948107e00 2024-07-27T06:48:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994ed96640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9948112580 con 0x7f99481024c0 2024-07-27T06:48:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994d333640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948108750 0x7f994819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994cb32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99481024c0 0x7f9948198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994cb32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99481024c0 0x7f9948198d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:44892/0 (socket says 172.21.15.29:44892) 2024-07-27T06:48:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994cb32640 1 -- 172.21.15.29:0/2776333303 learned_addr learned my addr 172.21.15.29:0/2776333303 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f994d333640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948108750 0x7f994819e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52854/0 (socket says 172.21.15.29:52854) 2024-07-27T06:48:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.285+0000 7f9947fff640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9948107e00 0x7f99481992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.286+0000 7f994cb32640 1 -- 172.21.15.29:0/2776333303 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9948107e00 msgr2=0x7f99481992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.286+0000 7f994cb32640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9948107e00 0x7f99481992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.286+0000 7f994cb32640 1 -- 172.21.15.29:0/2776333303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948108750 msgr2=0x7f994819e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.286+0000 7f994cb32640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948108750 0x7f994819e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.286+0000 7f994cb32640 1 -- 172.21.15.29:0/2776333303 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f994819ecc0 con 0x7f99481024c0 2024-07-27T06:48:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.286+0000 7f994cb32640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99481024c0 0x7f9948198d50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f993c00c890 tx=0x7f993c00cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.286+0000 7f9945ffb640 1 -- 172.21.15.29:0/2776333303 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c004220 con 0x7f99481024c0 2024-07-27T06:48:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.286+0000 7f994ed96640 1 -- 172.21.15.29:0/2776333303 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f994819efb0 con 0x7f99481024c0 2024-07-27T06:48:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.286+0000 7f9945ffb640 1 -- 172.21.15.29:0/2776333303 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f993c0043c0 con 0x7f99481024c0 2024-07-27T06:48:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.286+0000 7f9945ffb640 1 -- 172.21.15.29:0/2776333303 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c004a00 con 0x7f99481024c0 2024-07-27T06:48:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.286+0000 7f994ed96640 1 -- 172.21.15.29:0/2776333303 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f994806b5b0 con 0x7f99481024c0 2024-07-27T06:48:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.289+0000 7f9945ffb640 1 -- 172.21.15.29:0/2776333303 <== mon.1 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f993c0079b0 con 0x7f99481024c0 2024-07-27T06:48:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.290+0000 7f995531a740 1 Processor -- start 2024-07-27T06:48:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.290+0000 7f9945ffb640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f99380b3d40 0x7f99380b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.290+0000 7f995531a740 1 -- start start 2024-07-27T06:48:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.290+0000 7f994d333640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f99380b3d40 0x7f99380b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.290+0000 7f9945ffb640 1 -- 172.21.15.29:0/2776333303 <== mon.1 v2:172.21.15.77:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f993c0e3cc0 con 0x7f99481024c0 2024-07-27T06:48:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f995531a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bef070 0x5614f2f94570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f995531a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 0x5614f2f94ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f995531a740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bce390 0x5614f2f94ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f995531a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5614f2f0abb0 con 0x5614f2bef070 2024-07-27T06:48:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f995531a740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5614f2efa0b0 con 0x5614f2bce390 2024-07-27T06:48:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f995531a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5614f2efe0f0 con 0x5614f2bc61b0 2024-07-27T06:48:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f994cb32640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bef070 0x5614f2f94570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f994cb32640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bef070 0x5614f2f94570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52870/0 (socket says 172.21.15.29:52870) 2024-07-27T06:48:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f994cb32640 1 -- 172.21.15.29:0/2034121623 learned_addr learned my addr 172.21.15.29:0/2034121623 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f9947fff640 1 --2- 172.21.15.29:0/2034121623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 0x5614f2f94ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f994d333640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f99380b3d40 0x7f99380b6250 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9940009430 tx=0x7f9940009800 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f994cb32640 1 -- 172.21.15.29:0/2034121623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 msgr2=0x5614f2f94ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f994cb32640 1 --2- 172.21.15.29:0/2034121623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 0x5614f2f94ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f994cb32640 1 -- 172.21.15.29:0/2034121623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bce390 msgr2=0x5614f2f94ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f994cb32640 1 --2- 172.21.15.29:0/2034121623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bce390 0x5614f2f94ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f994cb32640 1 -- 172.21.15.29:0/2034121623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614f2ea0b90 con 0x5614f2bef070 2024-07-27T06:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.291+0000 7f994cb32640 1 --2- 172.21.15.29:0/2034121623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bef070 0x5614f2f94570 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f993c0aa920 tx=0x7f993c0b32d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b048715a070b26 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.292+0000 7f9932ffd640 1 -- 172.21.15.29:0/2034121623 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c01d040 con 0x5614f2bef070 2024-07-27T06:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.292+0000 7f9932ffd640 1 -- 172.21.15.29:0/2034121623 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f993c0920d0 con 0x5614f2bef070 2024-07-27T06:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.292+0000 7f9932ffd640 1 -- 172.21.15.29:0/2034121623 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c0e1830 con 0x5614f2bef070 2024-07-27T06:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.293+0000 7f995531a740 1 -- 172.21.15.29:0/2034121623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bef070 msgr2=0x5614f2f94570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.293+0000 7f995531a740 1 --2- 172.21.15.29:0/2034121623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bef070 0x5614f2f94570 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f993c0aa920 tx=0x7f993c0b32d0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.293+0000 7f995531a740 1 -- 172.21.15.29:0/2034121623 shutdown_connections 2024-07-27T06:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.293+0000 7f995531a740 1 --2- 172.21.15.29:0/2034121623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bce390 0x5614f2f94ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.293+0000 7f995531a740 1 --2- 172.21.15.29:0/2034121623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 0x5614f2f94ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.293+0000 7f995531a740 1 --2- 172.21.15.29:0/2034121623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bef070 0x5614f2f94570 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.293+0000 7f995531a740 1 -- 172.21.15.29:0/2034121623 >> 172.21.15.29:0/2034121623 conn(0x5614f2be3460 msgr2=0x5614f2f92460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.293+0000 7f995531a740 1 -- 172.21.15.29:0/2034121623 shutdown_connections 2024-07-27T06:48:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.293+0000 7f995531a740 1 -- 172.21.15.29:0/2034121623 wait complete. 2024-07-27T06:48:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.293+0000 7f995531a740 1 -- start start 2024-07-27T06:48:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f995531a740 1 Processor -- start 2024-07-27T06:48:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f995531a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 0x5614f2fa0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f995531a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 0x5614f2fa07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f995531a740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bef070 0x5614f2fa6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f995531a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5614f2d5d700 con 0x5614f2bce390 2024-07-27T06:48:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f995531a740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5614f2e9c9e0 con 0x5614f2bef070 2024-07-27T06:48:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f995531a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5614f2fa2d50 con 0x5614f2bc61b0 2024-07-27T06:48:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f995531a740 10 client.0 _ll_register_callbacks cb 0x5614f2eb74f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f9947fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 0x5614f2fa07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f9947fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 0x5614f2fa07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52882/0 (socket says 172.21.15.29:52882) 2024-07-27T06:48:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f9947fff640 1 -- 172.21.15.29:0/3834607722 learned_addr learned my addr 172.21.15.29:0/3834607722 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f994d333640 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bef070 0x5614f2fa6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.294+0000 7f994cb32640 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 0x5614f2fa0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f9947fff640 1 -- 172.21.15.29:0/3834607722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 msgr2=0x5614f2fa0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f9947fff640 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 0x5614f2fa0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f9947fff640 1 -- 172.21.15.29:0/3834607722 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bef070 msgr2=0x5614f2fa6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f9947fff640 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bef070 0x5614f2fa6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f9947fff640 1 -- 172.21.15.29:0/3834607722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614f2c5a650 con 0x5614f2bce390 2024-07-27T06:48:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f9947fff640 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 0x5614f2fa07f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9934010820 tx=0x7f9934010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eeb94d43d4286c4 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f9927fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f9927fff640 1 -- 172.21.15.29:0/3834607722 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993400fb10 con 0x5614f2bce390 2024-07-27T06:48:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f9927fff640 1 -- 172.21.15.29:0/3834607722 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f993400fcb0 con 0x5614f2bce390 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f995531a740 1 -- 172.21.15.29:0/3834607722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f993c0920d0 con 0x5614f2bce390 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f9927fff640 1 -- 172.21.15.29:0/3834607722 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9934005020 con 0x5614f2bce390 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f995531a740 10 client.15972 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f995531a740 1 -- 172.21.15.29:0/3834607722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614f2faef20 con 0x5614f2bce390 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.295+0000 7f9927fff640 1 -- 172.21.15.29:0/3834607722 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f993c0920d0 con 0x5614f2bce390 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.296+0000 7f9927fff640 1 -- 172.21.15.29:0/3834607722 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9934019410 con 0x5614f2bce390 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.296+0000 7f995531a740 10 client.15972 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.296+0000 7f995531a740 10 client.15972 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.296+0000 7f995531a740 20 client.15972 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.296+0000 7f995531a740 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x5614f2bad9f0 0x5614f2fb24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.296+0000 7f995531a740 4 client.15972 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.296+0000 7f995531a740 1 -- 172.21.15.29:0/3834607722 --> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614f2f0cb00 con 0x5614f2bad9f0 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.296+0000 7f994cb32640 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x5614f2bad9f0 0x5614f2fb24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.296+0000 7f994cb32640 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x5614f2bad9f0 0x5614f2fb24c0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e0888a39c6371992 server_cookie=8610c125d3b05a66 in_seq=0 out_seq=0 2024-07-27T06:48:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.296+0000 7f9927fff640 10 client.15972 ms_handle_connect on v2:172.21.15.77:6832/1085922080 2024-07-27T06:48:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.297+0000 7f9927fff640 1 -- 172.21.15.29:0/3834607722 <== mds.4 v2:172.21.15.77:6832/1085922080 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5614f2f0cb00 con 0x5614f2bad9f0 2024-07-27T06:48:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.297+0000 7f9927fff640 10 client.15972 handle_command_reply: tid=0 2024-07-27T06:48:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.297+0000 7f995531a740 1 client.15972 shutdown 2024-07-27T06:48:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.297+0000 7f995531a740 1 -- 172.21.15.29:0/3834607722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 msgr2=0x5614f2fa07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.297+0000 7f995531a740 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 0x5614f2fa07f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9934010820 tx=0x7f9934010c20 comp rx=0 tx=0).stop 2024-07-27T06:48:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.297+0000 7f995531a740 1 -- 172.21.15.29:0/3834607722 shutdown_connections 2024-07-27T06:48:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.297+0000 7f995531a740 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x5614f2bad9f0 0x5614f2fb24c0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.297+0000 7f995531a740 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bef070 0x5614f2fa6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.297+0000 7f995531a740 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 0x5614f2fa07f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.297+0000 7f9927fff640 0 client.15972 ms_handle_reset on v2:172.21.15.77:6832/1085922080 2024-07-27T06:48:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.297+0000 7f995531a740 1 --2- 172.21.15.29:0/3834607722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 0x5614f2fa0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.297+0000 7f995531a740 1 -- 172.21.15.29:0/3834607722 >> 172.21.15.29:0/3834607722 conn(0x5614f2be3460 msgr2=0x5614f2f97750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.298+0000 7f995531a740 1 -- 172.21.15.29:0/3834607722 shutdown_connections 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.298+0000 7f995531a740 1 -- 172.21.15.29:0/3834607722 wait complete. 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.298+0000 7f995531a740 20 client.15972 trim_cache size 0 max 16384 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.312+0000 7f995531a740 1 Processor -- start 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.312+0000 7f995531a740 1 -- start start 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.312+0000 7f995531a740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bad9f0 0x5614f2f92560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.312+0000 7f995531a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bc61b0 0x5614f2face30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.312+0000 7f995531a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bce390 0x5614f2f97670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.312+0000 7f995531a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f99481af1c0 con 0x5614f2bc61b0 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.312+0000 7f995531a740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f99481b0d00 con 0x5614f2bad9f0 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.312+0000 7f995531a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f99481af890 con 0x5614f2bce390 2024-07-27T06:48:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.312+0000 7f9947fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bc61b0 0x5614f2face30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.312+0000 7f9947fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bc61b0 0x5614f2face30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52886/0 (socket says 172.21.15.29:52886) 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.312+0000 7f9947fff640 1 -- 172.21.15.29:0/442614714 learned_addr learned my addr 172.21.15.29:0/442614714 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.312+0000 7f994d333640 1 --2- 172.21.15.29:0/442614714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bce390 0x5614f2f97670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.313+0000 7f994cb32640 1 --2- 172.21.15.29:0/442614714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bad9f0 0x5614f2f92560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.313+0000 7f9947fff640 1 -- 172.21.15.29:0/442614714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bce390 msgr2=0x5614f2f97670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.313+0000 7f9947fff640 1 --2- 172.21.15.29:0/442614714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bce390 0x5614f2f97670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.313+0000 7f9947fff640 1 -- 172.21.15.29:0/442614714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bad9f0 msgr2=0x5614f2f92560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.313+0000 7f9947fff640 1 --2- 172.21.15.29:0/442614714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bad9f0 0x5614f2f92560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.313+0000 7f9947fff640 1 -- 172.21.15.29:0/442614714 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614f2fa2670 con 0x5614f2bc61b0 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.313+0000 7f994d333640 1 --2- 172.21.15.29:0/442614714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bce390 0x5614f2f97670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.313+0000 7f994cb32640 1 --2- 172.21.15.29:0/442614714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bad9f0 0x5614f2f92560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.313+0000 7f9947fff640 1 --2- 172.21.15.29:0/442614714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bc61b0 0x5614f2face30 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9934000c00 tx=0x7f993400a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e991fcea38a29b5c server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.313+0000 7f99327fc640 1 -- 172.21.15.29:0/442614714 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99340193e0 con 0x5614f2bc61b0 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.313+0000 7f99327fc640 1 -- 172.21.15.29:0/442614714 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9934019580 con 0x5614f2bc61b0 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.313+0000 7f99327fc640 1 -- 172.21.15.29:0/442614714 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99340055e0 con 0x5614f2bc61b0 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.314+0000 7f995531a740 1 -- 172.21.15.29:0/442614714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bc61b0 msgr2=0x5614f2face30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.314+0000 7f995531a740 1 --2- 172.21.15.29:0/442614714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bc61b0 0x5614f2face30 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9934000c00 tx=0x7f993400a3c0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.314+0000 7f995531a740 1 -- 172.21.15.29:0/442614714 shutdown_connections 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.314+0000 7f995531a740 1 --2- 172.21.15.29:0/442614714 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bce390 0x5614f2f97670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.314+0000 7f995531a740 1 --2- 172.21.15.29:0/442614714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bc61b0 0x5614f2face30 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.314+0000 7f995531a740 1 --2- 172.21.15.29:0/442614714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bad9f0 0x5614f2f92560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.314+0000 7f995531a740 1 -- 172.21.15.29:0/442614714 >> 172.21.15.29:0/442614714 conn(0x5614f2be3460 msgr2=0x5614f2f91e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.314+0000 7f995531a740 1 -- 172.21.15.29:0/442614714 shutdown_connections 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.314+0000 7f995531a740 1 -- 172.21.15.29:0/442614714 wait complete. 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.315+0000 7f995531a740 1 -- start start 2024-07-27T06:48:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.315+0000 7f995531a740 1 Processor -- start 2024-07-27T06:48:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.315+0000 7f995531a740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bad9f0 0x5614f2fac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.315+0000 7f995531a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 0x5614f2facdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.315+0000 7f995531a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 0x5614f2fa5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.315+0000 7f995531a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f99481b0630 con 0x5614f2bce390 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.315+0000 7f995531a740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f99481aff60 con 0x5614f2bad9f0 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.315+0000 7f995531a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f99481ad1a0 con 0x5614f2bc61b0 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.315+0000 7f995531a740 10 client.0 _ll_register_callbacks cb 0x7f993400a6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f994d333640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 0x5614f2fa5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f994d333640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 0x5614f2fa5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52894/0 (socket says 172.21.15.29:52894) 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f994d333640 1 -- 172.21.15.29:0/148223281 learned_addr learned my addr 172.21.15.29:0/148223281 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f994cb32640 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bad9f0 0x5614f2fac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f9947fff640 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 0x5614f2facdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f994d333640 1 -- 172.21.15.29:0/148223281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 msgr2=0x5614f2facdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f994d333640 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 0x5614f2facdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f994d333640 1 -- 172.21.15.29:0/148223281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bad9f0 msgr2=0x5614f2fac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f994d333640 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bad9f0 0x5614f2fac7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f994d333640 1 -- 172.21.15.29:0/148223281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614f2fa65c0 con 0x5614f2bce390 2024-07-27T06:48:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f994d333640 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 0x5614f2fa5e30 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9940009a60 tx=0x7f994002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c04e032f85538 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f9927fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f9927fff640 1 -- 172.21.15.29:0/148223281 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9940056060 con 0x5614f2bce390 2024-07-27T06:48:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f9927fff640 1 -- 172.21.15.29:0/148223281 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f9940004450 con 0x5614f2bce390 2024-07-27T06:48:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.316+0000 7f995531a740 1 -- 172.21.15.29:0/148223281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9934019580 con 0x5614f2bce390 2024-07-27T06:48:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.317+0000 7f9927fff640 1 -- 172.21.15.29:0/148223281 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9940004650 con 0x5614f2bce390 2024-07-27T06:48:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.317+0000 7f995531a740 10 client.15978 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.317+0000 7f995531a740 1 -- 172.21.15.29:0/148223281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614f2fabfa0 con 0x5614f2bce390 2024-07-27T06:48:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.317+0000 7f9927fff640 1 -- 172.21.15.29:0/148223281 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9934019580 con 0x5614f2bce390 2024-07-27T06:48:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.317+0000 7f9927fff640 1 -- 172.21.15.29:0/148223281 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9940043040 con 0x5614f2bce390 2024-07-27T06:48:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.317+0000 7f995531a740 10 client.15978 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.317+0000 7f995531a740 10 client.15978 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.317+0000 7f995531a740 20 client.15978 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.317+0000 7f995531a740 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x5614f2bef070 0x5614f301d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.317+0000 7f995531a740 4 client.15978 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.317+0000 7f995531a740 1 -- 172.21.15.29:0/148223281 --> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5614f2f085a0 con 0x5614f2bef070 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.318+0000 7f994cb32640 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x5614f2bef070 0x5614f301d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.318+0000 7f9927fff640 10 client.15978 ms_handle_connect on v2:172.21.15.77:6832/1085922080 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.318+0000 7f994cb32640 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x5614f2bef070 0x5614f301d000 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1171127de72625f7 server_cookie=bf973998c432c2f1 in_seq=0 out_seq=0 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.318+0000 7f9927fff640 1 -- 172.21.15.29:0/148223281 <== mds.4 v2:172.21.15.77:6832/1085922080 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5614f2f085a0 con 0x5614f2bef070 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.318+0000 7f9927fff640 10 client.15978 handle_command_reply: tid=0 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.318+0000 7f995531a740 1 client.15978 shutdown 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.319+0000 7f995531a740 1 -- 172.21.15.29:0/148223281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 msgr2=0x5614f2fa5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.319+0000 7f995531a740 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 0x5614f2fa5e30 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9940009a60 tx=0x7f994002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.319+0000 7f995531a740 1 -- 172.21.15.29:0/148223281 shutdown_connections 2024-07-27T06:48:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.319+0000 7f995531a740 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x5614f2bef070 0x5614f301d000 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.319+0000 7f995531a740 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5614f2bce390 0x5614f2fa5e30 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.319+0000 7f995531a740 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5614f2bc61b0 0x5614f2facdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.319+0000 7f995531a740 1 --2- 172.21.15.29:0/148223281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5614f2bad9f0 0x5614f2fac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.319+0000 7f9927fff640 0 client.15978 ms_handle_reset on v2:172.21.15.77:6832/1085922080 2024-07-27T06:48:33.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.319+0000 7f995531a740 1 -- 172.21.15.29:0/148223281 >> 172.21.15.29:0/148223281 conn(0x5614f2be3460 msgr2=0x5614f2f91cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:33.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.319+0000 7f995531a740 1 -- 172.21.15.29:0/148223281 shutdown_connections 2024-07-27T06:48:33.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.319+0000 7f995531a740 1 -- 172.21.15.29:0/148223281 wait complete. 2024-07-27T06:48:33.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.319+0000 7f995531a740 20 client.15978 trim_cache size 0 max 16384 2024-07-27T06:48:33.336 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:48:33.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.320+0000 7f994ed96640 1 -- 172.21.15.29:0/2776333303 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f99380b3d40 msgr2=0x7f99380b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:33.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.320+0000 7f994ed96640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f99380b3d40 0x7f99380b6250 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9940009430 tx=0x7f9940009800 comp rx=0 tx=0).stop 2024-07-27T06:48:33.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.320+0000 7f994ed96640 1 -- 172.21.15.29:0/2776333303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99481024c0 msgr2=0x7f9948198d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:33.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.320+0000 7f994ed96640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99481024c0 0x7f9948198d50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f993c00c890 tx=0x7f993c00cd50 comp rx=0 tx=0).stop 2024-07-27T06:48:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.320+0000 7f994ed96640 1 -- 172.21.15.29:0/2776333303 shutdown_connections 2024-07-27T06:48:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.320+0000 7f994ed96640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9948108750 0x7f994819e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.320+0000 7f994ed96640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9948107e00 0x7f99481992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.320+0000 7f994ed96640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f99380b3d40 0x7f99380b6250 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.320+0000 7f994ed96640 1 --2- 172.21.15.29:0/2776333303 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99481024c0 0x7f9948198d50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.320+0000 7f994ed96640 1 -- 172.21.15.29:0/2776333303 >> 172.21.15.29:0/2776333303 conn(0x7f9948079180 msgr2=0x7f9948103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.320+0000 7f994ed96640 1 -- 172.21.15.29:0/2776333303 shutdown_connections 2024-07-27T06:48:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:33.320+0000 7f994ed96640 1 -- 172.21.15.29:0/2776333303 wait complete. 2024-07-27T06:48:33.338 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-07-27T06:48:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:32 smithi077 ceph-mon[37153]: pgmap v443: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-27T06:48:33.371 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-27T06:48:33.372 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-27T06:48:33.373 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-07-27T06:48:33.413 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-27T06:48:33.416 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-07-27T06:48:33.417 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-07-27T06:48:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:32 smithi047 ceph-mon[36835]: pgmap v443: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-27T06:48:33.473 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-07-27T06:48:33.474 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-07-27T06:48:33.474 DEBUG:teuthology.orchestra.run.smithi029:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-07-27T06:48:34.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-27T06:48:34.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.805+0000 7fd3f315a640 1 -- 172.21.15.29:0/1533194878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3ec109820 msgr2=0x7fd3ec110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:34.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.805+0000 7fd3f315a640 1 --2- 172.21.15.29:0/1533194878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3ec109820 0x7fd3ec110050 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009900 tx=0x7fd3e402a410 comp rx=0 tx=0).stop 2024-07-27T06:48:34.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.805+0000 7fd3f315a640 1 -- 172.21.15.29:0/1533194878 shutdown_connections 2024-07-27T06:48:34.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.805+0000 7fd3f315a640 1 --2- 172.21.15.29:0/1533194878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3ec109820 0x7fd3ec110050 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:34.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.805+0000 7fd3f315a640 1 --2- 172.21.15.29:0/1533194878 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3ec108ed0 0x7fd3ec1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.805+0000 7fd3f315a640 1 --2- 172.21.15.29:0/1533194878 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3ec103590 0x7fd3ec1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.805+0000 7fd3f315a640 1 -- 172.21.15.29:0/1533194878 >> 172.21.15.29:0/1533194878 conn(0x7fd3ec07b280 msgr2=0x7fd3ec07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.805+0000 7fd3f315a640 1 -- 172.21.15.29:0/1533194878 shutdown_connections 2024-07-27T06:48:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.805+0000 7fd3f315a640 1 -- 172.21.15.29:0/1533194878 wait complete. 2024-07-27T06:48:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3f315a640 1 Processor -- start 2024-07-27T06:48:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3f315a640 1 -- start start 2024-07-27T06:48:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3f315a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3ec103590 0x7fd3ec075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3f315a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3ec108ed0 0x7fd3ec075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3f315a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3ec109820 0x7fd3ec075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3f315a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec113520 con 0x7fd3ec108ed0 2024-07-27T06:48:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3f315a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1133a0 con 0x7fd3ec109820 2024-07-27T06:48:34.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3f315a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1136a0 con 0x7fd3ec103590 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3ec108ed0 0x7fd3ec075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3ec108ed0 0x7fd3ec075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52906/0 (socket says 172.21.15.29:52906) 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3ebfff640 1 -- 172.21.15.29:0/3945307924 learned_addr learned my addr 172.21.15.29:0/3945307924 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3f0ef6640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3ec103590 0x7fd3ec075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.806+0000 7fd3f16f7640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3ec109820 0x7fd3ec075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.807+0000 7fd3ebfff640 1 -- 172.21.15.29:0/3945307924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3ec103590 msgr2=0x7fd3ec075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.807+0000 7fd3ebfff640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3ec103590 0x7fd3ec075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.807+0000 7fd3ebfff640 1 -- 172.21.15.29:0/3945307924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3ec109820 msgr2=0x7fd3ec075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.807+0000 7fd3ebfff640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3ec109820 0x7fd3ec075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.807+0000 7fd3ebfff640 1 -- 172.21.15.29:0/3945307924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ec1a8100 con 0x7fd3ec108ed0 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.807+0000 7fd3ebfff640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3ec108ed0 0x7fd3ec075680 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4006fd0 tx=0x7fd3d400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.807+0000 7fd3e9ffb640 1 -- 172.21.15.29:0/3945307924 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4011830 con 0x7fd3ec108ed0 2024-07-27T06:48:34.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.807+0000 7fd3e9ffb640 1 -- 172.21.15.29:0/3945307924 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3d4016070 con 0x7fd3ec108ed0 2024-07-27T06:48:34.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.807+0000 7fd3f315a640 1 -- 172.21.15.29:0/3945307924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ec1a83f0 con 0x7fd3ec108ed0 2024-07-27T06:48:34.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.807+0000 7fd3e9ffb640 1 -- 172.21.15.29:0/3945307924 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4010e00 con 0x7fd3ec108ed0 2024-07-27T06:48:34.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.807+0000 7fd3f315a640 1 -- 172.21.15.29:0/3945307924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ec1a89b0 con 0x7fd3ec108ed0 2024-07-27T06:48:34.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.808+0000 7fd3f315a640 1 -- 172.21.15.29:0/3945307924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3ec1039a0 con 0x7fd3ec108ed0 2024-07-27T06:48:34.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.810+0000 7fd3e9ffb640 1 -- 172.21.15.29:0/3945307924 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fd3d401b460 con 0x7fd3ec108ed0 2024-07-27T06:48:34.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.811+0000 7fd3e9ffb640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3d80b3d40 0x7fd3d80b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:34.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.811+0000 7fd3f0ef6640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3d80b3d40 0x7fd3d80b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:34.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.811+0000 7fd3e9ffb640 1 -- 172.21.15.29:0/3945307924 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd3d40e3490 con 0x7fd3ec108ed0 2024-07-27T06:48:34.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.811+0000 7fd3f0ef6640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3d80b3d40 0x7fd3d80b6250 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec076b90 tx=0x7fd3dc0075b0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:34.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:34.815+0000 7fd3e9ffb640 1 -- 172.21.15.29:0/3945307924 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd3d40aa8c0 con 0x7fd3ec108ed0 2024-07-27T06:48:34.909 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.053+0000 7fd3f315a640 1 -- 172.21.15.29:0/3945307924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3ec02e0c0 con 0x7fd3ec108ed0 2024-07-27T06:48:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.054+0000 7fd3e9ffb640 1 -- 172.21.15.29:0/3945307924 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11957 (secure 0 0 0) 0x7fd3d40aa1c0 con 0x7fd3ec108ed0 2024-07-27T06:48:35.056 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:48:35.056 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-07-27T06:46:50:421100+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:46:49.561737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24670,24658,24646,24617,24613]},"id":1}]} 2024-07-27T06:48:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-07-27T06:48:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.056+0000 7fd3f315a640 1 -- 172.21.15.29:0/3945307924 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3d80b3d40 msgr2=0x7fd3d80b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.056+0000 7fd3f315a640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3d80b3d40 0x7fd3d80b6250 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec076b90 tx=0x7fd3dc0075b0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.056+0000 7fd3f315a640 1 -- 172.21.15.29:0/3945307924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3ec108ed0 msgr2=0x7fd3ec075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.056+0000 7fd3f315a640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3ec108ed0 0x7fd3ec075680 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4006fd0 tx=0x7fd3d400b8e0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.056+0000 7fd3f315a640 1 -- 172.21.15.29:0/3945307924 shutdown_connections 2024-07-27T06:48:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.056+0000 7fd3f315a640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd3ec109820 0x7fd3ec075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.056+0000 7fd3f315a640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3ec108ed0 0x7fd3ec075680 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.056+0000 7fd3f315a640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd3d80b3d40 0x7fd3d80b6250 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.056+0000 7fd3f315a640 1 --2- 172.21.15.29:0/3945307924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3ec103590 0x7fd3ec075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.056+0000 7fd3f315a640 1 -- 172.21.15.29:0/3945307924 >> 172.21.15.29:0/3945307924 conn(0x7fd3ec07b280 msgr2=0x7fd3ec106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:35.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.056+0000 7fd3f315a640 1 -- 172.21.15.29:0/3945307924 shutdown_connections 2024-07-27T06:48:35.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:34 smithi029 ceph-mon[32610]: pgmap v444: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-27T06:48:35.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:35.056+0000 7fd3f315a640 1 -- 172.21.15.29:0/3945307924 wait complete. 2024-07-27T06:48:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:34 smithi077 ceph-mon[37153]: pgmap v444: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-27T06:48:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:34 smithi047 ceph-mon[36835]: pgmap v444: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-27T06:48:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.773+0000 7fd5eb006640 1 -- 172.21.15.29:0/3751101007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5e4108670 msgr2=0x7fd5e410eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.773+0000 7fd5eb006640 1 --2- 172.21.15.29:0/3751101007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5e4108670 0x7fd5e410eea0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc009ca0 tx=0x7fd5dc02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.773+0000 7fd5eb006640 1 -- 172.21.15.29:0/3751101007 shutdown_connections 2024-07-27T06:48:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.773+0000 7fd5eb006640 1 --2- 172.21.15.29:0/3751101007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5e4108670 0x7fd5e410eea0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.773+0000 7fd5eb006640 1 --2- 172.21.15.29:0/3751101007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5e4107d20 0x7fd5e4108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.773+0000 7fd5eb006640 1 --2- 172.21.15.29:0/3751101007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5e41023f0 0x7fd5e4102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.773+0000 7fd5eb006640 1 -- 172.21.15.29:0/3751101007 >> 172.21.15.29:0/3751101007 conn(0x7fd5e40fddf0 msgr2=0x7fd5e4100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.773+0000 7fd5eb006640 1 -- 172.21.15.29:0/3751101007 shutdown_connections 2024-07-27T06:48:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.773+0000 7fd5eb006640 1 -- 172.21.15.29:0/3751101007 wait complete. 2024-07-27T06:48:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.773+0000 7fd5eb006640 1 Processor -- start 2024-07-27T06:48:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.773+0000 7fd5eb006640 1 -- start start 2024-07-27T06:48:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5eb006640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5e41023f0 0x7fd5e419ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5eb006640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5e4107d20 0x7fd5e419b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5eb006640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5e4108670 0x7fd5e419b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5eb006640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e41123e0 con 0x7fd5e41023f0 2024-07-27T06:48:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5eb006640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e4112260 con 0x7fd5e4107d20 2024-07-27T06:48:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5eb006640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e4112560 con 0x7fd5e4108670 2024-07-27T06:48:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5e8da2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5e41023f0 0x7fd5e419ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5e8da2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5e41023f0 0x7fd5e419ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52920/0 (socket says 172.21.15.29:52920) 2024-07-27T06:48:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5e8da2640 1 -- 172.21.15.29:0/1532299648 learned_addr learned my addr 172.21.15.29:0/1532299648 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5e4107d20 0x7fd5e419b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5e8da2640 1 -- 172.21.15.29:0/1532299648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5e4108670 msgr2=0x7fd5e419b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5e8da2640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5e4108670 0x7fd5e419b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5e8da2640 1 -- 172.21.15.29:0/1532299648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5e4107d20 msgr2=0x7fd5e419b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5e8da2640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5e4107d20 0x7fd5e419b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5e8da2640 1 -- 172.21.15.29:0/1532299648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e41a5ea0 con 0x7fd5e41023f0 2024-07-27T06:48:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5e8da2640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5e41023f0 0x7fd5e419ac70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800b4a0 tx=0x7fd5d800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5e1ffb640 1 -- 172.21.15.29:0/1532299648 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8014070 con 0x7fd5e41023f0 2024-07-27T06:48:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5e1ffb640 1 -- 172.21.15.29:0/1532299648 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5d8010ac0 con 0x7fd5e41023f0 2024-07-27T06:48:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.774+0000 7fd5eb006640 1 -- 172.21.15.29:0/1532299648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e41a6190 con 0x7fd5e41023f0 2024-07-27T06:48:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.775+0000 7fd5e1ffb640 1 -- 172.21.15.29:0/1532299648 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8017c30 con 0x7fd5e41023f0 2024-07-27T06:48:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.775+0000 7fd5eb006640 1 -- 172.21.15.29:0/1532299648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e41a6760 con 0x7fd5e41023f0 2024-07-27T06:48:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.777+0000 7fd5e1ffb640 1 -- 172.21.15.29:0/1532299648 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fd5d8017070 con 0x7fd5e41023f0 2024-07-27T06:48:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.777+0000 7fd5f1ff7740 1 Processor -- start 2024-07-27T06:48:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.777+0000 7fd5f1ff7740 1 -- start start 2024-07-27T06:48:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5f1ff7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab879a1b0 0x564ab8b3cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5f1ff7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87712f0 0x564ab8b3d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5f1ff7740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b3d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5f1ff7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564ab8ab23b0 con 0x564ab879a1b0 2024-07-27T06:48:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5f1ff7740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x564ab8aa18b0 con 0x564ab87794d0 2024-07-27T06:48:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5f1ff7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564ab8aa58f0 con 0x564ab87712f0 2024-07-27T06:48:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e95a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab879a1b0 0x564ab8b3cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e95a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab879a1b0 0x564ab8b3cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52928/0 (socket says 172.21.15.29:52928) 2024-07-27T06:48:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e95a3640 1 -- 172.21.15.29:0/2112186198 learned_addr learned my addr 172.21.15.29:0/2112186198 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e8da2640 1 --2- 172.21.15.29:0/2112186198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87712f0 0x564ab8b3d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/2112186198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b3d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e1ffb640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd5d40b3d40 0x7fd5d40b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e95a3640 1 -- 172.21.15.29:0/2112186198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87712f0 msgr2=0x564ab8b3d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e95a3640 1 --2- 172.21.15.29:0/2112186198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87712f0 0x564ab8b3d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e95a3640 1 -- 172.21.15.29:0/2112186198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 msgr2=0x564ab8b3d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e95a3640 1 --2- 172.21.15.29:0/2112186198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b3d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e95a3640 1 -- 172.21.15.29:0/2112186198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ab8a489a0 con 0x564ab879a1b0 2024-07-27T06:48:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/2112186198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b3d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:48:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e1ffb640 1 -- 172.21.15.29:0/1532299648 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd5d80e36c0 con 0x7fd5e41023f0 2024-07-27T06:48:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e95a3640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd5d40b3d40 0x7fd5d40b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.778+0000 7fd5e95a3640 1 --2- 172.21.15.29:0/2112186198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab879a1b0 0x564ab8b3cce0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc02f4a0 tx=0x7fd5dc009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c98512df5a3f14 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.779+0000 7fd5ceffd640 1 -- 172.21.15.29:0/2112186198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc04c070 con 0x564ab879a1b0 2024-07-27T06:48:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.779+0000 7fd5ceffd640 1 -- 172.21.15.29:0/2112186198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5dc036070 con 0x564ab879a1b0 2024-07-27T06:48:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.779+0000 7fd5ceffd640 1 -- 172.21.15.29:0/2112186198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc02f990 con 0x564ab879a1b0 2024-07-27T06:48:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.780+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2112186198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab879a1b0 msgr2=0x564ab8b3cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.780+0000 7fd5e95a3640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd5d40b3d40 0x7fd5d40b6250 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc033040 tx=0x7fd5dc04a000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.780+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/2112186198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab879a1b0 0x564ab8b3cce0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc02f4a0 tx=0x7fd5dc009a20 comp rx=0 tx=0).stop 2024-07-27T06:48:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.780+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2112186198 shutdown_connections 2024-07-27T06:48:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.780+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/2112186198 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b3d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.780+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/2112186198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87712f0 0x564ab8b3d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.780+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/2112186198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab879a1b0 0x564ab8b3cce0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.780+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2112186198 >> 172.21.15.29:0/2112186198 conn(0x564ab878e300 msgr2=0x564ab8b3abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.780+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2112186198 shutdown_connections 2024-07-27T06:48:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.780+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2112186198 wait complete. 2024-07-27T06:48:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.780+0000 7fd5f1ff7740 1 -- start start 2024-07-27T06:48:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.780+0000 7fd5f1ff7740 1 Processor -- start 2024-07-27T06:48:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5f1ff7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 0x564ab8b4cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5f1ff7740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b4d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5f1ff7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab879a1b0 0x564ab8b58b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5f1ff7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564ab89dbab0 con 0x564ab87712f0 2024-07-27T06:48:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5f1ff7740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x564ab8a29cf0 con 0x564ab87794d0 2024-07-27T06:48:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5f1ff7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564ab8b4f840 con 0x564ab879a1b0 2024-07-27T06:48:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5f1ff7740 10 client.0 _ll_register_callbacks cb 0x564ab8a9b9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e8da2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 0x564ab8b4cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e8da2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 0x564ab8b4cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52944/0 (socket says 172.21.15.29:52944) 2024-07-27T06:48:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e8da2640 1 -- 172.21.15.29:0/781756432 learned_addr learned my addr 172.21.15.29:0/781756432 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e95a3640 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab879a1b0 0x564ab8b58b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b4d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e8da2640 1 -- 172.21.15.29:0/781756432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab879a1b0 msgr2=0x564ab8b58b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e8da2640 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab879a1b0 0x564ab8b58b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e8da2640 1 -- 172.21.15.29:0/781756432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 msgr2=0x564ab8b4d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e8da2640 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b4d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e8da2640 1 -- 172.21.15.29:0/781756432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ab8805790 con 0x564ab87712f0 2024-07-27T06:48:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b4d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:48:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e95a3640 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab879a1b0 0x564ab8b58b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:48:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5e8da2640 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 0x564ab8b4cc20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8000c00 tx=0x7fd5d80a9e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b918d6bb372ec37 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.781+0000 7fd5c3fff640 1 -- 172.21.15.29:0/781756432 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d801e030 con 0x564ab87712f0 2024-07-27T06:48:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5c3fff640 1 -- 172.21.15.29:0/781756432 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd5d80b3010 con 0x564ab87712f0 2024-07-27T06:48:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/781756432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5dc036070 con 0x564ab87712f0 2024-07-27T06:48:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5c3fff640 1 -- 172.21.15.29:0/781756432 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd5d80a9f00 con 0x564ab87712f0 2024-07-27T06:48:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5f1ff7740 10 client.15996 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/781756432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564ab887e620 con 0x564ab87712f0 2024-07-27T06:48:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5c3fff640 1 -- 172.21.15.29:0/781756432 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5dc036070 con 0x564ab87712f0 2024-07-27T06:48:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5c3fff640 1 -- 172.21.15.29:0/781756432 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd5d80e3b70 con 0x564ab87712f0 2024-07-27T06:48:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5f1ff7740 10 client.15996 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5f1ff7740 10 client.15996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5f1ff7740 20 client.15996 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564ab8758d80 0x564ab8b5ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5f1ff7740 4 client.15996 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/781756432 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ab8ab4300 con 0x564ab8758d80 2024-07-27T06:48:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.782+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564ab8758d80 0x564ab8b5ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.783+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564ab8758d80 0x564ab8b5ea40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f024a5f82bd4ce server_cookie=97d79419351c432b in_seq=0 out_seq=0 2024-07-27T06:48:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.783+0000 7fd5c3fff640 10 client.15996 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5c3fff640 1 -- 172.21.15.29:0/781756432 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564ab8ab4300 con 0x564ab8758d80 2024-07-27T06:48:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5c3fff640 10 client.15996 handle_command_reply: tid=0 2024-07-27T06:48:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5f1ff7740 1 client.15996 shutdown 2024-07-27T06:48:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/781756432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 msgr2=0x564ab8b4cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 0x564ab8b4cc20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8000c00 tx=0x7fd5d80a9e00 comp rx=0 tx=0).stop 2024-07-27T06:48:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/781756432 shutdown_connections 2024-07-27T06:48:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564ab8758d80 0x564ab8b5ea40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab879a1b0 0x564ab8b58b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b4d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5c3fff640 0 client.15996 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/781756432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 0x564ab8b4cc20 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/781756432 >> 172.21.15.29:0/781756432 conn(0x564ab878e300 msgr2=0x564ab8b3e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/781756432 shutdown_connections 2024-07-27T06:48:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/781756432 wait complete. 2024-07-27T06:48:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.784+0000 7fd5f1ff7740 20 client.15996 trim_cache size 0 max 16384 2024-07-27T06:48:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.800+0000 7fd5f1ff7740 1 Processor -- start 2024-07-27T06:48:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.800+0000 7fd5f1ff7740 1 -- start start 2024-07-27T06:48:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.800+0000 7fd5f1ff7740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab8758d80 0x564ab8b58960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.800+0000 7fd5f1ff7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 0x564ab8b58ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.800+0000 7fd5f1ff7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87794d0 0x564ab8b64220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.800+0000 7fd5f1ff7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e41acfb0 con 0x564ab87712f0 2024-07-27T06:48:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.800+0000 7fd5f1ff7740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e41aeaf0 con 0x564ab8758d80 2024-07-27T06:48:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.800+0000 7fd5f1ff7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e41ad680 con 0x564ab87794d0 2024-07-27T06:48:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.800+0000 7fd5e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 0x564ab8b58ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.800+0000 7fd5e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 0x564ab8b58ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52948/0 (socket says 172.21.15.29:52948) 2024-07-27T06:48:35.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.800+0000 7fd5e3fff640 1 -- 172.21.15.29:0/4035018031 learned_addr learned my addr 172.21.15.29:0/4035018031 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:35.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.800+0000 7fd5e95a3640 1 --2- 172.21.15.29:0/4035018031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87794d0 0x564ab8b64220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.801+0000 7fd5e3fff640 1 -- 172.21.15.29:0/4035018031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87794d0 msgr2=0x564ab8b64220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:35.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.801+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/4035018031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87794d0 0x564ab8b64220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.801+0000 7fd5e8da2640 1 --2- 172.21.15.29:0/4035018031 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab8758d80 0x564ab8b58960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.801+0000 7fd5e3fff640 1 -- 172.21.15.29:0/4035018031 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab8758d80 msgr2=0x564ab8b58960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:35.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.801+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/4035018031 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab8758d80 0x564ab8b58960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.801+0000 7fd5e3fff640 1 -- 172.21.15.29:0/4035018031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ab8b3fb40 con 0x564ab87712f0 2024-07-27T06:48:35.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.801+0000 7fd5e95a3640 1 --2- 172.21.15.29:0/4035018031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87794d0 0x564ab8b64220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:48:35.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.801+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/4035018031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 0x564ab8b58ea0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd5e406c390 tx=0x7fd5d0005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d900ea2a323bf39 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:35.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.801+0000 7fd5ce7fc640 1 -- 172.21.15.29:0/4035018031 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0010a30 con 0x564ab87712f0 2024-07-27T06:48:35.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.801+0000 7fd5ce7fc640 1 -- 172.21.15.29:0/4035018031 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5d0010bd0 con 0x564ab87712f0 2024-07-27T06:48:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.801+0000 7fd5ce7fc640 1 -- 172.21.15.29:0/4035018031 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d000e670 con 0x564ab87712f0 2024-07-27T06:48:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.802+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/4035018031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 msgr2=0x564ab8b58ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.802+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/4035018031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 0x564ab8b58ea0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd5e406c390 tx=0x7fd5d0005d30 comp rx=0 tx=0).stop 2024-07-27T06:48:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.802+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/4035018031 shutdown_connections 2024-07-27T06:48:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.802+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/4035018031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87794d0 0x564ab8b64220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.802+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/4035018031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab87712f0 0x564ab8b58ea0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.802+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/4035018031 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab8758d80 0x564ab8b58960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.802+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/4035018031 >> 172.21.15.29:0/4035018031 conn(0x564ab878e300 msgr2=0x564ab8b47b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.802+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/4035018031 shutdown_connections 2024-07-27T06:48:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.802+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/4035018031 wait complete. 2024-07-27T06:48:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.802+0000 7fd5f1ff7740 1 -- start start 2024-07-27T06:48:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.802+0000 7fd5f1ff7740 1 Processor -- start 2024-07-27T06:48:35.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5f1ff7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab8758d80 0x564ab8b552e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5f1ff7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87712f0 0x564ab8b537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5f1ff7740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b53e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5f1ff7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e41ae420 con 0x564ab8758d80 2024-07-27T06:48:35.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5f1ff7740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e41add50 con 0x564ab87794d0 2024-07-27T06:48:35.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5f1ff7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e41aafb0 con 0x564ab87712f0 2024-07-27T06:48:35.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5f1ff7740 10 client.0 _ll_register_callbacks cb 0x7fd5d80a6ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:35.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5e8da2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab8758d80 0x564ab8b552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5e8da2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab8758d80 0x564ab8b552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52958/0 (socket says 172.21.15.29:52958) 2024-07-27T06:48:35.810 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5e8da2640 1 -- 172.21.15.29:0/2763464455 learned_addr learned my addr 172.21.15.29:0/2763464455 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5e95a3640 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b53e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87712f0 0x564ab8b537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5e8da2640 1 -- 172.21.15.29:0/2763464455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87712f0 msgr2=0x564ab8b537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5e8da2640 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87712f0 0x564ab8b537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5e8da2640 1 -- 172.21.15.29:0/2763464455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 msgr2=0x564ab8b53e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5e8da2640 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b53e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5e8da2640 1 -- 172.21.15.29:0/2763464455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ab8b545c0 con 0x564ab8758d80 2024-07-27T06:48:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87712f0 0x564ab8b537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:48:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5e8da2640 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab8758d80 0x564ab8b552e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80aa0a0 tx=0x7fd5d800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1e9d30c94c43fc server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5c3fff640 1 -- 172.21.15.29:0/2763464455 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d801e070 con 0x564ab8758d80 2024-07-27T06:48:35.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.803+0000 7fd5c3fff640 1 -- 172.21.15.29:0/2763464455 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fd5d80a98d0 con 0x564ab8758d80 2024-07-27T06:48:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2763464455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5d0010bd0 con 0x564ab8758d80 2024-07-27T06:48:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5c3fff640 1 -- 172.21.15.29:0/2763464455 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd5d80a62b0 con 0x564ab8758d80 2024-07-27T06:48:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5f1ff7740 10 client.16002 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2763464455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564ab8bc6200 con 0x564ab8758d80 2024-07-27T06:48:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5c3fff640 1 -- 172.21.15.29:0/2763464455 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5d0010bd0 con 0x564ab8758d80 2024-07-27T06:48:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5c3fff640 1 -- 172.21.15.29:0/2763464455 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd5d80a4c10 con 0x564ab8758d80 2024-07-27T06:48:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5f1ff7740 10 client.16002 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5f1ff7740 10 client.16002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5f1ff7740 20 client.16002 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564ab879a1b0 0x564ab8bc8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:35.812 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5f1ff7740 4 client.16002 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-27T06:48:35.813 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2763464455 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564ab8aafda0 con 0x564ab879a1b0 2024-07-27T06:48:35.813 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.804+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564ab879a1b0 0x564ab8bc8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:35.813 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.805+0000 7fd5c3fff640 10 client.16002 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:35.813 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.805+0000 7fd5e3fff640 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564ab879a1b0 0x564ab8bc8830 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c9520ceb2e6ded0 server_cookie=54cc729ea1c19ae5 in_seq=0 out_seq=0 2024-07-27T06:48:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5c3fff640 1 -- 172.21.15.29:0/2763464455 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564ab8aafda0 con 0x564ab879a1b0 2024-07-27T06:48:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5c3fff640 10 client.16002 handle_command_reply: tid=0 2024-07-27T06:48:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5f1ff7740 1 client.16002 shutdown 2024-07-27T06:48:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2763464455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab8758d80 msgr2=0x564ab8b552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:35.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab8758d80 0x564ab8b552e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80aa0a0 tx=0x7fd5d800bf90 comp rx=0 tx=0).stop 2024-07-27T06:48:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2763464455 shutdown_connections 2024-07-27T06:48:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564ab87794d0 0x564ab8b53e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x564ab879a1b0 0x564ab8bc8830 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ab87712f0 0x564ab8b537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5f1ff7740 1 --2- 172.21.15.29:0/2763464455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564ab8758d80 0x564ab8b552e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5c3fff640 0 client.16002 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2763464455 >> 172.21.15.29:0/2763464455 conn(0x564ab878e300 msgr2=0x564ab8b39c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2763464455 shutdown_connections 2024-07-27T06:48:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5f1ff7740 1 -- 172.21.15.29:0/2763464455 wait complete. 2024-07-27T06:48:35.815 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.811+0000 7fd5f1ff7740 20 client.16002 trim_cache size 0 max 16384 2024-07-27T06:48:35.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-07-27T06:48:35.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-07-27T06:48:35.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-07-27T06:48:35.816 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-07-27T06:48:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.812+0000 7fd5eb006640 1 -- 172.21.15.29:0/1532299648 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd5d40b3d40 msgr2=0x7fd5d40b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.812+0000 7fd5eb006640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd5d40b3d40 0x7fd5d40b6250 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc033040 tx=0x7fd5dc04a000 comp rx=0 tx=0).stop 2024-07-27T06:48:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.812+0000 7fd5eb006640 1 -- 172.21.15.29:0/1532299648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5e41023f0 msgr2=0x7fd5e419ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.812+0000 7fd5eb006640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5e41023f0 0x7fd5e419ac70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800b4a0 tx=0x7fd5d800b960 comp rx=0 tx=0).stop 2024-07-27T06:48:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.812+0000 7fd5eb006640 1 -- 172.21.15.29:0/1532299648 shutdown_connections 2024-07-27T06:48:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.812+0000 7fd5eb006640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fd5d40b3d40 0x7fd5d40b6250 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.817 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.812+0000 7fd5eb006640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5e4108670 0x7fd5e419b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.812+0000 7fd5eb006640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5e4107d20 0x7fd5e419b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.812+0000 7fd5eb006640 1 --2- 172.21.15.29:0/1532299648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5e41023f0 0x7fd5e419ac70 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:35.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.812+0000 7fd5eb006640 1 -- 172.21.15.29:0/1532299648 >> 172.21.15.29:0/1532299648 conn(0x7fd5e40fddf0 msgr2=0x7fd5e41072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:35.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.812+0000 7fd5eb006640 1 -- 172.21.15.29:0/1532299648 shutdown_connections 2024-07-27T06:48:35.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:35.812+0000 7fd5eb006640 1 -- 172.21.15.29:0/1532299648 wait complete. 2024-07-27T06:48:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:35 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3945307924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:48:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:35 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3945307924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:48:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:35 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3945307924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:48:37.126 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24670 quiesce db --set-id 7f206a77 --release --await' 2024-07-27T06:48:37.126 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- timeout 105 ceph tell mds.24670 quiesce db --set-id 7f206a77 --release --await 2024-07-27T06:48:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:36 smithi029 ceph-mon[32610]: pgmap v445: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-27T06:48:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:36 smithi077 ceph-mon[37153]: pgmap v445: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-27T06:48:37.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.378+0000 7fe1a2a82640 1 -- 172.21.15.29:0/514505282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c1024b0 msgr2=0x7fe19c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:37.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.378+0000 7fe1a2a82640 1 --2- 172.21.15.29:0/514505282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c1024b0 0x7fe19c1028c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe190009900 tx=0x7fe19002a410 comp rx=0 tx=0).stop 2024-07-27T06:48:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.378+0000 7fe1a2a82640 1 -- 172.21.15.29:0/514505282 shutdown_connections 2024-07-27T06:48:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.378+0000 7fe1a2a82640 1 --2- 172.21.15.29:0/514505282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe19c108740 0x7fe19c10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.378+0000 7fe1a2a82640 1 --2- 172.21.15.29:0/514505282 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe19c107df0 0x7fe19c108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.378+0000 7fe1a2a82640 1 --2- 172.21.15.29:0/514505282 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c1024b0 0x7fe19c1028c0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.378+0000 7fe1a2a82640 1 -- 172.21.15.29:0/514505282 >> 172.21.15.29:0/514505282 conn(0x7fe19c079170 msgr2=0x7fe19c079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.378+0000 7fe1a2a82640 1 -- 172.21.15.29:0/514505282 shutdown_connections 2024-07-27T06:48:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.378+0000 7fe1a2a82640 1 -- 172.21.15.29:0/514505282 wait complete. 2024-07-27T06:48:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a2a82640 1 Processor -- start 2024-07-27T06:48:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a2a82640 1 -- start start 2024-07-27T06:48:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a2a82640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c1024b0 0x7fe19c199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a2a82640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe19c107df0 0x7fe19c19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a2a82640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe19c108740 0x7fe19c19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a2a82640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c113560 con 0x7fe19c1024b0 2024-07-27T06:48:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a2a82640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c1133e0 con 0x7fe19c108740 2024-07-27T06:48:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a2a82640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c1136e0 con 0x7fe19c107df0 2024-07-27T06:48:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a081e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c1024b0 0x7fe19c199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a081e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c1024b0 0x7fe19c199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52976/0 (socket says 172.21.15.29:52976) 2024-07-27T06:48:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a081e640 1 -- 172.21.15.29:0/3336732706 learned_addr learned my addr 172.21.15.29:0/3336732706 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe19bfff640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe19c107df0 0x7fe19c19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe19c108740 0x7fe19c19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a081e640 1 -- 172.21.15.29:0/3336732706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe19c107df0 msgr2=0x7fe19c19a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a081e640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe19c107df0 0x7fe19c19a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a081e640 1 -- 172.21.15.29:0/3336732706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe19c108740 msgr2=0x7fe19c19f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a081e640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe19c108740 0x7fe19c19f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.379+0000 7fe1a081e640 1 -- 172.21.15.29:0/3336732706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19c19ff00 con 0x7fe19c1024b0 2024-07-27T06:48:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.380+0000 7fe1a081e640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c1024b0 0x7fe19c199f90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe190005ab0 tx=0x7fe190003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.380+0000 7fe199ffb640 1 -- 172.21.15.29:0/3336732706 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19003f070 con 0x7fe19c1024b0 2024-07-27T06:48:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.380+0000 7fe1a2a82640 1 -- 172.21.15.29:0/3336732706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19c1a0190 con 0x7fe19c1024b0 2024-07-27T06:48:37.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.380+0000 7fe199ffb640 1 -- 172.21.15.29:0/3336732706 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe190031d60 con 0x7fe19c1024b0 2024-07-27T06:48:37.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.380+0000 7fe199ffb640 1 -- 172.21.15.29:0/3336732706 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe190035030 con 0x7fe19c1024b0 2024-07-27T06:48:37.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.380+0000 7fe1a2a82640 1 -- 172.21.15.29:0/3336732706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe19c1a6950 con 0x7fe19c1024b0 2024-07-27T06:48:37.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe1a9006740 1 Processor -- start 2024-07-27T06:48:37.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe1a9006740 1 -- start start 2024-07-27T06:48:37.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe199ffb640 1 -- 172.21.15.29:0/3336732706 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fe190042070 con 0x7fe19c1024b0 2024-07-27T06:48:37.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe1a9006740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5fb0070 0x564fe6355570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe1a9006740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f871b0 0x564fe6355ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe1a9006740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f8f390 0x564fe6355ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe1a9006740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564fe62cbbb0 con 0x564fe5f871b0 2024-07-27T06:48:37.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe1a9006740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x564fe62bb0b0 con 0x564fe5fb0070 2024-07-27T06:48:37.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe1a9006740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564fe62bf0f0 con 0x564fe5f8f390 2024-07-27T06:48:37.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe1a081e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f871b0 0x564fe6355ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe1a081e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f871b0 0x564fe6355ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52978/0 (socket says 172.21.15.29:52978) 2024-07-27T06:48:37.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe1a081e640 1 -- 172.21.15.29:0/3650122475 learned_addr learned my addr 172.21.15.29:0/3650122475 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:37.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3650122475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5fb0070 0x564fe6355570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.383+0000 7fe19bfff640 1 --2- 172.21.15.29:0/3650122475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f8f390 0x564fe6355ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe1a101f640 1 -- 172.21.15.29:0/3650122475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f8f390 msgr2=0x564fe6355ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:37.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3650122475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f8f390 0x564fe6355ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe1a101f640 1 -- 172.21.15.29:0/3650122475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f871b0 msgr2=0x564fe6355ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:37.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3650122475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f871b0 0x564fe6355ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe1a101f640 1 -- 172.21.15.29:0/3650122475 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fe6261b90 con 0x564fe5fb0070 2024-07-27T06:48:37.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe19bfff640 1 --2- 172.21.15.29:0/3650122475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f8f390 0x564fe6355ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe1a081e640 1 --2- 172.21.15.29:0/3650122475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f871b0 0x564fe6355ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe199ffb640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe18c0b3d40 0x7fe18c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe1a081e640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe18c0b3d40 0x7fe18c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3650122475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5fb0070 0x564fe6355570 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe194008d00 tx=0x7fe19400e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7bc44b8b399f642 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe199ffb640 1 -- 172.21.15.29:0/3336732706 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fe1901047a0 con 0x7fe19c1024b0 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe186ffd640 1 -- 172.21.15.29:0/3650122475 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe194014070 con 0x564fe5fb0070 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe186ffd640 1 -- 172.21.15.29:0/3650122475 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe194003b70 con 0x564fe5fb0070 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.384+0000 7fe186ffd640 1 -- 172.21.15.29:0/3650122475 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe194010790 con 0x564fe5fb0070 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.385+0000 7fe1a9006740 1 -- 172.21.15.29:0/3650122475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5fb0070 msgr2=0x564fe6355570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.385+0000 7fe1a081e640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe18c0b3d40 0x7fe18c0b6250 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe190000c00 tx=0x7fe19010a550 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.385+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3650122475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5fb0070 0x564fe6355570 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe194008d00 tx=0x7fe19400e720 comp rx=0 tx=0).stop 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.385+0000 7fe1a9006740 1 -- 172.21.15.29:0/3650122475 shutdown_connections 2024-07-27T06:48:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.385+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3650122475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f8f390 0x564fe6355ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.385+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3650122475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f871b0 0x564fe6355ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.385+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3650122475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5fb0070 0x564fe6355570 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.385+0000 7fe1a9006740 1 -- 172.21.15.29:0/3650122475 >> 172.21.15.29:0/3650122475 conn(0x564fe5fa4460 msgr2=0x564fe6353460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.385+0000 7fe1a9006740 1 -- 172.21.15.29:0/3650122475 shutdown_connections 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.385+0000 7fe1a9006740 1 -- 172.21.15.29:0/3650122475 wait complete. 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a9006740 1 -- start start 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a9006740 1 Processor -- start 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a9006740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f871b0 0x564fe63611b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a9006740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f8f390 0x564fe6361710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a9006740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5fb0070 0x564fe636d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a9006740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564fe611e700 con 0x564fe5fb0070 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a9006740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x564fe625d9e0 con 0x564fe5f8f390 2024-07-27T06:48:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a9006740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564fe6363d50 con 0x564fe5f871b0 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a9006740 10 client.0 _ll_register_callbacks cb 0x564fe62784f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a101f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5fb0070 0x564fe636d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a101f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5fb0070 0x564fe636d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52986/0 (socket says 172.21.15.29:52986) 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a101f640 1 -- 172.21.15.29:0/3664558984 learned_addr learned my addr 172.21.15.29:0/3664558984 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe19bfff640 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f8f390 0x564fe6361710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.386+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f871b0 0x564fe63611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe1a101f640 1 -- 172.21.15.29:0/3664558984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f871b0 msgr2=0x564fe63611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f871b0 0x564fe63611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe1a101f640 1 -- 172.21.15.29:0/3664558984 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f8f390 msgr2=0x564fe6361710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f8f390 0x564fe6361710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe1a101f640 1 -- 172.21.15.29:0/3664558984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fe601b650 con 0x564fe5fb0070 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe19bfff640 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f8f390 0x564fe6361710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5fb0070 0x564fe636d0a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x564fe6363800 tx=0x7fe194010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cac00ca0e209b92 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe17bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe17bfff640 1 -- 172.21.15.29:0/3664558984 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe194014040 con 0x564fe5fb0070 2024-07-27T06:48:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe17bfff640 1 -- 172.21.15.29:0/3664558984 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fe194003970 con 0x564fe5fb0070 2024-07-27T06:48:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe1a9006740 1 -- 172.21.15.29:0/3664558984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe194003b70 con 0x564fe5fb0070 2024-07-27T06:48:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe17bfff640 1 -- 172.21.15.29:0/3664558984 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe1940044e0 con 0x564fe5fb0070 2024-07-27T06:48:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe1a9006740 10 client.16014 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe1a9006740 1 -- 172.21.15.29:0/3664558984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564fe636fdf0 con 0x564fe5fb0070 2024-07-27T06:48:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.387+0000 7fe17bfff640 1 -- 172.21.15.29:0/3664558984 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe194003b70 con 0x564fe5fb0070 2024-07-27T06:48:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.388+0000 7fe17bfff640 1 -- 172.21.15.29:0/3664558984 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe194026040 con 0x564fe5fb0070 2024-07-27T06:48:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.388+0000 7fe1a9006740 10 client.16014 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.388+0000 7fe1a9006740 10 client.16014 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-07-27T06:48:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.388+0000 7fe1a9006740 20 client.16014 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.388+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x564fe5f6e9f0 0x564fe63731b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.388+0000 7fe1a9006740 4 client.16014 mds_command: new command op to 24617 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.388+0000 7fe1a9006740 1 -- 172.21.15.29:0/3664558984 --> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564fe62cdb00 con 0x564fe5f6e9f0 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.388+0000 7fe19bfff640 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x564fe5f6e9f0 0x564fe63731b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.389+0000 7fe19bfff640 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x564fe5f6e9f0 0x564fe63731b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9f6d007fdcc61413 server_cookie=7ebc1e4765e12169 in_seq=0 out_seq=0 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.389+0000 7fe17bfff640 10 client.16014 ms_handle_connect on v2:172.21.15.47:6838/670107266 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.389+0000 7fe17bfff640 1 -- 172.21.15.29:0/3664558984 <== mds.3 v2:172.21.15.47:6838/670107266 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564fe62cdb00 con 0x564fe5f6e9f0 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.389+0000 7fe17bfff640 10 client.16014 handle_command_reply: tid=0 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.389+0000 7fe1a9006740 1 client.16014 shutdown 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.390+0000 7fe1a9006740 1 -- 172.21.15.29:0/3664558984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5fb0070 msgr2=0x564fe636d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.390+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5fb0070 0x564fe636d0a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x564fe6363800 tx=0x7fe194010430 comp rx=0 tx=0).stop 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.390+0000 7fe1a9006740 1 -- 172.21.15.29:0/3664558984 shutdown_connections 2024-07-27T06:48:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.390+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x564fe5f6e9f0 0x564fe63731b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.390+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5fb0070 0x564fe636d0a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.390+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f8f390 0x564fe6361710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.390+0000 7fe17bfff640 0 client.16014 ms_handle_reset on v2:172.21.15.47:6838/670107266 2024-07-27T06:48:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.390+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3664558984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f871b0 0x564fe63611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.390+0000 7fe1a9006740 1 -- 172.21.15.29:0/3664558984 >> 172.21.15.29:0/3664558984 conn(0x564fe5fa4460 msgr2=0x564fe6351c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.390+0000 7fe1a9006740 1 -- 172.21.15.29:0/3664558984 shutdown_connections 2024-07-27T06:48:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.390+0000 7fe1a9006740 1 -- 172.21.15.29:0/3664558984 wait complete. 2024-07-27T06:48:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.390+0000 7fe1a9006740 20 client.16014 trim_cache size 0 max 16384 2024-07-27T06:48:37.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.403+0000 7fe1a9006740 1 Processor -- start 2024-07-27T06:48:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.403+0000 7fe1a9006740 1 -- start start 2024-07-27T06:48:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.403+0000 7fe1a9006740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f6e9f0 0x564fe635ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.403+0000 7fe1a9006740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f871b0 0x564fe636ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.403+0000 7fe1a9006740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f8f390 0x564fe636d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.403+0000 7fe1a9006740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c1af1d0 con 0x564fe5f8f390 2024-07-27T06:48:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe1a9006740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c1b0d10 con 0x564fe5f871b0 2024-07-27T06:48:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe1a9006740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c1af8a0 con 0x564fe5f6e9f0 2024-07-27T06:48:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe1a101f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f8f390 0x564fe636d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe1a101f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f8f390 0x564fe636d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52998/0 (socket says 172.21.15.29:52998) 2024-07-27T06:48:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe1a101f640 1 -- 172.21.15.29:0/3213419519 learned_addr learned my addr 172.21.15.29:0/3213419519 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3213419519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f6e9f0 0x564fe635ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe19bfff640 1 --2- 172.21.15.29:0/3213419519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f871b0 0x564fe636ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe1a101f640 1 -- 172.21.15.29:0/3213419519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f6e9f0 msgr2=0x564fe635ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:37.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3213419519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f6e9f0 0x564fe635ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe1a101f640 1 -- 172.21.15.29:0/3213419519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f871b0 msgr2=0x564fe636ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:37.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3213419519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f871b0 0x564fe636ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe1a101f640 1 -- 172.21.15.29:0/3213419519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fe63583d0 con 0x564fe5f8f390 2024-07-27T06:48:37.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.404+0000 7fe1a101f640 1 --2- 172.21.15.29:0/3213419519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f8f390 0x564fe636d3d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe1940309e0 tx=0x7fe19400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dadb065157a650 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:37.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.405+0000 7fe1867fc640 1 -- 172.21.15.29:0/3213419519 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19400ec20 con 0x564fe5f8f390 2024-07-27T06:48:37.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.405+0000 7fe1867fc640 1 -- 172.21.15.29:0/3213419519 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe194003970 con 0x564fe5f8f390 2024-07-27T06:48:37.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.405+0000 7fe1867fc640 1 -- 172.21.15.29:0/3213419519 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe194004b00 con 0x564fe5f8f390 2024-07-27T06:48:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.406+0000 7fe1a9006740 1 -- 172.21.15.29:0/3213419519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f8f390 msgr2=0x564fe636d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.406+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3213419519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f8f390 0x564fe636d3d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe1940309e0 tx=0x7fe19400e720 comp rx=0 tx=0).stop 2024-07-27T06:48:37.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.406+0000 7fe1a9006740 1 -- 172.21.15.29:0/3213419519 shutdown_connections 2024-07-27T06:48:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.406+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3213419519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f8f390 0x564fe636d3d0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.406+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3213419519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f871b0 0x564fe636ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.406+0000 7fe1a9006740 1 --2- 172.21.15.29:0/3213419519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f6e9f0 0x564fe635ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.406+0000 7fe1a9006740 1 -- 172.21.15.29:0/3213419519 >> 172.21.15.29:0/3213419519 conn(0x564fe5fa4460 msgr2=0x564fe635a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.406+0000 7fe1a9006740 1 -- 172.21.15.29:0/3213419519 shutdown_connections 2024-07-27T06:48:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.406+0000 7fe1a9006740 1 -- 172.21.15.29:0/3213419519 wait complete. 2024-07-27T06:48:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.406+0000 7fe1a9006740 1 -- start start 2024-07-27T06:48:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.406+0000 7fe1a9006740 1 Processor -- start 2024-07-27T06:48:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a9006740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f6e9f0 0x564fe63694e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a9006740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f871b0 0x564fe6364990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a9006740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f8f390 0x564fe6364ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a9006740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c1b0640 con 0x564fe5f6e9f0 2024-07-27T06:48:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a9006740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c1aff70 con 0x564fe5f871b0 2024-07-27T06:48:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a9006740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c1ad1b0 con 0x564fe5f8f390 2024-07-27T06:48:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a9006740 10 client.0 _ll_register_callbacks cb 0x7fe19010a420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a101f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f6e9f0 0x564fe63694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a101f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f6e9f0 0x564fe63694e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53008/0 (socket says 172.21.15.29:53008) 2024-07-27T06:48:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a101f640 1 -- 172.21.15.29:0/4250691360 learned_addr learned my addr 172.21.15.29:0/4250691360 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe19bfff640 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f871b0 0x564fe6364990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a101f640 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f8f390 0x564fe6364ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a101f640 1 -- 172.21.15.29:0/4250691360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f8f390 msgr2=0x564fe6364ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a101f640 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f8f390 0x564fe6364ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a101f640 1 -- 172.21.15.29:0/4250691360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f871b0 msgr2=0x564fe6364990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a101f640 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f871b0 0x564fe6364990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe1a101f640 1 -- 172.21.15.29:0/4250691360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fe6357d80 con 0x564fe5f6e9f0 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.407+0000 7fe19bfff640 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f871b0 0x564fe6364990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.408+0000 7fe1a101f640 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f6e9f0 0x564fe63694e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe1940104e0 tx=0x7fe194030860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7245ad1dafbbe07c server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.408+0000 7fe17bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.408+0000 7fe17bfff640 1 -- 172.21.15.29:0/4250691360 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe194046060 con 0x564fe5f6e9f0 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.408+0000 7fe17bfff640 1 -- 172.21.15.29:0/4250691360 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fe194004380 con 0x564fe5f6e9f0 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.408+0000 7fe1a9006740 1 -- 172.21.15.29:0/4250691360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe194003970 con 0x564fe5f6e9f0 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.408+0000 7fe17bfff640 1 -- 172.21.15.29:0/4250691360 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe194004580 con 0x564fe5f6e9f0 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.408+0000 7fe1a9006740 10 client.16020 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.408+0000 7fe1a9006740 1 -- 172.21.15.29:0/4250691360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564fe63db330 con 0x564fe5f6e9f0 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.408+0000 7fe17bfff640 1 -- 172.21.15.29:0/4250691360 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe194003970 con 0x564fe5f6e9f0 2024-07-27T06:48:37.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.409+0000 7fe17bfff640 1 -- 172.21.15.29:0/4250691360 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe1940107b0 con 0x564fe5f6e9f0 2024-07-27T06:48:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.409+0000 7fe1a9006740 10 client.16020 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.409+0000 7fe1a9006740 10 client.16020 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-07-27T06:48:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.409+0000 7fe1a9006740 20 client.16020 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.409+0000 7fe1a9006740 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x564fe5fb0070 0x564fe63dd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.409+0000 7fe1a9006740 4 client.16020 mds_command: new command op to 24617 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:48:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.409+0000 7fe1a9006740 1 -- 172.21.15.29:0/4250691360 --> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564fe62c95a0 con 0x564fe5fb0070 2024-07-27T06:48:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.409+0000 7fe19bfff640 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x564fe5fb0070 0x564fe63dd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe17bfff640 10 client.16020 ms_handle_connect on v2:172.21.15.47:6838/670107266 2024-07-27T06:48:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe19bfff640 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x564fe5fb0070 0x564fe63dd9d0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d4f7410b15c4d91e server_cookie=d8785782279481c2 in_seq=0 out_seq=0 2024-07-27T06:48:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe17bfff640 1 -- 172.21.15.29:0/4250691360 <== mds.3 v2:172.21.15.47:6838/670107266 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564fe62c95a0 con 0x564fe5fb0070 2024-07-27T06:48:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe17bfff640 10 client.16020 handle_command_reply: tid=0 2024-07-27T06:48:37.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe1a9006740 1 client.16020 shutdown 2024-07-27T06:48:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe1a9006740 1 -- 172.21.15.29:0/4250691360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f6e9f0 msgr2=0x564fe63694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe1a9006740 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f6e9f0 0x564fe63694e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe1940104e0 tx=0x7fe194030860 comp rx=0 tx=0).stop 2024-07-27T06:48:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe1a9006740 1 -- 172.21.15.29:0/4250691360 shutdown_connections 2024-07-27T06:48:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe1a9006740 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x564fe5fb0070 0x564fe63dd9d0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe17bfff640 0 client.16020 ms_handle_reset on v2:172.21.15.47:6838/670107266 2024-07-27T06:48:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe1a9006740 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564fe5f8f390 0x564fe6364ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe1a9006740 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x564fe5f871b0 0x564fe6364990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe1a9006740 1 --2- 172.21.15.29:0/4250691360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564fe5f6e9f0 0x564fe63694e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe1a9006740 1 -- 172.21.15.29:0/4250691360 >> 172.21.15.29:0/4250691360 conn(0x564fe5fa4460 msgr2=0x564fe63565a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:37.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe1a9006740 1 -- 172.21.15.29:0/4250691360 shutdown_connections 2024-07-27T06:48:37.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.410+0000 7fe1a9006740 1 -- 172.21.15.29:0/4250691360 wait complete. 2024-07-27T06:48:37.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.411+0000 7fe1a9006740 20 client.16020 trim_cache size 0 max 16384 2024-07-27T06:48:37.423 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:48:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.411+0000 7fe1a2a82640 1 -- 172.21.15.29:0/3336732706 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe18c0b3d40 msgr2=0x7fe18c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.411+0000 7fe1a2a82640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe18c0b3d40 0x7fe18c0b6250 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe190000c00 tx=0x7fe19010a550 comp rx=0 tx=0).stop 2024-07-27T06:48:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.411+0000 7fe1a2a82640 1 -- 172.21.15.29:0/3336732706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c1024b0 msgr2=0x7fe19c199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.411+0000 7fe1a2a82640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c1024b0 0x7fe19c199f90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe190005ab0 tx=0x7fe190003d50 comp rx=0 tx=0).stop 2024-07-27T06:48:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.411+0000 7fe1a2a82640 1 -- 172.21.15.29:0/3336732706 shutdown_connections 2024-07-27T06:48:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.411+0000 7fe1a2a82640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe19c108740 0x7fe19c19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.411+0000 7fe1a2a82640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe18c0b3d40 0x7fe18c0b6250 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.411+0000 7fe1a2a82640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe19c107df0 0x7fe19c19a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.411+0000 7fe1a2a82640 1 --2- 172.21.15.29:0/3336732706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c1024b0 0x7fe19c199f90 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:37.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.411+0000 7fe1a2a82640 1 -- 172.21.15.29:0/3336732706 >> 172.21.15.29:0/3336732706 conn(0x7fe19c079170 msgr2=0x7fe19c100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:37.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.411+0000 7fe1a2a82640 1 -- 172.21.15.29:0/3336732706 shutdown_connections 2024-07-27T06:48:37.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:37.412+0000 7fe1a2a82640 1 -- 172.21.15.29:0/3336732706 wait complete. 2024-07-27T06:48:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:36 smithi047 ceph-mon[36835]: pgmap v445: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-27T06:48:38.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-27T06:48:38.419 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:39.152 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:38 smithi029 ceph-mon[32610]: pgmap v446: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-07-27T06:48:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:38 smithi077 ceph-mon[37153]: pgmap v446: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-07-27T06:48:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:38 smithi047 ceph-mon[36835]: pgmap v446: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-07-27T06:48:40.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.881+0000 7f44c567c640 1 -- 172.21.15.29:0/3610144850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c0103590 msgr2=0x7f44c01039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:40.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.881+0000 7f44c567c640 1 --2- 172.21.15.29:0/3610144850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c0103590 0x7f44c01039a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009900 tx=0x7f44b002a410 comp rx=0 tx=0).stop 2024-07-27T06:48:40.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.881+0000 7f44c567c640 1 -- 172.21.15.29:0/3610144850 shutdown_connections 2024-07-27T06:48:40.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.881+0000 7f44c567c640 1 --2- 172.21.15.29:0/3610144850 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44c0109820 0x7f44c0110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.881+0000 7f44c567c640 1 --2- 172.21.15.29:0/3610144850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0108ed0 0x7f44c01092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.881+0000 7f44c567c640 1 --2- 172.21.15.29:0/3610144850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c0103590 0x7f44c01039a0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.881+0000 7f44c567c640 1 -- 172.21.15.29:0/3610144850 >> 172.21.15.29:0/3610144850 conn(0x7f44c007b280 msgr2=0x7f44c007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:40.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.881+0000 7f44c567c640 1 -- 172.21.15.29:0/3610144850 shutdown_connections 2024-07-27T06:48:40.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.881+0000 7f44c567c640 1 -- 172.21.15.29:0/3610144850 wait complete. 2024-07-27T06:48:40.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.882+0000 7f44c567c640 1 Processor -- start 2024-07-27T06:48:40.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.882+0000 7f44c567c640 1 -- start start 2024-07-27T06:48:40.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.882+0000 7f44c567c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c0103590 0x7f44c019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.882+0000 7f44c567c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0108ed0 0x7f44c019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.882+0000 7f44c567c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44c0109820 0x7f44c019f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44c567c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01135a0 con 0x7f44c0103590 2024-07-27T06:48:40.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44c567c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f44c0113420 con 0x7f44c0109820 2024-07-27T06:48:40.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44c567c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f44c0113720 con 0x7f44c0108ed0 2024-07-27T06:48:40.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44beffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c0103590 0x7f44c019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44beffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c0103590 0x7f44c019a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41534/0 (socket says 172.21.15.29:41534) 2024-07-27T06:48:40.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44beffd640 1 -- 172.21.15.29:0/2766332465 learned_addr learned my addr 172.21.15.29:0/2766332465 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:40.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44b7fff640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0108ed0 0x7f44c019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44c0109820 0x7f44c019f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44beffd640 1 -- 172.21.15.29:0/2766332465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0108ed0 msgr2=0x7f44c019a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:40.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44beffd640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0108ed0 0x7f44c019a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44beffd640 1 -- 172.21.15.29:0/2766332465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44c0109820 msgr2=0x7f44c019f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:40.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44beffd640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44c0109820 0x7f44c019f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44beffd640 1 -- 172.21.15.29:0/2766332465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44c019ffc0 con 0x7f44c0103590 2024-07-27T06:48:40.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44c0109820 0x7f44c019f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:48:40.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44beffd640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c0103590 0x7f44c019a050 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f44b0006010 tx=0x7f44b00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:40.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.883+0000 7f44bcff9640 1 -- 172.21.15.29:0/2766332465 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b003f030 con 0x7f44c0103590 2024-07-27T06:48:40.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.884+0000 7f44bcff9640 1 -- 172.21.15.29:0/2766332465 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b0003800 con 0x7f44c0103590 2024-07-27T06:48:40.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.884+0000 7f44c567c640 1 -- 172.21.15.29:0/2766332465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44c01a0250 con 0x7f44c0103590 2024-07-27T06:48:40.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.884+0000 7f44bcff9640 1 -- 172.21.15.29:0/2766332465 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b0035070 con 0x7f44c0103590 2024-07-27T06:48:40.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.884+0000 7f44c567c640 1 -- 172.21.15.29:0/2766332465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44c01a6a10 con 0x7f44c0103590 2024-07-27T06:48:40.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.887+0000 7f44cbc00740 1 Processor -- start 2024-07-27T06:48:40.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.887+0000 7f44cbc00740 1 -- start start 2024-07-27T06:48:40.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44bcff9640 1 -- 172.21.15.29:0/2766332465 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f44b00039b0 con 0x7f44c0103590 2024-07-27T06:48:40.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44cbc00740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 0x556eb227c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44cbc00740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1ead660 0x556eb227c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44cbc00740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1eb5840 0x556eb227caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44cbc00740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556eb21f1e70 con 0x556eb1ed6520 2024-07-27T06:48:40.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44cbc00740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x556eb21e1370 con 0x556eb1eb5840 2024-07-27T06:48:40.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44cbc00740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556eb21e53b0 con 0x556eb1ead660 2024-07-27T06:48:40.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 0x556eb227c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 0x556eb227c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41544/0 (socket says 172.21.15.29:41544) 2024-07-27T06:48:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44bf7fe640 1 -- 172.21.15.29:0/1503211873 learned_addr learned my addr 172.21.15.29:0/1503211873 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44b7fff640 1 --2- 172.21.15.29:0/1503211873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1eb5840 0x556eb227caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44beffd640 1 --2- 172.21.15.29:0/1503211873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1ead660 0x556eb227c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44bf7fe640 1 -- 172.21.15.29:0/1503211873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1ead660 msgr2=0x556eb227c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/1503211873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1ead660 0x556eb227c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44bf7fe640 1 -- 172.21.15.29:0/1503211873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1eb5840 msgr2=0x556eb227caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/1503211873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1eb5840 0x556eb227caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44bf7fe640 1 -- 172.21.15.29:0/1503211873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556eb21ae040 con 0x556eb1ed6520 2024-07-27T06:48:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.888+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/1503211873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 0x556eb227c070 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f44b800c450 tx=0x7f44b800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6d9ca6fad97c9 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.889+0000 7f44bcff9640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f44ac0b3dc0 0x7f44ac0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.889+0000 7f44b57fa640 1 -- 172.21.15.29:0/1503211873 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b8007430 con 0x556eb1ed6520 2024-07-27T06:48:40.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.889+0000 7f44b57fa640 1 -- 172.21.15.29:0/1503211873 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b8007a80 con 0x556eb1ed6520 2024-07-27T06:48:40.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.889+0000 7f44b57fa640 1 -- 172.21.15.29:0/1503211873 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b800aa00 con 0x556eb1ed6520 2024-07-27T06:48:40.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.889+0000 7f44bcff9640 1 -- 172.21.15.29:0/2766332465 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f44b01048d0 con 0x7f44c0103590 2024-07-27T06:48:40.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.889+0000 7f44beffd640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f44ac0b3dc0 0x7f44ac0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.890+0000 7f44cbc00740 1 -- 172.21.15.29:0/1503211873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 msgr2=0x556eb227c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:40.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.890+0000 7f44beffd640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f44ac0b3dc0 0x7f44ac0b62d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f44b01020e0 tx=0x7f44b0031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:40.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.890+0000 7f44cbc00740 1 --2- 172.21.15.29:0/1503211873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 0x556eb227c070 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f44b800c450 tx=0x7f44b800c910 comp rx=0 tx=0).stop 2024-07-27T06:48:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.890+0000 7f44cbc00740 1 -- 172.21.15.29:0/1503211873 shutdown_connections 2024-07-27T06:48:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.890+0000 7f44cbc00740 1 --2- 172.21.15.29:0/1503211873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1eb5840 0x556eb227caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.890+0000 7f44cbc00740 1 --2- 172.21.15.29:0/1503211873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1ead660 0x556eb227c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.890+0000 7f44cbc00740 1 --2- 172.21.15.29:0/1503211873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 0x556eb227c070 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.890+0000 7f44cbc00740 1 -- 172.21.15.29:0/1503211873 >> 172.21.15.29:0/1503211873 conn(0x556eb1eca910 msgr2=0x556eb2279f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.890+0000 7f44cbc00740 1 -- 172.21.15.29:0/1503211873 shutdown_connections 2024-07-27T06:48:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.890+0000 7f44cbc00740 1 -- 172.21.15.29:0/1503211873 wait complete. 2024-07-27T06:48:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.890+0000 7f44cbc00740 1 -- start start 2024-07-27T06:48:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.890+0000 7f44cbc00740 1 Processor -- start 2024-07-27T06:48:40.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44cbc00740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ead660 0x556eb2287bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44cbc00740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1eb5840 0x556eb2288100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44cbc00740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 0x556eb2293b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44cbc00740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556eb2044bb0 con 0x556eb1ed6520 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44cbc00740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x556eb2183ac0 con 0x556eb1ead660 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44cbc00740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556eb228a850 con 0x556eb1eb5840 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44cbc00740 10 client.0 _ll_register_callbacks cb 0x556eb218c3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 0x556eb2293b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 0x556eb2293b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41546/0 (socket says 172.21.15.29:41546) 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44bf7fe640 1 -- 172.21.15.29:0/3774950035 learned_addr learned my addr 172.21.15.29:0/3774950035 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44b7fff640 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1eb5840 0x556eb2288100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ead660 0x556eb2287bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44bf7fe640 1 -- 172.21.15.29:0/3774950035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1eb5840 msgr2=0x556eb2288100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1eb5840 0x556eb2288100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44bf7fe640 1 -- 172.21.15.29:0/3774950035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ead660 msgr2=0x556eb2287bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ead660 0x556eb2287bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.891+0000 7f44bf7fe640 1 -- 172.21.15.29:0/3774950035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556eb1f41b00 con 0x556eb1ed6520 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.892+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 0x556eb2293b40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f44b800c420 tx=0x7f44b800a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7af109dae1e7147 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.892+0000 7f44a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.892+0000 7f44a67fc640 1 -- 172.21.15.29:0/3774950035 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b800a7e0 con 0x556eb1ed6520 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.892+0000 7f44a67fc640 1 -- 172.21.15.29:0/3774950035 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f44b8012040 con 0x556eb1ed6520 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.892+0000 7f44cbc00740 1 -- 172.21.15.29:0/3774950035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44b8007a80 con 0x556eb1ed6520 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.892+0000 7f44a67fc640 1 -- 172.21.15.29:0/3774950035 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f44b800a980 con 0x556eb1ed6520 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.892+0000 7f44cbc00740 10 client.16032 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.892+0000 7f44cbc00740 1 -- 172.21.15.29:0/3774950035 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556eb1fb8960 con 0x556eb1ed6520 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.892+0000 7f44a67fc640 1 -- 172.21.15.29:0/3774950035 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b8007a80 con 0x556eb1ed6520 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.892+0000 7f44a67fc640 1 -- 172.21.15.29:0/3774950035 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f44b8021040 con 0x556eb1ed6520 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.893+0000 7f44cbc00740 10 client.16032 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.893+0000 7f44cbc00740 10 client.16032 resolve_mds: validated gid 24670 aka daemon mds.i 2024-07-27T06:48:40.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.893+0000 7f44cbc00740 20 client.16032 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:40.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.893+0000 7f44cbc00740 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556eb1e94ea0 0x556eb2299a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.893+0000 7f44cbc00740 4 client.16032 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:40.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.893+0000 7f44cbc00740 1 -- 172.21.15.29:0/3774950035 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556eb21f3dc0 con 0x556eb1e94ea0 2024-07-27T06:48:40.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.893+0000 7f44b7fff640 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556eb1e94ea0 0x556eb2299a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.893+0000 7f44b7fff640 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556eb1e94ea0 0x556eb2299a50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe6c0f69d72389c0 server_cookie=1f1f84d2a014bf92 in_seq=0 out_seq=0 2024-07-27T06:48:40.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.893+0000 7f44a67fc640 10 client.16032 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:40.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.894+0000 7f44a67fc640 1 -- 172.21.15.29:0/3774950035 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556eb21f3dc0 con 0x556eb1e94ea0 2024-07-27T06:48:40.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.894+0000 7f44a67fc640 10 client.16032 handle_command_reply: tid=0 2024-07-27T06:48:40.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.894+0000 7f44cbc00740 1 client.16032 shutdown 2024-07-27T06:48:40.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.895+0000 7f44cbc00740 1 -- 172.21.15.29:0/3774950035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 msgr2=0x556eb2293b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:40.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.895+0000 7f44cbc00740 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 0x556eb2293b40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f44b800c420 tx=0x7f44b800a640 comp rx=0 tx=0).stop 2024-07-27T06:48:40.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.895+0000 7f44cbc00740 1 -- 172.21.15.29:0/3774950035 shutdown_connections 2024-07-27T06:48:40.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.895+0000 7f44cbc00740 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ed6520 0x556eb2293b40 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.895+0000 7f44cbc00740 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556eb1e94ea0 0x556eb2299a50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.895+0000 7f44cbc00740 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1eb5840 0x556eb2288100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.895+0000 7f44cbc00740 1 --2- 172.21.15.29:0/3774950035 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ead660 0x556eb2287bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.895+0000 7f44cbc00740 1 -- 172.21.15.29:0/3774950035 >> 172.21.15.29:0/3774950035 conn(0x556eb1eca910 msgr2=0x556eb227f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:40.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.895+0000 7f44a67fc640 0 client.16032 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:40.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.895+0000 7f44cbc00740 1 -- 172.21.15.29:0/3774950035 shutdown_connections 2024-07-27T06:48:40.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.895+0000 7f44cbc00740 1 -- 172.21.15.29:0/3774950035 wait complete. 2024-07-27T06:48:40.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.895+0000 7f44cbc00740 20 client.16032 trim_cache size 0 max 16384 2024-07-27T06:48:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.912+0000 7f44cbc00740 1 Processor -- start 2024-07-27T06:48:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.912+0000 7f44cbc00740 1 -- start start 2024-07-27T06:48:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44cbc00740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ead660 0x556eb227f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44cbc00740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1eb5840 0x556eb2293930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44cbc00740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1ed6520 0x556eb2293e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44cbc00740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01af2c0 con 0x556eb1eb5840 2024-07-27T06:48:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44cbc00740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01b0e00 con 0x556eb1ead660 2024-07-27T06:48:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44cbc00740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01af990 con 0x556eb1ed6520 2024-07-27T06:48:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1eb5840 0x556eb2293930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44b7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1eb5840 0x556eb2293930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41552/0 (socket says 172.21.15.29:41552) 2024-07-27T06:48:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44b7fff640 1 -- 172.21.15.29:0/2575930254 learned_addr learned my addr 172.21.15.29:0/2575930254 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/2575930254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1ed6520 0x556eb2293e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44b7fff640 1 -- 172.21.15.29:0/2575930254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1ed6520 msgr2=0x556eb2293e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/2575930254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ead660 0x556eb227f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44b7fff640 1 --2- 172.21.15.29:0/2575930254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1ed6520 0x556eb2293e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44b7fff640 1 -- 172.21.15.29:0/2575930254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ead660 msgr2=0x556eb227f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44b7fff640 1 --2- 172.21.15.29:0/2575930254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ead660 0x556eb227f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44b7fff640 1 -- 172.21.15.29:0/2575930254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556eb2288c80 con 0x556eb1eb5840 2024-07-27T06:48:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.913+0000 7f44b7fff640 1 --2- 172.21.15.29:0/2575930254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1eb5840 0x556eb2293930 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f44c006c420 tx=0x7f44a8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3797ae3ab1998b server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:40.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.914+0000 7f44b4ff9640 1 -- 172.21.15.29:0/2575930254 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8021070 con 0x556eb1eb5840 2024-07-27T06:48:40.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.914+0000 7f44b4ff9640 1 -- 172.21.15.29:0/2575930254 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44a8016da0 con 0x556eb1eb5840 2024-07-27T06:48:40.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.914+0000 7f44b4ff9640 1 -- 172.21.15.29:0/2575930254 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8015760 con 0x556eb1eb5840 2024-07-27T06:48:40.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.914+0000 7f44cbc00740 1 -- 172.21.15.29:0/2575930254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1eb5840 msgr2=0x556eb2293930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:40.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.914+0000 7f44cbc00740 1 --2- 172.21.15.29:0/2575930254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1eb5840 0x556eb2293930 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f44c006c420 tx=0x7f44a8008040 comp rx=0 tx=0).stop 2024-07-27T06:48:40.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.914+0000 7f44cbc00740 1 -- 172.21.15.29:0/2575930254 shutdown_connections 2024-07-27T06:48:40.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 --2- 172.21.15.29:0/2575930254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1ed6520 0x556eb2293e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 --2- 172.21.15.29:0/2575930254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1eb5840 0x556eb2293930 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 --2- 172.21.15.29:0/2575930254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ead660 0x556eb227f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 -- 172.21.15.29:0/2575930254 >> 172.21.15.29:0/2575930254 conn(0x556eb1eca910 msgr2=0x556eb227f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:40.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 -- 172.21.15.29:0/2575930254 shutdown_connections 2024-07-27T06:48:40.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 -- 172.21.15.29:0/2575930254 wait complete. 2024-07-27T06:48:40.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 -- start start 2024-07-27T06:48:40.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 Processor -- start 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ead660 0x556eb228ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1eb5840 0x556eb228b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ed6520 0x556eb228b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01b0730 con 0x556eb1ead660 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01b0060 con 0x556eb1ed6520 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f44c01ad2a0 con 0x556eb1eb5840 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.915+0000 7f44cbc00740 10 client.0 _ll_register_callbacks cb 0x7f44b8009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ead660 0x556eb228ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44bf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ead660 0x556eb228ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41566/0 (socket says 172.21.15.29:41566) 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44bf7fe640 1 -- 172.21.15.29:0/3083611774 learned_addr learned my addr 172.21.15.29:0/3083611774 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44b7fff640 1 --2- 172.21.15.29:0/3083611774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1eb5840 0x556eb228b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44bf7fe640 1 -- 172.21.15.29:0/3083611774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1eb5840 msgr2=0x556eb228b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:40.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/3083611774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1eb5840 0x556eb228b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44bf7fe640 1 -- 172.21.15.29:0/3083611774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ed6520 msgr2=0x556eb228b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/3083611774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ed6520 0x556eb228b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44bf7fe640 1 -- 172.21.15.29:0/3083611774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556eb228beb0 con 0x556eb1ead660 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44bf7fe640 1 --2- 172.21.15.29:0/3083611774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ead660 0x556eb228ff50 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f44b8004b90 tx=0x7f44b800ce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd62a8a47ad460a0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44a67fc640 1 -- 172.21.15.29:0/3083611774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b800a400 con 0x556eb1ead660 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44a67fc640 1 -- 172.21.15.29:0/3083611774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f44b801b070 con 0x556eb1ead660 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44cbc00740 1 -- 172.21.15.29:0/3083611774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44a8016da0 con 0x556eb1ead660 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44a67fc640 1 -- 172.21.15.29:0/3083611774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f44b8007740 con 0x556eb1ead660 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44cbc00740 10 client.16038 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.916+0000 7f44cbc00740 1 -- 172.21.15.29:0/3083611774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556eb2301e80 con 0x556eb1ead660 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.917+0000 7f44a67fc640 1 -- 172.21.15.29:0/3083611774 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44a8016da0 con 0x556eb1ead660 2024-07-27T06:48:40.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.917+0000 7f44a67fc640 1 -- 172.21.15.29:0/3083611774 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f44b8033020 con 0x556eb1ead660 2024-07-27T06:48:40.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.917+0000 7f44cbc00740 10 client.16038 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:40.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.917+0000 7f44cbc00740 10 client.16038 resolve_mds: validated gid 24670 aka daemon mds.i 2024-07-27T06:48:40.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.917+0000 7f44cbc00740 20 client.16038 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:40.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.917+0000 7f44cbc00740 1 --2- 172.21.15.29:0/3083611774 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556eb1e6a250 0x556eb2304600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:40.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.917+0000 7f44cbc00740 4 client.16038 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "7f206a77", "release": true, "await": true}] 2024-07-27T06:48:40.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.917+0000 7f44cbc00740 1 -- 172.21.15.29:0/3083611774 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "quiesce db", "set_id": "7f206a77", "release": true, "await": true}) -- 0x556eb21ef860 con 0x556eb1e6a250 2024-07-27T06:48:40.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.917+0000 7f44b7fff640 1 --2- 172.21.15.29:0/3083611774 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556eb1e6a250 0x556eb2304600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:40.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.918+0000 7f44b7fff640 1 --2- 172.21.15.29:0/3083611774 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556eb1e6a250 0x556eb2304600 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85faa5f9fd653d3d server_cookie=ccddb79e2d10504d in_seq=0 out_seq=0 2024-07-27T06:48:40.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.918+0000 7f44a67fc640 10 client.16038 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:40.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44a67fc640 1 -- 172.21.15.29:0/3083611774 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x556eb21ef860 con 0x556eb1e6a250 2024-07-27T06:48:40.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44a67fc640 10 client.16038 handle_command_reply: tid=0 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44cbc00740 1 client.16038 shutdown 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44cbc00740 1 -- 172.21.15.29:0/3083611774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ead660 msgr2=0x556eb228ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44cbc00740 1 --2- 172.21.15.29:0/3083611774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ead660 0x556eb228ff50 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f44b8004b90 tx=0x7f44b800ce10 comp rx=0 tx=0).stop 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44cbc00740 1 -- 172.21.15.29:0/3083611774 shutdown_connections 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44cbc00740 1 --2- 172.21.15.29:0/3083611774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556eb1ed6520 0x556eb228b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44cbc00740 1 --2- 172.21.15.29:0/3083611774 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x556eb1e6a250 0x556eb2304600 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44cbc00740 1 --2- 172.21.15.29:0/3083611774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556eb1eb5840 0x556eb228b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44cbc00740 1 --2- 172.21.15.29:0/3083611774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556eb1ead660 0x556eb228ff50 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44cbc00740 1 -- 172.21.15.29:0/3083611774 >> 172.21.15.29:0/3083611774 conn(0x556eb1eca910 msgr2=0x556eb227d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44a67fc640 0 client.16038 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44cbc00740 1 -- 172.21.15.29:0/3083611774 shutdown_connections 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44cbc00740 1 -- 172.21.15.29:0/3083611774 wait complete. 2024-07-27T06:48:40.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.922+0000 7f44cbc00740 20 client.16038 trim_cache size 0 max 16384 2024-07-27T06:48:40.926 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:48:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 29, 2024-07-27T06:48:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24670, 2024-07-27T06:48:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 4, 2024-07-27T06:48:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-07-27T06:48:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "7f206a77": { 2024-07-27T06:48:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 4, 2024-07-27T06:48:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-07-27T06:48:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 142.6, 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:40.928 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:48:40.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.923+0000 7f44c567c640 1 -- 172.21.15.29:0/2766332465 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f44ac0b3dc0 msgr2=0x7f44ac0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.923+0000 7f44c567c640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f44ac0b3dc0 0x7f44ac0b62d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f44b01020e0 tx=0x7f44b0031400 comp rx=0 tx=0).stop 2024-07-27T06:48:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.923+0000 7f44c567c640 1 -- 172.21.15.29:0/2766332465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c0103590 msgr2=0x7f44c019a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.923+0000 7f44c567c640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c0103590 0x7f44c019a050 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f44b0006010 tx=0x7f44b00042f0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.923+0000 7f44c567c640 1 -- 172.21.15.29:0/2766332465 shutdown_connections 2024-07-27T06:48:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.923+0000 7f44c567c640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44c0109820 0x7f44c019f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.923+0000 7f44c567c640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f44ac0b3dc0 0x7f44ac0b62d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.923+0000 7f44c567c640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44c0108ed0 0x7f44c019a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.923+0000 7f44c567c640 1 --2- 172.21.15.29:0/2766332465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c0103590 0x7f44c019a050 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.923+0000 7f44c567c640 1 -- 172.21.15.29:0/2766332465 >> 172.21.15.29:0/2766332465 conn(0x7f44c007b280 msgr2=0x7f44c0107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.923+0000 7f44c567c640 1 -- 172.21.15.29:0/2766332465 shutdown_connections 2024-07-27T06:48:40.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:40.923+0000 7f44c567c640 1 -- 172.21.15.29:0/2766332465 wait complete. 2024-07-27T06:48:40.930 INFO:tasks.workunit.client.0.smithi029.stderr:+ set -e 2024-07-27T06:48:40.930 INFO:tasks.workunit.client.0.smithi029.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-07-27T06:48:40.930 INFO:tasks.workunit.client.0.smithi029.stderr:Cloning into 'xfstests-dev'... 2024-07-27T06:48:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:40 smithi029 ceph-mon[32610]: pgmap v447: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 853 B/s wr, 8 op/s 2024-07-27T06:48:41.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:40 smithi047 ceph-mon[36835]: pgmap v447: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 853 B/s wr, 8 op/s 2024-07-27T06:48:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:40 smithi077 ceph-mon[37153]: pgmap v447: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 853 B/s wr, 8 op/s 2024-07-27T06:48:42.711 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '7f206a77', total seconds elapsed: 36.0 2024-07-27T06:48:42.712 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.55 2024-07-27T06:48:42.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:48:42.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.935+0000 7faa2de67640 1 -- 172.21.15.29:0/2335798648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa28107de0 msgr2=0x7faa281081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:42.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.935+0000 7faa2de67640 1 --2- 172.21.15.29:0/2335798648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa28107de0 0x7faa281081f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009900 tx=0x7faa1c02a410 comp rx=0 tx=0).stop 2024-07-27T06:48:42.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.935+0000 7faa2de67640 1 -- 172.21.15.29:0/2335798648 shutdown_connections 2024-07-27T06:48:42.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.935+0000 7faa2de67640 1 --2- 172.21.15.29:0/2335798648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa28108730 0x7faa2810ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.935+0000 7faa2de67640 1 --2- 172.21.15.29:0/2335798648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa28107de0 0x7faa281081f0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.935+0000 7faa2de67640 1 --2- 172.21.15.29:0/2335798648 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa281024a0 0x7faa281028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.935+0000 7faa2de67640 1 -- 172.21.15.29:0/2335798648 >> 172.21.15.29:0/2335798648 conn(0x7faa28079160 msgr2=0x7faa28079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:42.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.935+0000 7faa2de67640 1 -- 172.21.15.29:0/2335798648 shutdown_connections 2024-07-27T06:48:42.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.936+0000 7faa2de67640 1 -- 172.21.15.29:0/2335798648 wait complete. 2024-07-27T06:48:42.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.938+0000 7faa2de67640 1 Processor -- start 2024-07-27T06:48:42.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.938+0000 7faa2de67640 1 -- start start 2024-07-27T06:48:42.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.938+0000 7faa2de67640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa281024a0 0x7faa2819ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.938+0000 7faa2de67640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa28107de0 0x7faa2819b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.938+0000 7faa2de67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa28108730 0x7faa2819b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa2de67640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faa28112360 con 0x7faa28108730 2024-07-27T06:48:42.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa2de67640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7faa281121e0 con 0x7faa281024a0 2024-07-27T06:48:42.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa2de67640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faa281124e0 con 0x7faa28107de0 2024-07-27T06:48:42.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa28108730 0x7faa2819b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa28108730 0x7faa2819b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41580/0 (socket says 172.21.15.29:41580) 2024-07-27T06:48:42.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa27fff640 1 -- 172.21.15.29:0/915447578 learned_addr learned my addr 172.21.15.29:0/915447578 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:42.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa26ffd640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa28107de0 0x7faa2819b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa277fe640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa281024a0 0x7faa2819ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa27fff640 1 -- 172.21.15.29:0/915447578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa28107de0 msgr2=0x7faa2819b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:42.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa27fff640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa28107de0 0x7faa2819b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa27fff640 1 -- 172.21.15.29:0/915447578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa281024a0 msgr2=0x7faa2819ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:42.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa27fff640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa281024a0 0x7faa2819ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.939+0000 7faa27fff640 1 -- 172.21.15.29:0/915447578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa281a5fd0 con 0x7faa28108730 2024-07-27T06:48:42.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.940+0000 7faa27fff640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa28108730 0x7faa2819b790 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7faa2000cad0 tx=0x7faa2000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:42.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.940+0000 7faa24ff9640 1 -- 172.21.15.29:0/915447578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20004d20 con 0x7faa28108730 2024-07-27T06:48:42.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.940+0000 7faa24ff9640 1 -- 172.21.15.29:0/915447578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa2000ada0 con 0x7faa28108730 2024-07-27T06:48:42.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.940+0000 7faa24ff9640 1 -- 172.21.15.29:0/915447578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20005870 con 0x7faa28108730 2024-07-27T06:48:42.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.941+0000 7faa2de67640 1 -- 172.21.15.29:0/915447578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa28198b60 con 0x7faa28108730 2024-07-27T06:48:42.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.941+0000 7faa2de67640 1 -- 172.21.15.29:0/915447578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa28199020 con 0x7faa28108730 2024-07-27T06:48:42.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.945+0000 7faa343eb740 1 Processor -- start 2024-07-27T06:48:42.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.946+0000 7faa24ff9640 1 -- 172.21.15.29:0/915447578 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7faa2001d440 con 0x7faa28108730 2024-07-27T06:48:42.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.946+0000 7faa343eb740 1 -- start start 2024-07-27T06:48:42.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.946+0000 7faa343eb740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c96070 0x55760703b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.946+0000 7faa343eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c6d1b0 0x55760703bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.947+0000 7faa343eb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c75390 0x55760703c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.947+0000 7faa343eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557606fb1bb0 con 0x557606c6d1b0 2024-07-27T06:48:42.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.947+0000 7faa343eb740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x557606fa10b0 con 0x557606c96070 2024-07-27T06:48:42.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.947+0000 7faa343eb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557606fa50f0 con 0x557606c75390 2024-07-27T06:48:42.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.947+0000 7faa277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c6d1b0 0x55760703bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.947+0000 7faa277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c6d1b0 0x55760703bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41588/0 (socket says 172.21.15.29:41588) 2024-07-27T06:48:42.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.947+0000 7faa277fe640 1 -- 172.21.15.29:0/1325514488 learned_addr learned my addr 172.21.15.29:0/1325514488 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:42.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.947+0000 7faa26ffd640 1 --2- 172.21.15.29:0/1325514488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c96070 0x55760703b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.947+0000 7faa24ff9640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7faa100b3e90 0x7faa100b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.947+0000 7faa26ffd640 1 --2- 172.21.15.29:0/1325514488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c75390 0x55760703c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.948+0000 7faa24ff9640 1 -- 172.21.15.29:0/915447578 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7faa200ec2a0 con 0x7faa28108730 2024-07-27T06:48:42.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.948+0000 7faa277fe640 1 -- 172.21.15.29:0/1325514488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c75390 msgr2=0x55760703c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:42.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.948+0000 7faa277fe640 1 --2- 172.21.15.29:0/1325514488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c75390 0x55760703c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.948+0000 7faa277fe640 1 -- 172.21.15.29:0/1325514488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c96070 msgr2=0x55760703b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:42.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.948+0000 7faa277fe640 1 --2- 172.21.15.29:0/1325514488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c96070 0x55760703b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.948+0000 7faa277fe640 1 -- 172.21.15.29:0/1325514488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557606f47b90 con 0x557606c6d1b0 2024-07-27T06:48:42.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.948+0000 7faa26ffd640 1 --2- 172.21.15.29:0/1325514488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c96070 0x55760703b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:42.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.948+0000 7faa277fe640 1 --2- 172.21.15.29:0/1325514488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c6d1b0 0x55760703bbd0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7faa18009c90 tx=0x7faa18011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19bf42fed09236c6 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:42.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.948+0000 7faa27fff640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7faa100b3e90 0x7faa100b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.948+0000 7faa0dffb640 1 -- 172.21.15.29:0/1325514488 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18005220 con 0x557606c6d1b0 2024-07-27T06:48:42.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.948+0000 7faa0dffb640 1 -- 172.21.15.29:0/1325514488 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa18004c50 con 0x557606c6d1b0 2024-07-27T06:48:42.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.948+0000 7faa0dffb640 1 -- 172.21.15.29:0/1325514488 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa180197a0 con 0x557606c6d1b0 2024-07-27T06:48:42.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.950+0000 7faa27fff640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7faa100b3e90 0x7faa100b63a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7faa200af100 tx=0x7faa20017040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:42.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.950+0000 7faa343eb740 1 -- 172.21.15.29:0/1325514488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c6d1b0 msgr2=0x55760703bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:42.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.950+0000 7faa343eb740 1 --2- 172.21.15.29:0/1325514488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c6d1b0 0x55760703bbd0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7faa18009c90 tx=0x7faa18011810 comp rx=0 tx=0).stop 2024-07-27T06:48:42.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.950+0000 7faa343eb740 1 -- 172.21.15.29:0/1325514488 shutdown_connections 2024-07-27T06:48:42.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.950+0000 7faa343eb740 1 --2- 172.21.15.29:0/1325514488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c75390 0x55760703c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.950+0000 7faa343eb740 1 --2- 172.21.15.29:0/1325514488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c6d1b0 0x55760703bbd0 secure :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7faa18009c90 tx=0x7faa18011810 comp rx=0 tx=0).stop 2024-07-27T06:48:42.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.950+0000 7faa343eb740 1 --2- 172.21.15.29:0/1325514488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c96070 0x55760703b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.950+0000 7faa343eb740 1 -- 172.21.15.29:0/1325514488 >> 172.21.15.29:0/1325514488 conn(0x557606c8a460 msgr2=0x557607039580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:42.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.950+0000 7faa343eb740 1 -- 172.21.15.29:0/1325514488 shutdown_connections 2024-07-27T06:48:42.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.950+0000 7faa343eb740 1 -- 172.21.15.29:0/1325514488 wait complete. 2024-07-27T06:48:42.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.950+0000 7faa343eb740 1 -- start start 2024-07-27T06:48:42.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.951+0000 7faa343eb740 1 Processor -- start 2024-07-27T06:48:42.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.951+0000 7faa343eb740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c75390 0x557607047470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.951+0000 7faa343eb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c96070 0x557607047a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.951+0000 7faa343eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c549f0 0x55760704d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.951+0000 7faa343eb740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557606e04700 con 0x557606c549f0 2024-07-27T06:48:42.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.951+0000 7faa343eb740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x557606f439e0 con 0x557606c75390 2024-07-27T06:48:42.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.951+0000 7faa343eb740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557607049e70 con 0x557606c96070 2024-07-27T06:48:42.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.951+0000 7faa343eb740 10 client.0 _ll_register_callbacks cb 0x557606f5e4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:42.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.952+0000 7faa26ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c549f0 0x55760704d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.952+0000 7faa26ffd640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c75390 0x557607047470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.952+0000 7faa26ffd640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c75390 0x557607047470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.29:42726/0 (socket says 172.21.15.29:42726) 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.952+0000 7faa26ffd640 1 -- 172.21.15.29:0/1052669103 learned_addr learned my addr 172.21.15.29:0/1052669103 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.952+0000 7faa277fe640 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c96070 0x557607047a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.952+0000 7faa26ffd640 1 -- 172.21.15.29:0/1052669103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c96070 msgr2=0x557607047a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.952+0000 7faa26ffd640 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c96070 0x557607047a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.952+0000 7faa26ffd640 1 -- 172.21.15.29:0/1052669103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c549f0 msgr2=0x55760704d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.952+0000 7faa26ffd640 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c549f0 0x55760704d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.952+0000 7faa26ffd640 1 -- 172.21.15.29:0/1052669103 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557606d01650 con 0x557606c75390 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.952+0000 7faa277fe640 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c96070 0x557607047a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.953+0000 7faa26ffd640 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c75390 0x557607047470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009900 tx=0x7faa1c005e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c325164c6344fe6 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.953+0000 7faa06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.953+0000 7faa06ffd640 1 -- 172.21.15.29:0/1052669103 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c035040 con 0x557606c75390 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.953+0000 7faa343eb740 1 -- 172.21.15.29:0/1052669103 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa18004c50 con 0x557606c75390 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.953+0000 7faa06ffd640 1 -- 172.21.15.29:0/1052669103 <== mon.2 v2:172.21.15.47:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7faa1c005240 con 0x557606c75390 2024-07-27T06:48:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.953+0000 7faa06ffd640 1 -- 172.21.15.29:0/1052669103 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faa1c005440 con 0x557606c75390 2024-07-27T06:48:42.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.953+0000 7faa343eb740 10 client.25559 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:42.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.953+0000 7faa343eb740 1 -- 172.21.15.29:0/1052669103 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557607056090 con 0x557606c75390 2024-07-27T06:48:42.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.954+0000 7faa06ffd640 1 -- 172.21.15.29:0/1052669103 <== mon.2 v2:172.21.15.47:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa18004c50 con 0x557606c75390 2024-07-27T06:48:42.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.954+0000 7faa06ffd640 1 -- 172.21.15.29:0/1052669103 <== mon.2 v2:172.21.15.47:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7faa1c0526a0 con 0x557606c75390 2024-07-27T06:48:42.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.954+0000 7faa343eb740 10 client.25559 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:42.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.954+0000 7faa343eb740 10 client.25559 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-07-27T06:48:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.954+0000 7faa343eb740 20 client.25559 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.954+0000 7faa343eb740 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x557606c29da0 0x5576070594b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.955+0000 7faa343eb740 4 client.25559 mds_command: new command op to 24646 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.955+0000 7faa343eb740 1 -- 172.21.15.29:0/1052669103 --> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557606fb3b00 con 0x557606c29da0 2024-07-27T06:48:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.955+0000 7faa277fe640 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x557606c29da0 0x5576070594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.956+0000 7faa06ffd640 10 client.25559 ms_handle_connect on v2:172.21.15.77:6834/439792826 2024-07-27T06:48:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.956+0000 7faa277fe640 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x557606c29da0 0x5576070594b0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5c3c59070b01b63b server_cookie=c5821d03ac788898 in_seq=0 out_seq=0 2024-07-27T06:48:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.956+0000 7faa06ffd640 1 -- 172.21.15.29:0/1052669103 <== mds.2 v2:172.21.15.77:6834/439792826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557606fb3b00 con 0x557606c29da0 2024-07-27T06:48:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.956+0000 7faa06ffd640 10 client.25559 handle_command_reply: tid=0 2024-07-27T06:48:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.956+0000 7faa343eb740 1 client.25559 shutdown 2024-07-27T06:48:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.957+0000 7faa343eb740 1 -- 172.21.15.29:0/1052669103 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c75390 msgr2=0x557607047470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.957+0000 7faa343eb740 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c75390 0x557607047470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009900 tx=0x7faa1c005e90 comp rx=0 tx=0).stop 2024-07-27T06:48:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.957+0000 7faa343eb740 1 -- 172.21.15.29:0/1052669103 shutdown_connections 2024-07-27T06:48:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.957+0000 7faa343eb740 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x557606c29da0 0x5576070594b0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.957+0000 7faa06ffd640 0 client.25559 ms_handle_reset on v2:172.21.15.77:6834/439792826 2024-07-27T06:48:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.957+0000 7faa343eb740 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c549f0 0x55760704d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.957+0000 7faa343eb740 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c96070 0x557607047a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.957+0000 7faa343eb740 1 --2- 172.21.15.29:0/1052669103 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c75390 0x557607047470 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.957+0000 7faa343eb740 1 -- 172.21.15.29:0/1052669103 >> 172.21.15.29:0/1052669103 conn(0x557606c8a460 msgr2=0x557607037d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.957+0000 7faa343eb740 1 -- 172.21.15.29:0/1052669103 shutdown_connections 2024-07-27T06:48:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.957+0000 7faa343eb740 1 -- 172.21.15.29:0/1052669103 wait complete. 2024-07-27T06:48:42.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.957+0000 7faa343eb740 20 client.25559 trim_cache size 0 max 16384 2024-07-27T06:48:42.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.975+0000 7faa343eb740 1 Processor -- start 2024-07-27T06:48:42.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.975+0000 7faa343eb740 1 -- start start 2024-07-27T06:48:42.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.976+0000 7faa343eb740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c549f0 0x557607046140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.976+0000 7faa343eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 0x55760704d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.976+0000 7faa277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 0x55760704d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.976+0000 7faa277fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 0x55760704d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41614/0 (socket says 172.21.15.29:41614) 2024-07-27T06:48:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.976+0000 7faa277fe640 1 -- 172.21.15.29:0/3961157494 learned_addr learned my addr 172.21.15.29:0/3961157494 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.976+0000 7faa343eb740 1 --2- 172.21.15.29:0/3961157494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c96070 0x5576070395b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.976+0000 7faa26ffd640 1 --2- 172.21.15.29:0/3961157494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c549f0 0x557607046140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.976+0000 7faa343eb740 1 -- 172.21.15.29:0/3961157494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faa281acef0 con 0x557606c75390 2024-07-27T06:48:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.976+0000 7faa343eb740 1 -- 172.21.15.29:0/3961157494 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7faa281aea30 con 0x557606c549f0 2024-07-27T06:48:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.976+0000 7faa343eb740 1 -- 172.21.15.29:0/3961157494 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faa281ad5c0 con 0x557606c96070 2024-07-27T06:48:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.976+0000 7faa26ffd640 1 --2- 172.21.15.29:0/3961157494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c96070 0x5576070395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.977+0000 7faa277fe640 1 -- 172.21.15.29:0/3961157494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c96070 msgr2=0x5576070395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.977+0000 7faa277fe640 1 --2- 172.21.15.29:0/3961157494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c96070 0x5576070395b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.977+0000 7faa277fe640 1 -- 172.21.15.29:0/3961157494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c549f0 msgr2=0x557607046140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.977+0000 7faa277fe640 1 --2- 172.21.15.29:0/3961157494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c549f0 0x557607046140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.977+0000 7faa277fe640 1 -- 172.21.15.29:0/3961157494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55760703d980 con 0x557606c75390 2024-07-27T06:48:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.977+0000 7faa26ffd640 1 --2- 172.21.15.29:0/3961157494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c549f0 0x557607046140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.977+0000 7faa277fe640 1 --2- 172.21.15.29:0/3961157494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 0x55760704d0a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7faa18005720 tx=0x7faa180047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995f5cf822267233 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.977+0000 7faa0d7fa640 1 -- 172.21.15.29:0/3961157494 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18004a40 con 0x557606c75390 2024-07-27T06:48:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.977+0000 7faa0d7fa640 1 -- 172.21.15.29:0/3961157494 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa18004be0 con 0x557606c75390 2024-07-27T06:48:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.978+0000 7faa0d7fa640 1 -- 172.21.15.29:0/3961157494 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa180052f0 con 0x557606c75390 2024-07-27T06:48:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.979+0000 7faa343eb740 1 -- 172.21.15.29:0/3961157494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 msgr2=0x55760704d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.979+0000 7faa343eb740 1 --2- 172.21.15.29:0/3961157494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 0x55760704d0a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7faa18005720 tx=0x7faa180047d0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.979+0000 7faa343eb740 1 -- 172.21.15.29:0/3961157494 shutdown_connections 2024-07-27T06:48:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.979+0000 7faa343eb740 1 --2- 172.21.15.29:0/3961157494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c96070 0x5576070395b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.979+0000 7faa343eb740 1 --2- 172.21.15.29:0/3961157494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 0x55760704d0a0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.979+0000 7faa343eb740 1 --2- 172.21.15.29:0/3961157494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c549f0 0x557607046140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.979+0000 7faa343eb740 1 -- 172.21.15.29:0/3961157494 >> 172.21.15.29:0/3961157494 conn(0x557606c8a460 msgr2=0x557607038ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.980+0000 7faa343eb740 1 -- 172.21.15.29:0/3961157494 shutdown_connections 2024-07-27T06:48:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.980+0000 7faa343eb740 1 -- 172.21.15.29:0/3961157494 wait complete. 2024-07-27T06:48:42.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.980+0000 7faa343eb740 1 -- start start 2024-07-27T06:48:42.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.981+0000 7faa343eb740 1 Processor -- start 2024-07-27T06:48:42.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa343eb740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c549f0 0x55760704f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa343eb740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 0x55760704faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa343eb740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c96070 0x557607050030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa26ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c549f0 0x55760704f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa26ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c549f0 0x55760704f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:42830/0 (socket says 172.21.15.29:42830) 2024-07-27T06:48:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa26ffd640 1 -- 172.21.15.29:0/3069731764 learned_addr learned my addr 172.21.15.29:0/3069731764 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa277fe640 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 0x55760704faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa343eb740 1 -- 172.21.15.29:0/3069731764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faa281ae360 con 0x557606c75390 2024-07-27T06:48:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa343eb740 1 -- 172.21.15.29:0/3069731764 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7faa281adc90 con 0x557606c96070 2024-07-27T06:48:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa343eb740 1 -- 172.21.15.29:0/3069731764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faa281aaed0 con 0x557606c549f0 2024-07-27T06:48:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa26ffd640 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c96070 0x557607050030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa343eb740 10 client.0 _ll_register_callbacks cb 0x7faa1c032c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa26ffd640 1 -- 172.21.15.29:0/3069731764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c96070 msgr2=0x557607050030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa26ffd640 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c96070 0x557607050030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.982+0000 7faa26ffd640 1 -- 172.21.15.29:0/3069731764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 msgr2=0x55760704faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.983+0000 7faa26ffd640 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 0x55760704faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.983+0000 7faa26ffd640 1 -- 172.21.15.29:0/3069731764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576070c0e40 con 0x557606c549f0 2024-07-27T06:48:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.983+0000 7faa343eb740 1 -- 172.21.15.29:0/3069731764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa18004be0 con 0x557606c549f0 2024-07-27T06:48:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.983+0000 7faa277fe640 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 0x55760704faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.983+0000 7faa26ffd640 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c549f0 0x55760704f5b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009900 tx=0x7faa1c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23fd0795e20659ed server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.983+0000 7faa06ffd640 10 client.25720 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:48:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.983+0000 7faa06ffd640 1 -- 172.21.15.29:0/3069731764 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c058040 con 0x557606c549f0 2024-07-27T06:48:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.983+0000 7faa06ffd640 1 -- 172.21.15.29:0/3069731764 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7faa1c052470 con 0x557606c549f0 2024-07-27T06:48:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.983+0000 7faa06ffd640 1 -- 172.21.15.29:0/3069731764 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faa1c035040 con 0x557606c549f0 2024-07-27T06:48:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.984+0000 7faa343eb740 10 client.25720 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.984+0000 7faa343eb740 1 -- 172.21.15.29:0/3069731764 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5576070c1ff0 con 0x557606c549f0 2024-07-27T06:48:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.984+0000 7faa06ffd640 1 -- 172.21.15.29:0/3069731764 <== mon.1 v2:172.21.15.77:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7faa1c05e060 con 0x557606c549f0 2024-07-27T06:48:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.984+0000 7faa343eb740 10 client.25720 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.984+0000 7faa343eb740 10 client.25720 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-07-27T06:48:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.984+0000 7faa343eb740 20 client.25720 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.984+0000 7faa343eb740 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x5576070c2680 0x5576070c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:42.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.984+0000 7faa343eb740 4 client.25720 mds_command: new command op to 24646 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:48:42.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.984+0000 7faa343eb740 1 -- 172.21.15.29:0/3069731764 --> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557606faf5a0 con 0x5576070c2680 2024-07-27T06:48:42.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.984+0000 7faa277fe640 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x5576070c2680 0x5576070c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:42.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.985+0000 7faa06ffd640 10 client.25720 ms_handle_connect on v2:172.21.15.77:6834/439792826 2024-07-27T06:48:42.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.985+0000 7faa277fe640 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x5576070c2680 0x5576070c4aa0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a72c4fd07fa97c70 server_cookie=5d1cd2b42a28abc9 in_seq=0 out_seq=0 2024-07-27T06:48:42.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.985+0000 7faa06ffd640 1 -- 172.21.15.29:0/3069731764 <== mds.2 v2:172.21.15.77:6834/439792826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557606faf5a0 con 0x5576070c2680 2024-07-27T06:48:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.985+0000 7faa06ffd640 10 client.25720 handle_command_reply: tid=0 2024-07-27T06:48:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.985+0000 7faa343eb740 1 client.25720 shutdown 2024-07-27T06:48:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.986+0000 7faa343eb740 1 -- 172.21.15.29:0/3069731764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c549f0 msgr2=0x55760704f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.986+0000 7faa343eb740 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c549f0 0x55760704f5b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009900 tx=0x7faa1c000bc0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.986+0000 7faa343eb740 1 -- 172.21.15.29:0/3069731764 shutdown_connections 2024-07-27T06:48:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.986+0000 7faa343eb740 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x5576070c2680 0x5576070c4aa0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.986+0000 7faa06ffd640 0 client.25720 ms_handle_reset on v2:172.21.15.77:6834/439792826 2024-07-27T06:48:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.986+0000 7faa343eb740 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x557606c96070 0x557607050030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.986+0000 7faa343eb740 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x557606c75390 0x55760704faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.986+0000 7faa343eb740 1 --2- 172.21.15.29:0/3069731764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x557606c549f0 0x55760704f5b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.986+0000 7faa343eb740 1 -- 172.21.15.29:0/3069731764 >> 172.21.15.29:0/3069731764 conn(0x557606c8a460 msgr2=0x5576070407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.986+0000 7faa343eb740 1 -- 172.21.15.29:0/3069731764 shutdown_connections 2024-07-27T06:48:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.986+0000 7faa343eb740 1 -- 172.21.15.29:0/3069731764 wait complete. 2024-07-27T06:48:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.987+0000 7faa343eb740 20 client.25720 trim_cache size 0 max 16384 2024-07-27T06:48:42.994 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:48:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.987+0000 7faa2de67640 1 -- 172.21.15.29:0/915447578 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7faa100b3e90 msgr2=0x7faa100b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.987+0000 7faa2de67640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7faa100b3e90 0x7faa100b63a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7faa200af100 tx=0x7faa20017040 comp rx=0 tx=0).stop 2024-07-27T06:48:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.988+0000 7faa2de67640 1 -- 172.21.15.29:0/915447578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa28108730 msgr2=0x7faa2819b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.988+0000 7faa2de67640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa28108730 0x7faa2819b790 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7faa2000cad0 tx=0x7faa2000cf90 comp rx=0 tx=0).stop 2024-07-27T06:48:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.988+0000 7faa2de67640 1 -- 172.21.15.29:0/915447578 shutdown_connections 2024-07-27T06:48:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.988+0000 7faa2de67640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa28108730 0x7faa2819b790 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.988+0000 7faa2de67640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7faa100b3e90 0x7faa100b63a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.988+0000 7faa2de67640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faa28107de0 0x7faa2819b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.988+0000 7faa2de67640 1 --2- 172.21.15.29:0/915447578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa281024a0 0x7faa2819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:42.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.988+0000 7faa2de67640 1 -- 172.21.15.29:0/915447578 >> 172.21.15.29:0/915447578 conn(0x7faa28079160 msgr2=0x7faa28103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:42.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.988+0000 7faa2de67640 1 -- 172.21.15.29:0/915447578 shutdown_connections 2024-07-27T06:48:42.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:42.988+0000 7faa2de67640 1 -- 172.21.15.29:0/915447578 wait complete. 2024-07-27T06:48:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:42 smithi029 ceph-mon[32610]: pgmap v448: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-27T06:48:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:42 smithi077 ceph-mon[37153]: pgmap v448: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-27T06:48:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:42 smithi047 ceph-mon[36835]: pgmap v448: 97 pgs: 97 active+clean; 1.4 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-27T06:48:43.671 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-27T06:48:43.784 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:45.051 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:45.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:44 smithi029 ceph-mon[32610]: pgmap v449: 97 pgs: 97 active+clean; 1.6 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-07-27T06:48:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:44 smithi077 ceph-mon[37153]: pgmap v449: 97 pgs: 97 active+clean; 1.6 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-07-27T06:48:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:44 smithi047 ceph-mon[36835]: pgmap v449: 97 pgs: 97 active+clean; 1.6 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-07-27T06:48:47.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.992+0000 7f3b90e4f640 1 -- 172.21.15.29:0/1194814493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b8c107dd0 msgr2=0x7f3b8c1081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.992+0000 7f3b90e4f640 1 --2- 172.21.15.29:0/1194814493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b8c107dd0 0x7f3b8c1081e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009900 tx=0x7f3b7402a410 comp rx=0 tx=0).stop 2024-07-27T06:48:47.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.993+0000 7f3b90e4f640 1 -- 172.21.15.29:0/1194814493 shutdown_connections 2024-07-27T06:48:47.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.993+0000 7f3b90e4f640 1 --2- 172.21.15.29:0/1194814493 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c108720 0x7f3b8c10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.993+0000 7f3b90e4f640 1 --2- 172.21.15.29:0/1194814493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b8c107dd0 0x7f3b8c1081e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.993+0000 7f3b90e4f640 1 --2- 172.21.15.29:0/1194814493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b8c102490 0x7f3b8c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.993+0000 7f3b90e4f640 1 -- 172.21.15.29:0/1194814493 >> 172.21.15.29:0/1194814493 conn(0x7f3b8c079160 msgr2=0x7f3b8c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:47.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.993+0000 7f3b90e4f640 1 -- 172.21.15.29:0/1194814493 shutdown_connections 2024-07-27T06:48:47.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.993+0000 7f3b90e4f640 1 -- 172.21.15.29:0/1194814493 wait complete. 2024-07-27T06:48:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b90e4f640 1 Processor -- start 2024-07-27T06:48:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b90e4f640 1 -- start start 2024-07-27T06:48:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b90e4f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b8c102490 0x7f3b8c0ff0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b90e4f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c107dd0 0x7f3b8c0ff5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b90e4f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b8c108720 0x7f3b8c0ffb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b90e4f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c112370 con 0x7f3b8c102490 2024-07-27T06:48:47.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b90e4f640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c1121f0 con 0x7f3b8c108720 2024-07-27T06:48:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b90e4f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c1124f0 con 0x7f3b8c107dd0 2024-07-27T06:48:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b8a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b8c102490 0x7f3b8c0ff0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b8a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b8c102490 0x7f3b8c0ff0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41646/0 (socket says 172.21.15.29:41646) 2024-07-27T06:48:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b8a59c640 1 -- 172.21.15.29:0/934906209 learned_addr learned my addr 172.21.15.29:0/934906209 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b89d9b640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c107dd0 0x7f3b8c0ff5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b8ad9d640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b8c108720 0x7f3b8c0ffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b8a59c640 1 -- 172.21.15.29:0/934906209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c107dd0 msgr2=0x7f3b8c0ff5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b8a59c640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c107dd0 0x7f3b8c0ff5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b8a59c640 1 -- 172.21.15.29:0/934906209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b8c108720 msgr2=0x7f3b8c0ffb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b8a59c640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b8c108720 0x7f3b8c0ffb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.994+0000 7f3b8a59c640 1 -- 172.21.15.29:0/934906209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8c1a8080 con 0x7f3b8c102490 2024-07-27T06:48:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.995+0000 7f3b8a59c640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b8c102490 0x7f3b8c0ff0a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c00cce0 tx=0x7f3b7c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.995+0000 7f3b837fe640 1 -- 172.21.15.29:0/934906209 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c00f650 con 0x7f3b8c102490 2024-07-27T06:48:47.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.995+0000 7f3b837fe640 1 -- 172.21.15.29:0/934906209 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b7c00fca0 con 0x7f3b8c102490 2024-07-27T06:48:47.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.995+0000 7f3b837fe640 1 -- 172.21.15.29:0/934906209 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c018b40 con 0x7f3b8c102490 2024-07-27T06:48:47.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.995+0000 7f3b90e4f640 1 -- 172.21.15.29:0/934906209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8c1a8370 con 0x7f3b8c102490 2024-07-27T06:48:47.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.995+0000 7f3b90e4f640 1 -- 172.21.15.29:0/934906209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b8c1a8880 con 0x7f3b8c102490 2024-07-27T06:48:47.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.996+0000 7f3b90e4f640 1 -- 172.21.15.29:0/934906209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b8c1028a0 con 0x7f3b8c102490 2024-07-27T06:48:47.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.998+0000 7f3b837fe640 1 -- 172.21.15.29:0/934906209 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f3b7c018d00 con 0x7f3b8c102490 2024-07-27T06:48:47.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.999+0000 7f3b837fe640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3b780b3d40 0x7f3b780b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.999+0000 7f3b89d9b640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3b780b3d40 0x7f3b780b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.999+0000 7f3b837fe640 1 -- 172.21.15.29:0/934906209 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f3b7c0e4700 con 0x7f3b8c102490 2024-07-27T06:48:47.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:46.999+0000 7f3b89d9b640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3b780b3d40 0x7f3b780b6250 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3b74000c00 tx=0x7f3b74031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:47.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.002+0000 7f3b837fe640 1 -- 172.21.15.29:0/934906209 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3b7c014030 con 0x7f3b8c102490 2024-07-27T06:48:47.152 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:46 smithi029 ceph-mon[32610]: pgmap v450: 97 pgs: 97 active+clean; 15 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-07-27T06:48:47.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.226+0000 7f3b90e4f640 1 -- 172.21.15.29:0/934906209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b8c1003a0 con 0x7f3b8c102490 2024-07-27T06:48:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.227+0000 7f3b837fe640 1 -- 172.21.15.29:0/934906209 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11957 (secure 0 0 0) 0x7f3b7c0aba50 con 0x7f3b8c102490 2024-07-27T06:48:47.229 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:48:47.229 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"btime":"2024-07-27T06:46:50:421100+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:46:49.561737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24670,24658,24646,24617,24613]},"id":1}]} 2024-07-27T06:48:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-07-27T06:48:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.229+0000 7f3b90e4f640 1 -- 172.21.15.29:0/934906209 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3b780b3d40 msgr2=0x7f3b780b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.229+0000 7f3b90e4f640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3b780b3d40 0x7f3b780b6250 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3b74000c00 tx=0x7f3b74031400 comp rx=0 tx=0).stop 2024-07-27T06:48:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.229+0000 7f3b90e4f640 1 -- 172.21.15.29:0/934906209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b8c102490 msgr2=0x7f3b8c0ff0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.229+0000 7f3b90e4f640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b8c102490 0x7f3b8c0ff0a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c00cce0 tx=0x7f3b7c00a620 comp rx=0 tx=0).stop 2024-07-27T06:48:47.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.229+0000 7f3b90e4f640 1 -- 172.21.15.29:0/934906209 shutdown_connections 2024-07-27T06:48:47.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.229+0000 7f3b90e4f640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3b8c108720 0x7f3b8c0ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.230+0000 7f3b90e4f640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3b780b3d40 0x7f3b780b6250 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.230+0000 7f3b90e4f640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b8c107dd0 0x7f3b8c0ff5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.230+0000 7f3b90e4f640 1 --2- 172.21.15.29:0/934906209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b8c102490 0x7f3b8c0ff0a0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.230+0000 7f3b90e4f640 1 -- 172.21.15.29:0/934906209 >> 172.21.15.29:0/934906209 conn(0x7f3b8c079160 msgr2=0x7f3b8c101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:47.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.230+0000 7f3b90e4f640 1 -- 172.21.15.29:0/934906209 shutdown_connections 2024-07-27T06:48:47.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.230+0000 7f3b90e4f640 1 -- 172.21.15.29:0/934906209 wait complete. 2024-07-27T06:48:47.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.247+0000 7f17ccc42640 1 -- 172.21.15.29:0/2152885233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8102490 msgr2=0x7f17c81028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.247+0000 7f17ccc42640 1 --2- 172.21.15.29:0/2152885233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8102490 0x7f17c81028a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009900 tx=0x7f17b802a410 comp rx=0 tx=0).stop 2024-07-27T06:48:47.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.247+0000 7f17ccc42640 1 -- 172.21.15.29:0/2152885233 shutdown_connections 2024-07-27T06:48:47.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.247+0000 7f17ccc42640 1 --2- 172.21.15.29:0/2152885233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17c8108720 0x7f17c810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.247+0000 7f17ccc42640 1 --2- 172.21.15.29:0/2152885233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17c8107dd0 0x7f17c81081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.247+0000 7f17ccc42640 1 --2- 172.21.15.29:0/2152885233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8102490 0x7f17c81028a0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.247+0000 7f17ccc42640 1 -- 172.21.15.29:0/2152885233 >> 172.21.15.29:0/2152885233 conn(0x7f17c8079160 msgr2=0x7f17c8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:47.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.247+0000 7f17ccc42640 1 -- 172.21.15.29:0/2152885233 shutdown_connections 2024-07-27T06:48:47.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.247+0000 7f17ccc42640 1 -- 172.21.15.29:0/2152885233 wait complete. 2024-07-27T06:48:47.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.248+0000 7f17ccc42640 1 Processor -- start 2024-07-27T06:48:47.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.248+0000 7f17ccc42640 1 -- start start 2024-07-27T06:48:47.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17ccc42640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8102490 0x7f17c810f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17ccc42640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17c8107dd0 0x7f17c810ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17ccc42640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17c8108720 0x7f17c8110460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17ccc42640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17c81134b0 con 0x7f17c8102490 2024-07-27T06:48:47.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17ccc42640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f17c8113330 con 0x7f17c8107dd0 2024-07-27T06:48:47.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17ccc42640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f17c8113630 con 0x7f17c8108720 2024-07-27T06:48:47.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8102490 0x7f17c810f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8102490 0x7f17c810f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41670/0 (socket says 172.21.15.29:41670) 2024-07-27T06:48:47.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17c659c640 1 -- 172.21.15.29:0/4000930014 learned_addr learned my addr 172.21.15.29:0/4000930014 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:47.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17c8108720 0x7f17c8110460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17c5d9b640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17c8107dd0 0x7f17c810ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17c659c640 1 -- 172.21.15.29:0/4000930014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17c8108720 msgr2=0x7f17c8110460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17c659c640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17c8108720 0x7f17c8110460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17c659c640 1 -- 172.21.15.29:0/4000930014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17c8107dd0 msgr2=0x7f17c810ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17c659c640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17c8107dd0 0x7f17c810ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.249+0000 7f17c659c640 1 -- 172.21.15.29:0/4000930014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c81a8120 con 0x7f17c8102490 2024-07-27T06:48:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.250+0000 7f17c659c640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8102490 0x7f17c810f9e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f17b8005940 tx=0x7f17b80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:47.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.250+0000 7f17b77fe640 1 -- 172.21.15.29:0/4000930014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b803f030 con 0x7f17c8102490 2024-07-27T06:48:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.250+0000 7f17b77fe640 1 -- 172.21.15.29:0/4000930014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17b8003cb0 con 0x7f17c8102490 2024-07-27T06:48:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.250+0000 7f17b77fe640 1 -- 172.21.15.29:0/4000930014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8035070 con 0x7f17c8102490 2024-07-27T06:48:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.250+0000 7f17ccc42640 1 -- 172.21.15.29:0/4000930014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c81a83b0 con 0x7f17c8102490 2024-07-27T06:48:47.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.250+0000 7f17ccc42640 1 -- 172.21.15.29:0/4000930014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c81a8890 con 0x7f17c8102490 2024-07-27T06:48:47.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.253+0000 7f17d31c6740 1 Processor -- start 2024-07-27T06:48:47.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.253+0000 7f17d31c6740 1 -- start start 2024-07-27T06:48:47.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17d31c6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b081070 0x558a3b426690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17d31c6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 0x558a3b426bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17d31c6740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b427110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17d31c6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558a3b39cbb0 con 0x558a3b0581b0 2024-07-27T06:48:47.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17d31c6740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x558a3b38c0b0 con 0x558a3b060390 2024-07-27T06:48:47.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17d31c6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558a3b3900f0 con 0x558a3b081070 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17c6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b081070 0x558a3b426690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17b77fe640 1 -- 172.21.15.29:0/4000930014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f17b8042030 con 0x7f17c8102490 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17c5d9b640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b427110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17c6d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b081070 0x558a3b426690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:42878/0 (socket says 172.21.15.29:42878) 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17c6d9d640 1 -- 172.21.15.29:0/153196801 learned_addr learned my addr 172.21.15.29:0/153196801 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17c6d9d640 1 -- 172.21.15.29:0/153196801 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 msgr2=0x558a3b427110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/153196801 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b427110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17c6d9d640 1 -- 172.21.15.29:0/153196801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 msgr2=0x558a3b426bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.254+0000 7f17c659c640 1 --2- 172.21.15.29:0/153196801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 0x558a3b426bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.255+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/153196801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 0x558a3b426bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.255+0000 7f17c6d9d640 1 -- 172.21.15.29:0/153196801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a3b332b90 con 0x558a3b081070 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.255+0000 7f17c659c640 1 --2- 172.21.15.29:0/153196801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 0x558a3b426bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:48:47.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.255+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/153196801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b081070 0x558a3b426690 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f17c0010370 tx=0x7f17c0010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0709ec621d10000 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.255+0000 7f17b4ff9640 1 -- 172.21.15.29:0/153196801 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0016070 con 0x558a3b081070 2024-07-27T06:48:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.255+0000 7f17b4ff9640 1 -- 172.21.15.29:0/153196801 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c0004d20 con 0x558a3b081070 2024-07-27T06:48:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.255+0000 7f17b4ff9640 1 -- 172.21.15.29:0/153196801 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0005420 con 0x558a3b081070 2024-07-27T06:48:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.256+0000 7f17b77fe640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17b00b3d40 0x7f17b00b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.256+0000 7f17c659c640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17b00b3d40 0x7f17b00b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.256+0000 7f17b77fe640 1 -- 172.21.15.29:0/4000930014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f17b8104490 con 0x7f17c8102490 2024-07-27T06:48:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.256+0000 7f17d31c6740 1 -- 172.21.15.29:0/153196801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b081070 msgr2=0x558a3b426690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.256+0000 7f17d31c6740 1 --2- 172.21.15.29:0/153196801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b081070 0x558a3b426690 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f17c0010370 tx=0x7f17c0010830 comp rx=0 tx=0).stop 2024-07-27T06:48:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.256+0000 7f17d31c6740 1 -- 172.21.15.29:0/153196801 shutdown_connections 2024-07-27T06:48:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.257+0000 7f17d31c6740 1 --2- 172.21.15.29:0/153196801 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b427110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.257+0000 7f17d31c6740 1 --2- 172.21.15.29:0/153196801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 0x558a3b426bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.257+0000 7f17d31c6740 1 --2- 172.21.15.29:0/153196801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b081070 0x558a3b426690 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.257+0000 7f17d31c6740 1 -- 172.21.15.29:0/153196801 >> 172.21.15.29:0/153196801 conn(0x558a3b075460 msgr2=0x558a3b424580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.257+0000 7f17c659c640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17b00b3d40 0x7f17b00b6250 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f17b8101ca0 tx=0x7f17b80045a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.257+0000 7f17d31c6740 1 -- 172.21.15.29:0/153196801 shutdown_connections 2024-07-27T06:48:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.257+0000 7f17d31c6740 1 -- 172.21.15.29:0/153196801 wait complete. 2024-07-27T06:48:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.257+0000 7f17d31c6740 1 -- start start 2024-07-27T06:48:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.257+0000 7f17d31c6740 1 Processor -- start 2024-07-27T06:48:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17d31c6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 0x558a3b432230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17d31c6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b060390 0x558a3b432770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17c6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 0x558a3b432230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17c6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 0x558a3b432230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41700/0 (socket says 172.21.15.29:41700) 2024-07-27T06:48:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17c6d9d640 1 -- 172.21.15.29:0/3008252603 learned_addr learned my addr 172.21.15.29:0/3008252603 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:47.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b081070 0x558a3b43e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17d31c6740 1 -- 172.21.15.29:0/3008252603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x558a3b1ef700 con 0x558a3b0581b0 2024-07-27T06:48:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17d31c6740 1 -- 172.21.15.29:0/3008252603 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x558a3b32e9e0 con 0x558a3b081070 2024-07-27T06:48:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17d31c6740 1 -- 172.21.15.29:0/3008252603 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558a3b434e70 con 0x558a3b060390 2024-07-27T06:48:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17c5d9b640 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b060390 0x558a3b432770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17d31c6740 10 client.0 _ll_register_callbacks cb 0x558a3b3494f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b081070 0x558a3b43e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17c6d9d640 1 -- 172.21.15.29:0/3008252603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b060390 msgr2=0x558a3b432770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b060390 0x558a3b432770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17c6d9d640 1 -- 172.21.15.29:0/3008252603 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b081070 msgr2=0x558a3b43e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b081070 0x558a3b43e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17c6d9d640 1 -- 172.21.15.29:0/3008252603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a3b0ec650 con 0x558a3b0581b0 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.258+0000 7f17c5d9b640 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b060390 0x558a3b432770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.259+0000 7f17d31c6740 1 -- 172.21.15.29:0/3008252603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17c0004d20 con 0x558a3b0581b0 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.259+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 0x558a3b432230 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f17c0006010 tx=0x7f17c0009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11961c5b65e758a4 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.259+0000 7f17adffb640 10 client.16068 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.259+0000 7f17adffb640 1 -- 172.21.15.29:0/3008252603 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0016030 con 0x558a3b0581b0 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.259+0000 7f17adffb640 1 -- 172.21.15.29:0/3008252603 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f17c0004040 con 0x558a3b0581b0 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.259+0000 7f17adffb640 1 -- 172.21.15.29:0/3008252603 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f17c0004280 con 0x558a3b0581b0 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.259+0000 7f17d31c6740 10 client.16068 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.259+0000 7f17d31c6740 1 -- 172.21.15.29:0/3008252603 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({fsmap=29}) -- 0x558a3b440c90 con 0x558a3b0581b0 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.260+0000 7f17adffb640 1 -- 172.21.15.29:0/3008252603 <== mon.0 v2:172.21.15.29:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f17c000f040 con 0x558a3b0581b0 2024-07-27T06:48:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.260+0000 7f17d31c6740 10 client.16068 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.260+0000 7f17d31c6740 10 client.16068 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.260+0000 7f17d31c6740 20 client.16068 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.260+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x558a3b03f9f0 0x558a3b443ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.260+0000 7f17d31c6740 4 client.16068 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.260+0000 7f17d31c6740 1 -- 172.21.15.29:0/3008252603 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a3b39eb00 con 0x558a3b03f9f0 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.260+0000 7f17c5d9b640 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x558a3b03f9f0 0x558a3b443ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.260+0000 7f17c5d9b640 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x558a3b03f9f0 0x558a3b443ff0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4919db7ddccc3e01 server_cookie=128d2c9b1aa28a7c in_seq=0 out_seq=0 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.260+0000 7f17adffb640 10 client.16068 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.261+0000 7f17adffb640 1 -- 172.21.15.29:0/3008252603 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558a3b39eb00 con 0x558a3b03f9f0 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.261+0000 7f17adffb640 10 client.16068 handle_command_reply: tid=0 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.261+0000 7f17d31c6740 1 client.16068 shutdown 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.261+0000 7f17d31c6740 1 -- 172.21.15.29:0/3008252603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 msgr2=0x558a3b432230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.261+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 0x558a3b432230 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f17c0006010 tx=0x7f17c0009d20 comp rx=0 tx=0).stop 2024-07-27T06:48:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.262+0000 7f17d31c6740 1 -- 172.21.15.29:0/3008252603 shutdown_connections 2024-07-27T06:48:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.262+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x558a3b03f9f0 0x558a3b443ff0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.262+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b081070 0x558a3b43e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.262+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b060390 0x558a3b432770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.262+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3008252603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b0581b0 0x558a3b432230 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.262+0000 7f17adffb640 0 client.16068 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:48:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.262+0000 7f17d31c6740 1 -- 172.21.15.29:0/3008252603 >> 172.21.15.29:0/3008252603 conn(0x558a3b075460 msgr2=0x558a3b422d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.262+0000 7f17d31c6740 1 -- 172.21.15.29:0/3008252603 shutdown_connections 2024-07-27T06:48:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.262+0000 7f17d31c6740 1 -- 172.21.15.29:0/3008252603 wait complete. 2024-07-27T06:48:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.262+0000 7f17d31c6740 20 client.16068 trim_cache size 0 max 16384 2024-07-27T06:48:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17d31c6740 1 Processor -- start 2024-07-27T06:48:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17d31c6740 1 -- start start 2024-07-27T06:48:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17d31c6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b0581b0 0x558a3b423410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17d31c6740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b43df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17d31c6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 0x558a3b42b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17d31c6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17c81acef0 con 0x558a3b081070 2024-07-27T06:48:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17d31c6740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f17c81aea30 con 0x558a3b060390 2024-07-27T06:48:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17d31c6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f17c81ad5c0 con 0x558a3b0581b0 2024-07-27T06:48:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17c6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 0x558a3b42b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17c6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 0x558a3b42b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41714/0 (socket says 172.21.15.29:41714) 2024-07-27T06:48:47.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17c6d9d640 1 -- 172.21.15.29:0/3865965456 learned_addr learned my addr 172.21.15.29:0/3865965456 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/3865965456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b0581b0 0x558a3b423410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.276+0000 7f17c5d9b640 1 --2- 172.21.15.29:0/3865965456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b43df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.277+0000 7f17c6d9d640 1 -- 172.21.15.29:0/3865965456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b0581b0 msgr2=0x558a3b423410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.277+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/3865965456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b0581b0 0x558a3b423410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.277+0000 7f17c6d9d640 1 -- 172.21.15.29:0/3865965456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 msgr2=0x558a3b43df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.277+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/3865965456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b43df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.277+0000 7f17c6d9d640 1 -- 172.21.15.29:0/3865965456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a3b428bb0 con 0x558a3b081070 2024-07-27T06:48:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.277+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/3865965456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 0x558a3b42b910 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f17c0004630 tx=0x7f17c0025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c49499a0ddcb84 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.277+0000 7f17affff640 1 -- 172.21.15.29:0/3865965456 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0016070 con 0x558a3b081070 2024-07-27T06:48:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.277+0000 7f17affff640 1 -- 172.21.15.29:0/3865965456 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c0004d20 con 0x558a3b081070 2024-07-27T06:48:47.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.277+0000 7f17affff640 1 -- 172.21.15.29:0/3865965456 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0039040 con 0x558a3b081070 2024-07-27T06:48:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.278+0000 7f17d31c6740 1 -- 172.21.15.29:0/3865965456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 msgr2=0x558a3b42b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.278+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3865965456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 0x558a3b42b910 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f17c0004630 tx=0x7f17c0025000 comp rx=0 tx=0).stop 2024-07-27T06:48:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.278+0000 7f17d31c6740 1 -- 172.21.15.29:0/3865965456 shutdown_connections 2024-07-27T06:48:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.278+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3865965456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 0x558a3b42b910 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.278+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3865965456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b43df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.278+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3865965456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b0581b0 0x558a3b423410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.278+0000 7f17d31c6740 1 -- 172.21.15.29:0/3865965456 >> 172.21.15.29:0/3865965456 conn(0x558a3b075460 msgr2=0x558a3b422d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.278+0000 7f17d31c6740 1 -- 172.21.15.29:0/3865965456 shutdown_connections 2024-07-27T06:48:47.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.278+0000 7f17d31c6740 1 -- 172.21.15.29:0/3865965456 wait complete. 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.278+0000 7f17d31c6740 1 -- start start 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17d31c6740 1 Processor -- start 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17d31c6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b0581b0 0x558a3b448da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17d31c6740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b449320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17d31c6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 0x558a3b439090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17d31c6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f17c81ae360 con 0x558a3b081070 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17d31c6740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f17c81adc90 con 0x558a3b060390 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17d31c6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f17c81aaed0 con 0x558a3b0581b0 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17d31c6740 10 client.0 _ll_register_callbacks cb 0x7f17bc00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17c6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 0x558a3b439090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17c6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 0x558a3b439090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41730/0 (socket says 172.21.15.29:41730) 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17c6d9d640 1 -- 172.21.15.29:0/3460552765 learned_addr learned my addr 172.21.15.29:0/3460552765 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:47.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17c5d9b640 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b449320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b0581b0 0x558a3b448da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17c5d9b640 1 -- 172.21.15.29:0/3460552765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b0581b0 msgr2=0x558a3b448da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.279+0000 7f17c5d9b640 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b0581b0 0x558a3b448da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.280+0000 7f17c5d9b640 1 -- 172.21.15.29:0/3460552765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 msgr2=0x558a3b439090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.280+0000 7f17c5d9b640 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 0x558a3b439090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.280+0000 7f17c5d9b640 1 -- 172.21.15.29:0/3460552765 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a3b439680 con 0x558a3b060390 2024-07-27T06:48:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.280+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 0x558a3b439090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.280+0000 7f17c5d9b640 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b449320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x558a3b1ef850 tx=0x7f17bc00bc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ab7219caf23cde9 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.280+0000 7f17adffb640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-07-27T06:48:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.280+0000 7f17adffb640 1 -- 172.21.15.29:0/3460552765 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc005020 con 0x558a3b060390 2024-07-27T06:48:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.280+0000 7f17adffb640 1 -- 172.21.15.29:0/3460552765 <== mon.2 v2:172.21.15.47:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f17bc0051c0 con 0x558a3b060390 2024-07-27T06:48:47.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.280+0000 7f17d31c6740 1 -- 172.21.15.29:0/3460552765 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17c0004d20 con 0x558a3b060390 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.280+0000 7f17adffb640 1 -- 172.21.15.29:0/3460552765 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f17bc0053c0 con 0x558a3b060390 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.280+0000 7f17d31c6740 10 client.25586 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.280+0000 7f17d31c6740 1 -- 172.21.15.29:0/3460552765 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558a3b43a8e0 con 0x558a3b060390 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.281+0000 7f17adffb640 1 -- 172.21.15.29:0/3460552765 <== mon.2 v2:172.21.15.47:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c0004d20 con 0x558a3b060390 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.281+0000 7f17adffb640 1 -- 172.21.15.29:0/3460552765 <== mon.2 v2:172.21.15.47:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f17bc02c030 con 0x558a3b060390 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.281+0000 7f17d31c6740 10 client.25586 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.281+0000 7f17d31c6740 10 client.25586 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.281+0000 7f17d31c6740 20 client.25586 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.281+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x558a3b014da0 0x558a3b4ae780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.281+0000 7f17d31c6740 4 client.25586 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.281+0000 7f17d31c6740 1 -- 172.21.15.29:0/3460552765 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558a3b39a5a0 con 0x558a3b014da0 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.281+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x558a3b014da0 0x558a3b4ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:47.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.281+0000 7f17c6d9d640 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x558a3b014da0 0x558a3b4ae780 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=32dde692c12296f8 server_cookie=3ac4d08133974984 in_seq=0 out_seq=0 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17adffb640 10 client.25586 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17adffb640 1 -- 172.21.15.29:0/3460552765 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558a3b39a5a0 con 0x558a3b014da0 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17adffb640 10 client.25586 handle_command_reply: tid=0 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17d31c6740 1 client.25586 shutdown 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17d31c6740 1 -- 172.21.15.29:0/3460552765 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 msgr2=0x558a3b449320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b449320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x558a3b1ef850 tx=0x7f17bc00bc60 comp rx=0 tx=0).stop 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17d31c6740 1 -- 172.21.15.29:0/3460552765 shutdown_connections 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x558a3b014da0 0x558a3b4ae780 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558a3b081070 0x558a3b439090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x558a3b060390 0x558a3b449320 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17d31c6740 1 --2- 172.21.15.29:0/3460552765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558a3b0581b0 0x558a3b448da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17d31c6740 1 -- 172.21.15.29:0/3460552765 >> 172.21.15.29:0/3460552765 conn(0x558a3b075460 msgr2=0x558a3b423e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:47.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17adffb640 0 client.25586 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:48:47.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17d31c6740 1 -- 172.21.15.29:0/3460552765 shutdown_connections 2024-07-27T06:48:47.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17d31c6740 1 -- 172.21.15.29:0/3460552765 wait complete. 2024-07-27T06:48:47.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.282+0000 7f17d31c6740 20 client.25586 trim_cache size 0 max 16384 2024-07-27T06:48:47.289 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:48:47.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.283+0000 7f17ccc42640 1 -- 172.21.15.29:0/4000930014 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17b00b3d40 msgr2=0x7f17b00b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.283+0000 7f17ccc42640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17b00b3d40 0x7f17b00b6250 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f17b8101ca0 tx=0x7f17b80045a0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.283+0000 7f17ccc42640 1 -- 172.21.15.29:0/4000930014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8102490 msgr2=0x7f17c810f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.283+0000 7f17ccc42640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8102490 0x7f17c810f9e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f17b8005940 tx=0x7f17b80042f0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.283+0000 7f17ccc42640 1 -- 172.21.15.29:0/4000930014 shutdown_connections 2024-07-27T06:48:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.283+0000 7f17ccc42640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f17b00b3d40 0x7f17b00b6250 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.283+0000 7f17ccc42640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17c8108720 0x7f17c8110460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.283+0000 7f17ccc42640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f17c8107dd0 0x7f17c810ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.283+0000 7f17ccc42640 1 --2- 172.21.15.29:0/4000930014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17c8102490 0x7f17c810f9e0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.283+0000 7f17ccc42640 1 -- 172.21.15.29:0/4000930014 >> 172.21.15.29:0/4000930014 conn(0x7f17c8079160 msgr2=0x7f17c8100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.283+0000 7f17ccc42640 1 -- 172.21.15.29:0/4000930014 shutdown_connections 2024-07-27T06:48:47.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:47.283+0000 7f17ccc42640 1 -- 172.21.15.29:0/4000930014 wait complete. 2024-07-27T06:48:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:46 smithi077 ceph-mon[37153]: pgmap v450: 97 pgs: 97 active+clean; 15 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-07-27T06:48:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:46 smithi047 ceph-mon[36835]: pgmap v450: 97 pgs: 97 active+clean; 15 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2024-07-27T06:48:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:47 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/934906209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:48:48.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-27T06:48:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:47 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/934906209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:48:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:47 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/934906209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:48:48.874 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24670 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0' 2024-07-27T06:48:48.875 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- timeout 105 ceph tell mds.24670 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0 2024-07-27T06:48:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.927+0000 7ff2b31ba640 1 -- 172.21.15.29:0/1581817780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2ac1023f0 msgr2=0x7ff2ac102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.927+0000 7ff2b31ba640 1 --2- 172.21.15.29:0/1581817780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2ac1023f0 0x7ff2ac102800 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009900 tx=0x7ff2a002a410 comp rx=0 tx=0).stop 2024-07-27T06:48:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.927+0000 7ff2b31ba640 1 -- 172.21.15.29:0/1581817780 shutdown_connections 2024-07-27T06:48:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.927+0000 7ff2b31ba640 1 --2- 172.21.15.29:0/1581817780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2ac108670 0x7ff2ac10eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.927+0000 7ff2b31ba640 1 --2- 172.21.15.29:0/1581817780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2ac107d20 0x7ff2ac108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.927+0000 7ff2b31ba640 1 --2- 172.21.15.29:0/1581817780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2ac1023f0 0x7ff2ac102800 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.927+0000 7ff2b31ba640 1 -- 172.21.15.29:0/1581817780 >> 172.21.15.29:0/1581817780 conn(0x7ff2ac0fddf0 msgr2=0x7ff2ac100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.927+0000 7ff2b31ba640 1 -- 172.21.15.29:0/1581817780 shutdown_connections 2024-07-27T06:48:48.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.927+0000 7ff2b31ba640 1 -- 172.21.15.29:0/1581817780 wait complete. 2024-07-27T06:48:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.928+0000 7ff2b31ba640 1 Processor -- start 2024-07-27T06:48:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.928+0000 7ff2b31ba640 1 -- start start 2024-07-27T06:48:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.928+0000 7ff2b31ba640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2ac1023f0 0x7ff2ac19acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.928+0000 7ff2b31ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2ac107d20 0x7ff2ac19b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.928+0000 7ff2b31ba640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2ac108670 0x7ff2ac19b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2b31ba640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac112340 con 0x7ff2ac107d20 2024-07-27T06:48:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2b31ba640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac1121c0 con 0x7ff2ac108670 2024-07-27T06:48:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2b31ba640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac1124c0 con 0x7ff2ac1023f0 2024-07-27T06:48:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2ac107d20 0x7ff2ac19b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2ac107d20 0x7ff2ac19b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50210/0 (socket says 172.21.15.29:50210) 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2abfff640 1 -- 172.21.15.29:0/616888772 learned_addr learned my addr 172.21.15.29:0/616888772 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2b1757640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2ac108670 0x7ff2ac19b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2b0f56640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2ac1023f0 0x7ff2ac19acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2abfff640 1 -- 172.21.15.29:0/616888772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2ac1023f0 msgr2=0x7ff2ac19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2abfff640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2ac1023f0 0x7ff2ac19acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2abfff640 1 -- 172.21.15.29:0/616888772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2ac108670 msgr2=0x7ff2ac19b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2abfff640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2ac108670 0x7ff2ac19b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2abfff640 1 -- 172.21.15.29:0/616888772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ac1a5f40 con 0x7ff2ac107d20 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.929+0000 7ff2abfff640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2ac107d20 0x7ff2ac19b1f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff29800e9c0 tx=0x7ff29800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.930+0000 7ff2a9ffb640 1 -- 172.21.15.29:0/616888772 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29800ccc0 con 0x7ff2ac107d20 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.930+0000 7ff2a9ffb640 1 -- 172.21.15.29:0/616888772 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2980054d0 con 0x7ff2ac107d20 2024-07-27T06:48:48.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.930+0000 7ff2a9ffb640 1 -- 172.21.15.29:0/616888772 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff298010430 con 0x7ff2ac107d20 2024-07-27T06:48:48.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.930+0000 7ff2b31ba640 1 -- 172.21.15.29:0/616888772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2ac1a6230 con 0x7ff2ac107d20 2024-07-27T06:48:48.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.930+0000 7ff2b31ba640 1 -- 172.21.15.29:0/616888772 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2ac1a6730 con 0x7ff2ac107d20 2024-07-27T06:48:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2ba1ac740 1 Processor -- start 2024-07-27T06:48:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2ba1ac740 1 -- start start 2024-07-27T06:48:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2a9ffb640 1 -- 172.21.15.29:0/616888772 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7ff298005020 con 0x7ff2ac107d20 2024-07-27T06:48:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2ba1ac740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738791b0 0x55bb73c1b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2ba1ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738502f0 0x55bb73c1bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2ba1ac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738584d0 0x55bb73c1c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2ba1ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bb73b913b0 con 0x55bb738502f0 2024-07-27T06:48:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2ba1ac740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55bb73b808b0 con 0x55bb738791b0 2024-07-27T06:48:48.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2ba1ac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bb73b848f0 con 0x55bb738584d0 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2b0f56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738502f0 0x55bb73c1bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2b1757640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738791b0 0x55bb73c1b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2b0f56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738502f0 0x55bb73c1bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50216/0 (socket says 172.21.15.29:50216) 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2b0f56640 1 -- 172.21.15.29:0/262784608 learned_addr learned my addr 172.21.15.29:0/262784608 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.933+0000 7ff2abfff640 1 --2- 172.21.15.29:0/262784608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738584d0 0x55bb73c1c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff2b0f56640 1 -- 172.21.15.29:0/262784608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738584d0 msgr2=0x55bb73c1c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff2b0f56640 1 --2- 172.21.15.29:0/262784608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738584d0 0x55bb73c1c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff2b0f56640 1 -- 172.21.15.29:0/262784608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738791b0 msgr2=0x55bb73c1b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff2b0f56640 1 --2- 172.21.15.29:0/262784608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738791b0 0x55bb73c1b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff2b0f56640 1 -- 172.21.15.29:0/262784608 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb73b279a0 con 0x55bb738502f0 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff2b1757640 1 --2- 172.21.15.29:0/262784608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738791b0 0x55bb73c1b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff2a9ffb640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff29c0b3d40 0x7ff29c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff2b0f56640 1 --2- 172.21.15.29:0/262784608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738502f0 0x55bb73c1bbf0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009900 tx=0x7ff2a0032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb54011aa2dbf026 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff2b1757640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff29c0b3d40 0x7ff29c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff2a9ffb640 1 -- 172.21.15.29:0/616888772 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7ff2980e4330 con 0x7ff2ac107d20 2024-07-27T06:48:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff296ffd640 1 -- 172.21.15.29:0/262784608 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0040070 con 0x55bb738502f0 2024-07-27T06:48:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff296ffd640 1 -- 172.21.15.29:0/262784608 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2a0004030 con 0x55bb738502f0 2024-07-27T06:48:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.934+0000 7ff296ffd640 1 -- 172.21.15.29:0/262784608 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0040070 con 0x55bb738502f0 2024-07-27T06:48:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.936+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/262784608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738502f0 msgr2=0x55bb73c1bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.936+0000 7ff2b1757640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff29c0b3d40 0x7ff29c0b6250 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009fd0 tx=0x7ff2a4009500 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.936+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/262784608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738502f0 0x55bb73c1bbf0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009900 tx=0x7ff2a0032b20 comp rx=0 tx=0).stop 2024-07-27T06:48:48.939 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.936+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/262784608 shutdown_connections 2024-07-27T06:48:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.936+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/262784608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738584d0 0x55bb73c1c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.936+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/262784608 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738502f0 0x55bb73c1bbf0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.936+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/262784608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738791b0 0x55bb73c1b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.936+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/262784608 >> 172.21.15.29:0/262784608 conn(0x55bb7386d300 msgr2=0x55bb73c195a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.936+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/262784608 shutdown_connections 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.936+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/262784608 wait complete. 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.936+0000 7ff2ba1ac740 1 -- start start 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.937+0000 7ff2ba1ac740 1 Processor -- start 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.937+0000 7ff2ba1ac740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738502f0 0x55bb73c271a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.937+0000 7ff2ba1ac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738584d0 0x55bb73c276e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.937+0000 7ff2ba1ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 0x55bb73c33120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.937+0000 7ff2ba1ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bb73abaab0 con 0x55bb738791b0 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.937+0000 7ff2ba1ac740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55bb73b08cf0 con 0x55bb738502f0 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.937+0000 7ff2ba1ac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bb73c29dd0 con 0x55bb738584d0 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.937+0000 7ff2ba1ac740 10 client.0 _ll_register_callbacks cb 0x55bb73b7a3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:48.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.937+0000 7ff2b1757640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 0x55bb73c33120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.937+0000 7ff2b1757640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 0x55bb73c33120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50220/0 (socket says 172.21.15.29:50220) 2024-07-27T06:48:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff2b1757640 1 -- 172.21.15.29:0/3902538281 learned_addr learned my addr 172.21.15.29:0/3902538281 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff2b0f56640 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738502f0 0x55bb73c271a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff2abfff640 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738584d0 0x55bb73c276e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff2b1757640 1 -- 172.21.15.29:0/3902538281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738584d0 msgr2=0x55bb73c276e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff2b1757640 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738584d0 0x55bb73c276e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff2b1757640 1 -- 172.21.15.29:0/3902538281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738502f0 msgr2=0x55bb73c271a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff2b1757640 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738502f0 0x55bb73c271a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff2b1757640 1 -- 172.21.15.29:0/3902538281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb738e4790 con 0x55bb738791b0 2024-07-27T06:48:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff2b0f56640 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738502f0 0x55bb73c271a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff2b1757640 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 0x55bb73c33120 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4011e30 tx=0x7ff2a4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a2cf457122cf36 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:48.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff28bfff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff28bfff640 1 -- 172.21.15.29:0/3902538281 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a401f9b0 con 0x55bb738791b0 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff28bfff640 1 -- 172.21.15.29:0/3902538281 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7ff2a401fb50 con 0x55bb738791b0 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.938+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/3902538281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2a0004030 con 0x55bb738791b0 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.939+0000 7ff28bfff640 1 -- 172.21.15.29:0/3902538281 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2a401fd50 con 0x55bb738791b0 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.939+0000 7ff2ba1ac740 10 client.16086 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.939+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/3902538281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bb7395d620 con 0x55bb738791b0 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.939+0000 7ff28bfff640 1 -- 172.21.15.29:0/3902538281 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2a0004030 con 0x55bb738791b0 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.939+0000 7ff28bfff640 1 -- 172.21.15.29:0/3902538281 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff2a4026070 con 0x55bb738791b0 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.939+0000 7ff2ba1ac740 10 client.16086 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.939+0000 7ff2ba1ac740 10 client.16086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.939+0000 7ff2ba1ac740 20 client.16086 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.939+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb73837d80 0x55bb73c39270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.939+0000 7ff2ba1ac740 4 client.16086 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.939+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/3902538281 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb73b93300 con 0x55bb73837d80 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.940+0000 7ff2b0f56640 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb73837d80 0x55bb73c39270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.940+0000 7ff28bfff640 10 client.16086 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.940+0000 7ff2b0f56640 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb73837d80 0x55bb73c39270 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3653ae958687b2bd server_cookie=9fa3708347d43fce in_seq=0 out_seq=0 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.941+0000 7ff28bfff640 1 -- 172.21.15.29:0/3902538281 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bb73b93300 con 0x55bb73837d80 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.941+0000 7ff28bfff640 10 client.16086 handle_command_reply: tid=0 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.941+0000 7ff2ba1ac740 1 client.16086 shutdown 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.941+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/3902538281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 msgr2=0x55bb73c33120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.941+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 0x55bb73c33120 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4011e30 tx=0x7ff2a4004ab0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.941+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/3902538281 shutdown_connections 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.941+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 0x55bb73c33120 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.941+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb73837d80 0x55bb73c39270 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.941+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738584d0 0x55bb73c276e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.941+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/3902538281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738502f0 0x55bb73c271a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.941+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/3902538281 >> 172.21.15.29:0/3902538281 conn(0x55bb7386d300 msgr2=0x55bb73c1c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:48.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.941+0000 7ff28bfff640 0 client.16086 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:48.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.942+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/3902538281 shutdown_connections 2024-07-27T06:48:48.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.942+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/3902538281 wait complete. 2024-07-27T06:48:48.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.942+0000 7ff2ba1ac740 20 client.16086 trim_cache size 0 max 16384 2024-07-27T06:48:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2ba1ac740 1 Processor -- start 2024-07-27T06:48:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2ba1ac740 1 -- start start 2024-07-27T06:48:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2ba1ac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738502f0 0x55bb73c196a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2ba1ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738584d0 0x55bb73c217f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2ba1ac740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738791b0 0x55bb73c26800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2ba1ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac1af110 con 0x55bb738584d0 2024-07-27T06:48:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2ba1ac740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac1b0c50 con 0x55bb738791b0 2024-07-27T06:48:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2ba1ac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac1af7e0 con 0x55bb738502f0 2024-07-27T06:48:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738584d0 0x55bb73c217f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.965 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738584d0 0x55bb73c217f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50226/0 (socket says 172.21.15.29:50226) 2024-07-27T06:48:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2abfff640 1 -- 172.21.15.29:0/1893645870 learned_addr learned my addr 172.21.15.29:0/1893645870 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2b0f56640 1 --2- 172.21.15.29:0/1893645870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738502f0 0x55bb73c196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2b1757640 1 --2- 172.21.15.29:0/1893645870 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738791b0 0x55bb73c26800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2abfff640 1 -- 172.21.15.29:0/1893645870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738502f0 msgr2=0x55bb73c196a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2abfff640 1 --2- 172.21.15.29:0/1893645870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738502f0 0x55bb73c196a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2abfff640 1 -- 172.21.15.29:0/1893645870 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738791b0 msgr2=0x55bb73c26800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.962+0000 7ff2abfff640 1 --2- 172.21.15.29:0/1893645870 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738791b0 0x55bb73c26800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.963+0000 7ff2abfff640 1 -- 172.21.15.29:0/1893645870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb73c25470 con 0x55bb738584d0 2024-07-27T06:48:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.963+0000 7ff2abfff640 1 --2- 172.21.15.29:0/1893645870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738584d0 0x55bb73c217f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff2980e1af0 tx=0x7ff2980a6ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555bb05117bbc891 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.963+0000 7ff2967fc640 1 -- 172.21.15.29:0/1893645870 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29800ce60 con 0x55bb738584d0 2024-07-27T06:48:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.963+0000 7ff2967fc640 1 -- 172.21.15.29:0/1893645870 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff298005310 con 0x55bb738584d0 2024-07-27T06:48:48.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.963+0000 7ff2967fc640 1 -- 172.21.15.29:0/1893645870 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2980af550 con 0x55bb738584d0 2024-07-27T06:48:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.964+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/1893645870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738584d0 msgr2=0x55bb73c217f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.964+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/1893645870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738584d0 0x55bb73c217f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff2980e1af0 tx=0x7ff2980a6ee0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.969 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.964+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/1893645870 shutdown_connections 2024-07-27T06:48:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.964+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/1893645870 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738791b0 0x55bb73c26800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.964+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/1893645870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738584d0 0x55bb73c217f0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.964+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/1893645870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738502f0 0x55bb73c196a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.964+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/1893645870 >> 172.21.15.29:0/1893645870 conn(0x55bb7386d300 msgr2=0x55bb73c18f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.964+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/1893645870 shutdown_connections 2024-07-27T06:48:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.964+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/1893645870 wait complete. 2024-07-27T06:48:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.964+0000 7ff2ba1ac740 1 -- start start 2024-07-27T06:48:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2ba1ac740 1 Processor -- start 2024-07-27T06:48:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2ba1ac740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738502f0 0x55bb73c2b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2ba1ac740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738584d0 0x55bb73c2b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2ba1ac740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 0x55bb73c2bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.970 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2ba1ac740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac1b0580 con 0x55bb738791b0 2024-07-27T06:48:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2ba1ac740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac1afeb0 con 0x55bb738584d0 2024-07-27T06:48:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2ba1ac740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac1ad0f0 con 0x55bb738502f0 2024-07-27T06:48:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2ba1ac740 10 client.0 _ll_register_callbacks cb 0x7ff2a4009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2b1757640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 0x55bb73c2bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2abfff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738584d0 0x55bb73c2b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2b1757640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 0x55bb73c2bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50228/0 (socket says 172.21.15.29:50228) 2024-07-27T06:48:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2b1757640 1 -- 172.21.15.29:0/228078780 learned_addr learned my addr 172.21.15.29:0/228078780 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:48.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.965+0000 7ff2b0f56640 1 --2- 172.21.15.29:0/228078780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738502f0 0x55bb73c2b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff2abfff640 1 -- 172.21.15.29:0/228078780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738502f0 msgr2=0x55bb73c2b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff2abfff640 1 --2- 172.21.15.29:0/228078780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738502f0 0x55bb73c2b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff2abfff640 1 -- 172.21.15.29:0/228078780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 msgr2=0x55bb73c2bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff2abfff640 1 --2- 172.21.15.29:0/228078780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 0x55bb73c2bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff2abfff640 1 -- 172.21.15.29:0/228078780 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb73c9ec60 con 0x55bb738584d0 2024-07-27T06:48:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff2abfff640 1 --2- 172.21.15.29:0/228078780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738584d0 0x55bb73c2b6a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff2980e1af0 tx=0x7ff29800ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c6b0f975fef63df server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff28bfff640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-07-27T06:48:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff28bfff640 1 -- 172.21.15.29:0/228078780 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2980aaa60 con 0x55bb738584d0 2024-07-27T06:48:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/228078780 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bb73b8eda0 con 0x55bb738584d0 2024-07-27T06:48:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff28bfff640 1 -- 172.21.15.29:0/228078780 <== mon.2 v2:172.21.15.47:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7ff2980b3b70 con 0x55bb738584d0 2024-07-27T06:48:48.972 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff28bfff640 1 -- 172.21.15.29:0/228078780 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2980ab650 con 0x55bb738584d0 2024-07-27T06:48:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff2ba1ac740 10 client.25604 fetch_fsmap learned FSMap version 29 2024-07-27T06:48:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/228078780 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bb73c9fef0 con 0x55bb738584d0 2024-07-27T06:48:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.966+0000 7ff28bfff640 1 -- 172.21.15.29:0/228078780 <== mon.2 v2:172.21.15.47:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55bb73b8eda0 con 0x55bb738584d0 2024-07-27T06:48:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.967+0000 7ff28bfff640 1 -- 172.21.15.29:0/228078780 <== mon.2 v2:172.21.15.47:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff2980e7c00 con 0x55bb738584d0 2024-07-27T06:48:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.967+0000 7ff2ba1ac740 10 client.25604 fetch_fsmap finished waiting for FSMap version 29 2024-07-27T06:48:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.967+0000 7ff2ba1ac740 10 client.25604 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.967+0000 7ff2ba1ac740 20 client.25604 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.967+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/228078780 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb7380d1c0 0x55bb73ca25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.967+0000 7ff2ba1ac740 4 client.25604 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-27T06:48:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.967+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/228078780 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bb73b8c4e0 con 0x55bb7380d1c0 2024-07-27T06:48:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.967+0000 7ff2b0f56640 1 --2- 172.21.15.29:0/228078780 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb7380d1c0 0x55bb73ca25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.968+0000 7ff28bfff640 10 client.25604 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:48.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:48.968+0000 7ff2b0f56640 1 --2- 172.21.15.29:0/228078780 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb7380d1c0 0x55bb73ca25c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82db5c8a93c5f14e server_cookie=aa45be0ce86c529b in_seq=0 out_seq=0 2024-07-27T06:48:49.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:48 smithi029 ceph-mon[32610]: pgmap v451: 97 pgs: 97 active+clean; 16 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-07-27T06:48:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff28bfff640 1 -- 172.21.15.29:0/228078780 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bb73b8c4e0 con 0x55bb7380d1c0 2024-07-27T06:48:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff28bfff640 10 client.25604 handle_command_reply: tid=0 2024-07-27T06:48:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff2ba1ac740 1 client.25604 shutdown 2024-07-27T06:48:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/228078780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738584d0 msgr2=0x55bb73c2b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/228078780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738584d0 0x55bb73c2b6a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff2980e1af0 tx=0x7ff29800ce60 comp rx=0 tx=0).stop 2024-07-27T06:48:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/228078780 shutdown_connections 2024-07-27T06:48:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/228078780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bb738791b0 0x55bb73c2bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/228078780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55bb738584d0 0x55bb73c2b6a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/228078780 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55bb7380d1c0 0x55bb73ca25c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff2ba1ac740 1 --2- 172.21.15.29:0/228078780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bb738502f0 0x55bb73c2b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff28bfff640 0 client.25604 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:49.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/228078780 >> 172.21.15.29:0/228078780 conn(0x55bb7386d300 msgr2=0x55bb73c207e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/228078780 shutdown_connections 2024-07-27T06:48:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff2ba1ac740 1 -- 172.21.15.29:0/228078780 wait complete. 2024-07-27T06:48:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.176+0000 7ff2ba1ac740 20 client.25604 trim_cache size 0 max 16384 2024-07-27T06:48:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-07-27T06:48:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-07-27T06:48:49.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-07-27T06:48:49.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-07-27T06:48:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.177+0000 7ff2b31ba640 1 -- 172.21.15.29:0/616888772 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff29c0b3d40 msgr2=0x7ff29c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.177+0000 7ff2b31ba640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff29c0b3d40 0x7ff29c0b6250 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009fd0 tx=0x7ff2a4009500 comp rx=0 tx=0).stop 2024-07-27T06:48:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.177+0000 7ff2b31ba640 1 -- 172.21.15.29:0/616888772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2ac107d20 msgr2=0x7ff2ac19b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.177+0000 7ff2b31ba640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2ac107d20 0x7ff2ac19b1f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff29800e9c0 tx=0x7ff29800ee80 comp rx=0 tx=0).stop 2024-07-27T06:48:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.177+0000 7ff2b31ba640 1 -- 172.21.15.29:0/616888772 shutdown_connections 2024-07-27T06:48:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.177+0000 7ff2b31ba640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2ac108670 0x7ff2ac19b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:49.182 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.177+0000 7ff2b31ba640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2ac107d20 0x7ff2ac19b1f0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.177+0000 7ff2b31ba640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ff29c0b3d40 0x7ff29c0b6250 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.177+0000 7ff2b31ba640 1 --2- 172.21.15.29:0/616888772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2ac1023f0 0x7ff2ac19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.177+0000 7ff2b31ba640 1 -- 172.21.15.29:0/616888772 >> 172.21.15.29:0/616888772 conn(0x7ff2ac0fddf0 msgr2=0x7ff2ac1073f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.177+0000 7ff2b31ba640 1 -- 172.21.15.29:0/616888772 shutdown_connections 2024-07-27T06:48:49.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:48:49.177+0000 7ff2b31ba640 1 -- 172.21.15.29:0/616888772 wait complete. 2024-07-27T06:48:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:48 smithi077 ceph-mon[37153]: pgmap v451: 97 pgs: 97 active+clean; 16 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-07-27T06:48:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:48 smithi047 ceph-mon[36835]: pgmap v451: 97 pgs: 97 active+clean; 16 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-07-27T06:48:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:51.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:50 smithi029 ceph-mon[32610]: pgmap v452: 97 pgs: 97 active+clean; 16 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-07-27T06:48:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:50 smithi077 ceph-mon[37153]: pgmap v452: 97 pgs: 97 active+clean; 16 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-07-27T06:48:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:50 smithi047 ceph-mon[36835]: pgmap v452: 97 pgs: 97 active+clean; 16 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-07-27T06:48:52.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.129+0000 7f636fea6640 1 -- 172.21.15.29:0/1620918030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6368108740 msgr2=0x7f636810ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.129+0000 7f636fea6640 1 --2- 172.21.15.29:0/1620918030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6368108740 0x7f636810ef70 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6360009ca0 tx=0x7f636002a8c0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.129+0000 7f636fea6640 1 -- 172.21.15.29:0/1620918030 shutdown_connections 2024-07-27T06:48:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.129+0000 7f636fea6640 1 --2- 172.21.15.29:0/1620918030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6368108740 0x7f636810ef70 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.129+0000 7f636fea6640 1 --2- 172.21.15.29:0/1620918030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6368107df0 0x7f6368108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.129+0000 7f636fea6640 1 --2- 172.21.15.29:0/1620918030 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63681024b0 0x7f63681028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.129+0000 7f636fea6640 1 -- 172.21.15.29:0/1620918030 >> 172.21.15.29:0/1620918030 conn(0x7f6368079190 msgr2=0x7f63680795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.129+0000 7f636fea6640 1 -- 172.21.15.29:0/1620918030 shutdown_connections 2024-07-27T06:48:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.129+0000 7f636fea6640 1 -- 172.21.15.29:0/1620918030 wait complete. 2024-07-27T06:48:52.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.130+0000 7f636fea6640 1 Processor -- start 2024-07-27T06:48:52.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.130+0000 7f636fea6640 1 -- start start 2024-07-27T06:48:52.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.130+0000 7f636fea6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63681024b0 0x7f636819ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.130+0000 7f636fea6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6368107df0 0x7f636819b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636fea6640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6368108740 0x7f636819b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636fea6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f63681123c0 con 0x7f6368107df0 2024-07-27T06:48:52.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636fea6640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f6368112240 con 0x7f6368108740 2024-07-27T06:48:52.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636fea6640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6368112540 con 0x7f63681024b0 2024-07-27T06:48:52.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636d441640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6368107df0 0x7f636819b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636dc42640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63681024b0 0x7f636819ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636d441640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6368107df0 0x7f636819b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50244/0 (socket says 172.21.15.29:50244) 2024-07-27T06:48:52.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636d441640 1 -- 172.21.15.29:0/1142164798 learned_addr learned my addr 172.21.15.29:0/1142164798 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:52.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636e443640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6368108740 0x7f636819b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636d441640 1 -- 172.21.15.29:0/1142164798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63681024b0 msgr2=0x7f636819ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636d441640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63681024b0 0x7f636819ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636d441640 1 -- 172.21.15.29:0/1142164798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6368108740 msgr2=0x7f636819b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636d441640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6368108740 0x7f636819b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636d441640 1 -- 172.21.15.29:0/1142164798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63681a5f60 con 0x7f6368107df0 2024-07-27T06:48:52.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636dc42640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63681024b0 0x7f636819ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:52.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f636d441640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6368107df0 0x7f636819b190 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f635800e9f0 tx=0x7f635800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:52.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.131+0000 7f635effd640 1 -- 172.21.15.29:0/1142164798 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635800ccc0 con 0x7f6368107df0 2024-07-27T06:48:52.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.132+0000 7f636fea6640 1 -- 172.21.15.29:0/1142164798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63681a6250 con 0x7f6368107df0 2024-07-27T06:48:52.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.132+0000 7f635effd640 1 -- 172.21.15.29:0/1142164798 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63580054d0 con 0x7f6368107df0 2024-07-27T06:48:52.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.132+0000 7f635effd640 1 -- 172.21.15.29:0/1142164798 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358010430 con 0x7f6368107df0 2024-07-27T06:48:52.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.132+0000 7f636fea6640 1 -- 172.21.15.29:0/1142164798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63681a6790 con 0x7f6368107df0 2024-07-27T06:48:52.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.134+0000 7f635effd640 1 -- 172.21.15.29:0/1142164798 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f63580046c0 con 0x7f6368107df0 2024-07-27T06:48:52.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.134+0000 7f637642a740 1 Processor -- start 2024-07-27T06:48:52.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.135+0000 7f637642a740 1 -- start start 2024-07-27T06:48:52.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.135+0000 7f637642a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbfb850 0x56216ffa1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.135+0000 7f637642a740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbd2990 0x56216ffa1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.135+0000 7f637642a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffa1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.135+0000 7f637642a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56216ff16690 con 0x56216fbdab70 2024-07-27T06:48:52.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.135+0000 7f637642a740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x56216ff05b90 con 0x56216fbd2990 2024-07-27T06:48:52.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.135+0000 7f637642a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56216ff09bd0 con 0x56216fbfb850 2024-07-27T06:48:52.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.135+0000 7f635effd640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63540b3d40 0x7f63540b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.135+0000 7f636d441640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffa1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.135+0000 7f636e443640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbfb850 0x56216ffa1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.135+0000 7f636d441640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffa1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50250/0 (socket says 172.21.15.29:50250) 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.135+0000 7f636d441640 1 -- 172.21.15.29:0/3183424981 learned_addr learned my addr 172.21.15.29:0/3183424981 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f636e443640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63540b3d40 0x7f63540b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f636dc42640 1 --2- 172.21.15.29:0/3183424981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbd2990 0x56216ffa1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f635effd640 1 -- 172.21.15.29:0/1142164798 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f63580e4300 con 0x7f6368107df0 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f636e443640 1 -- 172.21.15.29:0/3183424981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbd2990 msgr2=0x56216ffa1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f636e443640 1 --2- 172.21.15.29:0/3183424981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbd2990 0x56216ffa1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f636e443640 1 -- 172.21.15.29:0/3183424981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 msgr2=0x56216ffa1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f636e443640 1 --2- 172.21.15.29:0/3183424981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffa1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f636e443640 1 -- 172.21.15.29:0/3183424981 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56216fe91a10 con 0x56216fbfb850 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f636e443640 1 --2- 172.21.15.29:0/3183424981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbfb850 0x56216ffa1510 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f636806b970 tx=0x7f6360031450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfd72e3750dd38a3 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f636e443640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63540b3d40 0x7f63540b6250 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6360005e30 tx=0x7f636003b000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f6353fff640 1 -- 172.21.15.29:0/3183424981 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636004e070 con 0x56216fbfb850 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f6353fff640 1 -- 172.21.15.29:0/3183424981 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6360036040 con 0x56216fbfb850 2024-07-27T06:48:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.136+0000 7f6353fff640 1 -- 172.21.15.29:0/3183424981 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6360004820 con 0x56216fbfb850 2024-07-27T06:48:52.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.137+0000 7f637642a740 1 -- 172.21.15.29:0/3183424981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbfb850 msgr2=0x56216ffa1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.137+0000 7f637642a740 1 --2- 172.21.15.29:0/3183424981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbfb850 0x56216ffa1510 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f636806b970 tx=0x7f6360031450 comp rx=0 tx=0).stop 2024-07-27T06:48:52.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.137+0000 7f637642a740 1 -- 172.21.15.29:0/3183424981 shutdown_connections 2024-07-27T06:48:52.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.137+0000 7f637642a740 1 --2- 172.21.15.29:0/3183424981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffa1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.137+0000 7f637642a740 1 --2- 172.21.15.29:0/3183424981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbd2990 0x56216ffa1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.137+0000 7f637642a740 1 --2- 172.21.15.29:0/3183424981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbfb850 0x56216ffa1510 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.137+0000 7f637642a740 1 -- 172.21.15.29:0/3183424981 >> 172.21.15.29:0/3183424981 conn(0x56216fbefc40 msgr2=0x56216ff9f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:52.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.138+0000 7f637642a740 1 -- 172.21.15.29:0/3183424981 shutdown_connections 2024-07-27T06:48:52.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.138+0000 7f637642a740 1 -- 172.21.15.29:0/3183424981 wait complete. 2024-07-27T06:48:52.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.138+0000 7f637642a740 1 -- start start 2024-07-27T06:48:52.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.138+0000 7f637642a740 1 Processor -- start 2024-07-27T06:48:52.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f637642a740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbd2990 0x56216ffb1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f637642a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffb15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f637642a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbfb850 0x56216ffbd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f637642a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56216fd697b0 con 0x56216fbdab70 2024-07-27T06:48:52.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f637642a740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x56216fea8210 con 0x56216fbd2990 2024-07-27T06:48:52.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f637642a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56216ffb3e10 con 0x56216fbfb850 2024-07-27T06:48:52.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f637642a740 10 client.0 _ll_register_callbacks cb 0x56216fe892c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:52.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f636d441640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffb15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f636d441640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffb15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50256/0 (socket says 172.21.15.29:50256) 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f636d441640 1 -- 172.21.15.29:0/2020807638 learned_addr learned my addr 172.21.15.29:0/2020807638 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f636e443640 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbfb850 0x56216ffbd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f636dc42640 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbd2990 0x56216ffb1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f636d441640 1 -- 172.21.15.29:0/2020807638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbfb850 msgr2=0x56216ffbd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f636d441640 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbfb850 0x56216ffbd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f636d441640 1 -- 172.21.15.29:0/2020807638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbd2990 msgr2=0x56216ffb1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f636d441640 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbd2990 0x56216ffb1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f636d441640 1 -- 172.21.15.29:0/2020807638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56216feb9710 con 0x56216fbdab70 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f636d441640 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffb15c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6358004aa0 tx=0x7f63580aa930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de99f5b3a13f96db server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f63517fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f63517fa640 1 -- 172.21.15.29:0/2020807638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63580c4020 con 0x56216fbdab70 2024-07-27T06:48:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.139+0000 7f63517fa640 1 -- 172.21.15.29:0/2020807638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f63580b3b40 con 0x56216fbdab70 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f637642a740 1 -- 172.21.15.29:0/2020807638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56216ff185e0 con 0x56216fbdab70 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f63517fa640 1 -- 172.21.15.29:0/2020807638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f63580cc050 con 0x56216fbdab70 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f637642a740 10 client.16104 fetch_fsmap learned FSMap version 30 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f637642a740 1 -- 172.21.15.29:0/2020807638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56216fc75ba0 con 0x56216fbdab70 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f63517fa640 1 -- 172.21.15.29:0/2020807638 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56216ff185e0 con 0x56216fbdab70 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f63517fa640 1 -- 172.21.15.29:0/2020807638 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f63580aa180 con 0x56216fbdab70 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f637642a740 10 client.16104 fetch_fsmap finished waiting for FSMap version 30 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f637642a740 10 client.16104 resolve_mds: validated gid 24670 aka daemon mds.i 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f637642a740 20 client.16104 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f637642a740 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x56216fbba1d0 0x56216ffc2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f637642a740 4 client.16104 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f637642a740 1 -- 172.21.15.29:0/2020807638 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56216ff14080 con 0x56216fbba1d0 2024-07-27T06:48:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.140+0000 7f636dc42640 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x56216fbba1d0 0x56216ffc2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.141+0000 7f63517fa640 10 client.16104 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.141+0000 7f636dc42640 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x56216fbba1d0 0x56216ffc2ef0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eddb11f6db885c84 server_cookie=afa12c95be6a250e in_seq=0 out_seq=0 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.141+0000 7f63517fa640 1 -- 172.21.15.29:0/2020807638 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56216ff14080 con 0x56216fbba1d0 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.141+0000 7f63517fa640 10 client.16104 handle_command_reply: tid=0 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f637642a740 1 client.16104 shutdown 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f637642a740 1 -- 172.21.15.29:0/2020807638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 msgr2=0x56216ffb15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f637642a740 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffb15c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6358004aa0 tx=0x7f63580aa930 comp rx=0 tx=0).stop 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f637642a740 1 -- 172.21.15.29:0/2020807638 shutdown_connections 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f637642a740 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x56216fbba1d0 0x56216ffc2ef0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f637642a740 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbfb850 0x56216ffbd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f637642a740 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffb15c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f637642a740 1 --2- 172.21.15.29:0/2020807638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbd2990 0x56216ffb1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f63517fa640 0 client.16104 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:52.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f637642a740 1 -- 172.21.15.29:0/2020807638 >> 172.21.15.29:0/2020807638 conn(0x56216fbefc40 msgr2=0x56216ff9da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:52.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f637642a740 1 -- 172.21.15.29:0/2020807638 shutdown_connections 2024-07-27T06:48:52.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f637642a740 1 -- 172.21.15.29:0/2020807638 wait complete. 2024-07-27T06:48:52.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.142+0000 7f637642a740 20 client.16104 trim_cache size 0 max 16384 2024-07-27T06:48:52.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.159+0000 7f637642a740 1 Processor -- start 2024-07-27T06:48:52.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.159+0000 7f637642a740 1 -- start start 2024-07-27T06:48:52.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f637642a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbd2990 0x56216ffac360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f637642a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbdab70 0x56216ffac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f637642a740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbfb850 0x56216ffbcf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f637642a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f63681acec0 con 0x56216fbd2990 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f637642a740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f63681aea30 con 0x56216fbfb850 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f637642a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f63681ad590 con 0x56216fbdab70 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f636dc42640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbd2990 0x56216ffac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f636dc42640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbd2990 0x56216ffac360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50266/0 (socket says 172.21.15.29:50266) 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f636d441640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbdab70 0x56216ffac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f636dc42640 1 -- 172.21.15.29:0/297121777 learned_addr learned my addr 172.21.15.29:0/297121777 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f636e443640 1 --2- 172.21.15.29:0/297121777 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbfb850 0x56216ffbcf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f636d441640 1 -- 172.21.15.29:0/297121777 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbfb850 msgr2=0x56216ffbcf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f636d441640 1 --2- 172.21.15.29:0/297121777 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbfb850 0x56216ffbcf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f636d441640 1 -- 172.21.15.29:0/297121777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbd2990 msgr2=0x56216ffac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f636d441640 1 --2- 172.21.15.29:0/297121777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbd2990 0x56216ffac360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f636d441640 1 -- 172.21.15.29:0/297121777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56216ffb37c0 con 0x56216fbdab70 2024-07-27T06:48:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f636dc42640 1 --2- 172.21.15.29:0/297121777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbd2990 0x56216ffac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:52.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.160+0000 7f636d441640 1 --2- 172.21.15.29:0/297121777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbdab70 0x56216ffac8a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f635800cfa0 tx=0x7f63580aa9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92001074bc730220 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:52.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.161+0000 7f63537fe640 1 -- 172.21.15.29:0/297121777 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63580de030 con 0x56216fbdab70 2024-07-27T06:48:52.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.161+0000 7f63537fe640 1 -- 172.21.15.29:0/297121777 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63580cc1f0 con 0x56216fbdab70 2024-07-27T06:48:52.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.161+0000 7f63537fe640 1 -- 172.21.15.29:0/297121777 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63580c4070 con 0x56216fbdab70 2024-07-27T06:48:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.162+0000 7f637642a740 1 -- 172.21.15.29:0/297121777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbdab70 msgr2=0x56216ffac8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.162+0000 7f637642a740 1 --2- 172.21.15.29:0/297121777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbdab70 0x56216ffac8a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f635800cfa0 tx=0x7f63580aa9a0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.162+0000 7f637642a740 1 -- 172.21.15.29:0/297121777 shutdown_connections 2024-07-27T06:48:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.162+0000 7f637642a740 1 --2- 172.21.15.29:0/297121777 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbfb850 0x56216ffbcf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.162+0000 7f637642a740 1 --2- 172.21.15.29:0/297121777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbdab70 0x56216ffac8a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.162+0000 7f637642a740 1 --2- 172.21.15.29:0/297121777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbd2990 0x56216ffac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.162+0000 7f637642a740 1 -- 172.21.15.29:0/297121777 >> 172.21.15.29:0/297121777 conn(0x56216fbefc40 msgr2=0x56216ffabc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.162+0000 7f637642a740 1 -- 172.21.15.29:0/297121777 shutdown_connections 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.162+0000 7f637642a740 1 -- 172.21.15.29:0/297121777 wait complete. 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.162+0000 7f637642a740 1 -- start start 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.162+0000 7f637642a740 1 Processor -- start 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f637642a740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbd2990 0x56216ffb57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f637642a740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffb5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f637642a740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbfb850 0x56217002a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f637642a740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f63681ae330 con 0x56216fbdab70 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f637642a740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f63681adc60 con 0x56216fbfb850 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f637642a740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f63681aaea0 con 0x56216fbd2990 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f637642a740 10 client.0 _ll_register_callbacks cb 0x7f63580e4600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f636d441640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffb5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f636d441640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffb5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50270/0 (socket says 172.21.15.29:50270) 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f636d441640 1 -- 172.21.15.29:0/2103701239 learned_addr learned my addr 172.21.15.29:0/2103701239 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f636dc42640 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbd2990 0x56216ffb57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f636e443640 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbfb850 0x56217002a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f636d441640 1 -- 172.21.15.29:0/2103701239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbd2990 msgr2=0x56216ffb57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f636d441640 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbd2990 0x56216ffb57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f636d441640 1 -- 172.21.15.29:0/2103701239 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbfb850 msgr2=0x56217002a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f636d441640 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbfb850 0x56217002a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f636d441640 1 -- 172.21.15.29:0/2103701239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56217002ac90 con 0x56216fbdab70 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f636e443640 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbfb850 0x56217002a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f636d441640 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffb5d40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f63580aaeb0 tx=0x7f63580b3df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cede50450f59e7a1 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.163+0000 7f63517fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f63517fa640 1 -- 172.21.15.29:0/2103701239 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63580de070 con 0x56216fbdab70 2024-07-27T06:48:52.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f63517fa640 1 -- 172.21.15.29:0/2103701239 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f6358005210 con 0x56216fbdab70 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f637642a740 1 -- 172.21.15.29:0/2103701239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63580cc1f0 con 0x56216fbdab70 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f63517fa640 1 -- 172.21.15.29:0/2103701239 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f63580c4050 con 0x56216fbdab70 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f637642a740 10 client.16110 fetch_fsmap learned FSMap version 30 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f637642a740 1 -- 172.21.15.29:0/2103701239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56217002bef0 con 0x56216fbdab70 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f63517fa640 1 -- 172.21.15.29:0/2103701239 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63580cc1f0 con 0x56216fbdab70 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f63517fa640 1 -- 172.21.15.29:0/2103701239 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f63580a9bf0 con 0x56216fbdab70 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f637642a740 10 client.16110 fetch_fsmap finished waiting for FSMap version 30 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f637642a740 10 client.16110 resolve_mds: validated gid 24670 aka daemon mds.i 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f637642a740 20 client.16110 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f637642a740 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x56216fb8f580 0x56217002e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f637642a740 4 client.16110 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}] 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.164+0000 7f637642a740 1 -- 172.21.15.29:0/2103701239 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}) -- 0x56216ff117c0 con 0x56216fb8f580 2024-07-27T06:48:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.165+0000 7f636dc42640 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x56216fb8f580 0x56217002e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.165+0000 7f63517fa640 10 client.16110 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:52.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.165+0000 7f636dc42640 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x56216fb8f580 0x56217002e3f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9672759a0f463261 server_cookie=60055726fd258f5d in_seq=0 out_seq=0 2024-07-27T06:48:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.411+0000 7fa90c4f2640 1 -- 172.21.15.29:0/3647100009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa904107df0 msgr2=0x7fa904108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.411+0000 7fa90c4f2640 1 --2- 172.21.15.29:0/3647100009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa904107df0 0x7fa904108200 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4009900 tx=0x7fa8f402a410 comp rx=0 tx=0).stop 2024-07-27T06:48:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.411+0000 7fa90c4f2640 1 -- 172.21.15.29:0/3647100009 shutdown_connections 2024-07-27T06:48:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.411+0000 7fa90c4f2640 1 --2- 172.21.15.29:0/3647100009 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa904108740 0x7fa90410ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.411+0000 7fa90c4f2640 1 --2- 172.21.15.29:0/3647100009 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa904107df0 0x7fa904108200 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.411+0000 7fa90c4f2640 1 --2- 172.21.15.29:0/3647100009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9041024b0 0x7fa9041028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.411+0000 7fa90c4f2640 1 -- 172.21.15.29:0/3647100009 >> 172.21.15.29:0/3647100009 conn(0x7fa904079170 msgr2=0x7fa904079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:52.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.411+0000 7fa90c4f2640 1 -- 172.21.15.29:0/3647100009 shutdown_connections 2024-07-27T06:48:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.411+0000 7fa90c4f2640 1 -- 172.21.15.29:0/3647100009 wait complete. 2024-07-27T06:48:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.411+0000 7fa90c4f2640 1 Processor -- start 2024-07-27T06:48:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.411+0000 7fa90c4f2640 1 -- start start 2024-07-27T06:48:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90c4f2640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9041024b0 0x7fa90410b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90c4f2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa904107df0 0x7fa90410b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90c4f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa904108740 0x7fa90410be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90c4f2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa904112370 con 0x7fa904108740 2024-07-27T06:48:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90c4f2640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fa9041121f0 con 0x7fa9041024b0 2024-07-27T06:48:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90c4f2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa9041124f0 con 0x7fa904107df0 2024-07-27T06:48:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90aa8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa904108740 0x7fa90410be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90aa8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa904108740 0x7fa90410be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50290/0 (socket says 172.21.15.29:50290) 2024-07-27T06:48:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90aa8f640 1 -- 172.21.15.29:0/238666226 learned_addr learned my addr 172.21.15.29:0/238666226 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa909a8d640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa904107df0 0x7fa90410b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90a28e640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9041024b0 0x7fa90410b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90aa8f640 1 -- 172.21.15.29:0/238666226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa904107df0 msgr2=0x7fa90410b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90aa8f640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa904107df0 0x7fa90410b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90aa8f640 1 -- 172.21.15.29:0/238666226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9041024b0 msgr2=0x7fa90410b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90aa8f640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9041024b0 0x7fa90410b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.412+0000 7fa90aa8f640 1 -- 172.21.15.29:0/238666226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9041ac5b0 con 0x7fa904108740 2024-07-27T06:48:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.413+0000 7fa90aa8f640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa904108740 0x7fa90410be20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00eb10 tx=0x7fa8fc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.413+0000 7fa8fb7fe640 1 -- 172.21.15.29:0/238666226 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc004b20 con 0x7fa904108740 2024-07-27T06:48:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.413+0000 7fa8fb7fe640 1 -- 172.21.15.29:0/238666226 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8fc004cc0 con 0x7fa904108740 2024-07-27T06:48:52.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.413+0000 7fa90c4f2640 1 -- 172.21.15.29:0/238666226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9041ac8a0 con 0x7fa904108740 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.413+0000 7fa8fb7fe640 1 -- 172.21.15.29:0/238666226 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc010670 con 0x7fa904108740 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.413+0000 7fa90c4f2640 1 -- 172.21.15.29:0/238666226 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9041acd60 con 0x7fa904108740 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.416+0000 7fa912a76740 1 Processor -- start 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.416+0000 7fa8fb7fe640 1 -- 172.21.15.29:0/238666226 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fa8fc005020 con 0x7fa904108740 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.416+0000 7fa912a76740 1 -- start start 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa912a76740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f93031a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa912a76740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f93031aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa912a76740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f93031aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa912a76740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f930290bb0 con 0x55f92ff75070 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa912a76740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55f9302800b0 con 0x55f92ff54390 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa912a76740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f9302840f0 con 0x55f92ff4c1b0 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa909a8d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f93031a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa909a8d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f93031a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50300/0 (socket says 172.21.15.29:50300) 2024-07-27T06:48:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa909a8d640 1 -- 172.21.15.29:0/1718394115 learned_addr learned my addr 172.21.15.29:0/1718394115 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa90a28e640 1 --2- 172.21.15.29:0/1718394115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f93031aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa909a8d640 1 --2- 172.21.15.29:0/1718394115 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f93031aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa909a8d640 1 -- 172.21.15.29:0/1718394115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 msgr2=0x55f93031aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa909a8d640 1 --2- 172.21.15.29:0/1718394115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f93031aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa909a8d640 1 -- 172.21.15.29:0/1718394115 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 msgr2=0x55f93031aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa909a8d640 1 --2- 172.21.15.29:0/1718394115 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f93031aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa909a8d640 1 -- 172.21.15.29:0/1718394115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f930226b90 con 0x55f92ff75070 2024-07-27T06:48:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.417+0000 7fa90a28e640 1 --2- 172.21.15.29:0/1718394115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f93031aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.418+0000 7fa8fb7fe640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa8f00b3d40 0x7fa8f00b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.418+0000 7fa909a8d640 1 --2- 172.21.15.29:0/1718394115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f93031a570 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4009900 tx=0x7fa8f4031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839d40438359708f server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.418+0000 7fa8f8ff9640 1 -- 172.21.15.29:0/1718394115 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4035040 con 0x55f92ff75070 2024-07-27T06:48:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.418+0000 7fa8fb7fe640 1 -- 172.21.15.29:0/238666226 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fa8fc0e4540 con 0x7fa904108740 2024-07-27T06:48:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.418+0000 7fa90a28e640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa8f00b3d40 0x7fa8f00b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.418+0000 7fa8f8ff9640 1 -- 172.21.15.29:0/1718394115 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8f4045070 con 0x55f92ff75070 2024-07-27T06:48:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.418+0000 7fa8f8ff9640 1 -- 172.21.15.29:0/1718394115 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4005420 con 0x55f92ff75070 2024-07-27T06:48:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.420+0000 7fa912a76740 1 -- 172.21.15.29:0/1718394115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 msgr2=0x55f93031a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.420+0000 7fa90a28e640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa8f00b3d40 0x7fa8f00b6250 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55f9303191f0 tx=0x7fa900005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.420+0000 7fa912a76740 1 --2- 172.21.15.29:0/1718394115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f93031a570 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4009900 tx=0x7fa8f4031400 comp rx=0 tx=0).stop 2024-07-27T06:48:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.420+0000 7fa912a76740 1 -- 172.21.15.29:0/1718394115 shutdown_connections 2024-07-27T06:48:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.420+0000 7fa912a76740 1 --2- 172.21.15.29:0/1718394115 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f93031aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.420+0000 7fa912a76740 1 --2- 172.21.15.29:0/1718394115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f93031aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.420+0000 7fa912a76740 1 --2- 172.21.15.29:0/1718394115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f93031a570 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.420+0000 7fa912a76740 1 -- 172.21.15.29:0/1718394115 >> 172.21.15.29:0/1718394115 conn(0x55f92ff69460 msgr2=0x55f930318460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.420+0000 7fa912a76740 1 -- 172.21.15.29:0/1718394115 shutdown_connections 2024-07-27T06:48:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.420+0000 7fa912a76740 1 -- 172.21.15.29:0/1718394115 wait complete. 2024-07-27T06:48:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.420+0000 7fa912a76740 1 -- start start 2024-07-27T06:48:52.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.420+0000 7fa912a76740 1 Processor -- start 2024-07-27T06:48:52.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa912a76740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f9303261f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa912a76740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f9303267d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa912a76740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f930331fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa912a76740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55f9300e3700 con 0x55f92ff75070 2024-07-27T06:48:52.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa912a76740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55f9302229e0 con 0x55f92ff54390 2024-07-27T06:48:52.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa912a76740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f930328cf0 con 0x55f92ff4c1b0 2024-07-27T06:48:52.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa912a76740 10 client.0 _ll_register_callbacks cb 0x55f93023d4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:52.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa90aa8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f930331fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa909a8d640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f9303267d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa90aa8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f930331fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50306/0 (socket says 172.21.15.29:50306) 2024-07-27T06:48:52.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa90aa8f640 1 -- 172.21.15.29:0/3097467832 learned_addr learned my addr 172.21.15.29:0/3097467832 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:52.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa90a28e640 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f9303261f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa909a8d640 1 -- 172.21.15.29:0/3097467832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 msgr2=0x55f9303261f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa909a8d640 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f9303261f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa909a8d640 1 -- 172.21.15.29:0/3097467832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 msgr2=0x55f930331fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa909a8d640 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f930331fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa909a8d640 1 -- 172.21.15.29:0/3097467832 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f92ffe0650 con 0x55f92ff54390 2024-07-27T06:48:52.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.421+0000 7fa90aa8f640 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f930331fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.422+0000 7fa909a8d640 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f9303267d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4006010 tx=0x7fa8f402a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de121fd5778ae87e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.422+0000 7fa8edffb640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.422+0000 7fa8edffb640 1 -- 172.21.15.29:0/3097467832 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4050060 con 0x55f92ff54390 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.422+0000 7fa8edffb640 1 -- 172.21.15.29:0/3097467832 <== mon.2 v2:172.21.15.47:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fa8f40055d0 con 0x55f92ff54390 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.422+0000 7fa912a76740 1 -- 172.21.15.29:0/3097467832 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8f4045070 con 0x55f92ff54390 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.422+0000 7fa8edffb640 1 -- 172.21.15.29:0/3097467832 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa8f4035040 con 0x55f92ff54390 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.422+0000 7fa912a76740 10 client.25634 fetch_fsmap learned FSMap version 30 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.422+0000 7fa912a76740 1 -- 172.21.15.29:0/3097467832 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f930334db0 con 0x55f92ff54390 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.422+0000 7fa8edffb640 1 -- 172.21.15.29:0/3097467832 <== mon.2 v2:172.21.15.47:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8f4045070 con 0x55f92ff54390 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.423+0000 7fa8edffb640 1 -- 172.21.15.29:0/3097467832 <== mon.2 v2:172.21.15.47:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7fa8f4049450 con 0x55f92ff54390 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.423+0000 7fa912a76740 10 client.25634 fetch_fsmap finished waiting for FSMap version 30 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.423+0000 7fa912a76740 10 client.25634 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.423+0000 7fa912a76740 20 client.25634 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.423+0000 7fa912a76740 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55f92ff339f0 0x55f930338110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.423+0000 7fa912a76740 4 client.25634 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.423+0000 7fa912a76740 1 -- 172.21.15.29:0/3097467832 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f930292b00 con 0x55f92ff339f0 2024-07-27T06:48:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.423+0000 7fa90a28e640 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55f92ff339f0 0x55f930338110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.424+0000 7fa8edffb640 10 client.25634 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.424+0000 7fa90a28e640 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55f92ff339f0 0x55f930338110 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32efd857750de02 server_cookie=6621e5fb5e7d9c8f in_seq=0 out_seq=0 2024-07-27T06:48:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.427+0000 7fa8edffb640 1 -- 172.21.15.29:0/3097467832 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f930292b00 con 0x55f92ff339f0 2024-07-27T06:48:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.427+0000 7fa8edffb640 10 client.25634 handle_command_reply: tid=0 2024-07-27T06:48:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.427+0000 7fa912a76740 1 client.25634 shutdown 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.427+0000 7fa912a76740 1 -- 172.21.15.29:0/3097467832 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 msgr2=0x55f9303267d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.427+0000 7fa912a76740 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f9303267d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4006010 tx=0x7fa8f402a410 comp rx=0 tx=0).stop 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.427+0000 7fa912a76740 1 -- 172.21.15.29:0/3097467832 shutdown_connections 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.427+0000 7fa912a76740 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f930331fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.427+0000 7fa912a76740 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f9303267d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.427+0000 7fa912a76740 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55f92ff339f0 0x55f930338110 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.427+0000 7fa912a76740 1 --2- 172.21.15.29:0/3097467832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f9303261f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.427+0000 7fa8edffb640 0 client.25634 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.427+0000 7fa912a76740 1 -- 172.21.15.29:0/3097467832 >> 172.21.15.29:0/3097467832 conn(0x55f92ff69460 msgr2=0x55f930316c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.428+0000 7fa912a76740 1 -- 172.21.15.29:0/3097467832 shutdown_connections 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.428+0000 7fa912a76740 1 -- 172.21.15.29:0/3097467832 wait complete. 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.428+0000 7fa912a76740 20 client.25634 trim_cache size 0 max 16384 2024-07-27T06:48:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.444+0000 7fa912a76740 1 Processor -- start 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.444+0000 7fa912a76740 1 -- start start 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa912a76740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f93031fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa912a76740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f930331f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa912a76740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f930332490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa912a76740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa9041b5580 con 0x55f92ff75070 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa912a76740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fa9041b70f0 con 0x55f92ff54390 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa912a76740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa9041b5c50 con 0x55f92ff4c1b0 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa90a28e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f93031fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa90aa8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f930332490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa90a28e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f93031fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:56820/0 (socket says 172.21.15.29:56820) 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa90a28e640 1 -- 172.21.15.29:0/2501152233 learned_addr learned my addr 172.21.15.29:0/2501152233 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa909a8d640 1 --2- 172.21.15.29:0/2501152233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f930331f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa90a28e640 1 -- 172.21.15.29:0/2501152233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 msgr2=0x55f930331f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa90a28e640 1 --2- 172.21.15.29:0/2501152233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f930331f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa90a28e640 1 -- 172.21.15.29:0/2501152233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 msgr2=0x55f930332490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa90a28e640 1 --2- 172.21.15.29:0/2501152233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f930332490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa90a28e640 1 -- 172.21.15.29:0/2501152233 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9303287e0 con 0x55f92ff4c1b0 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.445+0000 7fa90aa8f640 1 --2- 172.21.15.29:0/2501152233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f930332490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.446+0000 7fa90a28e640 1 --2- 172.21.15.29:0/2501152233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f93031fd90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa900009fd0 tx=0x7fa900019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a709fc74652a989 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.446+0000 7fa8effff640 1 -- 172.21.15.29:0/2501152233 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90001ddb0 con 0x55f92ff4c1b0 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.446+0000 7fa8effff640 1 -- 172.21.15.29:0/2501152233 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa90002a8c0 con 0x55f92ff4c1b0 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.446+0000 7fa8effff640 1 -- 172.21.15.29:0/2501152233 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90000f7e0 con 0x55f92ff4c1b0 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.447+0000 7fa912a76740 1 -- 172.21.15.29:0/2501152233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 msgr2=0x55f93031fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.447+0000 7fa912a76740 1 --2- 172.21.15.29:0/2501152233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f93031fd90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa900009fd0 tx=0x7fa900019000 comp rx=0 tx=0).stop 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.447+0000 7fa912a76740 1 -- 172.21.15.29:0/2501152233 shutdown_connections 2024-07-27T06:48:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.447+0000 7fa912a76740 1 --2- 172.21.15.29:0/2501152233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff75070 0x55f930332490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.447+0000 7fa912a76740 1 --2- 172.21.15.29:0/2501152233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff54390 0x55f930331f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.447+0000 7fa912a76740 1 --2- 172.21.15.29:0/2501152233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff4c1b0 0x55f93031fd90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.447+0000 7fa912a76740 1 -- 172.21.15.29:0/2501152233 >> 172.21.15.29:0/2501152233 conn(0x55f92ff69460 msgr2=0x55f93031f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:52.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.447+0000 7fa912a76740 1 -- 172.21.15.29:0/2501152233 shutdown_connections 2024-07-27T06:48:52.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.447+0000 7fa912a76740 1 -- 172.21.15.29:0/2501152233 wait complete. 2024-07-27T06:48:52.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.448+0000 7fa912a76740 1 -- start start 2024-07-27T06:48:52.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.448+0000 7fa912a76740 1 Processor -- start 2024-07-27T06:48:52.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.448+0000 7fa912a76740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff4c1b0 0x55f93033b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.448+0000 7fa912a76740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff54390 0x55f930338c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa912a76740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff75070 0x55f930339170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa912a76740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa9041b69f0 con 0x55f92ff54390 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa912a76740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fa9041b6320 con 0x55f92ff4c1b0 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa912a76740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa9041b3560 con 0x55f92ff75070 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa912a76740 10 client.0 _ll_register_callbacks cb 0x7fa9000097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa909a8d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff54390 0x55f930338c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa90a28e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff4c1b0 0x55f93033b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa909a8d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff54390 0x55f930338c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50324/0 (socket says 172.21.15.29:50324) 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa909a8d640 1 -- 172.21.15.29:0/2284808498 learned_addr learned my addr 172.21.15.29:0/2284808498 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa90aa8f640 1 --2- 172.21.15.29:0/2284808498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff75070 0x55f930339170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa909a8d640 1 -- 172.21.15.29:0/2284808498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff75070 msgr2=0x55f930339170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa909a8d640 1 --2- 172.21.15.29:0/2284808498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff75070 0x55f930339170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa909a8d640 1 -- 172.21.15.29:0/2284808498 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff4c1b0 msgr2=0x55f93033b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa909a8d640 1 --2- 172.21.15.29:0/2284808498 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff4c1b0 0x55f93033b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa909a8d640 1 -- 172.21.15.29:0/2284808498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f930339860 con 0x55f92ff54390 2024-07-27T06:48:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa909a8d640 1 --2- 172.21.15.29:0/2284808498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff54390 0x55f930338c30 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4049870 tx=0x7fa8f4005600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821dc9c4dade314f server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.449+0000 7fa8edffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.450+0000 7fa8edffb640 1 -- 172.21.15.29:0/2284808498 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4049ae0 con 0x55f92ff54390 2024-07-27T06:48:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.450+0000 7fa8edffb640 1 -- 172.21.15.29:0/2284808498 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fa8f4049c80 con 0x55f92ff54390 2024-07-27T06:48:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.450+0000 7fa912a76740 1 -- 172.21.15.29:0/2284808498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa90002a8c0 con 0x55f92ff54390 2024-07-27T06:48:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.450+0000 7fa8edffb640 1 -- 172.21.15.29:0/2284808498 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa8f4049e80 con 0x55f92ff54390 2024-07-27T06:48:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.450+0000 7fa912a76740 10 client.16128 fetch_fsmap learned FSMap version 30 2024-07-27T06:48:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.450+0000 7fa912a76740 1 -- 172.21.15.29:0/2284808498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f9303a04f0 con 0x55f92ff54390 2024-07-27T06:48:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.450+0000 7fa8edffb640 1 -- 172.21.15.29:0/2284808498 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa90002a8c0 con 0x55f92ff54390 2024-07-27T06:48:52.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.451+0000 7fa8edffb640 1 -- 172.21.15.29:0/2284808498 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7fa8f4004920 con 0x55f92ff54390 2024-07-27T06:48:52.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.451+0000 7fa912a76740 10 client.16128 fetch_fsmap finished waiting for FSMap version 30 2024-07-27T06:48:52.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.451+0000 7fa912a76740 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:52.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.451+0000 7fa912a76740 20 client.16128 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:52.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.451+0000 7fa912a76740 1 --2- 172.21.15.29:0/2284808498 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55f92ff08da0 0x55f9303a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:52.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.451+0000 7fa912a76740 4 client.16128 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:48:52.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.451+0000 7fa912a76740 1 -- 172.21.15.29:0/2284808498 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f93028e5a0 con 0x55f92ff08da0 2024-07-27T06:48:52.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.451+0000 7fa90a28e640 1 --2- 172.21.15.29:0/2284808498 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55f92ff08da0 0x55f9303a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:52.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.451+0000 7fa90a28e640 1 --2- 172.21.15.29:0/2284808498 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55f92ff08da0 0x55f9303a2cf0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f019f51ab01204a server_cookie=6a8e2ecd0f0e23cd in_seq=0 out_seq=0 2024-07-27T06:48:52.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.451+0000 7fa8edffb640 10 client.16128 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:52.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.456+0000 7fa8edffb640 1 -- 172.21.15.29:0/2284808498 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f93028e5a0 con 0x55f92ff08da0 2024-07-27T06:48:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.456+0000 7fa8edffb640 10 client.16128 handle_command_reply: tid=0 2024-07-27T06:48:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.456+0000 7fa912a76740 1 client.16128 shutdown 2024-07-27T06:48:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.457+0000 7fa912a76740 1 -- 172.21.15.29:0/2284808498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff54390 msgr2=0x55f930338c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.457+0000 7fa912a76740 1 --2- 172.21.15.29:0/2284808498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff54390 0x55f930338c30 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4049870 tx=0x7fa8f4005600 comp rx=0 tx=0).stop 2024-07-27T06:48:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.457+0000 7fa912a76740 1 -- 172.21.15.29:0/2284808498 shutdown_connections 2024-07-27T06:48:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.457+0000 7fa912a76740 1 --2- 172.21.15.29:0/2284808498 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55f92ff08da0 0x55f9303a2cf0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.457+0000 7fa912a76740 1 --2- 172.21.15.29:0/2284808498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f92ff75070 0x55f930339170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.457+0000 7fa912a76740 1 --2- 172.21.15.29:0/2284808498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55f92ff54390 0x55f930338c30 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.457+0000 7fa912a76740 1 --2- 172.21.15.29:0/2284808498 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55f92ff4c1b0 0x55f93033b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.457+0000 7fa912a76740 1 -- 172.21.15.29:0/2284808498 >> 172.21.15.29:0/2284808498 conn(0x55f92ff69460 msgr2=0x55f930327470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.457+0000 7fa8edffb640 0 client.16128 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.457+0000 7fa912a76740 1 -- 172.21.15.29:0/2284808498 shutdown_connections 2024-07-27T06:48:52.515 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:48:52.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.457+0000 7fa912a76740 1 -- 172.21.15.29:0/2284808498 wait complete. 2024-07-27T06:48:52.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.457+0000 7fa912a76740 20 client.16128 trim_cache size 0 max 16384 2024-07-27T06:48:52.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.458+0000 7fa90c4f2640 1 -- 172.21.15.29:0/238666226 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa8f00b3d40 msgr2=0x7fa8f00b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.458+0000 7fa90c4f2640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa8f00b3d40 0x7fa8f00b6250 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55f9303191f0 tx=0x7fa900005f90 comp rx=0 tx=0).stop 2024-07-27T06:48:52.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.458+0000 7fa90c4f2640 1 -- 172.21.15.29:0/238666226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa904108740 msgr2=0x7fa90410be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:52.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.458+0000 7fa90c4f2640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa904108740 0x7fa90410be20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00eb10 tx=0x7fa8fc00c2d0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.458+0000 7fa90c4f2640 1 -- 172.21.15.29:0/238666226 shutdown_connections 2024-07-27T06:48:52.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.458+0000 7fa90c4f2640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa904108740 0x7fa90410be20 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.458+0000 7fa90c4f2640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa8f00b3d40 0x7fa8f00b6250 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.458+0000 7fa90c4f2640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa904107df0 0x7fa90410b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.458+0000 7fa90c4f2640 1 --2- 172.21.15.29:0/238666226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa9041024b0 0x7fa90410b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:52.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.458+0000 7fa90c4f2640 1 -- 172.21.15.29:0/238666226 >> 172.21.15.29:0/238666226 conn(0x7fa904079170 msgr2=0x7fa90406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:52.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.458+0000 7fa90c4f2640 1 -- 172.21.15.29:0/238666226 shutdown_connections 2024-07-27T06:48:52.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:52.458+0000 7fa90c4f2640 1 -- 172.21.15.29:0/238666226 wait complete. 2024-07-27T06:48:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:52 smithi029 ceph-mon[32610]: pgmap v453: 97 pgs: 97 active+clean; 16 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.2 MiB/s wr, 90 op/s 2024-07-27T06:48:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:52 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:48:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:52 smithi077 ceph-mon[37153]: pgmap v453: 97 pgs: 97 active+clean; 16 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.2 MiB/s wr, 90 op/s 2024-07-27T06:48:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:52 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:48:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:52 smithi047 ceph-mon[36835]: pgmap v453: 97 pgs: 97 active+clean; 16 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 1.2 MiB/s wr, 90 op/s 2024-07-27T06:48:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:52 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:48:54.501 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-27T06:48:54.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-27T06:48:54.765 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:48:54.765 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 30, 2024-07-27T06:48:54.765 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24670, 2024-07-27T06:48:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 6, 2024-07-27T06:48:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-07-27T06:48:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: "e30c826e": { 2024-07-27T06:48:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 6, 2024-07-27T06:48:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-07-27T06:48:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-07-27T06:48:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-07-27T06:48:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-07-27T06:48:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-07-27T06:48:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-07-27T06:48:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 142.6, 2024-07-27T06:48:54.766 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-07-27T06:48:54.767 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-07-27T06:48:54.767 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-07-27T06:48:54.767 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-07-27T06:48:54.767 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "QUIESCED", 2024-07-27T06:48:54.767 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-07-27T06:48:54.767 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:54.767 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:54.767 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:54.767 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:54.767 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:48:54.767 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:48:54.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.763+0000 7f63517fa640 1 -- 172.21.15.29:0/2103701239 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x56216ff117c0 con 0x56216fb8f580 2024-07-27T06:48:54.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.763+0000 7f63517fa640 10 client.16110 handle_command_reply: tid=0 2024-07-27T06:48:54.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.763+0000 7f637642a740 1 client.16110 shutdown 2024-07-27T06:48:54.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.763+0000 7f637642a740 1 -- 172.21.15.29:0/2103701239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 msgr2=0x56216ffb5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:54.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.763+0000 7f637642a740 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffb5d40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f63580aaeb0 tx=0x7f63580b3df0 comp rx=0 tx=0).stop 2024-07-27T06:48:54.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.764+0000 7f637642a740 1 -- 172.21.15.29:0/2103701239 shutdown_connections 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.764+0000 7f637642a740 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x56216fbfb850 0x56217002a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.764+0000 7f637642a740 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56216fbdab70 0x56216ffb5d40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.764+0000 7f637642a740 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x56216fb8f580 0x56217002e3f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.764+0000 7f637642a740 1 --2- 172.21.15.29:0/2103701239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56216fbd2990 0x56216ffb57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.764+0000 7f637642a740 1 -- 172.21.15.29:0/2103701239 >> 172.21.15.29:0/2103701239 conn(0x56216fbefc40 msgr2=0x56216ffabc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.764+0000 7f63517fa640 0 client.16110 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.764+0000 7f637642a740 1 -- 172.21.15.29:0/2103701239 shutdown_connections 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.764+0000 7f637642a740 1 -- 172.21.15.29:0/2103701239 wait complete. 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.764+0000 7f637642a740 20 client.16110 trim_cache size 0 max 16384 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.765+0000 7f636fea6640 1 -- 172.21.15.29:0/1142164798 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63540b3d40 msgr2=0x7f63540b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.765+0000 7f636fea6640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63540b3d40 0x7f63540b6250 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6360005e30 tx=0x7f636003b000 comp rx=0 tx=0).stop 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.765+0000 7f636fea6640 1 -- 172.21.15.29:0/1142164798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6368107df0 msgr2=0x7f636819b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.765+0000 7f636fea6640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6368107df0 0x7f636819b190 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f635800e9f0 tx=0x7f635800eeb0 comp rx=0 tx=0).stop 2024-07-27T06:48:54.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.765+0000 7f636fea6640 1 -- 172.21.15.29:0/1142164798 shutdown_connections 2024-07-27T06:48:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.765+0000 7f636fea6640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6368108740 0x7f636819b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.765+0000 7f636fea6640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6368107df0 0x7f636819b190 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.765+0000 7f636fea6640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f63540b3d40 0x7f63540b6250 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.765+0000 7f636fea6640 1 --2- 172.21.15.29:0/1142164798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63681024b0 0x7f636819ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.765+0000 7f636fea6640 1 -- 172.21.15.29:0/1142164798 >> 172.21.15.29:0/1142164798 conn(0x7f6368079190 msgr2=0x7f6368104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.765+0000 7f636fea6640 1 -- 172.21.15.29:0/1142164798 shutdown_connections 2024-07-27T06:48:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:54.765+0000 7f636fea6640 1 -- 172.21.15.29:0/1142164798 wait complete. 2024-07-27T06:48:54.994 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:55.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:54 smithi029 ceph-mon[32610]: pgmap v454: 97 pgs: 97 active+clean; 18 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-07-27T06:48:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:54 smithi077 ceph-mon[37153]: pgmap v454: 97 pgs: 97 active+clean; 18 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-07-27T06:48:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:54 smithi047 ceph-mon[36835]: pgmap v454: 97 pgs: 97 active+clean; 18 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 1.5 MiB/s wr, 131 op/s 2024-07-27T06:48:57.092 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'e30c826e', quiesce took 14.4 seconds. Will release after: 8.15 2024-07-27T06:48:57.312 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:57 smithi029 ceph-mon[32610]: pgmap v455: 97 pgs: 97 active+clean; 24 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 2.0 MiB/s wr, 250 op/s 2024-07-27T06:48:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:57 smithi077 ceph-mon[37153]: pgmap v455: 97 pgs: 97 active+clean; 24 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 2.0 MiB/s wr, 250 op/s 2024-07-27T06:48:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:57 smithi047 ceph-mon[36835]: pgmap v455: 97 pgs: 97 active+clean; 24 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 2.0 MiB/s wr, 250 op/s 2024-07-27T06:48:57.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.695+0000 7f3a19b87640 1 -- 172.21.15.29:0/2992994169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a14107dd0 msgr2=0x7f3a141081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:57.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.695+0000 7f3a19b87640 1 --2- 172.21.15.29:0/2992994169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a14107dd0 0x7f3a141081e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009900 tx=0x7f3a0802a410 comp rx=0 tx=0).stop 2024-07-27T06:48:57.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.695+0000 7f3a19b87640 1 -- 172.21.15.29:0/2992994169 shutdown_connections 2024-07-27T06:48:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.695+0000 7f3a19b87640 1 --2- 172.21.15.29:0/2992994169 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a14108720 0x7f3a1410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.695+0000 7f3a19b87640 1 --2- 172.21.15.29:0/2992994169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a14107dd0 0x7f3a141081e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.695+0000 7f3a19b87640 1 --2- 172.21.15.29:0/2992994169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a14102490 0x7f3a141028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.695+0000 7f3a19b87640 1 -- 172.21.15.29:0/2992994169 >> 172.21.15.29:0/2992994169 conn(0x7f3a14079160 msgr2=0x7f3a14079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.695+0000 7f3a19b87640 1 -- 172.21.15.29:0/2992994169 shutdown_connections 2024-07-27T06:48:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.695+0000 7f3a19b87640 1 -- 172.21.15.29:0/2992994169 wait complete. 2024-07-27T06:48:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.696+0000 7f3a19b87640 1 Processor -- start 2024-07-27T06:48:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.696+0000 7f3a19b87640 1 -- start start 2024-07-27T06:48:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a19b87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a14102490 0x7f3a14199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a19b87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a14107dd0 0x7f3a1419a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a19b87640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a14108720 0x7f3a1419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a19b87640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3a14112370 con 0x7f3a14102490 2024-07-27T06:48:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a19b87640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141121f0 con 0x7f3a14108720 2024-07-27T06:48:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a19b87640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141124f0 con 0x7f3a14107dd0 2024-07-27T06:48:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a14102490 0x7f3a14199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a14102490 0x7f3a14199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50340/0 (socket says 172.21.15.29:50340) 2024-07-27T06:48:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a137fe640 1 -- 172.21.15.29:0/3514234379 learned_addr learned my addr 172.21.15.29:0/3514234379 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a12ffd640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a14107dd0 0x7f3a1419a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a13fff640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a14108720 0x7f3a1419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a137fe640 1 -- 172.21.15.29:0/3514234379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a14107dd0 msgr2=0x7f3a1419a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a137fe640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a14107dd0 0x7f3a1419a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a137fe640 1 -- 172.21.15.29:0/3514234379 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a14108720 msgr2=0x7f3a1419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a137fe640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a14108720 0x7f3a1419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a137fe640 1 -- 172.21.15.29:0/3514234379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1419fe20 con 0x7f3a14102490 2024-07-27T06:48:57.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.697+0000 7f3a137fe640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a14102490 0x7f3a14199f70 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400cce0 tx=0x7f3a0400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:57.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.698+0000 7f3a10ff9640 1 -- 172.21.15.29:0/3514234379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a04004b20 con 0x7f3a14102490 2024-07-27T06:48:57.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.698+0000 7f3a10ff9640 1 -- 172.21.15.29:0/3514234379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a04004cc0 con 0x7f3a14102490 2024-07-27T06:48:57.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.698+0000 7f3a19b87640 1 -- 172.21.15.29:0/3514234379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a141a0110 con 0x7f3a14102490 2024-07-27T06:48:57.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.698+0000 7f3a10ff9640 1 -- 172.21.15.29:0/3514234379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0400f7c0 con 0x7f3a14102490 2024-07-27T06:48:57.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.698+0000 7f3a19b87640 1 -- 172.21.15.29:0/3514234379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a141a05d0 con 0x7f3a14102490 2024-07-27T06:48:57.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.700+0000 7f3a2010b740 1 Processor -- start 2024-07-27T06:48:57.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.700+0000 7f3a2010b740 1 -- start start 2024-07-27T06:48:57.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a10ff9640 1 -- 172.21.15.29:0/3514234379 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f3a04005560 con 0x7f3a14102490 2024-07-27T06:48:57.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a2010b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac4c350 0x55e3eaff13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a2010b740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eaff18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a2010b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 0x55e3eaff1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a2010b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e3eaed92f0 con 0x55e3eac2b670 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a2010b740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55e3eaf67a80 con 0x55e3eac23490 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a2010b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e3eaf56f80 con 0x55e3eac4c350 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a12ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 0x55e3eaff1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a12ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 0x55e3eaff1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50346/0 (socket says 172.21.15.29:50346) 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a12ffd640 1 -- 172.21.15.29:0/1877996685 learned_addr learned my addr 172.21.15.29:0/1877996685 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a13fff640 1 --2- 172.21.15.29:0/1877996685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac4c350 0x55e3eaff13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a137fe640 1 --2- 172.21.15.29:0/1877996685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eaff18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a12ffd640 1 -- 172.21.15.29:0/1877996685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac4c350 msgr2=0x55e3eaff13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a12ffd640 1 --2- 172.21.15.29:0/1877996685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac4c350 0x55e3eaff13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a12ffd640 1 -- 172.21.15.29:0/1877996685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 msgr2=0x55e3eaff18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:57.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a12ffd640 1 --2- 172.21.15.29:0/1877996685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eaff18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a12ffd640 1 -- 172.21.15.29:0/1877996685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3eacb7930 con 0x55e3eac2b670 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.701+0000 7f3a137fe640 1 --2- 172.21.15.29:0/1877996685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eaff18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.702+0000 7f3a12ffd640 1 --2- 172.21.15.29:0/1877996685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 0x55e3eaff1e30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009900 tx=0x7f3a08031de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de550705c7321ed server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.702+0000 7f3a10ff9640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f39fc0b3d70 0x7f39fc0b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.702+0000 7f3a13fff640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f39fc0b3d70 0x7f39fc0b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.702+0000 7f39f9ffb640 1 -- 172.21.15.29:0/1877996685 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08040070 con 0x55e3eac2b670 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.702+0000 7f39f9ffb640 1 -- 172.21.15.29:0/1877996685 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a080316a0 con 0x55e3eac2b670 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.702+0000 7f3a10ff9640 1 -- 172.21.15.29:0/3514234379 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f3a040e4070 con 0x7f3a14102490 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.702+0000 7f39f9ffb640 1 -- 172.21.15.29:0/1877996685 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08035040 con 0x55e3eac2b670 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.703+0000 7f3a2010b740 1 -- 172.21.15.29:0/1877996685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 msgr2=0x55e3eaff1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.703+0000 7f3a13fff640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f39fc0b3d70 0x7f39fc0b6280 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0045d0 tx=0x7f3a0c005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.703+0000 7f3a2010b740 1 --2- 172.21.15.29:0/1877996685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 0x55e3eaff1e30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009900 tx=0x7f3a08031de0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.703+0000 7f3a2010b740 1 -- 172.21.15.29:0/1877996685 shutdown_connections 2024-07-27T06:48:57.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.703+0000 7f3a2010b740 1 --2- 172.21.15.29:0/1877996685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 0x55e3eaff1e30 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.703+0000 7f3a2010b740 1 --2- 172.21.15.29:0/1877996685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eaff18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.703+0000 7f3a2010b740 1 --2- 172.21.15.29:0/1877996685 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac4c350 0x55e3eaff13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.703+0000 7f3a2010b740 1 -- 172.21.15.29:0/1877996685 >> 172.21.15.29:0/1877996685 conn(0x55e3eac40740 msgr2=0x55e3eafef2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.703+0000 7f3a2010b740 1 -- 172.21.15.29:0/1877996685 shutdown_connections 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.703+0000 7f3a2010b740 1 -- 172.21.15.29:0/1877996685 wait complete. 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.703+0000 7f3a2010b740 1 -- start start 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a2010b740 1 Processor -- start 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a2010b740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eb0013b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a2010b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac2b670 0x55e3eb0018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a2010b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 0x55e3eb00d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a2010b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e3eaef94f0 con 0x55e3eac4c350 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a2010b740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55e3eaf5afc0 con 0x55e3eac23490 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a2010b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e3eadba890 con 0x55e3eac2b670 2024-07-27T06:48:57.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a2010b740 10 client.0 _ll_register_callbacks cb 0x55e3eaf03500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a13fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 0x55e3eb00d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a13fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 0x55e3eb00d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50348/0 (socket says 172.21.15.29:50348) 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a13fff640 1 -- 172.21.15.29:0/4115758706 learned_addr learned my addr 172.21.15.29:0/4115758706 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a137fe640 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eb0013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.704+0000 7f3a12ffd640 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac2b670 0x55e3eb0018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f3a13fff640 1 -- 172.21.15.29:0/4115758706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac2b670 msgr2=0x55e3eb0018f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f3a13fff640 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac2b670 0x55e3eb0018f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f3a13fff640 1 -- 172.21.15.29:0/4115758706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 msgr2=0x55e3eb0013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f3a13fff640 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eb0013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f3a13fff640 1 -- 172.21.15.29:0/4115758706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3eaf08c50 con 0x55e3eac4c350 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f3a137fe640 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eb0013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f3a13fff640 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 0x55e3eb00d040 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0116a0 tx=0x7f3a0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae487b2b7827ee server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:57.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f39f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:48:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f39f2ffd640 1 -- 172.21.15.29:0/4115758706 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c029060 con 0x55e3eac4c350 2024-07-27T06:48:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f3a2010b740 1 -- 172.21.15.29:0/4115758706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a080316a0 con 0x55e3eac4c350 2024-07-27T06:48:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f39f2ffd640 1 -- 172.21.15.29:0/4115758706 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f3a0c01c7d0 con 0x55e3eac4c350 2024-07-27T06:48:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f39f2ffd640 1 -- 172.21.15.29:0/4115758706 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3a0c01c9d0 con 0x55e3eac4c350 2024-07-27T06:48:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f3a2010b740 10 client.16140 fetch_fsmap learned FSMap version 31 2024-07-27T06:48:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.705+0000 7f3a2010b740 1 -- 172.21.15.29:0/4115758706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e3eb00fe20 con 0x55e3eac4c350 2024-07-27T06:48:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.706+0000 7f39f2ffd640 1 -- 172.21.15.29:0/4115758706 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a080316a0 con 0x55e3eac4c350 2024-07-27T06:48:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.706+0000 7f39f2ffd640 1 -- 172.21.15.29:0/4115758706 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f3a0c018040 con 0x55e3eac4c350 2024-07-27T06:48:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.706+0000 7f3a2010b740 10 client.16140 fetch_fsmap finished waiting for FSMap version 31 2024-07-27T06:48:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.706+0000 7f3a2010b740 10 client.16140 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:57.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.706+0000 7f3a2010b740 20 client.16140 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.706+0000 7f3a2010b740 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3eac0acd0 0x55e3eb013400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.706+0000 7f3a2010b740 4 client.16140 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.706+0000 7f3a2010b740 1 -- 172.21.15.29:0/4115758706 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3eaf699d0 con 0x55e3eac0acd0 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.706+0000 7f3a137fe640 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3eac0acd0 0x55e3eb013400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.706+0000 7f39f2ffd640 10 client.16140 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.706+0000 7f3a137fe640 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3eac0acd0 0x55e3eb013400 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cf4dde9ec57a577 server_cookie=71f33869662e0820 in_seq=0 out_seq=0 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.707+0000 7f39f2ffd640 1 -- 172.21.15.29:0/4115758706 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e3eaf699d0 con 0x55e3eac0acd0 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.707+0000 7f39f2ffd640 10 client.16140 handle_command_reply: tid=0 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.707+0000 7f3a2010b740 1 client.16140 shutdown 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.707+0000 7f3a2010b740 1 -- 172.21.15.29:0/4115758706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 msgr2=0x55e3eb00d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.707+0000 7f3a2010b740 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 0x55e3eb00d040 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0116a0 tx=0x7f3a0c0046c0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.708+0000 7f3a2010b740 1 -- 172.21.15.29:0/4115758706 shutdown_connections 2024-07-27T06:48:57.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.708+0000 7f3a2010b740 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 0x55e3eb00d040 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.708+0000 7f3a2010b740 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3eac0acd0 0x55e3eb013400 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.708+0000 7f3a2010b740 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac2b670 0x55e3eb0018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.708+0000 7f3a2010b740 1 --2- 172.21.15.29:0/4115758706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eb0013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.708+0000 7f3a2010b740 1 -- 172.21.15.29:0/4115758706 >> 172.21.15.29:0/4115758706 conn(0x55e3eac40740 msgr2=0x55e3eafed8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.708+0000 7f39f2ffd640 0 client.16140 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.708+0000 7f3a2010b740 1 -- 172.21.15.29:0/4115758706 shutdown_connections 2024-07-27T06:48:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.708+0000 7f3a2010b740 1 -- 172.21.15.29:0/4115758706 wait complete. 2024-07-27T06:48:57.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.708+0000 7f3a2010b740 20 client.16140 trim_cache size 0 max 16384 2024-07-27T06:48:57.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.725+0000 7f3a2010b740 1 Processor -- start 2024-07-27T06:48:57.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.725+0000 7f3a2010b740 1 -- start start 2024-07-27T06:48:57.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a2010b740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eafedfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a2010b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 0x55e3eb00ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a2010b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac4c350 0x55e3eb018bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a2010b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141ace50 con 0x55e3eac2b670 2024-07-27T06:48:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a2010b740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141aea50 con 0x55e3eac23490 2024-07-27T06:48:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a2010b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141ad520 con 0x55e3eac4c350 2024-07-27T06:48:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a12ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 0x55e3eb00ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a12ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 0x55e3eb00ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50350/0 (socket says 172.21.15.29:50350) 2024-07-27T06:48:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a12ffd640 1 -- 172.21.15.29:0/1696647809 learned_addr learned my addr 172.21.15.29:0/1696647809 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a137fe640 1 --2- 172.21.15.29:0/1696647809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eafedfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a13fff640 1 --2- 172.21.15.29:0/1696647809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac4c350 0x55e3eb018bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a12ffd640 1 -- 172.21.15.29:0/1696647809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac4c350 msgr2=0x55e3eb018bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a12ffd640 1 --2- 172.21.15.29:0/1696647809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac4c350 0x55e3eb018bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a12ffd640 1 -- 172.21.15.29:0/1696647809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 msgr2=0x55e3eafedfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a12ffd640 1 --2- 172.21.15.29:0/1696647809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eafedfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a12ffd640 1 -- 172.21.15.29:0/1696647809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3eb002020 con 0x55e3eac2b670 2024-07-27T06:48:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f3a12ffd640 1 --2- 172.21.15.29:0/1696647809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 0x55e3eb00ce30 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3a08003930 tx=0x7f3a08003a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577c45146146d5c3 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.726+0000 7f39f97fa640 1 -- 172.21.15.29:0/1696647809 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08035050 con 0x55e3eac2b670 2024-07-27T06:48:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.727+0000 7f39f97fa640 1 -- 172.21.15.29:0/1696647809 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a080035f0 con 0x55e3eac2b670 2024-07-27T06:48:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.727+0000 7f39f97fa640 1 -- 172.21.15.29:0/1696647809 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08040040 con 0x55e3eac2b670 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.727+0000 7f3a2010b740 1 -- 172.21.15.29:0/1696647809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 msgr2=0x55e3eb00ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.727+0000 7f3a2010b740 1 --2- 172.21.15.29:0/1696647809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 0x55e3eb00ce30 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3a08003930 tx=0x7f3a08003a00 comp rx=0 tx=0).stop 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.727+0000 7f3a2010b740 1 -- 172.21.15.29:0/1696647809 shutdown_connections 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.727+0000 7f3a2010b740 1 --2- 172.21.15.29:0/1696647809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac4c350 0x55e3eb018bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.727+0000 7f3a2010b740 1 --2- 172.21.15.29:0/1696647809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac2b670 0x55e3eb00ce30 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.727+0000 7f3a2010b740 1 --2- 172.21.15.29:0/1696647809 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eafedfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.727+0000 7f3a2010b740 1 -- 172.21.15.29:0/1696647809 >> 172.21.15.29:0/1696647809 conn(0x55e3eac40740 msgr2=0x55e3eafed8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.728+0000 7f3a2010b740 1 -- 172.21.15.29:0/1696647809 shutdown_connections 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.728+0000 7f3a2010b740 1 -- 172.21.15.29:0/1696647809 wait complete. 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.728+0000 7f3a2010b740 1 -- start start 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.728+0000 7f3a2010b740 1 Processor -- start 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.728+0000 7f3a2010b740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eb009360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.728+0000 7f3a2010b740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac2b670 0x55e3eb0048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.728+0000 7f3a2010b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 0x55e3eb004df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.728+0000 7f3a2010b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141ae2c0 con 0x55e3eac4c350 2024-07-27T06:48:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.728+0000 7f3a2010b740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141adbf0 con 0x55e3eac23490 2024-07-27T06:48:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.728+0000 7f3a2010b740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141aae30 con 0x55e3eac2b670 2024-07-27T06:48:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.728+0000 7f3a2010b740 10 client.0 _ll_register_callbacks cb 0x7f3a0c003f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:48:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a13fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 0x55e3eb004df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a13fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 0x55e3eb004df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50360/0 (socket says 172.21.15.29:50360) 2024-07-27T06:48:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a13fff640 1 -- 172.21.15.29:0/223609384 learned_addr learned my addr 172.21.15.29:0/223609384 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:48:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a12ffd640 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac2b670 0x55e3eb0048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a137fe640 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eb009360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a12ffd640 1 -- 172.21.15.29:0/223609384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 msgr2=0x55e3eb009360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a12ffd640 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eb009360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a12ffd640 1 -- 172.21.15.29:0/223609384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 msgr2=0x55e3eb004df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a12ffd640 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 0x55e3eb004df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a12ffd640 1 -- 172.21.15.29:0/223609384 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3eb005330 con 0x55e3eac2b670 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a13fff640 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 0x55e3eb004df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a12ffd640 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac2b670 0x55e3eb0048b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a0802a930 tx=0x7f3a08005280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=833ee0621d01547f server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f39f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f39f2ffd640 1 -- 172.21.15.29:0/223609384 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08040030 con 0x55e3eac2b670 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f3a2010b740 1 -- 172.21.15.29:0/223609384 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a080035f0 con 0x55e3eac2b670 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.729+0000 7f39f2ffd640 1 -- 172.21.15.29:0/223609384 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f3a080314a0 con 0x55e3eac2b670 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.730+0000 7f39f2ffd640 1 -- 172.21.15.29:0/223609384 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3a080316c0 con 0x55e3eac2b670 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.730+0000 7f3a2010b740 10 client.25813 fetch_fsmap learned FSMap version 31 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.730+0000 7f3a2010b740 1 -- 172.21.15.29:0/223609384 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e3eb07b7f0 con 0x55e3eac2b670 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.730+0000 7f39f2ffd640 1 -- 172.21.15.29:0/223609384 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a080035f0 con 0x55e3eac2b670 2024-07-27T06:48:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.730+0000 7f39f2ffd640 1 -- 172.21.15.29:0/223609384 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f3a08054030 con 0x55e3eac2b670 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.730+0000 7f3a2010b740 10 client.25813 fetch_fsmap finished waiting for FSMap version 31 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.730+0000 7f3a2010b740 10 client.25813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.730+0000 7f3a2010b740 20 client.25813 populate_metadata read hostname 'smithi029' 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.730+0000 7f3a2010b740 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3eabe0080 0x55e3eb07da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.730+0000 7f3a2010b740 4 client.25813 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.730+0000 7f3a2010b740 1 -- 172.21.15.29:0/223609384 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e3eaf65470 con 0x55e3eabe0080 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.730+0000 7f3a137fe640 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3eabe0080 0x55e3eb07da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.731+0000 7f39f2ffd640 10 client.25813 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.731+0000 7f3a137fe640 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3eabe0080 0x55e3eb07da80 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39b9cb3bc2b531db server_cookie=ac61517320d22240 in_seq=0 out_seq=0 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f39f2ffd640 1 -- 172.21.15.29:0/223609384 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e3eaf65470 con 0x55e3eabe0080 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f39f2ffd640 10 client.25813 handle_command_reply: tid=0 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f3a2010b740 1 client.25813 shutdown 2024-07-27T06:48:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f3a2010b740 1 -- 172.21.15.29:0/223609384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac2b670 msgr2=0x55e3eb0048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:57.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f3a2010b740 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac2b670 0x55e3eb0048b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a0802a930 tx=0x7f3a08005280 comp rx=0 tx=0).stop 2024-07-27T06:48:57.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f3a2010b740 1 -- 172.21.15.29:0/223609384 shutdown_connections 2024-07-27T06:48:57.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f3a2010b740 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e3eac4c350 0x55e3eb004df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f3a2010b740 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e3eabe0080 0x55e3eb07da80 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f3a2010b740 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e3eac2b670 0x55e3eb0048b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f3a2010b740 1 --2- 172.21.15.29:0/223609384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e3eac23490 0x55e3eb009360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f39f2ffd640 0 client.25813 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:48:57.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f3a2010b740 1 -- 172.21.15.29:0/223609384 >> 172.21.15.29:0/223609384 conn(0x55e3eac40740 msgr2=0x55e3eaff6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:57.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f3a2010b740 1 -- 172.21.15.29:0/223609384 shutdown_connections 2024-07-27T06:48:57.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f3a2010b740 1 -- 172.21.15.29:0/223609384 wait complete. 2024-07-27T06:48:57.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.732+0000 7f3a2010b740 20 client.25813 trim_cache size 0 max 16384 2024-07-27T06:48:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:48:57.738 INFO:teuthology.orchestra.run.smithi029.stdout: "return_code": 0, 2024-07-27T06:48:57.738 INFO:teuthology.orchestra.run.smithi029.stdout: "scrub_tag": "790380e7-074c-4c4a-ac94-7e7a6aac95d1", 2024-07-27T06:48:57.738 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "asynchronous" 2024-07-27T06:48:57.738 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:48:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.733+0000 7f3a19b87640 1 -- 172.21.15.29:0/3514234379 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f39fc0b3d70 msgr2=0x7f39fc0b6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.733+0000 7f3a19b87640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f39fc0b3d70 0x7f39fc0b6280 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0045d0 tx=0x7f3a0c005f90 comp rx=0 tx=0).stop 2024-07-27T06:48:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.733+0000 7f3a19b87640 1 -- 172.21.15.29:0/3514234379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a14102490 msgr2=0x7f3a14199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:48:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.733+0000 7f3a19b87640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a14102490 0x7f3a14199f70 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400cce0 tx=0x7f3a0400a620 comp rx=0 tx=0).stop 2024-07-27T06:48:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.733+0000 7f3a19b87640 1 -- 172.21.15.29:0/3514234379 shutdown_connections 2024-07-27T06:48:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.733+0000 7f3a19b87640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3a14108720 0x7f3a1419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.733+0000 7f3a19b87640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f39fc0b3d70 0x7f39fc0b6280 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.733+0000 7f3a19b87640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3a14107dd0 0x7f3a1419a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.733+0000 7f3a19b87640 1 --2- 172.21.15.29:0/3514234379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a14102490 0x7f3a14199f70 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:48:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.733+0000 7f3a19b87640 1 -- 172.21.15.29:0/3514234379 >> 172.21.15.29:0/3514234379 conn(0x7f3a14079160 msgr2=0x7f3a1406ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:48:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.733+0000 7f3a19b87640 1 -- 172.21.15.29:0/3514234379 shutdown_connections 2024-07-27T06:48:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:48:57.733+0000 7f3a19b87640 1 -- 172.21.15.29:0/3514234379 wait complete. 2024-07-27T06:48:58.312 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:58 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:48:58.312 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:58 smithi029 ceph-mon[32610]: scrub queued for path: / 2024-07-27T06:48:58.312 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:58 smithi029 ceph-mon[32610]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:58.312 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:58 smithi029 ceph-mon[32610]: scrub summary: active paths [/] 2024-07-27T06:48:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:58 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:48:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:58 smithi077 ceph-mon[37153]: scrub queued for path: / 2024-07-27T06:48:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:58 smithi077 ceph-mon[37153]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:58 smithi077 ceph-mon[37153]: scrub summary: active paths [/] 2024-07-27T06:48:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:58 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:48:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:58 smithi047 ceph-mon[36835]: scrub queued for path: / 2024-07-27T06:48:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:58 smithi047 ceph-mon[36835]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:58 smithi047 ceph-mon[36835]: scrub summary: active paths [/] 2024-07-27T06:48:58.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-27T06:48:58.995 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:48:59.291 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:59 smithi029 ceph-mon[32610]: pgmap v456: 97 pgs: 97 active+clean; 25 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 1.1 MiB/s wr, 293 op/s 2024-07-27T06:48:59.291 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:59 smithi029 ceph-mon[32610]: scrub summary: active paths [/] 2024-07-27T06:48:59.291 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:59 smithi029 ceph-mon[32610]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:59.291 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:48:59 smithi029 ceph-mon[32610]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:59 smithi077 ceph-mon[37153]: pgmap v456: 97 pgs: 97 active+clean; 25 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 1.1 MiB/s wr, 293 op/s 2024-07-27T06:48:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:59 smithi077 ceph-mon[37153]: scrub summary: active paths [/] 2024-07-27T06:48:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:59 smithi077 ceph-mon[37153]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:48:59 smithi077 ceph-mon[37153]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:59 smithi047 ceph-mon[36835]: pgmap v456: 97 pgs: 97 active+clean; 25 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 1.1 MiB/s wr, 293 op/s 2024-07-27T06:48:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:59 smithi047 ceph-mon[36835]: scrub summary: active paths [/] 2024-07-27T06:48:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:59 smithi047 ceph-mon[36835]: scrub summary: idle+waiting paths [/] 2024-07-27T06:48:59.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:48:59 smithi047 ceph-mon[36835]: scrub summary: idle+waiting paths [/] 2024-07-27T06:49:00.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.867+0000 7f3ad159f640 1 -- 172.21.15.29:0/2515331366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc106600 msgr2=0x7f3acc110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:00.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.867+0000 7f3ad159f640 1 --2- 172.21.15.29:0/2515331366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc106600 0x7f3acc110090 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009ca0 tx=0x7f3ac402a8c0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.867+0000 7f3ad159f640 1 -- 172.21.15.29:0/2515331366 shutdown_connections 2024-07-27T06:49:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.867+0000 7f3ad159f640 1 --2- 172.21.15.29:0/2515331366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc106600 0x7f3acc110090 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.867+0000 7f3ad159f640 1 --2- 172.21.15.29:0/2515331366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc105cb0 0x7f3acc1060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.867+0000 7f3ad159f640 1 --2- 172.21.15.29:0/2515331366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3acc101a60 0x7f3acc101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.867+0000 7f3ad159f640 1 -- 172.21.15.29:0/2515331366 >> 172.21.15.29:0/2515331366 conn(0x7f3acc077690 msgr2=0x7f3acc077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.867+0000 7f3ad159f640 1 -- 172.21.15.29:0/2515331366 shutdown_connections 2024-07-27T06:49:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.867+0000 7f3ad159f640 1 -- 172.21.15.29:0/2515331366 wait complete. 2024-07-27T06:49:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.868+0000 7f3ad159f640 1 Processor -- start 2024-07-27T06:49:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.868+0000 7f3ad159f640 1 -- start start 2024-07-27T06:49:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3ad159f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc101a60 0x7f3acc199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3ad159f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3acc105cb0 0x7f3acc19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3ad159f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc106600 0x7f3acc19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3ad159f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1134d0 con 0x7f3acc101a60 2024-07-27T06:49:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3ad159f640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc113350 con 0x7f3acc105cb0 2024-07-27T06:49:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3ad159f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc113650 con 0x7f3acc106600 2024-07-27T06:49:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3acaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc101a60 0x7f3acc199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3aca7fc640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3acc105cb0 0x7f3acc19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3acaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc101a60 0x7f3acc199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37936/0 (socket says 172.21.15.29:37936) 2024-07-27T06:49:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3acaffd640 1 -- 172.21.15.29:0/4175708491 learned_addr learned my addr 172.21.15.29:0/4175708491 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc106600 0x7f3acc19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3aca7fc640 1 -- 172.21.15.29:0/4175708491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc106600 msgr2=0x7f3acc19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3aca7fc640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc106600 0x7f3acc19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3aca7fc640 1 -- 172.21.15.29:0/4175708491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc101a60 msgr2=0x7f3acc199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3aca7fc640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc101a60 0x7f3acc199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3aca7fc640 1 -- 172.21.15.29:0/4175708491 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3acc19fe30 con 0x7f3acc105cb0 2024-07-27T06:49:00.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3acaffd640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc101a60 0x7f3acc199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:49:00.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.869+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc106600 0x7f3acc19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:49:00.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.870+0000 7f3aca7fc640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3acc105cb0 0x7f3acc19a460 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0754e0 tx=0x7f3ac0009e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:00.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.870+0000 7f3ab3fff640 1 -- 172.21.15.29:0/4175708491 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0007860 con 0x7f3acc105cb0 2024-07-27T06:49:00.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.870+0000 7f3ad159f640 1 -- 172.21.15.29:0/4175708491 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3acc1a0120 con 0x7f3acc105cb0 2024-07-27T06:49:00.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.870+0000 7f3ad159f640 1 -- 172.21.15.29:0/4175708491 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3acc1028e0 con 0x7f3acc105cb0 2024-07-27T06:49:00.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.870+0000 7f3ab3fff640 1 -- 172.21.15.29:0/4175708491 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ac0005020 con 0x7f3acc105cb0 2024-07-27T06:49:00.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.870+0000 7f3ab3fff640 1 -- 172.21.15.29:0/4175708491 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0004c60 con 0x7f3acc105cb0 2024-07-27T06:49:00.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.873+0000 7f3ad7b23740 1 Processor -- start 2024-07-27T06:49:00.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.873+0000 7f3ad7b23740 1 -- start start 2024-07-27T06:49:00.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.873+0000 7f3ad7b23740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 0x55de0bea2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.873+0000 7f3ad7b23740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0bad41b0 0x55de0bea2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.873+0000 7f3ad7b23740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0badc390 0x55de0bea2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.873+0000 7f3ad7b23740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55de0be18b60 con 0x55de0bafd070 2024-07-27T06:49:00.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.873+0000 7f3ad7b23740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55de0be08060 con 0x55de0badc390 2024-07-27T06:49:00.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.873+0000 7f3ad7b23740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55de0be0c0a0 con 0x55de0bad41b0 2024-07-27T06:49:00.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.874+0000 7f3acb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 0x55de0bea2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.874+0000 7f3acb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 0x55de0bea2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37938/0 (socket says 172.21.15.29:37938) 2024-07-27T06:49:00.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.874+0000 7f3acb7fe640 1 -- 172.21.15.29:0/893282851 learned_addr learned my addr 172.21.15.29:0/893282851 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:00.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.874+0000 7f3acaffd640 1 --2- 172.21.15.29:0/893282851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0bad41b0 0x55de0bea2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.874+0000 7f3ab3fff640 1 -- 172.21.15.29:0/4175708491 <== mon.2 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f3ac001a420 con 0x7f3acc105cb0 2024-07-27T06:49:00.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.874+0000 7f3acaffd640 1 -- 172.21.15.29:0/893282851 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0badc390 msgr2=0x55de0bea2fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-27T06:49:00.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.874+0000 7f3acaffd640 1 --2- 172.21.15.29:0/893282851 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0badc390 0x55de0bea2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.874+0000 7f3acaffd640 1 -- 172.21.15.29:0/893282851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 msgr2=0x55de0bea2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:00.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.874+0000 7f3acaffd640 1 --2- 172.21.15.29:0/893282851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 0x55de0bea2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.874+0000 7f3acaffd640 1 -- 172.21.15.29:0/893282851 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de0bdaeb40 con 0x55de0bad41b0 2024-07-27T06:49:00.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.874+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/893282851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 0x55de0bea2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:49:00.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.874+0000 7f3acaffd640 1 --2- 172.21.15.29:0/893282851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0bad41b0 0x55de0bea2a60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00afd0 tx=0x7f3abc00f850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1954c467b30259b3 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:00.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.875+0000 7f3ab17fa640 1 -- 172.21.15.29:0/893282851 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc01d070 con 0x55de0bad41b0 2024-07-27T06:49:00.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.875+0000 7f3ab17fa640 1 -- 172.21.15.29:0/893282851 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3abc010db0 con 0x55de0bad41b0 2024-07-27T06:49:00.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.875+0000 7f3ab17fa640 1 -- 172.21.15.29:0/893282851 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc016340 con 0x55de0bad41b0 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.875+0000 7f3ab3fff640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3ab40b3d40 0x7f3ab40b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.875+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3ab40b3d40 0x7f3ab40b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.875+0000 7f3ab3fff640 1 -- 172.21.15.29:0/4175708491 <== mon.2 v2:172.21.15.47:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f3ac00e7420 con 0x7f3acc105cb0 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.876+0000 7f3ad7b23740 1 -- 172.21.15.29:0/893282851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0bad41b0 msgr2=0x55de0bea2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.876+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3ab40b3d40 0x7f3ab40b6250 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4035000 tx=0x7f3ac4031590 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.876+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/893282851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0bad41b0 0x55de0bea2a60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00afd0 tx=0x7f3abc00f850 comp rx=0 tx=0).stop 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.876+0000 7f3ad7b23740 1 -- 172.21.15.29:0/893282851 shutdown_connections 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.876+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/893282851 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0badc390 0x55de0bea2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.876+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/893282851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0bad41b0 0x55de0bea2a60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.876+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/893282851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 0x55de0bea2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.876+0000 7f3ad7b23740 1 -- 172.21.15.29:0/893282851 >> 172.21.15.29:0/893282851 conn(0x55de0baf1460 msgr2=0x55de0bea0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.876+0000 7f3ad7b23740 1 -- 172.21.15.29:0/893282851 shutdown_connections 2024-07-27T06:49:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.876+0000 7f3ad7b23740 1 -- 172.21.15.29:0/893282851 wait complete. 2024-07-27T06:49:00.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.877+0000 7f3ad7b23740 1 -- start start 2024-07-27T06:49:00.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.877+0000 7f3ad7b23740 1 Processor -- start 2024-07-27T06:49:00.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.877+0000 7f3ad7b23740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bad41b0 0x55de0beae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.877+0000 7f3ad7b23740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0badc390 0x55de0beae750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.877+0000 7f3ad7b23740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 0x55de0beb3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.877+0000 7f3ad7b23740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55de0bc6b700 con 0x55de0bafd070 2024-07-27T06:49:00.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.877+0000 7f3ad7b23740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55de0bdaa990 con 0x55de0bad41b0 2024-07-27T06:49:00.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.877+0000 7f3ad7b23740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55de0beb0d00 con 0x55de0badc390 2024-07-27T06:49:00.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.877+0000 7f3ad7b23740 10 client.0 _ll_register_callbacks cb 0x55de0bdc54a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:00.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.877+0000 7f3aca7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0badc390 0x55de0beae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3aca7fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0badc390 0x55de0beae750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:41450/0 (socket says 172.21.15.29:41450) 2024-07-27T06:49:00.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3acaffd640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bad41b0 0x55de0beae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3aca7fc640 1 -- 172.21.15.29:0/3523387080 learned_addr learned my addr 172.21.15.29:0/3523387080 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 0x55de0beb3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3aca7fc640 1 -- 172.21.15.29:0/3523387080 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bad41b0 msgr2=0x55de0beae1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3aca7fc640 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bad41b0 0x55de0beae1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3aca7fc640 1 -- 172.21.15.29:0/3523387080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 msgr2=0x55de0beb3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3aca7fc640 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 0x55de0beb3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3aca7fc640 1 -- 172.21.15.29:0/3523387080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de0bb68650 con 0x55de0badc390 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 0x55de0beb3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3aca7fc640 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0badc390 0x55de0beae750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0007fd0 tx=0x7f3ac00adc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cbdcb48ade4e64d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3aaeff5640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3aaeff5640 1 -- 172.21.15.29:0/3523387080 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac00b70f0 con 0x55de0badc390 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3aaeff5640 1 -- 172.21.15.29:0/3523387080 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f3ac00051d0 con 0x55de0badc390 2024-07-27T06:49:00.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.878+0000 7f3ad7b23740 1 -- 172.21.15.29:0/3523387080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3abc010db0 con 0x55de0badc390 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.879+0000 7f3aaeff5640 1 -- 172.21.15.29:0/3523387080 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3ac00aa380 con 0x55de0badc390 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.879+0000 7f3ad7b23740 10 client.25825 fetch_fsmap learned FSMap version 31 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.879+0000 7f3ad7b23740 1 -- 172.21.15.29:0/3523387080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55de0bebce00 con 0x55de0badc390 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.879+0000 7f3aaeff5640 1 -- 172.21.15.29:0/3523387080 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3abc010db0 con 0x55de0badc390 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.879+0000 7f3aaeff5640 1 -- 172.21.15.29:0/3523387080 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f3ac00a9af0 con 0x55de0badc390 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.879+0000 7f3ad7b23740 10 client.25825 fetch_fsmap finished waiting for FSMap version 31 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.879+0000 7f3ad7b23740 10 client.25825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.879+0000 7f3ad7b23740 20 client.25825 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.879+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55de0babb9f0 0x55de0bec0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.879+0000 7f3ad7b23740 4 client.25825 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.879+0000 7f3ad7b23740 1 -- 172.21.15.29:0/3523387080 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de0be1aab0 con 0x55de0babb9f0 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.879+0000 7f3acaffd640 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55de0babb9f0 0x55de0bec0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.880+0000 7f3acaffd640 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55de0babb9f0 0x55de0bec0340 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=877954dbbf813074 server_cookie=96bbca60aada8933 in_seq=0 out_seq=0 2024-07-27T06:49:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.880+0000 7f3aaeff5640 10 client.25825 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3aaeff5640 1 -- 172.21.15.29:0/3523387080 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55de0be1aab0 con 0x55de0babb9f0 2024-07-27T06:49:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3aaeff5640 10 client.25825 handle_command_reply: tid=0 2024-07-27T06:49:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3ad7b23740 1 client.25825 shutdown 2024-07-27T06:49:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3ad7b23740 1 -- 172.21.15.29:0/3523387080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0badc390 msgr2=0x55de0beae750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0badc390 0x55de0beae750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0007fd0 tx=0x7f3ac00adc00 comp rx=0 tx=0).stop 2024-07-27T06:49:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3ad7b23740 1 -- 172.21.15.29:0/3523387080 shutdown_connections 2024-07-27T06:49:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bafd070 0x55de0beb3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55de0babb9f0 0x55de0bec0340 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0badc390 0x55de0beae750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/3523387080 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bad41b0 0x55de0beae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3ad7b23740 1 -- 172.21.15.29:0/3523387080 >> 172.21.15.29:0/3523387080 conn(0x55de0baf1460 msgr2=0x55de0be9ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:00.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3aaeff5640 0 client.25825 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:00.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3ad7b23740 1 -- 172.21.15.29:0/3523387080 shutdown_connections 2024-07-27T06:49:00.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3ad7b23740 1 -- 172.21.15.29:0/3523387080 wait complete. 2024-07-27T06:49:00.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.881+0000 7f3ad7b23740 20 client.25825 trim_cache size 0 max 16384 2024-07-27T06:49:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.898+0000 7f3ad7b23740 1 Processor -- start 2024-07-27T06:49:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.898+0000 7f3ad7b23740 1 -- start start 2024-07-27T06:49:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3ad7b23740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0bad41b0 0x55de0beb3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3ad7b23740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0badc390 0x55de0bebad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3ad7b23740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bafd070 0x55de0bea7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3ad7b23740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1acf80 con 0x55de0badc390 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3ad7b23740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1aeb80 con 0x55de0bafd070 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3ad7b23740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1ad650 con 0x55de0bad41b0 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3aca7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0badc390 0x55de0bebad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3aca7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0badc390 0x55de0bebad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37962/0 (socket says 172.21.15.29:37962) 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3aca7fc640 1 -- 172.21.15.29:0/2673969525 learned_addr learned my addr 172.21.15.29:0/2673969525 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3acaffd640 1 --2- 172.21.15.29:0/2673969525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0bad41b0 0x55de0beb3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/2673969525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bafd070 0x55de0bea7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3aca7fc640 1 -- 172.21.15.29:0/2673969525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0bad41b0 msgr2=0x55de0beb3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3aca7fc640 1 --2- 172.21.15.29:0/2673969525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0bad41b0 0x55de0beb3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3aca7fc640 1 -- 172.21.15.29:0/2673969525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bafd070 msgr2=0x55de0bea7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3aca7fc640 1 --2- 172.21.15.29:0/2673969525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bafd070 0x55de0bea7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3aca7fc640 1 -- 172.21.15.29:0/2673969525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de0bea4460 con 0x55de0badc390 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3aca7fc640 1 --2- 172.21.15.29:0/2673969525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0badc390 0x55de0bebad80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0007590 tx=0x7f3ac0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cad96495e15cf49 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.899+0000 7f3ab0ff9640 1 -- 172.21.15.29:0/2673969525 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac00b70f0 con 0x55de0badc390 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.900+0000 7f3ab0ff9640 1 -- 172.21.15.29:0/2673969525 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ac0007c80 con 0x55de0badc390 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.900+0000 7f3ab0ff9640 1 -- 172.21.15.29:0/2673969525 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac00c8b80 con 0x55de0badc390 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.900+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2673969525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0badc390 msgr2=0x55de0bebad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.900+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/2673969525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0badc390 0x55de0bebad80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0007590 tx=0x7f3ac0000bc0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.900+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2673969525 shutdown_connections 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.900+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/2673969525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bafd070 0x55de0bea7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.900+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/2673969525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0badc390 0x55de0bebad80 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.900+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/2673969525 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0bad41b0 0x55de0beb3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.900+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2673969525 >> 172.21.15.29:0/2673969525 conn(0x55de0baf1460 msgr2=0x55de0beac830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.900+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2673969525 shutdown_connections 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.900+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2673969525 wait complete. 2024-07-27T06:49:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3ad7b23740 1 -- start start 2024-07-27T06:49:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3ad7b23740 1 Processor -- start 2024-07-27T06:49:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3ad7b23740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bad41b0 0x55de0beba7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3ad7b23740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0badc390 0x55de0beb5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3ad7b23740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bafd070 0x55de0beb61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3ad7b23740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1ae3f0 con 0x55de0bad41b0 2024-07-27T06:49:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3ad7b23740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1add20 con 0x55de0bafd070 2024-07-27T06:49:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3ad7b23740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3acc1aaf60 con 0x55de0badc390 2024-07-27T06:49:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3ad7b23740 10 client.0 _ll_register_callbacks cb 0x7f3ac0009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3acaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bad41b0 0x55de0beba7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3acaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bad41b0 0x55de0beba7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37974/0 (socket says 172.21.15.29:37974) 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3acaffd640 1 -- 172.21.15.29:0/2074714691 learned_addr learned my addr 172.21.15.29:0/2074714691 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3acb7fe640 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bafd070 0x55de0beb61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.901+0000 7f3aca7fc640 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0badc390 0x55de0beb5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3acaffd640 1 -- 172.21.15.29:0/2074714691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0badc390 msgr2=0x55de0beb5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3acaffd640 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0badc390 0x55de0beb5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3acaffd640 1 -- 172.21.15.29:0/2074714691 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bafd070 msgr2=0x55de0beb61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3acaffd640 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bafd070 0x55de0beb61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3acaffd640 1 -- 172.21.15.29:0/2074714691 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de0beb6720 con 0x55de0bad41b0 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3acaffd640 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bad41b0 0x55de0beba7f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00fd50 tx=0x7f3abc00a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd0faeca2d031cc server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3aaeff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3aaeff5640 1 -- 172.21.15.29:0/2074714691 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc01d040 con 0x55de0bad41b0 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3aaeff5640 1 -- 172.21.15.29:0/2074714691 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f3abc018640 con 0x55de0bad41b0 2024-07-27T06:49:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2074714691 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ac0007c80 con 0x55de0bad41b0 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3aaeff5640 1 -- 172.21.15.29:0/2074714691 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3abc037050 con 0x55de0bad41b0 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3ad7b23740 10 client.16164 fetch_fsmap learned FSMap version 31 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.902+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2074714691 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55de0bf28680 con 0x55de0bad41b0 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.903+0000 7f3aaeff5640 1 -- 172.21.15.29:0/2074714691 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ac0007c80 con 0x55de0bad41b0 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.903+0000 7f3aaeff5640 1 -- 172.21.15.29:0/2074714691 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f3abc005570 con 0x55de0bad41b0 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.903+0000 7f3ad7b23740 10 client.16164 fetch_fsmap finished waiting for FSMap version 31 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.903+0000 7f3ad7b23740 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.903+0000 7f3ad7b23740 20 client.16164 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.903+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55de0ba90da0 0x55de0bf2acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.903+0000 7f3ad7b23740 4 client.16164 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.903+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2074714691 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55de0be16550 con 0x55de0ba90da0 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.903+0000 7f3aca7fc640 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55de0ba90da0 0x55de0bf2acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:00.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.903+0000 7f3aaeff5640 10 client.16164 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.903+0000 7f3aca7fc640 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55de0ba90da0 0x55de0bf2acd0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72a113ea8fc27f1c server_cookie=c8956e057281fc54 in_seq=0 out_seq=0 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3aaeff5640 1 -- 172.21.15.29:0/2074714691 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55de0be16550 con 0x55de0ba90da0 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3aaeff5640 10 client.16164 handle_command_reply: tid=0 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad7b23740 1 client.16164 shutdown 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2074714691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bad41b0 msgr2=0x55de0beba7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bad41b0 0x55de0beba7f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00fd50 tx=0x7f3abc00a560 comp rx=0 tx=0).stop 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2074714691 shutdown_connections 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55de0bafd070 0x55de0beb61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55de0ba90da0 0x55de0bf2acd0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55de0badc390 0x55de0beb5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad7b23740 1 --2- 172.21.15.29:0/2074714691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55de0bad41b0 0x55de0beba7f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2074714691 >> 172.21.15.29:0/2074714691 conn(0x55de0baf1460 msgr2=0x55de0bea7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3aaeff5640 0 client.16164 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:00.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2074714691 shutdown_connections 2024-07-27T06:49:00.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad7b23740 1 -- 172.21.15.29:0/2074714691 wait complete. 2024-07-27T06:49:00.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad7b23740 20 client.16164 trim_cache size 0 max 16384 2024-07-27T06:49:00.911 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:49:00.911 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-27T06:49:00.911 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": { 2024-07-27T06:49:00.911 INFO:teuthology.orchestra.run.smithi029.stdout: "790380e7-074c-4c4a-ac94-7e7a6aac95d1": { 2024-07-27T06:49:00.911 INFO:teuthology.orchestra.run.smithi029.stdout: "path": "/", 2024-07-27T06:49:00.911 INFO:teuthology.orchestra.run.smithi029.stdout: "tag": "790380e7-074c-4c4a-ac94-7e7a6aac95d1", 2024-07-27T06:49:00.911 INFO:teuthology.orchestra.run.smithi029.stdout: "options": "recursive,force" 2024-07-27T06:49:00.912 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:49:00.912 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:49:00.912 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:49:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad159f640 1 -- 172.21.15.29:0/4175708491 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3ab40b3d40 msgr2=0x7f3ab40b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad159f640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3ab40b3d40 0x7f3ab40b6250 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4035000 tx=0x7f3ac4031590 comp rx=0 tx=0).stop 2024-07-27T06:49:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.904+0000 7f3ad159f640 1 -- 172.21.15.29:0/4175708491 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3acc105cb0 msgr2=0x7f3acc19a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.905+0000 7f3ad159f640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3acc105cb0 0x7f3acc19a460 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0754e0 tx=0x7f3ac0009e20 comp rx=0 tx=0).stop 2024-07-27T06:49:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.905+0000 7f3ad159f640 1 -- 172.21.15.29:0/4175708491 shutdown_connections 2024-07-27T06:49:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.905+0000 7f3ad159f640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f3ab40b3d40 0x7f3ab40b6250 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.905+0000 7f3ad159f640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3acc106600 0x7f3acc19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.905+0000 7f3ad159f640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3acc105cb0 0x7f3acc19a460 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.905+0000 7f3ad159f640 1 --2- 172.21.15.29:0/4175708491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3acc101a60 0x7f3acc199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.905+0000 7f3ad159f640 1 -- 172.21.15.29:0/4175708491 >> 172.21.15.29:0/4175708491 conn(0x7f3acc077690 msgr2=0x7f3acc106ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.905+0000 7f3ad159f640 1 -- 172.21.15.29:0/4175708491 shutdown_connections 2024-07-27T06:49:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:00.905+0000 7f3ad159f640 1 -- 172.21.15.29:0/4175708491 wait complete. 2024-07-27T06:49:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:01 smithi029 ceph-mon[32610]: pgmap v457: 97 pgs: 97 active+clean; 25 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1004 KiB/s wr, 290 op/s 2024-07-27T06:49:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:01 smithi077 ceph-mon[37153]: pgmap v457: 97 pgs: 97 active+clean; 25 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1004 KiB/s wr, 290 op/s 2024-07-27T06:49:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:01 smithi047 ceph-mon[36835]: pgmap v457: 97 pgs: 97 active+clean; 25 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 1004 KiB/s wr, 290 op/s 2024-07-27T06:49:01.768 INFO:tasks.cephfs.filesystem:scrub status for tag:790380e7-074c-4c4a-ac94-7e7a6aac95d1 - {'path': '/', 'tag': '790380e7-074c-4c4a-ac94-7e7a6aac95d1', 'options': 'recursive,force'} 2024-07-27T06:49:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.291+0000 7efd97fad640 1 -- 172.21.15.29:0/2080289901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd90102420 msgr2=0x7efd90102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.291+0000 7efd97fad640 1 --2- 172.21.15.29:0/2080289901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd90102420 0x7efd90102830 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7efd8c009900 tx=0x7efd8c02a410 comp rx=0 tx=0).stop 2024-07-27T06:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.291+0000 7efd97fad640 1 -- 172.21.15.29:0/2080289901 shutdown_connections 2024-07-27T06:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.291+0000 7efd97fad640 1 --2- 172.21.15.29:0/2080289901 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd901086a0 0x7efd9010eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.291+0000 7efd97fad640 1 --2- 172.21.15.29:0/2080289901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd90107d50 0x7efd90108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.291+0000 7efd97fad640 1 --2- 172.21.15.29:0/2080289901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd90102420 0x7efd90102830 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.291+0000 7efd97fad640 1 -- 172.21.15.29:0/2080289901 >> 172.21.15.29:0/2080289901 conn(0x7efd900fdde0 msgr2=0x7efd90100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.292+0000 7efd97fad640 1 -- 172.21.15.29:0/2080289901 shutdown_connections 2024-07-27T06:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.292+0000 7efd97fad640 1 -- 172.21.15.29:0/2080289901 wait complete. 2024-07-27T06:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.292+0000 7efd97fad640 1 Processor -- start 2024-07-27T06:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.292+0000 7efd97fad640 1 -- start start 2024-07-27T06:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.292+0000 7efd97fad640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd90102420 0x7efd9019acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd97fad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd90107d50 0x7efd9019b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd97fad640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd901086a0 0x7efd9019b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd97fad640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efd90112370 con 0x7efd90107d50 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd97fad640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7efd901121f0 con 0x7efd901086a0 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd97fad640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efd901124f0 con 0x7efd90102420 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd95548640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd90107d50 0x7efd9019b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd95548640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd90107d50 0x7efd9019b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37986/0 (socket says 172.21.15.29:37986) 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd95548640 1 -- 172.21.15.29:0/1320576867 learned_addr learned my addr 172.21.15.29:0/1320576867 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd95d49640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd90102420 0x7efd9019acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd9654a640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd901086a0 0x7efd9019b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd95548640 1 -- 172.21.15.29:0/1320576867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd90102420 msgr2=0x7efd9019acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd95548640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd90102420 0x7efd9019acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd95548640 1 -- 172.21.15.29:0/1320576867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd901086a0 msgr2=0x7efd9019b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd95548640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd901086a0 0x7efd9019b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd95548640 1 -- 172.21.15.29:0/1320576867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd901a5f80 con 0x7efd90107d50 2024-07-27T06:49:02.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd9654a640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd901086a0 0x7efd9019b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:49:02.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd95548640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd90107d50 0x7efd9019b200 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7efd8000e9c0 tx=0x7efd8000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:02.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd86ffd640 1 -- 172.21.15.29:0/1320576867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8000ccc0 con 0x7efd90107d50 2024-07-27T06:49:02.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd86ffd640 1 -- 172.21.15.29:0/1320576867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd800054d0 con 0x7efd90107d50 2024-07-27T06:49:02.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd97fad640 1 -- 172.21.15.29:0/1320576867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd901a6270 con 0x7efd90107d50 2024-07-27T06:49:02.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.293+0000 7efd86ffd640 1 -- 172.21.15.29:0/1320576867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd80010430 con 0x7efd90107d50 2024-07-27T06:49:02.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.294+0000 7efd97fad640 1 -- 172.21.15.29:0/1320576867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd901a6770 con 0x7efd90107d50 2024-07-27T06:49:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.296+0000 7efd9efa7740 1 Processor -- start 2024-07-27T06:49:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.296+0000 7efd9efa7740 1 -- start start 2024-07-27T06:49:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.296+0000 7efd86ffd640 1 -- 172.21.15.29:0/1320576867 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7efd80005020 con 0x7efd90107d50 2024-07-27T06:49:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9efa7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83eeea6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9efa7740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eeeabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9efa7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 0x55a83eeeb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9efa7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a83ee603b0 con 0x55a83eb481b0 2024-07-27T06:49:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9efa7740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55a83ee4f8b0 con 0x55a83eb1f2f0 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9efa7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a83ee538f0 con 0x55a83eb274d0 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9654a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83eeea6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9654a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83eeea6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38002/0 (socket says 172.21.15.29:38002) 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9654a640 1 -- 172.21.15.29:0/3871994820 learned_addr learned my addr 172.21.15.29:0/3871994820 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd95548640 1 --2- 172.21.15.29:0/3871994820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 0x55a83eeeb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd95d49640 1 --2- 172.21.15.29:0/3871994820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eeeabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9654a640 1 -- 172.21.15.29:0/3871994820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 msgr2=0x55a83eeeb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9654a640 1 --2- 172.21.15.29:0/3871994820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 0x55a83eeeb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9654a640 1 -- 172.21.15.29:0/3871994820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 msgr2=0x55a83eeeabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9654a640 1 --2- 172.21.15.29:0/3871994820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eeeabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9654a640 1 -- 172.21.15.29:0/3871994820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a83edf69a0 con 0x55a83eb481b0 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd95d49640 1 --2- 172.21.15.29:0/3871994820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eeeabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd86ffd640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efd7c0b3d40 0x7efd7c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.297+0000 7efd9654a640 1 --2- 172.21.15.29:0/3871994820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83eeea6b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7efd88009f10 tx=0x7efd8800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831ecde3aa45404d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.298+0000 7efd95d49640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efd7c0b3d40 0x7efd7c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.298+0000 7efd86ffd640 1 -- 172.21.15.29:0/1320576867 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7efd800e43e0 con 0x7efd90107d50 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.298+0000 7efd7bfff640 1 -- 172.21.15.29:0/3871994820 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd88016070 con 0x55a83eb481b0 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.298+0000 7efd7bfff640 1 -- 172.21.15.29:0/3871994820 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd88004d20 con 0x55a83eb481b0 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.298+0000 7efd7bfff640 1 -- 172.21.15.29:0/3871994820 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd88011830 con 0x55a83eb481b0 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.299+0000 7efd9efa7740 1 -- 172.21.15.29:0/3871994820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 msgr2=0x55a83eeea6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.299+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3871994820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83eeea6b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7efd88009f10 tx=0x7efd8800e6c0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.299+0000 7efd95d49640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efd7c0b3d40 0x7efd7c0b6250 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd8c000c00 tx=0x7efd8c031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.299+0000 7efd9efa7740 1 -- 172.21.15.29:0/3871994820 shutdown_connections 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.299+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3871994820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 0x55a83eeeb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.299+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3871994820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eeeabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.299+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3871994820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83eeea6b0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.299+0000 7efd9efa7740 1 -- 172.21.15.29:0/3871994820 >> 172.21.15.29:0/3871994820 conn(0x55a83eb3c300 msgr2=0x55a83eee85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.299+0000 7efd9efa7740 1 -- 172.21.15.29:0/3871994820 shutdown_connections 2024-07-27T06:49:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.299+0000 7efd9efa7740 1 -- 172.21.15.29:0/3871994820 wait complete. 2024-07-27T06:49:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.299+0000 7efd9efa7740 1 -- start start 2024-07-27T06:49:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.299+0000 7efd9efa7740 1 Processor -- start 2024-07-27T06:49:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9efa7740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eef61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9efa7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 0x55a83eef6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9efa7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83ef02150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9efa7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a83ed89ab0 con 0x55a83eb481b0 2024-07-27T06:49:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9efa7740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55a83edd7cf0 con 0x55a83eb1f2f0 2024-07-27T06:49:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9efa7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a83eef8e30 con 0x55a83eb274d0 2024-07-27T06:49:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9efa7740 10 client.0 _ll_register_callbacks cb 0x55a83ee493e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9654a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83ef02150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9654a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83ef02150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38008/0 (socket says 172.21.15.29:38008) 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9654a640 1 -- 172.21.15.29:0/3036358347 learned_addr learned my addr 172.21.15.29:0/3036358347 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd95548640 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 0x55a83eef6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd95d49640 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eef61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9654a640 1 -- 172.21.15.29:0/3036358347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 msgr2=0x55a83eef6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9654a640 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 0x55a83eef6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9654a640 1 -- 172.21.15.29:0/3036358347 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 msgr2=0x55a83eef61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9654a640 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eef61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.300+0000 7efd9654a640 1 -- 172.21.15.29:0/3036358347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a83ebb3790 con 0x55a83eb481b0 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.301+0000 7efd9654a640 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83ef02150 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efd88009ee0 tx=0x7efd88005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f87550cdda6abf0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.301+0000 7efd797fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.301+0000 7efd797fa640 1 -- 172.21.15.29:0/3036358347 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd88028060 con 0x55a83eb481b0 2024-07-27T06:49:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.301+0000 7efd797fa640 1 -- 172.21.15.29:0/3036358347 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7efd88003970 con 0x55a83eb481b0 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.301+0000 7efd9efa7740 1 -- 172.21.15.29:0/3036358347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd88004d20 con 0x55a83eb481b0 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.301+0000 7efd797fa640 1 -- 172.21.15.29:0/3036358347 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efd88016040 con 0x55a83eb481b0 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.301+0000 7efd9efa7740 10 client.16176 fetch_fsmap learned FSMap version 31 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.301+0000 7efd9efa7740 1 -- 172.21.15.29:0/3036358347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a83ec2c620 con 0x55a83eb481b0 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.301+0000 7efd797fa640 1 -- 172.21.15.29:0/3036358347 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd88004d20 con 0x55a83eb481b0 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.301+0000 7efd797fa640 1 -- 172.21.15.29:0/3036358347 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7efd8800f040 con 0x55a83eb481b0 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.302+0000 7efd9efa7740 10 client.16176 fetch_fsmap finished waiting for FSMap version 31 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.302+0000 7efd9efa7740 10 client.16176 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.302+0000 7efd9efa7740 20 client.16176 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.302+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55a83eb06d80 0x55a83ef08070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.302+0000 7efd9efa7740 4 client.16176 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.302+0000 7efd9efa7740 1 -- 172.21.15.29:0/3036358347 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a83ee62300 con 0x55a83eb06d80 2024-07-27T06:49:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.302+0000 7efd95d49640 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55a83eb06d80 0x55a83ef08070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.302+0000 7efd95d49640 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55a83eb06d80 0x55a83ef08070 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fdbf029417b1804 server_cookie=ddaabbe8f1860037 in_seq=0 out_seq=0 2024-07-27T06:49:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.302+0000 7efd797fa640 10 client.16176 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd797fa640 1 -- 172.21.15.29:0/3036358347 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a83ee62300 con 0x55a83eb06d80 2024-07-27T06:49:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd797fa640 10 client.16176 handle_command_reply: tid=0 2024-07-27T06:49:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd9efa7740 1 client.16176 shutdown 2024-07-27T06:49:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd9efa7740 1 -- 172.21.15.29:0/3036358347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 msgr2=0x55a83ef02150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83ef02150 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efd88009ee0 tx=0x7efd88005750 comp rx=0 tx=0).stop 2024-07-27T06:49:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd9efa7740 1 -- 172.21.15.29:0/3036358347 shutdown_connections 2024-07-27T06:49:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83ef02150 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55a83eb06d80 0x55a83ef08070 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 0x55a83eef6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3036358347 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eef61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd797fa640 0 client.16176 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd9efa7740 1 -- 172.21.15.29:0/3036358347 >> 172.21.15.29:0/3036358347 conn(0x55a83eb3c300 msgr2=0x55a83eee6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd9efa7740 1 -- 172.21.15.29:0/3036358347 shutdown_connections 2024-07-27T06:49:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd9efa7740 1 -- 172.21.15.29:0/3036358347 wait complete. 2024-07-27T06:49:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.303+0000 7efd9efa7740 20 client.16176 trim_cache size 0 max 16384 2024-07-27T06:49:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.319+0000 7efd9efa7740 1 Processor -- start 2024-07-27T06:49:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.319+0000 7efd9efa7740 1 -- start start 2024-07-27T06:49:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.319+0000 7efd9efa7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb1f2f0 0x55a83eee85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.319+0000 7efd9efa7740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb274d0 0x55a83eef0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.319+0000 7efd9efa7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb481b0 0x55a83eef5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.319+0000 7efd9efa7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efd901af240 con 0x55a83eb1f2f0 2024-07-27T06:49:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.319+0000 7efd9efa7740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7efd901b0e40 con 0x55a83eb274d0 2024-07-27T06:49:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.319+0000 7efd9efa7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efd901af910 con 0x55a83eb481b0 2024-07-27T06:49:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.319+0000 7efd95d49640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb1f2f0 0x55a83eee85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.320+0000 7efd95d49640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb1f2f0 0x55a83eee85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38018/0 (socket says 172.21.15.29:38018) 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.320+0000 7efd95d49640 1 -- 172.21.15.29:0/1611544042 learned_addr learned my addr 172.21.15.29:0/1611544042 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.320+0000 7efd95d49640 1 -- 172.21.15.29:0/1611544042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb481b0 msgr2=0x55a83eef5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.320+0000 7efd95d49640 1 --2- 172.21.15.29:0/1611544042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb481b0 0x55a83eef5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.320+0000 7efd95d49640 1 -- 172.21.15.29:0/1611544042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb274d0 msgr2=0x55a83eef0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.320+0000 7efd95d49640 1 --2- 172.21.15.29:0/1611544042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb274d0 0x55a83eef0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.320+0000 7efd95d49640 1 -- 172.21.15.29:0/1611544042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a83ef083e0 con 0x55a83eb1f2f0 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.320+0000 7efd95d49640 1 --2- 172.21.15.29:0/1611544042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb1f2f0 0x55a83eee85b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7efd8c032fd0 tx=0x7efd8c03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa70b2f4102a698e server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.320+0000 7efd7b7fe640 1 -- 172.21.15.29:0/1611544042 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8c035040 con 0x55a83eb1f2f0 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.320+0000 7efd7b7fe640 1 -- 172.21.15.29:0/1611544042 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd8c058070 con 0x55a83eb1f2f0 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.320+0000 7efd7b7fe640 1 -- 172.21.15.29:0/1611544042 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8c052610 con 0x55a83eb1f2f0 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.321+0000 7efd9efa7740 1 -- 172.21.15.29:0/1611544042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb1f2f0 msgr2=0x55a83eee85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.321+0000 7efd9efa7740 1 --2- 172.21.15.29:0/1611544042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb1f2f0 0x55a83eee85b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7efd8c032fd0 tx=0x7efd8c03e000 comp rx=0 tx=0).stop 2024-07-27T06:49:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.321+0000 7efd9efa7740 1 -- 172.21.15.29:0/1611544042 shutdown_connections 2024-07-27T06:49:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.321+0000 7efd9efa7740 1 --2- 172.21.15.29:0/1611544042 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb481b0 0x55a83eef5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.321+0000 7efd9efa7740 1 --2- 172.21.15.29:0/1611544042 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb274d0 0x55a83eef0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.321+0000 7efd9efa7740 1 --2- 172.21.15.29:0/1611544042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb1f2f0 0x55a83eee85b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.321+0000 7efd9efa7740 1 -- 172.21.15.29:0/1611544042 >> 172.21.15.29:0/1611544042 conn(0x55a83eb3c300 msgr2=0x55a83eee7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.321+0000 7efd9efa7740 1 -- 172.21.15.29:0/1611544042 shutdown_connections 2024-07-27T06:49:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.321+0000 7efd9efa7740 1 -- 172.21.15.29:0/1611544042 wait complete. 2024-07-27T06:49:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.321+0000 7efd9efa7740 1 -- start start 2024-07-27T06:49:02.324 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.321+0000 7efd9efa7740 1 Processor -- start 2024-07-27T06:49:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9efa7740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eefe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9efa7740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 0x55a83eefe840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9efa7740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83eefb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9efa7740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efd901b06b0 con 0x55a83eb481b0 2024-07-27T06:49:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9efa7740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7efd901affe0 con 0x55a83eb1f2f0 2024-07-27T06:49:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9efa7740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7efd901ad220 con 0x55a83eb274d0 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9efa7740 10 client.0 _ll_register_callbacks cb 0x7efd88009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9654a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83eefb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9654a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83eefb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38034/0 (socket says 172.21.15.29:38034) 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9654a640 1 -- 172.21.15.29:0/3812293759 learned_addr learned my addr 172.21.15.29:0/3812293759 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd95548640 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 0x55a83eefe840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd95d49640 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eefe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9654a640 1 -- 172.21.15.29:0/3812293759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 msgr2=0x55a83eefe840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9654a640 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 0x55a83eefe840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9654a640 1 -- 172.21.15.29:0/3812293759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 msgr2=0x55a83eefe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9654a640 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eefe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9654a640 1 -- 172.21.15.29:0/3812293759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a83eefb850 con 0x55a83eb481b0 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd9654a640 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83eefb0c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7efd88009740 tx=0x7efd8802dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583713cb2f0e3c69 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.322+0000 7efd797fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd797fa640 1 -- 172.21.15.29:0/3812293759 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd88028070 con 0x55a83eb481b0 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd797fa640 1 -- 172.21.15.29:0/3812293759 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7efd880056e0 con 0x55a83eb481b0 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd9efa7740 1 -- 172.21.15.29:0/3812293759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd8c058070 con 0x55a83eb481b0 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd797fa640 1 -- 172.21.15.29:0/3812293759 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efd8803d050 con 0x55a83eb481b0 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd9efa7740 10 client.16182 fetch_fsmap learned FSMap version 31 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd9efa7740 1 -- 172.21.15.29:0/3812293759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a83ef6f030 con 0x55a83eb481b0 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd797fa640 1 -- 172.21.15.29:0/3812293759 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd8c058070 con 0x55a83eb481b0 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd797fa640 1 -- 172.21.15.29:0/3812293759 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7efd88034070 con 0x55a83eb481b0 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd9efa7740 10 client.16182 fetch_fsmap finished waiting for FSMap version 31 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd9efa7740 10 client.16182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd9efa7740 20 client.16182 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55a83eadc1c0 0x55a83ef71880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd9efa7740 4 client.16182 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-27T06:49:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd9efa7740 1 -- 172.21.15.29:0/3812293759 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a83ee5dda0 con 0x55a83eadc1c0 2024-07-27T06:49:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.323+0000 7efd95d49640 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55a83eadc1c0 0x55a83ef71880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.324+0000 7efd95d49640 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55a83eadc1c0 0x55a83ef71880 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf5dbd0448d4c249 server_cookie=da9d5718b51068f6 in_seq=0 out_seq=0 2024-07-27T06:49:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.324+0000 7efd797fa640 10 client.16182 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:02.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.526+0000 7efd797fa640 1 -- 172.21.15.29:0/3812293759 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a83ee5dda0 con 0x55a83eadc1c0 2024-07-27T06:49:02.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.526+0000 7efd797fa640 10 client.16182 handle_command_reply: tid=0 2024-07-27T06:49:02.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.526+0000 7efd9efa7740 1 client.16182 shutdown 2024-07-27T06:49:02.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.526+0000 7efd9efa7740 1 -- 172.21.15.29:0/3812293759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 msgr2=0x55a83eefb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:02.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.526+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83eefb0c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7efd88009740 tx=0x7efd8802dd70 comp rx=0 tx=0).stop 2024-07-27T06:49:02.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd9efa7740 1 -- 172.21.15.29:0/3812293759 shutdown_connections 2024-07-27T06:49:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a83eb481b0 0x55a83eefb0c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55a83eadc1c0 0x55a83ef71880 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a83eb274d0 0x55a83eefe840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd9efa7740 1 --2- 172.21.15.29:0/3812293759 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a83eb1f2f0 0x55a83eefe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd797fa640 0 client.16182 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd9efa7740 1 -- 172.21.15.29:0/3812293759 >> 172.21.15.29:0/3812293759 conn(0x55a83eb3c300 msgr2=0x55a83eeeb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd9efa7740 1 -- 172.21.15.29:0/3812293759 shutdown_connections 2024-07-27T06:49:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd9efa7740 1 -- 172.21.15.29:0/3812293759 wait complete. 2024-07-27T06:49:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd9efa7740 20 client.16182 trim_cache size 0 max 16384 2024-07-27T06:49:02.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-07-27T06:49:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-07-27T06:49:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-07-27T06:49:02.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-07-27T06:49:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd97fad640 1 -- 172.21.15.29:0/1320576867 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efd7c0b3d40 msgr2=0x7efd7c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd97fad640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efd7c0b3d40 0x7efd7c0b6250 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd8c000c00 tx=0x7efd8c031400 comp rx=0 tx=0).stop 2024-07-27T06:49:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd97fad640 1 -- 172.21.15.29:0/1320576867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd90107d50 msgr2=0x7efd9019b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd97fad640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd90107d50 0x7efd9019b200 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7efd8000e9c0 tx=0x7efd8000ee80 comp rx=0 tx=0).stop 2024-07-27T06:49:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd97fad640 1 -- 172.21.15.29:0/1320576867 shutdown_connections 2024-07-27T06:49:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd97fad640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd901086a0 0x7efd9019b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd97fad640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd90107d50 0x7efd9019b200 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd97fad640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7efd7c0b3d40 0x7efd7c0b6250 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.527+0000 7efd97fad640 1 --2- 172.21.15.29:0/1320576867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd90102420 0x7efd9019acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.528+0000 7efd97fad640 1 -- 172.21.15.29:0/1320576867 >> 172.21.15.29:0/1320576867 conn(0x7efd900fdde0 msgr2=0x7efd90100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.528+0000 7efd97fad640 1 -- 172.21.15.29:0/1320576867 shutdown_connections 2024-07-27T06:49:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:02.528+0000 7efd97fad640 1 -- 172.21.15.29:0/1320576867 wait complete. 2024-07-27T06:49:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:03 smithi077 ceph-mon[37153]: pgmap v458: 97 pgs: 97 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 1007 KiB/s wr, 320 op/s 2024-07-27T06:49:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:03 smithi047 ceph-mon[36835]: pgmap v458: 97 pgs: 97 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 1007 KiB/s wr, 320 op/s 2024-07-27T06:49:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:03 smithi029 ceph-mon[32610]: pgmap v458: 97 pgs: 97 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 1007 KiB/s wr, 320 op/s 2024-07-27T06:49:05.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:49:05.273 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:05 smithi029 ceph-mon[32610]: pgmap v459: 97 pgs: 97 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 994 KiB/s wr, 277 op/s 2024-07-27T06:49:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:05 smithi077 ceph-mon[37153]: pgmap v459: 97 pgs: 97 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 994 KiB/s wr, 277 op/s 2024-07-27T06:49:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:05 smithi047 ceph-mon[36835]: pgmap v459: 97 pgs: 97 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 994 KiB/s wr, 277 op/s 2024-07-27T06:49:05.527 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:49:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:07 smithi077 ceph-mon[37153]: pgmap v460: 97 pgs: 97 active+clean; 26 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 823 KiB/s wr, 325 op/s 2024-07-27T06:49:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:07 smithi077 ceph-mon[37153]: scrub summary: idle 2024-07-27T06:49:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:07 smithi047 ceph-mon[36835]: pgmap v460: 97 pgs: 97 active+clean; 26 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 823 KiB/s wr, 325 op/s 2024-07-27T06:49:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:07 smithi047 ceph-mon[36835]: scrub summary: idle 2024-07-27T06:49:07.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:07 smithi029 ceph-mon[32610]: pgmap v460: 97 pgs: 97 active+clean; 26 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 823 KiB/s wr, 325 op/s 2024-07-27T06:49:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:07 smithi029 ceph-mon[32610]: scrub summary: idle 2024-07-27T06:49:08.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:08 smithi029 ceph-mon[32610]: scrub summary: idle 2024-07-27T06:49:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:08 smithi077 ceph-mon[37153]: scrub summary: idle 2024-07-27T06:49:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:08 smithi047 ceph-mon[36835]: scrub summary: idle 2024-07-27T06:49:09.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.059+0000 7f1c572e9640 1 -- 172.21.15.29:0/1388118713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c50108720 msgr2=0x7f1c5010ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:09.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.059+0000 7f1c572e9640 1 --2- 172.21.15.29:0/1388118713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c50108720 0x7f1c5010ef50 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1c48009ca0 tx=0x7f1c4802a8c0 comp rx=0 tx=0).stop 2024-07-27T06:49:09.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.059+0000 7f1c572e9640 1 -- 172.21.15.29:0/1388118713 shutdown_connections 2024-07-27T06:49:09.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.059+0000 7f1c572e9640 1 --2- 172.21.15.29:0/1388118713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c50108720 0x7f1c5010ef50 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:09.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.059+0000 7f1c572e9640 1 --2- 172.21.15.29:0/1388118713 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c50107dd0 0x7f1c501081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:09.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.059+0000 7f1c572e9640 1 --2- 172.21.15.29:0/1388118713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c50102490 0x7f1c501028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:09.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.059+0000 7f1c572e9640 1 -- 172.21.15.29:0/1388118713 >> 172.21.15.29:0/1388118713 conn(0x7f1c50079160 msgr2=0x7f1c50079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:09.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.059+0000 7f1c572e9640 1 -- 172.21.15.29:0/1388118713 shutdown_connections 2024-07-27T06:49:09.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.059+0000 7f1c572e9640 1 -- 172.21.15.29:0/1388118713 wait complete. 2024-07-27T06:49:09.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.060+0000 7f1c572e9640 1 Processor -- start 2024-07-27T06:49:09.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.060+0000 7f1c572e9640 1 -- start start 2024-07-27T06:49:09.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.060+0000 7f1c572e9640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c50102490 0x7f1c50199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:09.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.060+0000 7f1c572e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c50107dd0 0x7f1c5019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:09.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.060+0000 7f1c572e9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c50108720 0x7f1c5019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:09.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.060+0000 7f1c572e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1c501134b0 con 0x7f1c50107dd0 2024-07-27T06:49:09.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.060+0000 7f1c572e9640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f1c50113330 con 0x7f1c50102490 2024-07-27T06:49:09.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.060+0000 7f1c572e9640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1c50113630 con 0x7f1c50108720 2024-07-27T06:49:09.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.060+0000 7f1c54884640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c50107dd0 0x7f1c5019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:09.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c54884640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c50107dd0 0x7f1c5019a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39834/0 (socket says 172.21.15.29:39834) 2024-07-27T06:49:09.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c54884640 1 -- 172.21.15.29:0/1724097257 learned_addr learned my addr 172.21.15.29:0/1724097257 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:09.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c55886640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c50108720 0x7f1c5019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:09.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c55085640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c50102490 0x7f1c50199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:09.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c54884640 1 -- 172.21.15.29:0/1724097257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c50108720 msgr2=0x7f1c5019f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:09.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c54884640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c50108720 0x7f1c5019f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:09.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c54884640 1 -- 172.21.15.29:0/1724097257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c50102490 msgr2=0x7f1c50199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c54884640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c50102490 0x7f1c50199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c54884640 1 -- 172.21.15.29:0/1724097257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5019fdb0 con 0x7f1c50107dd0 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c55085640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c50102490 0x7f1c50199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c55886640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c50108720 0x7f1c5019f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c54884640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c50107dd0 0x7f1c5019a3e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1c44005fd0 tx=0x7f1c4400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c427fc640 1 -- 172.21.15.29:0/1724097257 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c44017070 con 0x7f1c50107dd0 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c427fc640 1 -- 172.21.15.29:0/1724097257 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c44009c30 con 0x7f1c50107dd0 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c427fc640 1 -- 172.21.15.29:0/1724097257 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c44003d30 con 0x7f1c50107dd0 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c572e9640 1 -- 172.21.15.29:0/1724097257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c501a00a0 con 0x7f1c50107dd0 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.061+0000 7f1c572e9640 1 -- 172.21.15.29:0/1724097257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c501a6800 con 0x7f1c50107dd0 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.062+0000 7f1c572e9640 1 -- 172.21.15.29:0/1724097257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c501028a0 con 0x7f1c50107dd0 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.064+0000 7f1c427fc640 1 -- 172.21.15.29:0/1724097257 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f1c44003ed0 con 0x7f1c50107dd0 2024-07-27T06:49:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.065+0000 7f1c427fc640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1c3c0b3d40 0x7f1c3c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:09.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.065+0000 7f1c427fc640 1 -- 172.21.15.29:0/1724097257 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f1c440e4570 con 0x7f1c50107dd0 2024-07-27T06:49:09.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.065+0000 7f1c55085640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1c3c0b3d40 0x7f1c3c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:09.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.066+0000 7f1c55085640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1c3c0b3d40 0x7f1c3c0b6250 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c0058c0 tx=0x7f1c4c005850 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:09.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.068+0000 7f1c427fc640 1 -- 172.21.15.29:0/1724097257 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1c44020030 con 0x7f1c50107dd0 2024-07-27T06:49:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:09 smithi029 ceph-mon[32610]: pgmap v461: 97 pgs: 97 active+clean; 22 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 336 KiB/s wr, 311 op/s 2024-07-27T06:49:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.289+0000 7f1c572e9640 1 -- 172.21.15.29:0/1724097257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c5019af70 con 0x7f1c50107dd0 2024-07-27T06:49:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.290+0000 7f1c427fc640 1 -- 172.21.15.29:0/1724097257 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11964 (secure 0 0 0) 0x7f1c5019af70 con 0x7f1c50107dd0 2024-07-27T06:49:09.291 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:49:09.291 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"btime":"2024-07-27T06:48:56:994705+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:48:56.751760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24670,24658,24646,24617,24613]},"id":1}]} 2024-07-27T06:49:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-07-27T06:49:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.292+0000 7f1c572e9640 1 -- 172.21.15.29:0/1724097257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1c3c0b3d40 msgr2=0x7f1c3c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.292+0000 7f1c572e9640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1c3c0b3d40 0x7f1c3c0b6250 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c0058c0 tx=0x7f1c4c005850 comp rx=0 tx=0).stop 2024-07-27T06:49:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.292+0000 7f1c572e9640 1 -- 172.21.15.29:0/1724097257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c50107dd0 msgr2=0x7f1c5019a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.292+0000 7f1c572e9640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c50107dd0 0x7f1c5019a3e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1c44005fd0 tx=0x7f1c4400dde0 comp rx=0 tx=0).stop 2024-07-27T06:49:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.292+0000 7f1c572e9640 1 -- 172.21.15.29:0/1724097257 shutdown_connections 2024-07-27T06:49:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.292+0000 7f1c572e9640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f1c3c0b3d40 0x7f1c3c0b6250 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.292+0000 7f1c572e9640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1c50108720 0x7f1c5019f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.292+0000 7f1c572e9640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c50107dd0 0x7f1c5019a3e0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.292+0000 7f1c572e9640 1 --2- 172.21.15.29:0/1724097257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c50102490 0x7f1c50199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.292+0000 7f1c572e9640 1 -- 172.21.15.29:0/1724097257 >> 172.21.15.29:0/1724097257 conn(0x7f1c50079160 msgr2=0x7f1c50103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.293+0000 7f1c572e9640 1 -- 172.21.15.29:0/1724097257 shutdown_connections 2024-07-27T06:49:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:09.293+0000 7f1c572e9640 1 -- 172.21.15.29:0/1724097257 wait complete. 2024-07-27T06:49:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:09 smithi077 ceph-mon[37153]: pgmap v461: 97 pgs: 97 active+clean; 22 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 336 KiB/s wr, 311 op/s 2024-07-27T06:49:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:09 smithi047 ceph-mon[36835]: pgmap v461: 97 pgs: 97 active+clean; 22 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 336 KiB/s wr, 311 op/s 2024-07-27T06:49:10.040 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24670 quiesce db --set-id e30c826e --release --await' 2024-07-27T06:49:10.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- timeout 105 ceph tell mds.24670 quiesce db --set-id e30c826e --release --await 2024-07-27T06:49:10.354 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:49:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:10 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1724097257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:49:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:10 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1724097257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:49:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:10 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1724097257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:49:11.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:11 smithi047 ceph-mon[36835]: pgmap v462: 97 pgs: 97 active+clean; 22 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 174 KiB/s wr, 274 op/s 2024-07-27T06:49:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:11 smithi077 ceph-mon[37153]: pgmap v462: 97 pgs: 97 active+clean; 22 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 174 KiB/s wr, 274 op/s 2024-07-27T06:49:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:11 smithi029 ceph-mon[32610]: pgmap v462: 97 pgs: 97 active+clean; 22 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 174 KiB/s wr, 274 op/s 2024-07-27T06:49:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.190+0000 7f821b5dd640 1 -- 172.21.15.29:0/1818825266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8214105d20 msgr2=0x7f8214106130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.190+0000 7f821b5dd640 1 --2- 172.21.15.29:0/1818825266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8214105d20 0x7f8214106130 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8204009900 tx=0x7f820402a3f0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.190+0000 7f821b5dd640 1 -- 172.21.15.29:0/1818825266 shutdown_connections 2024-07-27T06:49:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.190+0000 7f821b5dd640 1 --2- 172.21.15.29:0/1818825266 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8214106670 0x7f82141100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.191+0000 7f821b5dd640 1 --2- 172.21.15.29:0/1818825266 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8214105d20 0x7f8214106130 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.191+0000 7f821b5dd640 1 --2- 172.21.15.29:0/1818825266 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8214101ad0 0x7f8214101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.191+0000 7f821b5dd640 1 -- 172.21.15.29:0/1818825266 >> 172.21.15.29:0/1818825266 conn(0x7f82140799c0 msgr2=0x7f8214079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.191+0000 7f821b5dd640 1 -- 172.21.15.29:0/1818825266 shutdown_connections 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.191+0000 7f821b5dd640 1 -- 172.21.15.29:0/1818825266 wait complete. 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.191+0000 7f821b5dd640 1 Processor -- start 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.191+0000 7f821b5dd640 1 -- start start 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f821b5dd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8214101ad0 0x7f821419c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f821b5dd640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8214105d20 0x7f821419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f821b5dd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8214106670 0x7f82141a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f821b5dd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f82141134e0 con 0x7f8214101ad0 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f821b5dd640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f8214113360 con 0x7f8214105d20 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f821b5dd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8214113660 con 0x7f8214106670 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f8219379640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8214101ad0 0x7f821419c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f8219379640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8214101ad0 0x7f821419c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39848/0 (socket says 172.21.15.29:39848) 2024-07-27T06:49:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f8219379640 1 -- 172.21.15.29:0/664705955 learned_addr learned my addr 172.21.15.29:0/664705955 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f8218b78640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8214105d20 0x7f821419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f8219b7a640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8214106670 0x7f82141a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f8219379640 1 -- 172.21.15.29:0/664705955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8214106670 msgr2=0x7f82141a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f8219379640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8214106670 0x7f82141a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f8219379640 1 -- 172.21.15.29:0/664705955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8214105d20 msgr2=0x7f821419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f8219379640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8214105d20 0x7f821419c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f8219379640 1 -- 172.21.15.29:0/664705955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82141a2000 con 0x7f8214101ad0 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f8218b78640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8214105d20 0x7f821419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f8219379640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8214101ad0 0x7f821419c150 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f821000c970 tx=0x7f821000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.192+0000 7f820a7fc640 1 -- 172.21.15.29:0/664705955 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8210007c10 con 0x7f8214101ad0 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.193+0000 7f820a7fc640 1 -- 172.21.15.29:0/664705955 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8210007db0 con 0x7f8214101ad0 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.193+0000 7f821b5dd640 1 -- 172.21.15.29:0/664705955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82141a22f0 con 0x7f8214101ad0 2024-07-27T06:49:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.193+0000 7f820a7fc640 1 -- 172.21.15.29:0/664705955 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821000f870 con 0x7f8214101ad0 2024-07-27T06:49:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.193+0000 7f821b5dd640 1 -- 172.21.15.29:0/664705955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82141a8b10 con 0x7f8214101ad0 2024-07-27T06:49:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.195+0000 7f8221b61740 1 Processor -- start 2024-07-27T06:49:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.195+0000 7f820a7fc640 1 -- 172.21.15.29:0/664705955 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f821000fa10 con 0x7f8214101ad0 2024-07-27T06:49:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.195+0000 7f8221b61740 1 -- start start 2024-07-27T06:49:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8221b61740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7b16520 0x55e0a7ebc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8221b61740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7aed660 0x55e0a7ebc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8221b61740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 0x55e0a7ebcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8221b61740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e0a7e31e70 con 0x55e0a7af5840 2024-07-27T06:49:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8221b61740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55e0a7e21370 con 0x55e0a7aed660 2024-07-27T06:49:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8221b61740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e0a7e253b0 con 0x55e0a7b16520 2024-07-27T06:49:12.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8218b78640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 0x55e0a7ebcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8218b78640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 0x55e0a7ebcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39862/0 (socket says 172.21.15.29:39862) 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8218b78640 1 -- 172.21.15.29:0/232297461 learned_addr learned my addr 172.21.15.29:0/232297461 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8219b7a640 1 --2- 172.21.15.29:0/232297461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7b16520 0x55e0a7ebc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8219379640 1 --2- 172.21.15.29:0/232297461 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7aed660 0x55e0a7ebc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8219b7a640 1 -- 172.21.15.29:0/232297461 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7aed660 msgr2=0x55e0a7ebc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8219b7a640 1 --2- 172.21.15.29:0/232297461 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7aed660 0x55e0a7ebc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8219b7a640 1 -- 172.21.15.29:0/232297461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 msgr2=0x55e0a7ebcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8219b7a640 1 --2- 172.21.15.29:0/232297461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 0x55e0a7ebcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8219b7a640 1 -- 172.21.15.29:0/232297461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0a7dee040 con 0x55e0a7b16520 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f820a7fc640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f82000b3d70 0x7f82000b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8218b78640 1 --2- 172.21.15.29:0/232297461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 0x55e0a7ebcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.196+0000 7f8219379640 1 --2- 172.21.15.29:0/232297461 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7aed660 0x55e0a7ebc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:12.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.197+0000 7f8219379640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f82000b3d70 0x7f82000b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.197+0000 7f8219b7a640 1 --2- 172.21.15.29:0/232297461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7b16520 0x55e0a7ebc070 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f820c009f30 tx=0x7f820c010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c4ddd4686be6f28 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.197+0000 7f820a7fc640 1 -- 172.21.15.29:0/664705955 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f82100e4390 con 0x7f8214101ad0 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.197+0000 7f81ff7fe640 1 -- 172.21.15.29:0/232297461 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c016070 con 0x55e0a7b16520 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.197+0000 7f81ff7fe640 1 -- 172.21.15.29:0/232297461 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f820c004d20 con 0x55e0a7b16520 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.197+0000 7f81ff7fe640 1 -- 172.21.15.29:0/232297461 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c0127d0 con 0x55e0a7b16520 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.198+0000 7f8221b61740 1 -- 172.21.15.29:0/232297461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7b16520 msgr2=0x55e0a7ebc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.198+0000 7f8219379640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f82000b3d70 0x7f82000b6280 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f82100e1ba0 tx=0x7f8210000bc0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.198+0000 7f8221b61740 1 --2- 172.21.15.29:0/232297461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7b16520 0x55e0a7ebc070 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f820c009f30 tx=0x7f820c010800 comp rx=0 tx=0).stop 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.198+0000 7f8221b61740 1 -- 172.21.15.29:0/232297461 shutdown_connections 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.198+0000 7f8221b61740 1 --2- 172.21.15.29:0/232297461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 0x55e0a7ebcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.198+0000 7f8221b61740 1 --2- 172.21.15.29:0/232297461 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7aed660 0x55e0a7ebc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.198+0000 7f8221b61740 1 --2- 172.21.15.29:0/232297461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7b16520 0x55e0a7ebc070 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.198+0000 7f8221b61740 1 -- 172.21.15.29:0/232297461 >> 172.21.15.29:0/232297461 conn(0x55e0a7b0a910 msgr2=0x55e0a7eb9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.198+0000 7f8221b61740 1 -- 172.21.15.29:0/232297461 shutdown_connections 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.198+0000 7f8221b61740 1 -- 172.21.15.29:0/232297461 wait complete. 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.198+0000 7f8221b61740 1 -- start start 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.198+0000 7f8221b61740 1 Processor -- start 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8221b61740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7aed660 0x55e0a7ec7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8221b61740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7af5840 0x55e0a7ec8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8221b61740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7b16520 0x55e0a7ed3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8221b61740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e0a7c84bb0 con 0x55e0a7aed660 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8221b61740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55e0a7dc3ac0 con 0x55e0a7b16520 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8221b61740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e0a7eca850 con 0x55e0a7af5840 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8221b61740 10 client.0 _ll_register_callbacks cb 0x55e0a7dcc3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8219b7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7aed660 0x55e0a7ec7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8219b7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7aed660 0x55e0a7ec7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39872/0 (socket says 172.21.15.29:39872) 2024-07-27T06:49:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8219b7a640 1 -- 172.21.15.29:0/2760630295 learned_addr learned my addr 172.21.15.29:0/2760630295 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8218b78640 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7af5840 0x55e0a7ec8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8219b7a640 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7b16520 0x55e0a7ed3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8219b7a640 1 -- 172.21.15.29:0/2760630295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7af5840 msgr2=0x55e0a7ec8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8219b7a640 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7af5840 0x55e0a7ec8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8219b7a640 1 -- 172.21.15.29:0/2760630295 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7b16520 msgr2=0x55e0a7ed3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8219b7a640 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7b16520 0x55e0a7ed3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.199+0000 7f8219b7a640 1 -- 172.21.15.29:0/2760630295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0a7b81b00 con 0x55e0a7aed660 2024-07-27T06:49:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.200+0000 7f8219b7a640 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7aed660 0x55e0a7ec7bc0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f820c005ab0 tx=0x7f820c012470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d2d977443afcb5a server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.200+0000 7f81fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.200+0000 7f81fcff9640 1 -- 172.21.15.29:0/2760630295 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c016070 con 0x55e0a7aed660 2024-07-27T06:49:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.200+0000 7f81fcff9640 1 -- 172.21.15.29:0/2760630295 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f820c005590 con 0x55e0a7aed660 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.200+0000 7f8221b61740 1 -- 172.21.15.29:0/2760630295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f820c004d20 con 0x55e0a7aed660 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.200+0000 7f81fcff9640 1 -- 172.21.15.29:0/2760630295 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f820c012550 con 0x55e0a7aed660 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.200+0000 7f8221b61740 10 client.16200 fetch_fsmap learned FSMap version 31 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.200+0000 7f8221b61740 1 -- 172.21.15.29:0/2760630295 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e0a7bf8960 con 0x55e0a7aed660 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.200+0000 7f81fcff9640 1 -- 172.21.15.29:0/2760630295 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f820c004d20 con 0x55e0a7aed660 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.200+0000 7f81fcff9640 1 -- 172.21.15.29:0/2760630295 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f820c00f040 con 0x55e0a7aed660 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.200+0000 7f8221b61740 10 client.16200 fetch_fsmap finished waiting for FSMap version 31 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.200+0000 7f8221b61740 10 client.16200 resolve_mds: validated gid 24670 aka daemon mds.i 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.201+0000 7f8221b61740 20 client.16200 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.201+0000 7f8221b61740 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e0a7ad4ea0 0x55e0a7ed99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.201+0000 7f8221b61740 4 client.16200 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.201+0000 7f8221b61740 1 -- 172.21.15.29:0/2760630295 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0a7e33dc0 con 0x55e0a7ad4ea0 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.201+0000 7f8218b78640 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e0a7ad4ea0 0x55e0a7ed99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.201+0000 7f81fcff9640 10 client.16200 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.201+0000 7f8218b78640 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e0a7ad4ea0 0x55e0a7ed99a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84bb9ae55e8048e3 server_cookie=3188b5ecc69d939e in_seq=0 out_seq=0 2024-07-27T06:49:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f81fcff9640 1 -- 172.21.15.29:0/2760630295 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e0a7e33dc0 con 0x55e0a7ad4ea0 2024-07-27T06:49:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f81fcff9640 10 client.16200 handle_command_reply: tid=0 2024-07-27T06:49:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f8221b61740 1 client.16200 shutdown 2024-07-27T06:49:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f8221b61740 1 -- 172.21.15.29:0/2760630295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7aed660 msgr2=0x55e0a7ec7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f8221b61740 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7aed660 0x55e0a7ec7bc0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f820c005ab0 tx=0x7f820c012470 comp rx=0 tx=0).stop 2024-07-27T06:49:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f8221b61740 1 -- 172.21.15.29:0/2760630295 shutdown_connections 2024-07-27T06:49:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f8221b61740 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7b16520 0x55e0a7ed3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f8221b61740 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e0a7ad4ea0 0x55e0a7ed99a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f8221b61740 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7af5840 0x55e0a7ec8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f8221b61740 1 --2- 172.21.15.29:0/2760630295 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7aed660 0x55e0a7ec7bc0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f8221b61740 1 -- 172.21.15.29:0/2760630295 >> 172.21.15.29:0/2760630295 conn(0x55e0a7b0a910 msgr2=0x55e0a7ebf250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f81fcff9640 0 client.16200 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f8221b61740 1 -- 172.21.15.29:0/2760630295 shutdown_connections 2024-07-27T06:49:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f8221b61740 1 -- 172.21.15.29:0/2760630295 wait complete. 2024-07-27T06:49:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.202+0000 7f8221b61740 20 client.16200 trim_cache size 0 max 16384 2024-07-27T06:49:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8221b61740 1 Processor -- start 2024-07-27T06:49:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8221b61740 1 -- start start 2024-07-27T06:49:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8221b61740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7aed660 0x55e0a7ebf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8221b61740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7af5840 0x55e0a7ed3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8221b61740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7b16520 0x55e0a7ed3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8221b61740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f82141b1370 con 0x55e0a7b16520 2024-07-27T06:49:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8221b61740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f82141b2f70 con 0x55e0a7af5840 2024-07-27T06:49:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8221b61740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f82141b1a40 con 0x55e0a7aed660 2024-07-27T06:49:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8219b7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7b16520 0x55e0a7ed3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8219b7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7b16520 0x55e0a7ed3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39874/0 (socket says 172.21.15.29:39874) 2024-07-27T06:49:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8219b7a640 1 -- 172.21.15.29:0/1260333101 learned_addr learned my addr 172.21.15.29:0/1260333101 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8219b7a640 1 --2- 172.21.15.29:0/1260333101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7aed660 0x55e0a7ebf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8218b78640 1 --2- 172.21.15.29:0/1260333101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7af5840 0x55e0a7ed3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8219b7a640 1 -- 172.21.15.29:0/1260333101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7aed660 msgr2=0x55e0a7ebf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8219b7a640 1 --2- 172.21.15.29:0/1260333101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7aed660 0x55e0a7ebf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8219b7a640 1 -- 172.21.15.29:0/1260333101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7af5840 msgr2=0x55e0a7ed3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8219b7a640 1 --2- 172.21.15.29:0/1260333101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7af5840 0x55e0a7ed3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8219b7a640 1 -- 172.21.15.29:0/1260333101 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0a7ec5ec0 con 0x55e0a7b16520 2024-07-27T06:49:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.220+0000 7f8218b78640 1 --2- 172.21.15.29:0/1260333101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7af5840 0x55e0a7ed3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.221+0000 7f8219b7a640 1 --2- 172.21.15.29:0/1260333101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7b16520 0x55e0a7ed3e70 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f820c005c50 tx=0x7f820c01d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55848e3a0d1a2915 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.221+0000 7f81feffd640 1 -- 172.21.15.29:0/1260333101 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c016070 con 0x55e0a7b16520 2024-07-27T06:49:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.221+0000 7f81feffd640 1 -- 172.21.15.29:0/1260333101 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f820c01f030 con 0x55e0a7b16520 2024-07-27T06:49:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.221+0000 7f81feffd640 1 -- 172.21.15.29:0/1260333101 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c012ba0 con 0x55e0a7b16520 2024-07-27T06:49:12.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.222+0000 7f8221b61740 1 -- 172.21.15.29:0/1260333101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7b16520 msgr2=0x55e0a7ed3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:12.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.222+0000 7f8221b61740 1 --2- 172.21.15.29:0/1260333101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7b16520 0x55e0a7ed3e70 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f820c005c50 tx=0x7f820c01d040 comp rx=0 tx=0).stop 2024-07-27T06:49:12.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.222+0000 7f8221b61740 1 -- 172.21.15.29:0/1260333101 shutdown_connections 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.222+0000 7f8221b61740 1 --2- 172.21.15.29:0/1260333101 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7b16520 0x55e0a7ed3e70 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.222+0000 7f8221b61740 1 --2- 172.21.15.29:0/1260333101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7af5840 0x55e0a7ed3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.222+0000 7f8221b61740 1 --2- 172.21.15.29:0/1260333101 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7aed660 0x55e0a7ebf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.222+0000 7f8221b61740 1 -- 172.21.15.29:0/1260333101 >> 172.21.15.29:0/1260333101 conn(0x55e0a7b0a910 msgr2=0x55e0a7ebf130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.222+0000 7f8221b61740 1 -- 172.21.15.29:0/1260333101 shutdown_connections 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.222+0000 7f8221b61740 1 -- 172.21.15.29:0/1260333101 wait complete. 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.222+0000 7f8221b61740 1 -- start start 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.222+0000 7f8221b61740 1 Processor -- start 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8221b61740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7aed660 0x55e0a7ecb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8221b61740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 0x55e0a7ecbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8221b61740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7b16520 0x55e0a7ecc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8221b61740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f82141b27e0 con 0x55e0a7af5840 2024-07-27T06:49:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8221b61740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f82141b2110 con 0x55e0a7b16520 2024-07-27T06:49:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8221b61740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f82141af350 con 0x55e0a7aed660 2024-07-27T06:49:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8221b61740 10 client.0 _ll_register_callbacks cb 0x7f8204009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8218b78640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 0x55e0a7ecbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8218b78640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 0x55e0a7ecbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39876/0 (socket says 172.21.15.29:39876) 2024-07-27T06:49:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8218b78640 1 -- 172.21.15.29:0/1116885891 learned_addr learned my addr 172.21.15.29:0/1116885891 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8219b7a640 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7aed660 0x55e0a7ecb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8219b7a640 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7b16520 0x55e0a7ecc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8218b78640 1 -- 172.21.15.29:0/1116885891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7aed660 msgr2=0x55e0a7ecb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8218b78640 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7aed660 0x55e0a7ecb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8218b78640 1 -- 172.21.15.29:0/1116885891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7b16520 msgr2=0x55e0a7ecc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8218b78640 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7b16520 0x55e0a7ecc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8218b78640 1 -- 172.21.15.29:0/1116885891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0a7f40f90 con 0x55e0a7af5840 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8219b7a640 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7aed660 0x55e0a7ecb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f8218b78640 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 0x55e0a7ecbf00 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8204009230 tx=0x7f8204004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0fe282a1808d310 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.223+0000 7f81fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f81fcff9640 1 -- 172.21.15.29:0/1116885891 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8204035040 con 0x55e0a7af5840 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f81fcff9640 1 -- 172.21.15.29:0/1116885891 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f8204005040 con 0x55e0a7af5840 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f8221b61740 1 -- 172.21.15.29:0/1116885891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f820c01f030 con 0x55e0a7af5840 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f81fcff9640 1 -- 172.21.15.29:0/1116885891 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8204053050 con 0x55e0a7af5840 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f8221b61740 10 client.16206 fetch_fsmap learned FSMap version 31 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f8221b61740 1 -- 172.21.15.29:0/1116885891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e0a7f422b0 con 0x55e0a7af5840 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f81fcff9640 1 -- 172.21.15.29:0/1116885891 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f820c01f030 con 0x55e0a7af5840 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f81fcff9640 1 -- 172.21.15.29:0/1116885891 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f82040035f0 con 0x55e0a7af5840 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f8221b61740 10 client.16206 fetch_fsmap finished waiting for FSMap version 31 2024-07-27T06:49:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f8221b61740 10 client.16206 resolve_mds: validated gid 24670 aka daemon mds.i 2024-07-27T06:49:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f8221b61740 20 client.16206 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f8221b61740 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e0a7aaa250 0x55e0a7f44bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f8221b61740 4 client.16206 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e30c826e", "release": true, "await": true}] 2024-07-27T06:49:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.224+0000 7f8221b61740 1 -- 172.21.15.29:0/1116885891 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e30c826e", "release": true, "await": true}) -- 0x55e0a7e2f860 con 0x55e0a7aaa250 2024-07-27T06:49:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.225+0000 7f8219b7a640 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e0a7aaa250 0x55e0a7f44bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.225+0000 7f8219b7a640 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e0a7aaa250 0x55e0a7f44bb0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4608c15eb97c8f40 server_cookie=49adbb86332f18e8 in_seq=0 out_seq=0 2024-07-27T06:49:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.225+0000 7f81fcff9640 10 client.16206 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:12.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f81fcff9640 1 -- 172.21.15.29:0/1116885891 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55e0a7e2f860 con 0x55e0a7aaa250 2024-07-27T06:49:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f81fcff9640 10 client.16206 handle_command_reply: tid=0 2024-07-27T06:49:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f8221b61740 1 client.16206 shutdown 2024-07-27T06:49:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f8221b61740 1 -- 172.21.15.29:0/1116885891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 msgr2=0x55e0a7ecbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f8221b61740 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 0x55e0a7ecbf00 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8204009230 tx=0x7f8204004040 comp rx=0 tx=0).stop 2024-07-27T06:49:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f8221b61740 1 -- 172.21.15.29:0/1116885891 shutdown_connections 2024-07-27T06:49:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f8221b61740 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e0a7b16520 0x55e0a7ecc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f8221b61740 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e0a7af5840 0x55e0a7ecbf00 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f8221b61740 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e0a7aaa250 0x55e0a7f44bb0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f8221b61740 1 --2- 172.21.15.29:0/1116885891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0a7aed660 0x55e0a7ecb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f8221b61740 1 -- 172.21.15.29:0/1116885891 >> 172.21.15.29:0/1116885891 conn(0x55e0a7b0a910 msgr2=0x55e0a7ebd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:12.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f81fcff9640 0 client.16206 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:12.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.312+0000 7f8221b61740 1 -- 172.21.15.29:0/1116885891 shutdown_connections 2024-07-27T06:49:12.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f8221b61740 1 -- 172.21.15.29:0/1116885891 wait complete. 2024-07-27T06:49:12.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f8221b61740 20 client.16206 trim_cache size 0 max 16384 2024-07-27T06:49:12.315 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:49:12.315 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 31, 2024-07-27T06:49:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 24670, 2024-07-27T06:49:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: "set_version": 8, 2024-07-27T06:49:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: "sets": { 2024-07-27T06:49:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: "e30c826e": { 2024-07-27T06:49:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: "version": 8, 2024-07-27T06:49:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: "age_ref": 0.0, 2024-07-27T06:49:12.316 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: "timeout": 90.0, 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: "expiration": 142.6, 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: "members": { 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: "file:/": { 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: "excluded": false, 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: "state": { 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "RELEASED", 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:49:12.317 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:49:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:49:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:49:12.318 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-07-27T06:49:12.318 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:49:12.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f821b5dd640 1 -- 172.21.15.29:0/664705955 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f82000b3d70 msgr2=0x7f82000b6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f821b5dd640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f82000b3d70 0x7f82000b6280 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f82100e1ba0 tx=0x7f8210000bc0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f821b5dd640 1 -- 172.21.15.29:0/664705955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8214101ad0 msgr2=0x7f821419c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f821b5dd640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8214101ad0 0x7f821419c150 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f821000c970 tx=0x7f821000ce30 comp rx=0 tx=0).stop 2024-07-27T06:49:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f821b5dd640 1 -- 172.21.15.29:0/664705955 shutdown_connections 2024-07-27T06:49:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f821b5dd640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f82000b3d70 0x7f82000b6280 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f821b5dd640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8214106670 0x7f82141a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f821b5dd640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8214105d20 0x7f821419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f821b5dd640 1 --2- 172.21.15.29:0/664705955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8214101ad0 0x7f821419c150 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f821b5dd640 1 -- 172.21.15.29:0/664705955 >> 172.21.15.29:0/664705955 conn(0x7f82140799c0 msgr2=0x7f8214107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f821b5dd640 1 -- 172.21.15.29:0/664705955 shutdown_connections 2024-07-27T06:49:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:12.313+0000 7f821b5dd640 1 -- 172.21.15.29:0/664705955 wait complete. 2024-07-27T06:49:13.241 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'e30c826e', total seconds elapsed: 30.5 2024-07-27T06:49:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:13 smithi077 ceph-mon[37153]: pgmap v463: 97 pgs: 97 active+clean; 23 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 319 KiB/s wr, 326 op/s 2024-07-27T06:49:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:13 smithi047 ceph-mon[36835]: pgmap v463: 97 pgs: 97 active+clean; 23 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 319 KiB/s wr, 326 op/s 2024-07-27T06:49:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:13 smithi029 ceph-mon[32610]: pgmap v463: 97 pgs: 97 active+clean; 23 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 319 KiB/s wr, 326 op/s 2024-07-27T06:49:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:15 smithi077 ceph-mon[37153]: pgmap v464: 97 pgs: 97 active+clean; 23 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 316 KiB/s wr, 294 op/s 2024-07-27T06:49:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:15 smithi047 ceph-mon[36835]: pgmap v464: 97 pgs: 97 active+clean; 23 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 316 KiB/s wr, 294 op/s 2024-07-27T06:49:15.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:15 smithi029 ceph-mon[32610]: pgmap v464: 97 pgs: 97 active+clean; 23 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 316 KiB/s wr, 294 op/s 2024-07-27T06:49:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.632+0000 7f1371716640 1 -- 172.21.15.29:0/1231551300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c101aa0 msgr2=0x7f136c101eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.632+0000 7f1371716640 1 --2- 172.21.15.29:0/1231551300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c101aa0 0x7f136c101eb0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f135c009ca0 tx=0x7f135c02a8c0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.632+0000 7f1371716640 1 -- 172.21.15.29:0/1231551300 shutdown_connections 2024-07-27T06:49:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.632+0000 7f1371716640 1 --2- 172.21.15.29:0/1231551300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f136c106560 0x7f136c10ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.632+0000 7f1371716640 1 --2- 172.21.15.29:0/1231551300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f136c105c10 0x7f136c106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.632+0000 7f1371716640 1 --2- 172.21.15.29:0/1231551300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c101aa0 0x7f136c101eb0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.632+0000 7f1371716640 1 -- 172.21.15.29:0/1231551300 >> 172.21.15.29:0/1231551300 conn(0x7f136c0798d0 msgr2=0x7f136c079ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.632+0000 7f1371716640 1 -- 172.21.15.29:0/1231551300 shutdown_connections 2024-07-27T06:49:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.632+0000 7f1371716640 1 -- 172.21.15.29:0/1231551300 wait complete. 2024-07-27T06:49:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.633+0000 7f1371716640 1 Processor -- start 2024-07-27T06:49:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.633+0000 7f1371716640 1 -- start start 2024-07-27T06:49:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.633+0000 7f1371716640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c101aa0 0x7f136c199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.633+0000 7f1371716640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f136c105c10 0x7f136c19a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.633+0000 7f1371716640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f136c106560 0x7f136c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f1371716640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f136c113450 con 0x7f136c101aa0 2024-07-27T06:49:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f1371716640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1132d0 con 0x7f136c106560 2024-07-27T06:49:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f1371716640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1135d0 con 0x7f136c105c10 2024-07-27T06:49:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f136affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c101aa0 0x7f136c199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f136affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c101aa0 0x7f136c199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39896/0 (socket says 172.21.15.29:39896) 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f136affd640 1 -- 172.21.15.29:0/124952233 learned_addr learned my addr 172.21.15.29:0/124952233 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f136a7fc640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f136c105c10 0x7f136c19a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f136affd640 1 -- 172.21.15.29:0/124952233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f136c105c10 msgr2=0x7f136c19a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f136affd640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f136c105c10 0x7f136c19a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f136affd640 1 -- 172.21.15.29:0/124952233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f136c106560 msgr2=0x7f136c19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f136affd640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f136c106560 0x7f136c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f136affd640 1 -- 172.21.15.29:0/124952233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c19fe20 con 0x7f136c101aa0 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f136affd640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c101aa0 0x7f136c199ef0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f135c006010 tx=0x7f135c0319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f1363fff640 1 -- 172.21.15.29:0/124952233 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c003bf0 con 0x7f136c101aa0 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f1363fff640 1 -- 172.21.15.29:0/124952233 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f135c003d90 con 0x7f136c101aa0 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f1371716640 1 -- 172.21.15.29:0/124952233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136c1a00b0 con 0x7f136c101aa0 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f1363fff640 1 -- 172.21.15.29:0/124952233 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c02fc20 con 0x7f136c101aa0 2024-07-27T06:49:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.634+0000 7f1371716640 1 -- 172.21.15.29:0/124952233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136c1a68a0 con 0x7f136c101aa0 2024-07-27T06:49:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.637+0000 7f1378631740 1 Processor -- start 2024-07-27T06:49:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.637+0000 7f1378631740 1 -- start start 2024-07-27T06:49:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.637+0000 7f1363fff640 1 -- 172.21.15.29:0/124952233 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f135c0046a0 con 0x7f136c101aa0 2024-07-27T06:49:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f1378631740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58241b0 0x55e1d5bc6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f1378631740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bc7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f1378631740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 0x55e1d5bc7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f1378631740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e1d5b3c3b0 con 0x55e1d58034d0 2024-07-27T06:49:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f1378631740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55e1d5b2b8b0 con 0x55e1d58241b0 2024-07-27T06:49:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f1378631740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e1d5b2f8f0 con 0x55e1d57fb2f0 2024-07-27T06:49:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f136a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 0x55e1d5bc7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f136a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 0x55e1d5bc7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39910/0 (socket says 172.21.15.29:39910) 2024-07-27T06:49:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f136a7fc640 1 -- 172.21.15.29:0/812162069 learned_addr learned my addr 172.21.15.29:0/812162069 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f136affd640 1 --2- 172.21.15.29:0/812162069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bc7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f136b7fe640 1 --2- 172.21.15.29:0/812162069 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58241b0 0x55e1d5bc6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f136a7fc640 1 -- 172.21.15.29:0/812162069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57fb2f0 msgr2=0x55e1d5bc7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f136a7fc640 1 --2- 172.21.15.29:0/812162069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bc7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f136a7fc640 1 -- 172.21.15.29:0/812162069 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58241b0 msgr2=0x55e1d5bc6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f136a7fc640 1 --2- 172.21.15.29:0/812162069 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58241b0 0x55e1d5bc6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.638+0000 7f136a7fc640 1 -- 172.21.15.29:0/812162069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1d5ad29a0 con 0x55e1d58034d0 2024-07-27T06:49:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.639+0000 7f1363fff640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f13580b3d40 0x7f13580b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.639+0000 7f136a7fc640 1 --2- 172.21.15.29:0/812162069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 0x55e1d5bc7760 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f135400bc10 tx=0x7f135400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e269b45989e253a server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.639+0000 7f136b7fe640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f13580b3d40 0x7f13580b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.639+0000 7f1363fff640 1 -- 172.21.15.29:0/124952233 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f135c105530 con 0x7f136c101aa0 2024-07-27T06:49:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.639+0000 7f13617fa640 1 -- 172.21.15.29:0/812162069 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1354015070 con 0x55e1d58034d0 2024-07-27T06:49:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.639+0000 7f13617fa640 1 -- 172.21.15.29:0/812162069 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f135401a910 con 0x55e1d58034d0 2024-07-27T06:49:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.639+0000 7f13617fa640 1 -- 172.21.15.29:0/812162069 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1354011960 con 0x55e1d58034d0 2024-07-27T06:49:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.640+0000 7f1378631740 1 -- 172.21.15.29:0/812162069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 msgr2=0x55e1d5bc7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.640+0000 7f136b7fe640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f13580b3d40 0x7f13580b6250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f136400b590 tx=0x7f136400b520 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.640+0000 7f1378631740 1 --2- 172.21.15.29:0/812162069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 0x55e1d5bc7760 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f135400bc10 tx=0x7f135400f730 comp rx=0 tx=0).stop 2024-07-27T06:49:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.640+0000 7f1378631740 1 -- 172.21.15.29:0/812162069 shutdown_connections 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.640+0000 7f1378631740 1 --2- 172.21.15.29:0/812162069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 0x55e1d5bc7760 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.640+0000 7f1378631740 1 --2- 172.21.15.29:0/812162069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bc7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.640+0000 7f1378631740 1 --2- 172.21.15.29:0/812162069 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58241b0 0x55e1d5bc6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.640+0000 7f1378631740 1 -- 172.21.15.29:0/812162069 >> 172.21.15.29:0/812162069 conn(0x55e1d5818300 msgr2=0x55e1d5bc4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.640+0000 7f1378631740 1 -- 172.21.15.29:0/812162069 shutdown_connections 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.640+0000 7f1378631740 1 -- 172.21.15.29:0/812162069 wait complete. 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.640+0000 7f1378631740 1 -- start start 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.640+0000 7f1378631740 1 Processor -- start 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f1378631740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f1378631740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d58034d0 0x55e1d5bd2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f1378631740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58241b0 0x55e1d5bde780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f1378631740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55e1d5a65ab0 con 0x55e1d57fb2f0 2024-07-27T06:49:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f1378631740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55e1d5ab3cf0 con 0x55e1d58241b0 2024-07-27T06:49:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f1378631740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e1d5bd5460 con 0x55e1d58034d0 2024-07-27T06:49:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f1378631740 10 client.0 _ll_register_callbacks cb 0x55e1d5b259f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f136affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f136affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39916/0 (socket says 172.21.15.29:39916) 2024-07-27T06:49:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f136affd640 1 -- 172.21.15.29:0/3244323139 learned_addr learned my addr 172.21.15.29:0/3244323139 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f136a7fc640 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d58034d0 0x55e1d5bd2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f136b7fe640 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58241b0 0x55e1d5bde780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f136affd640 1 -- 172.21.15.29:0/3244323139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d58034d0 msgr2=0x55e1d5bd2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f136affd640 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d58034d0 0x55e1d5bd2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f136affd640 1 -- 172.21.15.29:0/3244323139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58241b0 msgr2=0x55e1d5bde780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f136affd640 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58241b0 0x55e1d5bde780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f136affd640 1 -- 172.21.15.29:0/3244323139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1d588f790 con 0x55e1d57fb2f0 2024-07-27T06:49:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f136a7fc640 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d58034d0 0x55e1d5bd2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:49:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.641+0000 7f136b7fe640 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58241b0 0x55e1d5bde780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:49:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f136affd640 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd27e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f135c102e00 tx=0x7f135c0c8770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845ee628533752e2 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f13527fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f13527fc640 1 -- 172.21.15.29:0/3244323139 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c0cc030 con 0x55e1d57fb2f0 2024-07-27T06:49:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f13527fc640 1 -- 172.21.15.29:0/3244323139 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f135c0d3df0 con 0x55e1d57fb2f0 2024-07-27T06:49:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f1378631740 1 -- 172.21.15.29:0/3244323139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f135401a910 con 0x55e1d57fb2f0 2024-07-27T06:49:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f13527fc640 1 -- 172.21.15.29:0/3244323139 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f135c0cb9d0 con 0x55e1d57fb2f0 2024-07-27T06:49:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f1378631740 10 client.16218 fetch_fsmap learned FSMap version 31 2024-07-27T06:49:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f1378631740 1 -- 172.21.15.29:0/3244323139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e1d5908620 con 0x55e1d57fb2f0 2024-07-27T06:49:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f13527fc640 1 -- 172.21.15.29:0/3244323139 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f135401a910 con 0x55e1d57fb2f0 2024-07-27T06:49:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f13527fc640 1 -- 172.21.15.29:0/3244323139 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f135c0cc4c0 con 0x55e1d57fb2f0 2024-07-27T06:49:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f1378631740 10 client.16218 fetch_fsmap finished waiting for FSMap version 31 2024-07-27T06:49:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f1378631740 10 client.16218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:49:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f1378631740 20 client.16218 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f1378631740 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e1d57e2d80 0x55e1d5be47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f1378631740 4 client.16218 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.642+0000 7f1378631740 1 -- 172.21.15.29:0/3244323139 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1d5b3e300 con 0x55e1d57e2d80 2024-07-27T06:49:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.643+0000 7f136a7fc640 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e1d57e2d80 0x55e1d5be47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.643+0000 7f136a7fc640 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e1d57e2d80 0x55e1d5be47c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=522542b9cfebd2a0 server_cookie=34275ccd37deb4b3 in_seq=0 out_seq=0 2024-07-27T06:49:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.643+0000 7f13527fc640 10 client.16218 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f13527fc640 1 -- 172.21.15.29:0/3244323139 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e1d5b3e300 con 0x55e1d57e2d80 2024-07-27T06:49:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f13527fc640 10 client.16218 handle_command_reply: tid=0 2024-07-27T06:49:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f1378631740 1 client.16218 shutdown 2024-07-27T06:49:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f1378631740 1 -- 172.21.15.29:0/3244323139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 msgr2=0x55e1d5bd27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f1378631740 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd27e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f135c102e00 tx=0x7f135c0c8770 comp rx=0 tx=0).stop 2024-07-27T06:49:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f1378631740 1 -- 172.21.15.29:0/3244323139 shutdown_connections 2024-07-27T06:49:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f1378631740 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58241b0 0x55e1d5bde780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f1378631740 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e1d57e2d80 0x55e1d5be47c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f1378631740 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d58034d0 0x55e1d5bd2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f1378631740 1 --2- 172.21.15.29:0/3244323139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd27e0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f13527fc640 0 client.16218 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f1378631740 1 -- 172.21.15.29:0/3244323139 >> 172.21.15.29:0/3244323139 conn(0x55e1d5818300 msgr2=0x55e1d5bc7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f1378631740 1 -- 172.21.15.29:0/3244323139 shutdown_connections 2024-07-27T06:49:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f1378631740 1 -- 172.21.15.29:0/3244323139 wait complete. 2024-07-27T06:49:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.644+0000 7f1378631740 20 client.16218 trim_cache size 0 max 16384 2024-07-27T06:49:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f1378631740 1 Processor -- start 2024-07-27T06:49:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f1378631740 1 -- start start 2024-07-27T06:49:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f1378631740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57e2d80 0x55e1d5bc87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f1378631740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f1378631740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58034d0 0x55e1d5bd1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f1378631740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1af030 con 0x55e1d57fb2f0 2024-07-27T06:49:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f1378631740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1b0c30 con 0x55e1d58034d0 2024-07-27T06:49:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f1378631740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1af700 con 0x55e1d57e2d80 2024-07-27T06:49:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f136a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f136a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39926/0 (socket says 172.21.15.29:39926) 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f136a7fc640 1 -- 172.21.15.29:0/1511944441 learned_addr learned my addr 172.21.15.29:0/1511944441 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f136affd640 1 --2- 172.21.15.29:0/1511944441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57e2d80 0x55e1d5bc87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.660+0000 7f136b7fe640 1 --2- 172.21.15.29:0/1511944441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58034d0 0x55e1d5bd1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.661+0000 7f136a7fc640 1 -- 172.21.15.29:0/1511944441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57e2d80 msgr2=0x55e1d5bc87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.661+0000 7f136a7fc640 1 --2- 172.21.15.29:0/1511944441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57e2d80 0x55e1d5bc87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.661+0000 7f136a7fc640 1 -- 172.21.15.29:0/1511944441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58034d0 msgr2=0x55e1d5bd1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.661+0000 7f136a7fc640 1 --2- 172.21.15.29:0/1511944441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58034d0 0x55e1d5bd1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.661+0000 7f136a7fc640 1 -- 172.21.15.29:0/1511944441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1d5bd2d10 con 0x55e1d57fb2f0 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.661+0000 7f136a7fc640 1 --2- 172.21.15.29:0/1511944441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd44b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1354011520 tx=0x7f1354011620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b77e9140cf062ac server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.661+0000 7f1360ff9640 1 -- 172.21.15.29:0/1511944441 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1354015040 con 0x55e1d57fb2f0 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.661+0000 7f1360ff9640 1 -- 172.21.15.29:0/1511944441 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f135402b070 con 0x55e1d57fb2f0 2024-07-27T06:49:15.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.661+0000 7f1360ff9640 1 -- 172.21.15.29:0/1511944441 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1354011690 con 0x55e1d57fb2f0 2024-07-27T06:49:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.662+0000 7f1378631740 1 -- 172.21.15.29:0/1511944441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 msgr2=0x55e1d5bd44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.662+0000 7f1378631740 1 --2- 172.21.15.29:0/1511944441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd44b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1354011520 tx=0x7f1354011620 comp rx=0 tx=0).stop 2024-07-27T06:49:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.662+0000 7f1378631740 1 -- 172.21.15.29:0/1511944441 shutdown_connections 2024-07-27T06:49:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.662+0000 7f1378631740 1 --2- 172.21.15.29:0/1511944441 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d58034d0 0x55e1d5bd1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.662+0000 7f1378631740 1 --2- 172.21.15.29:0/1511944441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd44b0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.662+0000 7f1378631740 1 --2- 172.21.15.29:0/1511944441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57e2d80 0x55e1d5bc87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.662+0000 7f1378631740 1 -- 172.21.15.29:0/1511944441 >> 172.21.15.29:0/1511944441 conn(0x55e1d5818300 msgr2=0x55e1d5bcbda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.662+0000 7f1378631740 1 -- 172.21.15.29:0/1511944441 shutdown_connections 2024-07-27T06:49:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.662+0000 7f1378631740 1 -- 172.21.15.29:0/1511944441 wait complete. 2024-07-27T06:49:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.662+0000 7f1378631740 1 -- start start 2024-07-27T06:49:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.662+0000 7f1378631740 1 Processor -- start 2024-07-27T06:49:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f1378631740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d57e2d80 0x55e1d5bdaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f1378631740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f1378631740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 0x55e1d5bd7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f1378631740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1b04a0 con 0x55e1d58034d0 2024-07-27T06:49:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f1378631740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1afdd0 con 0x55e1d57e2d80 2024-07-27T06:49:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f1378631740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1ad010 con 0x55e1d57fb2f0 2024-07-27T06:49:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f1378631740 10 client.0 _ll_register_callbacks cb 0x7f135c009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f136b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 0x55e1d5bd7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f136b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 0x55e1d5bd7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39928/0 (socket says 172.21.15.29:39928) 2024-07-27T06:49:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f136b7fe640 1 -- 172.21.15.29:0/4185794543 learned_addr learned my addr 172.21.15.29:0/4185794543 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f136affd640 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d57e2d80 0x55e1d5bdaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f136a7fc640 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f136b7fe640 1 -- 172.21.15.29:0/4185794543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57fb2f0 msgr2=0x55e1d5bd7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f136b7fe640 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f136b7fe640 1 -- 172.21.15.29:0/4185794543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d57e2d80 msgr2=0x55e1d5bdaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f136b7fe640 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d57e2d80 0x55e1d5bdaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f136b7fe640 1 -- 172.21.15.29:0/4185794543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1d5c4af50 con 0x55e1d58034d0 2024-07-27T06:49:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f136b7fe640 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 0x55e1d5bd7e20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f13640116c0 tx=0x7f1364005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0dc99ce02a08f9 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.663+0000 7f13527fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f13527fc640 1 -- 172.21.15.29:0/4185794543 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364028420 con 0x55e1d58034d0 2024-07-27T06:49:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f13527fc640 1 -- 172.21.15.29:0/4185794543 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f136401e420 con 0x55e1d58034d0 2024-07-27T06:49:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f1378631740 1 -- 172.21.15.29:0/4185794543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f135402b070 con 0x55e1d58034d0 2024-07-27T06:49:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f13527fc640 1 -- 172.21.15.29:0/4185794543 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f136401e620 con 0x55e1d58034d0 2024-07-27T06:49:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f1378631740 10 client.16224 fetch_fsmap learned FSMap version 31 2024-07-27T06:49:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f1378631740 1 -- 172.21.15.29:0/4185794543 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e1d5c4c170 con 0x55e1d58034d0 2024-07-27T06:49:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f13527fc640 1 -- 172.21.15.29:0/4185794543 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f135402b070 con 0x55e1d58034d0 2024-07-27T06:49:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f13527fc640 1 -- 172.21.15.29:0/4185794543 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f1364028b20 con 0x55e1d58034d0 2024-07-27T06:49:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f1378631740 10 client.16224 fetch_fsmap finished waiting for FSMap version 31 2024-07-27T06:49:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f1378631740 10 client.16224 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:49:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f1378631740 20 client.16224 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f1378631740 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e1d58241b0 0x55e1d5c4eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f1378631740 4 client.16224 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-27T06:49:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f1378631740 1 -- 172.21.15.29:0/4185794543 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e1d5b39da0 con 0x55e1d58241b0 2024-07-27T06:49:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.664+0000 7f136affd640 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e1d58241b0 0x55e1d5c4eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.665+0000 7f136affd640 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e1d58241b0 0x55e1d5c4eaa0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34df9915b3f9bf4 server_cookie=16be4558f07cefb5 in_seq=0 out_seq=0 2024-07-27T06:49:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:15.665+0000 7f13527fc640 10 client.16224 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:16.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f13527fc640 1 -- 172.21.15.29:0/4185794543 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e1d5b39da0 con 0x55e1d58241b0 2024-07-27T06:49:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f13527fc640 10 client.16224 handle_command_reply: tid=0 2024-07-27T06:49:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f1378631740 1 client.16224 shutdown 2024-07-27T06:49:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f1378631740 1 -- 172.21.15.29:0/4185794543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 msgr2=0x55e1d5bd7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f1378631740 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 0x55e1d5bd7e20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f13640116c0 tx=0x7f1364005cc0 comp rx=0 tx=0).stop 2024-07-27T06:49:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f1378631740 1 -- 172.21.15.29:0/4185794543 shutdown_connections 2024-07-27T06:49:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f1378631740 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55e1d58034d0 0x55e1d5bd7e20 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f1378631740 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55e1d58241b0 0x55e1d5c4eaa0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f1378631740 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e1d57fb2f0 0x55e1d5bd7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f1378631740 1 --2- 172.21.15.29:0/4185794543 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55e1d57e2d80 0x55e1d5bdaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f13527fc640 0 client.16224 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f1378631740 1 -- 172.21.15.29:0/4185794543 >> 172.21.15.29:0/4185794543 conn(0x55e1d5818300 msgr2=0x55e1d5bd0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f1378631740 1 -- 172.21.15.29:0/4185794543 shutdown_connections 2024-07-27T06:49:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f1378631740 1 -- 172.21.15.29:0/4185794543 wait complete. 2024-07-27T06:49:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.020+0000 7f1378631740 20 client.16224 trim_cache size 0 max 16384 2024-07-27T06:49:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-07-27T06:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-07-27T06:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-07-27T06:49:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-07-27T06:49:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.021+0000 7f1371716640 1 -- 172.21.15.29:0/124952233 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f13580b3d40 msgr2=0x7f13580b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.021+0000 7f1371716640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f13580b3d40 0x7f13580b6250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f136400b590 tx=0x7f136400b520 comp rx=0 tx=0).stop 2024-07-27T06:49:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.021+0000 7f1371716640 1 -- 172.21.15.29:0/124952233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c101aa0 msgr2=0x7f136c199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.021+0000 7f1371716640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c101aa0 0x7f136c199ef0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f135c006010 tx=0x7f135c0319f0 comp rx=0 tx=0).stop 2024-07-27T06:49:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.021+0000 7f1371716640 1 -- 172.21.15.29:0/124952233 shutdown_connections 2024-07-27T06:49:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.021+0000 7f1371716640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f136c106560 0x7f136c19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.021+0000 7f1371716640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f13580b3d40 0x7f13580b6250 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.021+0000 7f1371716640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f136c105c10 0x7f136c19a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.021+0000 7f1371716640 1 --2- 172.21.15.29:0/124952233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f136c101aa0 0x7f136c199ef0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.021+0000 7f1371716640 1 -- 172.21.15.29:0/124952233 >> 172.21.15.29:0/124952233 conn(0x7f136c0798d0 msgr2=0x7f136c107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.021+0000 7f1371716640 1 -- 172.21.15.29:0/124952233 shutdown_connections 2024-07-27T06:49:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:16.021+0000 7f1371716640 1 -- 172.21.15.29:0/124952233 wait complete. 2024-07-27T06:49:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:17 smithi077 ceph-mon[37153]: pgmap v465: 97 pgs: 97 active+clean; 27 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 670 KiB/s wr, 273 op/s 2024-07-27T06:49:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:17 smithi047 ceph-mon[36835]: pgmap v465: 97 pgs: 97 active+clean; 27 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 670 KiB/s wr, 273 op/s 2024-07-27T06:49:17.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:17 smithi029 ceph-mon[32610]: pgmap v465: 97 pgs: 97 active+clean; 27 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 670 KiB/s wr, 273 op/s 2024-07-27T06:49:18.978 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 7% (322/4329) Updating files: 8% (347/4329) Updating files: 9% (390/4329) Updating files: 10% (433/4329) Updating files: 11% (477/4329) Updating files: 11% (491/4329) Updating files: 12% (520/4329) Updating files: 13% (563/4329) Updating files: 14% (607/4329) Updating files: 15% (650/4329) Updating files: 16% (693/4329) Updating files: 17% (736/4329) Updating files: 17% (740/4329) Updating files: 18% (780/4329) Updating files: 19% (823/4329) Updating files: 20% (866/4329) Updating files: 21% (910/4329) Updating files: 22% (953/4329) Updating files: 23% (996/4329) Updating files: 24% (1039/4329) Updating files: 25% (1083/4329) Updating files: 26% (1126/4329) Updating files: 27% (1169/4329) Updating files: 28% (1213/4329) Updating files: 29% (1256/4329) Updating files: 29% (1289/4329) Updating files: 30% (1299/4329) Updating files: 31% (1342/4329) Updating files: 32% (1386/4329) Updating files: 32% (1423/4329) Updating files: 33% (1429/4329) Updating files: 34% (1472/4329) Updating files: 35% (1516/4329) Updating files: 35% (1525/4329) Updating files: 36% (1559/4329) Updating files: 37% (1602/4329) Updating files: 38% (1646/4329) Updating files: 39% (1689/4329) Updating files: 40% (1732/4329) Updating files: 41% (1775/4329) Updating files: 42% (1819/4329) Updating files: 43% (1862/4329) Updating files: 44% (1905/4329) Updating files: 45% (1949/4329) Updating files: 46% (1992/4329) Updating files: 47% (2035/4329) Updating files: 48% (2078/4329) Updating files: 49% (2122/4329) Updating files: 50% (2165/4329) Updating files: 50% (2193/4329) Updating files: 51% (2208/4329) Updating files: 52% (2252/4329) Updating files: 53% (2295/4329) Updating files: 54% (2338/4329) Updating files: 55% (2381/4329) Updating files: 56% (2425/4329) Updating files: 57% (2468/4329) Updating files: 57% (2483/4329) Updating files: 58% (2511/4329) Updating files: 59% (2555/4329) Updating files: 60% (2598/4329) Updating files: 61% (2641/4329) Updating files: 62% (2684/4329) Updating files: 63% (2728/4329) Updating files: 64% (2771/4329) Updating files: 64% (2791/4329) Updating files: 65% (2814/4329) Updating files: 66% (2858/4329) Updating files: 67% (2901/4329) Updating files: 68% (2944/4329) Updating files: 69% (2988/4329) Updating files: 69% (3022/4329) Updating files: 70% (3031/4329) Updating files: 71% (3074/4329) Updating files: 72% (3117/4329) Updating files: 73% (3161/4329) Updating files: 74% (3204/4329) Updating files: 75% (3247/4329) Updating files: 76% (3291/4329) Updating files: 76% (3324/4329) Updating files: 77% (3334/4329) Updating files: 78% (3377/4329) Updating files: 79% (3420/4329) Updating files: 80% (3464/4329) Updating files: 81% (3507/4329) Updating files: 82% (3550/4329) Updating files: 83% (3594/4329) Updating files: 84% (3637/4329) Updating files: 85% (3680/4329) Updating files: 86% (3723/4329) Updating files: 87% (3767/4329) Updating files: 88% (3810/4329) Updating files: 89% (3853/4329) Updating files: 90% (3897/4329) Updating files: 91% (3940/4329) Updating files: 92% (3983/4329) Updating files: 92% (3991/4329) Updating files: 93% (4026/4329) Updating files: 94% (4070/4329) Updating files: 95% (4113/4329) Updating files: 96% (4156/4329) Updating files: 97% (4200/4329) Updating files: 98% (4243/4329) Updating files: 99% (4286/4329) Updating files: 100% (4329/4329) Updating files: 100% (4329/4329), done. 2024-07-27T06:49:18.991 INFO:tasks.workunit.client.0.smithi029.stderr:+ cd xfstests-dev 2024-07-27T06:49:18.991 INFO:tasks.workunit.client.0.smithi029.stderr:+ git checkout 0e5c12dfd008efc2848c98108c9237487e91ef35 2024-07-27T06:49:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:19 smithi077 ceph-mon[37153]: pgmap v466: 97 pgs: 97 active+clean; 31 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 965 KiB/s wr, 305 op/s 2024-07-27T06:49:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:19 smithi047 ceph-mon[36835]: pgmap v466: 97 pgs: 97 active+clean; 31 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 965 KiB/s wr, 305 op/s 2024-07-27T06:49:19.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:19 smithi029 ceph-mon[32610]: pgmap v466: 97 pgs: 97 active+clean; 31 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 965 KiB/s wr, 305 op/s 2024-07-27T06:49:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:21 smithi077 ceph-mon[37153]: pgmap v467: 97 pgs: 97 active+clean; 31 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 891 KiB/s wr, 220 op/s 2024-07-27T06:49:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:21 smithi047 ceph-mon[36835]: pgmap v467: 97 pgs: 97 active+clean; 31 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 891 KiB/s wr, 220 op/s 2024-07-27T06:49:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:21 smithi029 ceph-mon[32610]: pgmap v467: 97 pgs: 97 active+clean; 31 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 891 KiB/s wr, 220 op/s 2024-07-27T06:49:21.856 INFO:tasks.workunit.client.0.smithi029.stderr:Note: switching to '0e5c12dfd008efc2848c98108c9237487e91ef35'. 2024-07-27T06:49:21.856 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:49:21.857 INFO:tasks.workunit.client.0.smithi029.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-27T06:49:21.857 INFO:tasks.workunit.client.0.smithi029.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-27T06:49:21.857 INFO:tasks.workunit.client.0.smithi029.stderr:state without impacting any branches by switching back to a branch. 2024-07-27T06:49:21.857 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:49:21.857 INFO:tasks.workunit.client.0.smithi029.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-27T06:49:21.857 INFO:tasks.workunit.client.0.smithi029.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-27T06:49:21.857 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:49:21.857 INFO:tasks.workunit.client.0.smithi029.stderr: git switch -c 2024-07-27T06:49:21.857 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:49:21.857 INFO:tasks.workunit.client.0.smithi029.stderr:Or undo this operation with: 2024-07-27T06:49:21.857 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:49:21.857 INFO:tasks.workunit.client.0.smithi029.stderr: git switch - 2024-07-27T06:49:21.858 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:49:21.858 INFO:tasks.workunit.client.0.smithi029.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-27T06:49:21.858 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-07-27T06:49:21.858 INFO:tasks.workunit.client.0.smithi029.stderr:HEAD is now at 0e5c12df xfs/522: use reflink instead of crc as test feature 2024-07-27T06:49:21.862 INFO:tasks.workunit.client.0.smithi029.stderr:+ make -j4 2024-07-27T06:49:21.871 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize -cfi 2024-07-27T06:49:22.330 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize: putting auxiliary files in '.'. 2024-07-27T06:49:22.331 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize: copying file './config.guess' 2024-07-27T06:49:22.335 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize: copying file './config.sub' 2024-07-27T06:49:22.340 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize: copying file './install-sh' 2024-07-27T06:49:22.346 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize: copying file './ltmain.sh' 2024-07-27T06:49:22.382 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-07-27T06:49:22.382 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-07-27T06:49:22.397 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-07-27T06:49:22.409 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-07-27T06:49:22.421 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-07-27T06:49:22.433 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-07-27T06:49:22.439 INFO:tasks.workunit.client.0.smithi029.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-07-27T06:49:22.441 INFO:tasks.workunit.client.0.smithi029.stdout:cp include/install-sh . 2024-07-27T06:49:22.447 INFO:tasks.workunit.client.0.smithi029.stdout:aclocal -I m4 2024-07-27T06:49:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:23 smithi077 ceph-mon[37153]: pgmap v468: 97 pgs: 97 active+clean; 24 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 365 op/s 2024-07-27T06:49:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:23 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:49:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:23 smithi047 ceph-mon[36835]: pgmap v468: 97 pgs: 97 active+clean; 24 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 365 op/s 2024-07-27T06:49:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:23 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:49:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:23 smithi029 ceph-mon[32610]: pgmap v468: 97 pgs: 97 active+clean; 24 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 365 op/s 2024-07-27T06:49:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:23 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:49:25.118 INFO:tasks.workunit.client.0.smithi029.stdout:autoheader 2024-07-27T06:49:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:25 smithi077 ceph-mon[37153]: pgmap v469: 97 pgs: 97 active+clean; 28 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 327 op/s 2024-07-27T06:49:25.441 INFO:tasks.workunit.client.0.smithi029.stdout:autoconf 2024-07-27T06:49:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:25 smithi047 ceph-mon[36835]: pgmap v469: 97 pgs: 97 active+clean; 28 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 327 op/s 2024-07-27T06:49:25.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:25 smithi029 ceph-mon[32610]: pgmap v469: 97 pgs: 97 active+clean; 28 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 327 op/s 2024-07-27T06:49:25.626 INFO:tasks.workunit.client.0.smithi029.stdout:./configure \ 2024-07-27T06:49:25.626 INFO:tasks.workunit.client.0.smithi029.stdout: --libexecdir=/usr/lib \ 2024-07-27T06:49:25.626 INFO:tasks.workunit.client.0.smithi029.stdout: --exec_prefix=/var/lib 2024-07-27T06:49:25.772 INFO:tasks.workunit.client.0.smithi029.stdout:checking build system type... x86_64-pc-linux-gnu 2024-07-27T06:49:25.773 INFO:tasks.workunit.client.0.smithi029.stdout:checking host system type... x86_64-pc-linux-gnu 2024-07-27T06:49:25.774 INFO:tasks.workunit.client.0.smithi029.stdout:checking how to print strings... printf 2024-07-27T06:49:25.774 INFO:tasks.workunit.client.0.smithi029.stdout:checking for gcc... gcc 2024-07-27T06:49:25.863 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether the C compiler works... yes 2024-07-27T06:49:25.863 INFO:tasks.workunit.client.0.smithi029.stdout:checking for C compiler default output file name... a.out 2024-07-27T06:49:25.906 INFO:tasks.workunit.client.0.smithi029.stdout:checking for suffix of executables... 2024-07-27T06:49:25.947 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether we are cross compiling... no 2024-07-27T06:49:25.985 INFO:tasks.workunit.client.0.smithi029.stdout:checking for suffix of object files... o 2024-07-27T06:49:26.011 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether we are using the GNU C compiler... yes 2024-07-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether gcc accepts -g... yes 2024-07-27T06:49:26.125 INFO:tasks.workunit.client.0.smithi029.stdout:checking for gcc option to accept ISO C89... none needed 2024-07-27T06:49:26.128 INFO:tasks.workunit.client.0.smithi029.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-07-27T06:49:26.133 INFO:tasks.workunit.client.0.smithi029.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-07-27T06:49:26.134 INFO:tasks.workunit.client.0.smithi029.stdout:checking for egrep... /usr/bin/grep -E 2024-07-27T06:49:26.139 INFO:tasks.workunit.client.0.smithi029.stdout:checking for fgrep... /usr/bin/grep -F 2024-07-27T06:49:26.139 INFO:tasks.workunit.client.0.smithi029.stdout:checking for ld used by gcc... /usr/bin/ld 2024-07-27T06:49:26.140 INFO:tasks.workunit.client.0.smithi029.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-07-27T06:49:26.147 INFO:tasks.workunit.client.0.smithi029.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-07-27T06:49:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:27 smithi077 ceph-mon[37153]: pgmap v470: 97 pgs: 97 active+clean; 38 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 492 op/s 2024-07-27T06:49:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:49:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:27 smithi047 ceph-mon[36835]: pgmap v470: 97 pgs: 97 active+clean; 38 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 492 op/s 2024-07-27T06:49:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:49:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:27 smithi029 ceph-mon[32610]: pgmap v470: 97 pgs: 97 active+clean; 38 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 492 op/s 2024-07-27T06:49:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:49:27.996 INFO:tasks.workunit.client.0.smithi029.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-07-27T06:49:27.996 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether ln -s works... yes 2024-07-27T06:49:28.044 INFO:tasks.workunit.client.0.smithi029.stdout:checking the maximum length of command line arguments... 1572864 2024-07-27T06:49:28.047 INFO:tasks.workunit.client.0.smithi029.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-07-27T06:49:28.047 INFO:tasks.workunit.client.0.smithi029.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-07-27T06:49:28.047 INFO:tasks.workunit.client.0.smithi029.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-07-27T06:49:28.048 INFO:tasks.workunit.client.0.smithi029.stdout:checking for objdump... objdump 2024-07-27T06:49:28.048 INFO:tasks.workunit.client.0.smithi029.stdout:checking how to recognize dependent libraries... pass_all 2024-07-27T06:49:28.049 INFO:tasks.workunit.client.0.smithi029.stdout:checking for dlltool... no 2024-07-27T06:49:28.049 INFO:tasks.workunit.client.0.smithi029.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-07-27T06:49:28.049 INFO:tasks.workunit.client.0.smithi029.stdout:checking for ar... ar 2024-07-27T06:49:28.148 INFO:tasks.workunit.client.0.smithi029.stdout:checking for archiver @FILE support... @ 2024-07-27T06:49:28.148 INFO:tasks.workunit.client.0.smithi029.stdout:checking for strip... strip 2024-07-27T06:49:28.148 INFO:tasks.workunit.client.0.smithi029.stdout:checking for ranlib... ranlib 2024-07-27T06:49:28.148 INFO:tasks.workunit.client.0.smithi029.stdout:checking for gawk... gawk 2024-07-27T06:49:28.250 INFO:tasks.workunit.client.0.smithi029.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-07-27T06:49:28.251 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sysroot... no 2024-07-27T06:49:28.263 INFO:tasks.workunit.client.0.smithi029.stdout:checking for a working dd... /usr/bin/dd 2024-07-27T06:49:28.275 INFO:tasks.workunit.client.0.smithi029.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-07-27T06:49:28.309 INFO:tasks.workunit.client.0.smithi029.stdout:checking for mt... no 2024-07-27T06:49:28.322 INFO:tasks.workunit.client.0.smithi029.stdout:checking if : is a manifest tool... no 2024-07-27T06:49:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:49:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:49:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:49:28.418 INFO:tasks.workunit.client.0.smithi029.stdout:checking how to run the C preprocessor... gcc -E 2024-07-27T06:49:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:49:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:49:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:49:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:49:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:49:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:49:28.605 INFO:tasks.workunit.client.0.smithi029.stdout:checking for ANSI C header files... yes 2024-07-27T06:49:28.643 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/types.h... yes 2024-07-27T06:49:28.682 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/stat.h... yes 2024-07-27T06:49:28.721 INFO:tasks.workunit.client.0.smithi029.stdout:checking for stdlib.h... yes 2024-07-27T06:49:28.761 INFO:tasks.workunit.client.0.smithi029.stdout:checking for string.h... yes 2024-07-27T06:49:28.802 INFO:tasks.workunit.client.0.smithi029.stdout:checking for memory.h... yes 2024-07-27T06:49:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:checking for strings.h... yes 2024-07-27T06:49:28.896 INFO:tasks.workunit.client.0.smithi029.stdout:checking for inttypes.h... yes 2024-07-27T06:49:28.939 INFO:tasks.workunit.client.0.smithi029.stdout:checking for stdint.h... yes 2024-07-27T06:49:28.983 INFO:tasks.workunit.client.0.smithi029.stdout:checking for unistd.h... yes 2024-07-27T06:49:29.034 INFO:tasks.workunit.client.0.smithi029.stdout:checking for dlfcn.h... yes 2024-07-27T06:49:29.039 INFO:tasks.workunit.client.0.smithi029.stdout:checking for objdir... .libs 2024-07-27T06:49:29.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.125+0000 7faf0724b640 1 -- 172.21.15.29:0/15577781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf00102420 msgr2=0x7faf00102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:29.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.125+0000 7faf0724b640 1 --2- 172.21.15.29:0/15577781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf00102420 0x7faf00102830 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7faef4009980 tx=0x7faef402e820 comp rx=0 tx=0).stop 2024-07-27T06:49:29.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.125+0000 7faf0724b640 1 -- 172.21.15.29:0/15577781 shutdown_connections 2024-07-27T06:49:29.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.125+0000 7faf0724b640 1 --2- 172.21.15.29:0/15577781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf001086a0 0x7faf0010eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.125+0000 7faf0724b640 1 --2- 172.21.15.29:0/15577781 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf00107d50 0x7faf00108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.125+0000 7faf0724b640 1 --2- 172.21.15.29:0/15577781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf00102420 0x7faf00102830 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.125+0000 7faf0724b640 1 -- 172.21.15.29:0/15577781 >> 172.21.15.29:0/15577781 conn(0x7faf000fde20 msgr2=0x7faf00100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:29.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.125+0000 7faf0724b640 1 -- 172.21.15.29:0/15577781 shutdown_connections 2024-07-27T06:49:29.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.125+0000 7faf0724b640 1 -- 172.21.15.29:0/15577781 wait complete. 2024-07-27T06:49:29.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.126+0000 7faf0724b640 1 Processor -- start 2024-07-27T06:49:29.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.126+0000 7faf0724b640 1 -- start start 2024-07-27T06:49:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.126+0000 7faf0724b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf00102420 0x7faf00197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.126+0000 7faf0724b640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf00107d50 0x7faf00198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.126+0000 7faf0724b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf001086a0 0x7faf0019d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.126+0000 7faf0724b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faf00111260 con 0x7faf001086a0 2024-07-27T06:49:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.126+0000 7faf0724b640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7faf001110e0 con 0x7faf00107d50 2024-07-27T06:49:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.126+0000 7faf0724b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faf001113e0 con 0x7faf00102420 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.126+0000 7faf057e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf001086a0 0x7faf0019d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faf057e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf001086a0 0x7faf0019d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45640/0 (socket says 172.21.15.29:45640) 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faf057e8640 1 -- 172.21.15.29:0/3598184982 learned_addr learned my addr 172.21.15.29:0/3598184982 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faf04fe7640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf00102420 0x7faf00197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faefffff640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf00107d50 0x7faf00198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faf057e8640 1 -- 172.21.15.29:0/3598184982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf00102420 msgr2=0x7faf00197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faf057e8640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf00102420 0x7faf00197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faf057e8640 1 -- 172.21.15.29:0/3598184982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf00107d50 msgr2=0x7faf00198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faf057e8640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf00107d50 0x7faf00198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faf057e8640 1 -- 172.21.15.29:0/3598184982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0019dbb0 con 0x7faf001086a0 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faf057e8640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf001086a0 0x7faf0019d4b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7faef800abd0 tx=0x7faef8007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faefdffb640 1 -- 172.21.15.29:0/3598184982 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8007ca0 con 0x7faf001086a0 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faefdffb640 1 -- 172.21.15.29:0/3598184982 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef8007e40 con 0x7faf001086a0 2024-07-27T06:49:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faf0724b640 1 -- 172.21.15.29:0/3598184982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0019dea0 con 0x7faf001086a0 2024-07-27T06:49:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.127+0000 7faefdffb640 1 -- 172.21.15.29:0/3598184982 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8014410 con 0x7faf001086a0 2024-07-27T06:49:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.128+0000 7faf0724b640 1 -- 172.21.15.29:0/3598184982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf001a4630 con 0x7faf001086a0 2024-07-27T06:49:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.130+0000 7faf0e16f740 1 Processor -- start 2024-07-27T06:49:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.130+0000 7faefdffb640 1 -- 172.21.15.29:0/3598184982 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7faef80052b0 con 0x7faf001086a0 2024-07-27T06:49:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.130+0000 7faf0e16f740 1 -- start start 2024-07-27T06:49:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf0e16f740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa8d1b0 0x55d02ae2fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf0e16f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 0x55d02ae30220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf0e16f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa6c4d0 0x55d02ae30760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf0e16f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d02ada53b0 con 0x55d02aa6c4d0 2024-07-27T06:49:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf0e16f740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55d02ad948b0 con 0x55d02aa8d1b0 2024-07-27T06:49:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf0e16f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d02ad988f0 con 0x55d02aa642f0 2024-07-27T06:49:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faefffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa6c4d0 0x55d02ae30760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf04fe7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 0x55d02ae30220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf04fe7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 0x55d02ae30220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:36722/0 (socket says 172.21.15.29:36722) 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf04fe7640 1 -- 172.21.15.29:0/3079916632 learned_addr learned my addr 172.21.15.29:0/3079916632 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faefffff640 1 --2- 172.21.15.29:0/3079916632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa8d1b0 0x55d02ae2fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf04fe7640 1 -- 172.21.15.29:0/3079916632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa8d1b0 msgr2=0x55d02ae2fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf04fe7640 1 --2- 172.21.15.29:0/3079916632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa8d1b0 0x55d02ae2fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf04fe7640 1 -- 172.21.15.29:0/3079916632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa6c4d0 msgr2=0x55d02ae30760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf04fe7640 1 --2- 172.21.15.29:0/3079916632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa6c4d0 0x55d02ae30760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf04fe7640 1 -- 172.21.15.29:0/3079916632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d02ad3b9a0 con 0x55d02aa642f0 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faefdffb640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7faef00b3d40 0x7faef00b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faefffff640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7faef00b3d40 0x7faef00b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faefdffb640 1 -- 172.21.15.29:0/3598184982 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7faef80e3900 con 0x7faf001086a0 2024-07-27T06:49:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.131+0000 7faf04fe7640 1 --2- 172.21.15.29:0/3079916632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 0x55d02ae30220 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7faef4009980 tx=0x7faef4036c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72e1b8c1b6cc6c2a server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.132+0000 7faeeaffd640 1 -- 172.21.15.29:0/3079916632 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef4036410 con 0x55d02aa642f0 2024-07-27T06:49:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.132+0000 7faeeaffd640 1 -- 172.21.15.29:0/3079916632 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef40365b0 con 0x55d02aa642f0 2024-07-27T06:49:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.132+0000 7faeeaffd640 1 -- 172.21.15.29:0/3079916632 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef4005420 con 0x55d02aa642f0 2024-07-27T06:49:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.133+0000 7faf0e16f740 1 -- 172.21.15.29:0/3079916632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 msgr2=0x55d02ae30220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.133+0000 7faf0e16f740 1 --2- 172.21.15.29:0/3079916632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 0x55d02ae30220 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7faef4009980 tx=0x7faef4036c90 comp rx=0 tx=0).stop 2024-07-27T06:49:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.133+0000 7faefffff640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7faef00b3d40 0x7faef00b6250 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faeec00bfd0 tx=0x7faeec00b650 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.133+0000 7faf0e16f740 1 -- 172.21.15.29:0/3079916632 shutdown_connections 2024-07-27T06:49:29.140 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.133+0000 7faf0e16f740 1 --2- 172.21.15.29:0/3079916632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa6c4d0 0x55d02ae30760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.133+0000 7faf0e16f740 1 --2- 172.21.15.29:0/3079916632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 0x55d02ae30220 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.133+0000 7faf0e16f740 1 --2- 172.21.15.29:0/3079916632 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa8d1b0 0x55d02ae2fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.133+0000 7faf0e16f740 1 -- 172.21.15.29:0/3079916632 >> 172.21.15.29:0/3079916632 conn(0x55d02aa81300 msgr2=0x55d02ae2dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.133+0000 7faf0e16f740 1 -- 172.21.15.29:0/3079916632 shutdown_connections 2024-07-27T06:49:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.133+0000 7faf0e16f740 1 -- 172.21.15.29:0/3079916632 wait complete. 2024-07-27T06:49:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.133+0000 7faf0e16f740 1 -- start start 2024-07-27T06:49:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.133+0000 7faf0e16f740 1 Processor -- start 2024-07-27T06:49:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf0e16f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa642f0 0x55d02ae3b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf0e16f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa6c4d0 0x55d02ae3bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf0e16f740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa8d1b0 0x55d02ae47720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.141 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf0e16f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55d02acceab0 con 0x55d02aa642f0 2024-07-27T06:49:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf0e16f740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55d02ad1ccf0 con 0x55d02aa8d1b0 2024-07-27T06:49:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf0e16f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d02ae3e400 con 0x55d02aa6c4d0 2024-07-27T06:49:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf0e16f740 10 client.0 _ll_register_callbacks cb 0x55d02ad8e9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf04fe7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa642f0 0x55d02ae3b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf04fe7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa642f0 0x55d02ae3b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45652/0 (socket says 172.21.15.29:45652) 2024-07-27T06:49:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf04fe7640 1 -- 172.21.15.29:0/1074060438 learned_addr learned my addr 172.21.15.29:0/1074060438 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faefffff640 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa6c4d0 0x55d02ae3bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf057e8640 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa8d1b0 0x55d02ae47720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf04fe7640 1 -- 172.21.15.29:0/1074060438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa6c4d0 msgr2=0x55d02ae3bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf04fe7640 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa6c4d0 0x55d02ae3bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf04fe7640 1 -- 172.21.15.29:0/1074060438 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa8d1b0 msgr2=0x55d02ae47720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:29.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf04fe7640 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa8d1b0 0x55d02ae47720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf04fe7640 1 -- 172.21.15.29:0/1074060438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d02aaf8790 con 0x55d02aa642f0 2024-07-27T06:49:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf057e8640 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa8d1b0 0x55d02ae47720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:49:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faf04fe7640 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa642f0 0x55d02ae3b7a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7faef4009430 tx=0x7faef4036ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8437f463da3de26d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.134+0000 7faedffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faedffff640 1 -- 172.21.15.29:0/1074060438 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef4009520 con 0x55d02aa642f0 2024-07-27T06:49:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faedffff640 1 -- 172.21.15.29:0/1074060438 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7faef4004050 con 0x55d02aa642f0 2024-07-27T06:49:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faf0e16f740 1 -- 172.21.15.29:0/1074060438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faef40365b0 con 0x55d02aa642f0 2024-07-27T06:49:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faedffff640 1 -- 172.21.15.29:0/1074060438 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faef4004250 con 0x55d02aa642f0 2024-07-27T06:49:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faf0e16f740 10 client.16236 fetch_fsmap learned FSMap version 32 2024-07-27T06:49:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faf0e16f740 1 -- 172.21.15.29:0/1074060438 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d02ab71620 con 0x55d02aa642f0 2024-07-27T06:49:29.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faedffff640 1 -- 172.21.15.29:0/1074060438 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef40365b0 con 0x55d02aa642f0 2024-07-27T06:49:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faedffff640 1 -- 172.21.15.29:0/1074060438 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8382+0+0 (secure 0 0 0) 0x7faef4003760 con 0x55d02aa642f0 2024-07-27T06:49:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faf0e16f740 10 client.16236 fetch_fsmap finished waiting for FSMap version 32 2024-07-27T06:49:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faf0e16f740 10 client.16236 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:49:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faf0e16f740 20 client.16236 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faf0e16f740 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d02aa4bd80 0x55d02ae4d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faf0e16f740 4 client.16236 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.135+0000 7faf0e16f740 1 -- 172.21.15.29:0/1074060438 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d02ada7300 con 0x55d02aa4bd80 2024-07-27T06:49:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.136+0000 7faefffff640 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d02aa4bd80 0x55d02ae4d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.136+0000 7faedffff640 10 client.16236 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.136+0000 7faefffff640 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d02aa4bd80 0x55d02ae4d4d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=545aa46044524285 server_cookie=520ac19b382fe434 in_seq=0 out_seq=0 2024-07-27T06:49:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faedffff640 1 -- 172.21.15.29:0/1074060438 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d02ada7300 con 0x55d02aa4bd80 2024-07-27T06:49:29.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faedffff640 10 client.16236 handle_command_reply: tid=0 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faf0e16f740 1 client.16236 shutdown 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faf0e16f740 1 -- 172.21.15.29:0/1074060438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa642f0 msgr2=0x55d02ae3b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faf0e16f740 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa642f0 0x55d02ae3b7a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7faef4009430 tx=0x7faef4036ec0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faf0e16f740 1 -- 172.21.15.29:0/1074060438 shutdown_connections 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faf0e16f740 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa8d1b0 0x55d02ae47720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faf0e16f740 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d02aa4bd80 0x55d02ae4d4d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faf0e16f740 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa6c4d0 0x55d02ae3bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faedffff640 0 client.16236 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faf0e16f740 1 --2- 172.21.15.29:0/1074060438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa642f0 0x55d02ae3b7a0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faf0e16f740 1 -- 172.21.15.29:0/1074060438 >> 172.21.15.29:0/1074060438 conn(0x55d02aa81300 msgr2=0x55d02ae30d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faf0e16f740 1 -- 172.21.15.29:0/1074060438 shutdown_connections 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faf0e16f740 1 -- 172.21.15.29:0/1074060438 wait complete. 2024-07-27T06:49:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.137+0000 7faf0e16f740 20 client.16236 trim_cache size 0 max 16384 2024-07-27T06:49:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.153+0000 7faf0e16f740 1 Processor -- start 2024-07-27T06:49:29.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.153+0000 7faf0e16f740 1 -- start start 2024-07-27T06:49:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.153+0000 7faf0e16f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 0x55d02ae3a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.153+0000 7faf0e16f740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa6c4d0 0x55d02ae3bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.153+0000 7faf0e16f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 0x55d02ae52c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.153+0000 7faf0e16f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faf001ace80 con 0x55d02aa8d1b0 2024-07-27T06:49:29.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.153+0000 7faf0e16f740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7faf001aeab0 con 0x55d02aa6c4d0 2024-07-27T06:49:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.153+0000 7faf0e16f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faf001ad550 con 0x55d02aa642f0 2024-07-27T06:49:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faf057e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 0x55d02ae52c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faf057e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 0x55d02ae52c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45654/0 (socket says 172.21.15.29:45654) 2024-07-27T06:49:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faf057e8640 1 -- 172.21.15.29:0/1228488048 learned_addr learned my addr 172.21.15.29:0/1228488048 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faf04fe7640 1 --2- 172.21.15.29:0/1228488048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 0x55d02ae3a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faefffff640 1 --2- 172.21.15.29:0/1228488048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa6c4d0 0x55d02ae3bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faf057e8640 1 -- 172.21.15.29:0/1228488048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 msgr2=0x55d02ae3a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faf057e8640 1 --2- 172.21.15.29:0/1228488048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 0x55d02ae3a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faf057e8640 1 -- 172.21.15.29:0/1228488048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa6c4d0 msgr2=0x55d02ae3bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faf057e8640 1 --2- 172.21.15.29:0/1228488048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa6c4d0 0x55d02ae3bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faf057e8640 1 -- 172.21.15.29:0/1228488048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d02ae316d0 con 0x55d02aa8d1b0 2024-07-27T06:49:29.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faf04fe7640 1 --2- 172.21.15.29:0/1228488048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 0x55d02ae3a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:49:29.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faf057e8640 1 --2- 172.21.15.29:0/1228488048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 0x55d02ae52c00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7faef80aa2d0 tx=0x7faef80aa050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d65c4e39383ec server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:29.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faeea7fc640 1 -- 172.21.15.29:0/1228488048 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef80e12e0 con 0x55d02aa8d1b0 2024-07-27T06:49:29.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faeea7fc640 1 -- 172.21.15.29:0/1228488048 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef80a6860 con 0x55d02aa8d1b0 2024-07-27T06:49:29.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.154+0000 7faeea7fc640 1 -- 172.21.15.29:0/1228488048 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef80a92a0 con 0x55d02aa8d1b0 2024-07-27T06:49:29.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.155+0000 7faf0e16f740 1 -- 172.21.15.29:0/1228488048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 msgr2=0x55d02ae52c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:29.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.155+0000 7faf0e16f740 1 --2- 172.21.15.29:0/1228488048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 0x55d02ae52c00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7faef80aa2d0 tx=0x7faef80aa050 comp rx=0 tx=0).stop 2024-07-27T06:49:29.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.155+0000 7faf0e16f740 1 -- 172.21.15.29:0/1228488048 shutdown_connections 2024-07-27T06:49:29.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.155+0000 7faf0e16f740 1 --2- 172.21.15.29:0/1228488048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 0x55d02ae52c00 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.155+0000 7faf0e16f740 1 --2- 172.21.15.29:0/1228488048 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa6c4d0 0x55d02ae3bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.167 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.155+0000 7faf0e16f740 1 --2- 172.21.15.29:0/1228488048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa642f0 0x55d02ae3a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.155+0000 7faf0e16f740 1 -- 172.21.15.29:0/1228488048 >> 172.21.15.29:0/1228488048 conn(0x55d02aa81300 msgr2=0x55d02ae3b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.155+0000 7faf0e16f740 1 -- 172.21.15.29:0/1228488048 shutdown_connections 2024-07-27T06:49:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.155+0000 7faf0e16f740 1 -- 172.21.15.29:0/1228488048 wait complete. 2024-07-27T06:49:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.155+0000 7faf0e16f740 1 -- start start 2024-07-27T06:49:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf0e16f740 1 Processor -- start 2024-07-27T06:49:29.168 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf0e16f740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa642f0 0x55d02ae43f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf0e16f740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa6c4d0 0x55d02ae42430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf0e16f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 0x55d02ae42b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf0e16f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faf001ae320 con 0x55d02aa8d1b0 2024-07-27T06:49:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf0e16f740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7faf001adc20 con 0x55d02aa642f0 2024-07-27T06:49:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf0e16f740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faf001aae60 con 0x55d02aa6c4d0 2024-07-27T06:49:29.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf0e16f740 10 client.0 _ll_register_callbacks cb 0x7faeec00bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:29.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf057e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 0x55d02ae42b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf04fe7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa642f0 0x55d02ae43f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf057e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 0x55d02ae42b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45656/0 (socket says 172.21.15.29:45656) 2024-07-27T06:49:29.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf04fe7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa642f0 0x55d02ae43f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.29:35528/0 (socket says 172.21.15.29:35528) 2024-07-27T06:49:29.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faefffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa6c4d0 0x55d02ae42430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf057e8640 1 -- 172.21.15.29:0/2217384661 learned_addr learned my addr 172.21.15.29:0/2217384661 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:29.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf057e8640 1 -- 172.21.15.29:0/2217384661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa6c4d0 msgr2=0x55d02ae42430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:29.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf057e8640 1 --2- 172.21.15.29:0/2217384661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa6c4d0 0x55d02ae42430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.156+0000 7faf057e8640 1 -- 172.21.15.29:0/2217384661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa642f0 msgr2=0x55d02ae43f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:29.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faf057e8640 1 --2- 172.21.15.29:0/2217384661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa642f0 0x55d02ae43f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faf057e8640 1 -- 172.21.15.29:0/2217384661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d02aeb3be0 con 0x55d02aa8d1b0 2024-07-27T06:49:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faf04fe7640 1 --2- 172.21.15.29:0/2217384661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa642f0 0x55d02ae43f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:49:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faf057e8640 1 --2- 172.21.15.29:0/2217384661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 0x55d02ae42b40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7faef80e1110 tx=0x7faef8005650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3decb10e85065c82 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:29.172 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faedffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:29.172 INFO:tasks.workunit.client.0.smithi029.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-07-27T06:49:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faedffff640 1 -- 172.21.15.29:0/2217384661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef80e12e0 con 0x55d02aa8d1b0 2024-07-27T06:49:29.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faedffff640 1 -- 172.21.15.29:0/2217384661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7faef80bfbc0 con 0x55d02aa8d1b0 2024-07-27T06:49:29.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faf0e16f740 1 -- 172.21.15.29:0/2217384661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faef80a6860 con 0x55d02aa8d1b0 2024-07-27T06:49:29.174 INFO:tasks.workunit.client.0.smithi029.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-07-27T06:49:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faedffff640 1 -- 172.21.15.29:0/2217384661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faef80bfdc0 con 0x55d02aa8d1b0 2024-07-27T06:49:29.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faf0e16f740 10 client.16242 fetch_fsmap learned FSMap version 32 2024-07-27T06:49:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faf0e16f740 1 -- 172.21.15.29:0/2217384661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d02aeb4d20 con 0x55d02aa8d1b0 2024-07-27T06:49:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faedffff640 1 -- 172.21.15.29:0/2217384661 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faef80a6860 con 0x55d02aa8d1b0 2024-07-27T06:49:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.157+0000 7faedffff640 1 -- 172.21.15.29:0/2217384661 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8382+0+0 (secure 0 0 0) 0x7faef80afdd0 con 0x55d02aa8d1b0 2024-07-27T06:49:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.158+0000 7faf0e16f740 10 client.16242 fetch_fsmap finished waiting for FSMap version 32 2024-07-27T06:49:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.158+0000 7faf0e16f740 10 client.16242 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:49:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.158+0000 7faf0e16f740 20 client.16242 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.158+0000 7faf0e16f740 1 --2- 172.21.15.29:0/2217384661 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d02aa211c0 0x55d02aeb74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.158+0000 7faf0e16f740 4 client.16242 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-27T06:49:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.158+0000 7faf0e16f740 1 -- 172.21.15.29:0/2217384661 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d02ada2da0 con 0x55d02aa211c0 2024-07-27T06:49:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.158+0000 7faf04fe7640 1 --2- 172.21.15.29:0/2217384661 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d02aa211c0 0x55d02aeb74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.158+0000 7faedffff640 10 client.16242 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.158+0000 7faf04fe7640 1 --2- 172.21.15.29:0/2217384661 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d02aa211c0 0x55d02aeb74b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64e507f25d46d1b0 server_cookie=fac83bdaf48631ae in_seq=0 out_seq=0 2024-07-27T06:49:29.495 INFO:tasks.workunit.client.0.smithi029.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-07-27T06:49:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.520+0000 7faedffff640 1 -- 172.21.15.29:0/2217384661 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d02ada2da0 con 0x55d02aa211c0 2024-07-27T06:49:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.520+0000 7faedffff640 10 client.16242 handle_command_reply: tid=0 2024-07-27T06:49:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.521+0000 7faf0e16f740 1 client.16242 shutdown 2024-07-27T06:49:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.521+0000 7faf0e16f740 1 -- 172.21.15.29:0/2217384661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 msgr2=0x55d02ae42b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.521+0000 7faf0e16f740 1 --2- 172.21.15.29:0/2217384661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 0x55d02ae42b40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7faef80e1110 tx=0x7faef8005650 comp rx=0 tx=0).stop 2024-07-27T06:49:29.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.521+0000 7faf0e16f740 1 -- 172.21.15.29:0/2217384661 shutdown_connections 2024-07-27T06:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.521+0000 7faf0e16f740 1 --2- 172.21.15.29:0/2217384661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d02aa8d1b0 0x55d02ae42b40 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.521+0000 7faf0e16f740 1 --2- 172.21.15.29:0/2217384661 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x55d02aa211c0 0x55d02aeb74b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.521+0000 7faf0e16f740 1 --2- 172.21.15.29:0/2217384661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d02aa6c4d0 0x55d02ae42430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.521+0000 7faf0e16f740 1 --2- 172.21.15.29:0/2217384661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55d02aa642f0 0x55d02ae43f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.521+0000 7faedffff640 0 client.16242 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.521+0000 7faf0e16f740 1 -- 172.21.15.29:0/2217384661 >> 172.21.15.29:0/2217384661 conn(0x55d02aa81300 msgr2=0x55d02ae3c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.522+0000 7faf0e16f740 1 -- 172.21.15.29:0/2217384661 shutdown_connections 2024-07-27T06:49:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.522+0000 7faf0e16f740 1 -- 172.21.15.29:0/2217384661 wait complete. 2024-07-27T06:49:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.522+0000 7faf0e16f740 20 client.16242 trim_cache size 0 max 16384 2024-07-27T06:49:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:{ 2024-07-27T06:49:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "message": "", 2024-07-27T06:49:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout: "return_code": 0 2024-07-27T06:49:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stdout:} 2024-07-27T06:49:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.523+0000 7faf0724b640 1 -- 172.21.15.29:0/3598184982 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7faef00b3d40 msgr2=0x7faef00b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.523+0000 7faf0724b640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7faef00b3d40 0x7faef00b6250 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7faeec00bfd0 tx=0x7faeec00b650 comp rx=0 tx=0).stop 2024-07-27T06:49:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.523+0000 7faf0724b640 1 -- 172.21.15.29:0/3598184982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf001086a0 msgr2=0x7faf0019d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.523+0000 7faf0724b640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf001086a0 0x7faf0019d4b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7faef800abd0 tx=0x7faef8007590 comp rx=0 tx=0).stop 2024-07-27T06:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.523+0000 7faf0724b640 1 -- 172.21.15.29:0/3598184982 shutdown_connections 2024-07-27T06:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.523+0000 7faf0724b640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf001086a0 0x7faf0019d4b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.523+0000 7faf0724b640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf00107d50 0x7faf00198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.523+0000 7faf0724b640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7faef00b3d40 0x7faef00b6250 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.523+0000 7faf0724b640 1 --2- 172.21.15.29:0/3598184982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf00102420 0x7faf00197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.523+0000 7faf0724b640 1 -- 172.21.15.29:0/3598184982 >> 172.21.15.29:0/3598184982 conn(0x7faf000fde20 msgr2=0x7faf00107450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.523+0000 7faf0724b640 1 -- 172.21.15.29:0/3598184982 shutdown_connections 2024-07-27T06:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:29.523+0000 7faf0724b640 1 -- 172.21.15.29:0/3598184982 wait complete. 2024-07-27T06:49:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:29 smithi077 ceph-mon[37153]: pgmap v471: 97 pgs: 97 active+clean; 43 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 666 op/s 2024-07-27T06:49:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:29 smithi047 ceph-mon[36835]: pgmap v471: 97 pgs: 97 active+clean; 43 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 666 op/s 2024-07-27T06:49:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:29 smithi029 ceph-mon[32610]: pgmap v471: 97 pgs: 97 active+clean; 43 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 666 op/s 2024-07-27T06:49:30.134 INFO:tasks.workunit.client.0.smithi029.stdout:checking if gcc static flag -static works... no 2024-07-27T06:49:30.251 INFO:tasks.workunit.client.0.smithi029.stdout:checking if gcc supports -c -o file.o... yes 2024-07-27T06:49:30.251 INFO:tasks.workunit.client.0.smithi029.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-07-27T06:49:30.259 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-07-27T06:49:30.574 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether -lc should be explicitly linked in... no 2024-07-27T06:49:30.745 INFO:tasks.workunit.client.0.smithi029.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-07-27T06:49:30.745 INFO:tasks.workunit.client.0.smithi029.stdout:checking how to hardcode library paths into programs... immediate 2024-07-27T06:49:30.746 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether stripping libraries is possible... yes 2024-07-27T06:49:30.747 INFO:tasks.workunit.client.0.smithi029.stdout:checking if libtool supports shared libraries... yes 2024-07-27T06:49:30.747 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether to build shared libraries... yes 2024-07-27T06:49:30.747 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether to build static libraries... yes 2024-07-27T06:49:30.755 INFO:tasks.workunit.client.0.smithi029.stdout:checking for gcc... (cached) gcc 2024-07-27T06:49:30.794 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-07-27T06:49:30.794 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether gcc accepts -g... (cached) yes 2024-07-27T06:49:30.794 INFO:tasks.workunit.client.0.smithi029.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-07-27T06:49:30.795 INFO:tasks.workunit.client.0.smithi029.stdout:checking for gmake... /usr/bin/gmake 2024-07-27T06:49:30.795 INFO:tasks.workunit.client.0.smithi029.stderr:./configure: line 12082: test: $(SHELL): binary operator expected 2024-07-27T06:49:30.795 INFO:tasks.workunit.client.0.smithi029.stdout:checking for tar... /bin/tar 2024-07-27T06:49:30.796 INFO:tasks.workunit.client.0.smithi029.stdout:checking for gzip... /bin/gzip 2024-07-27T06:49:30.824 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether gcc -MM is supported... yes 2024-07-27T06:49:30.824 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sort... /bin/sort 2024-07-27T06:49:30.824 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether ln -s works... yes 2024-07-27T06:49:30.824 INFO:tasks.workunit.client.0.smithi029.stdout:checking for rpm... /bin/rpm 2024-07-27T06:49:30.828 INFO:tasks.workunit.client.0.smithi029.stdout:checking for rpmbuild... no 2024-07-27T06:49:30.886 INFO:tasks.workunit.client.0.smithi029.stdout:checking assert.h usability... yes 2024-07-27T06:49:30.903 INFO:tasks.workunit.client.0.smithi029.stdout:checking assert.h presence... yes 2024-07-27T06:49:30.903 INFO:tasks.workunit.client.0.smithi029.stdout:checking for assert.h... yes 2024-07-27T06:49:30.957 INFO:tasks.workunit.client.0.smithi029.stdout:checking bstring.h usability... no 2024-07-27T06:49:30.990 INFO:tasks.workunit.client.0.smithi029.stdout:checking bstring.h presence... no 2024-07-27T06:49:30.990 INFO:tasks.workunit.client.0.smithi029.stdout:checking for bstring.h... no 2024-07-27T06:49:31.031 INFO:tasks.workunit.client.0.smithi029.stdout:checking libgen.h usability... yes 2024-07-27T06:49:31.048 INFO:tasks.workunit.client.0.smithi029.stdout:checking libgen.h presence... yes 2024-07-27T06:49:31.048 INFO:tasks.workunit.client.0.smithi029.stdout:checking for libgen.h... yes 2024-07-27T06:49:31.098 INFO:tasks.workunit.client.0.smithi029.stdout:checking dirent.h usability... yes 2024-07-27T06:49:31.116 INFO:tasks.workunit.client.0.smithi029.stdout:checking dirent.h presence... yes 2024-07-27T06:49:31.116 INFO:tasks.workunit.client.0.smithi029.stdout:checking for dirent.h... yes 2024-07-27T06:49:31.168 INFO:tasks.workunit.client.0.smithi029.stdout:checking errno.h usability... yes 2024-07-27T06:49:31.188 INFO:tasks.workunit.client.0.smithi029.stdout:checking errno.h presence... yes 2024-07-27T06:49:31.188 INFO:tasks.workunit.client.0.smithi029.stdout:checking for errno.h... yes 2024-07-27T06:49:31.235 INFO:tasks.workunit.client.0.smithi029.stdout:checking malloc.h usability... yes 2024-07-27T06:49:31.253 INFO:tasks.workunit.client.0.smithi029.stdout:checking malloc.h presence... yes 2024-07-27T06:49:31.254 INFO:tasks.workunit.client.0.smithi029.stdout:checking for malloc.h... yes 2024-07-27T06:49:31.306 INFO:tasks.workunit.client.0.smithi029.stdout:checking uuid.h usability... no 2024-07-27T06:49:31.336 INFO:tasks.workunit.client.0.smithi029.stdout:checking uuid.h presence... no 2024-07-27T06:49:31.336 INFO:tasks.workunit.client.0.smithi029.stdout:checking for uuid.h... no 2024-07-27T06:49:31.387 INFO:tasks.workunit.client.0.smithi029.stdout:checking uuid/uuid.h usability... yes 2024-07-27T06:49:31.406 INFO:tasks.workunit.client.0.smithi029.stdout:checking uuid/uuid.h presence... yes 2024-07-27T06:49:31.406 INFO:tasks.workunit.client.0.smithi029.stdout:checking for uuid/uuid.h... yes 2024-07-27T06:49:31.459 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/uuid.h usability... no 2024-07-27T06:49:31.488 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/uuid.h presence... no 2024-07-27T06:49:31.488 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/uuid.h... no 2024-07-27T06:49:31.528 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/file.h usability... yes 2024-07-27T06:49:31.546 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/file.h presence... yes 2024-07-27T06:49:31.547 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/file.h... yes 2024-07-27T06:49:31.597 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/fcntl.h usability... yes 2024-07-27T06:49:31.614 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/fcntl.h presence... yes 2024-07-27T06:49:31.615 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/fcntl.h... yes 2024-07-27T06:49:31.667 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/syssgi.h usability... no 2024-07-27T06:49:31.697 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/syssgi.h presence... no 2024-07-27T06:49:31.697 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/syssgi.h... no 2024-07-27T06:49:31.770 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-27T06:49:31.783 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/param.h usability... yes 2024-07-27T06:49:31.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:31 smithi029 ceph-mon[32610]: pgmap v472: 97 pgs: 97 active+clean; 43 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 544 op/s 2024-07-27T06:49:31.816 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/param.h presence... yes 2024-07-27T06:49:31.816 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/param.h... yes 2024-07-27T06:49:31.824 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/stat.h... (cached) yes 2024-07-27T06:49:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:31 smithi077 ceph-mon[37153]: pgmap v472: 97 pgs: 97 active+clean; 43 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 544 op/s 2024-07-27T06:49:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:31 smithi047 ceph-mon[36835]: pgmap v472: 97 pgs: 97 active+clean; 43 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 544 op/s 2024-07-27T06:49:31.948 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/statvfs.h usability... yes 2024-07-27T06:49:32.064 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:49:32.182 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/statvfs.h presence... yes 2024-07-27T06:49:32.182 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/statvfs.h... yes 2024-07-27T06:49:32.309 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/time.h usability... yes 2024-07-27T06:49:32.326 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/time.h presence... yes 2024-07-27T06:49:32.326 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/time.h... yes 2024-07-27T06:49:32.373 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/ioctl.h usability... yes 2024-07-27T06:49:32.390 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/ioctl.h presence... yes 2024-07-27T06:49:32.391 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/ioctl.h... yes 2024-07-27T06:49:32.462 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/wait.h usability... yes 2024-07-27T06:49:32.482 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/wait.h presence... yes 2024-07-27T06:49:32.482 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/wait.h... yes 2024-07-27T06:49:32.496 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/types.h... (cached) yes 2024-07-27T06:49:32.500 INFO:tasks.workunit.client.0.smithi029.stdout:checking for strings.h... (cached) yes 2024-07-27T06:49:32.751 INFO:tasks.workunit.client.0.smithi029.stdout:checking err.h usability... yes 2024-07-27T06:49:32.768 INFO:tasks.workunit.client.0.smithi029.stdout:checking err.h presence... yes 2024-07-27T06:49:32.769 INFO:tasks.workunit.client.0.smithi029.stdout:checking for err.h... yes 2024-07-27T06:49:32.820 INFO:tasks.workunit.client.0.smithi029.stdout:checking linux/falloc.h usability... yes 2024-07-27T06:49:32.836 INFO:tasks.workunit.client.0.smithi029.stdout:checking linux/falloc.h presence... yes 2024-07-27T06:49:32.836 INFO:tasks.workunit.client.0.smithi029.stdout:checking for linux/falloc.h... yes 2024-07-27T06:49:32.897 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-07-27T06:49:32.927 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-07-27T06:49:32.928 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/fs/xfs_fsops.h... no 2024-07-27T06:49:32.979 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/fs/xfs_itable.h usability... no 2024-07-27T06:49:33.007 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/fs/xfs_itable.h presence... no 2024-07-27T06:49:33.007 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/fs/xfs_itable.h... no 2024-07-27T06:49:33.055 INFO:tasks.workunit.client.0.smithi029.stdout:checking xfs/platform_defs.h usability... no 2024-07-27T06:49:33.084 INFO:tasks.workunit.client.0.smithi029.stdout:checking xfs/platform_defs.h presence... no 2024-07-27T06:49:33.084 INFO:tasks.workunit.client.0.smithi029.stdout:checking for xfs/platform_defs.h... no 2024-07-27T06:49:33.134 INFO:tasks.workunit.client.0.smithi029.stdout:checking btrfs/ioctl.h usability... no 2024-07-27T06:49:33.167 INFO:tasks.workunit.client.0.smithi029.stdout:checking btrfs/ioctl.h presence... no 2024-07-27T06:49:33.168 INFO:tasks.workunit.client.0.smithi029.stdout:checking for btrfs/ioctl.h... no 2024-07-27T06:49:33.216 INFO:tasks.workunit.client.0.smithi029.stdout:checking cifs/ioctl.h usability... no 2024-07-27T06:49:33.246 INFO:tasks.workunit.client.0.smithi029.stdout:checking cifs/ioctl.h presence... no 2024-07-27T06:49:33.246 INFO:tasks.workunit.client.0.smithi029.stdout:checking for cifs/ioctl.h... no 2024-07-27T06:49:33.331 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/mman.h usability... yes 2024-07-27T06:49:33.348 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/mman.h presence... yes 2024-07-27T06:49:33.348 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/mman.h... yes 2024-07-27T06:49:33.405 INFO:tasks.workunit.client.0.smithi029.stdout:checking linux/ext4.h usability... no 2024-07-27T06:49:33.439 INFO:tasks.workunit.client.0.smithi029.stdout:checking linux/ext4.h presence... no 2024-07-27T06:49:33.439 INFO:tasks.workunit.client.0.smithi029.stdout:checking for linux/ext4.h... no 2024-07-27T06:49:33.476 INFO:tasks.workunit.client.0.smithi029.stdout:checking for xfs/xfs_log_format.h... no 2024-07-27T06:49:33.534 INFO:tasks.workunit.client.0.smithi029.stdout:checking for uuid_compare... no 2024-07-27T06:49:33.574 INFO:tasks.workunit.client.0.smithi029.stdout:checking for uuid_compare in -luuid... yes 2024-07-27T06:49:33.770 INFO:tasks.workunit.client.0.smithi029.stdout:checking for xfs/xfs.h... yes 2024-07-27T06:49:33.804 INFO:tasks.workunit.client.0.smithi029.stdout:checking for xfs/libxfs.h... no 2024-07-27T06:49:33.835 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether xlog_assign_lsn is declared... no 2024-07-27T06:49:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:33 smithi077 ceph-mon[37153]: pgmap v473: 97 pgs: 97 active+clean; 41 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 734 op/s 2024-07-27T06:49:33.874 INFO:tasks.workunit.client.0.smithi029.stdout:checking for xfs/xqm.h... yes 2024-07-27T06:49:33.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:33 smithi029 ceph-mon[32610]: pgmap v473: 97 pgs: 97 active+clean; 41 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 734 op/s 2024-07-27T06:49:33.929 INFO:tasks.workunit.client.0.smithi029.stdout:checking xfsctl from xfs/xfs.h... ok 2024-07-27T06:49:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:33 smithi047 ceph-mon[36835]: pgmap v473: 97 pgs: 97 active+clean; 41 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 734 op/s 2024-07-27T06:49:33.966 INFO:tasks.workunit.client.0.smithi029.stdout:checking xfs/handle.h usability... yes 2024-07-27T06:49:33.982 INFO:tasks.workunit.client.0.smithi029.stdout:checking xfs/handle.h presence... yes 2024-07-27T06:49:33.982 INFO:tasks.workunit.client.0.smithi029.stdout:checking for xfs/handle.h... yes 2024-07-27T06:49:34.026 INFO:tasks.workunit.client.0.smithi029.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-07-27T06:49:34.091 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/acl.h usability... yes 2024-07-27T06:49:34.108 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/acl.h presence... yes 2024-07-27T06:49:34.109 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/acl.h... yes 2024-07-27T06:49:34.145 INFO:tasks.workunit.client.0.smithi029.stdout:checking acl/libacl.h usability... yes 2024-07-27T06:49:34.164 INFO:tasks.workunit.client.0.smithi029.stdout:checking acl/libacl.h presence... yes 2024-07-27T06:49:34.165 INFO:tasks.workunit.client.0.smithi029.stdout:checking for acl/libacl.h... yes 2024-07-27T06:49:34.207 INFO:tasks.workunit.client.0.smithi029.stdout:checking for acl_init in -lacl... yes 2024-07-27T06:49:34.279 INFO:tasks.workunit.client.0.smithi029.stdout:checking gdbm-ndbm.h usability... no 2024-07-27T06:49:34.307 INFO:tasks.workunit.client.0.smithi029.stdout:checking gdbm-ndbm.h presence... no 2024-07-27T06:49:34.307 INFO:tasks.workunit.client.0.smithi029.stdout:checking for gdbm-ndbm.h... no 2024-07-27T06:49:34.359 INFO:tasks.workunit.client.0.smithi029.stdout:checking gdbm/ndbm.h usability... no 2024-07-27T06:49:34.390 INFO:tasks.workunit.client.0.smithi029.stdout:checking gdbm/ndbm.h presence... no 2024-07-27T06:49:34.390 INFO:tasks.workunit.client.0.smithi029.stdout:checking for gdbm/ndbm.h... no 2024-07-27T06:49:34.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.408+0000 7f1115e62640 1 -- 172.21.15.29:0/3316820879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f111010dff0 msgr2=0x7f111010e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:34.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.408+0000 7f1115e62640 1 --2- 172.21.15.29:0/3316820879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f111010dff0 0x7f111010e400 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1100009900 tx=0x7f110002a410 comp rx=0 tx=0).stop 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.408+0000 7f1115e62640 1 -- 172.21.15.29:0/3316820879 shutdown_connections 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.408+0000 7f1115e62640 1 --2- 172.21.15.29:0/3316820879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1110114230 0x7f111011aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.408+0000 7f1115e62640 1 --2- 172.21.15.29:0/3316820879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11101138e0 0x7f1110113cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.408+0000 7f1115e62640 1 --2- 172.21.15.29:0/3316820879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f111010dff0 0x7f111010e400 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.408+0000 7f1115e62640 1 -- 172.21.15.29:0/3316820879 >> 172.21.15.29:0/3316820879 conn(0x7f1110109df0 msgr2=0x7f111010c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.408+0000 7f1115e62640 1 -- 172.21.15.29:0/3316820879 shutdown_connections 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.408+0000 7f1115e62640 1 -- 172.21.15.29:0/3316820879 wait complete. 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.409+0000 7f1115e62640 1 Processor -- start 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.409+0000 7f1115e62640 1 -- start start 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.409+0000 7f1115e62640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f111010dff0 0x7f11101a48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.409+0000 7f1115e62640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11101138e0 0x7f11101a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.409+0000 7f1115e62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1110114230 0x7f11101aa120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.409+0000 7f1115e62640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f111011def0 con 0x7f111010dff0 2024-07-27T06:49:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.409+0000 7f1115e62640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f111011dd70 con 0x7f11101138e0 2024-07-27T06:49:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.409+0000 7f1115e62640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f111011e070 con 0x7f1110114230 2024-07-27T06:49:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.410+0000 7f110f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f111010dff0 0x7f11101a48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.410+0000 7f110f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f111010dff0 0x7f11101a48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45670/0 (socket says 172.21.15.29:45670) 2024-07-27T06:49:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.410+0000 7f110f7fe640 1 -- 172.21.15.29:0/1349370875 learned_addr learned my addr 172.21.15.29:0/1349370875 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.410+0000 7f1107fff640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11101138e0 0x7f11101a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.410+0000 7f110ffff640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1110114230 0x7f11101aa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.410+0000 7f110f7fe640 1 -- 172.21.15.29:0/1349370875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1110114230 msgr2=0x7f11101aa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.410+0000 7f110f7fe640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1110114230 0x7f11101aa120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.410+0000 7f110f7fe640 1 -- 172.21.15.29:0/1349370875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11101138e0 msgr2=0x7f11101a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.410+0000 7f110f7fe640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11101138e0 0x7f11101a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.410+0000 7f110f7fe640 1 -- 172.21.15.29:0/1349370875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11101aa850 con 0x7f111010dff0 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.410+0000 7f110f7fe640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f111010dff0 0x7f11101a48e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1100005940 tx=0x7f11000042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.410+0000 7f110d7fa640 1 -- 172.21.15.29:0/1349370875 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110003f030 con 0x7f111010dff0 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.411+0000 7f110d7fa640 1 -- 172.21.15.29:0/1349370875 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1100003cb0 con 0x7f111010dff0 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.411+0000 7f110d7fa640 1 -- 172.21.15.29:0/1349370875 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100035070 con 0x7f111010dff0 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.411+0000 7f1115e62640 1 -- 172.21.15.29:0/1349370875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11101aaae0 con 0x7f111010dff0 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.411+0000 7f1115e62640 1 -- 172.21.15.29:0/1349370875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11101b12a0 con 0x7f111010dff0 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.413+0000 7f111c3e6740 1 Processor -- start 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.413+0000 7f111c3e6740 1 -- start start 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f110d7fa640 1 -- 172.21.15.29:0/1349370875 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f1100042030 con 0x7f111010dff0 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f111c3e6740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 0x5631038c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f111c3e6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f111c3e6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f111c3e6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56310383bb90 con 0x5631034ff390 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f111c3e6740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x56310382b090 con 0x563103520070 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f111c3e6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56310382f0d0 con 0x5631034f71b0 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f1107fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f1107fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45680/0 (socket says 172.21.15.29:45680) 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f1107fff640 1 -- 172.21.15.29:0/2057002834 learned_addr learned my addr 172.21.15.29:0/2057002834 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f110f7fe640 1 --2- 172.21.15.29:0/2057002834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f110ffff640 1 --2- 172.21.15.29:0/2057002834 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 0x5631038c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f1107fff640 1 -- 172.21.15.29:0/2057002834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 msgr2=0x5631038c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f1107fff640 1 --2- 172.21.15.29:0/2057002834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f1107fff640 1 -- 172.21.15.29:0/2057002834 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 msgr2=0x5631038c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f1107fff640 1 --2- 172.21.15.29:0/2057002834 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 0x5631038c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f1107fff640 1 -- 172.21.15.29:0/2057002834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631037d1b70 con 0x5631034ff390 2024-07-27T06:49:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.414+0000 7f110f7fe640 1 --2- 172.21.15.29:0/2057002834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.415+0000 7f110d7fa640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f10fc0b3d40 0x7f10fc0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.415+0000 7f1107fff640 1 --2- 172.21.15.29:0/2057002834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038c60f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f10f800bc30 tx=0x7f10f80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec054631ec75a4b0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.415+0000 7f110ffff640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f10fc0b3d40 0x7f10fc0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.415+0000 7f110d7fa640 1 -- 172.21.15.29:0/1349370875 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f1100104840 con 0x7f111010dff0 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.415+0000 7f1105ffb640 1 -- 172.21.15.29:0/2057002834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f801c480 con 0x5631034ff390 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.415+0000 7f1105ffb640 1 -- 172.21.15.29:0/2057002834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f801cad0 con 0x5631034ff390 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.415+0000 7f1105ffb640 1 -- 172.21.15.29:0/2057002834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8011ae0 con 0x5631034ff390 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.416+0000 7f111c3e6740 1 -- 172.21.15.29:0/2057002834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 msgr2=0x5631038c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.416+0000 7f111c3e6740 1 --2- 172.21.15.29:0/2057002834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038c60f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f10f800bc30 tx=0x7f10f80107a0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.416+0000 7f110ffff640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f10fc0b3d40 0x7f10fc0b6250 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1108005fd0 tx=0x7f1108009600 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.416+0000 7f111c3e6740 1 -- 172.21.15.29:0/2057002834 shutdown_connections 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.416+0000 7f111c3e6740 1 --2- 172.21.15.29:0/2057002834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038c60f0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.416+0000 7f111c3e6740 1 --2- 172.21.15.29:0/2057002834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.416+0000 7f111c3e6740 1 --2- 172.21.15.29:0/2057002834 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 0x5631038c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.416+0000 7f111c3e6740 1 -- 172.21.15.29:0/2057002834 >> 172.21.15.29:0/2057002834 conn(0x563103514460 msgr2=0x5631038c3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.416+0000 7f111c3e6740 1 -- 172.21.15.29:0/2057002834 shutdown_connections 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.416+0000 7f111c3e6740 1 -- 172.21.15.29:0/2057002834 wait complete. 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f111c3e6740 1 -- start start 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f111c3e6740 1 Processor -- start 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f111c3e6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038d11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f111c3e6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038d1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f111c3e6740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 0x5631038dd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f111c3e6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x56310368e700 con 0x5631034ff390 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f111c3e6740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5631037cd9c0 con 0x563103520070 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f111c3e6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5631038d3e50 con 0x5631034f71b0 2024-07-27T06:49:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f111c3e6740 10 client.0 _ll_register_callbacks cb 0x5631037e84d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f1107fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038d1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f1107fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038d1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45682/0 (socket says 172.21.15.29:45682) 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f1107fff640 1 -- 172.21.15.29:0/1351746058 learned_addr learned my addr 172.21.15.29:0/1351746058 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.417+0000 7f110f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038d11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f110ffff640 1 --2- 172.21.15.29:0/1351746058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 0x5631038dd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f110f7fe640 1 -- 172.21.15.29:0/1351746058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 msgr2=0x5631038dd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f110f7fe640 1 --2- 172.21.15.29:0/1351746058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 0x5631038dd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f110f7fe640 1 -- 172.21.15.29:0/1351746058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 msgr2=0x5631038d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f110f7fe640 1 --2- 172.21.15.29:0/1351746058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038d1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f110f7fe640 1 -- 172.21.15.29:0/1351746058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56310358b650 con 0x5631034f71b0 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f110f7fe640 1 --2- 172.21.15.29:0/1351746058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038d11e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1100101fa0 tx=0x7f110002a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0a599bacd8cea09 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f10eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:49:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f10eeffd640 1 -- 172.21.15.29:0/1351746058 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100049070 con 0x5631034f71b0 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f10eeffd640 1 -- 172.21.15.29:0/1351746058 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f11000c77a0 con 0x5631034f71b0 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f111c3e6740 1 -- 172.21.15.29:0/1351746058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10f801cad0 con 0x5631034f71b0 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f10eeffd640 1 -- 172.21.15.29:0/1351746058 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f11000cb2e0 con 0x5631034f71b0 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f111c3e6740 10 client.25912 fetch_fsmap learned FSMap version 32 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.418+0000 7f111c3e6740 1 -- 172.21.15.29:0/1351746058 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5631038dfe10 con 0x5631034f71b0 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.419+0000 7f10eeffd640 1 -- 172.21.15.29:0/1351746058 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f801cad0 con 0x5631034f71b0 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.419+0000 7f10eeffd640 1 -- 172.21.15.29:0/1351746058 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 32) ==== 8382+0+0 (secure 0 0 0) 0x7f110003bbf0 con 0x5631034f71b0 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.419+0000 7f111c3e6740 10 client.25912 fetch_fsmap finished waiting for FSMap version 32 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.419+0000 7f111c3e6740 10 client.25912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.419+0000 7f111c3e6740 20 client.25912 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.419+0000 7f111c3e6740 1 --2- 172.21.15.29:0/1351746058 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5631034de9f0 0x5631038e3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.419+0000 7f111c3e6740 4 client.25912 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.419+0000 7f111c3e6740 1 -- 172.21.15.29:0/1351746058 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56310383dae0 con 0x5631034de9f0 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.419+0000 7f1107fff640 1 --2- 172.21.15.29:0/1351746058 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5631034de9f0 0x5631038e3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.420+0000 7f10eeffd640 10 client.25912 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.420+0000 7f1107fff640 1 --2- 172.21.15.29:0/1351746058 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5631034de9f0 0x5631038e3330 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce4e960dd9867eb server_cookie=d6ef211b7f141168 in_seq=0 out_seq=0 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.420+0000 7f10eeffd640 1 -- 172.21.15.29:0/1351746058 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56310383dae0 con 0x5631034de9f0 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.420+0000 7f10eeffd640 10 client.25912 handle_command_reply: tid=0 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.420+0000 7f111c3e6740 1 client.25912 shutdown 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.420+0000 7f111c3e6740 1 -- 172.21.15.29:0/1351746058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 msgr2=0x5631038d11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.420+0000 7f111c3e6740 1 --2- 172.21.15.29:0/1351746058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038d11e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1100101fa0 tx=0x7f110002a410 comp rx=0 tx=0).stop 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.421+0000 7f111c3e6740 1 -- 172.21.15.29:0/1351746058 shutdown_connections 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.421+0000 7f111c3e6740 1 --2- 172.21.15.29:0/1351746058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 0x5631038dd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.421+0000 7f111c3e6740 1 --2- 172.21.15.29:0/1351746058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038d1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.421+0000 7f111c3e6740 1 --2- 172.21.15.29:0/1351746058 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5631034de9f0 0x5631038e3330 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.421+0000 7f111c3e6740 1 --2- 172.21.15.29:0/1351746058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038d11e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.421+0000 7f10eeffd640 0 client.25912 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:34.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.421+0000 7f111c3e6740 1 -- 172.21.15.29:0/1351746058 >> 172.21.15.29:0/1351746058 conn(0x563103514460 msgr2=0x5631038c8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:34.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.421+0000 7f111c3e6740 1 -- 172.21.15.29:0/1351746058 shutdown_connections 2024-07-27T06:49:34.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.421+0000 7f111c3e6740 1 -- 172.21.15.29:0/1351746058 wait complete. 2024-07-27T06:49:34.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.421+0000 7f111c3e6740 20 client.25912 trim_cache size 0 max 16384 2024-07-27T06:49:34.445 INFO:tasks.workunit.client.0.smithi029.stdout:checking ndbm.h usability... no 2024-07-27T06:49:34.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.439+0000 7f111c3e6740 1 Processor -- start 2024-07-27T06:49:34.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.439+0000 7f111c3e6740 1 -- start start 2024-07-27T06:49:34.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.439+0000 7f111c3e6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038c23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.439+0000 7f111c3e6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.439+0000 7f111c3e6740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 0x5631038cae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.439+0000 7f111c3e6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f11101b9b70 con 0x5631034ff390 2024-07-27T06:49:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.439+0000 7f111c3e6740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f11101bb7a0 con 0x563103520070 2024-07-27T06:49:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.439+0000 7f111c3e6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f11101ba240 con 0x5631034f71b0 2024-07-27T06:49:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.439+0000 7f1107fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.439+0000 7f1107fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45694/0 (socket says 172.21.15.29:45694) 2024-07-27T06:49:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.439+0000 7f1107fff640 1 -- 172.21.15.29:0/793060404 learned_addr learned my addr 172.21.15.29:0/793060404 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.439+0000 7f110f7fe640 1 --2- 172.21.15.29:0/793060404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038c23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.440+0000 7f1107fff640 1 -- 172.21.15.29:0/793060404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 msgr2=0x5631038c23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.440+0000 7f1107fff640 1 --2- 172.21.15.29:0/793060404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038c23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.440+0000 7f1107fff640 1 -- 172.21.15.29:0/793060404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 msgr2=0x5631038cae30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.440+0000 7f1107fff640 1 --2- 172.21.15.29:0/793060404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 0x5631038cae30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.440+0000 7f1107fff640 1 -- 172.21.15.29:0/793060404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631034dec20 con 0x5631034ff390 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.440+0000 7f1107fff640 1 --2- 172.21.15.29:0/793060404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038ca8f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f10f800bb50 tx=0x7f10f800bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26af1efd4b7b171c server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.440+0000 7f11057fa640 1 -- 172.21.15.29:0/793060404 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8004020 con 0x5631034ff390 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.440+0000 7f11057fa640 1 -- 172.21.15.29:0/793060404 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f80054f0 con 0x5631034ff390 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.440+0000 7f11057fa640 1 -- 172.21.15.29:0/793060404 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f801c6c0 con 0x5631034ff390 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.441+0000 7f111c3e6740 1 -- 172.21.15.29:0/793060404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 msgr2=0x5631038ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.441+0000 7f111c3e6740 1 --2- 172.21.15.29:0/793060404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038ca8f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f10f800bb50 tx=0x7f10f800bf70 comp rx=0 tx=0).stop 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.442+0000 7f111c3e6740 1 -- 172.21.15.29:0/793060404 shutdown_connections 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.442+0000 7f111c3e6740 1 --2- 172.21.15.29:0/793060404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x563103520070 0x5631038cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.442+0000 7f111c3e6740 1 --2- 172.21.15.29:0/793060404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5631034ff390 0x5631038ca8f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.442+0000 7f111c3e6740 1 --2- 172.21.15.29:0/793060404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034f71b0 0x5631038c23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.442+0000 7f111c3e6740 1 -- 172.21.15.29:0/793060404 >> 172.21.15.29:0/793060404 conn(0x563103514460 msgr2=0x5631038c1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.442+0000 7f111c3e6740 1 -- 172.21.15.29:0/793060404 shutdown_connections 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.442+0000 7f111c3e6740 1 -- 172.21.15.29:0/793060404 wait complete. 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.443+0000 7f111c3e6740 1 -- start start 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.443+0000 7f111c3e6740 1 Processor -- start 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.443+0000 7f111c3e6740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5631034f71b0 0x5631038d9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.443+0000 7f111c3e6740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034ff390 0x5631038dc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.443+0000 7f111c3e6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563103520070 0x5631038dccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.443+0000 7f111c3e6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f11101bb010 con 0x563103520070 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.443+0000 7f111c3e6740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f11101ba910 con 0x5631034f71b0 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.443+0000 7f111c3e6740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f11101b7b50 con 0x5631034ff390 2024-07-27T06:49:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.443+0000 7f111c3e6740 10 client.0 _ll_register_callbacks cb 0x7f1100005780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f110f7fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5631034f71b0 0x5631038d9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f110f7fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5631034f71b0 0x5631038d9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.29:35574/0 (socket says 172.21.15.29:35574) 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f110f7fe640 1 -- 172.21.15.29:0/3378705976 learned_addr learned my addr 172.21.15.29:0/3378705976 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f1107fff640 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034ff390 0x5631038dc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f110ffff640 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563103520070 0x5631038dccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f110f7fe640 1 -- 172.21.15.29:0/3378705976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034ff390 msgr2=0x5631038dc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f110f7fe640 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034ff390 0x5631038dc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f110f7fe640 1 -- 172.21.15.29:0/3378705976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563103520070 msgr2=0x5631038dccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f110f7fe640 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563103520070 0x5631038dccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f110f7fe640 1 -- 172.21.15.29:0/3378705976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631038dd280 con 0x5631034f71b0 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f110ffff640 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563103520070 0x5631038dccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f110f7fe640 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5631034f71b0 0x5631038d9560 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f11000c7590 tx=0x7f11000c79d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57463cc82b698b94 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.444+0000 7f10eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.47:3300/0 2024-07-27T06:49:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.445+0000 7f10eeffd640 1 -- 172.21.15.29:0/3378705976 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100049070 con 0x5631034f71b0 2024-07-27T06:49:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.445+0000 7f10eeffd640 1 -- 172.21.15.29:0/3378705976 <== mon.2 v2:172.21.15.47:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f11000cb7c0 con 0x5631034f71b0 2024-07-27T06:49:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.445+0000 7f111c3e6740 1 -- 172.21.15.29:0/3378705976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10f80054f0 con 0x5631034f71b0 2024-07-27T06:49:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.445+0000 7f10eeffd640 1 -- 172.21.15.29:0/3378705976 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1100003e60 con 0x5631034f71b0 2024-07-27T06:49:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.445+0000 7f111c3e6740 10 client.25742 fetch_fsmap learned FSMap version 32 2024-07-27T06:49:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.445+0000 7f111c3e6740 1 -- 172.21.15.29:0/3378705976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5631038dd280 con 0x5631034f71b0 2024-07-27T06:49:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.445+0000 7f10eeffd640 1 -- 172.21.15.29:0/3378705976 <== mon.2 v2:172.21.15.47:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f80054f0 con 0x5631034f71b0 2024-07-27T06:49:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.445+0000 7f10eeffd640 1 -- 172.21.15.29:0/3378705976 <== mon.2 v2:172.21.15.47:3300/0 5 ==== fsmap(e 32) ==== 8382+0+0 (secure 0 0 0) 0x7f11000d4080 con 0x5631034f71b0 2024-07-27T06:49:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.446+0000 7f111c3e6740 10 client.25742 fetch_fsmap finished waiting for FSMap version 32 2024-07-27T06:49:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.446+0000 7f111c3e6740 10 client.25742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.i 2024-07-27T06:49:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.446+0000 7f111c3e6740 20 client.25742 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:34.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.446+0000 7f111c3e6740 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5631034b3da0 0x56310394dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.446+0000 7f111c3e6740 4 client.25742 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.446+0000 7f111c3e6740 1 -- 172.21.15.29:0/3378705976 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563103839580 con 0x5631034b3da0 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.446+0000 7f1107fff640 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5631034b3da0 0x56310394dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.446+0000 7f10eeffd640 10 client.25742 ms_handle_connect on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.446+0000 7f1107fff640 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5631034b3da0 0x56310394dce0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63a0e388a0f24d65 server_cookie=9b982f805a4ee7ca in_seq=0 out_seq=0 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f10eeffd640 1 -- 172.21.15.29:0/3378705976 <== mds.0 v2:172.21.15.77:6838/269899139 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563103839580 con 0x5631034b3da0 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f10eeffd640 10 client.25742 handle_command_reply: tid=0 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f111c3e6740 1 client.25742 shutdown 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f111c3e6740 1 -- 172.21.15.29:0/3378705976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5631034f71b0 msgr2=0x5631038d9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f111c3e6740 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5631034f71b0 0x5631038d9560 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f11000c7590 tx=0x7f11000c79d0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f111c3e6740 1 -- 172.21.15.29:0/3378705976 shutdown_connections 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f111c3e6740 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x563103520070 0x5631038dccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f111c3e6740 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5631034b3da0 0x56310394dce0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f111c3e6740 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5631034ff390 0x5631038dc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f111c3e6740 1 --2- 172.21.15.29:0/3378705976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5631034f71b0 0x5631038d9560 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f111c3e6740 1 -- 172.21.15.29:0/3378705976 >> 172.21.15.29:0/3378705976 conn(0x563103514460 msgr2=0x5631038c7060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f10eeffd640 0 client.25742 ms_handle_reset on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f111c3e6740 1 -- 172.21.15.29:0/3378705976 shutdown_connections 2024-07-27T06:49:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f111c3e6740 1 -- 172.21.15.29:0/3378705976 wait complete. 2024-07-27T06:49:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.447+0000 7f111c3e6740 20 client.25742 trim_cache size 0 max 16384 2024-07-27T06:49:34.456 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:49:34.456 INFO:teuthology.orchestra.run.smithi029.stdout: "status": "no active scrubs running", 2024-07-27T06:49:34.456 INFO:teuthology.orchestra.run.smithi029.stdout: "scrubs": {} 2024-07-27T06:49:34.457 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:49:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.448+0000 7f1115e62640 1 -- 172.21.15.29:0/1349370875 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f10fc0b3d40 msgr2=0x7f10fc0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.448+0000 7f1115e62640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f10fc0b3d40 0x7f10fc0b6250 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1108005fd0 tx=0x7f1108009600 comp rx=0 tx=0).stop 2024-07-27T06:49:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.448+0000 7f1115e62640 1 -- 172.21.15.29:0/1349370875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f111010dff0 msgr2=0x7f11101a48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.448+0000 7f1115e62640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f111010dff0 0x7f11101a48e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1100005940 tx=0x7f11000042f0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.448+0000 7f1115e62640 1 -- 172.21.15.29:0/1349370875 shutdown_connections 2024-07-27T06:49:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.448+0000 7f1115e62640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f10fc0b3d40 0x7f10fc0b6250 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.448+0000 7f1115e62640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1110114230 0x7f11101aa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.448+0000 7f1115e62640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11101138e0 0x7f11101a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.448+0000 7f1115e62640 1 --2- 172.21.15.29:0/1349370875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f111010dff0 0x7f11101a48e0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.448+0000 7f1115e62640 1 -- 172.21.15.29:0/1349370875 >> 172.21.15.29:0/1349370875 conn(0x7f1110109df0 msgr2=0x7f1110112f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.448+0000 7f1115e62640 1 -- 172.21.15.29:0/1349370875 shutdown_connections 2024-07-27T06:49:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:34.448+0000 7f1115e62640 1 -- 172.21.15.29:0/1349370875 wait complete. 2024-07-27T06:49:34.465 INFO:tasks.workunit.client.0.smithi029.stdout:checking ndbm.h presence... no 2024-07-27T06:49:34.465 INFO:tasks.workunit.client.0.smithi029.stdout:checking for ndbm.h... no 2024-07-27T06:49:34.628 INFO:tasks.workunit.client.0.smithi029.stdout:checking libaio.h usability... yes 2024-07-27T06:49:34.647 INFO:tasks.workunit.client.0.smithi029.stdout:checking libaio.h presence... yes 2024-07-27T06:49:34.647 INFO:tasks.workunit.client.0.smithi029.stdout:checking for libaio.h... yes 2024-07-27T06:49:34.777 INFO:tasks.workunit.client.0.smithi029.stdout:checking liburing.h usability... no 2024-07-27T06:49:34.803 INFO:tasks.workunit.client.0.smithi029.stdout:checking liburing.h presence... no 2024-07-27T06:49:34.803 INFO:tasks.workunit.client.0.smithi029.stdout:checking for liburing.h... no 2024-07-27T06:49:35.042 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/capability.h usability... no 2024-07-27T06:49:35.071 INFO:tasks.workunit.client.0.smithi029.stdout:checking sys/capability.h presence... no 2024-07-27T06:49:35.071 INFO:tasks.workunit.client.0.smithi029.stdout:checking for sys/capability.h... no 2024-07-27T06:49:35.296 INFO:tasks.workunit.client.0.smithi029.stdout:checking linux/fiemap.h usability... yes 2024-07-27T06:49:35.312 INFO:tasks.workunit.client.0.smithi029.stdout:checking linux/fiemap.h presence... yes 2024-07-27T06:49:35.312 INFO:tasks.workunit.client.0.smithi029.stdout:checking for linux/fiemap.h... yes 2024-07-27T06:49:35.375 INFO:tasks.workunit.client.0.smithi029.stdout:checking for fallocate... yes 2024-07-27T06:49:35.427 INFO:tasks.workunit.client.0.smithi029.stdout:checking for open_by_handle_at... yes 2024-07-27T06:49:35.465 INFO:tasks.workunit.client.0.smithi029.stdout:checking linux/fs.h usability... yes 2024-07-27T06:49:35.482 INFO:tasks.workunit.client.0.smithi029.stdout:checking linux/fs.h presence... yes 2024-07-27T06:49:35.482 INFO:tasks.workunit.client.0.smithi029.stdout:checking for linux/fs.h... yes 2024-07-27T06:49:35.549 INFO:tasks.workunit.client.0.smithi029.stdout:checking btrfsutil.h usability... no 2024-07-27T06:49:35.578 INFO:tasks.workunit.client.0.smithi029.stdout:checking btrfsutil.h presence... no 2024-07-27T06:49:35.578 INFO:tasks.workunit.client.0.smithi029.stdout:checking for btrfsutil.h... no 2024-07-27T06:49:35.655 INFO:tasks.workunit.client.0.smithi029.stdout:checking for copy_file_range... yes 2024-07-27T06:49:35.700 INFO:tasks.workunit.client.0.smithi029.stdout:checking for SEEK_DATA... yes 2024-07-27T06:49:35.766 INFO:tasks.workunit.client.0.smithi029.stdout:checking for BMV_OF_SHARED... yes 2024-07-27T06:49:35.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:35 smithi029 ceph-mon[32610]: pgmap v474: 97 pgs: 97 active+clean; 33 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 665 op/s 2024-07-27T06:49:35.819 INFO:tasks.workunit.client.0.smithi029.stdout:checking for nftw... yes 2024-07-27T06:49:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:35 smithi077 ceph-mon[37153]: pgmap v474: 97 pgs: 97 active+clean; 33 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 665 op/s 2024-07-27T06:49:35.903 INFO:tasks.workunit.client.0.smithi029.stdout:checking for RLIMIT_NOFILE... yes 2024-07-27T06:49:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:35 smithi047 ceph-mon[36835]: pgmap v474: 97 pgs: 97 active+clean; 33 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 665 op/s 2024-07-27T06:49:35.963 INFO:tasks.workunit.client.0.smithi029.stdout:checking for XFS_IOC_EXCHANGE_RANGE... no 2024-07-27T06:49:36.029 INFO:tasks.workunit.client.0.smithi029.stdout:checking for FICLONE... yes 2024-07-27T06:49:36.078 INFO:tasks.workunit.client.0.smithi029.stdout:checking for renameat2... yes 2024-07-27T06:49:36.121 INFO:tasks.workunit.client.0.smithi029.stdout:checking for reallocarray... yes 2024-07-27T06:49:36.197 INFO:tasks.workunit.client.0.smithi029.stdout:checking for struct mount_attr... yes 2024-07-27T06:49:36.278 INFO:tasks.workunit.client.0.smithi029.stdout:checking for struct btrfs_qgroup_limit... yes 2024-07-27T06:49:36.363 INFO:tasks.workunit.client.0.smithi029.stdout:checking for struct btrfs_qgroup_inherit... yes 2024-07-27T06:49:36.386 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-27T06:49:36.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:49:36.428 INFO:tasks.workunit.client.0.smithi029.stdout:checking for struct btrfs_ioctl_vol_args... yes 2024-07-27T06:49:36.487 INFO:tasks.workunit.client.0.smithi029.stdout:checking for struct btrfs_ioctl_vol_args_v2... yes 2024-07-27T06:49:36.546 INFO:tasks.workunit.client.0.smithi029.stdout:checking for struct btrfs_ioctl_ino_lookup_args... yes 2024-07-27T06:49:36.605 INFO:tasks.workunit.client.0.smithi029.stdout:checking for struct btrfs_ioctl_ino_lookup_user_args... yes 2024-07-27T06:49:36.667 INFO:tasks.workunit.client.0.smithi029.stdout:checking for struct btrfs_ioctl_get_subvol_rootref_args... yes 2024-07-27T06:49:36.705 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:49:36.709 INFO:tasks.workunit.client.0.smithi029.stdout:checking linux/btrfs.h usability... yes 2024-07-27T06:49:36.726 INFO:tasks.workunit.client.0.smithi029.stdout:checking linux/btrfs.h presence... yes 2024-07-27T06:49:36.726 INFO:tasks.workunit.client.0.smithi029.stdout:checking for linux/btrfs.h... yes 2024-07-27T06:49:36.789 INFO:tasks.workunit.client.0.smithi029.stdout:checking linux/btrfs_tree.h usability... yes 2024-07-27T06:49:36.807 INFO:tasks.workunit.client.0.smithi029.stdout:checking linux/btrfs_tree.h presence... yes 2024-07-27T06:49:36.807 INFO:tasks.workunit.client.0.smithi029.stdout:checking for linux/btrfs_tree.h... yes 2024-07-27T06:49:36.863 INFO:tasks.workunit.client.0.smithi029.stdout:checking for struct btrfs_ioctl_vol_args_v2.subvolid... yes 2024-07-27T06:49:36.906 INFO:tasks.workunit.client.0.smithi029.stdout:checking whether BTRFS_IOC_SNAP_DESTROY_V2 is declared... yes 2024-07-27T06:49:36.926 INFO:tasks.workunit.client.0.smithi029.stdout:configure: creating ./config.status 2024-07-27T06:49:37.277 INFO:tasks.workunit.client.0.smithi029.stdout:config.status: creating include/builddefs 2024-07-27T06:49:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-mon[37153]: pgmap v475: 97 pgs: 97 active+clean; 35 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 2.0 MiB/s wr, 685 op/s 2024-07-27T06:49:37.607 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:49:37.607 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: *** Caught signal (Aborted) ** 2024-07-27T06:49:37.607 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: in thread 7fd180281640 thread_name:ms_dispatch 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd1856586f0] 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd1856a594c] 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 3: raise() 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 4: abort() 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd18564271b] 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 6: /lib64/libc.so.6(+0x37386) [0x7fd185651386] 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5563b9df86b9] 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5563b9fb680c] 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5563b9f04cde] 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5563b9ec6763] 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5563b9ec7a1b] 2024-07-27T06:49:37.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5563b9eab752] 2024-07-27T06:49:37.609 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5563b9df9a56] 2024-07-27T06:49:37.609 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5563b9dfd637] 2024-07-27T06:49:37.609 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5563b9dfdf5e] 2024-07-27T06:49:37.609 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5563b9de103a] 2024-07-27T06:49:37.609 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd185f25407] 2024-07-27T06:49:37.609 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 18: (DispatchQueue::entry()+0x60f) [0x7fd185f25cf9] 2024-07-27T06:49:37.609 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd1860025a1] 2024-07-27T06:49:37.610 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 20: (Thread::entry_wrapper()+0x43) [0x7fd185dafcdd] 2024-07-27T06:49:37.610 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd185dafcf9] 2024-07-27T06:49:37.610 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd1856a3c02] 2024-07-27T06:49:37.611 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd185728c40] 2024-07-27T06:49:37.611 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2024-07-27T06:49:37.299+0000 7fd180281640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:49:37.611 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: in thread 7fd180281640 thread_name:ms_dispatch 2024-07-27T06:49:37.611 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2024-07-27T06:49:37.611 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:49:37.611 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd1856586f0] 2024-07-27T06:49:37.611 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd1856a594c] 2024-07-27T06:49:37.611 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 3: raise() 2024-07-27T06:49:37.612 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 4: abort() 2024-07-27T06:49:37.612 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd18564271b] 2024-07-27T06:49:37.612 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 6: /lib64/libc.so.6(+0x37386) [0x7fd185651386] 2024-07-27T06:49:37.612 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5563b9df86b9] 2024-07-27T06:49:37.612 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5563b9fb680c] 2024-07-27T06:49:37.612 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5563b9f04cde] 2024-07-27T06:49:37.613 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5563b9ec6763] 2024-07-27T06:49:37.613 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5563b9ec7a1b] 2024-07-27T06:49:37.613 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5563b9eab752] 2024-07-27T06:49:37.613 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5563b9df9a56] 2024-07-27T06:49:37.613 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5563b9dfd637] 2024-07-27T06:49:37.613 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5563b9dfdf5e] 2024-07-27T06:49:37.613 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5563b9de103a] 2024-07-27T06:49:37.614 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd185f25407] 2024-07-27T06:49:37.614 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 18: (DispatchQueue::entry()+0x60f) [0x7fd185f25cf9] 2024-07-27T06:49:37.614 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd1860025a1] 2024-07-27T06:49:37.614 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 20: (Thread::entry_wrapper()+0x43) [0x7fd185dafcdd] 2024-07-27T06:49:37.614 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd185dafcf9] 2024-07-27T06:49:37.614 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd1856a3c02] 2024-07-27T06:49:37.614 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd185728c40] 2024-07-27T06:49:37.615 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:49:37.615 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2024-07-27T06:49:37.615 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 0> 2024-07-27T06:49:37.299+0000 7fd180281640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:49:37.615 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: in thread 7fd180281640 thread_name:ms_dispatch 2024-07-27T06:49:37.615 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2024-07-27T06:49:37.615 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:49:37.615 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd1856586f0] 2024-07-27T06:49:37.615 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd1856a594c] 2024-07-27T06:49:37.616 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 3: raise() 2024-07-27T06:49:37.616 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 4: abort() 2024-07-27T06:49:37.617 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd18564271b] 2024-07-27T06:49:37.617 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 6: /lib64/libc.so.6(+0x37386) [0x7fd185651386] 2024-07-27T06:49:37.618 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5563b9df86b9] 2024-07-27T06:49:37.618 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5563b9fb680c] 2024-07-27T06:49:37.618 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5563b9f04cde] 2024-07-27T06:49:37.618 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5563b9ec6763] 2024-07-27T06:49:37.618 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5563b9ec7a1b] 2024-07-27T06:49:37.618 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5563b9eab752] 2024-07-27T06:49:37.618 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5563b9df9a56] 2024-07-27T06:49:37.618 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5563b9dfd637] 2024-07-27T06:49:37.618 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5563b9dfdf5e] 2024-07-27T06:49:37.618 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5563b9de103a] 2024-07-27T06:49:37.618 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd185f25407] 2024-07-27T06:49:37.619 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 18: (DispatchQueue::entry()+0x60f) [0x7fd185f25cf9] 2024-07-27T06:49:37.619 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd1860025a1] 2024-07-27T06:49:37.619 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 20: (Thread::entry_wrapper()+0x43) [0x7fd185dafcdd] 2024-07-27T06:49:37.619 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd185dafcf9] 2024-07-27T06:49:37.619 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd1856a3c02] 2024-07-27T06:49:37.619 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd185728c40] 2024-07-27T06:49:37.619 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:49:37.620 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2024-07-27T06:49:37.620 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 0> 2024-07-27T06:49:37.299+0000 7fd180281640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:49:37.620 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: in thread 7fd180281640 thread_name:ms_dispatch 2024-07-27T06:49:37.620 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2024-07-27T06:49:37.620 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:49:37.620 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd1856586f0] 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd1856a594c] 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 3: raise() 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 4: abort() 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd18564271b] 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 6: /lib64/libc.so.6(+0x37386) [0x7fd185651386] 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5563b9df86b9] 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5563b9fb680c] 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5563b9f04cde] 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5563b9ec6763] 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5563b9ec7a1b] 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5563b9eab752] 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5563b9df9a56] 2024-07-27T06:49:37.621 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5563b9dfd637] 2024-07-27T06:49:37.622 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5563b9dfdf5e] 2024-07-27T06:49:37.622 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5563b9de103a] 2024-07-27T06:49:37.622 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd185f25407] 2024-07-27T06:49:37.622 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 18: (DispatchQueue::entry()+0x60f) [0x7fd185f25cf9] 2024-07-27T06:49:37.622 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd1860025a1] 2024-07-27T06:49:37.622 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 20: (Thread::entry_wrapper()+0x43) [0x7fd185dafcdd] 2024-07-27T06:49:37.622 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd185dafcf9] 2024-07-27T06:49:37.622 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd1856a3c02] 2024-07-27T06:49:37.622 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd185728c40] 2024-07-27T06:49:37.623 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:49:37.623 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[69361]: 2024-07-27T06:49:37.623 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:37 smithi077 podman[71774]: 2024-07-27 06:49:37.359195466 +0000 UTC m=+0.020277343 container died 12d1fea815231b4a7ee0b98aa2eb7e4c055b4cb0bf7889434f6769af724287fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-27T06:49:37.831 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:37 smithi029 ceph-mon[32610]: pgmap v475: 97 pgs: 97 active+clean; 35 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 2.0 MiB/s wr, 685 op/s 2024-07-27T06:49:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:37 smithi047 ceph-mon[36835]: pgmap v475: 97 pgs: 97 active+clean; 35 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 2.0 MiB/s wr, 685 op/s 2024-07-27T06:49:38.607 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:38 smithi077 podman[71774]: 2024-07-27 06:49:38.109523442 +0000 UTC m=+0.770605307 container remove 12d1fea815231b4a7ee0b98aa2eb7e4c055b4cb0bf7889434f6769af724287fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-27T06:49:38.607 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:38 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:49:38.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:38 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-27T06:49:38.608 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:38 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Consumed 30.763s CPU time. 2024-07-27T06:49:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.607+0000 7fb38066c640 1 -- 172.21.15.29:0/1628100513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3781035b0 msgr2=0x7fb3781039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.607+0000 7fb38066c640 1 --2- 172.21.15.29:0/1628100513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3781035b0 0x7fb3781039c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb374009900 tx=0x7fb37402a410 comp rx=0 tx=0).stop 2024-07-27T06:49:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.607+0000 7fb38066c640 1 -- 172.21.15.29:0/1628100513 shutdown_connections 2024-07-27T06:49:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.607+0000 7fb38066c640 1 --2- 172.21.15.29:0/1628100513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb378109840 0x7fb378110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.607+0000 7fb38066c640 1 --2- 172.21.15.29:0/1628100513 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb378108ef0 0x7fb378109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.607+0000 7fb38066c640 1 --2- 172.21.15.29:0/1628100513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3781035b0 0x7fb3781039c0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.607+0000 7fb38066c640 1 -- 172.21.15.29:0/1628100513 >> 172.21.15.29:0/1628100513 conn(0x7fb3780ff040 msgr2=0x7fb378101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.607+0000 7fb38066c640 1 -- 172.21.15.29:0/1628100513 shutdown_connections 2024-07-27T06:49:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.607+0000 7fb38066c640 1 -- 172.21.15.29:0/1628100513 wait complete. 2024-07-27T06:49:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.608+0000 7fb38066c640 1 Processor -- start 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.608+0000 7fb38066c640 1 -- start start 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.608+0000 7fb38066c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3781035b0 0x7fb378076b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.608+0000 7fb38066c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb378108ef0 0x7fb378075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.608+0000 7fb38066c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb378109840 0x7fb378075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.608+0000 7fb38066c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb3781134f0 con 0x7fb378109840 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.608+0000 7fb38066c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fb378113370 con 0x7fb3781035b0 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.608+0000 7fb38066c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb378113670 con 0x7fb378108ef0 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb37ec09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb378109840 0x7fb378075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb37ec09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb378109840 0x7fb378075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45714/0 (socket says 172.21.15.29:45714) 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb37ec09640 1 -- 172.21.15.29:0/2389696983 learned_addr learned my addr 172.21.15.29:0/2389696983 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb37e408640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3781035b0 0x7fb378076b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb37dc07640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb378108ef0 0x7fb378075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb37ec09640 1 -- 172.21.15.29:0/2389696983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb378108ef0 msgr2=0x7fb378075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb37ec09640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb378108ef0 0x7fb378075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb37ec09640 1 -- 172.21.15.29:0/2389696983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3781035b0 msgr2=0x7fb378076b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb37ec09640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3781035b0 0x7fb378076b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb37ec09640 1 -- 172.21.15.29:0/2389696983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb378075e10 con 0x7fb378109840 2024-07-27T06:49:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb37e408640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3781035b0 0x7fb378076b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb37ec09640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb378109840 0x7fb378075680 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb37000eb10 tx=0x7fb37000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb36f7fe640 1 -- 172.21.15.29:0/2389696983 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb370004b20 con 0x7fb378109840 2024-07-27T06:49:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb36f7fe640 1 -- 172.21.15.29:0/2389696983 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb370004cc0 con 0x7fb378109840 2024-07-27T06:49:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.609+0000 7fb38066c640 1 -- 172.21.15.29:0/2389696983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3781ac840 con 0x7fb378109840 2024-07-27T06:49:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.610+0000 7fb36f7fe640 1 -- 172.21.15.29:0/2389696983 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb370010670 con 0x7fb378109840 2024-07-27T06:49:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.610+0000 7fb38066c640 1 -- 172.21.15.29:0/2389696983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3781acd50 con 0x7fb378109840 2024-07-27T06:49:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.611+0000 7fb38066c640 1 -- 172.21.15.29:0/2389696983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb378104830 con 0x7fb378109840 2024-07-27T06:49:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.613+0000 7fb36f7fe640 1 -- 172.21.15.29:0/2389696983 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fb370005000 con 0x7fb378109840 2024-07-27T06:49:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.613+0000 7fb36f7fe640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb3640b3dc0 0x7fb3640b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.614+0000 7fb36f7fe640 1 -- 172.21.15.29:0/2389696983 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7fb3700e48a0 con 0x7fb378109840 2024-07-27T06:49:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.614+0000 7fb37e408640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb3640b3dc0 0x7fb3640b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.614+0000 7fb37e408640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb3640b3dc0 0x7fb3640b62d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb374000c00 tx=0x7fb374009680 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:38.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.616+0000 7fb36f7fe640 1 -- 172.21.15.29:0/2389696983 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb370055600 con 0x7fb378109840 2024-07-27T06:49:38.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.838+0000 7fb38066c640 1 -- 172.21.15.29:0/2389696983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb37802e0c0 con 0x7fb378109840 2024-07-27T06:49:38.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.839+0000 7fb36f7fe640 1 -- 172.21.15.29:0/2389696983 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11965 (secure 0 0 0) 0x7fb3700abbc0 con 0x7fb378109840 2024-07-27T06:49:38.841 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:49:38.841 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"btime":"2024-07-27T06:49:22:056950+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:49:21.638183+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24670,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24584":{"gid":24584,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6837/2573307312","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":2573307312},{"type":"v1","addr":"172.21.15.47:6837","nonce":2573307312}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24670":{"gid":24670,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.77:6839/269899139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":269899139},{"type":"v1","addr":"172.21.15.77:6839","nonce":269899139}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24670,"qdb_cluster":[24613,24617,24646,24658,24670]},"id":1}]} 2024-07-27T06:49:38.841 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 32 2024-07-27T06:49:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.841+0000 7fb38066c640 1 -- 172.21.15.29:0/2389696983 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb3640b3dc0 msgr2=0x7fb3640b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.841+0000 7fb38066c640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb3640b3dc0 0x7fb3640b62d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb374000c00 tx=0x7fb374009680 comp rx=0 tx=0).stop 2024-07-27T06:49:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.841+0000 7fb38066c640 1 -- 172.21.15.29:0/2389696983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb378109840 msgr2=0x7fb378075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.841+0000 7fb38066c640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb378109840 0x7fb378075680 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb37000eb10 tx=0x7fb37000c2d0 comp rx=0 tx=0).stop 2024-07-27T06:49:38.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.841+0000 7fb38066c640 1 -- 172.21.15.29:0/2389696983 shutdown_connections 2024-07-27T06:49:38.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.841+0000 7fb38066c640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb378109840 0x7fb378075680 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:38.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.841+0000 7fb38066c640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fb3640b3dc0 0x7fb3640b62d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:38.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.841+0000 7fb38066c640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb378108ef0 0x7fb378075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:38.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.841+0000 7fb38066c640 1 --2- 172.21.15.29:0/2389696983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3781035b0 0x7fb378076b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:38.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.841+0000 7fb38066c640 1 -- 172.21.15.29:0/2389696983 >> 172.21.15.29:0/2389696983 conn(0x7fb3780ff040 msgr2=0x7fb378101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:38.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.841+0000 7fb38066c640 1 -- 172.21.15.29:0/2389696983 shutdown_connections 2024-07-27T06:49:38.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:38.841+0000 7fb38066c640 1 -- 172.21.15.29:0/2389696983 wait complete. 2024-07-27T06:49:39.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-27T06:49:39.817 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:39 smithi029 ceph-mon[32610]: pgmap v476: 97 pgs: 97 active+clean; 39 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 1.4 MiB/s wr, 543 op/s 2024-07-27T06:49:39.817 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:39 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/2389696983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:49:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:39 smithi077 ceph-mon[37153]: pgmap v476: 97 pgs: 97 active+clean; 39 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 1.4 MiB/s wr, 543 op/s 2024-07-27T06:49:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:39 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/2389696983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:49:39.860 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:49:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:39 smithi047 ceph-mon[36835]: pgmap v476: 97 pgs: 97 active+clean; 39 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 1.4 MiB/s wr, 543 op/s 2024-07-27T06:49:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:39 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/2389696983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:49:41.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:41 smithi029 ceph-mon[32610]: pgmap v477: 97 pgs: 97 active+clean; 39 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 864 KiB/s wr, 362 op/s 2024-07-27T06:49:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.753+0000 7f244111f640 1 -- 172.21.15.29:0/2866355690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f243c108ed0 msgr2=0x7f243c1092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.753+0000 7f244111f640 1 --2- 172.21.15.29:0/2866355690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f243c108ed0 0x7f243c1092e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2430009900 tx=0x7f243002a410 comp rx=0 tx=0).stop 2024-07-27T06:49:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.753+0000 7f244111f640 1 -- 172.21.15.29:0/2866355690 shutdown_connections 2024-07-27T06:49:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.753+0000 7f244111f640 1 --2- 172.21.15.29:0/2866355690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f243c109820 0x7f243c110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.753+0000 7f244111f640 1 --2- 172.21.15.29:0/2866355690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f243c108ed0 0x7f243c1092e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.753+0000 7f244111f640 1 --2- 172.21.15.29:0/2866355690 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f243c103590 0x7f243c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.753+0000 7f244111f640 1 -- 172.21.15.29:0/2866355690 >> 172.21.15.29:0/2866355690 conn(0x7f243c07b280 msgr2=0x7f243c07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.753+0000 7f244111f640 1 -- 172.21.15.29:0/2866355690 shutdown_connections 2024-07-27T06:49:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.753+0000 7f244111f640 1 -- 172.21.15.29:0/2866355690 wait complete. 2024-07-27T06:49:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.754+0000 7f244111f640 1 Processor -- start 2024-07-27T06:49:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.754+0000 7f244111f640 1 -- start start 2024-07-27T06:49:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.754+0000 7f244111f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f243c103590 0x7f243c199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.754+0000 7f244111f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f243c108ed0 0x7f243c19a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.754+0000 7f244111f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f243c109820 0x7f243c19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f244111f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f243c113490 con 0x7f243c103590 2024-07-27T06:49:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f244111f640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f243c113310 con 0x7f243c108ed0 2024-07-27T06:49:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f244111f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f243c113610 con 0x7f243c109820 2024-07-27T06:49:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f243ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f243c103590 0x7f243c199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f243ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f243c103590 0x7f243c199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59528/0 (socket says 172.21.15.29:59528) 2024-07-27T06:49:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f243ad9d640 1 -- 172.21.15.29:0/1241957718 learned_addr learned my addr 172.21.15.29:0/1241957718 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f243a59c640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f243c108ed0 0x7f243c19a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f243b59e640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f243c109820 0x7f243c19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f243ad9d640 1 -- 172.21.15.29:0/1241957718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f243c109820 msgr2=0x7f243c19f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:41.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f243ad9d640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f243c109820 0x7f243c19f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f243ad9d640 1 -- 172.21.15.29:0/1241957718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f243c108ed0 msgr2=0x7f243c19a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:41.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f243ad9d640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f243c108ed0 0x7f243c19a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f243ad9d640 1 -- 172.21.15.29:0/1241957718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f243c19fed0 con 0x7f243c103590 2024-07-27T06:49:41.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f243b59e640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f243c109820 0x7f243c19f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:49:41.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f243ad9d640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f243c103590 0x7f243c199f60 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f242c00d950 tx=0x7f242c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:41.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f2423fff640 1 -- 172.21.15.29:0/1241957718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c017030 con 0x7f243c103590 2024-07-27T06:49:41.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f244111f640 1 -- 172.21.15.29:0/1241957718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f243c1a01c0 con 0x7f243c103590 2024-07-27T06:49:41.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.755+0000 7f2423fff640 1 -- 172.21.15.29:0/1241957718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f242c004020 con 0x7f243c103590 2024-07-27T06:49:41.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.756+0000 7f2423fff640 1 -- 172.21.15.29:0/1241957718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c0047a0 con 0x7f243c103590 2024-07-27T06:49:41.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.756+0000 7f244111f640 1 -- 172.21.15.29:0/1241957718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f243c1a6920 con 0x7f243c103590 2024-07-27T06:49:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.758+0000 7f24476a3740 1 Processor -- start 2024-07-27T06:49:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.758+0000 7f24476a3740 1 -- start start 2024-07-27T06:49:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.758+0000 7f24476a3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681eddb070 0x55681f180570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.758+0000 7f24476a3740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edb21b0 0x55681f180ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.758+0000 7f24476a3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edba390 0x55681f180ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.758+0000 7f2423fff640 1 -- 172.21.15.29:0/1241957718 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f242c004940 con 0x7f243c103590 2024-07-27T06:49:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f24476a3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55681f0f6bb0 con 0x55681edba390 2024-07-27T06:49:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f24476a3740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55681f0e60b0 con 0x55681edb21b0 2024-07-27T06:49:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f24476a3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55681f0ea0f0 con 0x55681eddb070 2024-07-27T06:49:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f243a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edba390 0x55681f180ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f243a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edba390 0x55681f180ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59544/0 (socket says 172.21.15.29:59544) 2024-07-27T06:49:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f243a59c640 1 -- 172.21.15.29:0/270484520 learned_addr learned my addr 172.21.15.29:0/270484520 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f243b59e640 1 --2- 172.21.15.29:0/270484520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681eddb070 0x55681f180570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f243ad9d640 1 --2- 172.21.15.29:0/270484520 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edb21b0 0x55681f180ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f243a59c640 1 -- 172.21.15.29:0/270484520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681eddb070 msgr2=0x55681f180570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f243a59c640 1 --2- 172.21.15.29:0/270484520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681eddb070 0x55681f180570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f243a59c640 1 -- 172.21.15.29:0/270484520 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edb21b0 msgr2=0x55681f180ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f243a59c640 1 --2- 172.21.15.29:0/270484520 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edb21b0 0x55681f180ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f243a59c640 1 -- 172.21.15.29:0/270484520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55681f08cb90 con 0x55681edba390 2024-07-27T06:49:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f243b59e640 1 --2- 172.21.15.29:0/270484520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681eddb070 0x55681f180570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.759+0000 7f243a59c640 1 --2- 172.21.15.29:0/270484520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edba390 0x55681f180ff0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2430009900 tx=0x7f2430000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82e842c2b18168b server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.760+0000 7f24217fa640 1 -- 172.21.15.29:0/270484520 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2430040070 con 0x55681edba390 2024-07-27T06:49:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.760+0000 7f2423fff640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f24240b3d40 0x7f24240b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.760+0000 7f24217fa640 1 -- 172.21.15.29:0/270484520 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2430005db0 con 0x55681edba390 2024-07-27T06:49:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.760+0000 7f24217fa640 1 -- 172.21.15.29:0/270484520 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2430035040 con 0x55681edba390 2024-07-27T06:49:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.760+0000 7f243b59e640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f24240b3d40 0x7f24240b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.760+0000 7f2423fff640 1 -- 172.21.15.29:0/1241957718 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f242c0e3a30 con 0x7f243c103590 2024-07-27T06:49:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.761+0000 7f24476a3740 1 -- 172.21.15.29:0/270484520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edba390 msgr2=0x55681f180ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.761+0000 7f24476a3740 1 --2- 172.21.15.29:0/270484520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edba390 0x55681f180ff0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2430009900 tx=0x7f2430000bc0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.761+0000 7f243b59e640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f24240b3d40 0x7f24240b6250 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2434005fd0 tx=0x7f2434008450 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.761+0000 7f24476a3740 1 -- 172.21.15.29:0/270484520 shutdown_connections 2024-07-27T06:49:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.761+0000 7f24476a3740 1 --2- 172.21.15.29:0/270484520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edba390 0x55681f180ff0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.761+0000 7f24476a3740 1 --2- 172.21.15.29:0/270484520 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edb21b0 0x55681f180ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.761+0000 7f24476a3740 1 --2- 172.21.15.29:0/270484520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681eddb070 0x55681f180570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.761+0000 7f24476a3740 1 -- 172.21.15.29:0/270484520 >> 172.21.15.29:0/270484520 conn(0x55681edcf460 msgr2=0x55681f17e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.761+0000 7f24476a3740 1 -- 172.21.15.29:0/270484520 shutdown_connections 2024-07-27T06:49:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.761+0000 7f24476a3740 1 -- 172.21.15.29:0/270484520 wait complete. 2024-07-27T06:49:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.761+0000 7f24476a3740 1 -- start start 2024-07-27T06:49:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.761+0000 7f24476a3740 1 Processor -- start 2024-07-27T06:49:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f24476a3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681edb21b0 0x55681f1904d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f24476a3740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edba390 0x55681f190a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f24476a3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681eddb070 0x55681f19c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f24476a3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55681ef49700 con 0x55681eddb070 2024-07-27T06:49:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f24476a3740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55681f0889e0 con 0x55681edba390 2024-07-27T06:49:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f24476a3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55681f193050 con 0x55681edb21b0 2024-07-27T06:49:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f243b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681eddb070 0x55681f19c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f243b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681eddb070 0x55681f19c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59552/0 (socket says 172.21.15.29:59552) 2024-07-27T06:49:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f24476a3740 10 client.0 _ll_register_callbacks cb 0x55681f0a34f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f243b59e640 1 -- 172.21.15.29:0/2898538094 learned_addr learned my addr 172.21.15.29:0/2898538094 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f243ad9d640 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681edb21b0 0x55681f1904d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f243a59c640 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edba390 0x55681f190a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f243b59e640 1 -- 172.21.15.29:0/2898538094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681edb21b0 msgr2=0x55681f1904d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f243b59e640 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681edb21b0 0x55681f1904d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f243b59e640 1 -- 172.21.15.29:0/2898538094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edba390 msgr2=0x55681f190a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f243b59e640 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edba390 0x55681f190a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f243b59e640 1 -- 172.21.15.29:0/2898538094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55681ee46650 con 0x55681eddb070 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f243b59e640 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681eddb070 0x55681f19c3b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2434010d90 tx=0x7f2434004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1bfffc5e165a8e0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f241eff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.762+0000 7f241eff5640 1 -- 172.21.15.29:0/2898538094 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243401d620 con 0x55681eddb070 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f241eff5640 1 -- 172.21.15.29:0/2898538094 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f243401d7c0 con 0x55681eddb070 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f24476a3740 1 -- 172.21.15.29:0/2898538094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2430005db0 con 0x55681eddb070 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f241eff5640 1 -- 172.21.15.29:0/2898538094 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f243401d9c0 con 0x55681eddb070 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f24476a3740 10 client.16278 fetch_fsmap learned FSMap version 32 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f24476a3740 1 -- 172.21.15.29:0/2898538094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55681f19f090 con 0x55681eddb070 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f241eff5640 1 -- 172.21.15.29:0/2898538094 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2430005db0 con 0x55681eddb070 2024-07-27T06:49:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f241eff5640 1 -- 172.21.15.29:0/2898538094 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8382+0+0 (secure 0 0 0) 0x7f2434025b10 con 0x55681eddb070 2024-07-27T06:49:41.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f24476a3740 10 client.16278 fetch_fsmap finished waiting for FSMap version 32 2024-07-27T06:49:41.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f24476a3740 10 client.16278 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-07-27T06:49:41.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f24476a3740 20 client.16278 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:41.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f24476a3740 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55681ed999f0 0x55681f1a25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f24476a3740 4 client.16278 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:41.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.763+0000 7f24476a3740 1 -- 172.21.15.29:0/2898538094 --> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55681f0f8b00 con 0x55681ed999f0 2024-07-27T06:49:41.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.764+0000 7f243ad9d640 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55681ed999f0 0x55681f1a25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.764+0000 7f243ad9d640 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55681ed999f0 0x55681f1a25e0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cca4c38d77b551ce server_cookie=e1e4aa314f33ff02 in_seq=0 out_seq=0 2024-07-27T06:49:41.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.764+0000 7f241eff5640 10 client.16278 ms_handle_connect on v2:172.21.15.77:6832/1085922080 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f241eff5640 1 -- 172.21.15.29:0/2898538094 <== mds.4 v2:172.21.15.77:6832/1085922080 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55681f0f8b00 con 0x55681ed999f0 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f241eff5640 10 client.16278 handle_command_reply: tid=0 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f24476a3740 1 client.16278 shutdown 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f24476a3740 1 -- 172.21.15.29:0/2898538094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681eddb070 msgr2=0x55681f19c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f24476a3740 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681eddb070 0x55681f19c3b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2434010d90 tx=0x7f2434004a40 comp rx=0 tx=0).stop 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f24476a3740 1 -- 172.21.15.29:0/2898538094 shutdown_connections 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f24476a3740 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55681ed999f0 0x55681f1a25e0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f24476a3740 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681eddb070 0x55681f19c3b0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f241eff5640 0 client.16278 ms_handle_reset on v2:172.21.15.77:6832/1085922080 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f24476a3740 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edba390 0x55681f190a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f24476a3740 1 --2- 172.21.15.29:0/2898538094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681edb21b0 0x55681f1904d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f24476a3740 1 -- 172.21.15.29:0/2898538094 >> 172.21.15.29:0/2898538094 conn(0x55681edcf460 msgr2=0x55681f17cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f24476a3740 1 -- 172.21.15.29:0/2898538094 shutdown_connections 2024-07-27T06:49:41.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f24476a3740 1 -- 172.21.15.29:0/2898538094 wait complete. 2024-07-27T06:49:41.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.765+0000 7f24476a3740 20 client.16278 trim_cache size 0 max 16384 2024-07-27T06:49:41.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.778+0000 7f24476a3740 1 Processor -- start 2024-07-27T06:49:41.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.778+0000 7f24476a3740 1 -- start start 2024-07-27T06:49:41.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.778+0000 7f24476a3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681ed999f0 0x55681f17e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.778+0000 7f24476a3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 0x55681f17cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.778+0000 7f24476a3740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edba390 0x55681f17d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.778+0000 7f24476a3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f243c1acf10 con 0x55681edb21b0 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.778+0000 7f24476a3740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f243c1aeb40 con 0x55681edba390 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.778+0000 7f24476a3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f243c1ad5e0 con 0x55681ed999f0 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f243a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 0x55681f17cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f243a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 0x55681f17cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59562/0 (socket says 172.21.15.29:59562) 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f243a59c640 1 -- 172.21.15.29:0/214295179 learned_addr learned my addr 172.21.15.29:0/214295179 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f243ad9d640 1 --2- 172.21.15.29:0/214295179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681ed999f0 0x55681f17e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f243b59e640 1 --2- 172.21.15.29:0/214295179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edba390 0x55681f17d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f243a59c640 1 -- 172.21.15.29:0/214295179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681ed999f0 msgr2=0x55681f17e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f243a59c640 1 --2- 172.21.15.29:0/214295179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681ed999f0 0x55681f17e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f243a59c640 1 -- 172.21.15.29:0/214295179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edba390 msgr2=0x55681f17d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f243a59c640 1 --2- 172.21.15.29:0/214295179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edba390 0x55681f17d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f243a59c640 1 -- 172.21.15.29:0/214295179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55681f1798c0 con 0x55681edb21b0 2024-07-27T06:49:41.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f243b59e640 1 --2- 172.21.15.29:0/214295179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edba390 0x55681f17d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f243a59c640 1 --2- 172.21.15.29:0/214295179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 0x55681f17cc40 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2430033000 tx=0x7f2430005210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8416ba52cfec6661 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f2420ff9640 1 -- 172.21.15.29:0/214295179 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2430035070 con 0x55681edb21b0 2024-07-27T06:49:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f2420ff9640 1 -- 172.21.15.29:0/214295179 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2430031cd0 con 0x55681edb21b0 2024-07-27T06:49:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.779+0000 7f2420ff9640 1 -- 172.21.15.29:0/214295179 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2430040040 con 0x55681edb21b0 2024-07-27T06:49:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.780+0000 7f24476a3740 1 -- 172.21.15.29:0/214295179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 msgr2=0x55681f17cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.780+0000 7f24476a3740 1 --2- 172.21.15.29:0/214295179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 0x55681f17cc40 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2430033000 tx=0x7f2430005210 comp rx=0 tx=0).stop 2024-07-27T06:49:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.781+0000 7f24476a3740 1 -- 172.21.15.29:0/214295179 shutdown_connections 2024-07-27T06:49:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.781+0000 7f24476a3740 1 --2- 172.21.15.29:0/214295179 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681edba390 0x55681f17d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.781+0000 7f24476a3740 1 --2- 172.21.15.29:0/214295179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 0x55681f17cc40 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.781+0000 7f24476a3740 1 --2- 172.21.15.29:0/214295179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681ed999f0 0x55681f17e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.781+0000 7f24476a3740 1 -- 172.21.15.29:0/214295179 >> 172.21.15.29:0/214295179 conn(0x55681edcf460 msgr2=0x55681f17dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.781+0000 7f24476a3740 1 -- 172.21.15.29:0/214295179 shutdown_connections 2024-07-27T06:49:41.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.781+0000 7f24476a3740 1 -- 172.21.15.29:0/214295179 wait complete. 2024-07-27T06:49:41.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.781+0000 7f24476a3740 1 -- start start 2024-07-27T06:49:41.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.781+0000 7f24476a3740 1 Processor -- start 2024-07-27T06:49:41.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f24476a3740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681ed999f0 0x55681f198890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f24476a3740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 0x55681f193d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f24476a3740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681edba390 0x55681f194280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f24476a3740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f243c1ae3b0 con 0x55681edb21b0 2024-07-27T06:49:41.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f24476a3740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f243c1adcb0 con 0x55681ed999f0 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f24476a3740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f243c1aaef0 con 0x55681edba390 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f24476a3740 10 client.0 _ll_register_callbacks cb 0x7f2434008a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f243a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 0x55681f193d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f243b59e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681edba390 0x55681f194280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f243a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 0x55681f193d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59566/0 (socket says 172.21.15.29:59566) 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f243a59c640 1 -- 172.21.15.29:0/418035197 learned_addr learned my addr 172.21.15.29:0/418035197 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f243ad9d640 1 --2- 172.21.15.29:0/418035197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681ed999f0 0x55681f198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f243a59c640 1 -- 172.21.15.29:0/418035197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681edba390 msgr2=0x55681f194280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f243a59c640 1 --2- 172.21.15.29:0/418035197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681edba390 0x55681f194280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f243a59c640 1 -- 172.21.15.29:0/418035197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681ed999f0 msgr2=0x55681f198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f243a59c640 1 --2- 172.21.15.29:0/418035197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681ed999f0 0x55681f198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f243a59c640 1 -- 172.21.15.29:0/418035197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55681f1948c0 con 0x55681edb21b0 2024-07-27T06:49:41.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f243a59c640 1 --2- 172.21.15.29:0/418035197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 0x55681f193d40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f243002a930 tx=0x7f243002abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4dc6a4872454ee server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:41.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f241eff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:41.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.782+0000 7f241eff5640 1 -- 172.21.15.29:0/418035197 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2430056060 con 0x55681edb21b0 2024-07-27T06:49:41.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f241eff5640 1 -- 172.21.15.29:0/418035197 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f2430004330 con 0x55681edb21b0 2024-07-27T06:49:41.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f24476a3740 1 -- 172.21.15.29:0/418035197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2430031cd0 con 0x55681edb21b0 2024-07-27T06:49:41.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f241eff5640 1 -- 172.21.15.29:0/418035197 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2430040070 con 0x55681edb21b0 2024-07-27T06:49:41.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f24476a3740 10 client.16284 fetch_fsmap learned FSMap version 32 2024-07-27T06:49:41.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f24476a3740 1 -- 172.21.15.29:0/418035197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55681f20aba0 con 0x55681edb21b0 2024-07-27T06:49:41.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f241eff5640 1 -- 172.21.15.29:0/418035197 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2430031cd0 con 0x55681edb21b0 2024-07-27T06:49:41.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f241eff5640 1 -- 172.21.15.29:0/418035197 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8382+0+0 (secure 0 0 0) 0x7f2430003c40 con 0x55681edb21b0 2024-07-27T06:49:41.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f24476a3740 10 client.16284 fetch_fsmap finished waiting for FSMap version 32 2024-07-27T06:49:41.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f24476a3740 10 client.16284 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-07-27T06:49:41.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f24476a3740 20 client.16284 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:41.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f24476a3740 1 --2- 172.21.15.29:0/418035197 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55681eddb070 0x55681f20cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:41.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f24476a3740 4 client.16284 mds_command: new command op to 24613 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:49:41.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f24476a3740 1 -- 172.21.15.29:0/418035197 --> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55681f0f45a0 con 0x55681eddb070 2024-07-27T06:49:41.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.783+0000 7f243ad9d640 1 --2- 172.21.15.29:0/418035197 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55681eddb070 0x55681f20cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:41.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.784+0000 7f241eff5640 10 client.16284 ms_handle_connect on v2:172.21.15.77:6832/1085922080 2024-07-27T06:49:41.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.784+0000 7f243ad9d640 1 --2- 172.21.15.29:0/418035197 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55681eddb070 0x55681f20cdd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3d854e217f847c96 server_cookie=24d60bd12dec067f in_seq=0 out_seq=0 2024-07-27T06:49:41.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.784+0000 7f241eff5640 1 -- 172.21.15.29:0/418035197 <== mds.4 v2:172.21.15.77:6832/1085922080 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55681f0f45a0 con 0x55681eddb070 2024-07-27T06:49:41.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.784+0000 7f241eff5640 10 client.16284 handle_command_reply: tid=0 2024-07-27T06:49:41.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.784+0000 7f24476a3740 1 client.16284 shutdown 2024-07-27T06:49:41.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.784+0000 7f24476a3740 1 -- 172.21.15.29:0/418035197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 msgr2=0x55681f193d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:41.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.784+0000 7f24476a3740 1 --2- 172.21.15.29:0/418035197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 0x55681f193d40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f243002a930 tx=0x7f243002abc0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f24476a3740 1 -- 172.21.15.29:0/418035197 shutdown_connections 2024-07-27T06:49:41.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f24476a3740 1 --2- 172.21.15.29:0/418035197 >> [v2:172.21.15.77:6832/1085922080,v1:172.21.15.77:6833/1085922080] conn(0x55681eddb070 0x55681f20cdd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f24476a3740 1 --2- 172.21.15.29:0/418035197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55681edba390 0x55681f194280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f24476a3740 1 --2- 172.21.15.29:0/418035197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55681edb21b0 0x55681f193d40 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f241eff5640 0 client.16284 ms_handle_reset on v2:172.21.15.77:6832/1085922080 2024-07-27T06:49:41.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f24476a3740 1 --2- 172.21.15.29:0/418035197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55681ed999f0 0x55681f198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f24476a3740 1 -- 172.21.15.29:0/418035197 >> 172.21.15.29:0/418035197 conn(0x55681edcf460 msgr2=0x55681f17dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:41.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f24476a3740 1 -- 172.21.15.29:0/418035197 shutdown_connections 2024-07-27T06:49:41.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f24476a3740 1 -- 172.21.15.29:0/418035197 wait complete. 2024-07-27T06:49:41.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f24476a3740 20 client.16284 trim_cache size 0 max 16384 2024-07-27T06:49:41.796 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:49:41.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f244111f640 1 -- 172.21.15.29:0/1241957718 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f24240b3d40 msgr2=0x7f24240b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:41.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f244111f640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f24240b3d40 0x7f24240b6250 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2434005fd0 tx=0x7f2434008450 comp rx=0 tx=0).stop 2024-07-27T06:49:41.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f244111f640 1 -- 172.21.15.29:0/1241957718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f243c103590 msgr2=0x7f243c199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:41.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f244111f640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f243c103590 0x7f243c199f60 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f242c00d950 tx=0x7f242c00de10 comp rx=0 tx=0).stop 2024-07-27T06:49:41.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f244111f640 1 -- 172.21.15.29:0/1241957718 shutdown_connections 2024-07-27T06:49:41.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f244111f640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f24240b3d40 0x7f24240b6250 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f244111f640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f243c109820 0x7f243c19f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f244111f640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f243c108ed0 0x7f243c19a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f244111f640 1 --2- 172.21.15.29:0/1241957718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f243c103590 0x7f243c199f60 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:41.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.785+0000 7f244111f640 1 -- 172.21.15.29:0/1241957718 >> 172.21.15.29:0/1241957718 conn(0x7f243c07b280 msgr2=0x7f243c107020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:41.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.786+0000 7f244111f640 1 -- 172.21.15.29:0/1241957718 shutdown_connections 2024-07-27T06:49:41.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:41.786+0000 7f244111f640 1 -- 172.21.15.29:0/1241957718 wait complete. 2024-07-27T06:49:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:41 smithi077 ceph-mon[37153]: pgmap v477: 97 pgs: 97 active+clean; 39 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 864 KiB/s wr, 362 op/s 2024-07-27T06:49:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:41 smithi047 ceph-mon[36835]: pgmap v477: 97 pgs: 97 active+clean; 39 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 864 KiB/s wr, 362 op/s 2024-07-27T06:49:42.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-27T06:49:42.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.626+0000 7f734f0b7640 1 -- 172.21.15.29:0/3122847186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7348107d20 msgr2=0x7f7348108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:42.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.626+0000 7f734f0b7640 1 --2- 172.21.15.29:0/3122847186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7348107d20 0x7f7348108130 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f7334009900 tx=0x7f733402a410 comp rx=0 tx=0).stop 2024-07-27T06:49:42.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.626+0000 7f734f0b7640 1 -- 172.21.15.29:0/3122847186 shutdown_connections 2024-07-27T06:49:42.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.626+0000 7f734f0b7640 1 --2- 172.21.15.29:0/3122847186 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7348108670 0x7f734810eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.626+0000 7f734f0b7640 1 --2- 172.21.15.29:0/3122847186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7348107d20 0x7f7348108130 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.628 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.626+0000 7f734f0b7640 1 --2- 172.21.15.29:0/3122847186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73481023f0 0x7f7348102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.626+0000 7f734f0b7640 1 -- 172.21.15.29:0/3122847186 >> 172.21.15.29:0/3122847186 conn(0x7f73480fddf0 msgr2=0x7f7348100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.626+0000 7f734f0b7640 1 -- 172.21.15.29:0/3122847186 shutdown_connections 2024-07-27T06:49:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.626+0000 7f734f0b7640 1 -- 172.21.15.29:0/3122847186 wait complete. 2024-07-27T06:49:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734f0b7640 1 Processor -- start 2024-07-27T06:49:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734f0b7640 1 -- start start 2024-07-27T06:49:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734f0b7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73481023f0 0x7f7348198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734f0b7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7348107d20 0x7f7348199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734f0b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7348108670 0x7f734819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734f0b7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73481123a0 con 0x7f7348108670 2024-07-27T06:49:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734f0b7640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f7348112220 con 0x7f73481023f0 2024-07-27T06:49:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734f0b7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7348112520 con 0x7f7348107d20 2024-07-27T06:49:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734d654640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7348108670 0x7f734819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734d654640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7348108670 0x7f734819e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59590/0 (socket says 172.21.15.29:59590) 2024-07-27T06:49:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734d654640 1 -- 172.21.15.29:0/1357579498 learned_addr learned my addr 172.21.15.29:0/1357579498 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f7347fff640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7348107d20 0x7f7348199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73481023f0 0x7f7348198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734d654640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7348107d20 msgr2=0x7f7348199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734d654640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7348107d20 0x7f7348199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734d654640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73481023f0 msgr2=0x7f7348198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734d654640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73481023f0 0x7f7348198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.627+0000 7f734d654640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f734819ed00 con 0x7f7348108670 2024-07-27T06:49:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.628+0000 7f734d654640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7348108670 0x7f734819e690 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f734000b970 tx=0x7f734000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.628+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7340019070 con 0x7f7348108670 2024-07-27T06:49:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.628+0000 7f734f0b7640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f734819eff0 con 0x7f7348108670 2024-07-27T06:49:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.628+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7340014440 con 0x7f7348108670 2024-07-27T06:49:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.628+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73400075d0 con 0x7f7348108670 2024-07-27T06:49:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.628+0000 7f734f0b7640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73481a5810 con 0x7f7348108670 2024-07-27T06:49:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.630+0000 7f7355fcf740 1 Processor -- start 2024-07-27T06:49:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.630+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f734001c070 con 0x7f7348108670 2024-07-27T06:49:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7355fcf740 1 -- start start 2024-07-27T06:49:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7355fcf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645ed3621b0 0x5645ed7046b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7355fcf740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5645ed3392f0 0x5645ed704bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7355fcf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645ed3414d0 0x5645ed705130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7355fcf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5645ed67a3b0 con 0x5645ed3414d0 2024-07-27T06:49:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7355fcf740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5645ed6698b0 con 0x5645ed3392f0 2024-07-27T06:49:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7355fcf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5645ed66d8f0 con 0x5645ed3621b0 2024-07-27T06:49:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7347fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645ed3414d0 0x5645ed705130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7347fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645ed3414d0 0x5645ed705130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59600/0 (socket says 172.21.15.29:59600) 2024-07-27T06:49:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7347fff640 1 -- 172.21.15.29:0/3853638961 learned_addr learned my addr 172.21.15.29:0/3853638961 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7347fff640 1 --2- 172.21.15.29:0/3853638961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645ed3621b0 0x5645ed7046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f734ce53640 1 --2- 172.21.15.29:0/3853638961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5645ed3392f0 0x5645ed704bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7347fff640 1 -- 172.21.15.29:0/3853638961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645ed3621b0 msgr2=0x5645ed7046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7347fff640 1 --2- 172.21.15.29:0/3853638961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645ed3621b0 0x5645ed7046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7347fff640 1 -- 172.21.15.29:0/3853638961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5645ed3392f0 msgr2=0x5645ed704bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7347fff640 1 --2- 172.21.15.29:0/3853638961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5645ed3392f0 0x5645ed704bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7347fff640 1 -- 172.21.15.29:0/3853638961 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645ed6109a0 con 0x5645ed3414d0 2024-07-27T06:49:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.631+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.632+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.632+0000 7f7347fff640 1 --2- 172.21.15.29:0/3853638961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645ed3414d0 0x5645ed705130 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7334009900 tx=0x7f7334000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a482fe3515f55042 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.632+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f73400e34f0 con 0x7f7348108670 2024-07-27T06:49:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.632+0000 7f7332ffd640 1 -- 172.21.15.29:0/3853638961 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7334035030 con 0x5645ed3414d0 2024-07-27T06:49:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.632+0000 7f7332ffd640 1 -- 172.21.15.29:0/3853638961 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73340054d0 con 0x5645ed3414d0 2024-07-27T06:49:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.632+0000 7f7332ffd640 1 -- 172.21.15.29:0/3853638961 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73340047a0 con 0x5645ed3414d0 2024-07-27T06:49:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.633+0000 7f7355fcf740 1 -- 172.21.15.29:0/3853638961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645ed3414d0 msgr2=0x5645ed705130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.633+0000 7f7355fcf740 1 --2- 172.21.15.29:0/3853638961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645ed3414d0 0x5645ed705130 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7334009900 tx=0x7f7334000bc0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.633+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b6250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x5645ed703330 tx=0x7f733c00c4b0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.633+0000 7f7355fcf740 1 -- 172.21.15.29:0/3853638961 shutdown_connections 2024-07-27T06:49:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.633+0000 7f7355fcf740 1 --2- 172.21.15.29:0/3853638961 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645ed3414d0 0x5645ed705130 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.633+0000 7f7355fcf740 1 --2- 172.21.15.29:0/3853638961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5645ed3392f0 0x5645ed704bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.633+0000 7f7355fcf740 1 --2- 172.21.15.29:0/3853638961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645ed3621b0 0x5645ed7046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.633+0000 7f7355fcf740 1 -- 172.21.15.29:0/3853638961 >> 172.21.15.29:0/3853638961 conn(0x5645ed356300 msgr2=0x5645ed7025a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.633+0000 7f7355fcf740 1 -- 172.21.15.29:0/3853638961 shutdown_connections 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.633+0000 7f7355fcf740 1 -- 172.21.15.29:0/3853638961 wait complete. 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.633+0000 7f7355fcf740 1 -- start start 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.633+0000 7f7355fcf740 1 Processor -- start 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f7355fcf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645ed3392f0 0x5645ed710210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f7355fcf740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5645ed3414d0 0x5645ed710750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f7355fcf740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645ed3621b0 0x5645ed71c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f7355fcf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5645ed5a3ab0 con 0x5645ed3392f0 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f7355fcf740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5645ed5f1cf0 con 0x5645ed3414d0 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f7355fcf740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5645ed712dd0 con 0x5645ed3621b0 2024-07-27T06:49:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f7355fcf740 10 client.0 _ll_register_callbacks cb 0x5645ed6633e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f734ce53640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645ed3392f0 0x5645ed710210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f734ce53640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645ed3392f0 0x5645ed710210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59608/0 (socket says 172.21.15.29:59608) 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f734ce53640 1 -- 172.21.15.29:0/940768653 learned_addr learned my addr 172.21.15.29:0/940768653 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f734d654640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645ed3621b0 0x5645ed71c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5645ed3414d0 0x5645ed710750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f734ce53640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645ed3621b0 msgr2=0x5645ed71c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f734ce53640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5645ed3621b0 0x5645ed71c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f734ce53640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5645ed3414d0 msgr2=0x5645ed710750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f734ce53640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5645ed3414d0 0x5645ed710750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.634+0000 7f734ce53640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645ed3cd790 con 0x5645ed3392f0 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.635+0000 7f734ce53640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5645ed3392f0 0x5645ed710210 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f733c0117c0 tx=0x7f733c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb018909b7953c70 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.635+0000 7f7327fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.635+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c029060 con 0x5645ed3392f0 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.635+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f733c011e00 con 0x5645ed3392f0 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.635+0000 7f7355fcf740 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73340054d0 con 0x5645ed3392f0 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.635+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f733c01cb10 con 0x5645ed3392f0 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.635+0000 7f7355fcf740 10 client.16296 fetch_fsmap learned FSMap version 32 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.635+0000 7f7355fcf740 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5645ed446620 con 0x5645ed3392f0 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.635+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73340054d0 con 0x5645ed3392f0 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.635+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8382+0+0 (secure 0 0 0) 0x7f733c018040 con 0x5645ed3392f0 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.636+0000 7f7355fcf740 10 client.16296 fetch_fsmap finished waiting for FSMap version 32 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.636+0000 7f7355fcf740 10 client.16296 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.636+0000 7f7355fcf740 20 client.16296 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.636+0000 7f7355fcf740 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.636+0000 7f7355fcf740 4 client.16296 mds_command: new command op to 24670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.636+0000 7f7355fcf740 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645ed67c300 con 0x5645ed320d80 2024-07-27T06:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.636+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:49:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.636+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.636+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T06:49:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:49:42.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.837+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:49:42.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.837+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-27T06:49:42.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:42.837+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:43.238+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:49:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:43.238+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-27T06:49:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:43.238+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:43 smithi077 ceph-mon[37153]: pgmap v478: 97 pgs: 97 active+clean; 40 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 939 KiB/s wr, 368 op/s 2024-07-27T06:49:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:43 smithi047 ceph-mon[36835]: pgmap v478: 97 pgs: 97 active+clean; 40 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 939 KiB/s wr, 368 op/s 2024-07-27T06:49:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:43 smithi029 ceph-mon[32610]: pgmap v478: 97 pgs: 97 active+clean; 40 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 939 KiB/s wr, 368 op/s 2024-07-27T06:49:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:44.039+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:49:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:44.039+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-27T06:49:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:44.039+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:44.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.559+0000 7f5b1685e640 1 -- 172.21.15.29:0/62726864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b10108ed0 msgr2=0x7f5b101092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:44.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.559+0000 7f5b1685e640 1 --2- 172.21.15.29:0/62726864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b10108ed0 0x7f5b101092e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009900 tx=0x7f5af802a410 comp rx=0 tx=0).stop 2024-07-27T06:49:44.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.559+0000 7f5b1685e640 1 -- 172.21.15.29:0/62726864 shutdown_connections 2024-07-27T06:49:44.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.559+0000 7f5b1685e640 1 --2- 172.21.15.29:0/62726864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b10109820 0x7f5b10110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.559+0000 7f5b1685e640 1 --2- 172.21.15.29:0/62726864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b10108ed0 0x7f5b101092e0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.559+0000 7f5b1685e640 1 --2- 172.21.15.29:0/62726864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b10103590 0x7f5b101039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.559+0000 7f5b1685e640 1 -- 172.21.15.29:0/62726864 >> 172.21.15.29:0/62726864 conn(0x7f5b1007b280 msgr2=0x7f5b1007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:44.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.559+0000 7f5b1685e640 1 -- 172.21.15.29:0/62726864 shutdown_connections 2024-07-27T06:49:44.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.559+0000 7f5b1685e640 1 -- 172.21.15.29:0/62726864 wait complete. 2024-07-27T06:49:44.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b1685e640 1 Processor -- start 2024-07-27T06:49:44.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b1685e640 1 -- start start 2024-07-27T06:49:44.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b1685e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b10103590 0x7f5b10199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b1685e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b10108ed0 0x7f5b1019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b1685e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b10109820 0x7f5b1019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b1685e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5b10113490 con 0x7f5b10109820 2024-07-27T06:49:44.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b1685e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f5b10113310 con 0x7f5b10108ed0 2024-07-27T06:49:44.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b1685e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5b10113610 con 0x7f5b10103590 2024-07-27T06:49:44.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b14dfb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b10109820 0x7f5b1019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b14dfb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b10109820 0x7f5b1019f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59624/0 (socket says 172.21.15.29:59624) 2024-07-27T06:49:44.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b14dfb640 1 -- 172.21.15.29:0/3183765061 learned_addr learned my addr 172.21.15.29:0/3183765061 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:44.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b0ffff640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b10103590 0x7f5b10199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.560+0000 7f5b077fe640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b10108ed0 0x7f5b1019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.561+0000 7f5b14dfb640 1 -- 172.21.15.29:0/3183765061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b10103590 msgr2=0x7f5b10199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.561+0000 7f5b14dfb640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b10103590 0x7f5b10199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.561+0000 7f5b14dfb640 1 -- 172.21.15.29:0/3183765061 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b10108ed0 msgr2=0x7f5b1019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.561+0000 7f5b14dfb640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b10108ed0 0x7f5b1019a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.561+0000 7f5b14dfb640 1 -- 172.21.15.29:0/3183765061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1019fe90 con 0x7f5b10109820 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.561+0000 7f5b14dfb640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b10109820 0x7f5b1019f760 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800eaa0 tx=0x7f5b0800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.561+0000 7f5b0dffb640 1 -- 172.21.15.29:0/3183765061 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0800cdb0 con 0x7f5b10109820 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.561+0000 7f5b0dffb640 1 -- 172.21.15.29:0/3183765061 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b08005020 con 0x7f5b10109820 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.561+0000 7f5b1685e640 1 -- 172.21.15.29:0/3183765061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b101a0180 con 0x7f5b10109820 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.561+0000 7f5b0dffb640 1 -- 172.21.15.29:0/3183765061 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b080116d0 con 0x7f5b10109820 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.561+0000 7f5b1685e640 1 -- 172.21.15.29:0/3183765061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b101a68e0 con 0x7f5b10109820 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.564+0000 7f5b1cde2740 1 Processor -- start 2024-07-27T06:49:44.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.564+0000 7f5b0dffb640 1 -- 172.21.15.29:0/3183765061 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f5b080051d0 con 0x7f5b10109820 2024-07-27T06:49:44.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.564+0000 7f5b1cde2740 1 -- start start 2024-07-27T06:49:44.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b1cde2740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983872d070 0x559838ad2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b1cde2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5598387041b0 0x559838ad2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b1cde2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 0x559838ad3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b1cde2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559838a48bb0 con 0x55983870c390 2024-07-27T06:49:44.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b1cde2740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x559838a380b0 con 0x55983872d070 2024-07-27T06:49:44.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b1cde2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559838a3c0f0 con 0x5598387041b0 2024-07-27T06:49:44.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 0x559838ad3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 0x559838ad3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59638/0 (socket says 172.21.15.29:59638) 2024-07-27T06:49:44.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b077fe640 1 -- 172.21.15.29:0/3818461634 learned_addr learned my addr 172.21.15.29:0/3818461634 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:44.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b0ffff640 1 --2- 172.21.15.29:0/3818461634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5598387041b0 0x559838ad2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b077fe640 1 --2- 172.21.15.29:0/3818461634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983872d070 0x559838ad2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b077fe640 1 -- 172.21.15.29:0/3818461634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5598387041b0 msgr2=0x559838ad2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:44.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b077fe640 1 --2- 172.21.15.29:0/3818461634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5598387041b0 0x559838ad2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b077fe640 1 -- 172.21.15.29:0/3818461634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983872d070 msgr2=0x559838ad2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:44.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b077fe640 1 --2- 172.21.15.29:0/3818461634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983872d070 0x559838ad2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b077fe640 1 -- 172.21.15.29:0/3818461634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598389deb90 con 0x55983870c390 2024-07-27T06:49:44.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b0dffb640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5afc0b3d40 0x7f5afc0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.565+0000 7f5b077fe640 1 --2- 172.21.15.29:0/3818461634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 0x559838ad3110 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5b1006c420 tx=0x7f5af8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5bc0111bcf39bf server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:44.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.566+0000 7f5b0ffff640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5afc0b3d40 0x7f5afc0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.566+0000 7f5b0dffb640 1 -- 172.21.15.29:0/3183765061 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f5b080e40d0 con 0x7f5b10109820 2024-07-27T06:49:44.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.566+0000 7f5b067fc640 1 -- 172.21.15.29:0/3818461634 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8044020 con 0x55983870c390 2024-07-27T06:49:44.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.566+0000 7f5b067fc640 1 -- 172.21.15.29:0/3818461634 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5af80316a0 con 0x55983870c390 2024-07-27T06:49:44.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.566+0000 7f5b067fc640 1 -- 172.21.15.29:0/3818461634 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8035070 con 0x55983870c390 2024-07-27T06:49:44.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.567+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3818461634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 msgr2=0x559838ad3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:44.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.567+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/3818461634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 0x559838ad3110 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5b1006c420 tx=0x7f5af8000bc0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.567+0000 7f5b0ffff640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5afc0b3d40 0x7f5afc0b6250 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009da0 tx=0x7f5b00005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:44.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.567+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3818461634 shutdown_connections 2024-07-27T06:49:44.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.567+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/3818461634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 0x559838ad3110 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.567+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/3818461634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5598387041b0 0x559838ad2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.567+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/3818461634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983872d070 0x559838ad2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.567+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3818461634 >> 172.21.15.29:0/3818461634 conn(0x559838721460 msgr2=0x559838ad0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:44.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.567+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3818461634 shutdown_connections 2024-07-27T06:49:44.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.567+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3818461634 wait complete. 2024-07-27T06:49:44.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.567+0000 7f5b1cde2740 1 -- start start 2024-07-27T06:49:44.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.567+0000 7f5b1cde2740 1 Processor -- start 2024-07-27T06:49:44.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b1cde2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5598387041b0 0x559838ae2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b1cde2740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983870c390 0x559838ae2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b1cde2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983872d070 0x559838aee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b1cde2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55983889b700 con 0x55983872d070 2024-07-27T06:49:44.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b1cde2740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5598389da9e0 con 0x55983870c390 2024-07-27T06:49:44.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b1cde2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559838ae5140 con 0x5598387041b0 2024-07-27T06:49:44.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b1cde2740 10 client.0 _ll_register_callbacks cb 0x5598389f54f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b0ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5598387041b0 0x559838ae2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b0ffff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5598387041b0 0x559838ae2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:45342/0 (socket says 172.21.15.29:45342) 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b14dfb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983872d070 0x559838aee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b0ffff640 1 -- 172.21.15.29:0/388005584 learned_addr learned my addr 172.21.15.29:0/388005584 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b077fe640 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983870c390 0x559838ae2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b0ffff640 1 -- 172.21.15.29:0/388005584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983870c390 msgr2=0x559838ae2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b0ffff640 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983870c390 0x559838ae2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b0ffff640 1 -- 172.21.15.29:0/388005584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983872d070 msgr2=0x559838aee4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b0ffff640 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983872d070 0x559838aee4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b0ffff640 1 -- 172.21.15.29:0/388005584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559838798650 con 0x5598387041b0 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.568+0000 7f5b14dfb640 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983872d070 0x559838aee4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.569+0000 7f5b0ffff640 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5598387041b0 0x559838ae2650 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5b00004620 tx=0x7f5b00004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ef3f419a86ffa0c server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:44.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.569+0000 7f5aef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.569+0000 7f5aef7fe640 1 -- 172.21.15.29:0/388005584 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0001f640 con 0x5598387041b0 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.569+0000 7f5b1cde2740 1 -- 172.21.15.29:0/388005584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5af80316a0 con 0x5598387041b0 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.569+0000 7f5aef7fe640 1 -- 172.21.15.29:0/388005584 <== mon.1 v2:172.21.15.77:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f5b0001f7e0 con 0x5598387041b0 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.569+0000 7f5aef7fe640 1 -- 172.21.15.29:0/388005584 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5b0001f9e0 con 0x5598387041b0 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.569+0000 7f5b1cde2740 10 client.25957 fetch_fsmap learned FSMap version 32 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.569+0000 7f5b1cde2740 1 -- 172.21.15.29:0/388005584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559838af12a0 con 0x5598387041b0 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.569+0000 7f5aef7fe640 1 -- 172.21.15.29:0/388005584 <== mon.1 v2:172.21.15.77:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5af80316a0 con 0x5598387041b0 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.570+0000 7f5aef7fe640 1 -- 172.21.15.29:0/388005584 <== mon.1 v2:172.21.15.77:3300/0 5 ==== fsmap(e 32) ==== 8382+0+0 (secure 0 0 0) 0x7f5b00027020 con 0x5598387041b0 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.570+0000 7f5b1cde2740 10 client.25957 fetch_fsmap finished waiting for FSMap version 32 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.570+0000 7f5b1cde2740 10 client.25957 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.570+0000 7f5b1cde2740 20 client.25957 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.570+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5598386eb9f0 0x559838af4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.570+0000 7f5b1cde2740 4 client.25957 mds_command: new command op to 24617 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:44.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.570+0000 7f5b1cde2740 1 -- 172.21.15.29:0/388005584 --> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559838a4ab00 con 0x5598386eb9f0 2024-07-27T06:49:44.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.570+0000 7f5b077fe640 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5598386eb9f0 0x559838af4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.571+0000 7f5b077fe640 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5598386eb9f0 0x559838af4700 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=14ac0f39858729fa server_cookie=3c184be1cddde6d in_seq=0 out_seq=0 2024-07-27T06:49:44.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.571+0000 7f5aef7fe640 10 client.25957 ms_handle_connect on v2:172.21.15.47:6838/670107266 2024-07-27T06:49:44.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.571+0000 7f5aef7fe640 1 -- 172.21.15.29:0/388005584 <== mds.3 v2:172.21.15.47:6838/670107266 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559838a4ab00 con 0x5598386eb9f0 2024-07-27T06:49:44.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.571+0000 7f5aef7fe640 10 client.25957 handle_command_reply: tid=0 2024-07-27T06:49:44.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.571+0000 7f5b1cde2740 1 client.25957 shutdown 2024-07-27T06:49:44.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.572+0000 7f5b1cde2740 1 -- 172.21.15.29:0/388005584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5598387041b0 msgr2=0x559838ae2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:44.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.572+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5598387041b0 0x559838ae2650 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5b00004620 tx=0x7f5b00004730 comp rx=0 tx=0).stop 2024-07-27T06:49:44.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.572+0000 7f5b1cde2740 1 -- 172.21.15.29:0/388005584 shutdown_connections 2024-07-27T06:49:44.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.572+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5598386eb9f0 0x559838af4700 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.572+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983872d070 0x559838aee4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.572+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983870c390 0x559838ae2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.572+0000 7f5aef7fe640 0 client.25957 ms_handle_reset on v2:172.21.15.47:6838/670107266 2024-07-27T06:49:44.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.572+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/388005584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5598387041b0 0x559838ae2650 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.572+0000 7f5b1cde2740 1 -- 172.21.15.29:0/388005584 >> 172.21.15.29:0/388005584 conn(0x559838721460 msgr2=0x559838aced00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:44.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.572+0000 7f5b1cde2740 1 -- 172.21.15.29:0/388005584 shutdown_connections 2024-07-27T06:49:44.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.572+0000 7f5b1cde2740 1 -- 172.21.15.29:0/388005584 wait complete. 2024-07-27T06:49:44.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.572+0000 7f5b1cde2740 20 client.25957 trim_cache size 0 max 16384 2024-07-27T06:49:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b1cde2740 1 Processor -- start 2024-07-27T06:49:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b1cde2740 1 -- start start 2024-07-27T06:49:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b1cde2740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5598387041b0 0x559838ad0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b1cde2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 0x559838ad7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b1cde2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55983872d070 0x559838ad7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b1cde2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101acf20 con 0x55983870c390 2024-07-27T06:49:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b1cde2740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101aeb50 con 0x5598387041b0 2024-07-27T06:49:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b1cde2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101ad5f0 con 0x55983872d070 2024-07-27T06:49:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 0x559838ad7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b0ffff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5598387041b0 0x559838ad0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 0x559838ad7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59662/0 (socket says 172.21.15.29:59662) 2024-07-27T06:49:44.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b14dfb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55983872d070 0x559838ad7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.586+0000 7f5b077fe640 1 -- 172.21.15.29:0/1671086091 learned_addr learned my addr 172.21.15.29:0/1671086091 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:44.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.587+0000 7f5b077fe640 1 -- 172.21.15.29:0/1671086091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55983872d070 msgr2=0x559838ad7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.587+0000 7f5b077fe640 1 --2- 172.21.15.29:0/1671086091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55983872d070 0x559838ad7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.587+0000 7f5b077fe640 1 -- 172.21.15.29:0/1671086091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5598387041b0 msgr2=0x559838ad0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.587+0000 7f5b077fe640 1 --2- 172.21.15.29:0/1671086091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5598387041b0 0x559838ad0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.587+0000 7f5b077fe640 1 -- 172.21.15.29:0/1671086091 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559838acbfa0 con 0x55983870c390 2024-07-27T06:49:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.587+0000 7f5b14dfb640 1 --2- 172.21.15.29:0/1671086091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55983872d070 0x559838ad7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:49:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.587+0000 7f5b077fe640 1 --2- 172.21.15.29:0/1671086091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 0x559838ad7980 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009900 tx=0x7f5af8005330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3460805885bd6c2 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.587+0000 7f5b05ffb640 1 -- 172.21.15.29:0/1671086091 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8044050 con 0x55983870c390 2024-07-27T06:49:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.587+0000 7f5b05ffb640 1 -- 172.21.15.29:0/1671086091 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5af80055a0 con 0x55983870c390 2024-07-27T06:49:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.587+0000 7f5b05ffb640 1 -- 172.21.15.29:0/1671086091 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8035040 con 0x55983870c390 2024-07-27T06:49:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.588+0000 7f5b1cde2740 1 -- 172.21.15.29:0/1671086091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 msgr2=0x559838ad7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.588+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/1671086091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 0x559838ad7980 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009900 tx=0x7f5af8005330 comp rx=0 tx=0).stop 2024-07-27T06:49:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.588+0000 7f5b1cde2740 1 -- 172.21.15.29:0/1671086091 shutdown_connections 2024-07-27T06:49:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.588+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/1671086091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55983872d070 0x559838ad7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.588+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/1671086091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55983870c390 0x559838ad7980 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.588+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/1671086091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5598387041b0 0x559838ad0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.588+0000 7f5b1cde2740 1 -- 172.21.15.29:0/1671086091 >> 172.21.15.29:0/1671086091 conn(0x559838721460 msgr2=0x559838acfd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.588+0000 7f5b1cde2740 1 -- 172.21.15.29:0/1671086091 shutdown_connections 2024-07-27T06:49:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.588+0000 7f5b1cde2740 1 -- 172.21.15.29:0/1671086091 wait complete. 2024-07-27T06:49:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b1cde2740 1 -- start start 2024-07-27T06:49:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b1cde2740 1 Processor -- start 2024-07-27T06:49:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b1cde2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598387041b0 0x559838aeaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b1cde2740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55983870c390 0x559838adf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b1cde2740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983872d070 0x559838adf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b1cde2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101ae3c0 con 0x5598387041b0 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b1cde2740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101adcc0 con 0x55983872d070 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b1cde2740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101aaf00 con 0x55983870c390 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b1cde2740 10 client.0 _ll_register_callbacks cb 0x7f5b00009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b0ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598387041b0 0x559838aeaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b0ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598387041b0 0x559838aeaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59670/0 (socket says 172.21.15.29:59670) 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b0ffff640 1 -- 172.21.15.29:0/3542575780 learned_addr learned my addr 172.21.15.29:0/3542575780 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.589+0000 7f5b077fe640 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55983870c390 0x559838adf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5b14dfb640 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983872d070 0x559838adf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5b0ffff640 1 -- 172.21.15.29:0/3542575780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55983870c390 msgr2=0x559838adf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5b0ffff640 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55983870c390 0x559838adf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5b0ffff640 1 -- 172.21.15.29:0/3542575780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983872d070 msgr2=0x559838adf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5b0ffff640 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983872d070 0x559838adf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5b0ffff640 1 -- 172.21.15.29:0/3542575780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559838adfd10 con 0x5598387041b0 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5b0ffff640 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598387041b0 0x559838aeaa20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5b00012d50 tx=0x7f5b00029750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f81ac8fd6c9ba268 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5aef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5aef7fe640 1 -- 172.21.15.29:0/3542575780 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00005020 con 0x5598387041b0 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5aef7fe640 1 -- 172.21.15.29:0/3542575780 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7f5b000051c0 con 0x5598387041b0 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3542575780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5af80055a0 con 0x5598387041b0 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5aef7fe640 1 -- 172.21.15.29:0/3542575780 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5b00005400 con 0x5598387041b0 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5b1cde2740 10 client.16314 fetch_fsmap learned FSMap version 32 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3542575780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559838b5ce90 con 0x5598387041b0 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.590+0000 7f5aef7fe640 1 -- 172.21.15.29:0/3542575780 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5af80055a0 con 0x5598387041b0 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.591+0000 7f5aef7fe640 1 -- 172.21.15.29:0/3542575780 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8382+0+0 (secure 0 0 0) 0x7f5b00027040 con 0x5598387041b0 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.591+0000 7f5b1cde2740 10 client.16314 fetch_fsmap finished waiting for FSMap version 32 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.591+0000 7f5b1cde2740 10 client.16314 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.b 2024-07-27T06:49:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.591+0000 7f5b1cde2740 20 client.16314 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:44.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.591+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5598386c0da0 0x559838b5f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:44.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.591+0000 7f5b1cde2740 4 client.16314 mds_command: new command op to 24617 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:49:44.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.591+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3542575780 --> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559838a465a0 con 0x5598386c0da0 2024-07-27T06:49:44.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.591+0000 7f5b077fe640 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5598386c0da0 0x559838b5f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:44.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.591+0000 7f5b077fe640 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5598386c0da0 0x559838b5f560 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=61b045d170b21cc server_cookie=b3d8ac73dce3fb4 in_seq=0 out_seq=0 2024-07-27T06:49:44.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.591+0000 7f5aef7fe640 10 client.16314 ms_handle_connect on v2:172.21.15.47:6838/670107266 2024-07-27T06:49:44.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5aef7fe640 1 -- 172.21.15.29:0/3542575780 <== mds.3 v2:172.21.15.47:6838/670107266 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559838a465a0 con 0x5598386c0da0 2024-07-27T06:49:44.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5aef7fe640 10 client.16314 handle_command_reply: tid=0 2024-07-27T06:49:44.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5b1cde2740 1 client.16314 shutdown 2024-07-27T06:49:44.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3542575780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598387041b0 msgr2=0x559838aeaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598387041b0 0x559838aeaa20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5b00012d50 tx=0x7f5b00029750 comp rx=0 tx=0).stop 2024-07-27T06:49:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3542575780 shutdown_connections 2024-07-27T06:49:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.47:6838/670107266,v1:172.21.15.47:6839/670107266] conn(0x5598386c0da0 0x559838b5f560 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55983872d070 0x559838adf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55983870c390 0x559838adf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5b1cde2740 1 --2- 172.21.15.29:0/3542575780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5598387041b0 0x559838aeaa20 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3542575780 >> 172.21.15.29:0/3542575780 conn(0x559838721460 msgr2=0x559838ad36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5aef7fe640 0 client.16314 ms_handle_reset on v2:172.21.15.47:6838/670107266 2024-07-27T06:49:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3542575780 shutdown_connections 2024-07-27T06:49:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5b1cde2740 1 -- 172.21.15.29:0/3542575780 wait complete. 2024-07-27T06:49:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.592+0000 7f5b1cde2740 20 client.16314 trim_cache size 0 max 16384 2024-07-27T06:49:44.600 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:49:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.593+0000 7f5b1685e640 1 -- 172.21.15.29:0/3183765061 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5afc0b3d40 msgr2=0x7f5afc0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.593+0000 7f5b1685e640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5afc0b3d40 0x7f5afc0b6250 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009da0 tx=0x7f5b00005f90 comp rx=0 tx=0).stop 2024-07-27T06:49:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.593+0000 7f5b1685e640 1 -- 172.21.15.29:0/3183765061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b10109820 msgr2=0x7f5b1019f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:44.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.593+0000 7f5b1685e640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b10109820 0x7f5b1019f760 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800eaa0 tx=0x7f5b0800ef60 comp rx=0 tx=0).stop 2024-07-27T06:49:44.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.593+0000 7f5b1685e640 1 -- 172.21.15.29:0/3183765061 shutdown_connections 2024-07-27T06:49:44.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.593+0000 7f5b1685e640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b10109820 0x7f5b1019f760 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.593+0000 7f5b1685e640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b10108ed0 0x7f5b1019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.593+0000 7f5b1685e640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f5afc0b3d40 0x7f5afc0b6250 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.593+0000 7f5b1685e640 1 --2- 172.21.15.29:0/3183765061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b10103590 0x7f5b10199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:44.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.593+0000 7f5b1685e640 1 -- 172.21.15.29:0/3183765061 >> 172.21.15.29:0/3183765061 conn(0x7f5b1007b280 msgr2=0x7f5b10107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:44.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.593+0000 7f5b1685e640 1 -- 172.21.15.29:0/3183765061 shutdown_connections 2024-07-27T06:49:44.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:44.593+0000 7f5b1685e640 1 -- 172.21.15.29:0/3183765061 wait complete. 2024-07-27T06:49:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:44 smithi077 ceph-mon[37153]: pgmap v479: 97 pgs: 97 active+clean; 40 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 705 KiB/s wr, 156 op/s 2024-07-27T06:49:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:44 smithi047 ceph-mon[36835]: pgmap v479: 97 pgs: 97 active+clean; 40 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 705 KiB/s wr, 156 op/s 2024-07-27T06:49:44.958 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:44 smithi029 ceph-mon[32610]: pgmap v479: 97 pgs: 97 active+clean; 40 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 705 KiB/s wr, 156 op/s 2024-07-27T06:49:45.411 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-27T06:49:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:45.641+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:49:45.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:45.641+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-27T06:49:45.643 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:45.641+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:45.682 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:49:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:46 smithi077 ceph-mon[37153]: pgmap v480: 97 pgs: 97 active+clean; 42 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 845 KiB/s wr, 91 op/s 2024-07-27T06:49:47.107 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:46 smithi029 ceph-mon[32610]: pgmap v480: 97 pgs: 97 active+clean; 42 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 845 KiB/s wr, 91 op/s 2024-07-27T06:49:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:46 smithi047 ceph-mon[36835]: pgmap v480: 97 pgs: 97 active+clean; 42 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 845 KiB/s wr, 91 op/s 2024-07-27T06:49:47.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.501+0000 7ffa78449640 1 -- 172.21.15.29:0/4127275569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa70107e50 msgr2=0x7ffa70108260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:47.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.501+0000 7ffa78449640 1 --2- 172.21.15.29:0/4127275569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa70107e50 0x7ffa70108260 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009900 tx=0x7ffa6002a410 comp rx=0 tx=0).stop 2024-07-27T06:49:47.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.501+0000 7ffa78449640 1 -- 172.21.15.29:0/4127275569 shutdown_connections 2024-07-27T06:49:47.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.501+0000 7ffa78449640 1 --2- 172.21.15.29:0/4127275569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa701087a0 0x7ffa7010efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.501+0000 7ffa78449640 1 --2- 172.21.15.29:0/4127275569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa70107e50 0x7ffa70108260 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.501+0000 7ffa78449640 1 --2- 172.21.15.29:0/4127275569 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa70102510 0x7ffa70102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.501+0000 7ffa78449640 1 -- 172.21.15.29:0/4127275569 >> 172.21.15.29:0/4127275569 conn(0x7ffa700791a0 msgr2=0x7ffa700795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.501+0000 7ffa78449640 1 -- 172.21.15.29:0/4127275569 shutdown_connections 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.501+0000 7ffa78449640 1 -- 172.21.15.29:0/4127275569 wait complete. 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa78449640 1 Processor -- start 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa78449640 1 -- start start 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa78449640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa70102510 0x7ffa7019c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa78449640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa70107e50 0x7ffa7019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa78449640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa701087a0 0x7ffa701a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa78449640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa70111310 con 0x7ffa70102510 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa78449640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7ffa70111190 con 0x7ffa70107e50 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa78449640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa70111490 con 0x7ffa701087a0 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa761e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa70102510 0x7ffa7019c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa761e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa70102510 0x7ffa7019c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59684/0 (socket says 172.21.15.29:59684) 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa761e5640 1 -- 172.21.15.29:0/1444678639 learned_addr learned my addr 172.21.15.29:0/1444678639 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa759e4640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa70107e50 0x7ffa7019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa769e6640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa701087a0 0x7ffa701a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa761e5640 1 -- 172.21.15.29:0/1444678639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa701087a0 msgr2=0x7ffa701a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa761e5640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa701087a0 0x7ffa701a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa761e5640 1 -- 172.21.15.29:0/1444678639 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa70107e50 msgr2=0x7ffa7019c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa761e5640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa70107e50 0x7ffa7019c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.502+0000 7ffa761e5640 1 -- 172.21.15.29:0/1444678639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa701a2030 con 0x7ffa70102510 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.503+0000 7ffa761e5640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa70102510 0x7ffa7019c180 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c00c920 tx=0x7ffa6c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.503+0000 7ffa677fe640 1 -- 172.21.15.29:0/1444678639 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c007c70 con 0x7ffa70102510 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.503+0000 7ffa677fe640 1 -- 172.21.15.29:0/1444678639 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa6c007e10 con 0x7ffa70102510 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.503+0000 7ffa677fe640 1 -- 172.21.15.29:0/1444678639 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c018820 con 0x7ffa70102510 2024-07-27T06:49:47.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.503+0000 7ffa78449640 1 -- 172.21.15.29:0/1444678639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa701a2320 con 0x7ffa70102510 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.503+0000 7ffa78449640 1 -- 172.21.15.29:0/1444678639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa701a8b40 con 0x7ffa70102510 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.505+0000 7ffa7e9cd740 1 Processor -- start 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa7e9cd740 1 -- start start 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa677fe640 1 -- 172.21.15.29:0/1444678639 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7ffa6c00f450 con 0x7ffa70102510 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa7e9cd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 0x559043057570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa7e9cd740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042c891b0 0x559043057ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa7e9cd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 0x559043057ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa7e9cd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559042fcdbb0 con 0x559042cb2070 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa7e9cd740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x559042fbd0b0 con 0x559042c891b0 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa7e9cd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559042fc10f0 con 0x559042c91390 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa769e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 0x559043057570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa769e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 0x559043057570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59698/0 (socket says 172.21.15.29:59698) 2024-07-27T06:49:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa769e6640 1 -- 172.21.15.29:0/1479773567 learned_addr learned my addr 172.21.15.29:0/1479773567 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa761e5640 1 --2- 172.21.15.29:0/1479773567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042c891b0 0x559043057ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa759e4640 1 --2- 172.21.15.29:0/1479773567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 0x559043057ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.506+0000 7ffa769e6640 1 -- 172.21.15.29:0/1479773567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 msgr2=0x559043057ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.507+0000 7ffa769e6640 1 --2- 172.21.15.29:0/1479773567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 0x559043057ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.507+0000 7ffa769e6640 1 -- 172.21.15.29:0/1479773567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042c891b0 msgr2=0x559043057ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.507+0000 7ffa769e6640 1 --2- 172.21.15.29:0/1479773567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042c891b0 0x559043057ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.507+0000 7ffa769e6640 1 -- 172.21.15.29:0/1479773567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559042f63b90 con 0x559042cb2070 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.507+0000 7ffa769e6640 1 --2- 172.21.15.29:0/1479773567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 0x559043057570 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ffa68008dc0 tx=0x7ffa6800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fff842de08117f4 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.507+0000 7ffa677fe640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ffa5c0b3df0 0x7ffa5c0b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.507+0000 7ffa761e5640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ffa5c0b3df0 0x7ffa5c0b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.507+0000 7ffa64d9a640 1 -- 172.21.15.29:0/1479773567 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68014070 con 0x559042cb2070 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.507+0000 7ffa64d9a640 1 -- 172.21.15.29:0/1479773567 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa68003b70 con 0x559042cb2070 2024-07-27T06:49:47.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.507+0000 7ffa64d9a640 1 -- 172.21.15.29:0/1479773567 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68010790 con 0x559042cb2070 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.507+0000 7ffa677fe640 1 -- 172.21.15.29:0/1444678639 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7ffa6c0e4090 con 0x7ffa70102510 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.508+0000 7ffa761e5640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ffa5c0b3df0 0x7ffa5c0b6300 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c0e18a0 tx=0x7ffa6c092040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.508+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/1479773567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 msgr2=0x559043057570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.508+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/1479773567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 0x559043057570 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ffa68008dc0 tx=0x7ffa6800e720 comp rx=0 tx=0).stop 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.508+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/1479773567 shutdown_connections 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.508+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/1479773567 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 0x559043057ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.508+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/1479773567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042c891b0 0x559043057ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.508+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/1479773567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 0x559043057570 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.508+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/1479773567 >> 172.21.15.29:0/1479773567 conn(0x559042ca6460 msgr2=0x559043055460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.508+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/1479773567 shutdown_connections 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.508+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/1479773567 wait complete. 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa7e9cd740 1 -- start start 2024-07-27T06:49:47.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa7e9cd740 1 Processor -- start 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa7e9cd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c891b0 0x559043063180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa7e9cd740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042c91390 0x559043063700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa7e9cd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 0x55904306f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa7e9cd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x559042e20700 con 0x559042cb2070 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa7e9cd740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x559042f5f9e0 con 0x559042c91390 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa7e9cd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559043065d50 con 0x559042c891b0 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa7e9cd740 10 client.0 _ll_register_callbacks cb 0x559042f7a4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa769e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 0x55904306f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa769e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 0x55904306f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59714/0 (socket says 172.21.15.29:59714) 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa769e6640 1 -- 172.21.15.29:0/3012323256 learned_addr learned my addr 172.21.15.29:0/3012323256 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa759e4640 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042c91390 0x559043063700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.509+0000 7ffa769e6640 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c891b0 0x559043063180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa769e6640 1 -- 172.21.15.29:0/3012323256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c891b0 msgr2=0x559043063180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa769e6640 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c891b0 0x559043063180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa769e6640 1 -- 172.21.15.29:0/3012323256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042c91390 msgr2=0x559043063700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa769e6640 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042c91390 0x559043063700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa769e6640 1 -- 172.21.15.29:0/3012323256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559042d1d650 con 0x559042cb2070 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa769e6640 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 0x55904306f040 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ffa68010930 tx=0x7ffa68010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d771915fe43b933b server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa59ffb640 1 -- 172.21.15.29:0/3012323256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68014040 con 0x559042cb2070 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa59ffb640 1 -- 172.21.15.29:0/3012323256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7ffa68010bc0 con 0x559042cb2070 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3012323256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa68003b70 con 0x559042cb2070 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa59ffb640 1 -- 172.21.15.29:0/3012323256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa68010dc0 con 0x559042cb2070 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa7e9cd740 10 client.16326 fetch_fsmap learned FSMap version 32 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3012323256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559043071db0 con 0x559042cb2070 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.510+0000 7ffa59ffb640 1 -- 172.21.15.29:0/3012323256 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa68003b70 con 0x559042cb2070 2024-07-27T06:49:47.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.511+0000 7ffa59ffb640 1 -- 172.21.15.29:0/3012323256 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8382+0+0 (secure 0 0 0) 0x7ffa68023070 con 0x559042cb2070 2024-07-27T06:49:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.511+0000 7ffa7e9cd740 10 client.16326 fetch_fsmap finished waiting for FSMap version 32 2024-07-27T06:49:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.511+0000 7ffa7e9cd740 10 client.16326 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-07-27T06:49:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.511+0000 7ffa7e9cd740 20 client.16326 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.511+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x559042c709f0 0x559043075110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.511+0000 7ffa7e9cd740 4 client.16326 mds_command: new command op to 24646 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.511+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3012323256 --> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559042fcfb00 con 0x559042c709f0 2024-07-27T06:49:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.511+0000 7ffa759e4640 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x559042c709f0 0x559043075110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.512+0000 7ffa759e4640 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x559042c709f0 0x559043075110 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f0162e02c01e50a server_cookie=3e142927c2e4208a in_seq=0 out_seq=0 2024-07-27T06:49:47.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.512+0000 7ffa59ffb640 10 client.16326 ms_handle_connect on v2:172.21.15.77:6834/439792826 2024-07-27T06:49:47.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.512+0000 7ffa59ffb640 1 -- 172.21.15.29:0/3012323256 <== mds.2 v2:172.21.15.77:6834/439792826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559042fcfb00 con 0x559042c709f0 2024-07-27T06:49:47.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.512+0000 7ffa59ffb640 10 client.16326 handle_command_reply: tid=0 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.512+0000 7ffa7e9cd740 1 client.16326 shutdown 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.512+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3012323256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 msgr2=0x55904306f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.512+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 0x55904306f040 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ffa68010930 tx=0x7ffa68010430 comp rx=0 tx=0).stop 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.513+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3012323256 shutdown_connections 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.513+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x559042c709f0 0x559043075110 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.513+0000 7ffa59ffb640 0 client.16326 ms_handle_reset on v2:172.21.15.77:6834/439792826 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.513+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042cb2070 0x55904306f040 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.513+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042c91390 0x559043063700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.513+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/3012323256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c891b0 0x559043063180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.513+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3012323256 >> 172.21.15.29:0/3012323256 conn(0x559042ca6460 msgr2=0x55904305a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.513+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3012323256 shutdown_connections 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.513+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3012323256 wait complete. 2024-07-27T06:49:47.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.513+0000 7ffa7e9cd740 20 client.16326 trim_cache size 0 max 16384 2024-07-27T06:49:47.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.526+0000 7ffa7e9cd740 1 Processor -- start 2024-07-27T06:49:47.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.526+0000 7ffa7e9cd740 1 -- start start 2024-07-27T06:49:47.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.526+0000 7ffa7e9cd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 0x559043062490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.526+0000 7ffa7e9cd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 0x559043054350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.526+0000 7ffa7e9cd740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042cb2070 0x55904306ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.526+0000 7ffa7e9cd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701b12f0 con 0x559042c891b0 2024-07-27T06:49:47.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.526+0000 7ffa7e9cd740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701b2f20 con 0x559042cb2070 2024-07-27T06:49:47.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.526+0000 7ffa7e9cd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701b19c0 con 0x559042c91390 2024-07-27T06:49:47.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.526+0000 7ffa769e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 0x559043062490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.526+0000 7ffa769e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 0x559043062490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59726/0 (socket says 172.21.15.29:59726) 2024-07-27T06:49:47.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.526+0000 7ffa769e6640 1 -- 172.21.15.29:0/1872261347 learned_addr learned my addr 172.21.15.29:0/1872261347 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:47.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.527+0000 7ffa769e6640 1 -- 172.21.15.29:0/1872261347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 msgr2=0x559043054350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:47.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.527+0000 7ffa759e4640 1 --2- 172.21.15.29:0/1872261347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 0x559043054350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.527+0000 7ffa769e6640 1 --2- 172.21.15.29:0/1872261347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 0x559043054350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.527+0000 7ffa769e6640 1 -- 172.21.15.29:0/1872261347 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042cb2070 msgr2=0x55904306ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:47.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.527+0000 7ffa769e6640 1 --2- 172.21.15.29:0/1872261347 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042cb2070 0x55904306ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.527+0000 7ffa769e6640 1 -- 172.21.15.29:0/1872261347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559043059860 con 0x559042c891b0 2024-07-27T06:49:47.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.527+0000 7ffa769e6640 1 --2- 172.21.15.29:0/1872261347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 0x559043062490 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ffa68008dc0 tx=0x7ffa68004800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea13fa3fe2d8d8be server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:47.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.527+0000 7ffa5bfff640 1 -- 172.21.15.29:0/1872261347 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68034030 con 0x559042c891b0 2024-07-27T06:49:47.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.527+0000 7ffa5bfff640 1 -- 172.21.15.29:0/1872261347 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa6800ec20 con 0x559042c891b0 2024-07-27T06:49:47.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.527+0000 7ffa5bfff640 1 -- 172.21.15.29:0/1872261347 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68014040 con 0x559042c891b0 2024-07-27T06:49:47.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.528+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/1872261347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 msgr2=0x559043062490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:47.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.528+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/1872261347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 0x559043062490 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ffa68008dc0 tx=0x7ffa68004800 comp rx=0 tx=0).stop 2024-07-27T06:49:47.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.528+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/1872261347 shutdown_connections 2024-07-27T06:49:47.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.528+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/1872261347 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042cb2070 0x55904306ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.528+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/1872261347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 0x559043054350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.528+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/1872261347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 0x559043062490 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.528+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/1872261347 >> 172.21.15.29:0/1872261347 conn(0x559042ca6460 msgr2=0x559043053c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:47.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.528+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/1872261347 shutdown_connections 2024-07-27T06:49:47.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.528+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/1872261347 wait complete. 2024-07-27T06:49:47.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.528+0000 7ffa7e9cd740 1 -- start start 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.528+0000 7ffa7e9cd740 1 Processor -- start 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa7e9cd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 0x559043066590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa7e9cd740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 0x559043066ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa7e9cd740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042cb2070 0x5590430db760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa7e9cd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701b2790 con 0x559042c891b0 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa7e9cd740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701b2090 con 0x559042cb2070 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa7e9cd740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ffa701af2d0 con 0x559042c91390 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa7e9cd740 10 client.0 _ll_register_callbacks cb 0x7ffa68008c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa769e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 0x559043066590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa769e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 0x559043066590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59728/0 (socket says 172.21.15.29:59728) 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa769e6640 1 -- 172.21.15.29:0/3816483650 learned_addr learned my addr 172.21.15.29:0/3816483650 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa759e4640 1 --2- 172.21.15.29:0/3816483650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 0x559043066ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa769e6640 1 -- 172.21.15.29:0/3816483650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 msgr2=0x559043066ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa769e6640 1 --2- 172.21.15.29:0/3816483650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 0x559043066ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa769e6640 1 -- 172.21.15.29:0/3816483650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042cb2070 msgr2=0x5590430db760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa769e6640 1 --2- 172.21.15.29:0/3816483650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042cb2070 0x5590430db760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa769e6640 1 -- 172.21.15.29:0/3816483650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590430dbca0 con 0x559042c891b0 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa769e6640 1 --2- 172.21.15.29:0/3816483650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 0x559043066590 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ffa68004960 tx=0x7ffa68027980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c9d1293a62e76 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.529+0000 7ffa59ffb640 1 -- 172.21.15.29:0/3816483650 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6803d070 con 0x559042c891b0 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3816483650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa6800ec20 con 0x559042c891b0 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa59ffb640 1 -- 172.21.15.29:0/3816483650 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8500+0+0 (secure 0 0 0) 0x7ffa68004c10 con 0x559042c891b0 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa59ffb640 1 -- 172.21.15.29:0/3816483650 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa68034070 con 0x559042c891b0 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa7e9cd740 10 client.16332 fetch_fsmap learned FSMap version 32 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3816483650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5590430dced0 con 0x559042c891b0 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa59ffb640 1 -- 172.21.15.29:0/3816483650 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa6800ec20 con 0x559042c891b0 2024-07-27T06:49:47.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa59ffb640 1 -- 172.21.15.29:0/3816483650 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) ==== 8382+0+0 (secure 0 0 0) 0x7ffa68023070 con 0x559042c891b0 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa7e9cd740 10 client.16332 fetch_fsmap finished waiting for FSMap version 32 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa7e9cd740 10 client.16332 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa7e9cd740 20 client.16332 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/3816483650 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x559042c45da0 0x5590430df520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa7e9cd740 4 client.16332 mds_command: new command op to 24646 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3816483650 --> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559042fcb5a0 con 0x559042c45da0 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.530+0000 7ffa759e4640 1 --2- 172.21.15.29:0/3816483650 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x559042c45da0 0x5590430df520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa59ffb640 10 client.16332 ms_handle_connect on v2:172.21.15.77:6834/439792826 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa759e4640 1 --2- 172.21.15.29:0/3816483650 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x559042c45da0 0x5590430df520 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1fbf38218700020c server_cookie=25f5a3462b8eae78 in_seq=0 out_seq=0 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa59ffb640 1 -- 172.21.15.29:0/3816483650 <== mds.2 v2:172.21.15.77:6834/439792826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559042fcb5a0 con 0x559042c45da0 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa59ffb640 10 client.16332 handle_command_reply: tid=0 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa7e9cd740 1 client.16332 shutdown 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3816483650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 msgr2=0x559043066590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:47.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/3816483650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 0x559043066590 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ffa68004960 tx=0x7ffa68027980 comp rx=0 tx=0).stop 2024-07-27T06:49:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3816483650 shutdown_connections 2024-07-27T06:49:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/3816483650 >> [v2:172.21.15.77:6834/439792826,v1:172.21.15.77:6835/439792826] conn(0x559042c45da0 0x5590430df520 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/3816483650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x559042cb2070 0x5590430db760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/3816483650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559042c91390 0x559043066ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa7e9cd740 1 --2- 172.21.15.29:0/3816483650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x559042c891b0 0x559043066590 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa59ffb640 0 client.16332 ms_handle_reset on v2:172.21.15.77:6834/439792826 2024-07-27T06:49:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3816483650 >> 172.21.15.29:0/3816483650 conn(0x559042ca6460 msgr2=0x559043058460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3816483650 shutdown_connections 2024-07-27T06:49:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.531+0000 7ffa7e9cd740 1 -- 172.21.15.29:0/3816483650 wait complete. 2024-07-27T06:49:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa7e9cd740 20 client.16332 trim_cache size 0 max 16384 2024-07-27T06:49:47.539 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:49:47.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa78449640 1 -- 172.21.15.29:0/1444678639 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ffa5c0b3df0 msgr2=0x7ffa5c0b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa78449640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ffa5c0b3df0 0x7ffa5c0b6300 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c0e18a0 tx=0x7ffa6c092040 comp rx=0 tx=0).stop 2024-07-27T06:49:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa78449640 1 -- 172.21.15.29:0/1444678639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa70102510 msgr2=0x7ffa7019c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa78449640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa70102510 0x7ffa7019c180 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c00c920 tx=0x7ffa6c00cde0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa78449640 1 -- 172.21.15.29:0/1444678639 shutdown_connections 2024-07-27T06:49:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa78449640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7ffa5c0b3df0 0x7ffa5c0b6300 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa78449640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa701087a0 0x7ffa701a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa78449640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffa70107e50 0x7ffa7019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa78449640 1 --2- 172.21.15.29:0/1444678639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa70102510 0x7ffa7019c180 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa78449640 1 -- 172.21.15.29:0/1444678639 >> 172.21.15.29:0/1444678639 conn(0x7ffa700791a0 msgr2=0x7ffa70105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa78449640 1 -- 172.21.15.29:0/1444678639 shutdown_connections 2024-07-27T06:49:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:47.532+0000 7ffa78449640 1 -- 172.21.15.29:0/1444678639 wait complete. 2024-07-27T06:49:48.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-27T06:49:48.573 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:49:48.661 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:48 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-07-27T06:49:48.661 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:48 smithi077 systemd[1]: Stopped Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:49:48.661 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:48 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Consumed 30.763s CPU time. 2024-07-27T06:49:48.661 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:48 smithi077 systemd[1]: Starting Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:49:48.661 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:48 smithi077 podman[71875]: 2024-07-27 06:49:48.496053793 +0000 UTC m=+0.017565946 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:49:48.661 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:48 smithi077 podman[71875]: 2024-07-27 06:49:48.660662575 +0000 UTC m=+0.182174760 container create cb9afc3d6c6d34c7518cb88a625d3b3c4a538bcb7287f0a45ad1595db876e4db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-27T06:49:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:48.847+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:49:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:48.847+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:49:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:48.847+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-27T06:49:48.939 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:48 smithi077 ceph-mon[37153]: pgmap v481: 97 pgs: 97 active+clean; 44 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 801 KiB/s wr, 75 op/s 2024-07-27T06:49:48.939 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:48 smithi077 podman[71875]: 2024-07-27 06:49:48.905150386 +0000 UTC m=+0.426662543 container init cb9afc3d6c6d34c7518cb88a625d3b3c4a538bcb7287f0a45ad1595db876e4db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-07-27T06:49:48.940 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:48 smithi077 podman[71875]: 2024-07-27 06:49:48.907974165 +0000 UTC m=+0.429486308 container start cb9afc3d6c6d34c7518cb88a625d3b3c4a538bcb7287f0a45ad1595db876e4db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-07-27T06:49:48.947 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:48 smithi029 ceph-mon[32610]: pgmap v481: 97 pgs: 97 active+clean; 44 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 801 KiB/s wr, 75 op/s 2024-07-27T06:49:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:48 smithi047 ceph-mon[36835]: pgmap v481: 97 pgs: 97 active+clean; 44 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 801 KiB/s wr, 75 op/s 2024-07-27T06:49:49.357 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:48 smithi077 ceph-mds[71894]: -- 172.21.15.77:0/4082398155 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557981937860 con 0x557985ad9600 2024-07-27T06:49:49.357 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:48 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: starting mds.i at 2024-07-27T06:49:49.357 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:49 smithi077 bash[71875]: cb9afc3d6c6d34c7518cb88a625d3b3c4a538bcb7287f0a45ad1595db876e4db 2024-07-27T06:49:49.357 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:49:49 smithi077 systemd[1]: Started Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:49:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:49 smithi029 ceph-mon[32610]: daemon mds.i restarted 2024-07-27T06:49:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:49 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:49:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:49 smithi029 ceph-mon[32610]: osdmap e84: 12 total, 12 up, 12 in 2024-07-27T06:49:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:49 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:boot 2024-07-27T06:49:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:49 smithi029 ceph-mon[32610]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:49:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:49 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:49:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:49:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:49 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:49:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:49 smithi077 ceph-mon[37153]: daemon mds.i restarted 2024-07-27T06:49:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:49 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:49:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:49 smithi077 ceph-mon[37153]: osdmap e84: 12 total, 12 up, 12 in 2024-07-27T06:49:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:49 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:boot 2024-07-27T06:49:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:49 smithi077 ceph-mon[37153]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:49:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:49 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:49:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:49:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:49 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:49:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:49 smithi047 ceph-mon[36835]: daemon mds.i restarted 2024-07-27T06:49:50.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:49 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:49:50.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:49 smithi047 ceph-mon[36835]: osdmap e84: 12 total, 12 up, 12 in 2024-07-27T06:49:50.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:49 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:boot 2024-07-27T06:49:50.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:49 smithi047 ceph-mon[36835]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:49:50.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:49 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:49:50.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:49:50.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:49 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:49:50.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.296+0000 7fedef7b0640 1 -- 172.21.15.29:0/3586696747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fede8107df0 msgr2=0x7fede8108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:50.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.296+0000 7fedef7b0640 1 --2- 172.21.15.29:0/3586696747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fede8107df0 0x7fede8108200 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009900 tx=0x7fedd802a3f0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.296+0000 7fedef7b0640 1 -- 172.21.15.29:0/3586696747 shutdown_connections 2024-07-27T06:49:50.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.296+0000 7fedef7b0640 1 --2- 172.21.15.29:0/3586696747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fede8108740 0x7fede810ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.296+0000 7fedef7b0640 1 --2- 172.21.15.29:0/3586696747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fede8107df0 0x7fede8108200 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.296+0000 7fedef7b0640 1 --2- 172.21.15.29:0/3586696747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fede81024b0 0x7fede81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.296+0000 7fedef7b0640 1 -- 172.21.15.29:0/3586696747 >> 172.21.15.29:0/3586696747 conn(0x7fede8079170 msgr2=0x7fede8079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:50.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.296+0000 7fedef7b0640 1 -- 172.21.15.29:0/3586696747 shutdown_connections 2024-07-27T06:49:50.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.296+0000 7fedef7b0640 1 -- 172.21.15.29:0/3586696747 wait complete. 2024-07-27T06:49:50.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7fedef7b0640 1 Processor -- start 2024-07-27T06:49:50.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7fedef7b0640 1 -- start start 2024-07-27T06:49:50.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7fedef7b0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fede81024b0 0x7fede8199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7fedef7b0640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fede8107df0 0x7fede819a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7fedef7b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fede8108740 0x7fede819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7fedef7b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fede8112370 con 0x7fede8108740 2024-07-27T06:49:50.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7fedef7b0640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fede81121f0 con 0x7fede8107df0 2024-07-27T06:49:50.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7fedef7b0640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fede81124f0 con 0x7fede81024b0 2024-07-27T06:49:50.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7fededd4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fede8108740 0x7fede819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7fededd4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fede8108740 0x7fede819f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45198/0 (socket says 172.21.15.29:45198) 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7fededd4d640 1 -- 172.21.15.29:0/3862749157 learned_addr learned my addr 172.21.15.29:0/3862749157 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7fedecd4b640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fede8107df0 0x7fede819a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.297+0000 7feded54c640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fede81024b0 0x7fede8199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.298+0000 7fededd4d640 1 -- 172.21.15.29:0/3862749157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fede81024b0 msgr2=0x7fede8199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.298+0000 7fededd4d640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fede81024b0 0x7fede8199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.298+0000 7fededd4d640 1 -- 172.21.15.29:0/3862749157 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fede8107df0 msgr2=0x7fede819a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.298+0000 7fededd4d640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fede8107df0 0x7fede819a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.298+0000 7fededd4d640 1 -- 172.21.15.29:0/3862749157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede819fd40 con 0x7fede8108740 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.298+0000 7fededd4d640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fede8108740 0x7fede819f6d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fede000eaa0 tx=0x7fede000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.298+0000 7fedde7fc640 1 -- 172.21.15.29:0/3862749157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede0004d20 con 0x7fede8108740 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.298+0000 7fedde7fc640 1 -- 172.21.15.29:0/3862749157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fede0005020 con 0x7fede8108740 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.298+0000 7fedde7fc640 1 -- 172.21.15.29:0/3862749157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede00116b0 con 0x7fede8108740 2024-07-27T06:49:50.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.298+0000 7fedef7b0640 1 -- 172.21.15.29:0/3862749157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede81a0030 con 0x7fede8108740 2024-07-27T06:49:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.298+0000 7fedef7b0640 1 -- 172.21.15.29:0/3862749157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede81a04f0 con 0x7fede8108740 2024-07-27T06:49:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.301+0000 7fedf5d34740 1 Processor -- start 2024-07-27T06:49:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7fedde7fc640 1 -- 172.21.15.29:0/3862749157 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fede00051d0 con 0x7fede8108740 2024-07-27T06:49:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7fedf5d34740 1 -- start start 2024-07-27T06:49:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7fedf5d34740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce2581070 0x561ce2926570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7fedf5d34740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 0x561ce2926ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7fedf5d34740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2926ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7fedf5d34740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561ce289cbb0 con 0x561ce25581b0 2024-07-27T06:49:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7fedf5d34740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x561ce288c0b0 con 0x561ce2560390 2024-07-27T06:49:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7fedf5d34740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ce28900f0 con 0x561ce2581070 2024-07-27T06:49:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7feded54c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 0x561ce2926ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7feded54c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 0x561ce2926ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45214/0 (socket says 172.21.15.29:45214) 2024-07-27T06:49:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7feded54c640 1 -- 172.21.15.29:0/320357368 learned_addr learned my addr 172.21.15.29:0/320357368 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7fedecd4b640 1 --2- 172.21.15.29:0/320357368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce2581070 0x561ce2926570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.302+0000 7fedecd4b640 1 --2- 172.21.15.29:0/320357368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2926ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7feded54c640 1 -- 172.21.15.29:0/320357368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce2581070 msgr2=0x561ce2926570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7feded54c640 1 --2- 172.21.15.29:0/320357368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce2581070 0x561ce2926570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7fedde7fc640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fedd40b3d70 0x7fedd40b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7feded54c640 1 -- 172.21.15.29:0/320357368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 msgr2=0x561ce2926ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7feded54c640 1 --2- 172.21.15.29:0/320357368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2926ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7feded54c640 1 -- 172.21.15.29:0/320357368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ce2832b90 con 0x561ce25581b0 2024-07-27T06:49:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7fedecd4b640 1 --2- 172.21.15.29:0/320357368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2926ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7fededd4d640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fedd40b3d70 0x7fedd40b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7feded54c640 1 --2- 172.21.15.29:0/320357368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 0x561ce2926ab0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fede400ad00 tx=0x7fede400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1c6a8db6c0f7ad server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7fedde7fc640 1 -- 172.21.15.29:0/3862749157 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8586+0+0 (secure 0 0 0) 0x7fede00e4170 con 0x7fede8108740 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7fedd37fe640 1 -- 172.21.15.29:0/320357368 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede401d070 con 0x561ce25581b0 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7fedd37fe640 1 -- 172.21.15.29:0/320357368 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fede4018af0 con 0x561ce25581b0 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.303+0000 7fedd37fe640 1 -- 172.21.15.29:0/320357368 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede4010a70 con 0x561ce25581b0 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.304+0000 7fedf5d34740 1 -- 172.21.15.29:0/320357368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 msgr2=0x561ce2926ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.304+0000 7fededd4d640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fedd40b3d70 0x7fedd40b6280 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fede00e1e80 tx=0x7fede0092040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.304+0000 7fedf5d34740 1 --2- 172.21.15.29:0/320357368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 0x561ce2926ab0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fede400ad00 tx=0x7fede400f790 comp rx=0 tx=0).stop 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.304+0000 7fedf5d34740 1 -- 172.21.15.29:0/320357368 shutdown_connections 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.304+0000 7fedf5d34740 1 --2- 172.21.15.29:0/320357368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2926ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.304+0000 7fedf5d34740 1 --2- 172.21.15.29:0/320357368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 0x561ce2926ab0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.304+0000 7fedf5d34740 1 --2- 172.21.15.29:0/320357368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce2581070 0x561ce2926570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.304+0000 7fedf5d34740 1 -- 172.21.15.29:0/320357368 >> 172.21.15.29:0/320357368 conn(0x561ce2575460 msgr2=0x561ce2924460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.304+0000 7fedf5d34740 1 -- 172.21.15.29:0/320357368 shutdown_connections 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.304+0000 7fedf5d34740 1 -- 172.21.15.29:0/320357368 wait complete. 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.305+0000 7fedf5d34740 1 -- start start 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.305+0000 7fedf5d34740 1 Processor -- start 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.305+0000 7fedf5d34740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce25581b0 0x561ce2936550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.305+0000 7fedf5d34740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2936a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.305+0000 7fedf5d34740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 0x561ce29423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.305+0000 7fedf5d34740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x561ce26ef700 con 0x561ce2581070 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.305+0000 7fedf5d34740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x561ce282e9e0 con 0x561ce2560390 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.305+0000 7fedf5d34740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561ce29390b0 con 0x561ce25581b0 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.305+0000 7fedf5d34740 10 client.0 _ll_register_callbacks cb 0x561ce28494f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.305+0000 7fedecd4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 0x561ce29423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.305+0000 7fedecd4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 0x561ce29423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45228/0 (socket says 172.21.15.29:45228) 2024-07-27T06:49:50.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.305+0000 7fedecd4b640 1 -- 172.21.15.29:0/3193665132 learned_addr learned my addr 172.21.15.29:0/3193665132 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:50.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedecd4b640 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce25581b0 0x561ce2936550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7feded54c640 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2936a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedecd4b640 1 -- 172.21.15.29:0/3193665132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce25581b0 msgr2=0x561ce2936550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:50.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedecd4b640 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce25581b0 0x561ce2936550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedecd4b640 1 -- 172.21.15.29:0/3193665132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 msgr2=0x561ce2936a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:50.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedecd4b640 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2936a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedecd4b640 1 -- 172.21.15.29:0/3193665132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ce25ec650 con 0x561ce2581070 2024-07-27T06:49:50.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7feded54c640 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2936a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:50.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedecd4b640 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 0x561ce29423e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fede806b0f0 tx=0x7fedd8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff9cbcb71806a8d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:50.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedd0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:50.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedd0ff9640 1 -- 172.21.15.29:0/3193665132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8035040 con 0x561ce2581070 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedd0ff9640 1 -- 172.21.15.29:0/3193665132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8586+0+0 (secure 0 0 0) 0x7fedd8005050 con 0x561ce2581070 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedf5d34740 1 -- 172.21.15.29:0/3193665132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fede4018af0 con 0x561ce2581070 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedd0ff9640 1 -- 172.21.15.29:0/3193665132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fedd8053050 con 0x561ce2581070 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedf5d34740 10 client.16347 fetch_fsmap learned FSMap version 34 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.306+0000 7fedf5d34740 1 -- 172.21.15.29:0/3193665132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x561ce2945100 con 0x561ce2581070 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.307+0000 7fedd0ff9640 1 -- 172.21.15.29:0/3193665132 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fede4018af0 con 0x561ce2581070 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.307+0000 7fedd0ff9640 1 -- 172.21.15.29:0/3193665132 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fedd80035f0 con 0x561ce2581070 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.307+0000 7fedf5d34740 10 client.16347 fetch_fsmap finished waiting for FSMap version 34 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.307+0000 7fedf5d34740 10 client.16347 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.307+0000 7fedf5d34740 20 client.16347 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.307+0000 7fedf5d34740 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x561ce253f9f0 0x561ce29485f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.307+0000 7fedf5d34740 4 client.16347 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:50.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.307+0000 7fedf5d34740 1 -- 172.21.15.29:0/3193665132 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ce289eb00 con 0x561ce253f9f0 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.307+0000 7feded54c640 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x561ce253f9f0 0x561ce29485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.308+0000 7feded54c640 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x561ce253f9f0 0x561ce29485f0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=617a3ac53969782d server_cookie=fd2f2a73525f35ba in_seq=0 out_seq=0 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.308+0000 7fedd0ff9640 10 client.16347 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedd0ff9640 1 -- 172.21.15.29:0/3193665132 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561ce289eb00 con 0x561ce253f9f0 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedd0ff9640 10 client.16347 handle_command_reply: tid=0 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedf5d34740 1 client.16347 shutdown 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedf5d34740 1 -- 172.21.15.29:0/3193665132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 msgr2=0x561ce29423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedf5d34740 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 0x561ce29423e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fede806b0f0 tx=0x7fedd8009260 comp rx=0 tx=0).stop 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedf5d34740 1 -- 172.21.15.29:0/3193665132 shutdown_connections 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedf5d34740 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x561ce253f9f0 0x561ce29485f0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedf5d34740 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 0x561ce29423e0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedf5d34740 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2936a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedd0ff9640 0 client.16347 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:49:50.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedf5d34740 1 --2- 172.21.15.29:0/3193665132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce25581b0 0x561ce2936550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedf5d34740 1 -- 172.21.15.29:0/3193665132 >> 172.21.15.29:0/3193665132 conn(0x561ce2575460 msgr2=0x561ce2922c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:50.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedf5d34740 1 -- 172.21.15.29:0/3193665132 shutdown_connections 2024-07-27T06:49:50.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedf5d34740 1 -- 172.21.15.29:0/3193665132 wait complete. 2024-07-27T06:49:50.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.309+0000 7fedf5d34740 20 client.16347 trim_cache size 0 max 16384 2024-07-27T06:49:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.322+0000 7fedf5d34740 1 Processor -- start 2024-07-27T06:49:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.322+0000 7fedf5d34740 1 -- start start 2024-07-27T06:49:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedf5d34740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 0x561ce29232b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedf5d34740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce2560390 0x561ce2924440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedf5d34740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2581070 0x561ce29421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedf5d34740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fede81ad5e0 con 0x561ce25581b0 2024-07-27T06:49:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedf5d34740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fede81aea80 con 0x561ce2581070 2024-07-27T06:49:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedf5d34740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fede81adcb0 con 0x561ce2560390 2024-07-27T06:49:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedecd4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 0x561ce29232b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedecd4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 0x561ce29232b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45230/0 (socket says 172.21.15.29:45230) 2024-07-27T06:49:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedecd4b640 1 -- 172.21.15.29:0/1152538757 learned_addr learned my addr 172.21.15.29:0/1152538757 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:50.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedecd4b640 1 -- 172.21.15.29:0/1152538757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce2560390 msgr2=0x561ce2924440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7feded54c640 1 --2- 172.21.15.29:0/1152538757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce2560390 0x561ce2924440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedecd4b640 1 --2- 172.21.15.29:0/1152538757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce2560390 0x561ce2924440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedecd4b640 1 -- 172.21.15.29:0/1152538757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2581070 msgr2=0x561ce29421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedecd4b640 1 --2- 172.21.15.29:0/1152538757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2581070 0x561ce29421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedecd4b640 1 -- 172.21.15.29:0/1152538757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ce2927360 con 0x561ce25581b0 2024-07-27T06:49:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.323+0000 7fedecd4b640 1 --2- 172.21.15.29:0/1152538757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 0x561ce29232b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009850 tx=0x7fedd8031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9deaf370e07f9f31 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.324+0000 7fedd2ffd640 1 -- 172.21.15.29:0/1152538757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8035070 con 0x561ce25581b0 2024-07-27T06:49:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.324+0000 7fedd2ffd640 1 -- 172.21.15.29:0/1152538757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd80035f0 con 0x561ce25581b0 2024-07-27T06:49:50.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.324+0000 7fedd2ffd640 1 -- 172.21.15.29:0/1152538757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8004c20 con 0x561ce25581b0 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.324+0000 7fedf5d34740 1 -- 172.21.15.29:0/1152538757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 msgr2=0x561ce29232b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.324+0000 7fedf5d34740 1 --2- 172.21.15.29:0/1152538757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 0x561ce29232b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009850 tx=0x7fedd8031400 comp rx=0 tx=0).stop 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.324+0000 7fedf5d34740 1 -- 172.21.15.29:0/1152538757 shutdown_connections 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.324+0000 7fedf5d34740 1 --2- 172.21.15.29:0/1152538757 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2581070 0x561ce29421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.324+0000 7fedf5d34740 1 --2- 172.21.15.29:0/1152538757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce2560390 0x561ce2924440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.324+0000 7fedf5d34740 1 --2- 172.21.15.29:0/1152538757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce25581b0 0x561ce29232b0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.324+0000 7fedf5d34740 1 -- 172.21.15.29:0/1152538757 >> 172.21.15.29:0/1152538757 conn(0x561ce2575460 msgr2=0x561ce2923d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.325+0000 7fedf5d34740 1 -- 172.21.15.29:0/1152538757 shutdown_connections 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.325+0000 7fedf5d34740 1 -- 172.21.15.29:0/1152538757 wait complete. 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.325+0000 7fedf5d34740 1 -- start start 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.325+0000 7fedf5d34740 1 Processor -- start 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedf5d34740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce25581b0 0x561ce293e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedf5d34740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2939e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedf5d34740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 0x561ce293a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedf5d34740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fede81abc10 con 0x561ce2581070 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedf5d34740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fede81ae380 con 0x561ce2560390 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedf5d34740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fede81aaed0 con 0x561ce25581b0 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedf5d34740 10 client.0 _ll_register_callbacks cb 0x7fedd8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedecd4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 0x561ce293a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedecd4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 0x561ce293a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45232/0 (socket says 172.21.15.29:45232) 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedecd4b640 1 -- 172.21.15.29:0/2725575272 learned_addr learned my addr 172.21.15.29:0/2725575272 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedecd4b640 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce25581b0 0x561ce293e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7feded54c640 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2939e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedecd4b640 1 -- 172.21.15.29:0/2725575272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce25581b0 msgr2=0x561ce293e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedecd4b640 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce25581b0 0x561ce293e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedecd4b640 1 -- 172.21.15.29:0/2725575272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 msgr2=0x561ce2939e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedecd4b640 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2939e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedecd4b640 1 -- 172.21.15.29:0/2725575272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ce293aaf0 con 0x561ce2581070 2024-07-27T06:49:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedecd4b640 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 0x561ce293a3b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fedd8006010 tx=0x7fedd8032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0b7ce8c63db5cc server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.326+0000 7fedd0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedd0ff9640 1 -- 172.21.15.29:0/2725575272 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8035070 con 0x561ce2581070 2024-07-27T06:49:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedd0ff9640 1 -- 172.21.15.29:0/2725575272 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8586+0+0 (secure 0 0 0) 0x7fedd804c460 con 0x561ce2581070 2024-07-27T06:49:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedf5d34740 1 -- 172.21.15.29:0/2725575272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedd80035f0 con 0x561ce2581070 2024-07-27T06:49:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedd0ff9640 1 -- 172.21.15.29:0/2725575272 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fedd806c050 con 0x561ce2581070 2024-07-27T06:49:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedf5d34740 10 client.16353 fetch_fsmap learned FSMap version 34 2024-07-27T06:49:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedf5d34740 1 -- 172.21.15.29:0/2725575272 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x561ce29b0c70 con 0x561ce2581070 2024-07-27T06:49:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedd0ff9640 1 -- 172.21.15.29:0/2725575272 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd80035f0 con 0x561ce2581070 2024-07-27T06:49:50.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedd0ff9640 1 -- 172.21.15.29:0/2725575272 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 34) ==== 8370+0+0 (secure 0 0 0) 0x7fedd8064020 con 0x561ce2581070 2024-07-27T06:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedf5d34740 10 client.16353 fetch_fsmap finished waiting for FSMap version 34 2024-07-27T06:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedf5d34740 10 client.16353 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.c 2024-07-27T06:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedf5d34740 20 client.16353 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedf5d34740 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x561ce2514da0 0x561ce29b3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedf5d34740 4 client.16353 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.327+0000 7fedf5d34740 1 -- 172.21.15.29:0/2725575272 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ce289a5a0 con 0x561ce2514da0 2024-07-27T06:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7feded54c640 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x561ce2514da0 0x561ce29b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7feded54c640 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x561ce2514da0 0x561ce29b3280 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4590c40cb02c5f11 server_cookie=955c76077d22596b in_seq=0 out_seq=0 2024-07-27T06:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedd0ff9640 10 client.16353 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedd0ff9640 1 -- 172.21.15.29:0/2725575272 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ce289a5a0 con 0x561ce2514da0 2024-07-27T06:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedd0ff9640 10 client.16353 handle_command_reply: tid=0 2024-07-27T06:49:50.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedf5d34740 1 client.16353 shutdown 2024-07-27T06:49:50.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedf5d34740 1 -- 172.21.15.29:0/2725575272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 msgr2=0x561ce293a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:50.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedf5d34740 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 0x561ce293a3b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fedd8006010 tx=0x7fedd8032bd0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedf5d34740 1 -- 172.21.15.29:0/2725575272 shutdown_connections 2024-07-27T06:49:50.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedf5d34740 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x561ce2514da0 0x561ce29b3280 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedf5d34740 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x561ce2581070 0x561ce293a3b0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedf5d34740 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x561ce2560390 0x561ce2939e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedf5d34740 1 --2- 172.21.15.29:0/2725575272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561ce25581b0 0x561ce293e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedd0ff9640 0 client.16353 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:49:50.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.328+0000 7fedf5d34740 1 -- 172.21.15.29:0/2725575272 >> 172.21.15.29:0/2725575272 conn(0x561ce2575460 msgr2=0x561ce2927530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:50.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedf5d34740 1 -- 172.21.15.29:0/2725575272 shutdown_connections 2024-07-27T06:49:50.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedf5d34740 1 -- 172.21.15.29:0/2725575272 wait complete. 2024-07-27T06:49:50.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedf5d34740 20 client.16353 trim_cache size 0 max 16384 2024-07-27T06:49:50.344 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:49:50.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedef7b0640 1 -- 172.21.15.29:0/3862749157 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fedd40b3d70 msgr2=0x7fedd40b6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:50.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedef7b0640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fedd40b3d70 0x7fedd40b6280 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fede00e1e80 tx=0x7fede0092040 comp rx=0 tx=0).stop 2024-07-27T06:49:50.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedef7b0640 1 -- 172.21.15.29:0/3862749157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fede8108740 msgr2=0x7fede819f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:50.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedef7b0640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fede8108740 0x7fede819f6d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fede000eaa0 tx=0x7fede000eea0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedef7b0640 1 -- 172.21.15.29:0/3862749157 shutdown_connections 2024-07-27T06:49:50.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedef7b0640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fede8108740 0x7fede819f6d0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedef7b0640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fede8107df0 0x7fede819a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedef7b0640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fedd40b3d70 0x7fedd40b6280 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedef7b0640 1 --2- 172.21.15.29:0/3862749157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fede81024b0 0x7fede8199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:50.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedef7b0640 1 -- 172.21.15.29:0/3862749157 >> 172.21.15.29:0/3862749157 conn(0x7fede8079170 msgr2=0x7fede806b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:50.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedef7b0640 1 -- 172.21.15.29:0/3862749157 shutdown_connections 2024-07-27T06:49:50.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:50.329+0000 7fedef7b0640 1 -- 172.21.15.29:0/3862749157 wait complete. 2024-07-27T06:49:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:50 smithi029 ceph-mon[32610]: pgmap v483: 97 pgs: 97 active+clean; 44 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 453 KiB/s wr, 58 op/s 2024-07-27T06:49:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:50 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:resolve 2024-07-27T06:49:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:50 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:49:51.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-27T06:49:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:50 smithi077 ceph-mon[37153]: pgmap v483: 97 pgs: 97 active+clean; 44 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 453 KiB/s wr, 58 op/s 2024-07-27T06:49:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:50 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:resolve 2024-07-27T06:49:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:50 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:49:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:50 smithi047 ceph-mon[36835]: pgmap v483: 97 pgs: 97 active+clean; 44 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 453 KiB/s wr, 58 op/s 2024-07-27T06:49:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:50 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:resolve 2024-07-27T06:49:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:50 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:49:51.491 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:49:52.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:52 smithi029 ceph-mon[32610]: pgmap v484: 97 pgs: 97 active+clean; 44 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 361 KiB/s wr, 53 op/s 2024-07-27T06:49:52.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:52 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0 2024-07-27T06:49:52.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:52 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:reconnect 2024-07-27T06:49:52.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:52 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:49:52.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:52 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00599987 2024-07-27T06:49:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:52 smithi077 ceph-mon[37153]: pgmap v484: 97 pgs: 97 active+clean; 44 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 361 KiB/s wr, 53 op/s 2024-07-27T06:49:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:52 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0 2024-07-27T06:49:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:52 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:reconnect 2024-07-27T06:49:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:52 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:49:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:52 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00599987 2024-07-27T06:49:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:52 smithi047 ceph-mon[36835]: pgmap v484: 97 pgs: 97 active+clean; 44 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 361 KiB/s wr, 53 op/s 2024-07-27T06:49:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:52 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0 2024-07-27T06:49:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:52 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:reconnect 2024-07-27T06:49:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:52 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:49:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:52 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00599987 2024-07-27T06:49:53.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.297+0000 7fc3c29cc640 1 -- 172.21.15.29:0/3585875499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc102490 msgr2=0x7fc3bc1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:53.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.297+0000 7fc3c29cc640 1 --2- 172.21.15.29:0/3585875499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc102490 0x7fc3bc1028a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac009900 tx=0x7fc3ac02a410 comp rx=0 tx=0).stop 2024-07-27T06:49:53.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.297+0000 7fc3c29cc640 1 -- 172.21.15.29:0/3585875499 shutdown_connections 2024-07-27T06:49:53.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.297+0000 7fc3c29cc640 1 --2- 172.21.15.29:0/3585875499 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3bc108720 0x7fc3bc10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.297+0000 7fc3c29cc640 1 --2- 172.21.15.29:0/3585875499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3bc107dd0 0x7fc3bc1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.297+0000 7fc3c29cc640 1 --2- 172.21.15.29:0/3585875499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc102490 0x7fc3bc1028a0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.297+0000 7fc3c29cc640 1 -- 172.21.15.29:0/3585875499 >> 172.21.15.29:0/3585875499 conn(0x7fc3bc079160 msgr2=0x7fc3bc079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.297+0000 7fc3c29cc640 1 -- 172.21.15.29:0/3585875499 shutdown_connections 2024-07-27T06:49:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.297+0000 7fc3c29cc640 1 -- 172.21.15.29:0/3585875499 wait complete. 2024-07-27T06:49:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.298+0000 7fc3c29cc640 1 Processor -- start 2024-07-27T06:49:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.298+0000 7fc3c29cc640 1 -- start start 2024-07-27T06:49:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.298+0000 7fc3c29cc640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3bc102490 0x7fc3bc103c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.298+0000 7fc3c29cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3bc107dd0 0x7fc3bc1041c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c29cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc108720 0x7fc3bc104700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c29cc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc3bc1123a0 con 0x7fc3bc108720 2024-07-27T06:49:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c29cc640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fc3bc112220 con 0x7fc3bc102490 2024-07-27T06:49:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c29cc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc3bc112520 con 0x7fc3bc107dd0 2024-07-27T06:49:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c0f69640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc108720 0x7fc3bc104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c0f69640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc108720 0x7fc3bc104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45250/0 (socket says 172.21.15.29:45250) 2024-07-27T06:49:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c0f69640 1 -- 172.21.15.29:0/1087388199 learned_addr learned my addr 172.21.15.29:0/1087388199 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3bc107dd0 0x7fc3bc1041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3bc102490 0x7fc3bc103c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c0f69640 1 -- 172.21.15.29:0/1087388199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3bc107dd0 msgr2=0x7fc3bc1041c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c0f69640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3bc107dd0 0x7fc3bc1041c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c0f69640 1 -- 172.21.15.29:0/1087388199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3bc102490 msgr2=0x7fc3bc103c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:53.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c0f69640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3bc102490 0x7fc3bc103c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c0f69640 1 -- 172.21.15.29:0/1087388199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3bc1a8180 con 0x7fc3bc108720 2024-07-27T06:49:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.299+0000 7fc3c0f69640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc108720 0x7fc3bc104700 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc3b400cad0 tx=0x7fc3b400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.300+0000 7fc3b97fa640 1 -- 172.21.15.29:0/1087388199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b400ae20 con 0x7fc3bc108720 2024-07-27T06:49:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.300+0000 7fc3c29cc640 1 -- 172.21.15.29:0/1087388199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3bc1a8470 con 0x7fc3bc108720 2024-07-27T06:49:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.300+0000 7fc3b97fa640 1 -- 172.21.15.29:0/1087388199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b4005020 con 0x7fc3bc108720 2024-07-27T06:49:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.300+0000 7fc3c29cc640 1 -- 172.21.15.29:0/1087388199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3bc1a8a00 con 0x7fc3bc108720 2024-07-27T06:49:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.300+0000 7fc3b97fa640 1 -- 172.21.15.29:0/1087388199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b400f6b0 con 0x7fc3bc108720 2024-07-27T06:49:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.302+0000 7fc3c8f50740 1 Processor -- start 2024-07-27T06:49:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.303+0000 7fc3b97fa640 1 -- 172.21.15.29:0/1087388199 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fc3b4004b20 con 0x7fc3bc108720 2024-07-27T06:49:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.303+0000 7fc3c8f50740 1 -- start start 2024-07-27T06:49:53.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.303+0000 7fc3c8f50740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 0x560eee638570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.303+0000 7fc3c8f50740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee638ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.303+0000 7fc3c8f50740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee272390 0x560eee638ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.303+0000 7fc3c8f50740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560eee5aebb0 con 0x560eee293070 2024-07-27T06:49:53.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.303+0000 7fc3c8f50740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x560eee59e0b0 con 0x560eee272390 2024-07-27T06:49:53.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.303+0000 7fc3c8f50740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560eee5a20f0 con 0x560eee26a1b0 2024-07-27T06:49:53.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.303+0000 7fc3bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 0x560eee638570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.303+0000 7fc3bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 0x560eee638570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45256/0 (socket says 172.21.15.29:45256) 2024-07-27T06:49:53.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.303+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/4120803286 learned_addr learned my addr 172.21.15.29:0/4120803286 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:53.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.303+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/4120803286 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee272390 0x560eee638ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/4120803286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee638ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/4120803286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 msgr2=0x560eee638ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:53.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/4120803286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee638ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/4120803286 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee272390 msgr2=0x560eee638ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:53.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/4120803286 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee272390 0x560eee638ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/4120803286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eee544b90 con 0x560eee293070 2024-07-27T06:49:53.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3b97fa640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc3a40b3d40 0x7fc3a40b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/4120803286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee638ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T06:49:53.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc3a40b3d40 0x7fc3a40b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3b97fa640 1 -- 172.21.15.29:0/1087388199 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8586+0+0 (secure 0 0 0) 0x7fc3b40e3da0 con 0x7fc3bc108720 2024-07-27T06:49:53.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/4120803286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 0x560eee638570 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00138d0 tx=0x7fc3b000a720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbca26c2da485ee5 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3a27fc640 1 -- 172.21.15.29:0/4120803286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b00110d0 con 0x560eee293070 2024-07-27T06:49:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3a27fc640 1 -- 172.21.15.29:0/4120803286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b00054d0 con 0x560eee293070 2024-07-27T06:49:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.304+0000 7fc3a27fc640 1 -- 172.21.15.29:0/4120803286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0004b90 con 0x560eee293070 2024-07-27T06:49:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.305+0000 7fc3c8f50740 1 -- 172.21.15.29:0/4120803286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 msgr2=0x560eee638570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.305+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc3a40b3d40 0x7fc3a40b6250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac000c00 tx=0x7fc3ac031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.305+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/4120803286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 0x560eee638570 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00138d0 tx=0x7fc3b000a720 comp rx=0 tx=0).stop 2024-07-27T06:49:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.305+0000 7fc3c8f50740 1 -- 172.21.15.29:0/4120803286 shutdown_connections 2024-07-27T06:49:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.305+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/4120803286 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee272390 0x560eee638ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.305+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/4120803286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee638ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.305+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/4120803286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 0x560eee638570 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.305+0000 7fc3c8f50740 1 -- 172.21.15.29:0/4120803286 >> 172.21.15.29:0/4120803286 conn(0x560eee287460 msgr2=0x560eee636460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.305+0000 7fc3c8f50740 1 -- 172.21.15.29:0/4120803286 shutdown_connections 2024-07-27T06:49:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.305+0000 7fc3c8f50740 1 -- 172.21.15.29:0/4120803286 wait complete. 2024-07-27T06:49:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.306+0000 7fc3c8f50740 1 -- start start 2024-07-27T06:49:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.306+0000 7fc3c8f50740 1 Processor -- start 2024-07-27T06:49:53.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.306+0000 7fc3c8f50740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee644140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.306+0000 7fc3c8f50740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 0x560eee644680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3c8f50740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee293070 0x560eee650040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3c8f50740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560eee401700 con 0x560eee272390 2024-07-27T06:49:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3c8f50740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x560eee5409e0 con 0x560eee293070 2024-07-27T06:49:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3c8f50740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560eee646cf0 con 0x560eee26a1b0 2024-07-27T06:49:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 0x560eee644680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 0x560eee644680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45264/0 (socket says 172.21.15.29:45264) 2024-07-27T06:49:53.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/2762312144 learned_addr learned my addr 172.21.15.29:0/2762312144 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:53.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3c8f50740 10 client.0 _ll_register_callbacks cb 0x560eee55b4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:53.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee644140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3c0f69640 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee293070 0x560eee650040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/2762312144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 msgr2=0x560eee644140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:53.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee644140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/2762312144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee293070 msgr2=0x560eee650040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:53.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee293070 0x560eee650040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/2762312144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eee2fe650 con 0x560eee272390 2024-07-27T06:49:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3c0f69640 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee293070 0x560eee650040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 0x560eee644680 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0013f50 tx=0x7fc3b0005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792ffc0e15dbca99 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.307+0000 7fc39b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.308+0000 7fc39b7fe640 1 -- 172.21.15.29:0/2762312144 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b00046c0 con 0x560eee272390 2024-07-27T06:49:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.308+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2762312144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3b00054d0 con 0x560eee272390 2024-07-27T06:49:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.308+0000 7fc39b7fe640 1 -- 172.21.15.29:0/2762312144 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8586+0+0 (secure 0 0 0) 0x7fc3b0005230 con 0x560eee272390 2024-07-27T06:49:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.308+0000 7fc39b7fe640 1 -- 172.21.15.29:0/2762312144 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc3b0004910 con 0x560eee272390 2024-07-27T06:49:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.308+0000 7fc3c8f50740 10 client.16365 fetch_fsmap learned FSMap version 37 2024-07-27T06:49:53.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.308+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2762312144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560eee652c60 con 0x560eee272390 2024-07-27T06:49:53.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.309+0000 7fc39b7fe640 1 -- 172.21.15.29:0/2762312144 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b00054d0 con 0x560eee272390 2024-07-27T06:49:53.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.309+0000 7fc39b7fe640 1 -- 172.21.15.29:0/2762312144 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fc3b001d020 con 0x560eee272390 2024-07-27T06:49:53.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.309+0000 7fc3c8f50740 10 client.16365 fetch_fsmap finished waiting for FSMap version 37 2024-07-27T06:49:53.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.309+0000 7fc3c8f50740 10 client.16365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-27T06:49:53.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.309+0000 7fc3c8f50740 20 client.16365 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:53.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.309+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x560eee2519f0 0x560eee655fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.309+0000 7fc3c8f50740 4 client.16365 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:53.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.309+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2762312144 --> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560eee5b0b00 con 0x560eee2519f0 2024-07-27T06:49:53.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.309+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x560eee2519f0 0x560eee655fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.310+0000 7fc39b7fe640 10 client.16365 ms_handle_connect on v2:172.21.15.47:6836/2573307312 2024-07-27T06:49:53.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.310+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x560eee2519f0 0x560eee655fc0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=946320269e1d6261 server_cookie=8ba074a57814c31b in_seq=0 out_seq=0 2024-07-27T06:49:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc39b7fe640 1 -- 172.21.15.29:0/2762312144 <== mds.0 v2:172.21.15.47:6836/2573307312 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560eee5b0b00 con 0x560eee2519f0 2024-07-27T06:49:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc39b7fe640 10 client.16365 handle_command_reply: tid=0 2024-07-27T06:49:53.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc3c8f50740 1 client.16365 shutdown 2024-07-27T06:49:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2762312144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 msgr2=0x560eee644680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 0x560eee644680 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0013f50 tx=0x7fc3b0005e30 comp rx=0 tx=0).stop 2024-07-27T06:49:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2762312144 shutdown_connections 2024-07-27T06:49:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x560eee2519f0 0x560eee655fc0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee293070 0x560eee650040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 0x560eee644680 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/2762312144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee644140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2762312144 >> 172.21.15.29:0/2762312144 conn(0x560eee287460 msgr2=0x560eee634c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:53.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc39b7fe640 0 client.16365 ms_handle_reset on v2:172.21.15.47:6836/2573307312 2024-07-27T06:49:53.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2762312144 shutdown_connections 2024-07-27T06:49:53.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2762312144 wait complete. 2024-07-27T06:49:53.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.311+0000 7fc3c8f50740 20 client.16365 trim_cache size 0 max 16384 2024-07-27T06:49:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3c8f50740 1 Processor -- start 2024-07-27T06:49:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3c8f50740 1 -- start start 2024-07-27T06:49:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3c8f50740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee63dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3c8f50740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 0x560eee64fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3c8f50740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee293070 0x560eee635e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3c8f50740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a800ef40 con 0x560eee272390 2024-07-27T06:49:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3c8f50740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a80103e0 con 0x560eee293070 2024-07-27T06:49:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3c8f50740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a800f610 con 0x560eee26a1b0 2024-07-27T06:49:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 0x560eee64fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3bbfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee63dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 0x560eee64fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45280/0 (socket says 172.21.15.29:45280) 2024-07-27T06:49:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1798426538 learned_addr learned my addr 172.21.15.29:0/1798426538 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.325+0000 7fc3c0f69640 1 --2- 172.21.15.29:0/1798426538 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee293070 0x560eee635e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.326+0000 7fc3bbfff640 1 -- 172.21.15.29:0/1798426538 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee293070 msgr2=0x560eee635e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.326+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/1798426538 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee293070 0x560eee635e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.326+0000 7fc3bbfff640 1 -- 172.21.15.29:0/1798426538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 msgr2=0x560eee64fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.326+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/1798426538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 0x560eee64fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.326+0000 7fc3bbfff640 1 -- 172.21.15.29:0/1798426538 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eee63b3d0 con 0x560eee26a1b0 2024-07-27T06:49:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.326+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/1798426538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 0x560eee64fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:49:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.326+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/1798426538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee63dd90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc1056d0 tx=0x7fc3ac03f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cc982322fe14258 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.326+0000 7fc3a1ffb640 1 -- 172.21.15.29:0/1798426538 <== mon.1 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac035050 con 0x560eee26a1b0 2024-07-27T06:49:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.326+0000 7fc3a1ffb640 1 -- 172.21.15.29:0/1798426538 <== mon.1 v2:172.21.15.77:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3ac056070 con 0x560eee26a1b0 2024-07-27T06:49:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.326+0000 7fc3a1ffb640 1 -- 172.21.15.29:0/1798426538 <== mon.1 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac003d60 con 0x560eee26a1b0 2024-07-27T06:49:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.327+0000 7fc3c8f50740 1 -- 172.21.15.29:0/1798426538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 msgr2=0x560eee63dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.327+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/1798426538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee63dd90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc1056d0 tx=0x7fc3ac03f000 comp rx=0 tx=0).stop 2024-07-27T06:49:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.327+0000 7fc3c8f50740 1 -- 172.21.15.29:0/1798426538 shutdown_connections 2024-07-27T06:49:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.327+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/1798426538 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee293070 0x560eee635e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.327+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/1798426538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee272390 0x560eee64fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.327+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/1798426538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee63dd90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.327+0000 7fc3c8f50740 1 -- 172.21.15.29:0/1798426538 >> 172.21.15.29:0/1798426538 conn(0x560eee287460 msgr2=0x560eee63d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.327+0000 7fc3c8f50740 1 -- 172.21.15.29:0/1798426538 shutdown_connections 2024-07-27T06:49:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.327+0000 7fc3c8f50740 1 -- 172.21.15.29:0/1798426538 wait complete. 2024-07-27T06:49:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3c8f50740 1 -- start start 2024-07-27T06:49:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3c8f50740 1 Processor -- start 2024-07-27T06:49:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3c8f50740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee64c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3c8f50740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee272390 0x560eee647bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3c8f50740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 0x560eee6481b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3c8f50740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a800d570 con 0x560eee293070 2024-07-27T06:49:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3c8f50740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a800fce0 con 0x560eee272390 2024-07-27T06:49:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3c8f50740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a800c830 con 0x560eee26a1b0 2024-07-27T06:49:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3c8f50740 10 client.0 _ll_register_callbacks cb 0x7fc3b000ad60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3c0f69640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 0x560eee6481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3c0f69640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 0x560eee6481b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45288/0 (socket says 172.21.15.29:45288) 2024-07-27T06:49:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3c0f69640 1 -- 172.21.15.29:0/2910931994 learned_addr learned my addr 172.21.15.29:0/2910931994 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.328+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee272390 0x560eee647bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee64c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc3c0f69640 1 -- 172.21.15.29:0/2910931994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 msgr2=0x560eee64c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc3c0f69640 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee64c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc3c0f69640 1 -- 172.21.15.29:0/2910931994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee272390 msgr2=0x560eee647bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc3c0f69640 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee272390 0x560eee647bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc3c0f69640 1 -- 172.21.15.29:0/2910931994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eee648940 con 0x560eee293070 2024-07-27T06:49:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc3c0f69640 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 0x560eee6481b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fc3b40e1560 tx=0x7fc3b40a6eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c3fca38af1785f server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc39b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc39b7fe640 1 -- 172.21.15.29:0/2910931994 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b401d040 con 0x560eee293070 2024-07-27T06:49:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc39b7fe640 1 -- 172.21.15.29:0/2910931994 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8586+0+0 (secure 0 0 0) 0x7fc3b40c5240 con 0x560eee293070 2024-07-27T06:49:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2910931994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3ac056070 con 0x560eee293070 2024-07-27T06:49:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc39b7fe640 1 -- 172.21.15.29:0/2910931994 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc3b40b3360 con 0x560eee293070 2024-07-27T06:49:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc3c8f50740 10 client.16371 fetch_fsmap learned FSMap version 37 2024-07-27T06:49:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.329+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2910931994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560eee6be6a0 con 0x560eee293070 2024-07-27T06:49:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.330+0000 7fc39b7fe640 1 -- 172.21.15.29:0/2910931994 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3ac056070 con 0x560eee293070 2024-07-27T06:49:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.330+0000 7fc39b7fe640 1 -- 172.21.15.29:0/2910931994 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 37) ==== 8370+0+0 (secure 0 0 0) 0x7fc3b40b3040 con 0x560eee293070 2024-07-27T06:49:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.330+0000 7fc3c8f50740 10 client.16371 fetch_fsmap finished waiting for FSMap version 37 2024-07-27T06:49:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.330+0000 7fc3c8f50740 10 client.16371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-27T06:49:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.330+0000 7fc3c8f50740 20 client.16371 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.330+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x560eee226da0 0x560eee6c0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.330+0000 7fc3c8f50740 4 client.16371 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-27T06:49:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.330+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2910931994 --> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560eee5ac5a0 con 0x560eee226da0 2024-07-27T06:49:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.330+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x560eee226da0 0x560eee6c0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x560eee226da0 0x560eee6c0e90 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cadf9e9992cfc575 server_cookie=53f6c2238aada57c in_seq=0 out_seq=0 2024-07-27T06:49:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc39b7fe640 10 client.16371 ms_handle_connect on v2:172.21.15.47:6836/2573307312 2024-07-27T06:49:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc39b7fe640 1 -- 172.21.15.29:0/2910931994 <== mds.0 v2:172.21.15.47:6836/2573307312 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560eee5ac5a0 con 0x560eee226da0 2024-07-27T06:49:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc39b7fe640 10 client.16371 handle_command_reply: tid=0 2024-07-27T06:49:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3c8f50740 1 client.16371 shutdown 2024-07-27T06:49:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2910931994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 msgr2=0x560eee6481b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 0x560eee6481b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fc3b40e1560 tx=0x7fc3b40a6eb0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2910931994 shutdown_connections 2024-07-27T06:49:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x560eee226da0 0x560eee6c0e90 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560eee293070 0x560eee6481b0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560eee272390 0x560eee647bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3c8f50740 1 --2- 172.21.15.29:0/2910931994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560eee26a1b0 0x560eee64c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2910931994 >> 172.21.15.29:0/2910931994 conn(0x560eee287460 msgr2=0x560eee635d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc39b7fe640 0 client.16371 ms_handle_reset on v2:172.21.15.47:6836/2573307312 2024-07-27T06:49:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2910931994 shutdown_connections 2024-07-27T06:49:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3c8f50740 1 -- 172.21.15.29:0/2910931994 wait complete. 2024-07-27T06:49:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.331+0000 7fc3c8f50740 20 client.16371 trim_cache size 0 max 16384 2024-07-27T06:49:53.343 INFO:teuthology.orchestra.run.smithi029.stdout:[] 2024-07-27T06:49:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.332+0000 7fc3c29cc640 1 -- 172.21.15.29:0/1087388199 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc3a40b3d40 msgr2=0x7fc3a40b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.332+0000 7fc3c29cc640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc3a40b3d40 0x7fc3a40b6250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac000c00 tx=0x7fc3ac031400 comp rx=0 tx=0).stop 2024-07-27T06:49:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.332+0000 7fc3c29cc640 1 -- 172.21.15.29:0/1087388199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc108720 msgr2=0x7fc3bc104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.332+0000 7fc3c29cc640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc108720 0x7fc3bc104700 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc3b400cad0 tx=0x7fc3b400cf90 comp rx=0 tx=0).stop 2024-07-27T06:49:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.332+0000 7fc3c29cc640 1 -- 172.21.15.29:0/1087388199 shutdown_connections 2024-07-27T06:49:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.332+0000 7fc3c29cc640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc108720 0x7fc3bc104700 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.332+0000 7fc3c29cc640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fc3a40b3d40 0x7fc3a40b6250 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.332+0000 7fc3c29cc640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3bc107dd0 0x7fc3bc1041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.332+0000 7fc3c29cc640 1 --2- 172.21.15.29:0/1087388199 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3bc102490 0x7fc3bc103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.332+0000 7fc3c29cc640 1 -- 172.21.15.29:0/1087388199 >> 172.21.15.29:0/1087388199 conn(0x7fc3bc079160 msgr2=0x7fc3bc101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.332+0000 7fc3c29cc640 1 -- 172.21.15.29:0/1087388199 shutdown_connections 2024-07-27T06:49:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:53.332+0000 7fc3c29cc640 1 -- 172.21.15.29:0/1087388199 wait complete. 2024-07-27T06:49:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:53 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:rejoin 2024-07-27T06:49:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:53 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:49:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:53 smithi077 ceph-mon[37153]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:49:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:53 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:rejoin 2024-07-27T06:49:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:53 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:49:54.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:53 smithi029 ceph-mon[32610]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:49:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:rejoin 2024-07-27T06:49:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:49:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-mon[36835]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:49:54.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:49:54.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: *** Caught signal (Aborted) ** 2024-07-27T06:49:54.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: in thread 7f3bea407640 thread_name:ms_dispatch 2024-07-27T06:49:54.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:49:54.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3bef7de6f0] 2024-07-27T06:49:54.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3bef82b94c] 2024-07-27T06:49:54.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 3: raise() 2024-07-27T06:49:54.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 4: abort() 2024-07-27T06:49:54.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3bef7c871b] 2024-07-27T06:49:54.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 6: /lib64/libc.so.6(+0x37386) [0x7f3bef7d7386] 2024-07-27T06:49:54.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d4e5fc66b9] 2024-07-27T06:49:54.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d4e618480c] 2024-07-27T06:49:54.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d4e60d2cde] 2024-07-27T06:49:54.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d4e6094763] 2024-07-27T06:49:54.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d4e6095a1b] 2024-07-27T06:49:54.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d4e6079752] 2024-07-27T06:49:54.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d4e5fc7a56] 2024-07-27T06:49:54.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d4e5fcb637] 2024-07-27T06:49:54.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d4e5fcbf5e] 2024-07-27T06:49:54.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d4e5faf03a] 2024-07-27T06:49:54.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3bf00ab407] 2024-07-27T06:49:54.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 18: (DispatchQueue::entry()+0x60f) [0x7f3bf00abcf9] 2024-07-27T06:49:54.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3bf01885a1] 2024-07-27T06:49:54.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 20: (Thread::entry_wrapper()+0x43) [0x7f3beff35cdd] 2024-07-27T06:49:54.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3beff35cf9] 2024-07-27T06:49:54.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3bef829c02] 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3bef8aec40] 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 2024-07-27T06:49:53.841+0000 7f3bea407640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: in thread 7f3bea407640 thread_name:ms_dispatch 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3bef7de6f0] 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3bef82b94c] 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 3: raise() 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 4: abort() 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3bef7c871b] 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 6: /lib64/libc.so.6(+0x37386) [0x7f3bef7d7386] 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d4e5fc66b9] 2024-07-27T06:49:54.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d4e618480c] 2024-07-27T06:49:54.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d4e60d2cde] 2024-07-27T06:49:54.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d4e6094763] 2024-07-27T06:49:54.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d4e6095a1b] 2024-07-27T06:49:54.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d4e6079752] 2024-07-27T06:49:54.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d4e5fc7a56] 2024-07-27T06:49:54.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d4e5fcb637] 2024-07-27T06:49:54.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d4e5fcbf5e] 2024-07-27T06:49:54.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d4e5faf03a] 2024-07-27T06:49:54.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3bf00ab407] 2024-07-27T06:49:54.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 18: (DispatchQueue::entry()+0x60f) [0x7f3bf00abcf9] 2024-07-27T06:49:54.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3bf01885a1] 2024-07-27T06:49:54.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 20: (Thread::entry_wrapper()+0x43) [0x7f3beff35cdd] 2024-07-27T06:49:54.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3beff35cf9] 2024-07-27T06:49:54.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3bef829c02] 2024-07-27T06:49:54.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3bef8aec40] 2024-07-27T06:49:54.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:49:54.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 2024-07-27T06:49:54.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 0> 2024-07-27T06:49:53.841+0000 7f3bea407640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:49:54.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: in thread 7f3bea407640 thread_name:ms_dispatch 2024-07-27T06:49:54.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 2024-07-27T06:49:54.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:49:54.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3bef7de6f0] 2024-07-27T06:49:54.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3bef82b94c] 2024-07-27T06:49:54.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 3: raise() 2024-07-27T06:49:54.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 4: abort() 2024-07-27T06:49:54.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3bef7c871b] 2024-07-27T06:49:54.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 6: /lib64/libc.so.6(+0x37386) [0x7f3bef7d7386] 2024-07-27T06:49:54.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d4e5fc66b9] 2024-07-27T06:49:54.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d4e618480c] 2024-07-27T06:49:54.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d4e60d2cde] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d4e6094763] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d4e6095a1b] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d4e6079752] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d4e5fc7a56] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d4e5fcb637] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d4e5fcbf5e] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d4e5faf03a] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3bf00ab407] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 18: (DispatchQueue::entry()+0x60f) [0x7f3bf00abcf9] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3bf01885a1] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 20: (Thread::entry_wrapper()+0x43) [0x7f3beff35cdd] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3beff35cf9] 2024-07-27T06:49:54.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3bef829c02] 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3bef8aec40] 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: -3> 2024-07-27T06:49:53.841+0000 7f3bea407640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: in thread 7f3bea407640 thread_name:ms_dispatch 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3bef7de6f0] 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3bef82b94c] 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 3: raise() 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 4: abort() 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3bef7c871b] 2024-07-27T06:49:54.208 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 6: /lib64/libc.so.6(+0x37386) [0x7f3bef7d7386] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d4e5fc66b9] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d4e618480c] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d4e60d2cde] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d4e6094763] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d4e6095a1b] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d4e6079752] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d4e5fc7a56] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d4e5fcb637] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d4e5fcbf5e] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d4e5faf03a] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3bf00ab407] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 18: (DispatchQueue::entry()+0x60f) [0x7f3bf00abcf9] 2024-07-27T06:49:54.209 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3bf01885a1] 2024-07-27T06:49:54.210 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 20: (Thread::entry_wrapper()+0x43) [0x7f3beff35cdd] 2024-07-27T06:49:54.210 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3beff35cf9] 2024-07-27T06:49:54.210 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3bef829c02] 2024-07-27T06:49:54.210 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3bef8aec40] 2024-07-27T06:49:54.210 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:49:54.210 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[68346]: 2024-07-27T06:49:54.210 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:53 smithi047 podman[71168]: 2024-07-27 06:49:53.90527381 +0000 UTC m=+0.021884799 container died d8f4ee956badb55e0699ae09ded162d0cd9f9944acff00bd81f3e1b45513f034 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-07-27T06:49:55.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:54 smithi047 ceph-mon[36835]: pgmap v485: 97 pgs: 97 active+clean; 44 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 375 KiB/s wr, 59 op/s 2024-07-27T06:49:55.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:54 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:49:55.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:54 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:49:55.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:54 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:active 2024-07-27T06:49:55.060 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:54 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:49:55.061 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:54 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:49:55.061 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:54 smithi047 podman[71168]: 2024-07-27 06:49:54.79670516 +0000 UTC m=+0.913316186 container remove d8f4ee956badb55e0699ae09ded162d0cd9f9944acff00bd81f3e1b45513f034 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-07-27T06:49:55.061 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:54 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:49:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:54 smithi077 ceph-mon[37153]: pgmap v485: 97 pgs: 97 active+clean; 44 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 375 KiB/s wr, 59 op/s 2024-07-27T06:49:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:54 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:49:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:54 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:49:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:54 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:active 2024-07-27T06:49:55.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:54 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:49:55.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:54 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:49:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:54 smithi029 ceph-mon[32610]: pgmap v485: 97 pgs: 97 active+clean; 44 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 375 KiB/s wr, 59 op/s 2024-07-27T06:49:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:54 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:49:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:54 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:49:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:54 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] up:active 2024-07-27T06:49:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:54 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:49:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:54 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:49:55.254 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-27T06:49:55.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-27T06:49:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:55.255+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:55.256+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-27T06:49:55.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:55.256+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:49:55.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:55.256+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:49:55.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:49:55.256+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-27T06:49:55.446 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:55 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-27T06:49:55.446 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:49:55 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Consumed 3.880s CPU time. 2024-07-27T06:49:55.536 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:49:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:56 smithi029 ceph-mon[32610]: pgmap v486: 97 pgs: 97 active+clean; 44 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 204 KiB/s wr, 41 op/s 2024-07-27T06:49:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:56 smithi077 ceph-mon[37153]: pgmap v486: 97 pgs: 97 active+clean; 44 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 204 KiB/s wr, 41 op/s 2024-07-27T06:49:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:56 smithi047 ceph-mon[36835]: pgmap v486: 97 pgs: 97 active+clean; 44 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 204 KiB/s wr, 41 op/s 2024-07-27T06:49:57.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.639+0000 7eff549ff640 1 -- 172.21.15.29:0/3149654494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5010d630 msgr2=0x7eff5010fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:57.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.639+0000 7eff549ff640 1 --2- 172.21.15.29:0/3149654494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5010d630 0x7eff5010fb40 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7eff48009900 tx=0x7eff4802a410 comp rx=0 tx=0).stop 2024-07-27T06:49:57.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.639+0000 7eff549ff640 1 -- 172.21.15.29:0/3149654494 shutdown_connections 2024-07-27T06:49:57.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.639+0000 7eff549ff640 1 --2- 172.21.15.29:0/3149654494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff5010d630 0x7eff5010fb40 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.639+0000 7eff549ff640 1 --2- 172.21.15.29:0/3149654494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff50104ab0 0x7eff5010d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.639+0000 7eff549ff640 1 --2- 172.21.15.29:0/3149654494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff50104080 0x7eff50104490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.639+0000 7eff549ff640 1 -- 172.21.15.29:0/3149654494 >> 172.21.15.29:0/3149654494 conn(0x7eff500fdcf0 msgr2=0x7eff50100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:57.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.639+0000 7eff549ff640 1 -- 172.21.15.29:0/3149654494 shutdown_connections 2024-07-27T06:49:57.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.639+0000 7eff549ff640 1 -- 172.21.15.29:0/3149654494 wait complete. 2024-07-27T06:49:57.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.640+0000 7eff549ff640 1 Processor -- start 2024-07-27T06:49:57.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.640+0000 7eff549ff640 1 -- start start 2024-07-27T06:49:57.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.640+0000 7eff549ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff50104080 0x7eff50199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:57.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff549ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff50104ab0 0x7eff5019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:57.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff549ff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5010d630 0x7eff5019f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:57.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff549ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7eff501132c0 con 0x7eff50104ab0 2024-07-27T06:49:57.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff549ff640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7eff50113140 con 0x7eff5010d630 2024-07-27T06:49:57.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff549ff640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7eff50113440 con 0x7eff50104080 2024-07-27T06:49:57.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff4e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff50104ab0 0x7eff5019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:57.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff4e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff50104ab0 0x7eff5019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45298/0 (socket says 172.21.15.29:45298) 2024-07-27T06:49:57.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff4e59c640 1 -- 172.21.15.29:0/399112257 learned_addr learned my addr 172.21.15.29:0/399112257 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:57.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff50104080 0x7eff50199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:57.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff4f59e640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5010d630 0x7eff5019f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:57.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff4e59c640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff50104080 msgr2=0x7eff50199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:57.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff4e59c640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff50104080 0x7eff50199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff4e59c640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5010d630 msgr2=0x7eff5019f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:57.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff4e59c640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5010d630 0x7eff5019f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff4e59c640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff5019fc60 con 0x7eff50104ab0 2024-07-27T06:49:57.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff4f59e640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5010d630 0x7eff5019f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:57.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff4e59c640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff50104ab0 0x7eff5019a290 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7eff3800e9f0 tx=0x7eff3800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:57.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.641+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3800ccc0 con 0x7eff50104ab0 2024-07-27T06:49:57.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.642+0000 7eff549ff640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff5019ff50 con 0x7eff50104ab0 2024-07-27T06:49:57.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.642+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff380108e0 con 0x7eff50104ab0 2024-07-27T06:49:57.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.642+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff38018860 con 0x7eff50104ab0 2024-07-27T06:49:57.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.642+0000 7eff549ff640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff501a66b0 con 0x7eff50104ab0 2024-07-27T06:49:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.644+0000 7eff5b784740 1 Processor -- start 2024-07-27T06:49:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.644+0000 7eff5b784740 1 -- start start 2024-07-27T06:49:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.644+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7eff38010430 con 0x7eff50104ab0 2024-07-27T06:49:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff5b784740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cdf3b350 0x55a2ce2e0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff5b784740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a2cdf12490 0x55a2ce2e0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff5b784740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2cdf1a670 0x55a2ce2e1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff5b784740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a2ce1c82f0 con 0x55a2cdf3b350 2024-07-27T06:49:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff5b784740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55a2ce256a80 con 0x55a2cdf12490 2024-07-27T06:49:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff5b784740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a2ce245f80 con 0x55a2cdf1a670 2024-07-27T06:49:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff4f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cdf3b350 0x55a2ce2e0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff4f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cdf3b350 0x55a2ce2e0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45308/0 (socket says 172.21.15.29:45308) 2024-07-27T06:49:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff4f59e640 1 -- 172.21.15.29:0/2476625606 learned_addr learned my addr 172.21.15.29:0/2476625606 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff4e59c640 1 --2- 172.21.15.29:0/2476625606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2cdf1a670 0x55a2ce2e1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/2476625606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a2cdf12490 0x55a2ce2e0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff4f59e640 1 -- 172.21.15.29:0/2476625606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2cdf1a670 msgr2=0x55a2ce2e1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff4f59e640 1 --2- 172.21.15.29:0/2476625606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2cdf1a670 0x55a2ce2e1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff4f59e640 1 -- 172.21.15.29:0/2476625606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a2cdf12490 msgr2=0x55a2ce2e0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff4f59e640 1 --2- 172.21.15.29:0/2476625606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a2cdf12490 0x55a2ce2e0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff4f59e640 1 -- 172.21.15.29:0/2476625606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2cdfa6930 con 0x55a2cdf3b350 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/2476625606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a2cdf12490 0x55a2ce2e0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.645+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.646+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.646+0000 7eff4f59e640 1 --2- 172.21.15.29:0/2476625606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cdf3b350 0x55a2ce2e0580 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7eff48009900 tx=0x7eff48032bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa743085da97b53 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.646+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8586+0+0 (secure 0 0 0) 0x7eff380e4340 con 0x7eff50104ab0 2024-07-27T06:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.646+0000 7eff457fa640 1 -- 172.21.15.29:0/2476625606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4803f070 con 0x55a2cdf3b350 2024-07-27T06:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.646+0000 7eff457fa640 1 -- 172.21.15.29:0/2476625606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff48004030 con 0x55a2cdf3b350 2024-07-27T06:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.646+0000 7eff457fa640 1 -- 172.21.15.29:0/2476625606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48035070 con 0x55a2cdf3b350 2024-07-27T06:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.647+0000 7eff5b784740 1 -- 172.21.15.29:0/2476625606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cdf3b350 msgr2=0x55a2ce2e0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.647+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b6250 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x55a2ce2df200 tx=0x7eff40009680 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.647+0000 7eff5b784740 1 --2- 172.21.15.29:0/2476625606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cdf3b350 0x55a2ce2e0580 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7eff48009900 tx=0x7eff48032bb0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.647+0000 7eff5b784740 1 -- 172.21.15.29:0/2476625606 shutdown_connections 2024-07-27T06:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.647+0000 7eff5b784740 1 --2- 172.21.15.29:0/2476625606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2cdf1a670 0x55a2ce2e1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.647+0000 7eff5b784740 1 --2- 172.21.15.29:0/2476625606 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a2cdf12490 0x55a2ce2e0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.647+0000 7eff5b784740 1 --2- 172.21.15.29:0/2476625606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cdf3b350 0x55a2ce2e0580 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.647+0000 7eff5b784740 1 -- 172.21.15.29:0/2476625606 >> 172.21.15.29:0/2476625606 conn(0x55a2cdf2f740 msgr2=0x55a2ce2de470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.647+0000 7eff5b784740 1 -- 172.21.15.29:0/2476625606 shutdown_connections 2024-07-27T06:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.647+0000 7eff5b784740 1 -- 172.21.15.29:0/2476625606 wait complete. 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.647+0000 7eff5b784740 1 -- start start 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff5b784740 1 Processor -- start 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff5b784740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2cdf12490 0x55a2ce2ec130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff5b784740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a2cdf1a670 0x55a2ce2ec670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff5b784740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cdf3b350 0x55a2ce2f7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff5b784740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55a2ce1e84f0 con 0x55a2cdf3b350 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff5b784740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55a2ce249fc0 con 0x55a2cdf1a670 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff5b784740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a2ce0a9890 con 0x55a2cdf12490 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff5b784740 10 client.0 _ll_register_callbacks cb 0x55a2ce1f2500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff4f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cdf3b350 0x55a2ce2f7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff4ed9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2cdf12490 0x55a2ce2ec130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff4f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cdf3b350 0x55a2ce2f7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45324/0 (socket says 172.21.15.29:45324) 2024-07-27T06:49:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff4f59e640 1 -- 172.21.15.29:0/3150119665 learned_addr learned my addr 172.21.15.29:0/3150119665 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:49:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.648+0000 7eff4e59c640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a2cdf1a670 0x55a2ce2ec670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:49:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff4f59e640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2cdf12490 msgr2=0x55a2ce2ec130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff4f59e640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2cdf12490 0x55a2ce2ec130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff4f59e640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a2cdf1a670 msgr2=0x55a2ce2ec670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:49:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff4f59e640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55a2cdf1a670 0x55a2ce2ec670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:49:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff4f59e640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2ce1f7c50 con 0x55a2cdf3b350 2024-07-27T06:49:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a2cdf12490 0x55a2ce2ec130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:49:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff4f59e640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a2cdf3b350 0x55a2ce2f7ea0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7eff48009850 tx=0x7eff480051c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8605813b8edc7a server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:49:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff367fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:49:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48035070 con 0x55a2cdf3b350 2024-07-27T06:49:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8586+0+0 (secure 0 0 0) 0x7eff480316a0 con 0x55a2cdf3b350 2024-07-27T06:49:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff5b784740 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff48004030 con 0x55a2cdf3b350 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7eff48055050 con 0x55a2cdf3b350 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff5b784740 10 client.16383 fetch_fsmap learned FSMap version 39 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff5b784740 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55a2ce2fab70 con 0x55a2cdf3b350 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.649+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff48004030 con 0x55a2cdf3b350 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.650+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 39) ==== 8366+0+0 (secure 0 0 0) 0x7eff48004750 con 0x55a2cdf3b350 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.650+0000 7eff5b784740 10 client.16383 fetch_fsmap finished waiting for FSMap version 39 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.650+0000 7eff5b784740 10 client.16383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.h 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.650+0000 7eff5b784740 20 client.16383 populate_metadata read hostname 'smithi029' 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.650+0000 7eff5b784740 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.650+0000 7eff5b784740 4 client.16383 mds_command: new command op to 24584 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.650+0000 7eff5b784740 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2ce2589d0 con 0x55a2cdef9cd0 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.650+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:49:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.650+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:49:57.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.650+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T06:49:57.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.852+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:49:57.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.852+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:49:57.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:57.852+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-27T06:49:58.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:58.253+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:49:58.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:58.253+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:49:58.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:58.253+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-27T06:49:59.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:59.055+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:49:59.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:59.055+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:49:59.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:49:59.055+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-27T06:49:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:49:58 smithi077 ceph-mon[37153]: pgmap v487: 97 pgs: 97 active+clean; 44 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 74 KiB/s wr, 23 op/s 2024-07-27T06:49:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:49:58 smithi047 ceph-mon[36835]: pgmap v487: 97 pgs: 97 active+clean; 44 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 74 KiB/s wr, 23 op/s 2024-07-27T06:49:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:49:58 smithi029 ceph-mon[32610]: pgmap v487: 97 pgs: 97 active+clean; 44 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 74 KiB/s wr, 23 op/s 2024-07-27T06:50:00.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:00.656+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:50:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:00.657+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:50:00.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:00.657+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-27T06:50:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:00 smithi029 ceph-mon[32610]: pgmap v488: 97 pgs: 97 active+clean; 44 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 69 KiB/s wr, 21 op/s 2024-07-27T06:50:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:00 smithi029 ceph-mon[32610]: overall HEALTH_OK 2024-07-27T06:50:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:00 smithi077 ceph-mon[37153]: pgmap v488: 97 pgs: 97 active+clean; 44 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 69 KiB/s wr, 21 op/s 2024-07-27T06:50:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:00 smithi077 ceph-mon[37153]: overall HEALTH_OK 2024-07-27T06:50:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:00 smithi047 ceph-mon[36835]: pgmap v488: 97 pgs: 97 active+clean; 44 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 69 KiB/s wr, 21 op/s 2024-07-27T06:50:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:00 smithi047 ceph-mon[36835]: overall HEALTH_OK 2024-07-27T06:50:03.658 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:03 smithi029 ceph-mon[32610]: pgmap v489: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 139 KiB/s wr, 22 op/s 2024-07-27T06:50:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:03 smithi047 ceph-mon[36835]: pgmap v489: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 139 KiB/s wr, 22 op/s 2024-07-27T06:50:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:03 smithi077 ceph-mon[37153]: pgmap v489: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 139 KiB/s wr, 22 op/s 2024-07-27T06:50:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:03.860+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:50:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:03.860+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:50:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:03.860+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-27T06:50:05.457 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:05 smithi047 ceph-mon[36835]: pgmap v490: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 139 KiB/s wr, 22 op/s 2024-07-27T06:50:05.457 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:05 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-07-27T06:50:05.457 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:05 smithi047 systemd[1]: Stopped Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:50:05.457 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:05 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Consumed 3.880s CPU time. 2024-07-27T06:50:05.457 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:05 smithi047 systemd[1]: Starting Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:50:05.457 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:05 smithi047 podman[71264]: 2024-07-27 06:50:05.340732255 +0000 UTC m=+0.018581976 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:50:05.458 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:05 smithi047 podman[71264]: 2024-07-27 06:50:05.456683392 +0000 UTC m=+0.134533142 container create fc3876150fc9b11f27805441770a1a4daea308b55399d0e299cb1b5ec458355a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-07-27T06:50:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:05 smithi029 ceph-mon[32610]: pgmap v490: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 139 KiB/s wr, 22 op/s 2024-07-27T06:50:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:05 smithi077 ceph-mon[37153]: pgmap v490: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 139 KiB/s wr, 22 op/s 2024-07-27T06:50:06.196 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:05 smithi047 podman[71264]: 2024-07-27 06:50:05.717568189 +0000 UTC m=+0.395417906 container init fc3876150fc9b11f27805441770a1a4daea308b55399d0e299cb1b5ec458355a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, org.label-schema.build-date=20240724, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-07-27T06:50:06.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:05 smithi047 podman[71264]: 2024-07-27 06:50:05.720256658 +0000 UTC m=+0.398106366 container start fc3876150fc9b11f27805441770a1a4daea308b55399d0e299cb1b5ec458355a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-07-27T06:50:06.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:05 smithi047 ceph-mds[71284]: -- 172.21.15.47:0/2747462246 <== mon.2 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56491d937c20 con 0x564921ad9600 2024-07-27T06:50:06.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:05 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: starting mds.h at 2024-07-27T06:50:06.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:05 smithi047 bash[71264]: fc3876150fc9b11f27805441770a1a4daea308b55399d0e299cb1b5ec458355a 2024-07-27T06:50:06.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:05 smithi047 systemd[1]: Started Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:50:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:06 smithi047 ceph-mon[36835]: daemon mds.h restarted 2024-07-27T06:50:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:06 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:50:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:06 smithi047 ceph-mon[36835]: osdmap e85: 12 total, 12 up, 12 in 2024-07-27T06:50:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:06 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:boot 2024-07-27T06:50:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:06 smithi047 ceph-mon[36835]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:50:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:06 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:50:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:50:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:06 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:50:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:06 smithi029 ceph-mon[32610]: daemon mds.h restarted 2024-07-27T06:50:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:06 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:50:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:06 smithi029 ceph-mon[32610]: osdmap e85: 12 total, 12 up, 12 in 2024-07-27T06:50:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:06 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:boot 2024-07-27T06:50:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:06 smithi029 ceph-mon[32610]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:50:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:06 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:50:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:50:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:06 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:50:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:06 smithi077 ceph-mon[37153]: daemon mds.h restarted 2024-07-27T06:50:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:06 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:50:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:06 smithi077 ceph-mon[37153]: osdmap e85: 12 total, 12 up, 12 in 2024-07-27T06:50:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:06 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:boot 2024-07-27T06:50:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:06 smithi077 ceph-mon[37153]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:50:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:06 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:50:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:50:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:06 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:50:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:07 smithi047 ceph-mon[36835]: pgmap v491: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 127 KiB/s wr, 17 op/s 2024-07-27T06:50:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:07 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:resolve 2024-07-27T06:50:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:07 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:50:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:07 smithi029 ceph-mon[32610]: pgmap v491: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 127 KiB/s wr, 17 op/s 2024-07-27T06:50:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:07 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:resolve 2024-07-27T06:50:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:07 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:50:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:07 smithi077 ceph-mon[37153]: pgmap v491: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 127 KiB/s wr, 17 op/s 2024-07-27T06:50:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:07 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:resolve 2024-07-27T06:50:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:07 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:50:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:08.070+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:08.071+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-27T06:50:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:08.071+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:50:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:08.071+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:50:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:08.071+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:50:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:08 smithi077 ceph-mon[37153]: pgmap v493: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 93 KiB/s wr, 16 op/s 2024-07-27T06:50:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:08 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00099998 2024-07-27T06:50:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:08 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:reconnect 2024-07-27T06:50:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:08 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:50:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:08 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00799984 2024-07-27T06:50:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:08 smithi047 ceph-mon[36835]: pgmap v493: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 93 KiB/s wr, 16 op/s 2024-07-27T06:50:09.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:08 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00099998 2024-07-27T06:50:09.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:08 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:reconnect 2024-07-27T06:50:09.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:08 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:50:09.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:08 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00799984 2024-07-27T06:50:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:08 smithi029 ceph-mon[32610]: pgmap v493: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 93 KiB/s wr, 16 op/s 2024-07-27T06:50:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:08 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00099998 2024-07-27T06:50:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:08 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:reconnect 2024-07-27T06:50:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:08 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:50:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:08 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00799984 2024-07-27T06:50:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:rejoin 2024-07-27T06:50:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:50:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-mon[37153]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:50:10.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:50:10.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: *** Caught signal (Aborted) ** 2024-07-27T06:50:10.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: in thread 7feae7942640 thread_name:ms_dispatch 2024-07-27T06:50:10.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:10.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7feaecd196f0] 2024-07-27T06:50:10.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 2: /lib64/libc.so.6(+0x8b94c) [0x7feaecd6694c] 2024-07-27T06:50:10.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 3: raise() 2024-07-27T06:50:10.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 4: abort() 2024-07-27T06:50:10.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 5: /lib64/libc.so.6(+0x2871b) [0x7feaecd0371b] 2024-07-27T06:50:10.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 6: /lib64/libc.so.6(+0x37386) [0x7feaecd12386] 2024-07-27T06:50:10.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55797ff366b9] 2024-07-27T06:50:10.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5579800f480c] 2024-07-27T06:50:10.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557980042cde] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x557980004763] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557980005a1b] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55797ffe9752] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55797ff37a56] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55797ff3b637] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55797ff3bf5e] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55797ff1f03a] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7feaed5e6407] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 18: (DispatchQueue::entry()+0x60f) [0x7feaed5e6cf9] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7feaed6c35a1] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 20: (Thread::entry_wrapper()+0x43) [0x7feaed470cdd] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 21: (Thread::_entry_func(void*)+0xd) [0x7feaed470cf9] 2024-07-27T06:50:10.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 22: /lib64/libc.so.6(+0x89c02) [0x7feaecd64c02] 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 23: /lib64/libc.so.6(+0x10ec40) [0x7feaecde9c40] 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 2024-07-27T06:50:09.809+0000 7feae7942640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: in thread 7feae7942640 thread_name:ms_dispatch 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7feaecd196f0] 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 2: /lib64/libc.so.6(+0x8b94c) [0x7feaecd6694c] 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 3: raise() 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 4: abort() 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 5: /lib64/libc.so.6(+0x2871b) [0x7feaecd0371b] 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 6: /lib64/libc.so.6(+0x37386) [0x7feaecd12386] 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55797ff366b9] 2024-07-27T06:50:10.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5579800f480c] 2024-07-27T06:50:10.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557980042cde] 2024-07-27T06:50:10.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x557980004763] 2024-07-27T06:50:10.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557980005a1b] 2024-07-27T06:50:10.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55797ffe9752] 2024-07-27T06:50:10.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55797ff37a56] 2024-07-27T06:50:10.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55797ff3b637] 2024-07-27T06:50:10.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55797ff3bf5e] 2024-07-27T06:50:10.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55797ff1f03a] 2024-07-27T06:50:10.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7feaed5e6407] 2024-07-27T06:50:10.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 18: (DispatchQueue::entry()+0x60f) [0x7feaed5e6cf9] 2024-07-27T06:50:10.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7feaed6c35a1] 2024-07-27T06:50:10.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 20: (Thread::entry_wrapper()+0x43) [0x7feaed470cdd] 2024-07-27T06:50:10.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 21: (Thread::_entry_func(void*)+0xd) [0x7feaed470cf9] 2024-07-27T06:50:10.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 22: /lib64/libc.so.6(+0x89c02) [0x7feaecd64c02] 2024-07-27T06:50:10.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 23: /lib64/libc.so.6(+0x10ec40) [0x7feaecde9c40] 2024-07-27T06:50:10.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:50:10.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 2024-07-27T06:50:10.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 0> 2024-07-27T06:50:09.809+0000 7feae7942640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:50:10.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: in thread 7feae7942640 thread_name:ms_dispatch 2024-07-27T06:50:10.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 2024-07-27T06:50:10.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:10.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7feaecd196f0] 2024-07-27T06:50:10.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 2: /lib64/libc.so.6(+0x8b94c) [0x7feaecd6694c] 2024-07-27T06:50:10.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 3: raise() 2024-07-27T06:50:10.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 4: abort() 2024-07-27T06:50:10.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 5: /lib64/libc.so.6(+0x2871b) [0x7feaecd0371b] 2024-07-27T06:50:10.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 6: /lib64/libc.so.6(+0x37386) [0x7feaecd12386] 2024-07-27T06:50:10.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55797ff366b9] 2024-07-27T06:50:10.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5579800f480c] 2024-07-27T06:50:10.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557980042cde] 2024-07-27T06:50:10.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x557980004763] 2024-07-27T06:50:10.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557980005a1b] 2024-07-27T06:50:10.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55797ffe9752] 2024-07-27T06:50:10.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55797ff37a56] 2024-07-27T06:50:10.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55797ff3b637] 2024-07-27T06:50:10.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55797ff3bf5e] 2024-07-27T06:50:10.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55797ff1f03a] 2024-07-27T06:50:10.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7feaed5e6407] 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 18: (DispatchQueue::entry()+0x60f) [0x7feaed5e6cf9] 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7feaed6c35a1] 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 20: (Thread::entry_wrapper()+0x43) [0x7feaed470cdd] 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 21: (Thread::_entry_func(void*)+0xd) [0x7feaed470cf9] 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 22: /lib64/libc.so.6(+0x89c02) [0x7feaecd64c02] 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 23: /lib64/libc.so.6(+0x10ec40) [0x7feaecde9c40] 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 0> 2024-07-27T06:50:09.809+0000 7feae7942640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: in thread 7feae7942640 thread_name:ms_dispatch 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:10.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7feaecd196f0] 2024-07-27T06:50:10.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 2: /lib64/libc.so.6(+0x8b94c) [0x7feaecd6694c] 2024-07-27T06:50:10.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 3: raise() 2024-07-27T06:50:10.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 4: abort() 2024-07-27T06:50:10.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 5: /lib64/libc.so.6(+0x2871b) [0x7feaecd0371b] 2024-07-27T06:50:10.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 6: /lib64/libc.so.6(+0x37386) [0x7feaecd12386] 2024-07-27T06:50:10.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55797ff366b9] 2024-07-27T06:50:10.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5579800f480c] 2024-07-27T06:50:10.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557980042cde] 2024-07-27T06:50:10.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x557980004763] 2024-07-27T06:50:10.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557980005a1b] 2024-07-27T06:50:10.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55797ffe9752] 2024-07-27T06:50:10.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55797ff37a56] 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55797ff3b637] 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55797ff3bf5e] 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55797ff1f03a] 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7feaed5e6407] 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 18: (DispatchQueue::entry()+0x60f) [0x7feaed5e6cf9] 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7feaed6c35a1] 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 20: (Thread::entry_wrapper()+0x43) [0x7feaed470cdd] 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 21: (Thread::_entry_func(void*)+0xd) [0x7feaed470cf9] 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 22: /lib64/libc.so.6(+0x89c02) [0x7feaecd64c02] 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 23: /lib64/libc.so.6(+0x10ec40) [0x7feaecde9c40] 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[71890]: 2024-07-27T06:50:10.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:09 smithi077 podman[71931]: 2024-07-27 06:50:09.867936577 +0000 UTC m=+0.019240051 container died cb9afc3d6c6d34c7518cb88a625d3b3c4a538bcb7287f0a45ad1595db876e4db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-07-27T06:50:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:09 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:rejoin 2024-07-27T06:50:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:09 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:50:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:09 smithi047 ceph-mon[36835]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:50:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:09 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:rejoin 2024-07-27T06:50:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:09 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:50:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:09 smithi029 ceph-mon[32610]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:50:10.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:10.267+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:10.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:10.268+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-27T06:50:10.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:10.268+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:50:10.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:10.268+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:50:10.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:10.268+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-27T06:50:10.810 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:10 smithi077 ceph-mon[37153]: pgmap v494: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 93 KiB/s wr, 16 op/s 2024-07-27T06:50:10.810 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:10 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:50:10.810 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:10 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:50:10.810 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:10 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:active 2024-07-27T06:50:10.810 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:10 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:50:10.811 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:10 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:50:10.811 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:10 smithi077 podman[71931]: 2024-07-27 06:50:10.553166471 +0000 UTC m=+0.704469991 container remove cb9afc3d6c6d34c7518cb88a625d3b3c4a538bcb7287f0a45ad1595db876e4db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-07-27T06:50:10.811 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:10 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:50:11.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:10 smithi047 ceph-mon[36835]: pgmap v494: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 93 KiB/s wr, 16 op/s 2024-07-27T06:50:11.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:10 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:50:11.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:10 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:50:11.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:10 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:active 2024-07-27T06:50:11.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:10 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:50:11.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:10 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:50:11.106 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:10 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-27T06:50:11.106 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:10 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Consumed 1.092s CPU time. 2024-07-27T06:50:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:10 smithi029 ceph-mon[32610]: pgmap v494: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 93 KiB/s wr, 16 op/s 2024-07-27T06:50:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:10 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:50:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:10 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:50:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:10 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/4086885512,v1:172.21.15.77:6839/4086885512] up:active 2024-07-27T06:50:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:10 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:50:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:10 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:50:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:12 smithi077 ceph-mon[37153]: pgmap v495: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 102 B/s wr, 21 op/s 2024-07-27T06:50:13.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:12 smithi029 ceph-mon[32610]: pgmap v495: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 102 B/s wr, 21 op/s 2024-07-27T06:50:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:12 smithi047 ceph-mon[36835]: pgmap v495: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 102 B/s wr, 21 op/s 2024-07-27T06:50:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:14 smithi077 ceph-mon[37153]: pgmap v496: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:50:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:14 smithi047 ceph-mon[36835]: pgmap v496: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:50:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:14 smithi029 ceph-mon[32610]: pgmap v496: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:50:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:16 smithi077 ceph-mon[37153]: pgmap v497: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 204 B/s wr, 21 op/s 2024-07-27T06:50:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:16 smithi047 ceph-mon[36835]: pgmap v497: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 204 B/s wr, 21 op/s 2024-07-27T06:50:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:16 smithi029 ceph-mon[32610]: pgmap v497: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 204 B/s wr, 21 op/s 2024-07-27T06:50:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:18 smithi077 ceph-mon[37153]: pgmap v498: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 21 op/s 2024-07-27T06:50:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:18 smithi047 ceph-mon[36835]: pgmap v498: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 21 op/s 2024-07-27T06:50:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:18 smithi029 ceph-mon[32610]: pgmap v498: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 21 op/s 2024-07-27T06:50:21.093 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:20 smithi077 ceph-mon[37153]: pgmap v499: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:50:21.093 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:20 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 2. 2024-07-27T06:50:21.094 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:20 smithi077 systemd[1]: Stopped Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:50:21.094 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:20 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Consumed 1.092s CPU time. 2024-07-27T06:50:21.094 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:20 smithi077 systemd[1]: Starting Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:50:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:20 smithi047 ceph-mon[36835]: pgmap v499: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:50:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:20 smithi029 ceph-mon[32610]: pgmap v499: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:50:21.357 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:21 smithi077 podman[72025]: 2024-07-27 06:50:20.995369491 +0000 UTC m=+0.017678643 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:50:21.357 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:21 smithi077 podman[72025]: 2024-07-27 06:50:21.103426514 +0000 UTC m=+0.125735660 container create 671e0da12d33bdf6578bfed1c81bee8096c2862216e8f96c64083024e848683e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True) 2024-07-27T06:50:21.357 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:21 smithi077 podman[72025]: 2024-07-27 06:50:21.339487266 +0000 UTC m=+0.361796416 container init 671e0da12d33bdf6578bfed1c81bee8096c2862216e8f96c64083024e848683e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-07-27T06:50:21.357 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:21 smithi077 podman[72025]: 2024-07-27 06:50:21.341972225 +0000 UTC m=+0.364281376 container start 671e0da12d33bdf6578bfed1c81bee8096c2862216e8f96c64083024e848683e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-07-27T06:50:21.619 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:21 smithi077 ceph-mds[72044]: -- 172.21.15.77:0/2530780858 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5640de79f860 con 0x5640e2941600 2024-07-27T06:50:21.619 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:21 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: starting mds.i at 2024-07-27T06:50:21.619 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:21 smithi077 bash[72025]: 671e0da12d33bdf6578bfed1c81bee8096c2862216e8f96c64083024e848683e 2024-07-27T06:50:21.619 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:21 smithi077 systemd[1]: Started Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:50:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:21 smithi077 ceph-mon[37153]: daemon mds.i restarted 2024-07-27T06:50:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:21 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:50:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:21 smithi077 ceph-mon[37153]: osdmap e86: 12 total, 12 up, 12 in 2024-07-27T06:50:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:21 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:boot 2024-07-27T06:50:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:21 smithi077 ceph-mon[37153]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:50:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:21 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:50:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:50:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:21 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:50:22.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:21 smithi029 ceph-mon[32610]: daemon mds.i restarted 2024-07-27T06:50:22.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:21 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:50:22.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:21 smithi029 ceph-mon[32610]: osdmap e86: 12 total, 12 up, 12 in 2024-07-27T06:50:22.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:21 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:boot 2024-07-27T06:50:22.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:21 smithi029 ceph-mon[32610]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:50:22.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:21 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:50:22.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:50:22.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:21 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:50:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:21 smithi047 ceph-mon[36835]: daemon mds.i restarted 2024-07-27T06:50:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:21 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:50:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:21 smithi047 ceph-mon[36835]: osdmap e86: 12 total, 12 up, 12 in 2024-07-27T06:50:22.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:21 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:boot 2024-07-27T06:50:22.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:21 smithi047 ceph-mon[36835]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:50:22.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:21 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:50:22.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:50:22.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:21 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:50:23.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:23.080+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:23.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:23.081+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:23.081+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-27T06:50:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:23.081+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:50:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:23.081+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:50:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:23.081+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:50:23.084 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:23.081+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-27T06:50:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:23.081+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:50:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:23.081+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:50:23.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:23.081+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:50:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:23 smithi047 ceph-mon[36835]: pgmap v501: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:50:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:23 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:resolve 2024-07-27T06:50:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:23 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:50:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:23 smithi029 ceph-mon[32610]: pgmap v501: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:50:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:23 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:resolve 2024-07-27T06:50:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:23 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:50:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:23 smithi077 ceph-mon[37153]: pgmap v501: 97 pgs: 97 active+clean; 45 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:50:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:23 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:resolve 2024-07-27T06:50:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:23 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:50:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:24 smithi077 ceph-mon[37153]: pgmap v502: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:50:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:24 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:50:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:24 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999979 2024-07-27T06:50:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:24 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:reconnect 2024-07-27T06:50:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:24 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:50:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:24 smithi047 ceph-mon[36835]: pgmap v502: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:50:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:24 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:50:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:24 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999979 2024-07-27T06:50:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:24 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:reconnect 2024-07-27T06:50:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:24 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:50:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:24 smithi029 ceph-mon[32610]: pgmap v502: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:50:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:24 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:50:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:24 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999979 2024-07-27T06:50:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:24 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:reconnect 2024-07-27T06:50:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:24 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:50:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:25 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:rejoin 2024-07-27T06:50:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:25 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:50:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:25 smithi077 ceph-mon[37153]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:50:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:rejoin 2024-07-27T06:50:26.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:50:26.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-mon[36835]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:50:26.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:50:26.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: *** Caught signal (Aborted) ** 2024-07-27T06:50:26.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: in thread 7f5d4ce31640 thread_name:ms_dispatch 2024-07-27T06:50:26.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:26.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5d522086f0] 2024-07-27T06:50:26.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5d5225594c] 2024-07-27T06:50:26.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 3: raise() 2024-07-27T06:50:26.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 4: abort() 2024-07-27T06:50:26.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5d521f271b] 2024-07-27T06:50:26.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 6: /lib64/libc.so.6(+0x37386) [0x7f5d52201386] 2024-07-27T06:50:26.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56491c7bb6b9] 2024-07-27T06:50:26.198 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56491c97980c] 2024-07-27T06:50:26.199 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56491c8c7cde] 2024-07-27T06:50:26.199 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56491c889763] 2024-07-27T06:50:26.199 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56491c88aa1b] 2024-07-27T06:50:26.199 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56491c86e752] 2024-07-27T06:50:26.199 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56491c7bca56] 2024-07-27T06:50:26.199 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56491c7c0637] 2024-07-27T06:50:26.199 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56491c7c0f5e] 2024-07-27T06:50:26.199 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56491c7a403a] 2024-07-27T06:50:26.199 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5d52ad5407] 2024-07-27T06:50:26.199 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 18: (DispatchQueue::entry()+0x60f) [0x7f5d52ad5cf9] 2024-07-27T06:50:26.199 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5d52bb25a1] 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 20: (Thread::entry_wrapper()+0x43) [0x7f5d5295fcdd] 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5d5295fcf9] 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5d52253c02] 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5d522d8c40] 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 2024-07-27T06:50:25.846+0000 7f5d4ce31640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: in thread 7f5d4ce31640 thread_name:ms_dispatch 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5d522086f0] 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5d5225594c] 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 3: raise() 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 4: abort() 2024-07-27T06:50:26.200 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5d521f271b] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 6: /lib64/libc.so.6(+0x37386) [0x7f5d52201386] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56491c7bb6b9] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56491c97980c] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56491c8c7cde] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56491c889763] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56491c88aa1b] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56491c86e752] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56491c7bca56] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56491c7c0637] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56491c7c0f5e] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56491c7a403a] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5d52ad5407] 2024-07-27T06:50:26.201 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 18: (DispatchQueue::entry()+0x60f) [0x7f5d52ad5cf9] 2024-07-27T06:50:26.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5d52bb25a1] 2024-07-27T06:50:26.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 20: (Thread::entry_wrapper()+0x43) [0x7f5d5295fcdd] 2024-07-27T06:50:26.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5d5295fcf9] 2024-07-27T06:50:26.202 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5d52253c02] 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5d522d8c40] 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 0> 2024-07-27T06:50:25.846+0000 7f5d4ce31640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: in thread 7f5d4ce31640 thread_name:ms_dispatch 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5d522086f0] 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5d5225594c] 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 3: raise() 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 4: abort() 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5d521f271b] 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 6: /lib64/libc.so.6(+0x37386) [0x7f5d52201386] 2024-07-27T06:50:26.203 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56491c7bb6b9] 2024-07-27T06:50:26.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56491c97980c] 2024-07-27T06:50:26.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56491c8c7cde] 2024-07-27T06:50:26.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56491c889763] 2024-07-27T06:50:26.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56491c88aa1b] 2024-07-27T06:50:26.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56491c86e752] 2024-07-27T06:50:26.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56491c7bca56] 2024-07-27T06:50:26.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56491c7c0637] 2024-07-27T06:50:26.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56491c7c0f5e] 2024-07-27T06:50:26.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56491c7a403a] 2024-07-27T06:50:26.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5d52ad5407] 2024-07-27T06:50:26.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 18: (DispatchQueue::entry()+0x60f) [0x7f5d52ad5cf9] 2024-07-27T06:50:26.204 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5d52bb25a1] 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 20: (Thread::entry_wrapper()+0x43) [0x7f5d5295fcdd] 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5d5295fcf9] 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5d52253c02] 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5d522d8c40] 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: -1> 2024-07-27T06:50:25.846+0000 7f5d4ce31640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: in thread 7f5d4ce31640 thread_name:ms_dispatch 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5d522086f0] 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5d5225594c] 2024-07-27T06:50:26.205 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 3: raise() 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 4: abort() 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5d521f271b] 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 6: /lib64/libc.so.6(+0x37386) [0x7f5d52201386] 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56491c7bb6b9] 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56491c97980c] 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56491c8c7cde] 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56491c889763] 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56491c88aa1b] 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56491c86e752] 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56491c7bca56] 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56491c7c0637] 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56491c7c0f5e] 2024-07-27T06:50:26.206 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56491c7a403a] 2024-07-27T06:50:26.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5d52ad5407] 2024-07-27T06:50:26.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 18: (DispatchQueue::entry()+0x60f) [0x7f5d52ad5cf9] 2024-07-27T06:50:26.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5d52bb25a1] 2024-07-27T06:50:26.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 20: (Thread::entry_wrapper()+0x43) [0x7f5d5295fcdd] 2024-07-27T06:50:26.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5d5295fcf9] 2024-07-27T06:50:26.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5d52253c02] 2024-07-27T06:50:26.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5d522d8c40] 2024-07-27T06:50:26.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:50:26.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71280]: 2024-07-27T06:50:26.207 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:25 smithi047 podman[71319]: 2024-07-27 06:50:25.904365758 +0000 UTC m=+0.019486155 container died fc3876150fc9b11f27805441770a1a4daea308b55399d0e299cb1b5ec458355a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-07-27T06:50:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:25 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:rejoin 2024-07-27T06:50:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:25 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:50:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:25 smithi029 ceph-mon[32610]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:50:26.814 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:26 smithi047 podman[71319]: 2024-07-27 06:50:26.557069142 +0000 UTC m=+0.672189533 container remove fc3876150fc9b11f27805441770a1a4daea308b55399d0e299cb1b5ec458355a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, io.buildah.version=1.36.0) 2024-07-27T06:50:26.814 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:26 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:50:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:26 smithi077 ceph-mon[37153]: pgmap v503: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 op/s 2024-07-27T06:50:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:26 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:50:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:26 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:50:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:26 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:active 2024-07-27T06:50:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:26 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:50:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:26 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:50:27.196 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:26 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-27T06:50:27.196 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:26 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Consumed 1.164s CPU time. 2024-07-27T06:50:27.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:26 smithi047 ceph-mon[36835]: pgmap v503: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 op/s 2024-07-27T06:50:27.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:26 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:50:27.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:26 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:50:27.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:26 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:active 2024-07-27T06:50:27.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:26 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:50:27.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:26 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:50:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:26 smithi029 ceph-mon[32610]: pgmap v503: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 op/s 2024-07-27T06:50:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:26 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:50:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:26 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:50:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:26 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/1512140130,v1:172.21.15.47:6837/1512140130] up:active 2024-07-27T06:50:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:26 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:50:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:26 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:50:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:50:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:50:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:50:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:28 smithi077 ceph-mon[37153]: pgmap v504: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:50:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:50:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:50:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:50:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:28 smithi047 ceph-mon[36835]: pgmap v504: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:50:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:50:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:50:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:50:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:28 smithi029 ceph-mon[32610]: pgmap v504: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:50:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:50:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:50:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:50:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:30 smithi077 ceph-mon[37153]: pgmap v505: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:50:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:30 smithi047 ceph-mon[36835]: pgmap v505: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:50:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:30 smithi029 ceph-mon[32610]: pgmap v505: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:50:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:32 smithi077 ceph-mon[37153]: pgmap v506: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 3.7 KiB/s wr, 21 op/s 2024-07-27T06:50:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:32 smithi047 ceph-mon[36835]: pgmap v506: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 3.7 KiB/s wr, 21 op/s 2024-07-27T06:50:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:32 smithi029 ceph-mon[32610]: pgmap v506: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 3.7 KiB/s wr, 21 op/s 2024-07-27T06:50:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:34 smithi077 ceph-mon[37153]: pgmap v507: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 3.2 KiB/s wr, 20 op/s 2024-07-27T06:50:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:34 smithi047 ceph-mon[36835]: pgmap v507: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 3.2 KiB/s wr, 20 op/s 2024-07-27T06:50:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:34 smithi029 ceph-mon[32610]: pgmap v507: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 3.2 KiB/s wr, 20 op/s 2024-07-27T06:50:37.000 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 31.1 2024-07-27T06:50:37.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T06:50:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:36 smithi077 ceph-mon[37153]: pgmap v508: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-07-27T06:50:37.188 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:36 smithi047 ceph-mon[36835]: pgmap v508: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-07-27T06:50:37.188 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:36 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 2. 2024-07-27T06:50:37.188 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:36 smithi047 systemd[1]: Stopped Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:50:37.188 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:36 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Consumed 1.164s CPU time. 2024-07-27T06:50:37.188 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:36 smithi047 systemd[1]: Starting Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:50:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:36 smithi029 ceph-mon[32610]: pgmap v508: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-07-27T06:50:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:50:37.439 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:37 smithi047 podman[71541]: 2024-07-27 06:50:37.090023367 +0000 UTC m=+0.018513253 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:50:37.439 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:37 smithi047 podman[71541]: 2024-07-27 06:50:37.199806378 +0000 UTC m=+0.128296256 container create 83dfcdb1a72278c7dd2444cb1197f0254fb79092015d0df6c6aeb0e9d311a5b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, ceph=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-07-27T06:50:37.440 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:37 smithi047 podman[71541]: 2024-07-27 06:50:37.436322689 +0000 UTC m=+0.364812570 container init 83dfcdb1a72278c7dd2444cb1197f0254fb79092015d0df6c6aeb0e9d311a5b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-27T06:50:37.696 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:37 smithi047 podman[71541]: 2024-07-27 06:50:37.439175002 +0000 UTC m=+0.367664886 container start 83dfcdb1a72278c7dd2444cb1197f0254fb79092015d0df6c6aeb0e9d311a5b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-27T06:50:37.696 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:37 smithi047 ceph-mds[71560]: -- 172.21.15.47:0/2173676251 <== mon.2 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55655cab1860 con 0x556560c53180 2024-07-27T06:50:37.696 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:37 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: starting mds.h at 2024-07-27T06:50:37.697 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:37 smithi047 bash[71541]: 83dfcdb1a72278c7dd2444cb1197f0254fb79092015d0df6c6aeb0e9d311a5b5 2024-07-27T06:50:37.697 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:37 smithi047 systemd[1]: Started Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:50:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:38.091+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:38.092+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-27T06:50:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:38.092+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:50:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:38.092+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:50:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:38.092+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:50:38.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:38.097+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:38.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:37 smithi029 ceph-mon[32610]: daemon mds.h restarted 2024-07-27T06:50:38.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:37 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:50:38.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:37 smithi029 ceph-mon[32610]: osdmap e87: 12 total, 12 up, 12 in 2024-07-27T06:50:38.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:37 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:boot 2024-07-27T06:50:38.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:37 smithi029 ceph-mon[32610]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:50:38.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:37 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:50:38.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:50:38.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:37 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:50:38.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:38.097+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-27T06:50:38.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:38.097+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:50:38.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:38.097+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:50:38.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:38.097+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:50:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:37 smithi077 ceph-mon[37153]: daemon mds.h restarted 2024-07-27T06:50:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:37 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:50:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:37 smithi077 ceph-mon[37153]: osdmap e87: 12 total, 12 up, 12 in 2024-07-27T06:50:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:37 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:boot 2024-07-27T06:50:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:37 smithi077 ceph-mon[37153]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:50:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:37 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:50:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:50:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:37 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:50:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:37 smithi047 ceph-mon[36835]: daemon mds.h restarted 2024-07-27T06:50:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:37 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:50:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:37 smithi047 ceph-mon[36835]: osdmap e87: 12 total, 12 up, 12 in 2024-07-27T06:50:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:37 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:boot 2024-07-27T06:50:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:37 smithi047 ceph-mon[36835]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:50:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:37 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:50:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:50:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:37 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:50:39.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.079+0000 7fa479c8c640 1 -- 172.21.15.29:0/3308239829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa474107df0 msgr2=0x7fa474108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:39.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.079+0000 7fa479c8c640 1 --2- 172.21.15.29:0/3308239829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa474107df0 0x7fa474108200 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009900 tx=0x7fa45c02a410 comp rx=0 tx=0).stop 2024-07-27T06:50:39.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.079+0000 7fa479c8c640 1 -- 172.21.15.29:0/3308239829 shutdown_connections 2024-07-27T06:50:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.079+0000 7fa479c8c640 1 --2- 172.21.15.29:0/3308239829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa474108740 0x7fa47410ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.079+0000 7fa479c8c640 1 --2- 172.21.15.29:0/3308239829 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa474107df0 0x7fa474108200 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.079+0000 7fa479c8c640 1 --2- 172.21.15.29:0/3308239829 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4741024b0 0x7fa4741028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.079+0000 7fa479c8c640 1 -- 172.21.15.29:0/3308239829 >> 172.21.15.29:0/3308239829 conn(0x7fa474079170 msgr2=0x7fa474079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:50:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.079+0000 7fa479c8c640 1 -- 172.21.15.29:0/3308239829 shutdown_connections 2024-07-27T06:50:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.079+0000 7fa479c8c640 1 -- 172.21.15.29:0/3308239829 wait complete. 2024-07-27T06:50:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa479c8c640 1 Processor -- start 2024-07-27T06:50:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa479c8c640 1 -- start start 2024-07-27T06:50:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa479c8c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4741024b0 0x7fa47419a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa479c8c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa474107df0 0x7fa47419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa479c8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa474108740 0x7fa47419f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:39.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa479c8c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa4741134f0 con 0x7fa4741024b0 2024-07-27T06:50:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa479c8c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fa474113370 con 0x7fa474107df0 2024-07-27T06:50:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa479c8c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa474113670 con 0x7fa474108740 2024-07-27T06:50:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa4737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4741024b0 0x7fa47419a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa4737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4741024b0 0x7fa47419a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44718/0 (socket says 172.21.15.29:44718) 2024-07-27T06:50:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa4737fe640 1 -- 172.21.15.29:0/1425642344 learned_addr learned my addr 172.21.15.29:0/1425642344 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:50:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa472ffd640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa474107df0 0x7fa47419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.080+0000 7fa473fff640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa474108740 0x7fa47419f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.081+0000 7fa4737fe640 1 -- 172.21.15.29:0/1425642344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa474108740 msgr2=0x7fa47419f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.081+0000 7fa4737fe640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa474108740 0x7fa47419f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.081+0000 7fa4737fe640 1 -- 172.21.15.29:0/1425642344 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa474107df0 msgr2=0x7fa47419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.081+0000 7fa4737fe640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa474107df0 0x7fa47419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:39.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.081+0000 7fa4737fe640 1 -- 172.21.15.29:0/1425642344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4741a0010 con 0x7fa4741024b0 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.081+0000 7fa4737fe640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4741024b0 0x7fa47419a0a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa46400d950 tx=0x7fa46400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.081+0000 7fa470ff9640 1 -- 172.21.15.29:0/1425642344 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464017030 con 0x7fa4741024b0 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.081+0000 7fa479c8c640 1 -- 172.21.15.29:0/1425642344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4741a0300 con 0x7fa4741024b0 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.081+0000 7fa470ff9640 1 -- 172.21.15.29:0/1425642344 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa464004020 con 0x7fa4741024b0 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.081+0000 7fa470ff9640 1 -- 172.21.15.29:0/1425642344 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4640047a0 con 0x7fa4741024b0 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.081+0000 7fa479c8c640 1 -- 172.21.15.29:0/1425642344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4741a6a60 con 0x7fa4741024b0 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.082+0000 7fa479c8c640 1 -- 172.21.15.29:0/1425642344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa474103730 con 0x7fa4741024b0 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.084+0000 7fa470ff9640 1 -- 172.21.15.29:0/1425642344 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fa464009a30 con 0x7fa4741024b0 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.084+0000 7fa470ff9640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa4600b3d40 0x7fa4600b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.084+0000 7fa472ffd640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa4600b3d40 0x7fa4600b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.085+0000 7fa470ff9640 1 -- 172.21.15.29:0/1425642344 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8844+0+0 (secure 0 0 0) 0x7fa4640e4200 con 0x7fa4741024b0 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.085+0000 7fa472ffd640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa4600b3d40 0x7fa4600b6250 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa45c000c00 tx=0x7fa45c031400 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:50:39.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.087+0000 7fa470ff9640 1 -- 172.21.15.29:0/1425642344 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa4640ab3d0 con 0x7fa4741024b0 2024-07-27T06:50:39.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.313+0000 7fa479c8c640 1 -- 172.21.15.29:0/1425642344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4741028c0 con 0x7fa4741024b0 2024-07-27T06:50:39.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.314+0000 7fa470ff9640 1 -- 172.21.15.29:0/1425642344 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11950 (secure 0 0 0) 0x7fa4741028c0 con 0x7fa4741024b0 2024-07-27T06:50:39.316 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T06:50:39.316 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":56,"btime":"2024-07-27T06:50:38:841076+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14733,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3730476844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3730476844},{"type":"v1","addr":"172.21.15.29:6835","nonce":3730476844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14739,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1301177290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1301177290},{"type":"v1","addr":"172.21.15.29:6837","nonce":1301177290}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":25859,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6837/3034977063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6836","nonce":3034977063},{"type":"v1","addr":"172.21.15.47:6837","nonce":3034977063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:50:38.024145+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":26026,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26026":{"gid":26026,"name":"i","rank":0,"incarnation":55,"state":"up:resolve","state_seq":6,"addr":"172.21.15.77:6839/2224015949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6838","nonce":2224015949},{"type":"v1","addr":"172.21.15.77:6839","nonce":2224015949}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24658,"qdb_cluster":[24613,24617,24646,24658]},"id":1}]} 2024-07-27T06:50:39.317 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 56 2024-07-27T06:50:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.317+0000 7fa479c8c640 1 -- 172.21.15.29:0/1425642344 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa4600b3d40 msgr2=0x7fa4600b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.317+0000 7fa479c8c640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa4600b3d40 0x7fa4600b6250 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa45c000c00 tx=0x7fa45c031400 comp rx=0 tx=0).stop 2024-07-27T06:50:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.317+0000 7fa479c8c640 1 -- 172.21.15.29:0/1425642344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4741024b0 msgr2=0x7fa47419a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.317+0000 7fa479c8c640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4741024b0 0x7fa47419a0a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa46400d950 tx=0x7fa46400de10 comp rx=0 tx=0).stop 2024-07-27T06:50:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.317+0000 7fa479c8c640 1 -- 172.21.15.29:0/1425642344 shutdown_connections 2024-07-27T06:50:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.317+0000 7fa479c8c640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fa4600b3d40 0x7fa4600b6250 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.317+0000 7fa479c8c640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa474108740 0x7fa47419f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.317+0000 7fa479c8c640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa474107df0 0x7fa47419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.317+0000 7fa479c8c640 1 --2- 172.21.15.29:0/1425642344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4741024b0 0x7fa47419a0a0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.317+0000 7fa479c8c640 1 -- 172.21.15.29:0/1425642344 >> 172.21.15.29:0/1425642344 conn(0x7fa474079170 msgr2=0x7fa474104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:50:39.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.317+0000 7fa479c8c640 1 -- 172.21.15.29:0/1425642344 shutdown_connections 2024-07-27T06:50:39.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:39.317+0000 7fa479c8c640 1 -- 172.21.15.29:0/1425642344 wait complete. 2024-07-27T06:50:39.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:39 smithi029 ceph-mon[32610]: pgmap v510: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 3.6 KiB/s wr, 12 op/s 2024-07-27T06:50:39.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:39 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:resolve 2024-07-27T06:50:39.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:39 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:50:39.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:39 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/1425642344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:50:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:39 smithi077 ceph-mon[37153]: pgmap v510: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 3.6 KiB/s wr, 12 op/s 2024-07-27T06:50:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:39 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:resolve 2024-07-27T06:50:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:39 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:50:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:39 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/1425642344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:50:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:39 smithi047 ceph-mon[36835]: pgmap v510: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 3.6 KiB/s wr, 12 op/s 2024-07-27T06:50:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:39 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:resolve 2024-07-27T06:50:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:39 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:50:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:39 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/1425642344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T06:50:40.091 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24658 quiesce db / --timeout 90 --expiration 151.1 --await-for 45.0' 2024-07-27T06:50:40.091 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- timeout 105 ceph tell mds.24658 quiesce db / --timeout 90 --expiration 151.1 --await-for 45.0 2024-07-27T06:50:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:50:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:40 smithi077 ceph-mon[37153]: pgmap v511: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 3.6 KiB/s wr, 12 op/s 2024-07-27T06:50:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:40 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:50:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:40 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00099998 2024-07-27T06:50:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:40 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:reconnect 2024-07-27T06:50:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:40 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:50:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:40 smithi047 ceph-mon[36835]: pgmap v511: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 3.6 KiB/s wr, 12 op/s 2024-07-27T06:50:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:40 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:50:41.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:40 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00099998 2024-07-27T06:50:41.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:40 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:reconnect 2024-07-27T06:50:41.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:40 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:50:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:40 smithi029 ceph-mon[32610]: pgmap v511: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 3.6 KiB/s wr, 12 op/s 2024-07-27T06:50:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:40 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:50:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:40 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00099998 2024-07-27T06:50:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:40 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:reconnect 2024-07-27T06:50:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:40 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:50:41.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:41 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:rejoin 2024-07-27T06:50:41.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:41 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:50:41.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:41 smithi029 ceph-mon[32610]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:50:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:rejoin 2024-07-27T06:50:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:50:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-mon[37153]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:50:42.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:50:42.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: *** Caught signal (Aborted) ** 2024-07-27T06:50:42.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: in thread 7f3280445640 thread_name:ms_dispatch 2024-07-27T06:50:42.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:42.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f328581c6f0] 2024-07-27T06:50:42.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f328586994c] 2024-07-27T06:50:42.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 3: raise() 2024-07-27T06:50:42.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 4: abort() 2024-07-27T06:50:42.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 5: /lib64/libc.so.6(+0x2871b) [0x7f328580671b] 2024-07-27T06:50:42.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 6: /lib64/libc.so.6(+0x37386) [0x7f3285815386] 2024-07-27T06:50:42.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5640dd0c96b9] 2024-07-27T06:50:42.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5640dd28780c] 2024-07-27T06:50:42.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5640dd1d5cde] 2024-07-27T06:50:42.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5640dd197763] 2024-07-27T06:50:42.108 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5640dd198a1b] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5640dd17c752] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5640dd0caa56] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5640dd0ce637] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5640dd0cef5e] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5640dd0b203a] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f32860e9407] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 18: (DispatchQueue::entry()+0x60f) [0x7f32860e9cf9] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f32861c65a1] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 20: (Thread::entry_wrapper()+0x43) [0x7f3285f73cdd] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3285f73cf9] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3285867c02] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f32858ecc40] 2024-07-27T06:50:42.109 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 2024-07-27T06:50:41.884+0000 7f3280445640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:50:42.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: in thread 7f3280445640 thread_name:ms_dispatch 2024-07-27T06:50:42.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 2024-07-27T06:50:42.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:42.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f328581c6f0] 2024-07-27T06:50:42.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f328586994c] 2024-07-27T06:50:42.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 3: raise() 2024-07-27T06:50:42.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 4: abort() 2024-07-27T06:50:42.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 5: /lib64/libc.so.6(+0x2871b) [0x7f328580671b] 2024-07-27T06:50:42.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 6: /lib64/libc.so.6(+0x37386) [0x7f3285815386] 2024-07-27T06:50:42.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5640dd0c96b9] 2024-07-27T06:50:42.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5640dd28780c] 2024-07-27T06:50:42.110 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5640dd1d5cde] 2024-07-27T06:50:42.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5640dd197763] 2024-07-27T06:50:42.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5640dd198a1b] 2024-07-27T06:50:42.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5640dd17c752] 2024-07-27T06:50:42.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5640dd0caa56] 2024-07-27T06:50:42.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5640dd0ce637] 2024-07-27T06:50:42.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5640dd0cef5e] 2024-07-27T06:50:42.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5640dd0b203a] 2024-07-27T06:50:42.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f32860e9407] 2024-07-27T06:50:42.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 18: (DispatchQueue::entry()+0x60f) [0x7f32860e9cf9] 2024-07-27T06:50:42.111 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f32861c65a1] 2024-07-27T06:50:42.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 20: (Thread::entry_wrapper()+0x43) [0x7f3285f73cdd] 2024-07-27T06:50:42.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3285f73cf9] 2024-07-27T06:50:42.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3285867c02] 2024-07-27T06:50:42.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f32858ecc40] 2024-07-27T06:50:42.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:50:42.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 2024-07-27T06:50:42.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 0> 2024-07-27T06:50:41.884+0000 7f3280445640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:50:42.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: in thread 7f3280445640 thread_name:ms_dispatch 2024-07-27T06:50:42.112 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 2024-07-27T06:50:42.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:42.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f328581c6f0] 2024-07-27T06:50:42.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f328586994c] 2024-07-27T06:50:42.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 3: raise() 2024-07-27T06:50:42.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 4: abort() 2024-07-27T06:50:42.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 5: /lib64/libc.so.6(+0x2871b) [0x7f328580671b] 2024-07-27T06:50:42.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 6: /lib64/libc.so.6(+0x37386) [0x7f3285815386] 2024-07-27T06:50:42.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5640dd0c96b9] 2024-07-27T06:50:42.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5640dd28780c] 2024-07-27T06:50:42.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5640dd1d5cde] 2024-07-27T06:50:42.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5640dd197763] 2024-07-27T06:50:42.113 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5640dd198a1b] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5640dd17c752] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5640dd0caa56] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5640dd0ce637] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5640dd0cef5e] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5640dd0b203a] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f32860e9407] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 18: (DispatchQueue::entry()+0x60f) [0x7f32860e9cf9] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f32861c65a1] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 20: (Thread::entry_wrapper()+0x43) [0x7f3285f73cdd] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3285f73cf9] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3285867c02] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f32858ecc40] 2024-07-27T06:50:42.114 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 0> 2024-07-27T06:50:41.884+0000 7f3280445640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: in thread 7f3280445640 thread_name:ms_dispatch 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f328581c6f0] 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f328586994c] 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 3: raise() 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 4: abort() 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 5: /lib64/libc.so.6(+0x2871b) [0x7f328580671b] 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 6: /lib64/libc.so.6(+0x37386) [0x7f3285815386] 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5640dd0c96b9] 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5640dd28780c] 2024-07-27T06:50:42.115 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5640dd1d5cde] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5640dd197763] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5640dd198a1b] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5640dd17c752] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5640dd0caa56] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5640dd0ce637] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5640dd0cef5e] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5640dd0b203a] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f32860e9407] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 18: (DispatchQueue::entry()+0x60f) [0x7f32860e9cf9] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f32861c65a1] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 20: (Thread::entry_wrapper()+0x43) [0x7f3285f73cdd] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3285f73cf9] 2024-07-27T06:50:42.116 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3285867c02] 2024-07-27T06:50:42.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f32858ecc40] 2024-07-27T06:50:42.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:50:42.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72040]: 2024-07-27T06:50:42.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:41 smithi077 podman[72222]: 2024-07-27 06:50:41.944917591 +0000 UTC m=+0.019926133 container died 671e0da12d33bdf6578bfed1c81bee8096c2862216e8f96c64083024e848683e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-27T06:50:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.154+0000 7f6558a03640 1 -- 172.21.15.29:0/4268387288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6554108560 msgr2=0x7f655410ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.154+0000 7f6558a03640 1 --2- 172.21.15.29:0/4268387288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6554108560 0x7f655410ed90 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6544009ca0 tx=0x7f654402a8c0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.154+0000 7f6558a03640 1 -- 172.21.15.29:0/4268387288 shutdown_connections 2024-07-27T06:50:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.154+0000 7f6558a03640 1 --2- 172.21.15.29:0/4268387288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6554108560 0x7f655410ed90 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.154+0000 7f6558a03640 1 --2- 172.21.15.29:0/4268387288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6554107c10 0x7f6554108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.154+0000 7f6558a03640 1 --2- 172.21.15.29:0/4268387288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65541022d0 0x7f65541026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.154+0000 7f6558a03640 1 -- 172.21.15.29:0/4268387288 >> 172.21.15.29:0/4268387288 conn(0x7f65540fdd60 msgr2=0x7f65541001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.154+0000 7f6558a03640 1 -- 172.21.15.29:0/4268387288 shutdown_connections 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.154+0000 7f6558a03640 1 -- 172.21.15.29:0/4268387288 wait complete. 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.155+0000 7f6558a03640 1 Processor -- start 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.155+0000 7f6558a03640 1 -- start start 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.155+0000 7f6558a03640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65541022d0 0x7f6554199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.155+0000 7f6558a03640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6554107c10 0x7f655419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.155+0000 7f6558a03640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6554108560 0x7f655419f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.155+0000 7f6558a03640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6554113330 con 0x7f6554108560 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.155+0000 7f6558a03640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f65541131b0 con 0x7f65541022d0 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.155+0000 7f6558a03640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f65541134b0 con 0x7f6554107c10 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.155+0000 7f655359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6554108560 0x7f655419f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.155+0000 7f655359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6554108560 0x7f655419f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44728/0 (socket says 172.21.15.29:44728) 2024-07-27T06:50:42.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.155+0000 7f655359e640 1 -- 172.21.15.29:0/2008241281 learned_addr learned my addr 172.21.15.29:0/2008241281 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f655259c640 1 --2- 172.21.15.29:0/2008241281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6554107c10 0x7f655419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f6552d9d640 1 --2- 172.21.15.29:0/2008241281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65541022d0 0x7f6554199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f655359e640 1 -- 172.21.15.29:0/2008241281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6554107c10 msgr2=0x7f655419a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f655359e640 1 --2- 172.21.15.29:0/2008241281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6554107c10 0x7f655419a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f655359e640 1 -- 172.21.15.29:0/2008241281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65541022d0 msgr2=0x7f6554199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f655359e640 1 --2- 172.21.15.29:0/2008241281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65541022d0 0x7f6554199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f655359e640 1 -- 172.21.15.29:0/2008241281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655419fc90 con 0x7f6554108560 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f655359e640 1 --2- 172.21.15.29:0/2008241281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6554108560 0x7f655419f560 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f65440098d0 tx=0x7f65440317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f653bfff640 1 -- 172.21.15.29:0/2008241281 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6544003b70 con 0x7f6554108560 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f6558a03640 1 -- 172.21.15.29:0/2008241281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655419ff20 con 0x7f6554108560 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f653bfff640 1 -- 172.21.15.29:0/2008241281 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6544003d10 con 0x7f6554108560 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f653bfff640 1 -- 172.21.15.29:0/2008241281 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654402fa20 con 0x7f6554108560 2024-07-27T06:50:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.156+0000 7f6558a03640 1 -- 172.21.15.29:0/2008241281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65541a66e0 con 0x7f6554108560 2024-07-27T06:50:42.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.159+0000 7f655f788740 1 Processor -- start 2024-07-27T06:50:42.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.159+0000 7f653bfff640 1 -- 172.21.15.29:0/2008241281 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f654402fc40 con 0x7f6554108560 2024-07-27T06:50:42.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.159+0000 7f655f788740 1 -- start start 2024-07-27T06:50:42.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f655f788740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376f9840 0x557737a9fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f655f788740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376d0980 0x557737a9ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f655f788740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 0x557737aa04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f655f788740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x557737a14780 con 0x5577376d8b60 2024-07-27T06:50:42.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f655f788740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x557737a03c80 con 0x5577376d0980 2024-07-27T06:50:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f655f788740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557737a07cc0 con 0x5577376f9840 2024-07-27T06:50:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f655259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 0x557737aa04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f655259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 0x557737aa04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44744/0 (socket says 172.21.15.29:44744) 2024-07-27T06:50:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f655259c640 1 -- 172.21.15.29:0/3394574058 learned_addr learned my addr 172.21.15.29:0/3394574058 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:50:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f6552d9d640 1 --2- 172.21.15.29:0/3394574058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376d0980 0x557737a9ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f655259c640 1 --2- 172.21.15.29:0/3394574058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376f9840 0x557737a9fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f6552d9d640 1 -- 172.21.15.29:0/3394574058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376f9840 msgr2=0x557737a9fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:50:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f6552d9d640 1 --2- 172.21.15.29:0/3394574058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376f9840 0x557737a9fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f6552d9d640 1 -- 172.21.15.29:0/3394574058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 msgr2=0x557737aa04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:50:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f6552d9d640 1 --2- 172.21.15.29:0/3394574058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 0x557737aa04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f6552d9d640 1 -- 172.21.15.29:0/3394574058 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557737999080 con 0x5577376d0980 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.160+0000 7f655259c640 1 --2- 172.21.15.29:0/3394574058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 0x557737aa04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.161+0000 7f653bfff640 1 --2- 172.21.15.29:0/2008241281 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f653c0b3d30 0x7f653c0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.161+0000 7f6552d9d640 1 --2- 172.21.15.29:0/3394574058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376d0980 0x557737a9ff70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f654c009c00 tx=0x7f654c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45a313c794441c32 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.161+0000 7f655259c640 1 --2- 172.21.15.29:0/2008241281 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f653c0b3d30 0x7f653c0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.161+0000 7f653bfff640 1 -- 172.21.15.29:0/2008241281 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8844+0+0 (secure 0 0 0) 0x7f65441054c0 con 0x7f6554108560 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.161+0000 7f65397fa640 1 -- 172.21.15.29:0/3394574058 <== mon.2 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c004230 con 0x5577376d0980 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.161+0000 7f65397fa640 1 -- 172.21.15.29:0/3394574058 <== mon.2 v2:172.21.15.47:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f654c0043d0 con 0x5577376d0980 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.161+0000 7f65397fa640 1 -- 172.21.15.29:0/3394574058 <== mon.2 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c004b90 con 0x5577376d0980 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.162+0000 7f655f788740 1 -- 172.21.15.29:0/3394574058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376d0980 msgr2=0x557737a9ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.162+0000 7f655f788740 1 --2- 172.21.15.29:0/3394574058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376d0980 0x557737a9ff70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f654c009c00 tx=0x7f654c011810 comp rx=0 tx=0).stop 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.162+0000 7f655259c640 1 --2- 172.21.15.29:0/2008241281 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f653c0b3d30 0x7f653c0b6240 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f654800bfd0 tx=0x7f6548004730 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.162+0000 7f655f788740 1 -- 172.21.15.29:0/3394574058 shutdown_connections 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.162+0000 7f655f788740 1 --2- 172.21.15.29:0/3394574058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 0x557737aa04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.162+0000 7f655f788740 1 --2- 172.21.15.29:0/3394574058 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376d0980 0x557737a9ff70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.162+0000 7f655f788740 1 --2- 172.21.15.29:0/3394574058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376f9840 0x557737a9fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.162+0000 7f655f788740 1 -- 172.21.15.29:0/3394574058 >> 172.21.15.29:0/3394574058 conn(0x5577376ed990 msgr2=0x557737a9d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.162+0000 7f655f788740 1 -- 172.21.15.29:0/3394574058 shutdown_connections 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.162+0000 7f655f788740 1 -- 172.21.15.29:0/3394574058 wait complete. 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.162+0000 7f655f788740 1 -- start start 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655f788740 1 Processor -- start 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655f788740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d0980 0x557737aaaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655f788740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 0x557737aab470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655f788740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376f9840 0x557737aab9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655f788740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5577378677a0 con 0x5577376d8b60 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655f788740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x5577379a62f0 con 0x5577376f9840 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655f788740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x557737aae210 con 0x5577376d0980 2024-07-27T06:50:42.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 0x557737aab470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 0x557737aab470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44756/0 (socket says 172.21.15.29:44756) 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655259c640 1 -- 172.21.15.29:0/1022012330 learned_addr learned my addr 172.21.15.29:0/1022012330 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f6552d9d640 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d0980 0x557737aaaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655359e640 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376f9840 0x557737aab9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655f788740 10 client.0 _ll_register_callbacks cb 0x5577379b0110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655259c640 1 -- 172.21.15.29:0/1022012330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d0980 msgr2=0x557737aaaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655259c640 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d0980 0x557737aaaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655259c640 1 -- 172.21.15.29:0/1022012330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376f9840 msgr2=0x557737aab9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.163+0000 7f655259c640 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376f9840 0x557737aab9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.164+0000 7f655259c640 1 -- 172.21.15.29:0/1022012330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577379aa480 con 0x5577376d8b60 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.164+0000 7f655359e640 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376f9840 0x557737aab9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.164+0000 7f655259c640 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 0x557737aab470 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f654800eac0 tx=0x7f654800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a1acf9c840f897 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:50:42.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.164+0000 7f6536ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.164+0000 7f6536ff5640 1 -- 172.21.15.29:0/1022012330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548021ba0 con 0x5577376d8b60 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.164+0000 7f6536ff5640 1 -- 172.21.15.29:0/1022012330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8844+0+0 (secure 0 0 0) 0x7f654801b030 con 0x5577376d8b60 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.164+0000 7f655f788740 1 -- 172.21.15.29:0/1022012330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f654c0043d0 con 0x5577376d8b60 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.164+0000 7f6536ff5640 1 -- 172.21.15.29:0/1022012330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f6548034050 con 0x5577376d8b60 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.164+0000 7f655f788740 10 client.16410 fetch_fsmap learned FSMap version 60 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.164+0000 7f655f788740 1 -- 172.21.15.29:0/1022012330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x557737773b90 con 0x5577376d8b60 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.164+0000 7f6536ff5640 1 -- 172.21.15.29:0/1022012330 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f654c0043d0 con 0x5577376d8b60 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.165+0000 7f6536ff5640 1 -- 172.21.15.29:0/1022012330 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 60) ==== 8366+0+0 (secure 0 0 0) 0x7f6548029410 con 0x5577376d8b60 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.165+0000 7f655f788740 10 client.16410 fetch_fsmap finished waiting for FSMap version 60 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.165+0000 7f655f788740 10 client.16410 resolve_mds: validated gid 24658 aka daemon mds.c 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.165+0000 7f655f788740 20 client.16410 populate_metadata read hostname 'smithi029' 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.165+0000 7f655f788740 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x5577376b81c0 0x557737abd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.165+0000 7f655f788740 4 client.16410 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:50:42.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.165+0000 7f655f788740 1 -- 172.21.15.29:0/1022012330 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557737a166d0 con 0x5577376b81c0 2024-07-27T06:50:42.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.165+0000 7f6552d9d640 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x5577376b81c0 0x557737abd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.165+0000 7f6552d9d640 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x5577376b81c0 0x557737abd260 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=96397366d2ab7669 server_cookie=f69dddb7921acca5 in_seq=0 out_seq=0 2024-07-27T06:50:42.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.165+0000 7f6536ff5640 10 client.16410 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:50:42.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.166+0000 7f6536ff5640 1 -- 172.21.15.29:0/1022012330 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557737a166d0 con 0x5577376b81c0 2024-07-27T06:50:42.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.166+0000 7f6536ff5640 10 client.16410 handle_command_reply: tid=0 2024-07-27T06:50:42.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.166+0000 7f655f788740 1 client.16410 shutdown 2024-07-27T06:50:42.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.166+0000 7f655f788740 1 -- 172.21.15.29:0/1022012330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 msgr2=0x557737aab470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:42.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.166+0000 7f655f788740 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 0x557737aab470 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f654800eac0 tx=0x7f654800b410 comp rx=0 tx=0).stop 2024-07-27T06:50:42.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.166+0000 7f655f788740 1 -- 172.21.15.29:0/1022012330 shutdown_connections 2024-07-27T06:50:42.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.166+0000 7f655f788740 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x5577376b81c0 0x557737abd260 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.166+0000 7f655f788740 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376f9840 0x557737aab9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.166+0000 7f655f788740 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d8b60 0x557737aab470 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.166+0000 7f655f788740 1 --2- 172.21.15.29:0/1022012330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d0980 0x557737aaaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.166+0000 7f655f788740 1 -- 172.21.15.29:0/1022012330 >> 172.21.15.29:0/1022012330 conn(0x5577376ed990 msgr2=0x557737aa2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:50:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.167+0000 7f6536ff5640 0 client.16410 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:50:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.167+0000 7f655f788740 1 -- 172.21.15.29:0/1022012330 shutdown_connections 2024-07-27T06:50:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.167+0000 7f655f788740 1 -- 172.21.15.29:0/1022012330 wait complete. 2024-07-27T06:50:42.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.167+0000 7f655f788740 20 client.16410 trim_cache size 0 max 16384 2024-07-27T06:50:42.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.184+0000 7f655f788740 1 Processor -- start 2024-07-27T06:50:42.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.184+0000 7f655f788740 1 -- start start 2024-07-27T06:50:42.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.184+0000 7f655f788740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376d0980 0x557737aa1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.184+0000 7f655f788740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d8b60 0x557737ac2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.184+0000 7f655f788740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376f9840 0x557737ac2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.184+0000 7f655f788740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f65541af1e0 con 0x5577376f9840 2024-07-27T06:50:42.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.184+0000 7f655f788740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f65541b0d50 con 0x5577376d0980 2024-07-27T06:50:42.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.184+0000 7f655f788740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f65541af8b0 con 0x5577376d8b60 2024-07-27T06:50:42.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.184+0000 7f655359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376f9840 0x557737ac2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.184+0000 7f655359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376f9840 0x557737ac2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44772/0 (socket says 172.21.15.29:44772) 2024-07-27T06:50:42.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.184+0000 7f655359e640 1 -- 172.21.15.29:0/3413940149 learned_addr learned my addr 172.21.15.29:0/3413940149 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:50:42.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.184+0000 7f6552d9d640 1 --2- 172.21.15.29:0/3413940149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376d0980 0x557737aa1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.185+0000 7f655259c640 1 --2- 172.21.15.29:0/3413940149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d8b60 0x557737ac2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.185+0000 7f655359e640 1 -- 172.21.15.29:0/3413940149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d8b60 msgr2=0x557737ac2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:50:42.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.185+0000 7f655359e640 1 --2- 172.21.15.29:0/3413940149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d8b60 0x557737ac2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.185+0000 7f655359e640 1 -- 172.21.15.29:0/3413940149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376d0980 msgr2=0x557737aa1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:50:42.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.185+0000 7f655359e640 1 --2- 172.21.15.29:0/3413940149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376d0980 0x557737aa1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.185+0000 7f655359e640 1 -- 172.21.15.29:0/3413940149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557737aac1f0 con 0x5577376f9840 2024-07-27T06:50:42.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.185+0000 7f6552d9d640 1 --2- 172.21.15.29:0/3413940149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376d0980 0x557737aa1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:50:42.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.185+0000 7f655359e640 1 --2- 172.21.15.29:0/3413940149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376f9840 0x557737ac2f60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f65440094e0 tx=0x7f65440cbb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49eaccc984aba12a server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:50:42.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.185+0000 7f6538ff9640 1 -- 172.21.15.29:0/3413940149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65440cc7a0 con 0x5577376f9840 2024-07-27T06:50:42.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.185+0000 7f6538ff9640 1 -- 172.21.15.29:0/3413940149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6544036040 con 0x5577376f9840 2024-07-27T06:50:42.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.185+0000 7f6538ff9640 1 -- 172.21.15.29:0/3413940149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65440cc300 con 0x5577376f9840 2024-07-27T06:50:42.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.186+0000 7f655f788740 1 -- 172.21.15.29:0/3413940149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376f9840 msgr2=0x557737ac2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:42.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.186+0000 7f655f788740 1 --2- 172.21.15.29:0/3413940149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376f9840 0x557737ac2f60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f65440094e0 tx=0x7f65440cbb00 comp rx=0 tx=0).stop 2024-07-27T06:50:42.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.186+0000 7f655f788740 1 -- 172.21.15.29:0/3413940149 shutdown_connections 2024-07-27T06:50:42.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.186+0000 7f655f788740 1 --2- 172.21.15.29:0/3413940149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376f9840 0x557737ac2f60 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.186+0000 7f655f788740 1 --2- 172.21.15.29:0/3413940149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d8b60 0x557737ac2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.186+0000 7f655f788740 1 --2- 172.21.15.29:0/3413940149 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376d0980 0x557737aa1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.186+0000 7f655f788740 1 -- 172.21.15.29:0/3413940149 >> 172.21.15.29:0/3413940149 conn(0x5577376ed990 msgr2=0x557737a9c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:50:42.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.186+0000 7f655f788740 1 -- 172.21.15.29:0/3413940149 shutdown_connections 2024-07-27T06:50:42.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.186+0000 7f655f788740 1 -- 172.21.15.29:0/3413940149 wait complete. 2024-07-27T06:50:42.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.186+0000 7f655f788740 1 -- start start 2024-07-27T06:50:42.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.186+0000 7f655f788740 1 Processor -- start 2024-07-27T06:50:42.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f655f788740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d0980 0x557737ab3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f655f788740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d8b60 0x557737ab1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f655f788740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376f9840 0x557737ab1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f655f788740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f65541b0680 con 0x5577376d0980 2024-07-27T06:50:42.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f655f788740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f65541affb0 con 0x5577376f9840 2024-07-27T06:50:42.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f655f788740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f65541ad1c0 con 0x5577376d8b60 2024-07-27T06:50:42.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f655f788740 10 client.0 _ll_register_callbacks cb 0x7f6548012d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:50:42.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f6552d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d0980 0x557737ab3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f6552d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d0980 0x557737ab3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44788/0 (socket says 172.21.15.29:44788) 2024-07-27T06:50:42.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f6552d9d640 1 -- 172.21.15.29:0/1468130467 learned_addr learned my addr 172.21.15.29:0/1468130467 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:50:42.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f655259c640 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d8b60 0x557737ab1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f655359e640 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376f9840 0x557737ab1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f6552d9d640 1 -- 172.21.15.29:0/1468130467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d8b60 msgr2=0x557737ab1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:50:42.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f6552d9d640 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d8b60 0x557737ab1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f6552d9d640 1 -- 172.21.15.29:0/1468130467 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376f9840 msgr2=0x557737ab1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:50:42.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f6552d9d640 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376f9840 0x557737ab1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.187+0000 7f6552d9d640 1 -- 172.21.15.29:0/1468130467 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557737ab1cf0 con 0x5577376d0980 2024-07-27T06:50:42.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f6552d9d640 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d0980 0x557737ab3b70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f654c000c00 tx=0x7f654c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e7465eadde532d server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:50:42.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f6536ff5640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:50:42.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f6536ff5640 1 -- 172.21.15.29:0/1468130467 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c01e040 con 0x5577376d0980 2024-07-27T06:50:42.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f6536ff5640 1 -- 172.21.15.29:0/1468130467 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8844+0+0 (secure 0 0 0) 0x7f654c019880 con 0x5577376d0980 2024-07-27T06:50:42.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f655f788740 1 -- 172.21.15.29:0/1468130467 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6544036040 con 0x5577376d0980 2024-07-27T06:50:42.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f6536ff5640 1 -- 172.21.15.29:0/1468130467 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f654c019a80 con 0x5577376d0980 2024-07-27T06:50:42.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f655f788740 10 client.16416 fetch_fsmap learned FSMap version 60 2024-07-27T06:50:42.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f655f788740 1 -- 172.21.15.29:0/1468130467 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x557737b25900 con 0x5577376d0980 2024-07-27T06:50:42.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f6536ff5640 1 -- 172.21.15.29:0/1468130467 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6544036040 con 0x5577376d0980 2024-07-27T06:50:42.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f6536ff5640 1 -- 172.21.15.29:0/1468130467 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 60) ==== 8366+0+0 (secure 0 0 0) 0x7f654c024040 con 0x5577376d0980 2024-07-27T06:50:42.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f655f788740 10 client.16416 fetch_fsmap finished waiting for FSMap version 60 2024-07-27T06:50:42.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f655f788740 10 client.16416 resolve_mds: validated gid 24658 aka daemon mds.c 2024-07-27T06:50:42.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f655f788740 20 client.16416 populate_metadata read hostname 'smithi029' 2024-07-27T06:50:42.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f655f788740 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x55773768d570 0x557737b27fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:42.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f655f788740 4 client.16416 mds_command: new command op to 24658 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 151.1, "await_for": 45.0}] 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.188+0000 7f655f788740 1 -- 172.21.15.29:0/1468130467 --> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 151.1, "await_for": 45.0}) -- 0x557737a12170 con 0x55773768d570 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.189+0000 7f655259c640 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x55773768d570 0x557737b27fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.189+0000 7f655259c640 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x55773768d570 0x557737b27fa0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=31619dc839f5a9c3 server_cookie=be3d5cc0e436631e in_seq=0 out_seq=0 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.189+0000 7f6536ff5640 10 client.16416 ms_handle_connect on v2:172.21.15.77:6836/2959628937 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f6536ff5640 1 -- 172.21.15.29:0/1468130467 <== mds.1 v2:172.21.15.77:6836/2959628937 1 ==== command_reply(tid 0: -25 Error submitting the command to the local db manager) ==== 60+0+41 (crc 0 0 0) 0x557737a12170 con 0x55773768d570 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f6536ff5640 10 client.16416 handle_command_reply: tid=0 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f655f788740 1 client.16416 shutdown 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f655f788740 1 -- 172.21.15.29:0/1468130467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d0980 msgr2=0x557737ab3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f655f788740 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d0980 0x557737ab3b70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f654c000c00 tx=0x7f654c009b50 comp rx=0 tx=0).stop 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f655f788740 1 -- 172.21.15.29:0/1468130467 shutdown_connections 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f655f788740 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] conn(0x55773768d570 0x557737b27fa0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f655f788740 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x5577376f9840 0x557737ab1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f655f788740 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5577376d8b60 0x557737ab1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f655f788740 1 --2- 172.21.15.29:0/1468130467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5577376d0980 0x557737ab3b70 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f655f788740 1 -- 172.21.15.29:0/1468130467 >> 172.21.15.29:0/1468130467 conn(0x5577376ed990 msgr2=0x557737a9c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:50:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f6536ff5640 0 client.16416 ms_handle_reset on v2:172.21.15.77:6836/2959628937 2024-07-27T06:50:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f655f788740 1 -- 172.21.15.29:0/1468130467 shutdown_connections 2024-07-27T06:50:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f655f788740 1 -- 172.21.15.29:0/1468130467 wait complete. 2024-07-27T06:50:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:42.190+0000 7f655f788740 20 client.16416 trim_cache size 0 max 16384 2024-07-27T06:50:42.198 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-07-27T06:50:42.198 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 60, 2024-07-27T06:50:42.198 INFO:teuthology.orchestra.run.smithi029.stdout: "leader": 26026 2024-07-27T06:50:42.199 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-07-27T06:50:42.199 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOTTY: Error submitting the command to the local db manager 2024-07-27T06:50:42.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:41 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:rejoin 2024-07-27T06:50:42.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:41 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:50:42.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:41 smithi047 ceph-mon[36835]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:50:42.896 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:42 smithi077 podman[72222]: 2024-07-27 06:50:42.646021892 +0000 UTC m=+0.721030426 container remove 671e0da12d33bdf6578bfed1c81bee8096c2862216e8f96c64083024e848683e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-07-27T06:50:42.897 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:42 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:50:42.897 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:42 smithi077 ceph-mon[37153]: pgmap v512: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-27T06:50:42.897 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:42 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:50:42.897 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:42 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:50:42.897 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:42 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:active 2024-07-27T06:50:42.897 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:42 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:50:42.897 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:42 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:50:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:42 smithi029 ceph-mon[32610]: pgmap v512: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-27T06:50:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:42 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:50:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:42 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:50:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:42 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:active 2024-07-27T06:50:42.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:42 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:50:42.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:42 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:50:43.044 DEBUG:teuthology.orchestra.run:got remote process result: 25 2024-07-27T06:50:43.044 INFO:tasks.quiescer.fs.[cephfs]:Retrying a quiesce db command with leader 26026 2024-07-27T06:50:43.044 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.26026 quiesce db / --timeout 90 --expiration 151.1 --await-for 45.0' 2024-07-27T06:50:43.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- timeout 105 ceph tell mds.26026 quiesce db / --timeout 90 --expiration 151.1 --await-for 45.0 2024-07-27T06:50:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:42 smithi047 ceph-mon[36835]: pgmap v512: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-27T06:50:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:42 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:50:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:42 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:50:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:42 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] up:active 2024-07-27T06:50:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:42 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:50:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:42 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:50:43.320 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T06:50:43.356 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:42 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-27T06:50:43.357 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:42 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Consumed 1.083s CPU time. 2024-07-27T06:50:45.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:44 smithi029 ceph-mon[32610]: pgmap v513: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 102 B/s wr, 16 op/s 2024-07-27T06:50:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.115+0000 7f2c8e389640 1 -- 172.21.15.29:0/3636722934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c88107df0 msgr2=0x7f2c88108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.115+0000 7f2c8e389640 1 --2- 172.21.15.29:0/3636722934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c88107df0 0x7f2c88108200 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009900 tx=0x7f2c7c02a410 comp rx=0 tx=0).stop 2024-07-27T06:50:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.115+0000 7f2c8e389640 1 -- 172.21.15.29:0/3636722934 shutdown_connections 2024-07-27T06:50:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.115+0000 7f2c8e389640 1 --2- 172.21.15.29:0/3636722934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c88108740 0x7f2c8810ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.115+0000 7f2c8e389640 1 --2- 172.21.15.29:0/3636722934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c88107df0 0x7f2c88108200 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.115+0000 7f2c8e389640 1 --2- 172.21.15.29:0/3636722934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c881024b0 0x7f2c881028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.115+0000 7f2c8e389640 1 -- 172.21.15.29:0/3636722934 >> 172.21.15.29:0/3636722934 conn(0x7f2c88079190 msgr2=0x7f2c880795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:50:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.115+0000 7f2c8e389640 1 -- 172.21.15.29:0/3636722934 shutdown_connections 2024-07-27T06:50:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.116+0000 7f2c8e389640 1 -- 172.21.15.29:0/3636722934 wait complete. 2024-07-27T06:50:45.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.116+0000 7f2c8e389640 1 Processor -- start 2024-07-27T06:50:45.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.116+0000 7f2c8e389640 1 -- start start 2024-07-27T06:50:45.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c8e389640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c881024b0 0x7f2c88199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:45.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c8e389640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c88107df0 0x7f2c8819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:45.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c8e389640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c88108740 0x7f2c8819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:45.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c8e389640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2c88112390 con 0x7f2c88107df0 2024-07-27T06:50:45.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c8e389640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f2c88112210 con 0x7f2c881024b0 2024-07-27T06:50:45.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c8e389640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2c88112510 con 0x7f2c88108740 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c88107df0 0x7f2c8819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c88107df0 0x7f2c8819a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44816/0 (socket says 172.21.15.29:44816) 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c877fe640 1 -- 172.21.15.29:0/4225126872 learned_addr learned my addr 172.21.15.29:0/4225126872 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c87fff640 1 --2- 172.21.15.29:0/4225126872 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c881024b0 0x7f2c88199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c8c926640 1 --2- 172.21.15.29:0/4225126872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c88108740 0x7f2c8819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c877fe640 1 -- 172.21.15.29:0/4225126872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c88108740 msgr2=0x7f2c8819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c877fe640 1 --2- 172.21.15.29:0/4225126872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c88108740 0x7f2c8819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c877fe640 1 -- 172.21.15.29:0/4225126872 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c881024b0 msgr2=0x7f2c88199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c877fe640 1 --2- 172.21.15.29:0/4225126872 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2c881024b0 0x7f2c88199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c877fe640 1 -- 172.21.15.29:0/4225126872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c8819fea0 con 0x7f2c88107df0 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.117+0000 7f2c877fe640 1 --2- 172.21.15.29:0/4225126872 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c88107df0 0x7f2c8819a4d0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c006010 tx=0x7f2c7c004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.118+0000 7f2c857fa640 1 -- 172.21.15.29:0/4225126872 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c03f030 con 0x7f2c88107df0 2024-07-27T06:50:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.118+0000 7f2c857fa640 1 -- 172.21.15.29:0/4225126872 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c7c003aa0 con 0x7f2c88107df0 2024-07-27T06:50:45.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.118+0000 7f2c857fa640 1 -- 172.21.15.29:0/4225126872 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c035070 con 0x7f2c88107df0 2024-07-27T06:50:45.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.118+0000 7f2c8e389640 1 -- 172.21.15.29:0/4225126872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c881a0130 con 0x7f2c88107df0 2024-07-27T06:50:45.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.118+0000 7f2c8e389640 1 -- 172.21.15.29:0/4225126872 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c881a68f0 con 0x7f2c88107df0 2024-07-27T06:50:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.120+0000 7f2c9490d740 1 Processor -- start 2024-07-27T06:50:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.120+0000 7f2c857fa640 1 -- 172.21.15.29:0/4225126872 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f2c7c042030 con 0x7f2c88107df0 2024-07-27T06:50:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.120+0000 7f2c9490d740 1 -- start start 2024-07-27T06:50:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c9490d740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560bf314d840 0x560bf34f3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c9490d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560bf3124980 0x560bf34f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c9490d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560bf312cb60 0x560bf34f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c9490d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560bf3468780 con 0x560bf312cb60 2024-07-27T06:50:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c9490d740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x560bf3457c80 con 0x560bf314d840 2024-07-27T06:50:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c9490d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560bf345bcc0 con 0x560bf3124980 2024-07-27T06:50:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560bf312cb60 0x560bf34f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560bf312cb60 0x560bf34f42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44822/0 (socket says 172.21.15.29:44822) 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c877fe640 1 -- 172.21.15.29:0/475819579 learned_addr learned my addr 172.21.15.29:0/475819579 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c8c926640 1 --2- 172.21.15.29:0/475819579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560bf314d840 0x560bf34f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c87fff640 1 --2- 172.21.15.29:0/475819579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560bf3124980 0x560bf34f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c877fe640 1 -- 172.21.15.29:0/475819579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560bf3124980 msgr2=0x560bf34f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c877fe640 1 --2- 172.21.15.29:0/475819579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560bf3124980 0x560bf34f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c877fe640 1 -- 172.21.15.29:0/475819579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560bf314d840 msgr2=0x560bf34f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c877fe640 1 --2- 172.21.15.29:0/475819579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560bf314d840 0x560bf34f3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c877fe640 1 -- 172.21.15.29:0/475819579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bf33ed080 con 0x560bf312cb60 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c8c926640 1 --2- 172.21.15.29:0/475819579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560bf314d840 0x560bf34f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.121+0000 7f2c857fa640 1 --2- 172.21.15.29:0/4225126872 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2c700b3dc0 0x7f2c700b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.122+0000 7f2c877fe640 1 --2- 172.21.15.29:0/475819579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560bf312cb60 0x560bf34f42c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c101a30 tx=0x7f2c7c10a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d4bc0d8dbc0907 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:50:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.122+0000 7f2c8c926640 1 --2- 172.21.15.29:0/4225126872 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2c700b3dc0 0x7f2c700b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:45.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.122+0000 7f2c857fa640 1 -- 172.21.15.29:0/4225126872 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8844+0+0 (secure 0 0 0) 0x7f2c7c104480 con 0x7f2c88107df0 2024-07-27T06:50:45.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.122+0000 7f2c6e7fc640 1 -- 172.21.15.29:0/475819579 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c0c6ec0 con 0x560bf312cb60 2024-07-27T06:50:45.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.122+0000 7f2c6e7fc640 1 -- 172.21.15.29:0/475819579 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c7c03b460 con 0x560bf312cb60 2024-07-27T06:50:45.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.122+0000 7f2c6e7fc640 1 -- 172.21.15.29:0/475819579 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c0ca770 con 0x560bf312cb60 2024-07-27T06:50:45.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.123+0000 7f2c9490d740 1 -- 172.21.15.29:0/475819579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560bf312cb60 msgr2=0x560bf34f42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T06:50:45.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.123+0000 7f2c9490d740 1 --2- 172.21.15.29:0/475819579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560bf312cb60 0x560bf34f42c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c101a30 tx=0x7f2c7c10a730 comp rx=0 tx=0).stop 2024-07-27T06:50:45.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.123+0000 7f2c8c926640 1 --2- 172.21.15.29:0/4225126872 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f2c700b3dc0 0x7f2c700b62d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2c80009fd0 tx=0x7f2c80009500 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:50:45.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.123+0000 7f2c9490d740 1 -- 172.21.15.29:0/475819579 shutdown_connections 2024-07-27T06:50:45.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.123+0000 7f2c9490d740 1 --2- 172.21.15.29:0/475819579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560bf312cb60 0x560bf34f42c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:45.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.123+0000 7f2c9490d740 1 --2- 172.21.15.29:0/475819579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560bf3124980 0x560bf34f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:45.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.123+0000 7f2c9490d740 1 --2- 172.21.15.29:0/475819579 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560bf314d840 0x560bf34f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:45.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.123+0000 7f2c9490d740 1 -- 172.21.15.29:0/475819579 >> 172.21.15.29:0/475819579 conn(0x560bf3141990 msgr2=0x560bf34f1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.123+0000 7f2c9490d740 1 -- 172.21.15.29:0/475819579 shutdown_connections 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.123+0000 7f2c9490d740 1 -- 172.21.15.29:0/475819579 wait complete. 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.123+0000 7f2c9490d740 1 -- start start 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c9490d740 1 Processor -- start 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c9490d740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560bf3124980 0x560bf34fed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c9490d740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560bf312cb60 0x560bf34ff2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c9490d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560bf314d840 0x560bf34ff7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c9490d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x560bf32bb7a0 con 0x560bf314d840 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c9490d740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x560bf33fa2f0 con 0x560bf3124980 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c9490d740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560bf35061f0 con 0x560bf312cb60 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c9490d740 10 client.0 _ll_register_callbacks cb 0x560bf3404110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c8c926640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560bf314d840 0x560bf34ff7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c8c926640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560bf314d840 0x560bf34ff7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44836/0 (socket says 172.21.15.29:44836) 2024-07-27T06:50:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c8c926640 1 -- 172.21.15.29:0/3317683865 learned_addr learned my addr 172.21.15.29:0/3317683865 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T06:50:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560bf3124980 0x560bf34fed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.124+0000 7f2c877fe640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560bf312cb60 0x560bf34ff2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c8c926640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560bf312cb60 msgr2=0x560bf34ff2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:50:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c8c926640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560bf312cb60 0x560bf34ff2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c8c926640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560bf3124980 msgr2=0x560bf34fed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T06:50:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c8c926640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560bf3124980 0x560bf34fed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T06:50:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c8c926640 1 -- 172.21.15.29:0/3317683865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bf33fe480 con 0x560bf314d840 2024-07-27T06:50:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c877fe640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560bf312cb60 0x560bf34ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-27T06:50:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x560bf3124980 0x560bf34fed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T06:50:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c8c926640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x560bf314d840 0x560bf34ff7f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000f6a0 tx=0x7f2c80004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89aee4bccb9cecc2 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T06:50:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c677fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c677fe640 1 -- 172.21.15.29:0/3317683865 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8002c060 con 0x560bf314d840 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c677fe640 1 -- 172.21.15.29:0/3317683865 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8844+0+0 (secure 0 0 0) 0x7f2c80017030 con 0x560bf314d840 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c9490d740 1 -- 172.21.15.29:0/3317683865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560bf346a6d0 con 0x560bf314d840 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c677fe640 1 -- 172.21.15.29:0/3317683865 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f2c80033050 con 0x560bf314d840 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c9490d740 10 client.16428 fetch_fsmap learned FSMap version 60 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.125+0000 7f2c9490d740 1 -- 172.21.15.29:0/3317683865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=4+}) -- 0x560bf31c7b90 con 0x560bf314d840 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.126+0000 7f2c677fe640 1 -- 172.21.15.29:0/3317683865 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560bf346a6d0 con 0x560bf314d840 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.126+0000 7f2c677fe640 1 -- 172.21.15.29:0/3317683865 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 60) ==== 8366+0+0 (secure 0 0 0) 0x7f2c800279b0 con 0x560bf314d840 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.126+0000 7f2c9490d740 10 client.16428 fetch_fsmap finished waiting for FSMap version 60 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.126+0000 7f2c9490d740 10 client.16428 resolve_mds: validated gid 26026 aka daemon mds.i 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.126+0000 7f2c9490d740 20 client.16428 populate_metadata read hostname 'smithi029' 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.126+0000 7f2c9490d740 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.126+0000 7f2c9490d740 4 client.16428 mds_command: new command op to 26026 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-27T06:50:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.126+0000 7f2c9490d740 1 -- 172.21.15.29:0/3317683865 --> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bf3466170 con 0x560bf310c1c0 2024-07-27T06:50:45.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.126+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/2224015949 2024-07-27T06:50:45.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.126+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-27T06:50:45.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.126+0000 7f2c677fe640 1 client.16428 ms_handle_refused on v2:172.21.15.77:6838/2224015949 2024-07-27T06:50:45.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:44 smithi077 ceph-mon[37153]: pgmap v513: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 102 B/s wr, 16 op/s 2024-07-27T06:50:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:44 smithi047 ceph-mon[36835]: pgmap v513: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 102 B/s wr, 16 op/s 2024-07-27T06:50:45.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.327+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/2224015949 2024-07-27T06:50:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.327+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-27T06:50:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.327+0000 7f2c677fe640 1 client.16428 ms_handle_refused on v2:172.21.15.77:6838/2224015949 2024-07-27T06:50:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.728+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/2224015949 2024-07-27T06:50:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.728+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-27T06:50:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:45.728+0000 7f2c677fe640 1 client.16428 ms_handle_refused on v2:172.21.15.77:6838/2224015949 2024-07-27T06:50:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:46.530+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/2224015949 2024-07-27T06:50:46.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:46.530+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-27T06:50:46.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:46.530+0000 7f2c677fe640 1 client.16428 ms_handle_refused on v2:172.21.15.77:6838/2224015949 2024-07-27T06:50:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:46 smithi047 ceph-mon[36835]: pgmap v514: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 102 B/s wr, 19 op/s 2024-07-27T06:50:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:46 smithi029 ceph-mon[32610]: pgmap v514: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 102 B/s wr, 19 op/s 2024-07-27T06:50:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:46 smithi077 ceph-mon[37153]: pgmap v514: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 102 B/s wr, 19 op/s 2024-07-27T06:50:48.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:48.132+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/2224015949 2024-07-27T06:50:48.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:48.132+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-27T06:50:48.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:48.132+0000 7f2c677fe640 1 client.16428 ms_handle_refused on v2:172.21.15.77:6838/2224015949 2024-07-27T06:50:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:48 smithi047 ceph-mon[36835]: pgmap v515: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 199 B/s wr, 20 op/s 2024-07-27T06:50:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:48 smithi029 ceph-mon[32610]: pgmap v515: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 199 B/s wr, 20 op/s 2024-07-27T06:50:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:48 smithi077 ceph-mon[37153]: pgmap v515: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 199 B/s wr, 20 op/s 2024-07-27T06:50:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:50 smithi029 ceph-mon[32610]: pgmap v516: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:50:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:50 smithi047 ceph-mon[36835]: pgmap v516: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:50:51.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:51.335+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/2224015949 2024-07-27T06:50:51.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:51.335+0000 7f2c677fe640 1 client.16428 ms_handle_refused on v2:172.21.15.77:6838/2224015949 2024-07-27T06:50:51.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:51.335+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-27T06:50:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:50 smithi077 ceph-mon[37153]: pgmap v516: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:50:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:53.108+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:50:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:53.108+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:50:53.109 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:50:53.108+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:50:53.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:53.112+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:53.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:52 smithi029 ceph-mon[32610]: pgmap v517: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:50:53.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:53.112+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-27T06:50:53.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:53.112+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:50:53.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:53.112+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:50:53.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:53.112+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:50:53.121 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:52 smithi077 ceph-mon[37153]: pgmap v517: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:50:53.121 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:53 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 3. 2024-07-27T06:50:53.121 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:53 smithi077 systemd[1]: Stopped Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:50:53.121 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:53 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Consumed 1.083s CPU time. 2024-07-27T06:50:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:52 smithi047 ceph-mon[36835]: pgmap v517: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:50:53.481 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:53 smithi077 systemd[1]: Starting Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:50:53.481 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:53 smithi077 podman[72316]: 2024-07-27 06:50:53.246240732 +0000 UTC m=+0.017506022 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:50:53.481 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:53 smithi077 podman[72316]: 2024-07-27 06:50:53.363544382 +0000 UTC m=+0.134809659 container create 888e4a119d6ce17133e621381c8fb21e67c0e972363e6d114f3a11d5afdf82dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724) 2024-07-27T06:50:53.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:53 smithi077 podman[72316]: 2024-07-27 06:50:53.58288698 +0000 UTC m=+0.354152253 container init 888e4a119d6ce17133e621381c8fb21e67c0e972363e6d114f3a11d5afdf82dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-07-27T06:50:53.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:53 smithi077 podman[72316]: 2024-07-27 06:50:53.585476958 +0000 UTC m=+0.356742231 container start 888e4a119d6ce17133e621381c8fb21e67c0e972363e6d114f3a11d5afdf82dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-27T06:50:53.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:53 smithi077 ceph-mds[72334]: -- 172.21.15.77:0/229650024 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627ecf0d860 con 0x5627f10afa80 2024-07-27T06:50:53.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:53 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: starting mds.i at 2024-07-27T06:50:53.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:53 smithi077 bash[72316]: 888e4a119d6ce17133e621381c8fb21e67c0e972363e6d114f3a11d5afdf82dc 2024-07-27T06:50:53.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:50:53 smithi077 systemd[1]: Started Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:50:54.130 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:53 smithi029 ceph-mon[32610]: daemon mds.i restarted 2024-07-27T06:50:54.130 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:53 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:50:54.130 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:53 smithi029 ceph-mon[32610]: osdmap e88: 12 total, 12 up, 12 in 2024-07-27T06:50:54.130 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:53 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:boot 2024-07-27T06:50:54.130 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:53 smithi029 ceph-mon[32610]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:50:54.130 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:53 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:50:54.130 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:50:54.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:53 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:50:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:53 smithi047 ceph-mon[36835]: daemon mds.i restarted 2024-07-27T06:50:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:53 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:50:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:53 smithi047 ceph-mon[36835]: osdmap e88: 12 total, 12 up, 12 in 2024-07-27T06:50:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:53 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:boot 2024-07-27T06:50:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:53 smithi047 ceph-mon[36835]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:50:54.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:53 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:50:54.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:50:54.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:53 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:50:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:53 smithi077 ceph-mon[37153]: daemon mds.i restarted 2024-07-27T06:50:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:53 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:50:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:53 smithi077 ceph-mon[37153]: osdmap e88: 12 total, 12 up, 12 in 2024-07-27T06:50:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:53 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:boot 2024-07-27T06:50:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:53 smithi077 ceph-mon[37153]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:50:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:53 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:50:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:50:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:53 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:50:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:54 smithi047 ceph-mon[36835]: pgmap v519: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:50:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:54 smithi029 ceph-mon[32610]: pgmap v519: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:50:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:54 smithi077 ceph-mon[37153]: pgmap v519: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:50:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:55 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:resolve 2024-07-27T06:50:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:55 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:50:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:55 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:resolve 2024-07-27T06:50:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:55 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:50:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:55 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:resolve 2024-07-27T06:50:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:55 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:50:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:56 smithi047 ceph-mon[36835]: pgmap v520: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:50:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:56 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:50:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:56 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0 2024-07-27T06:50:57.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:56 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:reconnect 2024-07-27T06:50:57.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:56 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:50:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:56 smithi029 ceph-mon[32610]: pgmap v520: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:50:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:56 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:50:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:56 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0 2024-07-27T06:50:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:56 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:reconnect 2024-07-27T06:50:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:56 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:50:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:56 smithi077 ceph-mon[37153]: pgmap v520: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:50:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:56 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:50:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:56 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0 2024-07-27T06:50:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:56 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:reconnect 2024-07-27T06:50:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:56 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:50:57.697 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:50:57.697 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: *** Caught signal (Aborted) ** 2024-07-27T06:50:57.697 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: in thread 7fb716336640 thread_name:ms_dispatch 2024-07-27T06:50:57.697 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:57.697 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb71b70d6f0] 2024-07-27T06:50:57.697 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb71b75a94c] 2024-07-27T06:50:57.697 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 3: raise() 2024-07-27T06:50:57.698 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 4: abort() 2024-07-27T06:50:57.698 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb71b6f771b] 2024-07-27T06:50:57.698 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 6: /lib64/libc.so.6(+0x37386) [0x7fb71b706386] 2024-07-27T06:50:57.698 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55655a1766b9] 2024-07-27T06:50:57.698 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55655a33480c] 2024-07-27T06:50:57.698 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55655a282cde] 2024-07-27T06:50:57.698 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55655a244763] 2024-07-27T06:50:57.698 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55655a245a1b] 2024-07-27T06:50:57.698 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55655a229752] 2024-07-27T06:50:57.698 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55655a177a56] 2024-07-27T06:50:57.698 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55655a17b637] 2024-07-27T06:50:57.698 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55655a17bf5e] 2024-07-27T06:50:57.699 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55655a15f03a] 2024-07-27T06:50:57.699 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb71bfda407] 2024-07-27T06:50:57.699 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 18: (DispatchQueue::entry()+0x60f) [0x7fb71bfdacf9] 2024-07-27T06:50:57.699 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb71c0b75a1] 2024-07-27T06:50:57.700 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 20: (Thread::entry_wrapper()+0x43) [0x7fb71be64cdd] 2024-07-27T06:50:57.700 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb71be64cf9] 2024-07-27T06:50:57.700 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb71b758c02] 2024-07-27T06:50:57.700 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb71b7ddc40] 2024-07-27T06:50:57.700 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 2024-07-27T06:50:57.430+0000 7fb716336640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:50:57.700 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: in thread 7fb716336640 thread_name:ms_dispatch 2024-07-27T06:50:57.700 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 2024-07-27T06:50:57.700 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:57.700 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb71b70d6f0] 2024-07-27T06:50:57.700 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb71b75a94c] 2024-07-27T06:50:57.700 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 3: raise() 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 4: abort() 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb71b6f771b] 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 6: /lib64/libc.so.6(+0x37386) [0x7fb71b706386] 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55655a1766b9] 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55655a33480c] 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55655a282cde] 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55655a244763] 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55655a245a1b] 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55655a229752] 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55655a177a56] 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55655a17b637] 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55655a17bf5e] 2024-07-27T06:50:57.701 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55655a15f03a] 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb71bfda407] 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 18: (DispatchQueue::entry()+0x60f) [0x7fb71bfdacf9] 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb71c0b75a1] 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 20: (Thread::entry_wrapper()+0x43) [0x7fb71be64cdd] 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb71be64cf9] 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb71b758c02] 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb71b7ddc40] 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 0> 2024-07-27T06:50:57.430+0000 7fb716336640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: in thread 7fb716336640 thread_name:ms_dispatch 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:57.702 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb71b70d6f0] 2024-07-27T06:50:57.703 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb71b75a94c] 2024-07-27T06:50:57.703 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 3: raise() 2024-07-27T06:50:57.703 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 4: abort() 2024-07-27T06:50:57.703 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb71b6f771b] 2024-07-27T06:50:57.703 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 6: /lib64/libc.so.6(+0x37386) [0x7fb71b706386] 2024-07-27T06:50:57.703 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55655a1766b9] 2024-07-27T06:50:57.703 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55655a33480c] 2024-07-27T06:50:57.703 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55655a282cde] 2024-07-27T06:50:57.703 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55655a244763] 2024-07-27T06:50:57.704 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55655a245a1b] 2024-07-27T06:50:57.704 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55655a229752] 2024-07-27T06:50:57.704 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55655a177a56] 2024-07-27T06:50:57.704 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55655a17b637] 2024-07-27T06:50:57.704 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55655a17bf5e] 2024-07-27T06:50:57.704 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55655a15f03a] 2024-07-27T06:50:57.704 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb71bfda407] 2024-07-27T06:50:57.704 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 18: (DispatchQueue::entry()+0x60f) [0x7fb71bfdacf9] 2024-07-27T06:50:57.704 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb71c0b75a1] 2024-07-27T06:50:57.704 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 20: (Thread::entry_wrapper()+0x43) [0x7fb71be64cdd] 2024-07-27T06:50:57.705 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb71be64cf9] 2024-07-27T06:50:57.705 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb71b758c02] 2024-07-27T06:50:57.705 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb71b7ddc40] 2024-07-27T06:50:57.705 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:50:57.705 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 2024-07-27T06:50:57.705 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 0> 2024-07-27T06:50:57.430+0000 7fb716336640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:50:57.705 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: in thread 7fb716336640 thread_name:ms_dispatch 2024-07-27T06:50:57.705 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 2024-07-27T06:50:57.705 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:50:57.706 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb71b70d6f0] 2024-07-27T06:50:57.706 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb71b75a94c] 2024-07-27T06:50:57.706 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 3: raise() 2024-07-27T06:50:57.706 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 4: abort() 2024-07-27T06:50:57.706 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb71b6f771b] 2024-07-27T06:50:57.706 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 6: /lib64/libc.so.6(+0x37386) [0x7fb71b706386] 2024-07-27T06:50:57.706 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55655a1766b9] 2024-07-27T06:50:57.706 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55655a33480c] 2024-07-27T06:50:57.706 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55655a282cde] 2024-07-27T06:50:57.706 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55655a244763] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55655a245a1b] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55655a229752] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55655a177a56] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55655a17b637] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55655a17bf5e] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55655a15f03a] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb71bfda407] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 18: (DispatchQueue::entry()+0x60f) [0x7fb71bfdacf9] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb71c0b75a1] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 20: (Thread::entry_wrapper()+0x43) [0x7fb71be64cdd] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb71be64cf9] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb71b758c02] 2024-07-27T06:50:57.707 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb71b7ddc40] 2024-07-27T06:50:57.708 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:50:57.708 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71556]: 2024-07-27T06:50:57.708 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:57 smithi047 podman[71595]: 2024-07-27 06:50:57.489962084 +0000 UTC m=+0.019772584 container died 83dfcdb1a72278c7dd2444cb1197f0254fb79092015d0df6c6aeb0e9d311a5b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-07-27T06:50:57.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:57.743+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:50:57.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:57.743+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-27T06:50:57.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:57.743+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:50:57.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:57.743+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:50:57.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:50:57.743+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-27T06:50:58.117 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:rejoin 2024-07-27T06:50:58.117 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:50:58.117 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-mon[36835]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:50:58.118 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:50:58.118 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:50:58.118 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:active 2024-07-27T06:50:58.118 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:57 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:50:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:57 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:rejoin 2024-07-27T06:50:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:57 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:50:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:57 smithi029 ceph-mon[32610]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:50:58.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:57 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:50:58.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:57 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:50:58.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:57 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:active 2024-07-27T06:50:58.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:57 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:50:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:57 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:rejoin 2024-07-27T06:50:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:57 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:50:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:57 smithi077 ceph-mon[37153]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:50:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:57 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:50:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:57 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:50:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:57 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/3034977063,v1:172.21.15.47:6837/3034977063] up:active 2024-07-27T06:50:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:57 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:50:58.383 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:58 smithi047 podman[71595]: 2024-07-27 06:50:58.116802906 +0000 UTC m=+0.646613436 container remove 83dfcdb1a72278c7dd2444cb1197f0254fb79092015d0df6c6aeb0e9d311a5b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True) 2024-07-27T06:50:58.383 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:58 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:50:58.696 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:58 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-27T06:50:58.696 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:50:58 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Consumed 1.119s CPU time. 2024-07-27T06:50:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:50:59 smithi047 ceph-mon[36835]: pgmap v521: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:50:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:50:59 smithi029 ceph-mon[32610]: pgmap v521: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:50:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:50:59 smithi077 ceph-mon[37153]: pgmap v521: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:51:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:01 smithi047 ceph-mon[36835]: pgmap v522: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:51:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:01 smithi029 ceph-mon[32610]: pgmap v522: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:51:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:01 smithi077 ceph-mon[37153]: pgmap v522: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:51:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:03 smithi047 ceph-mon[36835]: pgmap v523: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:51:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:03 smithi029 ceph-mon[32610]: pgmap v523: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:51:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:03 smithi077 ceph-mon[37153]: pgmap v523: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:51:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:05 smithi047 ceph-mon[36835]: pgmap v524: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 202 B/s wr, 20 op/s 2024-07-27T06:51:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:05 smithi029 ceph-mon[32610]: pgmap v524: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 202 B/s wr, 20 op/s 2024-07-27T06:51:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:05 smithi077 ceph-mon[37153]: pgmap v524: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 202 B/s wr, 20 op/s 2024-07-27T06:51:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:07 smithi047 ceph-mon[36835]: pgmap v525: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:51:07.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:07 smithi029 ceph-mon[32610]: pgmap v525: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:51:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:07 smithi077 ceph-mon[37153]: pgmap v525: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:51:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:08.118+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:51:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:08.119+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-27T06:51:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:08.119+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:51:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:08.119+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:51:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:08.119+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:51:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:08.128+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:51:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:08.128+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:51:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:08.128+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:51:08.710 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:08 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 3. 2024-07-27T06:51:08.710 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:08 smithi047 systemd[1]: Stopped Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:51:08.710 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:08 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Consumed 1.119s CPU time. 2024-07-27T06:51:08.710 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:08 smithi047 systemd[1]: Starting Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:51:08.710 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:08 smithi047 podman[71688]: 2024-07-27 06:51:08.591414622 +0000 UTC m=+0.018706555 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:51:08.710 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:08 smithi047 podman[71688]: 2024-07-27 06:51:08.709577799 +0000 UTC m=+0.136869728 container create 79c033ee36962d005d570f0bbb01340c8348bb601c2c86f72dfca18571366282 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-07-27T06:51:09.429 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:08 smithi047 podman[71688]: 2024-07-27 06:51:08.96239278 +0000 UTC m=+0.389684728 container init 79c033ee36962d005d570f0bbb01340c8348bb601c2c86f72dfca18571366282 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-27T06:51:09.429 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:08 smithi047 podman[71688]: 2024-07-27 06:51:08.965572075 +0000 UTC m=+0.392864011 container start 79c033ee36962d005d570f0bbb01340c8348bb601c2c86f72dfca18571366282 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-27T06:51:09.429 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:08 smithi047 ceph-mds[71707]: -- 172.21.15.47:0/849315116 <== mon.2 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557f738e7c20 con 0x557f77a89180 2024-07-27T06:51:09.429 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:09 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: starting mds.h at 2024-07-27T06:51:09.429 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:09 smithi047 bash[71688]: 79c033ee36962d005d570f0bbb01340c8348bb601c2c86f72dfca18571366282 2024-07-27T06:51:09.429 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:09 smithi047 systemd[1]: Started Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:51:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:09 smithi047 ceph-mon[36835]: pgmap v526: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:51:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:09 smithi047 ceph-mon[36835]: daemon mds.h restarted 2024-07-27T06:51:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:09 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:51:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:09 smithi047 ceph-mon[36835]: osdmap e89: 12 total, 12 up, 12 in 2024-07-27T06:51:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:09 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:boot 2024-07-27T06:51:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:09 smithi047 ceph-mon[36835]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:51:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:09 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:51:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:51:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:09 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:51:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:09 smithi029 ceph-mon[32610]: pgmap v526: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:51:09.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:09 smithi029 ceph-mon[32610]: daemon mds.h restarted 2024-07-27T06:51:09.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:09 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:51:09.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:09 smithi029 ceph-mon[32610]: osdmap e89: 12 total, 12 up, 12 in 2024-07-27T06:51:09.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:09 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:boot 2024-07-27T06:51:09.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:09 smithi029 ceph-mon[32610]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:51:09.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:09 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:51:09.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:51:09.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:09 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:51:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:09 smithi077 ceph-mon[37153]: pgmap v526: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:51:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:09 smithi077 ceph-mon[37153]: daemon mds.h restarted 2024-07-27T06:51:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:09 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:51:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:09 smithi077 ceph-mon[37153]: osdmap e89: 12 total, 12 up, 12 in 2024-07-27T06:51:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:09 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:boot 2024-07-27T06:51:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:09 smithi077 ceph-mon[37153]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:51:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:09 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:51:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:51:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:09 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:51:10.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:10.556+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:51:10.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:10.556+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-27T06:51:10.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:10.556+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:51:10.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:10.556+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:51:10.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:10.556+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:51:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:11 smithi029 ceph-mon[32610]: pgmap v528: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:51:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:11 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:resolve 2024-07-27T06:51:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:11 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:51:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:11 smithi077 ceph-mon[37153]: pgmap v528: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:51:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:11 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:resolve 2024-07-27T06:51:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:11 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:51:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:11 smithi047 ceph-mon[36835]: pgmap v528: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:51:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:11 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:resolve 2024-07-27T06:51:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:11 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:51:12.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:12 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00099998 2024-07-27T06:51:12.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:12 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:reconnect 2024-07-27T06:51:12.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:12 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:51:12.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:12 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00399992 2024-07-27T06:51:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:12 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00099998 2024-07-27T06:51:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:12 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:reconnect 2024-07-27T06:51:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:12 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:51:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:12 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00399992 2024-07-27T06:51:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:12 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00099998 2024-07-27T06:51:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:12 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:reconnect 2024-07-27T06:51:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:12 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:51:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:12 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00399992 2024-07-27T06:51:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:13 smithi029 ceph-mon[32610]: pgmap v529: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-07-27T06:51:13.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:13 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:rejoin 2024-07-27T06:51:13.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:13 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:51:13.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:13 smithi029 ceph-mon[32610]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:51:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-mon[37153]: pgmap v529: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-07-27T06:51:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:rejoin 2024-07-27T06:51:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:51:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-mon[37153]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:51:13.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:51:13.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: *** Caught signal (Aborted) ** 2024-07-27T06:51:13.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: in thread 7f5f7551f640 thread_name:ms_dispatch 2024-07-27T06:51:13.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:51:13.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5f7a8f66f0] 2024-07-27T06:51:13.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5f7a94394c] 2024-07-27T06:51:13.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 3: raise() 2024-07-27T06:51:13.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 4: abort() 2024-07-27T06:51:13.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5f7a8e071b] 2024-07-27T06:51:13.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 6: /lib64/libc.so.6(+0x37386) [0x7f5f7a8ef386] 2024-07-27T06:51:13.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5627ea43a6b9] 2024-07-27T06:51:13.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5627ea5f880c] 2024-07-27T06:51:13.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5627ea546cde] 2024-07-27T06:51:13.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5627ea508763] 2024-07-27T06:51:13.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5627ea509a1b] 2024-07-27T06:51:13.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5627ea4ed752] 2024-07-27T06:51:13.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5627ea43ba56] 2024-07-27T06:51:13.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5627ea43f637] 2024-07-27T06:51:13.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5627ea43ff5e] 2024-07-27T06:51:13.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5627ea42303a] 2024-07-27T06:51:13.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5f7b1c3407] 2024-07-27T06:51:13.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 18: (DispatchQueue::entry()+0x60f) [0x7f5f7b1c3cf9] 2024-07-27T06:51:13.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5f7b2a05a1] 2024-07-27T06:51:13.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 20: (Thread::entry_wrapper()+0x43) [0x7f5f7b04dcdd] 2024-07-27T06:51:13.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5f7b04dcf9] 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5f7a941c02] 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5f7a9c6c40] 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 2024-07-27T06:51:13.475+0000 7f5f7551f640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: in thread 7f5f7551f640 thread_name:ms_dispatch 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5f7a8f66f0] 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5f7a94394c] 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 3: raise() 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 4: abort() 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5f7a8e071b] 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 6: /lib64/libc.so.6(+0x37386) [0x7f5f7a8ef386] 2024-07-27T06:51:13.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5627ea43a6b9] 2024-07-27T06:51:13.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5627ea5f880c] 2024-07-27T06:51:13.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5627ea546cde] 2024-07-27T06:51:13.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5627ea508763] 2024-07-27T06:51:13.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5627ea509a1b] 2024-07-27T06:51:13.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5627ea4ed752] 2024-07-27T06:51:13.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5627ea43ba56] 2024-07-27T06:51:13.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5627ea43f637] 2024-07-27T06:51:13.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5627ea43ff5e] 2024-07-27T06:51:13.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5627ea42303a] 2024-07-27T06:51:13.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5f7b1c3407] 2024-07-27T06:51:13.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 18: (DispatchQueue::entry()+0x60f) [0x7f5f7b1c3cf9] 2024-07-27T06:51:13.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5f7b2a05a1] 2024-07-27T06:51:13.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 20: (Thread::entry_wrapper()+0x43) [0x7f5f7b04dcdd] 2024-07-27T06:51:13.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5f7b04dcf9] 2024-07-27T06:51:13.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5f7a941c02] 2024-07-27T06:51:13.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5f7a9c6c40] 2024-07-27T06:51:13.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:51:13.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 2024-07-27T06:51:13.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 0> 2024-07-27T06:51:13.475+0000 7f5f7551f640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: in thread 7f5f7551f640 thread_name:ms_dispatch 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5f7a8f66f0] 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5f7a94394c] 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 3: raise() 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 4: abort() 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5f7a8e071b] 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 6: /lib64/libc.so.6(+0x37386) [0x7f5f7a8ef386] 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5627ea43a6b9] 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5627ea5f880c] 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5627ea546cde] 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5627ea508763] 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5627ea509a1b] 2024-07-27T06:51:13.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5627ea4ed752] 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5627ea43ba56] 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5627ea43f637] 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5627ea43ff5e] 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5627ea42303a] 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5f7b1c3407] 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 18: (DispatchQueue::entry()+0x60f) [0x7f5f7b1c3cf9] 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5f7b2a05a1] 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 20: (Thread::entry_wrapper()+0x43) [0x7f5f7b04dcdd] 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5f7b04dcf9] 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5f7a941c02] 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5f7a9c6c40] 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 2024-07-27T06:51:13.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 0> 2024-07-27T06:51:13.475+0000 7f5f7551f640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: in thread 7f5f7551f640 thread_name:ms_dispatch 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5f7a8f66f0] 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5f7a94394c] 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 3: raise() 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 4: abort() 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5f7a8e071b] 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 6: /lib64/libc.so.6(+0x37386) [0x7f5f7a8ef386] 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5627ea43a6b9] 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5627ea5f880c] 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5627ea546cde] 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5627ea508763] 2024-07-27T06:51:13.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5627ea509a1b] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5627ea4ed752] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5627ea43ba56] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5627ea43f637] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5627ea43ff5e] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5627ea42303a] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5f7b1c3407] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 18: (DispatchQueue::entry()+0x60f) [0x7f5f7b1c3cf9] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5f7b2a05a1] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 20: (Thread::entry_wrapper()+0x43) [0x7f5f7b04dcdd] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5f7b04dcf9] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5f7a941c02] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5f7a9c6c40] 2024-07-27T06:51:13.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:51:13.867 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72330]: 2024-07-27T06:51:13.867 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:13 smithi077 podman[72369]: 2024-07-27 06:51:13.535074535 +0000 UTC m=+0.018760636 container died 888e4a119d6ce17133e621381c8fb21e67c0e972363e6d114f3a11d5afdf82dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724) 2024-07-27T06:51:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:13 smithi047 ceph-mon[36835]: pgmap v529: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-07-27T06:51:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:13 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:rejoin 2024-07-27T06:51:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:13 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:51:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:13 smithi047 ceph-mon[36835]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:51:14.463 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:14 smithi077 podman[72369]: 2024-07-27 06:51:14.162906482 +0000 UTC m=+0.646592573 container remove 888e4a119d6ce17133e621381c8fb21e67c0e972363e6d114f3a11d5afdf82dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-07-27T06:51:14.463 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:14 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:51:14.463 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:14 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-27T06:51:14.463 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:14 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Consumed 1.067s CPU time. 2024-07-27T06:51:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:14 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:51:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:14 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:51:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:14 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:active 2024-07-27T06:51:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:14 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:51:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:14 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:51:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:14 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:51:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:14 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:51:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:14 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:active 2024-07-27T06:51:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:14 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:51:14.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:14 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:51:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:14 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:51:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:14 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:51:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:14 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/1858446246,v1:172.21.15.77:6839/1858446246] up:active 2024-07-27T06:51:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:14 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:51:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:14 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:51:15.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:15 smithi077 ceph-mon[37153]: pgmap v530: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:51:15.816 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:15 smithi029 ceph-mon[32610]: pgmap v530: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:51:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:15 smithi047 ceph-mon[36835]: pgmap v530: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:51:17.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:17 smithi029 ceph-mon[32610]: pgmap v531: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 op/s 2024-07-27T06:51:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:17 smithi077 ceph-mon[37153]: pgmap v531: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 op/s 2024-07-27T06:51:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:17 smithi047 ceph-mon[36835]: pgmap v531: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 op/s 2024-07-27T06:51:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:19 smithi077 ceph-mon[37153]: pgmap v532: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:51:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:19 smithi047 ceph-mon[36835]: pgmap v532: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:51:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:19 smithi029 ceph-mon[32610]: pgmap v532: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:51:21.831 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:21 smithi029 ceph-mon[32610]: pgmap v533: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 190 B/s wr, 19 op/s 2024-07-27T06:51:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:21 smithi077 ceph-mon[37153]: pgmap v533: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 190 B/s wr, 19 op/s 2024-07-27T06:51:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:21 smithi047 ceph-mon[36835]: pgmap v533: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 190 B/s wr, 19 op/s 2024-07-27T06:51:23.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:23.122+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:51:23.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:23.122+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:51:23.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:23.122+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:51:23.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:23.143+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:51:23.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:23.144+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-27T06:51:23.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:23.144+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:51:23.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:23.144+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:51:23.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:23.144+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:51:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:23 smithi077 ceph-mon[37153]: pgmap v534: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:51:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:23 smithi047 ceph-mon[36835]: pgmap v534: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:51:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:23 smithi029 ceph-mon[32610]: pgmap v534: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:51:24.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:24 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 4. 2024-07-27T06:51:24.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:24 smithi077 systemd[1]: Stopped Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:51:24.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:24 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Consumed 1.067s CPU time. 2024-07-27T06:51:24.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:24 smithi077 systemd[1]: Starting Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:51:24.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:24 smithi077 podman[72463]: 2024-07-27 06:51:24.743275705 +0000 UTC m=+0.018161559 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:51:25.117 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:24 smithi077 podman[72463]: 2024-07-27 06:51:24.863988159 +0000 UTC m=+0.138874011 container create 15a575fd63a3b11f5a4e4d2d0111c576c9fd5cf63aaf0f63f7e4282c826ac92d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-27T06:51:25.486 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:25 smithi077 podman[72463]: 2024-07-27 06:51:25.117013741 +0000 UTC m=+0.391899587 container init 15a575fd63a3b11f5a4e4d2d0111c576c9fd5cf63aaf0f63f7e4282c826ac92d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-07-27T06:51:25.486 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:25 smithi077 podman[72463]: 2024-07-27 06:51:25.119572597 +0000 UTC m=+0.394458442 container start 15a575fd63a3b11f5a4e4d2d0111c576c9fd5cf63aaf0f63f7e4282c826ac92d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-07-27T06:51:25.486 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:25 smithi077 ceph-mds[72482]: -- 172.21.15.77:0/1423002630 <== mon.1 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561819833860 con 0x56181d9d5600 2024-07-27T06:51:25.487 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:25 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: starting mds.i at 2024-07-27T06:51:25.487 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:25 smithi077 bash[72463]: 15a575fd63a3b11f5a4e4d2d0111c576c9fd5cf63aaf0f63f7e4282c826ac92d 2024-07-27T06:51:25.487 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:25 smithi077 systemd[1]: Started Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:51:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:25.571+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:51:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:25.571+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-27T06:51:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:25.571+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:51:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:25.571+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:51:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:25.571+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:51:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:25 smithi077 ceph-mon[37153]: pgmap v535: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:51:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:25 smithi077 ceph-mon[37153]: daemon mds.i restarted 2024-07-27T06:51:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:25 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:51:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:25 smithi077 ceph-mon[37153]: osdmap e90: 12 total, 12 up, 12 in 2024-07-27T06:51:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:25 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:boot 2024-07-27T06:51:25.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:25 smithi077 ceph-mon[37153]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:51:25.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:25 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:51:25.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:51:25.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:25 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:51:25.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:25 smithi047 ceph-mon[36835]: pgmap v535: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:51:25.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:25 smithi047 ceph-mon[36835]: daemon mds.i restarted 2024-07-27T06:51:25.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:25 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:51:25.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:25 smithi047 ceph-mon[36835]: osdmap e90: 12 total, 12 up, 12 in 2024-07-27T06:51:25.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:25 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:boot 2024-07-27T06:51:25.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:25 smithi047 ceph-mon[36835]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:51:25.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:25 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:51:25.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:51:25.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:25 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:51:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:25 smithi029 ceph-mon[32610]: pgmap v535: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:51:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:25 smithi029 ceph-mon[32610]: daemon mds.i restarted 2024-07-27T06:51:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:25 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:51:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:25 smithi029 ceph-mon[32610]: osdmap e90: 12 total, 12 up, 12 in 2024-07-27T06:51:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:25 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:boot 2024-07-27T06:51:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:25 smithi029 ceph-mon[32610]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:51:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:25 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:51:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-27T06:51:25.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:25 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:51:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:27 smithi077 ceph-mon[37153]: pgmap v537: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-27T06:51:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:27 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:resolve 2024-07-27T06:51:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:27 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:51:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:27 smithi047 ceph-mon[36835]: pgmap v537: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-27T06:51:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:27 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:resolve 2024-07-27T06:51:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:27 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:51:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:27 smithi029 ceph-mon[32610]: pgmap v537: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-27T06:51:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:27 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:resolve 2024-07-27T06:51:27.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:27 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:51:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:28 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999979 2024-07-27T06:51:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:28 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:reconnect 2024-07-27T06:51:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:28 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:51:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:28 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00399992 2024-07-27T06:51:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:51:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:28 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999979 2024-07-27T06:51:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:28 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:reconnect 2024-07-27T06:51:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:28 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:51:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:28 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00399992 2024-07-27T06:51:28.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:51:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:28 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999979 2024-07-27T06:51:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:28 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:reconnect 2024-07-27T06:51:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:28 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:51:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:28 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00399992 2024-07-27T06:51:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:51:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:29 smithi077 ceph-mon[37153]: pgmap v538: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:51:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:29 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:rejoin 2024-07-27T06:51:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:29 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:51:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:29 smithi077 ceph-mon[37153]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:51:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:51:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:51:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:51:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:51:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:51:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:51:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:51:29.936 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-mon[36835]: pgmap v538: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:51:29.936 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:rejoin 2024-07-27T06:51:29.936 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:51:29.936 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-mon[36835]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:51:29.936 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:51:29.937 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:51:29.937 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:51:29.937 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:51:29.937 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:51:29.937 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:51:29.937 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:51:29.939 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:51:29.939 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: *** Caught signal (Aborted) ** 2024-07-27T06:51:29.939 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: in thread 7fdd9d482640 thread_name:ms_dispatch 2024-07-27T06:51:29.940 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:51:29.940 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fdda28596f0] 2024-07-27T06:51:29.940 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fdda28a694c] 2024-07-27T06:51:29.940 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 3: raise() 2024-07-27T06:51:29.940 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 4: abort() 2024-07-27T06:51:29.940 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 5: /lib64/libc.so.6(+0x2871b) [0x7fdda284371b] 2024-07-27T06:51:29.940 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 6: /lib64/libc.so.6(+0x37386) [0x7fdda2852386] 2024-07-27T06:51:29.941 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557f717276b9] 2024-07-27T06:51:29.941 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557f718e580c] 2024-07-27T06:51:29.941 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557f71833cde] 2024-07-27T06:51:29.941 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x557f717f5763] 2024-07-27T06:51:29.941 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557f717f6a1b] 2024-07-27T06:51:29.941 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557f717da752] 2024-07-27T06:51:29.942 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557f71728a56] 2024-07-27T06:51:29.942 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557f7172c637] 2024-07-27T06:51:29.942 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557f7172cf5e] 2024-07-27T06:51:29.942 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557f7171003a] 2024-07-27T06:51:29.942 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fdda3126407] 2024-07-27T06:51:29.942 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 18: (DispatchQueue::entry()+0x60f) [0x7fdda3126cf9] 2024-07-27T06:51:29.943 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fdda32035a1] 2024-07-27T06:51:29.943 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 20: (Thread::entry_wrapper()+0x43) [0x7fdda2fb0cdd] 2024-07-27T06:51:29.943 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 21: (Thread::_entry_func(void*)+0xd) [0x7fdda2fb0cf9] 2024-07-27T06:51:29.943 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 22: /lib64/libc.so.6(+0x89c02) [0x7fdda28a4c02] 2024-07-27T06:51:29.943 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fdda2929c40] 2024-07-27T06:51:29.943 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 2024-07-27T06:51:29.531+0000 7fdd9d482640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:51:29.943 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: in thread 7fdd9d482640 thread_name:ms_dispatch 2024-07-27T06:51:29.944 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 2024-07-27T06:51:29.944 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:51:29.944 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fdda28596f0] 2024-07-27T06:51:29.944 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fdda28a694c] 2024-07-27T06:51:29.944 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 3: raise() 2024-07-27T06:51:29.944 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 4: abort() 2024-07-27T06:51:29.944 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 5: /lib64/libc.so.6(+0x2871b) [0x7fdda284371b] 2024-07-27T06:51:29.945 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 6: /lib64/libc.so.6(+0x37386) [0x7fdda2852386] 2024-07-27T06:51:29.945 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557f717276b9] 2024-07-27T06:51:29.945 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557f718e580c] 2024-07-27T06:51:29.945 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557f71833cde] 2024-07-27T06:51:29.945 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x557f717f5763] 2024-07-27T06:51:29.945 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557f717f6a1b] 2024-07-27T06:51:29.945 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557f717da752] 2024-07-27T06:51:29.946 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557f71728a56] 2024-07-27T06:51:29.946 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557f7172c637] 2024-07-27T06:51:29.946 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557f7172cf5e] 2024-07-27T06:51:29.946 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557f7171003a] 2024-07-27T06:51:29.946 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fdda3126407] 2024-07-27T06:51:29.946 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 18: (DispatchQueue::entry()+0x60f) [0x7fdda3126cf9] 2024-07-27T06:51:29.947 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fdda32035a1] 2024-07-27T06:51:29.947 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 20: (Thread::entry_wrapper()+0x43) [0x7fdda2fb0cdd] 2024-07-27T06:51:29.947 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 21: (Thread::_entry_func(void*)+0xd) [0x7fdda2fb0cf9] 2024-07-27T06:51:29.947 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 22: /lib64/libc.so.6(+0x89c02) [0x7fdda28a4c02] 2024-07-27T06:51:29.947 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fdda2929c40] 2024-07-27T06:51:29.947 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:51:29.947 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 2024-07-27T06:51:29.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 0> 2024-07-27T06:51:29.531+0000 7fdd9d482640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:51:29.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: in thread 7fdd9d482640 thread_name:ms_dispatch 2024-07-27T06:51:29.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 2024-07-27T06:51:29.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:51:29.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fdda28596f0] 2024-07-27T06:51:29.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fdda28a694c] 2024-07-27T06:51:29.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 3: raise() 2024-07-27T06:51:29.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 4: abort() 2024-07-27T06:51:29.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 5: /lib64/libc.so.6(+0x2871b) [0x7fdda284371b] 2024-07-27T06:51:29.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 6: /lib64/libc.so.6(+0x37386) [0x7fdda2852386] 2024-07-27T06:51:29.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557f717276b9] 2024-07-27T06:51:29.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557f718e580c] 2024-07-27T06:51:29.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557f71833cde] 2024-07-27T06:51:29.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x557f717f5763] 2024-07-27T06:51:29.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557f717f6a1b] 2024-07-27T06:51:29.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557f717da752] 2024-07-27T06:51:29.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557f71728a56] 2024-07-27T06:51:29.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557f7172c637] 2024-07-27T06:51:29.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557f7172cf5e] 2024-07-27T06:51:29.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557f7171003a] 2024-07-27T06:51:29.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fdda3126407] 2024-07-27T06:51:29.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 18: (DispatchQueue::entry()+0x60f) [0x7fdda3126cf9] 2024-07-27T06:51:29.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fdda32035a1] 2024-07-27T06:51:29.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 20: (Thread::entry_wrapper()+0x43) [0x7fdda2fb0cdd] 2024-07-27T06:51:29.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 21: (Thread::_entry_func(void*)+0xd) [0x7fdda2fb0cf9] 2024-07-27T06:51:29.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 22: /lib64/libc.so.6(+0x89c02) [0x7fdda28a4c02] 2024-07-27T06:51:29.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fdda2929c40] 2024-07-27T06:51:29.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:51:29.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 2024-07-27T06:51:29.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 0> 2024-07-27T06:51:29.531+0000 7fdd9d482640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:51:29.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: in thread 7fdd9d482640 thread_name:ms_dispatch 2024-07-27T06:51:29.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 2024-07-27T06:51:29.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:51:29.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fdda28596f0] 2024-07-27T06:51:29.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fdda28a694c] 2024-07-27T06:51:29.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 3: raise() 2024-07-27T06:51:29.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 4: abort() 2024-07-27T06:51:29.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 5: /lib64/libc.so.6(+0x2871b) [0x7fdda284371b] 2024-07-27T06:51:29.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 6: /lib64/libc.so.6(+0x37386) [0x7fdda2852386] 2024-07-27T06:51:29.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557f717276b9] 2024-07-27T06:51:29.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557f718e580c] 2024-07-27T06:51:29.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557f71833cde] 2024-07-27T06:51:29.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x557f717f5763] 2024-07-27T06:51:29.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557f717f6a1b] 2024-07-27T06:51:29.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557f717da752] 2024-07-27T06:51:29.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557f71728a56] 2024-07-27T06:51:29.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557f7172c637] 2024-07-27T06:51:29.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557f7172cf5e] 2024-07-27T06:51:29.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557f7171003a] 2024-07-27T06:51:29.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fdda3126407] 2024-07-27T06:51:29.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 18: (DispatchQueue::entry()+0x60f) [0x7fdda3126cf9] 2024-07-27T06:51:29.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fdda32035a1] 2024-07-27T06:51:29.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 20: (Thread::entry_wrapper()+0x43) [0x7fdda2fb0cdd] 2024-07-27T06:51:29.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 21: (Thread::_entry_func(void*)+0xd) [0x7fdda2fb0cf9] 2024-07-27T06:51:29.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 22: /lib64/libc.so.6(+0x89c02) [0x7fdda28a4c02] 2024-07-27T06:51:29.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fdda2929c40] 2024-07-27T06:51:29.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:51:29.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71703]: 2024-07-27T06:51:29.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:29 smithi047 podman[71871]: 2024-07-27 06:51:29.594416627 +0000 UTC m=+0.020183976 container died 79c033ee36962d005d570f0bbb01340c8348bb601c2c86f72dfca18571366282 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS) 2024-07-27T06:51:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:29 smithi029 ceph-mon[32610]: pgmap v538: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:51:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:29 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:rejoin 2024-07-27T06:51:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:29 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:51:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:29 smithi029 ceph-mon[32610]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:51:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:51:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:51:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:51:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:51:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:51:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:51:29.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:51:30.507 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:30 smithi047 podman[71871]: 2024-07-27 06:51:30.253766592 +0000 UTC m=+0.679533929 container remove 79c033ee36962d005d570f0bbb01340c8348bb601c2c86f72dfca18571366282 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-07-27T06:51:30.507 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:30 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:51:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:30 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T06:51:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:30 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T06:51:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:30 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:51:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:30 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:51:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:30 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:active 2024-07-27T06:51:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:30 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:51:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:30 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:51:30.946 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:30 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-27T06:51:30.947 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:30 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Consumed 1.131s CPU time. 2024-07-27T06:51:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:30 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T06:51:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:30 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T06:51:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:30 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:51:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:30 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:51:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:30 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:active 2024-07-27T06:51:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:30 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:51:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:30 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:51:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:30 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T06:51:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:30 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T06:51:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:30 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:51:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:30 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:51:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:30 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/2870455685,v1:172.21.15.47:6837/2870455685] up:active 2024-07-27T06:51:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:30 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:51:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:30 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:51:31.815 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:31 smithi029 ceph-mon[32610]: pgmap v539: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:51:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:31 smithi077 ceph-mon[37153]: pgmap v539: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:51:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:31 smithi047 ceph-mon[36835]: pgmap v539: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:51:33.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:33 smithi029 ceph-mon[32610]: pgmap v540: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 102 B/s wr, 18 op/s 2024-07-27T06:51:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:33 smithi077 ceph-mon[37153]: pgmap v540: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 102 B/s wr, 18 op/s 2024-07-27T06:51:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:33 smithi047 ceph-mon[36835]: pgmap v540: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 102 B/s wr, 18 op/s 2024-07-27T06:51:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:35 smithi077 ceph-mon[37153]: pgmap v541: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:51:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:35 smithi047 ceph-mon[36835]: pgmap v541: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:51:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:35 smithi029 ceph-mon[32610]: pgmap v541: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:51:37.832 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:37 smithi029 ceph-mon[32610]: pgmap v542: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 193 B/s wr, 20 op/s 2024-07-27T06:51:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:37 smithi077 ceph-mon[37153]: pgmap v542: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 193 B/s wr, 20 op/s 2024-07-27T06:51:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:37 smithi047 ceph-mon[36835]: pgmap v542: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 193 B/s wr, 20 op/s 2024-07-27T06:51:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:38.138+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:51:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:38.138+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-07-27T06:51:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:38.138+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:51:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:38.138+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:51:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:38.138+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:51:38.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:38.159+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:51:38.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:38.159+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:51:38.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:38.159+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:51:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:39 smithi077 ceph-mon[37153]: pgmap v543: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:51:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:39 smithi047 ceph-mon[36835]: pgmap v543: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:51:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:39 smithi029 ceph-mon[32610]: pgmap v543: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:51:40.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:40.587+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:51:40.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:40.588+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-27T06:51:40.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:40.588+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:51:40.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:40.588+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:51:40.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:40.588+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:51:40.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:40 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 4. 2024-07-27T06:51:40.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:40 smithi047 systemd[1]: Stopped Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:51:40.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:40 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Consumed 1.131s CPU time. 2024-07-27T06:51:40.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:40 smithi047 systemd[1]: Starting Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:51:40.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:40 smithi047 podman[71962]: 2024-07-27 06:51:40.845672137 +0000 UTC m=+0.018697123 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:51:40.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:40 smithi047 podman[71962]: 2024-07-27 06:51:40.955990188 +0000 UTC m=+0.129015159 container create 1d4b7f16cd7d3e0572275d1c47d0696d956ebe1038f272ce12427faba996adfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-07-27T06:51:41.227 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:41 smithi047 podman[71962]: 2024-07-27 06:51:41.192767323 +0000 UTC m=+0.365792313 container init 1d4b7f16cd7d3e0572275d1c47d0696d956ebe1038f272ce12427faba996adfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-07-27T06:51:41.228 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:41 smithi047 podman[71962]: 2024-07-27 06:51:41.19541372 +0000 UTC m=+0.368438694 container start 1d4b7f16cd7d3e0572275d1c47d0696d956ebe1038f272ce12427faba996adfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724) 2024-07-27T06:51:41.500 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:41 smithi047 ceph-mds[71981]: -- 172.21.15.47:0/1729148818 <== mon.2 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b6a312f860 con 0x55b6a72d1600 2024-07-27T06:51:41.500 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:41 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: starting mds.h at 2024-07-27T06:51:41.500 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:41 smithi047 bash[71962]: 1d4b7f16cd7d3e0572275d1c47d0696d956ebe1038f272ce12427faba996adfb 2024-07-27T06:51:41.501 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:51:41 smithi047 systemd[1]: Started Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:51:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:41 smithi077 ceph-mon[37153]: pgmap v544: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:51:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:41 smithi077 ceph-mon[37153]: daemon mds.h restarted 2024-07-27T06:51:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:41 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:51:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:41 smithi077 ceph-mon[37153]: osdmap e91: 12 total, 12 up, 12 in 2024-07-27T06:51:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:41 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:boot 2024-07-27T06:51:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:41 smithi077 ceph-mon[37153]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:51:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:41 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:51:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:51:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:41 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:51:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:41 smithi047 ceph-mon[36835]: pgmap v544: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:51:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:41 smithi047 ceph-mon[36835]: daemon mds.h restarted 2024-07-27T06:51:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:41 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:51:41.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:41 smithi047 ceph-mon[36835]: osdmap e91: 12 total, 12 up, 12 in 2024-07-27T06:51:41.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:41 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:boot 2024-07-27T06:51:41.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:41 smithi047 ceph-mon[36835]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:51:41.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:41 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:51:41.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:51:41.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:41 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:51:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:41 smithi029 ceph-mon[32610]: pgmap v544: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:51:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:41 smithi029 ceph-mon[32610]: daemon mds.h restarted 2024-07-27T06:51:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:41 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:51:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:41 smithi029 ceph-mon[32610]: osdmap e91: 12 total, 12 up, 12 in 2024-07-27T06:51:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:41 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:boot 2024-07-27T06:51:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:41 smithi029 ceph-mon[32610]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-07-27T06:51:41.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:41 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-27T06:51:41.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:51:41.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:41 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-27T06:51:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:43 smithi077 ceph-mon[37153]: pgmap v546: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:51:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:43 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:resolve 2024-07-27T06:51:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:43 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:51:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:43 smithi047 ceph-mon[36835]: pgmap v546: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:51:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:43 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:resolve 2024-07-27T06:51:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:43 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:51:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:43 smithi029 ceph-mon[32610]: pgmap v546: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:51:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:43 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:resolve 2024-07-27T06:51:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:43 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-27T06:51:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:44 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00099998 2024-07-27T06:51:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:44 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:reconnect 2024-07-27T06:51:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:44 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:51:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:44 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00399992 2024-07-27T06:51:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:44 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00099998 2024-07-27T06:51:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:44 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:reconnect 2024-07-27T06:51:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:44 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:51:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:44 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00399992 2024-07-27T06:51:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:44 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00099998 2024-07-27T06:51:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:44 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:reconnect 2024-07-27T06:51:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:44 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:51:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:44 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.00399992 2024-07-27T06:51:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-mon[37153]: pgmap v547: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:51:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:rejoin 2024-07-27T06:51:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:51:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-mon[37153]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:51:45.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:51:45.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: *** Caught signal (Aborted) ** 2024-07-27T06:51:45.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: in thread 7fd8c1c69640 thread_name:ms_dispatch 2024-07-27T06:51:45.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:51:45.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd8c70406f0] 2024-07-27T06:51:45.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd8c708d94c] 2024-07-27T06:51:45.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 3: raise() 2024-07-27T06:51:45.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 4: abort() 2024-07-27T06:51:45.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd8c702a71b] 2024-07-27T06:51:45.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 6: /lib64/libc.so.6(+0x37386) [0x7fd8c7039386] 2024-07-27T06:51:45.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56181736d6b9] 2024-07-27T06:51:45.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56181752b80c] 2024-07-27T06:51:45.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x561817479cde] 2024-07-27T06:51:45.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56181743b763] 2024-07-27T06:51:45.858 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56181743ca1b] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561817420752] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56181736ea56] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561817372637] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561817372f5e] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56181735603a] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd8c790d407] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 18: (DispatchQueue::entry()+0x60f) [0x7fd8c790dcf9] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd8c79ea5a1] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 20: (Thread::entry_wrapper()+0x43) [0x7fd8c7797cdd] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd8c7797cf9] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd8c708bc02] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd8c7110c40] 2024-07-27T06:51:45.859 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 2024-07-27T06:51:45.583+0000 7fd8c1c69640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:51:45.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: in thread 7fd8c1c69640 thread_name:ms_dispatch 2024-07-27T06:51:45.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 2024-07-27T06:51:45.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:51:45.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd8c70406f0] 2024-07-27T06:51:45.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd8c708d94c] 2024-07-27T06:51:45.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 3: raise() 2024-07-27T06:51:45.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 4: abort() 2024-07-27T06:51:45.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd8c702a71b] 2024-07-27T06:51:45.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 6: /lib64/libc.so.6(+0x37386) [0x7fd8c7039386] 2024-07-27T06:51:45.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56181736d6b9] 2024-07-27T06:51:45.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56181752b80c] 2024-07-27T06:51:45.860 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x561817479cde] 2024-07-27T06:51:45.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56181743b763] 2024-07-27T06:51:45.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56181743ca1b] 2024-07-27T06:51:45.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561817420752] 2024-07-27T06:51:45.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56181736ea56] 2024-07-27T06:51:45.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561817372637] 2024-07-27T06:51:45.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561817372f5e] 2024-07-27T06:51:45.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56181735603a] 2024-07-27T06:51:45.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd8c790d407] 2024-07-27T06:51:45.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 18: (DispatchQueue::entry()+0x60f) [0x7fd8c790dcf9] 2024-07-27T06:51:45.861 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd8c79ea5a1] 2024-07-27T06:51:45.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 20: (Thread::entry_wrapper()+0x43) [0x7fd8c7797cdd] 2024-07-27T06:51:45.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd8c7797cf9] 2024-07-27T06:51:45.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd8c708bc02] 2024-07-27T06:51:45.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd8c7110c40] 2024-07-27T06:51:45.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:51:45.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 2024-07-27T06:51:45.862 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 0> 2024-07-27T06:51:45.583+0000 7fd8c1c69640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: in thread 7fd8c1c69640 thread_name:ms_dispatch 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd8c70406f0] 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd8c708d94c] 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 3: raise() 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 4: abort() 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd8c702a71b] 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 6: /lib64/libc.so.6(+0x37386) [0x7fd8c7039386] 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56181736d6b9] 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56181752b80c] 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x561817479cde] 2024-07-27T06:51:45.863 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56181743b763] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56181743ca1b] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561817420752] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56181736ea56] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561817372637] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561817372f5e] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56181735603a] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd8c790d407] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 18: (DispatchQueue::entry()+0x60f) [0x7fd8c790dcf9] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd8c79ea5a1] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 20: (Thread::entry_wrapper()+0x43) [0x7fd8c7797cdd] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd8c7797cf9] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd8c708bc02] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd8c7110c40] 2024-07-27T06:51:45.864 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:51:45.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 2024-07-27T06:51:45.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 0> 2024-07-27T06:51:45.583+0000 7fd8c1c69640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:51:45.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: in thread 7fd8c1c69640 thread_name:ms_dispatch 2024-07-27T06:51:45.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 2024-07-27T06:51:45.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:51:45.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd8c70406f0] 2024-07-27T06:51:45.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd8c708d94c] 2024-07-27T06:51:45.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 3: raise() 2024-07-27T06:51:45.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 4: abort() 2024-07-27T06:51:45.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd8c702a71b] 2024-07-27T06:51:45.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 6: /lib64/libc.so.6(+0x37386) [0x7fd8c7039386] 2024-07-27T06:51:45.865 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56181736d6b9] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56181752b80c] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x561817479cde] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56181743b763] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56181743ca1b] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561817420752] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56181736ea56] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561817372637] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561817372f5e] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56181735603a] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd8c790d407] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 18: (DispatchQueue::entry()+0x60f) [0x7fd8c790dcf9] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd8c79ea5a1] 2024-07-27T06:51:45.866 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 20: (Thread::entry_wrapper()+0x43) [0x7fd8c7797cdd] 2024-07-27T06:51:45.867 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd8c7797cf9] 2024-07-27T06:51:45.867 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd8c708bc02] 2024-07-27T06:51:45.867 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd8c7110c40] 2024-07-27T06:51:45.867 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:51:45.867 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i[72478]: 2024-07-27T06:51:45.867 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:45 smithi077 podman[72646]: 2024-07-27 06:51:45.639611713 +0000 UTC m=+0.017672965 container died 15a575fd63a3b11f5a4e4d2d0111c576c9fd5cf63aaf0f63f7e4282c826ac92d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-07-27T06:51:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:45 smithi047 ceph-mon[36835]: pgmap v547: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:51:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:45 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:rejoin 2024-07-27T06:51:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:45 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:51:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:45 smithi047 ceph-mon[36835]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:51:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:45 smithi029 ceph-mon[32610]: pgmap v547: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:51:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:45 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:rejoin 2024-07-27T06:51:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:45 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:51:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:45 smithi029 ceph-mon[32610]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-07-27T06:51:46.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:46 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:51:46.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:46 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:51:46.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:46 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:active 2024-07-27T06:51:46.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:46 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:51:46.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:46 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:51:46.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:46 smithi029 ceph-mon[32610]: pgmap v548: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 op/s 2024-07-27T06:51:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:46 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:51:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:46 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:51:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:46 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:active 2024-07-27T06:51:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:46 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:51:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:46 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:51:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:46 smithi077 ceph-mon[37153]: pgmap v548: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 op/s 2024-07-27T06:51:46.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:46 smithi077 podman[72646]: 2024-07-27 06:51:46.364243868 +0000 UTC m=+0.742305114 container remove 15a575fd63a3b11f5a4e4d2d0111c576c9fd5cf63aaf0f63f7e4282c826ac92d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-i, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-07-27T06:51:46.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:46 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:51:46.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:46 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-27T06:51:46.857 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:46 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Consumed 1.078s CPU time. 2024-07-27T06:51:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:46 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:51:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:46 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:51:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:46 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6838/412097226,v1:172.21.15.77:6839/412097226] up:active 2024-07-27T06:51:46.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:46 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-27T06:51:46.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:46 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:51:46.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:46 smithi047 ceph-mon[36835]: pgmap v548: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 op/s 2024-07-27T06:51:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:48 smithi077 ceph-mon[37153]: pgmap v549: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:51:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:48 smithi047 ceph-mon[36835]: pgmap v549: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:51:49.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:48 smithi029 ceph-mon[32610]: pgmap v549: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:51:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:50 smithi077 ceph-mon[37153]: pgmap v550: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:51:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:50 smithi047 ceph-mon[36835]: pgmap v550: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:51:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:50 smithi029 ceph-mon[32610]: pgmap v550: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:51:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:52 smithi077 ceph-mon[37153]: pgmap v551: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 194 B/s wr, 19 op/s 2024-07-27T06:51:53.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:53.154+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:51:53.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:53.154+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:51:53.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:51:53.154+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:51:53.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:53.165+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:51:53.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:52 smithi029 ceph-mon[32610]: pgmap v551: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 194 B/s wr, 19 op/s 2024-07-27T06:51:53.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:53.165+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-27T06:51:53.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:53.165+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:51:53.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:53.165+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:51:53.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:53.165+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:51:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:52 smithi047 ceph-mon[36835]: pgmap v551: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 194 B/s wr, 19 op/s 2024-07-27T06:51:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:54 smithi077 ceph-mon[37153]: pgmap v552: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:51:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:54 smithi047 ceph-mon[36835]: pgmap v552: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:51:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:54 smithi029 ceph-mon[32610]: pgmap v552: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:51:55.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:55.604+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/2224015949 2024-07-27T06:51:55.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:55.604+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:51:55.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:51:55.604+0000 7f2c677fe640 1 client.16428 ms_handle_refused on v2:172.21.15.77:6838/2224015949 2024-07-27T06:51:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:56 smithi077 ceph-mon[37153]: pgmap v553: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:51:57.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:56 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 5. 2024-07-27T06:51:57.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:56 smithi077 systemd[1]: Stopped Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:51:57.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:56 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Consumed 1.078s CPU time. 2024-07-27T06:51:57.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:56 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Start request repeated too quickly. 2024-07-27T06:51:57.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:56 smithi077 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-27T06:51:57.107 INFO:journalctl@ceph.mds.i.smithi077.stdout:Jul 27 06:51:56 smithi077 systemd[1]: Failed to start Ceph mds.i for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:51:57.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:56 smithi047 ceph-mon[36835]: pgmap v553: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:51:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:56 smithi029 ceph-mon[32610]: pgmap v553: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:51:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:51:58 smithi077 ceph-mon[37153]: pgmap v554: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:51:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:51:58 smithi047 ceph-mon[36835]: pgmap v554: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:51:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:51:58 smithi029 ceph-mon[32610]: pgmap v554: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:52:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:00 smithi077 ceph-mon[37153]: pgmap v555: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:52:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:00 smithi047 ceph-mon[36835]: pgmap v555: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:52:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:00 smithi029 ceph-mon[32610]: pgmap v555: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:52:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:02 smithi077 ceph-mon[37153]: pgmap v556: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 11 op/s 2024-07-27T06:52:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:02 smithi077 ceph-mon[37153]: Replacing daemon mds.i as rank 0 with standby daemon mds.h 2024-07-27T06:52:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:02 smithi077 ceph-mon[37153]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-07-27T06:52:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:02 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:52:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:02 smithi077 ceph-mon[37153]: osdmap e92: 12 total, 12 up, 12 in 2024-07-27T06:52:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:02 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-27T06:52:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:02 smithi047 ceph-mon[36835]: pgmap v556: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 11 op/s 2024-07-27T06:52:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:02 smithi047 ceph-mon[36835]: Replacing daemon mds.i as rank 0 with standby daemon mds.h 2024-07-27T06:52:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:02 smithi047 ceph-mon[36835]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-07-27T06:52:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:02 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:52:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:02 smithi047 ceph-mon[36835]: osdmap e92: 12 total, 12 up, 12 in 2024-07-27T06:52:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:02 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-27T06:52:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:02 smithi029 ceph-mon[32610]: pgmap v556: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 11 op/s 2024-07-27T06:52:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:02 smithi029 ceph-mon[32610]: Replacing daemon mds.i as rank 0 with standby daemon mds.h 2024-07-27T06:52:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:02 smithi029 ceph-mon[32610]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-07-27T06:52:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:02 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:52:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:02 smithi029 ceph-mon[32610]: osdmap e92: 12 total, 12 up, 12 in 2024-07-27T06:52:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:02 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-27T06:52:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:04 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:resolve 2024-07-27T06:52:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:04 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-27T06:52:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:04 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:resolve 2024-07-27T06:52:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:04 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-27T06:52:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:04 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:resolve 2024-07-27T06:52:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:04 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-27T06:52:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:05 smithi047 ceph-mon[36835]: pgmap v558: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T06:52:05.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:05 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:52:05.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:05 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999979 2024-07-27T06:52:05.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:05 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:reconnect 2024-07-27T06:52:05.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:05 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:52:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:05 smithi029 ceph-mon[32610]: pgmap v558: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T06:52:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:05 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:52:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:05 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999979 2024-07-27T06:52:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:05 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:reconnect 2024-07-27T06:52:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:05 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:52:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:05 smithi077 ceph-mon[37153]: pgmap v558: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T06:52:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:05 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:52:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:05 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999979 2024-07-27T06:52:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:05 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:reconnect 2024-07-27T06:52:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:05 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:52:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:06 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:rejoin 2024-07-27T06:52:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:06 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:52:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:06 smithi029 ceph-mon[32610]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:52:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:06 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:rejoin 2024-07-27T06:52:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:06 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:52:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:06 smithi077 ceph-mon[37153]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:52:06.881 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:rejoin 2024-07-27T06:52:06.881 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:52:06.881 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-mon[36835]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:52:06.881 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:52:06.881 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: *** Caught signal (Aborted) ** 2024-07-27T06:52:06.882 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: in thread 7f522457c640 thread_name:ms_dispatch 2024-07-27T06:52:06.882 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:52:06.882 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f52299536f0] 2024-07-27T06:52:06.882 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f52299a094c] 2024-07-27T06:52:06.882 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 3: raise() 2024-07-27T06:52:06.882 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 4: abort() 2024-07-27T06:52:06.882 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 5: /lib64/libc.so.6(+0x2871b) [0x7f522993d71b] 2024-07-27T06:52:06.882 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 6: /lib64/libc.so.6(+0x37386) [0x7f522994c386] 2024-07-27T06:52:06.882 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b6a10886b9] 2024-07-27T06:52:06.883 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b6a124680c] 2024-07-27T06:52:06.883 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b6a1194cde] 2024-07-27T06:52:06.883 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b6a1156763] 2024-07-27T06:52:06.883 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b6a1157a1b] 2024-07-27T06:52:06.883 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b6a113b752] 2024-07-27T06:52:06.883 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b6a1089a56] 2024-07-27T06:52:06.883 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b6a108d637] 2024-07-27T06:52:06.883 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b6a108df5e] 2024-07-27T06:52:06.883 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b6a107103a] 2024-07-27T06:52:06.883 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f522a220407] 2024-07-27T06:52:06.883 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 18: (DispatchQueue::entry()+0x60f) [0x7f522a220cf9] 2024-07-27T06:52:06.883 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f522a2fd5a1] 2024-07-27T06:52:06.884 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 20: (Thread::entry_wrapper()+0x43) [0x7f522a0aacdd] 2024-07-27T06:52:06.884 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 21: (Thread::_entry_func(void*)+0xd) [0x7f522a0aacf9] 2024-07-27T06:52:06.885 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 22: /lib64/libc.so.6(+0x89c02) [0x7f522999ec02] 2024-07-27T06:52:06.885 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5229a23c40] 2024-07-27T06:52:06.885 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 2024-07-27T06:52:06.478+0000 7f522457c640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:52:06.885 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: in thread 7f522457c640 thread_name:ms_dispatch 2024-07-27T06:52:06.885 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 2024-07-27T06:52:06.885 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:52:06.885 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f52299536f0] 2024-07-27T06:52:06.885 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f52299a094c] 2024-07-27T06:52:06.885 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 3: raise() 2024-07-27T06:52:06.885 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 4: abort() 2024-07-27T06:52:06.885 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 5: /lib64/libc.so.6(+0x2871b) [0x7f522993d71b] 2024-07-27T06:52:06.885 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 6: /lib64/libc.so.6(+0x37386) [0x7f522994c386] 2024-07-27T06:52:06.886 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b6a10886b9] 2024-07-27T06:52:06.886 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b6a124680c] 2024-07-27T06:52:06.886 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b6a1194cde] 2024-07-27T06:52:06.886 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b6a1156763] 2024-07-27T06:52:06.886 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b6a1157a1b] 2024-07-27T06:52:06.886 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b6a113b752] 2024-07-27T06:52:06.886 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b6a1089a56] 2024-07-27T06:52:06.886 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b6a108d637] 2024-07-27T06:52:06.887 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b6a108df5e] 2024-07-27T06:52:06.887 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b6a107103a] 2024-07-27T06:52:06.887 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f522a220407] 2024-07-27T06:52:06.887 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 18: (DispatchQueue::entry()+0x60f) [0x7f522a220cf9] 2024-07-27T06:52:06.887 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f522a2fd5a1] 2024-07-27T06:52:06.887 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 20: (Thread::entry_wrapper()+0x43) [0x7f522a0aacdd] 2024-07-27T06:52:06.887 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 21: (Thread::_entry_func(void*)+0xd) [0x7f522a0aacf9] 2024-07-27T06:52:06.887 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 22: /lib64/libc.so.6(+0x89c02) [0x7f522999ec02] 2024-07-27T06:52:06.887 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5229a23c40] 2024-07-27T06:52:06.887 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:52:06.887 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 0> 2024-07-27T06:52:06.478+0000 7f522457c640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: in thread 7f522457c640 thread_name:ms_dispatch 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f52299536f0] 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f52299a094c] 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 3: raise() 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 4: abort() 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 5: /lib64/libc.so.6(+0x2871b) [0x7f522993d71b] 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 6: /lib64/libc.so.6(+0x37386) [0x7f522994c386] 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b6a10886b9] 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b6a124680c] 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b6a1194cde] 2024-07-27T06:52:06.888 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b6a1156763] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b6a1157a1b] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b6a113b752] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b6a1089a56] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b6a108d637] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b6a108df5e] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b6a107103a] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f522a220407] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 18: (DispatchQueue::entry()+0x60f) [0x7f522a220cf9] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f522a2fd5a1] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 20: (Thread::entry_wrapper()+0x43) [0x7f522a0aacdd] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 21: (Thread::_entry_func(void*)+0xd) [0x7f522a0aacf9] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 22: /lib64/libc.so.6(+0x89c02) [0x7f522999ec02] 2024-07-27T06:52:06.889 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5229a23c40] 2024-07-27T06:52:06.890 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:52:06.890 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 2024-07-27T06:52:06.890 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 0> 2024-07-27T06:52:06.478+0000 7f522457c640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:52:06.890 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: in thread 7f522457c640 thread_name:ms_dispatch 2024-07-27T06:52:06.890 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 2024-07-27T06:52:06.890 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:52:06.890 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f52299536f0] 2024-07-27T06:52:06.890 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f52299a094c] 2024-07-27T06:52:06.890 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 3: raise() 2024-07-27T06:52:06.890 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 4: abort() 2024-07-27T06:52:06.890 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 5: /lib64/libc.so.6(+0x2871b) [0x7f522993d71b] 2024-07-27T06:52:06.890 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 6: /lib64/libc.so.6(+0x37386) [0x7f522994c386] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b6a10886b9] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b6a124680c] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b6a1194cde] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b6a1156763] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b6a1157a1b] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b6a113b752] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b6a1089a56] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b6a108d637] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b6a108df5e] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b6a107103a] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f522a220407] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 18: (DispatchQueue::entry()+0x60f) [0x7f522a220cf9] 2024-07-27T06:52:06.891 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f522a2fd5a1] 2024-07-27T06:52:06.892 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 20: (Thread::entry_wrapper()+0x43) [0x7f522a0aacdd] 2024-07-27T06:52:06.892 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 21: (Thread::_entry_func(void*)+0xd) [0x7f522a0aacf9] 2024-07-27T06:52:06.892 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 22: /lib64/libc.so.6(+0x89c02) [0x7f522999ec02] 2024-07-27T06:52:06.892 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5229a23c40] 2024-07-27T06:52:06.892 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:52:06.892 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[71977]: 2024-07-27T06:52:06.892 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:06 smithi047 podman[72029]: 2024-07-27 06:52:06.536458912 +0000 UTC m=+0.019371781 container died 1d4b7f16cd7d3e0572275d1c47d0696d956ebe1038f272ce12427faba996adfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-07-27T06:52:07.437 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:07 smithi047 podman[72029]: 2024-07-27 06:52:07.173113977 +0000 UTC m=+0.656026848 container remove 1d4b7f16cd7d3e0572275d1c47d0696d956ebe1038f272ce12427faba996adfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-07-27T06:52:07.437 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:07 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:52:07.696 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:07 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-27T06:52:07.696 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:07 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Consumed 1.147s CPU time. 2024-07-27T06:52:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:07 smithi047 ceph-mon[36835]: pgmap v559: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-27T06:52:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:07 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:52:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:07 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:52:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:07 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:active 2024-07-27T06:52:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:07 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:52:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:07 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:52:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:07 smithi029 ceph-mon[32610]: pgmap v559: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-27T06:52:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:07 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:52:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:07 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:52:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:07 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:active 2024-07-27T06:52:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:07 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:52:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:07 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:52:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:07 smithi077 ceph-mon[37153]: pgmap v559: 97 pgs: 97 active+clean; 45 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-27T06:52:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:07 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:52:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:07 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:52:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:07 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/508610514,v1:172.21.15.47:6837/508610514] up:active 2024-07-27T06:52:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:07 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:52:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:07 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:52:08.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:08.165+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:52:08.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:08.165+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:52:08.166 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:08.165+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:52:08.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:08.172+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:52:08.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:08.172+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:52:08.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:08.172+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:52:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:09 smithi029 ceph-mon[32610]: pgmap v560: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 17 op/s 2024-07-27T06:52:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:09 smithi077 ceph-mon[37153]: pgmap v560: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 17 op/s 2024-07-27T06:52:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:09 smithi047 ceph-mon[36835]: pgmap v560: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 17 op/s 2024-07-27T06:52:10.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:10.620+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/2224015949 2024-07-27T06:52:10.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:10.620+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:52:10.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:10.620+0000 7f2c677fe640 1 client.16428 ms_handle_refused on v2:172.21.15.77:6838/2224015949 2024-07-27T06:52:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:11 smithi029 ceph-mon[32610]: pgmap v561: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 17 op/s 2024-07-27T06:52:11.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:11 smithi047 ceph-mon[36835]: pgmap v561: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 17 op/s 2024-07-27T06:52:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:11 smithi077 ceph-mon[37153]: pgmap v561: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 17 op/s 2024-07-27T06:52:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:13 smithi029 ceph-mon[32610]: pgmap v562: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:52:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:13 smithi077 ceph-mon[37153]: pgmap v562: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:52:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:13 smithi047 ceph-mon[36835]: pgmap v562: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:52:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:15 smithi029 ceph-mon[32610]: pgmap v563: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 180 B/s wr, 18 op/s 2024-07-27T06:52:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:15 smithi077 ceph-mon[37153]: pgmap v563: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 180 B/s wr, 18 op/s 2024-07-27T06:52:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:15 smithi047 ceph-mon[36835]: pgmap v563: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 180 B/s wr, 18 op/s 2024-07-27T06:52:17.708 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:17 smithi047 ceph-mon[36835]: pgmap v564: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:52:17.708 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:17 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 5. 2024-07-27T06:52:17.709 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:17 smithi047 systemd[1]: Stopped Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:52:17.709 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:17 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Consumed 1.147s CPU time. 2024-07-27T06:52:17.709 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:17 smithi047 systemd[1]: Starting Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:52:17.709 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:17 smithi047 podman[72120]: 2024-07-27 06:52:17.586431597 +0000 UTC m=+0.018317163 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:52:17.709 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:17 smithi047 podman[72120]: 2024-07-27 06:52:17.707990041 +0000 UTC m=+0.139875607 container create 81e583dddcb0013290c692390afc24d939a547c77ca42dd06fab327d35c7eee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-07-27T06:52:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:17 smithi077 ceph-mon[37153]: pgmap v564: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:52:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:17 smithi029 ceph-mon[32610]: pgmap v564: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:52:18.447 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:17 smithi047 podman[72120]: 2024-07-27 06:52:17.960833479 +0000 UTC m=+0.392719045 container init 81e583dddcb0013290c692390afc24d939a547c77ca42dd06fab327d35c7eee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-07-27T06:52:18.447 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:17 smithi047 podman[72120]: 2024-07-27 06:52:17.963672916 +0000 UTC m=+0.395558485 container start 81e583dddcb0013290c692390afc24d939a547c77ca42dd06fab327d35c7eee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-07-27T06:52:18.447 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:17 smithi047 ceph-mds[72139]: -- 172.21.15.47:0/3521884131 <== mon.2 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559eaad61860 con 0x559eaef03a80 2024-07-27T06:52:18.447 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:17 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: starting mds.h at 2024-07-27T06:52:18.447 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:18 smithi047 bash[72120]: 81e583dddcb0013290c692390afc24d939a547c77ca42dd06fab327d35c7eee4 2024-07-27T06:52:18.447 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:18 smithi047 systemd[1]: Started Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:52:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:18 smithi077 ceph-mon[37153]: daemon mds.h restarted 2024-07-27T06:52:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:18 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:52:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:18 smithi077 ceph-mon[37153]: osdmap e93: 12 total, 12 up, 12 in 2024-07-27T06:52:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:18 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:boot 2024-07-27T06:52:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:18 smithi077 ceph-mon[37153]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:52:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:18 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-27T06:52:18.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:52:18.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:18 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-27T06:52:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:18 smithi047 ceph-mon[36835]: daemon mds.h restarted 2024-07-27T06:52:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:18 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:52:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:18 smithi047 ceph-mon[36835]: osdmap e93: 12 total, 12 up, 12 in 2024-07-27T06:52:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:18 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:boot 2024-07-27T06:52:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:18 smithi047 ceph-mon[36835]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:52:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:18 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-27T06:52:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:52:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:18 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-27T06:52:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:18 smithi029 ceph-mon[32610]: daemon mds.h restarted 2024-07-27T06:52:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:18 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:52:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:18 smithi029 ceph-mon[32610]: osdmap e93: 12 total, 12 up, 12 in 2024-07-27T06:52:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:18 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:boot 2024-07-27T06:52:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:18 smithi029 ceph-mon[32610]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:52:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:18 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-27T06:52:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-27T06:52:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:18 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-27T06:52:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:19 smithi077 ceph-mon[37153]: pgmap v565: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:52:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:19 smithi047 ceph-mon[36835]: pgmap v565: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:52:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:19 smithi029 ceph-mon[32610]: pgmap v565: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:52:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:20 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:resolve 2024-07-27T06:52:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:20 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-27T06:52:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:20 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:resolve 2024-07-27T06:52:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:20 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-27T06:52:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:20 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:resolve 2024-07-27T06:52:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:20 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-27T06:52:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:21 smithi077 ceph-mon[37153]: pgmap v567: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 12 op/s 2024-07-27T06:52:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:21 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999984 2024-07-27T06:52:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:21 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:52:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:21 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:reconnect 2024-07-27T06:52:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:21 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:52:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:21 smithi047 ceph-mon[36835]: pgmap v567: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 12 op/s 2024-07-27T06:52:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:21 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999984 2024-07-27T06:52:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:21 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:52:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:21 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:reconnect 2024-07-27T06:52:21.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:21 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:52:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:21 smithi029 ceph-mon[32610]: pgmap v567: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 12 op/s 2024-07-27T06:52:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:21 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999984 2024-07-27T06:52:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:21 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:52:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:21 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:reconnect 2024-07-27T06:52:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:21 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:52:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:rejoin 2024-07-27T06:52:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:52:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-mon[32610]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:52:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:52:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:52:22.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:active 2024-07-27T06:52:22.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:52:22.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:52:22.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: *** Caught signal (Aborted) ** 2024-07-27T06:52:22.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: in thread 7f04ec90a640 thread_name:ms_dispatch 2024-07-27T06:52:22.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:52:22.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f04f1ce16f0] 2024-07-27T06:52:22.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f04f1d2e94c] 2024-07-27T06:52:22.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 3: raise() 2024-07-27T06:52:22.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 4: abort() 2024-07-27T06:52:22.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 5: /lib64/libc.so.6(+0x2871b) [0x7f04f1ccb71b] 2024-07-27T06:52:22.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 6: /lib64/libc.so.6(+0x37386) [0x7f04f1cda386] 2024-07-27T06:52:22.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55aaac15e6b9] 2024-07-27T06:52:22.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55aaac31c80c] 2024-07-27T06:52:22.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55aaac26acde] 2024-07-27T06:52:22.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55aaac22c763] 2024-07-27T06:52:22.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55aaac22da1b] 2024-07-27T06:52:22.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55aaac211752] 2024-07-27T06:52:22.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55aaac15fa56] 2024-07-27T06:52:22.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55aaac163637] 2024-07-27T06:52:22.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55aaac163f5e] 2024-07-27T06:52:22.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55aaac14703a] 2024-07-27T06:52:22.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f04f25ae407] 2024-07-27T06:52:22.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 18: (DispatchQueue::entry()+0x60f) [0x7f04f25aecf9] 2024-07-27T06:52:22.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f04f268b5a1] 2024-07-27T06:52:22.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 20: (Thread::entry_wrapper()+0x43) [0x7f04f2438cdd] 2024-07-27T06:52:22.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 21: (Thread::_entry_func(void*)+0xd) [0x7f04f2438cf9] 2024-07-27T06:52:22.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 22: /lib64/libc.so.6(+0x89c02) [0x7f04f1d2cc02] 2024-07-27T06:52:22.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f04f1db1c40] 2024-07-27T06:52:22.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 2024-07-27T06:52:22.449+0000 7f04ec90a640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:52:22.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: in thread 7f04ec90a640 thread_name:ms_dispatch 2024-07-27T06:52:22.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 2024-07-27T06:52:22.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:52:22.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f04f1ce16f0] 2024-07-27T06:52:22.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f04f1d2e94c] 2024-07-27T06:52:22.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 3: raise() 2024-07-27T06:52:22.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 4: abort() 2024-07-27T06:52:22.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 5: /lib64/libc.so.6(+0x2871b) [0x7f04f1ccb71b] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 6: /lib64/libc.so.6(+0x37386) [0x7f04f1cda386] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55aaac15e6b9] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55aaac31c80c] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55aaac26acde] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55aaac22c763] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55aaac22da1b] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55aaac211752] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55aaac15fa56] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55aaac163637] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55aaac163f5e] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55aaac14703a] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f04f25ae407] 2024-07-27T06:52:22.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 18: (DispatchQueue::entry()+0x60f) [0x7f04f25aecf9] 2024-07-27T06:52:22.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f04f268b5a1] 2024-07-27T06:52:22.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 20: (Thread::entry_wrapper()+0x43) [0x7f04f2438cdd] 2024-07-27T06:52:22.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 21: (Thread::_entry_func(void*)+0xd) [0x7f04f2438cf9] 2024-07-27T06:52:22.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 22: /lib64/libc.so.6(+0x89c02) [0x7f04f1d2cc02] 2024-07-27T06:52:22.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f04f1db1c40] 2024-07-27T06:52:22.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:52:22.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 2024-07-27T06:52:22.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 0> 2024-07-27T06:52:22.449+0000 7f04ec90a640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:52:22.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: in thread 7f04ec90a640 thread_name:ms_dispatch 2024-07-27T06:52:22.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 2024-07-27T06:52:22.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:52:22.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f04f1ce16f0] 2024-07-27T06:52:22.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f04f1d2e94c] 2024-07-27T06:52:22.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 3: raise() 2024-07-27T06:52:22.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 4: abort() 2024-07-27T06:52:22.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 5: /lib64/libc.so.6(+0x2871b) [0x7f04f1ccb71b] 2024-07-27T06:52:22.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 6: /lib64/libc.so.6(+0x37386) [0x7f04f1cda386] 2024-07-27T06:52:22.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55aaac15e6b9] 2024-07-27T06:52:22.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55aaac31c80c] 2024-07-27T06:52:22.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55aaac26acde] 2024-07-27T06:52:22.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55aaac22c763] 2024-07-27T06:52:22.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55aaac22da1b] 2024-07-27T06:52:22.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55aaac211752] 2024-07-27T06:52:22.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55aaac15fa56] 2024-07-27T06:52:22.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55aaac163637] 2024-07-27T06:52:22.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55aaac163f5e] 2024-07-27T06:52:22.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55aaac14703a] 2024-07-27T06:52:22.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f04f25ae407] 2024-07-27T06:52:22.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 18: (DispatchQueue::entry()+0x60f) [0x7f04f25aecf9] 2024-07-27T06:52:22.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f04f268b5a1] 2024-07-27T06:52:22.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 20: (Thread::entry_wrapper()+0x43) [0x7f04f2438cdd] 2024-07-27T06:52:22.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 21: (Thread::_entry_func(void*)+0xd) [0x7f04f2438cf9] 2024-07-27T06:52:22.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 22: /lib64/libc.so.6(+0x89c02) [0x7f04f1d2cc02] 2024-07-27T06:52:22.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f04f1db1c40] 2024-07-27T06:52:22.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:52:22.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 2024-07-27T06:52:22.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 0> 2024-07-27T06:52:22.449+0000 7f04ec90a640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:52:22.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: in thread 7f04ec90a640 thread_name:ms_dispatch 2024-07-27T06:52:22.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 2024-07-27T06:52:22.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:52:22.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f04f1ce16f0] 2024-07-27T06:52:22.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f04f1d2e94c] 2024-07-27T06:52:22.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 3: raise() 2024-07-27T06:52:22.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 4: abort() 2024-07-27T06:52:22.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 5: /lib64/libc.so.6(+0x2871b) [0x7f04f1ccb71b] 2024-07-27T06:52:22.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 6: /lib64/libc.so.6(+0x37386) [0x7f04f1cda386] 2024-07-27T06:52:22.723 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55aaac15e6b9] 2024-07-27T06:52:22.723 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55aaac31c80c] 2024-07-27T06:52:22.723 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55aaac26acde] 2024-07-27T06:52:22.723 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55aaac22c763] 2024-07-27T06:52:22.723 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55aaac22da1b] 2024-07-27T06:52:22.723 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55aaac211752] 2024-07-27T06:52:22.723 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55aaac15fa56] 2024-07-27T06:52:22.723 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55aaac163637] 2024-07-27T06:52:22.723 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55aaac163f5e] 2024-07-27T06:52:22.723 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55aaac14703a] 2024-07-27T06:52:22.723 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f04f25ae407] 2024-07-27T06:52:22.723 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 18: (DispatchQueue::entry()+0x60f) [0x7f04f25aecf9] 2024-07-27T06:52:22.724 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f04f268b5a1] 2024-07-27T06:52:22.724 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 20: (Thread::entry_wrapper()+0x43) [0x7f04f2438cdd] 2024-07-27T06:52:22.724 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 21: (Thread::_entry_func(void*)+0xd) [0x7f04f2438cf9] 2024-07-27T06:52:22.724 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 22: /lib64/libc.so.6(+0x89c02) [0x7f04f1d2cc02] 2024-07-27T06:52:22.724 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f04f1db1c40] 2024-07-27T06:52:22.724 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:52:22.724 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[77647]: 2024-07-27T06:52:22.724 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:22 smithi029 podman[107881]: 2024-07-27 06:52:22.524506864 +0000 UTC m=+0.023838243 container died d25557f5b03dff4cb63d83b456b53cf4d74f078b6f9eb0ea2266311e49e28069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240724, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-07-27T06:52:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:22 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:rejoin 2024-07-27T06:52:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:22 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:52:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:22 smithi077 ceph-mon[37153]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:52:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:22 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:52:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:22 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:52:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:22 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:active 2024-07-27T06:52:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:22 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:52:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:22 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:rejoin 2024-07-27T06:52:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:22 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:52:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:22 smithi047 ceph-mon[36835]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:52:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:22 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:52:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:22 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:52:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:22 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1301177290,v1:172.21.15.29:6837/1301177290] up:active 2024-07-27T06:52:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:22 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:52:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:23.175+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:52:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:23.175+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:52:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:23.175+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:52:23.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:23.184+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:52:23.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:23.185+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-27T06:52:23.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:23.185+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:52:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:23.185+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:52:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:23.185+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:52:23.462 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:23 smithi029 podman[107881]: 2024-07-27 06:52:23.280321918 +0000 UTC m=+0.779653298 container remove d25557f5b03dff4cb63d83b456b53cf4d74f078b6f9eb0ea2266311e49e28069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, ceph=True) 2024-07-27T06:52:23.463 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:23 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:52:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:23 smithi077 ceph-mon[37153]: pgmap v568: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-27T06:52:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:23 smithi047 ceph-mon[36835]: pgmap v568: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-27T06:52:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:23 smithi029 ceph-mon[32610]: pgmap v568: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-27T06:52:23.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:23 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-27T06:52:23.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:23 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Consumed 1.336s CPU time. 2024-07-27T06:52:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:25.634+0000 7f2c87fff640 1 -- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 msgr2=0x560bf35150c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/2224015949 2024-07-27T06:52:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:25.634+0000 7f2c87fff640 1 --2- 172.21.15.29:0/3317683865 >> [v2:172.21.15.77:6838/2224015949,v1:172.21.15.77:6839/2224015949] conn(0x560bf310c1c0 0x560bf35150c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:52:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:25.634+0000 7f2c677fe640 1 client.16428 ms_handle_refused on v2:172.21.15.77:6838/2224015949 2024-07-27T06:52:25.759 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:25 smithi029 ceph-mon[32610]: pgmap v569: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:52:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:25 smithi077 ceph-mon[37153]: pgmap v569: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:52:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:25 smithi047 ceph-mon[36835]: pgmap v569: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:52:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:27 smithi077 ceph-mon[37153]: pgmap v570: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 102 B/s wr, 17 op/s 2024-07-27T06:52:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:27 smithi047 ceph-mon[36835]: pgmap v570: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 102 B/s wr, 17 op/s 2024-07-27T06:52:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:27 smithi029 ceph-mon[32610]: pgmap v570: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 102 B/s wr, 17 op/s 2024-07-27T06:52:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: pgmap v571: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:52:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:52:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:52:29.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:52:29.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:52:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: pgmap v571: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:52:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:52:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:52:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:52:29.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:52:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: pgmap v571: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:52:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:52:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T06:52:29.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:52:29.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:52:29.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:52:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:30 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T06:52:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:30 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T06:52:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:30 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T06:52:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:30 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T06:52:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:30 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T06:52:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:30 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T06:52:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:30 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T06:52:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:30 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T06:52:30.858 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-07-27T06:52:30.859 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't parse response with error Expecting value: line 1 column 1 (char 0); rc: 124 (EMEDIUMTYPE); stdout: 2024-07-27T06:52:30.859 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 294, in do_quiesce response = json.loads(stdout) File "/usr/lib/python3.10/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.10/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 357, in _run self.do_quiesce(d) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 298, in do_quiesce raise RuntimeError(f"Error parsing quiesce response: {e}") RuntimeError: Error parsing quiesce response: Expecting value: line 1 column 1 (char 0) 2024-07-27T06:52:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:30 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T06:52:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:30 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T06:52:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:30 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T06:52:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:30 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T06:52:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:31 smithi077 ceph-mon[37153]: pgmap v572: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 173 B/s wr, 17 op/s 2024-07-27T06:52:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:31 smithi047 ceph-mon[36835]: pgmap v572: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 173 B/s wr, 17 op/s 2024-07-27T06:52:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:31 smithi029 ceph-mon[32610]: pgmap v572: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 173 B/s wr, 17 op/s 2024-07-27T06:52:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:33 smithi077 ceph-mon[37153]: pgmap v573: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:52:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:33 smithi047 ceph-mon[36835]: pgmap v573: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:52:33.957 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:33 smithi029 ceph-mon[32610]: pgmap v573: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:52:33.957 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:33 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-07-27T06:52:33.957 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:33 smithi029 systemd[1]: Stopped Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:52:33.957 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:33 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Consumed 1.336s CPU time. 2024-07-27T06:52:33.957 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:33 smithi029 systemd[1]: Starting Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:52:34.212 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:33 smithi029 podman[108120]: 2024-07-27 06:52:33.859028337 +0000 UTC m=+0.018593557 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:52:34.212 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:33 smithi029 podman[108120]: 2024-07-27 06:52:33.989514089 +0000 UTC m=+0.149079308 container create e66bbca00cc2ae798bde2203fba4a6ffb6776554b32eef73c91acf03475b68f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724) 2024-07-27T06:52:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:34 smithi029 ceph-mon[32610]: daemon mds.d restarted 2024-07-27T06:52:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:34 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:52:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:34 smithi029 ceph-mon[32610]: osdmap e94: 12 total, 12 up, 12 in 2024-07-27T06:52:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:34 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:boot 2024-07-27T06:52:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:34 smithi029 ceph-mon[32610]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:52:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:34 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-27T06:52:34.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:52:34.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:34 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-27T06:52:34.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:34 smithi029 podman[108120]: 2024-07-27 06:52:34.296952463 +0000 UTC m=+0.456517694 container init e66bbca00cc2ae798bde2203fba4a6ffb6776554b32eef73c91acf03475b68f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724) 2024-07-27T06:52:34.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:34 smithi029 podman[108120]: 2024-07-27 06:52:34.300101557 +0000 UTC m=+0.459666775 container start e66bbca00cc2ae798bde2203fba4a6ffb6776554b32eef73c91acf03475b68f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-07-27T06:52:34.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:34 smithi029 ceph-mds[108139]: -- 172.21.15.29:0/824967975 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d364e7bc20 con 0x55d36901d600 2024-07-27T06:52:34.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:34 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: starting mds.d at 2024-07-27T06:52:34.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:34 smithi029 bash[108120]: e66bbca00cc2ae798bde2203fba4a6ffb6776554b32eef73c91acf03475b68f5 2024-07-27T06:52:34.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:52:34 smithi029 systemd[1]: Started Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:52:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:34 smithi077 ceph-mon[37153]: daemon mds.d restarted 2024-07-27T06:52:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:34 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:52:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:34 smithi077 ceph-mon[37153]: osdmap e94: 12 total, 12 up, 12 in 2024-07-27T06:52:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:34 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:boot 2024-07-27T06:52:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:34 smithi077 ceph-mon[37153]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:52:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:34 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-27T06:52:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:52:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:34 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-27T06:52:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:34 smithi047 ceph-mon[36835]: daemon mds.d restarted 2024-07-27T06:52:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:34 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:52:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:34 smithi047 ceph-mon[36835]: osdmap e94: 12 total, 12 up, 12 in 2024-07-27T06:52:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:34 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:boot 2024-07-27T06:52:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:34 smithi047 ceph-mon[36835]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-27T06:52:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:34 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-27T06:52:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:52:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:34 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-27T06:52:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:35 smithi077 ceph-mon[37153]: pgmap v574: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:52:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:35 smithi047 ceph-mon[36835]: pgmap v574: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:52:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:35 smithi029 ceph-mon[32610]: pgmap v574: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:52:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:36 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:resolve 2024-07-27T06:52:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:36 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-27T06:52:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:36 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:resolve 2024-07-27T06:52:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:36 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-27T06:52:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:36 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:resolve 2024-07-27T06:52:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:36 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-27T06:52:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:37 smithi077 ceph-mon[37153]: pgmap v576: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 102 B/s wr, 12 op/s 2024-07-27T06:52:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:37 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:52:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:37 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999979 2024-07-27T06:52:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:37 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:reconnect 2024-07-27T06:52:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:37 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:52:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:37 smithi047 ceph-mon[36835]: pgmap v576: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 102 B/s wr, 12 op/s 2024-07-27T06:52:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:37 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:52:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:37 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999979 2024-07-27T06:52:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:37 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:reconnect 2024-07-27T06:52:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:37 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:52:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:37 smithi029 ceph-mon[32610]: pgmap v576: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 102 B/s wr, 12 op/s 2024-07-27T06:52:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:37 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:52:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:37 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999979 2024-07-27T06:52:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:37 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:reconnect 2024-07-27T06:52:37.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:37 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:52:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:38.181+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:52:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:38.182+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:52:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:38.182+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:52:38.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:38.200+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T06:52:38.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:38.201+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-27T06:52:38.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:38.201+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-27T06:52:38.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:38.201+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T06:52:38.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:38.201+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:52:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:38 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:rejoin 2024-07-27T06:52:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:38 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:52:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:38 smithi077 ceph-mon[37153]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:52:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:rejoin 2024-07-27T06:52:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:52:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-mon[36835]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:52:38.947 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:52:38.947 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: *** Caught signal (Aborted) ** 2024-07-27T06:52:38.947 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: in thread 7f7568844640 thread_name:ms_dispatch 2024-07-27T06:52:38.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:52:38.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f756dc1b6f0] 2024-07-27T06:52:38.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f756dc6894c] 2024-07-27T06:52:38.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 3: raise() 2024-07-27T06:52:38.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 4: abort() 2024-07-27T06:52:38.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 5: /lib64/libc.so.6(+0x2871b) [0x7f756dc0571b] 2024-07-27T06:52:38.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 6: /lib64/libc.so.6(+0x37386) [0x7f756dc14386] 2024-07-27T06:52:38.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559ea864a6b9] 2024-07-27T06:52:38.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559ea880880c] 2024-07-27T06:52:38.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559ea8756cde] 2024-07-27T06:52:38.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559ea8718763] 2024-07-27T06:52:38.948 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559ea8719a1b] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559ea86fd752] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559ea864ba56] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559ea864f637] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559ea864ff5e] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559ea863303a] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f756e4e8407] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 18: (DispatchQueue::entry()+0x60f) [0x7f756e4e8cf9] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f756e5c55a1] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 20: (Thread::entry_wrapper()+0x43) [0x7f756e372cdd] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 21: (Thread::_entry_func(void*)+0xd) [0x7f756e372cf9] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 22: /lib64/libc.so.6(+0x89c02) [0x7f756dc66c02] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f756dcebc40] 2024-07-27T06:52:38.949 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 2024-07-27T06:52:38.585+0000 7f7568844640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:52:38.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: in thread 7f7568844640 thread_name:ms_dispatch 2024-07-27T06:52:38.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 2024-07-27T06:52:38.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:52:38.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f756dc1b6f0] 2024-07-27T06:52:38.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f756dc6894c] 2024-07-27T06:52:38.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 3: raise() 2024-07-27T06:52:38.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 4: abort() 2024-07-27T06:52:38.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 5: /lib64/libc.so.6(+0x2871b) [0x7f756dc0571b] 2024-07-27T06:52:38.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 6: /lib64/libc.so.6(+0x37386) [0x7f756dc14386] 2024-07-27T06:52:38.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559ea864a6b9] 2024-07-27T06:52:38.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559ea880880c] 2024-07-27T06:52:38.950 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559ea8756cde] 2024-07-27T06:52:38.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559ea8718763] 2024-07-27T06:52:38.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559ea8719a1b] 2024-07-27T06:52:38.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559ea86fd752] 2024-07-27T06:52:38.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559ea864ba56] 2024-07-27T06:52:38.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559ea864f637] 2024-07-27T06:52:38.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559ea864ff5e] 2024-07-27T06:52:38.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559ea863303a] 2024-07-27T06:52:38.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f756e4e8407] 2024-07-27T06:52:38.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 18: (DispatchQueue::entry()+0x60f) [0x7f756e4e8cf9] 2024-07-27T06:52:38.951 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f756e5c55a1] 2024-07-27T06:52:38.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 20: (Thread::entry_wrapper()+0x43) [0x7f756e372cdd] 2024-07-27T06:52:38.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 21: (Thread::_entry_func(void*)+0xd) [0x7f756e372cf9] 2024-07-27T06:52:38.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 22: /lib64/libc.so.6(+0x89c02) [0x7f756dc66c02] 2024-07-27T06:52:38.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f756dcebc40] 2024-07-27T06:52:38.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:52:38.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 2024-07-27T06:52:38.952 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 0> 2024-07-27T06:52:38.585+0000 7f7568844640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: in thread 7f7568844640 thread_name:ms_dispatch 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f756dc1b6f0] 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f756dc6894c] 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 3: raise() 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 4: abort() 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 5: /lib64/libc.so.6(+0x2871b) [0x7f756dc0571b] 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 6: /lib64/libc.so.6(+0x37386) [0x7f756dc14386] 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559ea864a6b9] 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559ea880880c] 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559ea8756cde] 2024-07-27T06:52:38.953 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559ea8718763] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559ea8719a1b] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559ea86fd752] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559ea864ba56] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559ea864f637] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559ea864ff5e] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559ea863303a] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f756e4e8407] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 18: (DispatchQueue::entry()+0x60f) [0x7f756e4e8cf9] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f756e5c55a1] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 20: (Thread::entry_wrapper()+0x43) [0x7f756e372cdd] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 21: (Thread::_entry_func(void*)+0xd) [0x7f756e372cf9] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 22: /lib64/libc.so.6(+0x89c02) [0x7f756dc66c02] 2024-07-27T06:52:38.954 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f756dcebc40] 2024-07-27T06:52:38.955 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:52:38.955 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 2024-07-27T06:52:38.955 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 0> 2024-07-27T06:52:38.585+0000 7f7568844640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:52:38.955 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: in thread 7f7568844640 thread_name:ms_dispatch 2024-07-27T06:52:38.955 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 2024-07-27T06:52:38.955 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:52:38.955 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f756dc1b6f0] 2024-07-27T06:52:38.955 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f756dc6894c] 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 3: raise() 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 4: abort() 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 5: /lib64/libc.so.6(+0x2871b) [0x7f756dc0571b] 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 6: /lib64/libc.so.6(+0x37386) [0x7f756dc14386] 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559ea864a6b9] 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559ea880880c] 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559ea8756cde] 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559ea8718763] 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559ea8719a1b] 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559ea86fd752] 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559ea864ba56] 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559ea864f637] 2024-07-27T06:52:38.956 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559ea864ff5e] 2024-07-27T06:52:38.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559ea863303a] 2024-07-27T06:52:38.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f756e4e8407] 2024-07-27T06:52:38.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 18: (DispatchQueue::entry()+0x60f) [0x7f756e4e8cf9] 2024-07-27T06:52:38.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f756e5c55a1] 2024-07-27T06:52:38.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 20: (Thread::entry_wrapper()+0x43) [0x7f756e372cdd] 2024-07-27T06:52:38.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 21: (Thread::_entry_func(void*)+0xd) [0x7f756e372cf9] 2024-07-27T06:52:38.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 22: /lib64/libc.so.6(+0x89c02) [0x7f756dc66c02] 2024-07-27T06:52:38.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f756dcebc40] 2024-07-27T06:52:38.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:52:38.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h[72135]: 2024-07-27T06:52:38.957 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:38 smithi047 podman[72303]: 2024-07-27 06:52:38.645376983 +0000 UTC m=+0.020774726 container died 81e583dddcb0013290c692390afc24d939a547c77ca42dd06fab327d35c7eee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-27T06:52:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:38 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:rejoin 2024-07-27T06:52:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:38 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:52:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:38 smithi029 ceph-mon[32610]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-27T06:52:39.631 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:39 smithi047 ceph-mon[36835]: pgmap v577: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:52:39.631 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:39 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:52:39.631 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:39 smithi047 ceph-mon[36835]: Cluster is now healthy 2024-07-27T06:52:39.631 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:39 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:active 2024-07-27T06:52:39.631 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:39 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:52:39.632 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:39 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:52:39.632 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:39 smithi047 podman[72303]: 2024-07-27 06:52:39.36577296 +0000 UTC m=+0.741170693 container remove 81e583dddcb0013290c692390afc24d939a547c77ca42dd06fab327d35c7eee4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True) 2024-07-27T06:52:39.632 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:39 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:52:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:39 smithi077 ceph-mon[37153]: pgmap v577: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:52:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:39 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:52:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:39 smithi077 ceph-mon[37153]: Cluster is now healthy 2024-07-27T06:52:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:39 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:active 2024-07-27T06:52:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:39 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:52:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:39 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:52:39.946 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:39 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-27T06:52:39.946 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:39 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Consumed 1.134s CPU time. 2024-07-27T06:52:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:39 smithi029 ceph-mon[32610]: pgmap v577: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:52:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:39 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:52:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:39 smithi029 ceph-mon[32610]: Cluster is now healthy 2024-07-27T06:52:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:39 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.47:6836/1568196908,v1:172.21.15.47:6837/1568196908] up:active 2024-07-27T06:52:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:39 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-27T06:52:39.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:39 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:52:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:41 smithi077 ceph-mon[37153]: pgmap v578: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:52:41.895 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:41 smithi029 ceph-mon[32610]: pgmap v578: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:52:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:41 smithi047 ceph-mon[36835]: pgmap v578: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:52:42.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:42.653+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200034c0 con 0x7f7348108670 2024-07-27T06:52:42.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:42.653+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200034c0 con 0x7f7348108670 2024-07-27T06:52:42.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:42.657+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73180034c0 con 0x5645ed3392f0 2024-07-27T06:52:42.659 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:42.658+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f73180034c0 con 0x5645ed3392f0 2024-07-27T06:52:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:42 smithi077 ceph-mon[37153]: 1 slow requests, 1 included below; oldest blocked for > 184.354915 secs 2024-07-27T06:52:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:42 smithi077 ceph-mon[37153]: slow request 184.354914 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T06:52:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:42 smithi047 ceph-mon[36835]: 1 slow requests, 1 included below; oldest blocked for > 184.354915 secs 2024-07-27T06:52:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:42 smithi047 ceph-mon[36835]: slow request 184.354914 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T06:52:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:42 smithi029 ceph-mon[32610]: 1 slow requests, 1 included below; oldest blocked for > 184.354915 secs 2024-07-27T06:52:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:42 smithi029 ceph-mon[32610]: slow request 184.354914 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T06:52:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:43 smithi077 ceph-mon[37153]: pgmap v579: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 102 B/s wr, 19 op/s 2024-07-27T06:52:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:43 smithi077 ceph-mon[37153]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-27T06:52:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:43 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] up:active 2024-07-27T06:52:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:43 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:52:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:43 smithi047 ceph-mon[36835]: pgmap v579: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 102 B/s wr, 19 op/s 2024-07-27T06:52:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:43 smithi047 ceph-mon[36835]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-27T06:52:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:43 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] up:active 2024-07-27T06:52:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:43 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:52:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:43 smithi029 ceph-mon[32610]: pgmap v579: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 102 B/s wr, 19 op/s 2024-07-27T06:52:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:43 smithi029 ceph-mon[32610]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-27T06:52:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:43 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.77:6836/2959628937,v1:172.21.15.77:6837/2959628937] up:active 2024-07-27T06:52:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:43 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:52:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:44 smithi077 ceph-mon[37153]: pgmap v580: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:52:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:44 smithi047 ceph-mon[36835]: pgmap v580: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:52:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:44 smithi029 ceph-mon[32610]: pgmap v580: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:52:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:46 smithi077 ceph-mon[37153]: pgmap v581: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 178 B/s wr, 19 op/s 2024-07-27T06:52:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:46 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 189.354997 secs 2024-07-27T06:52:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:46 smithi047 ceph-mon[36835]: pgmap v581: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 178 B/s wr, 19 op/s 2024-07-27T06:52:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:46 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 189.354997 secs 2024-07-27T06:52:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:46 smithi029 ceph-mon[32610]: pgmap v581: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 178 B/s wr, 19 op/s 2024-07-27T06:52:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:46 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 189.354997 secs 2024-07-27T06:52:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:48 smithi077 ceph-mon[37153]: pgmap v582: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:52:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:48 smithi047 ceph-mon[36835]: pgmap v582: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:52:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:48 smithi029 ceph-mon[32610]: pgmap v582: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:52:50.196 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:49 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 6. 2024-07-27T06:52:50.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:49 smithi047 systemd[1]: Stopped Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:52:50.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:49 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Consumed 1.134s CPU time. 2024-07-27T06:52:50.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:49 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Start request repeated too quickly. 2024-07-27T06:52:50.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:49 smithi047 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-27T06:52:50.197 INFO:journalctl@ceph.mds.h.smithi047.stdout:Jul 27 06:52:49 smithi047 systemd[1]: Failed to start Ceph mds.h for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:52:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:50 smithi077 ceph-mon[37153]: pgmap v583: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:52:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:50 smithi047 ceph-mon[36835]: pgmap v583: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:52:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:50 smithi029 ceph-mon[32610]: pgmap v583: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:52:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:51 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 194.355140 secs 2024-07-27T06:52:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:51 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 194.355140 secs 2024-07-27T06:52:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:51 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 194.355140 secs 2024-07-27T06:52:53.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:52 smithi029 ceph-mon[32610]: pgmap v584: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:52:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:52 smithi047 ceph-mon[36835]: pgmap v584: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:52:53.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:53.198+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:52:53.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:53.198+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:52:53.199 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:52:53.198+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:52:53.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:53.217+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:52:53.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:53.217+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:52:53.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:53.217+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:52:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:52 smithi077 ceph-mon[37153]: pgmap v584: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:52:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:54 smithi047 ceph-mon[36835]: pgmap v585: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-27T06:52:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:54 smithi029 ceph-mon[32610]: pgmap v585: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-27T06:52:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:54 smithi077 ceph-mon[37153]: pgmap v585: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-27T06:52:57.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:56 smithi047 ceph-mon[36835]: pgmap v586: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:52:57.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:56 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 199.355262 secs 2024-07-27T06:52:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:56 smithi029 ceph-mon[32610]: pgmap v586: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:52:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:56 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 199.355262 secs 2024-07-27T06:52:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:56 smithi077 ceph-mon[37153]: pgmap v586: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-27T06:52:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:56 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 199.355262 secs 2024-07-27T06:52:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:57.666+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff240034c0 con 0x7eff50104ab0 2024-07-27T06:52:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:57.666+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff240034c0 con 0x7eff50104ab0 2024-07-27T06:52:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:57.672+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff280034c0 con 0x55a2cdf3b350 2024-07-27T06:52:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:52:57.672+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff280034c0 con 0x55a2cdf3b350 2024-07-27T06:52:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:57 smithi029 ceph-mon[32610]: Replacing daemon mds.h as rank 0 with standby daemon mds.d 2024-07-27T06:52:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:57 smithi029 ceph-mon[32610]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-27T06:52:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:57 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:52:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:57 smithi029 ceph-mon[32610]: osdmap e95: 12 total, 12 up, 12 in 2024-07-27T06:52:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:57 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:52:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:57 smithi047 ceph-mon[36835]: Replacing daemon mds.h as rank 0 with standby daemon mds.d 2024-07-27T06:52:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:57 smithi047 ceph-mon[36835]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-27T06:52:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:57 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:52:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:57 smithi047 ceph-mon[36835]: osdmap e95: 12 total, 12 up, 12 in 2024-07-27T06:52:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:57 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:52:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:57 smithi077 ceph-mon[37153]: Replacing daemon mds.h as rank 0 with standby daemon mds.d 2024-07-27T06:52:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:57 smithi077 ceph-mon[37153]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-27T06:52:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:57 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:52:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:57 smithi077 ceph-mon[37153]: osdmap e95: 12 total, 12 up, 12 in 2024-07-27T06:52:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:57 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:52:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:59 smithi029 ceph-mon[32610]: pgmap v588: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T06:52:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:59 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:resolve 2024-07-27T06:52:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:52:59 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:52:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:59 smithi077 ceph-mon[37153]: pgmap v588: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T06:52:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:59 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:resolve 2024-07-27T06:52:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:52:59 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:52:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:59 smithi047 ceph-mon[36835]: pgmap v588: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T06:52:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:59 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:resolve 2024-07-27T06:52:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:52:59 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:53:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:00 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999984 2024-07-27T06:53:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:00 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:53:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:00 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:reconnect 2024-07-27T06:53:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:00 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:53:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:00 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:rejoin 2024-07-27T06:53:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:00 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:53:00.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:00 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999984 2024-07-27T06:53:00.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:00 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:53:00.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:00 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:reconnect 2024-07-27T06:53:00.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:00 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:53:00.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:00 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:rejoin 2024-07-27T06:53:00.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:00 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:53:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:00 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999984 2024-07-27T06:53:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:00 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:53:00.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:00 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:reconnect 2024-07-27T06:53:00.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:00 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:53:00.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:00 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:rejoin 2024-07-27T06:53:00.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:00 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:53:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-mon[32610]: pgmap v589: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T06:53:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-mon[32610]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:53:01.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:53:01.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: *** Caught signal (Aborted) ** 2024-07-27T06:53:01.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: in thread 7f610a653640 thread_name:ms_dispatch 2024-07-27T06:53:01.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:01.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f610fa2a6f0] 2024-07-27T06:53:01.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f610fa7794c] 2024-07-27T06:53:01.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 3: raise() 2024-07-27T06:53:01.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 4: abort() 2024-07-27T06:53:01.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 5: /lib64/libc.so.6(+0x2871b) [0x7f610fa1471b] 2024-07-27T06:53:01.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 6: /lib64/libc.so.6(+0x37386) [0x7f610fa23386] 2024-07-27T06:53:01.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d3626c86b9] 2024-07-27T06:53:01.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d36288680c] 2024-07-27T06:53:01.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d3627d4cde] 2024-07-27T06:53:01.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d362796763] 2024-07-27T06:53:01.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d362797a1b] 2024-07-27T06:53:01.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d36277b752] 2024-07-27T06:53:01.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d3626c9a56] 2024-07-27T06:53:01.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d3626cd637] 2024-07-27T06:53:01.714 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d3626cdf5e] 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d3626b103a] 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f61102f7407] 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 18: (DispatchQueue::entry()+0x60f) [0x7f61102f7cf9] 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f61103d45a1] 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 20: (Thread::entry_wrapper()+0x43) [0x7f6110181cdd] 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6110181cf9] 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 22: /lib64/libc.so.6(+0x89c02) [0x7f610fa75c02] 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f610fafac40] 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 2024-07-27T06:53:01.503+0000 7f610a653640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: in thread 7f610a653640 thread_name:ms_dispatch 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f610fa2a6f0] 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f610fa7794c] 2024-07-27T06:53:01.715 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 3: raise() 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 4: abort() 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 5: /lib64/libc.so.6(+0x2871b) [0x7f610fa1471b] 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 6: /lib64/libc.so.6(+0x37386) [0x7f610fa23386] 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d3626c86b9] 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d36288680c] 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d3627d4cde] 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d362796763] 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d362797a1b] 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d36277b752] 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d3626c9a56] 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d3626cd637] 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d3626cdf5e] 2024-07-27T06:53:01.716 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d3626b103a] 2024-07-27T06:53:01.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f61102f7407] 2024-07-27T06:53:01.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 18: (DispatchQueue::entry()+0x60f) [0x7f61102f7cf9] 2024-07-27T06:53:01.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f61103d45a1] 2024-07-27T06:53:01.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 20: (Thread::entry_wrapper()+0x43) [0x7f6110181cdd] 2024-07-27T06:53:01.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6110181cf9] 2024-07-27T06:53:01.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 22: /lib64/libc.so.6(+0x89c02) [0x7f610fa75c02] 2024-07-27T06:53:01.717 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f610fafac40] 2024-07-27T06:53:01.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:53:01.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 2024-07-27T06:53:01.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 0> 2024-07-27T06:53:01.503+0000 7f610a653640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:53:01.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: in thread 7f610a653640 thread_name:ms_dispatch 2024-07-27T06:53:01.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 2024-07-27T06:53:01.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:01.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f610fa2a6f0] 2024-07-27T06:53:01.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f610fa7794c] 2024-07-27T06:53:01.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 3: raise() 2024-07-27T06:53:01.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 4: abort() 2024-07-27T06:53:01.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 5: /lib64/libc.so.6(+0x2871b) [0x7f610fa1471b] 2024-07-27T06:53:01.718 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 6: /lib64/libc.so.6(+0x37386) [0x7f610fa23386] 2024-07-27T06:53:01.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d3626c86b9] 2024-07-27T06:53:01.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d36288680c] 2024-07-27T06:53:01.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d3627d4cde] 2024-07-27T06:53:01.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d362796763] 2024-07-27T06:53:01.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d362797a1b] 2024-07-27T06:53:01.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d36277b752] 2024-07-27T06:53:01.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d3626c9a56] 2024-07-27T06:53:01.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d3626cd637] 2024-07-27T06:53:01.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d3626cdf5e] 2024-07-27T06:53:01.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d3626b103a] 2024-07-27T06:53:01.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f61102f7407] 2024-07-27T06:53:01.719 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 18: (DispatchQueue::entry()+0x60f) [0x7f61102f7cf9] 2024-07-27T06:53:01.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f61103d45a1] 2024-07-27T06:53:01.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 20: (Thread::entry_wrapper()+0x43) [0x7f6110181cdd] 2024-07-27T06:53:01.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6110181cf9] 2024-07-27T06:53:01.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 22: /lib64/libc.so.6(+0x89c02) [0x7f610fa75c02] 2024-07-27T06:53:01.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f610fafac40] 2024-07-27T06:53:01.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:53:01.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 2024-07-27T06:53:01.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 0> 2024-07-27T06:53:01.503+0000 7f610a653640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:53:01.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: in thread 7f610a653640 thread_name:ms_dispatch 2024-07-27T06:53:01.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 2024-07-27T06:53:01.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:01.720 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f610fa2a6f0] 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f610fa7794c] 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 3: raise() 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 4: abort() 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 5: /lib64/libc.so.6(+0x2871b) [0x7f610fa1471b] 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 6: /lib64/libc.so.6(+0x37386) [0x7f610fa23386] 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d3626c86b9] 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d36288680c] 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d3627d4cde] 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d362796763] 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d362797a1b] 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d36277b752] 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d3626c9a56] 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d3626cd637] 2024-07-27T06:53:01.721 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d3626cdf5e] 2024-07-27T06:53:01.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d3626b103a] 2024-07-27T06:53:01.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f61102f7407] 2024-07-27T06:53:01.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 18: (DispatchQueue::entry()+0x60f) [0x7f61102f7cf9] 2024-07-27T06:53:01.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f61103d45a1] 2024-07-27T06:53:01.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 20: (Thread::entry_wrapper()+0x43) [0x7f6110181cdd] 2024-07-27T06:53:01.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6110181cf9] 2024-07-27T06:53:01.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 22: /lib64/libc.so.6(+0x89c02) [0x7f610fa75c02] 2024-07-27T06:53:01.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f610fafac40] 2024-07-27T06:53:01.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:53:01.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108135]: 2024-07-27T06:53:01.722 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:01 smithi029 podman[108198]: 2024-07-27 06:53:01.594399006 +0000 UTC m=+0.027217652 container died e66bbca00cc2ae798bde2203fba4a6ffb6776554b32eef73c91acf03475b68f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-27T06:53:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:01 smithi077 ceph-mon[37153]: pgmap v589: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T06:53:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:01 smithi077 ceph-mon[37153]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:53:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:01 smithi047 ceph-mon[36835]: pgmap v589: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T06:53:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:01 smithi047 ceph-mon[36835]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:53:02.619 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:02 smithi029 podman[108198]: 2024-07-27 06:53:02.357222851 +0000 UTC m=+0.790041532 container remove e66bbca00cc2ae798bde2203fba4a6ffb6776554b32eef73c91acf03475b68f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-07-27T06:53:02.619 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:02 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:53:02.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:02 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:53:02.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:02 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:active 2024-07-27T06:53:02.619 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:02 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:53:02.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:02 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:53:02.620 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 204.355386 secs 2024-07-27T06:53:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:02 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:53:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:02 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:active 2024-07-27T06:53:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:02 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:53:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:02 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:53:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 204.355386 secs 2024-07-27T06:53:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:02 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:53:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:02 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/2183619755,v1:172.21.15.29:6837/2183619755] up:active 2024-07-27T06:53:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:02 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:53:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:02 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:53:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 204.355386 secs 2024-07-27T06:53:02.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:02 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-27T06:53:02.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:02 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Consumed 1.188s CPU time. 2024-07-27T06:53:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:03 smithi077 ceph-mon[37153]: pgmap v590: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:53:03.911 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:03 smithi029 ceph-mon[32610]: pgmap v590: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:53:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:03 smithi047 ceph-mon[36835]: pgmap v590: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:53:05.832 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:05 smithi029 ceph-mon[32610]: pgmap v591: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 17 op/s 2024-07-27T06:53:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:05 smithi077 ceph-mon[37153]: pgmap v591: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 17 op/s 2024-07-27T06:53:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:05 smithi047 ceph-mon[36835]: pgmap v591: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 17 op/s 2024-07-27T06:53:07.816 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:07 smithi029 ceph-mon[32610]: pgmap v592: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:53:07.816 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 209.355506 secs 2024-07-27T06:53:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:07 smithi077 ceph-mon[37153]: pgmap v592: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:53:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 209.355506 secs 2024-07-27T06:53:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:07 smithi047 ceph-mon[36835]: pgmap v592: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:53:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 209.355506 secs 2024-07-27T06:53:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:53:08.214+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:53:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:53:08.214+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:53:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:53:08.214+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:53:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:53:08.233+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:53:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:53:08.233+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:53:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:53:08.233+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:53:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:09 smithi077 ceph-mon[37153]: pgmap v593: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 197 B/s wr, 19 op/s 2024-07-27T06:53:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:09 smithi047 ceph-mon[36835]: pgmap v593: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 197 B/s wr, 19 op/s 2024-07-27T06:53:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:09 smithi029 ceph-mon[32610]: pgmap v593: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 197 B/s wr, 19 op/s 2024-07-27T06:53:11.749 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:11 smithi047 ceph-mon[36835]: pgmap v594: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:53:11.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:11 smithi029 ceph-mon[32610]: pgmap v594: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:53:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:11 smithi077 ceph-mon[37153]: pgmap v594: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:53:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 214.355629 secs 2024-07-27T06:53:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 214.355629 secs 2024-07-27T06:53:12.959 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 214.355629 secs 2024-07-27T06:53:12.960 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:12 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-07-27T06:53:12.960 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:12 smithi029 systemd[1]: Stopped Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:53:12.960 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:12 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Consumed 1.188s CPU time. 2024-07-27T06:53:12.960 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:12 smithi029 systemd[1]: Starting Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:53:13.212 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:12 smithi029 podman[108292]: 2024-07-27 06:53:12.861670969 +0000 UTC m=+0.018960768 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:53:13.212 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:12 smithi029 podman[108292]: 2024-07-27 06:53:12.970470426 +0000 UTC m=+0.127760264 container create a0462b7c10499a5fb110af677df75c3d3d3613a1824083b0e6ee9b305993ab9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-27T06:53:13.506 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:13 smithi029 podman[108292]: 2024-07-27 06:53:13.233880381 +0000 UTC m=+0.391170171 container init a0462b7c10499a5fb110af677df75c3d3d3613a1824083b0e6ee9b305993ab9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-27T06:53:13.507 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:13 smithi029 podman[108292]: 2024-07-27 06:53:13.236963516 +0000 UTC m=+0.394253310 container start a0462b7c10499a5fb110af677df75c3d3d3613a1824083b0e6ee9b305993ab9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-27T06:53:13.507 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:13 smithi029 ceph-mds[108311]: -- 172.21.15.29:0/1844873018 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560a08c43860 con 0x560a0cde5180 2024-07-27T06:53:13.507 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:13 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: starting mds.d at 2024-07-27T06:53:13.507 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:13 smithi029 bash[108292]: a0462b7c10499a5fb110af677df75c3d3d3613a1824083b0e6ee9b305993ab9d 2024-07-27T06:53:13.507 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:13 smithi029 systemd[1]: Started Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:53:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:13 smithi077 ceph-mon[37153]: pgmap v595: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:53:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:13 smithi077 ceph-mon[37153]: daemon mds.d restarted 2024-07-27T06:53:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:13 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:53:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:13 smithi077 ceph-mon[37153]: osdmap e96: 12 total, 12 up, 12 in 2024-07-27T06:53:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:13 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:boot 2024-07-27T06:53:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:13 smithi077 ceph-mon[37153]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:53:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:13 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:53:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:53:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:13 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:53:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:13 smithi047 ceph-mon[36835]: pgmap v595: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:53:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:13 smithi047 ceph-mon[36835]: daemon mds.d restarted 2024-07-27T06:53:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:13 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:53:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:13 smithi047 ceph-mon[36835]: osdmap e96: 12 total, 12 up, 12 in 2024-07-27T06:53:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:13 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:boot 2024-07-27T06:53:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:13 smithi047 ceph-mon[36835]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:53:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:13 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:53:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:53:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:13 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:53:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:13 smithi029 ceph-mon[32610]: pgmap v595: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:53:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:13 smithi029 ceph-mon[32610]: daemon mds.d restarted 2024-07-27T06:53:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:13 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:53:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:13 smithi029 ceph-mon[32610]: osdmap e96: 12 total, 12 up, 12 in 2024-07-27T06:53:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:13 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:boot 2024-07-27T06:53:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:13 smithi029 ceph-mon[32610]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:53:13.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:13 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:53:13.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:53:13.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:13 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:53:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:15 smithi077 ceph-mon[37153]: pgmap v597: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 13 op/s 2024-07-27T06:53:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:15 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:resolve 2024-07-27T06:53:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:15 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:53:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:15 smithi047 ceph-mon[36835]: pgmap v597: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 13 op/s 2024-07-27T06:53:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:15 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:resolve 2024-07-27T06:53:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:15 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:53:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:15 smithi029 ceph-mon[32610]: pgmap v597: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 13 op/s 2024-07-27T06:53:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:15 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:resolve 2024-07-27T06:53:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:15 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:53:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:16 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999983 2024-07-27T06:53:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:16 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:53:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:16 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:reconnect 2024-07-27T06:53:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:16 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:53:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:16 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999983 2024-07-27T06:53:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:16 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:53:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:16 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:reconnect 2024-07-27T06:53:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:16 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:53:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:16 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999983 2024-07-27T06:53:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:16 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:53:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:16 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:reconnect 2024-07-27T06:53:16.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:16 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:53:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-mon[32610]: pgmap v598: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 10 op/s 2024-07-27T06:53:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:rejoin 2024-07-27T06:53:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:53:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-mon[32610]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:53:17.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 219.355751 secs 2024-07-27T06:53:17.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:53:17.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:active 2024-07-27T06:53:17.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:53:17.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:53:17.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: *** Caught signal (Aborted) ** 2024-07-27T06:53:17.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: in thread 7f4e1002b640 thread_name:ms_dispatch 2024-07-27T06:53:17.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:17.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4e154026f0] 2024-07-27T06:53:17.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4e1544f94c] 2024-07-27T06:53:17.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 3: raise() 2024-07-27T06:53:17.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 4: abort() 2024-07-27T06:53:17.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4e153ec71b] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 6: /lib64/libc.so.6(+0x37386) [0x7f4e153fb386] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5576859806b9] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557685b3e80c] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557685a8ccde] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x557685a4e763] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557685a4fa1b] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557685a33752] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557685981a56] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557685985637] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557685985f5e] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55768596903a] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4e15ccf407] 2024-07-27T06:53:17.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 18: (DispatchQueue::entry()+0x60f) [0x7f4e15ccfcf9] 2024-07-27T06:53:17.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4e15dac5a1] 2024-07-27T06:53:17.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 20: (Thread::entry_wrapper()+0x43) [0x7f4e15b59cdd] 2024-07-27T06:53:17.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4e15b59cf9] 2024-07-27T06:53:17.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4e1544dc02] 2024-07-27T06:53:17.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4e154d2c40] 2024-07-27T06:53:17.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 2024-07-27T06:53:17.475+0000 7f4e1002b640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:53:17.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: in thread 7f4e1002b640 thread_name:ms_dispatch 2024-07-27T06:53:17.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 2024-07-27T06:53:17.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:17.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4e154026f0] 2024-07-27T06:53:17.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4e1544f94c] 2024-07-27T06:53:17.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 3: raise() 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 4: abort() 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4e153ec71b] 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 6: /lib64/libc.so.6(+0x37386) [0x7f4e153fb386] 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5576859806b9] 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557685b3e80c] 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557685a8ccde] 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x557685a4e763] 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557685a4fa1b] 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557685a33752] 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557685981a56] 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557685985637] 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557685985f5e] 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55768596903a] 2024-07-27T06:53:17.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4e15ccf407] 2024-07-27T06:53:17.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 18: (DispatchQueue::entry()+0x60f) [0x7f4e15ccfcf9] 2024-07-27T06:53:17.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4e15dac5a1] 2024-07-27T06:53:17.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 20: (Thread::entry_wrapper()+0x43) [0x7f4e15b59cdd] 2024-07-27T06:53:17.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4e15b59cf9] 2024-07-27T06:53:17.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4e1544dc02] 2024-07-27T06:53:17.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4e154d2c40] 2024-07-27T06:53:17.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:53:17.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 2024-07-27T06:53:17.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 0> 2024-07-27T06:53:17.475+0000 7f4e1002b640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:53:17.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: in thread 7f4e1002b640 thread_name:ms_dispatch 2024-07-27T06:53:17.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 2024-07-27T06:53:17.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:17.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4e154026f0] 2024-07-27T06:53:17.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4e1544f94c] 2024-07-27T06:53:17.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 3: raise() 2024-07-27T06:53:17.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 4: abort() 2024-07-27T06:53:17.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4e153ec71b] 2024-07-27T06:53:17.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 6: /lib64/libc.so.6(+0x37386) [0x7f4e153fb386] 2024-07-27T06:53:17.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5576859806b9] 2024-07-27T06:53:17.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557685b3e80c] 2024-07-27T06:53:17.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557685a8ccde] 2024-07-27T06:53:17.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x557685a4e763] 2024-07-27T06:53:17.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557685a4fa1b] 2024-07-27T06:53:17.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557685a33752] 2024-07-27T06:53:17.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557685981a56] 2024-07-27T06:53:17.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557685985637] 2024-07-27T06:53:17.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557685985f5e] 2024-07-27T06:53:17.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55768596903a] 2024-07-27T06:53:17.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4e15ccf407] 2024-07-27T06:53:17.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 18: (DispatchQueue::entry()+0x60f) [0x7f4e15ccfcf9] 2024-07-27T06:53:17.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4e15dac5a1] 2024-07-27T06:53:17.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 20: (Thread::entry_wrapper()+0x43) [0x7f4e15b59cdd] 2024-07-27T06:53:17.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4e15b59cf9] 2024-07-27T06:53:17.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4e1544dc02] 2024-07-27T06:53:17.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4e154d2c40] 2024-07-27T06:53:17.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:53:17.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 2024-07-27T06:53:17.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 0> 2024-07-27T06:53:17.475+0000 7f4e1002b640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:53:17.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: in thread 7f4e1002b640 thread_name:ms_dispatch 2024-07-27T06:53:17.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 2024-07-27T06:53:17.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:17.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4e154026f0] 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4e1544f94c] 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 3: raise() 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 4: abort() 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4e153ec71b] 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 6: /lib64/libc.so.6(+0x37386) [0x7f4e153fb386] 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5576859806b9] 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557685b3e80c] 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557685a8ccde] 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x557685a4e763] 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557685a4fa1b] 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557685a33752] 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557685981a56] 2024-07-27T06:53:17.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557685985637] 2024-07-27T06:53:17.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557685985f5e] 2024-07-27T06:53:17.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55768596903a] 2024-07-27T06:53:17.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4e15ccf407] 2024-07-27T06:53:17.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 18: (DispatchQueue::entry()+0x60f) [0x7f4e15ccfcf9] 2024-07-27T06:53:17.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4e15dac5a1] 2024-07-27T06:53:17.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 20: (Thread::entry_wrapper()+0x43) [0x7f4e15b59cdd] 2024-07-27T06:53:17.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4e15b59cf9] 2024-07-27T06:53:17.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4e1544dc02] 2024-07-27T06:53:17.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4e154d2c40] 2024-07-27T06:53:17.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:53:17.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[77233]: 2024-07-27T06:53:17.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:17 smithi029 podman[108346]: 2024-07-27 06:53:17.557960065 +0000 UTC m=+0.032211798 container died b33ce37cc3a07f27f4cbb90abcd7b020fcb4f5100260d2d57c22b6ee6c263859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-07-27T06:53:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:17 smithi077 ceph-mon[37153]: pgmap v598: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 10 op/s 2024-07-27T06:53:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:17 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:rejoin 2024-07-27T06:53:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:17 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:53:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:17 smithi077 ceph-mon[37153]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:53:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 219.355751 secs 2024-07-27T06:53:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:17 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:53:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:17 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:active 2024-07-27T06:53:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:17 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:53:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:17 smithi047 ceph-mon[36835]: pgmap v598: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 10 op/s 2024-07-27T06:53:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:17 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:rejoin 2024-07-27T06:53:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:17 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:53:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:17 smithi047 ceph-mon[36835]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:53:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 219.355751 secs 2024-07-27T06:53:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:17 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:53:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:17 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/3730476844,v1:172.21.15.29:6835/3730476844] up:active 2024-07-27T06:53:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:17 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:53:18.626 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:18 smithi029 podman[108346]: 2024-07-27 06:53:18.366947962 +0000 UTC m=+0.841199674 container remove b33ce37cc3a07f27f4cbb90abcd7b020fcb4f5100260d2d57c22b6ee6c263859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True) 2024-07-27T06:53:18.626 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:18 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:53:18.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:18 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-27T06:53:18.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:18 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Consumed 1.340s CPU time. 2024-07-27T06:53:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:19 smithi077 ceph-mon[37153]: pgmap v599: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:53:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:19 smithi047 ceph-mon[36835]: pgmap v599: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:53:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:19 smithi029 ceph-mon[32610]: pgmap v599: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:53:21.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:21 smithi029 ceph-mon[32610]: pgmap v600: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:53:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:21 smithi077 ceph-mon[37153]: pgmap v600: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:53:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:21 smithi047 ceph-mon[36835]: pgmap v600: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:53:22.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 224.355881 secs 2024-07-27T06:53:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 224.355881 secs 2024-07-27T06:53:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 224.355881 secs 2024-07-27T06:53:23.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:53:23.228+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:53:23.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:53:23.228+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:53:23.230 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:53:23.228+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:53:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:53:23.247+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:53:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:53:23.247+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:53:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:53:23.247+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:53:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:23 smithi077 ceph-mon[37153]: pgmap v601: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:53:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:23 smithi047 ceph-mon[36835]: pgmap v601: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:53:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:23 smithi029 ceph-mon[32610]: pgmap v601: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:53:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:25 smithi077 ceph-mon[37153]: pgmap v602: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 194 B/s wr, 19 op/s 2024-07-27T06:53:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:25 smithi047 ceph-mon[36835]: pgmap v602: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 194 B/s wr, 19 op/s 2024-07-27T06:53:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:25 smithi029 ceph-mon[32610]: pgmap v602: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 194 B/s wr, 19 op/s 2024-07-27T06:53:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:27 smithi077 ceph-mon[37153]: pgmap v603: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:53:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 229.356004 secs 2024-07-27T06:53:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:27 smithi047 ceph-mon[36835]: pgmap v603: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:53:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 229.356004 secs 2024-07-27T06:53:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:27 smithi029 ceph-mon[32610]: pgmap v603: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:53:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 229.356004 secs 2024-07-27T06:53:28.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:28 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-07-27T06:53:28.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:28 smithi029 systemd[1]: Stopped Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:53:28.967 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:28 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Consumed 1.340s CPU time. 2024-07-27T06:53:28.967 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:28 smithi029 systemd[1]: Starting Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:53:29.272 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:28 smithi029 podman[108450]: 2024-07-27 06:53:28.868568094 +0000 UTC m=+0.018561712 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:53:29.272 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:28 smithi029 podman[108450]: 2024-07-27 06:53:28.980963578 +0000 UTC m=+0.130957182 container create 73b48f662a7ed4e30752e25cc786f49923800618bb2de24f482c369e0358e09a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240724) 2024-07-27T06:53:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:29 smithi029 ceph-mon[32610]: pgmap v604: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:53:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:53:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:29 smithi029 ceph-mon[32610]: daemon mds.j restarted 2024-07-27T06:53:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:29 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:53:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:29 smithi029 ceph-mon[32610]: osdmap e97: 12 total, 12 up, 12 in 2024-07-27T06:53:29.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:29 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:boot 2024-07-27T06:53:29.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:29 smithi029 ceph-mon[32610]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:53:29.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:29 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:53:29.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:53:29.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:29 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:53:29.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:29 smithi029 podman[108450]: 2024-07-27 06:53:29.394400197 +0000 UTC m=+0.544393799 container init 73b48f662a7ed4e30752e25cc786f49923800618bb2de24f482c369e0358e09a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, io.buildah.version=1.36.0) 2024-07-27T06:53:29.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:29 smithi029 podman[108450]: 2024-07-27 06:53:29.397454661 +0000 UTC m=+0.547448263 container start 73b48f662a7ed4e30752e25cc786f49923800618bb2de24f482c369e0358e09a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-07-27T06:53:29.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:29 smithi029 ceph-mds[108520]: -- 172.21.15.29:0/2504902038 <== mon.2 v2:172.21.15.47:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558328243860 con 0x55832c3e5600 2024-07-27T06:53:29.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:29 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: starting mds.j at 2024-07-27T06:53:29.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:29 smithi029 bash[108450]: 73b48f662a7ed4e30752e25cc786f49923800618bb2de24f482c369e0358e09a 2024-07-27T06:53:29.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:29 smithi029 systemd[1]: Started Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:53:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:29 smithi077 ceph-mon[37153]: pgmap v604: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:53:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:53:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:29 smithi077 ceph-mon[37153]: daemon mds.j restarted 2024-07-27T06:53:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:29 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:53:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:29 smithi077 ceph-mon[37153]: osdmap e97: 12 total, 12 up, 12 in 2024-07-27T06:53:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:29 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:boot 2024-07-27T06:53:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:29 smithi077 ceph-mon[37153]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:53:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:29 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:53:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:53:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:29 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:53:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:29 smithi047 ceph-mon[36835]: pgmap v604: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:53:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:53:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:29 smithi047 ceph-mon[36835]: daemon mds.j restarted 2024-07-27T06:53:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:29 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:53:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:29 smithi047 ceph-mon[36835]: osdmap e97: 12 total, 12 up, 12 in 2024-07-27T06:53:29.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:29 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:boot 2024-07-27T06:53:29.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:29 smithi047 ceph-mon[36835]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:53:29.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:29 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:53:29.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:53:29.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:29 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:53:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:53:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:53:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:53:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:53:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:53:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:53:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:53:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:53:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:53:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:31 smithi077 ceph-mon[37153]: pgmap v606: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:53:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:31 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:resolve 2024-07-27T06:53:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:31 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:53:31.903 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:31 smithi029 ceph-mon[32610]: pgmap v606: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:53:31.903 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:31 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:resolve 2024-07-27T06:53:31.903 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:31 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:53:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:31 smithi047 ceph-mon[36835]: pgmap v606: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:53:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:31 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:resolve 2024-07-27T06:53:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:31 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:53:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:32 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:53:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:32 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:53:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:32 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:reconnect 2024-07-27T06:53:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:32 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:53:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 234.356143 secs 2024-07-27T06:53:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:32 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:rejoin 2024-07-27T06:53:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:32 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:53:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:32 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:53:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:32 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:53:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:32 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:reconnect 2024-07-27T06:53:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:32 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:53:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 234.356143 secs 2024-07-27T06:53:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:32 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:rejoin 2024-07-27T06:53:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:32 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:53:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:32 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:53:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:32 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:53:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:32 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:reconnect 2024-07-27T06:53:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:32 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:53:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 234.356143 secs 2024-07-27T06:53:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:32 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:rejoin 2024-07-27T06:53:32.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:32 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:53:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:33 smithi077 ceph-mon[37153]: pgmap v607: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-07-27T06:53:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:33 smithi077 ceph-mon[37153]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:53:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:33 smithi047 ceph-mon[36835]: pgmap v607: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-07-27T06:53:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:33 smithi047 ceph-mon[36835]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:53:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-mon[32610]: pgmap v607: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-07-27T06:53:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-mon[32610]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:53:33.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:53:33.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: *** Caught signal (Aborted) ** 2024-07-27T06:53:33.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: in thread 7f6c431a6640 thread_name:ms_dispatch 2024-07-27T06:53:33.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:33.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6c4857d6f0] 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6c485ca94c] 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 3: raise() 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 4: abort() 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6c4856771b] 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 6: /lib64/libc.so.6(+0x37386) [0x7f6c48576386] 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560a066996b9] 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560a0685780c] 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x560a067a5cde] 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x560a06767763] 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560a06768a1b] 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560a0674c752] 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560a0669aa56] 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560a0669e637] 2024-07-27T06:53:33.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560a0669ef5e] 2024-07-27T06:53:33.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560a0668203a] 2024-07-27T06:53:33.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6c48e4a407] 2024-07-27T06:53:33.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 18: (DispatchQueue::entry()+0x60f) [0x7f6c48e4acf9] 2024-07-27T06:53:33.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6c48f275a1] 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 20: (Thread::entry_wrapper()+0x43) [0x7f6c48cd4cdd] 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6c48cd4cf9] 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6c485c8c02] 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6c4864dc40] 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 2024-07-27T06:53:33.598+0000 7f6c431a6640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: in thread 7f6c431a6640 thread_name:ms_dispatch 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6c4857d6f0] 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6c485ca94c] 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 3: raise() 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 4: abort() 2024-07-27T06:53:33.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6c4856771b] 2024-07-27T06:53:33.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 6: /lib64/libc.so.6(+0x37386) [0x7f6c48576386] 2024-07-27T06:53:33.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560a066996b9] 2024-07-27T06:53:33.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560a0685780c] 2024-07-27T06:53:33.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x560a067a5cde] 2024-07-27T06:53:33.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x560a06767763] 2024-07-27T06:53:33.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560a06768a1b] 2024-07-27T06:53:33.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560a0674c752] 2024-07-27T06:53:33.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560a0669aa56] 2024-07-27T06:53:33.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560a0669e637] 2024-07-27T06:53:33.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560a0669ef5e] 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560a0668203a] 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6c48e4a407] 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 18: (DispatchQueue::entry()+0x60f) [0x7f6c48e4acf9] 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6c48f275a1] 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 20: (Thread::entry_wrapper()+0x43) [0x7f6c48cd4cdd] 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6c48cd4cf9] 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6c485c8c02] 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6c4864dc40] 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 0> 2024-07-27T06:53:33.598+0000 7f6c431a6640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: in thread 7f6c431a6640 thread_name:ms_dispatch 2024-07-27T06:53:33.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6c4857d6f0] 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6c485ca94c] 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 3: raise() 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 4: abort() 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6c4856771b] 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 6: /lib64/libc.so.6(+0x37386) [0x7f6c48576386] 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560a066996b9] 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560a0685780c] 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x560a067a5cde] 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x560a06767763] 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560a06768a1b] 2024-07-27T06:53:33.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560a0674c752] 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560a0669aa56] 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560a0669e637] 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560a0669ef5e] 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560a0668203a] 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6c48e4a407] 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 18: (DispatchQueue::entry()+0x60f) [0x7f6c48e4acf9] 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6c48f275a1] 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 20: (Thread::entry_wrapper()+0x43) [0x7f6c48cd4cdd] 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6c48cd4cf9] 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6c485c8c02] 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6c4864dc40] 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:53:33.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 0> 2024-07-27T06:53:33.598+0000 7f6c431a6640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: in thread 7f6c431a6640 thread_name:ms_dispatch 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6c4857d6f0] 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6c485ca94c] 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 3: raise() 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 4: abort() 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6c4856771b] 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 6: /lib64/libc.so.6(+0x37386) [0x7f6c48576386] 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560a066996b9] 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560a0685780c] 2024-07-27T06:53:33.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x560a067a5cde] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x560a06767763] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560a06768a1b] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560a0674c752] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560a0669aa56] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560a0669e637] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560a0669ef5e] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560a0668203a] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6c48e4a407] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 18: (DispatchQueue::entry()+0x60f) [0x7f6c48e4acf9] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6c48f275a1] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 20: (Thread::entry_wrapper()+0x43) [0x7f6c48cd4cdd] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6c48cd4cf9] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6c485c8c02] 2024-07-27T06:53:33.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6c4864dc40] 2024-07-27T06:53:33.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:53:33.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108307]: 2024-07-27T06:53:33.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:33 smithi029 podman[108635]: 2024-07-27 06:53:33.674231164 +0000 UTC m=+0.025224174 container died a0462b7c10499a5fb110af677df75c3d3d3613a1824083b0e6ee9b305993ab9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-27T06:53:34.690 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:34 smithi029 podman[108635]: 2024-07-27 06:53:34.427075991 +0000 UTC m=+0.778069004 container remove a0462b7c10499a5fb110af677df75c3d3d3613a1824083b0e6ee9b305993ab9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-07-27T06:53:34.690 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:34 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:53:34.690 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:34 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:53:34.690 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:34 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:active 2024-07-27T06:53:34.690 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:34 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:53:34.690 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:34 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:53:34.690 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:34 smithi029 ceph-mon[32610]: pgmap v608: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:53:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:34 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:53:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:34 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:active 2024-07-27T06:53:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:34 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:53:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:34 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:53:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:34 smithi077 ceph-mon[37153]: pgmap v608: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:53:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:34 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:53:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:34 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1949909510,v1:172.21.15.29:6837/1949909510] up:active 2024-07-27T06:53:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:34 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:53:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:34 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:53:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:34 smithi047 ceph-mon[36835]: pgmap v608: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:53:34.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:34 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-27T06:53:34.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:34 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Consumed 1.158s CPU time. 2024-07-27T06:53:37.191 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:36 smithi029 ceph-mon[32610]: pgmap v609: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 op/s 2024-07-27T06:53:37.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:36 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 239.356218 secs 2024-07-27T06:53:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:36 smithi047 ceph-mon[36835]: pgmap v609: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 op/s 2024-07-27T06:53:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:36 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 239.356218 secs 2024-07-27T06:53:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:36 smithi077 ceph-mon[37153]: pgmap v609: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 op/s 2024-07-27T06:53:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:36 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 239.356218 secs 2024-07-27T06:53:38.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:53:38.240+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:53:38.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:53:38.240+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:53:38.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:53:38.240+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:53:38.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:53:38.249+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:53:38.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:53:38.249+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:53:38.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:53:38.249+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:53:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:38 smithi047 ceph-mon[36835]: pgmap v610: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:53:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:38 smithi029 ceph-mon[32610]: pgmap v610: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:53:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:38 smithi077 ceph-mon[37153]: pgmap v610: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:53:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:40 smithi047 ceph-mon[36835]: pgmap v611: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 197 B/s wr, 20 op/s 2024-07-27T06:53:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:40 smithi029 ceph-mon[32610]: pgmap v611: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 197 B/s wr, 20 op/s 2024-07-27T06:53:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:40 smithi077 ceph-mon[37153]: pgmap v611: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 197 B/s wr, 20 op/s 2024-07-27T06:53:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:41 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 244.356341 secs 2024-07-27T06:53:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:41 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 244.356341 secs 2024-07-27T06:53:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:41 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 244.356341 secs 2024-07-27T06:53:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:42 smithi047 ceph-mon[36835]: pgmap v612: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:53:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:42 smithi029 ceph-mon[32610]: pgmap v612: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:53:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:42 smithi077 ceph-mon[37153]: pgmap v612: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:53:44.974 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:44 smithi029 ceph-mon[32610]: pgmap v613: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:53:44.974 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:44 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-07-27T06:53:44.974 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:44 smithi029 systemd[1]: Stopped Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:53:44.974 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:44 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Consumed 1.158s CPU time. 2024-07-27T06:53:44.974 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:44 smithi029 systemd[1]: Starting Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:53:44.975 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:44 smithi029 podman[108726]: 2024-07-27 06:53:44.860601083 +0000 UTC m=+0.018525504 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:53:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:44 smithi047 ceph-mon[36835]: pgmap v613: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:53:45.229 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:44 smithi029 podman[108726]: 2024-07-27 06:53:44.973710923 +0000 UTC m=+0.131635381 container create 55a24be59da248e60fd4bf75698ccf3370651f082318f169be0dedb8eb8c1a45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_CLEAN=True) 2024-07-27T06:53:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:44 smithi077 ceph-mon[37153]: pgmap v613: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:53:45.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:45 smithi029 podman[108726]: 2024-07-27 06:53:45.228972067 +0000 UTC m=+0.386896490 container init 55a24be59da248e60fd4bf75698ccf3370651f082318f169be0dedb8eb8c1a45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-07-27T06:53:45.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:45 smithi029 podman[108726]: 2024-07-27 06:53:45.231984205 +0000 UTC m=+0.389908615 container start 55a24be59da248e60fd4bf75698ccf3370651f082318f169be0dedb8eb8c1a45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-07-27T06:53:45.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:45 smithi029 ceph-mds[108745]: -- 172.21.15.29:0/3724350421 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560d1cc25c20 con 0x560d20dc7180 2024-07-27T06:53:45.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:45 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: starting mds.d at 2024-07-27T06:53:45.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:45 smithi029 bash[108726]: 55a24be59da248e60fd4bf75698ccf3370651f082318f169be0dedb8eb8c1a45 2024-07-27T06:53:45.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:53:45 smithi029 systemd[1]: Started Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:53:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:45 smithi047 ceph-mon[36835]: daemon mds.d restarted 2024-07-27T06:53:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:45 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:53:46.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:45 smithi047 ceph-mon[36835]: osdmap e98: 12 total, 12 up, 12 in 2024-07-27T06:53:46.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:45 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:boot 2024-07-27T06:53:46.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:45 smithi047 ceph-mon[36835]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:53:46.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:45 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:53:46.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:53:46.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:45 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:53:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:45 smithi029 ceph-mon[32610]: daemon mds.d restarted 2024-07-27T06:53:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:45 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:53:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:45 smithi029 ceph-mon[32610]: osdmap e98: 12 total, 12 up, 12 in 2024-07-27T06:53:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:45 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:boot 2024-07-27T06:53:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:45 smithi029 ceph-mon[32610]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:53:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:45 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:53:46.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:53:46.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:45 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:53:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:45 smithi077 ceph-mon[37153]: daemon mds.d restarted 2024-07-27T06:53:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:45 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:53:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:45 smithi077 ceph-mon[37153]: osdmap e98: 12 total, 12 up, 12 in 2024-07-27T06:53:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:45 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:boot 2024-07-27T06:53:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:45 smithi077 ceph-mon[37153]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:53:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:45 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:53:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:53:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:45 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:53:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:47 smithi077 ceph-mon[37153]: pgmap v615: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-27T06:53:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:47 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:resolve 2024-07-27T06:53:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:47 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:53:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 249.356462 secs 2024-07-27T06:53:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:47 smithi047 ceph-mon[36835]: pgmap v615: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-27T06:53:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:47 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:resolve 2024-07-27T06:53:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:47 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:53:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 249.356462 secs 2024-07-27T06:53:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:47 smithi029 ceph-mon[32610]: pgmap v615: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-27T06:53:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:47 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:resolve 2024-07-27T06:53:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:47 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:53:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 249.356462 secs 2024-07-27T06:53:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:48 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:53:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:48 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:53:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:48 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:reconnect 2024-07-27T06:53:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:48 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:53:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:48 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:53:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:48 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:53:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:48 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:reconnect 2024-07-27T06:53:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:48 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:53:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:48 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:53:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:48 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:53:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:48 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:reconnect 2024-07-27T06:53:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:48 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:53:49.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:49 smithi077 ceph-mon[37153]: pgmap v616: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:53:49.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:49 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:rejoin 2024-07-27T06:53:49.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:49 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:53:49.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:49 smithi077 ceph-mon[37153]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:53:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:49 smithi047 ceph-mon[36835]: pgmap v616: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:53:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:49 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:rejoin 2024-07-27T06:53:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:49 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:53:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:49 smithi047 ceph-mon[36835]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:53:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-mon[32610]: pgmap v616: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:53:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:rejoin 2024-07-27T06:53:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:53:49.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-mon[32610]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:53:49.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:53:49.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: *** Caught signal (Aborted) ** 2024-07-27T06:53:49.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: in thread 7f61296a9640 thread_name:ms_dispatch 2024-07-27T06:53:49.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:49.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f612ea806f0] 2024-07-27T06:53:49.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f612eacd94c] 2024-07-27T06:53:49.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 3: raise() 2024-07-27T06:53:49.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 4: abort() 2024-07-27T06:53:49.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 5: /lib64/libc.so.6(+0x2871b) [0x7f612ea6a71b] 2024-07-27T06:53:49.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 6: /lib64/libc.so.6(+0x37386) [0x7f612ea79386] 2024-07-27T06:53:49.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558326a986b9] 2024-07-27T06:53:49.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558326c5680c] 2024-07-27T06:53:49.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x558326ba4cde] 2024-07-27T06:53:49.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x558326b66763] 2024-07-27T06:53:49.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558326b67a1b] 2024-07-27T06:53:49.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558326b4b752] 2024-07-27T06:53:49.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558326a99a56] 2024-07-27T06:53:49.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558326a9d637] 2024-07-27T06:53:49.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558326a9df5e] 2024-07-27T06:53:49.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558326a8103a] 2024-07-27T06:53:49.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f612f34d407] 2024-07-27T06:53:49.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 18: (DispatchQueue::entry()+0x60f) [0x7f612f34dcf9] 2024-07-27T06:53:49.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f612f42a5a1] 2024-07-27T06:53:49.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 20: (Thread::entry_wrapper()+0x43) [0x7f612f1d7cdd] 2024-07-27T06:53:49.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 21: (Thread::_entry_func(void*)+0xd) [0x7f612f1d7cf9] 2024-07-27T06:53:49.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 22: /lib64/libc.so.6(+0x89c02) [0x7f612eacbc02] 2024-07-27T06:53:49.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f612eb50c40] 2024-07-27T06:53:49.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 2024-07-27T06:53:49.334+0000 7f61296a9640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:53:49.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: in thread 7f61296a9640 thread_name:ms_dispatch 2024-07-27T06:53:49.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 2024-07-27T06:53:49.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:49.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f612ea806f0] 2024-07-27T06:53:49.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f612eacd94c] 2024-07-27T06:53:49.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 3: raise() 2024-07-27T06:53:49.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 4: abort() 2024-07-27T06:53:49.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 5: /lib64/libc.so.6(+0x2871b) [0x7f612ea6a71b] 2024-07-27T06:53:49.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 6: /lib64/libc.so.6(+0x37386) [0x7f612ea79386] 2024-07-27T06:53:49.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558326a986b9] 2024-07-27T06:53:49.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558326c5680c] 2024-07-27T06:53:49.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x558326ba4cde] 2024-07-27T06:53:49.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x558326b66763] 2024-07-27T06:53:49.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558326b67a1b] 2024-07-27T06:53:49.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558326b4b752] 2024-07-27T06:53:49.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558326a99a56] 2024-07-27T06:53:49.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558326a9d637] 2024-07-27T06:53:49.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558326a9df5e] 2024-07-27T06:53:49.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558326a8103a] 2024-07-27T06:53:49.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f612f34d407] 2024-07-27T06:53:49.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 18: (DispatchQueue::entry()+0x60f) [0x7f612f34dcf9] 2024-07-27T06:53:49.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f612f42a5a1] 2024-07-27T06:53:49.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 20: (Thread::entry_wrapper()+0x43) [0x7f612f1d7cdd] 2024-07-27T06:53:49.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 21: (Thread::_entry_func(void*)+0xd) [0x7f612f1d7cf9] 2024-07-27T06:53:49.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 22: /lib64/libc.so.6(+0x89c02) [0x7f612eacbc02] 2024-07-27T06:53:49.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f612eb50c40] 2024-07-27T06:53:49.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:53:49.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 2024-07-27T06:53:49.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 0> 2024-07-27T06:53:49.334+0000 7f61296a9640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:53:49.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: in thread 7f61296a9640 thread_name:ms_dispatch 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f612ea806f0] 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f612eacd94c] 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 3: raise() 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 4: abort() 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 5: /lib64/libc.so.6(+0x2871b) [0x7f612ea6a71b] 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 6: /lib64/libc.so.6(+0x37386) [0x7f612ea79386] 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558326a986b9] 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558326c5680c] 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x558326ba4cde] 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x558326b66763] 2024-07-27T06:53:49.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558326b67a1b] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558326b4b752] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558326a99a56] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558326a9d637] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558326a9df5e] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558326a8103a] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f612f34d407] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 18: (DispatchQueue::entry()+0x60f) [0x7f612f34dcf9] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f612f42a5a1] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 20: (Thread::entry_wrapper()+0x43) [0x7f612f1d7cdd] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 21: (Thread::_entry_func(void*)+0xd) [0x7f612f1d7cf9] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 22: /lib64/libc.so.6(+0x89c02) [0x7f612eacbc02] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f612eb50c40] 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:53:49.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 0> 2024-07-27T06:53:49.334+0000 7f61296a9640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: in thread 7f61296a9640 thread_name:ms_dispatch 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f612ea806f0] 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f612eacd94c] 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 3: raise() 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 4: abort() 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 5: /lib64/libc.so.6(+0x2871b) [0x7f612ea6a71b] 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 6: /lib64/libc.so.6(+0x37386) [0x7f612ea79386] 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558326a986b9] 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558326c5680c] 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x558326ba4cde] 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x558326b66763] 2024-07-27T06:53:49.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558326b67a1b] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558326b4b752] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558326a99a56] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558326a9d637] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558326a9df5e] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558326a8103a] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f612f34d407] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 18: (DispatchQueue::entry()+0x60f) [0x7f612f34dcf9] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f612f42a5a1] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 20: (Thread::entry_wrapper()+0x43) [0x7f612f1d7cdd] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 21: (Thread::_entry_func(void*)+0xd) [0x7f612f1d7cf9] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 22: /lib64/libc.so.6(+0x89c02) [0x7f612eacbc02] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f612eb50c40] 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:53:49.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108466]: 2024-07-27T06:53:49.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:49 smithi029 podman[108780]: 2024-07-27 06:53:49.430126659 +0000 UTC m=+0.023163686 container died 73b48f662a7ed4e30752e25cc786f49923800618bb2de24f482c369e0358e09a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True) 2024-07-27T06:53:50.506 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:50 smithi029 podman[108780]: 2024-07-27 06:53:50.245031469 +0000 UTC m=+0.838068485 container remove 73b48f662a7ed4e30752e25cc786f49923800618bb2de24f482c369e0358e09a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True) 2024-07-27T06:53:50.506 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:50 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:53:50.506 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:50 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:53:50.506 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:50 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:active 2024-07-27T06:53:50.506 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:50 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:53:50.506 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:50 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:53:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:50 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:53:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:50 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:active 2024-07-27T06:53:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:50 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:53:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:50 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:53:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:50 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:53:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:50 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/382953937,v1:172.21.15.29:6835/382953937] up:active 2024-07-27T06:53:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:50 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:53:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:50 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:53:50.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:50 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-27T06:53:50.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:53:50 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Consumed 1.255s CPU time. 2024-07-27T06:53:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:51 smithi077 ceph-mon[37153]: pgmap v617: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:53:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:51 smithi047 ceph-mon[36835]: pgmap v617: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:53:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:51 smithi029 ceph-mon[32610]: pgmap v617: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:53:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 254.356589 secs 2024-07-27T06:53:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 254.356589 secs 2024-07-27T06:53:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 254.356589 secs 2024-07-27T06:53:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:53:53.255+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:53:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:53:53.255+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:53:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:53:53.255+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:53:53.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:53:53.265+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:53:53.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:53:53.265+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:53:53.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:53:53.265+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:53:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:53 smithi077 ceph-mon[37153]: pgmap v618: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 102 B/s wr, 18 op/s 2024-07-27T06:53:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:53 smithi047 ceph-mon[36835]: pgmap v618: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 102 B/s wr, 18 op/s 2024-07-27T06:53:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:53 smithi029 ceph-mon[32610]: pgmap v618: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 102 B/s wr, 18 op/s 2024-07-27T06:53:55.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:55 smithi029 ceph-mon[32610]: pgmap v619: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:53:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:55 smithi077 ceph-mon[37153]: pgmap v619: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:53:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:55 smithi047 ceph-mon[36835]: pgmap v619: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:53:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:57 smithi077 ceph-mon[37153]: pgmap v620: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 194 B/s wr, 20 op/s 2024-07-27T06:53:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 259.356713 secs 2024-07-27T06:53:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:57 smithi047 ceph-mon[36835]: pgmap v620: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 194 B/s wr, 20 op/s 2024-07-27T06:53:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 259.356713 secs 2024-07-27T06:53:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:57 smithi029 ceph-mon[32610]: pgmap v620: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 194 B/s wr, 20 op/s 2024-07-27T06:53:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 259.356713 secs 2024-07-27T06:53:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:53:59 smithi077 ceph-mon[37153]: pgmap v621: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:53:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:53:59 smithi047 ceph-mon[36835]: pgmap v621: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:53:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:53:59 smithi029 ceph-mon[32610]: pgmap v621: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:54:00.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:00 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-07-27T06:54:00.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:00 smithi029 systemd[1]: Stopped Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:54:00.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:00 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Consumed 1.255s CPU time. 2024-07-27T06:54:00.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:00 smithi029 systemd[1]: Starting Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:54:01.239 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:00 smithi029 podman[108885]: 2024-07-27 06:54:00.866462668 +0000 UTC m=+0.019014476 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:54:01.239 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:00 smithi029 podman[108885]: 2024-07-27 06:54:00.990628539 +0000 UTC m=+0.143180379 container create 7960c2831731debdfdc6cd575a1c8eecda5708248d353d65c181a488142c05d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-07-27T06:54:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:01 smithi077 ceph-mon[37153]: pgmap v622: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:54:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:01 smithi077 ceph-mon[37153]: daemon mds.j restarted 2024-07-27T06:54:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:01 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:54:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:01 smithi077 ceph-mon[37153]: osdmap e99: 12 total, 12 up, 12 in 2024-07-27T06:54:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:01 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:boot 2024-07-27T06:54:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:01 smithi077 ceph-mon[37153]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:54:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:01 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:54:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:54:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:01 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:54:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:01 smithi047 ceph-mon[36835]: pgmap v622: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:54:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:01 smithi047 ceph-mon[36835]: daemon mds.j restarted 2024-07-27T06:54:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:01 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:54:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:01 smithi047 ceph-mon[36835]: osdmap e99: 12 total, 12 up, 12 in 2024-07-27T06:54:01.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:01 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:boot 2024-07-27T06:54:01.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:01 smithi047 ceph-mon[36835]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:54:01.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:01 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:54:01.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:54:01.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:01 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:54:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:01 smithi029 ceph-mon[32610]: pgmap v622: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:54:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:01 smithi029 ceph-mon[32610]: daemon mds.j restarted 2024-07-27T06:54:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:01 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:54:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:01 smithi029 ceph-mon[32610]: osdmap e99: 12 total, 12 up, 12 in 2024-07-27T06:54:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:01 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:boot 2024-07-27T06:54:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:01 smithi029 ceph-mon[32610]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:54:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:01 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:54:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:54:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:01 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:54:01.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:01 smithi029 podman[108885]: 2024-07-27 06:54:01.238806333 +0000 UTC m=+0.391358134 container init 7960c2831731debdfdc6cd575a1c8eecda5708248d353d65c181a488142c05d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-27T06:54:01.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:01 smithi029 podman[108885]: 2024-07-27 06:54:01.24189285 +0000 UTC m=+0.394444659 container start 7960c2831731debdfdc6cd575a1c8eecda5708248d353d65c181a488142c05d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-27T06:54:01.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:01 smithi029 ceph-mds[108904]: -- 172.21.15.29:0/273787131 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559974a83c20 con 0x559978c25180 2024-07-27T06:54:01.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:01 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: starting mds.j at 2024-07-27T06:54:01.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:01 smithi029 bash[108885]: 7960c2831731debdfdc6cd575a1c8eecda5708248d353d65c181a488142c05d2 2024-07-27T06:54:01.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:01 smithi029 systemd[1]: Started Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:54:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 264.356839 secs 2024-07-27T06:54:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 264.356839 secs 2024-07-27T06:54:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 264.356839 secs 2024-07-27T06:54:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:03 smithi077 ceph-mon[37153]: pgmap v624: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:54:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:03 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:resolve 2024-07-27T06:54:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:03 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:54:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:03 smithi047 ceph-mon[36835]: pgmap v624: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:54:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:03 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:resolve 2024-07-27T06:54:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:03 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:54:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:03 smithi029 ceph-mon[32610]: pgmap v624: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-27T06:54:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:03 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:resolve 2024-07-27T06:54:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:03 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:54:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:04 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:54:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:04 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:54:04.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:04 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:reconnect 2024-07-27T06:54:04.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:04 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:54:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:04 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:54:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:04 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:54:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:04 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:reconnect 2024-07-27T06:54:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:04 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:54:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:04 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:54:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:04 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:54:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:04 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:reconnect 2024-07-27T06:54:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:04 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:54:05.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-mon[32610]: pgmap v625: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:54:05.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:rejoin 2024-07-27T06:54:05.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:54:05.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-mon[32610]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:54:05.651 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:54:05.651 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: *** Caught signal (Aborted) ** 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: in thread 7fefebcc2640 thread_name:ms_dispatch 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7feff10996f0] 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 2: /lib64/libc.so.6(+0x8b94c) [0x7feff10e694c] 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 3: raise() 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 4: abort() 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 5: /lib64/libc.so.6(+0x2871b) [0x7feff108371b] 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 6: /lib64/libc.so.6(+0x37386) [0x7feff1092386] 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560d1a3f26b9] 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560d1a5b080c] 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x560d1a4fecde] 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x560d1a4c0763] 2024-07-27T06:54:05.652 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560d1a4c1a1b] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560d1a4a5752] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560d1a3f3a56] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560d1a3f7637] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560d1a3f7f5e] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560d1a3db03a] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7feff1966407] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 18: (DispatchQueue::entry()+0x60f) [0x7feff1966cf9] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7feff1a435a1] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 20: (Thread::entry_wrapper()+0x43) [0x7feff17f0cdd] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 21: (Thread::_entry_func(void*)+0xd) [0x7feff17f0cf9] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 22: /lib64/libc.so.6(+0x89c02) [0x7feff10e4c02] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 23: /lib64/libc.so.6(+0x10ec40) [0x7feff1169c40] 2024-07-27T06:54:05.653 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 2024-07-27T06:54:05.385+0000 7fefebcc2640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:54:05.654 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: in thread 7fefebcc2640 thread_name:ms_dispatch 2024-07-27T06:54:05.654 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 2024-07-27T06:54:05.654 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:05.654 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7feff10996f0] 2024-07-27T06:54:05.654 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 2: /lib64/libc.so.6(+0x8b94c) [0x7feff10e694c] 2024-07-27T06:54:05.654 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 3: raise() 2024-07-27T06:54:05.654 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 4: abort() 2024-07-27T06:54:05.654 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 5: /lib64/libc.so.6(+0x2871b) [0x7feff108371b] 2024-07-27T06:54:05.654 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 6: /lib64/libc.so.6(+0x37386) [0x7feff1092386] 2024-07-27T06:54:05.654 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560d1a3f26b9] 2024-07-27T06:54:05.654 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560d1a5b080c] 2024-07-27T06:54:05.654 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x560d1a4fecde] 2024-07-27T06:54:05.655 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x560d1a4c0763] 2024-07-27T06:54:05.655 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560d1a4c1a1b] 2024-07-27T06:54:05.655 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560d1a4a5752] 2024-07-27T06:54:05.655 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560d1a3f3a56] 2024-07-27T06:54:05.655 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560d1a3f7637] 2024-07-27T06:54:05.655 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560d1a3f7f5e] 2024-07-27T06:54:05.655 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560d1a3db03a] 2024-07-27T06:54:05.655 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7feff1966407] 2024-07-27T06:54:05.655 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 18: (DispatchQueue::entry()+0x60f) [0x7feff1966cf9] 2024-07-27T06:54:05.655 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7feff1a435a1] 2024-07-27T06:54:05.656 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 20: (Thread::entry_wrapper()+0x43) [0x7feff17f0cdd] 2024-07-27T06:54:05.656 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 21: (Thread::_entry_func(void*)+0xd) [0x7feff17f0cf9] 2024-07-27T06:54:05.656 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 22: /lib64/libc.so.6(+0x89c02) [0x7feff10e4c02] 2024-07-27T06:54:05.656 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 23: /lib64/libc.so.6(+0x10ec40) [0x7feff1169c40] 2024-07-27T06:54:05.656 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:54:05.656 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 2024-07-27T06:54:05.656 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 0> 2024-07-27T06:54:05.385+0000 7fefebcc2640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:54:05.656 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: in thread 7fefebcc2640 thread_name:ms_dispatch 2024-07-27T06:54:05.656 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 2024-07-27T06:54:05.657 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:05.657 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7feff10996f0] 2024-07-27T06:54:05.657 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 2: /lib64/libc.so.6(+0x8b94c) [0x7feff10e694c] 2024-07-27T06:54:05.657 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 3: raise() 2024-07-27T06:54:05.657 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 4: abort() 2024-07-27T06:54:05.657 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 5: /lib64/libc.so.6(+0x2871b) [0x7feff108371b] 2024-07-27T06:54:05.657 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 6: /lib64/libc.so.6(+0x37386) [0x7feff1092386] 2024-07-27T06:54:05.657 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560d1a3f26b9] 2024-07-27T06:54:05.657 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560d1a5b080c] 2024-07-27T06:54:05.657 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x560d1a4fecde] 2024-07-27T06:54:05.657 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x560d1a4c0763] 2024-07-27T06:54:05.657 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560d1a4c1a1b] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560d1a4a5752] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560d1a3f3a56] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560d1a3f7637] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560d1a3f7f5e] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560d1a3db03a] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7feff1966407] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 18: (DispatchQueue::entry()+0x60f) [0x7feff1966cf9] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7feff1a435a1] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 20: (Thread::entry_wrapper()+0x43) [0x7feff17f0cdd] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 21: (Thread::_entry_func(void*)+0xd) [0x7feff17f0cf9] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 22: /lib64/libc.so.6(+0x89c02) [0x7feff10e4c02] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 23: /lib64/libc.so.6(+0x10ec40) [0x7feff1169c40] 2024-07-27T06:54:05.658 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 0> 2024-07-27T06:54:05.385+0000 7fefebcc2640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: in thread 7fefebcc2640 thread_name:ms_dispatch 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7feff10996f0] 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 2: /lib64/libc.so.6(+0x8b94c) [0x7feff10e694c] 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 3: raise() 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 4: abort() 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 5: /lib64/libc.so.6(+0x2871b) [0x7feff108371b] 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 6: /lib64/libc.so.6(+0x37386) [0x7feff1092386] 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560d1a3f26b9] 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560d1a5b080c] 2024-07-27T06:54:05.659 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x560d1a4fecde] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x560d1a4c0763] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560d1a4c1a1b] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560d1a4a5752] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560d1a3f3a56] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560d1a3f7637] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560d1a3f7f5e] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560d1a3db03a] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7feff1966407] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 18: (DispatchQueue::entry()+0x60f) [0x7feff1966cf9] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7feff1a435a1] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 20: (Thread::entry_wrapper()+0x43) [0x7feff17f0cdd] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 21: (Thread::_entry_func(void*)+0xd) [0x7feff17f0cf9] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 22: /lib64/libc.so.6(+0x89c02) [0x7feff10e4c02] 2024-07-27T06:54:05.660 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 23: /lib64/libc.so.6(+0x10ec40) [0x7feff1169c40] 2024-07-27T06:54:05.661 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:54:05.661 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[108741]: 2024-07-27T06:54:05.661 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:05 smithi029 podman[108939]: 2024-07-27 06:54:05.460162181 +0000 UTC m=+0.022458817 container died 55a24be59da248e60fd4bf75698ccf3370651f082318f169be0dedb8eb8c1a45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-07-27T06:54:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:05 smithi047 ceph-mon[36835]: pgmap v625: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:54:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:05 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:rejoin 2024-07-27T06:54:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:05 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:54:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:05 smithi047 ceph-mon[36835]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:54:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:05 smithi077 ceph-mon[37153]: pgmap v625: 97 pgs: 97 active+clean; 45 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:54:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:05 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:rejoin 2024-07-27T06:54:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:05 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:54:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:05 smithi077 ceph-mon[37153]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:54:06.468 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:06 smithi029 podman[108939]: 2024-07-27 06:54:06.213783362 +0000 UTC m=+0.776080031 container remove 55a24be59da248e60fd4bf75698ccf3370651f082318f169be0dedb8eb8c1a45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-07-27T06:54:06.469 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:06 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:54:06.469 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:06 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:54:06.469 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:06 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:active 2024-07-27T06:54:06.469 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:06 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:54:06.469 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:06 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:54:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:06 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:54:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:06 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:active 2024-07-27T06:54:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:06 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:54:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:06 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:54:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:06 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:54:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:06 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1202798464,v1:172.21.15.29:6837/1202798464] up:active 2024-07-27T06:54:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:06 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:54:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:06 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:54:06.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:06 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-27T06:54:06.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:06 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Consumed 1.154s CPU time. 2024-07-27T06:54:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:07 smithi047 ceph-mon[36835]: pgmap v626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 op/s 2024-07-27T06:54:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 269.356966 secs 2024-07-27T06:54:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:07 smithi029 ceph-mon[32610]: pgmap v626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 op/s 2024-07-27T06:54:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 269.356966 secs 2024-07-27T06:54:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:07 smithi077 ceph-mon[37153]: pgmap v626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 op/s 2024-07-27T06:54:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 269.356966 secs 2024-07-27T06:54:08.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:54:08.271+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:54:08.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:54:08.271+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:54:08.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:54:08.271+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:54:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:54:08.281+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:54:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:54:08.281+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:54:08.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:54:08.281+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:54:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:09 smithi047 ceph-mon[36835]: pgmap v627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:54:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:09 smithi029 ceph-mon[32610]: pgmap v627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:54:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:09 smithi077 ceph-mon[37153]: pgmap v627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:54:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:11 smithi047 ceph-mon[36835]: pgmap v628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:54:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:11 smithi029 ceph-mon[32610]: pgmap v628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:54:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:11 smithi077 ceph-mon[37153]: pgmap v628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:54:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 274.357067 secs 2024-07-27T06:54:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 274.357067 secs 2024-07-27T06:54:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 274.357067 secs 2024-07-27T06:54:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:13 smithi047 ceph-mon[36835]: pgmap v629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 194 B/s wr, 19 op/s 2024-07-27T06:54:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:13 smithi029 ceph-mon[32610]: pgmap v629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 194 B/s wr, 19 op/s 2024-07-27T06:54:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:13 smithi077 ceph-mon[37153]: pgmap v629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 194 B/s wr, 19 op/s 2024-07-27T06:54:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:15 smithi047 ceph-mon[36835]: pgmap v630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:54:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:15 smithi029 ceph-mon[32610]: pgmap v630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:54:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:15 smithi077 ceph-mon[37153]: pgmap v630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:54:16.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:16 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-07-27T06:54:16.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:16 smithi029 systemd[1]: Stopped Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:54:16.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:16 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Consumed 1.154s CPU time. 2024-07-27T06:54:16.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:16 smithi029 systemd[1]: Starting Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:54:16.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:16 smithi029 podman[109029]: 2024-07-27 06:54:16.950248648 +0000 UTC m=+0.102164929 container create 03dc51e099d1f97e6277a7f38065045a868d619970f7b27abde6180f9ec99ce4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-07-27T06:54:16.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:16 smithi029 podman[109029]: 2024-07-27 06:54:16.866645725 +0000 UTC m=+0.018561974 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:54:17.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:17 smithi077 ceph-mon[37153]: pgmap v631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:54:17.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 279.357136 secs 2024-07-27T06:54:17.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:17 smithi077 ceph-mon[37153]: daemon mds.d restarted 2024-07-27T06:54:17.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:17 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:54:17.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:17 smithi077 ceph-mon[37153]: osdmap e100: 12 total, 12 up, 12 in 2024-07-27T06:54:17.660 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:17 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:boot 2024-07-27T06:54:17.660 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:17 smithi077 ceph-mon[37153]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:54:17.660 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:17 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:54:17.660 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:54:17.660 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:17 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:54:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:17 smithi047 ceph-mon[36835]: pgmap v631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:54:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 279.357136 secs 2024-07-27T06:54:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:17 smithi047 ceph-mon[36835]: daemon mds.d restarted 2024-07-27T06:54:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:17 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:54:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:17 smithi047 ceph-mon[36835]: osdmap e100: 12 total, 12 up, 12 in 2024-07-27T06:54:17.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:17 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:boot 2024-07-27T06:54:17.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:17 smithi047 ceph-mon[36835]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:54:17.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:17 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:54:17.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:54:17.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:17 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:54:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:17 smithi029 ceph-mon[32610]: pgmap v631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:54:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 279.357136 secs 2024-07-27T06:54:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:17 smithi029 ceph-mon[32610]: daemon mds.d restarted 2024-07-27T06:54:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:17 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:54:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:17 smithi029 ceph-mon[32610]: osdmap e100: 12 total, 12 up, 12 in 2024-07-27T06:54:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:17 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:boot 2024-07-27T06:54:17.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:17 smithi029 ceph-mon[32610]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:54:17.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:17 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:54:17.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:54:17.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:17 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:54:17.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:17 smithi029 podman[109029]: 2024-07-27 06:54:17.223863702 +0000 UTC m=+0.375779938 container init 03dc51e099d1f97e6277a7f38065045a868d619970f7b27abde6180f9ec99ce4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-27T06:54:17.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:17 smithi029 podman[109029]: 2024-07-27 06:54:17.226881321 +0000 UTC m=+0.378797566 container start 03dc51e099d1f97e6277a7f38065045a868d619970f7b27abde6180f9ec99ce4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-27T06:54:17.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:17 smithi029 ceph-mds[109048]: -- 172.21.15.29:0/435568411 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc8e1ed860 con 0x55dc9238f180 2024-07-27T06:54:17.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:17 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: starting mds.d at 2024-07-27T06:54:17.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:17 smithi029 bash[109029]: 03dc51e099d1f97e6277a7f38065045a868d619970f7b27abde6180f9ec99ce4 2024-07-27T06:54:17.713 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:17 smithi029 systemd[1]: Started Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:54:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:19 smithi047 ceph-mon[36835]: pgmap v633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2024-07-27T06:54:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:19 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:resolve 2024-07-27T06:54:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:19 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:54:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:19 smithi029 ceph-mon[32610]: pgmap v633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2024-07-27T06:54:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:19 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:resolve 2024-07-27T06:54:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:19 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:54:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:19 smithi077 ceph-mon[37153]: pgmap v633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2024-07-27T06:54:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:19 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:resolve 2024-07-27T06:54:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:19 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:54:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:20 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:54:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:20 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999983 2024-07-27T06:54:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:20 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:reconnect 2024-07-27T06:54:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:20 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:54:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:20 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:54:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:20 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999983 2024-07-27T06:54:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:20 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:reconnect 2024-07-27T06:54:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:20 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:54:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:20 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:54:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:20 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999983 2024-07-27T06:54:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:20 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:reconnect 2024-07-27T06:54:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:20 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:54:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:21 smithi047 ceph-mon[36835]: pgmap v634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-27T06:54:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:21 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:rejoin 2024-07-27T06:54:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:21 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:54:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:21 smithi047 ceph-mon[36835]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:54:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-mon[32610]: pgmap v634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-27T06:54:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:rejoin 2024-07-27T06:54:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:54:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-mon[32610]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:54:21.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:54:21.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: *** Caught signal (Aborted) ** 2024-07-27T06:54:21.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: in thread 7f81f808e640 thread_name:ms_dispatch 2024-07-27T06:54:21.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:21.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f81fd4656f0] 2024-07-27T06:54:21.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f81fd4b294c] 2024-07-27T06:54:21.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 3: raise() 2024-07-27T06:54:21.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 4: abort() 2024-07-27T06:54:21.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 5: /lib64/libc.so.6(+0x2871b) [0x7f81fd44f71b] 2024-07-27T06:54:21.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 6: /lib64/libc.so.6(+0x37386) [0x7f81fd45e386] 2024-07-27T06:54:21.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559973b326b9] 2024-07-27T06:54:21.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559973cf080c] 2024-07-27T06:54:21.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559973c3ecde] 2024-07-27T06:54:21.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559973c00763] 2024-07-27T06:54:21.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559973c01a1b] 2024-07-27T06:54:21.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559973be5752] 2024-07-27T06:54:21.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559973b33a56] 2024-07-27T06:54:21.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559973b37637] 2024-07-27T06:54:21.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559973b37f5e] 2024-07-27T06:54:21.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559973b1b03a] 2024-07-27T06:54:21.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f81fdd32407] 2024-07-27T06:54:21.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 18: (DispatchQueue::entry()+0x60f) [0x7f81fdd32cf9] 2024-07-27T06:54:21.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f81fde0f5a1] 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 20: (Thread::entry_wrapper()+0x43) [0x7f81fdbbccdd] 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 21: (Thread::_entry_func(void*)+0xd) [0x7f81fdbbccf9] 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 22: /lib64/libc.so.6(+0x89c02) [0x7f81fd4b0c02] 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f81fd535c40] 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 2024-07-27T06:54:21.440+0000 7f81f808e640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: in thread 7f81f808e640 thread_name:ms_dispatch 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f81fd4656f0] 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f81fd4b294c] 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 3: raise() 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 4: abort() 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 5: /lib64/libc.so.6(+0x2871b) [0x7f81fd44f71b] 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 6: /lib64/libc.so.6(+0x37386) [0x7f81fd45e386] 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559973b326b9] 2024-07-27T06:54:21.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559973cf080c] 2024-07-27T06:54:21.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559973c3ecde] 2024-07-27T06:54:21.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559973c00763] 2024-07-27T06:54:21.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559973c01a1b] 2024-07-27T06:54:21.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559973be5752] 2024-07-27T06:54:21.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559973b33a56] 2024-07-27T06:54:21.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559973b37637] 2024-07-27T06:54:21.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559973b37f5e] 2024-07-27T06:54:21.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559973b1b03a] 2024-07-27T06:54:21.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f81fdd32407] 2024-07-27T06:54:21.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 18: (DispatchQueue::entry()+0x60f) [0x7f81fdd32cf9] 2024-07-27T06:54:21.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f81fde0f5a1] 2024-07-27T06:54:21.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 20: (Thread::entry_wrapper()+0x43) [0x7f81fdbbccdd] 2024-07-27T06:54:21.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 21: (Thread::_entry_func(void*)+0xd) [0x7f81fdbbccf9] 2024-07-27T06:54:21.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 22: /lib64/libc.so.6(+0x89c02) [0x7f81fd4b0c02] 2024-07-27T06:54:21.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f81fd535c40] 2024-07-27T06:54:21.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:54:21.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 2024-07-27T06:54:21.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 0> 2024-07-27T06:54:21.440+0000 7f81f808e640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:54:21.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: in thread 7f81f808e640 thread_name:ms_dispatch 2024-07-27T06:54:21.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 2024-07-27T06:54:21.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:21.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f81fd4656f0] 2024-07-27T06:54:21.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f81fd4b294c] 2024-07-27T06:54:21.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 3: raise() 2024-07-27T06:54:21.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 4: abort() 2024-07-27T06:54:21.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 5: /lib64/libc.so.6(+0x2871b) [0x7f81fd44f71b] 2024-07-27T06:54:21.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 6: /lib64/libc.so.6(+0x37386) [0x7f81fd45e386] 2024-07-27T06:54:21.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559973b326b9] 2024-07-27T06:54:21.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559973cf080c] 2024-07-27T06:54:21.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559973c3ecde] 2024-07-27T06:54:21.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559973c00763] 2024-07-27T06:54:21.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559973c01a1b] 2024-07-27T06:54:21.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559973be5752] 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559973b33a56] 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559973b37637] 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559973b37f5e] 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559973b1b03a] 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f81fdd32407] 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 18: (DispatchQueue::entry()+0x60f) [0x7f81fdd32cf9] 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f81fde0f5a1] 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 20: (Thread::entry_wrapper()+0x43) [0x7f81fdbbccdd] 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 21: (Thread::_entry_func(void*)+0xd) [0x7f81fdbbccf9] 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 22: /lib64/libc.so.6(+0x89c02) [0x7f81fd4b0c02] 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f81fd535c40] 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 0> 2024-07-27T06:54:21.440+0000 7f81f808e640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:54:21.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: in thread 7f81f808e640 thread_name:ms_dispatch 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f81fd4656f0] 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f81fd4b294c] 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 3: raise() 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 4: abort() 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 5: /lib64/libc.so.6(+0x2871b) [0x7f81fd44f71b] 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 6: /lib64/libc.so.6(+0x37386) [0x7f81fd45e386] 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559973b326b9] 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559973cf080c] 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559973c3ecde] 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559973c00763] 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559973c01a1b] 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559973be5752] 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559973b33a56] 2024-07-27T06:54:21.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559973b37637] 2024-07-27T06:54:21.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559973b37f5e] 2024-07-27T06:54:21.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559973b1b03a] 2024-07-27T06:54:21.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f81fdd32407] 2024-07-27T06:54:21.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 18: (DispatchQueue::entry()+0x60f) [0x7f81fdd32cf9] 2024-07-27T06:54:21.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f81fde0f5a1] 2024-07-27T06:54:21.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 20: (Thread::entry_wrapper()+0x43) [0x7f81fdbbccdd] 2024-07-27T06:54:21.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 21: (Thread::_entry_func(void*)+0xd) [0x7f81fdbbccf9] 2024-07-27T06:54:21.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 22: /lib64/libc.so.6(+0x89c02) [0x7f81fd4b0c02] 2024-07-27T06:54:21.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f81fd535c40] 2024-07-27T06:54:21.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:54:21.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[108900]: 2024-07-27T06:54:21.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:21 smithi029 podman[109083]: 2024-07-27 06:54:21.514996782 +0000 UTC m=+0.019335233 container died 7960c2831731debdfdc6cd575a1c8eecda5708248d353d65c181a488142c05d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240724) 2024-07-27T06:54:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:21 smithi077 ceph-mon[37153]: pgmap v634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-27T06:54:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:21 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:rejoin 2024-07-27T06:54:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:21 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:54:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:21 smithi077 ceph-mon[37153]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:54:22.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:22 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:54:22.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:22 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:active 2024-07-27T06:54:22.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:22 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:54:22.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:22 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:54:22.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 284.357295 secs 2024-07-27T06:54:22.624 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:22 smithi029 podman[109083]: 2024-07-27 06:54:22.36686595 +0000 UTC m=+0.871204439 container remove 7960c2831731debdfdc6cd575a1c8eecda5708248d353d65c181a488142c05d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=) 2024-07-27T06:54:22.624 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:22 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:54:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:22 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:54:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:22 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:active 2024-07-27T06:54:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:22 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:54:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:22 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:54:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 284.357295 secs 2024-07-27T06:54:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:22 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:54:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:22 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/1189882376,v1:172.21.15.29:6835/1189882376] up:active 2024-07-27T06:54:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:22 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:54:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:22 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:54:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 284.357295 secs 2024-07-27T06:54:22.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:22 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-27T06:54:22.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:22 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Consumed 1.165s CPU time. 2024-07-27T06:54:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:54:23.286+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:54:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:54:23.286+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:54:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:54:23.286+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:54:23.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:54:23.287+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:54:23.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:54:23.287+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:54:23.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:54:23.287+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:54:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:23 smithi047 ceph-mon[36835]: pgmap v635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-27T06:54:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:23 smithi029 ceph-mon[32610]: pgmap v635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-27T06:54:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:23 smithi077 ceph-mon[37153]: pgmap v635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-27T06:54:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:25 smithi047 ceph-mon[36835]: pgmap v636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-27T06:54:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:25 smithi029 ceph-mon[32610]: pgmap v636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-27T06:54:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:25 smithi077 ceph-mon[37153]: pgmap v636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-27T06:54:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:27 smithi047 ceph-mon[36835]: pgmap v637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:54:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 289.357397 secs 2024-07-27T06:54:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:27 smithi029 ceph-mon[32610]: pgmap v637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:54:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 289.357397 secs 2024-07-27T06:54:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:27 smithi077 ceph-mon[37153]: pgmap v637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 204 B/s wr, 19 op/s 2024-07-27T06:54:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 289.357397 secs 2024-07-27T06:54:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:29 smithi047 ceph-mon[36835]: pgmap v638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 194 B/s wr, 21 op/s 2024-07-27T06:54:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:29 smithi029 ceph-mon[32610]: pgmap v638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 194 B/s wr, 21 op/s 2024-07-27T06:54:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:29 smithi077 ceph-mon[37153]: pgmap v638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 194 B/s wr, 21 op/s 2024-07-27T06:54:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:54:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:54:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:54:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:54:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:54:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:54:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:54:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:54:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:54:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:54:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:54:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:54:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:31 smithi029 ceph-mon[32610]: pgmap v639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:54:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:31 smithi077 ceph-mon[37153]: pgmap v639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:54:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:31 smithi047 ceph-mon[36835]: pgmap v639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:54:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 294.357541 secs 2024-07-27T06:54:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 294.357541 secs 2024-07-27T06:54:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 294.357541 secs 2024-07-27T06:54:32.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:32 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-07-27T06:54:32.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:32 smithi029 systemd[1]: Stopped Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:54:32.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:32 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Consumed 1.165s CPU time. 2024-07-27T06:54:32.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:32 smithi029 systemd[1]: Starting Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:54:32.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:32 smithi029 podman[109303]: 2024-07-27 06:54:32.95220815 +0000 UTC m=+0.105335557 container create 12f4d8d68d98d1e9f8a340a362255b54477264ed0fd489527a79cf1a6299e741 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-07-27T06:54:32.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:32 smithi029 podman[109303]: 2024-07-27 06:54:32.865914311 +0000 UTC m=+0.019041677 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:54:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:33 smithi029 ceph-mon[32610]: pgmap v640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:54:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:33 smithi029 ceph-mon[32610]: daemon mds.j restarted 2024-07-27T06:54:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:33 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:54:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:33 smithi029 ceph-mon[32610]: osdmap e101: 12 total, 12 up, 12 in 2024-07-27T06:54:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:33 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:boot 2024-07-27T06:54:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:33 smithi029 ceph-mon[32610]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:54:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:33 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:54:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:54:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:33 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:54:33.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:33 smithi029 podman[109303]: 2024-07-27 06:54:33.268142918 +0000 UTC m=+0.421270302 container init 12f4d8d68d98d1e9f8a340a362255b54477264ed0fd489527a79cf1a6299e741 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-27T06:54:33.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:33 smithi029 podman[109303]: 2024-07-27 06:54:33.271197621 +0000 UTC m=+0.424324986 container start 12f4d8d68d98d1e9f8a340a362255b54477264ed0fd489527a79cf1a6299e741 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-07-27T06:54:33.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:33 smithi029 ceph-mds[109323]: -- 172.21.15.29:0/1044851376 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563d54bafc20 con 0x563d58d51180 2024-07-27T06:54:33.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:33 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: starting mds.j at 2024-07-27T06:54:33.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:33 smithi029 bash[109303]: 12f4d8d68d98d1e9f8a340a362255b54477264ed0fd489527a79cf1a6299e741 2024-07-27T06:54:33.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:33 smithi029 systemd[1]: Started Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:54:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:33 smithi077 ceph-mon[37153]: pgmap v640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:54:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:33 smithi077 ceph-mon[37153]: daemon mds.j restarted 2024-07-27T06:54:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:33 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:54:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:33 smithi077 ceph-mon[37153]: osdmap e101: 12 total, 12 up, 12 in 2024-07-27T06:54:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:33 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:boot 2024-07-27T06:54:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:33 smithi077 ceph-mon[37153]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:54:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:33 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:54:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:54:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:33 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:54:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:33 smithi047 ceph-mon[36835]: pgmap v640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:54:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:33 smithi047 ceph-mon[36835]: daemon mds.j restarted 2024-07-27T06:54:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:33 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:54:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:33 smithi047 ceph-mon[36835]: osdmap e101: 12 total, 12 up, 12 in 2024-07-27T06:54:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:33 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:boot 2024-07-27T06:54:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:33 smithi047 ceph-mon[36835]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:54:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:33 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:54:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:54:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:33 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:54:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:35 smithi077 ceph-mon[37153]: pgmap v642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 13 op/s 2024-07-27T06:54:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:35 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:resolve 2024-07-27T06:54:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:35 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:54:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:35 smithi047 ceph-mon[36835]: pgmap v642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 13 op/s 2024-07-27T06:54:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:35 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:resolve 2024-07-27T06:54:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:35 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:54:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:35 smithi029 ceph-mon[32610]: pgmap v642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 13 op/s 2024-07-27T06:54:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:35 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:resolve 2024-07-27T06:54:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:35 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:54:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:36 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:54:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:36 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999983 2024-07-27T06:54:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:36 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:reconnect 2024-07-27T06:54:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:36 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:54:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:36 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:rejoin 2024-07-27T06:54:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:36 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:54:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:36 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:54:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:36 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999983 2024-07-27T06:54:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:36 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:reconnect 2024-07-27T06:54:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:36 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:54:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:36 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:rejoin 2024-07-27T06:54:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:36 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:54:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:36 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:54:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:36 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999983 2024-07-27T06:54:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:36 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:reconnect 2024-07-27T06:54:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:36 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:54:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:36 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:rejoin 2024-07-27T06:54:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:36 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:54:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:37 smithi077 ceph-mon[37153]: pgmap v643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 10 op/s 2024-07-27T06:54:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:37 smithi077 ceph-mon[37153]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:54:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 299.357682 secs 2024-07-27T06:54:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:37 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:54:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:37 smithi047 ceph-mon[36835]: pgmap v643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 10 op/s 2024-07-27T06:54:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:37 smithi047 ceph-mon[36835]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:54:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 299.357682 secs 2024-07-27T06:54:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:37 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:54:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-mon[32610]: pgmap v643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 10 op/s 2024-07-27T06:54:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-mon[32610]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:54:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 299.357682 secs 2024-07-27T06:54:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:54:37.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:54:37.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: *** Caught signal (Aborted) ** 2024-07-27T06:54:37.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: in thread 7f8df2b8c640 thread_name:ms_dispatch 2024-07-27T06:54:37.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:37.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8df7f636f0] 2024-07-27T06:54:37.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8df7fb094c] 2024-07-27T06:54:37.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 3: raise() 2024-07-27T06:54:37.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 4: abort() 2024-07-27T06:54:37.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8df7f4d71b] 2024-07-27T06:54:37.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 6: /lib64/libc.so.6(+0x37386) [0x7f8df7f5c386] 2024-07-27T06:54:37.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55dc8d0eb6b9] 2024-07-27T06:54:37.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55dc8d2a980c] 2024-07-27T06:54:37.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55dc8d1f7cde] 2024-07-27T06:54:37.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55dc8d1b9763] 2024-07-27T06:54:37.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55dc8d1baa1b] 2024-07-27T06:54:37.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55dc8d19e752] 2024-07-27T06:54:37.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55dc8d0eca56] 2024-07-27T06:54:37.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55dc8d0f0637] 2024-07-27T06:54:37.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55dc8d0f0f5e] 2024-07-27T06:54:37.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55dc8d0d403a] 2024-07-27T06:54:37.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8df8830407] 2024-07-27T06:54:37.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 18: (DispatchQueue::entry()+0x60f) [0x7f8df8830cf9] 2024-07-27T06:54:37.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8df890d5a1] 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 20: (Thread::entry_wrapper()+0x43) [0x7f8df86bacdd] 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8df86bacf9] 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8df7faec02] 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8df8033c40] 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 2024-07-27T06:54:37.501+0000 7f8df2b8c640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: in thread 7f8df2b8c640 thread_name:ms_dispatch 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8df7f636f0] 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8df7fb094c] 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 3: raise() 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 4: abort() 2024-07-27T06:54:37.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8df7f4d71b] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 6: /lib64/libc.so.6(+0x37386) [0x7f8df7f5c386] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55dc8d0eb6b9] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55dc8d2a980c] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55dc8d1f7cde] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55dc8d1b9763] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55dc8d1baa1b] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55dc8d19e752] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55dc8d0eca56] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55dc8d0f0637] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55dc8d0f0f5e] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55dc8d0d403a] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8df8830407] 2024-07-27T06:54:37.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 18: (DispatchQueue::entry()+0x60f) [0x7f8df8830cf9] 2024-07-27T06:54:37.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8df890d5a1] 2024-07-27T06:54:37.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 20: (Thread::entry_wrapper()+0x43) [0x7f8df86bacdd] 2024-07-27T06:54:37.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8df86bacf9] 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8df7faec02] 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8df8033c40] 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 0> 2024-07-27T06:54:37.501+0000 7f8df2b8c640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: in thread 7f8df2b8c640 thread_name:ms_dispatch 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8df7f636f0] 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8df7fb094c] 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 3: raise() 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 4: abort() 2024-07-27T06:54:37.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8df7f4d71b] 2024-07-27T06:54:37.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 6: /lib64/libc.so.6(+0x37386) [0x7f8df7f5c386] 2024-07-27T06:54:37.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55dc8d0eb6b9] 2024-07-27T06:54:37.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55dc8d2a980c] 2024-07-27T06:54:37.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55dc8d1f7cde] 2024-07-27T06:54:37.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55dc8d1b9763] 2024-07-27T06:54:37.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55dc8d1baa1b] 2024-07-27T06:54:37.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55dc8d19e752] 2024-07-27T06:54:37.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55dc8d0eca56] 2024-07-27T06:54:37.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55dc8d0f0637] 2024-07-27T06:54:37.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55dc8d0f0f5e] 2024-07-27T06:54:37.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55dc8d0d403a] 2024-07-27T06:54:37.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8df8830407] 2024-07-27T06:54:37.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 18: (DispatchQueue::entry()+0x60f) [0x7f8df8830cf9] 2024-07-27T06:54:37.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8df890d5a1] 2024-07-27T06:54:37.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 20: (Thread::entry_wrapper()+0x43) [0x7f8df86bacdd] 2024-07-27T06:54:37.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8df86bacf9] 2024-07-27T06:54:37.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8df7faec02] 2024-07-27T06:54:37.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8df8033c40] 2024-07-27T06:54:37.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:54:37.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 2024-07-27T06:54:37.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 0> 2024-07-27T06:54:37.501+0000 7f8df2b8c640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:54:37.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: in thread 7f8df2b8c640 thread_name:ms_dispatch 2024-07-27T06:54:37.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 2024-07-27T06:54:37.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:37.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8df7f636f0] 2024-07-27T06:54:37.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f8df7fb094c] 2024-07-27T06:54:37.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 3: raise() 2024-07-27T06:54:37.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 4: abort() 2024-07-27T06:54:37.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 5: /lib64/libc.so.6(+0x2871b) [0x7f8df7f4d71b] 2024-07-27T06:54:37.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 6: /lib64/libc.so.6(+0x37386) [0x7f8df7f5c386] 2024-07-27T06:54:37.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55dc8d0eb6b9] 2024-07-27T06:54:37.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55dc8d2a980c] 2024-07-27T06:54:37.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55dc8d1f7cde] 2024-07-27T06:54:37.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55dc8d1b9763] 2024-07-27T06:54:37.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55dc8d1baa1b] 2024-07-27T06:54:37.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55dc8d19e752] 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55dc8d0eca56] 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55dc8d0f0637] 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55dc8d0f0f5e] 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55dc8d0d403a] 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f8df8830407] 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 18: (DispatchQueue::entry()+0x60f) [0x7f8df8830cf9] 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f8df890d5a1] 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 20: (Thread::entry_wrapper()+0x43) [0x7f8df86bacdd] 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8df86bacf9] 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 22: /lib64/libc.so.6(+0x89c02) [0x7f8df7faec02] 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8df8033c40] 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:54:37.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109044]: 2024-07-27T06:54:37.973 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:37 smithi029 podman[109358]: 2024-07-27 06:54:37.58834191 +0000 UTC m=+0.022988871 container died 03dc51e099d1f97e6277a7f38065045a868d619970f7b27abde6180f9ec99ce4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-07-27T06:54:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:54:38.301+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:54:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:54:38.301+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:54:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:54:38.301+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:54:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:54:38.302+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:54:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:54:38.302+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:54:38.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:54:38.302+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:54:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:38 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:active 2024-07-27T06:54:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:38 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:54:38.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:38 smithi029 podman[109358]: 2024-07-27 06:54:38.476734292 +0000 UTC m=+0.911381283 container remove 03dc51e099d1f97e6277a7f38065045a868d619970f7b27abde6180f9ec99ce4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-27T06:54:38.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:38 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:54:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:38 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:active 2024-07-27T06:54:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:38 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:54:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:38 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/43858083,v1:172.21.15.29:6837/43858083] up:active 2024-07-27T06:54:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:38 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:54:39.212 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:38 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-27T06:54:39.212 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:38 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Consumed 1.269s CPU time. 2024-07-27T06:54:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:39 smithi077 ceph-mon[37153]: pgmap v644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:54:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:39 smithi047 ceph-mon[36835]: pgmap v644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:54:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:39 smithi029 ceph-mon[32610]: pgmap v644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:54:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:41 smithi077 ceph-mon[37153]: pgmap v645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:54:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:41 smithi047 ceph-mon[36835]: pgmap v645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:54:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:41 smithi029 ceph-mon[32610]: pgmap v645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:54:42.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 304.357824 secs 2024-07-27T06:54:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 304.357824 secs 2024-07-27T06:54:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 304.357824 secs 2024-07-27T06:54:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:43 smithi077 ceph-mon[37153]: pgmap v646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 18 op/s 2024-07-27T06:54:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:43 smithi047 ceph-mon[36835]: pgmap v646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 18 op/s 2024-07-27T06:54:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:43 smithi029 ceph-mon[32610]: pgmap v646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 18 op/s 2024-07-27T06:54:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:45 smithi077 ceph-mon[37153]: pgmap v647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 194 B/s wr, 19 op/s 2024-07-27T06:54:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:45 smithi047 ceph-mon[36835]: pgmap v647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 194 B/s wr, 19 op/s 2024-07-27T06:54:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:45 smithi029 ceph-mon[32610]: pgmap v647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 194 B/s wr, 19 op/s 2024-07-27T06:54:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:47 smithi077 ceph-mon[37153]: pgmap v648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:54:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 309.357889 secs 2024-07-27T06:54:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:47 smithi047 ceph-mon[36835]: pgmap v648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:54:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 309.357889 secs 2024-07-27T06:54:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:47 smithi029 ceph-mon[32610]: pgmap v648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:54:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 309.357889 secs 2024-07-27T06:54:49.214 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:48 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-07-27T06:54:49.215 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:48 smithi029 systemd[1]: Stopped Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:54:49.215 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:48 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Consumed 1.269s CPU time. 2024-07-27T06:54:49.215 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:48 smithi029 systemd[1]: Starting Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:54:49.215 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:49 smithi029 podman[109449]: 2024-07-27 06:54:49.109509971 +0000 UTC m=+0.018505091 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:54:49.215 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:49 smithi029 podman[109449]: 2024-07-27 06:54:49.21392485 +0000 UTC m=+0.122919947 container create 7521981690ead910190638cd9565681a6cd1661e8a6f5ecf1caca7a8316667d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-07-27T06:54:49.465 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:49 smithi029 podman[109449]: 2024-07-27 06:54:49.461993712 +0000 UTC m=+0.370988822 container init 7521981690ead910190638cd9565681a6cd1661e8a6f5ecf1caca7a8316667d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-27T06:54:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:49 smithi077 ceph-mon[37153]: pgmap v649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:54:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:49 smithi077 ceph-mon[37153]: daemon mds.d restarted 2024-07-27T06:54:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:49 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:54:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:49 smithi077 ceph-mon[37153]: osdmap e102: 12 total, 12 up, 12 in 2024-07-27T06:54:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:49 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:boot 2024-07-27T06:54:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:49 smithi077 ceph-mon[37153]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:54:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:49 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:54:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:54:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:49 smithi047 ceph-mon[36835]: pgmap v649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:54:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:49 smithi047 ceph-mon[36835]: daemon mds.d restarted 2024-07-27T06:54:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:49 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:54:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:49 smithi047 ceph-mon[36835]: osdmap e102: 12 total, 12 up, 12 in 2024-07-27T06:54:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:49 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:boot 2024-07-27T06:54:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:49 smithi047 ceph-mon[36835]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:54:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:49 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:54:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:54:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:49 smithi029 ceph-mon[32610]: pgmap v649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:54:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:49 smithi029 ceph-mon[32610]: daemon mds.d restarted 2024-07-27T06:54:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:49 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:54:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:49 smithi029 ceph-mon[32610]: osdmap e102: 12 total, 12 up, 12 in 2024-07-27T06:54:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:49 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:boot 2024-07-27T06:54:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:49 smithi029 ceph-mon[32610]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:54:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:49 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:54:49.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-27T06:54:49.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:49 smithi029 podman[109449]: 2024-07-27 06:54:49.464929256 +0000 UTC m=+0.373924355 container start 7521981690ead910190638cd9565681a6cd1661e8a6f5ecf1caca7a8316667d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, ceph=True, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-27T06:54:49.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:49 smithi029 ceph-mds[109468]: -- 172.21.15.29:0/1924938805 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ddda0af520 con 0x55ddde247600 2024-07-27T06:54:49.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:49 smithi029 ceph-mds[109468]: -- 172.21.15.29:0/1924938805 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ddda0a5c20 con 0x55ddde247600 2024-07-27T06:54:49.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:49 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: starting mds.d at 2024-07-27T06:54:49.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:49 smithi029 bash[109449]: 7521981690ead910190638cd9565681a6cd1661e8a6f5ecf1caca7a8316667d5 2024-07-27T06:54:49.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:54:49 smithi029 systemd[1]: Started Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:54:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:50 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:54:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:50 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:54:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:50 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:54:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:51 smithi077 ceph-mon[37153]: pgmap v651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:54:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:51 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:resolve 2024-07-27T06:54:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:51 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:54:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:51 smithi047 ceph-mon[36835]: pgmap v651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:54:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:51 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:resolve 2024-07-27T06:54:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:51 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:54:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:51 smithi029 ceph-mon[32610]: pgmap v651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:54:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:51 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:resolve 2024-07-27T06:54:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:51 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:54:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:52 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999983 2024-07-27T06:54:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:52 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:54:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:52 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:reconnect 2024-07-27T06:54:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:52 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:54:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 314.357997 secs 2024-07-27T06:54:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:52 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999983 2024-07-27T06:54:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:52 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:54:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:52 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:reconnect 2024-07-27T06:54:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:52 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:54:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 314.357997 secs 2024-07-27T06:54:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:52 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999983 2024-07-27T06:54:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:52 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:54:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:52 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:reconnect 2024-07-27T06:54:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:52 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:54:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 314.357997 secs 2024-07-27T06:54:53.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:54:53.308+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:54:53.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:54:53.308+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:54:53.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:54:53.308+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:54:53.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:54:53.317+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:54:53.319 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:54:53.317+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:54:53.319 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:54:53.317+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:54:53.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-mon[32610]: pgmap v652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-07-27T06:54:53.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:rejoin 2024-07-27T06:54:53.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:54:53.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-mon[32610]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:54:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:53 smithi077 ceph-mon[37153]: pgmap v652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-07-27T06:54:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:53 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:rejoin 2024-07-27T06:54:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:53 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:54:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:53 smithi077 ceph-mon[37153]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:54:53.869 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:54:53.869 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: *** Caught signal (Aborted) ** 2024-07-27T06:54:53.870 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: in thread 7f585f236640 thread_name:ms_dispatch 2024-07-27T06:54:53.870 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:53.870 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f586460d6f0] 2024-07-27T06:54:53.870 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f586465a94c] 2024-07-27T06:54:53.870 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 3: raise() 2024-07-27T06:54:53.870 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 4: abort() 2024-07-27T06:54:53.870 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 5: /lib64/libc.so.6(+0x2871b) [0x7f58645f771b] 2024-07-27T06:54:53.870 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 6: /lib64/libc.so.6(+0x37386) [0x7f5864606386] 2024-07-27T06:54:53.870 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563d53a806b9] 2024-07-27T06:54:53.870 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563d53c3e80c] 2024-07-27T06:54:53.870 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x563d53b8ccde] 2024-07-27T06:54:53.871 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x563d53b4e763] 2024-07-27T06:54:53.871 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563d53b4fa1b] 2024-07-27T06:54:53.871 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563d53b33752] 2024-07-27T06:54:53.871 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563d53a81a56] 2024-07-27T06:54:53.871 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563d53a85637] 2024-07-27T06:54:53.871 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563d53a85f5e] 2024-07-27T06:54:53.871 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563d53a6903a] 2024-07-27T06:54:53.871 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5864eda407] 2024-07-27T06:54:53.871 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 18: (DispatchQueue::entry()+0x60f) [0x7f5864edacf9] 2024-07-27T06:54:53.871 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5864fb75a1] 2024-07-27T06:54:53.872 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 20: (Thread::entry_wrapper()+0x43) [0x7f5864d64cdd] 2024-07-27T06:54:53.872 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5864d64cf9] 2024-07-27T06:54:53.872 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5864658c02] 2024-07-27T06:54:53.872 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f58646ddc40] 2024-07-27T06:54:53.872 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 2024-07-27T06:54:53.571+0000 7f585f236640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:54:53.872 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: in thread 7f585f236640 thread_name:ms_dispatch 2024-07-27T06:54:53.873 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 2024-07-27T06:54:53.873 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:53.873 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f586460d6f0] 2024-07-27T06:54:53.873 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f586465a94c] 2024-07-27T06:54:53.873 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 3: raise() 2024-07-27T06:54:53.873 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 4: abort() 2024-07-27T06:54:53.873 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 5: /lib64/libc.so.6(+0x2871b) [0x7f58645f771b] 2024-07-27T06:54:53.873 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 6: /lib64/libc.so.6(+0x37386) [0x7f5864606386] 2024-07-27T06:54:53.873 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563d53a806b9] 2024-07-27T06:54:53.873 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563d53c3e80c] 2024-07-27T06:54:53.873 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x563d53b8ccde] 2024-07-27T06:54:53.873 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x563d53b4e763] 2024-07-27T06:54:53.874 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563d53b4fa1b] 2024-07-27T06:54:53.874 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563d53b33752] 2024-07-27T06:54:53.874 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563d53a81a56] 2024-07-27T06:54:53.874 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563d53a85637] 2024-07-27T06:54:53.874 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563d53a85f5e] 2024-07-27T06:54:53.874 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563d53a6903a] 2024-07-27T06:54:53.874 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5864eda407] 2024-07-27T06:54:53.874 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 18: (DispatchQueue::entry()+0x60f) [0x7f5864edacf9] 2024-07-27T06:54:53.874 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5864fb75a1] 2024-07-27T06:54:53.874 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 20: (Thread::entry_wrapper()+0x43) [0x7f5864d64cdd] 2024-07-27T06:54:53.874 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5864d64cf9] 2024-07-27T06:54:53.874 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5864658c02] 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f58646ddc40] 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 0> 2024-07-27T06:54:53.571+0000 7f585f236640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: in thread 7f585f236640 thread_name:ms_dispatch 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f586460d6f0] 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f586465a94c] 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 3: raise() 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 4: abort() 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 5: /lib64/libc.so.6(+0x2871b) [0x7f58645f771b] 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 6: /lib64/libc.so.6(+0x37386) [0x7f5864606386] 2024-07-27T06:54:53.875 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563d53a806b9] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563d53c3e80c] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x563d53b8ccde] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x563d53b4e763] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563d53b4fa1b] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563d53b33752] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563d53a81a56] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563d53a85637] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563d53a85f5e] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563d53a6903a] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5864eda407] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 18: (DispatchQueue::entry()+0x60f) [0x7f5864edacf9] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5864fb75a1] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 20: (Thread::entry_wrapper()+0x43) [0x7f5864d64cdd] 2024-07-27T06:54:53.876 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5864d64cf9] 2024-07-27T06:54:53.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5864658c02] 2024-07-27T06:54:53.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f58646ddc40] 2024-07-27T06:54:53.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:54:53.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 2024-07-27T06:54:53.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 0> 2024-07-27T06:54:53.571+0000 7f585f236640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:54:53.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: in thread 7f585f236640 thread_name:ms_dispatch 2024-07-27T06:54:53.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 2024-07-27T06:54:53.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:54:53.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f586460d6f0] 2024-07-27T06:54:53.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f586465a94c] 2024-07-27T06:54:53.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 3: raise() 2024-07-27T06:54:53.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 4: abort() 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 5: /lib64/libc.so.6(+0x2871b) [0x7f58645f771b] 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 6: /lib64/libc.so.6(+0x37386) [0x7f5864606386] 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563d53a806b9] 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563d53c3e80c] 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x563d53b8ccde] 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x563d53b4e763] 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563d53b4fa1b] 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563d53b33752] 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563d53a81a56] 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563d53a85637] 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563d53a85f5e] 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563d53a6903a] 2024-07-27T06:54:53.878 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5864eda407] 2024-07-27T06:54:53.879 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 18: (DispatchQueue::entry()+0x60f) [0x7f5864edacf9] 2024-07-27T06:54:53.879 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5864fb75a1] 2024-07-27T06:54:53.879 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 20: (Thread::entry_wrapper()+0x43) [0x7f5864d64cdd] 2024-07-27T06:54:53.879 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5864d64cf9] 2024-07-27T06:54:53.879 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5864658c02] 2024-07-27T06:54:53.879 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f58646ddc40] 2024-07-27T06:54:53.879 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:54:53.879 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109319]: 2024-07-27T06:54:53.879 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:53 smithi029 podman[109504]: 2024-07-27 06:54:53.651007549 +0000 UTC m=+0.026388381 container died 12f4d8d68d98d1e9f8a340a362255b54477264ed0fd489527a79cf1a6299e741 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-27T06:54:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:53 smithi047 ceph-mon[36835]: pgmap v652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-07-27T06:54:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:53 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:rejoin 2024-07-27T06:54:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:53 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:54:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:53 smithi047 ceph-mon[36835]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:54:54.541 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:54 smithi029 podman[109504]: 2024-07-27 06:54:54.428685842 +0000 UTC m=+0.804066664 container remove 12f4d8d68d98d1e9f8a340a362255b54477264ed0fd489527a79cf1a6299e741 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-07-27T06:54:54.541 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:54 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:54:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:54 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:54:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:54 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:active 2024-07-27T06:54:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:54 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:54:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:54 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:54:54.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:54 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-27T06:54:54.877 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:54:54 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Consumed 1.187s CPU time. 2024-07-27T06:54:54.878 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:54 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:54:54.878 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:54 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:active 2024-07-27T06:54:54.878 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:54 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:54:54.878 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:54 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:54:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:54 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:54:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:54 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/2259905677,v1:172.21.15.29:6835/2259905677] up:active 2024-07-27T06:54:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:54 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:54:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:54 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:54:55.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:55 smithi029 ceph-mon[32610]: pgmap v653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:54:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:55 smithi077 ceph-mon[37153]: pgmap v653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:54:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:55 smithi047 ceph-mon[36835]: pgmap v653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:54:57.832 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:57 smithi029 ceph-mon[32610]: pgmap v654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 op/s 2024-07-27T06:54:57.832 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 319.358102 secs 2024-07-27T06:54:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:57 smithi077 ceph-mon[37153]: pgmap v654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 op/s 2024-07-27T06:54:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 319.358102 secs 2024-07-27T06:54:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:57 smithi047 ceph-mon[36835]: pgmap v654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 op/s 2024-07-27T06:54:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 319.358102 secs 2024-07-27T06:54:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:54:59 smithi077 ceph-mon[37153]: pgmap v655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:54:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:54:59 smithi047 ceph-mon[36835]: pgmap v655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:54:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:54:59 smithi029 ceph-mon[32610]: pgmap v655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:55:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:01 smithi077 ceph-mon[37153]: pgmap v656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 198 B/s wr, 20 op/s 2024-07-27T06:55:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:01 smithi047 ceph-mon[36835]: pgmap v656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 198 B/s wr, 20 op/s 2024-07-27T06:55:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:01 smithi029 ceph-mon[32610]: pgmap v656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 198 B/s wr, 20 op/s 2024-07-27T06:55:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 324.358226 secs 2024-07-27T06:55:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 324.358226 secs 2024-07-27T06:55:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 324.358226 secs 2024-07-27T06:55:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:03 smithi077 ceph-mon[37153]: pgmap v657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:55:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:03 smithi047 ceph-mon[36835]: pgmap v657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:55:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:03 smithi029 ceph-mon[32610]: pgmap v657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 18 op/s 2024-07-27T06:55:04.967 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:04 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-07-27T06:55:04.967 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:04 smithi029 systemd[1]: Stopped Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:55:04.967 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:04 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Consumed 1.187s CPU time. 2024-07-27T06:55:04.967 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:04 smithi029 systemd[1]: Starting Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:55:05.231 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:04 smithi029 podman[109607]: 2024-07-27 06:55:04.868555746 +0000 UTC m=+0.018536844 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:55:05.231 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:04 smithi029 podman[109607]: 2024-07-27 06:55:04.982878605 +0000 UTC m=+0.132859688 container create 15fafc159898709efc11d2b3299011f407fab894a0e90f107ee702ab842c17a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=) 2024-07-27T06:55:05.576 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:05 smithi029 podman[109607]: 2024-07-27 06:55:05.231027693 +0000 UTC m=+0.381008790 container init 15fafc159898709efc11d2b3299011f407fab894a0e90f107ee702ab842c17a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-07-27T06:55:05.576 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:05 smithi029 podman[109607]: 2024-07-27 06:55:05.234392442 +0000 UTC m=+0.384373526 container start 15fafc159898709efc11d2b3299011f407fab894a0e90f107ee702ab842c17a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-27T06:55:05.576 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:05 smithi029 ceph-mds[109627]: -- 172.21.15.29:0/1612558799 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555c79919860 con 0x555c7dabba80 2024-07-27T06:55:05.576 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:05 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: starting mds.j at 2024-07-27T06:55:05.576 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:05 smithi029 bash[109607]: 15fafc159898709efc11d2b3299011f407fab894a0e90f107ee702ab842c17a6 2024-07-27T06:55:05.576 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:05 smithi029 systemd[1]: Started Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:55:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:05 smithi077 ceph-mon[37153]: pgmap v658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:55:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:05 smithi077 ceph-mon[37153]: daemon mds.j restarted 2024-07-27T06:55:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:05 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:55:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:05 smithi077 ceph-mon[37153]: osdmap e103: 12 total, 12 up, 12 in 2024-07-27T06:55:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:05 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:boot 2024-07-27T06:55:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:05 smithi077 ceph-mon[37153]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:55:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:05 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:55:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:55:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:05 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:55:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:05 smithi047 ceph-mon[36835]: pgmap v658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:55:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:05 smithi047 ceph-mon[36835]: daemon mds.j restarted 2024-07-27T06:55:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:05 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:55:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:05 smithi047 ceph-mon[36835]: osdmap e103: 12 total, 12 up, 12 in 2024-07-27T06:55:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:05 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:boot 2024-07-27T06:55:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:05 smithi047 ceph-mon[36835]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:55:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:05 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:55:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:55:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:05 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:55:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:05 smithi029 ceph-mon[32610]: pgmap v658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 170 B/s wr, 19 op/s 2024-07-27T06:55:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:05 smithi029 ceph-mon[32610]: daemon mds.j restarted 2024-07-27T06:55:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:05 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:55:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:05 smithi029 ceph-mon[32610]: osdmap e103: 12 total, 12 up, 12 in 2024-07-27T06:55:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:05 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:boot 2024-07-27T06:55:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:05 smithi029 ceph-mon[32610]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-27T06:55:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:05 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:55:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:55:05.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:05 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-27T06:55:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:07 smithi077 ceph-mon[37153]: pgmap v660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-27T06:55:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:07 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:resolve 2024-07-27T06:55:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:07 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:55:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 329.358356 secs 2024-07-27T06:55:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:07 smithi047 ceph-mon[36835]: pgmap v660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-27T06:55:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:07 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:resolve 2024-07-27T06:55:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:07 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:55:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 329.358356 secs 2024-07-27T06:55:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:07 smithi029 ceph-mon[32610]: pgmap v660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-27T06:55:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:07 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:resolve 2024-07-27T06:55:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:07 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-27T06:55:07.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 329.358356 secs 2024-07-27T06:55:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:08.324+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:55:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:08.324+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:55:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:08.324+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:55:08.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:08.324+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:55:08.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:08.324+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:55:08.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:08.324+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:55:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:08 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:55:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:08 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:55:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:08 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:reconnect 2024-07-27T06:55:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:08 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:55:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:08 smithi077 ceph-mon[37153]: pgmap v661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:55:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:08 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:55:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:08 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:55:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:08 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:reconnect 2024-07-27T06:55:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:08 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:55:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:08 smithi047 ceph-mon[36835]: pgmap v661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:55:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:08 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:55:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:08 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:55:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:08 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:reconnect 2024-07-27T06:55:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:08 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-27T06:55:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:08 smithi029 ceph-mon[32610]: pgmap v661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:55:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:09 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:rejoin 2024-07-27T06:55:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:09 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:55:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:09 smithi077 ceph-mon[37153]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:55:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:09 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:rejoin 2024-07-27T06:55:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:09 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:55:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:09 smithi047 ceph-mon[36835]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:55:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:rejoin 2024-07-27T06:55:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-27T06:55:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-mon[32610]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-27T06:55:09.962 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:55:09.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: *** Caught signal (Aborted) ** 2024-07-27T06:55:09.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: in thread 7f659114b640 thread_name:ms_dispatch 2024-07-27T06:55:09.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:55:09.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f65965226f0] 2024-07-27T06:55:09.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f659656f94c] 2024-07-27T06:55:09.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 3: raise() 2024-07-27T06:55:09.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 4: abort() 2024-07-27T06:55:09.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 5: /lib64/libc.so.6(+0x2871b) [0x7f659650c71b] 2024-07-27T06:55:09.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 6: /lib64/libc.so.6(+0x37386) [0x7f659651b386] 2024-07-27T06:55:09.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ddd7fe86b9] 2024-07-27T06:55:09.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ddd81a680c] 2024-07-27T06:55:09.963 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55ddd80f4cde] 2024-07-27T06:55:09.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55ddd80b6763] 2024-07-27T06:55:09.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ddd80b7a1b] 2024-07-27T06:55:09.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ddd809b752] 2024-07-27T06:55:09.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ddd7fe9a56] 2024-07-27T06:55:09.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ddd7fed637] 2024-07-27T06:55:09.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ddd7fedf5e] 2024-07-27T06:55:09.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ddd7fd103a] 2024-07-27T06:55:09.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6596def407] 2024-07-27T06:55:09.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 18: (DispatchQueue::entry()+0x60f) [0x7f6596defcf9] 2024-07-27T06:55:09.964 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6596ecc5a1] 2024-07-27T06:55:09.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 20: (Thread::entry_wrapper()+0x43) [0x7f6596c79cdd] 2024-07-27T06:55:09.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6596c79cf9] 2024-07-27T06:55:09.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 22: /lib64/libc.so.6(+0x89c02) [0x7f659656dc02] 2024-07-27T06:55:09.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f65965f2c40] 2024-07-27T06:55:09.965 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 2024-07-27T06:55:09.624+0000 7f659114b640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:55:09.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: in thread 7f659114b640 thread_name:ms_dispatch 2024-07-27T06:55:09.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 2024-07-27T06:55:09.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:55:09.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f65965226f0] 2024-07-27T06:55:09.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f659656f94c] 2024-07-27T06:55:09.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 3: raise() 2024-07-27T06:55:09.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 4: abort() 2024-07-27T06:55:09.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 5: /lib64/libc.so.6(+0x2871b) [0x7f659650c71b] 2024-07-27T06:55:09.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 6: /lib64/libc.so.6(+0x37386) [0x7f659651b386] 2024-07-27T06:55:09.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ddd7fe86b9] 2024-07-27T06:55:09.966 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ddd81a680c] 2024-07-27T06:55:09.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55ddd80f4cde] 2024-07-27T06:55:09.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55ddd80b6763] 2024-07-27T06:55:09.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ddd80b7a1b] 2024-07-27T06:55:09.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ddd809b752] 2024-07-27T06:55:09.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ddd7fe9a56] 2024-07-27T06:55:09.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ddd7fed637] 2024-07-27T06:55:09.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ddd7fedf5e] 2024-07-27T06:55:09.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ddd7fd103a] 2024-07-27T06:55:09.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6596def407] 2024-07-27T06:55:09.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 18: (DispatchQueue::entry()+0x60f) [0x7f6596defcf9] 2024-07-27T06:55:09.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6596ecc5a1] 2024-07-27T06:55:09.967 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 20: (Thread::entry_wrapper()+0x43) [0x7f6596c79cdd] 2024-07-27T06:55:09.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6596c79cf9] 2024-07-27T06:55:09.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 22: /lib64/libc.so.6(+0x89c02) [0x7f659656dc02] 2024-07-27T06:55:09.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f65965f2c40] 2024-07-27T06:55:09.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:55:09.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 2024-07-27T06:55:09.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 0> 2024-07-27T06:55:09.624+0000 7f659114b640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:55:09.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: in thread 7f659114b640 thread_name:ms_dispatch 2024-07-27T06:55:09.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 2024-07-27T06:55:09.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:55:09.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f65965226f0] 2024-07-27T06:55:09.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f659656f94c] 2024-07-27T06:55:09.968 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 3: raise() 2024-07-27T06:55:09.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 4: abort() 2024-07-27T06:55:09.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 5: /lib64/libc.so.6(+0x2871b) [0x7f659650c71b] 2024-07-27T06:55:09.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 6: /lib64/libc.so.6(+0x37386) [0x7f659651b386] 2024-07-27T06:55:09.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ddd7fe86b9] 2024-07-27T06:55:09.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ddd81a680c] 2024-07-27T06:55:09.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55ddd80f4cde] 2024-07-27T06:55:09.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55ddd80b6763] 2024-07-27T06:55:09.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ddd80b7a1b] 2024-07-27T06:55:09.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ddd809b752] 2024-07-27T06:55:09.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ddd7fe9a56] 2024-07-27T06:55:09.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ddd7fed637] 2024-07-27T06:55:09.969 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ddd7fedf5e] 2024-07-27T06:55:09.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ddd7fd103a] 2024-07-27T06:55:09.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6596def407] 2024-07-27T06:55:09.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 18: (DispatchQueue::entry()+0x60f) [0x7f6596defcf9] 2024-07-27T06:55:09.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6596ecc5a1] 2024-07-27T06:55:09.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 20: (Thread::entry_wrapper()+0x43) [0x7f6596c79cdd] 2024-07-27T06:55:09.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6596c79cf9] 2024-07-27T06:55:09.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 22: /lib64/libc.so.6(+0x89c02) [0x7f659656dc02] 2024-07-27T06:55:09.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f65965f2c40] 2024-07-27T06:55:09.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:55:09.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 2024-07-27T06:55:09.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 0> 2024-07-27T06:55:09.624+0000 7f659114b640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:55:09.970 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: in thread 7f659114b640 thread_name:ms_dispatch 2024-07-27T06:55:09.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 2024-07-27T06:55:09.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:55:09.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f65965226f0] 2024-07-27T06:55:09.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f659656f94c] 2024-07-27T06:55:09.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 3: raise() 2024-07-27T06:55:09.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 4: abort() 2024-07-27T06:55:09.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 5: /lib64/libc.so.6(+0x2871b) [0x7f659650c71b] 2024-07-27T06:55:09.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 6: /lib64/libc.so.6(+0x37386) [0x7f659651b386] 2024-07-27T06:55:09.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ddd7fe86b9] 2024-07-27T06:55:09.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ddd81a680c] 2024-07-27T06:55:09.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55ddd80f4cde] 2024-07-27T06:55:09.971 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55ddd80b6763] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ddd80b7a1b] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ddd809b752] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ddd7fe9a56] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ddd7fed637] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ddd7fedf5e] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ddd7fd103a] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6596def407] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 18: (DispatchQueue::entry()+0x60f) [0x7f6596defcf9] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6596ecc5a1] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 20: (Thread::entry_wrapper()+0x43) [0x7f6596c79cdd] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6596c79cf9] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 22: /lib64/libc.so.6(+0x89c02) [0x7f659656dc02] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f65965f2c40] 2024-07-27T06:55:09.972 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:55:09.973 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d[109464]: 2024-07-27T06:55:09.973 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:09 smithi029 podman[109662]: 2024-07-27 06:55:09.694456931 +0000 UTC m=+0.019772785 container died 7521981690ead910190638cd9565681a6cd1661e8a6f5ecf1caca7a8316667d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-27T06:55:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:10 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:55:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:10 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:active 2024-07-27T06:55:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:10 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:55:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:10 smithi029 ceph-mon[32610]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:55:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:10 smithi029 ceph-mon[32610]: pgmap v662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:55:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:10 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:55:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:10 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:active 2024-07-27T06:55:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:10 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:55:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:10 smithi077 ceph-mon[37153]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:55:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:10 smithi077 ceph-mon[37153]: pgmap v662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:55:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:10 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:55:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:10 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6836/1483763614,v1:172.21.15.29:6837/1483763614] up:active 2024-07-27T06:55:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:10 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-27T06:55:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:10 smithi047 ceph-mon[36835]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-27T06:55:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:10 smithi047 ceph-mon[36835]: pgmap v662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 op/s 2024-07-27T06:55:11.087 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:10 smithi029 podman[109662]: 2024-07-27 06:55:10.831610567 +0000 UTC m=+1.156926410 container remove 7521981690ead910190638cd9565681a6cd1661e8a6f5ecf1caca7a8316667d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724) 2024-07-27T06:55:11.088 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:10 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:55:11.462 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:11 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-27T06:55:11.462 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:11 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Consumed 1.165s CPU time. 2024-07-27T06:55:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:11 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 334.358479 secs 2024-07-27T06:55:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:11 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 334.358479 secs 2024-07-27T06:55:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:11 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 334.358479 secs 2024-07-27T06:55:13.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:12 smithi047 ceph-mon[36835]: pgmap v663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 102 B/s wr, 18 op/s 2024-07-27T06:55:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:12 smithi077 ceph-mon[37153]: pgmap v663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 102 B/s wr, 18 op/s 2024-07-27T06:55:13.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:12 smithi029 ceph-mon[32610]: pgmap v663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 102 B/s wr, 18 op/s 2024-07-27T06:55:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:14 smithi047 ceph-mon[36835]: pgmap v664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:55:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:14 smithi029 ceph-mon[32610]: pgmap v664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:55:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:14 smithi077 ceph-mon[37153]: pgmap v664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 204 B/s wr, 20 op/s 2024-07-27T06:55:17.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:16 smithi077 ceph-mon[37153]: pgmap v665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 193 B/s wr, 20 op/s 2024-07-27T06:55:17.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:16 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 339.358579 secs 2024-07-27T06:55:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:16 smithi047 ceph-mon[36835]: pgmap v665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 193 B/s wr, 20 op/s 2024-07-27T06:55:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:16 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 339.358579 secs 2024-07-27T06:55:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:16 smithi029 ceph-mon[32610]: pgmap v665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 193 B/s wr, 20 op/s 2024-07-27T06:55:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:16 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 339.358579 secs 2024-07-27T06:55:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:18 smithi047 ceph-mon[36835]: pgmap v666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:55:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:18 smithi029 ceph-mon[32610]: pgmap v666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:55:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:18 smithi077 ceph-mon[37153]: pgmap v666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 170 B/s wr, 20 op/s 2024-07-27T06:55:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:20 smithi047 ceph-mon[36835]: pgmap v667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:55:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:20 smithi029 ceph-mon[32610]: pgmap v667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:55:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:20 smithi077 ceph-mon[37153]: pgmap v667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-27T06:55:21.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:21 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-07-27T06:55:21.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:21 smithi029 systemd[1]: Stopped Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:55:21.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:21 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Consumed 1.165s CPU time. 2024-07-27T06:55:21.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:21 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Start request repeated too quickly. 2024-07-27T06:55:21.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:21 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-27T06:55:21.712 INFO:journalctl@ceph.mds.d.smithi029.stdout:Jul 27 06:55:21 smithi029 systemd[1]: Failed to start Ceph mds.d for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:55:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:21 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 344.358704 secs 2024-07-27T06:55:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:21 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 344.358704 secs 2024-07-27T06:55:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:21 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 344.358704 secs 2024-07-27T06:55:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:22 smithi047 ceph-mon[36835]: pgmap v668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:55:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:22 smithi029 ceph-mon[32610]: pgmap v668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:55:23.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:23.337+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:55:23.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:23.337+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:55:23.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:23.337+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:55:23.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:23.338+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:55:23.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:23.338+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:55:23.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:23.338+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:55:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:22 smithi077 ceph-mon[37153]: pgmap v668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-27T06:55:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:24 smithi047 ceph-mon[36835]: pgmap v669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-27T06:55:25.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:24 smithi029 ceph-mon[32610]: pgmap v669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-27T06:55:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:24 smithi077 ceph-mon[37153]: pgmap v669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-27T06:55:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:26 smithi047 ceph-mon[36835]: pgmap v670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 10 op/s 2024-07-27T06:55:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:26 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 349.358832 secs 2024-07-27T06:55:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:26 smithi029 ceph-mon[32610]: pgmap v670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 10 op/s 2024-07-27T06:55:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:26 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 349.358832 secs 2024-07-27T06:55:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:26 smithi077 ceph-mon[37153]: pgmap v670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 10 op/s 2024-07-27T06:55:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:26 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 349.358832 secs 2024-07-27T06:55:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:27 smithi047 ceph-mon[36835]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-07-27T06:55:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:27 smithi047 ceph-mon[36835]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-07-27T06:55:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:27 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:55:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:27 smithi047 ceph-mon[36835]: osdmap e104: 12 total, 12 up, 12 in 2024-07-27T06:55:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:27 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-27T06:55:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:27 smithi029 ceph-mon[32610]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-07-27T06:55:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:27 smithi029 ceph-mon[32610]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-07-27T06:55:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:27 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:55:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:27 smithi029 ceph-mon[32610]: osdmap e104: 12 total, 12 up, 12 in 2024-07-27T06:55:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:27 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-27T06:55:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:27 smithi077 ceph-mon[37153]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-07-27T06:55:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:27 smithi077 ceph-mon[37153]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-07-27T06:55:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:27 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:55:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:27 smithi077 ceph-mon[37153]: osdmap e104: 12 total, 12 up, 12 in 2024-07-27T06:55:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:27 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-27T06:55:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:29 smithi077 ceph-mon[37153]: pgmap v672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T06:55:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:29 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:resolve 2024-07-27T06:55:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:29 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-27T06:55:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:29 smithi047 ceph-mon[36835]: pgmap v672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T06:55:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:29 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:resolve 2024-07-27T06:55:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:29 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-27T06:55:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:29 smithi029 ceph-mon[32610]: pgmap v672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T06:55:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:29 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:resolve 2024-07-27T06:55:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:29 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-27T06:55:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:30 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999984 2024-07-27T06:55:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:30 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:55:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:30 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:reconnect 2024-07-27T06:55:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:30 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-27T06:55:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:55:30.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:30 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:rejoin 2024-07-27T06:55:30.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:30 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-27T06:55:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:30 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999984 2024-07-27T06:55:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:30 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:55:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:30 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:reconnect 2024-07-27T06:55:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:30 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-27T06:55:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:55:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:30 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:rejoin 2024-07-27T06:55:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:30 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-27T06:55:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:30 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0.000999984 2024-07-27T06:55:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:30 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.00199997 2024-07-27T06:55:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:30 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:reconnect 2024-07-27T06:55:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:30 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-27T06:55:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:55:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:30 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:rejoin 2024-07-27T06:55:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:30 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-27T06:55:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-mon[32610]: pgmap v673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-27T06:55:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-mon[32610]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:55:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:31.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:55:31.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: *** Caught signal (Aborted) ** 2024-07-27T06:55:31.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: in thread 7f33ff995640 thread_name:ms_dispatch 2024-07-27T06:55:31.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:55:31.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3404d6c6f0] 2024-07-27T06:55:31.713 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3404db994c] 2024-07-27T06:55:31.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 3: raise() 2024-07-27T06:55:31.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 4: abort() 2024-07-27T06:55:31.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3404d5671b] 2024-07-27T06:55:31.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 6: /lib64/libc.so.6(+0x37386) [0x7f3404d65386] 2024-07-27T06:55:31.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x555c7776b6b9] 2024-07-27T06:55:31.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x555c7792980c] 2024-07-27T06:55:31.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x555c77877cde] 2024-07-27T06:55:31.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x555c77839763] 2024-07-27T06:55:31.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x555c7783aa1b] 2024-07-27T06:55:31.714 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555c7781e752] 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x555c7776ca56] 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x555c77770637] 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x555c77770f5e] 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x555c7775403a] 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3405639407] 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 18: (DispatchQueue::entry()+0x60f) [0x7f3405639cf9] 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f34057165a1] 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 20: (Thread::entry_wrapper()+0x43) [0x7f34054c3cdd] 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 21: (Thread::_entry_func(void*)+0xd) [0x7f34054c3cf9] 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3404db7c02] 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3404e3cc40] 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 2024-07-27T06:55:31.548+0000 7f33ff995640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:55:31.715 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: in thread 7f33ff995640 thread_name:ms_dispatch 2024-07-27T06:55:31.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 2024-07-27T06:55:31.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:55:31.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3404d6c6f0] 2024-07-27T06:55:31.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3404db994c] 2024-07-27T06:55:31.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 3: raise() 2024-07-27T06:55:31.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 4: abort() 2024-07-27T06:55:31.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3404d5671b] 2024-07-27T06:55:31.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 6: /lib64/libc.so.6(+0x37386) [0x7f3404d65386] 2024-07-27T06:55:31.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x555c7776b6b9] 2024-07-27T06:55:31.716 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x555c7792980c] 2024-07-27T06:55:31.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x555c77877cde] 2024-07-27T06:55:31.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x555c77839763] 2024-07-27T06:55:31.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x555c7783aa1b] 2024-07-27T06:55:31.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555c7781e752] 2024-07-27T06:55:31.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x555c7776ca56] 2024-07-27T06:55:31.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x555c77770637] 2024-07-27T06:55:31.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x555c77770f5e] 2024-07-27T06:55:31.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x555c7775403a] 2024-07-27T06:55:31.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3405639407] 2024-07-27T06:55:31.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 18: (DispatchQueue::entry()+0x60f) [0x7f3405639cf9] 2024-07-27T06:55:31.717 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f34057165a1] 2024-07-27T06:55:31.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 20: (Thread::entry_wrapper()+0x43) [0x7f34054c3cdd] 2024-07-27T06:55:31.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 21: (Thread::_entry_func(void*)+0xd) [0x7f34054c3cf9] 2024-07-27T06:55:31.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3404db7c02] 2024-07-27T06:55:31.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3404e3cc40] 2024-07-27T06:55:31.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:55:31.718 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 0> 2024-07-27T06:55:31.548+0000 7f33ff995640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: in thread 7f33ff995640 thread_name:ms_dispatch 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3404d6c6f0] 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3404db994c] 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 3: raise() 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 4: abort() 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3404d5671b] 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 6: /lib64/libc.so.6(+0x37386) [0x7f3404d65386] 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x555c7776b6b9] 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x555c7792980c] 2024-07-27T06:55:31.719 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x555c77877cde] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x555c77839763] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x555c7783aa1b] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555c7781e752] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x555c7776ca56] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x555c77770637] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x555c77770f5e] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x555c7775403a] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3405639407] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 18: (DispatchQueue::entry()+0x60f) [0x7f3405639cf9] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f34057165a1] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 20: (Thread::entry_wrapper()+0x43) [0x7f34054c3cdd] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 21: (Thread::_entry_func(void*)+0xd) [0x7f34054c3cf9] 2024-07-27T06:55:31.720 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3404db7c02] 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3404e3cc40] 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 0> 2024-07-27T06:55:31.548+0000 7f33ff995640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: in thread 7f33ff995640 thread_name:ms_dispatch 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3404d6c6f0] 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3404db994c] 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 3: raise() 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 4: abort() 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3404d5671b] 2024-07-27T06:55:31.721 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 6: /lib64/libc.so.6(+0x37386) [0x7f3404d65386] 2024-07-27T06:55:31.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x555c7776b6b9] 2024-07-27T06:55:31.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x555c7792980c] 2024-07-27T06:55:31.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x555c77877cde] 2024-07-27T06:55:31.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x555c77839763] 2024-07-27T06:55:31.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x555c7783aa1b] 2024-07-27T06:55:31.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555c7781e752] 2024-07-27T06:55:31.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x555c7776ca56] 2024-07-27T06:55:31.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x555c77770637] 2024-07-27T06:55:31.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x555c77770f5e] 2024-07-27T06:55:31.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x555c7775403a] 2024-07-27T06:55:31.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3405639407] 2024-07-27T06:55:31.722 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 18: (DispatchQueue::entry()+0x60f) [0x7f3405639cf9] 2024-07-27T06:55:31.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f34057165a1] 2024-07-27T06:55:31.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 20: (Thread::entry_wrapper()+0x43) [0x7f34054c3cdd] 2024-07-27T06:55:31.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 21: (Thread::_entry_func(void*)+0xd) [0x7f34054c3cf9] 2024-07-27T06:55:31.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3404db7c02] 2024-07-27T06:55:31.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3404e3cc40] 2024-07-27T06:55:31.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:55:31.723 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[109623]: 2024-07-27T06:55:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:31 smithi077 ceph-mon[37153]: pgmap v673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-27T06:55:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:31 smithi077 ceph-mon[37153]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:55:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:31 smithi047 ceph-mon[36835]: pgmap v673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-27T06:55:31.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:31 smithi047 ceph-mon[36835]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:55:31.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:31.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:32.212 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:31 smithi029 podman[109979]: 2024-07-27 06:55:31.942528454 +0000 UTC m=+0.349543170 container died 15fafc159898709efc11d2b3299011f407fab894a0e90f107ee702ab842c17a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-07-27T06:55:32.821 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:32 smithi029 podman[109979]: 2024-07-27 06:55:32.551456958 +0000 UTC m=+0.958471663 container remove 15fafc159898709efc11d2b3299011f407fab894a0e90f107ee702ab842c17a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True) 2024-07-27T06:55:32.821 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:32 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:55:32.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:32 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:55:32.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:32 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:active 2024-07-27T06:55:32.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:32 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-27T06:55:32.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 354.358896 secs 2024-07-27T06:55:32.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:32.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:32.822 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:32.823 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:32 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:55:32.937 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:32 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:active 2024-07-27T06:55:32.937 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:32 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-27T06:55:32.937 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 354.358896 secs 2024-07-27T06:55:32.937 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:32.937 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:32.937 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:32.937 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:32 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:55:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:32 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/2836348309,v1:172.21.15.29:6835/2836348309] up:active 2024-07-27T06:55:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:32 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-27T06:55:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 354.358896 secs 2024-07-27T06:55:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:33.159 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:32 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-27T06:55:33.159 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:32 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Consumed 1.180s CPU time. 2024-07-27T06:55:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:33 smithi077 ceph-mon[37153]: pgmap v674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-27T06:55:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:33 smithi047 ceph-mon[36835]: pgmap v674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-27T06:55:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:33 smithi029 ceph-mon[32610]: pgmap v674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-27T06:55:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:34.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:55:34.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:55:34.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:55:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:55:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:55:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:55:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:55:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:35 smithi029 ceph-mon[32610]: pgmap v675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 205 B/s wr, 17 op/s 2024-07-27T06:55:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:35 smithi029 ceph-mon[32610]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-27T06:55:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:35 smithi077 ceph-mon[37153]: pgmap v675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 205 B/s wr, 17 op/s 2024-07-27T06:55:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:35 smithi077 ceph-mon[37153]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-27T06:55:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:35 smithi047 ceph-mon[36835]: pgmap v675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 205 B/s wr, 17 op/s 2024-07-27T06:55:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:35 smithi047 ceph-mon[36835]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-27T06:55:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:37 smithi077 ceph-mon[37153]: pgmap v676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 205 B/s wr, 19 op/s 2024-07-27T06:55:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 359.358978 secs 2024-07-27T06:55:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:37 smithi047 ceph-mon[36835]: pgmap v676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 205 B/s wr, 19 op/s 2024-07-27T06:55:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 359.358978 secs 2024-07-27T06:55:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:37 smithi029 ceph-mon[32610]: pgmap v676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 205 B/s wr, 19 op/s 2024-07-27T06:55:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 359.358978 secs 2024-07-27T06:55:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:38.352+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:55:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:38.352+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:55:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:38.352+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:55:38.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:38.352+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:55:38.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:38.352+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:55:38.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:38.352+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:55:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:39 smithi077 ceph-mon[37153]: pgmap v677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 198 B/s wr, 18 op/s 2024-07-27T06:55:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:39 smithi047 ceph-mon[36835]: pgmap v677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 198 B/s wr, 18 op/s 2024-07-27T06:55:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:39 smithi029 ceph-mon[32610]: pgmap v677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 198 B/s wr, 18 op/s 2024-07-27T06:55:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:41 smithi077 ceph-mon[37153]: pgmap v678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 B/s wr, 16 op/s 2024-07-27T06:55:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:41 smithi047 ceph-mon[36835]: pgmap v678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 B/s wr, 16 op/s 2024-07-27T06:55:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:41 smithi029 ceph-mon[32610]: pgmap v678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 B/s wr, 16 op/s 2024-07-27T06:55:42.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:42.678+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320005820 con 0x7f7348108670 2024-07-27T06:55:42.680 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:42.679+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320005820 con 0x7f7348108670 2024-07-27T06:55:42.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:42.681+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318005850 con 0x5645ed3392f0 2024-07-27T06:55:42.682 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:42.681+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318005850 con 0x5645ed3392f0 2024-07-27T06:55:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:42 smithi077 ceph-mon[37153]: 1 slow requests, 1 included below; oldest blocked for > 364.359064 secs 2024-07-27T06:55:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:42 smithi077 ceph-mon[37153]: slow request 364.359063 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T06:55:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:42 smithi047 ceph-mon[36835]: 1 slow requests, 1 included below; oldest blocked for > 364.359064 secs 2024-07-27T06:55:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:42 smithi047 ceph-mon[36835]: slow request 364.359063 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T06:55:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:42 smithi029 ceph-mon[32610]: 1 slow requests, 1 included below; oldest blocked for > 364.359064 secs 2024-07-27T06:55:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:42 smithi029 ceph-mon[32610]: slow request 364.359063 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T06:55:43.256 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:42 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-07-27T06:55:43.256 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:42 smithi029 systemd[1]: Stopped Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:55:43.256 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:42 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Consumed 1.180s CPU time. 2024-07-27T06:55:43.256 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:42 smithi029 systemd[1]: Starting Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968... 2024-07-27T06:55:43.256 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:43 smithi029 podman[110397]: 2024-07-27 06:55:43.113831912 +0000 UTC m=+0.018572399 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-27T06:55:43.521 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:43 smithi029 podman[110397]: 2024-07-27 06:55:43.255896468 +0000 UTC m=+0.160636955 container create d91ce3f82acca0048383ead862f346d526eda6ca21a9ca904aaa502c2cfda9d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-27T06:55:43.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:43 smithi047 ceph-mon[36835]: pgmap v679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 171 B/s wr, 16 op/s 2024-07-27T06:55:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:43 smithi077 ceph-mon[37153]: pgmap v679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 171 B/s wr, 16 op/s 2024-07-27T06:55:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:43 smithi029 ceph-mon[32610]: pgmap v679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 171 B/s wr, 16 op/s 2024-07-27T06:55:43.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:43 smithi029 podman[110397]: 2024-07-27 06:55:43.521266003 +0000 UTC m=+0.426006505 container init d91ce3f82acca0048383ead862f346d526eda6ca21a9ca904aaa502c2cfda9d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-07-27T06:55:43.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:43 smithi029 podman[110397]: 2024-07-27 06:55:43.524405205 +0000 UTC m=+0.429145698 container start d91ce3f82acca0048383ead862f346d526eda6ca21a9ca904aaa502c2cfda9d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-07-27T06:55:43.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:43 smithi029 ceph-mds[110416]: -- 172.21.15.29:0/2694894523 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b3b95b5860 con 0x55b3bd757a80 2024-07-27T06:55:43.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:43 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: starting mds.j at 2024-07-27T06:55:43.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:43 smithi029 bash[110397]: d91ce3f82acca0048383ead862f346d526eda6ca21a9ca904aaa502c2cfda9d5 2024-07-27T06:55:43.963 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:43 smithi029 systemd[1]: Started Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:55:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:44 smithi077 ceph-mon[37153]: daemon mds.j restarted 2024-07-27T06:55:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:44 smithi077 ceph-mon[37153]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:55:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:44 smithi077 ceph-mon[37153]: osdmap e105: 12 total, 12 up, 12 in 2024-07-27T06:55:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:44 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:boot 2024-07-27T06:55:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:44 smithi077 ceph-mon[37153]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:55:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:44 smithi077 ceph-mon[37153]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:55:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:55:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:44 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-27T06:55:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:44 smithi047 ceph-mon[36835]: daemon mds.j restarted 2024-07-27T06:55:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:44 smithi047 ceph-mon[36835]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:55:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:44 smithi047 ceph-mon[36835]: osdmap e105: 12 total, 12 up, 12 in 2024-07-27T06:55:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:44 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:boot 2024-07-27T06:55:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:44 smithi047 ceph-mon[36835]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:55:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:44 smithi047 ceph-mon[36835]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:55:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:55:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:44 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-27T06:55:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:44 smithi029 ceph-mon[32610]: daemon mds.j restarted 2024-07-27T06:55:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:44 smithi029 ceph-mon[32610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-27T06:55:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:44 smithi029 ceph-mon[32610]: osdmap e105: 12 total, 12 up, 12 in 2024-07-27T06:55:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:44 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:boot 2024-07-27T06:55:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:44 smithi029 ceph-mon[32610]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-07-27T06:55:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:44 smithi029 ceph-mon[32610]: fsmap cephfs:4/5 {1=c=up:active,2=f=up:active,3=b=up:active,4=l=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-27T06:55:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-27T06:55:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:44 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-27T06:55:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:45 smithi077 ceph-mon[37153]: pgmap v681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 op/s 2024-07-27T06:55:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:45 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:resolve 2024-07-27T06:55:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:45 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-27T06:55:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:45 smithi047 ceph-mon[36835]: pgmap v681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 op/s 2024-07-27T06:55:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:45 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:resolve 2024-07-27T06:55:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:45 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-27T06:55:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:45 smithi029 ceph-mon[32610]: pgmap v681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 op/s 2024-07-27T06:55:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:45 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:resolve 2024-07-27T06:55:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:45 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-27T06:55:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:46 smithi077 ceph-mon[37153]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:55:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:46 smithi077 ceph-mon[37153]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:55:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:46 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:reconnect 2024-07-27T06:55:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:46 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-27T06:55:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:46 smithi047 ceph-mon[36835]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:55:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:46 smithi047 ceph-mon[36835]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:55:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:46 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:reconnect 2024-07-27T06:55:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:46 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-27T06:55:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:46 smithi029 ceph-mon[32610]: reconnect by client.15402 v1:192.168.0.1:0/2451673514 after 0 2024-07-27T06:55:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:46 smithi029 ceph-mon[32610]: reconnect by client.15822 172.21.15.29:0/372739535 after 0.000999984 2024-07-27T06:55:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:46 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:reconnect 2024-07-27T06:55:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:46 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-27T06:55:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:47 smithi077 ceph-mon[37153]: pgmap v682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T06:55:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:47 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:rejoin 2024-07-27T06:55:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:47 smithi077 ceph-mon[37153]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-27T06:55:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:47 smithi077 ceph-mon[37153]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:55:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 369.359146 secs 2024-07-27T06:55:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:47 smithi047 ceph-mon[36835]: pgmap v682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T06:55:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:47 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:rejoin 2024-07-27T06:55:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:47 smithi047 ceph-mon[36835]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-27T06:55:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:47 smithi047 ceph-mon[36835]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:55:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 369.359146 secs 2024-07-27T06:55:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-mon[32610]: pgmap v682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T06:55:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:rejoin 2024-07-27T06:55:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-mon[32610]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-27T06:55:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-mon[32610]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-07-27T06:55:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 369.359146 secs 2024-07-27T06:55:47.963 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-27T06:55:47.963 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: *** Caught signal (Aborted) ** 2024-07-27T06:55:47.963 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: in thread 7f4fd9670640 thread_name:ms_dispatch 2024-07-27T06:55:47.963 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:55:47.963 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4fdea476f0] 2024-07-27T06:55:47.963 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4fdea9494c] 2024-07-27T06:55:47.963 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 3: raise() 2024-07-27T06:55:47.963 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 4: abort() 2024-07-27T06:55:47.963 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4fdea3171b] 2024-07-27T06:55:47.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 6: /lib64/libc.so.6(+0x37386) [0x7f4fdea40386] 2024-07-27T06:55:47.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b3b6eb16b9] 2024-07-27T06:55:47.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b3b706f80c] 2024-07-27T06:55:47.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b3b6fbdcde] 2024-07-27T06:55:47.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b3b6f7f763] 2024-07-27T06:55:47.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b3b6f80a1b] 2024-07-27T06:55:47.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b3b6f64752] 2024-07-27T06:55:47.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b3b6eb2a56] 2024-07-27T06:55:47.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b3b6eb6637] 2024-07-27T06:55:47.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b3b6eb6f5e] 2024-07-27T06:55:47.964 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b3b6e9a03a] 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4fdf314407] 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 18: (DispatchQueue::entry()+0x60f) [0x7f4fdf314cf9] 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4fdf3f15a1] 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 20: (Thread::entry_wrapper()+0x43) [0x7f4fdf19ecdd] 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4fdf19ecf9] 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4fdea92c02] 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4fdeb17c40] 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 2024-07-27T06:55:47.611+0000 7f4fd9670640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: in thread 7f4fd9670640 thread_name:ms_dispatch 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4fdea476f0] 2024-07-27T06:55:47.965 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4fdea9494c] 2024-07-27T06:55:47.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 3: raise() 2024-07-27T06:55:47.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 4: abort() 2024-07-27T06:55:47.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4fdea3171b] 2024-07-27T06:55:47.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 6: /lib64/libc.so.6(+0x37386) [0x7f4fdea40386] 2024-07-27T06:55:47.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b3b6eb16b9] 2024-07-27T06:55:47.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b3b706f80c] 2024-07-27T06:55:47.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b3b6fbdcde] 2024-07-27T06:55:47.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b3b6f7f763] 2024-07-27T06:55:47.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b3b6f80a1b] 2024-07-27T06:55:47.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b3b6f64752] 2024-07-27T06:55:47.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b3b6eb2a56] 2024-07-27T06:55:47.966 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b3b6eb6637] 2024-07-27T06:55:47.967 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b3b6eb6f5e] 2024-07-27T06:55:47.967 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b3b6e9a03a] 2024-07-27T06:55:47.967 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4fdf314407] 2024-07-27T06:55:47.967 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 18: (DispatchQueue::entry()+0x60f) [0x7f4fdf314cf9] 2024-07-27T06:55:47.967 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4fdf3f15a1] 2024-07-27T06:55:47.967 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 20: (Thread::entry_wrapper()+0x43) [0x7f4fdf19ecdd] 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4fdf19ecf9] 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4fdea92c02] 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4fdeb17c40] 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 0> 2024-07-27T06:55:47.611+0000 7f4fd9670640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: in thread 7f4fd9670640 thread_name:ms_dispatch 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4fdea476f0] 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4fdea9494c] 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 3: raise() 2024-07-27T06:55:47.968 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 4: abort() 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4fdea3171b] 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 6: /lib64/libc.so.6(+0x37386) [0x7f4fdea40386] 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b3b6eb16b9] 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b3b706f80c] 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b3b6fbdcde] 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b3b6f7f763] 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b3b6f80a1b] 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b3b6f64752] 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b3b6eb2a56] 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b3b6eb6637] 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b3b6eb6f5e] 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b3b6e9a03a] 2024-07-27T06:55:47.969 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4fdf314407] 2024-07-27T06:55:47.970 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 18: (DispatchQueue::entry()+0x60f) [0x7f4fdf314cf9] 2024-07-27T06:55:47.970 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4fdf3f15a1] 2024-07-27T06:55:47.970 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 20: (Thread::entry_wrapper()+0x43) [0x7f4fdf19ecdd] 2024-07-27T06:55:47.970 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4fdf19ecf9] 2024-07-27T06:55:47.970 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4fdea92c02] 2024-07-27T06:55:47.970 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4fdeb17c40] 2024-07-27T06:55:47.970 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:55:47.970 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 2024-07-27T06:55:47.970 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 0> 2024-07-27T06:55:47.611+0000 7f4fd9670640 -1 *** Caught signal (Aborted) ** 2024-07-27T06:55:47.970 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: in thread 7f4fd9670640 thread_name:ms_dispatch 2024-07-27T06:55:47.970 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 2024-07-27T06:55:47.970 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4fdea476f0] 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4fdea9494c] 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 3: raise() 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 4: abort() 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4fdea3171b] 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 6: /lib64/libc.so.6(+0x37386) [0x7f4fdea40386] 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b3b6eb16b9] 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b3b706f80c] 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55b3b6fbdcde] 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55b3b6f7f763] 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b3b6f80a1b] 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b3b6f64752] 2024-07-27T06:55:47.971 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b3b6eb2a56] 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b3b6eb6637] 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b3b6eb6f5e] 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b3b6e9a03a] 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4fdf314407] 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 18: (DispatchQueue::entry()+0x60f) [0x7f4fdf314cf9] 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4fdf3f15a1] 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 20: (Thread::entry_wrapper()+0x43) [0x7f4fdf19ecdd] 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4fdf19ecf9] 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4fdea92c02] 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4fdeb17c40] 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j[110412]: 2024-07-27T06:55:47.972 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:47 smithi029 podman[110451]: 2024-07-27 06:55:47.682429467 +0000 UTC m=+0.020304551 container died d91ce3f82acca0048383ead862f346d526eda6ca21a9ca904aaa502c2cfda9d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS) 2024-07-27T06:55:48.680 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:48 smithi029 podman[110451]: 2024-07-27 06:55:48.420221238 +0000 UTC m=+0.758096324 container remove d91ce3f82acca0048383ead862f346d526eda6ca21a9ca904aaa502c2cfda9d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968-mds-j, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True) 2024-07-27T06:55:48.680 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:48 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-27T06:55:48.680 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:48 smithi029 ceph-mon[32610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:55:48.680 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:48 smithi029 ceph-mon[32610]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:active 2024-07-27T06:55:48.680 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:48 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-27T06:55:48.680 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:48 smithi029 ceph-mon[32610]: pgmap v683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:55:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:48 smithi077 ceph-mon[37153]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:55:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:48 smithi077 ceph-mon[37153]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:active 2024-07-27T06:55:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:48 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-27T06:55:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:48 smithi077 ceph-mon[37153]: pgmap v683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:55:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:48 smithi047 ceph-mon[36835]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-27T06:55:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:48 smithi047 ceph-mon[36835]: mds.? [v2:172.21.15.29:6834/1620428612,v1:172.21.15.29:6835/1620428612] up:active 2024-07-27T06:55:48.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:48 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-27T06:55:48.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:48 smithi047 ceph-mon[36835]: pgmap v683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 102 B/s wr, 14 op/s 2024-07-27T06:55:48.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:48 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-27T06:55:48.962 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:48 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Consumed 1.129s CPU time. 2024-07-27T06:55:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:50 smithi047 ceph-mon[36835]: pgmap v684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:55:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:50 smithi029 ceph-mon[32610]: pgmap v684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:55:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:50 smithi077 ceph-mon[37153]: pgmap v684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 102 B/s wr, 15 op/s 2024-07-27T06:55:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:51 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 374.359235 secs 2024-07-27T06:55:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:51 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 374.359235 secs 2024-07-27T06:55:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:51 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 374.359235 secs 2024-07-27T06:55:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:52 smithi047 ceph-mon[36835]: pgmap v685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 17 op/s 2024-07-27T06:55:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:52 smithi029 ceph-mon[32610]: pgmap v685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 17 op/s 2024-07-27T06:55:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:52 smithi077 ceph-mon[37153]: pgmap v685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 204 B/s wr, 17 op/s 2024-07-27T06:55:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:53.357+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:55:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:53.357+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:55:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:55:53.357+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:55:53.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:53.357+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:55:53.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:53.357+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:55:53.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:53.357+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:55:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:54 smithi047 ceph-mon[36835]: pgmap v686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 199 B/s wr, 18 op/s 2024-07-27T06:55:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:54 smithi029 ceph-mon[32610]: pgmap v686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 199 B/s wr, 18 op/s 2024-07-27T06:55:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:54 smithi077 ceph-mon[37153]: pgmap v686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 199 B/s wr, 18 op/s 2024-07-27T06:55:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:56 smithi047 ceph-mon[36835]: pgmap v687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:55:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:56 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 379.359363 secs 2024-07-27T06:55:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:56 smithi029 ceph-mon[32610]: pgmap v687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:55:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:56 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 379.359363 secs 2024-07-27T06:55:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:56 smithi077 ceph-mon[37153]: pgmap v687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:55:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:56 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 379.359363 secs 2024-07-27T06:55:57.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:57.692+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24005820 con 0x7eff50104ab0 2024-07-27T06:55:57.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:57.692+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24005820 con 0x7eff50104ab0 2024-07-27T06:55:57.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:57.694+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28005850 con 0x55a2cdf3b350 2024-07-27T06:55:57.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:55:57.695+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28005850 con 0x55a2cdf3b350 2024-07-27T06:55:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:55:58 smithi047 ceph-mon[36835]: pgmap v688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:55:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:55:58 smithi029 ceph-mon[32610]: pgmap v688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:55:59.212 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:58 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-07-27T06:55:59.212 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:58 smithi029 systemd[1]: Stopped Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:55:59.212 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:58 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Consumed 1.129s CPU time. 2024-07-27T06:55:59.212 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:58 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Start request repeated too quickly. 2024-07-27T06:55:59.213 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:58 smithi029 systemd[1]: ceph-0a8a0a4c-4be2-11ef-bcc7-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-27T06:55:59.213 INFO:journalctl@ceph.mds.j.smithi029.stdout:Jul 27 06:55:58 smithi029 systemd[1]: Failed to start Ceph mds.j for 0a8a0a4c-4be2-11ef-bcc7-c7b262605968. 2024-07-27T06:55:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:55:58 smithi077 ceph-mon[37153]: pgmap v688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 170 B/s wr, 17 op/s 2024-07-27T06:56:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:00 smithi047 ceph-mon[36835]: pgmap v689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-07-27T06:56:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:00 smithi029 ceph-mon[32610]: pgmap v689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-07-27T06:56:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:00 smithi077 ceph-mon[37153]: pgmap v689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-07-27T06:56:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:01 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 384.359505 secs 2024-07-27T06:56:02.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:01 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 384.359505 secs 2024-07-27T06:56:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:01 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 384.359505 secs 2024-07-27T06:56:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:03 smithi077 ceph-mon[37153]: pgmap v690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-27T06:56:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:03 smithi077 ceph-mon[37153]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-27T06:56:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:03 smithi047 ceph-mon[36835]: pgmap v690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-27T06:56:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:03 smithi047 ceph-mon[36835]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-27T06:56:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:03 smithi029 ceph-mon[32610]: pgmap v690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-27T06:56:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:03 smithi029 ceph-mon[32610]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-27T06:56:05.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:05 smithi077 ceph-mon[37153]: pgmap v691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:56:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:05 smithi047 ceph-mon[36835]: pgmap v691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:56:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:05 smithi029 ceph-mon[32610]: pgmap v691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T06:56:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:07 smithi077 ceph-mon[37153]: pgmap v692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 389.359628 secs 2024-07-27T06:56:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:07 smithi047 ceph-mon[36835]: pgmap v692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 389.359628 secs 2024-07-27T06:56:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:07 smithi029 ceph-mon[32610]: pgmap v692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 389.359628 secs 2024-07-27T06:56:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:56:08.371+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:56:08.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:56:08.371+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:56:08.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:56:08.371+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:56:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:56:08.373+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:56:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:56:08.373+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:56:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:56:08.373+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:56:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:09 smithi077 ceph-mon[37153]: pgmap v693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:56:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:09 smithi047 ceph-mon[36835]: pgmap v693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:56:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:09 smithi029 ceph-mon[32610]: pgmap v693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:56:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:11 smithi077 ceph-mon[37153]: pgmap v694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:11 smithi047 ceph-mon[36835]: pgmap v694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:11 smithi029 ceph-mon[32610]: pgmap v694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 394.359747 secs 2024-07-27T06:56:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 394.359747 secs 2024-07-27T06:56:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 394.359747 secs 2024-07-27T06:56:13.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:13 smithi047 ceph-mon[36835]: pgmap v695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:56:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:13 smithi077 ceph-mon[37153]: pgmap v695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:56:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:13 smithi029 ceph-mon[32610]: pgmap v695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:56:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:15 smithi077 ceph-mon[37153]: pgmap v696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:56:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:15 smithi047 ceph-mon[36835]: pgmap v696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:56:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:15 smithi029 ceph-mon[32610]: pgmap v696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:56:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:17 smithi077 ceph-mon[37153]: pgmap v697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 399.359862 secs 2024-07-27T06:56:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:17 smithi047 ceph-mon[36835]: pgmap v697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 399.359862 secs 2024-07-27T06:56:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:17 smithi029 ceph-mon[32610]: pgmap v697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 399.359862 secs 2024-07-27T06:56:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:19 smithi077 ceph-mon[37153]: pgmap v698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T06:56:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:19 smithi047 ceph-mon[36835]: pgmap v698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T06:56:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:19 smithi029 ceph-mon[32610]: pgmap v698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T06:56:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:21 smithi077 ceph-mon[37153]: pgmap v699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T06:56:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:21 smithi047 ceph-mon[36835]: pgmap v699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T06:56:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:21 smithi029 ceph-mon[32610]: pgmap v699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T06:56:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 404.359931 secs 2024-07-27T06:56:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 404.359931 secs 2024-07-27T06:56:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 404.359931 secs 2024-07-27T06:56:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:56:23.386+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:56:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:56:23.386+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:56:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:56:23.386+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:56:23.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:56:23.388+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:56:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:56:23.388+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:56:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:56:23.388+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:56:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:23 smithi029 ceph-mon[32610]: pgmap v700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:56:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:23 smithi077 ceph-mon[37153]: pgmap v700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:56:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:23 smithi047 ceph-mon[36835]: pgmap v700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:56:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:25 smithi077 ceph-mon[37153]: pgmap v701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T06:56:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:25 smithi047 ceph-mon[36835]: pgmap v701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T06:56:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:25 smithi029 ceph-mon[32610]: pgmap v701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T06:56:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:27 smithi077 ceph-mon[37153]: pgmap v702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T06:56:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 409.360020 secs 2024-07-27T06:56:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:27 smithi047 ceph-mon[36835]: pgmap v702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T06:56:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 409.360020 secs 2024-07-27T06:56:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:27 smithi029 ceph-mon[32610]: pgmap v702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T06:56:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 409.360020 secs 2024-07-27T06:56:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:29 smithi077 ceph-mon[37153]: pgmap v703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T06:56:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:29 smithi047 ceph-mon[36835]: pgmap v703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T06:56:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:29 smithi029 ceph-mon[32610]: pgmap v703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T06:56:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:31 smithi077 ceph-mon[37153]: pgmap v704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:31 smithi047 ceph-mon[36835]: pgmap v704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:31 smithi029 ceph-mon[32610]: pgmap v704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 414.360094 secs 2024-07-27T06:56:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 414.360094 secs 2024-07-27T06:56:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 414.360094 secs 2024-07-27T06:56:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:33 smithi077 ceph-mon[37153]: pgmap v705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:56:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:33 smithi047 ceph-mon[36835]: pgmap v705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:56:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:33 smithi029 ceph-mon[32610]: pgmap v705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:56:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:56:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:56:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:56:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:35 smithi077 ceph-mon[37153]: pgmap v706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:56:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:56:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:56:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:35 smithi047 ceph-mon[36835]: pgmap v706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:56:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:56:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:56:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:35 smithi029 ceph-mon[32610]: pgmap v706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:56:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:56:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:56:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:56:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:36 smithi077 ceph-mon[37153]: pgmap v707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:56:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:36 smithi077 ceph-mon[37153]: pgmap v708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T06:56:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:36 smithi047 ceph-mon[36835]: pgmap v707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:56:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:36 smithi047 ceph-mon[36835]: pgmap v708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T06:56:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:36 smithi029 ceph-mon[32610]: pgmap v707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:56:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:36 smithi029 ceph-mon[32610]: pgmap v708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T06:56:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 419.360172 secs 2024-07-27T06:56:37.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 419.360172 secs 2024-07-27T06:56:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 419.360172 secs 2024-07-27T06:56:38.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:56:38.391+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:56:38.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:56:38.391+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:56:38.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:56:38.391+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:56:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:56:38.400+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:56:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:56:38.400+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:56:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:56:38.400+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:56:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:38 smithi029 ceph-mon[32610]: pgmap v709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T06:56:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:38 smithi077 ceph-mon[37153]: pgmap v709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T06:56:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:38 smithi047 ceph-mon[36835]: pgmap v709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T06:56:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:40 smithi077 ceph-mon[37153]: pgmap v710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T06:56:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:40 smithi047 ceph-mon[36835]: pgmap v710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T06:56:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:40 smithi029 ceph-mon[32610]: pgmap v710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T06:56:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:42 smithi077 ceph-mon[37153]: pgmap v711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T06:56:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 424.360257 secs 2024-07-27T06:56:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:42 smithi047 ceph-mon[36835]: pgmap v711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T06:56:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 424.360257 secs 2024-07-27T06:56:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:42 smithi029 ceph-mon[32610]: pgmap v711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T06:56:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 424.360257 secs 2024-07-27T06:56:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:44 smithi077 ceph-mon[37153]: pgmap v712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:56:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:44 smithi047 ceph-mon[36835]: pgmap v712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:56:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:44 smithi029 ceph-mon[32610]: pgmap v712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T06:56:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:46 smithi077 ceph-mon[37153]: pgmap v713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T06:56:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:46 smithi047 ceph-mon[36835]: pgmap v713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T06:56:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:46 smithi029 ceph-mon[32610]: pgmap v713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T06:56:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 429.360341 secs 2024-07-27T06:56:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 429.360341 secs 2024-07-27T06:56:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 429.360341 secs 2024-07-27T06:56:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:48 smithi077 ceph-mon[37153]: pgmap v714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:56:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:48 smithi047 ceph-mon[36835]: pgmap v714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:56:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:48 smithi029 ceph-mon[32610]: pgmap v714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:56:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:49 smithi047 ceph-mon[36835]: pgmap v715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:56:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:49 smithi029 ceph-mon[32610]: pgmap v715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:56:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:49 smithi077 ceph-mon[37153]: pgmap v715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:56:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:51 smithi077 ceph-mon[37153]: pgmap v716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:51 smithi047 ceph-mon[36835]: pgmap v716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:51 smithi029 ceph-mon[32610]: pgmap v716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:56:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 434.360483 secs 2024-07-27T06:56:53.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 434.360483 secs 2024-07-27T06:56:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 434.360483 secs 2024-07-27T06:56:53.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:56:53.399+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:56:53.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:56:53.399+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:56:53.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:56:53.399+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:56:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:56:53.415+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:56:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:56:53.415+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:56:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:56:53.415+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:56:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:53 smithi077 ceph-mon[37153]: pgmap v717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:56:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:53 smithi047 ceph-mon[36835]: pgmap v717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:56:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:53 smithi029 ceph-mon[32610]: pgmap v717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:56:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:55 smithi077 ceph-mon[37153]: pgmap v718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:56:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:55 smithi047 ceph-mon[36835]: pgmap v718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:56:56.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:55 smithi029 ceph-mon[32610]: pgmap v718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:56:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 439.360623 secs 2024-07-27T06:56:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:57 smithi077 ceph-mon[37153]: pgmap v719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:56:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 439.360623 secs 2024-07-27T06:56:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:57 smithi047 ceph-mon[36835]: pgmap v719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:56:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 439.360623 secs 2024-07-27T06:56:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:57 smithi029 ceph-mon[32610]: pgmap v719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:56:59 smithi077 ceph-mon[37153]: pgmap v720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:57:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:56:59 smithi047 ceph-mon[36835]: pgmap v720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:57:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:56:59 smithi029 ceph-mon[32610]: pgmap v720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:57:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:01 smithi077 ceph-mon[37153]: pgmap v721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:57:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:01 smithi047 ceph-mon[36835]: pgmap v721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:57:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:01 smithi029 ceph-mon[32610]: pgmap v721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:57:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 444.360761 secs 2024-07-27T06:57:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 444.360761 secs 2024-07-27T06:57:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 444.360761 secs 2024-07-27T06:57:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:03 smithi077 ceph-mon[37153]: pgmap v722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T06:57:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:03 smithi047 ceph-mon[36835]: pgmap v722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T06:57:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:03 smithi029 ceph-mon[32610]: pgmap v722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T06:57:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:05 smithi077 ceph-mon[37153]: pgmap v723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T06:57:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:05 smithi047 ceph-mon[36835]: pgmap v723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T06:57:06.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:05 smithi029 ceph-mon[32610]: pgmap v723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T06:57:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 449.360879 secs 2024-07-27T06:57:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:07 smithi077 ceph-mon[37153]: pgmap v724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 449.360879 secs 2024-07-27T06:57:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:07 smithi047 ceph-mon[36835]: pgmap v724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 449.360879 secs 2024-07-27T06:57:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:07 smithi029 ceph-mon[32610]: pgmap v724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:08.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:57:08.414+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:57:08.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:57:08.414+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:57:08.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:57:08.414+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:57:08.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:08.423+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:57:08.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:08.423+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:57:08.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:08.423+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:57:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:09 smithi077 ceph-mon[37153]: pgmap v725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T06:57:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:09 smithi047 ceph-mon[36835]: pgmap v725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T06:57:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:09 smithi029 ceph-mon[32610]: pgmap v725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T06:57:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:11 smithi077 ceph-mon[37153]: pgmap v726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T06:57:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:11 smithi047 ceph-mon[36835]: pgmap v726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T06:57:12.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:11 smithi029 ceph-mon[32610]: pgmap v726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T06:57:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 454.360951 secs 2024-07-27T06:57:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 454.360951 secs 2024-07-27T06:57:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 454.360951 secs 2024-07-27T06:57:14.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:13 smithi047 ceph-mon[36835]: pgmap v727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T06:57:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:13 smithi077 ceph-mon[37153]: pgmap v727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T06:57:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:13 smithi029 ceph-mon[32610]: pgmap v727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T06:57:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:15 smithi077 ceph-mon[37153]: pgmap v728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:57:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:15 smithi047 ceph-mon[36835]: pgmap v728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:57:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:15 smithi029 ceph-mon[32610]: pgmap v728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:57:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:16 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 459.361036 secs 2024-07-27T06:57:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:16 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 459.361036 secs 2024-07-27T06:57:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:16 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 459.361036 secs 2024-07-27T06:57:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:17 smithi077 ceph-mon[37153]: pgmap v729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:17 smithi047 ceph-mon[36835]: pgmap v729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:17 smithi029 ceph-mon[32610]: pgmap v729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:19 smithi077 ceph-mon[37153]: pgmap v730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:57:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:19 smithi047 ceph-mon[36835]: pgmap v730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:57:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:19 smithi029 ceph-mon[32610]: pgmap v730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:57:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:21 smithi077 ceph-mon[37153]: pgmap v731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:57:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:21 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 464.361120 secs 2024-07-27T06:57:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:21 smithi047 ceph-mon[36835]: pgmap v731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:57:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:21 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 464.361120 secs 2024-07-27T06:57:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:21 smithi029 ceph-mon[32610]: pgmap v731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:57:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:21 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 464.361120 secs 2024-07-27T06:57:23.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:57:23.424+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:57:23.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:57:23.424+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:57:23.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:57:23.424+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:57:23.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:23.439+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:57:23.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:23.439+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:57:23.439 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:23.439+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:57:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:23 smithi077 ceph-mon[37153]: pgmap v732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:57:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:23 smithi047 ceph-mon[36835]: pgmap v732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:57:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:23 smithi029 ceph-mon[32610]: pgmap v732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:57:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:25 smithi077 ceph-mon[37153]: pgmap v733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:57:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:25 smithi047 ceph-mon[36835]: pgmap v733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:57:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:25 smithi029 ceph-mon[32610]: pgmap v733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:57:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:26 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 469.361191 secs 2024-07-27T06:57:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:26 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 469.361191 secs 2024-07-27T06:57:27.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:26 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 469.361191 secs 2024-07-27T06:57:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:27 smithi077 ceph-mon[37153]: pgmap v734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:27 smithi047 ceph-mon[36835]: pgmap v734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:27 smithi029 ceph-mon[32610]: pgmap v734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:29 smithi077 ceph-mon[37153]: pgmap v735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:57:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:29 smithi047 ceph-mon[36835]: pgmap v735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:57:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:29 smithi029 ceph-mon[32610]: pgmap v735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:57:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:31 smithi077 ceph-mon[37153]: pgmap v736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:57:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:31 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 474.361255 secs 2024-07-27T06:57:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:31 smithi047 ceph-mon[36835]: pgmap v736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:57:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:31 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 474.361255 secs 2024-07-27T06:57:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:31 smithi029 ceph-mon[32610]: pgmap v736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:57:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:31 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 474.361255 secs 2024-07-27T06:57:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:33 smithi077 ceph-mon[37153]: pgmap v737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:57:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:33 smithi047 ceph-mon[36835]: pgmap v737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:57:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:33 smithi029 ceph-mon[32610]: pgmap v737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:57:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:57:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:57:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:57:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:35 smithi077 ceph-mon[37153]: pgmap v738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:57:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:57:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:57:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:35 smithi077 ceph-mon[37153]: pgmap v739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T06:57:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:35 smithi077 ceph-mon[37153]: pgmap v740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T06:57:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:57:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:35 smithi047 ceph-mon[36835]: pgmap v738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:57:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:57:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:57:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:35 smithi047 ceph-mon[36835]: pgmap v739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T06:57:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:35 smithi047 ceph-mon[36835]: pgmap v740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T06:57:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:57:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:35 smithi029 ceph-mon[32610]: pgmap v738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:57:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:57:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:57:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:35 smithi029 ceph-mon[32610]: pgmap v739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T06:57:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:35 smithi029 ceph-mon[32610]: pgmap v740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T06:57:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:57:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:36 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 479.361319 secs 2024-07-27T06:57:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:36 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 479.361319 secs 2024-07-27T06:57:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:36 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 479.361319 secs 2024-07-27T06:57:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:37 smithi077 ceph-mon[37153]: pgmap v741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T06:57:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:37 smithi047 ceph-mon[36835]: pgmap v741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T06:57:38.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:37 smithi029 ceph-mon[32610]: pgmap v741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T06:57:38.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:57:38.439+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:57:38.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:57:38.439+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:57:38.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:57:38.439+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:57:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:38.453+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:57:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:38.453+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:57:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:38.453+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:57:40.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:40 smithi077 ceph-mon[37153]: pgmap v742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T06:57:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:40 smithi047 ceph-mon[36835]: pgmap v742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T06:57:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:40 smithi029 ceph-mon[32610]: pgmap v742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T06:57:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:42 smithi077 ceph-mon[37153]: pgmap v743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T06:57:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 484.361434 secs 2024-07-27T06:57:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:42 smithi047 ceph-mon[36835]: pgmap v743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T06:57:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 484.361434 secs 2024-07-27T06:57:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:42 smithi029 ceph-mon[32610]: pgmap v743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T06:57:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 484.361434 secs 2024-07-27T06:57:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:44 smithi077 ceph-mon[37153]: pgmap v744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T06:57:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:44 smithi047 ceph-mon[36835]: pgmap v744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T06:57:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:44 smithi029 ceph-mon[32610]: pgmap v744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T06:57:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:46 smithi077 ceph-mon[37153]: pgmap v745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T06:57:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:46 smithi047 ceph-mon[36835]: pgmap v745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T06:57:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:46 smithi029 ceph-mon[32610]: pgmap v745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T06:57:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 489.361511 secs 2024-07-27T06:57:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 489.361511 secs 2024-07-27T06:57:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 489.361511 secs 2024-07-27T06:57:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:48 smithi077 ceph-mon[37153]: pgmap v746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:57:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:48 smithi047 ceph-mon[36835]: pgmap v746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:57:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:48 smithi029 ceph-mon[32610]: pgmap v746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:57:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:50 smithi077 ceph-mon[37153]: pgmap v747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:57:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:50 smithi047 ceph-mon[36835]: pgmap v747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:57:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:50 smithi029 ceph-mon[32610]: pgmap v747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:57:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:52 smithi077 ceph-mon[37153]: pgmap v748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 494.361600 secs 2024-07-27T06:57:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:52 smithi047 ceph-mon[36835]: pgmap v748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 494.361600 secs 2024-07-27T06:57:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:52 smithi029 ceph-mon[32610]: pgmap v748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:57:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 494.361600 secs 2024-07-27T06:57:53.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:57:53.449+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:57:53.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:57:53.449+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:57:53.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:57:53.449+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:57:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:53.469+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:57:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:53.469+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:57:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:53.469+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:57:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:54 smithi077 ceph-mon[37153]: pgmap v749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:57:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:54 smithi047 ceph-mon[36835]: pgmap v749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:57:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:54 smithi029 ceph-mon[32610]: pgmap v749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:57:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:56 smithi077 ceph-mon[37153]: pgmap v750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:57:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:56 smithi047 ceph-mon[36835]: pgmap v750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:57:56.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:56 smithi029 ceph-mon[32610]: pgmap v750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:57:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 499.361679 secs 2024-07-27T06:57:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 499.361679 secs 2024-07-27T06:57:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:57.700+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7320005b60 con 0x7f7348108670 2024-07-27T06:57:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:57.700+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318005b30 con 0x5645ed3392f0 2024-07-27T06:57:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:57.700+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7320005b60 con 0x7f7348108670 2024-07-27T06:57:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:57:57.700+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318005b30 con 0x5645ed3392f0 2024-07-27T06:57:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 499.361679 secs 2024-07-27T06:57:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:57:58 smithi077 ceph-mon[37153]: pgmap v751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:57:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:57:58 smithi047 ceph-mon[36835]: pgmap v751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:57:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:57:58 smithi029 ceph-mon[32610]: pgmap v751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:58:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:00 smithi077 ceph-mon[37153]: pgmap v752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:58:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:00 smithi047 ceph-mon[36835]: pgmap v752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:58:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:00 smithi029 ceph-mon[32610]: pgmap v752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:58:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:02 smithi077 ceph-mon[37153]: pgmap v753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:58:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 504.361752 secs 2024-07-27T06:58:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:02 smithi047 ceph-mon[36835]: pgmap v753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:58:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 504.361752 secs 2024-07-27T06:58:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:02 smithi029 ceph-mon[32610]: pgmap v753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:58:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 504.361752 secs 2024-07-27T06:58:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:04 smithi077 ceph-mon[37153]: pgmap v754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:04 smithi047 ceph-mon[36835]: pgmap v754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:04 smithi029 ceph-mon[32610]: pgmap v754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:06 smithi077 ceph-mon[37153]: pgmap v755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:06 smithi047 ceph-mon[36835]: pgmap v755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:06 smithi029 ceph-mon[32610]: pgmap v755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 509.361810 secs 2024-07-27T06:58:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 509.361810 secs 2024-07-27T06:58:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 509.361810 secs 2024-07-27T06:58:08.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:08.463+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:58:08.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:08.464+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:58:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:08.464+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:58:08.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:08.478+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:58:08.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:08.478+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:58:08.479 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:08.478+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:58:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:08 smithi077 ceph-mon[37153]: pgmap v756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:58:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:08 smithi047 ceph-mon[36835]: pgmap v756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:58:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:08 smithi029 ceph-mon[32610]: pgmap v756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:58:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:10 smithi077 ceph-mon[37153]: pgmap v757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:58:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:10 smithi047 ceph-mon[36835]: pgmap v757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:58:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:10 smithi029 ceph-mon[32610]: pgmap v757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:58:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:12.712+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24005ac0 con 0x7eff50104ab0 2024-07-27T06:58:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:12.712+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24005ac0 con 0x7eff50104ab0 2024-07-27T06:58:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:12.713+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28005b30 con 0x55a2cdf3b350 2024-07-27T06:58:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:12.714+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28005b30 con 0x55a2cdf3b350 2024-07-27T06:58:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:12 smithi077 ceph-mon[37153]: pgmap v758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:58:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 514.361885 secs 2024-07-27T06:58:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:12 smithi047 ceph-mon[36835]: pgmap v758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:58:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 514.361885 secs 2024-07-27T06:58:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:12 smithi029 ceph-mon[32610]: pgmap v758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:58:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 514.361885 secs 2024-07-27T06:58:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:14 smithi077 ceph-mon[37153]: pgmap v759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:14 smithi047 ceph-mon[36835]: pgmap v759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:14 smithi029 ceph-mon[32610]: pgmap v759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:16 smithi077 ceph-mon[37153]: pgmap v760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:16 smithi047 ceph-mon[36835]: pgmap v760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:16 smithi029 ceph-mon[32610]: pgmap v760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 519.361960 secs 2024-07-27T06:58:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 519.361960 secs 2024-07-27T06:58:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 519.361960 secs 2024-07-27T06:58:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:18 smithi077 ceph-mon[37153]: pgmap v761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:58:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:18 smithi047 ceph-mon[36835]: pgmap v761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:58:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:18 smithi029 ceph-mon[32610]: pgmap v761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:58:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:20 smithi077 ceph-mon[37153]: pgmap v762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:58:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:20 smithi047 ceph-mon[36835]: pgmap v762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:58:20.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:20 smithi029 ceph-mon[32610]: pgmap v762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:58:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:22 smithi077 ceph-mon[37153]: pgmap v763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:58:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 524.362024 secs 2024-07-27T06:58:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:22 smithi047 ceph-mon[36835]: pgmap v763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:58:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 524.362024 secs 2024-07-27T06:58:22.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:22 smithi029 ceph-mon[32610]: pgmap v763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:58:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 524.362024 secs 2024-07-27T06:58:23.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:23.474+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:58:23.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:23.474+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:58:23.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:23.474+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:58:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:23.493+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:58:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:23.493+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:58:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:23.493+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:58:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:24 smithi077 ceph-mon[37153]: pgmap v764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:24 smithi047 ceph-mon[36835]: pgmap v764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:24 smithi029 ceph-mon[32610]: pgmap v764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:26 smithi077 ceph-mon[37153]: pgmap v765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:26 smithi047 ceph-mon[36835]: pgmap v765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:26 smithi029 ceph-mon[32610]: pgmap v765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 529.362143 secs 2024-07-27T06:58:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 529.362143 secs 2024-07-27T06:58:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 529.362143 secs 2024-07-27T06:58:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:28 smithi077 ceph-mon[37153]: pgmap v766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:58:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:28 smithi047 ceph-mon[36835]: pgmap v766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:58:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:28 smithi029 ceph-mon[32610]: pgmap v766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:58:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:30 smithi077 ceph-mon[37153]: pgmap v767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:58:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:30 smithi047 ceph-mon[36835]: pgmap v767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:58:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:30 smithi029 ceph-mon[32610]: pgmap v767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:58:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:32 smithi077 ceph-mon[37153]: pgmap v768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:58:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 534.362243 secs 2024-07-27T06:58:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:32 smithi047 ceph-mon[36835]: pgmap v768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:58:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 534.362243 secs 2024-07-27T06:58:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:32 smithi029 ceph-mon[32610]: pgmap v768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:58:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 534.362243 secs 2024-07-27T06:58:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:34 smithi077 ceph-mon[37153]: pgmap v769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:34 smithi047 ceph-mon[36835]: pgmap v769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:34 smithi029 ceph-mon[32610]: pgmap v769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:58:35.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:58:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:58:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:36 smithi047 ceph-mon[36835]: pgmap v770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:58:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:58:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:58:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:36 smithi029 ceph-mon[32610]: pgmap v770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:58:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:58:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:58:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:36 smithi077 ceph-mon[37153]: pgmap v770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:58:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:58:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:58:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:37 smithi047 ceph-mon[36835]: pgmap v771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T06:58:37.951 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:37 smithi047 ceph-mon[36835]: pgmap v772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T06:58:37.952 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 539.362343 secs 2024-07-27T06:58:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:37 smithi029 ceph-mon[32610]: pgmap v771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T06:58:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:37 smithi029 ceph-mon[32610]: pgmap v772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T06:58:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 539.362343 secs 2024-07-27T06:58:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:37 smithi077 ceph-mon[37153]: pgmap v771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T06:58:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:37 smithi077 ceph-mon[37153]: pgmap v772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T06:58:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 539.362343 secs 2024-07-27T06:58:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:38.474+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:58:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:38.474+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:58:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:38.474+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:58:38.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:38.508+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:58:38.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:38.508+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:58:38.509 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:38.508+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:58:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:39 smithi047 ceph-mon[36835]: pgmap v773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T06:58:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:39 smithi029 ceph-mon[32610]: pgmap v773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T06:58:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:39 smithi077 ceph-mon[37153]: pgmap v773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T06:58:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:41 smithi047 ceph-mon[36835]: pgmap v774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T06:58:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:41 smithi029 ceph-mon[32610]: pgmap v774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T06:58:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:41 smithi077 ceph-mon[37153]: pgmap v774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T06:58:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:42.705+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318005e30 con 0x5645ed3392f0 2024-07-27T06:58:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:42.705+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320005e90 con 0x7f7348108670 2024-07-27T06:58:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:42.706+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320005e90 con 0x7f7348108670 2024-07-27T06:58:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:42.706+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318005e30 con 0x5645ed3392f0 2024-07-27T06:58:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 544.362458 secs 2024-07-27T06:58:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 544.362458 secs 2024-07-27T06:58:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 544.362458 secs 2024-07-27T06:58:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:43 smithi047 ceph-mon[36835]: pgmap v775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T06:58:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:43 smithi029 ceph-mon[32610]: pgmap v775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T06:58:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:43 smithi077 ceph-mon[37153]: pgmap v775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T06:58:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:44 smithi047 ceph-mon[36835]: pgmap v776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T06:58:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:44 smithi029 ceph-mon[32610]: pgmap v776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T06:58:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:44 smithi077 ceph-mon[37153]: pgmap v776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T06:58:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:46 smithi077 ceph-mon[37153]: pgmap v777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T06:58:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:46 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 549.362588 secs 2024-07-27T06:58:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:46 smithi047 ceph-mon[36835]: pgmap v777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T06:58:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:46 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 549.362588 secs 2024-07-27T06:58:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:46 smithi029 ceph-mon[32610]: pgmap v777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T06:58:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:46 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 549.362588 secs 2024-07-27T06:58:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:48 smithi077 ceph-mon[37153]: pgmap v778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:58:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:48 smithi047 ceph-mon[36835]: pgmap v778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:58:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:48 smithi029 ceph-mon[32610]: pgmap v778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:58:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:50 smithi077 ceph-mon[37153]: pgmap v779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:50 smithi047 ceph-mon[36835]: pgmap v779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:50 smithi029 ceph-mon[32610]: pgmap v779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:51 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 554.362718 secs 2024-07-27T06:58:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:51 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 554.362718 secs 2024-07-27T06:58:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:51 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 554.362718 secs 2024-07-27T06:58:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:52 smithi077 ceph-mon[37153]: pgmap v780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:58:53.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:52 smithi029 ceph-mon[32610]: pgmap v780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:58:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:52 smithi047 ceph-mon[36835]: pgmap v780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:58:53.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:53.485+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:58:53.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:53.485+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:58:53.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:53.485+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:58:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:53.523+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:58:53.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:53.523+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:58:53.525 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:58:53.523+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:58:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:54 smithi077 ceph-mon[37153]: pgmap v781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:55.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:54 smithi047 ceph-mon[36835]: pgmap v781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:54 smithi029 ceph-mon[32610]: pgmap v781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:58:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:56 smithi077 ceph-mon[37153]: pgmap v782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:56 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 559.362853 secs 2024-07-27T06:58:57.191 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:56 smithi029 ceph-mon[32610]: pgmap v782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:57.191 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:56 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 559.362853 secs 2024-07-27T06:58:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:56 smithi047 ceph-mon[36835]: pgmap v782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:58:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:56 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 559.362853 secs 2024-07-27T06:58:57.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:57.718+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24005e20 con 0x7eff50104ab0 2024-07-27T06:58:57.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:57.718+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24005e20 con 0x7eff50104ab0 2024-07-27T06:58:57.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:57.719+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28005e30 con 0x55a2cdf3b350 2024-07-27T06:58:57.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:58:57.719+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28005e30 con 0x55a2cdf3b350 2024-07-27T06:58:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:58:58 smithi077 ceph-mon[37153]: pgmap v783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:58:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:58:58 smithi047 ceph-mon[36835]: pgmap v783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:58:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:58:58 smithi029 ceph-mon[32610]: pgmap v783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:59:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:00 smithi047 ceph-mon[36835]: pgmap v784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:00 smithi029 ceph-mon[32610]: pgmap v784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:00 smithi077 ceph-mon[37153]: pgmap v784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:01 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 564.362994 secs 2024-07-27T06:59:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:01 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 564.362994 secs 2024-07-27T06:59:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:01 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 564.362994 secs 2024-07-27T06:59:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:02 smithi047 ceph-mon[36835]: pgmap v785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:59:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:02 smithi029 ceph-mon[32610]: pgmap v785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:59:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:02 smithi077 ceph-mon[37153]: pgmap v785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:59:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:04 smithi047 ceph-mon[36835]: pgmap v786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:05.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:04 smithi029 ceph-mon[32610]: pgmap v786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:04 smithi077 ceph-mon[37153]: pgmap v786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:06 smithi047 ceph-mon[36835]: pgmap v787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:06 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 569.363089 secs 2024-07-27T06:59:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:06 smithi029 ceph-mon[32610]: pgmap v787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:06 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 569.363089 secs 2024-07-27T06:59:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:06 smithi077 ceph-mon[37153]: pgmap v787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:06 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 569.363089 secs 2024-07-27T06:59:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:59:08.501+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:59:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:59:08.501+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:59:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:59:08.501+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:59:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:59:08.539+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:59:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:59:08.539+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:59:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:59:08.539+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:59:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:08 smithi047 ceph-mon[36835]: pgmap v788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:59:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:08 smithi029 ceph-mon[32610]: pgmap v788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:59:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:08 smithi077 ceph-mon[37153]: pgmap v788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:59:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:10 smithi047 ceph-mon[36835]: pgmap v789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:10 smithi029 ceph-mon[32610]: pgmap v789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:10 smithi077 ceph-mon[37153]: pgmap v789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:11 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 574.363214 secs 2024-07-27T06:59:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:11 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 574.363214 secs 2024-07-27T06:59:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:11 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 574.363214 secs 2024-07-27T06:59:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:12 smithi047 ceph-mon[36835]: pgmap v790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:59:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:12 smithi029 ceph-mon[32610]: pgmap v790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:59:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:12 smithi077 ceph-mon[37153]: pgmap v790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:59:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:14 smithi047 ceph-mon[36835]: pgmap v791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:14 smithi029 ceph-mon[32610]: pgmap v791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:14 smithi077 ceph-mon[37153]: pgmap v791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:16 smithi047 ceph-mon[36835]: pgmap v792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:17.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:16 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 579.363279 secs 2024-07-27T06:59:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:16 smithi029 ceph-mon[32610]: pgmap v792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:16 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 579.363279 secs 2024-07-27T06:59:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:16 smithi077 ceph-mon[37153]: pgmap v792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:16 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 579.363279 secs 2024-07-27T06:59:19.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:18 smithi077 ceph-mon[37153]: pgmap v793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:59:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:18 smithi047 ceph-mon[36835]: pgmap v793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:59:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:18 smithi029 ceph-mon[32610]: pgmap v793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:59:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:20 smithi047 ceph-mon[36835]: pgmap v794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:59:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:20 smithi029 ceph-mon[32610]: pgmap v794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:59:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:20 smithi077 ceph-mon[37153]: pgmap v794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T06:59:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:21 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 584.363431 secs 2024-07-27T06:59:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:21 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 584.363431 secs 2024-07-27T06:59:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:21 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 584.363431 secs 2024-07-27T06:59:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:59:23.517+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:59:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:59:23.517+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:59:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:59:23.517+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:59:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:59:23.540+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:59:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:59:23.540+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:59:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:59:23.541+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:59:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:23 smithi077 ceph-mon[37153]: pgmap v795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:59:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:23 smithi047 ceph-mon[36835]: pgmap v795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:59:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:23 smithi029 ceph-mon[32610]: pgmap v795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:59:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:25 smithi077 ceph-mon[37153]: pgmap v796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:25 smithi047 ceph-mon[36835]: pgmap v796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:25 smithi029 ceph-mon[32610]: pgmap v796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:27 smithi077 ceph-mon[37153]: pgmap v797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 589.363537 secs 2024-07-27T06:59:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:27 smithi047 ceph-mon[36835]: pgmap v797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 589.363537 secs 2024-07-27T06:59:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:27 smithi029 ceph-mon[32610]: pgmap v797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 589.363537 secs 2024-07-27T06:59:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:29 smithi077 ceph-mon[37153]: pgmap v798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:59:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:29 smithi047 ceph-mon[36835]: pgmap v798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:59:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:29 smithi029 ceph-mon[32610]: pgmap v798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T06:59:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:31 smithi077 ceph-mon[37153]: pgmap v799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:31 smithi047 ceph-mon[36835]: pgmap v799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:31 smithi029 ceph-mon[32610]: pgmap v799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 594.363675 secs 2024-07-27T06:59:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 594.363675 secs 2024-07-27T06:59:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 594.363675 secs 2024-07-27T06:59:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:33 smithi077 ceph-mon[37153]: pgmap v800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:59:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:33 smithi047 ceph-mon[36835]: pgmap v800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:59:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:33 smithi029 ceph-mon[32610]: pgmap v800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T06:59:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:35 smithi077 ceph-mon[37153]: pgmap v801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:35 smithi047 ceph-mon[36835]: pgmap v801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:35 smithi029 ceph-mon[32610]: pgmap v801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:59:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:59:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:59:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:59:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:59:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:59:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:59:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:59:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T06:59:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T06:59:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T06:59:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T06:59:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:37 smithi077 ceph-mon[37153]: pgmap v802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:37 smithi077 ceph-mon[37153]: pgmap v803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T06:59:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:37 smithi077 ceph-mon[37153]: pgmap v804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T06:59:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 599.363806 secs 2024-07-27T06:59:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:37 smithi047 ceph-mon[36835]: pgmap v802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:37 smithi047 ceph-mon[36835]: pgmap v803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T06:59:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:37 smithi047 ceph-mon[36835]: pgmap v804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T06:59:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 599.363806 secs 2024-07-27T06:59:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:37 smithi029 ceph-mon[32610]: pgmap v802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:37 smithi029 ceph-mon[32610]: pgmap v803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T06:59:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:37 smithi029 ceph-mon[32610]: pgmap v804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T06:59:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 599.363806 secs 2024-07-27T06:59:38.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:59:38.524+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:59:38.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:59:38.524+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:59:38.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:59:38.524+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:59:38.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:59:38.555+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:59:38.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:59:38.555+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:59:38.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:59:38.555+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:59:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:39 smithi077 ceph-mon[37153]: pgmap v805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T06:59:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:39 smithi047 ceph-mon[36835]: pgmap v805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T06:59:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:39 smithi029 ceph-mon[32610]: pgmap v805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T06:59:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:41 smithi077 ceph-mon[37153]: pgmap v806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T06:59:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:41 smithi047 ceph-mon[36835]: pgmap v806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T06:59:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:41 smithi029 ceph-mon[32610]: pgmap v806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T06:59:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 604.363937 secs 2024-07-27T06:59:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 604.363937 secs 2024-07-27T06:59:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 604.363937 secs 2024-07-27T06:59:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:43 smithi047 ceph-mon[36835]: pgmap v807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T06:59:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:43 smithi029 ceph-mon[32610]: pgmap v807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T06:59:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:43 smithi077 ceph-mon[37153]: pgmap v807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T06:59:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:45 smithi047 ceph-mon[36835]: pgmap v808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T06:59:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:45 smithi029 ceph-mon[32610]: pgmap v808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T06:59:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:45 smithi077 ceph-mon[37153]: pgmap v808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T06:59:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:47 smithi047 ceph-mon[36835]: pgmap v809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T06:59:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 609.364085 secs 2024-07-27T06:59:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:47 smithi029 ceph-mon[32610]: pgmap v809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T06:59:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 609.364085 secs 2024-07-27T06:59:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:47 smithi077 ceph-mon[37153]: pgmap v809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T06:59:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 609.364085 secs 2024-07-27T06:59:49.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:49 smithi077 ceph-mon[37153]: pgmap v810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:49 smithi047 ceph-mon[36835]: pgmap v810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:49 smithi029 ceph-mon[32610]: pgmap v810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:51 smithi047 ceph-mon[36835]: pgmap v811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:51.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:51 smithi029 ceph-mon[32610]: pgmap v811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:51 smithi077 ceph-mon[37153]: pgmap v811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T06:59:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 614.364229 secs 2024-07-27T06:59:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 614.364229 secs 2024-07-27T06:59:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 614.364229 secs 2024-07-27T06:59:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:59:53.540+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T06:59:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:59:53.540+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:59:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T06:59:53.540+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T06:59:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:59:53.570+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T06:59:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:59:53.570+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T06:59:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T06:59:53.570+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T06:59:53.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:53 smithi029 ceph-mon[32610]: pgmap v812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:59:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:53 smithi047 ceph-mon[36835]: pgmap v812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:59:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:53 smithi077 ceph-mon[37153]: pgmap v812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T06:59:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:55 smithi047 ceph-mon[36835]: pgmap v813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:59:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:55 smithi029 ceph-mon[32610]: pgmap v813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:59:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:55 smithi077 ceph-mon[37153]: pgmap v813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T06:59:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:57 smithi047 ceph-mon[36835]: pgmap v814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:59:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 619.364367 secs 2024-07-27T06:59:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:57 smithi029 ceph-mon[32610]: pgmap v814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:59:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 619.364367 secs 2024-07-27T06:59:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:57 smithi077 ceph-mon[37153]: pgmap v814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T06:59:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 619.364367 secs 2024-07-27T06:59:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 06:59:59 smithi047 ceph-mon[36835]: pgmap v815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T06:59:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 06:59:59 smithi029 ceph-mon[32610]: pgmap v815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 06:59:59 smithi077 ceph-mon[37153]: pgmap v815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:00 smithi047 ceph-mon[36835]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:00:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:00 smithi047 ceph-mon[36835]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-27T07:00:00.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:00 smithi047 ceph-mon[36835]: daemon mds.j on smithi029 is in unknown state 2024-07-27T07:00:00.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:00 smithi047 ceph-mon[36835]: daemon mds.d on smithi029 is in error state 2024-07-27T07:00:00.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:00 smithi047 ceph-mon[36835]: daemon mds.h on smithi047 is in error state 2024-07-27T07:00:00.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:00 smithi047 ceph-mon[36835]: daemon mds.i on smithi077 is in error state 2024-07-27T07:00:00.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:00 smithi047 ceph-mon[36835]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-27T07:00:00.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:00 smithi047 ceph-mon[36835]: mds.c(mds.1): 1 slow requests are blocked > 180 secs 2024-07-27T07:00:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:00 smithi029 ceph-mon[32610]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:00:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:00 smithi029 ceph-mon[32610]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-27T07:00:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:00 smithi029 ceph-mon[32610]: daemon mds.j on smithi029 is in unknown state 2024-07-27T07:00:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:00 smithi029 ceph-mon[32610]: daemon mds.d on smithi029 is in error state 2024-07-27T07:00:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:00 smithi029 ceph-mon[32610]: daemon mds.h on smithi047 is in error state 2024-07-27T07:00:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:00 smithi029 ceph-mon[32610]: daemon mds.i on smithi077 is in error state 2024-07-27T07:00:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:00 smithi029 ceph-mon[32610]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-27T07:00:00.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:00 smithi029 ceph-mon[32610]: mds.c(mds.1): 1 slow requests are blocked > 180 secs 2024-07-27T07:00:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:00 smithi077 ceph-mon[37153]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:00:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:00 smithi077 ceph-mon[37153]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-27T07:00:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:00 smithi077 ceph-mon[37153]: daemon mds.j on smithi029 is in unknown state 2024-07-27T07:00:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:00 smithi077 ceph-mon[37153]: daemon mds.d on smithi029 is in error state 2024-07-27T07:00:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:00 smithi077 ceph-mon[37153]: daemon mds.h on smithi047 is in error state 2024-07-27T07:00:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:00 smithi077 ceph-mon[37153]: daemon mds.i on smithi077 is in error state 2024-07-27T07:00:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:00 smithi077 ceph-mon[37153]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-27T07:00:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:00 smithi077 ceph-mon[37153]: mds.c(mds.1): 1 slow requests are blocked > 180 secs 2024-07-27T07:00:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:01 smithi047 ceph-mon[36835]: pgmap v816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:01 smithi029 ceph-mon[32610]: pgmap v816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:01 smithi077 ceph-mon[37153]: pgmap v816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 624.364504 secs 2024-07-27T07:00:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:02 smithi047 ceph-mon[36835]: pgmap v817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:00:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 624.364504 secs 2024-07-27T07:00:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:02 smithi029 ceph-mon[32610]: pgmap v817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:00:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 624.364504 secs 2024-07-27T07:00:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:02 smithi077 ceph-mon[37153]: pgmap v817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:00:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:05 smithi047 ceph-mon[36835]: pgmap v818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:00:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:05 smithi029 ceph-mon[32610]: pgmap v818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:00:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:05 smithi077 ceph-mon[37153]: pgmap v818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:00:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:07 smithi047 ceph-mon[36835]: pgmap v819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:00:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 629.364645 secs 2024-07-27T07:00:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:07 smithi029 ceph-mon[32610]: pgmap v819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:00:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 629.364645 secs 2024-07-27T07:00:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:07 smithi077 ceph-mon[37153]: pgmap v819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:00:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 629.364645 secs 2024-07-27T07:00:08.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:00:08.556+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:00:08.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:00:08.556+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:00:08.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:00:08.556+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:00:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:00:08.573+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:00:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:00:08.573+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:00:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:00:08.573+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:00:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:09 smithi047 ceph-mon[36835]: pgmap v820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:09 smithi029 ceph-mon[32610]: pgmap v820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:09 smithi077 ceph-mon[37153]: pgmap v820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:11 smithi047 ceph-mon[36835]: pgmap v821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:11 smithi029 ceph-mon[32610]: pgmap v821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:11 smithi077 ceph-mon[37153]: pgmap v821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 634.364775 secs 2024-07-27T07:00:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 634.364775 secs 2024-07-27T07:00:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 634.364775 secs 2024-07-27T07:00:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:13 smithi047 ceph-mon[36835]: pgmap v822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:00:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:13 smithi029 ceph-mon[32610]: pgmap v822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:00:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:13 smithi077 ceph-mon[37153]: pgmap v822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:00:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:15 smithi047 ceph-mon[36835]: pgmap v823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:00:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:15 smithi029 ceph-mon[32610]: pgmap v823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:00:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:15 smithi077 ceph-mon[37153]: pgmap v823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:00:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:17 smithi047 ceph-mon[36835]: pgmap v824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:00:17.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 639.364902 secs 2024-07-27T07:00:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:17 smithi029 ceph-mon[32610]: pgmap v824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:00:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 639.364902 secs 2024-07-27T07:00:17.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:17 smithi077 ceph-mon[37153]: pgmap v824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:00:17.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 639.364902 secs 2024-07-27T07:00:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:19 smithi047 ceph-mon[36835]: pgmap v825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:19 smithi029 ceph-mon[32610]: pgmap v825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:19 smithi077 ceph-mon[37153]: pgmap v825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:21 smithi047 ceph-mon[36835]: pgmap v826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:21 smithi029 ceph-mon[32610]: pgmap v826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:21 smithi077 ceph-mon[37153]: pgmap v826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 644.365023 secs 2024-07-27T07:00:22.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 644.365023 secs 2024-07-27T07:00:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 644.365023 secs 2024-07-27T07:00:23.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:00:23.571+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:00:23.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:00:23.571+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:00:23.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:00:23.571+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:00:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:00:23.586+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:00:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:00:23.586+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:00:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:00:23.586+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:00:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:23 smithi047 ceph-mon[36835]: pgmap v827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:00:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:23 smithi029 ceph-mon[32610]: pgmap v827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:00:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:23 smithi077 ceph-mon[37153]: pgmap v827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:00:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:25 smithi047 ceph-mon[36835]: pgmap v828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:00:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:25 smithi029 ceph-mon[32610]: pgmap v828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:00:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:25 smithi077 ceph-mon[37153]: pgmap v828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:00:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:27 smithi047 ceph-mon[36835]: pgmap v829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:00:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 649.365162 secs 2024-07-27T07:00:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:27 smithi029 ceph-mon[32610]: pgmap v829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:00:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 649.365162 secs 2024-07-27T07:00:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:27 smithi077 ceph-mon[37153]: pgmap v829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:00:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 649.365162 secs 2024-07-27T07:00:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:29 smithi047 ceph-mon[36835]: pgmap v830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:29 smithi029 ceph-mon[32610]: pgmap v830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:29 smithi077 ceph-mon[37153]: pgmap v830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:31 smithi047 ceph-mon[36835]: pgmap v831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:00:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:31 smithi029 ceph-mon[32610]: pgmap v831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:00:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:31 smithi077 ceph-mon[37153]: pgmap v831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:00:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 654.365297 secs 2024-07-27T07:00:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 654.365297 secs 2024-07-27T07:00:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 654.365297 secs 2024-07-27T07:00:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:33 smithi029 ceph-mon[32610]: pgmap v832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:00:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:33 smithi077 ceph-mon[37153]: pgmap v832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:00:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:33 smithi047 ceph-mon[36835]: pgmap v832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:00:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:35 smithi029 ceph-mon[32610]: pgmap v833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:00:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:35 smithi077 ceph-mon[37153]: pgmap v833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:00:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:35 smithi047 ceph-mon[36835]: pgmap v833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:00:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:00:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:00:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:00:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:37 smithi077 ceph-mon[37153]: pgmap v834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:00:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 659.365389 secs 2024-07-27T07:00:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:00:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:00:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:00:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:37 smithi047 ceph-mon[36835]: pgmap v834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:00:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 659.365389 secs 2024-07-27T07:00:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:00:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:00:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:00:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:37 smithi029 ceph-mon[32610]: pgmap v834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:00:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 659.365389 secs 2024-07-27T07:00:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:00:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:00:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:00:38.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:00:38.586+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:00:38.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:00:38.586+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:00:38.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:00:38.586+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:00:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:00:38.587+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:00:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:00:38.587+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:00:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:00:38.587+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:00:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:38 smithi077 ceph-mon[37153]: pgmap v835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:00:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:38 smithi077 ceph-mon[37153]: pgmap v836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:00:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:38 smithi047 ceph-mon[36835]: pgmap v835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:00:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:38 smithi047 ceph-mon[36835]: pgmap v836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:00:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:38 smithi029 ceph-mon[32610]: pgmap v835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:00:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:38 smithi029 ceph-mon[32610]: pgmap v836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:00:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:40 smithi077 ceph-mon[37153]: pgmap v837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:00:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:40 smithi047 ceph-mon[36835]: pgmap v837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:00:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:40 smithi029 ceph-mon[32610]: pgmap v837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:00:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:42 smithi077 ceph-mon[37153]: pgmap v838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T07:00:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 664.365536 secs 2024-07-27T07:00:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:42 smithi047 ceph-mon[36835]: pgmap v838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T07:00:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 664.365536 secs 2024-07-27T07:00:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:42 smithi029 ceph-mon[32610]: pgmap v838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T07:00:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 664.365536 secs 2024-07-27T07:00:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:44 smithi077 ceph-mon[37153]: pgmap v839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:00:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:44 smithi047 ceph-mon[36835]: pgmap v839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:00:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:44 smithi029 ceph-mon[32610]: pgmap v839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:00:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:46 smithi029 ceph-mon[32610]: pgmap v840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:00:46.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:46 smithi047 ceph-mon[36835]: pgmap v840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:00:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:46 smithi077 ceph-mon[37153]: pgmap v840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:00:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 669.365644 secs 2024-07-27T07:00:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 669.365644 secs 2024-07-27T07:00:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 669.365644 secs 2024-07-27T07:00:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:48 smithi077 ceph-mon[37153]: pgmap v841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:00:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:48 smithi047 ceph-mon[36835]: pgmap v841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:00:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:48 smithi029 ceph-mon[32610]: pgmap v841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:00:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:50 smithi077 ceph-mon[37153]: pgmap v842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:50 smithi047 ceph-mon[36835]: pgmap v842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:50 smithi029 ceph-mon[32610]: pgmap v842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:00:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:52 smithi077 ceph-mon[37153]: pgmap v843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 674.365792 secs 2024-07-27T07:00:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:52 smithi047 ceph-mon[36835]: pgmap v843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 674.365792 secs 2024-07-27T07:00:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:52 smithi029 ceph-mon[32610]: pgmap v843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 674.365792 secs 2024-07-27T07:00:53.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:00:53.590+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:00:53.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:00:53.590+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:00:53.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:00:53.590+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:00:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:00:53.602+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:00:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:00:53.602+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:00:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:00:53.602+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:00:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:54 smithi077 ceph-mon[37153]: pgmap v844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:00:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:54 smithi047 ceph-mon[36835]: pgmap v844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:00:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:54 smithi029 ceph-mon[32610]: pgmap v844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:00:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:56 smithi077 ceph-mon[37153]: pgmap v845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:56 smithi047 ceph-mon[36835]: pgmap v845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:56 smithi029 ceph-mon[32610]: pgmap v845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:00:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 679.365908 secs 2024-07-27T07:00:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 679.365908 secs 2024-07-27T07:00:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 679.365908 secs 2024-07-27T07:00:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:00:58 smithi077 ceph-mon[37153]: pgmap v846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:00:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:00:58 smithi047 ceph-mon[36835]: pgmap v846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:00:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:00:58 smithi029 ceph-mon[32610]: pgmap v846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:00.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:00 smithi047 ceph-mon[36835]: pgmap v847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:01:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:00 smithi029 ceph-mon[32610]: pgmap v847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:01:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:00 smithi077 ceph-mon[37153]: pgmap v847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:01:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:02 smithi047 ceph-mon[36835]: pgmap v848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 684.366001 secs 2024-07-27T07:01:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:02 smithi029 ceph-mon[32610]: pgmap v848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 684.366001 secs 2024-07-27T07:01:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:02 smithi077 ceph-mon[37153]: pgmap v848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 684.366001 secs 2024-07-27T07:01:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:04 smithi047 ceph-mon[36835]: pgmap v849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:01:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:04 smithi029 ceph-mon[32610]: pgmap v849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:01:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:04 smithi077 ceph-mon[37153]: pgmap v849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:01:06.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:06 smithi047 ceph-mon[36835]: pgmap v850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:07.216 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:06 smithi077 ceph-mon[37153]: pgmap v850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:07.216 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:06 smithi029 ceph-mon[32610]: pgmap v850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 689.366106 secs 2024-07-27T07:01:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 689.366106 secs 2024-07-27T07:01:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 689.366106 secs 2024-07-27T07:01:08.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:08.596+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:01:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:08.596+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:01:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:08.596+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:01:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:08.614+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:01:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:08.614+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:01:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:08.614+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:01:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:08 smithi047 ceph-mon[36835]: pgmap v851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:08 smithi029 ceph-mon[32610]: pgmap v851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:08 smithi077 ceph-mon[37153]: pgmap v851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:10 smithi047 ceph-mon[36835]: pgmap v852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:01:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:10 smithi029 ceph-mon[32610]: pgmap v852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:01:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:10 smithi077 ceph-mon[37153]: pgmap v852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:01:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:12 smithi047 ceph-mon[36835]: pgmap v853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 694.366238 secs 2024-07-27T07:01:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:12 smithi029 ceph-mon[32610]: pgmap v853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 694.366238 secs 2024-07-27T07:01:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:12 smithi077 ceph-mon[37153]: pgmap v853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 694.366238 secs 2024-07-27T07:01:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:14 smithi047 ceph-mon[36835]: pgmap v854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:01:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:14 smithi029 ceph-mon[32610]: pgmap v854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:01:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:14 smithi077 ceph-mon[37153]: pgmap v854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:01:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:16 smithi047 ceph-mon[36835]: pgmap v855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:16 smithi029 ceph-mon[32610]: pgmap v855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:16 smithi077 ceph-mon[37153]: pgmap v855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 699.366396 secs 2024-07-27T07:01:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 699.366396 secs 2024-07-27T07:01:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 699.366396 secs 2024-07-27T07:01:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:18 smithi047 ceph-mon[36835]: pgmap v856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:18 smithi029 ceph-mon[32610]: pgmap v856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:18 smithi077 ceph-mon[37153]: pgmap v856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:20.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:20 smithi077 ceph-mon[37153]: pgmap v857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:01:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:20 smithi047 ceph-mon[36835]: pgmap v857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:01:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:20 smithi029 ceph-mon[32610]: pgmap v857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:01:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:22 smithi047 ceph-mon[36835]: pgmap v858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 704.366491 secs 2024-07-27T07:01:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:22 smithi029 ceph-mon[32610]: pgmap v858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 704.366491 secs 2024-07-27T07:01:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:22 smithi077 ceph-mon[37153]: pgmap v858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 704.366491 secs 2024-07-27T07:01:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:23.611+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:01:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:23.611+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:01:23.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:23.611+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:01:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:23.622+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:01:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:23.622+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:01:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:23.622+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:01:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:24 smithi047 ceph-mon[36835]: pgmap v859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:01:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:24 smithi029 ceph-mon[32610]: pgmap v859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:01:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:24 smithi077 ceph-mon[37153]: pgmap v859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:01:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:25 smithi029 ceph-mon[32610]: pgmap v860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:25 smithi077 ceph-mon[37153]: pgmap v860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:25 smithi047 ceph-mon[36835]: pgmap v860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:26 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 709.366614 secs 2024-07-27T07:01:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:26 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 709.366614 secs 2024-07-27T07:01:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:26 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 709.366614 secs 2024-07-27T07:01:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:27 smithi047 ceph-mon[36835]: pgmap v861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:27 smithi029 ceph-mon[32610]: pgmap v861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:27 smithi077 ceph-mon[37153]: pgmap v861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:29 smithi077 ceph-mon[37153]: pgmap v862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:01:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:29 smithi047 ceph-mon[36835]: pgmap v862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:01:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:29 smithi029 ceph-mon[32610]: pgmap v862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:01:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:31 smithi077 ceph-mon[37153]: pgmap v863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:01:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:31 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 714.366717 secs 2024-07-27T07:01:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:31 smithi047 ceph-mon[36835]: pgmap v863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:01:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:31 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 714.366717 secs 2024-07-27T07:01:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:31 smithi029 ceph-mon[32610]: pgmap v863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:01:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:31 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 714.366717 secs 2024-07-27T07:01:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:33 smithi077 ceph-mon[37153]: pgmap v864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:01:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:33 smithi047 ceph-mon[36835]: pgmap v864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:01:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:33 smithi029 ceph-mon[32610]: pgmap v864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:01:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:35 smithi077 ceph-mon[37153]: pgmap v865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:01:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:35 smithi047 ceph-mon[36835]: pgmap v865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:01:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:35 smithi029 ceph-mon[32610]: pgmap v865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:01:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:36 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 719.366866 secs 2024-07-27T07:01:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:36 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 719.366866 secs 2024-07-27T07:01:37.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:36 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 719.366866 secs 2024-07-27T07:01:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: pgmap v866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:01:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:01:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:01:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:01:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:01:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:01:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:01:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:01:38.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:01:38.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: pgmap v867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T07:01:38.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: pgmap v868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:01:38.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:01:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: pgmap v866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:01:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:01:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:01:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:01:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:01:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:01:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:01:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:01:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:01:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: pgmap v867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T07:01:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: pgmap v868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:01:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: pgmap v866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: pgmap v867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T07:01:38.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: pgmap v868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:01:38.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:01:38.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:38.619+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:01:38.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:38.619+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:01:38.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:38.619+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:01:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:38.623+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:01:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:38.623+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:01:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:38.623+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:01:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:40 smithi077 ceph-mon[37153]: pgmap v869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:01:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:40 smithi047 ceph-mon[36835]: pgmap v869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:01:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:40 smithi029 ceph-mon[32610]: pgmap v869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:01:42.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:42.732+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320006110 con 0x7f7348108670 2024-07-27T07:01:42.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:42.732+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006000 con 0x5645ed3392f0 2024-07-27T07:01:42.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:42.732+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320006110 con 0x7f7348108670 2024-07-27T07:01:42.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:42.732+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006000 con 0x5645ed3392f0 2024-07-27T07:01:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:42 smithi077 ceph-mon[37153]: pgmap v870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:01:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:42 smithi077 ceph-mon[37153]: 1 slow requests, 1 included below; oldest blocked for > 724.366953 secs 2024-07-27T07:01:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:42 smithi077 ceph-mon[37153]: slow request 724.366952 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T07:01:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:42 smithi047 ceph-mon[36835]: pgmap v870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:01:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:42 smithi047 ceph-mon[36835]: 1 slow requests, 1 included below; oldest blocked for > 724.366953 secs 2024-07-27T07:01:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:42 smithi047 ceph-mon[36835]: slow request 724.366952 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T07:01:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:42 smithi029 ceph-mon[32610]: pgmap v870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:01:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:42 smithi029 ceph-mon[32610]: 1 slow requests, 1 included below; oldest blocked for > 724.366953 secs 2024-07-27T07:01:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:42 smithi029 ceph-mon[32610]: slow request 724.366952 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T07:01:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:44 smithi077 ceph-mon[37153]: pgmap v871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:01:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:44 smithi047 ceph-mon[36835]: pgmap v871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:01:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:44 smithi029 ceph-mon[32610]: pgmap v871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:01:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:46 smithi077 ceph-mon[37153]: pgmap v872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:01:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:46 smithi047 ceph-mon[36835]: pgmap v872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:01:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:46 smithi029 ceph-mon[32610]: pgmap v872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:01:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 729.367080 secs 2024-07-27T07:01:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 729.367080 secs 2024-07-27T07:01:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 729.367080 secs 2024-07-27T07:01:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:48 smithi077 ceph-mon[37153]: pgmap v873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:01:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:48 smithi047 ceph-mon[36835]: pgmap v873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:01:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:48 smithi029 ceph-mon[32610]: pgmap v873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:01:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:50 smithi077 ceph-mon[37153]: pgmap v874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:01:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:50 smithi047 ceph-mon[36835]: pgmap v874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:01:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:50 smithi029 ceph-mon[32610]: pgmap v874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:01:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:52 smithi077 ceph-mon[37153]: pgmap v875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:01:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 734.367209 secs 2024-07-27T07:01:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:52 smithi047 ceph-mon[36835]: pgmap v875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:01:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 734.367209 secs 2024-07-27T07:01:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:52 smithi029 ceph-mon[32610]: pgmap v875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:01:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 734.367209 secs 2024-07-27T07:01:53.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:53.624+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:01:53.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:53.624+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:01:53.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:01:53.624+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:01:53.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:53.635+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:01:53.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:53.635+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:01:53.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:53.635+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:01:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:54 smithi077 ceph-mon[37153]: pgmap v876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:01:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:54 smithi047 ceph-mon[36835]: pgmap v876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:01:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:54 smithi029 ceph-mon[32610]: pgmap v876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:01:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:56 smithi077 ceph-mon[37153]: pgmap v877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:56 smithi047 ceph-mon[36835]: pgmap v877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:56 smithi029 ceph-mon[32610]: pgmap v877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:01:57.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:57.745+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006070 con 0x55a2cdf3b350 2024-07-27T07:01:57.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:57.745+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006060 con 0x7eff50104ab0 2024-07-27T07:01:57.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:57.745+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006060 con 0x7eff50104ab0 2024-07-27T07:01:57.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:01:57.745+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006070 con 0x55a2cdf3b350 2024-07-27T07:01:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 739.367336 secs 2024-07-27T07:01:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 739.367336 secs 2024-07-27T07:01:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 739.367336 secs 2024-07-27T07:01:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:01:58 smithi077 ceph-mon[37153]: pgmap v878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:01:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:01:58 smithi047 ceph-mon[36835]: pgmap v878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:01:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:01:58 smithi029 ceph-mon[32610]: pgmap v878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:00 smithi077 ceph-mon[37153]: pgmap v879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:02:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:00 smithi047 ceph-mon[36835]: pgmap v879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:02:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:00 smithi029 ceph-mon[32610]: pgmap v879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:02:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:02 smithi077 ceph-mon[37153]: pgmap v880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:02:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 744.367485 secs 2024-07-27T07:02:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:02 smithi047 ceph-mon[36835]: pgmap v880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:02:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 744.367485 secs 2024-07-27T07:02:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:02 smithi029 ceph-mon[32610]: pgmap v880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:02:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 744.367485 secs 2024-07-27T07:02:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:04 smithi047 ceph-mon[36835]: pgmap v881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:04 smithi029 ceph-mon[32610]: pgmap v881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:04 smithi077 ceph-mon[37153]: pgmap v881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:06 smithi047 ceph-mon[36835]: pgmap v882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:06 smithi029 ceph-mon[32610]: pgmap v882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:06 smithi077 ceph-mon[37153]: pgmap v882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 749.367636 secs 2024-07-27T07:02:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 749.367636 secs 2024-07-27T07:02:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 749.367636 secs 2024-07-27T07:02:08.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:02:08.640+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:02:08.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:02:08.640+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:02:08.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:02:08.640+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:02:08.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:02:08.651+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:02:08.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:02:08.651+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:02:08.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:02:08.651+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:02:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:08 smithi047 ceph-mon[36835]: pgmap v883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:08 smithi029 ceph-mon[32610]: pgmap v883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:08 smithi077 ceph-mon[37153]: pgmap v883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:10 smithi047 ceph-mon[36835]: pgmap v884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:02:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:10 smithi029 ceph-mon[32610]: pgmap v884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:02:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:10 smithi077 ceph-mon[37153]: pgmap v884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:02:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:12 smithi047 ceph-mon[36835]: pgmap v885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:02:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 754.367793 secs 2024-07-27T07:02:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:12 smithi029 ceph-mon[32610]: pgmap v885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:02:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 754.367793 secs 2024-07-27T07:02:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:12 smithi077 ceph-mon[37153]: pgmap v885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:02:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 754.367793 secs 2024-07-27T07:02:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:14 smithi047 ceph-mon[36835]: pgmap v886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:14 smithi029 ceph-mon[32610]: pgmap v886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:14 smithi077 ceph-mon[37153]: pgmap v886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:16 smithi047 ceph-mon[36835]: pgmap v887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:16 smithi029 ceph-mon[32610]: pgmap v887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:16 smithi077 ceph-mon[37153]: pgmap v887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 759.367865 secs 2024-07-27T07:02:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 759.367865 secs 2024-07-27T07:02:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 759.367865 secs 2024-07-27T07:02:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:18 smithi047 ceph-mon[36835]: pgmap v888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:18 smithi029 ceph-mon[32610]: pgmap v888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:18 smithi077 ceph-mon[37153]: pgmap v888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:20.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:20 smithi077 ceph-mon[37153]: pgmap v889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:02:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:20 smithi047 ceph-mon[36835]: pgmap v889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:02:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:20 smithi029 ceph-mon[32610]: pgmap v889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:02:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:22 smithi047 ceph-mon[36835]: pgmap v890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:02:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 764.367969 secs 2024-07-27T07:02:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:22 smithi029 ceph-mon[32610]: pgmap v890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:02:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 764.367969 secs 2024-07-27T07:02:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:22 smithi077 ceph-mon[37153]: pgmap v890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:02:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 764.367969 secs 2024-07-27T07:02:23.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:02:23.655+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:02:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:02:23.655+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:02:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:02:23.655+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:02:23.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:02:23.666+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:02:23.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:02:23.666+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:02:23.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:02:23.666+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:02:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:24 smithi047 ceph-mon[36835]: pgmap v891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:24.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:24 smithi029 ceph-mon[32610]: pgmap v891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:24 smithi077 ceph-mon[37153]: pgmap v891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:26 smithi047 ceph-mon[36835]: pgmap v892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:26 smithi029 ceph-mon[32610]: pgmap v892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:26 smithi077 ceph-mon[37153]: pgmap v892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 769.368087 secs 2024-07-27T07:02:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 769.368087 secs 2024-07-27T07:02:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 769.368087 secs 2024-07-27T07:02:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:28 smithi047 ceph-mon[36835]: pgmap v893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:28 smithi029 ceph-mon[32610]: pgmap v893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:28 smithi077 ceph-mon[37153]: pgmap v893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:30 smithi047 ceph-mon[36835]: pgmap v894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:02:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:30 smithi029 ceph-mon[32610]: pgmap v894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:02:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:30 smithi077 ceph-mon[37153]: pgmap v894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:02:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:31 smithi047 ceph-mon[36835]: pgmap v895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:02:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:31 smithi029 ceph-mon[32610]: pgmap v895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:02:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:31 smithi077 ceph-mon[37153]: pgmap v895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:02:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 774.368174 secs 2024-07-27T07:02:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 774.368174 secs 2024-07-27T07:02:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 774.368174 secs 2024-07-27T07:02:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:33 smithi047 ceph-mon[36835]: pgmap v896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:33 smithi029 ceph-mon[32610]: pgmap v896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:33 smithi077 ceph-mon[37153]: pgmap v896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:36 smithi077 ceph-mon[37153]: pgmap v897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:36 smithi047 ceph-mon[36835]: pgmap v897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:36 smithi029 ceph-mon[32610]: pgmap v897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 779.368306 secs 2024-07-27T07:02:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 779.368306 secs 2024-07-27T07:02:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 779.368306 secs 2024-07-27T07:02:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:02:38.669+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:02:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:02:38.669+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:02:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:02:38.669+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:02:38.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:02:38.680+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:02:38.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:02:38.680+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:02:38.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:02:38.680+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:02:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: pgmap v898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:02:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:02:38.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:02:38.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:02:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: pgmap v898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:02:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:02:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:02:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:02:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: pgmap v898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:02:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:02:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:02:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:02:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:02:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:02:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:39 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:02:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:39 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:02:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:39 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:02:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:39 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:02:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:39 smithi077 ceph-mon[37153]: pgmap v899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:02:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:39 smithi077 ceph-mon[37153]: pgmap v900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:02:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:39 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:02:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:39 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:02:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:39 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:02:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:39 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:02:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:39 smithi047 ceph-mon[36835]: pgmap v899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:02:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:39 smithi047 ceph-mon[36835]: pgmap v900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:02:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:39 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:02:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:39 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:02:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:39 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:02:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:39 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:02:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:39 smithi029 ceph-mon[32610]: pgmap v899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:02:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:39 smithi029 ceph-mon[32610]: pgmap v900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:02:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:41 smithi077 ceph-mon[37153]: pgmap v901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:02:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:41 smithi047 ceph-mon[36835]: pgmap v901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:02:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:41 smithi029 ceph-mon[32610]: pgmap v901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:02:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 784.368432 secs 2024-07-27T07:02:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 784.368432 secs 2024-07-27T07:02:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 784.368432 secs 2024-07-27T07:02:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:43 smithi077 ceph-mon[37153]: pgmap v902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:02:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:43 smithi047 ceph-mon[36835]: pgmap v902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:02:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:43 smithi029 ceph-mon[32610]: pgmap v902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:02:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:45 smithi077 ceph-mon[37153]: pgmap v903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:02:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:45 smithi047 ceph-mon[36835]: pgmap v903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:02:45.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:45 smithi029 ceph-mon[32610]: pgmap v903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:02:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:47 smithi077 ceph-mon[37153]: pgmap v904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:02:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 789.368535 secs 2024-07-27T07:02:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:47 smithi047 ceph-mon[36835]: pgmap v904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:02:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 789.368535 secs 2024-07-27T07:02:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:47 smithi029 ceph-mon[32610]: pgmap v904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:02:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 789.368535 secs 2024-07-27T07:02:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:49 smithi077 ceph-mon[37153]: pgmap v905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:02:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:49 smithi047 ceph-mon[36835]: pgmap v905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:02:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:49 smithi029 ceph-mon[32610]: pgmap v905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:02:51.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:51 smithi077 ceph-mon[37153]: pgmap v906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:51 smithi047 ceph-mon[36835]: pgmap v906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:51 smithi029 ceph-mon[32610]: pgmap v906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 794.368644 secs 2024-07-27T07:02:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 794.368644 secs 2024-07-27T07:02:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 794.368644 secs 2024-07-27T07:02:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:02:53.685+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:02:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:02:53.685+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:02:53.686 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:02:53.685+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:02:53.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:02:53.696+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:02:53.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:02:53.696+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:02:53.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:02:53.696+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:02:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:53 smithi077 ceph-mon[37153]: pgmap v907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:02:53.864 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:53 smithi029 ceph-mon[32610]: pgmap v907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:02:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:53 smithi047 ceph-mon[36835]: pgmap v907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:02:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:55 smithi077 ceph-mon[37153]: pgmap v908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:55 smithi047 ceph-mon[36835]: pgmap v908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:55 smithi029 ceph-mon[32610]: pgmap v908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:02:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:57 smithi077 ceph-mon[37153]: pgmap v909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 799.368766 secs 2024-07-27T07:02:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:57 smithi047 ceph-mon[36835]: pgmap v909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 799.368766 secs 2024-07-27T07:02:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:57 smithi029 ceph-mon[32610]: pgmap v909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:02:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 799.368766 secs 2024-07-27T07:02:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:02:59 smithi077 ceph-mon[37153]: pgmap v910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:02:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:02:59 smithi047 ceph-mon[36835]: pgmap v910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:02:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:02:59 smithi029 ceph-mon[32610]: pgmap v910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:01 smithi077 ceph-mon[37153]: pgmap v911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:01 smithi047 ceph-mon[36835]: pgmap v911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:01 smithi029 ceph-mon[32610]: pgmap v911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 804.368890 secs 2024-07-27T07:03:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 804.368890 secs 2024-07-27T07:03:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 804.368890 secs 2024-07-27T07:03:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:03 smithi077 ceph-mon[37153]: pgmap v912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:03 smithi047 ceph-mon[36835]: pgmap v912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:03 smithi029 ceph-mon[32610]: pgmap v912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:05 smithi077 ceph-mon[37153]: pgmap v913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:03:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:05 smithi047 ceph-mon[36835]: pgmap v913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:03:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:05 smithi029 ceph-mon[32610]: pgmap v913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:03:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:07 smithi077 ceph-mon[37153]: pgmap v914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 809.369018 secs 2024-07-27T07:03:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:07 smithi047 ceph-mon[36835]: pgmap v914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 809.369018 secs 2024-07-27T07:03:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:07 smithi029 ceph-mon[32610]: pgmap v914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 809.369018 secs 2024-07-27T07:03:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:03:08.701+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:03:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:03:08.701+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:03:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:03:08.701+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:03:08.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:03:08.711+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:03:08.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:03:08.711+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:03:08.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:03:08.712+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:03:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:09 smithi077 ceph-mon[37153]: pgmap v915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:09 smithi047 ceph-mon[36835]: pgmap v915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:09 smithi029 ceph-mon[32610]: pgmap v915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:11 smithi077 ceph-mon[37153]: pgmap v916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:11 smithi047 ceph-mon[36835]: pgmap v916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:11 smithi029 ceph-mon[32610]: pgmap v916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 814.369083 secs 2024-07-27T07:03:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 814.369083 secs 2024-07-27T07:03:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 814.369083 secs 2024-07-27T07:03:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:13 smithi047 ceph-mon[36835]: pgmap v917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:13 smithi029 ceph-mon[32610]: pgmap v917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:13 smithi077 ceph-mon[37153]: pgmap v917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:15 smithi047 ceph-mon[36835]: pgmap v918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:03:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:15 smithi029 ceph-mon[32610]: pgmap v918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:03:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:15 smithi077 ceph-mon[37153]: pgmap v918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:03:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:17 smithi047 ceph-mon[36835]: pgmap v919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 819.369204 secs 2024-07-27T07:03:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:17 smithi029 ceph-mon[32610]: pgmap v919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 819.369204 secs 2024-07-27T07:03:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:17 smithi077 ceph-mon[37153]: pgmap v919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 819.369204 secs 2024-07-27T07:03:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:19 smithi047 ceph-mon[36835]: pgmap v920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:19 smithi029 ceph-mon[32610]: pgmap v920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:19 smithi077 ceph-mon[37153]: pgmap v920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:21.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:21 smithi077 ceph-mon[37153]: pgmap v921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:21 smithi047 ceph-mon[36835]: pgmap v921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:21 smithi029 ceph-mon[32610]: pgmap v921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:22.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 824.369345 secs 2024-07-27T07:03:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 824.369345 secs 2024-07-27T07:03:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 824.369345 secs 2024-07-27T07:03:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:03:23.716+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:03:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:03:23.716+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:03:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:03:23.716+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:03:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:03:23.725+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:03:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:03:23.725+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:03:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:03:23.725+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:03:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:23 smithi047 ceph-mon[36835]: pgmap v922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:23 smithi029 ceph-mon[32610]: pgmap v922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:23 smithi077 ceph-mon[37153]: pgmap v922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:25 smithi047 ceph-mon[36835]: pgmap v923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:03:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:25 smithi029 ceph-mon[32610]: pgmap v923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:03:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:25 smithi077 ceph-mon[37153]: pgmap v923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:03:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:27 smithi047 ceph-mon[36835]: pgmap v924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 829.369472 secs 2024-07-27T07:03:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:27 smithi029 ceph-mon[32610]: pgmap v924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 829.369472 secs 2024-07-27T07:03:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:27 smithi077 ceph-mon[37153]: pgmap v924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 829.369472 secs 2024-07-27T07:03:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:29 smithi047 ceph-mon[36835]: pgmap v925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:29 smithi029 ceph-mon[32610]: pgmap v925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:29 smithi077 ceph-mon[37153]: pgmap v925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:31 smithi047 ceph-mon[36835]: pgmap v926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:31 smithi029 ceph-mon[32610]: pgmap v926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:31 smithi077 ceph-mon[37153]: pgmap v926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 834.369599 secs 2024-07-27T07:03:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 834.369599 secs 2024-07-27T07:03:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 834.369599 secs 2024-07-27T07:03:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:33 smithi047 ceph-mon[36835]: pgmap v927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:33 smithi029 ceph-mon[32610]: pgmap v927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:33 smithi077 ceph-mon[37153]: pgmap v927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:35 smithi047 ceph-mon[36835]: pgmap v928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:03:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:35 smithi029 ceph-mon[32610]: pgmap v928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:03:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:35 smithi077 ceph-mon[37153]: pgmap v928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:03:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:37 smithi047 ceph-mon[36835]: pgmap v929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 839.369664 secs 2024-07-27T07:03:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:37 smithi029 ceph-mon[32610]: pgmap v929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 839.369664 secs 2024-07-27T07:03:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:37 smithi077 ceph-mon[37153]: pgmap v929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 839.369664 secs 2024-07-27T07:03:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:03:38.730+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:03:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:03:38.730+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:03:38.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:03:38.730+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:03:38.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:03:38.739+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:03:38.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:03:38.739+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:03:38.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:03:38.739+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:03:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:03:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:03:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:03:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:03:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:03:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:03:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:03:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:03:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:03:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:03:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:03:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:03:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:39 smithi047 ceph-mon[36835]: pgmap v930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:39 smithi047 ceph-mon[36835]: pgmap v931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:03:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:39 smithi047 ceph-mon[36835]: pgmap v932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:03:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:39 smithi029 ceph-mon[32610]: pgmap v930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:39 smithi029 ceph-mon[32610]: pgmap v931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:03:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:39 smithi029 ceph-mon[32610]: pgmap v932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:03:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:39 smithi077 ceph-mon[37153]: pgmap v930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:03:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:39 smithi077 ceph-mon[37153]: pgmap v931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:03:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:39 smithi077 ceph-mon[37153]: pgmap v932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:03:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:41 smithi047 ceph-mon[36835]: pgmap v933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:03:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:41 smithi029 ceph-mon[32610]: pgmap v933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:03:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:41 smithi077 ceph-mon[37153]: pgmap v933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:03:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 844.369809 secs 2024-07-27T07:03:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 844.369809 secs 2024-07-27T07:03:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 844.369809 secs 2024-07-27T07:03:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:43 smithi047 ceph-mon[36835]: pgmap v934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:03:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:43 smithi029 ceph-mon[32610]: pgmap v934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:03:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:43 smithi077 ceph-mon[37153]: pgmap v934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:03:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:45 smithi047 ceph-mon[36835]: pgmap v935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:03:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:45 smithi029 ceph-mon[32610]: pgmap v935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:03:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:45 smithi077 ceph-mon[37153]: pgmap v935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:03:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:46 smithi047 ceph-mon[36835]: pgmap v936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:03:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:46 smithi029 ceph-mon[32610]: pgmap v936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:03:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:46 smithi077 ceph-mon[37153]: pgmap v936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:03:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 849.369926 secs 2024-07-27T07:03:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 849.369926 secs 2024-07-27T07:03:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 849.369926 secs 2024-07-27T07:03:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:48 smithi047 ceph-mon[36835]: pgmap v937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:03:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:48 smithi029 ceph-mon[32610]: pgmap v937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:03:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:48 smithi077 ceph-mon[37153]: pgmap v937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:03:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:51 smithi047 ceph-mon[36835]: pgmap v938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:51 smithi029 ceph-mon[32610]: pgmap v938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:51 smithi077 ceph-mon[37153]: pgmap v938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:03:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 854.370023 secs 2024-07-27T07:03:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 854.370023 secs 2024-07-27T07:03:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 854.370023 secs 2024-07-27T07:03:53.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:03:53.742+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:03:53.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:03:53.742+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:03:53.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:03:53.742+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:03:53.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:03:53.748+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:03:53.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:03:53.748+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:03:53.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:03:53.748+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:03:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:53 smithi047 ceph-mon[36835]: pgmap v939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:03:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:53 smithi029 ceph-mon[32610]: pgmap v939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:03:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:53 smithi077 ceph-mon[37153]: pgmap v939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:03:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:55 smithi047 ceph-mon[36835]: pgmap v940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:03:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:55 smithi029 ceph-mon[32610]: pgmap v940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:03:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:55 smithi077 ceph-mon[37153]: pgmap v940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:03:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:57 smithi047 ceph-mon[36835]: pgmap v941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 859.370149 secs 2024-07-27T07:03:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:57 smithi029 ceph-mon[32610]: pgmap v941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 859.370149 secs 2024-07-27T07:03:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:57 smithi077 ceph-mon[37153]: pgmap v941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:03:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 859.370149 secs 2024-07-27T07:03:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:03:59 smithi047 ceph-mon[36835]: pgmap v942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:03:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:03:59 smithi029 ceph-mon[32610]: pgmap v942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:04:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:03:59 smithi077 ceph-mon[37153]: pgmap v942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:04:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:01 smithi047 ceph-mon[36835]: pgmap v943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:01 smithi029 ceph-mon[32610]: pgmap v943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:01 smithi077 ceph-mon[37153]: pgmap v943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 864.370226 secs 2024-07-27T07:04:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 864.370226 secs 2024-07-27T07:04:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 864.370226 secs 2024-07-27T07:04:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:03 smithi047 ceph-mon[36835]: pgmap v944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:04:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:03 smithi029 ceph-mon[32610]: pgmap v944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:04:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:03 smithi077 ceph-mon[37153]: pgmap v944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:04:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:05 smithi047 ceph-mon[36835]: pgmap v945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:04:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:05 smithi029 ceph-mon[32610]: pgmap v945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:04:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:05 smithi077 ceph-mon[37153]: pgmap v945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:04:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:07 smithi047 ceph-mon[36835]: pgmap v946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 869.370310 secs 2024-07-27T07:04:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:07 smithi029 ceph-mon[32610]: pgmap v946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 869.370310 secs 2024-07-27T07:04:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:07 smithi077 ceph-mon[37153]: pgmap v946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 869.370310 secs 2024-07-27T07:04:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:08.754+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:04:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:08.754+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:04:08.755 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:08.754+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:04:08.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:08.764+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:04:08.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:08.764+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:04:08.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:08.764+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:04:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:09 smithi047 ceph-mon[36835]: pgmap v947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:04:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:09 smithi029 ceph-mon[32610]: pgmap v947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:04:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:09 smithi077 ceph-mon[37153]: pgmap v947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:04:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:11 smithi047 ceph-mon[36835]: pgmap v948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:11 smithi029 ceph-mon[32610]: pgmap v948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:11 smithi077 ceph-mon[37153]: pgmap v948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 874.370468 secs 2024-07-27T07:04:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 874.370468 secs 2024-07-27T07:04:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 874.370468 secs 2024-07-27T07:04:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:13 smithi047 ceph-mon[36835]: pgmap v949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:04:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:13 smithi029 ceph-mon[32610]: pgmap v949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:04:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:13 smithi077 ceph-mon[37153]: pgmap v949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:04:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:14 smithi047 ceph-mon[36835]: pgmap v950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:04:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:14 smithi029 ceph-mon[32610]: pgmap v950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:04:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:14 smithi077 ceph-mon[37153]: pgmap v950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:04:17.646 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:17 smithi047 ceph-mon[36835]: pgmap v951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 879.370595 secs 2024-07-27T07:04:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:17 smithi029 ceph-mon[32610]: pgmap v951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 879.370595 secs 2024-07-27T07:04:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:17 smithi077 ceph-mon[37153]: pgmap v951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 879.370595 secs 2024-07-27T07:04:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:19 smithi047 ceph-mon[36835]: pgmap v952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:04:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:19 smithi029 ceph-mon[32610]: pgmap v952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:04:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:19 smithi077 ceph-mon[37153]: pgmap v952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:04:21.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:21 smithi077 ceph-mon[37153]: pgmap v953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:04:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:21 smithi047 ceph-mon[36835]: pgmap v953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:04:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:21 smithi029 ceph-mon[32610]: pgmap v953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:04:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 884.370663 secs 2024-07-27T07:04:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 884.370663 secs 2024-07-27T07:04:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 884.370663 secs 2024-07-27T07:04:23.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:23.770+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:04:23.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:23.770+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:04:23.771 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:23.770+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:04:23.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:23.777+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:04:23.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:23.777+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:04:23.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:23.777+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:04:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:23 smithi047 ceph-mon[36835]: pgmap v954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:04:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:23 smithi029 ceph-mon[32610]: pgmap v954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:04:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:23 smithi077 ceph-mon[37153]: pgmap v954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:04:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:25 smithi047 ceph-mon[36835]: pgmap v955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:04:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:25 smithi029 ceph-mon[32610]: pgmap v955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:04:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:25 smithi077 ceph-mon[37153]: pgmap v955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:04:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:27 smithi047 ceph-mon[36835]: pgmap v956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 889.370807 secs 2024-07-27T07:04:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:27 smithi029 ceph-mon[32610]: pgmap v956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:27.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 889.370807 secs 2024-07-27T07:04:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:27 smithi077 ceph-mon[37153]: pgmap v956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 889.370807 secs 2024-07-27T07:04:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:29 smithi047 ceph-mon[36835]: pgmap v957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:04:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:29 smithi029 ceph-mon[32610]: pgmap v957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:04:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:29 smithi077 ceph-mon[37153]: pgmap v957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:04:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:31 smithi047 ceph-mon[36835]: pgmap v958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:31 smithi029 ceph-mon[32610]: pgmap v958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:31 smithi077 ceph-mon[37153]: pgmap v958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 894.370954 secs 2024-07-27T07:04:32.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 894.370954 secs 2024-07-27T07:04:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 894.370954 secs 2024-07-27T07:04:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:33 smithi047 ceph-mon[36835]: pgmap v959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:04:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:33 smithi029 ceph-mon[32610]: pgmap v959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:04:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:33 smithi077 ceph-mon[37153]: pgmap v959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:04:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:35 smithi047 ceph-mon[36835]: pgmap v960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:04:35.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:35 smithi029 ceph-mon[32610]: pgmap v960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:04:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:35 smithi077 ceph-mon[37153]: pgmap v960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:04:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:37 smithi047 ceph-mon[36835]: pgmap v961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 899.371072 secs 2024-07-27T07:04:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:37 smithi029 ceph-mon[32610]: pgmap v961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 899.371072 secs 2024-07-27T07:04:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:37 smithi077 ceph-mon[37153]: pgmap v961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 899.371072 secs 2024-07-27T07:04:38.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:38.784+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:04:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:38.784+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:04:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:38.784+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:04:38.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:38.791+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:04:38.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:38.791+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:04:38.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:38.791+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:04:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:38 smithi077 ceph-mon[37153]: pgmap v962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:04:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:04:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:38 smithi047 ceph-mon[36835]: pgmap v962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:04:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:04:39.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:38 smithi029 ceph-mon[32610]: pgmap v962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:04:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:04:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:04:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:04:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:39 smithi077 ceph-mon[37153]: pgmap v963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:04:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:39 smithi077 ceph-mon[37153]: pgmap v964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:04:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:04:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:04:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:04:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:39 smithi047 ceph-mon[36835]: pgmap v963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:04:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:39 smithi047 ceph-mon[36835]: pgmap v964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:04:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:04:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:04:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:04:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:39 smithi029 ceph-mon[32610]: pgmap v963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:04:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:39 smithi029 ceph-mon[32610]: pgmap v964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:04:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:04:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:42 smithi047 ceph-mon[36835]: pgmap v965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:04:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 904.371185 secs 2024-07-27T07:04:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:42 smithi029 ceph-mon[32610]: pgmap v965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:04:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 904.371185 secs 2024-07-27T07:04:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:42 smithi077 ceph-mon[37153]: pgmap v965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:04:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 904.371185 secs 2024-07-27T07:04:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.632+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-27T07:04:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.632+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T07:04:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.632+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b6250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x5645ed703330 tx=0x7f733c00c4b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T07:04:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.632+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b6250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x5645ed703330 tx=0x7f733c00c4b0 comp rx=0 tx=0).stop 2024-07-27T07:04:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.632+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b6250 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T07:04:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.632+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b6250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T07:04:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.632+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b6250 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T07:04:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.632+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T07:04:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.632+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T07:04:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.633+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f733c02b3a0 tx=0x7f733c037000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T07:04:42.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.759+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200063d0 con 0x7f7348108670 2024-07-27T07:04:42.760 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.759+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73180062c0 con 0x5645ed3392f0 2024-07-27T07:04:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.759+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f73180062c0 con 0x5645ed3392f0 2024-07-27T07:04:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:42.759+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200063d0 con 0x7f7348108670 2024-07-27T07:04:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:44 smithi047 ceph-mon[36835]: pgmap v966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:04:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:44 smithi029 ceph-mon[32610]: pgmap v966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:04:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:44 smithi077 ceph-mon[37153]: pgmap v966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:04:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:46 smithi047 ceph-mon[36835]: pgmap v967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:04:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:46 smithi029 ceph-mon[32610]: pgmap v967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:04:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:46 smithi077 ceph-mon[37153]: pgmap v967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:04:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 909.371298 secs 2024-07-27T07:04:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 909.371298 secs 2024-07-27T07:04:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 909.371298 secs 2024-07-27T07:04:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:48 smithi029 ceph-mon[32610]: pgmap v968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:04:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:48 smithi077 ceph-mon[37153]: pgmap v968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:04:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:48 smithi047 ceph-mon[36835]: pgmap v968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:04:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:50 smithi029 ceph-mon[32610]: pgmap v969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:04:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:50 smithi077 ceph-mon[37153]: pgmap v969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:04:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:50 smithi047 ceph-mon[36835]: pgmap v969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:04:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:52 smithi029 ceph-mon[32610]: pgmap v970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 914.371422 secs 2024-07-27T07:04:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:52 smithi077 ceph-mon[37153]: pgmap v970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 914.371422 secs 2024-07-27T07:04:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:52 smithi047 ceph-mon[36835]: pgmap v970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 914.371422 secs 2024-07-27T07:04:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:53.799+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:04:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:53.799+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:04:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:04:53.799+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:04:53.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:53.806+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:04:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:53.806+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:04:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:53.806+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:04:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:54 smithi077 ceph-mon[37153]: pgmap v971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:04:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:54 smithi047 ceph-mon[36835]: pgmap v971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:04:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:54 smithi029 ceph-mon[32610]: pgmap v971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:04:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:56 smithi029 ceph-mon[32610]: pgmap v972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:56 smithi077 ceph-mon[37153]: pgmap v972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:56 smithi047 ceph-mon[36835]: pgmap v972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:04:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 919.371545 secs 2024-07-27T07:04:57.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:57.646+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000186) for more than 900000000 us, fault. 2024-07-27T07:04:57.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:57.646+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b6250 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x55a2ce2df200 tx=0x7eff40009680 comp rx=0 tx=0).stop 2024-07-27T07:04:57.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:57.646+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b6250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T07:04:57.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:57.646+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b6250 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T07:04:57.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:57.646+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T07:04:57.648 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 919.371545 secs 2024-07-27T07:04:57.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:57.646+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T07:04:57.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:57.647+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7eff40003b30 tx=0x7eff40010000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T07:04:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 919.371545 secs 2024-07-27T07:04:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:57.772+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006330 con 0x55a2cdf3b350 2024-07-27T07:04:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:57.772+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006320 con 0x7eff50104ab0 2024-07-27T07:04:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:57.772+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006330 con 0x55a2cdf3b350 2024-07-27T07:04:57.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:04:57.772+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006320 con 0x7eff50104ab0 2024-07-27T07:04:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:04:58 smithi077 ceph-mon[37153]: pgmap v973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:04:58 smithi047 ceph-mon[36835]: pgmap v973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:04:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:04:58 smithi029 ceph-mon[32610]: pgmap v973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:00 smithi077 ceph-mon[37153]: pgmap v974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:00 smithi047 ceph-mon[36835]: pgmap v974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:00 smithi029 ceph-mon[32610]: pgmap v974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:02 smithi077 ceph-mon[37153]: pgmap v975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 924.371610 secs 2024-07-27T07:05:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:02 smithi047 ceph-mon[36835]: pgmap v975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 924.371610 secs 2024-07-27T07:05:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:02 smithi029 ceph-mon[32610]: pgmap v975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 924.371610 secs 2024-07-27T07:05:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:04 smithi077 ceph-mon[37153]: pgmap v976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:05:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:04 smithi047 ceph-mon[36835]: pgmap v976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:05:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:04 smithi029 ceph-mon[32610]: pgmap v976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:05:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:06 smithi077 ceph-mon[37153]: pgmap v977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:06 smithi047 ceph-mon[36835]: pgmap v977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:06 smithi029 ceph-mon[32610]: pgmap v977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 929.371755 secs 2024-07-27T07:05:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 929.371755 secs 2024-07-27T07:05:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 929.371755 secs 2024-07-27T07:05:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:08 smithi077 ceph-mon[37153]: pgmap v978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:08 smithi047 ceph-mon[36835]: pgmap v978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:08 smithi029 ceph-mon[32610]: pgmap v978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:05:08.806+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:05:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:05:08.806+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:05:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:05:08.806+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:05:08.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:05:08.818+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:05:08.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:05:08.818+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:05:08.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:05:08.818+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:05:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:10 smithi077 ceph-mon[37153]: pgmap v979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:10 smithi047 ceph-mon[36835]: pgmap v979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:10 smithi029 ceph-mon[32610]: pgmap v979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:12 smithi077 ceph-mon[37153]: pgmap v980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 934.371914 secs 2024-07-27T07:05:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:12 smithi047 ceph-mon[36835]: pgmap v980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:12.702 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 934.371914 secs 2024-07-27T07:05:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:12 smithi029 ceph-mon[32610]: pgmap v980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 934.371914 secs 2024-07-27T07:05:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:14 smithi077 ceph-mon[37153]: pgmap v981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:05:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:14 smithi047 ceph-mon[36835]: pgmap v981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:05:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:14 smithi029 ceph-mon[32610]: pgmap v981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:05:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:16 smithi077 ceph-mon[37153]: pgmap v982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:16 smithi047 ceph-mon[36835]: pgmap v982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:16 smithi029 ceph-mon[32610]: pgmap v982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:17.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 939.372019 secs 2024-07-27T07:05:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 939.372019 secs 2024-07-27T07:05:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 939.372019 secs 2024-07-27T07:05:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:18 smithi077 ceph-mon[37153]: pgmap v983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:18 smithi047 ceph-mon[36835]: pgmap v983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:18 smithi029 ceph-mon[32610]: pgmap v983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:20 smithi077 ceph-mon[37153]: pgmap v984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:20 smithi047 ceph-mon[36835]: pgmap v984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:20 smithi029 ceph-mon[32610]: pgmap v984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:22 smithi077 ceph-mon[37153]: pgmap v985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 944.372122 secs 2024-07-27T07:05:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:22 smithi047 ceph-mon[36835]: pgmap v985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 944.372122 secs 2024-07-27T07:05:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:22 smithi029 ceph-mon[32610]: pgmap v985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 944.372122 secs 2024-07-27T07:05:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:05:23.821+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:05:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:05:23.821+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:05:23.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:05:23.821+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:05:23.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:05:23.825+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:05:23.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:05:23.825+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:05:23.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:05:23.825+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:05:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:24 smithi077 ceph-mon[37153]: pgmap v986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:05:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:24 smithi047 ceph-mon[36835]: pgmap v986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:05:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:24 smithi029 ceph-mon[32610]: pgmap v986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:05:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:26 smithi077 ceph-mon[37153]: pgmap v987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:26 smithi047 ceph-mon[36835]: pgmap v987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:26 smithi029 ceph-mon[32610]: pgmap v987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 949.372205 secs 2024-07-27T07:05:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 949.372205 secs 2024-07-27T07:05:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 949.372205 secs 2024-07-27T07:05:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:28 smithi077 ceph-mon[37153]: pgmap v988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:28 smithi047 ceph-mon[36835]: pgmap v988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:28 smithi029 ceph-mon[32610]: pgmap v988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:30 smithi077 ceph-mon[37153]: pgmap v989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:30 smithi047 ceph-mon[36835]: pgmap v989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:30.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:30 smithi029 ceph-mon[32610]: pgmap v989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:32 smithi077 ceph-mon[37153]: pgmap v990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 954.372333 secs 2024-07-27T07:05:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:32 smithi047 ceph-mon[36835]: pgmap v990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 954.372333 secs 2024-07-27T07:05:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:32 smithi029 ceph-mon[32610]: pgmap v990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 954.372333 secs 2024-07-27T07:05:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:34 smithi077 ceph-mon[37153]: pgmap v991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:05:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:34 smithi047 ceph-mon[36835]: pgmap v991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:05:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:34 smithi029 ceph-mon[32610]: pgmap v991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:05:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:36 smithi077 ceph-mon[37153]: pgmap v992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:36 smithi047 ceph-mon[36835]: pgmap v992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:36 smithi029 ceph-mon[32610]: pgmap v992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 959.372473 secs 2024-07-27T07:05:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 959.372473 secs 2024-07-27T07:05:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 959.372473 secs 2024-07-27T07:05:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:38 smithi077 ceph-mon[37153]: pgmap v993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:38 smithi047 ceph-mon[36835]: pgmap v993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:38 smithi029 ceph-mon[32610]: pgmap v993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:05:38.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:05:38.836+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:05:38.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:05:38.836+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:05:38.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:05:38.836+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:05:38.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:05:38.838+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:05:38.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:05:38.839+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:05:38.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:05:38.839+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:05:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:05:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:05:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:05:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:40 smithi077 ceph-mon[37153]: pgmap v994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:40 smithi047 ceph-mon[36835]: pgmap v994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:40 smithi029 ceph-mon[32610]: pgmap v994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:05:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:41 smithi029 ceph-mon[32610]: pgmap v995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:42.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:41 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 964.372600 secs 2024-07-27T07:05:42.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:05:42.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:05:42.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:41 smithi047 ceph-mon[36835]: pgmap v995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:41 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 964.372600 secs 2024-07-27T07:05:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:05:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:05:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:41 smithi077 ceph-mon[37153]: pgmap v995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:05:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:41 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 964.372600 secs 2024-07-27T07:05:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:05:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:05:42.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:05:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:42 smithi047 ceph-mon[36835]: pgmap v996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:05:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:42 smithi047 ceph-mon[36835]: pgmap v997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:05:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:42 smithi029 ceph-mon[32610]: pgmap v996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:05:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:42 smithi029 ceph-mon[32610]: pgmap v997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:05:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:42 smithi077 ceph-mon[37153]: pgmap v996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:05:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:42 smithi077 ceph-mon[37153]: pgmap v997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:05:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:44 smithi047 ceph-mon[36835]: pgmap v998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:05:45.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:44 smithi029 ceph-mon[32610]: pgmap v998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:05:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:44 smithi077 ceph-mon[37153]: pgmap v998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:05:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:46 smithi047 ceph-mon[36835]: pgmap v999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T07:05:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:46 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 969.372720 secs 2024-07-27T07:05:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:46 smithi029 ceph-mon[32610]: pgmap v999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T07:05:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:46 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 969.372720 secs 2024-07-27T07:05:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:46 smithi077 ceph-mon[37153]: pgmap v999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T07:05:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:46 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 969.372720 secs 2024-07-27T07:05:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:48 smithi047 ceph-mon[36835]: pgmap v1000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:05:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:48 smithi029 ceph-mon[32610]: pgmap v1000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:05:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:48 smithi077 ceph-mon[37153]: pgmap v1000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:05:51.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:50 smithi047 ceph-mon[36835]: pgmap v1001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:05:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:50 smithi029 ceph-mon[32610]: pgmap v1001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:05:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:50 smithi077 ceph-mon[37153]: pgmap v1001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:05:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:51 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 974.372864 secs 2024-07-27T07:05:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:51 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 974.372864 secs 2024-07-27T07:05:52.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:51 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 974.372864 secs 2024-07-27T07:05:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:52 smithi047 ceph-mon[36835]: pgmap v1002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:05:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:52 smithi029 ceph-mon[32610]: pgmap v1002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:05:53.273 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:52 smithi077 ceph-mon[37153]: pgmap v1002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:05:53.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:05:53.840+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:05:53.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:05:53.840+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:05:53.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:05:53.840+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:05:53.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:05:53.845+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:05:53.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:05:53.845+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:05:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:05:53.845+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:05:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:54 smithi047 ceph-mon[36835]: pgmap v1003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:05:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:54 smithi029 ceph-mon[32610]: pgmap v1003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:05:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:54 smithi077 ceph-mon[37153]: pgmap v1003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:05:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:56 smithi047 ceph-mon[36835]: pgmap v1004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:05:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:56 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 979.372986 secs 2024-07-27T07:05:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:56 smithi029 ceph-mon[32610]: pgmap v1004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:05:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:56 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 979.372986 secs 2024-07-27T07:05:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:56 smithi077 ceph-mon[37153]: pgmap v1004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:05:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:56 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 979.372986 secs 2024-07-27T07:05:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:05:58 smithi047 ceph-mon[36835]: pgmap v1005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:05:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:05:58 smithi029 ceph-mon[32610]: pgmap v1005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:05:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:05:58 smithi077 ceph-mon[37153]: pgmap v1005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:06:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:00 smithi047 ceph-mon[36835]: pgmap v1006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:00 smithi029 ceph-mon[32610]: pgmap v1006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:00 smithi077 ceph-mon[37153]: pgmap v1006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:01 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 984.373106 secs 2024-07-27T07:06:02.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:01 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 984.373106 secs 2024-07-27T07:06:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:01 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 984.373106 secs 2024-07-27T07:06:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:03 smithi047 ceph-mon[36835]: pgmap v1007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:03.959 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:03 smithi077 ceph-mon[37153]: pgmap v1007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:03 smithi029 ceph-mon[32610]: pgmap v1007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:05 smithi047 ceph-mon[36835]: pgmap v1008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:06:05.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:05 smithi029 ceph-mon[32610]: pgmap v1008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:06:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:05 smithi077 ceph-mon[37153]: pgmap v1008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:06:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:07 smithi047 ceph-mon[36835]: pgmap v1009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 989.373230 secs 2024-07-27T07:06:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:07 smithi029 ceph-mon[32610]: pgmap v1009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 989.373230 secs 2024-07-27T07:06:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:07 smithi077 ceph-mon[37153]: pgmap v1009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 989.373230 secs 2024-07-27T07:06:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:08.856+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:06:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:08.856+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:06:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:08.856+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:06:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:08.861+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:06:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:08.861+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:06:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:08.861+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:06:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:09 smithi047 ceph-mon[36835]: pgmap v1010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:06:09.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:09 smithi029 ceph-mon[32610]: pgmap v1010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:06:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:09 smithi077 ceph-mon[37153]: pgmap v1010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:06:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:11 smithi047 ceph-mon[36835]: pgmap v1011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:11 smithi029 ceph-mon[32610]: pgmap v1011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:11 smithi077 ceph-mon[37153]: pgmap v1011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:12.773+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:06:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:12.773+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f73200065f0 con 0x7f7348108670 2024-07-27T07:06:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:12.774+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:06:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:12.774+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f73200065f0 con 0x7f7348108670 2024-07-27T07:06:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 994.373356 secs 2024-07-27T07:06:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 994.373356 secs 2024-07-27T07:06:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 994.373356 secs 2024-07-27T07:06:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:13 smithi029 ceph-mon[32610]: pgmap v1012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:13 smithi077 ceph-mon[37153]: pgmap v1012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:13 smithi047 ceph-mon[36835]: pgmap v1012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:15 smithi029 ceph-mon[32610]: pgmap v1013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:06:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:15 smithi077 ceph-mon[37153]: pgmap v1013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:06:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:15 smithi047 ceph-mon[36835]: pgmap v1013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:06:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:17 smithi029 ceph-mon[32610]: pgmap v1014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 999.373476 secs 2024-07-27T07:06:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:17 smithi077 ceph-mon[37153]: pgmap v1014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 999.373476 secs 2024-07-27T07:06:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:17 smithi047 ceph-mon[36835]: pgmap v1014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 999.373476 secs 2024-07-27T07:06:19.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:18 smithi047 ceph-mon[36835]: pgmap v1015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:06:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:18 smithi077 ceph-mon[37153]: pgmap v1015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:06:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:18 smithi029 ceph-mon[32610]: pgmap v1015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:06:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:20 smithi047 ceph-mon[36835]: pgmap v1016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:20 smithi029 ceph-mon[32610]: pgmap v1016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:20 smithi077 ceph-mon[37153]: pgmap v1016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:21 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1004.373602 secs 2024-07-27T07:06:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:21 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1004.373602 secs 2024-07-27T07:06:22.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:21 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1004.373602 secs 2024-07-27T07:06:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:22 smithi047 ceph-mon[36835]: pgmap v1017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:22 smithi029 ceph-mon[32610]: pgmap v1017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:22 smithi077 ceph-mon[37153]: pgmap v1017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:23.871+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:06:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:23.871+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:06:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:23.871+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:06:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:23.877+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:06:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:23.877+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:06:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:23.877+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:06:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:24 smithi047 ceph-mon[36835]: pgmap v1018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:06:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:24 smithi029 ceph-mon[32610]: pgmap v1018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:06:25.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:24 smithi077 ceph-mon[37153]: pgmap v1018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:06:27.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:26 smithi047 ceph-mon[36835]: pgmap v1019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:27.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:26 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1009.373738 secs 2024-07-27T07:06:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:26 smithi029 ceph-mon[32610]: pgmap v1019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:26 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1009.373738 secs 2024-07-27T07:06:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:26 smithi077 ceph-mon[37153]: pgmap v1019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:26 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1009.373738 secs 2024-07-27T07:06:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:27.786+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:06:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:27.786+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff240065e0 con 0x7eff50104ab0 2024-07-27T07:06:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:27.786+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff240065e0 con 0x7eff50104ab0 2024-07-27T07:06:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:27.786+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:06:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:28 smithi047 ceph-mon[36835]: pgmap v1020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:06:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:28 smithi029 ceph-mon[32610]: pgmap v1020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:06:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:28 smithi077 ceph-mon[37153]: pgmap v1020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:06:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:30 smithi047 ceph-mon[36835]: pgmap v1021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:30 smithi029 ceph-mon[32610]: pgmap v1021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:30 smithi077 ceph-mon[37153]: pgmap v1021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:31 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1014.373883 secs 2024-07-27T07:06:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:31 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1014.373883 secs 2024-07-27T07:06:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:31 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1014.373883 secs 2024-07-27T07:06:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:32 smithi047 ceph-mon[36835]: pgmap v1022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:32 smithi029 ceph-mon[32610]: pgmap v1022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:32 smithi077 ceph-mon[37153]: pgmap v1022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:35.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:34 smithi047 ceph-mon[36835]: pgmap v1023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:06:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:34 smithi029 ceph-mon[32610]: pgmap v1023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:06:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:34 smithi077 ceph-mon[37153]: pgmap v1023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:06:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:36 smithi029 ceph-mon[32610]: pgmap v1024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:36 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1019.373978 secs 2024-07-27T07:06:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:36 smithi077 ceph-mon[37153]: pgmap v1024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:36 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1019.373978 secs 2024-07-27T07:06:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:36 smithi047 ceph-mon[36835]: pgmap v1024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:36 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1019.373978 secs 2024-07-27T07:06:38.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:38.886+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:06:38.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:38.886+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:06:38.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:38.886+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:06:38.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:38.891+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:06:38.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:38.891+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:06:38.893 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:38.891+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:06:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:38 smithi029 ceph-mon[32610]: pgmap v1025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:06:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:38 smithi077 ceph-mon[37153]: pgmap v1025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:06:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:38 smithi047 ceph-mon[36835]: pgmap v1025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:06:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:40 smithi029 ceph-mon[32610]: pgmap v1026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:40 smithi077 ceph-mon[37153]: pgmap v1026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:40 smithi047 ceph-mon[36835]: pgmap v1026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:41 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1024.374099 secs 2024-07-27T07:06:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:41 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1024.374099 secs 2024-07-27T07:06:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:41 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1024.374099 secs 2024-07-27T07:06:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:42 smithi029 ceph-mon[32610]: pgmap v1027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:06:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:42 smithi077 ceph-mon[37153]: pgmap v1027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:42 smithi047 ceph-mon[36835]: pgmap v1027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:06:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:06:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:06:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:43 smithi077 ceph-mon[37153]: pgmap v1028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:06:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:43 smithi077 ceph-mon[37153]: pgmap v1029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:06:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:06:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:06:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:43 smithi047 ceph-mon[36835]: pgmap v1028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:06:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:43 smithi047 ceph-mon[36835]: pgmap v1029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:06:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:06:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:06:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:43 smithi029 ceph-mon[32610]: pgmap v1028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:06:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:43 smithi029 ceph-mon[32610]: pgmap v1029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:06:44.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:06:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:46 smithi077 ceph-mon[37153]: pgmap v1030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T07:06:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:46 smithi047 ceph-mon[36835]: pgmap v1030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T07:06:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:46 smithi029 ceph-mon[32610]: pgmap v1030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T07:06:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1029.374228 secs 2024-07-27T07:06:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1029.374228 secs 2024-07-27T07:06:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1029.374228 secs 2024-07-27T07:06:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:48 smithi077 ceph-mon[37153]: pgmap v1031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T07:06:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:48 smithi047 ceph-mon[36835]: pgmap v1031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T07:06:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:48 smithi029 ceph-mon[32610]: pgmap v1031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T07:06:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:50 smithi077 ceph-mon[37153]: pgmap v1032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:06:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:50 smithi047 ceph-mon[36835]: pgmap v1032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:06:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:50 smithi029 ceph-mon[32610]: pgmap v1032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:06:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:52 smithi077 ceph-mon[37153]: pgmap v1033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:06:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1034.374347 secs 2024-07-27T07:06:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:52 smithi047 ceph-mon[36835]: pgmap v1033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:06:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1034.374347 secs 2024-07-27T07:06:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:52 smithi029 ceph-mon[32610]: pgmap v1033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:06:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1034.374347 secs 2024-07-27T07:06:53.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:53.900+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:06:53.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:53.900+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:06:53.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:06:53.900+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:06:53.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:53.901+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:06:53.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:53.901+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:06:53.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:06:53.901+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:06:54.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:54 smithi029 ceph-mon[32610]: pgmap v1034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:06:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:54 smithi077 ceph-mon[37153]: pgmap v1034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:06:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:54 smithi047 ceph-mon[36835]: pgmap v1034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:06:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:56 smithi077 ceph-mon[37153]: pgmap v1035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:56 smithi047 ceph-mon[36835]: pgmap v1035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:56 smithi029 ceph-mon[32610]: pgmap v1035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:06:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1039.374469 secs 2024-07-27T07:06:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1039.374469 secs 2024-07-27T07:06:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1039.374469 secs 2024-07-27T07:06:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:06:58 smithi029 ceph-mon[32610]: pgmap v1036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:06:58 smithi077 ceph-mon[37153]: pgmap v1036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:06:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:06:58 smithi047 ceph-mon[36835]: pgmap v1036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:00 smithi029 ceph-mon[32610]: pgmap v1037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:07:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:00 smithi077 ceph-mon[37153]: pgmap v1037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:07:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:00 smithi047 ceph-mon[36835]: pgmap v1037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:07:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:02 smithi029 ceph-mon[32610]: pgmap v1038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1044.374579 secs 2024-07-27T07:07:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:02 smithi077 ceph-mon[37153]: pgmap v1038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1044.374579 secs 2024-07-27T07:07:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:02 smithi047 ceph-mon[36835]: pgmap v1038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1044.374579 secs 2024-07-27T07:07:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:04 smithi029 ceph-mon[32610]: pgmap v1039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:07:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:04 smithi077 ceph-mon[37153]: pgmap v1039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:07:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:04 smithi047 ceph-mon[36835]: pgmap v1039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:07:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:06 smithi029 ceph-mon[32610]: pgmap v1040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:06 smithi077 ceph-mon[37153]: pgmap v1040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:06 smithi047 ceph-mon[36835]: pgmap v1040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1049.374712 secs 2024-07-27T07:07:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1049.374712 secs 2024-07-27T07:07:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1049.374712 secs 2024-07-27T07:07:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:08.906+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:07:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:08.906+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:07:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:08.906+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:07:08.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:08.914+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:07:08.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:08.914+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:07:08.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:08.914+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:07:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:08 smithi077 ceph-mon[37153]: pgmap v1041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:08 smithi047 ceph-mon[36835]: pgmap v1041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:09.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:08 smithi029 ceph-mon[32610]: pgmap v1041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:09 smithi077 ceph-mon[37153]: pgmap v1042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:07:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:09 smithi047 ceph-mon[36835]: pgmap v1042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:07:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:09 smithi029 ceph-mon[32610]: pgmap v1042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:07:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:12 smithi077 ceph-mon[37153]: pgmap v1043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1054.374838 secs 2024-07-27T07:07:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:12 smithi047 ceph-mon[36835]: pgmap v1043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:12.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1054.374838 secs 2024-07-27T07:07:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:12 smithi029 ceph-mon[32610]: pgmap v1043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1054.374838 secs 2024-07-27T07:07:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:14 smithi077 ceph-mon[37153]: pgmap v1044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:07:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:14 smithi047 ceph-mon[36835]: pgmap v1044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:07:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:14 smithi029 ceph-mon[32610]: pgmap v1044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:07:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:16 smithi029 ceph-mon[32610]: pgmap v1045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:16 smithi077 ceph-mon[37153]: pgmap v1045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:16 smithi047 ceph-mon[36835]: pgmap v1045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1059.374956 secs 2024-07-27T07:07:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1059.374956 secs 2024-07-27T07:07:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1059.374956 secs 2024-07-27T07:07:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:18 smithi077 ceph-mon[37153]: pgmap v1046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:18 smithi047 ceph-mon[36835]: pgmap v1046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:18 smithi029 ceph-mon[32610]: pgmap v1046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:20 smithi077 ceph-mon[37153]: pgmap v1047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:07:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:20 smithi047 ceph-mon[36835]: pgmap v1047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:07:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:20 smithi029 ceph-mon[32610]: pgmap v1047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:07:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:22 smithi077 ceph-mon[37153]: pgmap v1048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1064.375081 secs 2024-07-27T07:07:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:22 smithi047 ceph-mon[36835]: pgmap v1048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1064.375081 secs 2024-07-27T07:07:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:22 smithi029 ceph-mon[32610]: pgmap v1048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1064.375081 secs 2024-07-27T07:07:23.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:23.912+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:07:23.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:23.912+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:07:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:23.912+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:07:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:23.920+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:07:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:23.920+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:07:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:23.920+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:07:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:24 smithi029 ceph-mon[32610]: pgmap v1049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:07:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:24 smithi077 ceph-mon[37153]: pgmap v1049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:07:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:24 smithi047 ceph-mon[36835]: pgmap v1049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:07:26.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:26 smithi077 ceph-mon[37153]: pgmap v1050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:26 smithi047 ceph-mon[36835]: pgmap v1050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:26 smithi029 ceph-mon[32610]: pgmap v1050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1069.375214 secs 2024-07-27T07:07:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1069.375214 secs 2024-07-27T07:07:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1069.375214 secs 2024-07-27T07:07:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:28 smithi077 ceph-mon[37153]: pgmap v1051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:28 smithi047 ceph-mon[36835]: pgmap v1051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:28 smithi029 ceph-mon[32610]: pgmap v1051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:30 smithi077 ceph-mon[37153]: pgmap v1052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:07:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:30 smithi047 ceph-mon[36835]: pgmap v1052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:07:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:30 smithi029 ceph-mon[32610]: pgmap v1052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:07:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:32 smithi077 ceph-mon[37153]: pgmap v1053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1074.375347 secs 2024-07-27T07:07:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:32 smithi047 ceph-mon[36835]: pgmap v1053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:32.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1074.375347 secs 2024-07-27T07:07:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:32 smithi029 ceph-mon[32610]: pgmap v1053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1074.375347 secs 2024-07-27T07:07:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:34 smithi047 ceph-mon[36835]: pgmap v1054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:07:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:34 smithi029 ceph-mon[32610]: pgmap v1054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:07:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:34 smithi077 ceph-mon[37153]: pgmap v1054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:07:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:36 smithi047 ceph-mon[36835]: pgmap v1055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:07:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:36 smithi029 ceph-mon[32610]: pgmap v1055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:07:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:36 smithi077 ceph-mon[37153]: pgmap v1055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:07:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1079.375489 secs 2024-07-27T07:07:37.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1079.375489 secs 2024-07-27T07:07:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1079.375489 secs 2024-07-27T07:07:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:38 smithi047 ceph-mon[36835]: pgmap v1056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:38 smithi029 ceph-mon[32610]: pgmap v1056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:38 smithi077 ceph-mon[37153]: pgmap v1056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:07:38.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:38.927+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:07:38.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:38.927+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:07:38.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:38.927+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:07:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:38.933+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:07:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:38.933+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:07:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:38.933+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:07:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:40 smithi047 ceph-mon[36835]: pgmap v1057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:07:40.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:40 smithi029 ceph-mon[32610]: pgmap v1057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:07:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:40 smithi077 ceph-mon[37153]: pgmap v1057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:07:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:42 smithi047 ceph-mon[36835]: pgmap v1058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:07:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1084.375628 secs 2024-07-27T07:07:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:42 smithi029 ceph-mon[32610]: pgmap v1058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:07:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1084.375628 secs 2024-07-27T07:07:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:42 smithi077 ceph-mon[37153]: pgmap v1058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:07:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1084.375628 secs 2024-07-27T07:07:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:42.786+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:07:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:42.786+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:07:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:42.787+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:07:42.789 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:42.787+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:07:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:07:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:07:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:07:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:44 smithi047 ceph-mon[36835]: pgmap v1059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:07:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:07:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:07:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:44 smithi047 ceph-mon[36835]: pgmap v1060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:07:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:44 smithi047 ceph-mon[36835]: pgmap v1061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:07:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:07:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:44 smithi029 ceph-mon[32610]: pgmap v1059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:07:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:07:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:07:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:44 smithi029 ceph-mon[32610]: pgmap v1060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:07:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:44 smithi029 ceph-mon[32610]: pgmap v1061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:07:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:07:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:44 smithi077 ceph-mon[37153]: pgmap v1059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:07:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:07:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:07:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:44 smithi077 ceph-mon[37153]: pgmap v1060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:07:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:44 smithi077 ceph-mon[37153]: pgmap v1061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:07:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:07:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:46 smithi047 ceph-mon[36835]: pgmap v1062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:07:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:46 smithi029 ceph-mon[32610]: pgmap v1062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:07:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:46 smithi077 ceph-mon[37153]: pgmap v1062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:07:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1089.375753 secs 2024-07-27T07:07:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1089.375753 secs 2024-07-27T07:07:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1089.375753 secs 2024-07-27T07:07:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:48 smithi047 ceph-mon[36835]: pgmap v1063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:07:48.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:48 smithi029 ceph-mon[32610]: pgmap v1063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:07:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:48 smithi077 ceph-mon[37153]: pgmap v1063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:07:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:50 smithi047 ceph-mon[36835]: pgmap v1064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:07:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:50 smithi029 ceph-mon[32610]: pgmap v1064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:07:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:50 smithi077 ceph-mon[37153]: pgmap v1064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:07:52.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:52 smithi077 ceph-mon[37153]: pgmap v1065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:07:52.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1094.375863 secs 2024-07-27T07:07:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:52 smithi047 ceph-mon[36835]: pgmap v1065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:07:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1094.375863 secs 2024-07-27T07:07:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:52 smithi029 ceph-mon[32610]: pgmap v1065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:07:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1094.375863 secs 2024-07-27T07:07:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:53.935+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:07:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:53.935+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:07:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:07:53.935+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:07:53.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:53.943+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:07:53.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:53.943+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:07:53.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:53.943+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:07:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:54 smithi047 ceph-mon[36835]: pgmap v1066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:07:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:54 smithi029 ceph-mon[32610]: pgmap v1066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:07:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:54 smithi077 ceph-mon[37153]: pgmap v1066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:07:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:56 smithi047 ceph-mon[36835]: pgmap v1067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:56 smithi029 ceph-mon[32610]: pgmap v1067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:56 smithi077 ceph-mon[37153]: pgmap v1067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:07:57.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:57.800+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:07:57.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:57.800+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:07:57.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:57.800+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:07:57.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:07:57.800+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:07:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1099.376017 secs 2024-07-27T07:07:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1099.376017 secs 2024-07-27T07:07:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1099.376017 secs 2024-07-27T07:07:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:07:58 smithi047 ceph-mon[36835]: pgmap v1068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:07:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:07:58 smithi029 ceph-mon[32610]: pgmap v1068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:07:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:07:58 smithi077 ceph-mon[37153]: pgmap v1068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:00 smithi047 ceph-mon[36835]: pgmap v1069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:00 smithi029 ceph-mon[32610]: pgmap v1069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:00 smithi077 ceph-mon[37153]: pgmap v1069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:02 smithi047 ceph-mon[36835]: pgmap v1070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1104.376169 secs 2024-07-27T07:08:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:02 smithi029 ceph-mon[32610]: pgmap v1070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1104.376169 secs 2024-07-27T07:08:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:02 smithi077 ceph-mon[37153]: pgmap v1070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1104.376169 secs 2024-07-27T07:08:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:04 smithi047 ceph-mon[36835]: pgmap v1071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:04 smithi029 ceph-mon[32610]: pgmap v1071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:04 smithi077 ceph-mon[37153]: pgmap v1071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:06 smithi047 ceph-mon[36835]: pgmap v1072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:06 smithi029 ceph-mon[32610]: pgmap v1072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:06 smithi077 ceph-mon[37153]: pgmap v1072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1109.376265 secs 2024-07-27T07:08:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1109.376265 secs 2024-07-27T07:08:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1109.376265 secs 2024-07-27T07:08:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:08 smithi047 ceph-mon[36835]: pgmap v1073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:08.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:08:08.951+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:08:08.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:08:08.951+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:08:08.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:08:08.951+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:08:08.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:08:08.958+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:08:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:08:08.958+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:08:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:08:08.958+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:08:08.959 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:08 smithi029 ceph-mon[32610]: pgmap v1073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:08 smithi077 ceph-mon[37153]: pgmap v1073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:10 smithi047 ceph-mon[36835]: pgmap v1074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:10 smithi029 ceph-mon[32610]: pgmap v1074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:10 smithi077 ceph-mon[37153]: pgmap v1074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:12 smithi047 ceph-mon[36835]: pgmap v1075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1114.376391 secs 2024-07-27T07:08:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:12 smithi029 ceph-mon[32610]: pgmap v1075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1114.376391 secs 2024-07-27T07:08:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:12 smithi077 ceph-mon[37153]: pgmap v1075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1114.376391 secs 2024-07-27T07:08:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:14 smithi047 ceph-mon[36835]: pgmap v1076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:14 smithi029 ceph-mon[32610]: pgmap v1076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:14 smithi077 ceph-mon[37153]: pgmap v1076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:16 smithi047 ceph-mon[36835]: pgmap v1077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:16 smithi029 ceph-mon[32610]: pgmap v1077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:16 smithi077 ceph-mon[37153]: pgmap v1077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1119.376540 secs 2024-07-27T07:08:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1119.376540 secs 2024-07-27T07:08:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1119.376540 secs 2024-07-27T07:08:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:18 smithi047 ceph-mon[36835]: pgmap v1078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:18 smithi029 ceph-mon[32610]: pgmap v1078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:18 smithi077 ceph-mon[37153]: pgmap v1078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:20 smithi047 ceph-mon[36835]: pgmap v1079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:20 smithi029 ceph-mon[32610]: pgmap v1079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:20 smithi077 ceph-mon[37153]: pgmap v1079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:22 smithi047 ceph-mon[36835]: pgmap v1080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1124.376667 secs 2024-07-27T07:08:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:22 smithi029 ceph-mon[32610]: pgmap v1080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1124.376667 secs 2024-07-27T07:08:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:22 smithi077 ceph-mon[37153]: pgmap v1080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1124.376667 secs 2024-07-27T07:08:23.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:08:23.967+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:08:23.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:08:23.967+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:08:23.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:08:23.967+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:08:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:08:23.971+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:08:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:08:23.971+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:08:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:08:23.971+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:08:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:24 smithi029 ceph-mon[32610]: pgmap v1081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:24 smithi077 ceph-mon[37153]: pgmap v1081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:24 smithi047 ceph-mon[36835]: pgmap v1081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:26 smithi029 ceph-mon[32610]: pgmap v1082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:26 smithi077 ceph-mon[37153]: pgmap v1082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:26 smithi047 ceph-mon[36835]: pgmap v1082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1129.376818 secs 2024-07-27T07:08:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1129.376818 secs 2024-07-27T07:08:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1129.376818 secs 2024-07-27T07:08:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:28 smithi029 ceph-mon[32610]: pgmap v1083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:28 smithi077 ceph-mon[37153]: pgmap v1083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:28 smithi047 ceph-mon[36835]: pgmap v1083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:30 smithi077 ceph-mon[37153]: pgmap v1084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:30 smithi047 ceph-mon[36835]: pgmap v1084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:30 smithi029 ceph-mon[32610]: pgmap v1084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:32 smithi077 ceph-mon[37153]: pgmap v1085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1134.376925 secs 2024-07-27T07:08:33.191 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:32 smithi029 ceph-mon[32610]: pgmap v1085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:33.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1134.376925 secs 2024-07-27T07:08:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:32 smithi047 ceph-mon[36835]: pgmap v1085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1134.376925 secs 2024-07-27T07:08:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:34 smithi077 ceph-mon[37153]: pgmap v1086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:34 smithi047 ceph-mon[36835]: pgmap v1086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:34 smithi029 ceph-mon[32610]: pgmap v1086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:36 smithi077 ceph-mon[37153]: pgmap v1087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:36 smithi047 ceph-mon[36835]: pgmap v1087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:36 smithi029 ceph-mon[32610]: pgmap v1087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1139.377047 secs 2024-07-27T07:08:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1139.377047 secs 2024-07-27T07:08:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1139.377047 secs 2024-07-27T07:08:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:08:38.976+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:08:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:08:38.976+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:08:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:08:38.976+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:08:38.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:08:38.978+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:08:38.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:08:38.978+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:08:38.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:08:38.978+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:08:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:38 smithi077 ceph-mon[37153]: pgmap v1088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:38 smithi047 ceph-mon[36835]: pgmap v1088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:39.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:38 smithi029 ceph-mon[32610]: pgmap v1088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:08:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:39 smithi077 ceph-mon[37153]: pgmap v1089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:39 smithi047 ceph-mon[36835]: pgmap v1089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:39 smithi029 ceph-mon[32610]: pgmap v1089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:08:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:42 smithi047 ceph-mon[36835]: pgmap v1090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1144.377172 secs 2024-07-27T07:08:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:42 smithi029 ceph-mon[32610]: pgmap v1090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1144.377172 secs 2024-07-27T07:08:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:42 smithi077 ceph-mon[37153]: pgmap v1090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:08:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1144.377172 secs 2024-07-27T07:08:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:44 smithi047 ceph-mon[36835]: pgmap v1091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:08:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:08:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:08:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:08:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:44 smithi029 ceph-mon[32610]: pgmap v1091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:08:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:08:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:08:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:08:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:44 smithi077 ceph-mon[37153]: pgmap v1091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:08:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:08:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:08:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:08:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:08:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:45 smithi047 ceph-mon[36835]: pgmap v1092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:08:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:45 smithi047 ceph-mon[36835]: pgmap v1093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:08:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:45 smithi029 ceph-mon[32610]: pgmap v1092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:08:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:45 smithi029 ceph-mon[32610]: pgmap v1093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:08:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:45 smithi077 ceph-mon[37153]: pgmap v1092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:08:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:45 smithi077 ceph-mon[37153]: pgmap v1093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:08:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:47 smithi047 ceph-mon[36835]: pgmap v1094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:08:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1149.377250 secs 2024-07-27T07:08:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:47 smithi029 ceph-mon[32610]: pgmap v1094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:08:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1149.377250 secs 2024-07-27T07:08:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:47 smithi077 ceph-mon[37153]: pgmap v1094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:08:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1149.377250 secs 2024-07-27T07:08:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:49 smithi047 ceph-mon[36835]: pgmap v1095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:08:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:49 smithi029 ceph-mon[32610]: pgmap v1095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:08:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:49 smithi077 ceph-mon[37153]: pgmap v1095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:08:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:51 smithi047 ceph-mon[36835]: pgmap v1096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:08:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:51 smithi029 ceph-mon[32610]: pgmap v1096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:08:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:51 smithi077 ceph-mon[37153]: pgmap v1096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:08:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1154.377336 secs 2024-07-27T07:08:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1154.377336 secs 2024-07-27T07:08:53.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1154.377336 secs 2024-07-27T07:08:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:53 smithi047 ceph-mon[36835]: pgmap v1097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:08:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:53 smithi029 ceph-mon[32610]: pgmap v1097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:08:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:08:53.983+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:08:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:08:53.983+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:08:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:08:53.983+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:08:53.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:08:53.984+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:08:53.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:08:53.984+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:08:53.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:08:53.984+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:08:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:53 smithi077 ceph-mon[37153]: pgmap v1097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:08:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:55 smithi047 ceph-mon[36835]: pgmap v1098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:08:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:55 smithi029 ceph-mon[32610]: pgmap v1098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:08:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:55 smithi077 ceph-mon[37153]: pgmap v1098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:08:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:57 smithi047 ceph-mon[36835]: pgmap v1099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1159.377401 secs 2024-07-27T07:08:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:57 smithi029 ceph-mon[32610]: pgmap v1099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1159.377401 secs 2024-07-27T07:08:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:57 smithi077 ceph-mon[37153]: pgmap v1099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:08:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1159.377401 secs 2024-07-27T07:08:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:08:59 smithi047 ceph-mon[36835]: pgmap v1100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:08:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:08:59 smithi029 ceph-mon[32610]: pgmap v1100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:08:59 smithi077 ceph-mon[37153]: pgmap v1100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:01 smithi047 ceph-mon[36835]: pgmap v1101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:09:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:01 smithi029 ceph-mon[32610]: pgmap v1101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:09:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:01 smithi077 ceph-mon[37153]: pgmap v1101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:09:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1164.377541 secs 2024-07-27T07:09:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1164.377541 secs 2024-07-27T07:09:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1164.377541 secs 2024-07-27T07:09:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:03 smithi047 ceph-mon[36835]: pgmap v1102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:03 smithi029 ceph-mon[32610]: pgmap v1102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:03 smithi077 ceph-mon[37153]: pgmap v1102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:05 smithi047 ceph-mon[36835]: pgmap v1103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:05 smithi029 ceph-mon[32610]: pgmap v1103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:05 smithi077 ceph-mon[37153]: pgmap v1103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:07 smithi047 ceph-mon[36835]: pgmap v1104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1169.377683 secs 2024-07-27T07:09:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:07 smithi029 ceph-mon[32610]: pgmap v1104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1169.377683 secs 2024-07-27T07:09:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:07 smithi077 ceph-mon[37153]: pgmap v1104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1169.377683 secs 2024-07-27T07:09:08.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:09:08.991+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:09:08.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:09:08.991+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:09:08.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:09:08.991+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:09:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:09:08.996+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:09:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:09:08.996+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:09:08.996 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:09:08.996+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:09:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:09 smithi077 ceph-mon[37153]: pgmap v1105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:09 smithi047 ceph-mon[36835]: pgmap v1105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:09 smithi029 ceph-mon[32610]: pgmap v1105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:10 smithi077 ceph-mon[37153]: pgmap v1106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:10 smithi047 ceph-mon[36835]: pgmap v1106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:10 smithi029 ceph-mon[32610]: pgmap v1106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:11 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1174.377776 secs 2024-07-27T07:09:12.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:11 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1174.377776 secs 2024-07-27T07:09:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:11 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1174.377776 secs 2024-07-27T07:09:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:12 smithi047 ceph-mon[36835]: pgmap v1107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:12 smithi029 ceph-mon[32610]: pgmap v1107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:12 smithi077 ceph-mon[37153]: pgmap v1107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:15 smithi047 ceph-mon[36835]: pgmap v1108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:15 smithi029 ceph-mon[32610]: pgmap v1108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:15 smithi077 ceph-mon[37153]: pgmap v1108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:17 smithi047 ceph-mon[36835]: pgmap v1109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1179.377844 secs 2024-07-27T07:09:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:17 smithi029 ceph-mon[32610]: pgmap v1109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1179.377844 secs 2024-07-27T07:09:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:17 smithi077 ceph-mon[37153]: pgmap v1109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1179.377844 secs 2024-07-27T07:09:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:19 smithi047 ceph-mon[36835]: pgmap v1110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:19 smithi029 ceph-mon[32610]: pgmap v1110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:19 smithi077 ceph-mon[37153]: pgmap v1110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:21 smithi047 ceph-mon[36835]: pgmap v1111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:21 smithi029 ceph-mon[32610]: pgmap v1111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:21 smithi077 ceph-mon[37153]: pgmap v1111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1184.377966 secs 2024-07-27T07:09:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1184.377966 secs 2024-07-27T07:09:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1184.377966 secs 2024-07-27T07:09:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:23 smithi029 ceph-mon[32610]: pgmap v1112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:23.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:23 smithi077 ceph-mon[37153]: pgmap v1112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:23 smithi047 ceph-mon[36835]: pgmap v1112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:09:23.998+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:09:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:09:23.998+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:09:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:09:23.998+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:09:24.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:09:24.003+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:09:24.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:09:24.003+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:09:24.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:09:24.003+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:09:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:25 smithi029 ceph-mon[32610]: pgmap v1113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:25 smithi077 ceph-mon[37153]: pgmap v1113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:25 smithi047 ceph-mon[36835]: pgmap v1113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:27 smithi029 ceph-mon[32610]: pgmap v1114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1189.378069 secs 2024-07-27T07:09:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:27 smithi077 ceph-mon[37153]: pgmap v1114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1189.378069 secs 2024-07-27T07:09:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:27 smithi047 ceph-mon[36835]: pgmap v1114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1189.378069 secs 2024-07-27T07:09:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:29 smithi029 ceph-mon[32610]: pgmap v1115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:29 smithi077 ceph-mon[37153]: pgmap v1115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:29 smithi047 ceph-mon[36835]: pgmap v1115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:31 smithi077 ceph-mon[37153]: pgmap v1116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:31 smithi047 ceph-mon[36835]: pgmap v1116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:31 smithi029 ceph-mon[32610]: pgmap v1116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1194.378190 secs 2024-07-27T07:09:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1194.378190 secs 2024-07-27T07:09:32.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1194.378190 secs 2024-07-27T07:09:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:33 smithi077 ceph-mon[37153]: pgmap v1117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:33 smithi047 ceph-mon[36835]: pgmap v1117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:33 smithi029 ceph-mon[32610]: pgmap v1117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:35 smithi077 ceph-mon[37153]: pgmap v1118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:35 smithi047 ceph-mon[36835]: pgmap v1118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:35 smithi029 ceph-mon[32610]: pgmap v1118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:37 smithi077 ceph-mon[37153]: pgmap v1119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1199.378318 secs 2024-07-27T07:09:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:37 smithi047 ceph-mon[36835]: pgmap v1119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1199.378318 secs 2024-07-27T07:09:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:37 smithi029 ceph-mon[32610]: pgmap v1119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1199.378318 secs 2024-07-27T07:09:39.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:09:39.012+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:09:39.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:09:39.012+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:09:39.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:09:39.012+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:09:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:09:39.012+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:09:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:09:39.012+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:09:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:09:39.012+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:09:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:39 smithi029 ceph-mon[32610]: pgmap v1120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:39 smithi077 ceph-mon[37153]: pgmap v1120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:39 smithi047 ceph-mon[36835]: pgmap v1120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:09:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:41 smithi077 ceph-mon[37153]: pgmap v1121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:41 smithi047 ceph-mon[36835]: pgmap v1121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:41 smithi029 ceph-mon[32610]: pgmap v1121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:09:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1204.378439 secs 2024-07-27T07:09:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1204.378439 secs 2024-07-27T07:09:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1204.378439 secs 2024-07-27T07:09:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:43 smithi077 ceph-mon[37153]: pgmap v1122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:43 smithi047 ceph-mon[36835]: pgmap v1122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:43 smithi029 ceph-mon[32610]: pgmap v1122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:09:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:09:44.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:09:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:45 smithi077 ceph-mon[37153]: pgmap v1123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:09:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:09:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:45 smithi077 ceph-mon[37153]: pgmap v1124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:09:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:45 smithi077 ceph-mon[37153]: pgmap v1125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:09:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:09:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:45 smithi047 ceph-mon[36835]: pgmap v1123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:09:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:09:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:45 smithi047 ceph-mon[36835]: pgmap v1124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:09:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:45 smithi047 ceph-mon[36835]: pgmap v1125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:09:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:09:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:45 smithi029 ceph-mon[32610]: pgmap v1123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:09:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:09:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:09:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:45 smithi029 ceph-mon[32610]: pgmap v1124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:09:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:45 smithi029 ceph-mon[32610]: pgmap v1125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:09:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:09:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1209.378558 secs 2024-07-27T07:09:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:47 smithi077 ceph-mon[37153]: pgmap v1126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:09:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1209.378558 secs 2024-07-27T07:09:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:47 smithi047 ceph-mon[36835]: pgmap v1126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:09:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1209.378558 secs 2024-07-27T07:09:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:47 smithi029 ceph-mon[32610]: pgmap v1126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:09:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:49 smithi077 ceph-mon[37153]: pgmap v1127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:09:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:49 smithi047 ceph-mon[36835]: pgmap v1127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:09:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:49 smithi029 ceph-mon[32610]: pgmap v1127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:09:51.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:51 smithi047 ceph-mon[36835]: pgmap v1128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:09:52.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:51 smithi029 ceph-mon[32610]: pgmap v1128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:09:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:51 smithi077 ceph-mon[37153]: pgmap v1128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:09:53.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1214.378714 secs 2024-07-27T07:09:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1214.378714 secs 2024-07-27T07:09:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1214.378714 secs 2024-07-27T07:09:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:09:54.014+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:09:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:09:54.014+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:09:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:09:54.014+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:09:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:09:54.014+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:09:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:09:54.014+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:09:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:09:54.014+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:09:54.017 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:53 smithi029 ceph-mon[32610]: pgmap v1129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:09:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:53 smithi077 ceph-mon[37153]: pgmap v1129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:09:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:53 smithi047 ceph-mon[36835]: pgmap v1129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:09:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:54 smithi077 ceph-mon[37153]: pgmap v1130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:09:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:54 smithi047 ceph-mon[36835]: pgmap v1130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:09:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:54 smithi029 ceph-mon[32610]: pgmap v1130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:09:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1219.378832 secs 2024-07-27T07:09:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:57 smithi077 ceph-mon[37153]: pgmap v1131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1219.378832 secs 2024-07-27T07:09:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:57 smithi047 ceph-mon[36835]: pgmap v1131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:09:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1219.378832 secs 2024-07-27T07:09:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:57 smithi029 ceph-mon[32610]: pgmap v1131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:09:59 smithi077 ceph-mon[37153]: pgmap v1132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:10:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:09:59 smithi047 ceph-mon[36835]: pgmap v1132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:10:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:09:59 smithi029 ceph-mon[32610]: pgmap v1132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:10:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:10:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:10:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:10:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:01 smithi077 ceph-mon[37153]: pgmap v1133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:10:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:01 smithi047 ceph-mon[36835]: pgmap v1133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:10:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:01 smithi029 ceph-mon[32610]: pgmap v1133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:10:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1224.378959 secs 2024-07-27T07:10:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:02 smithi077 ceph-mon[37153]: pgmap v1134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:10:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1224.378959 secs 2024-07-27T07:10:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:02 smithi047 ceph-mon[36835]: pgmap v1134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:10:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1224.378959 secs 2024-07-27T07:10:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:02 smithi029 ceph-mon[32610]: pgmap v1134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:10:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:05 smithi077 ceph-mon[37153]: pgmap v1135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:10:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:05 smithi047 ceph-mon[36835]: pgmap v1135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:10:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:05 smithi029 ceph-mon[32610]: pgmap v1135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:10:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1229.379082 secs 2024-07-27T07:10:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:07 smithi077 ceph-mon[37153]: pgmap v1136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1229.379082 secs 2024-07-27T07:10:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:07 smithi047 ceph-mon[36835]: pgmap v1136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1229.379082 secs 2024-07-27T07:10:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:07 smithi029 ceph-mon[32610]: pgmap v1136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:09.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:09.026+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:10:09.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:09.026+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:10:09.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:09.026+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:10:09.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:09.030+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:10:09.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:09.030+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:10:09.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:09.030+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:10:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:09 smithi077 ceph-mon[37153]: pgmap v1137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:10:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:09 smithi047 ceph-mon[36835]: pgmap v1137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:10:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:09 smithi029 ceph-mon[32610]: pgmap v1137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:10:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:10 smithi077 ceph-mon[37153]: pgmap v1138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:10:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:10 smithi047 ceph-mon[36835]: pgmap v1138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:10:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:10 smithi029 ceph-mon[32610]: pgmap v1138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:10:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:11 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1234.379204 secs 2024-07-27T07:10:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:11 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1234.379204 secs 2024-07-27T07:10:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:11 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1234.379204 secs 2024-07-27T07:10:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:12 smithi047 ceph-mon[36835]: pgmap v1139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:10:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:12 smithi029 ceph-mon[32610]: pgmap v1139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:10:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:12 smithi077 ceph-mon[37153]: pgmap v1139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:10:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:15 smithi077 ceph-mon[37153]: pgmap v1140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:10:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:15 smithi047 ceph-mon[36835]: pgmap v1140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:10:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:15 smithi029 ceph-mon[32610]: pgmap v1140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:10:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1239.379335 secs 2024-07-27T07:10:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:17 smithi077 ceph-mon[37153]: pgmap v1141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1239.379335 secs 2024-07-27T07:10:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:17 smithi047 ceph-mon[36835]: pgmap v1141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1239.379335 secs 2024-07-27T07:10:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:17 smithi029 ceph-mon[32610]: pgmap v1141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:19 smithi077 ceph-mon[37153]: pgmap v1142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:10:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:19 smithi047 ceph-mon[36835]: pgmap v1142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:10:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:19 smithi029 ceph-mon[32610]: pgmap v1142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:10:21.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:20 smithi047 ceph-mon[36835]: pgmap v1143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:10:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:20 smithi077 ceph-mon[37153]: pgmap v1143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:10:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:20 smithi029 ceph-mon[32610]: pgmap v1143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:10:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:21 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1244.379458 secs 2024-07-27T07:10:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:21 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1244.379458 secs 2024-07-27T07:10:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:21 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1244.379458 secs 2024-07-27T07:10:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:22 smithi047 ceph-mon[36835]: pgmap v1144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:10:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:22 smithi029 ceph-mon[32610]: pgmap v1144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:10:23.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:22 smithi077 ceph-mon[37153]: pgmap v1144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:10:24.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:24.030+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:10:24.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:24.030+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:10:24.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:24.030+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:10:24.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:24.045+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:10:24.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:24.045+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:10:24.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:24.045+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:10:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:25 smithi077 ceph-mon[37153]: pgmap v1145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:10:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:25 smithi047 ceph-mon[36835]: pgmap v1145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:10:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:25 smithi029 ceph-mon[32610]: pgmap v1145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:10:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1249.379582 secs 2024-07-27T07:10:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:27 smithi077 ceph-mon[37153]: pgmap v1146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1249.379582 secs 2024-07-27T07:10:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:27 smithi047 ceph-mon[36835]: pgmap v1146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1249.379582 secs 2024-07-27T07:10:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:27 smithi029 ceph-mon[32610]: pgmap v1146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:28 smithi077 ceph-mon[37153]: pgmap v1147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:10:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:28 smithi047 ceph-mon[36835]: pgmap v1147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:10:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:28 smithi029 ceph-mon[32610]: pgmap v1147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:10:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:31 smithi077 ceph-mon[37153]: pgmap v1148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:10:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:31 smithi047 ceph-mon[36835]: pgmap v1148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:10:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:31 smithi029 ceph-mon[32610]: pgmap v1148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:10:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1254.379738 secs 2024-07-27T07:10:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1254.379738 secs 2024-07-27T07:10:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1254.379738 secs 2024-07-27T07:10:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:33 smithi077 ceph-mon[37153]: pgmap v1149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:10:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:33 smithi047 ceph-mon[36835]: pgmap v1149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:10:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:33 smithi029 ceph-mon[32610]: pgmap v1149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:10:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:34 smithi077 ceph-mon[37153]: pgmap v1150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:10:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:34 smithi047 ceph-mon[36835]: pgmap v1150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:10:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:34 smithi029 ceph-mon[32610]: pgmap v1150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:10:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1259.379865 secs 2024-07-27T07:10:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:37 smithi077 ceph-mon[37153]: pgmap v1151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:38.119 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1259.379865 secs 2024-07-27T07:10:38.119 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:37 smithi029 ceph-mon[32610]: pgmap v1151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1259.379865 secs 2024-07-27T07:10:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:37 smithi047 ceph-mon[36835]: pgmap v1151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:39.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:39.036+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:10:39.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:39.036+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:10:39.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:39.036+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:10:39.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:38 smithi029 ceph-mon[32610]: pgmap v1152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:10:39.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:39.059+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:10:39.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:39.059+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:10:39.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:39.059+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:10:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:38 smithi077 ceph-mon[37153]: pgmap v1152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:10:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:38 smithi047 ceph-mon[36835]: pgmap v1152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:10:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:41 smithi077 ceph-mon[37153]: pgmap v1153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:10:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:41 smithi047 ceph-mon[36835]: pgmap v1153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:10:42.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:41 smithi029 ceph-mon[32610]: pgmap v1153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:10:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:42.815+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:10:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:42.815+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:10:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:42.815+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:10:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:42.815+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:10:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1264.379952 secs 2024-07-27T07:10:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:42 smithi077 ceph-mon[37153]: pgmap v1154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:10:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1264.379952 secs 2024-07-27T07:10:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:42 smithi047 ceph-mon[36835]: pgmap v1154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:10:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1264.379952 secs 2024-07-27T07:10:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:42 smithi029 ceph-mon[32610]: pgmap v1154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:10:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:44 smithi077 ceph-mon[37153]: pgmap v1155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:10:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:10:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:44 smithi047 ceph-mon[36835]: pgmap v1155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:10:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:10:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:44 smithi029 ceph-mon[32610]: pgmap v1155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:10:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:10:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:10:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:10:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:45 smithi077 ceph-mon[37153]: pgmap v1156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:10:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:45 smithi077 ceph-mon[37153]: pgmap v1157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:10:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:10:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:10:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:10:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:45 smithi047 ceph-mon[36835]: pgmap v1156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:10:46.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:45 smithi047 ceph-mon[36835]: pgmap v1157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:10:46.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:10:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:10:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:10:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:45 smithi029 ceph-mon[32610]: pgmap v1156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:10:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:45 smithi029 ceph-mon[32610]: pgmap v1157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:10:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:10:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:46 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1269.380074 secs 2024-07-27T07:10:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:46 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1269.380074 secs 2024-07-27T07:10:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:46 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1269.380074 secs 2024-07-27T07:10:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:47 smithi047 ceph-mon[36835]: pgmap v1158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:10:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:47 smithi029 ceph-mon[32610]: pgmap v1158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:10:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:47 smithi077 ceph-mon[37153]: pgmap v1158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:10:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:50 smithi077 ceph-mon[37153]: pgmap v1159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:10:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:50 smithi047 ceph-mon[36835]: pgmap v1159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:10:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:50 smithi029 ceph-mon[32610]: pgmap v1159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:10:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:52 smithi077 ceph-mon[37153]: pgmap v1160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:10:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1274.380195 secs 2024-07-27T07:10:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:52 smithi047 ceph-mon[36835]: pgmap v1160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:10:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1274.380195 secs 2024-07-27T07:10:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:52 smithi029 ceph-mon[32610]: pgmap v1160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:10:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1274.380195 secs 2024-07-27T07:10:54.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:54.051+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:10:54.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:54.051+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:10:54.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:54.051+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:10:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:54.070+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:10:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:54.070+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:10:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:10:54.070+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:10:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:54 smithi077 ceph-mon[37153]: pgmap v1161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:10:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:54 smithi047 ceph-mon[36835]: pgmap v1161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:10:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:54 smithi029 ceph-mon[32610]: pgmap v1161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:10:56.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:56 smithi077 ceph-mon[37153]: pgmap v1162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:10:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:56 smithi047 ceph-mon[36835]: pgmap v1162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:10:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:56 smithi029 ceph-mon[32610]: pgmap v1162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:10:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1279.380336 secs 2024-07-27T07:10:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1279.380336 secs 2024-07-27T07:10:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1279.380336 secs 2024-07-27T07:10:57.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:57.827+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:10:57.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:57.827+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:10:57.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:57.827+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:10:57.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:10:57.827+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:10:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:10:58 smithi077 ceph-mon[37153]: pgmap v1163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:10:58 smithi047 ceph-mon[36835]: pgmap v1163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:10:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:10:58 smithi029 ceph-mon[32610]: pgmap v1163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:00 smithi077 ceph-mon[37153]: pgmap v1164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:00 smithi047 ceph-mon[36835]: pgmap v1164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:00 smithi029 ceph-mon[32610]: pgmap v1164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:02 smithi077 ceph-mon[37153]: pgmap v1165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1284.380475 secs 2024-07-27T07:11:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:02 smithi047 ceph-mon[36835]: pgmap v1165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1284.380475 secs 2024-07-27T07:11:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:02 smithi029 ceph-mon[32610]: pgmap v1165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1284.380475 secs 2024-07-27T07:11:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:04 smithi077 ceph-mon[37153]: pgmap v1166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:04 smithi047 ceph-mon[36835]: pgmap v1166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:04 smithi029 ceph-mon[32610]: pgmap v1166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:06 smithi077 ceph-mon[37153]: pgmap v1167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:11:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:06 smithi047 ceph-mon[36835]: pgmap v1167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:11:06.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:06 smithi029 ceph-mon[32610]: pgmap v1167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:11:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1289.380605 secs 2024-07-27T07:11:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1289.380605 secs 2024-07-27T07:11:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1289.380605 secs 2024-07-27T07:11:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:08 smithi077 ceph-mon[37153]: pgmap v1168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:08 smithi047 ceph-mon[36835]: pgmap v1168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:08 smithi029 ceph-mon[32610]: pgmap v1168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:09.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:11:09.054+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:11:09.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:11:09.054+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:11:09.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:11:09.054+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:11:09.085 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:11:09.085+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:11:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:11:09.085+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:11:09.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:11:09.085+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:11:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:10 smithi077 ceph-mon[37153]: pgmap v1169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:10 smithi047 ceph-mon[36835]: pgmap v1169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:10 smithi029 ceph-mon[32610]: pgmap v1169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:12 smithi077 ceph-mon[37153]: pgmap v1170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1294.380731 secs 2024-07-27T07:11:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:12 smithi047 ceph-mon[36835]: pgmap v1170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1294.380731 secs 2024-07-27T07:11:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:12 smithi029 ceph-mon[32610]: pgmap v1170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1294.380731 secs 2024-07-27T07:11:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:14 smithi077 ceph-mon[37153]: pgmap v1171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:14 smithi047 ceph-mon[36835]: pgmap v1171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:14 smithi029 ceph-mon[32610]: pgmap v1171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:16 smithi077 ceph-mon[37153]: pgmap v1172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:11:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:16 smithi047 ceph-mon[36835]: pgmap v1172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:11:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:16 smithi029 ceph-mon[32610]: pgmap v1172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:11:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1299.380863 secs 2024-07-27T07:11:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1299.380863 secs 2024-07-27T07:11:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1299.380863 secs 2024-07-27T07:11:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:18 smithi077 ceph-mon[37153]: pgmap v1173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:18 smithi047 ceph-mon[36835]: pgmap v1173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:18 smithi029 ceph-mon[32610]: pgmap v1173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:20 smithi077 ceph-mon[37153]: pgmap v1174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:20 smithi047 ceph-mon[36835]: pgmap v1174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:20 smithi029 ceph-mon[32610]: pgmap v1174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:22 smithi077 ceph-mon[37153]: pgmap v1175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1304.380983 secs 2024-07-27T07:11:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:22 smithi047 ceph-mon[36835]: pgmap v1175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1304.380983 secs 2024-07-27T07:11:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:22 smithi029 ceph-mon[32610]: pgmap v1175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1304.380983 secs 2024-07-27T07:11:24.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:11:24.069+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:11:24.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:11:24.069+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:11:24.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:11:24.069+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:11:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:11:24.100+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:11:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:11:24.100+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:11:24.101 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:11:24.100+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:11:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:24 smithi047 ceph-mon[36835]: pgmap v1176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:24 smithi029 ceph-mon[32610]: pgmap v1176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:24 smithi077 ceph-mon[37153]: pgmap v1176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:26 smithi047 ceph-mon[36835]: pgmap v1177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:11:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:26 smithi029 ceph-mon[32610]: pgmap v1177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:11:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:26 smithi077 ceph-mon[37153]: pgmap v1177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:11:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1309.381067 secs 2024-07-27T07:11:27.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1309.381067 secs 2024-07-27T07:11:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1309.381067 secs 2024-07-27T07:11:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:28 smithi047 ceph-mon[36835]: pgmap v1178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:28 smithi029 ceph-mon[32610]: pgmap v1178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:28 smithi077 ceph-mon[37153]: pgmap v1178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:30 smithi047 ceph-mon[36835]: pgmap v1179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:30 smithi029 ceph-mon[32610]: pgmap v1179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:30 smithi077 ceph-mon[37153]: pgmap v1179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:32 smithi047 ceph-mon[36835]: pgmap v1180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1314.381187 secs 2024-07-27T07:11:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:32 smithi029 ceph-mon[32610]: pgmap v1180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1314.381187 secs 2024-07-27T07:11:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:32 smithi077 ceph-mon[37153]: pgmap v1180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1314.381187 secs 2024-07-27T07:11:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:34 smithi047 ceph-mon[36835]: pgmap v1181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:34 smithi029 ceph-mon[32610]: pgmap v1181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:34 smithi077 ceph-mon[37153]: pgmap v1181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:36 smithi047 ceph-mon[36835]: pgmap v1182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:11:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:36 smithi029 ceph-mon[32610]: pgmap v1182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:11:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:36 smithi077 ceph-mon[37153]: pgmap v1182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:11:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1319.381307 secs 2024-07-27T07:11:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1319.381307 secs 2024-07-27T07:11:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1319.381307 secs 2024-07-27T07:11:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:38 smithi047 ceph-mon[36835]: pgmap v1183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:38 smithi029 ceph-mon[32610]: pgmap v1183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:38 smithi077 ceph-mon[37153]: pgmap v1183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:39.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:11:39.084+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:11:39.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:11:39.084+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:11:39.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:11:39.084+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:11:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:11:39.114+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:11:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:11:39.114+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:11:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:11:39.114+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:11:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:40 smithi047 ceph-mon[36835]: pgmap v1184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:40 smithi029 ceph-mon[32610]: pgmap v1184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:40 smithi077 ceph-mon[37153]: pgmap v1184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:42 smithi047 ceph-mon[36835]: pgmap v1185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1324.381457 secs 2024-07-27T07:11:42.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:42 smithi029 ceph-mon[32610]: pgmap v1185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1324.381457 secs 2024-07-27T07:11:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:42 smithi077 ceph-mon[37153]: pgmap v1185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1324.381457 secs 2024-07-27T07:11:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:44 smithi047 ceph-mon[36835]: pgmap v1186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:44 smithi029 ceph-mon[32610]: pgmap v1186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:44 smithi077 ceph-mon[37153]: pgmap v1186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:11:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:11:45.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:11:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:11:46.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:46 smithi029 ceph-mon[32610]: pgmap v1187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:11:46.513 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:46 smithi047 ceph-mon[36835]: pgmap v1187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:11:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:46 smithi077 ceph-mon[37153]: pgmap v1187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:11:47.506 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1329.381570 secs 2024-07-27T07:11:47.586 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1329.381570 secs 2024-07-27T07:11:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1329.381570 secs 2024-07-27T07:11:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:48 smithi047 ceph-mon[36835]: pgmap v1188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:48 smithi029 ceph-mon[32610]: pgmap v1188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:48 smithi077 ceph-mon[37153]: pgmap v1188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:11:50.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: pgmap v1189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:50.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:11:50.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:11:50.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:11:50.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:11:50.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:11:50.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:11:50.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:11:50.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:11:50.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: pgmap v1190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:11:50.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: pgmap v1191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:11:50.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: pgmap v1189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:50.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:11:50.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:11:50.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:11:50.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:11:50.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:11:50.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:11:50.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:11:50.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:11:50.465 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: pgmap v1190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:11:50.465 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: pgmap v1191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:11:50.465 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: pgmap v1189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:11:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:11:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:11:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:11:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:11:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:11:50.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:11:50.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:11:50.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:11:50.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: pgmap v1190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:11:50.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: pgmap v1191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:11:50.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:11:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:52 smithi047 ceph-mon[36835]: pgmap v1192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T07:11:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1334.381694 secs 2024-07-27T07:11:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:52 smithi029 ceph-mon[32610]: pgmap v1192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T07:11:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1334.381694 secs 2024-07-27T07:11:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:52 smithi077 ceph-mon[37153]: pgmap v1192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T07:11:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1334.381694 secs 2024-07-27T07:11:54.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:11:54.093+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:11:54.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:11:54.093+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:11:54.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:11:54.093+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:11:54.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:11:54.128+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:11:54.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:11:54.128+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:11:54.129 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:11:54.128+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:11:54.756 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:54 smithi029 ceph-mon[32610]: pgmap v1193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T07:11:54.908 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:54 smithi077 ceph-mon[37153]: pgmap v1193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T07:11:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:54 smithi047 ceph-mon[36835]: pgmap v1193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T07:11:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:56 smithi047 ceph-mon[36835]: pgmap v1194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:11:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:56 smithi029 ceph-mon[32610]: pgmap v1194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:11:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:56 smithi077 ceph-mon[37153]: pgmap v1194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:11:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1339.381827 secs 2024-07-27T07:11:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1339.381827 secs 2024-07-27T07:11:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1339.381827 secs 2024-07-27T07:11:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:11:58 smithi047 ceph-mon[36835]: pgmap v1195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:11:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:11:58 smithi029 ceph-mon[32610]: pgmap v1195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:11:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:11:58 smithi077 ceph-mon[37153]: pgmap v1195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:12:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:00 smithi047 ceph-mon[36835]: pgmap v1196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:12:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:00 smithi029 ceph-mon[32610]: pgmap v1196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:12:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:00 smithi077 ceph-mon[37153]: pgmap v1196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:12:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:02 smithi047 ceph-mon[36835]: pgmap v1197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1344.381950 secs 2024-07-27T07:12:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:02 smithi029 ceph-mon[32610]: pgmap v1197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1344.381950 secs 2024-07-27T07:12:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:02 smithi077 ceph-mon[37153]: pgmap v1197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1344.381950 secs 2024-07-27T07:12:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:04 smithi047 ceph-mon[36835]: pgmap v1198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:12:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:04 smithi029 ceph-mon[32610]: pgmap v1198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:12:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:04 smithi077 ceph-mon[37153]: pgmap v1198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:12:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:06 smithi047 ceph-mon[36835]: pgmap v1199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:12:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:06 smithi029 ceph-mon[32610]: pgmap v1199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:12:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:06 smithi077 ceph-mon[37153]: pgmap v1199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:12:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1349.382019 secs 2024-07-27T07:12:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1349.382019 secs 2024-07-27T07:12:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1349.382019 secs 2024-07-27T07:12:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:08 smithi047 ceph-mon[36835]: pgmap v1200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:12:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:08 smithi029 ceph-mon[32610]: pgmap v1200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:12:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:12:09.105+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:12:09.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:12:09.105+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:12:09.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:12:09.105+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:12:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:08 smithi077 ceph-mon[37153]: pgmap v1200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:12:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:12:09.142+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:12:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:12:09.142+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:12:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:12:09.142+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:12:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:10 smithi047 ceph-mon[36835]: pgmap v1201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:10 smithi029 ceph-mon[32610]: pgmap v1201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:10 smithi077 ceph-mon[37153]: pgmap v1201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:12 smithi047 ceph-mon[36835]: pgmap v1202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1354.382085 secs 2024-07-27T07:12:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:12 smithi029 ceph-mon[32610]: pgmap v1202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1354.382085 secs 2024-07-27T07:12:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:12 smithi077 ceph-mon[37153]: pgmap v1202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1354.382085 secs 2024-07-27T07:12:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:14 smithi047 ceph-mon[36835]: pgmap v1203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:12:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:14 smithi029 ceph-mon[32610]: pgmap v1203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:12:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:14 smithi077 ceph-mon[37153]: pgmap v1203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:12:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:16 smithi047 ceph-mon[36835]: pgmap v1204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:12:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:16 smithi029 ceph-mon[32610]: pgmap v1204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:12:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:16 smithi077 ceph-mon[37153]: pgmap v1204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:12:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1359.382171 secs 2024-07-27T07:12:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1359.382171 secs 2024-07-27T07:12:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1359.382171 secs 2024-07-27T07:12:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:18 smithi047 ceph-mon[36835]: pgmap v1205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:12:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:18 smithi029 ceph-mon[32610]: pgmap v1205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:12:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:18 smithi077 ceph-mon[37153]: pgmap v1205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:12:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:20 smithi047 ceph-mon[36835]: pgmap v1206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:20 smithi029 ceph-mon[32610]: pgmap v1206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:20 smithi077 ceph-mon[37153]: pgmap v1206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:22 smithi029 ceph-mon[32610]: pgmap v1207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1364.382295 secs 2024-07-27T07:12:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:22 smithi077 ceph-mon[37153]: pgmap v1207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1364.382295 secs 2024-07-27T07:12:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:22 smithi047 ceph-mon[36835]: pgmap v1207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1364.382295 secs 2024-07-27T07:12:24.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:12:24.110+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:12:24.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:12:24.110+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:12:24.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:12:24.110+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:12:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:12:24.143+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:12:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:12:24.143+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:12:24.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:12:24.143+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:12:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:24 smithi029 ceph-mon[32610]: pgmap v1208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:12:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:24 smithi077 ceph-mon[37153]: pgmap v1208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:12:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:24 smithi047 ceph-mon[36835]: pgmap v1208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:12:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:26 smithi029 ceph-mon[32610]: pgmap v1209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:12:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:26 smithi077 ceph-mon[37153]: pgmap v1209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:12:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:26 smithi047 ceph-mon[36835]: pgmap v1209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:12:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1369.382414 secs 2024-07-27T07:12:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1369.382414 secs 2024-07-27T07:12:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1369.382414 secs 2024-07-27T07:12:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:28 smithi029 ceph-mon[32610]: pgmap v1210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:12:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:28 smithi077 ceph-mon[37153]: pgmap v1210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:12:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:28 smithi047 ceph-mon[36835]: pgmap v1210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:12:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:30 smithi077 ceph-mon[37153]: pgmap v1211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:30 smithi047 ceph-mon[36835]: pgmap v1211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:30 smithi029 ceph-mon[32610]: pgmap v1211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:32 smithi077 ceph-mon[37153]: pgmap v1212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1374.382536 secs 2024-07-27T07:12:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:32 smithi047 ceph-mon[36835]: pgmap v1212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1374.382536 secs 2024-07-27T07:12:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:32 smithi029 ceph-mon[32610]: pgmap v1212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1374.382536 secs 2024-07-27T07:12:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:34 smithi077 ceph-mon[37153]: pgmap v1213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:12:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:34 smithi047 ceph-mon[36835]: pgmap v1213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:12:35.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:34 smithi029 ceph-mon[32610]: pgmap v1213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:12:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:36 smithi077 ceph-mon[37153]: pgmap v1214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:12:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:36 smithi047 ceph-mon[36835]: pgmap v1214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:12:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:36 smithi029 ceph-mon[32610]: pgmap v1214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:12:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1379.382656 secs 2024-07-27T07:12:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1379.382656 secs 2024-07-27T07:12:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1379.382656 secs 2024-07-27T07:12:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:38 smithi077 ceph-mon[37153]: pgmap v1215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:12:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:12:39.116+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:12:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:12:39.116+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:12:39.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:12:39.116+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:12:39.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:38 smithi029 ceph-mon[32610]: pgmap v1215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:12:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:12:39.157+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:12:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:12:39.157+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:12:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:12:39.157+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:12:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:38 smithi047 ceph-mon[36835]: pgmap v1215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:12:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:40 smithi077 ceph-mon[37153]: pgmap v1216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:40 smithi047 ceph-mon[36835]: pgmap v1216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:40 smithi029 ceph-mon[32610]: pgmap v1216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:43.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:42 smithi029 ceph-mon[32610]: pgmap v1217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:43.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1384.382817 secs 2024-07-27T07:12:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:42 smithi077 ceph-mon[37153]: pgmap v1217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1384.382817 secs 2024-07-27T07:12:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:42 smithi047 ceph-mon[36835]: pgmap v1217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1384.382817 secs 2024-07-27T07:12:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:44 smithi077 ceph-mon[37153]: pgmap v1218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:12:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:44 smithi047 ceph-mon[36835]: pgmap v1218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:12:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:44 smithi029 ceph-mon[32610]: pgmap v1218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:12:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:46 smithi077 ceph-mon[37153]: pgmap v1219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:12:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:46 smithi047 ceph-mon[36835]: pgmap v1219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:12:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:46 smithi029 ceph-mon[32610]: pgmap v1219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:12:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1389.382910 secs 2024-07-27T07:12:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1389.382910 secs 2024-07-27T07:12:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1389.382910 secs 2024-07-27T07:12:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:48 smithi077 ceph-mon[37153]: pgmap v1220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:12:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:48 smithi047 ceph-mon[36835]: pgmap v1220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:12:49.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:48 smithi029 ceph-mon[32610]: pgmap v1220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:12:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:12:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:12:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:12:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:50 smithi077 ceph-mon[37153]: pgmap v1221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:50 smithi047 ceph-mon[36835]: pgmap v1221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:51.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:51.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:51.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:50 smithi029 ceph-mon[32610]: pgmap v1221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:12:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:51 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:12:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:51 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:12:52.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:51 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:12:52.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:51 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:12:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:51 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:12:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:51 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:12:52.897 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:52 smithi077 ceph-mon[37153]: pgmap v1222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:52.897 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1394.382984 secs 2024-07-27T07:12:53.113 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:52 smithi047 ceph-mon[36835]: pgmap v1222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:53.113 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1394.382984 secs 2024-07-27T07:12:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:52 smithi029 ceph-mon[32610]: pgmap v1222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:12:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1394.382984 secs 2024-07-27T07:12:54.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:12:54.117+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:12:54.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:12:54.117+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:12:54.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:12:54.117+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:12:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:12:54.172+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:12:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:12:54.172+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:12:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:12:54.172+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:12:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:12:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:12:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:12:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:12:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:12:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:12:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: pgmap v1223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:12:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: pgmap v1224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:12:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:12:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:12:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:12:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:12:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:12:54.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:12:54.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:12:54.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: pgmap v1223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:12:54.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: pgmap v1224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:12:54.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:12:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:12:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:12:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:12:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:12:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:12:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:12:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:12:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: pgmap v1223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:12:54.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: pgmap v1224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:12:54.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:12:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:56 smithi077 ceph-mon[37153]: pgmap v1225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:12:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:56 smithi047 ceph-mon[36835]: pgmap v1225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:12:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:56 smithi029 ceph-mon[32610]: pgmap v1225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:12:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1399.383109 secs 2024-07-27T07:12:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1399.383109 secs 2024-07-27T07:12:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1399.383109 secs 2024-07-27T07:12:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:12:58 smithi077 ceph-mon[37153]: pgmap v1226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s 2024-07-27T07:12:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:12:58 smithi047 ceph-mon[36835]: pgmap v1226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s 2024-07-27T07:12:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:12:58 smithi029 ceph-mon[32610]: pgmap v1226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s 2024-07-27T07:13:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:00 smithi077 ceph-mon[37153]: pgmap v1227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:13:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:00 smithi047 ceph-mon[36835]: pgmap v1227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:13:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:00 smithi029 ceph-mon[32610]: pgmap v1227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:13:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:02 smithi077 ceph-mon[37153]: pgmap v1228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:13:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1404.383234 secs 2024-07-27T07:13:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:02 smithi047 ceph-mon[36835]: pgmap v1228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:13:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1404.383234 secs 2024-07-27T07:13:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:02 smithi029 ceph-mon[32610]: pgmap v1228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:13:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1404.383234 secs 2024-07-27T07:13:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:04 smithi077 ceph-mon[37153]: pgmap v1229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:13:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:04 smithi047 ceph-mon[36835]: pgmap v1229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:13:04.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:04 smithi029 ceph-mon[32610]: pgmap v1229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:13:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:06 smithi077 ceph-mon[37153]: pgmap v1230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:13:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:06 smithi047 ceph-mon[36835]: pgmap v1230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:13:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:06 smithi029 ceph-mon[32610]: pgmap v1230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:13:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1409.383317 secs 2024-07-27T07:13:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1409.383317 secs 2024-07-27T07:13:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1409.383317 secs 2024-07-27T07:13:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:08 smithi077 ceph-mon[37153]: pgmap v1231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:08 smithi047 ceph-mon[36835]: pgmap v1231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:08 smithi029 ceph-mon[32610]: pgmap v1231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:09.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:09.133+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:13:09.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:09.133+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:13:09.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:09.133+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:13:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:09.187+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:13:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:09.187+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:13:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:09.187+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:13:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:10 smithi077 ceph-mon[37153]: pgmap v1232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:10 smithi047 ceph-mon[36835]: pgmap v1232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:10 smithi029 ceph-mon[32610]: pgmap v1232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:12 smithi077 ceph-mon[37153]: pgmap v1233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1414.383457 secs 2024-07-27T07:13:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:12 smithi047 ceph-mon[36835]: pgmap v1233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1414.383457 secs 2024-07-27T07:13:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:12 smithi029 ceph-mon[32610]: pgmap v1233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1414.383457 secs 2024-07-27T07:13:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:14 smithi077 ceph-mon[37153]: pgmap v1234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:14 smithi047 ceph-mon[36835]: pgmap v1234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:14 smithi029 ceph-mon[32610]: pgmap v1234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:16 smithi077 ceph-mon[37153]: pgmap v1235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:13:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:16 smithi047 ceph-mon[36835]: pgmap v1235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:13:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:16 smithi029 ceph-mon[32610]: pgmap v1235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:13:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1419.383583 secs 2024-07-27T07:13:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1419.383583 secs 2024-07-27T07:13:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1419.383583 secs 2024-07-27T07:13:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:18 smithi077 ceph-mon[37153]: pgmap v1236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:18 smithi047 ceph-mon[36835]: pgmap v1236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:18 smithi029 ceph-mon[32610]: pgmap v1236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:20 smithi077 ceph-mon[37153]: pgmap v1237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:20 smithi047 ceph-mon[36835]: pgmap v1237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:20 smithi029 ceph-mon[32610]: pgmap v1237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:22 smithi077 ceph-mon[37153]: pgmap v1238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1424.383704 secs 2024-07-27T07:13:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:22 smithi047 ceph-mon[36835]: pgmap v1238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1424.383704 secs 2024-07-27T07:13:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:22 smithi029 ceph-mon[32610]: pgmap v1238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1424.383704 secs 2024-07-27T07:13:24.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:24.149+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:13:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:24.149+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:13:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:24.149+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:13:24.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:24.194+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:13:24.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:24.194+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:13:24.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:24.194+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:13:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:24 smithi029 ceph-mon[32610]: pgmap v1239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:24 smithi077 ceph-mon[37153]: pgmap v1239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:24 smithi047 ceph-mon[36835]: pgmap v1239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:26 smithi077 ceph-mon[37153]: pgmap v1240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:13:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:26 smithi047 ceph-mon[36835]: pgmap v1240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:13:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:26 smithi029 ceph-mon[32610]: pgmap v1240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:13:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1429.383826 secs 2024-07-27T07:13:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1429.383826 secs 2024-07-27T07:13:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1429.383826 secs 2024-07-27T07:13:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:28 smithi047 ceph-mon[36835]: pgmap v1241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:28 smithi029 ceph-mon[32610]: pgmap v1241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:28 smithi077 ceph-mon[37153]: pgmap v1241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:30 smithi029 ceph-mon[32610]: pgmap v1242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:30 smithi047 ceph-mon[36835]: pgmap v1242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:30 smithi077 ceph-mon[37153]: pgmap v1242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:32 smithi047 ceph-mon[36835]: pgmap v1243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:32.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1434.383947 secs 2024-07-27T07:13:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:32 smithi029 ceph-mon[32610]: pgmap v1243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1434.383947 secs 2024-07-27T07:13:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:32 smithi077 ceph-mon[37153]: pgmap v1243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1434.383947 secs 2024-07-27T07:13:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:34 smithi047 ceph-mon[36835]: pgmap v1244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:34.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:34 smithi029 ceph-mon[32610]: pgmap v1244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:34 smithi077 ceph-mon[37153]: pgmap v1244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:36 smithi047 ceph-mon[36835]: pgmap v1245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:13:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:36 smithi029 ceph-mon[32610]: pgmap v1245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:13:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:36 smithi077 ceph-mon[37153]: pgmap v1245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:13:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1439.384061 secs 2024-07-27T07:13:37.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1439.384061 secs 2024-07-27T07:13:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1439.384061 secs 2024-07-27T07:13:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:38 smithi047 ceph-mon[36835]: pgmap v1246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:38 smithi029 ceph-mon[32610]: pgmap v1246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:38 smithi077 ceph-mon[37153]: pgmap v1246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:39.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:39.159+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:13:39.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:39.159+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:13:39.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:39.159+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:13:39.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:39.207+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:13:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:39.207+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:13:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:39.207+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:13:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:40 smithi047 ceph-mon[36835]: pgmap v1247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:40 smithi029 ceph-mon[32610]: pgmap v1247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:40 smithi077 ceph-mon[37153]: pgmap v1247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:42 smithi047 ceph-mon[36835]: pgmap v1248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:42 smithi047 ceph-mon[36835]: 1 slow requests, 1 included below; oldest blocked for > 1444.384134 secs 2024-07-27T07:13:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:42 smithi047 ceph-mon[36835]: slow request 1444.384133 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T07:13:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:42 smithi029 ceph-mon[32610]: pgmap v1248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:42 smithi029 ceph-mon[32610]: 1 slow requests, 1 included below; oldest blocked for > 1444.384134 secs 2024-07-27T07:13:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:42 smithi029 ceph-mon[32610]: slow request 1444.384133 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T07:13:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:42.842+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:13:42.844 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:42.842+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:13:42.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:42.843+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:13:42.845 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:42.843+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:13:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:42 smithi077 ceph-mon[37153]: pgmap v1248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:42 smithi077 ceph-mon[37153]: 1 slow requests, 1 included below; oldest blocked for > 1444.384134 secs 2024-07-27T07:13:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:42 smithi077 ceph-mon[37153]: slow request 1444.384133 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T07:13:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:44 smithi047 ceph-mon[36835]: pgmap v1249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:44.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:44 smithi029 ceph-mon[32610]: pgmap v1249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:44 smithi077 ceph-mon[37153]: pgmap v1249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:46 smithi047 ceph-mon[36835]: pgmap v1250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:13:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:46 smithi029 ceph-mon[32610]: pgmap v1250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:13:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:46 smithi077 ceph-mon[37153]: pgmap v1250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:13:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1449.384207 secs 2024-07-27T07:13:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1449.384207 secs 2024-07-27T07:13:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1449.384207 secs 2024-07-27T07:13:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:48 smithi047 ceph-mon[36835]: pgmap v1251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:48 smithi029 ceph-mon[32610]: pgmap v1251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:48 smithi077 ceph-mon[37153]: pgmap v1251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:50 smithi047 ceph-mon[36835]: pgmap v1252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:50 smithi029 ceph-mon[32610]: pgmap v1252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:50 smithi077 ceph-mon[37153]: pgmap v1252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:13:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:52 smithi047 ceph-mon[36835]: pgmap v1253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1454.384307 secs 2024-07-27T07:13:52.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:52 smithi029 ceph-mon[32610]: pgmap v1253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1454.384307 secs 2024-07-27T07:13:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:52 smithi077 ceph-mon[37153]: pgmap v1253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:13:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1454.384307 secs 2024-07-27T07:13:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:13:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:13:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:13:54.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:54.168+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:13:54.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:54.168+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:13:54.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:54.168+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:13:54.209 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:54.208+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:13:54.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:54.208+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:13:54.210 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:13:54.208+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:13:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:54 smithi047 ceph-mon[36835]: pgmap v1254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:13:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:13:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:13:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:54 smithi029 ceph-mon[32610]: pgmap v1254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:13:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:13:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:13:54.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:54 smithi077 ceph-mon[37153]: pgmap v1254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:13:54.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:13:54.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:13:54.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:13:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:55 smithi047 ceph-mon[36835]: pgmap v1255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:13:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:55 smithi047 ceph-mon[36835]: pgmap v1256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:13:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:55 smithi029 ceph-mon[32610]: pgmap v1255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:13:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:55 smithi029 ceph-mon[32610]: pgmap v1256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:13:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:55 smithi077 ceph-mon[37153]: pgmap v1255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:13:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:55 smithi077 ceph-mon[37153]: pgmap v1256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:13:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:57 smithi047 ceph-mon[36835]: pgmap v1257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:13:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1459.384426 secs 2024-07-27T07:13:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:57 smithi029 ceph-mon[32610]: pgmap v1257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:13:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1459.384426 secs 2024-07-27T07:13:57.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:57.853+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:13:57.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:57.853+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:13:57.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:57.854+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:13:57.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:13:57.854+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:13:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:57 smithi077 ceph-mon[37153]: pgmap v1257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:13:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1459.384426 secs 2024-07-27T07:13:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:13:59 smithi047 ceph-mon[36835]: pgmap v1258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:13:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:13:59 smithi029 ceph-mon[32610]: pgmap v1258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:13:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:13:59 smithi077 ceph-mon[37153]: pgmap v1258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:14:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:01 smithi047 ceph-mon[36835]: pgmap v1259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:14:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:01 smithi029 ceph-mon[32610]: pgmap v1259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:14:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:01 smithi077 ceph-mon[37153]: pgmap v1259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:14:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1464.384570 secs 2024-07-27T07:14:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1464.384570 secs 2024-07-27T07:14:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1464.384570 secs 2024-07-27T07:14:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:03 smithi047 ceph-mon[36835]: pgmap v1260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:14:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:03 smithi029 ceph-mon[32610]: pgmap v1260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:14:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:03 smithi077 ceph-mon[37153]: pgmap v1260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:14:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:05 smithi047 ceph-mon[36835]: pgmap v1261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:14:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:05 smithi029 ceph-mon[32610]: pgmap v1261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:14:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:05 smithi077 ceph-mon[37153]: pgmap v1261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:14:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:07 smithi029 ceph-mon[32610]: pgmap v1262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1469.384716 secs 2024-07-27T07:14:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:07 smithi077 ceph-mon[37153]: pgmap v1262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1469.384716 secs 2024-07-27T07:14:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:07 smithi047 ceph-mon[36835]: pgmap v1262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1469.384716 secs 2024-07-27T07:14:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:09.174+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:14:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:09.174+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:14:09.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:09.174+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:14:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:09.220+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:14:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:09.220+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:14:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:09.220+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:14:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:09 smithi029 ceph-mon[32610]: pgmap v1263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:09 smithi077 ceph-mon[37153]: pgmap v1263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:09 smithi047 ceph-mon[36835]: pgmap v1263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:11 smithi029 ceph-mon[32610]: pgmap v1264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:11 smithi077 ceph-mon[37153]: pgmap v1264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:11 smithi047 ceph-mon[36835]: pgmap v1264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1474.384813 secs 2024-07-27T07:14:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1474.384813 secs 2024-07-27T07:14:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1474.384813 secs 2024-07-27T07:14:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:13 smithi077 ceph-mon[37153]: pgmap v1265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:13 smithi047 ceph-mon[36835]: pgmap v1265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:13 smithi029 ceph-mon[32610]: pgmap v1265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:15 smithi077 ceph-mon[37153]: pgmap v1266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:15 smithi047 ceph-mon[36835]: pgmap v1266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:15 smithi029 ceph-mon[32610]: pgmap v1266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:17 smithi077 ceph-mon[37153]: pgmap v1267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1479.384912 secs 2024-07-27T07:14:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:17 smithi047 ceph-mon[36835]: pgmap v1267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1479.384912 secs 2024-07-27T07:14:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:17 smithi029 ceph-mon[32610]: pgmap v1267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1479.384912 secs 2024-07-27T07:14:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:19 smithi077 ceph-mon[37153]: pgmap v1268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:19 smithi047 ceph-mon[36835]: pgmap v1268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:19 smithi029 ceph-mon[32610]: pgmap v1268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:21 smithi077 ceph-mon[37153]: pgmap v1269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:21 smithi047 ceph-mon[36835]: pgmap v1269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:21 smithi029 ceph-mon[32610]: pgmap v1269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:22.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1484.385028 secs 2024-07-27T07:14:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1484.385028 secs 2024-07-27T07:14:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1484.385028 secs 2024-07-27T07:14:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:23 smithi077 ceph-mon[37153]: pgmap v1270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:23 smithi047 ceph-mon[36835]: pgmap v1270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:23 smithi029 ceph-mon[32610]: pgmap v1270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:24.189+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:14:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:24.189+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:14:24.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:24.189+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:14:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:24.236+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:14:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:24.236+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:14:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:24.236+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:14:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:25 smithi077 ceph-mon[37153]: pgmap v1271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:25 smithi047 ceph-mon[36835]: pgmap v1271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:25 smithi029 ceph-mon[32610]: pgmap v1271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:27.851 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:27.849+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:14:27.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:27.849+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:14:27.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:27.850+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:14:27.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:27.850+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:14:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:27 smithi077 ceph-mon[37153]: pgmap v1272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1489.385157 secs 2024-07-27T07:14:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:27 smithi047 ceph-mon[36835]: pgmap v1272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1489.385157 secs 2024-07-27T07:14:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:27 smithi029 ceph-mon[32610]: pgmap v1272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1489.385157 secs 2024-07-27T07:14:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:29 smithi077 ceph-mon[37153]: pgmap v1273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:29 smithi047 ceph-mon[36835]: pgmap v1273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:29 smithi029 ceph-mon[32610]: pgmap v1273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:31 smithi077 ceph-mon[37153]: pgmap v1274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:31 smithi047 ceph-mon[36835]: pgmap v1274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:31 smithi029 ceph-mon[32610]: pgmap v1274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1494.385258 secs 2024-07-27T07:14:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1494.385258 secs 2024-07-27T07:14:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1494.385258 secs 2024-07-27T07:14:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:33 smithi077 ceph-mon[37153]: pgmap v1275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:33 smithi047 ceph-mon[36835]: pgmap v1275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:33 smithi029 ceph-mon[32610]: pgmap v1275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:35 smithi077 ceph-mon[37153]: pgmap v1276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:35 smithi047 ceph-mon[36835]: pgmap v1276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:35 smithi029 ceph-mon[32610]: pgmap v1276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:37 smithi077 ceph-mon[37153]: pgmap v1277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1499.385383 secs 2024-07-27T07:14:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:37 smithi047 ceph-mon[36835]: pgmap v1277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1499.385383 secs 2024-07-27T07:14:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:37 smithi029 ceph-mon[32610]: pgmap v1277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1499.385383 secs 2024-07-27T07:14:39.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:39.194+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:14:39.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:39.194+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:14:39.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:39.194+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:14:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:39.245+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:14:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:39.245+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:14:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:39.245+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:14:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:39 smithi077 ceph-mon[37153]: pgmap v1278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:39 smithi047 ceph-mon[36835]: pgmap v1278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:39 smithi029 ceph-mon[32610]: pgmap v1278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:41 smithi077 ceph-mon[37153]: pgmap v1279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:41.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:41 smithi047 ceph-mon[36835]: pgmap v1279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:41 smithi029 ceph-mon[32610]: pgmap v1279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1504.385452 secs 2024-07-27T07:14:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:42.860+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:14:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:42.860+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:14:42.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1504.385452 secs 2024-07-27T07:14:42.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:42.860+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:14:42.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:42.861+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:14:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1504.385452 secs 2024-07-27T07:14:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:43 smithi077 ceph-mon[37153]: pgmap v1280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:43 smithi047 ceph-mon[36835]: pgmap v1280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:43 smithi029 ceph-mon[32610]: pgmap v1280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:45 smithi077 ceph-mon[37153]: pgmap v1281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:45 smithi047 ceph-mon[36835]: pgmap v1281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:45.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:45 smithi029 ceph-mon[32610]: pgmap v1281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:47 smithi077 ceph-mon[37153]: pgmap v1282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1509.385594 secs 2024-07-27T07:14:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:47 smithi047 ceph-mon[36835]: pgmap v1282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1509.385594 secs 2024-07-27T07:14:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:47 smithi029 ceph-mon[32610]: pgmap v1282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:14:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1509.385594 secs 2024-07-27T07:14:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:49 smithi077 ceph-mon[37153]: pgmap v1283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:49 smithi047 ceph-mon[36835]: pgmap v1283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:49 smithi029 ceph-mon[32610]: pgmap v1283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:14:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:51 smithi077 ceph-mon[37153]: pgmap v1284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:51 smithi047 ceph-mon[36835]: pgmap v1284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:51 smithi029 ceph-mon[32610]: pgmap v1284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:14:52.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1514.385722 secs 2024-07-27T07:14:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1514.385722 secs 2024-07-27T07:14:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1514.385722 secs 2024-07-27T07:14:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:53 smithi077 ceph-mon[37153]: pgmap v1285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:53 smithi047 ceph-mon[36835]: pgmap v1285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:53 smithi029 ceph-mon[32610]: pgmap v1285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:14:54.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:54.205+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:14:54.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:54.205+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:14:54.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:14:54.205+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:14:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:54.260+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:14:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:54.260+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:14:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:14:54.260+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:14:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:14:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:14:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:14:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:14:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:14:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:14:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:14:54.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:14:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:14:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:14:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:14:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:14:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:55 smithi077 ceph-mon[37153]: pgmap v1286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:55 smithi077 ceph-mon[37153]: pgmap v1287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:14:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:55 smithi077 ceph-mon[37153]: pgmap v1288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:14:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:55 smithi047 ceph-mon[36835]: pgmap v1286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:55 smithi047 ceph-mon[36835]: pgmap v1287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:14:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:55 smithi047 ceph-mon[36835]: pgmap v1288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:14:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:55 smithi029 ceph-mon[32610]: pgmap v1286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:14:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:55 smithi029 ceph-mon[32610]: pgmap v1287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:14:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:55 smithi029 ceph-mon[32610]: pgmap v1288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:14:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:57 smithi077 ceph-mon[37153]: pgmap v1289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:14:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1519.385848 secs 2024-07-27T07:14:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:57 smithi047 ceph-mon[36835]: pgmap v1289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:14:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1519.385848 secs 2024-07-27T07:14:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:57 smithi029 ceph-mon[32610]: pgmap v1289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:14:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1519.385848 secs 2024-07-27T07:14:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:14:59 smithi077 ceph-mon[37153]: pgmap v1290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:14:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:14:59 smithi047 ceph-mon[36835]: pgmap v1290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:14:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:14:59 smithi029 ceph-mon[32610]: pgmap v1290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:15:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:01 smithi077 ceph-mon[37153]: pgmap v1291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:15:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:01 smithi047 ceph-mon[36835]: pgmap v1291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:15:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:01 smithi029 ceph-mon[32610]: pgmap v1291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:15:02.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1524.385965 secs 2024-07-27T07:15:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1524.385965 secs 2024-07-27T07:15:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1524.385965 secs 2024-07-27T07:15:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:03 smithi077 ceph-mon[37153]: pgmap v1292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:15:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:03 smithi047 ceph-mon[36835]: pgmap v1292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:15:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:03 smithi029 ceph-mon[32610]: pgmap v1292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:15:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:05 smithi077 ceph-mon[37153]: pgmap v1293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:15:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:05 smithi047 ceph-mon[36835]: pgmap v1293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:15:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:05 smithi029 ceph-mon[32610]: pgmap v1293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:15:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:07 smithi077 ceph-mon[37153]: pgmap v1294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1529.386107 secs 2024-07-27T07:15:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:07 smithi047 ceph-mon[36835]: pgmap v1294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1529.386107 secs 2024-07-27T07:15:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:07 smithi029 ceph-mon[32610]: pgmap v1294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1529.386107 secs 2024-07-27T07:15:09.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:15:09.216+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:15:09.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:15:09.216+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:15:09.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:15:09.216+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:15:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:15:09.274+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:15:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:15:09.274+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:15:09.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:15:09.274+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:15:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:09 smithi077 ceph-mon[37153]: pgmap v1295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:09 smithi047 ceph-mon[36835]: pgmap v1295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:09 smithi029 ceph-mon[32610]: pgmap v1295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:11 smithi077 ceph-mon[37153]: pgmap v1296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:11 smithi047 ceph-mon[36835]: pgmap v1296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:11.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:11 smithi029 ceph-mon[32610]: pgmap v1296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1534.386243 secs 2024-07-27T07:15:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1534.386243 secs 2024-07-27T07:15:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1534.386243 secs 2024-07-27T07:15:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:13 smithi077 ceph-mon[37153]: pgmap v1297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:13 smithi047 ceph-mon[36835]: pgmap v1297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:13.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:13 smithi029 ceph-mon[32610]: pgmap v1297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:15 smithi077 ceph-mon[37153]: pgmap v1298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:15 smithi047 ceph-mon[36835]: pgmap v1298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:15 smithi029 ceph-mon[32610]: pgmap v1298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:17 smithi077 ceph-mon[37153]: pgmap v1299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1539.386368 secs 2024-07-27T07:15:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:17 smithi047 ceph-mon[36835]: pgmap v1299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1539.386368 secs 2024-07-27T07:15:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:17 smithi029 ceph-mon[32610]: pgmap v1299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1539.386368 secs 2024-07-27T07:15:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:19 smithi077 ceph-mon[37153]: pgmap v1300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:19 smithi047 ceph-mon[36835]: pgmap v1300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:19 smithi029 ceph-mon[32610]: pgmap v1300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:21 smithi077 ceph-mon[37153]: pgmap v1301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:21 smithi047 ceph-mon[36835]: pgmap v1301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:21 smithi029 ceph-mon[32610]: pgmap v1301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1544.386491 secs 2024-07-27T07:15:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1544.386491 secs 2024-07-27T07:15:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1544.386491 secs 2024-07-27T07:15:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:23 smithi047 ceph-mon[36835]: pgmap v1302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:23 smithi029 ceph-mon[32610]: pgmap v1302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:23 smithi077 ceph-mon[37153]: pgmap v1302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:15:24.231+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:15:24.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:15:24.231+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:15:24.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:15:24.231+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:15:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:15:24.281+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:15:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:15:24.281+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:15:24.282 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:15:24.281+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:15:25.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:25 smithi077 ceph-mon[37153]: pgmap v1303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:25 smithi047 ceph-mon[36835]: pgmap v1303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:25 smithi029 ceph-mon[32610]: pgmap v1303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:27 smithi047 ceph-mon[36835]: pgmap v1304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1549.386615 secs 2024-07-27T07:15:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:27 smithi029 ceph-mon[32610]: pgmap v1304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1549.386615 secs 2024-07-27T07:15:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:27 smithi077 ceph-mon[37153]: pgmap v1304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1549.386615 secs 2024-07-27T07:15:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:29 smithi047 ceph-mon[36835]: pgmap v1305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:29 smithi029 ceph-mon[32610]: pgmap v1305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:29 smithi077 ceph-mon[37153]: pgmap v1305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:31 smithi047 ceph-mon[36835]: pgmap v1306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:31 smithi029 ceph-mon[32610]: pgmap v1306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:31 smithi077 ceph-mon[37153]: pgmap v1306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1554.386740 secs 2024-07-27T07:15:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1554.386740 secs 2024-07-27T07:15:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1554.386740 secs 2024-07-27T07:15:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:33 smithi047 ceph-mon[36835]: pgmap v1307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:33 smithi029 ceph-mon[32610]: pgmap v1307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:33 smithi077 ceph-mon[37153]: pgmap v1307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:35 smithi047 ceph-mon[36835]: pgmap v1308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:35.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:35 smithi029 ceph-mon[32610]: pgmap v1308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:35 smithi077 ceph-mon[37153]: pgmap v1308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:37 smithi047 ceph-mon[36835]: pgmap v1309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1559.386865 secs 2024-07-27T07:15:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:37 smithi029 ceph-mon[32610]: pgmap v1309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1559.386865 secs 2024-07-27T07:15:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:37 smithi077 ceph-mon[37153]: pgmap v1309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1559.386865 secs 2024-07-27T07:15:39.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:15:39.246+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:15:39.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:15:39.246+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:15:39.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:15:39.246+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:15:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:15:39.295+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:15:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:15:39.295+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:15:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:15:39.295+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:15:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:39 smithi047 ceph-mon[36835]: pgmap v1310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:39.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:39 smithi029 ceph-mon[32610]: pgmap v1310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:39 smithi077 ceph-mon[37153]: pgmap v1310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:41 smithi047 ceph-mon[36835]: pgmap v1311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:41 smithi029 ceph-mon[32610]: pgmap v1311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:41 smithi077 ceph-mon[37153]: pgmap v1311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1564.386931 secs 2024-07-27T07:15:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1564.386931 secs 2024-07-27T07:15:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1564.386931 secs 2024-07-27T07:15:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:43 smithi047 ceph-mon[36835]: pgmap v1312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:43 smithi029 ceph-mon[32610]: pgmap v1312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:43 smithi077 ceph-mon[37153]: pgmap v1312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:45 smithi047 ceph-mon[36835]: pgmap v1313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:45.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:45 smithi029 ceph-mon[32610]: pgmap v1313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:45 smithi077 ceph-mon[37153]: pgmap v1313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:47 smithi047 ceph-mon[36835]: pgmap v1314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1569.387023 secs 2024-07-27T07:15:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:47 smithi029 ceph-mon[32610]: pgmap v1314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1569.387023 secs 2024-07-27T07:15:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:47 smithi077 ceph-mon[37153]: pgmap v1314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1569.387023 secs 2024-07-27T07:15:49.928 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:49 smithi029 ceph-mon[32610]: pgmap v1315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:49 smithi047 ceph-mon[36835]: pgmap v1315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:49 smithi077 ceph-mon[37153]: pgmap v1315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:15:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:51 smithi047 ceph-mon[36835]: pgmap v1316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:51 smithi029 ceph-mon[32610]: pgmap v1316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:51 smithi077 ceph-mon[37153]: pgmap v1316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:15:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1574.387122 secs 2024-07-27T07:15:52.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1574.387122 secs 2024-07-27T07:15:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1574.387122 secs 2024-07-27T07:15:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:53 smithi047 ceph-mon[36835]: pgmap v1317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:53 smithi029 ceph-mon[32610]: pgmap v1317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:53 smithi077 ceph-mon[37153]: pgmap v1317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:15:54.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:15:54.261+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:15:54.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:15:54.261+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:15:54.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:15:54.261+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:15:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:15:54.310+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:15:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:15:54.310+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:15:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:15:54.310+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:15:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:15:54.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:15:55.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:15:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:55 smithi077 ceph-mon[37153]: pgmap v1318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:55 smithi047 ceph-mon[36835]: pgmap v1318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:56.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:55 smithi029 ceph-mon[32610]: pgmap v1318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:15:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:57 smithi047 ceph-mon[36835]: pgmap v1319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1579.387243 secs 2024-07-27T07:15:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:15:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:15:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:57 smithi029 ceph-mon[32610]: pgmap v1319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1579.387243 secs 2024-07-27T07:15:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:15:57.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:15:57.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:57 smithi077 ceph-mon[37153]: pgmap v1319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1579.387243 secs 2024-07-27T07:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:15:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:58 smithi029 ceph-mon[32610]: pgmap v1320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:15:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:15:58 smithi029 ceph-mon[32610]: pgmap v1321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:15:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:58 smithi077 ceph-mon[37153]: pgmap v1320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:15:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:15:58 smithi077 ceph-mon[37153]: pgmap v1321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:15:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:58 smithi047 ceph-mon[36835]: pgmap v1320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:15:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:15:58 smithi047 ceph-mon[36835]: pgmap v1321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:16:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:00 smithi029 ceph-mon[32610]: pgmap v1322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:16:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:00 smithi077 ceph-mon[37153]: pgmap v1322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:16:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:00 smithi047 ceph-mon[36835]: pgmap v1322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:16:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:02 smithi029 ceph-mon[32610]: pgmap v1323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:16:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1584.387366 secs 2024-07-27T07:16:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:02 smithi077 ceph-mon[37153]: pgmap v1323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:16:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1584.387366 secs 2024-07-27T07:16:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:02 smithi047 ceph-mon[36835]: pgmap v1323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:16:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1584.387366 secs 2024-07-27T07:16:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:04 smithi077 ceph-mon[37153]: pgmap v1324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:16:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:04 smithi047 ceph-mon[36835]: pgmap v1324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:16:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:04 smithi029 ceph-mon[32610]: pgmap v1324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:16:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:06 smithi077 ceph-mon[37153]: pgmap v1325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:16:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:06 smithi047 ceph-mon[36835]: pgmap v1325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:16:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:06 smithi029 ceph-mon[32610]: pgmap v1325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:16:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1589.387502 secs 2024-07-27T07:16:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1589.387502 secs 2024-07-27T07:16:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1589.387502 secs 2024-07-27T07:16:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:08 smithi077 ceph-mon[37153]: pgmap v1326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:16:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:08 smithi047 ceph-mon[36835]: pgmap v1326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:16:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:08 smithi029 ceph-mon[32610]: pgmap v1326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:16:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:09.263+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:16:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:09.263+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:16:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:09.263+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:16:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:09.318+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:16:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:09.318+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:16:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:09.318+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:16:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:10 smithi077 ceph-mon[37153]: pgmap v1327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:16:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:10 smithi047 ceph-mon[36835]: pgmap v1327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:16:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:10 smithi029 ceph-mon[32610]: pgmap v1327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:16:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:12 smithi077 ceph-mon[37153]: pgmap v1328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:16:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1594.387647 secs 2024-07-27T07:16:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:12 smithi047 ceph-mon[36835]: pgmap v1328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:16:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1594.387647 secs 2024-07-27T07:16:12.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:12 smithi029 ceph-mon[32610]: pgmap v1328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:16:12.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1594.387647 secs 2024-07-27T07:16:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:14 smithi077 ceph-mon[37153]: pgmap v1329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:16:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:14 smithi047 ceph-mon[36835]: pgmap v1329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:16:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:14 smithi029 ceph-mon[32610]: pgmap v1329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:16:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:16 smithi077 ceph-mon[37153]: pgmap v1330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:16 smithi047 ceph-mon[36835]: pgmap v1330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:16 smithi029 ceph-mon[32610]: pgmap v1330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1599.387775 secs 2024-07-27T07:16:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1599.387775 secs 2024-07-27T07:16:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1599.387775 secs 2024-07-27T07:16:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:18 smithi077 ceph-mon[37153]: pgmap v1331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:18 smithi047 ceph-mon[36835]: pgmap v1331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:18 smithi029 ceph-mon[32610]: pgmap v1331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:20 smithi077 ceph-mon[37153]: pgmap v1332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:16:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:20 smithi047 ceph-mon[36835]: pgmap v1332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:16:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:20 smithi029 ceph-mon[32610]: pgmap v1332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:16:22.502 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:22 smithi047 ceph-mon[36835]: pgmap v1333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:16:22.502 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1604.387918 secs 2024-07-27T07:16:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:22 smithi077 ceph-mon[37153]: pgmap v1333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:16:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1604.387918 secs 2024-07-27T07:16:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:22 smithi029 ceph-mon[32610]: pgmap v1333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:16:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1604.387918 secs 2024-07-27T07:16:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:24.268+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:16:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:24.268+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:16:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:24.268+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:16:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:24.333+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:16:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:24.333+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:16:24.337 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:24.333+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:16:24.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:24 smithi029 ceph-mon[32610]: pgmap v1334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:16:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:24 smithi077 ceph-mon[37153]: pgmap v1334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:16:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:24 smithi047 ceph-mon[36835]: pgmap v1334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:16:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:26 smithi077 ceph-mon[37153]: pgmap v1335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:16:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:26 smithi047 ceph-mon[36835]: pgmap v1335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:16:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:26 smithi029 ceph-mon[32610]: pgmap v1335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:16:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1609.388041 secs 2024-07-27T07:16:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1609.388041 secs 2024-07-27T07:16:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1609.388041 secs 2024-07-27T07:16:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:28 smithi077 ceph-mon[37153]: pgmap v1336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:28 smithi047 ceph-mon[36835]: pgmap v1336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:28 smithi029 ceph-mon[32610]: pgmap v1336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:30 smithi077 ceph-mon[37153]: pgmap v1337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:16:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:30 smithi047 ceph-mon[36835]: pgmap v1337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:16:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:30 smithi029 ceph-mon[32610]: pgmap v1337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:16:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:32 smithi077 ceph-mon[37153]: pgmap v1338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1614.388112 secs 2024-07-27T07:16:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:32 smithi047 ceph-mon[36835]: pgmap v1338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1614.388112 secs 2024-07-27T07:16:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:32 smithi029 ceph-mon[32610]: pgmap v1338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1614.388112 secs 2024-07-27T07:16:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:34 smithi077 ceph-mon[37153]: pgmap v1339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:16:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:34 smithi047 ceph-mon[36835]: pgmap v1339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:16:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:34 smithi029 ceph-mon[32610]: pgmap v1339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:16:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:36 smithi077 ceph-mon[37153]: pgmap v1340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:36 smithi047 ceph-mon[36835]: pgmap v1340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:36 smithi029 ceph-mon[32610]: pgmap v1340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1619.388238 secs 2024-07-27T07:16:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1619.388238 secs 2024-07-27T07:16:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1619.388238 secs 2024-07-27T07:16:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:38 smithi077 ceph-mon[37153]: pgmap v1341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:38 smithi047 ceph-mon[36835]: pgmap v1341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:38 smithi029 ceph-mon[32610]: pgmap v1341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:39.276+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:16:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:39.276+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:16:39.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:39.276+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:16:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:39.347+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:16:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:39.347+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:16:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:39.347+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:16:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:40 smithi077 ceph-mon[37153]: pgmap v1342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:16:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:40 smithi047 ceph-mon[36835]: pgmap v1342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:16:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:40 smithi029 ceph-mon[32610]: pgmap v1342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:16:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:42 smithi077 ceph-mon[37153]: pgmap v1343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1624.388364 secs 2024-07-27T07:16:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:42 smithi047 ceph-mon[36835]: pgmap v1343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1624.388364 secs 2024-07-27T07:16:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:42 smithi029 ceph-mon[32610]: pgmap v1343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1624.388364 secs 2024-07-27T07:16:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:42.871+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:16:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:42.871+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:16:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:42.871+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:16:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:42.871+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:16:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:44 smithi077 ceph-mon[37153]: pgmap v1344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:16:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:44 smithi047 ceph-mon[36835]: pgmap v1344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:16:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:44 smithi029 ceph-mon[32610]: pgmap v1344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:16:46.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:46 smithi077 ceph-mon[37153]: pgmap v1345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:46 smithi047 ceph-mon[36835]: pgmap v1345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:46 smithi029 ceph-mon[32610]: pgmap v1345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1629.388495 secs 2024-07-27T07:16:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1629.388495 secs 2024-07-27T07:16:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1629.388495 secs 2024-07-27T07:16:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:48 smithi077 ceph-mon[37153]: pgmap v1346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:48 smithi047 ceph-mon[36835]: pgmap v1346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:48 smithi029 ceph-mon[32610]: pgmap v1346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:50 smithi077 ceph-mon[37153]: pgmap v1347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:16:50.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:50 smithi047 ceph-mon[36835]: pgmap v1347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:16:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:50 smithi029 ceph-mon[32610]: pgmap v1347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:16:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:52 smithi077 ceph-mon[37153]: pgmap v1348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:52.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1634.388615 secs 2024-07-27T07:16:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:52 smithi047 ceph-mon[36835]: pgmap v1348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1634.388615 secs 2024-07-27T07:16:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:52 smithi029 ceph-mon[32610]: pgmap v1348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1634.388615 secs 2024-07-27T07:16:54.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:54.292+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:16:54.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:54.292+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:16:54.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:54.292+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:16:54.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:54.349+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:16:54.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:54.349+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:16:54.351 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:16:54.349+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:16:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:54 smithi077 ceph-mon[37153]: pgmap v1349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:16:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:54 smithi047 ceph-mon[36835]: pgmap v1349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:16:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:54 smithi029 ceph-mon[32610]: pgmap v1349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:16:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:56 smithi077 ceph-mon[37153]: pgmap v1350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:56 smithi047 ceph-mon[36835]: pgmap v1350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:56 smithi029 ceph-mon[32610]: pgmap v1350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:16:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1639.388749 secs 2024-07-27T07:16:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:16:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1639.388749 secs 2024-07-27T07:16:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:16:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1639.388749 secs 2024-07-27T07:16:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:16:57.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:57.880+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:16:57.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:57.880+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:16:57.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:57.880+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:16:57.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:16:57.880+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:16:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:58 smithi077 ceph-mon[37153]: pgmap v1351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:16:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:16:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:58 smithi047 ceph-mon[36835]: pgmap v1351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:16:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:16:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:58 smithi029 ceph-mon[32610]: pgmap v1351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:16:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:16:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:16:58.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:16:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:59 smithi077 ceph-mon[37153]: pgmap v1352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T07:16:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:16:59 smithi077 ceph-mon[37153]: pgmap v1353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:16:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:59 smithi047 ceph-mon[36835]: pgmap v1352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T07:16:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:16:59 smithi047 ceph-mon[36835]: pgmap v1353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:16:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:59 smithi029 ceph-mon[32610]: pgmap v1352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T07:16:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:16:59 smithi029 ceph-mon[32610]: pgmap v1353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:17:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:01 smithi077 ceph-mon[37153]: pgmap v1354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:17:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:01 smithi047 ceph-mon[36835]: pgmap v1354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:17:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:01 smithi029 ceph-mon[32610]: pgmap v1354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:17:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1644.388879 secs 2024-07-27T07:17:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1644.388879 secs 2024-07-27T07:17:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1644.388879 secs 2024-07-27T07:17:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:03 smithi077 ceph-mon[37153]: pgmap v1355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T07:17:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:03 smithi047 ceph-mon[36835]: pgmap v1355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T07:17:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:03 smithi029 ceph-mon[32610]: pgmap v1355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T07:17:05.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:05 smithi077 ceph-mon[37153]: pgmap v1356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:17:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:05 smithi047 ceph-mon[36835]: pgmap v1356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:17:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:05 smithi029 ceph-mon[32610]: pgmap v1356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:17:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:07 smithi077 ceph-mon[37153]: pgmap v1357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:17:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1649.388976 secs 2024-07-27T07:17:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:07 smithi047 ceph-mon[36835]: pgmap v1357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:17:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1649.388976 secs 2024-07-27T07:17:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:07 smithi029 ceph-mon[32610]: pgmap v1357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:17:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1649.388976 secs 2024-07-27T07:17:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:17:09.298+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:17:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:17:09.298+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:17:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:17:09.298+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:17:09.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:17:09.365+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:17:09.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:17:09.365+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:17:09.366 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:17:09.365+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:17:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:09 smithi029 ceph-mon[32610]: pgmap v1358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:17:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:09 smithi077 ceph-mon[37153]: pgmap v1358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:17:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:09 smithi047 ceph-mon[36835]: pgmap v1358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:17:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:11 smithi077 ceph-mon[37153]: pgmap v1359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:11 smithi047 ceph-mon[36835]: pgmap v1359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:11 smithi029 ceph-mon[32610]: pgmap v1359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1654.389116 secs 2024-07-27T07:17:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1654.389116 secs 2024-07-27T07:17:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1654.389116 secs 2024-07-27T07:17:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:13 smithi077 ceph-mon[37153]: pgmap v1360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:13 smithi047 ceph-mon[36835]: pgmap v1360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:13 smithi029 ceph-mon[32610]: pgmap v1360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:15 smithi077 ceph-mon[37153]: pgmap v1361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:17:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:15 smithi047 ceph-mon[36835]: pgmap v1361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:17:15.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:15 smithi029 ceph-mon[32610]: pgmap v1361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:17:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:17 smithi077 ceph-mon[37153]: pgmap v1362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1659.389241 secs 2024-07-27T07:17:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:17 smithi047 ceph-mon[36835]: pgmap v1362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1659.389241 secs 2024-07-27T07:17:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:17 smithi029 ceph-mon[32610]: pgmap v1362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1659.389241 secs 2024-07-27T07:17:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:19 smithi077 ceph-mon[37153]: pgmap v1363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:17:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:19 smithi047 ceph-mon[36835]: pgmap v1363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:17:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:19 smithi029 ceph-mon[32610]: pgmap v1363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:17:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:21 smithi077 ceph-mon[37153]: pgmap v1364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:17:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:21 smithi047 ceph-mon[36835]: pgmap v1364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:17:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:21 smithi029 ceph-mon[32610]: pgmap v1364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:17:22.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1664.389319 secs 2024-07-27T07:17:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1664.389319 secs 2024-07-27T07:17:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1664.389319 secs 2024-07-27T07:17:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:23 smithi077 ceph-mon[37153]: pgmap v1365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:23 smithi047 ceph-mon[36835]: pgmap v1365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:23 smithi029 ceph-mon[32610]: pgmap v1365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:24.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:17:24.313+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:17:24.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:17:24.313+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:17:24.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:17:24.313+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:17:24.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:17:24.381+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:17:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:17:24.381+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:17:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:17:24.381+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:17:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:25 smithi077 ceph-mon[37153]: pgmap v1366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:17:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:25 smithi047 ceph-mon[36835]: pgmap v1366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:17:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:25 smithi029 ceph-mon[32610]: pgmap v1366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:17:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:27 smithi077 ceph-mon[37153]: pgmap v1367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:17:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1669.389483 secs 2024-07-27T07:17:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:27 smithi047 ceph-mon[36835]: pgmap v1367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:17:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1669.389483 secs 2024-07-27T07:17:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:27 smithi029 ceph-mon[32610]: pgmap v1367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:17:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1669.389483 secs 2024-07-27T07:17:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:29 smithi077 ceph-mon[37153]: pgmap v1368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:17:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:29 smithi047 ceph-mon[36835]: pgmap v1368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:17:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:29 smithi029 ceph-mon[32610]: pgmap v1368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:17:31.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:31 smithi077 ceph-mon[37153]: pgmap v1369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:31 smithi047 ceph-mon[36835]: pgmap v1369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:32.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:31 smithi029 ceph-mon[32610]: pgmap v1369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1674.389626 secs 2024-07-27T07:17:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1674.389626 secs 2024-07-27T07:17:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1674.389626 secs 2024-07-27T07:17:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:33 smithi077 ceph-mon[37153]: pgmap v1370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:33 smithi047 ceph-mon[36835]: pgmap v1370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:33 smithi029 ceph-mon[32610]: pgmap v1370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:35 smithi077 ceph-mon[37153]: pgmap v1371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:17:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:35 smithi047 ceph-mon[36835]: pgmap v1371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:17:35.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:35 smithi029 ceph-mon[32610]: pgmap v1371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:17:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:37 smithi077 ceph-mon[37153]: pgmap v1372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1679.389779 secs 2024-07-27T07:17:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:37 smithi047 ceph-mon[36835]: pgmap v1372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1679.389779 secs 2024-07-27T07:17:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:37 smithi029 ceph-mon[32610]: pgmap v1372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1679.389779 secs 2024-07-27T07:17:39.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:17:39.322+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:17:39.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:17:39.322+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:17:39.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:17:39.322+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:17:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:17:39.395+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:17:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:17:39.395+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:17:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:17:39.395+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:17:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:39 smithi047 ceph-mon[36835]: pgmap v1373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:17:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:39 smithi029 ceph-mon[32610]: pgmap v1373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:17:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:39 smithi077 ceph-mon[37153]: pgmap v1373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:17:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:41 smithi047 ceph-mon[36835]: pgmap v1374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:17:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:41 smithi029 ceph-mon[32610]: pgmap v1374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:17:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:41 smithi077 ceph-mon[37153]: pgmap v1374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:17:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1684.389906 secs 2024-07-27T07:17:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1684.389906 secs 2024-07-27T07:17:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1684.389906 secs 2024-07-27T07:17:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:43 smithi047 ceph-mon[36835]: pgmap v1375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:43 smithi029 ceph-mon[32610]: pgmap v1375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:43 smithi077 ceph-mon[37153]: pgmap v1375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:45 smithi047 ceph-mon[36835]: pgmap v1376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:17:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:45 smithi029 ceph-mon[32610]: pgmap v1376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:17:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:45 smithi077 ceph-mon[37153]: pgmap v1376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:17:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:47 smithi047 ceph-mon[36835]: pgmap v1377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1689.390042 secs 2024-07-27T07:17:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:47 smithi029 ceph-mon[32610]: pgmap v1377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1689.390042 secs 2024-07-27T07:17:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:47 smithi077 ceph-mon[37153]: pgmap v1377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1689.390042 secs 2024-07-27T07:17:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:49 smithi047 ceph-mon[36835]: pgmap v1378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:17:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:49 smithi029 ceph-mon[32610]: pgmap v1378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:17:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:49 smithi077 ceph-mon[37153]: pgmap v1378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:17:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:51 smithi047 ceph-mon[36835]: pgmap v1379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:51 smithi029 ceph-mon[32610]: pgmap v1379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:51 smithi077 ceph-mon[37153]: pgmap v1379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1694.390156 secs 2024-07-27T07:17:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1694.390156 secs 2024-07-27T07:17:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1694.390156 secs 2024-07-27T07:17:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:53 smithi047 ceph-mon[36835]: pgmap v1380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:53 smithi029 ceph-mon[32610]: pgmap v1380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:53 smithi077 ceph-mon[37153]: pgmap v1380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:17:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:17:54.327+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:17:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:17:54.327+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:17:54.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:17:54.327+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:17:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:17:54.410+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:17:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:17:54.411+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:17:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:17:54.411+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:17:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:55 smithi047 ceph-mon[36835]: pgmap v1381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:17:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:55 smithi029 ceph-mon[32610]: pgmap v1381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:17:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:55 smithi077 ceph-mon[37153]: pgmap v1381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:17:57.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:57 smithi077 ceph-mon[37153]: pgmap v1382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:57.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1699.390284 secs 2024-07-27T07:17:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:57 smithi047 ceph-mon[36835]: pgmap v1382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1699.390284 secs 2024-07-27T07:17:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:57 smithi029 ceph-mon[32610]: pgmap v1382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:17:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1699.390284 secs 2024-07-27T07:17:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:17:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:17:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:17:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:17:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:17:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:17:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:17:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:17:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:17:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:17:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:17:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:17:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:59 smithi047 ceph-mon[36835]: pgmap v1383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:17:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:59 smithi047 ceph-mon[36835]: pgmap v1384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:17:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:17:59 smithi047 ceph-mon[36835]: pgmap v1385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:17:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:59 smithi029 ceph-mon[32610]: pgmap v1383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:17:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:59 smithi029 ceph-mon[32610]: pgmap v1384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:17:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:17:59 smithi029 ceph-mon[32610]: pgmap v1385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:18:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:59 smithi077 ceph-mon[37153]: pgmap v1383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:18:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:59 smithi077 ceph-mon[37153]: pgmap v1384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:18:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:17:59 smithi077 ceph-mon[37153]: pgmap v1385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:18:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:01 smithi047 ceph-mon[36835]: pgmap v1386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:18:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:01 smithi029 ceph-mon[32610]: pgmap v1386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:18:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:01 smithi077 ceph-mon[37153]: pgmap v1386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:18:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1704.390430 secs 2024-07-27T07:18:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1704.390430 secs 2024-07-27T07:18:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1704.390430 secs 2024-07-27T07:18:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:03 smithi047 ceph-mon[36835]: pgmap v1387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:18:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:03 smithi029 ceph-mon[32610]: pgmap v1387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:18:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:03 smithi077 ceph-mon[37153]: pgmap v1387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:18:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:05 smithi047 ceph-mon[36835]: pgmap v1388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:18:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:05 smithi029 ceph-mon[32610]: pgmap v1388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:18:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:05 smithi077 ceph-mon[37153]: pgmap v1388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:18:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:07 smithi047 ceph-mon[36835]: pgmap v1389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:18:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1709.390561 secs 2024-07-27T07:18:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:07 smithi029 ceph-mon[32610]: pgmap v1389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:18:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1709.390561 secs 2024-07-27T07:18:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:07 smithi077 ceph-mon[37153]: pgmap v1389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:18:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1709.390561 secs 2024-07-27T07:18:09.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:18:09.342+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:18:09.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:18:09.342+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:18:09.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:18:09.342+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:18:09.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:18:09.422+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:18:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:18:09.422+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:18:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:18:09.422+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:18:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:09 smithi047 ceph-mon[36835]: pgmap v1390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:18:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:09 smithi029 ceph-mon[32610]: pgmap v1390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:18:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:09 smithi077 ceph-mon[37153]: pgmap v1390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:18:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:11 smithi047 ceph-mon[36835]: pgmap v1391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:11 smithi029 ceph-mon[32610]: pgmap v1391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:11 smithi077 ceph-mon[37153]: pgmap v1391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1714.390658 secs 2024-07-27T07:18:12.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1714.390658 secs 2024-07-27T07:18:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1714.390658 secs 2024-07-27T07:18:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:13 smithi047 ceph-mon[36835]: pgmap v1392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:13 smithi029 ceph-mon[32610]: pgmap v1392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:13 smithi077 ceph-mon[37153]: pgmap v1392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:15 smithi047 ceph-mon[36835]: pgmap v1393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:15 smithi029 ceph-mon[32610]: pgmap v1393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:15 smithi077 ceph-mon[37153]: pgmap v1393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:17 smithi047 ceph-mon[36835]: pgmap v1394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1719.390807 secs 2024-07-27T07:18:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:17 smithi029 ceph-mon[32610]: pgmap v1394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1719.390807 secs 2024-07-27T07:18:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:17 smithi077 ceph-mon[37153]: pgmap v1394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1719.390807 secs 2024-07-27T07:18:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:19 smithi047 ceph-mon[36835]: pgmap v1395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:18:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:19 smithi029 ceph-mon[32610]: pgmap v1395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:18:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:19 smithi077 ceph-mon[37153]: pgmap v1395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:18:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:21 smithi047 ceph-mon[36835]: pgmap v1396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:21 smithi029 ceph-mon[32610]: pgmap v1396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:21 smithi077 ceph-mon[37153]: pgmap v1396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1724.390922 secs 2024-07-27T07:18:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1724.390922 secs 2024-07-27T07:18:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1724.390922 secs 2024-07-27T07:18:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:23 smithi029 ceph-mon[32610]: pgmap v1397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:23 smithi077 ceph-mon[37153]: pgmap v1397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:23 smithi047 ceph-mon[36835]: pgmap v1397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:24.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:18:24.357+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:18:24.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:18:24.357+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:18:24.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:18:24.357+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:18:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:18:24.437+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:18:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:18:24.437+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:18:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:18:24.437+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:18:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:25 smithi029 ceph-mon[32610]: pgmap v1398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:25 smithi077 ceph-mon[37153]: pgmap v1398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:25 smithi047 ceph-mon[36835]: pgmap v1398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:27 smithi029 ceph-mon[32610]: pgmap v1399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1729.391042 secs 2024-07-27T07:18:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:27 smithi077 ceph-mon[37153]: pgmap v1399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1729.391042 secs 2024-07-27T07:18:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:27 smithi047 ceph-mon[36835]: pgmap v1399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1729.391042 secs 2024-07-27T07:18:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:29 smithi077 ceph-mon[37153]: pgmap v1400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:18:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:29 smithi047 ceph-mon[36835]: pgmap v1400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:18:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:29 smithi029 ceph-mon[32610]: pgmap v1400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:18:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:31 smithi077 ceph-mon[37153]: pgmap v1401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:31 smithi047 ceph-mon[36835]: pgmap v1401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:31 smithi029 ceph-mon[32610]: pgmap v1401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1734.391165 secs 2024-07-27T07:18:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1734.391165 secs 2024-07-27T07:18:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1734.391165 secs 2024-07-27T07:18:34.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:33 smithi077 ceph-mon[37153]: pgmap v1402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:34.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:33 smithi047 ceph-mon[36835]: pgmap v1402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:34.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:33 smithi029 ceph-mon[32610]: pgmap v1402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:35 smithi077 ceph-mon[37153]: pgmap v1403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:35 smithi047 ceph-mon[36835]: pgmap v1403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:35 smithi029 ceph-mon[32610]: pgmap v1403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:37 smithi077 ceph-mon[37153]: pgmap v1404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1739.391287 secs 2024-07-27T07:18:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:37 smithi047 ceph-mon[36835]: pgmap v1404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1739.391287 secs 2024-07-27T07:18:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:37 smithi029 ceph-mon[32610]: pgmap v1404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1739.391287 secs 2024-07-27T07:18:39.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:18:39.372+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:18:39.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:18:39.372+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:18:39.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:18:39.372+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:18:39.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:18:39.452+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:18:39.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:18:39.452+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:18:39.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:18:39.452+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:18:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:39 smithi077 ceph-mon[37153]: pgmap v1405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:18:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:39 smithi047 ceph-mon[36835]: pgmap v1405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:18:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:39 smithi029 ceph-mon[32610]: pgmap v1405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:18:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:41 smithi077 ceph-mon[37153]: pgmap v1406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:41 smithi047 ceph-mon[36835]: pgmap v1406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:41 smithi029 ceph-mon[32610]: pgmap v1406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1744.391410 secs 2024-07-27T07:18:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1744.391410 secs 2024-07-27T07:18:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1744.391410 secs 2024-07-27T07:18:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:43 smithi077 ceph-mon[37153]: pgmap v1407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:43 smithi047 ceph-mon[36835]: pgmap v1407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:43 smithi029 ceph-mon[32610]: pgmap v1407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:45 smithi077 ceph-mon[37153]: pgmap v1408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:45 smithi047 ceph-mon[36835]: pgmap v1408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:45 smithi029 ceph-mon[32610]: pgmap v1408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:47 smithi077 ceph-mon[37153]: pgmap v1409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1749.391532 secs 2024-07-27T07:18:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:47 smithi047 ceph-mon[36835]: pgmap v1409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1749.391532 secs 2024-07-27T07:18:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:47 smithi029 ceph-mon[32610]: pgmap v1409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1749.391532 secs 2024-07-27T07:18:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:49 smithi077 ceph-mon[37153]: pgmap v1410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:18:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:49 smithi047 ceph-mon[36835]: pgmap v1410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:18:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:49 smithi029 ceph-mon[32610]: pgmap v1410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:18:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:51 smithi077 ceph-mon[37153]: pgmap v1411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:51 smithi047 ceph-mon[36835]: pgmap v1411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:51 smithi029 ceph-mon[32610]: pgmap v1411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:18:53.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1754.391608 secs 2024-07-27T07:18:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1754.391608 secs 2024-07-27T07:18:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1754.391608 secs 2024-07-27T07:18:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:53 smithi077 ceph-mon[37153]: pgmap v1412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:54.112 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:53 smithi029 ceph-mon[32610]: pgmap v1412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:53 smithi047 ceph-mon[36835]: pgmap v1412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:18:54.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:18:54.388+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:18:54.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:18:54.388+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:18:54.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:18:54.388+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:18:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:18:54.467+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:18:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:18:54.467+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:18:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:18:54.467+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:18:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:55 smithi077 ceph-mon[37153]: pgmap v1413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:55 smithi047 ceph-mon[36835]: pgmap v1413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:55 smithi029 ceph-mon[32610]: pgmap v1413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:18:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:57 smithi077 ceph-mon[37153]: pgmap v1414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1759.391710 secs 2024-07-27T07:18:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:57 smithi047 ceph-mon[36835]: pgmap v1414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1759.391710 secs 2024-07-27T07:18:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:57 smithi029 ceph-mon[32610]: pgmap v1414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:18:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1759.391710 secs 2024-07-27T07:18:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:18:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:18:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:19:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:59 smithi077 ceph-mon[37153]: pgmap v1415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:19:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:19:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:18:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:19:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:59 smithi047 ceph-mon[36835]: pgmap v1415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:19:00.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:19:00.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:18:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:19:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:59 smithi029 ceph-mon[32610]: pgmap v1415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:19:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:19:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:18:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:19:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:00 smithi077 ceph-mon[37153]: pgmap v1416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:19:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:00 smithi077 ceph-mon[37153]: pgmap v1417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:19:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:00 smithi047 ceph-mon[36835]: pgmap v1416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:19:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:00 smithi047 ceph-mon[36835]: pgmap v1417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:19:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:00 smithi029 ceph-mon[32610]: pgmap v1416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:19:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:00 smithi029 ceph-mon[32610]: pgmap v1417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:19:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:02 smithi077 ceph-mon[37153]: pgmap v1418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:19:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1764.391836 secs 2024-07-27T07:19:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:02 smithi047 ceph-mon[36835]: pgmap v1418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:19:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1764.391836 secs 2024-07-27T07:19:03.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:02 smithi029 ceph-mon[32610]: pgmap v1418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:19:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1764.391836 secs 2024-07-27T07:19:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:04 smithi077 ceph-mon[37153]: pgmap v1419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:19:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:04 smithi047 ceph-mon[36835]: pgmap v1419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:19:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:04 smithi029 ceph-mon[32610]: pgmap v1419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:19:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:06 smithi077 ceph-mon[37153]: pgmap v1420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:19:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:06 smithi047 ceph-mon[36835]: pgmap v1420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:19:07.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:06 smithi029 ceph-mon[32610]: pgmap v1420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:19:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1769.391879 secs 2024-07-27T07:19:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1769.391879 secs 2024-07-27T07:19:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1769.391879 secs 2024-07-27T07:19:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:08 smithi077 ceph-mon[37153]: pgmap v1421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:19:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:08 smithi047 ceph-mon[36835]: pgmap v1421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:19:09.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:08 smithi029 ceph-mon[32610]: pgmap v1421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:19:09.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:09.403+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:19:09.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:09.403+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:19:09.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:09.403+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:19:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:09.473+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:19:09.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:09.473+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:19:09.474 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:09.473+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:19:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:10 smithi077 ceph-mon[37153]: pgmap v1422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:19:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:10 smithi047 ceph-mon[36835]: pgmap v1422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:19:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:10 smithi029 ceph-mon[32610]: pgmap v1422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:19:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:12 smithi077 ceph-mon[37153]: pgmap v1423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1774.392005 secs 2024-07-27T07:19:13.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:12 smithi047 ceph-mon[36835]: pgmap v1423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:13.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1774.392005 secs 2024-07-27T07:19:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:12 smithi029 ceph-mon[32610]: pgmap v1423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1774.392005 secs 2024-07-27T07:19:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:14 smithi077 ceph-mon[37153]: pgmap v1424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:14 smithi047 ceph-mon[36835]: pgmap v1424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:14 smithi029 ceph-mon[32610]: pgmap v1424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:16 smithi077 ceph-mon[37153]: pgmap v1425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:16 smithi047 ceph-mon[36835]: pgmap v1425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:16 smithi029 ceph-mon[32610]: pgmap v1425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1779.392092 secs 2024-07-27T07:19:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1779.392092 secs 2024-07-27T07:19:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1779.392092 secs 2024-07-27T07:19:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:18 smithi077 ceph-mon[37153]: pgmap v1426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:18 smithi047 ceph-mon[36835]: pgmap v1426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:19.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:18 smithi029 ceph-mon[32610]: pgmap v1426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:20 smithi077 ceph-mon[37153]: pgmap v1427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:20 smithi047 ceph-mon[36835]: pgmap v1427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:20 smithi029 ceph-mon[32610]: pgmap v1427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:22 smithi077 ceph-mon[37153]: pgmap v1428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1784.392214 secs 2024-07-27T07:19:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:22 smithi047 ceph-mon[36835]: pgmap v1428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1784.392214 secs 2024-07-27T07:19:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:22 smithi029 ceph-mon[32610]: pgmap v1428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1784.392214 secs 2024-07-27T07:19:24.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:24.417+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:19:24.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:24.417+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:19:24.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:24.417+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:19:24.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:24.488+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:19:24.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:24.488+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:19:24.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:24.488+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:19:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:24 smithi077 ceph-mon[37153]: pgmap v1429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:24 smithi047 ceph-mon[36835]: pgmap v1429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:24 smithi029 ceph-mon[32610]: pgmap v1429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:26 smithi077 ceph-mon[37153]: pgmap v1430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:26 smithi047 ceph-mon[36835]: pgmap v1430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:26 smithi029 ceph-mon[32610]: pgmap v1430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1789.392333 secs 2024-07-27T07:19:28.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1789.392333 secs 2024-07-27T07:19:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1789.392333 secs 2024-07-27T07:19:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:28 smithi077 ceph-mon[37153]: pgmap v1431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:28 smithi047 ceph-mon[36835]: pgmap v1431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:28 smithi029 ceph-mon[32610]: pgmap v1431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:30 smithi077 ceph-mon[37153]: pgmap v1432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:30 smithi047 ceph-mon[36835]: pgmap v1432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:30 smithi029 ceph-mon[32610]: pgmap v1432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:31 smithi047 ceph-mon[36835]: pgmap v1433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:31 smithi029 ceph-mon[32610]: pgmap v1433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:31 smithi077 ceph-mon[37153]: pgmap v1433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1794.392455 secs 2024-07-27T07:19:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1794.392455 secs 2024-07-27T07:19:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1794.392455 secs 2024-07-27T07:19:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:33 smithi047 ceph-mon[36835]: pgmap v1434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:33 smithi029 ceph-mon[32610]: pgmap v1434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:33 smithi077 ceph-mon[37153]: pgmap v1434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:36 smithi077 ceph-mon[37153]: pgmap v1435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:36 smithi047 ceph-mon[36835]: pgmap v1435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:36 smithi029 ceph-mon[32610]: pgmap v1435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1799.392549 secs 2024-07-27T07:19:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1799.392549 secs 2024-07-27T07:19:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1799.392549 secs 2024-07-27T07:19:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:38 smithi077 ceph-mon[37153]: pgmap v1436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:38 smithi047 ceph-mon[36835]: pgmap v1436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:38 smithi029 ceph-mon[32610]: pgmap v1436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:39.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:39.431+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:19:39.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:39.431+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:19:39.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:39.431+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:19:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:39.502+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:19:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:39.502+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:19:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:39.502+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:19:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:40 smithi077 ceph-mon[37153]: pgmap v1437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:40 smithi047 ceph-mon[36835]: pgmap v1437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:40 smithi029 ceph-mon[32610]: pgmap v1437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:42 smithi077 ceph-mon[37153]: pgmap v1438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1804.392671 secs 2024-07-27T07:19:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:42 smithi047 ceph-mon[36835]: pgmap v1438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1804.392671 secs 2024-07-27T07:19:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:42 smithi029 ceph-mon[32610]: pgmap v1438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1804.392671 secs 2024-07-27T07:19:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.634+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-27T07:19:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.634+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T07:19:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.634+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f733c02b3a0 tx=0x7f733c037000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T07:19:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.634+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f733c02b3a0 tx=0x7f733c037000 comp rx=0 tx=0).stop 2024-07-27T07:19:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.634+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T07:19:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.634+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T07:19:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.634+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T07:19:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.634+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380c0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T07:19:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.634+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380c0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T07:19:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.635+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380c0770 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f733c004010 tx=0x7f733c03b000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T07:19:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.898+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:19:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.898+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:19:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.899+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:19:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:42.899+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:19:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:43 smithi047 ceph-mon[36835]: pgmap v1439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:43 smithi029 ceph-mon[32610]: pgmap v1439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:43 smithi077 ceph-mon[37153]: pgmap v1439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:46 smithi077 ceph-mon[37153]: pgmap v1440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:46.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:46 smithi047 ceph-mon[36835]: pgmap v1440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:46.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:46 smithi029 ceph-mon[32610]: pgmap v1440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1809.392814 secs 2024-07-27T07:19:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1809.392814 secs 2024-07-27T07:19:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1809.392814 secs 2024-07-27T07:19:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:48 smithi077 ceph-mon[37153]: pgmap v1441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:48 smithi047 ceph-mon[36835]: pgmap v1441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:48 smithi029 ceph-mon[32610]: pgmap v1441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:50 smithi077 ceph-mon[37153]: pgmap v1442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:50 smithi047 ceph-mon[36835]: pgmap v1442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:50 smithi029 ceph-mon[32610]: pgmap v1442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:19:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:52 smithi077 ceph-mon[37153]: pgmap v1443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1814.392929 secs 2024-07-27T07:19:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:52 smithi047 ceph-mon[36835]: pgmap v1443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1814.392929 secs 2024-07-27T07:19:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:52 smithi029 ceph-mon[32610]: pgmap v1443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1814.392929 secs 2024-07-27T07:19:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:54 smithi077 ceph-mon[37153]: pgmap v1444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:54 smithi047 ceph-mon[36835]: pgmap v1444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:54.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:54.447+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:19:54.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:54.447+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:19:54.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:54.447+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:19:54.448 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:54 smithi029 ceph-mon[32610]: pgmap v1444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:19:54.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:54.518+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:19:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:54.518+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:19:54.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:19:54.518+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:19:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:56 smithi077 ceph-mon[37153]: pgmap v1445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:56 smithi047 ceph-mon[36835]: pgmap v1445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:56 smithi029 ceph-mon[32610]: pgmap v1445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:19:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1819.393022 secs 2024-07-27T07:19:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1819.393022 secs 2024-07-27T07:19:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1819.393022 secs 2024-07-27T07:19:57.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.648+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T07:19:57.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.648+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T07:19:57.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.648+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7eff40003b30 tx=0x7eff40010000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T07:19:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.648+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7eff40003b30 tx=0x7eff40010000 comp rx=0 tx=0).stop 2024-07-27T07:19:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.648+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T07:19:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.648+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T07:19:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.648+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T07:19:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.648+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0c0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T07:19:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.648+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T07:19:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.648+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0c0810 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7eff40004480 tx=0x7eff4002b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T07:19:57.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.906+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:19:57.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.906+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:19:57.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.906+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:19:57.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:19:57.906+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:19:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:19:58 smithi029 ceph-mon[32610]: pgmap v1446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:19:58 smithi077 ceph-mon[37153]: pgmap v1446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:19:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:19:58 smithi047 ceph-mon[36835]: pgmap v1446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:20:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:00 smithi047 ceph-mon[36835]: pgmap v1447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:20:00.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:20:00.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:20:00.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:00 smithi047 ceph-mon[36835]: pgmap v1448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:20:00.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:00 smithi047 ceph-mon[36835]: pgmap v1449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:20:00.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:20:00.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:20:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:00 smithi029 ceph-mon[32610]: pgmap v1447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:20:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:20:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:20:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:00 smithi029 ceph-mon[32610]: pgmap v1448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:20:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:00 smithi029 ceph-mon[32610]: pgmap v1449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:20:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:20:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:20:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:00 smithi077 ceph-mon[37153]: pgmap v1447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:20:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:20:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:20:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:00 smithi077 ceph-mon[37153]: pgmap v1448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:20:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:00 smithi077 ceph-mon[37153]: pgmap v1449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:20:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:20:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:20:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:02 smithi047 ceph-mon[36835]: pgmap v1450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:20:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1824.393165 secs 2024-07-27T07:20:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:02 smithi029 ceph-mon[32610]: pgmap v1450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:20:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1824.393165 secs 2024-07-27T07:20:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:02 smithi077 ceph-mon[37153]: pgmap v1450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:20:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1824.393165 secs 2024-07-27T07:20:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:04 smithi047 ceph-mon[36835]: pgmap v1451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:20:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:04 smithi029 ceph-mon[32610]: pgmap v1451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:20:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:04 smithi077 ceph-mon[37153]: pgmap v1451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:20:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:06 smithi047 ceph-mon[36835]: pgmap v1452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:20:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:06 smithi029 ceph-mon[32610]: pgmap v1452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:20:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:06 smithi077 ceph-mon[37153]: pgmap v1452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:20:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1829.393291 secs 2024-07-27T07:20:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1829.393291 secs 2024-07-27T07:20:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1829.393291 secs 2024-07-27T07:20:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:08 smithi047 ceph-mon[36835]: pgmap v1453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:20:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:08 smithi029 ceph-mon[32610]: pgmap v1453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:20:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:08 smithi077 ceph-mon[37153]: pgmap v1453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:20:09.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:20:09.453+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:20:09.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:20:09.453+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:20:09.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:20:09.453+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:20:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:20:09.534+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:20:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:20:09.534+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:20:09.535 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:20:09.534+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:20:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:10 smithi047 ceph-mon[36835]: pgmap v1454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:20:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:10 smithi029 ceph-mon[32610]: pgmap v1454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:20:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:10 smithi077 ceph-mon[37153]: pgmap v1454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:20:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:12 smithi047 ceph-mon[36835]: pgmap v1455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1834.393420 secs 2024-07-27T07:20:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:12 smithi029 ceph-mon[32610]: pgmap v1455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1834.393420 secs 2024-07-27T07:20:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:12 smithi077 ceph-mon[37153]: pgmap v1455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1834.393420 secs 2024-07-27T07:20:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:14 smithi047 ceph-mon[36835]: pgmap v1456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:14 smithi029 ceph-mon[32610]: pgmap v1456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:14 smithi077 ceph-mon[37153]: pgmap v1456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:16 smithi047 ceph-mon[36835]: pgmap v1457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:16 smithi029 ceph-mon[32610]: pgmap v1457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:16 smithi077 ceph-mon[37153]: pgmap v1457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1839.393548 secs 2024-07-27T07:20:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1839.393548 secs 2024-07-27T07:20:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1839.393548 secs 2024-07-27T07:20:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:18 smithi047 ceph-mon[36835]: pgmap v1458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:18 smithi029 ceph-mon[32610]: pgmap v1458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:18 smithi077 ceph-mon[37153]: pgmap v1458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:20 smithi047 ceph-mon[36835]: pgmap v1459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:20 smithi029 ceph-mon[32610]: pgmap v1459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:20 smithi077 ceph-mon[37153]: pgmap v1459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:22 smithi047 ceph-mon[36835]: pgmap v1460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1844.393674 secs 2024-07-27T07:20:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:22 smithi029 ceph-mon[32610]: pgmap v1460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1844.393674 secs 2024-07-27T07:20:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:22 smithi077 ceph-mon[37153]: pgmap v1460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1844.393674 secs 2024-07-27T07:20:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:20:24.468+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:20:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:20:24.468+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:20:24.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:20:24.468+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:20:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:20:24.549+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:20:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:20:24.549+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:20:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:20:24.549+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:20:24.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:24 smithi029 ceph-mon[32610]: pgmap v1461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:24 smithi047 ceph-mon[36835]: pgmap v1461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:24 smithi077 ceph-mon[37153]: pgmap v1461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:26 smithi047 ceph-mon[36835]: pgmap v1462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:20:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:26 smithi029 ceph-mon[32610]: pgmap v1462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:20:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:26 smithi077 ceph-mon[37153]: pgmap v1462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:20:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1849.393821 secs 2024-07-27T07:20:27.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1849.393821 secs 2024-07-27T07:20:28.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1849.393821 secs 2024-07-27T07:20:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:28 smithi047 ceph-mon[36835]: pgmap v1463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:28 smithi029 ceph-mon[32610]: pgmap v1463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:28 smithi077 ceph-mon[37153]: pgmap v1463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:30 smithi047 ceph-mon[36835]: pgmap v1464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:30 smithi029 ceph-mon[32610]: pgmap v1464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:30 smithi077 ceph-mon[37153]: pgmap v1464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:32 smithi047 ceph-mon[36835]: pgmap v1465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1854.393957 secs 2024-07-27T07:20:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:32 smithi029 ceph-mon[32610]: pgmap v1465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1854.393957 secs 2024-07-27T07:20:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:32 smithi077 ceph-mon[37153]: pgmap v1465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1854.393957 secs 2024-07-27T07:20:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:34 smithi047 ceph-mon[36835]: pgmap v1466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:34 smithi029 ceph-mon[32610]: pgmap v1466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:34 smithi077 ceph-mon[37153]: pgmap v1466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:36 smithi047 ceph-mon[36835]: pgmap v1467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:20:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:36 smithi029 ceph-mon[32610]: pgmap v1467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:20:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:36 smithi077 ceph-mon[37153]: pgmap v1467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:20:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1859.394028 secs 2024-07-27T07:20:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1859.394028 secs 2024-07-27T07:20:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1859.394028 secs 2024-07-27T07:20:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:38 smithi047 ceph-mon[36835]: pgmap v1468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:38 smithi029 ceph-mon[32610]: pgmap v1468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:38 smithi077 ceph-mon[37153]: pgmap v1468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:39.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:20:39.483+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:20:39.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:20:39.483+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:20:39.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:20:39.483+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:20:39.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:20:39.564+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:20:39.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:20:39.564+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:20:39.566 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:20:39.564+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:20:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:40 smithi047 ceph-mon[36835]: pgmap v1469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:40.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:40 smithi029 ceph-mon[32610]: pgmap v1469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:40 smithi077 ceph-mon[37153]: pgmap v1469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:42 smithi047 ceph-mon[36835]: pgmap v1470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1864.394143 secs 2024-07-27T07:20:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:42 smithi029 ceph-mon[32610]: pgmap v1470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1864.394143 secs 2024-07-27T07:20:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:42 smithi077 ceph-mon[37153]: pgmap v1470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1864.394143 secs 2024-07-27T07:20:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:44 smithi047 ceph-mon[36835]: pgmap v1471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:44 smithi029 ceph-mon[32610]: pgmap v1471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:44 smithi077 ceph-mon[37153]: pgmap v1471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:46 smithi029 ceph-mon[32610]: pgmap v1472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:20:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:46 smithi077 ceph-mon[37153]: pgmap v1472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:20:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:46 smithi047 ceph-mon[36835]: pgmap v1472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:20:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1869.394220 secs 2024-07-27T07:20:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1869.394220 secs 2024-07-27T07:20:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1869.394220 secs 2024-07-27T07:20:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:48 smithi029 ceph-mon[32610]: pgmap v1473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:48 smithi077 ceph-mon[37153]: pgmap v1473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:48 smithi047 ceph-mon[36835]: pgmap v1473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:50 smithi029 ceph-mon[32610]: pgmap v1474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:50 smithi077 ceph-mon[37153]: pgmap v1474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:50 smithi047 ceph-mon[36835]: pgmap v1474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:20:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:52 smithi029 ceph-mon[32610]: pgmap v1475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1874.394338 secs 2024-07-27T07:20:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:52 smithi077 ceph-mon[37153]: pgmap v1475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1874.394338 secs 2024-07-27T07:20:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:52 smithi047 ceph-mon[36835]: pgmap v1475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:20:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1874.394338 secs 2024-07-27T07:20:54.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:20:54.498+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:20:54.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:20:54.498+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:20:54.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:20:54.498+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:20:54.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:20:54.575+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:20:54.576 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:20:54.575+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:20:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:20:54.575+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:20:54.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:54 smithi029 ceph-mon[32610]: pgmap v1476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:54.998 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:54 smithi047 ceph-mon[36835]: pgmap v1476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:54 smithi077 ceph-mon[37153]: pgmap v1476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:20:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:56 smithi077 ceph-mon[37153]: pgmap v1477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:20:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:56 smithi047 ceph-mon[36835]: pgmap v1477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:20:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:56 smithi029 ceph-mon[32610]: pgmap v1477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:20:58.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1879.394477 secs 2024-07-27T07:20:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1879.394477 secs 2024-07-27T07:20:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1879.394477 secs 2024-07-27T07:20:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:58 smithi077 ceph-mon[37153]: pgmap v1478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:58 smithi047 ceph-mon[36835]: pgmap v1478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:20:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:58 smithi029 ceph-mon[32610]: pgmap v1478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:21:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:20:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:21:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:20:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:21:00.199 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:20:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:21:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:00 smithi077 ceph-mon[37153]: pgmap v1479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:21:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:21:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:21:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:21:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:00 smithi047 ceph-mon[36835]: pgmap v1479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:21:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:21:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:21:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:21:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:00 smithi029 ceph-mon[32610]: pgmap v1479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:21:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:21:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:21:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:21:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:01 smithi077 ceph-mon[37153]: pgmap v1480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:21:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:01 smithi077 ceph-mon[37153]: pgmap v1481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:21:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:01 smithi047 ceph-mon[36835]: pgmap v1480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:21:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:01 smithi047 ceph-mon[36835]: pgmap v1481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:21:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:01 smithi029 ceph-mon[32610]: pgmap v1480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:21:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:01 smithi029 ceph-mon[32610]: pgmap v1481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:21:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1884.394597 secs 2024-07-27T07:21:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1884.394597 secs 2024-07-27T07:21:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1884.394597 secs 2024-07-27T07:21:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:03 smithi077 ceph-mon[37153]: pgmap v1482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:21:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:03 smithi047 ceph-mon[36835]: pgmap v1482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:21:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:03 smithi029 ceph-mon[32610]: pgmap v1482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:21:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:05 smithi077 ceph-mon[37153]: pgmap v1483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:21:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:05 smithi047 ceph-mon[36835]: pgmap v1483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:21:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:05 smithi029 ceph-mon[32610]: pgmap v1483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:21:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:07 smithi077 ceph-mon[37153]: pgmap v1484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:21:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1889.394661 secs 2024-07-27T07:21:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:07 smithi047 ceph-mon[36835]: pgmap v1484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:21:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1889.394661 secs 2024-07-27T07:21:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:07 smithi029 ceph-mon[32610]: pgmap v1484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:21:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1889.394661 secs 2024-07-27T07:21:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:21:09.514+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:21:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:21:09.514+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:21:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:21:09.514+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:21:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:21:09.582+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:21:09.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:21:09.582+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:21:09.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:21:09.582+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:21:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:09 smithi029 ceph-mon[32610]: pgmap v1485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:21:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:09 smithi077 ceph-mon[37153]: pgmap v1485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:21:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:09 smithi047 ceph-mon[36835]: pgmap v1485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:21:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:11 smithi077 ceph-mon[37153]: pgmap v1486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:21:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:11 smithi047 ceph-mon[36835]: pgmap v1486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:21:12.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:11 smithi029 ceph-mon[32610]: pgmap v1486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:21:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1894.394819 secs 2024-07-27T07:21:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1894.394819 secs 2024-07-27T07:21:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1894.394819 secs 2024-07-27T07:21:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:13 smithi077 ceph-mon[37153]: pgmap v1487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:13 smithi047 ceph-mon[36835]: pgmap v1487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:13 smithi029 ceph-mon[32610]: pgmap v1487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:16.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:15 smithi029 ceph-mon[32610]: pgmap v1488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:21:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:15 smithi077 ceph-mon[37153]: pgmap v1488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:21:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:15 smithi047 ceph-mon[36835]: pgmap v1488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:21:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:17 smithi077 ceph-mon[37153]: pgmap v1489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1899.394939 secs 2024-07-27T07:21:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:17 smithi047 ceph-mon[36835]: pgmap v1489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1899.394939 secs 2024-07-27T07:21:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:17 smithi029 ceph-mon[32610]: pgmap v1489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1899.394939 secs 2024-07-27T07:21:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:19 smithi077 ceph-mon[37153]: pgmap v1490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:21:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:19 smithi047 ceph-mon[36835]: pgmap v1490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:21:20.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:19 smithi029 ceph-mon[32610]: pgmap v1490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:21:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:21 smithi077 ceph-mon[37153]: pgmap v1491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:21 smithi047 ceph-mon[36835]: pgmap v1491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:21 smithi029 ceph-mon[32610]: pgmap v1491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1904.395066 secs 2024-07-27T07:21:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1904.395066 secs 2024-07-27T07:21:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1904.395066 secs 2024-07-27T07:21:24.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:23 smithi047 ceph-mon[36835]: pgmap v1492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:23 smithi077 ceph-mon[37153]: pgmap v1492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:24.112 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:23 smithi029 ceph-mon[32610]: pgmap v1492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:24.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:21:24.530+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:21:24.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:21:24.530+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:21:24.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:21:24.530+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:21:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:21:24.589+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:21:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:21:24.589+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:21:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:21:24.589+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:21:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:25 smithi077 ceph-mon[37153]: pgmap v1493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:21:26.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:25 smithi047 ceph-mon[36835]: pgmap v1493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:21:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:25 smithi029 ceph-mon[32610]: pgmap v1493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:21:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:27 smithi077 ceph-mon[37153]: pgmap v1494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1909.395178 secs 2024-07-27T07:21:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:27 smithi047 ceph-mon[36835]: pgmap v1494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1909.395178 secs 2024-07-27T07:21:28.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:27 smithi029 ceph-mon[32610]: pgmap v1494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1909.395178 secs 2024-07-27T07:21:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:29 smithi077 ceph-mon[37153]: pgmap v1495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:21:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:29 smithi047 ceph-mon[36835]: pgmap v1495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:21:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:29 smithi029 ceph-mon[32610]: pgmap v1495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:21:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:31 smithi077 ceph-mon[37153]: pgmap v1496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:31 smithi047 ceph-mon[36835]: pgmap v1496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:32.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:31 smithi029 ceph-mon[32610]: pgmap v1496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1914.395270 secs 2024-07-27T07:21:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1914.395270 secs 2024-07-27T07:21:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1914.395270 secs 2024-07-27T07:21:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:33 smithi077 ceph-mon[37153]: pgmap v1497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:33 smithi047 ceph-mon[36835]: pgmap v1497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:34.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:33 smithi029 ceph-mon[32610]: pgmap v1497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:35 smithi077 ceph-mon[37153]: pgmap v1498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:21:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:35 smithi047 ceph-mon[36835]: pgmap v1498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:21:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:35 smithi029 ceph-mon[32610]: pgmap v1498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:21:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:37 smithi077 ceph-mon[37153]: pgmap v1499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1919.395389 secs 2024-07-27T07:21:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:37 smithi047 ceph-mon[36835]: pgmap v1499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1919.395389 secs 2024-07-27T07:21:38.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:37 smithi029 ceph-mon[32610]: pgmap v1499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1919.395389 secs 2024-07-27T07:21:39.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:21:39.544+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:21:39.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:21:39.544+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:21:39.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:21:39.544+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:21:39.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:21:39.602+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:21:39.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:21:39.602+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:21:39.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:21:39.602+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:21:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:39 smithi077 ceph-mon[37153]: pgmap v1500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:21:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:39 smithi047 ceph-mon[36835]: pgmap v1500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:21:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:39 smithi029 ceph-mon[32610]: pgmap v1500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:21:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:41 smithi077 ceph-mon[37153]: pgmap v1501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:21:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:41 smithi047 ceph-mon[36835]: pgmap v1501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:21:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:41 smithi029 ceph-mon[32610]: pgmap v1501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:21:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1924.395541 secs 2024-07-27T07:21:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1924.395541 secs 2024-07-27T07:21:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1924.395541 secs 2024-07-27T07:21:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:43 smithi077 ceph-mon[37153]: pgmap v1502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:43 smithi047 ceph-mon[36835]: pgmap v1502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:43 smithi029 ceph-mon[32610]: pgmap v1502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:45 smithi077 ceph-mon[37153]: pgmap v1503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:21:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:45 smithi047 ceph-mon[36835]: pgmap v1503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:21:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:45 smithi029 ceph-mon[32610]: pgmap v1503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:21:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:47 smithi077 ceph-mon[37153]: pgmap v1504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:21:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1929.395667 secs 2024-07-27T07:21:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:47 smithi047 ceph-mon[36835]: pgmap v1504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:21:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1929.395667 secs 2024-07-27T07:21:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:47 smithi029 ceph-mon[32610]: pgmap v1504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:21:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1929.395667 secs 2024-07-27T07:21:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:49 smithi077 ceph-mon[37153]: pgmap v1505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:21:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:49 smithi047 ceph-mon[36835]: pgmap v1505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:21:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:49 smithi029 ceph-mon[32610]: pgmap v1505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:21:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:51 smithi077 ceph-mon[37153]: pgmap v1506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:21:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:51 smithi047 ceph-mon[36835]: pgmap v1506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:21:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:51 smithi029 ceph-mon[32610]: pgmap v1506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:21:53.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1934.395773 secs 2024-07-27T07:21:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1934.395773 secs 2024-07-27T07:21:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1934.395773 secs 2024-07-27T07:21:54.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:53 smithi047 ceph-mon[36835]: pgmap v1507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:53 smithi077 ceph-mon[37153]: pgmap v1507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:54.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:53 smithi029 ceph-mon[32610]: pgmap v1507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:21:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:21:54.560+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:21:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:21:54.560+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:21:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:21:54.560+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:21:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:21:54.603+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:21:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:21:54.603+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:21:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:21:54.603+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:21:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:55 smithi077 ceph-mon[37153]: pgmap v1508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:21:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:55 smithi047 ceph-mon[36835]: pgmap v1508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:21:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:55 smithi029 ceph-mon[32610]: pgmap v1508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:21:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:57 smithi077 ceph-mon[37153]: pgmap v1509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1939.395867 secs 2024-07-27T07:21:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:57 smithi047 ceph-mon[36835]: pgmap v1509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1939.395867 secs 2024-07-27T07:21:58.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:57 smithi029 ceph-mon[32610]: pgmap v1509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:21:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1939.395867 secs 2024-07-27T07:22:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:21:59 smithi047 ceph-mon[36835]: pgmap v1510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:22:00.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:21:59 smithi029 ceph-mon[32610]: pgmap v1510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:22:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:21:59 smithi077 ceph-mon[37153]: pgmap v1510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:22:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:22:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:22:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:22:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:22:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:22:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:22:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:22:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:22:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:22:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:22:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:22:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:22:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:22:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:22:01.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:22:01.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:22:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:22:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:22:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:22:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:22:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:22:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:22:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:22:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:22:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:01 smithi047 ceph-mon[36835]: pgmap v1511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:01 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:22:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:01 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:22:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:01 smithi047 ceph-mon[36835]: pgmap v1512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:22:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:01 smithi047 ceph-mon[36835]: pgmap v1513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:22:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:01 smithi029 ceph-mon[32610]: pgmap v1511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:01 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:22:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:01 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:22:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:01 smithi029 ceph-mon[32610]: pgmap v1512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:22:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:01 smithi029 ceph-mon[32610]: pgmap v1513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:22:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:01 smithi077 ceph-mon[37153]: pgmap v1511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:01 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:22:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:01 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:22:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:01 smithi077 ceph-mon[37153]: pgmap v1512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:22:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:01 smithi077 ceph-mon[37153]: pgmap v1513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:22:03.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1944.395936 secs 2024-07-27T07:22:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1944.395936 secs 2024-07-27T07:22:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1944.395936 secs 2024-07-27T07:22:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:03 smithi047 ceph-mon[36835]: pgmap v1514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:22:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:03 smithi029 ceph-mon[32610]: pgmap v1514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:22:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:03 smithi077 ceph-mon[37153]: pgmap v1514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:22:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:04 smithi047 ceph-mon[36835]: pgmap v1515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:22:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:04 smithi029 ceph-mon[32610]: pgmap v1515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:22:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:04 smithi077 ceph-mon[37153]: pgmap v1515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:22:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:07 smithi077 ceph-mon[37153]: pgmap v1516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:22:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1949.396032 secs 2024-07-27T07:22:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:07 smithi047 ceph-mon[36835]: pgmap v1516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:22:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1949.396032 secs 2024-07-27T07:22:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:07 smithi029 ceph-mon[32610]: pgmap v1516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:22:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1949.396032 secs 2024-07-27T07:22:09.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:09.576+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:22:09.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:09.576+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:22:09.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:09.576+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:22:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:09.619+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:22:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:09.619+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:22:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:09.619+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:22:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:09 smithi029 ceph-mon[32610]: pgmap v1517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:22:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:09 smithi077 ceph-mon[37153]: pgmap v1517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:22:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:09 smithi047 ceph-mon[36835]: pgmap v1517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:22:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:11 smithi077 ceph-mon[37153]: pgmap v1518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:22:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:11 smithi047 ceph-mon[36835]: pgmap v1518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:22:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:11 smithi029 ceph-mon[32610]: pgmap v1518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:22:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1954.396180 secs 2024-07-27T07:22:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1954.396180 secs 2024-07-27T07:22:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1954.396180 secs 2024-07-27T07:22:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:13 smithi077 ceph-mon[37153]: pgmap v1519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:13 smithi047 ceph-mon[36835]: pgmap v1519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:13 smithi029 ceph-mon[32610]: pgmap v1519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:15 smithi077 ceph-mon[37153]: pgmap v1520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:15 smithi047 ceph-mon[36835]: pgmap v1520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:16.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:15 smithi029 ceph-mon[32610]: pgmap v1520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:17 smithi077 ceph-mon[37153]: pgmap v1521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1959.396262 secs 2024-07-27T07:22:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:17 smithi047 ceph-mon[36835]: pgmap v1521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1959.396262 secs 2024-07-27T07:22:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:17 smithi029 ceph-mon[32610]: pgmap v1521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1959.396262 secs 2024-07-27T07:22:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:19 smithi077 ceph-mon[37153]: pgmap v1522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:22:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:19 smithi047 ceph-mon[36835]: pgmap v1522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:22:20.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:19 smithi029 ceph-mon[32610]: pgmap v1522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:22:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:21 smithi077 ceph-mon[37153]: pgmap v1523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:21 smithi047 ceph-mon[36835]: pgmap v1523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:21 smithi029 ceph-mon[32610]: pgmap v1523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1964.396362 secs 2024-07-27T07:22:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1964.396362 secs 2024-07-27T07:22:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1964.396362 secs 2024-07-27T07:22:24.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:23 smithi047 ceph-mon[36835]: pgmap v1524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:23 smithi077 ceph-mon[37153]: pgmap v1524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:24.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:23 smithi029 ceph-mon[32610]: pgmap v1524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:24.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:24.582+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:22:24.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:24.582+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:22:24.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:24.582+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:22:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:24.634+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:22:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:24.634+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:22:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:24.634+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:22:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:25 smithi077 ceph-mon[37153]: pgmap v1525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:25 smithi047 ceph-mon[36835]: pgmap v1525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:25 smithi029 ceph-mon[32610]: pgmap v1525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:27 smithi077 ceph-mon[37153]: pgmap v1526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1969.396511 secs 2024-07-27T07:22:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:27 smithi047 ceph-mon[36835]: pgmap v1526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1969.396511 secs 2024-07-27T07:22:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:27 smithi029 ceph-mon[32610]: pgmap v1526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1969.396511 secs 2024-07-27T07:22:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:29 smithi077 ceph-mon[37153]: pgmap v1527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:22:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:29 smithi047 ceph-mon[36835]: pgmap v1527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:22:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:29 smithi029 ceph-mon[32610]: pgmap v1527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:22:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:31 smithi077 ceph-mon[37153]: pgmap v1528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:31 smithi047 ceph-mon[36835]: pgmap v1528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:31 smithi029 ceph-mon[32610]: pgmap v1528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1974.396631 secs 2024-07-27T07:22:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1974.396631 secs 2024-07-27T07:22:33.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1974.396631 secs 2024-07-27T07:22:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:33 smithi077 ceph-mon[37153]: pgmap v1529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:33 smithi047 ceph-mon[36835]: pgmap v1529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:33 smithi029 ceph-mon[32610]: pgmap v1529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:35 smithi077 ceph-mon[37153]: pgmap v1530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:35 smithi047 ceph-mon[36835]: pgmap v1530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:35 smithi029 ceph-mon[32610]: pgmap v1530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:37 smithi077 ceph-mon[37153]: pgmap v1531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1979.396752 secs 2024-07-27T07:22:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:37 smithi047 ceph-mon[36835]: pgmap v1531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1979.396752 secs 2024-07-27T07:22:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:37 smithi029 ceph-mon[32610]: pgmap v1531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1979.396752 secs 2024-07-27T07:22:39.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:39.591+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:22:39.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:39.591+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:22:39.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:39.591+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:22:39.650 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:39.649+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:22:39.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:39.649+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:22:39.651 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:39.649+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:22:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:39 smithi029 ceph-mon[32610]: pgmap v1532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:22:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:39 smithi077 ceph-mon[37153]: pgmap v1532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:22:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:39 smithi047 ceph-mon[36835]: pgmap v1532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:22:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:41 smithi077 ceph-mon[37153]: pgmap v1533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:41 smithi047 ceph-mon[36835]: pgmap v1533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:41 smithi029 ceph-mon[32610]: pgmap v1533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:42.926+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:22:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:42.926+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:22:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:42.926+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:22:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:42.926+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:22:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:42.926+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065f0 con 0x7f7348108670 2024-07-27T07:22:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:42.926+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:22:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:42.926+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065f0 con 0x7f7348108670 2024-07-27T07:22:42.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:42.926+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:22:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1984.396874 secs 2024-07-27T07:22:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1984.396874 secs 2024-07-27T07:22:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1984.396874 secs 2024-07-27T07:22:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:43 smithi077 ceph-mon[37153]: pgmap v1534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:43 smithi047 ceph-mon[36835]: pgmap v1534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:43 smithi029 ceph-mon[32610]: pgmap v1534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:45 smithi077 ceph-mon[37153]: pgmap v1535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:45 smithi047 ceph-mon[36835]: pgmap v1535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:46.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:45 smithi029 ceph-mon[32610]: pgmap v1535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1989.396998 secs 2024-07-27T07:22:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:47 smithi077 ceph-mon[37153]: pgmap v1536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1989.396998 secs 2024-07-27T07:22:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:47 smithi047 ceph-mon[36835]: pgmap v1536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1989.396998 secs 2024-07-27T07:22:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:47 smithi029 ceph-mon[32610]: pgmap v1536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:50.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:49 smithi077 ceph-mon[37153]: pgmap v1537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:22:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:49 smithi047 ceph-mon[36835]: pgmap v1537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:22:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:49 smithi029 ceph-mon[32610]: pgmap v1537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:22:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:51 smithi077 ceph-mon[37153]: pgmap v1538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:51 smithi047 ceph-mon[36835]: pgmap v1538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:51 smithi029 ceph-mon[32610]: pgmap v1538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:22:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1994.397105 secs 2024-07-27T07:22:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1994.397105 secs 2024-07-27T07:22:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1994.397105 secs 2024-07-27T07:22:54.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:53 smithi047 ceph-mon[36835]: pgmap v1539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:53 smithi077 ceph-mon[37153]: pgmap v1539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:54.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:53 smithi029 ceph-mon[32610]: pgmap v1539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:22:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:54.606+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:22:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:54.606+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:22:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:54.606+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:22:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:54.664+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:22:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:54.664+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:22:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:22:54.664+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:22:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:55 smithi077 ceph-mon[37153]: pgmap v1540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:55 smithi047 ceph-mon[36835]: pgmap v1540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:55 smithi029 ceph-mon[32610]: pgmap v1540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:22:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:57.934+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:22:57.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:57.934+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:22:57.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:57.934+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:22:57.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:57.934+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff240065e0 con 0x7eff50104ab0 2024-07-27T07:22:57.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:57.934+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:22:57.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:57.934+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:22:57.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:57.934+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff240065e0 con 0x7eff50104ab0 2024-07-27T07:22:57.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:22:57.934+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:22:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 1999.397248 secs 2024-07-27T07:22:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:57 smithi077 ceph-mon[37153]: pgmap v1541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 1999.397248 secs 2024-07-27T07:22:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:57 smithi047 ceph-mon[36835]: pgmap v1541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:22:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 1999.397248 secs 2024-07-27T07:22:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:57 smithi029 ceph-mon[32610]: pgmap v1541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:22:59 smithi077 ceph-mon[37153]: pgmap v1542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:22:59 smithi047 ceph-mon[36835]: pgmap v1542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:00.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:22:59 smithi029 ceph-mon[32610]: pgmap v1542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:23:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:23:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:23:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:01 smithi077 ceph-mon[37153]: pgmap v1543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:23:02.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:23:02.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:23:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:01 smithi047 ceph-mon[36835]: pgmap v1543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:23:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:23:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:23:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:01 smithi029 ceph-mon[32610]: pgmap v1543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:23:02.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:23:02.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:23:02.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:23:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:02 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:23:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:02 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:23:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:02 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:23:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:02 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:23:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:02 smithi077 ceph-mon[37153]: pgmap v1544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T07:23:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:02 smithi077 ceph-mon[37153]: pgmap v1545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:23:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2004.397408 secs 2024-07-27T07:23:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:02 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:23:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:02 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:23:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:02 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:23:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:02 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:23:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:02 smithi047 ceph-mon[36835]: pgmap v1544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T07:23:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:02 smithi047 ceph-mon[36835]: pgmap v1545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:23:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2004.397408 secs 2024-07-27T07:23:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:02 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:23:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:02 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:23:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:02 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:23:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:02 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:23:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:02 smithi029 ceph-mon[32610]: pgmap v1544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T07:23:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:02 smithi029 ceph-mon[32610]: pgmap v1545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:23:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2004.397408 secs 2024-07-27T07:23:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:04 smithi077 ceph-mon[37153]: pgmap v1546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:23:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:04 smithi047 ceph-mon[36835]: pgmap v1546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:23:05.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:04 smithi029 ceph-mon[32610]: pgmap v1546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:23:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:06 smithi077 ceph-mon[37153]: pgmap v1547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:23:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:06 smithi047 ceph-mon[36835]: pgmap v1547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:23:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:06 smithi029 ceph-mon[32610]: pgmap v1547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:23:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2009.397556 secs 2024-07-27T07:23:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2009.397556 secs 2024-07-27T07:23:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2009.397556 secs 2024-07-27T07:23:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:08 smithi047 ceph-mon[36835]: pgmap v1548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:23:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:08 smithi029 ceph-mon[32610]: pgmap v1548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:23:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:08 smithi077 ceph-mon[37153]: pgmap v1548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:23:09.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:23:09.622+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:23:09.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:23:09.622+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:23:09.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:23:09.622+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:23:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:23:09.670+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:23:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:23:09.670+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:23:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:23:09.670+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:23:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:10 smithi047 ceph-mon[36835]: pgmap v1549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:23:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:10 smithi029 ceph-mon[32610]: pgmap v1549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:23:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:10 smithi077 ceph-mon[37153]: pgmap v1549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:23:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:12 smithi047 ceph-mon[36835]: pgmap v1550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:23:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2014.397687 secs 2024-07-27T07:23:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:12 smithi029 ceph-mon[32610]: pgmap v1550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:23:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2014.397687 secs 2024-07-27T07:23:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:12 smithi077 ceph-mon[37153]: pgmap v1550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:23:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2014.397687 secs 2024-07-27T07:23:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:14 smithi047 ceph-mon[36835]: pgmap v1551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:14 smithi029 ceph-mon[32610]: pgmap v1551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:14 smithi077 ceph-mon[37153]: pgmap v1551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:16 smithi047 ceph-mon[36835]: pgmap v1552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:16 smithi029 ceph-mon[32610]: pgmap v1552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:16 smithi077 ceph-mon[37153]: pgmap v1552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2019.397800 secs 2024-07-27T07:23:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2019.397800 secs 2024-07-27T07:23:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2019.397800 secs 2024-07-27T07:23:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:18 smithi047 ceph-mon[36835]: pgmap v1553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:18 smithi029 ceph-mon[32610]: pgmap v1553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:18 smithi077 ceph-mon[37153]: pgmap v1553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:21.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:20 smithi047 ceph-mon[36835]: pgmap v1554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:23:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:20 smithi029 ceph-mon[32610]: pgmap v1554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:23:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:20 smithi077 ceph-mon[37153]: pgmap v1554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:23:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:21 smithi047 ceph-mon[36835]: pgmap v1555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:21 smithi029 ceph-mon[32610]: pgmap v1555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:21 smithi077 ceph-mon[37153]: pgmap v1555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2024.397931 secs 2024-07-27T07:23:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2024.397931 secs 2024-07-27T07:23:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2024.397931 secs 2024-07-27T07:23:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:23 smithi047 ceph-mon[36835]: pgmap v1556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:24.215 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:23 smithi029 ceph-mon[32610]: pgmap v1556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:23 smithi077 ceph-mon[37153]: pgmap v1556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:24.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:23:24.636+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:23:24.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:23:24.636+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:23:24.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:23:24.636+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:23:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:23:24.673+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:23:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:23:24.673+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:23:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:23:24.673+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:23:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:26 smithi077 ceph-mon[37153]: pgmap v1557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:26 smithi047 ceph-mon[36835]: pgmap v1557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:26 smithi029 ceph-mon[32610]: pgmap v1557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2029.398063 secs 2024-07-27T07:23:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2029.398063 secs 2024-07-27T07:23:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2029.398063 secs 2024-07-27T07:23:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:28 smithi077 ceph-mon[37153]: pgmap v1558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:28 smithi047 ceph-mon[36835]: pgmap v1558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:28 smithi029 ceph-mon[32610]: pgmap v1558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:30 smithi077 ceph-mon[37153]: pgmap v1559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:23:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:30 smithi047 ceph-mon[36835]: pgmap v1559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:23:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:30 smithi029 ceph-mon[32610]: pgmap v1559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:23:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:32 smithi077 ceph-mon[37153]: pgmap v1560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2034.398195 secs 2024-07-27T07:23:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:32 smithi047 ceph-mon[36835]: pgmap v1560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2034.398195 secs 2024-07-27T07:23:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:32 smithi029 ceph-mon[32610]: pgmap v1560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:32.743 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2034.398195 secs 2024-07-27T07:23:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:34 smithi077 ceph-mon[37153]: pgmap v1561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:34 smithi047 ceph-mon[36835]: pgmap v1561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:34 smithi029 ceph-mon[32610]: pgmap v1561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:36 smithi077 ceph-mon[37153]: pgmap v1562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:23:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:36 smithi047 ceph-mon[36835]: pgmap v1562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:23:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:36 smithi029 ceph-mon[32610]: pgmap v1562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:23:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2039.398304 secs 2024-07-27T07:23:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2039.398304 secs 2024-07-27T07:23:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2039.398304 secs 2024-07-27T07:23:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:38 smithi077 ceph-mon[37153]: pgmap v1563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:38 smithi047 ceph-mon[36835]: pgmap v1563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:38 smithi029 ceph-mon[32610]: pgmap v1563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:39.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:23:39.640+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:23:39.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:23:39.640+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:23:39.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:23:39.640+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:23:39.684 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:23:39.683+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:23:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:23:39.683+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:23:39.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:23:39.683+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:23:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:40 smithi077 ceph-mon[37153]: pgmap v1564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:23:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:40 smithi047 ceph-mon[36835]: pgmap v1564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:23:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:40 smithi029 ceph-mon[32610]: pgmap v1564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:23:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:42 smithi077 ceph-mon[37153]: pgmap v1565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2044.398443 secs 2024-07-27T07:23:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:42 smithi047 ceph-mon[36835]: pgmap v1565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2044.398443 secs 2024-07-27T07:23:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:42 smithi029 ceph-mon[32610]: pgmap v1565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2044.398443 secs 2024-07-27T07:23:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:44 smithi077 ceph-mon[37153]: pgmap v1566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:44 smithi047 ceph-mon[36835]: pgmap v1566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:44.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:44 smithi029 ceph-mon[32610]: pgmap v1566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:46 smithi077 ceph-mon[37153]: pgmap v1567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:46 smithi047 ceph-mon[36835]: pgmap v1567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:46.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:46 smithi029 ceph-mon[32610]: pgmap v1567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2049.398591 secs 2024-07-27T07:23:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2049.398591 secs 2024-07-27T07:23:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2049.398591 secs 2024-07-27T07:23:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:48 smithi077 ceph-mon[37153]: pgmap v1568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:48 smithi047 ceph-mon[36835]: pgmap v1568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:48 smithi029 ceph-mon[32610]: pgmap v1568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:50 smithi077 ceph-mon[37153]: pgmap v1569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:23:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:50 smithi047 ceph-mon[36835]: pgmap v1569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:23:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:50 smithi029 ceph-mon[32610]: pgmap v1569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:23:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:52 smithi077 ceph-mon[37153]: pgmap v1570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2054.398744 secs 2024-07-27T07:23:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:52 smithi047 ceph-mon[36835]: pgmap v1570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2054.398744 secs 2024-07-27T07:23:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:52 smithi029 ceph-mon[32610]: pgmap v1570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2054.398744 secs 2024-07-27T07:23:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:54 smithi077 ceph-mon[37153]: pgmap v1571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:54.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:23:54.655+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:23:54.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:23:54.655+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:23:54.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:23:54.655+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:23:54.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:54 smithi029 ceph-mon[32610]: pgmap v1571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:54 smithi047 ceph-mon[36835]: pgmap v1571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:23:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:23:54.697+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:23:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:23:54.697+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:23:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:23:54.697+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:23:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:56 smithi077 ceph-mon[37153]: pgmap v1572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:56 smithi047 ceph-mon[36835]: pgmap v1572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:56 smithi029 ceph-mon[32610]: pgmap v1572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:23:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2059.398871 secs 2024-07-27T07:23:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2059.398871 secs 2024-07-27T07:23:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2059.398871 secs 2024-07-27T07:23:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:23:58 smithi047 ceph-mon[36835]: pgmap v1573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:23:58 smithi029 ceph-mon[32610]: pgmap v1573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:23:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:23:58 smithi077 ceph-mon[37153]: pgmap v1573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:00.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:00 smithi077 ceph-mon[37153]: pgmap v1574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:24:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:00 smithi047 ceph-mon[36835]: pgmap v1574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:24:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:00 smithi029 ceph-mon[32610]: pgmap v1574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:24:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:24:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:24:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:24:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:02 smithi047 ceph-mon[36835]: pgmap v1575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2064.398944 secs 2024-07-27T07:24:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:24:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:24:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:02 smithi047 ceph-mon[36835]: pgmap v1576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:24:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:02 smithi047 ceph-mon[36835]: pgmap v1577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:24:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:24:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:02 smithi029 ceph-mon[32610]: pgmap v1575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2064.398944 secs 2024-07-27T07:24:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:24:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:24:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:02 smithi029 ceph-mon[32610]: pgmap v1576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:24:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:02 smithi029 ceph-mon[32610]: pgmap v1577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:24:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:24:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:02 smithi077 ceph-mon[37153]: pgmap v1575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2064.398944 secs 2024-07-27T07:24:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:24:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:24:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:02 smithi077 ceph-mon[37153]: pgmap v1576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:24:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:02 smithi077 ceph-mon[37153]: pgmap v1577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:24:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:24:05.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:04 smithi077 ceph-mon[37153]: pgmap v1578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:24:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:04 smithi047 ceph-mon[36835]: pgmap v1578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:24:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:04 smithi029 ceph-mon[32610]: pgmap v1578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:24:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:06 smithi077 ceph-mon[37153]: pgmap v1579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:24:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:06 smithi047 ceph-mon[36835]: pgmap v1579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:24:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:06 smithi029 ceph-mon[32610]: pgmap v1579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:24:08.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2069.399086 secs 2024-07-27T07:24:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2069.399086 secs 2024-07-27T07:24:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2069.399086 secs 2024-07-27T07:24:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:08 smithi077 ceph-mon[37153]: pgmap v1580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:24:09.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:08 smithi047 ceph-mon[36835]: pgmap v1580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:24:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:08 smithi029 ceph-mon[32610]: pgmap v1580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:24:09.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:24:09.668+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:24:09.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:24:09.668+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:24:09.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:24:09.668+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:24:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:24:09.713+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:24:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:24:09.713+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:24:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:24:09.713+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:24:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:10 smithi077 ceph-mon[37153]: pgmap v1581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:24:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:10 smithi047 ceph-mon[36835]: pgmap v1581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:24:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:10 smithi029 ceph-mon[32610]: pgmap v1581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:24:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2074.399225 secs 2024-07-27T07:24:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:12 smithi047 ceph-mon[36835]: pgmap v1582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:24:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2074.399225 secs 2024-07-27T07:24:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:12 smithi029 ceph-mon[32610]: pgmap v1582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:24:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2074.399225 secs 2024-07-27T07:24:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:12 smithi077 ceph-mon[37153]: pgmap v1582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:24:15.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:14 smithi047 ceph-mon[36835]: pgmap v1583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:14 smithi029 ceph-mon[32610]: pgmap v1583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:14 smithi077 ceph-mon[37153]: pgmap v1583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:16 smithi047 ceph-mon[36835]: pgmap v1584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:17.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:16 smithi029 ceph-mon[32610]: pgmap v1584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:16 smithi077 ceph-mon[37153]: pgmap v1584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2079.399360 secs 2024-07-27T07:24:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2079.399360 secs 2024-07-27T07:24:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2079.399360 secs 2024-07-27T07:24:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:18 smithi047 ceph-mon[36835]: pgmap v1585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:18 smithi029 ceph-mon[32610]: pgmap v1585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:18 smithi077 ceph-mon[37153]: pgmap v1585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:20 smithi047 ceph-mon[36835]: pgmap v1586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:24:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:20 smithi029 ceph-mon[32610]: pgmap v1586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:24:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:20 smithi077 ceph-mon[37153]: pgmap v1586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:24:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2084.399497 secs 2024-07-27T07:24:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:22 smithi047 ceph-mon[36835]: pgmap v1587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2084.399497 secs 2024-07-27T07:24:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:22 smithi029 ceph-mon[32610]: pgmap v1587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2084.399497 secs 2024-07-27T07:24:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:22 smithi077 ceph-mon[37153]: pgmap v1587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:24.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:24:24.683+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:24:24.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:24:24.683+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:24:24.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:24:24.683+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:24:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:24:24.729+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:24:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:24:24.729+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:24:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:24:24.729+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:24:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:24 smithi029 ceph-mon[32610]: pgmap v1588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:24 smithi047 ceph-mon[36835]: pgmap v1588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:24 smithi077 ceph-mon[37153]: pgmap v1588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:26 smithi047 ceph-mon[36835]: pgmap v1589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:26 smithi029 ceph-mon[32610]: pgmap v1589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:26 smithi077 ceph-mon[37153]: pgmap v1589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2089.399600 secs 2024-07-27T07:24:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2089.399600 secs 2024-07-27T07:24:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2089.399600 secs 2024-07-27T07:24:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:28 smithi047 ceph-mon[36835]: pgmap v1590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:28 smithi029 ceph-mon[32610]: pgmap v1590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:28 smithi077 ceph-mon[37153]: pgmap v1590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:31.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:30 smithi077 ceph-mon[37153]: pgmap v1591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:24:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:30 smithi047 ceph-mon[36835]: pgmap v1591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:24:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:30 smithi029 ceph-mon[32610]: pgmap v1591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:24:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2094.399702 secs 2024-07-27T07:24:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:32 smithi047 ceph-mon[36835]: pgmap v1592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2094.399702 secs 2024-07-27T07:24:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:32 smithi029 ceph-mon[32610]: pgmap v1592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2094.399702 secs 2024-07-27T07:24:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:32 smithi077 ceph-mon[37153]: pgmap v1592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:34 smithi047 ceph-mon[36835]: pgmap v1593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:34 smithi029 ceph-mon[32610]: pgmap v1593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:34 smithi077 ceph-mon[37153]: pgmap v1593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:35 smithi047 ceph-mon[36835]: pgmap v1594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:36.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:35 smithi029 ceph-mon[32610]: pgmap v1594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:35 smithi077 ceph-mon[37153]: pgmap v1594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2099.399829 secs 2024-07-27T07:24:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2099.399829 secs 2024-07-27T07:24:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2099.399829 secs 2024-07-27T07:24:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:38 smithi077 ceph-mon[37153]: pgmap v1595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:38 smithi047 ceph-mon[36835]: pgmap v1595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:38 smithi029 ceph-mon[32610]: pgmap v1595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:39.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:24:39.690+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:24:39.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:24:39.690+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:24:39.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:24:39.690+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:24:39.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:24:39.741+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:24:39.742 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:24:39.741+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:24:39.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:24:39.741+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:24:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:40 smithi077 ceph-mon[37153]: pgmap v1596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:40 smithi047 ceph-mon[36835]: pgmap v1596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:40 smithi029 ceph-mon[32610]: pgmap v1596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2104.399914 secs 2024-07-27T07:24:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:42 smithi077 ceph-mon[37153]: pgmap v1597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2104.399914 secs 2024-07-27T07:24:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:42 smithi047 ceph-mon[36835]: pgmap v1597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2104.399914 secs 2024-07-27T07:24:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:42 smithi029 ceph-mon[32610]: pgmap v1597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:24:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:44 smithi047 ceph-mon[36835]: pgmap v1598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:44 smithi029 ceph-mon[32610]: pgmap v1598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:44 smithi077 ceph-mon[37153]: pgmap v1598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:46 smithi047 ceph-mon[36835]: pgmap v1599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:46 smithi029 ceph-mon[32610]: pgmap v1599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:46 smithi077 ceph-mon[37153]: pgmap v1599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2109.400044 secs 2024-07-27T07:24:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2109.400044 secs 2024-07-27T07:24:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2109.400044 secs 2024-07-27T07:24:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:48 smithi047 ceph-mon[36835]: pgmap v1600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:48 smithi029 ceph-mon[32610]: pgmap v1600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:48 smithi077 ceph-mon[37153]: pgmap v1600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:50 smithi047 ceph-mon[36835]: pgmap v1601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:50 smithi029 ceph-mon[32610]: pgmap v1601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:50 smithi077 ceph-mon[37153]: pgmap v1601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2114.400117 secs 2024-07-27T07:24:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:52 smithi047 ceph-mon[36835]: pgmap v1602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:24:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2114.400117 secs 2024-07-27T07:24:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:52 smithi029 ceph-mon[32610]: pgmap v1602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:24:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2114.400117 secs 2024-07-27T07:24:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:52 smithi077 ceph-mon[37153]: pgmap v1602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:24:54.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:24:54.706+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:24:54.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:24:54.706+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:24:54.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:24:54.706+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:24:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:24:54.744+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:24:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:24:54.744+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:24:54.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:24:54.744+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:24:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:54 smithi029 ceph-mon[32610]: pgmap v1603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:54 smithi047 ceph-mon[36835]: pgmap v1603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:54 smithi077 ceph-mon[37153]: pgmap v1603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:24:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:56 smithi047 ceph-mon[36835]: pgmap v1604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:56 smithi029 ceph-mon[32610]: pgmap v1604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:56 smithi077 ceph-mon[37153]: pgmap v1604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:24:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2119.400242 secs 2024-07-27T07:24:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2119.400242 secs 2024-07-27T07:24:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2119.400242 secs 2024-07-27T07:24:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:24:58 smithi047 ceph-mon[36835]: pgmap v1605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:24:58 smithi029 ceph-mon[32610]: pgmap v1605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:24:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:24:58 smithi077 ceph-mon[37153]: pgmap v1605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:25:01.158 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:00 smithi077 ceph-mon[37153]: pgmap v1606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:00 smithi047 ceph-mon[36835]: pgmap v1606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:00 smithi029 ceph-mon[32610]: pgmap v1606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:25:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:25:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:25:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2124.400367 secs 2024-07-27T07:25:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:02 smithi047 ceph-mon[36835]: pgmap v1607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:25:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:25:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:25:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:25:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2124.400367 secs 2024-07-27T07:25:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:02 smithi029 ceph-mon[32610]: pgmap v1607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:25:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:25:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:25:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:25:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2124.400367 secs 2024-07-27T07:25:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:02 smithi077 ceph-mon[37153]: pgmap v1607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:25:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:25:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:25:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:25:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:03 smithi047 ceph-mon[36835]: pgmap v1608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:25:04.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:03 smithi047 ceph-mon[36835]: pgmap v1609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:25:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:03 smithi029 ceph-mon[32610]: pgmap v1608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:25:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:03 smithi029 ceph-mon[32610]: pgmap v1609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:25:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:03 smithi077 ceph-mon[37153]: pgmap v1608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:25:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:03 smithi077 ceph-mon[37153]: pgmap v1609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:25:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:04 smithi047 ceph-mon[36835]: pgmap v1610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:25:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:04 smithi029 ceph-mon[32610]: pgmap v1610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:25:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:04 smithi077 ceph-mon[37153]: pgmap v1610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:25:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:07 smithi047 ceph-mon[36835]: pgmap v1611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:25:08.053 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2129.400492 secs 2024-07-27T07:25:08.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:07 smithi077 ceph-mon[37153]: pgmap v1611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:25:08.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2129.400492 secs 2024-07-27T07:25:08.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:07 smithi029 ceph-mon[32610]: pgmap v1611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:25:08.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2129.400492 secs 2024-07-27T07:25:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:09 smithi047 ceph-mon[36835]: pgmap v1612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:25:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:09 smithi029 ceph-mon[32610]: pgmap v1612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:25:09.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:09.720+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:25:09.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:09.720+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:25:09.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:09.720+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:25:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:09.758+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:25:09.758 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:09.758+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:25:09.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:09.758+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:25:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:09 smithi077 ceph-mon[37153]: pgmap v1612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:25:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:11 smithi047 ceph-mon[36835]: pgmap v1613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:25:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:11 smithi029 ceph-mon[32610]: pgmap v1613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:25:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:11 smithi077 ceph-mon[37153]: pgmap v1613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:25:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2134.400596 secs 2024-07-27T07:25:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2134.400596 secs 2024-07-27T07:25:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2134.400596 secs 2024-07-27T07:25:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:13 smithi047 ceph-mon[36835]: pgmap v1614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:25:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:13 smithi029 ceph-mon[32610]: pgmap v1614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:25:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:13 smithi077 ceph-mon[37153]: pgmap v1614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:25:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:15 smithi047 ceph-mon[36835]: pgmap v1615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:15 smithi029 ceph-mon[32610]: pgmap v1615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:15 smithi077 ceph-mon[37153]: pgmap v1615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:17 smithi047 ceph-mon[36835]: pgmap v1616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2139.400717 secs 2024-07-27T07:25:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:17 smithi029 ceph-mon[32610]: pgmap v1616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2139.400717 secs 2024-07-27T07:25:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:17 smithi077 ceph-mon[37153]: pgmap v1616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2139.400717 secs 2024-07-27T07:25:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:19 smithi047 ceph-mon[36835]: pgmap v1617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:19 smithi029 ceph-mon[32610]: pgmap v1617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:19 smithi077 ceph-mon[37153]: pgmap v1617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:21 smithi047 ceph-mon[36835]: pgmap v1618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:21 smithi029 ceph-mon[32610]: pgmap v1618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:21 smithi077 ceph-mon[37153]: pgmap v1618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2144.400843 secs 2024-07-27T07:25:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2144.400843 secs 2024-07-27T07:25:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2144.400843 secs 2024-07-27T07:25:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:23 smithi047 ceph-mon[36835]: pgmap v1619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:23 smithi029 ceph-mon[32610]: pgmap v1619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:23 smithi077 ceph-mon[37153]: pgmap v1619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:24.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:24.735+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:25:24.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:24.735+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:25:24.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:24.735+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:25:24.773 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:24.773+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:25:24.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:24.773+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:25:24.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:24.773+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:25:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:25 smithi047 ceph-mon[36835]: pgmap v1620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:25.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:25 smithi029 ceph-mon[32610]: pgmap v1620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:25 smithi077 ceph-mon[37153]: pgmap v1620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:27 smithi047 ceph-mon[36835]: pgmap v1621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2149.400930 secs 2024-07-27T07:25:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:27 smithi029 ceph-mon[32610]: pgmap v1621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2149.400930 secs 2024-07-27T07:25:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:27 smithi077 ceph-mon[37153]: pgmap v1621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2149.400930 secs 2024-07-27T07:25:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:29 smithi047 ceph-mon[36835]: pgmap v1622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:29 smithi029 ceph-mon[32610]: pgmap v1622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:29.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:29 smithi077 ceph-mon[37153]: pgmap v1622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:31 smithi047 ceph-mon[36835]: pgmap v1623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:31 smithi029 ceph-mon[32610]: pgmap v1623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:31 smithi077 ceph-mon[37153]: pgmap v1623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2154.401016 secs 2024-07-27T07:25:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2154.401016 secs 2024-07-27T07:25:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2154.401016 secs 2024-07-27T07:25:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:33 smithi029 ceph-mon[32610]: pgmap v1624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:33 smithi077 ceph-mon[37153]: pgmap v1624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:33 smithi047 ceph-mon[36835]: pgmap v1624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:35 smithi029 ceph-mon[32610]: pgmap v1625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:35 smithi077 ceph-mon[37153]: pgmap v1625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:35 smithi047 ceph-mon[36835]: pgmap v1625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:37 smithi029 ceph-mon[32610]: pgmap v1626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2159.401143 secs 2024-07-27T07:25:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:37 smithi077 ceph-mon[37153]: pgmap v1626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2159.401143 secs 2024-07-27T07:25:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:37 smithi047 ceph-mon[36835]: pgmap v1626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2159.401143 secs 2024-07-27T07:25:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:39 smithi077 ceph-mon[37153]: pgmap v1627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:39 smithi047 ceph-mon[36835]: pgmap v1627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:39 smithi029 ceph-mon[32610]: pgmap v1627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:39.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:39.749+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:25:39.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:39.749+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:25:39.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:39.749+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:25:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:39.773+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:25:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:39.773+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:25:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:39.773+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:25:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:41 smithi047 ceph-mon[36835]: pgmap v1628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:41 smithi029 ceph-mon[32610]: pgmap v1628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:41 smithi077 ceph-mon[37153]: pgmap v1628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2164.401229 secs 2024-07-27T07:25:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2164.401229 secs 2024-07-27T07:25:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2164.401229 secs 2024-07-27T07:25:42.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:42.954+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:25:42.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:42.954+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:25:42.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:42.954+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:25:42.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:42.954+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:25:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:43 smithi047 ceph-mon[36835]: pgmap v1629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:43 smithi029 ceph-mon[32610]: pgmap v1629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:43 smithi077 ceph-mon[37153]: pgmap v1629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:45 smithi047 ceph-mon[36835]: pgmap v1630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:45 smithi029 ceph-mon[32610]: pgmap v1630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:45 smithi077 ceph-mon[37153]: pgmap v1630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:47 smithi047 ceph-mon[36835]: pgmap v1631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2169.401356 secs 2024-07-27T07:25:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:47 smithi029 ceph-mon[32610]: pgmap v1631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2169.401356 secs 2024-07-27T07:25:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:47 smithi077 ceph-mon[37153]: pgmap v1631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2169.401356 secs 2024-07-27T07:25:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:49 smithi047 ceph-mon[36835]: pgmap v1632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:49 smithi029 ceph-mon[32610]: pgmap v1632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:49 smithi077 ceph-mon[37153]: pgmap v1632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:51 smithi047 ceph-mon[36835]: pgmap v1633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:51 smithi029 ceph-mon[32610]: pgmap v1633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:51 smithi077 ceph-mon[37153]: pgmap v1633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:25:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2174.401483 secs 2024-07-27T07:25:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2174.401483 secs 2024-07-27T07:25:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2174.401483 secs 2024-07-27T07:25:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:53 smithi047 ceph-mon[36835]: pgmap v1634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:53 smithi029 ceph-mon[32610]: pgmap v1634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:53 smithi077 ceph-mon[37153]: pgmap v1634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:54.754+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:25:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:54.754+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:25:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:54.754+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:25:54.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:54.788+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:25:54.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:54.788+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:25:54.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:25:54.788+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:25:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:55 smithi047 ceph-mon[36835]: pgmap v1635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:55 smithi029 ceph-mon[32610]: pgmap v1635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:55 smithi077 ceph-mon[37153]: pgmap v1635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:25:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:57 smithi047 ceph-mon[36835]: pgmap v1636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2179.401608 secs 2024-07-27T07:25:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:57 smithi029 ceph-mon[32610]: pgmap v1636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2179.401608 secs 2024-07-27T07:25:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:57 smithi077 ceph-mon[37153]: pgmap v1636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:25:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2179.401608 secs 2024-07-27T07:25:57.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:57.960+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff240065e0 con 0x7eff50104ab0 2024-07-27T07:25:57.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:57.960+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:25:57.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:57.960+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff240065e0 con 0x7eff50104ab0 2024-07-27T07:25:57.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:25:57.960+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:25:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:25:59 smithi029 ceph-mon[32610]: pgmap v1637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:25:59 smithi077 ceph-mon[37153]: pgmap v1637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:25:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:25:59 smithi047 ceph-mon[36835]: pgmap v1637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:26:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:01 smithi029 ceph-mon[32610]: pgmap v1638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:01 smithi077 ceph-mon[37153]: pgmap v1638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:01 smithi047 ceph-mon[36835]: pgmap v1638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2184.401752 secs 2024-07-27T07:26:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:26:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2184.401752 secs 2024-07-27T07:26:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:26:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2184.401752 secs 2024-07-27T07:26:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:26:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:03 smithi077 ceph-mon[37153]: pgmap v1639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:26:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:03 smithi047 ceph-mon[36835]: pgmap v1639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:26:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:03 smithi029 ceph-mon[32610]: pgmap v1639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:26:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:05 smithi047 ceph-mon[36835]: pgmap v1640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:26:05.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:26:05.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:26:05.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:05 smithi029 ceph-mon[32610]: pgmap v1640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:26:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:26:05.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:26:05.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:05 smithi077 ceph-mon[37153]: pgmap v1640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:26:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:06 smithi047 ceph-mon[36835]: pgmap v1641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:26:06.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:06 smithi047 ceph-mon[36835]: pgmap v1642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:26:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:06 smithi029 ceph-mon[32610]: pgmap v1641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:26:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:06 smithi029 ceph-mon[32610]: pgmap v1642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:26:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:06 smithi077 ceph-mon[37153]: pgmap v1641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:26:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:06 smithi077 ceph-mon[37153]: pgmap v1642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:26:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2189.401908 secs 2024-07-27T07:26:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2189.401908 secs 2024-07-27T07:26:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2189.401908 secs 2024-07-27T07:26:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:08 smithi047 ceph-mon[36835]: pgmap v1643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:26:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:08 smithi029 ceph-mon[32610]: pgmap v1643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:26:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:08 smithi077 ceph-mon[37153]: pgmap v1643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:26:09.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:26:09.770+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:26:09.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:26:09.770+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:26:09.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:26:09.770+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:26:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:26:09.801+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:26:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:26:09.801+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:26:09.802 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:26:09.801+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:26:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:10 smithi047 ceph-mon[36835]: pgmap v1644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:26:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:10 smithi029 ceph-mon[32610]: pgmap v1644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:26:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:10 smithi077 ceph-mon[37153]: pgmap v1644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:26:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:12 smithi047 ceph-mon[36835]: pgmap v1645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T07:26:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2194.402031 secs 2024-07-27T07:26:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:12 smithi029 ceph-mon[32610]: pgmap v1645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T07:26:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2194.402031 secs 2024-07-27T07:26:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:12 smithi077 ceph-mon[37153]: pgmap v1645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T07:26:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2194.402031 secs 2024-07-27T07:26:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:14 smithi047 ceph-mon[36835]: pgmap v1646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T07:26:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:14 smithi029 ceph-mon[32610]: pgmap v1646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T07:26:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:14 smithi077 ceph-mon[37153]: pgmap v1646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T07:26:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:16 smithi047 ceph-mon[36835]: pgmap v1647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:26:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:16 smithi029 ceph-mon[32610]: pgmap v1647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:26:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:16 smithi077 ceph-mon[37153]: pgmap v1647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:26:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2199.402157 secs 2024-07-27T07:26:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2199.402157 secs 2024-07-27T07:26:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2199.402157 secs 2024-07-27T07:26:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:18 smithi047 ceph-mon[36835]: pgmap v1648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:26:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:18 smithi029 ceph-mon[32610]: pgmap v1648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:26:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:18 smithi077 ceph-mon[37153]: pgmap v1648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:26:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:20 smithi047 ceph-mon[36835]: pgmap v1649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:26:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:20 smithi029 ceph-mon[32610]: pgmap v1649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:26:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:20 smithi077 ceph-mon[37153]: pgmap v1649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:26:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:22 smithi029 ceph-mon[32610]: pgmap v1650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:26:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2204.402299 secs 2024-07-27T07:26:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:22 smithi077 ceph-mon[37153]: pgmap v1650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:26:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2204.402299 secs 2024-07-27T07:26:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:22 smithi047 ceph-mon[36835]: pgmap v1650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:26:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2204.402299 secs 2024-07-27T07:26:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:24 smithi029 ceph-mon[32610]: pgmap v1651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:26:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:24 smithi077 ceph-mon[37153]: pgmap v1651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:26:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:24 smithi047 ceph-mon[36835]: pgmap v1651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:26:24.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:26:24.782+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:26:24.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:26:24.782+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:26:24.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:26:24.782+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:26:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:26:24.806+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:26:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:26:24.806+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:26:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:26:24.806+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:26:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:26 smithi029 ceph-mon[32610]: pgmap v1652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:26:26.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:26 smithi047 ceph-mon[36835]: pgmap v1652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:26:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:26 smithi077 ceph-mon[37153]: pgmap v1652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:26:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2209.402423 secs 2024-07-27T07:26:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2209.402423 secs 2024-07-27T07:26:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2209.402423 secs 2024-07-27T07:26:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:28 smithi029 ceph-mon[32610]: pgmap v1653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:26:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:28 smithi077 ceph-mon[37153]: pgmap v1653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:26:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:28 smithi047 ceph-mon[36835]: pgmap v1653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:26:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:30 smithi077 ceph-mon[37153]: pgmap v1654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:26:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:30 smithi047 ceph-mon[36835]: pgmap v1654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:26:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:30 smithi029 ceph-mon[32610]: pgmap v1654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:26:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:32 smithi077 ceph-mon[37153]: pgmap v1655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2214.402548 secs 2024-07-27T07:26:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:32 smithi047 ceph-mon[36835]: pgmap v1655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2214.402548 secs 2024-07-27T07:26:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:32 smithi029 ceph-mon[32610]: pgmap v1655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2214.402548 secs 2024-07-27T07:26:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:34 smithi077 ceph-mon[37153]: pgmap v1656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:26:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:34 smithi047 ceph-mon[36835]: pgmap v1656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:26:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:34 smithi029 ceph-mon[32610]: pgmap v1656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:26:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:36 smithi077 ceph-mon[37153]: pgmap v1657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:36 smithi047 ceph-mon[36835]: pgmap v1657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:36 smithi029 ceph-mon[32610]: pgmap v1657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2219.402675 secs 2024-07-27T07:26:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2219.402675 secs 2024-07-27T07:26:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2219.402675 secs 2024-07-27T07:26:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:38 smithi077 ceph-mon[37153]: pgmap v1658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:26:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:38 smithi047 ceph-mon[36835]: pgmap v1658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:26:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:38 smithi029 ceph-mon[32610]: pgmap v1658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:26:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:26:39.797+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:26:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:26:39.797+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:26:39.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:26:39.797+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:26:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:26:39.821+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:26:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:26:39.821+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:26:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:26:39.821+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:26:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:40 smithi077 ceph-mon[37153]: pgmap v1659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:26:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:40 smithi047 ceph-mon[36835]: pgmap v1659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:26:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:40 smithi029 ceph-mon[32610]: pgmap v1659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:26:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:42 smithi077 ceph-mon[37153]: pgmap v1660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:42.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2224.402779 secs 2024-07-27T07:26:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:42 smithi047 ceph-mon[36835]: pgmap v1660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2224.402779 secs 2024-07-27T07:26:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:42 smithi029 ceph-mon[32610]: pgmap v1660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2224.402779 secs 2024-07-27T07:26:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:44 smithi077 ceph-mon[37153]: pgmap v1661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T07:26:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:44 smithi047 ceph-mon[36835]: pgmap v1661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T07:26:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:44 smithi029 ceph-mon[32610]: pgmap v1661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T07:26:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:46 smithi077 ceph-mon[37153]: pgmap v1662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:26:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:46 smithi047 ceph-mon[36835]: pgmap v1662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:26:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:46 smithi029 ceph-mon[32610]: pgmap v1662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:26:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2229.402914 secs 2024-07-27T07:26:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2229.402914 secs 2024-07-27T07:26:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2229.402914 secs 2024-07-27T07:26:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:48 smithi077 ceph-mon[37153]: pgmap v1663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T07:26:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:48 smithi047 ceph-mon[36835]: pgmap v1663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T07:26:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:48 smithi029 ceph-mon[32610]: pgmap v1663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T07:26:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:50 smithi077 ceph-mon[37153]: pgmap v1664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:26:50.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:50 smithi047 ceph-mon[36835]: pgmap v1664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:26:50.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:50 smithi029 ceph-mon[32610]: pgmap v1664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:26:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:52 smithi077 ceph-mon[37153]: pgmap v1665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:26:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2234.403042 secs 2024-07-27T07:26:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:52 smithi047 ceph-mon[36835]: pgmap v1665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:26:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2234.403042 secs 2024-07-27T07:26:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:52 smithi029 ceph-mon[32610]: pgmap v1665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:26:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2234.403042 secs 2024-07-27T07:26:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:54 smithi077 ceph-mon[37153]: pgmap v1666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T07:26:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:54 smithi047 ceph-mon[36835]: pgmap v1666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T07:26:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:54 smithi029 ceph-mon[32610]: pgmap v1666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T07:26:54.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:26:54.812+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:26:54.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:26:54.812+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:26:54.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:26:54.812+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:26:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:26:54.836+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:26:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:26:54.836+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:26:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:26:54.836+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:26:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:56 smithi077 ceph-mon[37153]: pgmap v1667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:56.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:56 smithi047 ceph-mon[36835]: pgmap v1667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:56 smithi029 ceph-mon[32610]: pgmap v1667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:26:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2239.403140 secs 2024-07-27T07:26:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2239.403140 secs 2024-07-27T07:26:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2239.403140 secs 2024-07-27T07:26:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:26:58 smithi077 ceph-mon[37153]: pgmap v1668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:26:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:26:58 smithi047 ceph-mon[36835]: pgmap v1668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:26:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:26:58 smithi029 ceph-mon[32610]: pgmap v1668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:00 smithi077 ceph-mon[37153]: pgmap v1669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:00 smithi047 ceph-mon[36835]: pgmap v1669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:00 smithi029 ceph-mon[32610]: pgmap v1669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:02 smithi077 ceph-mon[37153]: pgmap v1670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2244.403242 secs 2024-07-27T07:27:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:02 smithi047 ceph-mon[36835]: pgmap v1670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2244.403242 secs 2024-07-27T07:27:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:02 smithi029 ceph-mon[32610]: pgmap v1670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2244.403242 secs 2024-07-27T07:27:04.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:04 smithi077 ceph-mon[37153]: pgmap v1671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:04 smithi047 ceph-mon[36835]: pgmap v1671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:04 smithi029 ceph-mon[32610]: pgmap v1671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:27:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:27:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:27:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:06 smithi047 ceph-mon[36835]: pgmap v1672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:27:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:27:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:06 smithi047 ceph-mon[36835]: pgmap v1673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:27:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:06 smithi047 ceph-mon[36835]: pgmap v1674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T07:27:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:06 smithi029 ceph-mon[32610]: pgmap v1672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:27:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:27:06.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:06 smithi029 ceph-mon[32610]: pgmap v1673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:27:06.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:06 smithi029 ceph-mon[32610]: pgmap v1674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T07:27:06.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:06 smithi077 ceph-mon[37153]: pgmap v1672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:27:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:27:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:06 smithi077 ceph-mon[37153]: pgmap v1673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:27:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:06 smithi077 ceph-mon[37153]: pgmap v1674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T07:27:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:27:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2249.403375 secs 2024-07-27T07:27:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2249.403375 secs 2024-07-27T07:27:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2249.403375 secs 2024-07-27T07:27:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:08 smithi077 ceph-mon[37153]: pgmap v1675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:27:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:08 smithi047 ceph-mon[36835]: pgmap v1675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:27:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:08 smithi029 ceph-mon[32610]: pgmap v1675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:27:09.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:27:09.817+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:27:09.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:27:09.817+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:27:09.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:27:09.817+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:27:09.852 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:27:09.852+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:27:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:27:09.852+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:27:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:27:09.852+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:27:11.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:10 smithi047 ceph-mon[36835]: pgmap v1676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:27:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:10 smithi029 ceph-mon[32610]: pgmap v1676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:27:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:10 smithi077 ceph-mon[37153]: pgmap v1676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:27:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2254.403509 secs 2024-07-27T07:27:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:12 smithi047 ceph-mon[36835]: pgmap v1677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T07:27:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2254.403509 secs 2024-07-27T07:27:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:12 smithi029 ceph-mon[32610]: pgmap v1677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T07:27:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2254.403509 secs 2024-07-27T07:27:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:12 smithi077 ceph-mon[37153]: pgmap v1677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T07:27:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:14 smithi047 ceph-mon[36835]: pgmap v1678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:27:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:14 smithi029 ceph-mon[32610]: pgmap v1678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:27:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:14 smithi077 ceph-mon[37153]: pgmap v1678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:27:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:15 smithi047 ceph-mon[36835]: pgmap v1679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:27:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:15 smithi029 ceph-mon[32610]: pgmap v1679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:27:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:15 smithi077 ceph-mon[37153]: pgmap v1679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:27:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2259.403636 secs 2024-07-27T07:27:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2259.403636 secs 2024-07-27T07:27:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2259.403636 secs 2024-07-27T07:27:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:18 smithi077 ceph-mon[37153]: pgmap v1680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:18 smithi047 ceph-mon[36835]: pgmap v1680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:18 smithi029 ceph-mon[32610]: pgmap v1680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:20 smithi047 ceph-mon[36835]: pgmap v1681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:20 smithi029 ceph-mon[32610]: pgmap v1681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:20 smithi077 ceph-mon[37153]: pgmap v1681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2264.403761 secs 2024-07-27T07:27:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:22 smithi047 ceph-mon[36835]: pgmap v1682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2264.403761 secs 2024-07-27T07:27:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:22 smithi029 ceph-mon[32610]: pgmap v1682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2264.403761 secs 2024-07-27T07:27:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:22 smithi077 ceph-mon[37153]: pgmap v1682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:23 smithi077 ceph-mon[37153]: pgmap v1683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:23 smithi047 ceph-mon[36835]: pgmap v1683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:24.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:23 smithi029 ceph-mon[32610]: pgmap v1683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:24.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:27:24.829+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:27:24.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:27:24.829+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:27:24.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:27:24.829+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:27:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:27:24.855+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:27:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:27:24.855+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:27:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:27:24.855+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:27:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:26 smithi047 ceph-mon[36835]: pgmap v1684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:26 smithi029 ceph-mon[32610]: pgmap v1684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:26 smithi077 ceph-mon[37153]: pgmap v1684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2269.403883 secs 2024-07-27T07:27:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2269.403883 secs 2024-07-27T07:27:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2269.403883 secs 2024-07-27T07:27:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:28 smithi047 ceph-mon[36835]: pgmap v1685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:28 smithi029 ceph-mon[32610]: pgmap v1685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:28 smithi077 ceph-mon[37153]: pgmap v1685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:29 smithi047 ceph-mon[36835]: pgmap v1686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:29 smithi029 ceph-mon[32610]: pgmap v1686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:29 smithi077 ceph-mon[37153]: pgmap v1686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2274.404015 secs 2024-07-27T07:27:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:32 smithi047 ceph-mon[36835]: pgmap v1687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2274.404015 secs 2024-07-27T07:27:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:32 smithi029 ceph-mon[32610]: pgmap v1687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2274.404015 secs 2024-07-27T07:27:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:32 smithi077 ceph-mon[37153]: pgmap v1687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:33 smithi077 ceph-mon[37153]: pgmap v1688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:33 smithi047 ceph-mon[36835]: pgmap v1688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:33 smithi029 ceph-mon[32610]: pgmap v1688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:36 smithi047 ceph-mon[36835]: pgmap v1689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:36 smithi029 ceph-mon[32610]: pgmap v1689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:36 smithi077 ceph-mon[37153]: pgmap v1689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2279.404122 secs 2024-07-27T07:27:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2279.404122 secs 2024-07-27T07:27:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2279.404122 secs 2024-07-27T07:27:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:38 smithi047 ceph-mon[36835]: pgmap v1690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:38 smithi029 ceph-mon[32610]: pgmap v1690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:38 smithi077 ceph-mon[37153]: pgmap v1690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:39.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:27:39.844+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:27:39.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:27:39.844+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:27:39.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:27:39.844+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:27:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:27:39.857+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:27:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:27:39.857+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:27:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:27:39.857+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:27:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:39 smithi047 ceph-mon[36835]: pgmap v1691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:39 smithi029 ceph-mon[32610]: pgmap v1691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:39 smithi077 ceph-mon[37153]: pgmap v1691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2284.404253 secs 2024-07-27T07:27:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:42 smithi047 ceph-mon[36835]: pgmap v1692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2284.404253 secs 2024-07-27T07:27:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:42 smithi029 ceph-mon[32610]: pgmap v1692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2284.404253 secs 2024-07-27T07:27:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:42 smithi077 ceph-mon[37153]: pgmap v1692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:43 smithi047 ceph-mon[36835]: pgmap v1693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:43 smithi029 ceph-mon[32610]: pgmap v1693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:43 smithi077 ceph-mon[37153]: pgmap v1693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:45 smithi077 ceph-mon[37153]: pgmap v1694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:45 smithi047 ceph-mon[36835]: pgmap v1694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:45 smithi029 ceph-mon[32610]: pgmap v1694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2289.404377 secs 2024-07-27T07:27:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2289.404377 secs 2024-07-27T07:27:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2289.404377 secs 2024-07-27T07:27:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:48 smithi077 ceph-mon[37153]: pgmap v1695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:48 smithi047 ceph-mon[36835]: pgmap v1695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:48 smithi029 ceph-mon[32610]: pgmap v1695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:49 smithi077 ceph-mon[37153]: pgmap v1696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:49 smithi047 ceph-mon[36835]: pgmap v1696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:49 smithi029 ceph-mon[32610]: pgmap v1696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:51 smithi077 ceph-mon[37153]: pgmap v1697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:51 smithi047 ceph-mon[36835]: pgmap v1697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:51 smithi029 ceph-mon[32610]: pgmap v1697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:27:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2294.404506 secs 2024-07-27T07:27:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2294.404506 secs 2024-07-27T07:27:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2294.404506 secs 2024-07-27T07:27:54.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:53 smithi077 ceph-mon[37153]: pgmap v1698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:53 smithi047 ceph-mon[36835]: pgmap v1698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:53 smithi029 ceph-mon[32610]: pgmap v1698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:27:54.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:27:54.857+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:27:54.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:27:54.857+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:27:54.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:27:54.857+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:27:54.873 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:27:54.872+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:27:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:27:54.872+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:27:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:27:54.872+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:27:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:55 smithi077 ceph-mon[37153]: pgmap v1699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:55 smithi047 ceph-mon[36835]: pgmap v1699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:55 smithi029 ceph-mon[32610]: pgmap v1699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:27:57.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2299.404654 secs 2024-07-27T07:27:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2299.404654 secs 2024-07-27T07:27:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2299.404654 secs 2024-07-27T07:27:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:58 smithi077 ceph-mon[37153]: pgmap v1700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:58 smithi047 ceph-mon[36835]: pgmap v1700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:27:58.539 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:58 smithi029 ceph-mon[32610]: pgmap v1700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:28:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:27:59 smithi077 ceph-mon[37153]: pgmap v1701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:27:59 smithi047 ceph-mon[36835]: pgmap v1701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:27:59 smithi029 ceph-mon[32610]: pgmap v1701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:01 smithi077 ceph-mon[37153]: pgmap v1702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:28:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:01 smithi047 ceph-mon[36835]: pgmap v1702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:28:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:01 smithi029 ceph-mon[32610]: pgmap v1702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:28:03.257 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2304.404781 secs 2024-07-27T07:28:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2304.404781 secs 2024-07-27T07:28:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2304.404781 secs 2024-07-27T07:28:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:03 smithi077 ceph-mon[37153]: pgmap v1703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:03 smithi047 ceph-mon[36835]: pgmap v1703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:03 smithi029 ceph-mon[32610]: pgmap v1703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:05 smithi077 ceph-mon[37153]: pgmap v1704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:28:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:05 smithi047 ceph-mon[36835]: pgmap v1704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:28:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:05 smithi029 ceph-mon[32610]: pgmap v1704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:28:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:28:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:28:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:06 smithi077 ceph-mon[37153]: pgmap v1705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T07:28:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:06 smithi077 ceph-mon[37153]: pgmap v1706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:28:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:28:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:28:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:28:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:06 smithi047 ceph-mon[36835]: pgmap v1705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T07:28:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:06 smithi047 ceph-mon[36835]: pgmap v1706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:28:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:28:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:28:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:28:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:06 smithi029 ceph-mon[32610]: pgmap v1705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T07:28:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:06 smithi029 ceph-mon[32610]: pgmap v1706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:28:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:28:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2309.404908 secs 2024-07-27T07:28:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2309.404908 secs 2024-07-27T07:28:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2309.404908 secs 2024-07-27T07:28:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:09 smithi077 ceph-mon[37153]: pgmap v1707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:28:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:08 smithi047 ceph-mon[36835]: pgmap v1707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:28:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:08 smithi029 ceph-mon[32610]: pgmap v1707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:28:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:09.871+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:28:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:09.871+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:28:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:09.871+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:28:09.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:09.888+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:28:09.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:09.888+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:28:09.889 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:09.888+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:28:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:11 smithi077 ceph-mon[37153]: pgmap v1708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:28:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:11 smithi047 ceph-mon[36835]: pgmap v1708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:28:11.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:11 smithi029 ceph-mon[32610]: pgmap v1708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:28:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2314.405036 secs 2024-07-27T07:28:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2314.405036 secs 2024-07-27T07:28:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2314.405036 secs 2024-07-27T07:28:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:13 smithi077 ceph-mon[37153]: pgmap v1709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:28:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:13 smithi047 ceph-mon[36835]: pgmap v1709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:28:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:13 smithi029 ceph-mon[32610]: pgmap v1709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:28:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:15 smithi077 ceph-mon[37153]: pgmap v1710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:28:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:15 smithi047 ceph-mon[36835]: pgmap v1710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:28:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:15 smithi029 ceph-mon[32610]: pgmap v1710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:28:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:17 smithi077 ceph-mon[37153]: pgmap v1711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:28:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2319.405165 secs 2024-07-27T07:28:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:17 smithi047 ceph-mon[36835]: pgmap v1711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:28:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2319.405165 secs 2024-07-27T07:28:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:17 smithi029 ceph-mon[32610]: pgmap v1711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:28:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2319.405165 secs 2024-07-27T07:28:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:19 smithi077 ceph-mon[37153]: pgmap v1712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:19 smithi047 ceph-mon[36835]: pgmap v1712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:19 smithi029 ceph-mon[32610]: pgmap v1712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:21 smithi077 ceph-mon[37153]: pgmap v1713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:21 smithi047 ceph-mon[36835]: pgmap v1713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:21 smithi029 ceph-mon[32610]: pgmap v1713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2324.405290 secs 2024-07-27T07:28:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2324.405290 secs 2024-07-27T07:28:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2324.405290 secs 2024-07-27T07:28:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:23 smithi077 ceph-mon[37153]: pgmap v1714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:28:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:23 smithi047 ceph-mon[36835]: pgmap v1714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:28:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:23 smithi029 ceph-mon[32610]: pgmap v1714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:28:24.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:24.878+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:28:24.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:24.878+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:28:24.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:24.878+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:28:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:24.889+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:28:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:24.889+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:28:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:24.889+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:28:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:25 smithi077 ceph-mon[37153]: pgmap v1715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:28:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:25 smithi047 ceph-mon[36835]: pgmap v1715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:28:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:25 smithi029 ceph-mon[32610]: pgmap v1715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:28:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:27 smithi077 ceph-mon[37153]: pgmap v1716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:28:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2329.405415 secs 2024-07-27T07:28:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:27 smithi047 ceph-mon[36835]: pgmap v1716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:28:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2329.405415 secs 2024-07-27T07:28:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:27 smithi029 ceph-mon[32610]: pgmap v1716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:28:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2329.405415 secs 2024-07-27T07:28:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:29 smithi077 ceph-mon[37153]: pgmap v1717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:29 smithi047 ceph-mon[36835]: pgmap v1717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:29 smithi029 ceph-mon[32610]: pgmap v1717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:31 smithi077 ceph-mon[37153]: pgmap v1718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:31 smithi047 ceph-mon[36835]: pgmap v1718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:31 smithi029 ceph-mon[32610]: pgmap v1718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2334.405577 secs 2024-07-27T07:28:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2334.405577 secs 2024-07-27T07:28:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2334.405577 secs 2024-07-27T07:28:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:33 smithi077 ceph-mon[37153]: pgmap v1719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:28:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:33 smithi047 ceph-mon[36835]: pgmap v1719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:28:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:33 smithi029 ceph-mon[32610]: pgmap v1719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:28:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:35 smithi077 ceph-mon[37153]: pgmap v1720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:28:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:35 smithi047 ceph-mon[36835]: pgmap v1720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:28:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:35 smithi029 ceph-mon[32610]: pgmap v1720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:28:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:37 smithi077 ceph-mon[37153]: pgmap v1721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:28:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2339.405726 secs 2024-07-27T07:28:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:37 smithi047 ceph-mon[36835]: pgmap v1721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:28:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2339.405726 secs 2024-07-27T07:28:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:37 smithi029 ceph-mon[32610]: pgmap v1721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:28:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2339.405726 secs 2024-07-27T07:28:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:39 smithi077 ceph-mon[37153]: pgmap v1722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:39.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:39.893+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:28:39.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:39.893+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:28:39.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:39.893+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:28:39.895 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:39 smithi029 ceph-mon[32610]: pgmap v1722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:39.905+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:28:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:39.905+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:28:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:39.905+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:28:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:39 smithi047 ceph-mon[36835]: pgmap v1722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:41 smithi077 ceph-mon[37153]: pgmap v1723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:41 smithi047 ceph-mon[36835]: pgmap v1723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:41 smithi029 ceph-mon[32610]: pgmap v1723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2344.405857 secs 2024-07-27T07:28:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2344.405857 secs 2024-07-27T07:28:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2344.405857 secs 2024-07-27T07:28:42.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:42.981+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:28:42.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:42.981+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:28:42.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:42.982+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:28:42.984 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:42.982+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:28:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:43 smithi077 ceph-mon[37153]: pgmap v1724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:28:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:43 smithi047 ceph-mon[36835]: pgmap v1724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:28:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:43 smithi029 ceph-mon[32610]: pgmap v1724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:28:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:45 smithi077 ceph-mon[37153]: pgmap v1725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:28:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:45 smithi047 ceph-mon[36835]: pgmap v1725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:28:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:45 smithi029 ceph-mon[32610]: pgmap v1725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:28:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:47 smithi077 ceph-mon[37153]: pgmap v1726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:28:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2349.405989 secs 2024-07-27T07:28:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:47 smithi047 ceph-mon[36835]: pgmap v1726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:28:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2349.405989 secs 2024-07-27T07:28:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:47 smithi029 ceph-mon[32610]: pgmap v1726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:28:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2349.405989 secs 2024-07-27T07:28:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:49 smithi077 ceph-mon[37153]: pgmap v1727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:49 smithi047 ceph-mon[36835]: pgmap v1727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:49 smithi029 ceph-mon[32610]: pgmap v1727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:51 smithi077 ceph-mon[37153]: pgmap v1728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:51 smithi047 ceph-mon[36835]: pgmap v1728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:51 smithi029 ceph-mon[32610]: pgmap v1728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:28:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2354.406122 secs 2024-07-27T07:28:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2354.406122 secs 2024-07-27T07:28:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2354.406122 secs 2024-07-27T07:28:53.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:53 smithi077 ceph-mon[37153]: pgmap v1729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:28:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:53 smithi047 ceph-mon[36835]: pgmap v1729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:28:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:53 smithi029 ceph-mon[32610]: pgmap v1729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:28:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:54.909+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:28:54.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:54.909+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:28:54.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:54.909+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:28:54.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:54.921+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:28:54.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:54.921+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:28:54.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:28:54.921+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:28:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:55 smithi077 ceph-mon[37153]: pgmap v1730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:28:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:55 smithi047 ceph-mon[36835]: pgmap v1730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:28:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:55 smithi029 ceph-mon[32610]: pgmap v1730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:28:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:57 smithi077 ceph-mon[37153]: pgmap v1731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:28:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2359.406281 secs 2024-07-27T07:28:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:57 smithi047 ceph-mon[36835]: pgmap v1731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:28:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2359.406281 secs 2024-07-27T07:28:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:57 smithi029 ceph-mon[32610]: pgmap v1731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:28:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2359.406281 secs 2024-07-27T07:28:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:57.987+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:28:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:57.987+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff240065b0 con 0x7eff50104ab0 2024-07-27T07:28:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:57.987+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff240065b0 con 0x7eff50104ab0 2024-07-27T07:28:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:28:57.987+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:28:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:28:59 smithi077 ceph-mon[37153]: pgmap v1732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:28:59 smithi047 ceph-mon[36835]: pgmap v1732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:28:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:28:59 smithi029 ceph-mon[32610]: pgmap v1732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:01 smithi077 ceph-mon[37153]: pgmap v1733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:01 smithi047 ceph-mon[36835]: pgmap v1733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:01 smithi029 ceph-mon[32610]: pgmap v1733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2364.406440 secs 2024-07-27T07:29:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2364.406440 secs 2024-07-27T07:29:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2364.406440 secs 2024-07-27T07:29:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:03 smithi077 ceph-mon[37153]: pgmap v1734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:29:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:03 smithi047 ceph-mon[36835]: pgmap v1734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:29:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:03 smithi029 ceph-mon[32610]: pgmap v1734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:29:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:05 smithi077 ceph-mon[37153]: pgmap v1735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:29:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:05 smithi047 ceph-mon[36835]: pgmap v1735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:29:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:05 smithi029 ceph-mon[32610]: pgmap v1735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:29:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:29:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:29:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:29:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:07 smithi077 ceph-mon[37153]: pgmap v1736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:29:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2369.406538 secs 2024-07-27T07:29:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:29:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:29:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:29:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:07 smithi047 ceph-mon[36835]: pgmap v1736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:29:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2369.406538 secs 2024-07-27T07:29:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:29:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:29:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:29:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:07 smithi029 ceph-mon[32610]: pgmap v1736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:29:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2369.406538 secs 2024-07-27T07:29:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:29:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:29:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:29:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:08 smithi047 ceph-mon[36835]: pgmap v1737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:29:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:08 smithi047 ceph-mon[36835]: pgmap v1738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:29:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:08 smithi029 ceph-mon[32610]: pgmap v1737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:29:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:08 smithi029 ceph-mon[32610]: pgmap v1738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:29:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:08 smithi077 ceph-mon[37153]: pgmap v1737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:29:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:08 smithi077 ceph-mon[37153]: pgmap v1738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:29:09.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:29:09.924+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:29:09.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:29:09.924+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:29:09.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:29:09.924+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:29:09.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:29:09.934+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:29:09.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:29:09.934+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:29:09.935 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:29:09.934+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:29:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:10 smithi047 ceph-mon[36835]: pgmap v1739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:29:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:10 smithi029 ceph-mon[32610]: pgmap v1739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:29:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:10 smithi077 ceph-mon[37153]: pgmap v1739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:29:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:12 smithi047 ceph-mon[36835]: pgmap v1740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:29:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2374.406676 secs 2024-07-27T07:29:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:12 smithi029 ceph-mon[32610]: pgmap v1740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:29:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2374.406676 secs 2024-07-27T07:29:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:12 smithi077 ceph-mon[37153]: pgmap v1740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:29:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2374.406676 secs 2024-07-27T07:29:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:14 smithi047 ceph-mon[36835]: pgmap v1741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:29:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:14 smithi029 ceph-mon[32610]: pgmap v1741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:29:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:14 smithi077 ceph-mon[37153]: pgmap v1741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:29:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:16 smithi047 ceph-mon[36835]: pgmap v1742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:29:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:16 smithi029 ceph-mon[32610]: pgmap v1742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:29:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:16 smithi077 ceph-mon[37153]: pgmap v1742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:29:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2379.406867 secs 2024-07-27T07:29:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2379.406867 secs 2024-07-27T07:29:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2379.406867 secs 2024-07-27T07:29:18.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:18 smithi029 ceph-mon[32610]: pgmap v1743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:29:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:18 smithi047 ceph-mon[36835]: pgmap v1743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:29:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:18 smithi077 ceph-mon[37153]: pgmap v1743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:29:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:20 smithi047 ceph-mon[36835]: pgmap v1744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:20 smithi029 ceph-mon[32610]: pgmap v1744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:20 smithi077 ceph-mon[37153]: pgmap v1744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:22 smithi047 ceph-mon[36835]: pgmap v1745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2384.406917 secs 2024-07-27T07:29:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:22 smithi029 ceph-mon[32610]: pgmap v1745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2384.406917 secs 2024-07-27T07:29:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:22 smithi077 ceph-mon[37153]: pgmap v1745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2384.406917 secs 2024-07-27T07:29:24.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:29:24.939+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:29:24.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:29:24.939+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:29:24.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:29:24.939+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:29:24.941 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:24 smithi029 ceph-mon[32610]: pgmap v1746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:29:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:24 smithi047 ceph-mon[36835]: pgmap v1746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:29:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:29:24.948+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:29:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:29:24.948+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:29:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:29:24.948+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:29:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:24 smithi077 ceph-mon[37153]: pgmap v1746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:29:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:26 smithi047 ceph-mon[36835]: pgmap v1747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:26 smithi029 ceph-mon[32610]: pgmap v1747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:26 smithi077 ceph-mon[37153]: pgmap v1747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2389.407052 secs 2024-07-27T07:29:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2389.407052 secs 2024-07-27T07:29:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2389.407052 secs 2024-07-27T07:29:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:28 smithi047 ceph-mon[36835]: pgmap v1748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:29:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:28 smithi029 ceph-mon[32610]: pgmap v1748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:29:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:28 smithi077 ceph-mon[37153]: pgmap v1748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:29:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:30 smithi047 ceph-mon[36835]: pgmap v1749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:30 smithi029 ceph-mon[32610]: pgmap v1749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:30 smithi077 ceph-mon[37153]: pgmap v1749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:32 smithi047 ceph-mon[36835]: pgmap v1750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2394.407198 secs 2024-07-27T07:29:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:32 smithi029 ceph-mon[32610]: pgmap v1750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2394.407198 secs 2024-07-27T07:29:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:32 smithi077 ceph-mon[37153]: pgmap v1750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2394.407198 secs 2024-07-27T07:29:34.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:34 smithi077 ceph-mon[37153]: pgmap v1751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:29:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:34 smithi047 ceph-mon[36835]: pgmap v1751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:29:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:34 smithi029 ceph-mon[32610]: pgmap v1751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:29:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:36 smithi047 ceph-mon[36835]: pgmap v1752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:29:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:36 smithi029 ceph-mon[32610]: pgmap v1752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:29:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:36 smithi077 ceph-mon[37153]: pgmap v1752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:29:37.951 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2399.407286 secs 2024-07-27T07:29:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2399.407286 secs 2024-07-27T07:29:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2399.407286 secs 2024-07-27T07:29:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:38 smithi047 ceph-mon[36835]: pgmap v1753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:29:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:38 smithi029 ceph-mon[32610]: pgmap v1753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:29:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:38 smithi077 ceph-mon[37153]: pgmap v1753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:29:39.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:29:39.941+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:29:39.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:29:39.941+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:29:39.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:29:39.941+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:29:39.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:29:39.950+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:29:39.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:29:39.950+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:29:39.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:29:39.950+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:29:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:40 smithi047 ceph-mon[36835]: pgmap v1754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:40 smithi029 ceph-mon[32610]: pgmap v1754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:40 smithi077 ceph-mon[37153]: pgmap v1754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:42 smithi047 ceph-mon[36835]: pgmap v1755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2404.407355 secs 2024-07-27T07:29:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:42 smithi029 ceph-mon[32610]: pgmap v1755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2404.407355 secs 2024-07-27T07:29:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:42 smithi077 ceph-mon[37153]: pgmap v1755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2404.407355 secs 2024-07-27T07:29:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:44 smithi047 ceph-mon[36835]: pgmap v1756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:29:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:44 smithi029 ceph-mon[32610]: pgmap v1756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:29:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:44 smithi077 ceph-mon[37153]: pgmap v1756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:29:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:46 smithi047 ceph-mon[36835]: pgmap v1757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:46 smithi029 ceph-mon[32610]: pgmap v1757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:46 smithi077 ceph-mon[37153]: pgmap v1757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2409.407458 secs 2024-07-27T07:29:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2409.407458 secs 2024-07-27T07:29:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2409.407458 secs 2024-07-27T07:29:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:48 smithi047 ceph-mon[36835]: pgmap v1758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:29:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:48 smithi029 ceph-mon[32610]: pgmap v1758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:29:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:48 smithi077 ceph-mon[37153]: pgmap v1758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:29:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:50 smithi029 ceph-mon[32610]: pgmap v1759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:50 smithi077 ceph-mon[37153]: pgmap v1759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:50 smithi047 ceph-mon[36835]: pgmap v1759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:29:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:52 smithi029 ceph-mon[32610]: pgmap v1760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2414.407577 secs 2024-07-27T07:29:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:52 smithi077 ceph-mon[37153]: pgmap v1760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2414.407577 secs 2024-07-27T07:29:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:52 smithi047 ceph-mon[36835]: pgmap v1760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2414.407577 secs 2024-07-27T07:29:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:29:54.957+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:29:54.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:29:54.957+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:29:54.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:29:54.957+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:29:54.959 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:54 smithi029 ceph-mon[32610]: pgmap v1761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:29:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:29:54.965+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:29:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:29:54.965+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:29:54.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:29:54.965+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:29:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:54 smithi077 ceph-mon[37153]: pgmap v1761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:29:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:54 smithi047 ceph-mon[36835]: pgmap v1761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:29:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:56 smithi029 ceph-mon[32610]: pgmap v1762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:56 smithi077 ceph-mon[37153]: pgmap v1762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:56 smithi047 ceph-mon[36835]: pgmap v1762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:29:58.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2419.407709 secs 2024-07-27T07:29:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2419.407709 secs 2024-07-27T07:29:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2419.407709 secs 2024-07-27T07:29:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:29:58 smithi077 ceph-mon[37153]: pgmap v1763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:29:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:29:58 smithi047 ceph-mon[36835]: pgmap v1763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:29:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:29:58 smithi029 ceph-mon[32610]: pgmap v1763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:30:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:00 smithi077 ceph-mon[37153]: pgmap v1764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:30:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:00 smithi047 ceph-mon[36835]: pgmap v1764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:30:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:00 smithi029 ceph-mon[32610]: pgmap v1764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:30:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:02 smithi077 ceph-mon[37153]: pgmap v1765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2424.407840 secs 2024-07-27T07:30:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:02 smithi047 ceph-mon[36835]: pgmap v1765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2424.407840 secs 2024-07-27T07:30:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:02 smithi029 ceph-mon[32610]: pgmap v1765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2424.407840 secs 2024-07-27T07:30:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:04 smithi077 ceph-mon[37153]: pgmap v1766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:30:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:04 smithi047 ceph-mon[36835]: pgmap v1766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:30:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:04 smithi029 ceph-mon[32610]: pgmap v1766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:30:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:06 smithi077 ceph-mon[37153]: pgmap v1767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:06 smithi047 ceph-mon[36835]: pgmap v1767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:07.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:06 smithi029 ceph-mon[32610]: pgmap v1767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2429.407910 secs 2024-07-27T07:30:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:30:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:30:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:30:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:30:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2429.407910 secs 2024-07-27T07:30:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:30:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:30:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:30:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:30:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2429.407910 secs 2024-07-27T07:30:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:30:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:30:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:30:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:30:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:08 smithi077 ceph-mon[37153]: pgmap v1768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:30:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:08 smithi077 ceph-mon[37153]: pgmap v1769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T07:30:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:08 smithi077 ceph-mon[37153]: pgmap v1770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:30:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:08 smithi047 ceph-mon[36835]: pgmap v1768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:30:09.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:08 smithi047 ceph-mon[36835]: pgmap v1769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T07:30:09.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:08 smithi047 ceph-mon[36835]: pgmap v1770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:30:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:08 smithi029 ceph-mon[32610]: pgmap v1768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:30:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:08 smithi029 ceph-mon[32610]: pgmap v1769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T07:30:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:08 smithi029 ceph-mon[32610]: pgmap v1770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:30:09.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:30:09.969+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:30:09.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:30:09.969+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:30:09.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:30:09.969+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:30:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:09.981+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:30:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:09.981+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:30:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:09.981+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:30:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:10 smithi077 ceph-mon[37153]: pgmap v1771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:30:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:10 smithi047 ceph-mon[36835]: pgmap v1771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:30:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:10 smithi029 ceph-mon[32610]: pgmap v1771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:30:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:12 smithi077 ceph-mon[37153]: pgmap v1772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:30:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2434.407998 secs 2024-07-27T07:30:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:12 smithi047 ceph-mon[36835]: pgmap v1772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:30:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2434.407998 secs 2024-07-27T07:30:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:12 smithi029 ceph-mon[32610]: pgmap v1772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:30:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2434.407998 secs 2024-07-27T07:30:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:14 smithi077 ceph-mon[37153]: pgmap v1773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:30:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:14 smithi047 ceph-mon[36835]: pgmap v1773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:30:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:14 smithi029 ceph-mon[32610]: pgmap v1773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:30:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:16 smithi077 ceph-mon[37153]: pgmap v1774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:30:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:16 smithi047 ceph-mon[36835]: pgmap v1774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:30:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:16 smithi029 ceph-mon[32610]: pgmap v1774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:30:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2439.408139 secs 2024-07-27T07:30:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2439.408139 secs 2024-07-27T07:30:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2439.408139 secs 2024-07-27T07:30:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:18 smithi077 ceph-mon[37153]: pgmap v1775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:30:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:18 smithi047 ceph-mon[36835]: pgmap v1775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:30:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:18 smithi029 ceph-mon[32610]: pgmap v1775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:30:21.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:20 smithi029 ceph-mon[32610]: pgmap v1776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:20 smithi077 ceph-mon[37153]: pgmap v1776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:20 smithi047 ceph-mon[36835]: pgmap v1776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:22 smithi077 ceph-mon[37153]: pgmap v1777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:30:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2444.408267 secs 2024-07-27T07:30:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:22 smithi047 ceph-mon[36835]: pgmap v1777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:30:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2444.408267 secs 2024-07-27T07:30:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:22 smithi029 ceph-mon[32610]: pgmap v1777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:30:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2444.408267 secs 2024-07-27T07:30:24.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:30:24.983+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:30:24.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:30:24.983+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:30:24.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:30:24.983+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:30:24.989 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:24.988+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:30:24.989 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:24.988+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:30:24.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:24.988+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:30:25.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:24 smithi029 ceph-mon[32610]: pgmap v1778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:24 smithi077 ceph-mon[37153]: pgmap v1778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:24 smithi047 ceph-mon[36835]: pgmap v1778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:26 smithi077 ceph-mon[37153]: pgmap v1779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:26 smithi047 ceph-mon[36835]: pgmap v1779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:26 smithi029 ceph-mon[32610]: pgmap v1779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:28.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2449.408396 secs 2024-07-27T07:30:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2449.408396 secs 2024-07-27T07:30:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2449.408396 secs 2024-07-27T07:30:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:28 smithi077 ceph-mon[37153]: pgmap v1780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:30:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:28 smithi047 ceph-mon[36835]: pgmap v1780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:30:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:28 smithi029 ceph-mon[32610]: pgmap v1780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:30:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:30 smithi077 ceph-mon[37153]: pgmap v1781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:30 smithi047 ceph-mon[36835]: pgmap v1781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:30 smithi029 ceph-mon[32610]: pgmap v1781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:32 smithi077 ceph-mon[37153]: pgmap v1782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:30:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2454.408519 secs 2024-07-27T07:30:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:32 smithi047 ceph-mon[36835]: pgmap v1782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:30:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2454.408519 secs 2024-07-27T07:30:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:32 smithi029 ceph-mon[32610]: pgmap v1782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:30:33.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2454.408519 secs 2024-07-27T07:30:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:34 smithi077 ceph-mon[37153]: pgmap v1783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:34 smithi047 ceph-mon[36835]: pgmap v1783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:34 smithi029 ceph-mon[32610]: pgmap v1783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:36 smithi077 ceph-mon[37153]: pgmap v1784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:36 smithi047 ceph-mon[36835]: pgmap v1784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:36 smithi029 ceph-mon[32610]: pgmap v1784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2459.408609 secs 2024-07-27T07:30:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2459.408609 secs 2024-07-27T07:30:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2459.408609 secs 2024-07-27T07:30:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:38 smithi077 ceph-mon[37153]: pgmap v1785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:30:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:38 smithi047 ceph-mon[36835]: pgmap v1785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:30:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:38 smithi029 ceph-mon[32610]: pgmap v1785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:30:39.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:39.989+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:30:39.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:39.989+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:30:39.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:39.989+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:30:40.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:30:39.999+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:30:40.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:30:39.999+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:30:40.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:30:39.999+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:30:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:40 smithi077 ceph-mon[37153]: pgmap v1786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:40 smithi047 ceph-mon[36835]: pgmap v1786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:40 smithi029 ceph-mon[32610]: pgmap v1786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:42 smithi077 ceph-mon[37153]: pgmap v1787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:30:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2464.408711 secs 2024-07-27T07:30:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:42 smithi047 ceph-mon[36835]: pgmap v1787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:30:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2464.408711 secs 2024-07-27T07:30:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:42 smithi029 ceph-mon[32610]: pgmap v1787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:30:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2464.408711 secs 2024-07-27T07:30:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:44 smithi077 ceph-mon[37153]: pgmap v1788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:44 smithi047 ceph-mon[36835]: pgmap v1788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:44 smithi029 ceph-mon[32610]: pgmap v1788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:46 smithi077 ceph-mon[37153]: pgmap v1789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:46 smithi047 ceph-mon[36835]: pgmap v1789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:46 smithi029 ceph-mon[32610]: pgmap v1789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2469.408830 secs 2024-07-27T07:30:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2469.408830 secs 2024-07-27T07:30:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2469.408830 secs 2024-07-27T07:30:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:48 smithi077 ceph-mon[37153]: pgmap v1790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:30:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:48 smithi047 ceph-mon[36835]: pgmap v1790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:30:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:48 smithi029 ceph-mon[32610]: pgmap v1790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:30:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:50 smithi077 ceph-mon[37153]: pgmap v1791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:50 smithi047 ceph-mon[36835]: pgmap v1791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:50 smithi029 ceph-mon[32610]: pgmap v1791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:30:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:52 smithi077 ceph-mon[37153]: pgmap v1792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:30:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2474.408911 secs 2024-07-27T07:30:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:52 smithi047 ceph-mon[36835]: pgmap v1792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:30:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2474.408911 secs 2024-07-27T07:30:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:52 smithi029 ceph-mon[32610]: pgmap v1792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:30:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2474.408911 secs 2024-07-27T07:30:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:55.002+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:30:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:55.002+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:30:55.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:55.002+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:30:55.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:30:55.014+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:30:55.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:30:55.014+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:30:55.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:30:55.014+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:30:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:54 smithi077 ceph-mon[37153]: pgmap v1793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:55.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:54 smithi047 ceph-mon[36835]: pgmap v1793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:54 smithi029 ceph-mon[32610]: pgmap v1793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:30:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:56 smithi077 ceph-mon[37153]: pgmap v1794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:56 smithi047 ceph-mon[36835]: pgmap v1794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:56 smithi029 ceph-mon[32610]: pgmap v1794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:30:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:58.002+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:30:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:58.002+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:30:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:58.002+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:30:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:30:58.002+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:30:58.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2479.409037 secs 2024-07-27T07:30:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2479.409037 secs 2024-07-27T07:30:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2479.409037 secs 2024-07-27T07:30:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:30:58 smithi077 ceph-mon[37153]: pgmap v1795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:30:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:30:58 smithi047 ceph-mon[36835]: pgmap v1795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:30:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:30:58 smithi029 ceph-mon[32610]: pgmap v1795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:31:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:00 smithi077 ceph-mon[37153]: pgmap v1796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:31:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:00 smithi047 ceph-mon[36835]: pgmap v1796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:31:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:00 smithi029 ceph-mon[32610]: pgmap v1796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:31:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:02 smithi077 ceph-mon[37153]: pgmap v1797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:31:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2484.409120 secs 2024-07-27T07:31:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:02 smithi047 ceph-mon[36835]: pgmap v1797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:31:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2484.409120 secs 2024-07-27T07:31:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:02 smithi029 ceph-mon[32610]: pgmap v1797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:31:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2484.409120 secs 2024-07-27T07:31:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:04 smithi077 ceph-mon[37153]: pgmap v1798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:31:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:04 smithi047 ceph-mon[36835]: pgmap v1798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:31:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:04 smithi029 ceph-mon[32610]: pgmap v1798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:31:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:06 smithi077 ceph-mon[37153]: pgmap v1799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:06 smithi047 ceph-mon[36835]: pgmap v1799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:06 smithi029 ceph-mon[32610]: pgmap v1799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2489.409244 secs 2024-07-27T07:31:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:31:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2489.409244 secs 2024-07-27T07:31:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:31:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2489.409244 secs 2024-07-27T07:31:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:31:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:08 smithi077 ceph-mon[37153]: pgmap v1800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:31:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:31:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:31:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:31:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:08 smithi047 ceph-mon[36835]: pgmap v1800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:31:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:31:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:31:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:31:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:08 smithi029 ceph-mon[32610]: pgmap v1800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:31:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:31:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:31:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:31:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:10.015+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:31:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:10.015+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:31:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:10.015+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:31:10.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:10.030+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:31:10.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:10.030+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:31:10.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:10.030+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:31:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:09 smithi047 ceph-mon[36835]: pgmap v1801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:31:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:09 smithi047 ceph-mon[36835]: pgmap v1802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:31:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:09 smithi029 ceph-mon[32610]: pgmap v1801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:31:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:09 smithi029 ceph-mon[32610]: pgmap v1802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:31:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:09 smithi077 ceph-mon[37153]: pgmap v1801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:31:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:09 smithi077 ceph-mon[37153]: pgmap v1802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:31:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:11 smithi047 ceph-mon[36835]: pgmap v1803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:31:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:11 smithi029 ceph-mon[32610]: pgmap v1803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:31:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:11 smithi077 ceph-mon[37153]: pgmap v1803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:31:13.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:13.005+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:31:13.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:13.005+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff240065b0 con 0x7eff50104ab0 2024-07-27T07:31:13.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:13.006+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff240065b0 con 0x7eff50104ab0 2024-07-27T07:31:13.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:13.006+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:31:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2494.409342 secs 2024-07-27T07:31:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2494.409342 secs 2024-07-27T07:31:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2494.409342 secs 2024-07-27T07:31:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:13 smithi047 ceph-mon[36835]: pgmap v1804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:31:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:13 smithi029 ceph-mon[32610]: pgmap v1804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:31:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:13 smithi077 ceph-mon[37153]: pgmap v1804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:31:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:15 smithi047 ceph-mon[36835]: pgmap v1805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:31:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:15 smithi029 ceph-mon[32610]: pgmap v1805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:31:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:15 smithi077 ceph-mon[37153]: pgmap v1805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:31:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:17 smithi047 ceph-mon[36835]: pgmap v1806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:31:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2499.409466 secs 2024-07-27T07:31:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:17 smithi029 ceph-mon[32610]: pgmap v1806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:31:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2499.409466 secs 2024-07-27T07:31:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:17 smithi077 ceph-mon[37153]: pgmap v1806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:31:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2499.409466 secs 2024-07-27T07:31:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:19 smithi047 ceph-mon[36835]: pgmap v1807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:31:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:19 smithi029 ceph-mon[32610]: pgmap v1807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:31:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:19 smithi077 ceph-mon[37153]: pgmap v1807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:31:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:21 smithi047 ceph-mon[36835]: pgmap v1808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:21 smithi029 ceph-mon[32610]: pgmap v1808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:21 smithi077 ceph-mon[37153]: pgmap v1808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2504.409591 secs 2024-07-27T07:31:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2504.409591 secs 2024-07-27T07:31:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2504.409591 secs 2024-07-27T07:31:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:23 smithi047 ceph-mon[36835]: pgmap v1809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:31:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:23 smithi029 ceph-mon[32610]: pgmap v1809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:31:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:23 smithi077 ceph-mon[37153]: pgmap v1809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:31:25.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:25.029+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:31:25.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:25.029+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:31:25.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:25.029+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:31:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:25.044+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:31:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:25.044+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:31:25.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:25.044+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:31:26.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:25 smithi047 ceph-mon[36835]: pgmap v1810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:31:26.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:25 smithi029 ceph-mon[32610]: pgmap v1810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:31:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:25 smithi077 ceph-mon[37153]: pgmap v1810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:31:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:27 smithi047 ceph-mon[36835]: pgmap v1811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:31:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2509.409712 secs 2024-07-27T07:31:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:27 smithi029 ceph-mon[32610]: pgmap v1811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:31:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2509.409712 secs 2024-07-27T07:31:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:27 smithi077 ceph-mon[37153]: pgmap v1811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:31:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2509.409712 secs 2024-07-27T07:31:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:29 smithi047 ceph-mon[36835]: pgmap v1812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:31:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:29 smithi029 ceph-mon[32610]: pgmap v1812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:31:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:29 smithi077 ceph-mon[37153]: pgmap v1812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:31:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:31 smithi047 ceph-mon[36835]: pgmap v1813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:31:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:31 smithi029 ceph-mon[32610]: pgmap v1813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:31:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:31 smithi077 ceph-mon[37153]: pgmap v1813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:31:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2514.409840 secs 2024-07-27T07:31:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2514.409840 secs 2024-07-27T07:31:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2514.409840 secs 2024-07-27T07:31:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:33 smithi047 ceph-mon[36835]: pgmap v1814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:31:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:33 smithi029 ceph-mon[32610]: pgmap v1814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:31:34.257 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:33 smithi077 ceph-mon[37153]: pgmap v1814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:31:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:35 smithi047 ceph-mon[36835]: pgmap v1815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:31:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:35 smithi029 ceph-mon[32610]: pgmap v1815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:31:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:35 smithi077 ceph-mon[37153]: pgmap v1815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:31:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:37 smithi047 ceph-mon[36835]: pgmap v1816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2519.409963 secs 2024-07-27T07:31:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:37 smithi029 ceph-mon[32610]: pgmap v1816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2519.409963 secs 2024-07-27T07:31:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:37 smithi077 ceph-mon[37153]: pgmap v1816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2519.409963 secs 2024-07-27T07:31:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:40.045+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:31:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:40.045+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:31:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:40.045+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:31:40.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:40.060+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:31:40.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:40.060+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:31:40.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:40.060+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:31:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:39 smithi047 ceph-mon[36835]: pgmap v1817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:31:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:39 smithi029 ceph-mon[32610]: pgmap v1817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:31:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:39 smithi077 ceph-mon[37153]: pgmap v1817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:31:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:41 smithi047 ceph-mon[36835]: pgmap v1818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:41 smithi029 ceph-mon[32610]: pgmap v1818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:41 smithi077 ceph-mon[37153]: pgmap v1818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:43.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:43.010+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:31:43.011 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:43.010+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:31:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:43.010+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:31:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:43.010+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:31:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2524.410065 secs 2024-07-27T07:31:43.215 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2524.410065 secs 2024-07-27T07:31:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2524.410065 secs 2024-07-27T07:31:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:43 smithi047 ceph-mon[36835]: pgmap v1819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:31:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:43 smithi029 ceph-mon[32610]: pgmap v1819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:31:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:43 smithi077 ceph-mon[37153]: pgmap v1819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:31:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:45 smithi047 ceph-mon[36835]: pgmap v1820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:31:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:45 smithi029 ceph-mon[32610]: pgmap v1820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:31:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:45 smithi077 ceph-mon[37153]: pgmap v1820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:31:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:47 smithi047 ceph-mon[36835]: pgmap v1821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2529.410188 secs 2024-07-27T07:31:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:47 smithi029 ceph-mon[32610]: pgmap v1821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2529.410188 secs 2024-07-27T07:31:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:47 smithi077 ceph-mon[37153]: pgmap v1821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2529.410188 secs 2024-07-27T07:31:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:49 smithi047 ceph-mon[36835]: pgmap v1822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:31:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:49 smithi029 ceph-mon[32610]: pgmap v1822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:31:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:49 smithi077 ceph-mon[37153]: pgmap v1822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:31:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:51 smithi047 ceph-mon[36835]: pgmap v1823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:51 smithi029 ceph-mon[32610]: pgmap v1823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:51 smithi077 ceph-mon[37153]: pgmap v1823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2534.410324 secs 2024-07-27T07:31:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2534.410324 secs 2024-07-27T07:31:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2534.410324 secs 2024-07-27T07:31:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:53 smithi029 ceph-mon[32610]: pgmap v1824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:31:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:53 smithi077 ceph-mon[37153]: pgmap v1824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:31:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:53 smithi047 ceph-mon[36835]: pgmap v1824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:31:55.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:55.060+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:31:55.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:55.060+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:31:55.062 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:31:55.060+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:31:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:55.075+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:31:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:55.075+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:31:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:55.075+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:31:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:55 smithi029 ceph-mon[32610]: pgmap v1825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:31:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:55 smithi077 ceph-mon[37153]: pgmap v1825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:31:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:55 smithi047 ceph-mon[36835]: pgmap v1825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:31:58.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:58.013+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:31:58.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:58.013+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff240065b0 con 0x7eff50104ab0 2024-07-27T07:31:58.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:58.013+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:31:58.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:31:58.013+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff240065b0 con 0x7eff50104ab0 2024-07-27T07:31:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:57 smithi029 ceph-mon[32610]: pgmap v1826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2539.410479 secs 2024-07-27T07:31:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:57 smithi077 ceph-mon[37153]: pgmap v1826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2539.410479 secs 2024-07-27T07:31:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:57 smithi047 ceph-mon[36835]: pgmap v1826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:31:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2539.410479 secs 2024-07-27T07:32:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:31:59 smithi029 ceph-mon[32610]: pgmap v1827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:32:00.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:31:59 smithi047 ceph-mon[36835]: pgmap v1827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:32:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:31:59 smithi077 ceph-mon[37153]: pgmap v1827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:32:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:01 smithi077 ceph-mon[37153]: pgmap v1828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:01 smithi047 ceph-mon[36835]: pgmap v1828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:01 smithi029 ceph-mon[32610]: pgmap v1828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2544.410608 secs 2024-07-27T07:32:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2544.410608 secs 2024-07-27T07:32:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2544.410608 secs 2024-07-27T07:32:04.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:03 smithi077 ceph-mon[37153]: pgmap v1829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:32:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:03 smithi047 ceph-mon[36835]: pgmap v1829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:32:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:03 smithi029 ceph-mon[32610]: pgmap v1829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:32:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:05 smithi077 ceph-mon[37153]: pgmap v1830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:32:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:05 smithi047 ceph-mon[36835]: pgmap v1830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:32:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:05 smithi029 ceph-mon[32610]: pgmap v1830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:32:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:07 smithi077 ceph-mon[37153]: pgmap v1831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2549.410714 secs 2024-07-27T07:32:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:07 smithi047 ceph-mon[36835]: pgmap v1831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2549.410714 secs 2024-07-27T07:32:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:07 smithi029 ceph-mon[32610]: pgmap v1831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2549.410714 secs 2024-07-27T07:32:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:32:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:32:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:32:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:32:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:32:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:32:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:32:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:32:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:32:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:32:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:32:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:32:09.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:32:09.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:32:09.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:32:09.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:32:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:32:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:32:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:32:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:32:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:32:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:32:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:32:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:32:10.077 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:32:10.077+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:32:10.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:32:10.077+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:32:10.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:32:10.077+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:32:10.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:32:10.091+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:32:10.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:32:10.091+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:32:10.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:32:10.091+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:32:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:09 smithi077 ceph-mon[37153]: pgmap v1832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:32:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:09 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:32:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:09 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:32:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:09 smithi077 ceph-mon[37153]: pgmap v1833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:32:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:09 smithi077 ceph-mon[37153]: pgmap v1834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:32:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:09 smithi047 ceph-mon[36835]: pgmap v1832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:32:10.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:09 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:32:10.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:09 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:32:10.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:09 smithi047 ceph-mon[36835]: pgmap v1833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:32:10.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:09 smithi047 ceph-mon[36835]: pgmap v1834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:32:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:09 smithi029 ceph-mon[32610]: pgmap v1832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:32:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:09 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:32:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:09 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:32:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:09 smithi029 ceph-mon[32610]: pgmap v1833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:32:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:09 smithi029 ceph-mon[32610]: pgmap v1834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:32:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:10 smithi029 ceph-mon[32610]: pgmap v1835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:32:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:10 smithi077 ceph-mon[37153]: pgmap v1835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:32:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:10 smithi047 ceph-mon[36835]: pgmap v1835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:32:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2554.410859 secs 2024-07-27T07:32:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2554.410859 secs 2024-07-27T07:32:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2554.410859 secs 2024-07-27T07:32:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:13 smithi047 ceph-mon[36835]: pgmap v1836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:32:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:13 smithi029 ceph-mon[32610]: pgmap v1836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:32:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:13 smithi077 ceph-mon[37153]: pgmap v1836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:32:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:15 smithi077 ceph-mon[37153]: pgmap v1837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:32:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:15 smithi047 ceph-mon[36835]: pgmap v1837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:32:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:15 smithi029 ceph-mon[32610]: pgmap v1837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:32:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:17 smithi077 ceph-mon[37153]: pgmap v1838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:32:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2559.410984 secs 2024-07-27T07:32:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:17 smithi047 ceph-mon[36835]: pgmap v1838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:32:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2559.410984 secs 2024-07-27T07:32:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:17 smithi029 ceph-mon[32610]: pgmap v1838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:32:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2559.410984 secs 2024-07-27T07:32:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:19 smithi077 ceph-mon[37153]: pgmap v1839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:32:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:19 smithi047 ceph-mon[36835]: pgmap v1839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:32:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:19 smithi029 ceph-mon[32610]: pgmap v1839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:32:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:21 smithi077 ceph-mon[37153]: pgmap v1840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:21 smithi047 ceph-mon[36835]: pgmap v1840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:21 smithi029 ceph-mon[32610]: pgmap v1840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2564.411092 secs 2024-07-27T07:32:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2564.411092 secs 2024-07-27T07:32:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2564.411092 secs 2024-07-27T07:32:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:23 smithi077 ceph-mon[37153]: pgmap v1841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:32:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:23 smithi047 ceph-mon[36835]: pgmap v1841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:32:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:23 smithi029 ceph-mon[32610]: pgmap v1841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:32:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:32:25.091+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:32:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:32:25.091+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:32:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:32:25.091+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:32:25.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:32:25.106+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:32:25.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:32:25.106+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:32:25.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:32:25.106+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:32:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:25 smithi077 ceph-mon[37153]: pgmap v1842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:32:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:25 smithi047 ceph-mon[36835]: pgmap v1842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:32:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:25 smithi029 ceph-mon[32610]: pgmap v1842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:32:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:27 smithi077 ceph-mon[37153]: pgmap v1843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:32:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2569.411192 secs 2024-07-27T07:32:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:27 smithi047 ceph-mon[36835]: pgmap v1843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:32:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2569.411192 secs 2024-07-27T07:32:28.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:27 smithi029 ceph-mon[32610]: pgmap v1843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:32:28.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2569.411192 secs 2024-07-27T07:32:30.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:29 smithi077 ceph-mon[37153]: pgmap v1844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:32:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:29 smithi047 ceph-mon[36835]: pgmap v1844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:32:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:29 smithi029 ceph-mon[32610]: pgmap v1844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:32:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:31 smithi077 ceph-mon[37153]: pgmap v1845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:31 smithi047 ceph-mon[36835]: pgmap v1845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:31 smithi029 ceph-mon[32610]: pgmap v1845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2574.411315 secs 2024-07-27T07:32:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2574.411315 secs 2024-07-27T07:32:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2574.411315 secs 2024-07-27T07:32:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:33 smithi077 ceph-mon[37153]: pgmap v1846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:32:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:33 smithi047 ceph-mon[36835]: pgmap v1846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:32:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:33 smithi029 ceph-mon[32610]: pgmap v1846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:32:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:35 smithi077 ceph-mon[37153]: pgmap v1847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:32:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:35 smithi047 ceph-mon[36835]: pgmap v1847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:32:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:35 smithi029 ceph-mon[32610]: pgmap v1847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:32:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:37 smithi077 ceph-mon[37153]: pgmap v1848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:32:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2579.411434 secs 2024-07-27T07:32:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:37 smithi047 ceph-mon[36835]: pgmap v1848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:32:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2579.411434 secs 2024-07-27T07:32:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:37 smithi029 ceph-mon[32610]: pgmap v1848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:32:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2579.411434 secs 2024-07-27T07:32:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:39 smithi077 ceph-mon[37153]: pgmap v1849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:32:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:32:40.107+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:32:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:32:40.107+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:32:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:32:40.107+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:32:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:32:40.121+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:32:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:32:40.121+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:32:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:32:40.121+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:32:40.123 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:39 smithi029 ceph-mon[32610]: pgmap v1849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:32:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:39 smithi047 ceph-mon[36835]: pgmap v1849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:32:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:41 smithi077 ceph-mon[37153]: pgmap v1850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:42.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:41 smithi029 ceph-mon[32610]: pgmap v1850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:41 smithi047 ceph-mon[36835]: pgmap v1850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2584.411555 secs 2024-07-27T07:32:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2584.411555 secs 2024-07-27T07:32:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2584.411555 secs 2024-07-27T07:32:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:43 smithi077 ceph-mon[37153]: pgmap v1851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:32:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:43 smithi047 ceph-mon[36835]: pgmap v1851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:32:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:43 smithi029 ceph-mon[32610]: pgmap v1851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:32:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:45 smithi077 ceph-mon[37153]: pgmap v1852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:32:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:45 smithi047 ceph-mon[36835]: pgmap v1852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:32:46.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:45 smithi029 ceph-mon[32610]: pgmap v1852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:32:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:47 smithi077 ceph-mon[37153]: pgmap v1853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:32:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2589.411693 secs 2024-07-27T07:32:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:47 smithi047 ceph-mon[36835]: pgmap v1853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:32:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2589.411693 secs 2024-07-27T07:32:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:47 smithi029 ceph-mon[32610]: pgmap v1853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:32:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2589.411693 secs 2024-07-27T07:32:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:49 smithi077 ceph-mon[37153]: pgmap v1854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:32:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:49 smithi047 ceph-mon[36835]: pgmap v1854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:32:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:49 smithi029 ceph-mon[32610]: pgmap v1854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:32:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:51 smithi077 ceph-mon[37153]: pgmap v1855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:51 smithi047 ceph-mon[36835]: pgmap v1855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:51 smithi029 ceph-mon[32610]: pgmap v1855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:32:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2594.411756 secs 2024-07-27T07:32:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2594.411756 secs 2024-07-27T07:32:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2594.411756 secs 2024-07-27T07:32:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:53 smithi077 ceph-mon[37153]: pgmap v1856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:32:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:53 smithi047 ceph-mon[36835]: pgmap v1856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:32:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:53 smithi029 ceph-mon[32610]: pgmap v1856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:32:55.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:32:55.122+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:32:55.123 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:32:55.122+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:32:55.124 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:32:55.122+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:32:55.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:32:55.137+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:32:55.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:32:55.137+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:32:55.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:32:55.137+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:32:56.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:55 smithi077 ceph-mon[37153]: pgmap v1857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:32:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:55 smithi047 ceph-mon[36835]: pgmap v1857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:32:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:55 smithi029 ceph-mon[32610]: pgmap v1857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:32:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:57 smithi077 ceph-mon[37153]: pgmap v1858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:32:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2599.411877 secs 2024-07-27T07:32:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:57 smithi047 ceph-mon[36835]: pgmap v1858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:32:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2599.411877 secs 2024-07-27T07:32:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:57 smithi029 ceph-mon[32610]: pgmap v1858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:32:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2599.411877 secs 2024-07-27T07:33:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:32:59 smithi077 ceph-mon[37153]: pgmap v1859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:32:59 smithi047 ceph-mon[36835]: pgmap v1859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:32:59 smithi029 ceph-mon[32610]: pgmap v1859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:01 smithi077 ceph-mon[37153]: pgmap v1860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:01 smithi047 ceph-mon[36835]: pgmap v1860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:01 smithi029 ceph-mon[32610]: pgmap v1860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2604.412007 secs 2024-07-27T07:33:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2604.412007 secs 2024-07-27T07:33:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2604.412007 secs 2024-07-27T07:33:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:03 smithi077 ceph-mon[37153]: pgmap v1861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:33:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:03 smithi047 ceph-mon[36835]: pgmap v1861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:33:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:03 smithi029 ceph-mon[32610]: pgmap v1861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:33:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:05 smithi077 ceph-mon[37153]: pgmap v1862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:33:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:05 smithi047 ceph-mon[36835]: pgmap v1862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:33:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:05 smithi029 ceph-mon[32610]: pgmap v1862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:33:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:07 smithi077 ceph-mon[37153]: pgmap v1863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2609.412128 secs 2024-07-27T07:33:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:07 smithi047 ceph-mon[36835]: pgmap v1863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2609.412128 secs 2024-07-27T07:33:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:07 smithi029 ceph-mon[32610]: pgmap v1863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2609.412128 secs 2024-07-27T07:33:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:33:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:33:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:33:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:09 smithi077 ceph-mon[37153]: pgmap v1864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:33:10.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:33:10.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:33:10.138 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:33:10.138+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:33:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:33:10.138+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:33:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:33:10.138+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:33:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:33:10.153+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:33:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:33:10.153+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:33:10.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:33:10.153+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:33:10.154 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:09 smithi029 ceph-mon[32610]: pgmap v1864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:10.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:33:10.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:33:10.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:33:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:09 smithi047 ceph-mon[36835]: pgmap v1864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:33:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:33:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:33:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:33:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:10 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:33:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:10 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:33:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:10 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:33:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:10 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:33:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:10 smithi077 ceph-mon[37153]: pgmap v1865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:33:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:10 smithi077 ceph-mon[37153]: pgmap v1866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:33:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:10 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:33:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:10 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:33:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:10 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:33:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:10 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:33:11.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:10 smithi047 ceph-mon[36835]: pgmap v1865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:33:11.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:10 smithi047 ceph-mon[36835]: pgmap v1866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:33:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:10 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:33:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:10 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:33:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:10 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:33:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:10 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:33:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:10 smithi029 ceph-mon[32610]: pgmap v1865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:33:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:10 smithi029 ceph-mon[32610]: pgmap v1866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:33:13.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:12 smithi077 ceph-mon[37153]: pgmap v1867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:33:13.111 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2614.412252 secs 2024-07-27T07:33:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:12 smithi047 ceph-mon[36835]: pgmap v1867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:33:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2614.412252 secs 2024-07-27T07:33:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:12 smithi029 ceph-mon[32610]: pgmap v1867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:33:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2614.412252 secs 2024-07-27T07:33:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:14 smithi047 ceph-mon[36835]: pgmap v1868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:33:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:14 smithi029 ceph-mon[32610]: pgmap v1868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:33:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:14 smithi077 ceph-mon[37153]: pgmap v1868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:33:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:16 smithi077 ceph-mon[37153]: pgmap v1869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:33:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:16 smithi047 ceph-mon[36835]: pgmap v1869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:33:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:16 smithi029 ceph-mon[32610]: pgmap v1869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:33:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2619.412374 secs 2024-07-27T07:33:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2619.412374 secs 2024-07-27T07:33:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2619.412374 secs 2024-07-27T07:33:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:18 smithi077 ceph-mon[37153]: pgmap v1870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:33:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:18 smithi047 ceph-mon[36835]: pgmap v1870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:33:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:18 smithi029 ceph-mon[32610]: pgmap v1870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:33:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:20 smithi077 ceph-mon[37153]: pgmap v1871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:33:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:20 smithi047 ceph-mon[36835]: pgmap v1871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:33:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:20 smithi029 ceph-mon[32610]: pgmap v1871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:33:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:22 smithi077 ceph-mon[37153]: pgmap v1872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2624.412475 secs 2024-07-27T07:33:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:22 smithi047 ceph-mon[36835]: pgmap v1872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2624.412475 secs 2024-07-27T07:33:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:22 smithi029 ceph-mon[32610]: pgmap v1872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2624.412475 secs 2024-07-27T07:33:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:24 smithi077 ceph-mon[37153]: pgmap v1873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:24 smithi047 ceph-mon[36835]: pgmap v1873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:24 smithi029 ceph-mon[32610]: pgmap v1873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:33:25.152+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:33:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:33:25.152+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:33:25.154 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:33:25.152+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:33:25.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:33:25.160+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:33:25.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:33:25.160+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:33:25.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:33:25.160+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:33:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:26 smithi077 ceph-mon[37153]: pgmap v1874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:26 smithi047 ceph-mon[36835]: pgmap v1874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:26 smithi029 ceph-mon[32610]: pgmap v1874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2629.412598 secs 2024-07-27T07:33:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2629.412598 secs 2024-07-27T07:33:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2629.412598 secs 2024-07-27T07:33:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:28 smithi077 ceph-mon[37153]: pgmap v1875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:28.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:28 smithi029 ceph-mon[32610]: pgmap v1875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:28 smithi047 ceph-mon[36835]: pgmap v1875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:30 smithi077 ceph-mon[37153]: pgmap v1876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:33:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:30 smithi047 ceph-mon[36835]: pgmap v1876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:33:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:30 smithi029 ceph-mon[32610]: pgmap v1876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:33:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:32 smithi077 ceph-mon[37153]: pgmap v1877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2634.412717 secs 2024-07-27T07:33:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:32 smithi047 ceph-mon[36835]: pgmap v1877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2634.412717 secs 2024-07-27T07:33:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:32 smithi029 ceph-mon[32610]: pgmap v1877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2634.412717 secs 2024-07-27T07:33:34.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:34 smithi077 ceph-mon[37153]: pgmap v1878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:34 smithi047 ceph-mon[36835]: pgmap v1878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:34 smithi029 ceph-mon[32610]: pgmap v1878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:36 smithi077 ceph-mon[37153]: pgmap v1879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:36 smithi047 ceph-mon[36835]: pgmap v1879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:36 smithi029 ceph-mon[32610]: pgmap v1879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2639.412840 secs 2024-07-27T07:33:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2639.412840 secs 2024-07-27T07:33:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2639.412840 secs 2024-07-27T07:33:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:38 smithi077 ceph-mon[37153]: pgmap v1880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:38 smithi047 ceph-mon[36835]: pgmap v1880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:38 smithi029 ceph-mon[32610]: pgmap v1880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:33:40.168+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:33:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:33:40.168+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:33:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:33:40.168+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:33:40.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:33:40.174+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:33:40.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:33:40.174+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:33:40.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:33:40.174+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:33:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:40 smithi029 ceph-mon[32610]: pgmap v1881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:33:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:40 smithi077 ceph-mon[37153]: pgmap v1881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:33:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:40 smithi047 ceph-mon[36835]: pgmap v1881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:33:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:42 smithi077 ceph-mon[37153]: pgmap v1882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2644.412961 secs 2024-07-27T07:33:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:42 smithi047 ceph-mon[36835]: pgmap v1882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2644.412961 secs 2024-07-27T07:33:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:42 smithi029 ceph-mon[32610]: pgmap v1882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2644.412961 secs 2024-07-27T07:33:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:44 smithi077 ceph-mon[37153]: pgmap v1883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:44 smithi047 ceph-mon[36835]: pgmap v1883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:44 smithi029 ceph-mon[32610]: pgmap v1883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:46.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:46 smithi047 ceph-mon[36835]: pgmap v1884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:46 smithi029 ceph-mon[32610]: pgmap v1884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:46 smithi077 ceph-mon[37153]: pgmap v1884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2649.413028 secs 2024-07-27T07:33:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2649.413028 secs 2024-07-27T07:33:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2649.413028 secs 2024-07-27T07:33:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:48 smithi047 ceph-mon[36835]: pgmap v1885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:33:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:48 smithi029 ceph-mon[32610]: pgmap v1885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:33:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:48 smithi077 ceph-mon[37153]: pgmap v1885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:33:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:50 smithi047 ceph-mon[36835]: pgmap v1886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:33:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:50 smithi029 ceph-mon[32610]: pgmap v1886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:33:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:50 smithi077 ceph-mon[37153]: pgmap v1886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:33:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:52 smithi047 ceph-mon[36835]: pgmap v1887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2654.413151 secs 2024-07-27T07:33:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:52 smithi029 ceph-mon[32610]: pgmap v1887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2654.413151 secs 2024-07-27T07:33:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:52 smithi077 ceph-mon[37153]: pgmap v1887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:33:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2654.413151 secs 2024-07-27T07:33:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:54 smithi047 ceph-mon[36835]: pgmap v1888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:54 smithi029 ceph-mon[32610]: pgmap v1888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:54 smithi077 ceph-mon[37153]: pgmap v1888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:33:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:33:55.169+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:33:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:33:55.169+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:33:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:33:55.169+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:33:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:33:55.190+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:33:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:33:55.190+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:33:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:33:55.190+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:33:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:56 smithi077 ceph-mon[37153]: pgmap v1889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:56 smithi047 ceph-mon[36835]: pgmap v1889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:56 smithi029 ceph-mon[32610]: pgmap v1889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:33:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2659.413287 secs 2024-07-27T07:33:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2659.413287 secs 2024-07-27T07:33:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2659.413287 secs 2024-07-27T07:33:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:33:58 smithi077 ceph-mon[37153]: pgmap v1890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:33:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:33:58 smithi047 ceph-mon[36835]: pgmap v1890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:33:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:33:58 smithi029 ceph-mon[32610]: pgmap v1890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:34:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:00 smithi077 ceph-mon[37153]: pgmap v1891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:34:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:00 smithi047 ceph-mon[36835]: pgmap v1891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:34:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:00 smithi029 ceph-mon[32610]: pgmap v1891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:34:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:02 smithi077 ceph-mon[37153]: pgmap v1892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:34:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2664.413426 secs 2024-07-27T07:34:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:02 smithi047 ceph-mon[36835]: pgmap v1892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:34:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2664.413426 secs 2024-07-27T07:34:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:02 smithi029 ceph-mon[32610]: pgmap v1892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:34:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2664.413426 secs 2024-07-27T07:34:04.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:04 smithi077 ceph-mon[37153]: pgmap v1893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:04 smithi047 ceph-mon[36835]: pgmap v1893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:04 smithi029 ceph-mon[32610]: pgmap v1893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:06 smithi077 ceph-mon[37153]: pgmap v1894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:06 smithi047 ceph-mon[36835]: pgmap v1894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:06 smithi029 ceph-mon[32610]: pgmap v1894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2669.413553 secs 2024-07-27T07:34:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2669.413553 secs 2024-07-27T07:34:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2669.413553 secs 2024-07-27T07:34:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:08 smithi077 ceph-mon[37153]: pgmap v1895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:34:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:08 smithi047 ceph-mon[36835]: pgmap v1895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:34:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:08 smithi029 ceph-mon[32610]: pgmap v1895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:34:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:10.172+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:34:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:10.172+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:34:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:10.172+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:34:10.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:10.205+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:34:10.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:10.205+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:34:10.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:10.205+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:34:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:10 smithi029 ceph-mon[32610]: pgmap v1896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:34:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:34:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:34:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:34:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:10 smithi029 ceph-mon[32610]: pgmap v1897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:34:10.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:10 smithi029 ceph-mon[32610]: pgmap v1898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:34:10.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:34:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:10 smithi077 ceph-mon[37153]: pgmap v1896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:34:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:34:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:34:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:34:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:10 smithi077 ceph-mon[37153]: pgmap v1897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:34:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:10 smithi077 ceph-mon[37153]: pgmap v1898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:34:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:34:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:10 smithi047 ceph-mon[36835]: pgmap v1896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:34:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:34:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:34:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:34:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:10 smithi047 ceph-mon[36835]: pgmap v1897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:34:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:10 smithi047 ceph-mon[36835]: pgmap v1898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:34:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:34:13.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2674.413650 secs 2024-07-27T07:34:13.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:12 smithi077 ceph-mon[37153]: pgmap v1899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:34:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2674.413650 secs 2024-07-27T07:34:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:12 smithi047 ceph-mon[36835]: pgmap v1899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:34:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2674.413650 secs 2024-07-27T07:34:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:12 smithi029 ceph-mon[32610]: pgmap v1899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:34:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:14 smithi077 ceph-mon[37153]: pgmap v1900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:34:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:14 smithi047 ceph-mon[36835]: pgmap v1900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:34:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:14 smithi029 ceph-mon[32610]: pgmap v1900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:34:17.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:16 smithi047 ceph-mon[36835]: pgmap v1901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T07:34:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:16 smithi029 ceph-mon[32610]: pgmap v1901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T07:34:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:16 smithi077 ceph-mon[37153]: pgmap v1901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T07:34:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2679.413828 secs 2024-07-27T07:34:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2679.413828 secs 2024-07-27T07:34:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2679.413828 secs 2024-07-27T07:34:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:18 smithi047 ceph-mon[36835]: pgmap v1902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:34:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:18 smithi029 ceph-mon[32610]: pgmap v1902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:34:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:18 smithi077 ceph-mon[37153]: pgmap v1902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:34:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:20 smithi047 ceph-mon[36835]: pgmap v1903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T07:34:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:20 smithi029 ceph-mon[32610]: pgmap v1903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T07:34:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:20 smithi077 ceph-mon[37153]: pgmap v1903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T07:34:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2684.413941 secs 2024-07-27T07:34:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:22 smithi047 ceph-mon[36835]: pgmap v1904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:34:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2684.413941 secs 2024-07-27T07:34:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:22 smithi029 ceph-mon[32610]: pgmap v1904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:34:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2684.413941 secs 2024-07-27T07:34:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:22 smithi077 ceph-mon[37153]: pgmap v1904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:34:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:25.172+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:34:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:25.172+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:34:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:25.172+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:34:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:24 smithi047 ceph-mon[36835]: pgmap v1905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:24 smithi029 ceph-mon[32610]: pgmap v1905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:25.219+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:34:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:25.219+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:34:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:25.219+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:34:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:24 smithi077 ceph-mon[37153]: pgmap v1905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:26 smithi047 ceph-mon[36835]: pgmap v1906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:26 smithi029 ceph-mon[32610]: pgmap v1906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:26 smithi077 ceph-mon[37153]: pgmap v1906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2689.414054 secs 2024-07-27T07:34:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2689.414054 secs 2024-07-27T07:34:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2689.414054 secs 2024-07-27T07:34:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:28 smithi047 ceph-mon[36835]: pgmap v1907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:34:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:28 smithi029 ceph-mon[32610]: pgmap v1907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:34:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:28 smithi077 ceph-mon[37153]: pgmap v1907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:34:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:30 smithi047 ceph-mon[36835]: pgmap v1908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:34:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:30 smithi029 ceph-mon[32610]: pgmap v1908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:34:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:30 smithi077 ceph-mon[37153]: pgmap v1908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:34:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2694.414176 secs 2024-07-27T07:34:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:32 smithi047 ceph-mon[36835]: pgmap v1909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:34:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2694.414176 secs 2024-07-27T07:34:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:32 smithi029 ceph-mon[32610]: pgmap v1909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:34:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2694.414176 secs 2024-07-27T07:34:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:32 smithi077 ceph-mon[37153]: pgmap v1909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:34:35.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:34 smithi077 ceph-mon[37153]: pgmap v1910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:34 smithi047 ceph-mon[36835]: pgmap v1910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:34 smithi029 ceph-mon[32610]: pgmap v1910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:36 smithi047 ceph-mon[36835]: pgmap v1911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:36 smithi029 ceph-mon[32610]: pgmap v1911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:36 smithi077 ceph-mon[37153]: pgmap v1911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2699.414296 secs 2024-07-27T07:34:38.215 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2699.414296 secs 2024-07-27T07:34:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2699.414296 secs 2024-07-27T07:34:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:38 smithi047 ceph-mon[36835]: pgmap v1912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:34:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:38 smithi029 ceph-mon[32610]: pgmap v1912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:34:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:38 smithi077 ceph-mon[37153]: pgmap v1912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:34:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:40.187+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:34:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:40.187+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:34:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:40.187+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:34:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:40.221+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:34:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:40.221+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:34:40.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:40.221+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:34:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:40 smithi047 ceph-mon[36835]: pgmap v1913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:40 smithi029 ceph-mon[32610]: pgmap v1913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:40 smithi077 ceph-mon[37153]: pgmap v1913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:42.635+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380c0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-27T07:34:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:42.635+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380c0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T07:34:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:42.635+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380c0770 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f733c004010 tx=0x7f733c03b000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T07:34:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:42.635+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380c0770 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f733c004010 tx=0x7f733c03b000 comp rx=0 tx=0).stop 2024-07-27T07:34:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:42.635+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380c0770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T07:34:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:42.635+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380c0770 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T07:34:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:42.635+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T07:34:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:42.635+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T07:34:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:42.636+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f733c02b4a0 tx=0x7f733c046000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T07:34:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:43.037+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:34:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:43.037+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:34:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:43.037+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:34:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:43.037+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:34:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2704.414416 secs 2024-07-27T07:34:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:42 smithi047 ceph-mon[36835]: pgmap v1914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:34:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2704.414416 secs 2024-07-27T07:34:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:42 smithi029 ceph-mon[32610]: pgmap v1914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:34:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2704.414416 secs 2024-07-27T07:34:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:42 smithi077 ceph-mon[37153]: pgmap v1914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:34:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:44 smithi047 ceph-mon[36835]: pgmap v1915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:44 smithi029 ceph-mon[32610]: pgmap v1915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:44 smithi077 ceph-mon[37153]: pgmap v1915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:46 smithi047 ceph-mon[36835]: pgmap v1916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:46 smithi029 ceph-mon[32610]: pgmap v1916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:46 smithi077 ceph-mon[37153]: pgmap v1916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2709.414540 secs 2024-07-27T07:34:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2709.414540 secs 2024-07-27T07:34:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2709.414540 secs 2024-07-27T07:34:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:48 smithi047 ceph-mon[36835]: pgmap v1917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:34:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:48 smithi029 ceph-mon[32610]: pgmap v1917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:34:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:48 smithi077 ceph-mon[37153]: pgmap v1917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:34:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:50 smithi047 ceph-mon[36835]: pgmap v1918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:50 smithi029 ceph-mon[32610]: pgmap v1918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:50 smithi077 ceph-mon[37153]: pgmap v1918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2714.414682 secs 2024-07-27T07:34:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:52 smithi047 ceph-mon[36835]: pgmap v1919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:34:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2714.414682 secs 2024-07-27T07:34:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:52 smithi029 ceph-mon[32610]: pgmap v1919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:34:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2714.414682 secs 2024-07-27T07:34:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:52 smithi077 ceph-mon[37153]: pgmap v1919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:34:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:54 smithi047 ceph-mon[36835]: pgmap v1920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:55.202+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:34:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:55.202+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:34:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:34:55.202+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:34:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:54 smithi029 ceph-mon[32610]: pgmap v1920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:55.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:55.236+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:34:55.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:55.236+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:34:55.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:55.236+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:34:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:54 smithi077 ceph-mon[37153]: pgmap v1920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:34:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:56 smithi047 ceph-mon[36835]: pgmap v1921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:56 smithi029 ceph-mon[32610]: pgmap v1921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:56 smithi077 ceph-mon[37153]: pgmap v1921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:34:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:57.649+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0c0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000967) for more than 900000000 us, fault. 2024-07-27T07:34:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:57.650+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0c0810 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7eff40004480 tx=0x7eff4002b040 comp rx=0 tx=0).stop 2024-07-27T07:34:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:57.650+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0c0810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T07:34:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:57.650+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0c0810 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T07:34:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:57.650+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T07:34:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:57.650+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T07:34:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:57.651+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7eff40009ee0 tx=0x7eff4001e040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T07:34:58.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:58.039+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff240065b0 con 0x7eff50104ab0 2024-07-27T07:34:58.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:58.039+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:34:58.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:58.040+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff240065b0 con 0x7eff50104ab0 2024-07-27T07:34:58.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:34:58.040+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:34:58.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2719.414821 secs 2024-07-27T07:34:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2719.414821 secs 2024-07-27T07:34:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2719.414821 secs 2024-07-27T07:34:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:34:58 smithi047 ceph-mon[36835]: pgmap v1922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:34:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:34:58 smithi029 ceph-mon[32610]: pgmap v1922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:34:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:34:58 smithi077 ceph-mon[37153]: pgmap v1922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:35:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:00 smithi029 ceph-mon[32610]: pgmap v1923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:00 smithi077 ceph-mon[37153]: pgmap v1923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:00 smithi047 ceph-mon[36835]: pgmap v1923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2724.414916 secs 2024-07-27T07:35:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:02 smithi029 ceph-mon[32610]: pgmap v1924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:35:03.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2724.414916 secs 2024-07-27T07:35:03.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:02 smithi047 ceph-mon[36835]: pgmap v1924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:35:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2724.414916 secs 2024-07-27T07:35:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:02 smithi077 ceph-mon[37153]: pgmap v1924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:35:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:04 smithi029 ceph-mon[32610]: pgmap v1925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:35:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:04 smithi077 ceph-mon[37153]: pgmap v1925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:35:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:04 smithi047 ceph-mon[36835]: pgmap v1925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:35:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:06 smithi029 ceph-mon[32610]: pgmap v1926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:06 smithi077 ceph-mon[37153]: pgmap v1926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:06 smithi047 ceph-mon[36835]: pgmap v1926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2729.415016 secs 2024-07-27T07:35:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2729.415016 secs 2024-07-27T07:35:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2729.415016 secs 2024-07-27T07:35:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:08 smithi077 ceph-mon[37153]: pgmap v1927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:35:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:08 smithi047 ceph-mon[36835]: pgmap v1927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:35:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:08 smithi029 ceph-mon[32610]: pgmap v1927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:35:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:35:10.218+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:35:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:35:10.218+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:35:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:35:10.218+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:35:10.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:35:10.249+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:35:10.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:35:10.249+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:35:10.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:35:10.249+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:35:10.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:35:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:35:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:35:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:10 smithi077 ceph-mon[37153]: pgmap v1928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:35:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:35:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:35:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:10 smithi047 ceph-mon[36835]: pgmap v1928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:35:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:35:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:35:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:10 smithi029 ceph-mon[32610]: pgmap v1928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:35:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:35:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:35:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:11 smithi077 ceph-mon[37153]: pgmap v1929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:35:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:11 smithi077 ceph-mon[37153]: pgmap v1930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:35:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:11 smithi047 ceph-mon[36835]: pgmap v1929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:35:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:11 smithi047 ceph-mon[36835]: pgmap v1930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:35:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:11 smithi029 ceph-mon[32610]: pgmap v1929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:35:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:11 smithi029 ceph-mon[32610]: pgmap v1930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:35:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2734.415114 secs 2024-07-27T07:35:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2734.415114 secs 2024-07-27T07:35:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2734.415114 secs 2024-07-27T07:35:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:13 smithi077 ceph-mon[37153]: pgmap v1931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T07:35:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:13 smithi047 ceph-mon[36835]: pgmap v1931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T07:35:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:13 smithi029 ceph-mon[32610]: pgmap v1931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T07:35:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:15 smithi077 ceph-mon[37153]: pgmap v1932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:35:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:15 smithi047 ceph-mon[36835]: pgmap v1932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:35:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:15 smithi029 ceph-mon[32610]: pgmap v1932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:35:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:17 smithi077 ceph-mon[37153]: pgmap v1933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:35:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2739.415237 secs 2024-07-27T07:35:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:17 smithi047 ceph-mon[36835]: pgmap v1933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:35:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2739.415237 secs 2024-07-27T07:35:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:17 smithi029 ceph-mon[32610]: pgmap v1933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:35:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2739.415237 secs 2024-07-27T07:35:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:19 smithi077 ceph-mon[37153]: pgmap v1934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:35:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:19 smithi047 ceph-mon[36835]: pgmap v1934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:35:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:19 smithi029 ceph-mon[32610]: pgmap v1934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:35:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:21 smithi077 ceph-mon[37153]: pgmap v1935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:35:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:21 smithi047 ceph-mon[36835]: pgmap v1935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:35:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:21 smithi029 ceph-mon[32610]: pgmap v1935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:35:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2744.415387 secs 2024-07-27T07:35:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2744.415387 secs 2024-07-27T07:35:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2744.415387 secs 2024-07-27T07:35:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:24 smithi077 ceph-mon[37153]: pgmap v1936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:35:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:24 smithi047 ceph-mon[36835]: pgmap v1936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:35:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:24 smithi029 ceph-mon[32610]: pgmap v1936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:35:25.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:35:25.230+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:35:25.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:35:25.230+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:35:25.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:35:25.230+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:35:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:35:25.253+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:35:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:35:25.253+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:35:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:35:25.253+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:35:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:26 smithi077 ceph-mon[37153]: pgmap v1937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:35:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:26 smithi047 ceph-mon[36835]: pgmap v1937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:35:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:26 smithi029 ceph-mon[32610]: pgmap v1937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:35:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:28 smithi077 ceph-mon[37153]: pgmap v1938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:35:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2749.415511 secs 2024-07-27T07:35:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:28 smithi047 ceph-mon[36835]: pgmap v1938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:35:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2749.415511 secs 2024-07-27T07:35:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:28 smithi029 ceph-mon[32610]: pgmap v1938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:35:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2749.415511 secs 2024-07-27T07:35:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:30 smithi077 ceph-mon[37153]: pgmap v1939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:35:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:30 smithi047 ceph-mon[36835]: pgmap v1939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:35:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:30 smithi029 ceph-mon[32610]: pgmap v1939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:35:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:31 smithi077 ceph-mon[37153]: pgmap v1940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:35:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:31 smithi047 ceph-mon[36835]: pgmap v1940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:35:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:31 smithi029 ceph-mon[32610]: pgmap v1940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:35:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2754.415631 secs 2024-07-27T07:35:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2754.415631 secs 2024-07-27T07:35:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2754.415631 secs 2024-07-27T07:35:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:33 smithi047 ceph-mon[36835]: pgmap v1941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:35:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:33 smithi029 ceph-mon[32610]: pgmap v1941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:35:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:33 smithi077 ceph-mon[37153]: pgmap v1941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:35:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:35 smithi047 ceph-mon[36835]: pgmap v1942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:35:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:35 smithi029 ceph-mon[32610]: pgmap v1942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:35:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:35 smithi077 ceph-mon[37153]: pgmap v1942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:35:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:37 smithi047 ceph-mon[36835]: pgmap v1943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:35:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2759.415772 secs 2024-07-27T07:35:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:37 smithi029 ceph-mon[32610]: pgmap v1943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:35:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2759.415772 secs 2024-07-27T07:35:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:37 smithi077 ceph-mon[37153]: pgmap v1943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:35:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2759.415772 secs 2024-07-27T07:35:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:39 smithi047 ceph-mon[36835]: pgmap v1944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:35:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:39 smithi029 ceph-mon[32610]: pgmap v1944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:35:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:39 smithi077 ceph-mon[37153]: pgmap v1944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:35:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:35:40.245+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:35:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:35:40.245+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:35:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:35:40.245+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:35:40.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:35:40.263+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:35:40.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:35:40.263+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:35:40.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:35:40.263+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:35:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:41 smithi047 ceph-mon[36835]: pgmap v1945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:41 smithi029 ceph-mon[32610]: pgmap v1945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:41 smithi077 ceph-mon[37153]: pgmap v1945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2764.415906 secs 2024-07-27T07:35:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2764.415906 secs 2024-07-27T07:35:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2764.415906 secs 2024-07-27T07:35:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:43 smithi047 ceph-mon[36835]: pgmap v1946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:35:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:43 smithi029 ceph-mon[32610]: pgmap v1946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:35:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:43 smithi077 ceph-mon[37153]: pgmap v1946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:35:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:45 smithi047 ceph-mon[36835]: pgmap v1947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:35:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:45 smithi029 ceph-mon[32610]: pgmap v1947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:35:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:45 smithi077 ceph-mon[37153]: pgmap v1947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:35:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:47 smithi047 ceph-mon[36835]: pgmap v1948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:35:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2769.416011 secs 2024-07-27T07:35:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:47 smithi029 ceph-mon[32610]: pgmap v1948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:35:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2769.416011 secs 2024-07-27T07:35:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:47 smithi077 ceph-mon[37153]: pgmap v1948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:35:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2769.416011 secs 2024-07-27T07:35:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:49 smithi047 ceph-mon[36835]: pgmap v1949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:35:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:49 smithi029 ceph-mon[32610]: pgmap v1949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:35:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:49 smithi077 ceph-mon[37153]: pgmap v1949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:35:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:51 smithi029 ceph-mon[32610]: pgmap v1950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:51 smithi077 ceph-mon[37153]: pgmap v1950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:51 smithi047 ceph-mon[36835]: pgmap v1950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:35:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2774.416098 secs 2024-07-27T07:35:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2774.416098 secs 2024-07-27T07:35:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2774.416098 secs 2024-07-27T07:35:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:53 smithi029 ceph-mon[32610]: pgmap v1951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:35:53.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:53 smithi077 ceph-mon[37153]: pgmap v1951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:35:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:53 smithi047 ceph-mon[36835]: pgmap v1951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:35:55.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:35:55.254+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:35:55.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:35:55.254+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:35:55.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:35:55.254+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:35:55.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:35:55.269+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:35:55.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:35:55.269+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:35:55.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:35:55.269+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:35:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:55 smithi029 ceph-mon[32610]: pgmap v1952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:35:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:55 smithi077 ceph-mon[37153]: pgmap v1952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:35:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:55 smithi047 ceph-mon[36835]: pgmap v1952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:35:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:57 smithi029 ceph-mon[32610]: pgmap v1953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:35:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2779.416216 secs 2024-07-27T07:35:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:57 smithi077 ceph-mon[37153]: pgmap v1953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:35:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2779.416216 secs 2024-07-27T07:35:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:57 smithi047 ceph-mon[36835]: pgmap v1953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:35:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2779.416216 secs 2024-07-27T07:35:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:35:59 smithi077 ceph-mon[37153]: pgmap v1954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:35:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:35:59 smithi047 ceph-mon[36835]: pgmap v1954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:35:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:35:59 smithi029 ceph-mon[32610]: pgmap v1954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:36:01.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:01 smithi077 ceph-mon[37153]: pgmap v1955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:36:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:01 smithi047 ceph-mon[36835]: pgmap v1955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:36:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:01 smithi029 ceph-mon[32610]: pgmap v1955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:36:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2784.416344 secs 2024-07-27T07:36:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2784.416344 secs 2024-07-27T07:36:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2784.416344 secs 2024-07-27T07:36:03.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:03 smithi047 ceph-mon[36835]: pgmap v1956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:36:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:03 smithi077 ceph-mon[37153]: pgmap v1956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:36:03.879 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:03 smithi029 ceph-mon[32610]: pgmap v1956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:36:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:05 smithi077 ceph-mon[37153]: pgmap v1957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:36:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:05 smithi047 ceph-mon[36835]: pgmap v1957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:36:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:05 smithi029 ceph-mon[32610]: pgmap v1957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:36:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:07 smithi077 ceph-mon[37153]: pgmap v1958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:36:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2789.416476 secs 2024-07-27T07:36:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:07 smithi047 ceph-mon[36835]: pgmap v1958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:36:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2789.416476 secs 2024-07-27T07:36:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:07 smithi029 ceph-mon[32610]: pgmap v1958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:36:07.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2789.416476 secs 2024-07-27T07:36:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:09 smithi077 ceph-mon[37153]: pgmap v1959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:36:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:09 smithi047 ceph-mon[36835]: pgmap v1959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:36:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:09 smithi029 ceph-mon[32610]: pgmap v1959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:36:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:36:10.261+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:36:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:36:10.262+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:36:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:36:10.262+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:36:10.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:36:10.285+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:36:10.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:36:10.285+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:36:10.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:36:10.285+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:36:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:36:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:36:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:36:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:11 smithi077 ceph-mon[37153]: pgmap v1960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:11 smithi047 ceph-mon[36835]: pgmap v1960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:11 smithi029 ceph-mon[32610]: pgmap v1960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2794.416533 secs 2024-07-27T07:36:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: pgmap v1961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:36:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:36:13.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:36:13.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: pgmap v1962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:36:13.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: pgmap v1963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:36:13.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2794.416533 secs 2024-07-27T07:36:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: pgmap v1961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:36:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:36:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:36:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: pgmap v1962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:36:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: pgmap v1963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:36:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2794.416533 secs 2024-07-27T07:36:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: pgmap v1961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:36:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:36:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:36:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: pgmap v1962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:36:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: pgmap v1963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:36:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:36:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:15 smithi077 ceph-mon[37153]: pgmap v1964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:36:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:15 smithi047 ceph-mon[36835]: pgmap v1964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:36:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:15 smithi029 ceph-mon[32610]: pgmap v1964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:36:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:17 smithi077 ceph-mon[37153]: pgmap v1965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:36:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:17 smithi047 ceph-mon[36835]: pgmap v1965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:36:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:17 smithi029 ceph-mon[32610]: pgmap v1965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:36:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2799.416687 secs 2024-07-27T07:36:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2799.416687 secs 2024-07-27T07:36:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2799.416687 secs 2024-07-27T07:36:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:19 smithi077 ceph-mon[37153]: pgmap v1966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:36:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:19 smithi047 ceph-mon[36835]: pgmap v1966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:36:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:19 smithi029 ceph-mon[32610]: pgmap v1966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:36:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:21 smithi077 ceph-mon[37153]: pgmap v1967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T07:36:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:21 smithi047 ceph-mon[36835]: pgmap v1967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T07:36:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:21 smithi029 ceph-mon[32610]: pgmap v1967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T07:36:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2804.416829 secs 2024-07-27T07:36:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2804.416829 secs 2024-07-27T07:36:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2804.416829 secs 2024-07-27T07:36:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:23 smithi047 ceph-mon[36835]: pgmap v1968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:36:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:23 smithi029 ceph-mon[32610]: pgmap v1968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:36:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:23 smithi077 ceph-mon[37153]: pgmap v1968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:36:25.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:36:25.276+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:36:25.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:36:25.276+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:36:25.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:36:25.276+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:36:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:36:25.299+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:36:25.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:36:25.299+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:36:25.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:36:25.299+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:36:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:25 smithi077 ceph-mon[37153]: pgmap v1969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:36:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:25 smithi047 ceph-mon[36835]: pgmap v1969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:36:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:25 smithi029 ceph-mon[32610]: pgmap v1969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:36:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:27 smithi077 ceph-mon[37153]: pgmap v1970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:36:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:27 smithi047 ceph-mon[36835]: pgmap v1970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:36:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:27 smithi029 ceph-mon[32610]: pgmap v1970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:36:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2809.416954 secs 2024-07-27T07:36:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2809.416954 secs 2024-07-27T07:36:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2809.416954 secs 2024-07-27T07:36:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:29 smithi077 ceph-mon[37153]: pgmap v1971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:36:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:29 smithi047 ceph-mon[36835]: pgmap v1971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:36:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:29 smithi029 ceph-mon[32610]: pgmap v1971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:36:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:31 smithi077 ceph-mon[37153]: pgmap v1972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:31 smithi047 ceph-mon[36835]: pgmap v1972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:31 smithi029 ceph-mon[32610]: pgmap v1972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2814.417076 secs 2024-07-27T07:36:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2814.417076 secs 2024-07-27T07:36:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2814.417076 secs 2024-07-27T07:36:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:33 smithi047 ceph-mon[36835]: pgmap v1973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:36:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:33 smithi029 ceph-mon[32610]: pgmap v1973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:36:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:33 smithi077 ceph-mon[37153]: pgmap v1973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:36:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:35 smithi077 ceph-mon[37153]: pgmap v1974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:35 smithi047 ceph-mon[36835]: pgmap v1974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:35 smithi029 ceph-mon[32610]: pgmap v1974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:37 smithi077 ceph-mon[37153]: pgmap v1975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:36:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:37 smithi047 ceph-mon[36835]: pgmap v1975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:36:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:37 smithi029 ceph-mon[32610]: pgmap v1975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:36:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2819.417181 secs 2024-07-27T07:36:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2819.417181 secs 2024-07-27T07:36:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2819.417181 secs 2024-07-27T07:36:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:39 smithi077 ceph-mon[37153]: pgmap v1976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:36:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:39 smithi047 ceph-mon[36835]: pgmap v1976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:36:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:39 smithi029 ceph-mon[32610]: pgmap v1976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:36:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:36:40.288+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:36:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:36:40.288+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:36:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:36:40.288+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:36:40.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:36:40.315+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:36:40.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:36:40.315+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:36:40.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:36:40.315+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:36:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:41 smithi077 ceph-mon[37153]: pgmap v1977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:41 smithi047 ceph-mon[36835]: pgmap v1977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:41 smithi029 ceph-mon[32610]: pgmap v1977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2824.417341 secs 2024-07-27T07:36:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2824.417341 secs 2024-07-27T07:36:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2824.417341 secs 2024-07-27T07:36:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:43 smithi047 ceph-mon[36835]: pgmap v1978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:36:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:43 smithi029 ceph-mon[32610]: pgmap v1978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:36:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:43 smithi077 ceph-mon[37153]: pgmap v1978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:36:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:45 smithi077 ceph-mon[37153]: pgmap v1979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:45 smithi047 ceph-mon[36835]: pgmap v1979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:45 smithi029 ceph-mon[32610]: pgmap v1979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:36:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:47 smithi077 ceph-mon[37153]: pgmap v1980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:36:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:47 smithi047 ceph-mon[36835]: pgmap v1980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:36:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:47 smithi029 ceph-mon[32610]: pgmap v1980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:36:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2829.417474 secs 2024-07-27T07:36:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2829.417474 secs 2024-07-27T07:36:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2829.417474 secs 2024-07-27T07:36:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:49 smithi077 ceph-mon[37153]: pgmap v1981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:36:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:49 smithi047 ceph-mon[36835]: pgmap v1981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:36:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:49 smithi029 ceph-mon[32610]: pgmap v1981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:36:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:51 smithi077 ceph-mon[37153]: pgmap v1982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:36:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:51 smithi047 ceph-mon[36835]: pgmap v1982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:36:51.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:51 smithi029 ceph-mon[32610]: pgmap v1982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T07:36:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2834.417602 secs 2024-07-27T07:36:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2834.417602 secs 2024-07-27T07:36:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2834.417602 secs 2024-07-27T07:36:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:53 smithi047 ceph-mon[36835]: pgmap v1983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:36:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:53 smithi029 ceph-mon[32610]: pgmap v1983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:36:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:53 smithi077 ceph-mon[37153]: pgmap v1983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:36:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:36:55.303+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:36:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:36:55.303+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:36:55.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:36:55.303+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:36:55.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:36:55.330+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:36:55.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:36:55.330+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:36:55.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:36:55.330+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:36:55.332 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:55 smithi029 ceph-mon[32610]: pgmap v1984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:36:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:55 smithi077 ceph-mon[37153]: pgmap v1984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:36:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:55 smithi047 ceph-mon[36835]: pgmap v1984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:36:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:57 smithi077 ceph-mon[37153]: pgmap v1985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:36:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:57 smithi047 ceph-mon[36835]: pgmap v1985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:36:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:57 smithi029 ceph-mon[32610]: pgmap v1985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:36:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2839.417723 secs 2024-07-27T07:36:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2839.417723 secs 2024-07-27T07:36:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2839.417723 secs 2024-07-27T07:36:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:36:59 smithi077 ceph-mon[37153]: pgmap v1986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:36:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:36:59 smithi047 ceph-mon[36835]: pgmap v1986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:36:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:36:59 smithi029 ceph-mon[32610]: pgmap v1986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:37:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:01 smithi077 ceph-mon[37153]: pgmap v1987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:01 smithi047 ceph-mon[36835]: pgmap v1987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:01 smithi029 ceph-mon[32610]: pgmap v1987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2844.417839 secs 2024-07-27T07:37:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2844.417839 secs 2024-07-27T07:37:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2844.417839 secs 2024-07-27T07:37:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:03 smithi047 ceph-mon[36835]: pgmap v1988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:03 smithi029 ceph-mon[32610]: pgmap v1988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:03 smithi077 ceph-mon[37153]: pgmap v1988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:05 smithi077 ceph-mon[37153]: pgmap v1989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:05 smithi047 ceph-mon[36835]: pgmap v1989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:05 smithi029 ceph-mon[32610]: pgmap v1989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:07 smithi077 ceph-mon[37153]: pgmap v1990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:37:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:07 smithi047 ceph-mon[36835]: pgmap v1990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:37:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:07 smithi029 ceph-mon[32610]: pgmap v1990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:37:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2849.417961 secs 2024-07-27T07:37:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2849.417961 secs 2024-07-27T07:37:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2849.417961 secs 2024-07-27T07:37:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:09 smithi077 ceph-mon[37153]: pgmap v1991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:37:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:09 smithi047 ceph-mon[36835]: pgmap v1991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:37:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:09 smithi029 ceph-mon[32610]: pgmap v1991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:37:10.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:10.306+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:37:10.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:10.306+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:37:10.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:10.306+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:37:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:10.337+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:37:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:10.337+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:37:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:10.337+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:37:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:11 smithi077 ceph-mon[37153]: pgmap v1992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:11 smithi047 ceph-mon[36835]: pgmap v1992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:11 smithi029 ceph-mon[32610]: pgmap v1992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2854.418042 secs 2024-07-27T07:37:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2854.418042 secs 2024-07-27T07:37:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2854.418042 secs 2024-07-27T07:37:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:13 smithi047 ceph-mon[36835]: pgmap v1993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:37:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:13 smithi029 ceph-mon[32610]: pgmap v1993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:37:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:13 smithi077 ceph-mon[37153]: pgmap v1993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:37:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:37:14.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:37:14.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:37:14.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:37:14.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:14.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:37:14.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:37:14.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:37:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:15 smithi047 ceph-mon[36835]: pgmap v1994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T07:37:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:15 smithi047 ceph-mon[36835]: pgmap v1995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:37:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:15 smithi029 ceph-mon[32610]: pgmap v1994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T07:37:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:15 smithi029 ceph-mon[32610]: pgmap v1995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:37:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:15 smithi077 ceph-mon[37153]: pgmap v1994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T07:37:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:15 smithi077 ceph-mon[37153]: pgmap v1995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:37:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:17 smithi047 ceph-mon[36835]: pgmap v1996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:37:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2859.418165 secs 2024-07-27T07:37:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:17 smithi029 ceph-mon[32610]: pgmap v1996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:37:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2859.418165 secs 2024-07-27T07:37:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:17 smithi077 ceph-mon[37153]: pgmap v1996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:37:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2859.418165 secs 2024-07-27T07:37:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:19 smithi047 ceph-mon[36835]: pgmap v1997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:37:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:19 smithi029 ceph-mon[32610]: pgmap v1997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:37:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:19 smithi077 ceph-mon[37153]: pgmap v1997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:37:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:21 smithi029 ceph-mon[32610]: pgmap v1998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T07:37:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:21 smithi047 ceph-mon[36835]: pgmap v1998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T07:37:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:21 smithi077 ceph-mon[37153]: pgmap v1998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T07:37:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2864.418279 secs 2024-07-27T07:37:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2864.418279 secs 2024-07-27T07:37:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2864.418279 secs 2024-07-27T07:37:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:23 smithi047 ceph-mon[36835]: pgmap v1999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T07:37:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:23 smithi029 ceph-mon[32610]: pgmap v1999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T07:37:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:23 smithi077 ceph-mon[37153]: pgmap v1999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T07:37:25.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:25.320+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:37:25.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:25.320+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:37:25.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:25.320+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:37:25.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:25.349+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:37:25.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:25.349+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:37:25.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:25.349+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:37:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:25 smithi047 ceph-mon[36835]: pgmap v2000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:37:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:25 smithi029 ceph-mon[32610]: pgmap v2000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:37:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:25 smithi077 ceph-mon[37153]: pgmap v2000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:37:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:27 smithi047 ceph-mon[36835]: pgmap v2001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2869.418358 secs 2024-07-27T07:37:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:27 smithi029 ceph-mon[32610]: pgmap v2001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2869.418358 secs 2024-07-27T07:37:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:27 smithi077 ceph-mon[37153]: pgmap v2001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2869.418358 secs 2024-07-27T07:37:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:29 smithi047 ceph-mon[36835]: pgmap v2002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:29 smithi029 ceph-mon[32610]: pgmap v2002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:29 smithi077 ceph-mon[37153]: pgmap v2002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:31 smithi047 ceph-mon[36835]: pgmap v2003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:31 smithi029 ceph-mon[32610]: pgmap v2003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:31 smithi077 ceph-mon[37153]: pgmap v2003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2874.418462 secs 2024-07-27T07:37:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2874.418462 secs 2024-07-27T07:37:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2874.418462 secs 2024-07-27T07:37:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:33 smithi047 ceph-mon[36835]: pgmap v2004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:37:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:33 smithi029 ceph-mon[32610]: pgmap v2004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:37:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:33 smithi077 ceph-mon[37153]: pgmap v2004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:37:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:35 smithi029 ceph-mon[32610]: pgmap v2005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:37:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:35 smithi077 ceph-mon[37153]: pgmap v2005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:37:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:35 smithi047 ceph-mon[36835]: pgmap v2005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:37:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:37 smithi029 ceph-mon[32610]: pgmap v2006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2879.418618 secs 2024-07-27T07:37:37.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:37 smithi077 ceph-mon[37153]: pgmap v2006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:37.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2879.418618 secs 2024-07-27T07:37:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:37 smithi047 ceph-mon[36835]: pgmap v2006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2879.418618 secs 2024-07-27T07:37:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:39 smithi029 ceph-mon[32610]: pgmap v2007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:39 smithi077 ceph-mon[37153]: pgmap v2007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:39 smithi047 ceph-mon[36835]: pgmap v2007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:40.326+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:37:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:40.326+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:37:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:40.326+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:37:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:40.364+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:37:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:40.364+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:37:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:40.364+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:37:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:41 smithi029 ceph-mon[32610]: pgmap v2008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:41 smithi077 ceph-mon[37153]: pgmap v2008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:41 smithi047 ceph-mon[36835]: pgmap v2008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:42 smithi077 ceph-mon[37153]: 1 slow requests, 1 included below; oldest blocked for > 2884.418729 secs 2024-07-27T07:37:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:42 smithi077 ceph-mon[37153]: slow request 2884.418728 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T07:37:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:42 smithi047 ceph-mon[36835]: 1 slow requests, 1 included below; oldest blocked for > 2884.418729 secs 2024-07-27T07:37:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:42 smithi047 ceph-mon[36835]: slow request 2884.418728 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T07:37:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:42 smithi029 ceph-mon[32610]: 1 slow requests, 1 included below; oldest blocked for > 2884.418729 secs 2024-07-27T07:37:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:42 smithi029 ceph-mon[32610]: slow request 2884.418728 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T07:37:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:43.065+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:37:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:43.065+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:37:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:43.065+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:37:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:43.065+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:37:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:43 smithi077 ceph-mon[37153]: pgmap v2009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:37:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:43 smithi047 ceph-mon[36835]: pgmap v2009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:37:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:43 smithi029 ceph-mon[32610]: pgmap v2009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:37:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:45 smithi077 ceph-mon[37153]: pgmap v2010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:37:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:45 smithi047 ceph-mon[36835]: pgmap v2010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:37:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:45 smithi029 ceph-mon[32610]: pgmap v2010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:37:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:47 smithi077 ceph-mon[37153]: pgmap v2011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2889.418858 secs 2024-07-27T07:37:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:47 smithi047 ceph-mon[36835]: pgmap v2011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2889.418858 secs 2024-07-27T07:37:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:47 smithi029 ceph-mon[32610]: pgmap v2011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2889.418858 secs 2024-07-27T07:37:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:49 smithi077 ceph-mon[37153]: pgmap v2012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:49 smithi047 ceph-mon[36835]: pgmap v2012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:49 smithi029 ceph-mon[32610]: pgmap v2012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:51 smithi077 ceph-mon[37153]: pgmap v2013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:51 smithi047 ceph-mon[36835]: pgmap v2013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:51 smithi029 ceph-mon[32610]: pgmap v2013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2894.418998 secs 2024-07-27T07:37:52.936 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2894.418998 secs 2024-07-27T07:37:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2894.418998 secs 2024-07-27T07:37:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:53 smithi077 ceph-mon[37153]: pgmap v2014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:37:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:53 smithi047 ceph-mon[36835]: pgmap v2014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:37:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:53 smithi029 ceph-mon[32610]: pgmap v2014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:37:55.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:55.341+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:37:55.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:55.341+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:37:55.344 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:37:55.341+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:37:55.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:55.380+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:37:55.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:55.380+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:37:55.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:55.380+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:37:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:55 smithi029 ceph-mon[32610]: pgmap v2015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:37:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:55 smithi077 ceph-mon[37153]: pgmap v2015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:37:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:55 smithi047 ceph-mon[36835]: pgmap v2015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:37:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:57 smithi077 ceph-mon[37153]: pgmap v2016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2899.419122 secs 2024-07-27T07:37:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:57 smithi047 ceph-mon[36835]: pgmap v2016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2899.419122 secs 2024-07-27T07:37:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:57 smithi029 ceph-mon[32610]: pgmap v2016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:37:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2899.419122 secs 2024-07-27T07:37:58.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:58.068+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff240065b0 con 0x7eff50104ab0 2024-07-27T07:37:58.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:58.068+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff240065b0 con 0x7eff50104ab0 2024-07-27T07:37:58.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:58.070+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:37:58.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:37:58.070+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:37:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:37:59 smithi077 ceph-mon[37153]: pgmap v2017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:37:59 smithi047 ceph-mon[36835]: pgmap v2017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:37:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:37:59 smithi029 ceph-mon[32610]: pgmap v2017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:38:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:01 smithi077 ceph-mon[37153]: pgmap v2018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:01 smithi047 ceph-mon[36835]: pgmap v2018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:01 smithi029 ceph-mon[32610]: pgmap v2018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2904.419198 secs 2024-07-27T07:38:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2904.419198 secs 2024-07-27T07:38:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2904.419198 secs 2024-07-27T07:38:03.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:03 smithi047 ceph-mon[36835]: pgmap v2019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:38:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:03 smithi077 ceph-mon[37153]: pgmap v2019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:38:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:03 smithi029 ceph-mon[32610]: pgmap v2019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:38:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:05 smithi077 ceph-mon[37153]: pgmap v2020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:05 smithi047 ceph-mon[36835]: pgmap v2020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:05 smithi029 ceph-mon[32610]: pgmap v2020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:07 smithi077 ceph-mon[37153]: pgmap v2021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2909.419324 secs 2024-07-27T07:38:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:07 smithi047 ceph-mon[36835]: pgmap v2021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2909.419324 secs 2024-07-27T07:38:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:07 smithi029 ceph-mon[32610]: pgmap v2021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2909.419324 secs 2024-07-27T07:38:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:09 smithi077 ceph-mon[37153]: pgmap v2022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:38:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:09 smithi047 ceph-mon[36835]: pgmap v2022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:38:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:09 smithi029 ceph-mon[32610]: pgmap v2022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:38:10.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:38:10.358+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:38:10.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:38:10.358+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:38:10.359 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:38:10.358+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:38:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:38:10.382+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:38:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:38:10.382+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:38:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:38:10.382+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:38:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:11 smithi077 ceph-mon[37153]: pgmap v2023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:11 smithi047 ceph-mon[36835]: pgmap v2023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:11 smithi029 ceph-mon[32610]: pgmap v2023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2914.419469 secs 2024-07-27T07:38:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2914.419469 secs 2024-07-27T07:38:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2914.419469 secs 2024-07-27T07:38:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:13 smithi077 ceph-mon[37153]: pgmap v2024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:38:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:13 smithi047 ceph-mon[36835]: pgmap v2024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:38:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:13 smithi029 ceph-mon[32610]: pgmap v2024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:38:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:38:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:38:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:38:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:15 smithi077 ceph-mon[37153]: pgmap v2025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:38:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:38:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:15 smithi077 ceph-mon[37153]: pgmap v2026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:38:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:15 smithi077 ceph-mon[37153]: pgmap v2027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:38:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:38:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:15 smithi047 ceph-mon[36835]: pgmap v2025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:38:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:38:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:15 smithi047 ceph-mon[36835]: pgmap v2026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:38:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:15 smithi047 ceph-mon[36835]: pgmap v2027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:38:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:38:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:15 smithi029 ceph-mon[32610]: pgmap v2025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:38:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:38:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:15 smithi029 ceph-mon[32610]: pgmap v2026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:38:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:15 smithi029 ceph-mon[32610]: pgmap v2027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:38:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:38:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:17 smithi029 ceph-mon[32610]: pgmap v2028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:38:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2919.419605 secs 2024-07-27T07:38:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:17 smithi077 ceph-mon[37153]: pgmap v2028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:38:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2919.419605 secs 2024-07-27T07:38:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:17 smithi047 ceph-mon[36835]: pgmap v2028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:38:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2919.419605 secs 2024-07-27T07:38:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:19 smithi029 ceph-mon[32610]: pgmap v2029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:38:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:19 smithi077 ceph-mon[37153]: pgmap v2029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:38:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:19 smithi047 ceph-mon[36835]: pgmap v2029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:38:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:21 smithi029 ceph-mon[32610]: pgmap v2030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:38:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:21 smithi077 ceph-mon[37153]: pgmap v2030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:38:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:21 smithi047 ceph-mon[36835]: pgmap v2030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:38:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2924.419701 secs 2024-07-27T07:38:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2924.419701 secs 2024-07-27T07:38:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2924.419701 secs 2024-07-27T07:38:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:23 smithi029 ceph-mon[32610]: pgmap v2031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:38:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:23 smithi077 ceph-mon[37153]: pgmap v2031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:38:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:23 smithi047 ceph-mon[36835]: pgmap v2031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:38:25.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:38:25.372+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:38:25.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:38:25.372+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:38:25.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:38:25.372+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:38:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:38:25.384+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:38:25.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:38:25.384+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:38:25.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:38:25.384+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:38:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:25 smithi077 ceph-mon[37153]: pgmap v2032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:38:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:25 smithi047 ceph-mon[36835]: pgmap v2032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:38:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:25 smithi029 ceph-mon[32610]: pgmap v2032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:38:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:27 smithi077 ceph-mon[37153]: pgmap v2033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:38:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2929.419827 secs 2024-07-27T07:38:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:27 smithi047 ceph-mon[36835]: pgmap v2033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:38:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2929.419827 secs 2024-07-27T07:38:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:27 smithi029 ceph-mon[32610]: pgmap v2033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:38:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2929.419827 secs 2024-07-27T07:38:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:29 smithi077 ceph-mon[37153]: pgmap v2034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:29 smithi047 ceph-mon[36835]: pgmap v2034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:29 smithi029 ceph-mon[32610]: pgmap v2034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:31 smithi077 ceph-mon[37153]: pgmap v2035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:31 smithi047 ceph-mon[36835]: pgmap v2035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:31 smithi029 ceph-mon[32610]: pgmap v2035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:33.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2934.419948 secs 2024-07-27T07:38:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2934.419948 secs 2024-07-27T07:38:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2934.419948 secs 2024-07-27T07:38:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:33 smithi077 ceph-mon[37153]: pgmap v2036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:38:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:33 smithi047 ceph-mon[36835]: pgmap v2036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:38:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:33 smithi029 ceph-mon[32610]: pgmap v2036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:38:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:35 smithi077 ceph-mon[37153]: pgmap v2037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:38:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:35 smithi047 ceph-mon[36835]: pgmap v2037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:38:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:35 smithi029 ceph-mon[32610]: pgmap v2037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:38:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:37 smithi077 ceph-mon[37153]: pgmap v2038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:38:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2939.420034 secs 2024-07-27T07:38:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:37 smithi047 ceph-mon[36835]: pgmap v2038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:38:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2939.420034 secs 2024-07-27T07:38:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:37 smithi029 ceph-mon[32610]: pgmap v2038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:38:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2939.420034 secs 2024-07-27T07:38:40.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:39 smithi029 ceph-mon[32610]: pgmap v2039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:39 smithi077 ceph-mon[37153]: pgmap v2039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:39 smithi047 ceph-mon[36835]: pgmap v2039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:38:40.387+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:38:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:38:40.387+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:38:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:38:40.387+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:38:40.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:38:40.387+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:38:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:38:40.387+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:38:40.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:38:40.387+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:38:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:41 smithi077 ceph-mon[37153]: pgmap v2040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:41 smithi047 ceph-mon[36835]: pgmap v2040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:41 smithi029 ceph-mon[32610]: pgmap v2040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2944.420104 secs 2024-07-27T07:38:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2944.420104 secs 2024-07-27T07:38:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2944.420104 secs 2024-07-27T07:38:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:43 smithi077 ceph-mon[37153]: pgmap v2041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:38:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:43 smithi047 ceph-mon[36835]: pgmap v2041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:38:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:43 smithi029 ceph-mon[32610]: pgmap v2041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:38:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:45 smithi077 ceph-mon[37153]: pgmap v2042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:38:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:45 smithi047 ceph-mon[36835]: pgmap v2042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:38:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:45 smithi029 ceph-mon[32610]: pgmap v2042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:38:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:47 smithi077 ceph-mon[37153]: pgmap v2043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:38:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2949.420220 secs 2024-07-27T07:38:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:47 smithi047 ceph-mon[36835]: pgmap v2043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:38:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2949.420220 secs 2024-07-27T07:38:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:47 smithi029 ceph-mon[32610]: pgmap v2043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:38:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2949.420220 secs 2024-07-27T07:38:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:49 smithi077 ceph-mon[37153]: pgmap v2044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:49 smithi047 ceph-mon[36835]: pgmap v2044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:49 smithi029 ceph-mon[32610]: pgmap v2044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:38:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:51 smithi077 ceph-mon[37153]: pgmap v2045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:51 smithi047 ceph-mon[36835]: pgmap v2045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:51 smithi029 ceph-mon[32610]: pgmap v2045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:38:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2954.420342 secs 2024-07-27T07:38:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2954.420342 secs 2024-07-27T07:38:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2954.420342 secs 2024-07-27T07:38:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:53 smithi077 ceph-mon[37153]: pgmap v2046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:38:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:53 smithi047 ceph-mon[36835]: pgmap v2046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:38:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:53 smithi029 ceph-mon[32610]: pgmap v2046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:38:55.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:38:55.397+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:38:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:38:55.397+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:38:55.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:38:55.397+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:38:55.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:38:55.402+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:38:55.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:38:55.402+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:38:55.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:38:55.402+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:38:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:55 smithi077 ceph-mon[37153]: pgmap v2047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:38:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:55 smithi047 ceph-mon[36835]: pgmap v2047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:38:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:55 smithi029 ceph-mon[32610]: pgmap v2047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:38:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:57 smithi077 ceph-mon[37153]: pgmap v2048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:38:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2959.420490 secs 2024-07-27T07:38:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:57 smithi047 ceph-mon[36835]: pgmap v2048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:38:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2959.420490 secs 2024-07-27T07:38:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:57 smithi029 ceph-mon[32610]: pgmap v2048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:38:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2959.420490 secs 2024-07-27T07:39:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:38:59 smithi077 ceph-mon[37153]: pgmap v2049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:39:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:38:59 smithi047 ceph-mon[36835]: pgmap v2049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:39:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:38:59 smithi029 ceph-mon[32610]: pgmap v2049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:39:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:01 smithi077 ceph-mon[37153]: pgmap v2050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:01 smithi047 ceph-mon[36835]: pgmap v2050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:01 smithi029 ceph-mon[32610]: pgmap v2050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:03.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2964.420611 secs 2024-07-27T07:39:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2964.420611 secs 2024-07-27T07:39:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2964.420611 secs 2024-07-27T07:39:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:03 smithi077 ceph-mon[37153]: pgmap v2051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:39:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:03 smithi047 ceph-mon[36835]: pgmap v2051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:39:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:03 smithi029 ceph-mon[32610]: pgmap v2051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:39:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:05 smithi077 ceph-mon[37153]: pgmap v2052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:39:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:05 smithi047 ceph-mon[36835]: pgmap v2052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:39:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:05 smithi029 ceph-mon[32610]: pgmap v2052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:39:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:07 smithi077 ceph-mon[37153]: pgmap v2053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2969.420762 secs 2024-07-27T07:39:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:07 smithi047 ceph-mon[36835]: pgmap v2053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2969.420762 secs 2024-07-27T07:39:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:07 smithi029 ceph-mon[32610]: pgmap v2053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2969.420762 secs 2024-07-27T07:39:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:09 smithi077 ceph-mon[37153]: pgmap v2054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:39:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:09 smithi047 ceph-mon[36835]: pgmap v2054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:39:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:09 smithi029 ceph-mon[32610]: pgmap v2054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:39:10.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:10.413+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:39:10.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:10.413+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:39:10.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:10.413+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:39:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:10.419+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:39:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:10.419+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:39:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:10.419+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:39:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:11 smithi077 ceph-mon[37153]: pgmap v2055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:11 smithi047 ceph-mon[36835]: pgmap v2055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:11 smithi029 ceph-mon[32610]: pgmap v2055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:13.079+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:39:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:13.079+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:39:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:13.079+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:39:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:13.079+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:39:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2974.420903 secs 2024-07-27T07:39:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2974.420903 secs 2024-07-27T07:39:13.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2974.420903 secs 2024-07-27T07:39:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:13 smithi077 ceph-mon[37153]: pgmap v2056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:39:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:13 smithi047 ceph-mon[36835]: pgmap v2056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:39:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:13 smithi029 ceph-mon[32610]: pgmap v2056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:39:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:39:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:39:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:39:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:15 smithi077 ceph-mon[37153]: pgmap v2057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:39:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:39:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:39:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:39:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:15 smithi047 ceph-mon[36835]: pgmap v2057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:39:16.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:39:16.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:39:16.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:39:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:15 smithi029 ceph-mon[32610]: pgmap v2057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:39:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:39:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:39:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:39:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:16 smithi077 ceph-mon[37153]: pgmap v2058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:39:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:16 smithi077 ceph-mon[37153]: pgmap v2059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:39:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:16 smithi047 ceph-mon[36835]: pgmap v2058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:39:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:16 smithi047 ceph-mon[36835]: pgmap v2059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:39:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:16 smithi029 ceph-mon[32610]: pgmap v2058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:39:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:16 smithi029 ceph-mon[32610]: pgmap v2059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:39:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2979.421029 secs 2024-07-27T07:39:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2979.421029 secs 2024-07-27T07:39:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2979.421029 secs 2024-07-27T07:39:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:18 smithi077 ceph-mon[37153]: pgmap v2060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:39:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:18 smithi047 ceph-mon[36835]: pgmap v2060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:39:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:18 smithi029 ceph-mon[32610]: pgmap v2060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:39:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:20 smithi077 ceph-mon[37153]: pgmap v2061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:39:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:20 smithi047 ceph-mon[36835]: pgmap v2061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:39:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:20 smithi029 ceph-mon[32610]: pgmap v2061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:39:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:22 smithi077 ceph-mon[37153]: pgmap v2062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:39:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2984.421159 secs 2024-07-27T07:39:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:22 smithi047 ceph-mon[36835]: pgmap v2062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:39:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2984.421159 secs 2024-07-27T07:39:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:22 smithi029 ceph-mon[32610]: pgmap v2062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:39:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2984.421159 secs 2024-07-27T07:39:25.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:24 smithi077 ceph-mon[37153]: pgmap v2063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:39:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:24 smithi047 ceph-mon[36835]: pgmap v2063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:39:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:24 smithi029 ceph-mon[32610]: pgmap v2063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:39:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:25.427+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:39:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:25.427+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:39:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:25.427+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:25.427+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:25.427+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:39:25.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:25.427+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:39:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:26 smithi077 ceph-mon[37153]: pgmap v2064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:39:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:26 smithi047 ceph-mon[36835]: pgmap v2064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:39:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:26 smithi029 ceph-mon[32610]: pgmap v2064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:39:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:28.080+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:39:28.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:28.080+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:39:28.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:28.083+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:39:28.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:28.083+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:39:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2989.421288 secs 2024-07-27T07:39:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2989.421288 secs 2024-07-27T07:39:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2989.421288 secs 2024-07-27T07:39:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:28 smithi077 ceph-mon[37153]: pgmap v2065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:28 smithi047 ceph-mon[36835]: pgmap v2065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:28 smithi029 ceph-mon[32610]: pgmap v2065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:30 smithi047 ceph-mon[36835]: pgmap v2066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:39:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:30 smithi029 ceph-mon[32610]: pgmap v2066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:39:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:30 smithi077 ceph-mon[37153]: pgmap v2066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:39:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:32 smithi047 ceph-mon[36835]: pgmap v2067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2994.421351 secs 2024-07-27T07:39:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:32 smithi029 ceph-mon[32610]: pgmap v2067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2994.421351 secs 2024-07-27T07:39:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:32 smithi077 ceph-mon[37153]: pgmap v2067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2994.421351 secs 2024-07-27T07:39:35.160 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:34 smithi077 ceph-mon[37153]: pgmap v2068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:39:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:34 smithi047 ceph-mon[36835]: pgmap v2068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:39:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:34 smithi029 ceph-mon[32610]: pgmap v2068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:39:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:36 smithi047 ceph-mon[36835]: pgmap v2069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:36 smithi029 ceph-mon[32610]: pgmap v2069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:36 smithi077 ceph-mon[37153]: pgmap v2069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 2999.421460 secs 2024-07-27T07:39:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 2999.421460 secs 2024-07-27T07:39:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 2999.421460 secs 2024-07-27T07:39:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:38 smithi047 ceph-mon[36835]: pgmap v2070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:38 smithi029 ceph-mon[32610]: pgmap v2070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:38 smithi077 ceph-mon[37153]: pgmap v2070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:40.432+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:39:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:40.432+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:39:40.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:40.432+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:39:40.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:40.443+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:39:40.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:40.443+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:39:40.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:40.443+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:39:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:40 smithi047 ceph-mon[36835]: pgmap v2071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:39:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:40 smithi029 ceph-mon[32610]: pgmap v2071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:39:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:40 smithi077 ceph-mon[37153]: pgmap v2071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:39:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:42 smithi047 ceph-mon[36835]: pgmap v2072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3004.421561 secs 2024-07-27T07:39:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:42 smithi029 ceph-mon[32610]: pgmap v2072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3004.421561 secs 2024-07-27T07:39:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:42 smithi077 ceph-mon[37153]: pgmap v2072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3004.421561 secs 2024-07-27T07:39:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:44 smithi047 ceph-mon[36835]: pgmap v2073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:39:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:44 smithi029 ceph-mon[32610]: pgmap v2073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:39:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:44 smithi077 ceph-mon[37153]: pgmap v2073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:39:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:46 smithi047 ceph-mon[36835]: pgmap v2074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:46 smithi029 ceph-mon[32610]: pgmap v2074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:46 smithi077 ceph-mon[37153]: pgmap v2074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3009.421683 secs 2024-07-27T07:39:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3009.421683 secs 2024-07-27T07:39:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3009.421683 secs 2024-07-27T07:39:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:48 smithi047 ceph-mon[36835]: pgmap v2075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:48 smithi029 ceph-mon[32610]: pgmap v2075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:48 smithi077 ceph-mon[37153]: pgmap v2075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:50 smithi047 ceph-mon[36835]: pgmap v2076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:39:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:50 smithi029 ceph-mon[32610]: pgmap v2076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:39:51.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:50 smithi077 ceph-mon[37153]: pgmap v2076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:39:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:52 smithi047 ceph-mon[36835]: pgmap v2077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3014.421822 secs 2024-07-27T07:39:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:52 smithi029 ceph-mon[32610]: pgmap v2077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3014.421822 secs 2024-07-27T07:39:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:52 smithi077 ceph-mon[37153]: pgmap v2077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3014.421822 secs 2024-07-27T07:39:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:54 smithi047 ceph-mon[36835]: pgmap v2078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:39:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:54 smithi029 ceph-mon[32610]: pgmap v2078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:39:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:54 smithi077 ceph-mon[37153]: pgmap v2078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:39:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:55.445+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:39:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:55.445+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:39:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:39:55.445+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:39:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:55.458+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:39:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:55.458+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:39:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:39:55.458+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:39:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:56 smithi047 ceph-mon[36835]: pgmap v2079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:56 smithi029 ceph-mon[32610]: pgmap v2079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:56 smithi077 ceph-mon[37153]: pgmap v2079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:39:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3019.421891 secs 2024-07-27T07:39:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3019.421891 secs 2024-07-27T07:39:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3019.421891 secs 2024-07-27T07:39:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:39:58 smithi047 ceph-mon[36835]: pgmap v2080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:39:58 smithi029 ceph-mon[32610]: pgmap v2080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:39:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:39:58 smithi077 ceph-mon[37153]: pgmap v2080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:00 smithi047 ceph-mon[36835]: pgmap v2081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:40:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:40:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:00 smithi029 ceph-mon[32610]: pgmap v2081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:40:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:40:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:00 smithi077 ceph-mon[37153]: pgmap v2081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:40:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:40:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:02 smithi047 ceph-mon[36835]: pgmap v2082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3024.422013 secs 2024-07-27T07:40:03.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:02 smithi029 ceph-mon[32610]: pgmap v2082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:03.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3024.422013 secs 2024-07-27T07:40:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:02 smithi077 ceph-mon[37153]: pgmap v2082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3024.422013 secs 2024-07-27T07:40:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:04 smithi047 ceph-mon[36835]: pgmap v2083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:40:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:04 smithi029 ceph-mon[32610]: pgmap v2083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:40:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:04 smithi077 ceph-mon[37153]: pgmap v2083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T07:40:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:06 smithi047 ceph-mon[36835]: pgmap v2084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:40:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:06 smithi029 ceph-mon[32610]: pgmap v2084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:40:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:06 smithi077 ceph-mon[37153]: pgmap v2084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:40:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3029.422141 secs 2024-07-27T07:40:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3029.422141 secs 2024-07-27T07:40:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3029.422141 secs 2024-07-27T07:40:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:08 smithi047 ceph-mon[36835]: pgmap v2085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:08 smithi029 ceph-mon[32610]: pgmap v2085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:09.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:08 smithi077 ceph-mon[37153]: pgmap v2085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:10.461+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:40:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:10.461+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:40:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:10.461+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:40:10.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:10.474+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:40:10.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:10.474+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:40:10.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:10.474+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:40:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:10 smithi047 ceph-mon[36835]: pgmap v2086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:40:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:10 smithi029 ceph-mon[32610]: pgmap v2086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:40:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:10 smithi077 ceph-mon[37153]: pgmap v2086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:40:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:12 smithi047 ceph-mon[36835]: pgmap v2087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3034.422275 secs 2024-07-27T07:40:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:12 smithi029 ceph-mon[32610]: pgmap v2087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3034.422275 secs 2024-07-27T07:40:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:12 smithi077 ceph-mon[37153]: pgmap v2087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3034.422275 secs 2024-07-27T07:40:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:14 smithi029 ceph-mon[32610]: pgmap v2088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:40:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:14 smithi077 ceph-mon[37153]: pgmap v2088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:40:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:14 smithi047 ceph-mon[36835]: pgmap v2088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:40:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:40:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:40:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:40:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:40:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:40:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:40:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:40:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:40:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:40:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:40:16.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:40:16.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:40:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:16 smithi029 ceph-mon[32610]: pgmap v2089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:40:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:16 smithi029 ceph-mon[32610]: pgmap v2090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:40:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:16 smithi029 ceph-mon[32610]: pgmap v2091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:40:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:16 smithi077 ceph-mon[37153]: pgmap v2089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:40:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:16 smithi077 ceph-mon[37153]: pgmap v2090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:40:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:16 smithi077 ceph-mon[37153]: pgmap v2091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:40:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:16 smithi047 ceph-mon[36835]: pgmap v2089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:40:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:16 smithi047 ceph-mon[36835]: pgmap v2090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:40:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:16 smithi047 ceph-mon[36835]: pgmap v2091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:40:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3039.422415 secs 2024-07-27T07:40:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3039.422415 secs 2024-07-27T07:40:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3039.422415 secs 2024-07-27T07:40:19.226 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:18 smithi029 ceph-mon[32610]: pgmap v2092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:40:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:18 smithi077 ceph-mon[37153]: pgmap v2092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:40:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:18 smithi047 ceph-mon[36835]: pgmap v2092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:40:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:20 smithi077 ceph-mon[37153]: pgmap v2093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s 2024-07-27T07:40:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:20 smithi047 ceph-mon[36835]: pgmap v2093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s 2024-07-27T07:40:21.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:20 smithi029 ceph-mon[32610]: pgmap v2093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s 2024-07-27T07:40:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3044.422537 secs 2024-07-27T07:40:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:22 smithi077 ceph-mon[37153]: pgmap v2094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:40:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3044.422537 secs 2024-07-27T07:40:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:22 smithi047 ceph-mon[36835]: pgmap v2094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:40:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3044.422537 secs 2024-07-27T07:40:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:22 smithi029 ceph-mon[32610]: pgmap v2094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:40:25.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:24 smithi029 ceph-mon[32610]: pgmap v2095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:40:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:24 smithi077 ceph-mon[37153]: pgmap v2095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:40:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:24 smithi047 ceph-mon[36835]: pgmap v2095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:40:25.477 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:25.476+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:40:25.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:25.476+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:40:25.478 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:25.476+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:40:25.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:25.489+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:40:25.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:25.489+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:40:25.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:25.489+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:40:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:26 smithi077 ceph-mon[37153]: pgmap v2096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:40:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:26 smithi047 ceph-mon[36835]: pgmap v2096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:26 smithi029 ceph-mon[32610]: pgmap v2096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:40:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3049.422634 secs 2024-07-27T07:40:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3049.422634 secs 2024-07-27T07:40:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3049.422634 secs 2024-07-27T07:40:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:28 smithi077 ceph-mon[37153]: pgmap v2097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:40:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:28 smithi047 ceph-mon[36835]: pgmap v2097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:40:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:28 smithi029 ceph-mon[32610]: pgmap v2097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:40:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:30 smithi077 ceph-mon[37153]: pgmap v2098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:40:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:30 smithi047 ceph-mon[36835]: pgmap v2098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:40:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:30 smithi029 ceph-mon[32610]: pgmap v2098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:40:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3054.422757 secs 2024-07-27T07:40:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:32 smithi077 ceph-mon[37153]: pgmap v2099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3054.422757 secs 2024-07-27T07:40:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:32 smithi047 ceph-mon[36835]: pgmap v2099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3054.422757 secs 2024-07-27T07:40:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:32 smithi029 ceph-mon[32610]: pgmap v2099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:35.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:34 smithi047 ceph-mon[36835]: pgmap v2100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:40:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:34 smithi077 ceph-mon[37153]: pgmap v2100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:40:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:34 smithi029 ceph-mon[32610]: pgmap v2100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:40:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:36 smithi077 ceph-mon[37153]: pgmap v2101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:40:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:36 smithi047 ceph-mon[36835]: pgmap v2101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:40:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:36 smithi029 ceph-mon[32610]: pgmap v2101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:40:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3059.422878 secs 2024-07-27T07:40:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3059.422878 secs 2024-07-27T07:40:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3059.422878 secs 2024-07-27T07:40:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:39 smithi077 ceph-mon[37153]: pgmap v2102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:40:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:39 smithi047 ceph-mon[36835]: pgmap v2102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:40:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:39 smithi029 ceph-mon[32610]: pgmap v2102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:40:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:40.491+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:40:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:40.491+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:40:40.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:40.491+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:40:40.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:40.491+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:40:40.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:40.491+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:40:40.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:40.491+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:40:41.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:41 smithi077 ceph-mon[37153]: pgmap v2103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:40:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:41 smithi047 ceph-mon[36835]: pgmap v2103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:40:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:41 smithi029 ceph-mon[32610]: pgmap v2103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:40:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:43.092+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:40:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:43.092+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:40:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:43.092+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:40:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:43.092+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:40:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3064.422999 secs 2024-07-27T07:40:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:43 smithi077 ceph-mon[37153]: pgmap v2104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3064.422999 secs 2024-07-27T07:40:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:43 smithi047 ceph-mon[36835]: pgmap v2104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3064.422999 secs 2024-07-27T07:40:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:43 smithi029 ceph-mon[32610]: pgmap v2104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:45 smithi077 ceph-mon[37153]: pgmap v2105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:40:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:45 smithi047 ceph-mon[36835]: pgmap v2105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:40:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:45 smithi029 ceph-mon[32610]: pgmap v2105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:40:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:47 smithi077 ceph-mon[37153]: pgmap v2106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:40:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:47 smithi047 ceph-mon[36835]: pgmap v2106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:40:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:47 smithi029 ceph-mon[32610]: pgmap v2106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:40:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3069.423123 secs 2024-07-27T07:40:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3069.423123 secs 2024-07-27T07:40:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3069.423123 secs 2024-07-27T07:40:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:49 smithi077 ceph-mon[37153]: pgmap v2107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:40:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:49 smithi047 ceph-mon[36835]: pgmap v2107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:40:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:49 smithi029 ceph-mon[32610]: pgmap v2107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:40:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:51 smithi077 ceph-mon[37153]: pgmap v2108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:40:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:51 smithi047 ceph-mon[36835]: pgmap v2108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:40:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:51 smithi029 ceph-mon[32610]: pgmap v2108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:40:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3074.423272 secs 2024-07-27T07:40:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:53 smithi077 ceph-mon[37153]: pgmap v2109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3074.423272 secs 2024-07-27T07:40:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:53 smithi047 ceph-mon[36835]: pgmap v2109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3074.423272 secs 2024-07-27T07:40:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:53 smithi029 ceph-mon[32610]: pgmap v2109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:40:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:55 smithi077 ceph-mon[37153]: pgmap v2110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:40:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:55 smithi047 ceph-mon[36835]: pgmap v2110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:40:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:55 smithi029 ceph-mon[32610]: pgmap v2110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:40:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:55.492+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:40:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:55.492+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:40:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:40:55.492+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:40:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:55.492+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:40:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:55.492+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:40:55.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:55.492+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:40:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:57 smithi077 ceph-mon[37153]: pgmap v2111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:40:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:57 smithi047 ceph-mon[36835]: pgmap v2111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:40:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:57 smithi029 ceph-mon[32610]: pgmap v2111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:40:58.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:58.095+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:40:58.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:58.095+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:40:58.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:58.095+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:40:58.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:40:58.095+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:40:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3079.423415 secs 2024-07-27T07:40:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3079.423415 secs 2024-07-27T07:40:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3079.423415 secs 2024-07-27T07:40:59.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:40:59 smithi029 ceph-mon[32610]: pgmap v2112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:40:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:40:59 smithi077 ceph-mon[37153]: pgmap v2112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:40:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:40:59 smithi047 ceph-mon[36835]: pgmap v2112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:41:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:01 smithi077 ceph-mon[37153]: pgmap v2113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:41:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:01 smithi047 ceph-mon[36835]: pgmap v2113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:41:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:01 smithi029 ceph-mon[32610]: pgmap v2113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:41:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3084.423538 secs 2024-07-27T07:41:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:03 smithi077 ceph-mon[37153]: pgmap v2114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3084.423538 secs 2024-07-27T07:41:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:03 smithi047 ceph-mon[36835]: pgmap v2114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3084.423538 secs 2024-07-27T07:41:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:03 smithi029 ceph-mon[32610]: pgmap v2114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:05 smithi077 ceph-mon[37153]: pgmap v2115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:05 smithi047 ceph-mon[36835]: pgmap v2115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:05 smithi029 ceph-mon[32610]: pgmap v2115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:07 smithi077 ceph-mon[37153]: pgmap v2116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:07 smithi047 ceph-mon[36835]: pgmap v2116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:07 smithi029 ceph-mon[32610]: pgmap v2116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3089.423664 secs 2024-07-27T07:41:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3089.423664 secs 2024-07-27T07:41:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3089.423664 secs 2024-07-27T07:41:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:09 smithi077 ceph-mon[37153]: pgmap v2117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:41:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:09 smithi047 ceph-mon[36835]: pgmap v2117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:41:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:09 smithi029 ceph-mon[32610]: pgmap v2117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:41:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:41:10.502+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:41:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:41:10.502+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:41:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:41:10.502+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:41:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:41:10.502+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:41:10.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:41:10.502+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:41:10.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:41:10.502+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:41:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:11 smithi077 ceph-mon[37153]: pgmap v2118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:41:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:11 smithi047 ceph-mon[36835]: pgmap v2118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:41:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:11 smithi029 ceph-mon[32610]: pgmap v2118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:41:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3094.423822 secs 2024-07-27T07:41:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:13 smithi077 ceph-mon[37153]: pgmap v2119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3094.423822 secs 2024-07-27T07:41:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:13 smithi047 ceph-mon[36835]: pgmap v2119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3094.423822 secs 2024-07-27T07:41:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:13 smithi029 ceph-mon[32610]: pgmap v2119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:15 smithi077 ceph-mon[37153]: pgmap v2120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:15 smithi047 ceph-mon[36835]: pgmap v2120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:15 smithi029 ceph-mon[32610]: pgmap v2120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:41:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:41:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:41:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:17 smithi077 ceph-mon[37153]: pgmap v2121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:41:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:41:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:17 smithi077 ceph-mon[37153]: pgmap v2122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T07:41:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:17 smithi077 ceph-mon[37153]: pgmap v2123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:41:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:41:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:17 smithi047 ceph-mon[36835]: pgmap v2121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:41:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:41:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:17 smithi047 ceph-mon[36835]: pgmap v2122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T07:41:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:17 smithi047 ceph-mon[36835]: pgmap v2123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:41:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:41:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:17 smithi029 ceph-mon[32610]: pgmap v2121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:41:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:41:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:17 smithi029 ceph-mon[32610]: pgmap v2122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T07:41:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:17 smithi029 ceph-mon[32610]: pgmap v2123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:41:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:41:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3099.423933 secs 2024-07-27T07:41:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3099.423933 secs 2024-07-27T07:41:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3099.423933 secs 2024-07-27T07:41:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:19 smithi077 ceph-mon[37153]: pgmap v2124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:41:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:19 smithi047 ceph-mon[36835]: pgmap v2124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:41:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:19 smithi029 ceph-mon[32610]: pgmap v2124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:41:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:21 smithi047 ceph-mon[36835]: pgmap v2125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:41:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:21 smithi029 ceph-mon[32610]: pgmap v2125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:41:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:21 smithi077 ceph-mon[37153]: pgmap v2125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:41:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3104.424065 secs 2024-07-27T07:41:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3104.424065 secs 2024-07-27T07:41:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3104.424065 secs 2024-07-27T07:41:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:23 smithi047 ceph-mon[36835]: pgmap v2126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T07:41:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:23 smithi029 ceph-mon[32610]: pgmap v2126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T07:41:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:23 smithi077 ceph-mon[37153]: pgmap v2126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T07:41:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:41:25.516+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:41:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:41:25.516+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:41:25.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:41:25.516+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:41:25.516+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:41:25.516+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:41:25.516+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:41:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:25 smithi047 ceph-mon[36835]: pgmap v2127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:41:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:25 smithi029 ceph-mon[32610]: pgmap v2127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:41:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:25 smithi077 ceph-mon[37153]: pgmap v2127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:41:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:27 smithi047 ceph-mon[36835]: pgmap v2128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:41:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3109.424191 secs 2024-07-27T07:41:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:27 smithi029 ceph-mon[32610]: pgmap v2128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:41:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3109.424191 secs 2024-07-27T07:41:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:27 smithi077 ceph-mon[37153]: pgmap v2128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:41:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3109.424191 secs 2024-07-27T07:41:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:29 smithi047 ceph-mon[36835]: pgmap v2129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:29 smithi029 ceph-mon[32610]: pgmap v2129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:29 smithi077 ceph-mon[37153]: pgmap v2129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:31 smithi047 ceph-mon[36835]: pgmap v2130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:31 smithi029 ceph-mon[32610]: pgmap v2130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:31 smithi077 ceph-mon[37153]: pgmap v2130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3114.424313 secs 2024-07-27T07:41:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3114.424313 secs 2024-07-27T07:41:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3114.424313 secs 2024-07-27T07:41:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:33 smithi047 ceph-mon[36835]: pgmap v2131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:41:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:33 smithi029 ceph-mon[32610]: pgmap v2131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:41:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:33 smithi077 ceph-mon[37153]: pgmap v2131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:41:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:35 smithi047 ceph-mon[36835]: pgmap v2132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:41:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:35 smithi029 ceph-mon[32610]: pgmap v2132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:41:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:35 smithi077 ceph-mon[37153]: pgmap v2132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:41:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:37 smithi047 ceph-mon[36835]: pgmap v2133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3119.424437 secs 2024-07-27T07:41:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:37 smithi029 ceph-mon[32610]: pgmap v2133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3119.424437 secs 2024-07-27T07:41:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:37 smithi077 ceph-mon[37153]: pgmap v2133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3119.424437 secs 2024-07-27T07:41:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:39 smithi047 ceph-mon[36835]: pgmap v2134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:39 smithi029 ceph-mon[32610]: pgmap v2134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:39 smithi077 ceph-mon[37153]: pgmap v2134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:41:40.526+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:41:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:41:40.526+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:41:40.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:41:40.526+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:41:40.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:41:40.531+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:41:40.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:41:40.531+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:41:40.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:41:40.531+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:41:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:41 smithi047 ceph-mon[36835]: pgmap v2135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:41 smithi029 ceph-mon[32610]: pgmap v2135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:41 smithi077 ceph-mon[37153]: pgmap v2135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3124.424578 secs 2024-07-27T07:41:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3124.424578 secs 2024-07-27T07:41:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3124.424578 secs 2024-07-27T07:41:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:43 smithi029 ceph-mon[32610]: pgmap v2136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:41:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:43 smithi077 ceph-mon[37153]: pgmap v2136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:41:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:43 smithi047 ceph-mon[36835]: pgmap v2136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:41:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:45 smithi029 ceph-mon[32610]: pgmap v2137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:41:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:45 smithi077 ceph-mon[37153]: pgmap v2137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:41:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:45 smithi047 ceph-mon[36835]: pgmap v2137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T07:41:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:47 smithi029 ceph-mon[32610]: pgmap v2138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3129.424696 secs 2024-07-27T07:41:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:47 smithi077 ceph-mon[37153]: pgmap v2138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3129.424696 secs 2024-07-27T07:41:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:47 smithi047 ceph-mon[36835]: pgmap v2138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3129.424696 secs 2024-07-27T07:41:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:49 smithi029 ceph-mon[32610]: pgmap v2139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:49 smithi077 ceph-mon[37153]: pgmap v2139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:49 smithi047 ceph-mon[36835]: pgmap v2139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:51 smithi077 ceph-mon[37153]: pgmap v2140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:51 smithi047 ceph-mon[36835]: pgmap v2140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:51 smithi029 ceph-mon[32610]: pgmap v2140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:41:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3134.424869 secs 2024-07-27T07:41:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3134.424869 secs 2024-07-27T07:41:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3134.424869 secs 2024-07-27T07:41:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:53 smithi077 ceph-mon[37153]: pgmap v2141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:41:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:53 smithi047 ceph-mon[36835]: pgmap v2141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:41:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:53 smithi029 ceph-mon[32610]: pgmap v2141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:41:55.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:41:55.532+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:41:55.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:41:55.532+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:41:55.533 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:41:55.532+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:41:55.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:41:55.541+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:41:55.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:41:55.541+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:41:55.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:41:55.541+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:41:55.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:55 smithi029 ceph-mon[32610]: pgmap v2142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:41:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:55 smithi077 ceph-mon[37153]: pgmap v2142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:41:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:55 smithi047 ceph-mon[36835]: pgmap v2142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:41:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:57 smithi077 ceph-mon[37153]: pgmap v2143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3139.424979 secs 2024-07-27T07:41:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:57 smithi047 ceph-mon[36835]: pgmap v2143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3139.424979 secs 2024-07-27T07:41:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:57 smithi029 ceph-mon[32610]: pgmap v2143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:41:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3139.424979 secs 2024-07-27T07:41:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:41:59 smithi077 ceph-mon[37153]: pgmap v2144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:41:59 smithi047 ceph-mon[36835]: pgmap v2144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:41:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:41:59 smithi029 ceph-mon[32610]: pgmap v2144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:01.807 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:01 smithi047 ceph-mon[36835]: pgmap v2145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:42:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:01 smithi077 ceph-mon[37153]: pgmap v2145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:42:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:01 smithi029 ceph-mon[32610]: pgmap v2145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:42:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3144.425114 secs 2024-07-27T07:42:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3144.425114 secs 2024-07-27T07:42:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3144.425114 secs 2024-07-27T07:42:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:03 smithi077 ceph-mon[37153]: pgmap v2146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:03 smithi047 ceph-mon[36835]: pgmap v2146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:03 smithi029 ceph-mon[32610]: pgmap v2146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:05.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:05 smithi047 ceph-mon[36835]: pgmap v2147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:05 smithi077 ceph-mon[37153]: pgmap v2147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:05 smithi029 ceph-mon[32610]: pgmap v2147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:07 smithi077 ceph-mon[37153]: pgmap v2148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3149.425227 secs 2024-07-27T07:42:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:07 smithi047 ceph-mon[36835]: pgmap v2148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3149.425227 secs 2024-07-27T07:42:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:07 smithi029 ceph-mon[32610]: pgmap v2148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3149.425227 secs 2024-07-27T07:42:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:09 smithi077 ceph-mon[37153]: pgmap v2149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:09 smithi047 ceph-mon[36835]: pgmap v2149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:09 smithi029 ceph-mon[32610]: pgmap v2149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:42:10.537+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:42:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:42:10.537+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:42:10.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:42:10.537+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:42:10.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:42:10.551+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:42:10.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:42:10.551+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:42:10.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:42:10.551+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:42:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:11 smithi047 ceph-mon[36835]: pgmap v2150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:42:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:11 smithi029 ceph-mon[32610]: pgmap v2150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:42:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:11 smithi077 ceph-mon[37153]: pgmap v2150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:42:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3154.425354 secs 2024-07-27T07:42:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3154.425354 secs 2024-07-27T07:42:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3154.425354 secs 2024-07-27T07:42:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:13 smithi047 ceph-mon[36835]: pgmap v2151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:13 smithi029 ceph-mon[32610]: pgmap v2151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:13 smithi077 ceph-mon[37153]: pgmap v2151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:15 smithi047 ceph-mon[36835]: pgmap v2152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:15 smithi029 ceph-mon[32610]: pgmap v2152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:15 smithi077 ceph-mon[37153]: pgmap v2152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:17.583 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:17 smithi047 ceph-mon[36835]: pgmap v2153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:17.583 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:42:17.583 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3159.425424 secs 2024-07-27T07:42:17.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:17 smithi029 ceph-mon[32610]: pgmap v2153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:17.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:42:17.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3159.425424 secs 2024-07-27T07:42:17.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:17 smithi077 ceph-mon[37153]: pgmap v2153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:17.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:42:17.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3159.425424 secs 2024-07-27T07:42:19.688 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:19 smithi047 ceph-mon[36835]: pgmap v2154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:19 smithi029 ceph-mon[32610]: pgmap v2154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:19 smithi077 ceph-mon[37153]: pgmap v2154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: pgmap v2155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:42:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:42:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:42:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:42:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:42:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:42:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:42:21.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:42:21.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:42:21.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: pgmap v2156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:42:21.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: pgmap v2157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-27T07:42:21.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: pgmap v2155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:42:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: pgmap v2156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:42:21.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: pgmap v2157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-27T07:42:21.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: pgmap v2155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:42:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:42:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:42:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:42:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:42:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:42:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:42:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:42:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:42:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: pgmap v2156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:42:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: pgmap v2157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-27T07:42:21.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:42:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3164.425558 secs 2024-07-27T07:42:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3164.425558 secs 2024-07-27T07:42:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3164.425558 secs 2024-07-27T07:42:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:23 smithi077 ceph-mon[37153]: pgmap v2158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:42:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:23 smithi047 ceph-mon[36835]: pgmap v2158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:42:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:23 smithi029 ceph-mon[32610]: pgmap v2158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:42:25.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:42:25.552+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:42:25.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:42:25.552+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:42:25.554 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:42:25.552+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:42:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:42:25.559+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:42:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:42:25.559+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:42:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:42:25.559+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:42:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:25 smithi077 ceph-mon[37153]: pgmap v2159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T07:42:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:25 smithi047 ceph-mon[36835]: pgmap v2159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T07:42:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:25 smithi029 ceph-mon[32610]: pgmap v2159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T07:42:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:27 smithi077 ceph-mon[37153]: pgmap v2160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-27T07:42:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3169.425686 secs 2024-07-27T07:42:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:27 smithi047 ceph-mon[36835]: pgmap v2160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-27T07:42:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3169.425686 secs 2024-07-27T07:42:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:27 smithi029 ceph-mon[32610]: pgmap v2160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-27T07:42:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3169.425686 secs 2024-07-27T07:42:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:29 smithi077 ceph-mon[37153]: pgmap v2161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:42:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:29 smithi047 ceph-mon[36835]: pgmap v2161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:42:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:29 smithi029 ceph-mon[32610]: pgmap v2161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:42:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:31 smithi077 ceph-mon[37153]: pgmap v2162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:42:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:31 smithi047 ceph-mon[36835]: pgmap v2162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:42:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:31 smithi029 ceph-mon[32610]: pgmap v2162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:42:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3174.425833 secs 2024-07-27T07:42:32.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3174.425833 secs 2024-07-27T07:42:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3174.425833 secs 2024-07-27T07:42:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:33 smithi047 ceph-mon[36835]: pgmap v2163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:33 smithi029 ceph-mon[32610]: pgmap v2163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:33 smithi077 ceph-mon[37153]: pgmap v2163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:35.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:35 smithi077 ceph-mon[37153]: pgmap v2164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:35 smithi047 ceph-mon[36835]: pgmap v2164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:35 smithi029 ceph-mon[32610]: pgmap v2164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:37 smithi047 ceph-mon[36835]: pgmap v2165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3179.425957 secs 2024-07-27T07:42:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:37 smithi029 ceph-mon[32610]: pgmap v2165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3179.425957 secs 2024-07-27T07:42:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:37 smithi077 ceph-mon[37153]: pgmap v2165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3179.425957 secs 2024-07-27T07:42:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:39 smithi047 ceph-mon[36835]: pgmap v2166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:39 smithi029 ceph-mon[32610]: pgmap v2166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:39 smithi077 ceph-mon[37153]: pgmap v2166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:40.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:42:40.568+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:42:40.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:42:40.568+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:42:40.569 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:42:40.568+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:42:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:42:40.572+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:42:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:42:40.572+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:42:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:42:40.572+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:42:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:41 smithi047 ceph-mon[36835]: pgmap v2167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:42:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:41 smithi029 ceph-mon[32610]: pgmap v2167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:42:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:41 smithi077 ceph-mon[37153]: pgmap v2167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:42:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3184.426085 secs 2024-07-27T07:42:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3184.426085 secs 2024-07-27T07:42:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3184.426085 secs 2024-07-27T07:42:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:43 smithi047 ceph-mon[36835]: pgmap v2168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:43 smithi029 ceph-mon[32610]: pgmap v2168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:43 smithi077 ceph-mon[37153]: pgmap v2168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:45 smithi047 ceph-mon[36835]: pgmap v2169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:45 smithi029 ceph-mon[32610]: pgmap v2169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:45 smithi077 ceph-mon[37153]: pgmap v2169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:47 smithi047 ceph-mon[36835]: pgmap v2170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3189.426179 secs 2024-07-27T07:42:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:47 smithi029 ceph-mon[32610]: pgmap v2170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3189.426179 secs 2024-07-27T07:42:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:47 smithi077 ceph-mon[37153]: pgmap v2170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3189.426179 secs 2024-07-27T07:42:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:49 smithi047 ceph-mon[36835]: pgmap v2171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:49 smithi029 ceph-mon[32610]: pgmap v2171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:49 smithi077 ceph-mon[37153]: pgmap v2171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:51 smithi047 ceph-mon[36835]: pgmap v2172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:51 smithi029 ceph-mon[32610]: pgmap v2172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:51 smithi077 ceph-mon[37153]: pgmap v2172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3194.426309 secs 2024-07-27T07:42:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3194.426309 secs 2024-07-27T07:42:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3194.426309 secs 2024-07-27T07:42:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:53 smithi047 ceph-mon[36835]: pgmap v2173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:53 smithi029 ceph-mon[32610]: pgmap v2173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:53 smithi077 ceph-mon[37153]: pgmap v2173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:42:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:42:55.582+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:42:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:42:55.582+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:42:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:42:55.582+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:42:55.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:42:55.587+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:42:55.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:42:55.587+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:42:55.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:42:55.587+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:42:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:55 smithi047 ceph-mon[36835]: pgmap v2174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:55 smithi029 ceph-mon[32610]: pgmap v2174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:55 smithi077 ceph-mon[37153]: pgmap v2174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:42:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:57 smithi047 ceph-mon[36835]: pgmap v2175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3199.426413 secs 2024-07-27T07:42:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:57 smithi029 ceph-mon[32610]: pgmap v2175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3199.426413 secs 2024-07-27T07:42:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:57 smithi077 ceph-mon[37153]: pgmap v2175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:42:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3199.426413 secs 2024-07-27T07:42:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:42:59 smithi047 ceph-mon[36835]: pgmap v2176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:42:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:42:59 smithi029 ceph-mon[32610]: pgmap v2176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:42:59 smithi077 ceph-mon[37153]: pgmap v2176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:01 smithi047 ceph-mon[36835]: pgmap v2177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:01 smithi029 ceph-mon[32610]: pgmap v2177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:01 smithi077 ceph-mon[37153]: pgmap v2177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3204.426524 secs 2024-07-27T07:43:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3204.426524 secs 2024-07-27T07:43:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3204.426524 secs 2024-07-27T07:43:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:03 smithi047 ceph-mon[36835]: pgmap v2178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:43:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:03 smithi029 ceph-mon[32610]: pgmap v2178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:43:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:03 smithi077 ceph-mon[37153]: pgmap v2178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:43:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:05 smithi047 ceph-mon[36835]: pgmap v2179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:43:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:05 smithi029 ceph-mon[32610]: pgmap v2179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:43:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:05 smithi077 ceph-mon[37153]: pgmap v2179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:43:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:07 smithi047 ceph-mon[36835]: pgmap v2180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3209.426654 secs 2024-07-27T07:43:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:07 smithi029 ceph-mon[32610]: pgmap v2180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3209.426654 secs 2024-07-27T07:43:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:07 smithi077 ceph-mon[37153]: pgmap v2180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3209.426654 secs 2024-07-27T07:43:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:09 smithi047 ceph-mon[36835]: pgmap v2181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:09 smithi029 ceph-mon[32610]: pgmap v2181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:09 smithi077 ceph-mon[37153]: pgmap v2181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:10.598+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:43:10.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:10.598+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:43:10.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:10.598+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:43:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:10.603+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:43:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:10.603+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:43:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:10.603+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:43:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:11 smithi047 ceph-mon[36835]: pgmap v2182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:11 smithi029 ceph-mon[32610]: pgmap v2182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:11 smithi077 ceph-mon[37153]: pgmap v2182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3214.426805 secs 2024-07-27T07:43:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3214.426805 secs 2024-07-27T07:43:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3214.426805 secs 2024-07-27T07:43:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:13 smithi047 ceph-mon[36835]: pgmap v2183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:43:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:13 smithi029 ceph-mon[32610]: pgmap v2183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:43:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:13 smithi077 ceph-mon[37153]: pgmap v2183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:43:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:15 smithi047 ceph-mon[36835]: pgmap v2184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:43:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:15 smithi029 ceph-mon[32610]: pgmap v2184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:43:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:15 smithi077 ceph-mon[37153]: pgmap v2184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:43:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:17 smithi029 ceph-mon[32610]: pgmap v2185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3219.426870 secs 2024-07-27T07:43:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:17 smithi077 ceph-mon[37153]: pgmap v2185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3219.426870 secs 2024-07-27T07:43:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:17 smithi047 ceph-mon[36835]: pgmap v2185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3219.426870 secs 2024-07-27T07:43:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:19 smithi029 ceph-mon[32610]: pgmap v2186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:19 smithi077 ceph-mon[37153]: pgmap v2186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:19 smithi047 ceph-mon[36835]: pgmap v2186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:43:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:43:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:43:21.807 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:21 smithi077 ceph-mon[37153]: pgmap v2187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:21.807 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:21.807 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:21.807 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:21.808 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:21 smithi029 ceph-mon[32610]: pgmap v2187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:21 smithi047 ceph-mon[36835]: pgmap v2187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:22.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:22 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:43:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:22 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:43:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3224.426936 secs 2024-07-27T07:43:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:22 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:43:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:22 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:43:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3224.426936 secs 2024-07-27T07:43:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:22 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:43:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:22 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:43:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3224.426936 secs 2024-07-27T07:43:23.943 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:23 smithi077 ceph-mon[37153]: pgmap v2188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:43:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:23 smithi047 ceph-mon[36835]: pgmap v2188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:43:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:23 smithi029 ceph-mon[32610]: pgmap v2188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:43:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:43:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:43:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:43:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:43:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:43:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:43:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: pgmap v2189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-27T07:43:25.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: pgmap v2190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:43:25.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:43:25.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:25.613+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:43:25.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:25.613+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:43:25.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:25.613+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:43:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:25.612+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:43:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:25.612+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:43:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:25.612+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:43:25.615 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:43:25.615 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:43:25.615 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:25.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:25.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:25.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:25.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:43:25.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:43:25.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:43:25.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:43:25.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: pgmap v2189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-27T07:43:25.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: pgmap v2190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:43:25.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:43:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:43:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:43:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:43:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:43:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:43:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:43:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:43:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: pgmap v2189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-27T07:43:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: pgmap v2190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:43:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:43:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:27 smithi077 ceph-mon[37153]: pgmap v2191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:43:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3229.427077 secs 2024-07-27T07:43:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:27 smithi047 ceph-mon[36835]: pgmap v2191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:43:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3229.427077 secs 2024-07-27T07:43:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:27 smithi029 ceph-mon[32610]: pgmap v2191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:43:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3229.427077 secs 2024-07-27T07:43:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:29 smithi077 ceph-mon[37153]: pgmap v2192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:43:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:29 smithi047 ceph-mon[36835]: pgmap v2192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:43:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:29 smithi029 ceph-mon[32610]: pgmap v2192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:43:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:31 smithi077 ceph-mon[37153]: pgmap v2193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:43:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:31 smithi047 ceph-mon[36835]: pgmap v2193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:43:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:31 smithi029 ceph-mon[32610]: pgmap v2193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:43:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3234.427160 secs 2024-07-27T07:43:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3234.427160 secs 2024-07-27T07:43:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3234.427160 secs 2024-07-27T07:43:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:33 smithi077 ceph-mon[37153]: pgmap v2194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s 2024-07-27T07:43:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:33 smithi047 ceph-mon[36835]: pgmap v2194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s 2024-07-27T07:43:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:33 smithi029 ceph-mon[32610]: pgmap v2194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s 2024-07-27T07:43:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:35 smithi077 ceph-mon[37153]: pgmap v2195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:43:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:35 smithi047 ceph-mon[36835]: pgmap v2195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:43:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:35 smithi029 ceph-mon[32610]: pgmap v2195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:43:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:37 smithi077 ceph-mon[37153]: pgmap v2196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3239.427283 secs 2024-07-27T07:43:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:37 smithi047 ceph-mon[36835]: pgmap v2196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3239.427283 secs 2024-07-27T07:43:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:37 smithi029 ceph-mon[32610]: pgmap v2196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3239.427283 secs 2024-07-27T07:43:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:39 smithi077 ceph-mon[37153]: pgmap v2197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:39 smithi047 ceph-mon[36835]: pgmap v2197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:39 smithi029 ceph-mon[32610]: pgmap v2197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:40.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:40.613+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:43:40.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:40.613+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:43:40.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:40.613+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:43:40.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:40.621+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:43:40.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:40.621+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:43:40.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:40.621+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:43:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:41 smithi077 ceph-mon[37153]: pgmap v2198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:41 smithi047 ceph-mon[36835]: pgmap v2198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:41 smithi029 ceph-mon[32610]: pgmap v2198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3244.427413 secs 2024-07-27T07:43:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3244.427413 secs 2024-07-27T07:43:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3244.427413 secs 2024-07-27T07:43:43.121 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:43.120+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:43:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:43.120+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:43:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:43.120+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:43:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:43.120+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:43:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:43 smithi077 ceph-mon[37153]: pgmap v2199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:43 smithi047 ceph-mon[36835]: pgmap v2199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:43 smithi029 ceph-mon[32610]: pgmap v2199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:45 smithi077 ceph-mon[37153]: pgmap v2200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:43:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:45 smithi047 ceph-mon[36835]: pgmap v2200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:43:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:45 smithi029 ceph-mon[32610]: pgmap v2200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:43:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:47 smithi077 ceph-mon[37153]: pgmap v2201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3249.427543 secs 2024-07-27T07:43:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:47 smithi047 ceph-mon[36835]: pgmap v2201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3249.427543 secs 2024-07-27T07:43:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:47 smithi029 ceph-mon[32610]: pgmap v2201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3249.427543 secs 2024-07-27T07:43:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:49 smithi077 ceph-mon[37153]: pgmap v2202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:49 smithi047 ceph-mon[36835]: pgmap v2202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:49.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:49 smithi029 ceph-mon[32610]: pgmap v2202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:51 smithi077 ceph-mon[37153]: pgmap v2203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:51.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:51 smithi047 ceph-mon[36835]: pgmap v2203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:51 smithi029 ceph-mon[32610]: pgmap v2203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:43:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3254.427693 secs 2024-07-27T07:43:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3254.427693 secs 2024-07-27T07:43:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3254.427693 secs 2024-07-27T07:43:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:53 smithi077 ceph-mon[37153]: pgmap v2204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:53 smithi047 ceph-mon[36835]: pgmap v2204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:53 smithi029 ceph-mon[32610]: pgmap v2204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:55.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:55.626+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:43:55.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:55.626+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:43:55.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:55.626+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:43:55.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:55 smithi029 ceph-mon[32610]: pgmap v2205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:43:55.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:55.636+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:43:55.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:55.636+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:43:55.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:43:55.636+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:43:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:55 smithi047 ceph-mon[36835]: pgmap v2205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:43:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:55 smithi077 ceph-mon[37153]: pgmap v2205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:43:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:57 smithi047 ceph-mon[36835]: pgmap v2206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3259.427851 secs 2024-07-27T07:43:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:57 smithi029 ceph-mon[32610]: pgmap v2206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3259.427851 secs 2024-07-27T07:43:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:57 smithi077 ceph-mon[37153]: pgmap v2206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:43:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3259.427851 secs 2024-07-27T07:43:58.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:58.123+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:43:58.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:58.123+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:43:58.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:58.123+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:43:58.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:43:58.123+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:43:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:43:59 smithi047 ceph-mon[36835]: pgmap v2207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:43:59 smithi029 ceph-mon[32610]: pgmap v2207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:43:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:43:59 smithi077 ceph-mon[37153]: pgmap v2207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:01 smithi047 ceph-mon[36835]: pgmap v2208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:01 smithi029 ceph-mon[32610]: pgmap v2208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:01 smithi077 ceph-mon[37153]: pgmap v2208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3264.427955 secs 2024-07-27T07:44:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3264.427955 secs 2024-07-27T07:44:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3264.427955 secs 2024-07-27T07:44:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:03 smithi047 ceph-mon[36835]: pgmap v2209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:03 smithi029 ceph-mon[32610]: pgmap v2209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:03 smithi077 ceph-mon[37153]: pgmap v2209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:05.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:05 smithi077 ceph-mon[37153]: pgmap v2210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:44:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:05 smithi047 ceph-mon[36835]: pgmap v2210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:44:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:05 smithi029 ceph-mon[32610]: pgmap v2210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:44:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:07 smithi047 ceph-mon[36835]: pgmap v2211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:07.866 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3269.428024 secs 2024-07-27T07:44:07.867 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:07 smithi077 ceph-mon[37153]: pgmap v2211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:07.867 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3269.428024 secs 2024-07-27T07:44:07.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:07 smithi029 ceph-mon[32610]: pgmap v2211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:07.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3269.428024 secs 2024-07-27T07:44:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:09 smithi047 ceph-mon[36835]: pgmap v2212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:09 smithi029 ceph-mon[32610]: pgmap v2212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:09 smithi077 ceph-mon[37153]: pgmap v2212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:10.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:44:10.638+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:44:10.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:44:10.638+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:44:10.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:44:10.638+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:44:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:44:10.647+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:44:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:44:10.647+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:44:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:44:10.647+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:44:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:11 smithi047 ceph-mon[36835]: pgmap v2213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:11 smithi029 ceph-mon[32610]: pgmap v2213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:11 smithi077 ceph-mon[37153]: pgmap v2213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3274.428156 secs 2024-07-27T07:44:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3274.428156 secs 2024-07-27T07:44:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3274.428156 secs 2024-07-27T07:44:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:13 smithi047 ceph-mon[36835]: pgmap v2214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:13 smithi029 ceph-mon[32610]: pgmap v2214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:13 smithi077 ceph-mon[37153]: pgmap v2214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:15 smithi047 ceph-mon[36835]: pgmap v2215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:44:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:15 smithi029 ceph-mon[32610]: pgmap v2215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:44:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:15 smithi077 ceph-mon[37153]: pgmap v2215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:44:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:17 smithi047 ceph-mon[36835]: pgmap v2216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3279.428288 secs 2024-07-27T07:44:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:17 smithi029 ceph-mon[32610]: pgmap v2216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3279.428288 secs 2024-07-27T07:44:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:17 smithi077 ceph-mon[37153]: pgmap v2216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3279.428288 secs 2024-07-27T07:44:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:19 smithi047 ceph-mon[36835]: pgmap v2217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:19 smithi029 ceph-mon[32610]: pgmap v2217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:19 smithi077 ceph-mon[37153]: pgmap v2217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:21 smithi047 ceph-mon[36835]: pgmap v2218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:21 smithi029 ceph-mon[32610]: pgmap v2218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:21 smithi077 ceph-mon[37153]: pgmap v2218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3284.428417 secs 2024-07-27T07:44:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3284.428417 secs 2024-07-27T07:44:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3284.428417 secs 2024-07-27T07:44:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:23 smithi047 ceph-mon[36835]: pgmap v2219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:23 smithi029 ceph-mon[32610]: pgmap v2219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:23 smithi077 ceph-mon[37153]: pgmap v2219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:44:24.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:44:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:44:25.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:44:25.639+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:44:25.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:44:25.639+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:44:25.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:44:25.639+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:44:25.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:44:25.661+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:44:25.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:44:25.661+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:44:25.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:44:25.661+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:44:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:25 smithi047 ceph-mon[36835]: pgmap v2220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:44:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:44:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:44:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:25 smithi047 ceph-mon[36835]: pgmap v2221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:44:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:25 smithi047 ceph-mon[36835]: pgmap v2222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:44:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:44:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:25 smithi029 ceph-mon[32610]: pgmap v2220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:44:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:44:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:44:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:25 smithi029 ceph-mon[32610]: pgmap v2221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:44:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:25 smithi029 ceph-mon[32610]: pgmap v2222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:44:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:44:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:25 smithi077 ceph-mon[37153]: pgmap v2220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:44:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:44:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:44:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:25 smithi077 ceph-mon[37153]: pgmap v2221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:44:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:25 smithi077 ceph-mon[37153]: pgmap v2222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:44:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:44:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3289.428559 secs 2024-07-27T07:44:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:27 smithi047 ceph-mon[36835]: pgmap v2223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:44:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3289.428559 secs 2024-07-27T07:44:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:27 smithi029 ceph-mon[32610]: pgmap v2223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:44:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3289.428559 secs 2024-07-27T07:44:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:27 smithi077 ceph-mon[37153]: pgmap v2223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:44:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:29 smithi047 ceph-mon[36835]: pgmap v2224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:44:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:29 smithi029 ceph-mon[32610]: pgmap v2224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:44:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:29 smithi077 ceph-mon[37153]: pgmap v2224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:44:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:31 smithi047 ceph-mon[36835]: pgmap v2225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:44:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:31 smithi029 ceph-mon[32610]: pgmap v2225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:44:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:31 smithi077 ceph-mon[37153]: pgmap v2225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:44:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3294.428707 secs 2024-07-27T07:44:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3294.428707 secs 2024-07-27T07:44:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3294.428707 secs 2024-07-27T07:44:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:33 smithi047 ceph-mon[36835]: pgmap v2226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:44:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:33 smithi029 ceph-mon[32610]: pgmap v2226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:44:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:33 smithi077 ceph-mon[37153]: pgmap v2226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:44:36.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:35 smithi077 ceph-mon[37153]: pgmap v2227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:44:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:35 smithi047 ceph-mon[36835]: pgmap v2227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:44:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:35 smithi029 ceph-mon[32610]: pgmap v2227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:44:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3299.428822 secs 2024-07-27T07:44:38.243 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:37 smithi047 ceph-mon[36835]: pgmap v2228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:38.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3299.428822 secs 2024-07-27T07:44:38.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:37 smithi029 ceph-mon[32610]: pgmap v2228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3299.428822 secs 2024-07-27T07:44:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:37 smithi077 ceph-mon[37153]: pgmap v2228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:39 smithi047 ceph-mon[36835]: pgmap v2229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:39 smithi029 ceph-mon[32610]: pgmap v2229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:39 smithi077 ceph-mon[37153]: pgmap v2229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:40.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:44:40.654+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:44:40.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:44:40.654+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:44:40.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:44:40.654+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:44:40.678 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:44:40.677+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:44:40.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:44:40.677+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:44:40.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:44:40.677+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:44:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:41 smithi047 ceph-mon[36835]: pgmap v2230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:41 smithi029 ceph-mon[32610]: pgmap v2230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:41 smithi077 ceph-mon[37153]: pgmap v2230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3304.428945 secs 2024-07-27T07:44:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3304.428945 secs 2024-07-27T07:44:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3304.428945 secs 2024-07-27T07:44:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:43 smithi047 ceph-mon[36835]: pgmap v2231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:44:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:43 smithi029 ceph-mon[32610]: pgmap v2231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:44:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:43 smithi077 ceph-mon[37153]: pgmap v2231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:44:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:45 smithi047 ceph-mon[36835]: pgmap v2232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:44:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:45 smithi029 ceph-mon[32610]: pgmap v2232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:44:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:45 smithi077 ceph-mon[37153]: pgmap v2232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:44:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3309.429094 secs 2024-07-27T07:44:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:47 smithi047 ceph-mon[36835]: pgmap v2233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3309.429094 secs 2024-07-27T07:44:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:47 smithi029 ceph-mon[32610]: pgmap v2233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3309.429094 secs 2024-07-27T07:44:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:47 smithi077 ceph-mon[37153]: pgmap v2233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:49 smithi047 ceph-mon[36835]: pgmap v2234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:49 smithi029 ceph-mon[32610]: pgmap v2234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:49 smithi077 ceph-mon[37153]: pgmap v2234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:44:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:51 smithi047 ceph-mon[36835]: pgmap v2235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:51 smithi029 ceph-mon[32610]: pgmap v2235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:51 smithi077 ceph-mon[37153]: pgmap v2235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:44:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3314.429223 secs 2024-07-27T07:44:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3314.429223 secs 2024-07-27T07:44:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3314.429223 secs 2024-07-27T07:44:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:53 smithi047 ceph-mon[36835]: pgmap v2236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:44:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:53 smithi029 ceph-mon[32610]: pgmap v2236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:44:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:53 smithi077 ceph-mon[37153]: pgmap v2236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:44:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:44:55.669+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:44:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:44:55.669+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:44:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:44:55.669+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:44:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:44:55.689+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:44:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:44:55.689+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:44:55.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:44:55.689+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:44:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:55 smithi047 ceph-mon[36835]: pgmap v2237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:44:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:55 smithi029 ceph-mon[32610]: pgmap v2237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:44:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:55 smithi077 ceph-mon[37153]: pgmap v2237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:44:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3319.429357 secs 2024-07-27T07:44:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:57 smithi047 ceph-mon[36835]: pgmap v2238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3319.429357 secs 2024-07-27T07:44:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:57 smithi029 ceph-mon[32610]: pgmap v2238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:44:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3319.429357 secs 2024-07-27T07:44:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:57 smithi077 ceph-mon[37153]: pgmap v2238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:44:59 smithi047 ceph-mon[36835]: pgmap v2239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:44:59 smithi029 ceph-mon[32610]: pgmap v2239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:44:59 smithi077 ceph-mon[37153]: pgmap v2239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:01 smithi047 ceph-mon[36835]: pgmap v2240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:01 smithi029 ceph-mon[32610]: pgmap v2240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:01 smithi077 ceph-mon[37153]: pgmap v2240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3324.429484 secs 2024-07-27T07:45:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3324.429484 secs 2024-07-27T07:45:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3324.429484 secs 2024-07-27T07:45:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:03 smithi029 ceph-mon[32610]: pgmap v2241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:45:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:03 smithi077 ceph-mon[37153]: pgmap v2241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:45:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:03 smithi047 ceph-mon[36835]: pgmap v2241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:45:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:05 smithi029 ceph-mon[32610]: pgmap v2242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:06.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:05 smithi047 ceph-mon[36835]: pgmap v2242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:05 smithi077 ceph-mon[37153]: pgmap v2242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3329.429611 secs 2024-07-27T07:45:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:07 smithi029 ceph-mon[32610]: pgmap v2243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3329.429611 secs 2024-07-27T07:45:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:07 smithi077 ceph-mon[37153]: pgmap v2243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3329.429611 secs 2024-07-27T07:45:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:07 smithi047 ceph-mon[36835]: pgmap v2243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:09 smithi029 ceph-mon[32610]: pgmap v2244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:09 smithi077 ceph-mon[37153]: pgmap v2244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:09 smithi047 ceph-mon[36835]: pgmap v2244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:10.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:45:10.685+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:45:10.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:45:10.685+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:45:10.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:45:10.685+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:45:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:45:10.705+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:45:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:45:10.705+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:45:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:45:10.705+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:45:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:11 smithi077 ceph-mon[37153]: pgmap v2245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:11 smithi047 ceph-mon[36835]: pgmap v2245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:11 smithi029 ceph-mon[32610]: pgmap v2245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3334.429777 secs 2024-07-27T07:45:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3334.429777 secs 2024-07-27T07:45:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3334.429777 secs 2024-07-27T07:45:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:13 smithi077 ceph-mon[37153]: pgmap v2246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:45:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:13 smithi047 ceph-mon[36835]: pgmap v2246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:45:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:13 smithi029 ceph-mon[32610]: pgmap v2246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:45:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:15 smithi077 ceph-mon[37153]: pgmap v2247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:15 smithi047 ceph-mon[36835]: pgmap v2247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:15 smithi029 ceph-mon[32610]: pgmap v2247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3339.429861 secs 2024-07-27T07:45:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:17 smithi077 ceph-mon[37153]: pgmap v2248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3339.429861 secs 2024-07-27T07:45:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:17 smithi047 ceph-mon[36835]: pgmap v2248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3339.429861 secs 2024-07-27T07:45:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:17 smithi029 ceph-mon[32610]: pgmap v2248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:19 smithi077 ceph-mon[37153]: pgmap v2249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:19 smithi047 ceph-mon[36835]: pgmap v2249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:19 smithi029 ceph-mon[32610]: pgmap v2249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:21 smithi077 ceph-mon[37153]: pgmap v2250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:21 smithi047 ceph-mon[36835]: pgmap v2250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:21 smithi029 ceph-mon[32610]: pgmap v2250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3344.429982 secs 2024-07-27T07:45:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3344.429982 secs 2024-07-27T07:45:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3344.429982 secs 2024-07-27T07:45:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:23 smithi077 ceph-mon[37153]: pgmap v2251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:45:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:23 smithi047 ceph-mon[36835]: pgmap v2251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:45:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:23 smithi029 ceph-mon[32610]: pgmap v2251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:45:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:45:25.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:45:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:45:25.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:45:25.691+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:45:25.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:45:25.691+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:45:25.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:45:25.691+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:45:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:45:25.718+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:45:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:45:25.718+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:45:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:45:25.718+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:45:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:26 smithi077 ceph-mon[37153]: pgmap v2252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:45:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:45:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:45:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:26 smithi047 ceph-mon[36835]: pgmap v2252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:45:26.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:45:26.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:45:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:26 smithi029 ceph-mon[32610]: pgmap v2252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:45:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:45:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:45:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:27 smithi077 ceph-mon[37153]: pgmap v2253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:45:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:27 smithi077 ceph-mon[37153]: pgmap v2254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:45:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:27 smithi047 ceph-mon[36835]: pgmap v2253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:45:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:27 smithi047 ceph-mon[36835]: pgmap v2254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:45:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:27 smithi029 ceph-mon[32610]: pgmap v2253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:45:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:27 smithi029 ceph-mon[32610]: pgmap v2254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:45:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3349.430103 secs 2024-07-27T07:45:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3349.430103 secs 2024-07-27T07:45:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3349.430103 secs 2024-07-27T07:45:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:29 smithi077 ceph-mon[37153]: pgmap v2255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:45:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:29 smithi047 ceph-mon[36835]: pgmap v2255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:45:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:29 smithi029 ceph-mon[32610]: pgmap v2255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:45:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:31 smithi077 ceph-mon[37153]: pgmap v2256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:45:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:31 smithi047 ceph-mon[36835]: pgmap v2256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:45:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:31 smithi029 ceph-mon[32610]: pgmap v2256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:45:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:33 smithi077 ceph-mon[37153]: pgmap v2257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:45:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3354.430226 secs 2024-07-27T07:45:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:33 smithi047 ceph-mon[36835]: pgmap v2257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:45:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3354.430226 secs 2024-07-27T07:45:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:33 smithi029 ceph-mon[32610]: pgmap v2257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:45:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3354.430226 secs 2024-07-27T07:45:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:35 smithi077 ceph-mon[37153]: pgmap v2258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:45:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:35 smithi047 ceph-mon[36835]: pgmap v2258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:45:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:35 smithi029 ceph-mon[32610]: pgmap v2258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:45:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:37 smithi077 ceph-mon[37153]: pgmap v2259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:45:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:37 smithi047 ceph-mon[36835]: pgmap v2259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:45:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:37 smithi029 ceph-mon[32610]: pgmap v2259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:45:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3359.430352 secs 2024-07-27T07:45:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3359.430352 secs 2024-07-27T07:45:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3359.430352 secs 2024-07-27T07:45:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:39 smithi077 ceph-mon[37153]: pgmap v2260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:45:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:39 smithi047 ceph-mon[36835]: pgmap v2260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:45:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:39 smithi029 ceph-mon[32610]: pgmap v2260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:45:40.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:45:40.701+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:45:40.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:45:40.701+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:45:40.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:45:40.701+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:45:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:45:40.732+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:45:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:45:40.732+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:45:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:45:40.732+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:45:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:41 smithi077 ceph-mon[37153]: pgmap v2261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:41 smithi047 ceph-mon[36835]: pgmap v2261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:41 smithi029 ceph-mon[32610]: pgmap v2261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:43 smithi077 ceph-mon[37153]: pgmap v2262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3364.430473 secs 2024-07-27T07:45:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:43 smithi047 ceph-mon[36835]: pgmap v2262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3364.430473 secs 2024-07-27T07:45:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:43 smithi029 ceph-mon[32610]: pgmap v2262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3364.430473 secs 2024-07-27T07:45:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:45 smithi077 ceph-mon[37153]: pgmap v2263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:45 smithi047 ceph-mon[36835]: pgmap v2263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:45 smithi029 ceph-mon[32610]: pgmap v2263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:47 smithi077 ceph-mon[37153]: pgmap v2264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:47 smithi047 ceph-mon[36835]: pgmap v2264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:47 smithi029 ceph-mon[32610]: pgmap v2264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3369.430594 secs 2024-07-27T07:45:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3369.430594 secs 2024-07-27T07:45:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3369.430594 secs 2024-07-27T07:45:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:49 smithi077 ceph-mon[37153]: pgmap v2265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:45:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:49 smithi047 ceph-mon[36835]: pgmap v2265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:45:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:49 smithi029 ceph-mon[32610]: pgmap v2265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:45:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:51 smithi077 ceph-mon[37153]: pgmap v2266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:51 smithi047 ceph-mon[36835]: pgmap v2266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:51 smithi029 ceph-mon[32610]: pgmap v2266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:45:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:53 smithi077 ceph-mon[37153]: pgmap v2267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3374.430713 secs 2024-07-27T07:45:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:53 smithi047 ceph-mon[36835]: pgmap v2267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3374.430713 secs 2024-07-27T07:45:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:53 smithi029 ceph-mon[32610]: pgmap v2267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:45:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3374.430713 secs 2024-07-27T07:45:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:55 smithi077 ceph-mon[37153]: pgmap v2268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:55 smithi047 ceph-mon[36835]: pgmap v2268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:55 smithi029 ceph-mon[32610]: pgmap v2268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:45:55.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:45:55.717+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:45:55.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:45:55.717+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:45:55.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:45:55.717+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:45:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:45:55.748+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:45:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:45:55.748+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:45:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:45:55.748+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:45:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:57 smithi077 ceph-mon[37153]: pgmap v2269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:57 smithi047 ceph-mon[36835]: pgmap v2269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:57 smithi029 ceph-mon[32610]: pgmap v2269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:45:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3379.430837 secs 2024-07-27T07:45:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3379.430837 secs 2024-07-27T07:45:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3379.430837 secs 2024-07-27T07:45:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:45:59 smithi077 ceph-mon[37153]: pgmap v2270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:45:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:45:59 smithi047 ceph-mon[36835]: pgmap v2270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:45:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:45:59 smithi029 ceph-mon[32610]: pgmap v2270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:46:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:01 smithi077 ceph-mon[37153]: pgmap v2271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:46:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:01 smithi047 ceph-mon[36835]: pgmap v2271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:46:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:01 smithi029 ceph-mon[32610]: pgmap v2271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:46:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:03 smithi077 ceph-mon[37153]: pgmap v2272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3384.430958 secs 2024-07-27T07:46:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:03 smithi047 ceph-mon[36835]: pgmap v2272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3384.430958 secs 2024-07-27T07:46:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:03 smithi029 ceph-mon[32610]: pgmap v2272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3384.430958 secs 2024-07-27T07:46:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:04 smithi077 ceph-mon[37153]: pgmap v2273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:46:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:04 smithi047 ceph-mon[36835]: pgmap v2273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:46:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:04 smithi029 ceph-mon[32610]: pgmap v2273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:46:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:06 smithi029 ceph-mon[32610]: pgmap v2274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:46:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:06 smithi077 ceph-mon[37153]: pgmap v2274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:46:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:06 smithi047 ceph-mon[36835]: pgmap v2274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:46:07.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3389.431076 secs 2024-07-27T07:46:07.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3389.431076 secs 2024-07-27T07:46:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3389.431076 secs 2024-07-27T07:46:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:08 smithi077 ceph-mon[37153]: pgmap v2275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:46:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:08 smithi047 ceph-mon[36835]: pgmap v2275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:46:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:08 smithi029 ceph-mon[32610]: pgmap v2275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:46:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:10.733+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:46:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:10.733+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:46:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:10.733+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:46:10.734 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:10 smithi029 ceph-mon[32610]: pgmap v2276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:46:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:10.760+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:46:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:10.760+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:46:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:10.760+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:46:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:10 smithi077 ceph-mon[37153]: pgmap v2276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:46:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:10 smithi047 ceph-mon[36835]: pgmap v2276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:46:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:12 smithi077 ceph-mon[37153]: pgmap v2277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3394.431180 secs 2024-07-27T07:46:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:12 smithi047 ceph-mon[36835]: pgmap v2277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3394.431180 secs 2024-07-27T07:46:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:12 smithi029 ceph-mon[32610]: pgmap v2277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3394.431180 secs 2024-07-27T07:46:14.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:14 smithi077 ceph-mon[37153]: pgmap v2278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:46:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:14 smithi047 ceph-mon[36835]: pgmap v2278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:46:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:14 smithi029 ceph-mon[32610]: pgmap v2278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:46:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:16 smithi077 ceph-mon[37153]: pgmap v2279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:46:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:16 smithi047 ceph-mon[36835]: pgmap v2279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:46:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:16 smithi029 ceph-mon[32610]: pgmap v2279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:46:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3399.431278 secs 2024-07-27T07:46:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3399.431278 secs 2024-07-27T07:46:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3399.431278 secs 2024-07-27T07:46:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:18 smithi077 ceph-mon[37153]: pgmap v2280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:46:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:18 smithi047 ceph-mon[36835]: pgmap v2280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:46:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:18 smithi029 ceph-mon[32610]: pgmap v2280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:46:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:20 smithi077 ceph-mon[37153]: pgmap v2281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:46:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:20 smithi047 ceph-mon[36835]: pgmap v2281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:46:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:20 smithi029 ceph-mon[32610]: pgmap v2281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:46:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:22 smithi077 ceph-mon[37153]: pgmap v2282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3404.431403 secs 2024-07-27T07:46:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:22 smithi047 ceph-mon[36835]: pgmap v2282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3404.431403 secs 2024-07-27T07:46:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:22 smithi029 ceph-mon[32610]: pgmap v2282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3404.431403 secs 2024-07-27T07:46:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:24 smithi077 ceph-mon[37153]: pgmap v2283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:46:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:24 smithi047 ceph-mon[36835]: pgmap v2283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:46:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:24 smithi029 ceph-mon[32610]: pgmap v2283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:46:25.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:25.733+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:46:25.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:25.733+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:46:25.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:25.733+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:46:25.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:25.774+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:46:25.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:25.774+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:46:25.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:25.774+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:46:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:26 smithi047 ceph-mon[36835]: pgmap v2284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:46:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:46:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:26 smithi029 ceph-mon[32610]: pgmap v2284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:46:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:46:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:26 smithi077 ceph-mon[37153]: pgmap v2284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:46:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:46:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3409.431473 secs 2024-07-27T07:46:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: pgmap v2285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T07:46:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:46:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:46:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: pgmap v2286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T07:46:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: pgmap v2287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:46:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3409.431473 secs 2024-07-27T07:46:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: pgmap v2285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T07:46:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:46:28.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:46:28.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: pgmap v2286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T07:46:28.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: pgmap v2287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:46:28.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3409.431473 secs 2024-07-27T07:46:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: pgmap v2285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T07:46:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:46:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:46:28.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: pgmap v2286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T07:46:28.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: pgmap v2287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:46:28.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:46:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:30 smithi047 ceph-mon[36835]: pgmap v2288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:46:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:30 smithi029 ceph-mon[32610]: pgmap v2288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:46:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:30 smithi077 ceph-mon[37153]: pgmap v2288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:46:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:32 smithi047 ceph-mon[36835]: pgmap v2289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:46:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:32 smithi029 ceph-mon[32610]: pgmap v2289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:46:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:32 smithi077 ceph-mon[37153]: pgmap v2289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:46:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3414.431596 secs 2024-07-27T07:46:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3414.431596 secs 2024-07-27T07:46:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3414.431596 secs 2024-07-27T07:46:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:34 smithi047 ceph-mon[36835]: pgmap v2290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:46:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:34 smithi029 ceph-mon[32610]: pgmap v2290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:46:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:34 smithi077 ceph-mon[37153]: pgmap v2290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:46:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:36 smithi047 ceph-mon[36835]: pgmap v2291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:46:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:36 smithi029 ceph-mon[32610]: pgmap v2291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:46:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:36 smithi077 ceph-mon[37153]: pgmap v2291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:46:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3419.431734 secs 2024-07-27T07:46:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3419.431734 secs 2024-07-27T07:46:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3419.431734 secs 2024-07-27T07:46:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:38 smithi077 ceph-mon[37153]: pgmap v2292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:46:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:38 smithi047 ceph-mon[36835]: pgmap v2292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:46:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:38 smithi029 ceph-mon[32610]: pgmap v2292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:46:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:40 smithi047 ceph-mon[36835]: pgmap v2293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:46:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:40 smithi029 ceph-mon[32610]: pgmap v2293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:46:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:40 smithi077 ceph-mon[37153]: pgmap v2293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:46:40.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:40.748+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:46:40.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:40.748+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:46:40.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:40.748+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:46:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:40.789+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:46:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:40.789+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:46:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:40.789+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:46:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:42 smithi047 ceph-mon[36835]: pgmap v2294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:42 smithi029 ceph-mon[32610]: pgmap v2294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:42 smithi077 ceph-mon[37153]: pgmap v2294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:43.148+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:46:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:43.148+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:46:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:43.148+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:46:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:43.148+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:46:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3424.431840 secs 2024-07-27T07:46:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3424.431840 secs 2024-07-27T07:46:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3424.431840 secs 2024-07-27T07:46:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:44 smithi047 ceph-mon[36835]: pgmap v2295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:46:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:44 smithi029 ceph-mon[32610]: pgmap v2295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:46:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:44 smithi077 ceph-mon[37153]: pgmap v2295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:46:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:46 smithi047 ceph-mon[36835]: pgmap v2296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:46 smithi029 ceph-mon[32610]: pgmap v2296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:46 smithi077 ceph-mon[37153]: pgmap v2296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3429.431938 secs 2024-07-27T07:46:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3429.431938 secs 2024-07-27T07:46:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3429.431938 secs 2024-07-27T07:46:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:48 smithi077 ceph-mon[37153]: pgmap v2297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:46:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:48 smithi047 ceph-mon[36835]: pgmap v2297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:46:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:48 smithi029 ceph-mon[32610]: pgmap v2297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:46:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:50 smithi047 ceph-mon[36835]: pgmap v2298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:46:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:50 smithi029 ceph-mon[32610]: pgmap v2298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:46:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:50 smithi077 ceph-mon[37153]: pgmap v2298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:46:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:52 smithi047 ceph-mon[36835]: pgmap v2299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:52 smithi029 ceph-mon[32610]: pgmap v2299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:52 smithi077 ceph-mon[37153]: pgmap v2299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3434.432083 secs 2024-07-27T07:46:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3434.432083 secs 2024-07-27T07:46:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3434.432083 secs 2024-07-27T07:46:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:54 smithi029 ceph-mon[32610]: pgmap v2300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:46:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:54 smithi077 ceph-mon[37153]: pgmap v2300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:46:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:54 smithi047 ceph-mon[36835]: pgmap v2300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:46:55.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:55.749+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:46:55.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:55.749+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:46:55.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:55.749+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:46:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:55.805+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:46:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:55.805+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:46:55.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:46:55.805+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:46:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:56 smithi047 ceph-mon[36835]: pgmap v2301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:56 smithi029 ceph-mon[32610]: pgmap v2301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:56 smithi077 ceph-mon[37153]: pgmap v2301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:46:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3439.432149 secs 2024-07-27T07:46:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3439.432149 secs 2024-07-27T07:46:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3439.432149 secs 2024-07-27T07:46:58.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:58.151+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:46:58.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:58.151+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:46:58.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:58.151+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:46:58.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:46:58.151+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:46:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:46:58 smithi029 ceph-mon[32610]: pgmap v2302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:46:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:46:58 smithi077 ceph-mon[37153]: pgmap v2302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:46:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:46:58 smithi047 ceph-mon[36835]: pgmap v2302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:00 smithi047 ceph-mon[36835]: pgmap v2303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:00 smithi029 ceph-mon[32610]: pgmap v2303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:00 smithi077 ceph-mon[37153]: pgmap v2303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:02 smithi047 ceph-mon[36835]: pgmap v2304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:02 smithi029 ceph-mon[32610]: pgmap v2304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:02 smithi077 ceph-mon[37153]: pgmap v2304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3444.432273 secs 2024-07-27T07:47:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3444.432273 secs 2024-07-27T07:47:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3444.432273 secs 2024-07-27T07:47:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:04 smithi029 ceph-mon[32610]: pgmap v2305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:47:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:04 smithi077 ceph-mon[37153]: pgmap v2305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:47:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:04 smithi047 ceph-mon[36835]: pgmap v2305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:47:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:06 smithi047 ceph-mon[36835]: pgmap v2306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:06 smithi029 ceph-mon[32610]: pgmap v2306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:06 smithi077 ceph-mon[37153]: pgmap v2306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3449.432418 secs 2024-07-27T07:47:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3449.432418 secs 2024-07-27T07:47:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3449.432418 secs 2024-07-27T07:47:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:08 smithi077 ceph-mon[37153]: pgmap v2307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:08 smithi047 ceph-mon[36835]: pgmap v2307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:08 smithi029 ceph-mon[32610]: pgmap v2307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:10 smithi047 ceph-mon[36835]: pgmap v2308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:10 smithi029 ceph-mon[32610]: pgmap v2308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:10 smithi077 ceph-mon[37153]: pgmap v2308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:10.765+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:47:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:10.765+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:47:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:10.765+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:47:10.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:10.806+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:47:10.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:10.807+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:47:10.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:10.807+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:47:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:12 smithi047 ceph-mon[36835]: pgmap v2309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:12 smithi029 ceph-mon[32610]: pgmap v2309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:12 smithi077 ceph-mon[37153]: pgmap v2309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3454.432589 secs 2024-07-27T07:47:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3454.432589 secs 2024-07-27T07:47:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3454.432589 secs 2024-07-27T07:47:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:14 smithi029 ceph-mon[32610]: pgmap v2310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:47:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:14 smithi077 ceph-mon[37153]: pgmap v2310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:47:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:14 smithi047 ceph-mon[36835]: pgmap v2310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:47:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:16 smithi047 ceph-mon[36835]: pgmap v2311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:16 smithi029 ceph-mon[32610]: pgmap v2311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:16 smithi077 ceph-mon[37153]: pgmap v2311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3459.432705 secs 2024-07-27T07:47:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3459.432705 secs 2024-07-27T07:47:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3459.432705 secs 2024-07-27T07:47:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:18 smithi077 ceph-mon[37153]: pgmap v2312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:18 smithi047 ceph-mon[36835]: pgmap v2312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:18 smithi029 ceph-mon[32610]: pgmap v2312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:20 smithi047 ceph-mon[36835]: pgmap v2313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:20 smithi029 ceph-mon[32610]: pgmap v2313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:20 smithi077 ceph-mon[37153]: pgmap v2313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:22 smithi047 ceph-mon[36835]: pgmap v2314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:22.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:22 smithi029 ceph-mon[32610]: pgmap v2314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:22 smithi077 ceph-mon[37153]: pgmap v2314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3464.432834 secs 2024-07-27T07:47:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3464.432834 secs 2024-07-27T07:47:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3464.432834 secs 2024-07-27T07:47:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:24 smithi029 ceph-mon[32610]: pgmap v2315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:47:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:24 smithi077 ceph-mon[37153]: pgmap v2315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:47:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:24 smithi047 ceph-mon[36835]: pgmap v2315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:47:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:25.769+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:47:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:25.769+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:47:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:25.769+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:47:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:25.820+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:47:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:25.820+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:47:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:25.820+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:47:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:26 smithi047 ceph-mon[36835]: pgmap v2316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:26 smithi029 ceph-mon[32610]: pgmap v2316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:26 smithi077 ceph-mon[37153]: pgmap v2316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:47:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3469.432899 secs 2024-07-27T07:47:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3469.432899 secs 2024-07-27T07:47:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3469.432899 secs 2024-07-27T07:47:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:28.156+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:47:28.157 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:28.156+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:47:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:28.156+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:47:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:28.156+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:47:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:28 smithi077 ceph-mon[37153]: pgmap v2317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:47:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:28 smithi047 ceph-mon[36835]: pgmap v2317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:47:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:28 smithi029 ceph-mon[32610]: pgmap v2317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:47:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:47:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:47:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:29 smithi077 ceph-mon[37153]: pgmap v2318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:47:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:29 smithi077 ceph-mon[37153]: pgmap v2319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:47:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:47:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:47:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:29 smithi047 ceph-mon[36835]: pgmap v2318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:47:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:29 smithi047 ceph-mon[36835]: pgmap v2319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:47:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:47:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:47:29.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:29 smithi029 ceph-mon[32610]: pgmap v2318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:47:29.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:29 smithi029 ceph-mon[32610]: pgmap v2319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:47:29.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:47:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:32 smithi077 ceph-mon[37153]: pgmap v2320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:47:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:32 smithi047 ceph-mon[36835]: pgmap v2320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:47:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:32 smithi029 ceph-mon[32610]: pgmap v2320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:47:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3474.433048 secs 2024-07-27T07:47:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3474.433048 secs 2024-07-27T07:47:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3474.433048 secs 2024-07-27T07:47:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:34 smithi077 ceph-mon[37153]: pgmap v2321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:47:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:34 smithi047 ceph-mon[36835]: pgmap v2321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:47:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:34 smithi029 ceph-mon[32610]: pgmap v2321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:47:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:36 smithi077 ceph-mon[37153]: pgmap v2322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:47:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:36 smithi047 ceph-mon[36835]: pgmap v2322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:47:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:36 smithi029 ceph-mon[32610]: pgmap v2322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:47:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3479.433112 secs 2024-07-27T07:47:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:38 smithi077 ceph-mon[37153]: pgmap v2323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T07:47:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3479.433112 secs 2024-07-27T07:47:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:38 smithi047 ceph-mon[36835]: pgmap v2323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T07:47:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3479.433112 secs 2024-07-27T07:47:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:38 smithi029 ceph-mon[32610]: pgmap v2323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T07:47:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:40 smithi077 ceph-mon[37153]: pgmap v2324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:47:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:40 smithi047 ceph-mon[36835]: pgmap v2324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:47:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:40 smithi029 ceph-mon[32610]: pgmap v2324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:47:40.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:40.772+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:47:40.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:40.772+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:47:40.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:40.772+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:47:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:40.821+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:47:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:40.821+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:47:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:40.821+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:47:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:42 smithi077 ceph-mon[37153]: pgmap v2325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:42 smithi047 ceph-mon[36835]: pgmap v2325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:42 smithi029 ceph-mon[32610]: pgmap v2325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:43.158+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:47:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:43.158+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:47:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:43.158+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:47:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:43.158+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:47:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3484.433243 secs 2024-07-27T07:47:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3484.433243 secs 2024-07-27T07:47:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3484.433243 secs 2024-07-27T07:47:44.296 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:44 smithi029 ceph-mon[32610]: pgmap v2326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:47:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:44 smithi077 ceph-mon[37153]: pgmap v2326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:47:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:44 smithi047 ceph-mon[36835]: pgmap v2326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:47:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:46 smithi077 ceph-mon[37153]: pgmap v2327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:46 smithi047 ceph-mon[36835]: pgmap v2327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:46 smithi029 ceph-mon[32610]: pgmap v2327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3489.433397 secs 2024-07-27T07:47:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:48 smithi077 ceph-mon[37153]: pgmap v2328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:47:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3489.433397 secs 2024-07-27T07:47:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:48 smithi047 ceph-mon[36835]: pgmap v2328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:47:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3489.433397 secs 2024-07-27T07:47:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:48 smithi029 ceph-mon[32610]: pgmap v2328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:47:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:50 smithi077 ceph-mon[37153]: pgmap v2329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:50 smithi047 ceph-mon[36835]: pgmap v2329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:50 smithi029 ceph-mon[32610]: pgmap v2329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:47:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:52 smithi077 ceph-mon[37153]: pgmap v2330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:52 smithi047 ceph-mon[36835]: pgmap v2330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:52 smithi029 ceph-mon[32610]: pgmap v2330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3494.433509 secs 2024-07-27T07:47:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3494.433509 secs 2024-07-27T07:47:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3494.433509 secs 2024-07-27T07:47:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:54 smithi077 ceph-mon[37153]: pgmap v2331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:47:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:54 smithi047 ceph-mon[36835]: pgmap v2331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:47:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:54 smithi029 ceph-mon[32610]: pgmap v2331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:47:55.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:55.775+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:47:55.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:55.775+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:47:55.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:47:55.775+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:55.837+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:55.837+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:47:55.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:47:55.837+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:47:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:56 smithi077 ceph-mon[37153]: pgmap v2332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:56 smithi047 ceph-mon[36835]: pgmap v2332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:56 smithi029 ceph-mon[32610]: pgmap v2332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:47:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3499.433640 secs 2024-07-27T07:47:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:47:58 smithi077 ceph-mon[37153]: pgmap v2333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:47:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3499.433640 secs 2024-07-27T07:47:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:47:58 smithi047 ceph-mon[36835]: pgmap v2333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:47:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3499.433640 secs 2024-07-27T07:47:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:47:58 smithi029 ceph-mon[32610]: pgmap v2333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:48:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:00 smithi077 ceph-mon[37153]: pgmap v2334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:00 smithi047 ceph-mon[36835]: pgmap v2334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:00 smithi029 ceph-mon[32610]: pgmap v2334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:02 smithi077 ceph-mon[37153]: pgmap v2335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:02 smithi047 ceph-mon[36835]: pgmap v2335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:02 smithi029 ceph-mon[32610]: pgmap v2335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3504.433772 secs 2024-07-27T07:48:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3504.433772 secs 2024-07-27T07:48:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3504.433772 secs 2024-07-27T07:48:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:04 smithi077 ceph-mon[37153]: pgmap v2336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:48:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:04 smithi047 ceph-mon[36835]: pgmap v2336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:48:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:04 smithi029 ceph-mon[32610]: pgmap v2336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:48:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:06 smithi077 ceph-mon[37153]: pgmap v2337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:06 smithi047 ceph-mon[36835]: pgmap v2337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:06 smithi029 ceph-mon[32610]: pgmap v2337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3509.433861 secs 2024-07-27T07:48:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:08 smithi077 ceph-mon[37153]: pgmap v2338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:48:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3509.433861 secs 2024-07-27T07:48:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:08 smithi047 ceph-mon[36835]: pgmap v2338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:48:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3509.433861 secs 2024-07-27T07:48:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:08 smithi029 ceph-mon[32610]: pgmap v2338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:48:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:10 smithi077 ceph-mon[37153]: pgmap v2339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:10 smithi047 ceph-mon[36835]: pgmap v2339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:10 smithi029 ceph-mon[32610]: pgmap v2339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:10.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:48:10.790+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:48:10.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:48:10.790+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:48:10.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:48:10.790+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:48:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:48:10.852+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:48:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:48:10.853+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:48:10.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:48:10.853+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:48:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:12 smithi077 ceph-mon[37153]: pgmap v2340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:12 smithi047 ceph-mon[36835]: pgmap v2340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:12 smithi029 ceph-mon[32610]: pgmap v2340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3514.434018 secs 2024-07-27T07:48:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3514.434018 secs 2024-07-27T07:48:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3514.434018 secs 2024-07-27T07:48:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:14 smithi077 ceph-mon[37153]: pgmap v2341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:48:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:14 smithi047 ceph-mon[36835]: pgmap v2341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:48:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:14 smithi029 ceph-mon[32610]: pgmap v2341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:48:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:16 smithi077 ceph-mon[37153]: pgmap v2342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:16 smithi047 ceph-mon[36835]: pgmap v2342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:16 smithi029 ceph-mon[32610]: pgmap v2342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3519.434084 secs 2024-07-27T07:48:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:18 smithi047 ceph-mon[36835]: pgmap v2343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:48:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3519.434084 secs 2024-07-27T07:48:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:18 smithi029 ceph-mon[32610]: pgmap v2343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:48:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3519.434084 secs 2024-07-27T07:48:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:18 smithi077 ceph-mon[37153]: pgmap v2343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:48:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:20 smithi047 ceph-mon[36835]: pgmap v2344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:20 smithi029 ceph-mon[32610]: pgmap v2344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:20 smithi077 ceph-mon[37153]: pgmap v2344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:22 smithi047 ceph-mon[36835]: pgmap v2345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:22 smithi029 ceph-mon[32610]: pgmap v2345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:22 smithi077 ceph-mon[37153]: pgmap v2345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3524.434185 secs 2024-07-27T07:48:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3524.434185 secs 2024-07-27T07:48:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3524.434185 secs 2024-07-27T07:48:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:24 smithi047 ceph-mon[36835]: pgmap v2346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:48:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:24 smithi029 ceph-mon[32610]: pgmap v2346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:48:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:24 smithi077 ceph-mon[37153]: pgmap v2346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:48:25.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:48:25.791+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:48:25.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:48:25.791+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:48:25.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:48:25.791+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:48:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:48:25.864+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:48:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:48:25.864+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:48:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:48:25.864+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:48:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:26 smithi047 ceph-mon[36835]: pgmap v2347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:26 smithi029 ceph-mon[32610]: pgmap v2347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:26 smithi077 ceph-mon[37153]: pgmap v2347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3529.434309 secs 2024-07-27T07:48:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:28 smithi047 ceph-mon[36835]: pgmap v2348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:48:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3529.434309 secs 2024-07-27T07:48:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:28 smithi029 ceph-mon[32610]: pgmap v2348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:48:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3529.434309 secs 2024-07-27T07:48:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:28 smithi077 ceph-mon[37153]: pgmap v2348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:48:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:48:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:48:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:48:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:30 smithi047 ceph-mon[36835]: pgmap v2349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:48:30.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:48:30.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:48:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:30 smithi029 ceph-mon[32610]: pgmap v2349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:48:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:48:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:48:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:30 smithi077 ceph-mon[37153]: pgmap v2349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:48:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:48:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:48:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:31 smithi047 ceph-mon[36835]: pgmap v2350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:48:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:31 smithi047 ceph-mon[36835]: pgmap v2351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:48:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:31 smithi029 ceph-mon[32610]: pgmap v2350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:48:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:31 smithi029 ceph-mon[32610]: pgmap v2351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:48:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:31 smithi077 ceph-mon[37153]: pgmap v2350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:48:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:31 smithi077 ceph-mon[37153]: pgmap v2351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:48:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:33 smithi047 ceph-mon[36835]: pgmap v2352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:48:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3534.434380 secs 2024-07-27T07:48:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:33 smithi029 ceph-mon[32610]: pgmap v2352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:48:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3534.434380 secs 2024-07-27T07:48:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:33 smithi077 ceph-mon[37153]: pgmap v2352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:48:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3534.434380 secs 2024-07-27T07:48:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:35 smithi047 ceph-mon[36835]: pgmap v2353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:48:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:35 smithi029 ceph-mon[32610]: pgmap v2353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:48:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:35 smithi077 ceph-mon[37153]: pgmap v2353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:48:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:37 smithi047 ceph-mon[36835]: pgmap v2354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:48:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:37 smithi029 ceph-mon[32610]: pgmap v2354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:48:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:37 smithi077 ceph-mon[37153]: pgmap v2354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:48:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3539.434502 secs 2024-07-27T07:48:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3539.434502 secs 2024-07-27T07:48:38.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3539.434502 secs 2024-07-27T07:48:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:39 smithi047 ceph-mon[36835]: pgmap v2355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:48:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:39 smithi029 ceph-mon[32610]: pgmap v2355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:48:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:39 smithi077 ceph-mon[37153]: pgmap v2355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:48:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:40 smithi047 ceph-mon[36835]: pgmap v2356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:48:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:40 smithi029 ceph-mon[32610]: pgmap v2356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:48:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:40 smithi077 ceph-mon[37153]: pgmap v2356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:48:40.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:48:40.806+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:48:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:48:40.806+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:48:40.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:48:40.806+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:48:40.868 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:48:40.867+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:48:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:48:40.867+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:48:40.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:48:40.867+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:48:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:42 smithi077 ceph-mon[37153]: pgmap v2357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:48:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3544.434643 secs 2024-07-27T07:48:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:42 smithi047 ceph-mon[36835]: pgmap v2357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:48:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3544.434643 secs 2024-07-27T07:48:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:42 smithi029 ceph-mon[32610]: pgmap v2357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:48:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3544.434643 secs 2024-07-27T07:48:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:44 smithi077 ceph-mon[37153]: pgmap v2358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:48:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:44 smithi047 ceph-mon[36835]: pgmap v2358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:48:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:44 smithi029 ceph-mon[32610]: pgmap v2358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:48:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:46 smithi077 ceph-mon[37153]: pgmap v2359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:48:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:46 smithi047 ceph-mon[36835]: pgmap v2359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:48:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:46 smithi029 ceph-mon[32610]: pgmap v2359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:48:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3549.434765 secs 2024-07-27T07:48:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3549.434765 secs 2024-07-27T07:48:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3549.434765 secs 2024-07-27T07:48:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:48 smithi077 ceph-mon[37153]: pgmap v2360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:48:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:48 smithi047 ceph-mon[36835]: pgmap v2360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:48:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:48 smithi029 ceph-mon[32610]: pgmap v2360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:48:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:50 smithi077 ceph-mon[37153]: pgmap v2361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:48:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:50 smithi047 ceph-mon[36835]: pgmap v2361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:48:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:50 smithi029 ceph-mon[32610]: pgmap v2361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:48:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:52 smithi077 ceph-mon[37153]: pgmap v2362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:48:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3554.434898 secs 2024-07-27T07:48:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:52 smithi047 ceph-mon[36835]: pgmap v2362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:48:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3554.434898 secs 2024-07-27T07:48:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:52 smithi029 ceph-mon[32610]: pgmap v2362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:48:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3554.434898 secs 2024-07-27T07:48:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:54 smithi077 ceph-mon[37153]: pgmap v2363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:54 smithi047 ceph-mon[36835]: pgmap v2363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:54 smithi029 ceph-mon[32610]: pgmap v2363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:48:55.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:48:55.813+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:48:55.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:48:55.813+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:48:55.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:48:55.813+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:48:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:48:55.883+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:48:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:48:55.883+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:48:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:48:55.883+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:48:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:56 smithi077 ceph-mon[37153]: pgmap v2364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:56 smithi047 ceph-mon[36835]: pgmap v2364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:56 smithi029 ceph-mon[32610]: pgmap v2364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:48:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3559.435021 secs 2024-07-27T07:48:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3559.435021 secs 2024-07-27T07:48:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3559.435021 secs 2024-07-27T07:48:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:48:58 smithi047 ceph-mon[36835]: pgmap v2365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:48:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:48:58 smithi029 ceph-mon[32610]: pgmap v2365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:48:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:48:58 smithi077 ceph-mon[37153]: pgmap v2365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:49:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:00 smithi047 ceph-mon[36835]: pgmap v2366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:49:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:00 smithi029 ceph-mon[32610]: pgmap v2366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:49:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:00 smithi077 ceph-mon[37153]: pgmap v2366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:49:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:02 smithi047 ceph-mon[36835]: pgmap v2367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3564.435143 secs 2024-07-27T07:49:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:02 smithi029 ceph-mon[32610]: pgmap v2367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3564.435143 secs 2024-07-27T07:49:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:02 smithi077 ceph-mon[37153]: pgmap v2367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3564.435143 secs 2024-07-27T07:49:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:04 smithi047 ceph-mon[36835]: pgmap v2368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:49:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:04 smithi029 ceph-mon[32610]: pgmap v2368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:49:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:04 smithi077 ceph-mon[37153]: pgmap v2368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:49:06.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:06 smithi047 ceph-mon[36835]: pgmap v2369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:06 smithi029 ceph-mon[32610]: pgmap v2369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:06 smithi077 ceph-mon[37153]: pgmap v2369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3569.435265 secs 2024-07-27T07:49:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3569.435265 secs 2024-07-27T07:49:08.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3569.435265 secs 2024-07-27T07:49:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:08 smithi047 ceph-mon[36835]: pgmap v2370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:49:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:08 smithi029 ceph-mon[32610]: pgmap v2370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:49:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:08 smithi077 ceph-mon[37153]: pgmap v2370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:49:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:10.822+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:49:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:10.822+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:49:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:10.822+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:49:10.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:10.898+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:49:10.899 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:10.899+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:49:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:10.899+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:49:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:10 smithi047 ceph-mon[36835]: pgmap v2371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:49:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:10 smithi029 ceph-mon[32610]: pgmap v2371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:49:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:10 smithi077 ceph-mon[37153]: pgmap v2371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:49:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:12 smithi047 ceph-mon[36835]: pgmap v2372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3574.435386 secs 2024-07-27T07:49:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:12 smithi029 ceph-mon[32610]: pgmap v2372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3574.435386 secs 2024-07-27T07:49:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:12 smithi077 ceph-mon[37153]: pgmap v2372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3574.435386 secs 2024-07-27T07:49:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:14 smithi047 ceph-mon[36835]: pgmap v2373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:49:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:14 smithi029 ceph-mon[32610]: pgmap v2373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:49:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:14 smithi077 ceph-mon[37153]: pgmap v2373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:49:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:16 smithi047 ceph-mon[36835]: pgmap v2374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:16 smithi029 ceph-mon[32610]: pgmap v2374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:16 smithi077 ceph-mon[37153]: pgmap v2374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3579.435508 secs 2024-07-27T07:49:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3579.435508 secs 2024-07-27T07:49:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3579.435508 secs 2024-07-27T07:49:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:18 smithi047 ceph-mon[36835]: pgmap v2375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:49:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:18 smithi029 ceph-mon[32610]: pgmap v2375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:49:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:18 smithi077 ceph-mon[37153]: pgmap v2375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:49:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:20 smithi047 ceph-mon[36835]: pgmap v2376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:49:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:20 smithi029 ceph-mon[32610]: pgmap v2376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:49:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:20 smithi077 ceph-mon[37153]: pgmap v2376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:49:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:22 smithi047 ceph-mon[36835]: pgmap v2377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3584.435614 secs 2024-07-27T07:49:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:22 smithi029 ceph-mon[32610]: pgmap v2377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3584.435614 secs 2024-07-27T07:49:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:22 smithi077 ceph-mon[37153]: pgmap v2377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3584.435614 secs 2024-07-27T07:49:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:24 smithi047 ceph-mon[36835]: pgmap v2378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:49:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:24 smithi029 ceph-mon[32610]: pgmap v2378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:49:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:24 smithi077 ceph-mon[37153]: pgmap v2378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:49:25.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:25.829+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:49:25.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:25.829+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:49:25.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:25.829+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:49:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:25.913+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:49:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:25.913+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:49:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:25.913+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:49:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:26 smithi047 ceph-mon[36835]: pgmap v2379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:26 smithi029 ceph-mon[32610]: pgmap v2379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:26 smithi077 ceph-mon[37153]: pgmap v2379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3589.435742 secs 2024-07-27T07:49:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3589.435742 secs 2024-07-27T07:49:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3589.435742 secs 2024-07-27T07:49:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:28 smithi047 ceph-mon[36835]: pgmap v2380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:49:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:28 smithi029 ceph-mon[32610]: pgmap v2380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:49:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:28 smithi077 ceph-mon[37153]: pgmap v2380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:49:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:49:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:49:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:49:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:30 smithi047 ceph-mon[36835]: pgmap v2381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:49:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:49:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:49:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:30 smithi047 ceph-mon[36835]: pgmap v2382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:49:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:30 smithi047 ceph-mon[36835]: pgmap v2383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:49:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:49:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:30 smithi029 ceph-mon[32610]: pgmap v2381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:49:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:49:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:49:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:30 smithi029 ceph-mon[32610]: pgmap v2382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:49:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:30 smithi029 ceph-mon[32610]: pgmap v2383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:49:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:49:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:30 smithi077 ceph-mon[37153]: pgmap v2381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:49:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:49:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:49:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:30 smithi077 ceph-mon[37153]: pgmap v2382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:49:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:30 smithi077 ceph-mon[37153]: pgmap v2383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:49:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:49:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3594.435864 secs 2024-07-27T07:49:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:33 smithi047 ceph-mon[36835]: pgmap v2384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:49:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3594.435864 secs 2024-07-27T07:49:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:33 smithi029 ceph-mon[32610]: pgmap v2384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:49:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3594.435864 secs 2024-07-27T07:49:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:33 smithi077 ceph-mon[37153]: pgmap v2384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:49:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:35 smithi047 ceph-mon[36835]: pgmap v2385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:49:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:35 smithi029 ceph-mon[32610]: pgmap v2385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:49:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:35 smithi077 ceph-mon[37153]: pgmap v2385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:49:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:37 smithi047 ceph-mon[36835]: pgmap v2386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:49:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:37 smithi029 ceph-mon[32610]: pgmap v2386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:49:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:37 smithi077 ceph-mon[37153]: pgmap v2386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T07:49:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3599.435894 secs 2024-07-27T07:49:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3599.435894 secs 2024-07-27T07:49:38.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3599.435894 secs 2024-07-27T07:49:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:39 smithi047 ceph-mon[36835]: pgmap v2387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:49:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:39 smithi029 ceph-mon[32610]: pgmap v2387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:49:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:39 smithi077 ceph-mon[37153]: pgmap v2387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:49:40.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:40.845+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:49:40.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:40.845+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:49:40.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:40.845+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:49:40.919 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:40.918+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:49:40.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:40.918+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:49:40.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:40.918+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:49:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:41 smithi047 ceph-mon[36835]: pgmap v2388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:49:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:41 smithi029 ceph-mon[32610]: pgmap v2388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:49:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:41 smithi077 ceph-mon[37153]: pgmap v2388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:49:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:42.636+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-27T07:49:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:42.636+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T07:49:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:42.636+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f733c02b4a0 tx=0x7f733c046000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T07:49:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:42.636+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f733c02b4a0 tx=0x7f733c046000 comp rx=0 tx=0).stop 2024-07-27T07:49:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:42.636+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T07:49:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:42.636+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T07:49:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:42.636+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T07:49:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:42.636+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T07:49:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:42.636+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T07:49:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:42.636+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4730 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f733c02b320 tx=0x7f733c07a000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T07:49:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:43.176+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:49:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:43.176+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:49:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:43.176+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:49:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:43.176+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:49:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3604.435991 secs 2024-07-27T07:49:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:43 smithi047 ceph-mon[36835]: pgmap v2389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3604.435991 secs 2024-07-27T07:49:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:43 smithi029 ceph-mon[32610]: pgmap v2389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3604.435991 secs 2024-07-27T07:49:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:43 smithi077 ceph-mon[37153]: pgmap v2389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:45 smithi047 ceph-mon[36835]: pgmap v2390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:49:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:45 smithi029 ceph-mon[32610]: pgmap v2390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:49:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:45 smithi077 ceph-mon[37153]: pgmap v2390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:49:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:47 smithi047 ceph-mon[36835]: pgmap v2391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:47 smithi029 ceph-mon[32610]: pgmap v2391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:47 smithi077 ceph-mon[37153]: pgmap v2391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3609.436118 secs 2024-07-27T07:49:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3609.436118 secs 2024-07-27T07:49:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3609.436118 secs 2024-07-27T07:49:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:49 smithi047 ceph-mon[36835]: pgmap v2392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:49:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:49 smithi029 ceph-mon[32610]: pgmap v2392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:49:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:49 smithi077 ceph-mon[37153]: pgmap v2392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:49:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:50 smithi047 ceph-mon[36835]: pgmap v2393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:50 smithi029 ceph-mon[32610]: pgmap v2393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:50 smithi077 ceph-mon[37153]: pgmap v2393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3614.436259 secs 2024-07-27T07:49:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:53 smithi047 ceph-mon[36835]: pgmap v2394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3614.436259 secs 2024-07-27T07:49:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:53 smithi029 ceph-mon[32610]: pgmap v2394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3614.436259 secs 2024-07-27T07:49:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:53 smithi077 ceph-mon[37153]: pgmap v2394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:49:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:55 smithi047 ceph-mon[36835]: pgmap v2395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:49:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:55 smithi029 ceph-mon[32610]: pgmap v2395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:49:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:55 smithi077 ceph-mon[37153]: pgmap v2395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:49:55.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:55.853+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:49:55.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:55.853+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:49:55.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:55.853+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:49:55.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:55.921+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:49:55.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:55.921+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:49:55.922 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:49:55.921+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:49:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:57 smithi047 ceph-mon[36835]: pgmap v2396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:57 smithi029 ceph-mon[32610]: pgmap v2396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:57 smithi077 ceph-mon[37153]: pgmap v2396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:57.651+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T07:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:57.651+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T07:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:57.651+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7eff40009ee0 tx=0x7eff4001e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T07:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:57.651+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7eff40009ee0 tx=0x7eff4001e040 comp rx=0 tx=0).stop 2024-07-27T07:49:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:57.651+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T07:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:57.651+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T07:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:57.651+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T07:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:57.651+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T07:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:57.651+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T07:49:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:57.651+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b4730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7eff400049e0 tx=0x7eff40038040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T07:49:58.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:58.178+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:49:58.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:58.178+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:49:58.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:58.178+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:49:58.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:49:58.179+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:49:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3619.436414 secs 2024-07-27T07:49:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3619.436414 secs 2024-07-27T07:49:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3619.436414 secs 2024-07-27T07:49:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:49:59 smithi047 ceph-mon[36835]: pgmap v2397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:49:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:49:59 smithi029 ceph-mon[32610]: pgmap v2397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:49:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:49:59 smithi077 ceph-mon[37153]: pgmap v2397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:50:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:50:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:50:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T07:50:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:01 smithi047 ceph-mon[36835]: pgmap v2398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:01 smithi029 ceph-mon[32610]: pgmap v2398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:01 smithi077 ceph-mon[37153]: pgmap v2398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:02 smithi077 ceph-mon[37153]: pgmap v2399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:02 smithi047 ceph-mon[36835]: pgmap v2399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:02 smithi029 ceph-mon[32610]: pgmap v2399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3624.436556 secs 2024-07-27T07:50:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3624.436556 secs 2024-07-27T07:50:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3624.436556 secs 2024-07-27T07:50:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:04 smithi047 ceph-mon[36835]: pgmap v2400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:50:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:04 smithi029 ceph-mon[32610]: pgmap v2400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:50:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:04 smithi077 ceph-mon[37153]: pgmap v2400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:50:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:07 smithi047 ceph-mon[36835]: pgmap v2401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:07 smithi029 ceph-mon[32610]: pgmap v2401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:07 smithi077 ceph-mon[37153]: pgmap v2401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3629.436649 secs 2024-07-27T07:50:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3629.436649 secs 2024-07-27T07:50:08.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3629.436649 secs 2024-07-27T07:50:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:09 smithi047 ceph-mon[36835]: pgmap v2402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:50:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:09 smithi029 ceph-mon[32610]: pgmap v2402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:50:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:09 smithi077 ceph-mon[37153]: pgmap v2402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:50:10.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:50:10.862+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:50:10.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:50:10.862+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:50:10.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:50:10.862+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:50:10.936 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:50:10.936+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:50:10.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:50:10.936+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:50:10.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:50:10.936+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:50:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:11 smithi047 ceph-mon[36835]: pgmap v2403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:11 smithi029 ceph-mon[32610]: pgmap v2403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:11 smithi077 ceph-mon[37153]: pgmap v2403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3634.436809 secs 2024-07-27T07:50:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:13 smithi047 ceph-mon[36835]: pgmap v2404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3634.436809 secs 2024-07-27T07:50:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:13 smithi029 ceph-mon[32610]: pgmap v2404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3634.436809 secs 2024-07-27T07:50:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:13 smithi077 ceph-mon[37153]: pgmap v2404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:15 smithi047 ceph-mon[36835]: pgmap v2405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:50:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:15 smithi029 ceph-mon[32610]: pgmap v2405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:50:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:15 smithi077 ceph-mon[37153]: pgmap v2405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:50:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:17 smithi047 ceph-mon[36835]: pgmap v2406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:17 smithi029 ceph-mon[32610]: pgmap v2406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:17 smithi077 ceph-mon[37153]: pgmap v2406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3639.436931 secs 2024-07-27T07:50:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3639.436931 secs 2024-07-27T07:50:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3639.436931 secs 2024-07-27T07:50:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:19 smithi047 ceph-mon[36835]: pgmap v2407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:50:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:19 smithi029 ceph-mon[32610]: pgmap v2407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:50:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:19 smithi077 ceph-mon[37153]: pgmap v2407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:50:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:20 smithi029 ceph-mon[32610]: pgmap v2408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:20 smithi047 ceph-mon[36835]: pgmap v2408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:20 smithi077 ceph-mon[37153]: pgmap v2408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3644.437090 secs 2024-07-27T07:50:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:23 smithi047 ceph-mon[36835]: pgmap v2409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3644.437090 secs 2024-07-27T07:50:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:23 smithi029 ceph-mon[32610]: pgmap v2409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3644.437090 secs 2024-07-27T07:50:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:23 smithi077 ceph-mon[37153]: pgmap v2409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:25 smithi047 ceph-mon[36835]: pgmap v2410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:50:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:25 smithi029 ceph-mon[32610]: pgmap v2410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:50:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:25 smithi077 ceph-mon[37153]: pgmap v2410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:50:25.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:50:25.868+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:50:25.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:50:25.868+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:50:25.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:50:25.868+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:50:25.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:50:25.950+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:50:25.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:50:25.950+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:50:25.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:50:25.950+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:50:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:27 smithi047 ceph-mon[36835]: pgmap v2411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:27 smithi029 ceph-mon[32610]: pgmap v2411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:27 smithi077 ceph-mon[37153]: pgmap v2411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3649.437219 secs 2024-07-27T07:50:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3649.437219 secs 2024-07-27T07:50:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3649.437219 secs 2024-07-27T07:50:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:29 smithi047 ceph-mon[36835]: pgmap v2412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:50:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:29 smithi029 ceph-mon[32610]: pgmap v2412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:50:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:29 smithi077 ceph-mon[37153]: pgmap v2412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:50:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:31 smithi047 ceph-mon[36835]: pgmap v2413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:50:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:50:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:50:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:50:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:31 smithi029 ceph-mon[32610]: pgmap v2413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:50:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:50:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:50:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:50:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:31 smithi077 ceph-mon[37153]: pgmap v2413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:50:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:50:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:50:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:50:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:32 smithi047 ceph-mon[36835]: pgmap v2414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:50:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:32 smithi047 ceph-mon[36835]: pgmap v2415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:50:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:32 smithi029 ceph-mon[32610]: pgmap v2414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:50:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:32 smithi029 ceph-mon[32610]: pgmap v2415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:50:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:32 smithi077 ceph-mon[37153]: pgmap v2414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:50:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:32 smithi077 ceph-mon[37153]: pgmap v2415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:50:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3654.437299 secs 2024-07-27T07:50:33.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:33 smithi047 ceph-mon[36835]: pgmap v2416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:50:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3654.437299 secs 2024-07-27T07:50:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:33 smithi029 ceph-mon[32610]: pgmap v2416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:50:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3654.437299 secs 2024-07-27T07:50:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:33 smithi077 ceph-mon[37153]: pgmap v2416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:50:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:35 smithi029 ceph-mon[32610]: pgmap v2417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:50:36.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:35 smithi077 ceph-mon[37153]: pgmap v2417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:50:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:35 smithi047 ceph-mon[36835]: pgmap v2417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:50:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:37 smithi029 ceph-mon[32610]: pgmap v2418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:50:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3659.437428 secs 2024-07-27T07:50:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:37 smithi077 ceph-mon[37153]: pgmap v2418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:50:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3659.437428 secs 2024-07-27T07:50:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:37 smithi047 ceph-mon[36835]: pgmap v2418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:50:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3659.437428 secs 2024-07-27T07:50:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:39 smithi029 ceph-mon[32610]: pgmap v2419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:50:39.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:39 smithi047 ceph-mon[36835]: pgmap v2419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:50:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:39 smithi077 ceph-mon[37153]: pgmap v2419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:50:40.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:50:40.877+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:50:40.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:50:40.877+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:50:40.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:50:40.877+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:50:40.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:50:40.965+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:50:40.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:50:40.965+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:50:40.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:50:40.965+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:50:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:41 smithi077 ceph-mon[37153]: pgmap v2420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:50:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:41 smithi047 ceph-mon[36835]: pgmap v2420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:50:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:41 smithi029 ceph-mon[32610]: pgmap v2420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:50:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3664.437516 secs 2024-07-27T07:50:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3664.437516 secs 2024-07-27T07:50:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3664.437516 secs 2024-07-27T07:50:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:43 smithi077 ceph-mon[37153]: pgmap v2421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:50:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:43 smithi047 ceph-mon[36835]: pgmap v2421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:50:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:43 smithi029 ceph-mon[32610]: pgmap v2421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:50:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:45 smithi077 ceph-mon[37153]: pgmap v2422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:50:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:45 smithi047 ceph-mon[36835]: pgmap v2422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:50:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:45 smithi029 ceph-mon[32610]: pgmap v2422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:50:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:47 smithi077 ceph-mon[37153]: pgmap v2423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3669.437674 secs 2024-07-27T07:50:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:47 smithi047 ceph-mon[36835]: pgmap v2423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3669.437674 secs 2024-07-27T07:50:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:47 smithi029 ceph-mon[32610]: pgmap v2423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3669.437674 secs 2024-07-27T07:50:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:49 smithi077 ceph-mon[37153]: pgmap v2424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:50:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:49 smithi047 ceph-mon[36835]: pgmap v2424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:50:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:49 smithi029 ceph-mon[32610]: pgmap v2424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:50:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:51 smithi077 ceph-mon[37153]: pgmap v2425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:51 smithi047 ceph-mon[36835]: pgmap v2425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:51 smithi029 ceph-mon[32610]: pgmap v2425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:50:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3674.437774 secs 2024-07-27T07:50:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3674.437774 secs 2024-07-27T07:50:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3674.437774 secs 2024-07-27T07:50:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:53 smithi077 ceph-mon[37153]: pgmap v2426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:50:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:53 smithi047 ceph-mon[36835]: pgmap v2426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:50:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:53 smithi029 ceph-mon[32610]: pgmap v2426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:50:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:50:55.893+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:50:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:50:55.893+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:50:55.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:50:55.893+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:50:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:50:55.981+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:50:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:50:55.981+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:50:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:50:55.981+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:50:56.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:55 smithi029 ceph-mon[32610]: pgmap v2427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:50:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:55 smithi077 ceph-mon[37153]: pgmap v2427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:50:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:55 smithi047 ceph-mon[36835]: pgmap v2427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:50:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:57 smithi077 ceph-mon[37153]: pgmap v2428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3679.437898 secs 2024-07-27T07:50:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:57 smithi047 ceph-mon[36835]: pgmap v2428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3679.437898 secs 2024-07-27T07:50:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:57 smithi029 ceph-mon[32610]: pgmap v2428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:50:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3679.437898 secs 2024-07-27T07:51:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:50:59 smithi077 ceph-mon[37153]: pgmap v2429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:50:59 smithi047 ceph-mon[36835]: pgmap v2429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:50:59 smithi029 ceph-mon[32610]: pgmap v2429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:01 smithi077 ceph-mon[37153]: pgmap v2430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:51:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:01 smithi047 ceph-mon[36835]: pgmap v2430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:51:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:01 smithi029 ceph-mon[32610]: pgmap v2430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:51:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3684.437984 secs 2024-07-27T07:51:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3684.437984 secs 2024-07-27T07:51:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3684.437984 secs 2024-07-27T07:51:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:03 smithi077 ceph-mon[37153]: pgmap v2431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:51:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:03 smithi047 ceph-mon[36835]: pgmap v2431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:51:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:03 smithi029 ceph-mon[32610]: pgmap v2431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:51:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:05 smithi077 ceph-mon[37153]: pgmap v2432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:05 smithi047 ceph-mon[36835]: pgmap v2432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:05 smithi029 ceph-mon[32610]: pgmap v2432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:07 smithi077 ceph-mon[37153]: pgmap v2433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3689.438110 secs 2024-07-27T07:51:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:07 smithi047 ceph-mon[36835]: pgmap v2433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3689.438110 secs 2024-07-27T07:51:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:07 smithi029 ceph-mon[32610]: pgmap v2433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3689.438110 secs 2024-07-27T07:51:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:09 smithi077 ceph-mon[37153]: pgmap v2434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:09 smithi047 ceph-mon[36835]: pgmap v2434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:09 smithi029 ceph-mon[32610]: pgmap v2434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:51:10.908+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:51:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:51:10.908+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:51:10.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:51:10.908+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:51:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:51:10.997+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:51:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:51:10.997+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:51:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:51:10.997+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:51:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:11 smithi077 ceph-mon[37153]: pgmap v2435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:11 smithi047 ceph-mon[36835]: pgmap v2435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:11 smithi029 ceph-mon[32610]: pgmap v2435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3694.438233 secs 2024-07-27T07:51:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3694.438233 secs 2024-07-27T07:51:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3694.438233 secs 2024-07-27T07:51:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:13 smithi077 ceph-mon[37153]: pgmap v2436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:51:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:13 smithi047 ceph-mon[36835]: pgmap v2436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:51:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:13 smithi029 ceph-mon[32610]: pgmap v2436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:51:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:15 smithi077 ceph-mon[37153]: pgmap v2437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:15 smithi047 ceph-mon[36835]: pgmap v2437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:15 smithi029 ceph-mon[32610]: pgmap v2437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:17 smithi077 ceph-mon[37153]: pgmap v2438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3699.438357 secs 2024-07-27T07:51:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:17 smithi047 ceph-mon[36835]: pgmap v2438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3699.438357 secs 2024-07-27T07:51:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:17 smithi029 ceph-mon[32610]: pgmap v2438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3699.438357 secs 2024-07-27T07:51:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:19 smithi077 ceph-mon[37153]: pgmap v2439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:19 smithi047 ceph-mon[36835]: pgmap v2439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:19 smithi029 ceph-mon[32610]: pgmap v2439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:21 smithi077 ceph-mon[37153]: pgmap v2440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:51:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:21 smithi047 ceph-mon[36835]: pgmap v2440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:51:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:21 smithi029 ceph-mon[32610]: pgmap v2440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:51:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3704.438477 secs 2024-07-27T07:51:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3704.438477 secs 2024-07-27T07:51:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3704.438477 secs 2024-07-27T07:51:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:23 smithi077 ceph-mon[37153]: pgmap v2441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:51:24.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:23 smithi047 ceph-mon[36835]: pgmap v2441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:51:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:23 smithi029 ceph-mon[32610]: pgmap v2441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:51:25.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:51:25.923+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:51:25.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:51:25.923+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:51:25.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:51:25.923+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:51:26.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:51:26.012+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:51:26.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:51:26.012+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:51:26.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:51:26.012+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:51:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:25 smithi077 ceph-mon[37153]: pgmap v2442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:25 smithi047 ceph-mon[36835]: pgmap v2442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:25 smithi029 ceph-mon[32610]: pgmap v2442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:27 smithi077 ceph-mon[37153]: pgmap v2443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3709.438604 secs 2024-07-27T07:51:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:27 smithi047 ceph-mon[36835]: pgmap v2443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3709.438604 secs 2024-07-27T07:51:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:27 smithi029 ceph-mon[32610]: pgmap v2443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3709.438604 secs 2024-07-27T07:51:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:29 smithi077 ceph-mon[37153]: pgmap v2444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:29 smithi047 ceph-mon[36835]: pgmap v2444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:29 smithi029 ceph-mon[32610]: pgmap v2444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:51:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:51:31.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:51:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:31 smithi077 ceph-mon[37153]: pgmap v2445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:51:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:51:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:51:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:51:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:31 smithi047 ceph-mon[36835]: pgmap v2445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:51:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:51:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:51:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:51:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:31 smithi029 ceph-mon[32610]: pgmap v2445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:51:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:51:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:51:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:51:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:32 smithi077 ceph-mon[37153]: pgmap v2446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T07:51:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:32 smithi077 ceph-mon[37153]: pgmap v2447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:51:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3714.438703 secs 2024-07-27T07:51:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:32 smithi047 ceph-mon[36835]: pgmap v2446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T07:51:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:32 smithi047 ceph-mon[36835]: pgmap v2447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:51:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3714.438703 secs 2024-07-27T07:51:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:32 smithi029 ceph-mon[32610]: pgmap v2446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T07:51:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:32 smithi029 ceph-mon[32610]: pgmap v2447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:51:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3714.438703 secs 2024-07-27T07:51:35.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:34 smithi077 ceph-mon[37153]: pgmap v2448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:51:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:34 smithi047 ceph-mon[36835]: pgmap v2448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:51:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:34 smithi029 ceph-mon[32610]: pgmap v2448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:51:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:36 smithi077 ceph-mon[37153]: pgmap v2449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:51:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:36 smithi047 ceph-mon[36835]: pgmap v2449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:51:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:36 smithi029 ceph-mon[32610]: pgmap v2449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:51:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3719.438827 secs 2024-07-27T07:51:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3719.438827 secs 2024-07-27T07:51:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3719.438827 secs 2024-07-27T07:51:39.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:38 smithi047 ceph-mon[36835]: pgmap v2450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:51:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:38 smithi077 ceph-mon[37153]: pgmap v2450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:51:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:38 smithi029 ceph-mon[32610]: pgmap v2450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:51:40.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:51:40.939+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:51:40.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:51:40.939+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:51:40.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:51:40.939+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:51:41.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:51:41.023+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:51:41.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:51:41.023+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:51:41.025 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:51:41.023+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:51:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:40 smithi077 ceph-mon[37153]: pgmap v2451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:51:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:40 smithi047 ceph-mon[36835]: pgmap v2451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:51:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:40 smithi029 ceph-mon[32610]: pgmap v2451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:51:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:42 smithi077 ceph-mon[37153]: pgmap v2452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:51:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3724.438952 secs 2024-07-27T07:51:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:42 smithi047 ceph-mon[36835]: pgmap v2452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:51:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3724.438952 secs 2024-07-27T07:51:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:42 smithi029 ceph-mon[32610]: pgmap v2452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:51:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3724.438952 secs 2024-07-27T07:51:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:44 smithi077 ceph-mon[37153]: pgmap v2453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:44 smithi047 ceph-mon[36835]: pgmap v2453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:44 smithi029 ceph-mon[32610]: pgmap v2453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:46 smithi047 ceph-mon[36835]: pgmap v2454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:46 smithi029 ceph-mon[32610]: pgmap v2454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:46 smithi077 ceph-mon[37153]: pgmap v2454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3729.439080 secs 2024-07-27T07:51:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3729.439080 secs 2024-07-27T07:51:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3729.439080 secs 2024-07-27T07:51:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:48 smithi047 ceph-mon[36835]: pgmap v2455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:48 smithi029 ceph-mon[32610]: pgmap v2455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:48 smithi077 ceph-mon[37153]: pgmap v2455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:50 smithi047 ceph-mon[36835]: pgmap v2456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:51:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:50 smithi029 ceph-mon[32610]: pgmap v2456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:51:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:50 smithi077 ceph-mon[37153]: pgmap v2456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:51:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:52 smithi047 ceph-mon[36835]: pgmap v2457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3734.439162 secs 2024-07-27T07:51:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:52 smithi029 ceph-mon[32610]: pgmap v2457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3734.439162 secs 2024-07-27T07:51:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:52 smithi077 ceph-mon[37153]: pgmap v2457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3734.439162 secs 2024-07-27T07:51:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:54 smithi047 ceph-mon[36835]: pgmap v2458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:54 smithi029 ceph-mon[32610]: pgmap v2458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:54 smithi077 ceph-mon[37153]: pgmap v2458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:51:55.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:51:55.954+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:51:55.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:51:55.954+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:51:55.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:51:55.954+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:51:56.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:51:56.029+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:51:56.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:51:56.029+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:51:56.031 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:51:56.029+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:51:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:56 smithi047 ceph-mon[36835]: pgmap v2459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:56 smithi029 ceph-mon[32610]: pgmap v2459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:56 smithi077 ceph-mon[37153]: pgmap v2459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:51:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3739.439261 secs 2024-07-27T07:51:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3739.439261 secs 2024-07-27T07:51:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3739.439261 secs 2024-07-27T07:51:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:51:58 smithi047 ceph-mon[36835]: pgmap v2460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:51:58 smithi029 ceph-mon[32610]: pgmap v2460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:51:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:51:58 smithi077 ceph-mon[37153]: pgmap v2460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:00 smithi047 ceph-mon[36835]: pgmap v2461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:52:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:00 smithi029 ceph-mon[32610]: pgmap v2461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:52:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:00 smithi077 ceph-mon[37153]: pgmap v2461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:52:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:02 smithi047 ceph-mon[36835]: pgmap v2462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3744.439400 secs 2024-07-27T07:52:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:02 smithi029 ceph-mon[32610]: pgmap v2462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3744.439400 secs 2024-07-27T07:52:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:02 smithi077 ceph-mon[37153]: pgmap v2462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3744.439400 secs 2024-07-27T07:52:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:04 smithi047 ceph-mon[36835]: pgmap v2463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:04 smithi029 ceph-mon[32610]: pgmap v2463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:04 smithi077 ceph-mon[37153]: pgmap v2463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:06 smithi047 ceph-mon[36835]: pgmap v2464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:06 smithi029 ceph-mon[32610]: pgmap v2464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:06 smithi077 ceph-mon[37153]: pgmap v2464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3749.439524 secs 2024-07-27T07:52:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3749.439524 secs 2024-07-27T07:52:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3749.439524 secs 2024-07-27T07:52:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:08 smithi047 ceph-mon[36835]: pgmap v2465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:08 smithi029 ceph-mon[32610]: pgmap v2465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:09.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:08 smithi077 ceph-mon[37153]: pgmap v2465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:10.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:10.969+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:52:10.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:10.969+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:52:10.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:10.969+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:52:11.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:11.035+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:52:11.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:11.035+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:52:11.036 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:11.035+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:52:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:10 smithi047 ceph-mon[36835]: pgmap v2466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:52:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:10 smithi029 ceph-mon[32610]: pgmap v2466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:52:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:10 smithi077 ceph-mon[37153]: pgmap v2466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:52:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:12 smithi047 ceph-mon[36835]: pgmap v2467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3754.439645 secs 2024-07-27T07:52:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:12 smithi029 ceph-mon[32610]: pgmap v2467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3754.439645 secs 2024-07-27T07:52:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:12 smithi077 ceph-mon[37153]: pgmap v2467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3754.439645 secs 2024-07-27T07:52:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:14 smithi047 ceph-mon[36835]: pgmap v2468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:14 smithi029 ceph-mon[32610]: pgmap v2468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:14 smithi077 ceph-mon[37153]: pgmap v2468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:16 smithi047 ceph-mon[36835]: pgmap v2469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:16 smithi029 ceph-mon[32610]: pgmap v2469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:16 smithi077 ceph-mon[37153]: pgmap v2469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3759.439768 secs 2024-07-27T07:52:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3759.439768 secs 2024-07-27T07:52:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3759.439768 secs 2024-07-27T07:52:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:18 smithi047 ceph-mon[36835]: pgmap v2470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:18 smithi029 ceph-mon[32610]: pgmap v2470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:18 smithi077 ceph-mon[37153]: pgmap v2470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:20 smithi047 ceph-mon[36835]: pgmap v2471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:52:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:20 smithi029 ceph-mon[32610]: pgmap v2471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:52:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:20 smithi077 ceph-mon[37153]: pgmap v2471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:52:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:22 smithi047 ceph-mon[36835]: pgmap v2472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3764.439889 secs 2024-07-27T07:52:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:22 smithi029 ceph-mon[32610]: pgmap v2472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3764.439889 secs 2024-07-27T07:52:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:22 smithi077 ceph-mon[37153]: pgmap v2472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3764.439889 secs 2024-07-27T07:52:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:24 smithi047 ceph-mon[36835]: pgmap v2473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:24 smithi029 ceph-mon[32610]: pgmap v2473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:24 smithi077 ceph-mon[37153]: pgmap v2473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:25.977+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:52:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:25.977+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:52:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:25.977+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:52:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:26.049+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:52:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:26.049+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:52:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:26.049+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:52:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:26 smithi047 ceph-mon[36835]: pgmap v2474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:26 smithi029 ceph-mon[32610]: pgmap v2474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:26 smithi077 ceph-mon[37153]: pgmap v2474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3769.440005 secs 2024-07-27T07:52:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3769.440005 secs 2024-07-27T07:52:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3769.440005 secs 2024-07-27T07:52:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:28 smithi029 ceph-mon[32610]: pgmap v2475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:28 smithi077 ceph-mon[37153]: pgmap v2475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:28 smithi047 ceph-mon[36835]: pgmap v2475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:30 smithi029 ceph-mon[32610]: pgmap v2476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:52:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:30 smithi077 ceph-mon[37153]: pgmap v2476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:52:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:30 smithi047 ceph-mon[36835]: pgmap v2476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:52:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:52:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:52:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:52:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:52:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:52:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:52:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:52:32.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:52:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:52:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:52:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:52:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:52:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:52:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:52:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:52:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:52:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:52:32.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:52:32.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:52:32.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:52:32.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:52:32.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:52:32.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:52:32.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:52:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:32 smithi029 ceph-mon[32610]: pgmap v2477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3774.440072 secs 2024-07-27T07:52:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:32 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:52:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:32 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:52:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:32 smithi029 ceph-mon[32610]: pgmap v2478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:52:33.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:32 smithi029 ceph-mon[32610]: pgmap v2479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:52:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:32 smithi077 ceph-mon[37153]: pgmap v2477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3774.440072 secs 2024-07-27T07:52:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:32 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:52:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:32 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:52:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:32 smithi077 ceph-mon[37153]: pgmap v2478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:52:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:32 smithi077 ceph-mon[37153]: pgmap v2479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:52:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:32 smithi047 ceph-mon[36835]: pgmap v2477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3774.440072 secs 2024-07-27T07:52:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:32 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T07:52:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:32 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T07:52:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:32 smithi047 ceph-mon[36835]: pgmap v2478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:52:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:32 smithi047 ceph-mon[36835]: pgmap v2479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:52:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:34 smithi029 ceph-mon[32610]: pgmap v2480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:52:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:34 smithi077 ceph-mon[37153]: pgmap v2480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:52:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:34 smithi047 ceph-mon[36835]: pgmap v2480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:52:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:36 smithi029 ceph-mon[32610]: pgmap v2481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:52:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:36 smithi077 ceph-mon[37153]: pgmap v2481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:52:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:36 smithi047 ceph-mon[36835]: pgmap v2481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:52:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3779.440196 secs 2024-07-27T07:52:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3779.440196 secs 2024-07-27T07:52:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3779.440196 secs 2024-07-27T07:52:39.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:38 smithi077 ceph-mon[37153]: pgmap v2482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:52:39.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:38 smithi047 ceph-mon[36835]: pgmap v2482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:52:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:38 smithi029 ceph-mon[32610]: pgmap v2482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:52:40.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:40.992+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:52:40.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:40.992+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:52:40.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:40.992+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:52:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:41.065+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:52:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:41.065+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:52:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:41.065+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:52:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:40 smithi077 ceph-mon[37153]: pgmap v2483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:52:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:40 smithi047 ceph-mon[36835]: pgmap v2483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:52:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:40 smithi029 ceph-mon[32610]: pgmap v2483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:52:43.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:43.203+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:52:43.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:43.203+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:52:43.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:43.203+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:52:43.205 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:43.203+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:52:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3784.440296 secs 2024-07-27T07:52:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:42 smithi077 ceph-mon[37153]: pgmap v2484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:52:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3784.440296 secs 2024-07-27T07:52:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:42 smithi047 ceph-mon[36835]: pgmap v2484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:52:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3784.440296 secs 2024-07-27T07:52:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:42 smithi029 ceph-mon[32610]: pgmap v2484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:52:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:44 smithi077 ceph-mon[37153]: pgmap v2485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:44 smithi047 ceph-mon[36835]: pgmap v2485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:44 smithi029 ceph-mon[32610]: pgmap v2485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:46 smithi077 ceph-mon[37153]: pgmap v2486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:46 smithi047 ceph-mon[36835]: pgmap v2486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:46 smithi029 ceph-mon[32610]: pgmap v2486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3789.440445 secs 2024-07-27T07:52:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3789.440445 secs 2024-07-27T07:52:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3789.440445 secs 2024-07-27T07:52:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:48 smithi077 ceph-mon[37153]: pgmap v2487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:52:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:48 smithi047 ceph-mon[36835]: pgmap v2487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:52:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:48 smithi029 ceph-mon[32610]: pgmap v2487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:52:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:50 smithi077 ceph-mon[37153]: pgmap v2488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:52:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:50 smithi047 ceph-mon[36835]: pgmap v2488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:52:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:50 smithi029 ceph-mon[32610]: pgmap v2488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:52:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3794.440588 secs 2024-07-27T07:52:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:52 smithi077 ceph-mon[37153]: pgmap v2489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3794.440588 secs 2024-07-27T07:52:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:52 smithi047 ceph-mon[36835]: pgmap v2489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3794.440588 secs 2024-07-27T07:52:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:52 smithi029 ceph-mon[32610]: pgmap v2489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:52:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:54 smithi077 ceph-mon[37153]: pgmap v2490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:54 smithi047 ceph-mon[36835]: pgmap v2490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:54 smithi029 ceph-mon[32610]: pgmap v2490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:52:56.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:56.007+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:52:56.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:56.007+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:52:56.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:56.007+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:52:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:56.076+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:52:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:56.076+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:52:56.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:52:56.076+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:52:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:57 smithi077 ceph-mon[37153]: pgmap v2491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:57 smithi047 ceph-mon[36835]: pgmap v2491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:57 smithi029 ceph-mon[32610]: pgmap v2491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:52:58.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:58.205+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:52:58.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:58.205+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:52:58.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:58.205+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:52:58.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:52:58.205+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:52:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3799.440706 secs 2024-07-27T07:52:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3799.440706 secs 2024-07-27T07:52:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3799.440706 secs 2024-07-27T07:52:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:52:59 smithi077 ceph-mon[37153]: pgmap v2492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:52:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:52:59 smithi047 ceph-mon[36835]: pgmap v2492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:52:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:52:59 smithi029 ceph-mon[32610]: pgmap v2492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:53:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:01 smithi077 ceph-mon[37153]: pgmap v2493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:53:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:01 smithi047 ceph-mon[36835]: pgmap v2493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:53:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:01 smithi029 ceph-mon[32610]: pgmap v2493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:53:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3804.440864 secs 2024-07-27T07:53:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:03 smithi077 ceph-mon[37153]: pgmap v2494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3804.440864 secs 2024-07-27T07:53:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:03 smithi047 ceph-mon[36835]: pgmap v2494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3804.440864 secs 2024-07-27T07:53:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:03 smithi029 ceph-mon[32610]: pgmap v2494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:05 smithi077 ceph-mon[37153]: pgmap v2495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:53:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:05 smithi047 ceph-mon[36835]: pgmap v2495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:53:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:05 smithi029 ceph-mon[32610]: pgmap v2495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:53:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:07 smithi077 ceph-mon[37153]: pgmap v2496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:53:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:07 smithi047 ceph-mon[36835]: pgmap v2496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:53:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:07 smithi029 ceph-mon[32610]: pgmap v2496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:53:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3809.440964 secs 2024-07-27T07:53:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3809.440964 secs 2024-07-27T07:53:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3809.440964 secs 2024-07-27T07:53:09.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:09 smithi047 ceph-mon[36835]: pgmap v2497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:53:09.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:09 smithi077 ceph-mon[37153]: pgmap v2497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:53:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:09 smithi029 ceph-mon[32610]: pgmap v2497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:53:11.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:53:11.014+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:53:11.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:53:11.014+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:53:11.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:53:11.014+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:53:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:53:11.092+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:53:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:53:11.092+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:53:11.093 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:53:11.092+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:53:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:11 smithi077 ceph-mon[37153]: pgmap v2498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:53:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:11 smithi047 ceph-mon[36835]: pgmap v2498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:53:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:11 smithi029 ceph-mon[32610]: pgmap v2498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:53:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3814.441067 secs 2024-07-27T07:53:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:13 smithi077 ceph-mon[37153]: pgmap v2499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3814.441067 secs 2024-07-27T07:53:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:13 smithi047 ceph-mon[36835]: pgmap v2499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3814.441067 secs 2024-07-27T07:53:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:13 smithi029 ceph-mon[32610]: pgmap v2499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:15 smithi077 ceph-mon[37153]: pgmap v2500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:53:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:15 smithi047 ceph-mon[36835]: pgmap v2500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:53:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:15 smithi029 ceph-mon[32610]: pgmap v2500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:53:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:17 smithi077 ceph-mon[37153]: pgmap v2501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:53:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:17 smithi047 ceph-mon[36835]: pgmap v2501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:53:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:17 smithi029 ceph-mon[32610]: pgmap v2501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:53:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3819.441193 secs 2024-07-27T07:53:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3819.441193 secs 2024-07-27T07:53:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3819.441193 secs 2024-07-27T07:53:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:19 smithi077 ceph-mon[37153]: pgmap v2502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:53:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:19 smithi047 ceph-mon[36835]: pgmap v2502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:53:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:19 smithi029 ceph-mon[32610]: pgmap v2502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:53:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:21 smithi077 ceph-mon[37153]: pgmap v2503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:53:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:21 smithi047 ceph-mon[36835]: pgmap v2503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:53:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:21 smithi029 ceph-mon[32610]: pgmap v2503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:53:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3824.441315 secs 2024-07-27T07:53:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:23 smithi077 ceph-mon[37153]: pgmap v2504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3824.441315 secs 2024-07-27T07:53:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:23 smithi047 ceph-mon[36835]: pgmap v2504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3824.441315 secs 2024-07-27T07:53:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:23 smithi029 ceph-mon[32610]: pgmap v2504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:25 smithi077 ceph-mon[37153]: pgmap v2505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:53:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:25 smithi047 ceph-mon[36835]: pgmap v2505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:53:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:25 smithi029 ceph-mon[32610]: pgmap v2505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:53:26.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:53:26.029+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:53:26.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:53:26.029+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:53:26.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:53:26.029+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:53:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:53:26.102+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:53:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:53:26.102+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:53:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:53:26.102+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:53:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:27 smithi029 ceph-mon[32610]: pgmap v2506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:53:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:27 smithi077 ceph-mon[37153]: pgmap v2506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:53:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:27 smithi047 ceph-mon[36835]: pgmap v2506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:53:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3829.441457 secs 2024-07-27T07:53:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3829.441457 secs 2024-07-27T07:53:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3829.441457 secs 2024-07-27T07:53:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:29 smithi077 ceph-mon[37153]: pgmap v2507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:53:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:29 smithi047 ceph-mon[36835]: pgmap v2507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:53:29.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:29 smithi029 ceph-mon[32610]: pgmap v2507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:53:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:31 smithi077 ceph-mon[37153]: pgmap v2508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:53:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:31 smithi047 ceph-mon[36835]: pgmap v2508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:53:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:31 smithi029 ceph-mon[32610]: pgmap v2508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:53:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:53:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:53:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:53:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3834.441598 secs 2024-07-27T07:53:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: pgmap v2509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:53:33.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:53:33.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:53:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3834.441598 secs 2024-07-27T07:53:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: pgmap v2509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:53:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:53:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:53:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3834.441598 secs 2024-07-27T07:53:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: pgmap v2509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T07:53:33.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:53:33.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:53:33.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:53:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:34 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:53:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:34 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:53:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:34 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:53:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:34 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:53:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:34 smithi077 ceph-mon[37153]: pgmap v2510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:53:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:34 smithi077 ceph-mon[37153]: pgmap v2511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:53:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:34 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:53:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:34 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:53:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:34 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:53:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:34 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:53:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:34 smithi047 ceph-mon[36835]: pgmap v2510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:53:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:34 smithi047 ceph-mon[36835]: pgmap v2511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:53:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:34 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T07:53:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:34 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T07:53:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:34 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T07:53:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:34 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T07:53:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:34 smithi029 ceph-mon[32610]: pgmap v2510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T07:53:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:34 smithi029 ceph-mon[32610]: pgmap v2511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:53:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:36 smithi077 ceph-mon[37153]: pgmap v2512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:53:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:36 smithi047 ceph-mon[36835]: pgmap v2512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:53:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:36 smithi029 ceph-mon[32610]: pgmap v2512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:53:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:38 smithi077 ceph-mon[37153]: pgmap v2513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:53:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3839.441751 secs 2024-07-27T07:53:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:38 smithi047 ceph-mon[36835]: pgmap v2513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:53:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3839.441751 secs 2024-07-27T07:53:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:38 smithi029 ceph-mon[32610]: pgmap v2513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:53:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3839.441751 secs 2024-07-27T07:53:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:40 smithi077 ceph-mon[37153]: pgmap v2514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:53:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:40 smithi047 ceph-mon[36835]: pgmap v2514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:53:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:40 smithi029 ceph-mon[32610]: pgmap v2514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:53:41.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:53:41.039+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:53:41.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:53:41.039+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:53:41.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:53:41.039+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:53:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:53:41.111+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:53:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:53:41.111+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:53:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:53:41.111+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:53:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:42 smithi077 ceph-mon[37153]: pgmap v2515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:53:42.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:42 smithi029 ceph-mon[32610]: pgmap v2515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:53:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:42 smithi047 ceph-mon[36835]: pgmap v2515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:53:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3844.441873 secs 2024-07-27T07:53:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3844.441873 secs 2024-07-27T07:53:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3844.441873 secs 2024-07-27T07:53:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:44 smithi047 ceph-mon[36835]: pgmap v2516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:53:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:44 smithi029 ceph-mon[32610]: pgmap v2516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:53:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:44 smithi077 ceph-mon[37153]: pgmap v2516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:53:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:46 smithi047 ceph-mon[36835]: pgmap v2517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:53:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:46 smithi029 ceph-mon[32610]: pgmap v2517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:53:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:46 smithi077 ceph-mon[37153]: pgmap v2517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:53:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:48 smithi047 ceph-mon[36835]: pgmap v2518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3849.441941 secs 2024-07-27T07:53:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:48 smithi029 ceph-mon[32610]: pgmap v2518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3849.441941 secs 2024-07-27T07:53:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:48 smithi077 ceph-mon[37153]: pgmap v2518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3849.441941 secs 2024-07-27T07:53:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:50 smithi047 ceph-mon[36835]: pgmap v2519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:53:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:50 smithi029 ceph-mon[32610]: pgmap v2519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:53:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:50 smithi077 ceph-mon[37153]: pgmap v2519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:53:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:52 smithi047 ceph-mon[36835]: pgmap v2520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:53:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:52 smithi029 ceph-mon[32610]: pgmap v2520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:53:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:52 smithi077 ceph-mon[37153]: pgmap v2520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:53:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3854.442065 secs 2024-07-27T07:53:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3854.442065 secs 2024-07-27T07:53:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3854.442065 secs 2024-07-27T07:53:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:54 smithi047 ceph-mon[36835]: pgmap v2521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:54 smithi029 ceph-mon[32610]: pgmap v2521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:54 smithi077 ceph-mon[37153]: pgmap v2521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:56.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:53:56.043+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:53:56.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:53:56.043+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:53:56.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:53:56.043+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:53:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:53:56.127+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:53:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:53:56.127+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:53:56.128 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:53:56.127+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:53:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:56 smithi047 ceph-mon[36835]: pgmap v2522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:53:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:56 smithi029 ceph-mon[32610]: pgmap v2522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:53:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:56 smithi077 ceph-mon[37153]: pgmap v2522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:53:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:58 smithi047 ceph-mon[36835]: pgmap v2523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:53:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3859.442214 secs 2024-07-27T07:53:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:58 smithi029 ceph-mon[32610]: pgmap v2523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:53:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3859.442214 secs 2024-07-27T07:53:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:58 smithi077 ceph-mon[37153]: pgmap v2523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:53:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:53:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3859.442214 secs 2024-07-27T07:54:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:00 smithi047 ceph-mon[36835]: pgmap v2524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:00 smithi029 ceph-mon[32610]: pgmap v2524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:00 smithi077 ceph-mon[37153]: pgmap v2524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:02 smithi047 ceph-mon[36835]: pgmap v2525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:02 smithi029 ceph-mon[32610]: pgmap v2525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:02 smithi077 ceph-mon[37153]: pgmap v2525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3864.442337 secs 2024-07-27T07:54:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3864.442337 secs 2024-07-27T07:54:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3864.442337 secs 2024-07-27T07:54:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:04 smithi047 ceph-mon[36835]: pgmap v2526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:04 smithi029 ceph-mon[32610]: pgmap v2526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:04 smithi077 ceph-mon[37153]: pgmap v2526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:06 smithi047 ceph-mon[36835]: pgmap v2527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:54:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:06 smithi029 ceph-mon[32610]: pgmap v2527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:54:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:06 smithi077 ceph-mon[37153]: pgmap v2527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:54:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:08 smithi047 ceph-mon[36835]: pgmap v2528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3869.442491 secs 2024-07-27T07:54:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:08 smithi029 ceph-mon[32610]: pgmap v2528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3869.442491 secs 2024-07-27T07:54:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:08 smithi077 ceph-mon[37153]: pgmap v2528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3869.442491 secs 2024-07-27T07:54:10.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:10 smithi077 ceph-mon[37153]: pgmap v2529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:10 smithi047 ceph-mon[36835]: pgmap v2529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:10 smithi029 ceph-mon[32610]: pgmap v2529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:11.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:54:11.059+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:54:11.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:54:11.059+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:54:11.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:54:11.059+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:54:11.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:54:11.143+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:54:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:54:11.143+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:54:11.144 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:54:11.143+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:54:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:12 smithi047 ceph-mon[36835]: pgmap v2530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:12 smithi029 ceph-mon[32610]: pgmap v2530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:12 smithi077 ceph-mon[37153]: pgmap v2530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3874.442623 secs 2024-07-27T07:54:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3874.442623 secs 2024-07-27T07:54:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3874.442623 secs 2024-07-27T07:54:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:14 smithi029 ceph-mon[32610]: pgmap v2531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:14 smithi047 ceph-mon[36835]: pgmap v2531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:14 smithi077 ceph-mon[37153]: pgmap v2531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:16 smithi047 ceph-mon[36835]: pgmap v2532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:54:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:16 smithi029 ceph-mon[32610]: pgmap v2532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:54:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:16 smithi077 ceph-mon[37153]: pgmap v2532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:54:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:18 smithi047 ceph-mon[36835]: pgmap v2533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3879.442757 secs 2024-07-27T07:54:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:18 smithi029 ceph-mon[32610]: pgmap v2533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3879.442757 secs 2024-07-27T07:54:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:18 smithi077 ceph-mon[37153]: pgmap v2533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3879.442757 secs 2024-07-27T07:54:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:20 smithi047 ceph-mon[36835]: pgmap v2534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:20 smithi029 ceph-mon[32610]: pgmap v2534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:20 smithi077 ceph-mon[37153]: pgmap v2534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:22 smithi047 ceph-mon[36835]: pgmap v2535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:54:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:22 smithi029 ceph-mon[32610]: pgmap v2535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:54:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:22 smithi077 ceph-mon[37153]: pgmap v2535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:54:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3884.442886 secs 2024-07-27T07:54:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3884.442886 secs 2024-07-27T07:54:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3884.442886 secs 2024-07-27T07:54:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:24 smithi029 ceph-mon[32610]: pgmap v2536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:24 smithi077 ceph-mon[37153]: pgmap v2536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:24 smithi047 ceph-mon[36835]: pgmap v2536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:26.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:54:26.059+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:54:26.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:54:26.059+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:54:26.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:54:26.059+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:54:26.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:54:26.154+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:54:26.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:54:26.154+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:54:26.156 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:54:26.154+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:54:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:26 smithi029 ceph-mon[32610]: pgmap v2537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:54:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:26 smithi077 ceph-mon[37153]: pgmap v2537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:54:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:26 smithi047 ceph-mon[36835]: pgmap v2537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:54:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:28 smithi029 ceph-mon[32610]: pgmap v2538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3889.443006 secs 2024-07-27T07:54:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:28 smithi077 ceph-mon[37153]: pgmap v2538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3889.443006 secs 2024-07-27T07:54:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:28 smithi047 ceph-mon[36835]: pgmap v2538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3889.443006 secs 2024-07-27T07:54:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:30 smithi029 ceph-mon[32610]: pgmap v2539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:30 smithi077 ceph-mon[37153]: pgmap v2539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:30 smithi047 ceph-mon[36835]: pgmap v2539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:32 smithi029 ceph-mon[32610]: pgmap v2540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:32 smithi077 ceph-mon[37153]: pgmap v2540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:32 smithi047 ceph-mon[36835]: pgmap v2540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3894.443075 secs 2024-07-27T07:54:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:54:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:54:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:54:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:54:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3894.443075 secs 2024-07-27T07:54:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:54:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:54:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:54:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:54:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3894.443075 secs 2024-07-27T07:54:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:54:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:54:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:54:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:54:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:34 smithi077 ceph-mon[37153]: pgmap v2541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:34 smithi077 ceph-mon[37153]: pgmap v2542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8 op/s 2024-07-27T07:54:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:34 smithi077 ceph-mon[37153]: pgmap v2543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:54:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:34 smithi047 ceph-mon[36835]: pgmap v2541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:34 smithi047 ceph-mon[36835]: pgmap v2542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8 op/s 2024-07-27T07:54:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:34 smithi047 ceph-mon[36835]: pgmap v2543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:54:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:34 smithi029 ceph-mon[32610]: pgmap v2541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:54:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:34 smithi029 ceph-mon[32610]: pgmap v2542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8 op/s 2024-07-27T07:54:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:34 smithi029 ceph-mon[32610]: pgmap v2543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:54:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:35 smithi077 ceph-mon[37153]: pgmap v2544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:54:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:35 smithi047 ceph-mon[36835]: pgmap v2544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:54:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:35 smithi029 ceph-mon[32610]: pgmap v2544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:54:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3899.443158 secs 2024-07-27T07:54:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:37 smithi077 ceph-mon[37153]: pgmap v2545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:54:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3899.443158 secs 2024-07-27T07:54:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:37 smithi047 ceph-mon[36835]: pgmap v2545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:54:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3899.443158 secs 2024-07-27T07:54:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:37 smithi029 ceph-mon[32610]: pgmap v2545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:54:40.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:39 smithi047 ceph-mon[36835]: pgmap v2546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:54:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:39 smithi077 ceph-mon[37153]: pgmap v2546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:54:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:39 smithi029 ceph-mon[32610]: pgmap v2546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:54:41.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:54:41.061+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:54:41.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:54:41.061+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:54:41.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:54:41.061+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:54:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:54:41.169+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:54:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:54:41.169+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:54:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:54:41.169+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:54:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:41 smithi077 ceph-mon[37153]: pgmap v2547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:54:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:41 smithi047 ceph-mon[36835]: pgmap v2547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:54:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:41 smithi029 ceph-mon[32610]: pgmap v2547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T07:54:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3904.443289 secs 2024-07-27T07:54:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3904.443289 secs 2024-07-27T07:54:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3904.443289 secs 2024-07-27T07:54:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:43 smithi077 ceph-mon[37153]: pgmap v2548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:54:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:43 smithi047 ceph-mon[36835]: pgmap v2548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:54:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:43 smithi029 ceph-mon[32610]: pgmap v2548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:54:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:45 smithi077 ceph-mon[37153]: pgmap v2549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:45 smithi047 ceph-mon[36835]: pgmap v2549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:45 smithi029 ceph-mon[32610]: pgmap v2549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3909.443432 secs 2024-07-27T07:54:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:47 smithi077 ceph-mon[37153]: pgmap v2550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:54:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3909.443432 secs 2024-07-27T07:54:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:47 smithi047 ceph-mon[36835]: pgmap v2550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:54:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3909.443432 secs 2024-07-27T07:54:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:47 smithi029 ceph-mon[32610]: pgmap v2550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:54:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:49 smithi077 ceph-mon[37153]: pgmap v2551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:49 smithi047 ceph-mon[36835]: pgmap v2551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:49 smithi029 ceph-mon[32610]: pgmap v2551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:54:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:51 smithi077 ceph-mon[37153]: pgmap v2552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:51 smithi047 ceph-mon[36835]: pgmap v2552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:51 smithi029 ceph-mon[32610]: pgmap v2552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3914.443573 secs 2024-07-27T07:54:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3914.443573 secs 2024-07-27T07:54:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3914.443573 secs 2024-07-27T07:54:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:54 smithi077 ceph-mon[37153]: pgmap v2553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:54:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:54 smithi047 ceph-mon[36835]: pgmap v2553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:54:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:54 smithi029 ceph-mon[32610]: pgmap v2553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:54:56.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:54:56.077+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:54:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:54:56.077+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:54:56.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:54:56.077+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:54:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:54:56.171+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:54:56.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:54:56.171+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:54:56.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:54:56.171+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:54:56.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:56 smithi029 ceph-mon[32610]: pgmap v2554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:56 smithi077 ceph-mon[37153]: pgmap v2554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:56 smithi047 ceph-mon[36835]: pgmap v2554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:54:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3919.443725 secs 2024-07-27T07:54:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:54:58 smithi077 ceph-mon[37153]: pgmap v2555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:54:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3919.443725 secs 2024-07-27T07:54:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:54:58 smithi047 ceph-mon[36835]: pgmap v2555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:54:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3919.443725 secs 2024-07-27T07:54:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:54:58 smithi029 ceph-mon[32610]: pgmap v2555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:55:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:00 smithi077 ceph-mon[37153]: pgmap v2556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:00 smithi047 ceph-mon[36835]: pgmap v2556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:00 smithi029 ceph-mon[32610]: pgmap v2556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:02 smithi077 ceph-mon[37153]: pgmap v2557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:02 smithi047 ceph-mon[36835]: pgmap v2557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:02 smithi029 ceph-mon[32610]: pgmap v2557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3924.443853 secs 2024-07-27T07:55:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3924.443853 secs 2024-07-27T07:55:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3924.443853 secs 2024-07-27T07:55:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:04 smithi077 ceph-mon[37153]: pgmap v2558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:55:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:04 smithi047 ceph-mon[36835]: pgmap v2558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:55:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:04 smithi029 ceph-mon[32610]: pgmap v2558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:55:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:06 smithi077 ceph-mon[37153]: pgmap v2559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:06.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:06 smithi047 ceph-mon[36835]: pgmap v2559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:06 smithi029 ceph-mon[32610]: pgmap v2559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3929.443936 secs 2024-07-27T07:55:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:08 smithi077 ceph-mon[37153]: pgmap v2560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:55:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3929.443936 secs 2024-07-27T07:55:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:08 smithi047 ceph-mon[36835]: pgmap v2560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:55:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3929.443936 secs 2024-07-27T07:55:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:08 smithi029 ceph-mon[32610]: pgmap v2560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:55:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:10 smithi077 ceph-mon[37153]: pgmap v2561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:10 smithi047 ceph-mon[36835]: pgmap v2561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:10 smithi029 ceph-mon[32610]: pgmap v2561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:11.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:11.078+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:55:11.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:11.078+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:55:11.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:11.078+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:55:11.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:11.182+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:55:11.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:11.182+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:55:11.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:11.182+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:55:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:12 smithi077 ceph-mon[37153]: pgmap v2562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:12 smithi047 ceph-mon[36835]: pgmap v2562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:12 smithi029 ceph-mon[32610]: pgmap v2562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3934.444065 secs 2024-07-27T07:55:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3934.444065 secs 2024-07-27T07:55:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3934.444065 secs 2024-07-27T07:55:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:14 smithi077 ceph-mon[37153]: pgmap v2563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:55:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:14 smithi047 ceph-mon[36835]: pgmap v2563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:55:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:14 smithi029 ceph-mon[32610]: pgmap v2563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:55:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:16 smithi029 ceph-mon[32610]: pgmap v2564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:16 smithi077 ceph-mon[37153]: pgmap v2564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:16 smithi047 ceph-mon[36835]: pgmap v2564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3939.444191 secs 2024-07-27T07:55:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:18 smithi077 ceph-mon[37153]: pgmap v2565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:55:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3939.444191 secs 2024-07-27T07:55:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:18 smithi047 ceph-mon[36835]: pgmap v2565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:55:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3939.444191 secs 2024-07-27T07:55:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:18 smithi029 ceph-mon[32610]: pgmap v2565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:55:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:20 smithi077 ceph-mon[37153]: pgmap v2566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:20 smithi047 ceph-mon[36835]: pgmap v2566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:20 smithi029 ceph-mon[32610]: pgmap v2566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:22 smithi077 ceph-mon[37153]: pgmap v2567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:22 smithi047 ceph-mon[36835]: pgmap v2567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:22 smithi029 ceph-mon[32610]: pgmap v2567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3944.444317 secs 2024-07-27T07:55:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3944.444317 secs 2024-07-27T07:55:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3944.444317 secs 2024-07-27T07:55:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:24 smithi077 ceph-mon[37153]: pgmap v2568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:55:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:24 smithi047 ceph-mon[36835]: pgmap v2568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:55:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:24 smithi029 ceph-mon[32610]: pgmap v2568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:55:26.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:26.092+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:55:26.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:26.092+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:55:26.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:26.092+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:55:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:26.189+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:55:26.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:26.189+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:55:26.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:26.189+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:55:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:26 smithi077 ceph-mon[37153]: pgmap v2569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:26 smithi047 ceph-mon[36835]: pgmap v2569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:26 smithi029 ceph-mon[32610]: pgmap v2569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3949.444440 secs 2024-07-27T07:55:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:28 smithi077 ceph-mon[37153]: pgmap v2570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:55:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3949.444440 secs 2024-07-27T07:55:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:28 smithi047 ceph-mon[36835]: pgmap v2570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:55:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3949.444440 secs 2024-07-27T07:55:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:28 smithi029 ceph-mon[32610]: pgmap v2570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:55:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:30 smithi077 ceph-mon[37153]: pgmap v2571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:30 smithi047 ceph-mon[36835]: pgmap v2571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:30 smithi029 ceph-mon[32610]: pgmap v2571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:32 smithi077 ceph-mon[37153]: pgmap v2572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:32 smithi047 ceph-mon[36835]: pgmap v2572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:32 smithi029 ceph-mon[32610]: pgmap v2572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3954.444541 secs 2024-07-27T07:55:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:55:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3954.444541 secs 2024-07-27T07:55:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:55:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3954.444541 secs 2024-07-27T07:55:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:55:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:34 smithi077 ceph-mon[37153]: pgmap v2573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:55:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:55:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:55:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:55:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:34 smithi047 ceph-mon[36835]: pgmap v2573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:55:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:55:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:55:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:55:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:34 smithi029 ceph-mon[32610]: pgmap v2573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:55:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:55:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:55:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:55:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:35 smithi077 ceph-mon[37153]: pgmap v2574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:55:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:35 smithi077 ceph-mon[37153]: pgmap v2575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:55:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:35 smithi047 ceph-mon[36835]: pgmap v2574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:55:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:35 smithi047 ceph-mon[36835]: pgmap v2575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:55:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:35 smithi029 ceph-mon[32610]: pgmap v2574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:55:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:35 smithi029 ceph-mon[32610]: pgmap v2575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:55:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:37 smithi077 ceph-mon[37153]: pgmap v2576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:55:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:37 smithi047 ceph-mon[36835]: pgmap v2576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:55:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:37 smithi029 ceph-mon[32610]: pgmap v2576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:55:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3959.444644 secs 2024-07-27T07:55:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3959.444644 secs 2024-07-27T07:55:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3959.444644 secs 2024-07-27T07:55:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:39 smithi077 ceph-mon[37153]: pgmap v2577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:55:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:39 smithi047 ceph-mon[36835]: pgmap v2577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:55:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:39 smithi029 ceph-mon[32610]: pgmap v2577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:55:41.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:41.097+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:55:41.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:41.097+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:55:41.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:41.097+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:55:41.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:41.204+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:55:41.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:41.204+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:55:41.206 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:41.204+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:55:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:41 smithi077 ceph-mon[37153]: pgmap v2578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:55:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:41 smithi047 ceph-mon[36835]: pgmap v2578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:55:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:41 smithi029 ceph-mon[32610]: pgmap v2578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:55:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:43.229+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:43.229+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:43.229+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T07:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:43.229+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:43.229+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T07:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:43.229+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:43.230+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:55:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:43.230+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:55:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:43 smithi047 ceph-mon[36835]: pgmap v2579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:55:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3964.444769 secs 2024-07-27T07:55:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:43 smithi029 ceph-mon[32610]: pgmap v2579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:55:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3964.444769 secs 2024-07-27T07:55:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:43 smithi077 ceph-mon[37153]: pgmap v2579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:55:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3964.444769 secs 2024-07-27T07:55:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:45 smithi047 ceph-mon[36835]: pgmap v2580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:55:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:45 smithi029 ceph-mon[32610]: pgmap v2580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:55:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:45 smithi077 ceph-mon[37153]: pgmap v2580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:55:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:47 smithi047 ceph-mon[36835]: pgmap v2581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:47 smithi029 ceph-mon[32610]: pgmap v2581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:47 smithi077 ceph-mon[37153]: pgmap v2581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3969.444929 secs 2024-07-27T07:55:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3969.444929 secs 2024-07-27T07:55:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3969.444929 secs 2024-07-27T07:55:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:49 smithi047 ceph-mon[36835]: pgmap v2582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:55:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:49 smithi029 ceph-mon[32610]: pgmap v2582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:55:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:49 smithi077 ceph-mon[37153]: pgmap v2582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:55:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:51 smithi047 ceph-mon[36835]: pgmap v2583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:55:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:51 smithi029 ceph-mon[32610]: pgmap v2583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:55:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:51 smithi077 ceph-mon[37153]: pgmap v2583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:55:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:53 smithi047 ceph-mon[36835]: pgmap v2584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:55:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3974.445012 secs 2024-07-27T07:55:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:53 smithi029 ceph-mon[32610]: pgmap v2584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:55:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3974.445012 secs 2024-07-27T07:55:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:53 smithi077 ceph-mon[37153]: pgmap v2584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:55:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3974.445012 secs 2024-07-27T07:55:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:55 smithi047 ceph-mon[36835]: pgmap v2585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:55 smithi029 ceph-mon[32610]: pgmap v2585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:55 smithi077 ceph-mon[37153]: pgmap v2585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:55:56.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:56.109+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:55:56.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:56.109+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:55:56.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:56.109+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:55:56.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:56.209+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:55:56.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:56.209+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:55:56.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:55:56.209+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:55:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:57 smithi047 ceph-mon[36835]: pgmap v2586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:57 smithi029 ceph-mon[32610]: pgmap v2586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:57 smithi077 ceph-mon[37153]: pgmap v2586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:55:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:58.231+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:55:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:58.231+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:55:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:58.231+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T07:55:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:58.231+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:55:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:58.231+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T07:55:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:58.232+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:55:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:58.232+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:55:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:55:58.232+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:55:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3979.445135 secs 2024-07-27T07:55:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3979.445135 secs 2024-07-27T07:55:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3979.445135 secs 2024-07-27T07:55:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:55:59 smithi047 ceph-mon[36835]: pgmap v2587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:55:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:55:59 smithi029 ceph-mon[32610]: pgmap v2587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:55:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:55:59 smithi077 ceph-mon[37153]: pgmap v2587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:56:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:01 smithi047 ceph-mon[36835]: pgmap v2588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:56:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:01 smithi029 ceph-mon[32610]: pgmap v2588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:56:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:01 smithi077 ceph-mon[37153]: pgmap v2588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:56:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:03 smithi047 ceph-mon[36835]: pgmap v2589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3984.445261 secs 2024-07-27T07:56:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:03 smithi029 ceph-mon[32610]: pgmap v2589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3984.445261 secs 2024-07-27T07:56:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:03 smithi077 ceph-mon[37153]: pgmap v2589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3984.445261 secs 2024-07-27T07:56:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:05 smithi047 ceph-mon[36835]: pgmap v2590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:56:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:05 smithi029 ceph-mon[32610]: pgmap v2590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:56:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:05 smithi077 ceph-mon[37153]: pgmap v2590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:56:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:07 smithi047 ceph-mon[36835]: pgmap v2591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:56:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:07 smithi029 ceph-mon[32610]: pgmap v2591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:56:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:07 smithi077 ceph-mon[37153]: pgmap v2591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:56:08.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3989.445338 secs 2024-07-27T07:56:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3989.445338 secs 2024-07-27T07:56:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3989.445338 secs 2024-07-27T07:56:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:09 smithi047 ceph-mon[36835]: pgmap v2592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:56:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:09 smithi029 ceph-mon[32610]: pgmap v2592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:56:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:09 smithi077 ceph-mon[37153]: pgmap v2592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:56:11.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:56:11.125+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:56:11.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:56:11.125+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:56:11.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:56:11.125+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:56:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:56:11.225+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:56:11.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:56:11.225+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:56:11.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:56:11.225+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:56:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:11 smithi047 ceph-mon[36835]: pgmap v2593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:56:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:11 smithi029 ceph-mon[32610]: pgmap v2593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:56:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:11 smithi077 ceph-mon[37153]: pgmap v2593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:56:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:13 smithi047 ceph-mon[36835]: pgmap v2594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3994.445476 secs 2024-07-27T07:56:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:13 smithi029 ceph-mon[32610]: pgmap v2594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3994.445476 secs 2024-07-27T07:56:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:13 smithi077 ceph-mon[37153]: pgmap v2594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3994.445476 secs 2024-07-27T07:56:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:15 smithi047 ceph-mon[36835]: pgmap v2595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:56:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:15 smithi029 ceph-mon[32610]: pgmap v2595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:56:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:15 smithi077 ceph-mon[37153]: pgmap v2595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:56:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:17 smithi047 ceph-mon[36835]: pgmap v2596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:56:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:17 smithi029 ceph-mon[32610]: pgmap v2596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:56:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:17 smithi077 ceph-mon[37153]: pgmap v2596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:56:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 3999.445599 secs 2024-07-27T07:56:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 3999.445599 secs 2024-07-27T07:56:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 3999.445599 secs 2024-07-27T07:56:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:19 smithi047 ceph-mon[36835]: pgmap v2597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:56:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:19 smithi029 ceph-mon[32610]: pgmap v2597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:56:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:19 smithi077 ceph-mon[37153]: pgmap v2597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:56:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:21 smithi047 ceph-mon[36835]: pgmap v2598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:56:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:21 smithi029 ceph-mon[32610]: pgmap v2598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:56:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:21 smithi077 ceph-mon[37153]: pgmap v2598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:56:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:23 smithi047 ceph-mon[36835]: pgmap v2599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4004.445718 secs 2024-07-27T07:56:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:23 smithi029 ceph-mon[32610]: pgmap v2599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4004.445718 secs 2024-07-27T07:56:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:23 smithi077 ceph-mon[37153]: pgmap v2599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4004.445718 secs 2024-07-27T07:56:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:25 smithi047 ceph-mon[36835]: pgmap v2600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:56:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:25 smithi029 ceph-mon[32610]: pgmap v2600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:56:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:25 smithi077 ceph-mon[37153]: pgmap v2600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:56:26.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:56:26.140+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:56:26.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:56:26.140+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:56:26.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:56:26.140+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:56:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:56:26.235+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:56:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:56:26.235+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:56:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:56:26.235+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:56:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:27 smithi047 ceph-mon[36835]: pgmap v2601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:56:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:27 smithi029 ceph-mon[32610]: pgmap v2601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:56:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:27 smithi077 ceph-mon[37153]: pgmap v2601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:56:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4009.445823 secs 2024-07-27T07:56:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4009.445823 secs 2024-07-27T07:56:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4009.445823 secs 2024-07-27T07:56:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:29 smithi029 ceph-mon[32610]: pgmap v2602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:56:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:29 smithi077 ceph-mon[37153]: pgmap v2602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:56:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:29 smithi047 ceph-mon[36835]: pgmap v2602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T07:56:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:31 smithi029 ceph-mon[32610]: pgmap v2603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:56:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:31 smithi077 ceph-mon[37153]: pgmap v2603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:56:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:31 smithi047 ceph-mon[36835]: pgmap v2603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:56:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:33 smithi029 ceph-mon[32610]: pgmap v2604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4014.445954 secs 2024-07-27T07:56:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:33 smithi077 ceph-mon[37153]: pgmap v2604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4014.445954 secs 2024-07-27T07:56:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:33 smithi047 ceph-mon[36835]: pgmap v2604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4014.445954 secs 2024-07-27T07:56:34.541 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:56:34.542 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:34.542 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:56:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:56:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:35 smithi077 ceph-mon[37153]: pgmap v2605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:56:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:35 smithi047 ceph-mon[36835]: pgmap v2605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:56:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:35 smithi029 ceph-mon[32610]: pgmap v2605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:56:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:56:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:56:36.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:56:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:56:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:36.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:56:36.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:56:36.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:56:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:37 smithi077 ceph-mon[37153]: pgmap v2606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:56:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:37 smithi077 ceph-mon[37153]: pgmap v2607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:56:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:37 smithi077 ceph-mon[37153]: pgmap v2608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:56:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:37 smithi047 ceph-mon[36835]: pgmap v2606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:56:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:37 smithi047 ceph-mon[36835]: pgmap v2607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:56:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:37 smithi047 ceph-mon[36835]: pgmap v2608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:56:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:37 smithi029 ceph-mon[32610]: pgmap v2606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:56:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:37 smithi029 ceph-mon[32610]: pgmap v2607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T07:56:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:37 smithi029 ceph-mon[32610]: pgmap v2608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:56:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4019.446039 secs 2024-07-27T07:56:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4019.446039 secs 2024-07-27T07:56:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4019.446039 secs 2024-07-27T07:56:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:39 smithi077 ceph-mon[37153]: pgmap v2609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:56:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:39 smithi047 ceph-mon[36835]: pgmap v2609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:56:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:39 smithi029 ceph-mon[32610]: pgmap v2609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:56:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:56:41.155+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:56:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:56:41.155+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:56:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:56:41.155+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:56:41.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:56:41.251+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:56:41.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:56:41.251+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:56:41.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:56:41.251+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:56:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:41 smithi029 ceph-mon[32610]: pgmap v2610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:56:41.498 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:41 smithi047 ceph-mon[36835]: pgmap v2610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:56:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:41 smithi077 ceph-mon[37153]: pgmap v2610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T07:56:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4024.446166 secs 2024-07-27T07:56:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:43 smithi077 ceph-mon[37153]: pgmap v2611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T07:56:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4024.446166 secs 2024-07-27T07:56:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:43 smithi047 ceph-mon[36835]: pgmap v2611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T07:56:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4024.446166 secs 2024-07-27T07:56:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:43 smithi029 ceph-mon[32610]: pgmap v2611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T07:56:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:45 smithi077 ceph-mon[37153]: pgmap v2612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:56:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:45 smithi047 ceph-mon[36835]: pgmap v2612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:56:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:45 smithi029 ceph-mon[32610]: pgmap v2612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:56:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:46 smithi077 ceph-mon[37153]: pgmap v2613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:56:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:46 smithi047 ceph-mon[36835]: pgmap v2613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:56:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:46 smithi029 ceph-mon[32610]: pgmap v2613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:56:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4029.446237 secs 2024-07-27T07:56:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4029.446237 secs 2024-07-27T07:56:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4029.446237 secs 2024-07-27T07:56:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:48 smithi047 ceph-mon[36835]: pgmap v2614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:56:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:48 smithi029 ceph-mon[32610]: pgmap v2614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:56:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:48 smithi077 ceph-mon[37153]: pgmap v2614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T07:56:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:50 smithi077 ceph-mon[37153]: pgmap v2615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:56:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:50 smithi047 ceph-mon[36835]: pgmap v2615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:56:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:50 smithi029 ceph-mon[32610]: pgmap v2615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:56:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:52 smithi077 ceph-mon[37153]: pgmap v2616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:52 smithi047 ceph-mon[36835]: pgmap v2616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:52 smithi029 ceph-mon[32610]: pgmap v2616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4034.446379 secs 2024-07-27T07:56:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4034.446379 secs 2024-07-27T07:56:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4034.446379 secs 2024-07-27T07:56:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:54 smithi077 ceph-mon[37153]: pgmap v2617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:56:54.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:54 smithi047 ceph-mon[36835]: pgmap v2617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:56:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:54 smithi029 ceph-mon[32610]: pgmap v2617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:56:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:56:56.171+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:56:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:56:56.171+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:56:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:56:56.171+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:56:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:56:56.267+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:56:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:56:56.267+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:56:56.268 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:56:56.267+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:56:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:56 smithi029 ceph-mon[32610]: pgmap v2618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:56 smithi077 ceph-mon[37153]: pgmap v2618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:56 smithi047 ceph-mon[36835]: pgmap v2618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:56:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4039.446501 secs 2024-07-27T07:56:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4039.446501 secs 2024-07-27T07:56:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4039.446501 secs 2024-07-27T07:56:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:56:58 smithi047 ceph-mon[36835]: pgmap v2619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:56:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:56:58 smithi029 ceph-mon[32610]: pgmap v2619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:56:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:56:58 smithi077 ceph-mon[37153]: pgmap v2619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:57:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:00 smithi077 ceph-mon[37153]: pgmap v2620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:00 smithi047 ceph-mon[36835]: pgmap v2620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:00 smithi029 ceph-mon[32610]: pgmap v2620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:02 smithi077 ceph-mon[37153]: pgmap v2621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:02 smithi047 ceph-mon[36835]: pgmap v2621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:02 smithi029 ceph-mon[32610]: pgmap v2621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4044.446622 secs 2024-07-27T07:57:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4044.446622 secs 2024-07-27T07:57:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4044.446622 secs 2024-07-27T07:57:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:04 smithi077 ceph-mon[37153]: pgmap v2622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:57:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:04 smithi047 ceph-mon[36835]: pgmap v2622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:57:04.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:04 smithi029 ceph-mon[32610]: pgmap v2622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:57:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:06 smithi077 ceph-mon[37153]: pgmap v2623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:06 smithi047 ceph-mon[36835]: pgmap v2623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:06 smithi029 ceph-mon[32610]: pgmap v2623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4049.446771 secs 2024-07-27T07:57:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4049.446771 secs 2024-07-27T07:57:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4049.446771 secs 2024-07-27T07:57:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:08 smithi047 ceph-mon[36835]: pgmap v2624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:57:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:08 smithi029 ceph-mon[32610]: pgmap v2624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:57:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:08 smithi077 ceph-mon[37153]: pgmap v2624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:57:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:10 smithi077 ceph-mon[37153]: pgmap v2625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:10 smithi047 ceph-mon[36835]: pgmap v2625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:10 smithi029 ceph-mon[32610]: pgmap v2625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:11.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:57:11.187+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:57:11.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:57:11.187+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:57:11.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:57:11.187+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:57:11.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:57:11.273+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:57:11.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:57:11.273+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:57:11.274 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:57:11.273+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:57:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:12 smithi077 ceph-mon[37153]: pgmap v2626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:12 smithi047 ceph-mon[36835]: pgmap v2626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:12 smithi029 ceph-mon[32610]: pgmap v2626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4054.446894 secs 2024-07-27T07:57:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4054.446894 secs 2024-07-27T07:57:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4054.446894 secs 2024-07-27T07:57:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:14 smithi077 ceph-mon[37153]: pgmap v2627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:57:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:14 smithi047 ceph-mon[36835]: pgmap v2627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:57:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:14 smithi029 ceph-mon[32610]: pgmap v2627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:57:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:16 smithi077 ceph-mon[37153]: pgmap v2628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:16 smithi047 ceph-mon[36835]: pgmap v2628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:16 smithi029 ceph-mon[32610]: pgmap v2628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4059.447012 secs 2024-07-27T07:57:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4059.447012 secs 2024-07-27T07:57:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4059.447012 secs 2024-07-27T07:57:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:18 smithi047 ceph-mon[36835]: pgmap v2629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:57:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:18 smithi029 ceph-mon[32610]: pgmap v2629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:57:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:18 smithi077 ceph-mon[37153]: pgmap v2629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:57:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:20 smithi077 ceph-mon[37153]: pgmap v2630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:20 smithi047 ceph-mon[36835]: pgmap v2630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:20 smithi029 ceph-mon[32610]: pgmap v2630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:22 smithi077 ceph-mon[37153]: pgmap v2631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:22 smithi047 ceph-mon[36835]: pgmap v2631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:22 smithi029 ceph-mon[32610]: pgmap v2631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4064.447141 secs 2024-07-27T07:57:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4064.447141 secs 2024-07-27T07:57:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4064.447141 secs 2024-07-27T07:57:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:24 smithi077 ceph-mon[37153]: pgmap v2632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:57:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:24 smithi047 ceph-mon[36835]: pgmap v2632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:57:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:24 smithi029 ceph-mon[32610]: pgmap v2632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T07:57:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:57:26.201+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:57:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:57:26.201+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:57:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:57:26.201+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:57:26.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:57:26.277+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:57:26.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:57:26.277+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:57:26.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:57:26.277+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:57:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:26 smithi029 ceph-mon[32610]: pgmap v2633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:26 smithi077 ceph-mon[37153]: pgmap v2633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:26 smithi047 ceph-mon[36835]: pgmap v2633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4069.447264 secs 2024-07-27T07:57:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4069.447264 secs 2024-07-27T07:57:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4069.447264 secs 2024-07-27T07:57:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:28 smithi047 ceph-mon[36835]: pgmap v2634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:57:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:28 smithi029 ceph-mon[32610]: pgmap v2634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:57:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:28 smithi077 ceph-mon[37153]: pgmap v2634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:57:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:30 smithi077 ceph-mon[37153]: pgmap v2635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:57:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:30 smithi047 ceph-mon[36835]: pgmap v2635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:57:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:30 smithi029 ceph-mon[32610]: pgmap v2635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T07:57:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:32 smithi077 ceph-mon[37153]: pgmap v2636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:32 smithi047 ceph-mon[36835]: pgmap v2636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:32 smithi029 ceph-mon[32610]: pgmap v2636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4074.447394 secs 2024-07-27T07:57:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4074.447394 secs 2024-07-27T07:57:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4074.447394 secs 2024-07-27T07:57:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:34 smithi077 ceph-mon[37153]: pgmap v2637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:57:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:34 smithi047 ceph-mon[36835]: pgmap v2637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:57:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:34 smithi029 ceph-mon[32610]: pgmap v2637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:57:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:36 smithi029 ceph-mon[32610]: pgmap v2638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:36 smithi077 ceph-mon[37153]: pgmap v2638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:36 smithi047 ceph-mon[36835]: pgmap v2638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:57:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:57:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:57:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:37 smithi077 ceph-mon[37153]: pgmap v2639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:57:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:37 smithi077 ceph-mon[37153]: pgmap v2640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T07:57:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:57:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:57:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:57:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:37 smithi047 ceph-mon[36835]: pgmap v2639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:57:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:37 smithi047 ceph-mon[36835]: pgmap v2640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T07:57:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:57:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:37.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:57:37.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:57:37.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:37 smithi029 ceph-mon[32610]: pgmap v2639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T07:57:37.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:37 smithi029 ceph-mon[32610]: pgmap v2640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T07:57:37.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:57:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4079.447485 secs 2024-07-27T07:57:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4079.447485 secs 2024-07-27T07:57:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4079.447485 secs 2024-07-27T07:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:39 smithi077 ceph-mon[37153]: pgmap v2641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:57:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:39 smithi047 ceph-mon[36835]: pgmap v2641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:57:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:39 smithi029 ceph-mon[32610]: pgmap v2641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T07:57:41.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:57:41.203+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:57:41.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:57:41.203+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:57:41.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:57:41.203+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:57:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:57:41.292+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:57:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:57:41.292+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:57:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:57:41.292+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:57:42.310 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:42 smithi077 ceph-mon[37153]: pgmap v2642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:57:42.399 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:42 smithi047 ceph-mon[36835]: pgmap v2642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:57:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:42 smithi029 ceph-mon[32610]: pgmap v2642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:57:43.322 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4084.447610 secs 2024-07-27T07:57:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4084.447610 secs 2024-07-27T07:57:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4084.447610 secs 2024-07-27T07:57:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:44 smithi077 ceph-mon[37153]: pgmap v2643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:57:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:44 smithi047 ceph-mon[36835]: pgmap v2643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:57:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:44 smithi029 ceph-mon[32610]: pgmap v2643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T07:57:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:46 smithi077 ceph-mon[37153]: pgmap v2644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:57:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:46 smithi047 ceph-mon[36835]: pgmap v2644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:57:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:46 smithi029 ceph-mon[32610]: pgmap v2644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:57:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4089.447739 secs 2024-07-27T07:57:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:48 smithi077 ceph-mon[37153]: pgmap v2645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:57:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4089.447739 secs 2024-07-27T07:57:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:48 smithi047 ceph-mon[36835]: pgmap v2645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:57:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4089.447739 secs 2024-07-27T07:57:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:48 smithi029 ceph-mon[32610]: pgmap v2645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:57:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:50 smithi077 ceph-mon[37153]: pgmap v2646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:57:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:50 smithi047 ceph-mon[36835]: pgmap v2646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:57:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:50 smithi029 ceph-mon[32610]: pgmap v2646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:57:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:52 smithi077 ceph-mon[37153]: pgmap v2647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:52 smithi047 ceph-mon[36835]: pgmap v2647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:52 smithi029 ceph-mon[32610]: pgmap v2647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4094.447883 secs 2024-07-27T07:57:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4094.447883 secs 2024-07-27T07:57:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4094.447883 secs 2024-07-27T07:57:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:54 smithi077 ceph-mon[37153]: pgmap v2648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:57:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:54 smithi047 ceph-mon[36835]: pgmap v2648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:57:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:54 smithi029 ceph-mon[32610]: pgmap v2648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:57:56.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:57:56.218+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:57:56.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:57:56.218+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:57:56.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:57:56.218+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:57:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:57:56.308+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:57:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:57:56.308+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:57:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:57:56.308+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:57:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:56 smithi077 ceph-mon[37153]: pgmap v2649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:56 smithi047 ceph-mon[36835]: pgmap v2649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:56 smithi029 ceph-mon[32610]: pgmap v2649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:57:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4099.447984 secs 2024-07-27T07:57:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:57:58 smithi077 ceph-mon[37153]: pgmap v2650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4099.447984 secs 2024-07-27T07:57:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:57:58 smithi047 ceph-mon[36835]: pgmap v2650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:57:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4099.447984 secs 2024-07-27T07:57:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:57:58 smithi029 ceph-mon[32610]: pgmap v2650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:00 smithi077 ceph-mon[37153]: pgmap v2651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:58:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:00 smithi047 ceph-mon[36835]: pgmap v2651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:58:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:00 smithi029 ceph-mon[32610]: pgmap v2651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:58:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:02 smithi077 ceph-mon[37153]: pgmap v2652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:02 smithi047 ceph-mon[36835]: pgmap v2652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:02 smithi029 ceph-mon[32610]: pgmap v2652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4104.448100 secs 2024-07-27T07:58:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4104.448100 secs 2024-07-27T07:58:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4104.448100 secs 2024-07-27T07:58:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:04 smithi047 ceph-mon[36835]: pgmap v2653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:58:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:04 smithi029 ceph-mon[32610]: pgmap v2653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:58:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:04 smithi077 ceph-mon[37153]: pgmap v2653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:58:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:06 smithi047 ceph-mon[36835]: pgmap v2654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:06 smithi029 ceph-mon[32610]: pgmap v2654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:06 smithi077 ceph-mon[37153]: pgmap v2654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4109.448222 secs 2024-07-27T07:58:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:08 smithi047 ceph-mon[36835]: pgmap v2655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4109.448222 secs 2024-07-27T07:58:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:08 smithi029 ceph-mon[32610]: pgmap v2655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4109.448222 secs 2024-07-27T07:58:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:08 smithi077 ceph-mon[37153]: pgmap v2655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:10 smithi047 ceph-mon[36835]: pgmap v2656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:58:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:10 smithi029 ceph-mon[32610]: pgmap v2656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:58:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:10 smithi077 ceph-mon[37153]: pgmap v2656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:58:11.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:11.234+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:58:11.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:11.234+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:58:11.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:11.234+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:58:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:11.310+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:58:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:11.310+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:58:11.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:11.310+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:58:12.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:12 smithi077 ceph-mon[37153]: pgmap v2657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:12 smithi047 ceph-mon[36835]: pgmap v2657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:12 smithi029 ceph-mon[32610]: pgmap v2657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4114.448307 secs 2024-07-27T07:58:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4114.448307 secs 2024-07-27T07:58:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4114.448307 secs 2024-07-27T07:58:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:14 smithi047 ceph-mon[36835]: pgmap v2658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:58:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:14 smithi029 ceph-mon[32610]: pgmap v2658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:58:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:14 smithi077 ceph-mon[37153]: pgmap v2658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:58:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:16 smithi047 ceph-mon[36835]: pgmap v2659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:16 smithi029 ceph-mon[32610]: pgmap v2659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:16 smithi077 ceph-mon[37153]: pgmap v2659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4119.448444 secs 2024-07-27T07:58:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:18 smithi047 ceph-mon[36835]: pgmap v2660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4119.448444 secs 2024-07-27T07:58:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:18 smithi029 ceph-mon[32610]: pgmap v2660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4119.448444 secs 2024-07-27T07:58:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:18 smithi077 ceph-mon[37153]: pgmap v2660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:20 smithi047 ceph-mon[36835]: pgmap v2661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:58:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:20 smithi029 ceph-mon[32610]: pgmap v2661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:58:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:20 smithi077 ceph-mon[37153]: pgmap v2661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:58:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:22 smithi047 ceph-mon[36835]: pgmap v2662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:22 smithi029 ceph-mon[32610]: pgmap v2662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:22 smithi077 ceph-mon[37153]: pgmap v2662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4124.448543 secs 2024-07-27T07:58:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4124.448543 secs 2024-07-27T07:58:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4124.448543 secs 2024-07-27T07:58:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:24 smithi047 ceph-mon[36835]: pgmap v2663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:58:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:24 smithi029 ceph-mon[32610]: pgmap v2663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:58:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:24 smithi077 ceph-mon[37153]: pgmap v2663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:58:26.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:26.249+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:58:26.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:26.249+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:58:26.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:26.249+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:58:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:26.323+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:58:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:26.323+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:58:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:26.323+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:58:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:26 smithi047 ceph-mon[36835]: pgmap v2664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:26 smithi029 ceph-mon[32610]: pgmap v2664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:26 smithi077 ceph-mon[37153]: pgmap v2664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4129.448665 secs 2024-07-27T07:58:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:28 smithi047 ceph-mon[36835]: pgmap v2665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4129.448665 secs 2024-07-27T07:58:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:28 smithi029 ceph-mon[32610]: pgmap v2665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4129.448665 secs 2024-07-27T07:58:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:28 smithi077 ceph-mon[37153]: pgmap v2665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:30 smithi047 ceph-mon[36835]: pgmap v2666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:58:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:30 smithi029 ceph-mon[32610]: pgmap v2666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:58:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:30 smithi077 ceph-mon[37153]: pgmap v2666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:58:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:32 smithi047 ceph-mon[36835]: pgmap v2667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:32 smithi029 ceph-mon[32610]: pgmap v2667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:32 smithi077 ceph-mon[37153]: pgmap v2667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4134.448763 secs 2024-07-27T07:58:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4134.448763 secs 2024-07-27T07:58:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4134.448763 secs 2024-07-27T07:58:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:34 smithi047 ceph-mon[36835]: pgmap v2668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:58:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:34 smithi029 ceph-mon[32610]: pgmap v2668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:58:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:34 smithi077 ceph-mon[37153]: pgmap v2668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T07:58:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:36 smithi047 ceph-mon[36835]: pgmap v2669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:36 smithi029 ceph-mon[32610]: pgmap v2669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:36 smithi077 ceph-mon[37153]: pgmap v2669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:58:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:58:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:58:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4139.448887 secs 2024-07-27T07:58:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:38 smithi047 ceph-mon[36835]: pgmap v2670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:58:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:58:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:58:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4139.448887 secs 2024-07-27T07:58:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:38 smithi029 ceph-mon[32610]: pgmap v2670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:58:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:58:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:58:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4139.448887 secs 2024-07-27T07:58:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:38 smithi077 ceph-mon[37153]: pgmap v2670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:58:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:58:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:58:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:39 smithi047 ceph-mon[36835]: pgmap v2671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T07:58:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:39 smithi047 ceph-mon[36835]: pgmap v2672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:58:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:39 smithi029 ceph-mon[32610]: pgmap v2671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T07:58:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:39 smithi029 ceph-mon[32610]: pgmap v2672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:58:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:39 smithi077 ceph-mon[37153]: pgmap v2671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T07:58:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:39 smithi077 ceph-mon[37153]: pgmap v2672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:58:41.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:41.264+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:58:41.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:41.264+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:58:41.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:41.264+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:58:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:41.338+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:58:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:41.338+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:58:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:41.338+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:58:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:41 smithi047 ceph-mon[36835]: pgmap v2673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:58:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:41 smithi029 ceph-mon[32610]: pgmap v2673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:58:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:41 smithi077 ceph-mon[37153]: pgmap v2673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T07:58:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:43.256+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:58:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:43.256+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:58:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:43.256+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T07:58:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:43.256+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T07:58:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:43 smithi047 ceph-mon[36835]: pgmap v2674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:58:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4144.449009 secs 2024-07-27T07:58:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:43 smithi029 ceph-mon[32610]: pgmap v2674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:58:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4144.449009 secs 2024-07-27T07:58:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:43 smithi077 ceph-mon[37153]: pgmap v2674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:58:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4144.449009 secs 2024-07-27T07:58:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:45 smithi029 ceph-mon[32610]: pgmap v2675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:58:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:45 smithi077 ceph-mon[37153]: pgmap v2675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:58:45.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:45 smithi047 ceph-mon[36835]: pgmap v2675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:58:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:47 smithi029 ceph-mon[32610]: pgmap v2676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:58:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:47 smithi077 ceph-mon[37153]: pgmap v2676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:58:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:47 smithi047 ceph-mon[36835]: pgmap v2676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:58:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4149.449101 secs 2024-07-27T07:58:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4149.449101 secs 2024-07-27T07:58:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4149.449101 secs 2024-07-27T07:58:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:49 smithi029 ceph-mon[32610]: pgmap v2677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:58:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:49 smithi077 ceph-mon[37153]: pgmap v2677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:58:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:49 smithi047 ceph-mon[36835]: pgmap v2677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:58:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:51 smithi029 ceph-mon[32610]: pgmap v2678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:58:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:51 smithi077 ceph-mon[37153]: pgmap v2678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:58:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:51 smithi047 ceph-mon[36835]: pgmap v2678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:58:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:53 smithi077 ceph-mon[37153]: pgmap v2679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4154.449216 secs 2024-07-27T07:58:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:53 smithi047 ceph-mon[36835]: pgmap v2679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4154.449216 secs 2024-07-27T07:58:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:53 smithi029 ceph-mon[32610]: pgmap v2679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:58:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4154.449216 secs 2024-07-27T07:58:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:55 smithi077 ceph-mon[37153]: pgmap v2680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:58:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:55 smithi047 ceph-mon[36835]: pgmap v2680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:58:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:55 smithi029 ceph-mon[32610]: pgmap v2680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:58:56.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:56.280+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:58:56.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:56.280+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:58:56.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:56.280+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:58:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:56.341+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:58:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:56.341+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:58:56.343 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:58:56.341+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:58:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:57 smithi077 ceph-mon[37153]: pgmap v2681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:57 smithi047 ceph-mon[36835]: pgmap v2681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:57 smithi029 ceph-mon[32610]: pgmap v2681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:58:58.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:58.258+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:58:58.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:58.258+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:58:58.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:58.258+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T07:58:58.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:58:58.258+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T07:58:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4159.449299 secs 2024-07-27T07:58:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4159.449299 secs 2024-07-27T07:58:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4159.449299 secs 2024-07-27T07:58:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:58:59 smithi077 ceph-mon[37153]: pgmap v2682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:58:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:58:59 smithi047 ceph-mon[36835]: pgmap v2682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:58:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:58:59 smithi029 ceph-mon[32610]: pgmap v2682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:01 smithi077 ceph-mon[37153]: pgmap v2683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:01 smithi047 ceph-mon[36835]: pgmap v2683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:01 smithi029 ceph-mon[32610]: pgmap v2683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:03 smithi077 ceph-mon[37153]: pgmap v2684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4164.449435 secs 2024-07-27T07:59:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:03 smithi047 ceph-mon[36835]: pgmap v2684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4164.449435 secs 2024-07-27T07:59:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:03 smithi029 ceph-mon[32610]: pgmap v2684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4164.449435 secs 2024-07-27T07:59:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:05 smithi077 ceph-mon[37153]: pgmap v2685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:05 smithi047 ceph-mon[36835]: pgmap v2685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:05 smithi029 ceph-mon[32610]: pgmap v2685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:07 smithi077 ceph-mon[37153]: pgmap v2686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:59:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:07 smithi047 ceph-mon[36835]: pgmap v2686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:59:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:07 smithi029 ceph-mon[32610]: pgmap v2686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:59:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4169.449495 secs 2024-07-27T07:59:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4169.449495 secs 2024-07-27T07:59:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4169.449495 secs 2024-07-27T07:59:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:09 smithi077 ceph-mon[37153]: pgmap v2687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:09 smithi047 ceph-mon[36835]: pgmap v2687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:09 smithi029 ceph-mon[32610]: pgmap v2687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:11.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:59:11.288+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:59:11.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:59:11.288+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:59:11.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:59:11.288+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:59:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:59:11.357+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:59:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:59:11.357+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:59:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:59:11.357+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:59:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:11 smithi077 ceph-mon[37153]: pgmap v2688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:11 smithi047 ceph-mon[36835]: pgmap v2688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:11 smithi029 ceph-mon[32610]: pgmap v2688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:13 smithi077 ceph-mon[37153]: pgmap v2689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4174.449616 secs 2024-07-27T07:59:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:13 smithi047 ceph-mon[36835]: pgmap v2689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4174.449616 secs 2024-07-27T07:59:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:13 smithi029 ceph-mon[32610]: pgmap v2689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4174.449616 secs 2024-07-27T07:59:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:15 smithi077 ceph-mon[37153]: pgmap v2690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:15 smithi047 ceph-mon[36835]: pgmap v2690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:15 smithi029 ceph-mon[32610]: pgmap v2690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:17 smithi077 ceph-mon[37153]: pgmap v2691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:59:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:17 smithi047 ceph-mon[36835]: pgmap v2691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:59:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:17 smithi029 ceph-mon[32610]: pgmap v2691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:59:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4179.449753 secs 2024-07-27T07:59:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4179.449753 secs 2024-07-27T07:59:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4179.449753 secs 2024-07-27T07:59:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:19 smithi077 ceph-mon[37153]: pgmap v2692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:19 smithi047 ceph-mon[36835]: pgmap v2692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:19 smithi029 ceph-mon[32610]: pgmap v2692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:20 smithi077 ceph-mon[37153]: pgmap v2693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:20 smithi047 ceph-mon[36835]: pgmap v2693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:20.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:20 smithi029 ceph-mon[32610]: pgmap v2693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:22 smithi047 ceph-mon[36835]: pgmap v2694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4184.449883 secs 2024-07-27T07:59:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:22 smithi029 ceph-mon[32610]: pgmap v2694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4184.449883 secs 2024-07-27T07:59:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:22 smithi077 ceph-mon[37153]: pgmap v2694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:23.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4184.449883 secs 2024-07-27T07:59:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:24 smithi047 ceph-mon[36835]: pgmap v2695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:24 smithi029 ceph-mon[32610]: pgmap v2695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:24 smithi077 ceph-mon[37153]: pgmap v2695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:26.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:59:26.295+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:59:26.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:59:26.295+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:59:26.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:59:26.295+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:59:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:59:26.372+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:59:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:59:26.372+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:59:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:59:26.372+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:59:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:26 smithi047 ceph-mon[36835]: pgmap v2696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:59:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:26 smithi029 ceph-mon[32610]: pgmap v2696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:59:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:26 smithi077 ceph-mon[37153]: pgmap v2696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:59:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4189.449950 secs 2024-07-27T07:59:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4189.449950 secs 2024-07-27T07:59:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4189.449950 secs 2024-07-27T07:59:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:28 smithi047 ceph-mon[36835]: pgmap v2697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:28 smithi029 ceph-mon[32610]: pgmap v2697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:28 smithi077 ceph-mon[37153]: pgmap v2697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:30 smithi047 ceph-mon[36835]: pgmap v2698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:30 smithi029 ceph-mon[32610]: pgmap v2698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:30 smithi077 ceph-mon[37153]: pgmap v2698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:32 smithi047 ceph-mon[36835]: pgmap v2699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4194.450073 secs 2024-07-27T07:59:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:32 smithi029 ceph-mon[32610]: pgmap v2699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4194.450073 secs 2024-07-27T07:59:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:32 smithi077 ceph-mon[37153]: pgmap v2699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4194.450073 secs 2024-07-27T07:59:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:34 smithi047 ceph-mon[36835]: pgmap v2700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:34 smithi029 ceph-mon[32610]: pgmap v2700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:34 smithi077 ceph-mon[37153]: pgmap v2700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:36 smithi047 ceph-mon[36835]: pgmap v2701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:59:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:36 smithi029 ceph-mon[32610]: pgmap v2701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:59:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:36 smithi077 ceph-mon[37153]: pgmap v2701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T07:59:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4199.450196 secs 2024-07-27T07:59:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4199.450196 secs 2024-07-27T07:59:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4199.450196 secs 2024-07-27T07:59:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:38 smithi047 ceph-mon[36835]: pgmap v2702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:59:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:59:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:59:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:38 smithi047 ceph-mon[36835]: pgmap v2703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:59:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:38 smithi047 ceph-mon[36835]: pgmap v2704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:59:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:59:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:38 smithi029 ceph-mon[32610]: pgmap v2702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:59:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:59:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:59:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:38 smithi029 ceph-mon[32610]: pgmap v2703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:59:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:38 smithi029 ceph-mon[32610]: pgmap v2704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:59:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:59:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:38 smithi077 ceph-mon[37153]: pgmap v2702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T07:59:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T07:59:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T07:59:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T07:59:39.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:38 smithi077 ceph-mon[37153]: pgmap v2703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T07:59:39.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:38 smithi077 ceph-mon[37153]: pgmap v2704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T07:59:39.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T07:59:41.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:59:41.311+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:59:41.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:59:41.311+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:59:41.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:59:41.311+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:59:41.387+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:59:41.387+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:59:41.389 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:59:41.387+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:59:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:41 smithi047 ceph-mon[36835]: pgmap v2705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:59:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:41 smithi029 ceph-mon[32610]: pgmap v2705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:59:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:41 smithi077 ceph-mon[37153]: pgmap v2705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T07:59:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4204.450318 secs 2024-07-27T07:59:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:43 smithi047 ceph-mon[36835]: pgmap v2706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:59:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4204.450318 secs 2024-07-27T07:59:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:43 smithi029 ceph-mon[32610]: pgmap v2706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:59:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4204.450318 secs 2024-07-27T07:59:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:43 smithi077 ceph-mon[37153]: pgmap v2706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T07:59:45.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:45 smithi047 ceph-mon[36835]: pgmap v2707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:59:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:45 smithi029 ceph-mon[32610]: pgmap v2707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:59:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:45 smithi077 ceph-mon[37153]: pgmap v2707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T07:59:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:47 smithi047 ceph-mon[36835]: pgmap v2708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:59:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:47 smithi029 ceph-mon[32610]: pgmap v2708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:59:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:47 smithi077 ceph-mon[37153]: pgmap v2708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T07:59:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4209.450382 secs 2024-07-27T07:59:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4209.450382 secs 2024-07-27T07:59:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4209.450382 secs 2024-07-27T07:59:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:49 smithi047 ceph-mon[36835]: pgmap v2709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:59:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:49 smithi029 ceph-mon[32610]: pgmap v2709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:59:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:49 smithi077 ceph-mon[37153]: pgmap v2709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T07:59:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:51 smithi047 ceph-mon[36835]: pgmap v2710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:51 smithi029 ceph-mon[32610]: pgmap v2710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:51 smithi077 ceph-mon[37153]: pgmap v2710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T07:59:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4214.450505 secs 2024-07-27T07:59:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:53 smithi029 ceph-mon[32610]: pgmap v2711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4214.450505 secs 2024-07-27T07:59:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:53 smithi077 ceph-mon[37153]: pgmap v2711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:53.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4214.450505 secs 2024-07-27T07:59:53.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:53 smithi047 ceph-mon[36835]: pgmap v2711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:55 smithi029 ceph-mon[32610]: pgmap v2712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:59:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:55 smithi077 ceph-mon[37153]: pgmap v2712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:59:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:55 smithi047 ceph-mon[36835]: pgmap v2712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T07:59:56.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:59:56.314+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T07:59:56.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:59:56.314+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:59:56.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T07:59:56.314+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T07:59:56.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:59:56.397+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T07:59:56.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:59:56.397+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T07:59:56.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T07:59:56.397+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T07:59:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:57 smithi029 ceph-mon[32610]: pgmap v2713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:57 smithi077 ceph-mon[37153]: pgmap v2713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:57 smithi047 ceph-mon[36835]: pgmap v2713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T07:59:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4219.450646 secs 2024-07-27T07:59:58.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4219.450646 secs 2024-07-27T07:59:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4219.450646 secs 2024-07-27T07:59:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 07:59:59 smithi029 ceph-mon[32610]: pgmap v2714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 07:59:59 smithi077 ceph-mon[37153]: pgmap v2714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T07:59:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 07:59:59 smithi047 ceph-mon[36835]: pgmap v2714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:00:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:00:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:00:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:01 smithi077 ceph-mon[37153]: pgmap v2715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:00:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:01 smithi047 ceph-mon[36835]: pgmap v2715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:00:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:01 smithi029 ceph-mon[32610]: pgmap v2715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:00:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4224.450767 secs 2024-07-27T08:00:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:03 smithi077 ceph-mon[37153]: pgmap v2716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4224.450767 secs 2024-07-27T08:00:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:03 smithi047 ceph-mon[36835]: pgmap v2716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4224.450767 secs 2024-07-27T08:00:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:03 smithi029 ceph-mon[32610]: pgmap v2716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:05 smithi077 ceph-mon[37153]: pgmap v2717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:00:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:05 smithi047 ceph-mon[36835]: pgmap v2717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:00:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:05 smithi029 ceph-mon[32610]: pgmap v2717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:00:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:07 smithi077 ceph-mon[37153]: pgmap v2718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:07 smithi047 ceph-mon[36835]: pgmap v2718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:07 smithi029 ceph-mon[32610]: pgmap v2718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4229.450858 secs 2024-07-27T08:00:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4229.450858 secs 2024-07-27T08:00:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4229.450858 secs 2024-07-27T08:00:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:09 smithi077 ceph-mon[37153]: pgmap v2719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:09 smithi047 ceph-mon[36835]: pgmap v2719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:09 smithi029 ceph-mon[32610]: pgmap v2719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:11.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:00:11.329+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:00:11.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:00:11.329+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:00:11.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:00:11.329+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:00:11.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:00:11.403+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:00:11.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:00:11.403+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:00:11.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:00:11.403+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:00:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:11 smithi077 ceph-mon[37153]: pgmap v2720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:00:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:11 smithi047 ceph-mon[36835]: pgmap v2720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:00:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:11 smithi029 ceph-mon[32610]: pgmap v2720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:00:13.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4234.450979 secs 2024-07-27T08:00:13.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:13 smithi047 ceph-mon[36835]: pgmap v2721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4234.450979 secs 2024-07-27T08:00:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:13 smithi077 ceph-mon[37153]: pgmap v2721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4234.450979 secs 2024-07-27T08:00:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:13 smithi029 ceph-mon[32610]: pgmap v2721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:15 smithi077 ceph-mon[37153]: pgmap v2722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:00:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:15 smithi047 ceph-mon[36835]: pgmap v2722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:00:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:15 smithi029 ceph-mon[32610]: pgmap v2722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:00:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:17 smithi077 ceph-mon[37153]: pgmap v2723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:17 smithi047 ceph-mon[36835]: pgmap v2723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:17 smithi029 ceph-mon[32610]: pgmap v2723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4239.451092 secs 2024-07-27T08:00:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4239.451092 secs 2024-07-27T08:00:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4239.451092 secs 2024-07-27T08:00:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:19 smithi077 ceph-mon[37153]: pgmap v2724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:19 smithi047 ceph-mon[36835]: pgmap v2724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:19 smithi029 ceph-mon[32610]: pgmap v2724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:21 smithi077 ceph-mon[37153]: pgmap v2725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:00:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:21 smithi047 ceph-mon[36835]: pgmap v2725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:00:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:21 smithi029 ceph-mon[32610]: pgmap v2725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:00:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4244.451238 secs 2024-07-27T08:00:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:23 smithi077 ceph-mon[37153]: pgmap v2726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4244.451238 secs 2024-07-27T08:00:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:23 smithi047 ceph-mon[36835]: pgmap v2726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4244.451238 secs 2024-07-27T08:00:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:23 smithi029 ceph-mon[32610]: pgmap v2726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:25 smithi077 ceph-mon[37153]: pgmap v2727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:00:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:25 smithi047 ceph-mon[36835]: pgmap v2727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:00:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:25 smithi029 ceph-mon[32610]: pgmap v2727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:00:26.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:00:26.335+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:00:26.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:00:26.335+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:00:26.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:00:26.335+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:00:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:00:26.417+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:00:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:00:26.417+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:00:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:00:26.417+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:00:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:27 smithi029 ceph-mon[32610]: pgmap v2728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:27 smithi077 ceph-mon[37153]: pgmap v2728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:27 smithi047 ceph-mon[36835]: pgmap v2728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4249.451359 secs 2024-07-27T08:00:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4249.451359 secs 2024-07-27T08:00:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4249.451359 secs 2024-07-27T08:00:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:29 smithi077 ceph-mon[37153]: pgmap v2729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:29 smithi047 ceph-mon[36835]: pgmap v2729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:29 smithi029 ceph-mon[32610]: pgmap v2729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:30 smithi077 ceph-mon[37153]: pgmap v2730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:00:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:30 smithi047 ceph-mon[36835]: pgmap v2730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:00:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:30 smithi029 ceph-mon[32610]: pgmap v2730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:00:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4254.451463 secs 2024-07-27T08:00:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:33 smithi047 ceph-mon[36835]: pgmap v2731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4254.451463 secs 2024-07-27T08:00:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:33 smithi029 ceph-mon[32610]: pgmap v2731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4254.451463 secs 2024-07-27T08:00:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:33 smithi077 ceph-mon[37153]: pgmap v2731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:35 smithi047 ceph-mon[36835]: pgmap v2732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:00:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:35 smithi029 ceph-mon[32610]: pgmap v2732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:00:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:35 smithi077 ceph-mon[37153]: pgmap v2732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:00:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:37 smithi029 ceph-mon[32610]: pgmap v2733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:37 smithi077 ceph-mon[37153]: pgmap v2733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:37 smithi047 ceph-mon[36835]: pgmap v2733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4259.451591 secs 2024-07-27T08:00:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4259.451591 secs 2024-07-27T08:00:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4259.451591 secs 2024-07-27T08:00:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:39 smithi029 ceph-mon[32610]: pgmap v2734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:00:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:00:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:00:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:00:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:39 smithi077 ceph-mon[37153]: pgmap v2734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:00:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:00:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:00:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:00:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:39 smithi047 ceph-mon[36835]: pgmap v2734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:00:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:00:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:00:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:00:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:00:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:40 smithi029 ceph-mon[32610]: pgmap v2735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:00:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:40 smithi029 ceph-mon[32610]: pgmap v2736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:00:40.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:40 smithi077 ceph-mon[37153]: pgmap v2735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:00:40.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:40 smithi077 ceph-mon[37153]: pgmap v2736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:00:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:40 smithi047 ceph-mon[36835]: pgmap v2735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:00:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:40 smithi047 ceph-mon[36835]: pgmap v2736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:00:41.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:00:41.340+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:00:41.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:00:41.340+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:00:41.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:00:41.340+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:00:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:00:41.432+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:00:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:00:41.432+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:00:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:00:41.432+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:00:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:42 smithi077 ceph-mon[37153]: pgmap v2737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:00:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:42 smithi047 ceph-mon[36835]: pgmap v2737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:00:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:42 smithi029 ceph-mon[32610]: pgmap v2737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:00:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4264.451717 secs 2024-07-27T08:00:43.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4264.451717 secs 2024-07-27T08:00:43.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4264.451717 secs 2024-07-27T08:00:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:44 smithi077 ceph-mon[37153]: pgmap v2738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:00:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:44 smithi047 ceph-mon[36835]: pgmap v2738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:00:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:44 smithi029 ceph-mon[32610]: pgmap v2738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:00:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:46 smithi077 ceph-mon[37153]: pgmap v2739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:00:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:46 smithi047 ceph-mon[36835]: pgmap v2739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:00:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:46 smithi029 ceph-mon[32610]: pgmap v2739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:00:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:48 smithi077 ceph-mon[37153]: pgmap v2740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:00:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4269.451842 secs 2024-07-27T08:00:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:48 smithi047 ceph-mon[36835]: pgmap v2740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:00:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4269.451842 secs 2024-07-27T08:00:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:48 smithi029 ceph-mon[32610]: pgmap v2740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:00:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4269.451842 secs 2024-07-27T08:00:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:50 smithi077 ceph-mon[37153]: pgmap v2741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:00:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:50 smithi047 ceph-mon[36835]: pgmap v2741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:00:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:50 smithi029 ceph-mon[32610]: pgmap v2741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:00:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:52 smithi077 ceph-mon[37153]: pgmap v2742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:00:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:52 smithi047 ceph-mon[36835]: pgmap v2742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:00:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:52 smithi029 ceph-mon[32610]: pgmap v2742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:00:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4274.451965 secs 2024-07-27T08:00:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4274.451965 secs 2024-07-27T08:00:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4274.451965 secs 2024-07-27T08:00:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:54 smithi077 ceph-mon[37153]: pgmap v2743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:00:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:54 smithi047 ceph-mon[36835]: pgmap v2743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:00:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:54 smithi029 ceph-mon[32610]: pgmap v2743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:00:56.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:00:56.356+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:00:56.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:00:56.356+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:00:56.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:00:56.356+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:00:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:00:56.447+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:00:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:00:56.447+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:00:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:00:56.447+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:00:56.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:56 smithi029 ceph-mon[32610]: pgmap v2744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:00:56.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:56 smithi077 ceph-mon[37153]: pgmap v2744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:00:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:56 smithi047 ceph-mon[36835]: pgmap v2744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:00:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:58 smithi077 ceph-mon[37153]: pgmap v2745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:00:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4279.452060 secs 2024-07-27T08:00:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:58 smithi047 ceph-mon[36835]: pgmap v2745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:00:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4279.452060 secs 2024-07-27T08:00:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:58 smithi029 ceph-mon[32610]: pgmap v2745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:00:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:00:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4279.452060 secs 2024-07-27T08:01:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:00 smithi077 ceph-mon[37153]: pgmap v2746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:00 smithi047 ceph-mon[36835]: pgmap v2746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:00 smithi029 ceph-mon[32610]: pgmap v2746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:02 smithi077 ceph-mon[37153]: pgmap v2747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:02 smithi047 ceph-mon[36835]: pgmap v2747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:02 smithi029 ceph-mon[32610]: pgmap v2747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4284.452191 secs 2024-07-27T08:01:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4284.452191 secs 2024-07-27T08:01:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4284.452191 secs 2024-07-27T08:01:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:04 smithi077 ceph-mon[37153]: pgmap v2748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:01:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:04 smithi047 ceph-mon[36835]: pgmap v2748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:01:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:04 smithi029 ceph-mon[32610]: pgmap v2748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:01:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:06 smithi077 ceph-mon[37153]: pgmap v2749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:01:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:06 smithi047 ceph-mon[36835]: pgmap v2749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:01:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:06 smithi029 ceph-mon[32610]: pgmap v2749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:01:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:08 smithi077 ceph-mon[37153]: pgmap v2750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4289.452338 secs 2024-07-27T08:01:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:08 smithi047 ceph-mon[36835]: pgmap v2750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4289.452338 secs 2024-07-27T08:01:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:08 smithi029 ceph-mon[32610]: pgmap v2750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4289.452338 secs 2024-07-27T08:01:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:10 smithi077 ceph-mon[37153]: pgmap v2751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:10 smithi047 ceph-mon[36835]: pgmap v2751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:10 smithi029 ceph-mon[32610]: pgmap v2751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:11.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:11.371+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:01:11.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:11.372+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:01:11.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:11.372+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:01:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:11.453+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:01:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:11.453+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:01:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:11.453+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:01:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:12 smithi077 ceph-mon[37153]: pgmap v2752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:12 smithi047 ceph-mon[36835]: pgmap v2752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:12 smithi029 ceph-mon[32610]: pgmap v2752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:13.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4294.452459 secs 2024-07-27T08:01:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4294.452459 secs 2024-07-27T08:01:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4294.452459 secs 2024-07-27T08:01:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:14 smithi077 ceph-mon[37153]: pgmap v2753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:01:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:14 smithi047 ceph-mon[36835]: pgmap v2753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:01:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:14 smithi029 ceph-mon[32610]: pgmap v2753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:01:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:15 smithi077 ceph-mon[37153]: pgmap v2754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:01:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:15 smithi047 ceph-mon[36835]: pgmap v2754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:01:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:15 smithi029 ceph-mon[32610]: pgmap v2754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:01:18.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:17 smithi077 ceph-mon[37153]: pgmap v2755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:18.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4299.452554 secs 2024-07-27T08:01:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:17 smithi047 ceph-mon[36835]: pgmap v2755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4299.452554 secs 2024-07-27T08:01:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:17 smithi029 ceph-mon[32610]: pgmap v2755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4299.452554 secs 2024-07-27T08:01:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:19 smithi077 ceph-mon[37153]: pgmap v2756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:19 smithi047 ceph-mon[36835]: pgmap v2756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:19 smithi029 ceph-mon[32610]: pgmap v2756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:21 smithi077 ceph-mon[37153]: pgmap v2757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:21 smithi047 ceph-mon[36835]: pgmap v2757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:22.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:21 smithi029 ceph-mon[32610]: pgmap v2757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4304.452677 secs 2024-07-27T08:01:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4304.452677 secs 2024-07-27T08:01:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4304.452677 secs 2024-07-27T08:01:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:23 smithi077 ceph-mon[37153]: pgmap v2758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:01:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:23 smithi047 ceph-mon[36835]: pgmap v2758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:01:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:23 smithi029 ceph-mon[32610]: pgmap v2758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:01:26.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:25 smithi029 ceph-mon[32610]: pgmap v2759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:01:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:25 smithi077 ceph-mon[37153]: pgmap v2759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:01:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:25 smithi047 ceph-mon[36835]: pgmap v2759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:01:26.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:26.386+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:01:26.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:26.386+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:01:26.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:26.386+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:01:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:26.467+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:01:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:26.467+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:01:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:26.467+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:01:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:27 smithi077 ceph-mon[37153]: pgmap v2760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4309.452819 secs 2024-07-27T08:01:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:27 smithi047 ceph-mon[36835]: pgmap v2760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4309.452819 secs 2024-07-27T08:01:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:27 smithi029 ceph-mon[32610]: pgmap v2760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4309.452819 secs 2024-07-27T08:01:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:29 smithi077 ceph-mon[37153]: pgmap v2761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:30.152 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:29 smithi029 ceph-mon[32610]: pgmap v2761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:29 smithi047 ceph-mon[36835]: pgmap v2761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:31 smithi077 ceph-mon[37153]: pgmap v2762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:31 smithi047 ceph-mon[36835]: pgmap v2762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:31 smithi029 ceph-mon[32610]: pgmap v2762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4314.452919 secs 2024-07-27T08:01:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4314.452919 secs 2024-07-27T08:01:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4314.452919 secs 2024-07-27T08:01:34.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:33 smithi077 ceph-mon[37153]: pgmap v2763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:01:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:33 smithi047 ceph-mon[36835]: pgmap v2763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:01:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:33 smithi029 ceph-mon[32610]: pgmap v2763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:01:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:35 smithi077 ceph-mon[37153]: pgmap v2764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:01:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:35 smithi047 ceph-mon[36835]: pgmap v2764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:01:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:35 smithi029 ceph-mon[32610]: pgmap v2764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:01:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:37 smithi077 ceph-mon[37153]: pgmap v2765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4319.453039 secs 2024-07-27T08:01:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:37 smithi047 ceph-mon[36835]: pgmap v2765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4319.453039 secs 2024-07-27T08:01:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:37 smithi029 ceph-mon[32610]: pgmap v2765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4319.453039 secs 2024-07-27T08:01:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:01:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:01:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:01:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:39 smithi077 ceph-mon[37153]: pgmap v2766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:01:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:01:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:39 smithi077 ceph-mon[37153]: pgmap v2767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:01:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:39 smithi077 ceph-mon[37153]: pgmap v2768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:01:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:01:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:39 smithi047 ceph-mon[36835]: pgmap v2766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:01:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:01:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:39 smithi047 ceph-mon[36835]: pgmap v2767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:01:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:39 smithi047 ceph-mon[36835]: pgmap v2768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:01:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:01:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:39 smithi029 ceph-mon[32610]: pgmap v2766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:01:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:01:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:39 smithi029 ceph-mon[32610]: pgmap v2767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:01:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:39 smithi029 ceph-mon[32610]: pgmap v2768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:01:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:01:41.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:41.401+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:01:41.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:41.401+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:01:41.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:41.401+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:41.483+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:01:41.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:41.483+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:01:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:41.483+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:01:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:42 smithi077 ceph-mon[37153]: pgmap v2769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:01:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:42 smithi047 ceph-mon[36835]: pgmap v2769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:01:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:42 smithi029 ceph-mon[32610]: pgmap v2769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:01:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:43.282+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:01:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:43.283+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:01:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:43.287+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:01:43.289 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:43.288+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:01:43.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4324.453163 secs 2024-07-27T08:01:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4324.453163 secs 2024-07-27T08:01:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4324.453163 secs 2024-07-27T08:01:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:44 smithi077 ceph-mon[37153]: pgmap v2770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:01:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:44 smithi047 ceph-mon[36835]: pgmap v2770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:01:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:44 smithi029 ceph-mon[32610]: pgmap v2770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:01:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:45 smithi077 ceph-mon[37153]: pgmap v2771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:01:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:45 smithi047 ceph-mon[36835]: pgmap v2771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:01:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:45 smithi029 ceph-mon[32610]: pgmap v2771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:01:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4329.453289 secs 2024-07-27T08:01:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:48 smithi077 ceph-mon[37153]: pgmap v2772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:01:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4329.453289 secs 2024-07-27T08:01:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:48 smithi047 ceph-mon[36835]: pgmap v2772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:01:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4329.453289 secs 2024-07-27T08:01:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:48 smithi029 ceph-mon[32610]: pgmap v2772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:01:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:49 smithi077 ceph-mon[37153]: pgmap v2773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:01:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:49 smithi047 ceph-mon[36835]: pgmap v2773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:01:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:49 smithi029 ceph-mon[32610]: pgmap v2773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:01:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:51 smithi077 ceph-mon[37153]: pgmap v2774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:51 smithi047 ceph-mon[36835]: pgmap v2774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:51 smithi029 ceph-mon[32610]: pgmap v2774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:01:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4334.453432 secs 2024-07-27T08:01:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4334.453432 secs 2024-07-27T08:01:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4334.453432 secs 2024-07-27T08:01:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:53 smithi047 ceph-mon[36835]: pgmap v2775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:53 smithi029 ceph-mon[32610]: pgmap v2775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:53 smithi077 ceph-mon[37153]: pgmap v2775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:01:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:55 smithi077 ceph-mon[37153]: pgmap v2776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:55 smithi047 ceph-mon[36835]: pgmap v2776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:55 smithi029 ceph-mon[32610]: pgmap v2776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:01:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:56.402+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:01:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:56.402+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:01:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:56.402+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:01:56.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:56.494+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:01:56.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:56.494+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:01:56.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:01:56.494+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:01:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:57 smithi077 ceph-mon[37153]: pgmap v2777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:01:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:57 smithi047 ceph-mon[36835]: pgmap v2777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:01:57.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:57 smithi029 ceph-mon[32610]: pgmap v2777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:01:58.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:58.285+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:01:58.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:58.285+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:01:58.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:58.285+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:01:58.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:01:58.285+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:01:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4339.453571 secs 2024-07-27T08:01:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4339.453571 secs 2024-07-27T08:01:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4339.453571 secs 2024-07-27T08:01:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:01:59 smithi077 ceph-mon[37153]: pgmap v2778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:01:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:01:59 smithi047 ceph-mon[36835]: pgmap v2778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:01:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:01:59 smithi029 ceph-mon[32610]: pgmap v2778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:02:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:01 smithi077 ceph-mon[37153]: pgmap v2779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:02:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:01 smithi047 ceph-mon[36835]: pgmap v2779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:02:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:01 smithi029 ceph-mon[32610]: pgmap v2779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:02:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4344.453696 secs 2024-07-27T08:02:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4344.453696 secs 2024-07-27T08:02:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4344.453696 secs 2024-07-27T08:02:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:03 smithi047 ceph-mon[36835]: pgmap v2780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:03 smithi029 ceph-mon[32610]: pgmap v2780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:03 smithi077 ceph-mon[37153]: pgmap v2780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:05 smithi077 ceph-mon[37153]: pgmap v2781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:05 smithi047 ceph-mon[36835]: pgmap v2781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:05 smithi029 ceph-mon[32610]: pgmap v2781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:07 smithi077 ceph-mon[37153]: pgmap v2782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:02:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:07 smithi047 ceph-mon[36835]: pgmap v2782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:02:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:07 smithi029 ceph-mon[32610]: pgmap v2782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:02:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4349.453854 secs 2024-07-27T08:02:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4349.453854 secs 2024-07-27T08:02:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4349.453854 secs 2024-07-27T08:02:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:09 smithi077 ceph-mon[37153]: pgmap v2783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:02:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:09 smithi047 ceph-mon[36835]: pgmap v2783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:02:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:09 smithi029 ceph-mon[32610]: pgmap v2783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:02:11.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:02:11.409+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:02:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:02:11.409+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:02:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:02:11.409+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:02:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:02:11.510+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:02:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:02:11.510+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:02:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:02:11.510+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:02:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:11 smithi077 ceph-mon[37153]: pgmap v2784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:02:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:11 smithi047 ceph-mon[36835]: pgmap v2784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:02:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:11 smithi029 ceph-mon[32610]: pgmap v2784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:02:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4354.453975 secs 2024-07-27T08:02:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4354.453975 secs 2024-07-27T08:02:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4354.453975 secs 2024-07-27T08:02:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:13 smithi047 ceph-mon[36835]: pgmap v2785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:13 smithi029 ceph-mon[32610]: pgmap v2785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:13.757 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:13 smithi077 ceph-mon[37153]: pgmap v2785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:15 smithi077 ceph-mon[37153]: pgmap v2786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:15 smithi047 ceph-mon[36835]: pgmap v2786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:15 smithi029 ceph-mon[32610]: pgmap v2786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:17 smithi077 ceph-mon[37153]: pgmap v2787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:02:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:17 smithi047 ceph-mon[36835]: pgmap v2787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:02:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:17 smithi029 ceph-mon[32610]: pgmap v2787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:02:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4359.454095 secs 2024-07-27T08:02:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4359.454095 secs 2024-07-27T08:02:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4359.454095 secs 2024-07-27T08:02:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:19 smithi077 ceph-mon[37153]: pgmap v2788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:02:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:19 smithi047 ceph-mon[36835]: pgmap v2788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:02:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:19 smithi029 ceph-mon[32610]: pgmap v2788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:02:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:21 smithi077 ceph-mon[37153]: pgmap v2789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:02:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:21 smithi047 ceph-mon[36835]: pgmap v2789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:02:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:21 smithi029 ceph-mon[32610]: pgmap v2789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:02:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4364.454231 secs 2024-07-27T08:02:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4364.454231 secs 2024-07-27T08:02:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4364.454231 secs 2024-07-27T08:02:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:23 smithi047 ceph-mon[36835]: pgmap v2790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:23 smithi029 ceph-mon[32610]: pgmap v2790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:23 smithi077 ceph-mon[37153]: pgmap v2790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:25 smithi077 ceph-mon[37153]: pgmap v2791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:25 smithi047 ceph-mon[36835]: pgmap v2791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:25 smithi029 ceph-mon[32610]: pgmap v2791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:26.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:02:26.424+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:02:26.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:02:26.424+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:02:26.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:02:26.424+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:02:26.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:02:26.518+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:02:26.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:02:26.518+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:02:26.520 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:02:26.518+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:02:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:27 smithi077 ceph-mon[37153]: pgmap v2792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:02:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:27 smithi047 ceph-mon[36835]: pgmap v2792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:02:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:27 smithi029 ceph-mon[32610]: pgmap v2792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:02:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4369.454335 secs 2024-07-27T08:02:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4369.454335 secs 2024-07-27T08:02:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4369.454335 secs 2024-07-27T08:02:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:29 smithi077 ceph-mon[37153]: pgmap v2793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:02:29.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:29 smithi047 ceph-mon[36835]: pgmap v2793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:02:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:29 smithi029 ceph-mon[32610]: pgmap v2793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:02:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:31 smithi077 ceph-mon[37153]: pgmap v2794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:02:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:31 smithi047 ceph-mon[36835]: pgmap v2794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:02:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:31 smithi029 ceph-mon[32610]: pgmap v2794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:02:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4374.454473 secs 2024-07-27T08:02:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4374.454473 secs 2024-07-27T08:02:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4374.454473 secs 2024-07-27T08:02:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:33 smithi047 ceph-mon[36835]: pgmap v2795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:33 smithi029 ceph-mon[32610]: pgmap v2795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:33 smithi077 ceph-mon[37153]: pgmap v2795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:35 smithi077 ceph-mon[37153]: pgmap v2796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:35 smithi047 ceph-mon[36835]: pgmap v2796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:35 smithi029 ceph-mon[32610]: pgmap v2796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:37 smithi077 ceph-mon[37153]: pgmap v2797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:02:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:37 smithi047 ceph-mon[36835]: pgmap v2797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:02:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:37 smithi029 ceph-mon[32610]: pgmap v2797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:02:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4379.454626 secs 2024-07-27T08:02:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4379.454626 secs 2024-07-27T08:02:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4379.454626 secs 2024-07-27T08:02:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:39 smithi077 ceph-mon[37153]: pgmap v2798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:02:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:02:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:39 smithi047 ceph-mon[36835]: pgmap v2798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:02:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:02:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:39 smithi029 ceph-mon[32610]: pgmap v2798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:02:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:02:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:02:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:02:40.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:02:40.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:02:40.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:40 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:02:40.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:40 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:02:40.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:02:40.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:02:40.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:40 smithi047 ceph-mon[36835]: pgmap v2799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:02:40.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:40 smithi047 ceph-mon[36835]: pgmap v2800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:02:40.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:02:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:02:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:02:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:02:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:02:40.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:40 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:02:40.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:40 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:02:40.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:02:40.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:02:40.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:40 smithi029 ceph-mon[32610]: pgmap v2799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:02:40.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:40 smithi029 ceph-mon[32610]: pgmap v2800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:02:40.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:02:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:02:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:02:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:02:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:02:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:40 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:02:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:40 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:02:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:02:40.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:02:40.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:40 smithi077 ceph-mon[37153]: pgmap v2799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:02:40.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:40 smithi077 ceph-mon[37153]: pgmap v2800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:02:40.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:02:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:02:41.439+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:02:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:02:41.439+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:02:41.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:02:41.439+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:02:41.523 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:02:41.522+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:02:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:02:41.522+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:02:41.524 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:02:41.522+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:02:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4384.454749 secs 2024-07-27T08:02:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:42 smithi077 ceph-mon[37153]: pgmap v2801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:02:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4384.454749 secs 2024-07-27T08:02:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:42 smithi047 ceph-mon[36835]: pgmap v2801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:02:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4384.454749 secs 2024-07-27T08:02:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:42 smithi029 ceph-mon[32610]: pgmap v2801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:02:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:44 smithi077 ceph-mon[37153]: pgmap v2802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:02:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:44 smithi047 ceph-mon[36835]: pgmap v2802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:02:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:44 smithi029 ceph-mon[32610]: pgmap v2802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:02:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:46 smithi077 ceph-mon[37153]: pgmap v2803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:02:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:46 smithi047 ceph-mon[36835]: pgmap v2803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:02:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:46 smithi029 ceph-mon[32610]: pgmap v2803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:02:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4389.454800 secs 2024-07-27T08:02:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4389.454800 secs 2024-07-27T08:02:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4389.454800 secs 2024-07-27T08:02:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:48 smithi047 ceph-mon[36835]: pgmap v2804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:02:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:48 smithi029 ceph-mon[32610]: pgmap v2804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:02:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:48 smithi077 ceph-mon[37153]: pgmap v2804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:02:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:50 smithi047 ceph-mon[36835]: pgmap v2805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:02:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:50 smithi029 ceph-mon[32610]: pgmap v2805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:02:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:50 smithi077 ceph-mon[37153]: pgmap v2805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:02:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4394.454932 secs 2024-07-27T08:02:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:52 smithi047 ceph-mon[36835]: pgmap v2806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:02:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4394.454932 secs 2024-07-27T08:02:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:52 smithi029 ceph-mon[32610]: pgmap v2806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:02:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4394.454932 secs 2024-07-27T08:02:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:52 smithi077 ceph-mon[37153]: pgmap v2806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:02:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:54 smithi047 ceph-mon[36835]: pgmap v2807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:54 smithi029 ceph-mon[32610]: pgmap v2807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:54 smithi077 ceph-mon[37153]: pgmap v2807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:02:56.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:02:56.455+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:02:56.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:02:56.455+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:02:56.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:02:56.455+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:02:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:02:56.538+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:02:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:02:56.538+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:02:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:02:56.538+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:02:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:56 smithi047 ceph-mon[36835]: pgmap v2808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:56 smithi029 ceph-mon[32610]: pgmap v2808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:56 smithi077 ceph-mon[37153]: pgmap v2808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:02:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4399.455053 secs 2024-07-27T08:02:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4399.455053 secs 2024-07-27T08:02:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4399.455053 secs 2024-07-27T08:02:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:02:58 smithi047 ceph-mon[36835]: pgmap v2809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:02:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:02:58 smithi029 ceph-mon[32610]: pgmap v2809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:02:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:02:58 smithi077 ceph-mon[37153]: pgmap v2809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:00 smithi047 ceph-mon[36835]: pgmap v2810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:03:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:00 smithi029 ceph-mon[32610]: pgmap v2810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:03:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:00 smithi077 ceph-mon[37153]: pgmap v2810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:03:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4404.455189 secs 2024-07-27T08:03:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:02 smithi047 ceph-mon[36835]: pgmap v2811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:03:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4404.455189 secs 2024-07-27T08:03:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:02 smithi029 ceph-mon[32610]: pgmap v2811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:03:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4404.455189 secs 2024-07-27T08:03:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:02 smithi077 ceph-mon[37153]: pgmap v2811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:03:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:04 smithi047 ceph-mon[36835]: pgmap v2812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:03:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:04 smithi029 ceph-mon[32610]: pgmap v2812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:03:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:04 smithi077 ceph-mon[37153]: pgmap v2812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:03:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:06 smithi047 ceph-mon[36835]: pgmap v2813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:06 smithi029 ceph-mon[32610]: pgmap v2813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:06 smithi077 ceph-mon[37153]: pgmap v2813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4409.455311 secs 2024-07-27T08:03:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4409.455311 secs 2024-07-27T08:03:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4409.455311 secs 2024-07-27T08:03:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:08 smithi047 ceph-mon[36835]: pgmap v2814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:08 smithi029 ceph-mon[32610]: pgmap v2814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:08 smithi077 ceph-mon[37153]: pgmap v2814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:10 smithi047 ceph-mon[36835]: pgmap v2815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:10 smithi029 ceph-mon[32610]: pgmap v2815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:10 smithi077 ceph-mon[37153]: pgmap v2815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:11.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:03:11.471+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:03:11.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:03:11.471+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:03:11.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:03:11.471+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:03:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:11.550+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:03:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:11.550+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:03:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:11.550+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:03:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4414.455432 secs 2024-07-27T08:03:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:12 smithi047 ceph-mon[36835]: pgmap v2816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:03:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4414.455432 secs 2024-07-27T08:03:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:12 smithi029 ceph-mon[32610]: pgmap v2816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:03:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4414.455432 secs 2024-07-27T08:03:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:12 smithi077 ceph-mon[37153]: pgmap v2816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:03:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:14 smithi047 ceph-mon[36835]: pgmap v2817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:03:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:14 smithi029 ceph-mon[32610]: pgmap v2817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:03:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:14 smithi077 ceph-mon[37153]: pgmap v2817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:03:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:16 smithi047 ceph-mon[36835]: pgmap v2818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:16 smithi029 ceph-mon[32610]: pgmap v2818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:16 smithi077 ceph-mon[37153]: pgmap v2818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:18.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4419.455570 secs 2024-07-27T08:03:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4419.455570 secs 2024-07-27T08:03:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4419.455570 secs 2024-07-27T08:03:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:18 smithi047 ceph-mon[36835]: pgmap v2819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:18 smithi029 ceph-mon[32610]: pgmap v2819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:18 smithi077 ceph-mon[37153]: pgmap v2819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:20 smithi047 ceph-mon[36835]: pgmap v2820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:03:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:20 smithi029 ceph-mon[32610]: pgmap v2820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:03:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:20 smithi077 ceph-mon[37153]: pgmap v2820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:03:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4424.455697 secs 2024-07-27T08:03:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:22 smithi047 ceph-mon[36835]: pgmap v2821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:03:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4424.455697 secs 2024-07-27T08:03:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:22 smithi029 ceph-mon[32610]: pgmap v2821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:03:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4424.455697 secs 2024-07-27T08:03:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:22 smithi077 ceph-mon[37153]: pgmap v2821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:03:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:24 smithi047 ceph-mon[36835]: pgmap v2822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:03:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:24 smithi029 ceph-mon[32610]: pgmap v2822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:03:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:24 smithi077 ceph-mon[37153]: pgmap v2822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:03:26.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:03:26.473+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:03:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:03:26.473+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:03:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:03:26.473+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:03:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:26.555+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:03:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:26.555+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:03:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:26.555+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:03:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:26 smithi047 ceph-mon[36835]: pgmap v2823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:26 smithi029 ceph-mon[32610]: pgmap v2823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:26 smithi077 ceph-mon[37153]: pgmap v2823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4429.455763 secs 2024-07-27T08:03:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4429.455763 secs 2024-07-27T08:03:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4429.455763 secs 2024-07-27T08:03:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:28 smithi047 ceph-mon[36835]: pgmap v2824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:28 smithi029 ceph-mon[32610]: pgmap v2824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:28 smithi077 ceph-mon[37153]: pgmap v2824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:30 smithi029 ceph-mon[32610]: pgmap v2825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:30 smithi077 ceph-mon[37153]: pgmap v2825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:30 smithi047 ceph-mon[36835]: pgmap v2825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4434.455868 secs 2024-07-27T08:03:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:32 smithi029 ceph-mon[32610]: pgmap v2826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:03:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4434.455868 secs 2024-07-27T08:03:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:32 smithi077 ceph-mon[37153]: pgmap v2826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:03:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4434.455868 secs 2024-07-27T08:03:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:32 smithi047 ceph-mon[36835]: pgmap v2826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:03:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:34 smithi029 ceph-mon[32610]: pgmap v2827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:03:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:34 smithi077 ceph-mon[37153]: pgmap v2827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:03:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:34 smithi047 ceph-mon[36835]: pgmap v2827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:03:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:36 smithi029 ceph-mon[32610]: pgmap v2828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:36 smithi077 ceph-mon[37153]: pgmap v2828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:36 smithi047 ceph-mon[36835]: pgmap v2828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4439.456024 secs 2024-07-27T08:03:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4439.456024 secs 2024-07-27T08:03:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4439.456024 secs 2024-07-27T08:03:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:38 smithi077 ceph-mon[37153]: pgmap v2829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:38 smithi047 ceph-mon[36835]: pgmap v2829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:38 smithi029 ceph-mon[32610]: pgmap v2829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:03:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:03:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:03:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:03:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:40 smithi077 ceph-mon[37153]: pgmap v2830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:03:41.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:03:41.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:03:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:40 smithi047 ceph-mon[36835]: pgmap v2830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:03:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:03:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:03:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:40 smithi029 ceph-mon[32610]: pgmap v2830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:03:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:03:41.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:03:41.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:03:41.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:03:41.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:03:41.488+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:03:41.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:03:41.488+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:03:41.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:03:41.488+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:03:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:41.571+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:03:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:41.571+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:03:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:41.571+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:03:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:41 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:03:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:41 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:03:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:41 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:03:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:41 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:03:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:41 smithi077 ceph-mon[37153]: pgmap v2831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:03:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:41 smithi077 ceph-mon[37153]: pgmap v2832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:03:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:41 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:03:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:41 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:03:42.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:41 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:03:42.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:41 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:03:42.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:41 smithi047 ceph-mon[36835]: pgmap v2831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:03:42.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:41 smithi047 ceph-mon[36835]: pgmap v2832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:03:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:41 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:03:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:41 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:03:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:41 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:03:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:41 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:03:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:41 smithi029 ceph-mon[32610]: pgmap v2831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:03:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:41 smithi029 ceph-mon[32610]: pgmap v2832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:03:43.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4444.456145 secs 2024-07-27T08:03:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4444.456145 secs 2024-07-27T08:03:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4444.456145 secs 2024-07-27T08:03:44.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:43 smithi047 ceph-mon[36835]: pgmap v2833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:03:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:43 smithi077 ceph-mon[37153]: pgmap v2833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:03:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:43 smithi029 ceph-mon[32610]: pgmap v2833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:03:46.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:45 smithi077 ceph-mon[37153]: pgmap v2834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:03:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:45 smithi047 ceph-mon[36835]: pgmap v2834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:03:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:45 smithi029 ceph-mon[32610]: pgmap v2834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:03:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:47 smithi077 ceph-mon[37153]: pgmap v2835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:03:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4449.456225 secs 2024-07-27T08:03:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:47 smithi047 ceph-mon[36835]: pgmap v2835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:03:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4449.456225 secs 2024-07-27T08:03:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:47 smithi029 ceph-mon[32610]: pgmap v2835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:03:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4449.456225 secs 2024-07-27T08:03:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:49 smithi029 ceph-mon[32610]: pgmap v2836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:03:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:49 smithi077 ceph-mon[37153]: pgmap v2836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:03:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:49 smithi047 ceph-mon[36835]: pgmap v2836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:03:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:51 smithi077 ceph-mon[37153]: pgmap v2837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:03:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:51 smithi047 ceph-mon[36835]: pgmap v2837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:03:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:51 smithi029 ceph-mon[32610]: pgmap v2837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:03:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4454.456342 secs 2024-07-27T08:03:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4454.456342 secs 2024-07-27T08:03:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4454.456342 secs 2024-07-27T08:03:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:54 smithi077 ceph-mon[37153]: pgmap v2838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:03:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:54 smithi047 ceph-mon[36835]: pgmap v2838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:03:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:54 smithi029 ceph-mon[32610]: pgmap v2838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:03:56.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:56 smithi029 ceph-mon[32610]: pgmap v2839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:03:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:56 smithi077 ceph-mon[37153]: pgmap v2839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:03:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:56 smithi047 ceph-mon[36835]: pgmap v2839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:03:56.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:03:56.504+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:03:56.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:03:56.504+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:03:56.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:03:56.504+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:03:56.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:56.581+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:03:56.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:56.581+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:03:56.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:56.581+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:03:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:58.304+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:03:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:58.304+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:03:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:58.306+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:03:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:03:58.306+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:03:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:58 smithi077 ceph-mon[37153]: pgmap v2840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:03:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:03:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4459.456462 secs 2024-07-27T08:03:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:58 smithi047 ceph-mon[36835]: pgmap v2840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:03:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:03:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4459.456462 secs 2024-07-27T08:03:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:58 smithi029 ceph-mon[32610]: pgmap v2840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:03:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:03:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4459.456462 secs 2024-07-27T08:04:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:00 smithi077 ceph-mon[37153]: pgmap v2841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:00 smithi047 ceph-mon[36835]: pgmap v2841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:00 smithi029 ceph-mon[32610]: pgmap v2841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:02 smithi077 ceph-mon[37153]: pgmap v2842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:02 smithi047 ceph-mon[36835]: pgmap v2842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:02 smithi029 ceph-mon[32610]: pgmap v2842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4464.456602 secs 2024-07-27T08:04:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4464.456602 secs 2024-07-27T08:04:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4464.456602 secs 2024-07-27T08:04:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:04 smithi077 ceph-mon[37153]: pgmap v2843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:04:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:04 smithi047 ceph-mon[36835]: pgmap v2843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:04:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:04 smithi029 ceph-mon[32610]: pgmap v2843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:04:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:06 smithi077 ceph-mon[37153]: pgmap v2844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:04:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:06 smithi047 ceph-mon[36835]: pgmap v2844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:04:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:06 smithi029 ceph-mon[32610]: pgmap v2844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:04:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:08 smithi077 ceph-mon[37153]: pgmap v2845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:04:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4469.456667 secs 2024-07-27T08:04:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:08 smithi047 ceph-mon[36835]: pgmap v2845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:04:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4469.456667 secs 2024-07-27T08:04:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:08 smithi029 ceph-mon[32610]: pgmap v2845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:04:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4469.456667 secs 2024-07-27T08:04:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:10 smithi077 ceph-mon[37153]: pgmap v2846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:10 smithi047 ceph-mon[36835]: pgmap v2846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:10 smithi029 ceph-mon[32610]: pgmap v2846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:11.520+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:04:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:11.520+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:04:11.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:11.520+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:04:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:11.584+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:04:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:11.584+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:04:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:11.584+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:04:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:12 smithi077 ceph-mon[37153]: pgmap v2847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:12 smithi047 ceph-mon[36835]: pgmap v2847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:12 smithi029 ceph-mon[32610]: pgmap v2847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:13.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:13.306+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:04:13.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:13.306+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:04:13.308 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4474.456799 secs 2024-07-27T08:04:13.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:13.307+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:04:13.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:13.307+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:04:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4474.456799 secs 2024-07-27T08:04:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4474.456799 secs 2024-07-27T08:04:14.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:14 smithi047 ceph-mon[36835]: pgmap v2848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:04:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:14 smithi077 ceph-mon[37153]: pgmap v2848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:04:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:14 smithi029 ceph-mon[32610]: pgmap v2848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:04:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:16 smithi077 ceph-mon[37153]: pgmap v2849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:04:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:16 smithi047 ceph-mon[36835]: pgmap v2849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:04:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:16 smithi029 ceph-mon[32610]: pgmap v2849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:04:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:18 smithi077 ceph-mon[37153]: pgmap v2850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:04:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4479.456935 secs 2024-07-27T08:04:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:18 smithi047 ceph-mon[36835]: pgmap v2850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:04:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4479.456935 secs 2024-07-27T08:04:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:18 smithi029 ceph-mon[32610]: pgmap v2850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:04:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4479.456935 secs 2024-07-27T08:04:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:20 smithi077 ceph-mon[37153]: pgmap v2851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:20 smithi047 ceph-mon[36835]: pgmap v2851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:20 smithi029 ceph-mon[32610]: pgmap v2851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:22 smithi077 ceph-mon[37153]: pgmap v2852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:22 smithi047 ceph-mon[36835]: pgmap v2852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:22 smithi029 ceph-mon[32610]: pgmap v2852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4484.457031 secs 2024-07-27T08:04:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4484.457031 secs 2024-07-27T08:04:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4484.457031 secs 2024-07-27T08:04:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:24 smithi077 ceph-mon[37153]: pgmap v2853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:04:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:24 smithi047 ceph-mon[36835]: pgmap v2853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:04:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:24 smithi029 ceph-mon[32610]: pgmap v2853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:04:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:26 smithi077 ceph-mon[37153]: pgmap v2854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:04:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:26 smithi047 ceph-mon[36835]: pgmap v2854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:04:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:26 smithi029 ceph-mon[32610]: pgmap v2854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:04:26.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:26.534+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:04:26.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:26.534+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:04:26.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:26.534+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:04:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:26.598+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:04:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:26.598+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:04:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:26.598+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:04:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:28 smithi077 ceph-mon[37153]: pgmap v2855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:04:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4489.457146 secs 2024-07-27T08:04:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:28 smithi047 ceph-mon[36835]: pgmap v2855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:04:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4489.457146 secs 2024-07-27T08:04:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:28 smithi029 ceph-mon[32610]: pgmap v2855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:04:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4489.457146 secs 2024-07-27T08:04:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:30 smithi077 ceph-mon[37153]: pgmap v2856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:30 smithi047 ceph-mon[36835]: pgmap v2856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:30 smithi029 ceph-mon[32610]: pgmap v2856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:32 smithi077 ceph-mon[37153]: pgmap v2857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:32 smithi047 ceph-mon[36835]: pgmap v2857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:32 smithi029 ceph-mon[32610]: pgmap v2857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4494.457266 secs 2024-07-27T08:04:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4494.457266 secs 2024-07-27T08:04:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4494.457266 secs 2024-07-27T08:04:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:34 smithi077 ceph-mon[37153]: pgmap v2858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:04:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:34 smithi047 ceph-mon[36835]: pgmap v2858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:04:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:34 smithi029 ceph-mon[32610]: pgmap v2858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:04:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:36 smithi077 ceph-mon[37153]: pgmap v2859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:04:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:36 smithi047 ceph-mon[36835]: pgmap v2859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:04:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:36 smithi029 ceph-mon[32610]: pgmap v2859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:04:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:38 smithi077 ceph-mon[37153]: pgmap v2860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:04:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4499.457384 secs 2024-07-27T08:04:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:38 smithi047 ceph-mon[36835]: pgmap v2860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:04:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4499.457384 secs 2024-07-27T08:04:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:38 smithi029 ceph-mon[32610]: pgmap v2860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:04:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4499.457384 secs 2024-07-27T08:04:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:40 smithi077 ceph-mon[37153]: pgmap v2861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:40 smithi047 ceph-mon[36835]: pgmap v2861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:40 smithi029 ceph-mon[32610]: pgmap v2861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:04:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:04:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:04:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:04:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:04:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:04:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:04:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:04:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:04:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:04:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:04:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:04:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:04:41.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:41.541+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:04:41.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:41.541+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:04:41.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:41.541+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:04:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:41.599+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:04:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:41.599+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:04:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:41.599+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:04:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:42 smithi077 ceph-mon[37153]: pgmap v2862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:42 smithi077 ceph-mon[37153]: pgmap v2863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:04:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:42 smithi077 ceph-mon[37153]: pgmap v2864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:04:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:42 smithi047 ceph-mon[36835]: pgmap v2862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:42.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:42 smithi047 ceph-mon[36835]: pgmap v2863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:04:42.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:42 smithi047 ceph-mon[36835]: pgmap v2864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:04:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:42 smithi029 ceph-mon[32610]: pgmap v2862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:42 smithi029 ceph-mon[32610]: pgmap v2863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:04:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:42 smithi029 ceph-mon[32610]: pgmap v2864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:04:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:42.636+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-27T08:04:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:42.636+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T08:04:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:42.636+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4730 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f733c02b320 tx=0x7f733c07a000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T08:04:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:42.636+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4730 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f733c02b320 tx=0x7f733c07a000 comp rx=0 tx=0).stop 2024-07-27T08:04:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:42.636+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4730 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T08:04:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:42.636+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T08:04:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:42.636+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4730 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T08:04:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:42.636+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T08:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:42.636+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T08:04:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:42.636+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f733c00c750 tx=0x7f733c03f000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T08:04:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:43.310+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:04:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:43.311+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:04:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:43.311+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:04:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:43.312+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:04:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4504.457503 secs 2024-07-27T08:04:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4504.457503 secs 2024-07-27T08:04:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4504.457503 secs 2024-07-27T08:04:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:44 smithi047 ceph-mon[36835]: pgmap v2865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:04:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:44 smithi029 ceph-mon[32610]: pgmap v2865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:04:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:44 smithi077 ceph-mon[37153]: pgmap v2865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:04:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:46 smithi047 ceph-mon[36835]: pgmap v2866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:04:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:46 smithi029 ceph-mon[32610]: pgmap v2866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:04:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:46 smithi077 ceph-mon[37153]: pgmap v2866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:04:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4509.457567 secs 2024-07-27T08:04:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:48 smithi047 ceph-mon[36835]: pgmap v2867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:04:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4509.457567 secs 2024-07-27T08:04:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:48 smithi029 ceph-mon[32610]: pgmap v2867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:04:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4509.457567 secs 2024-07-27T08:04:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:48 smithi077 ceph-mon[37153]: pgmap v2867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:04:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:50 smithi047 ceph-mon[36835]: pgmap v2868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:04:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:50 smithi029 ceph-mon[32610]: pgmap v2868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:04:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:50 smithi077 ceph-mon[37153]: pgmap v2868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:04:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:52 smithi029 ceph-mon[32610]: pgmap v2869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:04:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:52 smithi047 ceph-mon[36835]: pgmap v2869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:04:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:52 smithi077 ceph-mon[37153]: pgmap v2869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:04:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4514.457692 secs 2024-07-27T08:04:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4514.457692 secs 2024-07-27T08:04:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4514.457692 secs 2024-07-27T08:04:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:54 smithi047 ceph-mon[36835]: pgmap v2870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:04:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:54 smithi029 ceph-mon[32610]: pgmap v2870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:04:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:54 smithi077 ceph-mon[37153]: pgmap v2870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:04:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:56 smithi047 ceph-mon[36835]: pgmap v2871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:56 smithi029 ceph-mon[32610]: pgmap v2871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:56.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:56.550+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:04:56.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:56.550+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:04:56.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:56.550+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:04:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:56 smithi077 ceph-mon[37153]: pgmap v2871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:04:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:56.615+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:04:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:56.615+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:04:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:04:56.615+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:04:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:57.651+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T08:04:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:57.651+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T08:04:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:57.651+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b4730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7eff400049e0 tx=0x7eff40038040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T08:04:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:57.651+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b4730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7eff400049e0 tx=0x7eff40038040 comp rx=0 tx=0).stop 2024-07-27T08:04:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:57.651+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b4730 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T08:04:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:57.651+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b4730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T08:04:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:57.651+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b4730 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T08:04:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:57.651+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T08:04:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:57.651+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T08:04:57.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:57.652+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7eff40009780 tx=0x7eff4004f040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T08:04:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:58.313+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:04:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:58.313+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:04:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:58.313+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:04:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:04:58.313+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:04:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4519.457840 secs 2024-07-27T08:04:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:04:58 smithi047 ceph-mon[36835]: pgmap v2872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:04:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4519.457840 secs 2024-07-27T08:04:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:04:58 smithi029 ceph-mon[32610]: pgmap v2872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:04:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4519.457840 secs 2024-07-27T08:04:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:04:58 smithi077 ceph-mon[37153]: pgmap v2872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:00 smithi047 ceph-mon[36835]: pgmap v2873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:00 smithi029 ceph-mon[32610]: pgmap v2873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:00 smithi077 ceph-mon[37153]: pgmap v2873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:02 smithi047 ceph-mon[36835]: pgmap v2874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:02 smithi029 ceph-mon[32610]: pgmap v2874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:02 smithi077 ceph-mon[37153]: pgmap v2874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4524.457960 secs 2024-07-27T08:05:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4524.457960 secs 2024-07-27T08:05:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4524.457960 secs 2024-07-27T08:05:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:04 smithi047 ceph-mon[36835]: pgmap v2875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:05:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:04 smithi029 ceph-mon[32610]: pgmap v2875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:05:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:04 smithi077 ceph-mon[37153]: pgmap v2875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:05:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:06 smithi047 ceph-mon[36835]: pgmap v2876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:06 smithi029 ceph-mon[32610]: pgmap v2876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:06 smithi077 ceph-mon[37153]: pgmap v2876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4529.458079 secs 2024-07-27T08:05:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:08 smithi047 ceph-mon[36835]: pgmap v2877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4529.458079 secs 2024-07-27T08:05:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:08 smithi029 ceph-mon[32610]: pgmap v2877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4529.458079 secs 2024-07-27T08:05:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:08 smithi077 ceph-mon[37153]: pgmap v2877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:10 smithi047 ceph-mon[36835]: pgmap v2878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:10 smithi029 ceph-mon[32610]: pgmap v2878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:10 smithi077 ceph-mon[37153]: pgmap v2878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:05:11.565+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:05:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:05:11.565+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:05:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:05:11.565+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:05:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:05:11.619+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:05:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:05:11.619+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:05:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:05:11.619+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:05:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:12 smithi047 ceph-mon[36835]: pgmap v2879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:12 smithi029 ceph-mon[32610]: pgmap v2879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:12 smithi077 ceph-mon[37153]: pgmap v2879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4534.458179 secs 2024-07-27T08:05:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4534.458179 secs 2024-07-27T08:05:13.507 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4534.458179 secs 2024-07-27T08:05:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:14 smithi047 ceph-mon[36835]: pgmap v2880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:05:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:14 smithi029 ceph-mon[32610]: pgmap v2880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:05:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:14 smithi077 ceph-mon[37153]: pgmap v2880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:05:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:16 smithi047 ceph-mon[36835]: pgmap v2881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:16 smithi029 ceph-mon[32610]: pgmap v2881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:16 smithi077 ceph-mon[37153]: pgmap v2881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4539.458303 secs 2024-07-27T08:05:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:18 smithi047 ceph-mon[36835]: pgmap v2882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4539.458303 secs 2024-07-27T08:05:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:18 smithi029 ceph-mon[32610]: pgmap v2882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4539.458303 secs 2024-07-27T08:05:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:18 smithi077 ceph-mon[37153]: pgmap v2882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:20 smithi047 ceph-mon[36835]: pgmap v2883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:20 smithi029 ceph-mon[32610]: pgmap v2883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:20 smithi077 ceph-mon[37153]: pgmap v2883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:22 smithi047 ceph-mon[36835]: pgmap v2884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:22 smithi029 ceph-mon[32610]: pgmap v2884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:22 smithi077 ceph-mon[37153]: pgmap v2884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4544.458429 secs 2024-07-27T08:05:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4544.458429 secs 2024-07-27T08:05:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4544.458429 secs 2024-07-27T08:05:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:24 smithi029 ceph-mon[32610]: pgmap v2885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:05:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:24 smithi077 ceph-mon[37153]: pgmap v2885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:05:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:24 smithi047 ceph-mon[36835]: pgmap v2885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:05:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:26 smithi029 ceph-mon[32610]: pgmap v2886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:26.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:05:26.566+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:05:26.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:05:26.566+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:05:26.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:05:26.566+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:05:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:26 smithi077 ceph-mon[37153]: pgmap v2886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:05:26.632+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:05:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:05:26.632+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:05:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:05:26.632+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:05:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:26 smithi047 ceph-mon[36835]: pgmap v2886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4549.458495 secs 2024-07-27T08:05:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:28 smithi029 ceph-mon[32610]: pgmap v2887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4549.458495 secs 2024-07-27T08:05:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:28 smithi077 ceph-mon[37153]: pgmap v2887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4549.458495 secs 2024-07-27T08:05:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:28 smithi047 ceph-mon[36835]: pgmap v2887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:30 smithi029 ceph-mon[32610]: pgmap v2888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:30 smithi077 ceph-mon[37153]: pgmap v2888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:30 smithi047 ceph-mon[36835]: pgmap v2888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:32 smithi029 ceph-mon[32610]: pgmap v2889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:32 smithi077 ceph-mon[37153]: pgmap v2889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:32 smithi047 ceph-mon[36835]: pgmap v2889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4554.458620 secs 2024-07-27T08:05:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4554.458620 secs 2024-07-27T08:05:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4554.458620 secs 2024-07-27T08:05:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:34 smithi077 ceph-mon[37153]: pgmap v2890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:05:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:34 smithi047 ceph-mon[36835]: pgmap v2890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:05:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:34 smithi029 ceph-mon[32610]: pgmap v2890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:05:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:36 smithi077 ceph-mon[37153]: pgmap v2891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:36 smithi047 ceph-mon[36835]: pgmap v2891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:36 smithi029 ceph-mon[32610]: pgmap v2891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4559.458719 secs 2024-07-27T08:05:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:38 smithi077 ceph-mon[37153]: pgmap v2892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4559.458719 secs 2024-07-27T08:05:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:38 smithi047 ceph-mon[36835]: pgmap v2892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4559.458719 secs 2024-07-27T08:05:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:38 smithi029 ceph-mon[32610]: pgmap v2892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:05:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:40 smithi077 ceph-mon[37153]: pgmap v2893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:05:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:40 smithi047 ceph-mon[36835]: pgmap v2893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:05:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:40 smithi029 ceph-mon[32610]: pgmap v2893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:05:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:05:41.581+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:05:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:05:41.581+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:05:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:05:41.581+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:05:41.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:05:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:05:41.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:05:41.647+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:05:41.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:05:41.647+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:05:41.649 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:05:41.647+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:05:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:05:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:42 smithi077 ceph-mon[37153]: pgmap v2894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:05:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:05:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:05:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:05:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:42 smithi047 ceph-mon[36835]: pgmap v2894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:05:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:05:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:05:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:05:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:42 smithi029 ceph-mon[32610]: pgmap v2894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:05:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:05:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:05:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:05:43.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:43 smithi077 ceph-mon[37153]: pgmap v2895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:05:43.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:43 smithi077 ceph-mon[37153]: pgmap v2896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:05:43.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4564.458812 secs 2024-07-27T08:05:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:43 smithi047 ceph-mon[36835]: pgmap v2895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:05:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:43 smithi047 ceph-mon[36835]: pgmap v2896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:05:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4564.458812 secs 2024-07-27T08:05:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:43 smithi029 ceph-mon[32610]: pgmap v2895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:05:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:43 smithi029 ceph-mon[32610]: pgmap v2896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:05:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4564.458812 secs 2024-07-27T08:05:45.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:45 smithi047 ceph-mon[36835]: pgmap v2897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:45 smithi077 ceph-mon[37153]: pgmap v2897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:45 smithi029 ceph-mon[32610]: pgmap v2897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:47 smithi077 ceph-mon[37153]: pgmap v2898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:05:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:47 smithi047 ceph-mon[36835]: pgmap v2898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:05:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:47 smithi029 ceph-mon[32610]: pgmap v2898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:05:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4569.458941 secs 2024-07-27T08:05:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4569.458941 secs 2024-07-27T08:05:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4569.458941 secs 2024-07-27T08:05:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:49 smithi077 ceph-mon[37153]: pgmap v2899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:05:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:49 smithi047 ceph-mon[36835]: pgmap v2899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:05:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:49 smithi029 ceph-mon[32610]: pgmap v2899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:05:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:51 smithi077 ceph-mon[37153]: pgmap v2900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:05:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:51 smithi047 ceph-mon[36835]: pgmap v2900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:05:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:51 smithi029 ceph-mon[32610]: pgmap v2900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:05:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:53 smithi077 ceph-mon[37153]: pgmap v2901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:05:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4574.459011 secs 2024-07-27T08:05:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:53 smithi047 ceph-mon[36835]: pgmap v2901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:05:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4574.459011 secs 2024-07-27T08:05:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:53 smithi029 ceph-mon[32610]: pgmap v2901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:05:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4574.459011 secs 2024-07-27T08:05:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:55 smithi077 ceph-mon[37153]: pgmap v2902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:55 smithi047 ceph-mon[36835]: pgmap v2902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:55 smithi029 ceph-mon[32610]: pgmap v2902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:05:56.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:05:56.588+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:05:56.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:05:56.588+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:05:56.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:05:56.588+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:05:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:05:56.653+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:05:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:05:56.653+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:05:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:05:56.653+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:05:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:57 smithi077 ceph-mon[37153]: pgmap v2903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:57 smithi047 ceph-mon[36835]: pgmap v2903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:57 smithi029 ceph-mon[32610]: pgmap v2903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:05:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4579.459138 secs 2024-07-27T08:05:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4579.459138 secs 2024-07-27T08:05:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4579.459138 secs 2024-07-27T08:05:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:05:59 smithi077 ceph-mon[37153]: pgmap v2904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:05:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:05:59 smithi047 ceph-mon[36835]: pgmap v2904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:05:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:05:59 smithi029 ceph-mon[32610]: pgmap v2904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:06:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:01 smithi077 ceph-mon[37153]: pgmap v2905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:06:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:01 smithi047 ceph-mon[36835]: pgmap v2905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:06:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:01 smithi029 ceph-mon[32610]: pgmap v2905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:06:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:03 smithi077 ceph-mon[37153]: pgmap v2906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4584.459241 secs 2024-07-27T08:06:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:03 smithi047 ceph-mon[36835]: pgmap v2906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4584.459241 secs 2024-07-27T08:06:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:03 smithi029 ceph-mon[32610]: pgmap v2906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4584.459241 secs 2024-07-27T08:06:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:05 smithi077 ceph-mon[37153]: pgmap v2907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:06:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:05 smithi047 ceph-mon[36835]: pgmap v2907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:06:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:05 smithi029 ceph-mon[32610]: pgmap v2907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:06:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:07 smithi077 ceph-mon[37153]: pgmap v2908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:06:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:07 smithi047 ceph-mon[36835]: pgmap v2908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:06:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:07 smithi029 ceph-mon[32610]: pgmap v2908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:06:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4589.459360 secs 2024-07-27T08:06:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4589.459360 secs 2024-07-27T08:06:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4589.459360 secs 2024-07-27T08:06:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:09 smithi029 ceph-mon[32610]: pgmap v2909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:06:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:09 smithi077 ceph-mon[37153]: pgmap v2909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:06:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:09 smithi047 ceph-mon[36835]: pgmap v2909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:06:11.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:06:11.600+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:06:11.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:06:11.600+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:06:11.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:06:11.600+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:06:11.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:11 smithi029 ceph-mon[32610]: pgmap v2910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:06:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:11 smithi077 ceph-mon[37153]: pgmap v2910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:06:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:06:11.669+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:06:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:06:11.669+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:06:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:06:11.669+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:06:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:11 smithi047 ceph-mon[36835]: pgmap v2910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:06:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:13 smithi077 ceph-mon[37153]: pgmap v2911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4594.459486 secs 2024-07-27T08:06:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:13 smithi047 ceph-mon[36835]: pgmap v2911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4594.459486 secs 2024-07-27T08:06:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:13 smithi029 ceph-mon[32610]: pgmap v2911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4594.459486 secs 2024-07-27T08:06:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:15 smithi077 ceph-mon[37153]: pgmap v2912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:06:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:15 smithi047 ceph-mon[36835]: pgmap v2912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:06:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:15 smithi029 ceph-mon[32610]: pgmap v2912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:06:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:17 smithi077 ceph-mon[37153]: pgmap v2913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:06:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:17 smithi047 ceph-mon[36835]: pgmap v2913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:06:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:17 smithi029 ceph-mon[32610]: pgmap v2913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:06:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4599.459586 secs 2024-07-27T08:06:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4599.459586 secs 2024-07-27T08:06:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4599.459586 secs 2024-07-27T08:06:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:19 smithi077 ceph-mon[37153]: pgmap v2914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:06:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:19 smithi047 ceph-mon[36835]: pgmap v2914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:06:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:19 smithi029 ceph-mon[32610]: pgmap v2914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:06:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:21 smithi077 ceph-mon[37153]: pgmap v2915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:06:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:21 smithi047 ceph-mon[36835]: pgmap v2915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:06:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:21 smithi029 ceph-mon[32610]: pgmap v2915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:06:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:23 smithi077 ceph-mon[37153]: pgmap v2916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4604.459704 secs 2024-07-27T08:06:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:23 smithi047 ceph-mon[36835]: pgmap v2916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4604.459704 secs 2024-07-27T08:06:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:23 smithi029 ceph-mon[32610]: pgmap v2916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4604.459704 secs 2024-07-27T08:06:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:25 smithi077 ceph-mon[37153]: pgmap v2917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:06:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:25 smithi047 ceph-mon[36835]: pgmap v2917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:06:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:25 smithi029 ceph-mon[32610]: pgmap v2917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:06:26.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:06:26.614+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:06:26.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:06:26.614+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:06:26.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:06:26.614+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:06:26.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:06:26.677+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:06:26.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:06:26.677+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:06:26.679 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:06:26.677+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:06:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:27 smithi077 ceph-mon[37153]: pgmap v2918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:06:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:27 smithi047 ceph-mon[36835]: pgmap v2918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:06:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:27 smithi029 ceph-mon[32610]: pgmap v2918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:06:28.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4609.459856 secs 2024-07-27T08:06:28.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:28 smithi077 ceph-mon[37153]: pgmap v2919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:06:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4609.459856 secs 2024-07-27T08:06:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:28 smithi047 ceph-mon[36835]: pgmap v2919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:06:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4609.459856 secs 2024-07-27T08:06:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:28 smithi029 ceph-mon[32610]: pgmap v2919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:06:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:30 smithi077 ceph-mon[37153]: pgmap v2920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:06:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:30 smithi047 ceph-mon[36835]: pgmap v2920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:06:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:30 smithi029 ceph-mon[32610]: pgmap v2920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:06:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:32 smithi077 ceph-mon[37153]: pgmap v2921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4614.459982 secs 2024-07-27T08:06:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:32 smithi047 ceph-mon[36835]: pgmap v2921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4614.459982 secs 2024-07-27T08:06:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:32 smithi029 ceph-mon[32610]: pgmap v2921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4614.459982 secs 2024-07-27T08:06:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:34 smithi077 ceph-mon[37153]: pgmap v2922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:06:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:34 smithi047 ceph-mon[36835]: pgmap v2922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:06:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:34 smithi029 ceph-mon[32610]: pgmap v2922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:06:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:36 smithi077 ceph-mon[37153]: pgmap v2923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:06:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:36 smithi047 ceph-mon[36835]: pgmap v2923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:06:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:36 smithi029 ceph-mon[32610]: pgmap v2923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:06:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4619.460052 secs 2024-07-27T08:06:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4619.460052 secs 2024-07-27T08:06:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4619.460052 secs 2024-07-27T08:06:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:38 smithi077 ceph-mon[37153]: pgmap v2924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:06:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:38 smithi047 ceph-mon[36835]: pgmap v2924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:06:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:38 smithi029 ceph-mon[32610]: pgmap v2924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:06:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:40 smithi047 ceph-mon[36835]: pgmap v2925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:06:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:40 smithi029 ceph-mon[32610]: pgmap v2925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:06:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:40 smithi077 ceph-mon[37153]: pgmap v2925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:06:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:06:41.629+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:06:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:06:41.629+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:06:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:06:41.629+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:06:41.689 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:06:41.688+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:06:41.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:06:41.688+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:06:41.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:06:41.688+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:06:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:06:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:06:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:06:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:42 smithi047 ceph-mon[36835]: pgmap v2926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4624.460153 secs 2024-07-27T08:06:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:42 smithi029 ceph-mon[32610]: pgmap v2926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4624.460153 secs 2024-07-27T08:06:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:42 smithi077 ceph-mon[37153]: pgmap v2926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4624.460153 secs 2024-07-27T08:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.572 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.572 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.572 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.572 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.572 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.572 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:44.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:45.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:45 smithi047 ceph-mon[36835]: pgmap v2927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:06:45.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:06:45.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:06:45.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:45 smithi047 ceph-mon[36835]: pgmap v2928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:06:45.599 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:45 smithi047 ceph-mon[36835]: pgmap v2929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:06:45.599 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:45 smithi077 ceph-mon[37153]: pgmap v2927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:06:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:06:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:06:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:45 smithi077 ceph-mon[37153]: pgmap v2928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:06:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:45 smithi077 ceph-mon[37153]: pgmap v2929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:06:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:45 smithi029 ceph-mon[32610]: pgmap v2927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:06:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:06:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:06:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:45 smithi029 ceph-mon[32610]: pgmap v2928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:06:45.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:45 smithi029 ceph-mon[32610]: pgmap v2929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:06:45.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:06:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:46 smithi047 ceph-mon[36835]: pgmap v2930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:06:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:46 smithi029 ceph-mon[32610]: pgmap v2930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:06:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:46 smithi077 ceph-mon[37153]: pgmap v2930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:06:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4629.460219 secs 2024-07-27T08:06:47.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4629.460219 secs 2024-07-27T08:06:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4629.460219 secs 2024-07-27T08:06:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:48 smithi029 ceph-mon[32610]: pgmap v2931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:06:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:48 smithi077 ceph-mon[37153]: pgmap v2931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:06:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:48 smithi047 ceph-mon[36835]: pgmap v2931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:06:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:50 smithi047 ceph-mon[36835]: pgmap v2932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:06:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:50 smithi029 ceph-mon[32610]: pgmap v2932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:06:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:50 smithi077 ceph-mon[37153]: pgmap v2932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:06:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:52 smithi047 ceph-mon[36835]: pgmap v2933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T08:06:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:52 smithi029 ceph-mon[32610]: pgmap v2933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T08:06:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:52 smithi077 ceph-mon[37153]: pgmap v2933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T08:06:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4634.460359 secs 2024-07-27T08:06:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4634.460359 secs 2024-07-27T08:06:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4634.460359 secs 2024-07-27T08:06:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:54 smithi047 ceph-mon[36835]: pgmap v2934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T08:06:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:54 smithi029 ceph-mon[32610]: pgmap v2934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T08:06:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:54 smithi077 ceph-mon[37153]: pgmap v2934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T08:06:56.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:06:56.633+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:06:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:06:56.633+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:06:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:06:56.633+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:06:56.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:56 smithi029 ceph-mon[32610]: pgmap v2935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:56 smithi047 ceph-mon[36835]: pgmap v2935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:56.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:06:56.703+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:06:56.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:06:56.703+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:06:56.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:06:56.703+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:06:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:56 smithi077 ceph-mon[37153]: pgmap v2935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:06:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4639.460493 secs 2024-07-27T08:06:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4639.460493 secs 2024-07-27T08:06:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4639.460493 secs 2024-07-27T08:06:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:06:58 smithi029 ceph-mon[32610]: pgmap v2936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:06:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:06:58 smithi077 ceph-mon[37153]: pgmap v2936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:06:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:06:58 smithi047 ceph-mon[36835]: pgmap v2936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:07:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:00 smithi047 ceph-mon[36835]: pgmap v2937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:07:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:00 smithi029 ceph-mon[32610]: pgmap v2937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:07:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:00 smithi077 ceph-mon[37153]: pgmap v2937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:07:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:02 smithi047 ceph-mon[36835]: pgmap v2938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T08:07:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:02 smithi029 ceph-mon[32610]: pgmap v2938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T08:07:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:02 smithi077 ceph-mon[37153]: pgmap v2938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T08:07:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4644.460615 secs 2024-07-27T08:07:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4644.460615 secs 2024-07-27T08:07:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4644.460615 secs 2024-07-27T08:07:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:04 smithi047 ceph-mon[36835]: pgmap v2939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:07:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:04 smithi029 ceph-mon[32610]: pgmap v2939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:07:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:04 smithi077 ceph-mon[37153]: pgmap v2939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:07:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:06 smithi047 ceph-mon[36835]: pgmap v2940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:07:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:06 smithi029 ceph-mon[32610]: pgmap v2940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:07:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:06 smithi077 ceph-mon[37153]: pgmap v2940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:07:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4649.460721 secs 2024-07-27T08:07:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4649.460721 secs 2024-07-27T08:07:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4649.460721 secs 2024-07-27T08:07:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:08 smithi029 ceph-mon[32610]: pgmap v2941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:07:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:08 smithi077 ceph-mon[37153]: pgmap v2941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:07:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:08 smithi047 ceph-mon[36835]: pgmap v2941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:07:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:10 smithi047 ceph-mon[36835]: pgmap v2942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:07:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:10 smithi029 ceph-mon[32610]: pgmap v2942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:07:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:10 smithi077 ceph-mon[37153]: pgmap v2942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:07:11.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:11.649+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:07:11.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:11.649+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:07:11.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:11.649+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:07:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:11.704+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:07:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:11.704+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:07:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:11.704+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:07:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:12 smithi029 ceph-mon[32610]: pgmap v2943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:07:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:12 smithi047 ceph-mon[36835]: pgmap v2943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:07:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:12 smithi077 ceph-mon[37153]: pgmap v2943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:07:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4654.460842 secs 2024-07-27T08:07:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4654.460842 secs 2024-07-27T08:07:13.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4654.460842 secs 2024-07-27T08:07:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:14 smithi047 ceph-mon[36835]: pgmap v2944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:07:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:14 smithi029 ceph-mon[32610]: pgmap v2944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:07:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:14 smithi077 ceph-mon[37153]: pgmap v2944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:07:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:16 smithi047 ceph-mon[36835]: pgmap v2945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:07:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:16 smithi029 ceph-mon[32610]: pgmap v2945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:07:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:16 smithi077 ceph-mon[37153]: pgmap v2945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:07:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4659.460926 secs 2024-07-27T08:07:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4659.460926 secs 2024-07-27T08:07:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4659.460926 secs 2024-07-27T08:07:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:18 smithi077 ceph-mon[37153]: pgmap v2946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:07:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:18 smithi047 ceph-mon[36835]: pgmap v2946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:07:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:18 smithi029 ceph-mon[32610]: pgmap v2946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:07:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:20 smithi047 ceph-mon[36835]: pgmap v2947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:07:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:20 smithi029 ceph-mon[32610]: pgmap v2947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:07:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:20 smithi077 ceph-mon[37153]: pgmap v2947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:07:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:22 smithi047 ceph-mon[36835]: pgmap v2948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:07:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:22 smithi029 ceph-mon[32610]: pgmap v2948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:07:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:22 smithi077 ceph-mon[37153]: pgmap v2948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:07:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4664.461026 secs 2024-07-27T08:07:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4664.461026 secs 2024-07-27T08:07:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4664.461026 secs 2024-07-27T08:07:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:24 smithi047 ceph-mon[36835]: pgmap v2949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:07:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:24 smithi029 ceph-mon[32610]: pgmap v2949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:07:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:24 smithi077 ceph-mon[37153]: pgmap v2949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:07:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:26.660+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:07:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:26.660+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:07:26.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:26.660+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:07:26.663 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:26 smithi029 ceph-mon[32610]: pgmap v2950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:07:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:26 smithi047 ceph-mon[36835]: pgmap v2950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:07:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:26.719+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:07:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:26.719+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:07:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:26.719+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:07:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:26 smithi077 ceph-mon[37153]: pgmap v2950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:07:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4669.461109 secs 2024-07-27T08:07:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4669.461109 secs 2024-07-27T08:07:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4669.461109 secs 2024-07-27T08:07:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:28 smithi077 ceph-mon[37153]: pgmap v2951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:07:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:28 smithi047 ceph-mon[36835]: pgmap v2951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:07:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:28 smithi029 ceph-mon[32610]: pgmap v2951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:07:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:30 smithi047 ceph-mon[36835]: pgmap v2952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:07:30.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:30 smithi029 ceph-mon[32610]: pgmap v2952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:07:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:30 smithi077 ceph-mon[37153]: pgmap v2952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:07:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:32 smithi047 ceph-mon[36835]: pgmap v2953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:07:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:32 smithi029 ceph-mon[32610]: pgmap v2953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:07:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:32 smithi077 ceph-mon[37153]: pgmap v2953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:07:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4674.461197 secs 2024-07-27T08:07:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4674.461197 secs 2024-07-27T08:07:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4674.461197 secs 2024-07-27T08:07:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:34 smithi047 ceph-mon[36835]: pgmap v2954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:07:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:34 smithi029 ceph-mon[32610]: pgmap v2954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:07:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:34 smithi077 ceph-mon[37153]: pgmap v2954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:07:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:36 smithi047 ceph-mon[36835]: pgmap v2955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:07:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:36 smithi029 ceph-mon[32610]: pgmap v2955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:07:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:36 smithi077 ceph-mon[37153]: pgmap v2955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:07:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4679.461327 secs 2024-07-27T08:07:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4679.461327 secs 2024-07-27T08:07:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4679.461327 secs 2024-07-27T08:07:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:38 smithi077 ceph-mon[37153]: pgmap v2956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:07:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:38 smithi047 ceph-mon[36835]: pgmap v2956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:07:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:38 smithi029 ceph-mon[32610]: pgmap v2956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:07:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:40 smithi047 ceph-mon[36835]: pgmap v2957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:07:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:40 smithi029 ceph-mon[32610]: pgmap v2957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:07:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:40 smithi077 ceph-mon[37153]: pgmap v2957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:07:41.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:41.664+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:07:41.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:41.664+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:07:41.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:41.664+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:07:41.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:41.726+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:07:41.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:41.726+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:07:41.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:41.726+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:07:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:42 smithi047 ceph-mon[36835]: pgmap v2958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:07:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:42 smithi029 ceph-mon[32610]: pgmap v2958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:07:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:42 smithi077 ceph-mon[37153]: pgmap v2958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:07:43.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:43.338+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:07:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:43.338+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:07:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:43.338+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:07:43.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:43.338+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:07:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4684.461446 secs 2024-07-27T08:07:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4684.461446 secs 2024-07-27T08:07:43.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4684.461446 secs 2024-07-27T08:07:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:44 smithi047 ceph-mon[36835]: pgmap v2959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:07:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:44 smithi029 ceph-mon[32610]: pgmap v2959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:07:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:44 smithi077 ceph-mon[37153]: pgmap v2959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:07:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:07:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:07:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:07:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:07:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:07:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:46 smithi047 ceph-mon[36835]: pgmap v2960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T08:07:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:46 smithi047 ceph-mon[36835]: pgmap v2961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:07:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:07:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:07:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:46 smithi029 ceph-mon[32610]: pgmap v2960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T08:07:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:46 smithi029 ceph-mon[32610]: pgmap v2961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:07:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:07:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:07:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:46 smithi077 ceph-mon[37153]: pgmap v2960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T08:07:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:46 smithi077 ceph-mon[37153]: pgmap v2961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:07:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:07:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4689.461568 secs 2024-07-27T08:07:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4689.461568 secs 2024-07-27T08:07:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4689.461568 secs 2024-07-27T08:07:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:48 smithi047 ceph-mon[36835]: pgmap v2962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:07:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:48 smithi029 ceph-mon[32610]: pgmap v2962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:07:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:48 smithi077 ceph-mon[37153]: pgmap v2962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:07:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:50 smithi047 ceph-mon[36835]: pgmap v2963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:07:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:50 smithi029 ceph-mon[32610]: pgmap v2963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:07:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:50 smithi077 ceph-mon[37153]: pgmap v2963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:07:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:52 smithi029 ceph-mon[32610]: pgmap v2964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:07:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4694.461693 secs 2024-07-27T08:07:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:52 smithi077 ceph-mon[37153]: pgmap v2964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:07:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4694.461693 secs 2024-07-27T08:07:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:52 smithi047 ceph-mon[36835]: pgmap v2964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:07:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4694.461693 secs 2024-07-27T08:07:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:54 smithi029 ceph-mon[32610]: pgmap v2965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:07:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:54 smithi077 ceph-mon[37153]: pgmap v2965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:07:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:54 smithi047 ceph-mon[36835]: pgmap v2965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:07:56.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:56.668+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:07:56.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:56.668+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:07:56.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:56.668+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:07:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:56 smithi029 ceph-mon[32610]: pgmap v2966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:07:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:56.742+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:07:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:56.742+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:07:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:07:56.742+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:07:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:56 smithi077 ceph-mon[37153]: pgmap v2966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:07:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:56 smithi047 ceph-mon[36835]: pgmap v2966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:07:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4699.461835 secs 2024-07-27T08:07:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4699.461835 secs 2024-07-27T08:07:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4699.461835 secs 2024-07-27T08:07:58.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:58.340+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:07:58.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:58.340+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:07:58.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:58.340+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:07:58.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:07:58.340+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:07:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:07:58 smithi029 ceph-mon[32610]: pgmap v2967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:07:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:07:58 smithi077 ceph-mon[37153]: pgmap v2967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:07:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:07:58 smithi047 ceph-mon[36835]: pgmap v2967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:00.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:00 smithi077 ceph-mon[37153]: pgmap v2968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:00 smithi047 ceph-mon[36835]: pgmap v2968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:00 smithi029 ceph-mon[32610]: pgmap v2968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:02 smithi077 ceph-mon[37153]: pgmap v2969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4704.461952 secs 2024-07-27T08:08:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:02 smithi047 ceph-mon[36835]: pgmap v2969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4704.461952 secs 2024-07-27T08:08:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:02 smithi029 ceph-mon[32610]: pgmap v2969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4704.461952 secs 2024-07-27T08:08:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:04 smithi077 ceph-mon[37153]: pgmap v2970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:04.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:04 smithi047 ceph-mon[36835]: pgmap v2970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:04 smithi029 ceph-mon[32610]: pgmap v2970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:06 smithi077 ceph-mon[37153]: pgmap v2971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:06 smithi047 ceph-mon[36835]: pgmap v2971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:06 smithi029 ceph-mon[32610]: pgmap v2971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4709.462052 secs 2024-07-27T08:08:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4709.462052 secs 2024-07-27T08:08:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4709.462052 secs 2024-07-27T08:08:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:08 smithi077 ceph-mon[37153]: pgmap v2972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:08 smithi047 ceph-mon[36835]: pgmap v2972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:08 smithi029 ceph-mon[32610]: pgmap v2972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:10 smithi077 ceph-mon[37153]: pgmap v2973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:10 smithi047 ceph-mon[36835]: pgmap v2973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:10 smithi029 ceph-mon[32610]: pgmap v2973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:11.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:08:11.673+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:08:11.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:08:11.674+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:08:11.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:08:11.674+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:08:11.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:08:11.758+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:08:11.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:08:11.758+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:08:11.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:08:11.758+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:08:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:12 smithi077 ceph-mon[37153]: pgmap v2974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4714.462174 secs 2024-07-27T08:08:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:12 smithi047 ceph-mon[36835]: pgmap v2974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4714.462174 secs 2024-07-27T08:08:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:12 smithi029 ceph-mon[32610]: pgmap v2974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4714.462174 secs 2024-07-27T08:08:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:14 smithi077 ceph-mon[37153]: pgmap v2975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:14 smithi047 ceph-mon[36835]: pgmap v2975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:14 smithi029 ceph-mon[32610]: pgmap v2975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:16.749 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:16 smithi047 ceph-mon[36835]: pgmap v2976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:16 smithi077 ceph-mon[37153]: pgmap v2976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:16 smithi029 ceph-mon[32610]: pgmap v2976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4719.462322 secs 2024-07-27T08:08:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4719.462322 secs 2024-07-27T08:08:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4719.462322 secs 2024-07-27T08:08:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:18 smithi077 ceph-mon[37153]: pgmap v2977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:18 smithi047 ceph-mon[36835]: pgmap v2977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:18 smithi029 ceph-mon[32610]: pgmap v2977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:20 smithi077 ceph-mon[37153]: pgmap v2978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:20 smithi047 ceph-mon[36835]: pgmap v2978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:20 smithi029 ceph-mon[32610]: pgmap v2978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:22 smithi077 ceph-mon[37153]: pgmap v2979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4724.462405 secs 2024-07-27T08:08:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:22 smithi047 ceph-mon[36835]: pgmap v2979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4724.462405 secs 2024-07-27T08:08:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:22 smithi029 ceph-mon[32610]: pgmap v2979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4724.462405 secs 2024-07-27T08:08:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:24 smithi077 ceph-mon[37153]: pgmap v2980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:24 smithi047 ceph-mon[36835]: pgmap v2980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:24 smithi029 ceph-mon[32610]: pgmap v2980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:26.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:08:26.677+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:08:26.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:08:26.677+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:08:26.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:08:26.677+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:08:26.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:08:26.772+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:08:26.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:08:26.772+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:08:26.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:08:26.772+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:08:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:26 smithi077 ceph-mon[37153]: pgmap v2981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:26 smithi047 ceph-mon[36835]: pgmap v2981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:26 smithi029 ceph-mon[32610]: pgmap v2981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4729.462524 secs 2024-07-27T08:08:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4729.462524 secs 2024-07-27T08:08:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4729.462524 secs 2024-07-27T08:08:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:28 smithi077 ceph-mon[37153]: pgmap v2982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:28.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:28 smithi047 ceph-mon[36835]: pgmap v2982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:28 smithi029 ceph-mon[32610]: pgmap v2982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:30 smithi077 ceph-mon[37153]: pgmap v2983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:30 smithi047 ceph-mon[36835]: pgmap v2983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:30 smithi029 ceph-mon[32610]: pgmap v2983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:32 smithi077 ceph-mon[37153]: pgmap v2984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4734.462654 secs 2024-07-27T08:08:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:32 smithi047 ceph-mon[36835]: pgmap v2984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4734.462654 secs 2024-07-27T08:08:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:32 smithi029 ceph-mon[32610]: pgmap v2984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4734.462654 secs 2024-07-27T08:08:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:34 smithi077 ceph-mon[37153]: pgmap v2985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:34 smithi047 ceph-mon[36835]: pgmap v2985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:34 smithi029 ceph-mon[32610]: pgmap v2985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:36 smithi077 ceph-mon[37153]: pgmap v2986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:36 smithi047 ceph-mon[36835]: pgmap v2986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:36 smithi029 ceph-mon[32610]: pgmap v2986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4739.462825 secs 2024-07-27T08:08:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4739.462825 secs 2024-07-27T08:08:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4739.462825 secs 2024-07-27T08:08:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:38 smithi077 ceph-mon[37153]: pgmap v2987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:38 smithi047 ceph-mon[36835]: pgmap v2987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:38 smithi029 ceph-mon[32610]: pgmap v2987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:08:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:40 smithi077 ceph-mon[37153]: pgmap v2988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:40 smithi047 ceph-mon[36835]: pgmap v2988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:40 smithi029 ceph-mon[32610]: pgmap v2988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:08:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:08:41.693+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:08:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:08:41.693+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:08:41.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:08:41.693+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:08:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:08:41.773+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:08:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:08:41.773+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:08:41.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:08:41.773+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:08:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:42 smithi077 ceph-mon[37153]: pgmap v2989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4744.462947 secs 2024-07-27T08:08:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:42 smithi047 ceph-mon[36835]: pgmap v2989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4744.462947 secs 2024-07-27T08:08:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:42 smithi029 ceph-mon[32610]: pgmap v2989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:08:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4744.462947 secs 2024-07-27T08:08:44.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:44 smithi029 ceph-mon[32610]: pgmap v2990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:44 smithi077 ceph-mon[37153]: pgmap v2990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:44 smithi047 ceph-mon[36835]: pgmap v2990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:08:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:08:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:08:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:08:46.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:46 smithi047 ceph-mon[36835]: pgmap v2991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:46.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:08:46.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:08:46.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:46 smithi047 ceph-mon[36835]: pgmap v2992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:08:46.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:46 smithi047 ceph-mon[36835]: pgmap v2993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:08:46.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:08:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:46 smithi077 ceph-mon[37153]: pgmap v2991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:08:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:08:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:46 smithi077 ceph-mon[37153]: pgmap v2992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:08:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:46 smithi077 ceph-mon[37153]: pgmap v2993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:08:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:08:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:46 smithi029 ceph-mon[32610]: pgmap v2991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:08:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:08:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:08:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:46 smithi029 ceph-mon[32610]: pgmap v2992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:08:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:46 smithi029 ceph-mon[32610]: pgmap v2993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:08:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:08:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4749.463075 secs 2024-07-27T08:08:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4749.463075 secs 2024-07-27T08:08:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4749.463075 secs 2024-07-27T08:08:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:48 smithi077 ceph-mon[37153]: pgmap v2994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:08:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:48 smithi047 ceph-mon[36835]: pgmap v2994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:08:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:48 smithi029 ceph-mon[32610]: pgmap v2994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:08:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:51 smithi077 ceph-mon[37153]: pgmap v2995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:08:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:51 smithi047 ceph-mon[36835]: pgmap v2995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:08:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:51 smithi029 ceph-mon[32610]: pgmap v2995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:08:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4754.463200 secs 2024-07-27T08:08:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:53 smithi077 ceph-mon[37153]: pgmap v2996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:08:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4754.463200 secs 2024-07-27T08:08:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:53 smithi047 ceph-mon[36835]: pgmap v2996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:08:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4754.463200 secs 2024-07-27T08:08:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:53 smithi029 ceph-mon[32610]: pgmap v2996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:08:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:55 smithi077 ceph-mon[37153]: pgmap v2997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:08:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:55 smithi047 ceph-mon[36835]: pgmap v2997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:08:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:55 smithi029 ceph-mon[32610]: pgmap v2997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:08:56.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:08:56.708+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:08:56.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:08:56.708+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:08:56.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:08:56.708+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:08:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:08:56.789+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:08:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:08:56.789+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:08:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:08:56.789+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:08:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:57 smithi077 ceph-mon[37153]: pgmap v2998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:08:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:57 smithi047 ceph-mon[36835]: pgmap v2998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:08:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:57 smithi029 ceph-mon[32610]: pgmap v2998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:08:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4759.463296 secs 2024-07-27T08:08:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4759.463296 secs 2024-07-27T08:08:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4759.463296 secs 2024-07-27T08:08:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:08:59 smithi077 ceph-mon[37153]: pgmap v2999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:08:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:08:59 smithi047 ceph-mon[36835]: pgmap v2999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:08:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:08:59 smithi029 ceph-mon[32610]: pgmap v2999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:01 smithi077 ceph-mon[37153]: pgmap v3000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:01 smithi047 ceph-mon[36835]: pgmap v3000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:01 smithi029 ceph-mon[32610]: pgmap v3000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4764.463422 secs 2024-07-27T08:09:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:03 smithi077 ceph-mon[37153]: pgmap v3001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4764.463422 secs 2024-07-27T08:09:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:03 smithi047 ceph-mon[36835]: pgmap v3001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4764.463422 secs 2024-07-27T08:09:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:03 smithi029 ceph-mon[32610]: pgmap v3001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:05 smithi077 ceph-mon[37153]: pgmap v3002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:05 smithi047 ceph-mon[36835]: pgmap v3002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:05 smithi029 ceph-mon[32610]: pgmap v3002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:07 smithi077 ceph-mon[37153]: pgmap v3003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:07 smithi047 ceph-mon[36835]: pgmap v3003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:07 smithi029 ceph-mon[32610]: pgmap v3003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4769.463566 secs 2024-07-27T08:09:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4769.463566 secs 2024-07-27T08:09:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4769.463566 secs 2024-07-27T08:09:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:09 smithi077 ceph-mon[37153]: pgmap v3004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:09 smithi047 ceph-mon[36835]: pgmap v3004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:09 smithi029 ceph-mon[32610]: pgmap v3004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:11 smithi077 ceph-mon[37153]: pgmap v3005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:11 smithi047 ceph-mon[36835]: pgmap v3005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:11 smithi029 ceph-mon[32610]: pgmap v3005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:11.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:09:11.724+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:09:11.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:09:11.724+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:09:11.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:09:11.724+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:09:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:09:11.796+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:09:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:09:11.796+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:09:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:09:11.796+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:09:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4774.463694 secs 2024-07-27T08:09:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:13 smithi077 ceph-mon[37153]: pgmap v3006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4774.463694 secs 2024-07-27T08:09:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:13 smithi047 ceph-mon[36835]: pgmap v3006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4774.463694 secs 2024-07-27T08:09:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:13 smithi029 ceph-mon[32610]: pgmap v3006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:15 smithi077 ceph-mon[37153]: pgmap v3007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:15 smithi047 ceph-mon[36835]: pgmap v3007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:15 smithi029 ceph-mon[32610]: pgmap v3007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:17 smithi077 ceph-mon[37153]: pgmap v3008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:17 smithi047 ceph-mon[36835]: pgmap v3008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:17 smithi029 ceph-mon[32610]: pgmap v3008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4779.463816 secs 2024-07-27T08:09:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4779.463816 secs 2024-07-27T08:09:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4779.463816 secs 2024-07-27T08:09:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:19 smithi047 ceph-mon[36835]: pgmap v3009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:19 smithi029 ceph-mon[32610]: pgmap v3009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:19 smithi077 ceph-mon[37153]: pgmap v3009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:21 smithi047 ceph-mon[36835]: pgmap v3010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:21 smithi029 ceph-mon[32610]: pgmap v3010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:21 smithi077 ceph-mon[37153]: pgmap v3010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:23.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4784.463845 secs 2024-07-27T08:09:23.450 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:23 smithi047 ceph-mon[36835]: pgmap v3011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4784.463845 secs 2024-07-27T08:09:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:23 smithi029 ceph-mon[32610]: pgmap v3011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4784.463845 secs 2024-07-27T08:09:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:23 smithi077 ceph-mon[37153]: pgmap v3011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:25 smithi047 ceph-mon[36835]: pgmap v3012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:25 smithi029 ceph-mon[32610]: pgmap v3012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:25 smithi077 ceph-mon[37153]: pgmap v3012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:26.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:09:26.738+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:09:26.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:09:26.738+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:09:26.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:09:26.738+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:09:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:09:26.804+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:09:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:09:26.804+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:09:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:09:26.804+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:09:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:27 smithi047 ceph-mon[36835]: pgmap v3013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:27 smithi029 ceph-mon[32610]: pgmap v3013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:27 smithi077 ceph-mon[37153]: pgmap v3013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4789.463983 secs 2024-07-27T08:09:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4789.463983 secs 2024-07-27T08:09:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4789.463983 secs 2024-07-27T08:09:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:29 smithi047 ceph-mon[36835]: pgmap v3014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:29 smithi029 ceph-mon[32610]: pgmap v3014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:29 smithi077 ceph-mon[37153]: pgmap v3014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:31 smithi047 ceph-mon[36835]: pgmap v3015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:09:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:31 smithi029 ceph-mon[32610]: pgmap v3015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:09:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:31 smithi077 ceph-mon[37153]: pgmap v3015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:09:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4794.464103 secs 2024-07-27T08:09:33.458 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:33 smithi047 ceph-mon[36835]: pgmap v3016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4794.464103 secs 2024-07-27T08:09:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:33 smithi029 ceph-mon[32610]: pgmap v3016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4794.464103 secs 2024-07-27T08:09:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:33 smithi077 ceph-mon[37153]: pgmap v3016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:35 smithi047 ceph-mon[36835]: pgmap v3017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:35 smithi029 ceph-mon[32610]: pgmap v3017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:35 smithi077 ceph-mon[37153]: pgmap v3017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:37 smithi047 ceph-mon[36835]: pgmap v3018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:37 smithi029 ceph-mon[32610]: pgmap v3018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:37 smithi077 ceph-mon[37153]: pgmap v3018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4799.464185 secs 2024-07-27T08:09:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4799.464185 secs 2024-07-27T08:09:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4799.464185 secs 2024-07-27T08:09:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:39 smithi047 ceph-mon[36835]: pgmap v3019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:39 smithi029 ceph-mon[32610]: pgmap v3019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:39 smithi077 ceph-mon[37153]: pgmap v3019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:09:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:41 smithi047 ceph-mon[36835]: pgmap v3020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:41 smithi029 ceph-mon[32610]: pgmap v3020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:41 smithi077 ceph-mon[37153]: pgmap v3020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:41.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:09:41.754+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:09:41.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:09:41.754+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:09:41.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:09:41.754+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:09:41.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:09:41.820+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:09:41.821 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:09:41.820+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:09:41.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:09:41.820+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:09:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4804.464268 secs 2024-07-27T08:09:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:43 smithi047 ceph-mon[36835]: pgmap v3021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4804.464268 secs 2024-07-27T08:09:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:43 smithi029 ceph-mon[32610]: pgmap v3021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4804.464268 secs 2024-07-27T08:09:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:43 smithi077 ceph-mon[37153]: pgmap v3021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:09:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:45 smithi047 ceph-mon[36835]: pgmap v3022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:45 smithi029 ceph-mon[32610]: pgmap v3022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:45 smithi077 ceph-mon[37153]: pgmap v3022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:09:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:09:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:09:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:09:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:47 smithi047 ceph-mon[36835]: pgmap v3023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:09:47.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:09:47.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:09:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:47 smithi029 ceph-mon[32610]: pgmap v3023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:09:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:09:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:09:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:47 smithi077 ceph-mon[37153]: pgmap v3023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:09:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:09:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:09:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:09:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:48 smithi047 ceph-mon[36835]: pgmap v3024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:09:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:48 smithi047 ceph-mon[36835]: pgmap v3025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:09:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4809.464359 secs 2024-07-27T08:09:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:48 smithi029 ceph-mon[32610]: pgmap v3024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:09:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:48 smithi029 ceph-mon[32610]: pgmap v3025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:09:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4809.464359 secs 2024-07-27T08:09:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:48 smithi077 ceph-mon[37153]: pgmap v3024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:09:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:48 smithi077 ceph-mon[37153]: pgmap v3025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:09:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4809.464359 secs 2024-07-27T08:09:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:50 smithi047 ceph-mon[36835]: pgmap v3026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:09:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:50 smithi029 ceph-mon[32610]: pgmap v3026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:09:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:50 smithi077 ceph-mon[37153]: pgmap v3026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:09:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:52 smithi047 ceph-mon[36835]: pgmap v3027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:09:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:52 smithi029 ceph-mon[32610]: pgmap v3027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:09:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:52 smithi077 ceph-mon[37153]: pgmap v3027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:09:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4814.464510 secs 2024-07-27T08:09:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4814.464510 secs 2024-07-27T08:09:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4814.464510 secs 2024-07-27T08:09:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:54 smithi047 ceph-mon[36835]: pgmap v3028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:09:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:54 smithi029 ceph-mon[32610]: pgmap v3028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:09:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:54 smithi077 ceph-mon[37153]: pgmap v3028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:09:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:56 smithi047 ceph-mon[36835]: pgmap v3029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:09:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:56 smithi029 ceph-mon[32610]: pgmap v3029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:09:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:56 smithi077 ceph-mon[37153]: pgmap v3029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:09:56.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:09:56.770+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:09:56.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:09:56.770+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:09:56.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:09:56.770+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:09:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:09:56.836+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:09:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:09:56.836+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:09:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:09:56.836+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:09:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:58 smithi047 ceph-mon[36835]: pgmap v3030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:09:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:09:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4819.464631 secs 2024-07-27T08:09:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:58 smithi029 ceph-mon[32610]: pgmap v3030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:09:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:09:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4819.464631 secs 2024-07-27T08:09:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:58 smithi077 ceph-mon[37153]: pgmap v3030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:09:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:09:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4819.464631 secs 2024-07-27T08:10:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:00 smithi047 ceph-mon[36835]: pgmap v3031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:10:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:00 smithi029 ceph-mon[32610]: pgmap v3031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:10:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:00 smithi077 ceph-mon[37153]: pgmap v3031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:10:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:02 smithi029 ceph-mon[32610]: pgmap v3032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:02 smithi077 ceph-mon[37153]: pgmap v3032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:02 smithi047 ceph-mon[36835]: pgmap v3032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4824.464816 secs 2024-07-27T08:10:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4824.464816 secs 2024-07-27T08:10:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4824.464816 secs 2024-07-27T08:10:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:04 smithi029 ceph-mon[32610]: pgmap v3033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:04 smithi077 ceph-mon[37153]: pgmap v3033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:04 smithi047 ceph-mon[36835]: pgmap v3033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:06 smithi029 ceph-mon[32610]: pgmap v3034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:06 smithi077 ceph-mon[37153]: pgmap v3034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:06 smithi047 ceph-mon[36835]: pgmap v3034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:08 smithi077 ceph-mon[37153]: pgmap v3035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4829.464932 secs 2024-07-27T08:10:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:08 smithi047 ceph-mon[36835]: pgmap v3035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4829.464932 secs 2024-07-27T08:10:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:08 smithi029 ceph-mon[32610]: pgmap v3035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4829.464932 secs 2024-07-27T08:10:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:10 smithi077 ceph-mon[37153]: pgmap v3036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:10 smithi047 ceph-mon[36835]: pgmap v3036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:10 smithi029 ceph-mon[32610]: pgmap v3036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:11.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:11.785+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:10:11.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:11.785+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:10:11.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:11.785+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:10:11.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:11.837+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:10:11.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:11.837+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:10:11.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:11.837+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:10:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:12 smithi077 ceph-mon[37153]: pgmap v3037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:12 smithi047 ceph-mon[36835]: pgmap v3037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:12 smithi029 ceph-mon[32610]: pgmap v3037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4834.465043 secs 2024-07-27T08:10:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4834.465043 secs 2024-07-27T08:10:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4834.465043 secs 2024-07-27T08:10:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:14 smithi077 ceph-mon[37153]: pgmap v3038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:14.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:14 smithi047 ceph-mon[36835]: pgmap v3038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:14 smithi029 ceph-mon[32610]: pgmap v3038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:16 smithi077 ceph-mon[37153]: pgmap v3039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:16 smithi047 ceph-mon[36835]: pgmap v3039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:16 smithi029 ceph-mon[32610]: pgmap v3039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:18.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:18 smithi047 ceph-mon[36835]: pgmap v3040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:18.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4839.465128 secs 2024-07-27T08:10:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:18 smithi077 ceph-mon[37153]: pgmap v3040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4839.465128 secs 2024-07-27T08:10:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:18 smithi029 ceph-mon[32610]: pgmap v3040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4839.465128 secs 2024-07-27T08:10:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:20 smithi077 ceph-mon[37153]: pgmap v3041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:20 smithi047 ceph-mon[36835]: pgmap v3041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:20 smithi029 ceph-mon[32610]: pgmap v3041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:22 smithi077 ceph-mon[37153]: pgmap v3042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:22 smithi047 ceph-mon[36835]: pgmap v3042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:22 smithi029 ceph-mon[32610]: pgmap v3042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4844.465244 secs 2024-07-27T08:10:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4844.465244 secs 2024-07-27T08:10:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4844.465244 secs 2024-07-27T08:10:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:24 smithi077 ceph-mon[37153]: pgmap v3043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:24 smithi047 ceph-mon[36835]: pgmap v3043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:24 smithi029 ceph-mon[32610]: pgmap v3043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:26 smithi077 ceph-mon[37153]: pgmap v3044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:26 smithi047 ceph-mon[36835]: pgmap v3044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:26 smithi029 ceph-mon[32610]: pgmap v3044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:26.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:26.800+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:10:26.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:26.800+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:10:26.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:26.800+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:10:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:26.837+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:10:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:26.837+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:10:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:26.837+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:10:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:28 smithi077 ceph-mon[37153]: pgmap v3045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4849.465362 secs 2024-07-27T08:10:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:28 smithi047 ceph-mon[36835]: pgmap v3045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4849.465362 secs 2024-07-27T08:10:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:28 smithi029 ceph-mon[32610]: pgmap v3045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4849.465362 secs 2024-07-27T08:10:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:30 smithi047 ceph-mon[36835]: pgmap v3046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:30 smithi077 ceph-mon[37153]: pgmap v3046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:30 smithi029 ceph-mon[32610]: pgmap v3046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:32 smithi077 ceph-mon[37153]: pgmap v3047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:32 smithi047 ceph-mon[36835]: pgmap v3047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:32 smithi029 ceph-mon[32610]: pgmap v3047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4854.465458 secs 2024-07-27T08:10:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4854.465458 secs 2024-07-27T08:10:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4854.465458 secs 2024-07-27T08:10:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:34 smithi077 ceph-mon[37153]: pgmap v3048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:34 smithi047 ceph-mon[36835]: pgmap v3048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:34 smithi029 ceph-mon[32610]: pgmap v3048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:36 smithi077 ceph-mon[37153]: pgmap v3049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:36 smithi047 ceph-mon[36835]: pgmap v3049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:36 smithi029 ceph-mon[32610]: pgmap v3049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:38 smithi077 ceph-mon[37153]: pgmap v3050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4859.465578 secs 2024-07-27T08:10:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:38 smithi047 ceph-mon[36835]: pgmap v3050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4859.465578 secs 2024-07-27T08:10:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:38 smithi029 ceph-mon[32610]: pgmap v3050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4859.465578 secs 2024-07-27T08:10:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:40 smithi077 ceph-mon[37153]: pgmap v3051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:40 smithi047 ceph-mon[36835]: pgmap v3051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:40 smithi029 ceph-mon[32610]: pgmap v3051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:10:41.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:41.806+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:10:41.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:41.806+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:10:41.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:41.806+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:10:41.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:41.842+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:10:41.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:41.842+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:10:41.843 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:41.842+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:10:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:42 smithi077 ceph-mon[37153]: pgmap v3052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:42 smithi047 ceph-mon[36835]: pgmap v3052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:42 smithi029 ceph-mon[32610]: pgmap v3052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:10:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:43.365+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:10:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:43.365+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:10:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:43.365+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:10:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:43.365+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:10:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4864.465704 secs 2024-07-27T08:10:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4864.465704 secs 2024-07-27T08:10:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4864.465704 secs 2024-07-27T08:10:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:44 smithi077 ceph-mon[37153]: pgmap v3053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:44 smithi047 ceph-mon[36835]: pgmap v3053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:44 smithi029 ceph-mon[32610]: pgmap v3053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:10:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:46 smithi077 ceph-mon[37153]: pgmap v3054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:46 smithi047 ceph-mon[36835]: pgmap v3054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:46 smithi029 ceph-mon[32610]: pgmap v3054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:10:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:10:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:10:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:10:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:10:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:10:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:10:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:10:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:10:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:10:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:10:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:10:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:10:48.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:48 smithi047 ceph-mon[36835]: pgmap v3055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:48.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4869.465833 secs 2024-07-27T08:10:48.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:48 smithi047 ceph-mon[36835]: pgmap v3056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:10:48.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:48 smithi047 ceph-mon[36835]: pgmap v3057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:10:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:48 smithi077 ceph-mon[37153]: pgmap v3055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4869.465833 secs 2024-07-27T08:10:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:48 smithi077 ceph-mon[37153]: pgmap v3056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:10:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:48 smithi077 ceph-mon[37153]: pgmap v3057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:10:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:48 smithi029 ceph-mon[32610]: pgmap v3055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:10:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4869.465833 secs 2024-07-27T08:10:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:48 smithi029 ceph-mon[32610]: pgmap v3056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:10:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:48 smithi029 ceph-mon[32610]: pgmap v3057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:10:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:50 smithi077 ceph-mon[37153]: pgmap v3058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:10:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:50 smithi047 ceph-mon[36835]: pgmap v3058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:10:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:50 smithi029 ceph-mon[32610]: pgmap v3058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:10:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:52 smithi077 ceph-mon[37153]: pgmap v3059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:10:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:52 smithi047 ceph-mon[36835]: pgmap v3059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:10:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:52 smithi029 ceph-mon[32610]: pgmap v3059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:10:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4874.465922 secs 2024-07-27T08:10:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4874.465922 secs 2024-07-27T08:10:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4874.465922 secs 2024-07-27T08:10:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:54 smithi077 ceph-mon[37153]: pgmap v3060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:10:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:54 smithi047 ceph-mon[36835]: pgmap v3060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:10:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:54 smithi029 ceph-mon[32610]: pgmap v3060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:10:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:56 smithi077 ceph-mon[37153]: pgmap v3061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:10:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:56 smithi047 ceph-mon[36835]: pgmap v3061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:10:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:56 smithi029 ceph-mon[32610]: pgmap v3061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:10:56.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:56.810+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:10:56.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:56.810+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:10:56.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:56.810+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:10:56.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:56.857+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:10:56.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:56.857+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:10:56.859 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:10:56.857+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:10:58.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:58.367+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:10:58.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:58.367+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:10:58.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:58.367+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:10:58.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:10:58.367+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:10:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4879.466067 secs 2024-07-27T08:10:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:10:58 smithi077 ceph-mon[37153]: pgmap v3062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:10:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4879.466067 secs 2024-07-27T08:10:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:10:58 smithi047 ceph-mon[36835]: pgmap v3062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:10:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4879.466067 secs 2024-07-27T08:10:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:10:58 smithi029 ceph-mon[32610]: pgmap v3062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:11:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:00 smithi077 ceph-mon[37153]: pgmap v3063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:00 smithi047 ceph-mon[36835]: pgmap v3063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:00 smithi029 ceph-mon[32610]: pgmap v3063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:02 smithi077 ceph-mon[37153]: pgmap v3064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:02 smithi047 ceph-mon[36835]: pgmap v3064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:02 smithi029 ceph-mon[32610]: pgmap v3064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4884.466152 secs 2024-07-27T08:11:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4884.466152 secs 2024-07-27T08:11:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4884.466152 secs 2024-07-27T08:11:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:04 smithi077 ceph-mon[37153]: pgmap v3065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:04 smithi047 ceph-mon[36835]: pgmap v3065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:04 smithi029 ceph-mon[32610]: pgmap v3065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:06 smithi029 ceph-mon[32610]: pgmap v3066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:11:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:06 smithi077 ceph-mon[37153]: pgmap v3066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:11:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:06 smithi047 ceph-mon[36835]: pgmap v3066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:11:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4889.466250 secs 2024-07-27T08:11:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:08 smithi077 ceph-mon[37153]: pgmap v3067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4889.466250 secs 2024-07-27T08:11:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:08 smithi047 ceph-mon[36835]: pgmap v3067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4889.466250 secs 2024-07-27T08:11:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:08 smithi029 ceph-mon[32610]: pgmap v3067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:10 smithi077 ceph-mon[37153]: pgmap v3068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:10 smithi047 ceph-mon[36835]: pgmap v3068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:10 smithi029 ceph-mon[32610]: pgmap v3068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:11.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:11:11.826+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:11:11.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:11:11.826+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:11:11.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:11:11.826+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:11:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:11:11.873+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:11:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:11:11.873+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:11:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:11:11.873+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:11:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:12 smithi077 ceph-mon[37153]: pgmap v3069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:12 smithi047 ceph-mon[36835]: pgmap v3069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:12 smithi029 ceph-mon[32610]: pgmap v3069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4894.466378 secs 2024-07-27T08:11:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4894.466378 secs 2024-07-27T08:11:13.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4894.466378 secs 2024-07-27T08:11:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:14 smithi077 ceph-mon[37153]: pgmap v3070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:14 smithi047 ceph-mon[36835]: pgmap v3070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:14 smithi029 ceph-mon[32610]: pgmap v3070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:16 smithi077 ceph-mon[37153]: pgmap v3071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:11:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:16 smithi047 ceph-mon[36835]: pgmap v3071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:11:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:16 smithi029 ceph-mon[32610]: pgmap v3071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:11:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4899.466505 secs 2024-07-27T08:11:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:18 smithi047 ceph-mon[36835]: pgmap v3072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4899.466505 secs 2024-07-27T08:11:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:18 smithi029 ceph-mon[32610]: pgmap v3072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4899.466505 secs 2024-07-27T08:11:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:18 smithi077 ceph-mon[37153]: pgmap v3072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:20 smithi047 ceph-mon[36835]: pgmap v3073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:20 smithi029 ceph-mon[32610]: pgmap v3073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:20 smithi077 ceph-mon[37153]: pgmap v3073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:22 smithi047 ceph-mon[36835]: pgmap v3074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:22 smithi029 ceph-mon[32610]: pgmap v3074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:22 smithi077 ceph-mon[37153]: pgmap v3074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4904.466632 secs 2024-07-27T08:11:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4904.466632 secs 2024-07-27T08:11:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4904.466632 secs 2024-07-27T08:11:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:24 smithi047 ceph-mon[36835]: pgmap v3075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:24 smithi029 ceph-mon[32610]: pgmap v3075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:24 smithi077 ceph-mon[37153]: pgmap v3075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:25 smithi047 ceph-mon[36835]: pgmap v3076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:11:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:25 smithi029 ceph-mon[32610]: pgmap v3076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:11:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:25 smithi077 ceph-mon[37153]: pgmap v3076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:11:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:11:26.837+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:11:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:11:26.837+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:11:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:11:26.837+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:11:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:11:26.888+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:11:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:11:26.888+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:11:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:11:26.888+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:11:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4909.466756 secs 2024-07-27T08:11:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:28 smithi047 ceph-mon[36835]: pgmap v3077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4909.466756 secs 2024-07-27T08:11:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:28 smithi029 ceph-mon[32610]: pgmap v3077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4909.466756 secs 2024-07-27T08:11:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:28 smithi077 ceph-mon[37153]: pgmap v3077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:30 smithi047 ceph-mon[36835]: pgmap v3078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:30 smithi029 ceph-mon[32610]: pgmap v3078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:30 smithi077 ceph-mon[37153]: pgmap v3078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:32 smithi047 ceph-mon[36835]: pgmap v3079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:32 smithi029 ceph-mon[32610]: pgmap v3079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:32 smithi077 ceph-mon[37153]: pgmap v3079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4914.466861 secs 2024-07-27T08:11:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4914.466861 secs 2024-07-27T08:11:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4914.466861 secs 2024-07-27T08:11:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:34 smithi047 ceph-mon[36835]: pgmap v3080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:34 smithi029 ceph-mon[32610]: pgmap v3080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:34 smithi077 ceph-mon[37153]: pgmap v3080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:36 smithi029 ceph-mon[32610]: pgmap v3081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:11:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:36 smithi077 ceph-mon[37153]: pgmap v3081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:11:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:36 smithi047 ceph-mon[36835]: pgmap v3081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:11:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4919.466996 secs 2024-07-27T08:11:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:38 smithi029 ceph-mon[32610]: pgmap v3082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4919.466996 secs 2024-07-27T08:11:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:38 smithi077 ceph-mon[37153]: pgmap v3082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4919.466996 secs 2024-07-27T08:11:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:38 smithi047 ceph-mon[36835]: pgmap v3082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:40 smithi029 ceph-mon[32610]: pgmap v3083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:40 smithi077 ceph-mon[37153]: pgmap v3083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:40 smithi047 ceph-mon[36835]: pgmap v3083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:11:41.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:11:41.853+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:11:41.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:11:41.853+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:11:41.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:11:41.853+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:11:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:11:41.903+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:11:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:11:41.903+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:11:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:11:41.903+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:11:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:42 smithi029 ceph-mon[32610]: pgmap v3084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:42 smithi077 ceph-mon[37153]: pgmap v3084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:42 smithi047 ceph-mon[36835]: pgmap v3084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4924.467115 secs 2024-07-27T08:11:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4924.467115 secs 2024-07-27T08:11:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4924.467115 secs 2024-07-27T08:11:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:44 smithi077 ceph-mon[37153]: pgmap v3085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:44 smithi047 ceph-mon[36835]: pgmap v3085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:44 smithi029 ceph-mon[32610]: pgmap v3085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:11:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:46 smithi077 ceph-mon[37153]: pgmap v3086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:11:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:46 smithi047 ceph-mon[36835]: pgmap v3086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:11:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:46 smithi029 ceph-mon[32610]: pgmap v3086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:11:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:11:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:11:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:11:48.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4929.467241 secs 2024-07-27T08:11:48.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:48 smithi047 ceph-mon[36835]: pgmap v3087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:48.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:11:48.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:11:48.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:11:48.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4929.467241 secs 2024-07-27T08:11:48.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:48 smithi077 ceph-mon[37153]: pgmap v3087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:48.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:11:48.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:11:48.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:11:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4929.467241 secs 2024-07-27T08:11:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:48 smithi029 ceph-mon[32610]: pgmap v3087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:11:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:11:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:11:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:11:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:49 smithi077 ceph-mon[37153]: pgmap v3088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T08:11:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:49 smithi077 ceph-mon[37153]: pgmap v3089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:11:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:49 smithi047 ceph-mon[36835]: pgmap v3088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T08:11:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:49 smithi047 ceph-mon[36835]: pgmap v3089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:11:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:49 smithi029 ceph-mon[32610]: pgmap v3088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T08:11:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:49 smithi029 ceph-mon[32610]: pgmap v3089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:11:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:51 smithi077 ceph-mon[37153]: pgmap v3090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:11:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:51 smithi047 ceph-mon[36835]: pgmap v3090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:11:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:51 smithi029 ceph-mon[32610]: pgmap v3090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:11:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:53 smithi077 ceph-mon[37153]: pgmap v3091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:11:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4934.467366 secs 2024-07-27T08:11:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:53 smithi047 ceph-mon[36835]: pgmap v3091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:11:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4934.467366 secs 2024-07-27T08:11:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:53 smithi029 ceph-mon[32610]: pgmap v3091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:11:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4934.467366 secs 2024-07-27T08:11:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:55 smithi077 ceph-mon[37153]: pgmap v3092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:11:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:55 smithi047 ceph-mon[36835]: pgmap v3092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:11:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:55 smithi029 ceph-mon[32610]: pgmap v3092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:11:56.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:11:56.868+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:11:56.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:11:56.868+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:11:56.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:11:56.868+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:11:56.920 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:11:56.919+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:11:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:11:56.919+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:11:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:11:56.919+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:11:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:57 smithi077 ceph-mon[37153]: pgmap v3093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:11:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:57 smithi047 ceph-mon[36835]: pgmap v3093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:11:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:57 smithi029 ceph-mon[32610]: pgmap v3093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:11:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4939.467508 secs 2024-07-27T08:11:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4939.467508 secs 2024-07-27T08:11:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4939.467508 secs 2024-07-27T08:11:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:11:59 smithi077 ceph-mon[37153]: pgmap v3094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:11:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:11:59 smithi047 ceph-mon[36835]: pgmap v3094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:11:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:11:59 smithi029 ceph-mon[32610]: pgmap v3094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:12:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:01 smithi077 ceph-mon[37153]: pgmap v3095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:01 smithi047 ceph-mon[36835]: pgmap v3095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:01 smithi029 ceph-mon[32610]: pgmap v3095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:03 smithi077 ceph-mon[37153]: pgmap v3096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4944.467637 secs 2024-07-27T08:12:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:03 smithi047 ceph-mon[36835]: pgmap v3096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:03.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4944.467637 secs 2024-07-27T08:12:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:03 smithi029 ceph-mon[32610]: pgmap v3096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4944.467637 secs 2024-07-27T08:12:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:05 smithi077 ceph-mon[37153]: pgmap v3097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:05 smithi047 ceph-mon[36835]: pgmap v3097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:05 smithi029 ceph-mon[32610]: pgmap v3097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:07 smithi077 ceph-mon[37153]: pgmap v3098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:07 smithi047 ceph-mon[36835]: pgmap v3098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:07 smithi029 ceph-mon[32610]: pgmap v3098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4949.467763 secs 2024-07-27T08:12:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4949.467763 secs 2024-07-27T08:12:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4949.467763 secs 2024-07-27T08:12:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:09 smithi077 ceph-mon[37153]: pgmap v3099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:09 smithi047 ceph-mon[36835]: pgmap v3099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:09 smithi029 ceph-mon[32610]: pgmap v3099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:11 smithi077 ceph-mon[37153]: pgmap v3100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:11 smithi047 ceph-mon[36835]: pgmap v3100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:11 smithi029 ceph-mon[32610]: pgmap v3100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:11.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:11.884+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:12:11.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:11.884+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:12:11.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:11.884+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:12:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:11.929+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:12:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:11.929+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:12:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:11.929+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:12:13.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:13.379+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:12:13.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:13.379+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:12:13.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:13.380+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:12:13.381 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:13.380+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:12:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:13 smithi077 ceph-mon[37153]: pgmap v3101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4954.467887 secs 2024-07-27T08:12:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:13 smithi047 ceph-mon[36835]: pgmap v3101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4954.467887 secs 2024-07-27T08:12:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:13 smithi029 ceph-mon[32610]: pgmap v3101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4954.467887 secs 2024-07-27T08:12:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:15 smithi077 ceph-mon[37153]: pgmap v3102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:15 smithi047 ceph-mon[36835]: pgmap v3102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:15 smithi029 ceph-mon[32610]: pgmap v3102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:17 smithi077 ceph-mon[37153]: pgmap v3103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:17 smithi047 ceph-mon[36835]: pgmap v3103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:17 smithi029 ceph-mon[32610]: pgmap v3103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:18.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4959.468019 secs 2024-07-27T08:12:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4959.468019 secs 2024-07-27T08:12:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4959.468019 secs 2024-07-27T08:12:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:19 smithi077 ceph-mon[37153]: pgmap v3104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:19 smithi047 ceph-mon[36835]: pgmap v3104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:19 smithi029 ceph-mon[32610]: pgmap v3104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:21 smithi077 ceph-mon[37153]: pgmap v3105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:21 smithi047 ceph-mon[36835]: pgmap v3105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:21 smithi029 ceph-mon[32610]: pgmap v3105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:23 smithi077 ceph-mon[37153]: pgmap v3106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4964.468100 secs 2024-07-27T08:12:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:23 smithi047 ceph-mon[36835]: pgmap v3106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4964.468100 secs 2024-07-27T08:12:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:23 smithi029 ceph-mon[32610]: pgmap v3106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4964.468100 secs 2024-07-27T08:12:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:25 smithi077 ceph-mon[37153]: pgmap v3107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:25 smithi047 ceph-mon[36835]: pgmap v3107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:25 smithi029 ceph-mon[32610]: pgmap v3107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:26.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:26.899+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:12:26.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:26.899+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:12:26.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:26.899+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:12:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:26.930+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:12:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:26.930+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:12:26.932 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:26.930+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:12:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:27 smithi077 ceph-mon[37153]: pgmap v3108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:27 smithi047 ceph-mon[36835]: pgmap v3108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:27 smithi029 ceph-mon[32610]: pgmap v3108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:28.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:28.381+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:12:28.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:28.381+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:12:28.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:28.381+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:12:28.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:28.381+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:12:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4969.468228 secs 2024-07-27T08:12:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4969.468228 secs 2024-07-27T08:12:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4969.468228 secs 2024-07-27T08:12:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:29 smithi047 ceph-mon[36835]: pgmap v3109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:29 smithi029 ceph-mon[32610]: pgmap v3109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:29 smithi077 ceph-mon[37153]: pgmap v3109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:31 smithi047 ceph-mon[36835]: pgmap v3110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:31 smithi029 ceph-mon[32610]: pgmap v3110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:31 smithi077 ceph-mon[37153]: pgmap v3110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:33 smithi047 ceph-mon[36835]: pgmap v3111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4974.468357 secs 2024-07-27T08:12:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:33 smithi029 ceph-mon[32610]: pgmap v3111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4974.468357 secs 2024-07-27T08:12:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:33 smithi077 ceph-mon[37153]: pgmap v3111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4974.468357 secs 2024-07-27T08:12:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:35 smithi047 ceph-mon[36835]: pgmap v3112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:35 smithi029 ceph-mon[32610]: pgmap v3112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:35 smithi077 ceph-mon[37153]: pgmap v3112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:37 smithi047 ceph-mon[36835]: pgmap v3113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:37 smithi029 ceph-mon[32610]: pgmap v3113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:37 smithi077 ceph-mon[37153]: pgmap v3113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4979.468492 secs 2024-07-27T08:12:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4979.468492 secs 2024-07-27T08:12:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4979.468492 secs 2024-07-27T08:12:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:39 smithi047 ceph-mon[36835]: pgmap v3114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:39 smithi029 ceph-mon[32610]: pgmap v3114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:39 smithi077 ceph-mon[37153]: pgmap v3114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:41 smithi047 ceph-mon[36835]: pgmap v3115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:41 smithi029 ceph-mon[32610]: pgmap v3115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:41 smithi077 ceph-mon[37153]: pgmap v3115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:41.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:41.914+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:12:41.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:41.914+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:12:41.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:41.914+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:12:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:41.932+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:12:41.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:41.932+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:12:41.934 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:41.932+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:12:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:42 smithi047 ceph-mon[36835]: pgmap v3116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:42 smithi029 ceph-mon[32610]: pgmap v3116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:42 smithi077 ceph-mon[37153]: pgmap v3116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4984.468624 secs 2024-07-27T08:12:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4984.468624 secs 2024-07-27T08:12:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4984.468624 secs 2024-07-27T08:12:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:44 smithi047 ceph-mon[36835]: pgmap v3117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:44 smithi029 ceph-mon[32610]: pgmap v3117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:44 smithi077 ceph-mon[37153]: pgmap v3117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:12:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:46 smithi077 ceph-mon[37153]: pgmap v3118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:46 smithi047 ceph-mon[36835]: pgmap v3118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:46 smithi029 ceph-mon[32610]: pgmap v3118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:12:48.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4989.468815 secs 2024-07-27T08:12:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4989.468815 secs 2024-07-27T08:12:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4989.468815 secs 2024-07-27T08:12:48.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:48 smithi029 ceph-mon[32610]: pgmap v3119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:48.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:12:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:48 smithi047 ceph-mon[36835]: pgmap v3119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:12:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:48 smithi077 ceph-mon[37153]: pgmap v3119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:12:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:12:51.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:50 smithi029 ceph-mon[32610]: pgmap v3120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:50 smithi077 ceph-mon[37153]: pgmap v3120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:50 smithi047 ceph-mon[36835]: pgmap v3120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:12:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: pgmap v3121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4994.468905 secs 2024-07-27T08:12:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:12:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:12:52.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:12:52.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:12:52.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:12:52.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:12:52.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:12:52.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:12:52.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: pgmap v3122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:12:52.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: pgmap v3123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-27T08:12:52.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: pgmap v3121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4994.468905 secs 2024-07-27T08:12:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:12:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:12:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:12:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:12:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:12:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:12:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:12:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:12:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: pgmap v3122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:12:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: pgmap v3123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-27T08:12:52.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: pgmap v3121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:12:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4994.468905 secs 2024-07-27T08:12:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:53.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:12:53.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:12:53.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:12:53.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:12:53.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:12:53.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:12:53.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:12:53.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:12:53.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: pgmap v3122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:12:53.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: pgmap v3123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-27T08:12:53.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:12:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:54 smithi077 ceph-mon[37153]: pgmap v3124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:12:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:54 smithi047 ceph-mon[36835]: pgmap v3124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:12:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:54 smithi029 ceph-mon[32610]: pgmap v3124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:12:56.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:56.925+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:12:56.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:56.925+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:12:56.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:12:56.925+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:12:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:56.942+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:12:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:56.942+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:12:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:12:56.942+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:12:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:56 smithi077 ceph-mon[37153]: pgmap v3125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-27T08:12:57.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:56 smithi029 ceph-mon[32610]: pgmap v3125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-27T08:12:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:56 smithi047 ceph-mon[36835]: pgmap v3125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-27T08:12:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 4999.469030 secs 2024-07-27T08:12:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 4999.469030 secs 2024-07-27T08:12:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 4999.469030 secs 2024-07-27T08:12:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:12:58 smithi047 ceph-mon[36835]: pgmap v3126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:12:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:12:58 smithi029 ceph-mon[32610]: pgmap v3126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:12:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:12:58 smithi077 ceph-mon[37153]: pgmap v3126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:13:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:00 smithi047 ceph-mon[36835]: pgmap v3127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T08:13:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:00 smithi029 ceph-mon[32610]: pgmap v3127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T08:13:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:00 smithi077 ceph-mon[37153]: pgmap v3127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T08:13:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5004.469159 secs 2024-07-27T08:13:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:02 smithi047 ceph-mon[36835]: pgmap v3128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:13:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5004.469159 secs 2024-07-27T08:13:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:02 smithi029 ceph-mon[32610]: pgmap v3128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:13:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5004.469159 secs 2024-07-27T08:13:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:02 smithi077 ceph-mon[37153]: pgmap v3128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:13:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:04 smithi077 ceph-mon[37153]: pgmap v3129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:04 smithi047 ceph-mon[36835]: pgmap v3129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:04 smithi029 ceph-mon[32610]: pgmap v3129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:06 smithi077 ceph-mon[37153]: pgmap v3130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:06 smithi047 ceph-mon[36835]: pgmap v3130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:06 smithi029 ceph-mon[32610]: pgmap v3130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5009.469284 secs 2024-07-27T08:13:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5009.469284 secs 2024-07-27T08:13:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5009.469284 secs 2024-07-27T08:13:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:08 smithi077 ceph-mon[37153]: pgmap v3131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:09.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:08 smithi047 ceph-mon[36835]: pgmap v3131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:08 smithi029 ceph-mon[32610]: pgmap v3131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:10 smithi077 ceph-mon[37153]: pgmap v3132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:10 smithi047 ceph-mon[36835]: pgmap v3132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:10 smithi029 ceph-mon[32610]: pgmap v3132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:11.941+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:13:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:11.941+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:13:11.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:11.941+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:13:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:11.958+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:13:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:11.958+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:13:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:11.958+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:13:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5014.469406 secs 2024-07-27T08:13:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:12 smithi077 ceph-mon[37153]: pgmap v3133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5014.469406 secs 2024-07-27T08:13:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:12 smithi047 ceph-mon[36835]: pgmap v3133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5014.469406 secs 2024-07-27T08:13:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:12 smithi029 ceph-mon[32610]: pgmap v3133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:14 smithi077 ceph-mon[37153]: pgmap v3134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:14 smithi047 ceph-mon[36835]: pgmap v3134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:14 smithi029 ceph-mon[32610]: pgmap v3134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:16 smithi077 ceph-mon[37153]: pgmap v3135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:16 smithi047 ceph-mon[36835]: pgmap v3135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:16 smithi029 ceph-mon[32610]: pgmap v3135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:18.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5019.469529 secs 2024-07-27T08:13:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5019.469529 secs 2024-07-27T08:13:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5019.469529 secs 2024-07-27T08:13:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:18 smithi077 ceph-mon[37153]: pgmap v3136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:18 smithi047 ceph-mon[36835]: pgmap v3136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:18 smithi029 ceph-mon[32610]: pgmap v3136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:20 smithi077 ceph-mon[37153]: pgmap v3137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:20 smithi047 ceph-mon[36835]: pgmap v3137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:20 smithi029 ceph-mon[32610]: pgmap v3137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5024.469670 secs 2024-07-27T08:13:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:22 smithi077 ceph-mon[37153]: pgmap v3138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5024.469670 secs 2024-07-27T08:13:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:22 smithi047 ceph-mon[36835]: pgmap v3138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5024.469670 secs 2024-07-27T08:13:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:22 smithi029 ceph-mon[32610]: pgmap v3138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:24 smithi077 ceph-mon[37153]: pgmap v3139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:24 smithi047 ceph-mon[36835]: pgmap v3139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:24 smithi029 ceph-mon[32610]: pgmap v3139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:26.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:26.956+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:13:26.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:26.956+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:13:26.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:26.956+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:13:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:26.972+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:13:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:26.972+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:13:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:26.972+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:13:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:26 smithi077 ceph-mon[37153]: pgmap v3140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:26 smithi047 ceph-mon[36835]: pgmap v3140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:26 smithi029 ceph-mon[32610]: pgmap v3140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5029.469765 secs 2024-07-27T08:13:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5029.469765 secs 2024-07-27T08:13:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5029.469765 secs 2024-07-27T08:13:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:28 smithi077 ceph-mon[37153]: pgmap v3141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:28 smithi047 ceph-mon[36835]: pgmap v3141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:28 smithi029 ceph-mon[32610]: pgmap v3141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:30 smithi077 ceph-mon[37153]: pgmap v3142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:30 smithi047 ceph-mon[36835]: pgmap v3142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:30 smithi029 ceph-mon[32610]: pgmap v3142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5034.469886 secs 2024-07-27T08:13:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:32 smithi077 ceph-mon[37153]: pgmap v3143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5034.469886 secs 2024-07-27T08:13:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:32 smithi047 ceph-mon[36835]: pgmap v3143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5034.469886 secs 2024-07-27T08:13:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:32 smithi029 ceph-mon[32610]: pgmap v3143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:34 smithi077 ceph-mon[37153]: pgmap v3144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:34 smithi047 ceph-mon[36835]: pgmap v3144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:34 smithi029 ceph-mon[32610]: pgmap v3144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:36 smithi077 ceph-mon[37153]: pgmap v3145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:36 smithi047 ceph-mon[36835]: pgmap v3145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:36 smithi029 ceph-mon[32610]: pgmap v3145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5039.469951 secs 2024-07-27T08:13:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5039.469951 secs 2024-07-27T08:13:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5039.469951 secs 2024-07-27T08:13:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:38 smithi047 ceph-mon[36835]: pgmap v3146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:38 smithi029 ceph-mon[32610]: pgmap v3146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:38 smithi077 ceph-mon[37153]: pgmap v3146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:40 smithi047 ceph-mon[36835]: pgmap v3147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:40 smithi029 ceph-mon[32610]: pgmap v3147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:40 smithi077 ceph-mon[37153]: pgmap v3147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:41.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:41.971+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:13:41.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:41.971+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:13:41.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:41.971+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:13:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:41.988+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:13:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:41.988+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:13:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:41.988+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:13:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5044.470078 secs 2024-07-27T08:13:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:42 smithi047 ceph-mon[36835]: pgmap v3148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5044.470078 secs 2024-07-27T08:13:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:42 smithi029 ceph-mon[32610]: pgmap v3148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5044.470078 secs 2024-07-27T08:13:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:42 smithi077 ceph-mon[37153]: pgmap v3148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:43.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:43.392+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:13:43.393 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:43.392+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:13:43.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:43.392+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:13:43.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:43.392+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:13:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:44 smithi047 ceph-mon[36835]: pgmap v3149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:44 smithi029 ceph-mon[32610]: pgmap v3149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:44 smithi077 ceph-mon[37153]: pgmap v3149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:46 smithi047 ceph-mon[36835]: pgmap v3150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:46 smithi029 ceph-mon[32610]: pgmap v3150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:46 smithi077 ceph-mon[37153]: pgmap v3150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:13:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5049.470140 secs 2024-07-27T08:13:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5049.470140 secs 2024-07-27T08:13:48.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5049.470140 secs 2024-07-27T08:13:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:48 smithi047 ceph-mon[36835]: pgmap v3151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:48 smithi029 ceph-mon[32610]: pgmap v3151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:48 smithi077 ceph-mon[37153]: pgmap v3151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:13:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:50 smithi047 ceph-mon[36835]: pgmap v3152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:50 smithi029 ceph-mon[32610]: pgmap v3152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:50 smithi077 ceph-mon[37153]: pgmap v3152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:13:53.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5054.470237 secs 2024-07-27T08:13:53.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:52 smithi077 ceph-mon[37153]: pgmap v3153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:53.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:13:53.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:53.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:53.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:53.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:53.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:52 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:13:53.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:52 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:13:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5054.470237 secs 2024-07-27T08:13:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:52 smithi047 ceph-mon[36835]: pgmap v3153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:13:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:53.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:53.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:53.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:52 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:13:53.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:52 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:13:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5054.470237 secs 2024-07-27T08:13:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:52 smithi029 ceph-mon[32610]: pgmap v3153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:13:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:13:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:53.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:52 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:13:53.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:52 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:13:55.142 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:54 smithi077 ceph-mon[37153]: pgmap v3154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:54 smithi047 ceph-mon[36835]: pgmap v3154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:54 smithi029 ceph-mon[32610]: pgmap v3154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:13:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:13:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:13:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:56.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:13:56.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:13:56.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:13:56.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:13:56.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: pgmap v3155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T08:13:56.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: pgmap v3156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:13:56.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:13:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:13:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:13:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:13:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:13:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:13:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:13:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: pgmap v3155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T08:13:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: pgmap v3156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:13:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:13:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:13:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:13:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:56.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:13:56.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:13:56.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:13:56.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:13:56.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:13:56.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: pgmap v3155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T08:13:56.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: pgmap v3156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:13:56.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:13:56.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:56.987+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:13:56.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:56.987+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:13:56.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:56.987+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:13:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:56.991+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:13:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:56.991+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:13:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:13:56.991+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:13:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5059.470380 secs 2024-07-27T08:13:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5059.470380 secs 2024-07-27T08:13:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5059.470380 secs 2024-07-27T08:13:58.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:58.394+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:13:58.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:58.394+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:13:58.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:58.394+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:13:58.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:13:58.394+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:13:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:13:58 smithi029 ceph-mon[32610]: pgmap v3157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:13:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:13:58 smithi077 ceph-mon[37153]: pgmap v3157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:13:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:13:58 smithi047 ceph-mon[36835]: pgmap v3157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:14:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:00 smithi077 ceph-mon[37153]: pgmap v3158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:14:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:00 smithi047 ceph-mon[36835]: pgmap v3158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:14:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:00 smithi029 ceph-mon[32610]: pgmap v3158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:14:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:02 smithi077 ceph-mon[37153]: pgmap v3159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:14:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5064.470531 secs 2024-07-27T08:14:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:02 smithi047 ceph-mon[36835]: pgmap v3159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:14:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5064.470531 secs 2024-07-27T08:14:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:02 smithi029 ceph-mon[32610]: pgmap v3159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:14:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5064.470531 secs 2024-07-27T08:14:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:04 smithi077 ceph-mon[37153]: pgmap v3160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:14:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:04 smithi047 ceph-mon[36835]: pgmap v3160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:14:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:04 smithi029 ceph-mon[32610]: pgmap v3160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:14:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:06 smithi077 ceph-mon[37153]: pgmap v3161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:14:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:06 smithi047 ceph-mon[36835]: pgmap v3161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:14:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:06 smithi029 ceph-mon[32610]: pgmap v3161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:14:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5069.470674 secs 2024-07-27T08:14:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5069.470674 secs 2024-07-27T08:14:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5069.470674 secs 2024-07-27T08:14:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:08 smithi077 ceph-mon[37153]: pgmap v3162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:08 smithi047 ceph-mon[36835]: pgmap v3162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:08 smithi029 ceph-mon[32610]: pgmap v3162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:10 smithi077 ceph-mon[37153]: pgmap v3163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:10 smithi047 ceph-mon[36835]: pgmap v3163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:10 smithi029 ceph-mon[32610]: pgmap v3163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:11.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:14:11.998+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:14:11.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:14:11.998+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:14:11.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:14:11.998+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:14:12.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:14:12.002+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:14:12.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:14:12.002+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:14:12.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:14:12.002+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:14:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:12 smithi077 ceph-mon[37153]: pgmap v3164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5074.470821 secs 2024-07-27T08:14:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:12 smithi047 ceph-mon[36835]: pgmap v3164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5074.470821 secs 2024-07-27T08:14:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:12 smithi029 ceph-mon[32610]: pgmap v3164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5074.470821 secs 2024-07-27T08:14:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:14 smithi077 ceph-mon[37153]: pgmap v3165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:14 smithi047 ceph-mon[36835]: pgmap v3165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:14 smithi029 ceph-mon[32610]: pgmap v3165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:16 smithi077 ceph-mon[37153]: pgmap v3166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:14:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:16 smithi047 ceph-mon[36835]: pgmap v3166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:14:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:16 smithi029 ceph-mon[32610]: pgmap v3166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:14:18.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5079.470941 secs 2024-07-27T08:14:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5079.470941 secs 2024-07-27T08:14:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5079.470941 secs 2024-07-27T08:14:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:18 smithi077 ceph-mon[37153]: pgmap v3167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:18 smithi047 ceph-mon[36835]: pgmap v3167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:18 smithi029 ceph-mon[32610]: pgmap v3167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:20 smithi077 ceph-mon[37153]: pgmap v3168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:20 smithi047 ceph-mon[36835]: pgmap v3168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:20 smithi029 ceph-mon[32610]: pgmap v3168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:22 smithi077 ceph-mon[37153]: pgmap v3169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5084.471009 secs 2024-07-27T08:14:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:22 smithi047 ceph-mon[36835]: pgmap v3169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5084.471009 secs 2024-07-27T08:14:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:22 smithi029 ceph-mon[32610]: pgmap v3169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5084.471009 secs 2024-07-27T08:14:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:24 smithi047 ceph-mon[36835]: pgmap v3170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:24 smithi029 ceph-mon[32610]: pgmap v3170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:24 smithi077 ceph-mon[37153]: pgmap v3170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:14:27.010+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:14:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:14:27.010+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:14:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:14:27.010+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:14:27.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:14:27.012+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:14:27.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:14:27.012+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:14:27.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:14:27.012+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:14:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:26 smithi047 ceph-mon[36835]: pgmap v3171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:26 smithi029 ceph-mon[32610]: pgmap v3171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:26 smithi077 ceph-mon[37153]: pgmap v3171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:28.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5089.471099 secs 2024-07-27T08:14:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5089.471099 secs 2024-07-27T08:14:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5089.471099 secs 2024-07-27T08:14:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:28 smithi047 ceph-mon[36835]: pgmap v3172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:28 smithi029 ceph-mon[32610]: pgmap v3172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:28 smithi077 ceph-mon[37153]: pgmap v3172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:30 smithi047 ceph-mon[36835]: pgmap v3173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:30 smithi029 ceph-mon[32610]: pgmap v3173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:30 smithi077 ceph-mon[37153]: pgmap v3173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:32 smithi047 ceph-mon[36835]: pgmap v3174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5094.471229 secs 2024-07-27T08:14:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:32 smithi029 ceph-mon[32610]: pgmap v3174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5094.471229 secs 2024-07-27T08:14:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:32 smithi077 ceph-mon[37153]: pgmap v3174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5094.471229 secs 2024-07-27T08:14:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:34 smithi047 ceph-mon[36835]: pgmap v3175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:34 smithi029 ceph-mon[32610]: pgmap v3175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:34 smithi077 ceph-mon[37153]: pgmap v3175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:36 smithi047 ceph-mon[36835]: pgmap v3176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:14:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:36 smithi029 ceph-mon[32610]: pgmap v3176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:14:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:36 smithi077 ceph-mon[37153]: pgmap v3176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:14:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5099.471370 secs 2024-07-27T08:14:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5099.471370 secs 2024-07-27T08:14:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5099.471370 secs 2024-07-27T08:14:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:38 smithi047 ceph-mon[36835]: pgmap v3177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:38 smithi029 ceph-mon[32610]: pgmap v3177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:38 smithi077 ceph-mon[37153]: pgmap v3177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:40 smithi047 ceph-mon[36835]: pgmap v3178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:40 smithi029 ceph-mon[32610]: pgmap v3178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:40 smithi077 ceph-mon[37153]: pgmap v3178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:42.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:14:42.018+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:14:42.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:14:42.018+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:14:42.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:14:42.018+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:14:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:14:42.025+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:14:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:14:42.025+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:14:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:14:42.025+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:14:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:42 smithi047 ceph-mon[36835]: pgmap v3179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5104.471504 secs 2024-07-27T08:14:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:42 smithi029 ceph-mon[32610]: pgmap v3179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5104.471504 secs 2024-07-27T08:14:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:42 smithi077 ceph-mon[37153]: pgmap v3179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5104.471504 secs 2024-07-27T08:14:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:44 smithi047 ceph-mon[36835]: pgmap v3180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:44 smithi029 ceph-mon[32610]: pgmap v3180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:44 smithi077 ceph-mon[37153]: pgmap v3180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:46 smithi047 ceph-mon[36835]: pgmap v3181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:14:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:46 smithi029 ceph-mon[32610]: pgmap v3181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:14:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:46 smithi077 ceph-mon[37153]: pgmap v3181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:14:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5109.471615 secs 2024-07-27T08:14:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5109.471615 secs 2024-07-27T08:14:48.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5109.471615 secs 2024-07-27T08:14:49.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:48 smithi047 ceph-mon[36835]: pgmap v3182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:48 smithi029 ceph-mon[32610]: pgmap v3182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:48 smithi077 ceph-mon[37153]: pgmap v3182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:14:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:50 smithi047 ceph-mon[36835]: pgmap v3183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:50 smithi029 ceph-mon[32610]: pgmap v3183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:50 smithi077 ceph-mon[37153]: pgmap v3183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:14:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:52 smithi047 ceph-mon[36835]: pgmap v3184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5114.471754 secs 2024-07-27T08:14:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:52 smithi029 ceph-mon[32610]: pgmap v3184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5114.471754 secs 2024-07-27T08:14:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:52 smithi077 ceph-mon[37153]: pgmap v3184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:14:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5114.471754 secs 2024-07-27T08:14:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:54 smithi047 ceph-mon[36835]: pgmap v3185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:54 smithi029 ceph-mon[32610]: pgmap v3185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:54 smithi077 ceph-mon[37153]: pgmap v3185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:14:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:14:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:14:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:14:57.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:14:57.033+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:14:57.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:14:57.033+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:14:57.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:14:57.033+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:14:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:14:57.040+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:14:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:14:57.040+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:14:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:14:57.040+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:14:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:56 smithi047 ceph-mon[36835]: pgmap v3186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:14:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:14:57.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:14:57.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:56 smithi047 ceph-mon[36835]: pgmap v3187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:14:57.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:56 smithi047 ceph-mon[36835]: pgmap v3188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:14:57.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:14:57.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:56 smithi029 ceph-mon[32610]: pgmap v3186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:14:57.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:14:57.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:14:57.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:56 smithi029 ceph-mon[32610]: pgmap v3187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:14:57.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:56 smithi029 ceph-mon[32610]: pgmap v3188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:14:57.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:14:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:56 smithi077 ceph-mon[37153]: pgmap v3186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:14:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:14:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:14:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:56 smithi077 ceph-mon[37153]: pgmap v3187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:14:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:56 smithi077 ceph-mon[37153]: pgmap v3188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:14:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:14:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5119.471872 secs 2024-07-27T08:14:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5119.471872 secs 2024-07-27T08:14:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5119.471872 secs 2024-07-27T08:14:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:14:58 smithi047 ceph-mon[36835]: pgmap v3189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:14:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:14:58 smithi029 ceph-mon[32610]: pgmap v3189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:14:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:14:58 smithi077 ceph-mon[37153]: pgmap v3189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:15:01.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:01 smithi047 ceph-mon[36835]: pgmap v3190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:15:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:01 smithi029 ceph-mon[32610]: pgmap v3190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:15:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:01 smithi077 ceph-mon[37153]: pgmap v3190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:15:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5124.471935 secs 2024-07-27T08:15:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:03 smithi047 ceph-mon[36835]: pgmap v3191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:15:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5124.471935 secs 2024-07-27T08:15:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:03 smithi029 ceph-mon[32610]: pgmap v3191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:15:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5124.471935 secs 2024-07-27T08:15:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:03 smithi077 ceph-mon[37153]: pgmap v3191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:15:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:04 smithi077 ceph-mon[37153]: pgmap v3192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:15:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:04 smithi047 ceph-mon[36835]: pgmap v3192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:15:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:04 smithi029 ceph-mon[32610]: pgmap v3192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:15:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:07 smithi047 ceph-mon[36835]: pgmap v3193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:15:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:07 smithi029 ceph-mon[32610]: pgmap v3193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:15:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:07 smithi077 ceph-mon[37153]: pgmap v3193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:15:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5129.472031 secs 2024-07-27T08:15:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5129.472031 secs 2024-07-27T08:15:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5129.472031 secs 2024-07-27T08:15:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:09 smithi047 ceph-mon[36835]: pgmap v3194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:15:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:09 smithi029 ceph-mon[32610]: pgmap v3194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:15:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:09 smithi077 ceph-mon[37153]: pgmap v3194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:15:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:11 smithi047 ceph-mon[36835]: pgmap v3195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:11 smithi029 ceph-mon[32610]: pgmap v3195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:11 smithi077 ceph-mon[37153]: pgmap v3195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:12.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:15:12.049+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:15:12.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:15:12.049+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:15:12.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:15:12.049+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:15:12.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:15:12.056+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:15:12.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:15:12.056+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:15:12.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:15:12.056+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:15:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5134.472097 secs 2024-07-27T08:15:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:13 smithi047 ceph-mon[36835]: pgmap v3196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5134.472097 secs 2024-07-27T08:15:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:13 smithi029 ceph-mon[32610]: pgmap v3196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5134.472097 secs 2024-07-27T08:15:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:13 smithi077 ceph-mon[37153]: pgmap v3196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:15 smithi047 ceph-mon[36835]: pgmap v3197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:15:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:15 smithi029 ceph-mon[32610]: pgmap v3197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:15:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:15 smithi077 ceph-mon[37153]: pgmap v3197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:15:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:17 smithi047 ceph-mon[36835]: pgmap v3198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:17 smithi029 ceph-mon[32610]: pgmap v3198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:17 smithi077 ceph-mon[37153]: pgmap v3198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:18.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5139.472254 secs 2024-07-27T08:15:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5139.472254 secs 2024-07-27T08:15:18.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5139.472254 secs 2024-07-27T08:15:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:19 smithi047 ceph-mon[36835]: pgmap v3199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:15:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:19 smithi029 ceph-mon[32610]: pgmap v3199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:15:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:19 smithi077 ceph-mon[37153]: pgmap v3199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:15:21.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:21 smithi047 ceph-mon[36835]: pgmap v3200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:21 smithi029 ceph-mon[32610]: pgmap v3200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:21 smithi077 ceph-mon[37153]: pgmap v3200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5144.472399 secs 2024-07-27T08:15:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:23 smithi047 ceph-mon[36835]: pgmap v3201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5144.472399 secs 2024-07-27T08:15:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:23 smithi029 ceph-mon[32610]: pgmap v3201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5144.472399 secs 2024-07-27T08:15:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:23 smithi077 ceph-mon[37153]: pgmap v3201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:25 smithi047 ceph-mon[36835]: pgmap v3202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:15:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:25 smithi029 ceph-mon[32610]: pgmap v3202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:15:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:25 smithi077 ceph-mon[37153]: pgmap v3202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:15:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:15:27.063+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:15:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:15:27.063+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:15:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:15:27.063+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:15:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:15:27.069+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:15:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:15:27.069+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:15:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:15:27.069+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:15:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:27 smithi047 ceph-mon[36835]: pgmap v3203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:27 smithi029 ceph-mon[32610]: pgmap v3203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:27 smithi077 ceph-mon[37153]: pgmap v3203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5149.472549 secs 2024-07-27T08:15:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5149.472549 secs 2024-07-27T08:15:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5149.472549 secs 2024-07-27T08:15:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:29 smithi047 ceph-mon[36835]: pgmap v3204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:15:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:29 smithi029 ceph-mon[32610]: pgmap v3204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:15:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:29 smithi077 ceph-mon[37153]: pgmap v3204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:15:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:31 smithi047 ceph-mon[36835]: pgmap v3205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:31 smithi029 ceph-mon[32610]: pgmap v3205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:31 smithi077 ceph-mon[37153]: pgmap v3205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5154.472692 secs 2024-07-27T08:15:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:33 smithi047 ceph-mon[36835]: pgmap v3206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5154.472692 secs 2024-07-27T08:15:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:33 smithi029 ceph-mon[32610]: pgmap v3206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5154.472692 secs 2024-07-27T08:15:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:33 smithi077 ceph-mon[37153]: pgmap v3206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:35 smithi047 ceph-mon[36835]: pgmap v3207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:15:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:35 smithi029 ceph-mon[32610]: pgmap v3207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:15:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:35 smithi077 ceph-mon[37153]: pgmap v3207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:15:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:37 smithi047 ceph-mon[36835]: pgmap v3208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:37 smithi029 ceph-mon[32610]: pgmap v3208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:37 smithi077 ceph-mon[37153]: pgmap v3208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5159.472841 secs 2024-07-27T08:15:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5159.472841 secs 2024-07-27T08:15:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5159.472841 secs 2024-07-27T08:15:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:39 smithi029 ceph-mon[32610]: pgmap v3209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:15:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:39 smithi077 ceph-mon[37153]: pgmap v3209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:15:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:39 smithi047 ceph-mon[36835]: pgmap v3209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:15:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:41 smithi029 ceph-mon[32610]: pgmap v3210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:15:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:41 smithi077 ceph-mon[37153]: pgmap v3210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:15:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:41 smithi047 ceph-mon[36835]: pgmap v3210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:15:42.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:15:42.064+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:15:42.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:15:42.064+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:15:42.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:15:42.064+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:15:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:15:42.085+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:15:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:15:42.085+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:15:42.086 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:15:42.085+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:15:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5164.472960 secs 2024-07-27T08:15:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:43 smithi029 ceph-mon[32610]: pgmap v3211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5164.472960 secs 2024-07-27T08:15:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:43 smithi077 ceph-mon[37153]: pgmap v3211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5164.472960 secs 2024-07-27T08:15:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:43 smithi047 ceph-mon[36835]: pgmap v3211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:45 smithi029 ceph-mon[32610]: pgmap v3212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:15:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:45 smithi077 ceph-mon[37153]: pgmap v3212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:15:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:45 smithi047 ceph-mon[36835]: pgmap v3212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:15:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:47 smithi029 ceph-mon[32610]: pgmap v3213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:15:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:47 smithi077 ceph-mon[37153]: pgmap v3213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:15:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:47 smithi047 ceph-mon[36835]: pgmap v3213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:15:48.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5169.473083 secs 2024-07-27T08:15:48.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5169.473083 secs 2024-07-27T08:15:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5169.473083 secs 2024-07-27T08:15:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:49 smithi077 ceph-mon[37153]: pgmap v3214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:15:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:49 smithi047 ceph-mon[36835]: pgmap v3214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:15:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:49 smithi029 ceph-mon[32610]: pgmap v3214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:15:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:51 smithi077 ceph-mon[37153]: pgmap v3215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:51 smithi047 ceph-mon[36835]: pgmap v3215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:51 smithi029 ceph-mon[32610]: pgmap v3215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5174.473190 secs 2024-07-27T08:15:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:53 smithi077 ceph-mon[37153]: pgmap v3216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5174.473190 secs 2024-07-27T08:15:53.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:53 smithi047 ceph-mon[36835]: pgmap v3216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5174.473190 secs 2024-07-27T08:15:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:53 smithi029 ceph-mon[32610]: pgmap v3216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:15:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:55 smithi077 ceph-mon[37153]: pgmap v3217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:15:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:55 smithi047 ceph-mon[36835]: pgmap v3217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:15:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:55 smithi029 ceph-mon[32610]: pgmap v3217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:15:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:15:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:15:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:15:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:15:57.080+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:15:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:15:57.080+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:15:57.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:15:57.080+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:15:57.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:15:57.097+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:15:57.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:15:57.097+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:15:57.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:15:57.097+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:15:57.465 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:57 smithi029 ceph-mon[32610]: pgmap v3218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:57.470 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:15:57.470 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:15:57.471 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:57 smithi077 ceph-mon[37153]: pgmap v3218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:15:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:15:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:57 smithi047 ceph-mon[36835]: pgmap v3218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:15:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:15:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:15:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:15:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:58 smithi077 ceph-mon[37153]: pgmap v3219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:15:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:58 smithi077 ceph-mon[37153]: pgmap v3220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:15:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:15:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5179.473283 secs 2024-07-27T08:15:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:58 smithi047 ceph-mon[36835]: pgmap v3219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:15:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:58 smithi047 ceph-mon[36835]: pgmap v3220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:15:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:15:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5179.473283 secs 2024-07-27T08:15:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:58 smithi029 ceph-mon[32610]: pgmap v3219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:15:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:58 smithi029 ceph-mon[32610]: pgmap v3220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:15:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:15:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5179.473283 secs 2024-07-27T08:16:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:00 smithi077 ceph-mon[37153]: pgmap v3221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:16:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:00 smithi047 ceph-mon[36835]: pgmap v3221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:16:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:00 smithi029 ceph-mon[32610]: pgmap v3221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:16:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:02 smithi077 ceph-mon[37153]: pgmap v3222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:16:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:02 smithi047 ceph-mon[36835]: pgmap v3222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:16:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:02 smithi029 ceph-mon[32610]: pgmap v3222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:16:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5184.473411 secs 2024-07-27T08:16:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5184.473411 secs 2024-07-27T08:16:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5184.473411 secs 2024-07-27T08:16:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:04 smithi077 ceph-mon[37153]: pgmap v3223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:16:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:04 smithi047 ceph-mon[36835]: pgmap v3223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:16:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:04 smithi029 ceph-mon[32610]: pgmap v3223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:16:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:06 smithi077 ceph-mon[37153]: pgmap v3224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:16:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:06 smithi047 ceph-mon[36835]: pgmap v3224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:16:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:06 smithi029 ceph-mon[32610]: pgmap v3224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:16:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:08 smithi077 ceph-mon[37153]: pgmap v3225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T08:16:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5189.473490 secs 2024-07-27T08:16:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:08 smithi047 ceph-mon[36835]: pgmap v3225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T08:16:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5189.473490 secs 2024-07-27T08:16:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:08 smithi029 ceph-mon[32610]: pgmap v3225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T08:16:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5189.473490 secs 2024-07-27T08:16:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:10 smithi077 ceph-mon[37153]: pgmap v3226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:16:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:10 smithi047 ceph-mon[36835]: pgmap v3226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:16:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:10 smithi029 ceph-mon[32610]: pgmap v3226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:16:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:12.087+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:16:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:12.087+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:16:12.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:12.087+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:16:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:12.114+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:16:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:12.114+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:16:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:12.114+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:16:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:12 smithi029 ceph-mon[32610]: pgmap v3227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:16:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:12 smithi077 ceph-mon[37153]: pgmap v3227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:16:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:12 smithi047 ceph-mon[36835]: pgmap v3227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:16:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5194.473608 secs 2024-07-27T08:16:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5194.473608 secs 2024-07-27T08:16:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5194.473608 secs 2024-07-27T08:16:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:14 smithi077 ceph-mon[37153]: pgmap v3228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:16:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:14 smithi047 ceph-mon[36835]: pgmap v3228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:16:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:14 smithi029 ceph-mon[32610]: pgmap v3228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:16:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:16 smithi077 ceph-mon[37153]: pgmap v3229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:16:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:16 smithi047 ceph-mon[36835]: pgmap v3229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:16:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:16 smithi029 ceph-mon[32610]: pgmap v3229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:16:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:18 smithi077 ceph-mon[37153]: pgmap v3230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5199.473743 secs 2024-07-27T08:16:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:18 smithi047 ceph-mon[36835]: pgmap v3230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5199.473743 secs 2024-07-27T08:16:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:18 smithi029 ceph-mon[32610]: pgmap v3230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5199.473743 secs 2024-07-27T08:16:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:20 smithi077 ceph-mon[37153]: pgmap v3231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:16:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:20 smithi047 ceph-mon[36835]: pgmap v3231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:16:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:20 smithi029 ceph-mon[32610]: pgmap v3231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:16:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:22 smithi077 ceph-mon[37153]: pgmap v3232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:16:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:22 smithi047 ceph-mon[36835]: pgmap v3232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:16:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:22 smithi029 ceph-mon[32610]: pgmap v3232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:16:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5204.473893 secs 2024-07-27T08:16:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5204.473893 secs 2024-07-27T08:16:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5204.473893 secs 2024-07-27T08:16:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:24 smithi077 ceph-mon[37153]: pgmap v3233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:16:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:24 smithi047 ceph-mon[36835]: pgmap v3233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:16:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:24 smithi029 ceph-mon[32610]: pgmap v3233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:16:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:26 smithi077 ceph-mon[37153]: pgmap v3234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:16:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:26 smithi047 ceph-mon[36835]: pgmap v3234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:16:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:26 smithi029 ceph-mon[32610]: pgmap v3234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:16:27.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:27.101+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:16:27.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:27.101+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:16:27.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:27.101+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:16:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:27.115+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:16:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:27.115+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:16:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:27.115+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:16:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:28 smithi077 ceph-mon[37153]: pgmap v3235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5209.473980 secs 2024-07-27T08:16:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:28 smithi047 ceph-mon[36835]: pgmap v3235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5209.473980 secs 2024-07-27T08:16:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:28 smithi029 ceph-mon[32610]: pgmap v3235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5209.473980 secs 2024-07-27T08:16:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:30 smithi077 ceph-mon[37153]: pgmap v3236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:16:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:30 smithi047 ceph-mon[36835]: pgmap v3236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:16:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:30 smithi029 ceph-mon[32610]: pgmap v3236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:16:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:32 smithi077 ceph-mon[37153]: pgmap v3237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:16:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:32 smithi047 ceph-mon[36835]: pgmap v3237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:16:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:32 smithi029 ceph-mon[32610]: pgmap v3237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:16:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5214.474116 secs 2024-07-27T08:16:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5214.474116 secs 2024-07-27T08:16:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5214.474116 secs 2024-07-27T08:16:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:34 smithi077 ceph-mon[37153]: pgmap v3238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:16:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:34 smithi047 ceph-mon[36835]: pgmap v3238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:16:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:34 smithi029 ceph-mon[32610]: pgmap v3238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:16:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:36 smithi077 ceph-mon[37153]: pgmap v3239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:16:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:36 smithi047 ceph-mon[36835]: pgmap v3239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:16:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:36 smithi029 ceph-mon[32610]: pgmap v3239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:16:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:38 smithi077 ceph-mon[37153]: pgmap v3240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5219.474184 secs 2024-07-27T08:16:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:38 smithi047 ceph-mon[36835]: pgmap v3240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5219.474184 secs 2024-07-27T08:16:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:38 smithi029 ceph-mon[32610]: pgmap v3240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5219.474184 secs 2024-07-27T08:16:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:40 smithi077 ceph-mon[37153]: pgmap v3241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:16:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:40 smithi047 ceph-mon[36835]: pgmap v3241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:16:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:40 smithi029 ceph-mon[32610]: pgmap v3241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:16:42.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:42.117+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:16:42.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:42.117+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:16:42.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:42.117+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:16:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:42.130+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:16:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:42.130+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:16:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:42.130+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:16:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:42 smithi029 ceph-mon[32610]: pgmap v3242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:16:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:42 smithi077 ceph-mon[37153]: pgmap v3242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:16:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:42 smithi047 ceph-mon[36835]: pgmap v3242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:16:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:43.420+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:16:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:43.420+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:16:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:43.420+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:16:43.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:43.420+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:16:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5224.474309 secs 2024-07-27T08:16:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5224.474309 secs 2024-07-27T08:16:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5224.474309 secs 2024-07-27T08:16:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:44 smithi077 ceph-mon[37153]: pgmap v3243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T08:16:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:44 smithi047 ceph-mon[36835]: pgmap v3243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T08:16:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:44 smithi029 ceph-mon[32610]: pgmap v3243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T08:16:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:46 smithi077 ceph-mon[37153]: pgmap v3244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:16:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:46 smithi047 ceph-mon[36835]: pgmap v3244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:16:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:46 smithi029 ceph-mon[32610]: pgmap v3244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:16:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:48 smithi029 ceph-mon[32610]: pgmap v3245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5229.474427 secs 2024-07-27T08:16:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:48 smithi077 ceph-mon[37153]: pgmap v3245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5229.474427 secs 2024-07-27T08:16:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:48 smithi047 ceph-mon[36835]: pgmap v3245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5229.474427 secs 2024-07-27T08:16:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:50 smithi077 ceph-mon[37153]: pgmap v3246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:16:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:50 smithi047 ceph-mon[36835]: pgmap v3246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:16:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:50 smithi029 ceph-mon[32610]: pgmap v3246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:16:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:52 smithi077 ceph-mon[37153]: pgmap v3247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:16:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:52 smithi047 ceph-mon[36835]: pgmap v3247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:16:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:52 smithi029 ceph-mon[32610]: pgmap v3247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:16:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5234.474558 secs 2024-07-27T08:16:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5234.474558 secs 2024-07-27T08:16:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5234.474558 secs 2024-07-27T08:16:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:54 smithi077 ceph-mon[37153]: pgmap v3248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:16:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:54 smithi047 ceph-mon[36835]: pgmap v3248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:16:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:54 smithi029 ceph-mon[32610]: pgmap v3248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:16:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:56 smithi047 ceph-mon[36835]: pgmap v3249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:16:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:56 smithi029 ceph-mon[32610]: pgmap v3249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:16:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:56 smithi077 ceph-mon[37153]: pgmap v3249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:16:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:57.132+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:16:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:57.132+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:16:57.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:57.132+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:16:57.146 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:57.145+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:16:57.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:57.145+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:16:57.147 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:16:57.145+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:16:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:16:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:57.618 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:16:57.618 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:57.618 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:16:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:58.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:58.422+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:16:58.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:58.422+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:16:58.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:58.422+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:16:58.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:16:58.422+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:16:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:58 smithi047 ceph-mon[36835]: pgmap v3250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5239.474649 secs 2024-07-27T08:16:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:58 smithi029 ceph-mon[32610]: pgmap v3250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5239.474649 secs 2024-07-27T08:16:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:58 smithi077 ceph-mon[37153]: pgmap v3250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:16:58.887 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5239.474649 secs 2024-07-27T08:16:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:16:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:16:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:16:59.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:16:59.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:16:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:16:59.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:16:59.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:17:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:00 smithi047 ceph-mon[36835]: pgmap v3251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:00.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:00 smithi047 ceph-mon[36835]: pgmap v3252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:17:00.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:00 smithi047 ceph-mon[36835]: pgmap v3253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:00 smithi029 ceph-mon[32610]: pgmap v3251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:00 smithi029 ceph-mon[32610]: pgmap v3252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:17:00.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:00 smithi029 ceph-mon[32610]: pgmap v3253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:00 smithi077 ceph-mon[37153]: pgmap v3251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:00 smithi077 ceph-mon[37153]: pgmap v3252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:17:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:00 smithi077 ceph-mon[37153]: pgmap v3253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:02 smithi047 ceph-mon[36835]: pgmap v3254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:17:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:02 smithi029 ceph-mon[32610]: pgmap v3254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:17:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:02 smithi077 ceph-mon[37153]: pgmap v3254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:17:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5244.474821 secs 2024-07-27T08:17:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5244.474821 secs 2024-07-27T08:17:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5244.474821 secs 2024-07-27T08:17:04.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:04 smithi047 ceph-mon[36835]: pgmap v3255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:17:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:04 smithi029 ceph-mon[32610]: pgmap v3255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:17:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:04 smithi077 ceph-mon[37153]: pgmap v3255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:17:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:06 smithi047 ceph-mon[36835]: pgmap v3256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:17:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:06 smithi029 ceph-mon[32610]: pgmap v3256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:17:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:06 smithi077 ceph-mon[37153]: pgmap v3256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:17:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5249.474890 secs 2024-07-27T08:17:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:08 smithi047 ceph-mon[36835]: pgmap v3257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:17:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5249.474890 secs 2024-07-27T08:17:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:08 smithi029 ceph-mon[32610]: pgmap v3257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:17:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5249.474890 secs 2024-07-27T08:17:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:08 smithi077 ceph-mon[37153]: pgmap v3257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:17:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:10 smithi047 ceph-mon[36835]: pgmap v3258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:17:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:10 smithi029 ceph-mon[32610]: pgmap v3258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:17:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:10 smithi077 ceph-mon[37153]: pgmap v3258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:17:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:17:12.147+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:17:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:17:12.147+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:17:12.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:17:12.147+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:17:12.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:17:12.161+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:17:12.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:17:12.161+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:17:12.162 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:17:12.161+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:17:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:12 smithi047 ceph-mon[36835]: pgmap v3259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:12 smithi029 ceph-mon[32610]: pgmap v3259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:12 smithi077 ceph-mon[37153]: pgmap v3259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5254.475015 secs 2024-07-27T08:17:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5254.475015 secs 2024-07-27T08:17:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5254.475015 secs 2024-07-27T08:17:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:14 smithi047 ceph-mon[36835]: pgmap v3260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:14 smithi029 ceph-mon[32610]: pgmap v3260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:14 smithi077 ceph-mon[37153]: pgmap v3260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:15 smithi047 ceph-mon[36835]: pgmap v3261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:15 smithi029 ceph-mon[32610]: pgmap v3261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:15 smithi077 ceph-mon[37153]: pgmap v3261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:17 smithi077 ceph-mon[37153]: pgmap v3262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:17 smithi047 ceph-mon[36835]: pgmap v3262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:17 smithi029 ceph-mon[32610]: pgmap v3262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:18.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5259.475099 secs 2024-07-27T08:17:18.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5259.475099 secs 2024-07-27T08:17:18.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5259.475099 secs 2024-07-27T08:17:19.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:19 smithi077 ceph-mon[37153]: pgmap v3263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:19 smithi047 ceph-mon[36835]: pgmap v3263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:19 smithi029 ceph-mon[32610]: pgmap v3263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:21 smithi077 ceph-mon[37153]: pgmap v3264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:21 smithi047 ceph-mon[36835]: pgmap v3264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:21 smithi029 ceph-mon[32610]: pgmap v3264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5264.475243 secs 2024-07-27T08:17:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5264.475243 secs 2024-07-27T08:17:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5264.475243 secs 2024-07-27T08:17:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:23 smithi077 ceph-mon[37153]: pgmap v3265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:23 smithi047 ceph-mon[36835]: pgmap v3265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:23 smithi029 ceph-mon[32610]: pgmap v3265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:25 smithi077 ceph-mon[37153]: pgmap v3266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:25 smithi047 ceph-mon[36835]: pgmap v3266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:25 smithi029 ceph-mon[32610]: pgmap v3266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:17:27.161+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:17:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:17:27.161+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:17:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:17:27.161+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:17:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:17:27.173+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:17:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:17:27.173+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:17:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:17:27.173+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:17:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:27 smithi077 ceph-mon[37153]: pgmap v3267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:27 smithi047 ceph-mon[36835]: pgmap v3267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:27 smithi029 ceph-mon[32610]: pgmap v3267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5269.475371 secs 2024-07-27T08:17:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5269.475371 secs 2024-07-27T08:17:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5269.475371 secs 2024-07-27T08:17:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:29 smithi077 ceph-mon[37153]: pgmap v3268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:29 smithi047 ceph-mon[36835]: pgmap v3268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:29 smithi029 ceph-mon[32610]: pgmap v3268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:31 smithi077 ceph-mon[37153]: pgmap v3269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:31 smithi047 ceph-mon[36835]: pgmap v3269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:31 smithi029 ceph-mon[32610]: pgmap v3269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5274.475446 secs 2024-07-27T08:17:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5274.475446 secs 2024-07-27T08:17:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5274.475446 secs 2024-07-27T08:17:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:33 smithi077 ceph-mon[37153]: pgmap v3270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:33 smithi047 ceph-mon[36835]: pgmap v3270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:33 smithi029 ceph-mon[32610]: pgmap v3270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:35 smithi077 ceph-mon[37153]: pgmap v3271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:35 smithi047 ceph-mon[36835]: pgmap v3271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:35 smithi029 ceph-mon[32610]: pgmap v3271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:37 smithi077 ceph-mon[37153]: pgmap v3272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:37 smithi047 ceph-mon[36835]: pgmap v3272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:37 smithi029 ceph-mon[32610]: pgmap v3272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5279.475591 secs 2024-07-27T08:17:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5279.475591 secs 2024-07-27T08:17:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5279.475591 secs 2024-07-27T08:17:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:39 smithi077 ceph-mon[37153]: pgmap v3273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:39 smithi047 ceph-mon[36835]: pgmap v3273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:39 smithi029 ceph-mon[32610]: pgmap v3273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:41 smithi077 ceph-mon[37153]: pgmap v3274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:41 smithi047 ceph-mon[36835]: pgmap v3274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:41 smithi029 ceph-mon[32610]: pgmap v3274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:17:42.176+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:17:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:17:42.176+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:17:42.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:17:42.176+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:17:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:17:42.181+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:17:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:17:42.181+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:17:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:17:42.181+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:17:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5284.475707 secs 2024-07-27T08:17:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5284.475707 secs 2024-07-27T08:17:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5284.475707 secs 2024-07-27T08:17:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:43 smithi077 ceph-mon[37153]: pgmap v3275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:43 smithi047 ceph-mon[36835]: pgmap v3275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:43 smithi029 ceph-mon[32610]: pgmap v3275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:45 smithi077 ceph-mon[37153]: pgmap v3276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:45 smithi047 ceph-mon[36835]: pgmap v3276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:45 smithi029 ceph-mon[32610]: pgmap v3276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:47 smithi077 ceph-mon[37153]: pgmap v3277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:47 smithi047 ceph-mon[36835]: pgmap v3277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:47 smithi029 ceph-mon[32610]: pgmap v3277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:48.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5289.475859 secs 2024-07-27T08:17:48.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5289.475859 secs 2024-07-27T08:17:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5289.475859 secs 2024-07-27T08:17:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:50 smithi029 ceph-mon[32610]: pgmap v3278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:50 smithi077 ceph-mon[37153]: pgmap v3278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:50 smithi047 ceph-mon[36835]: pgmap v3278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:51 smithi077 ceph-mon[37153]: pgmap v3279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:51 smithi047 ceph-mon[36835]: pgmap v3279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:51 smithi029 ceph-mon[32610]: pgmap v3279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:17:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5294.475983 secs 2024-07-27T08:17:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5294.475983 secs 2024-07-27T08:17:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5294.475983 secs 2024-07-27T08:17:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:53 smithi077 ceph-mon[37153]: pgmap v3280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:53 smithi047 ceph-mon[36835]: pgmap v3280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:53 smithi029 ceph-mon[32610]: pgmap v3280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:17:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:55 smithi077 ceph-mon[37153]: pgmap v3281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:55 smithi047 ceph-mon[36835]: pgmap v3281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:55 smithi029 ceph-mon[32610]: pgmap v3281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:17:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:17:57.185+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:17:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:17:57.185+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:17:57.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:17:57.185+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:17:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:17:57.197+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:17:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:17:57.197+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:17:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:17:57.197+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:17:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:57 smithi077 ceph-mon[37153]: pgmap v3282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:57 smithi047 ceph-mon[36835]: pgmap v3282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:57 smithi029 ceph-mon[32610]: pgmap v3282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:17:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5299.476077 secs 2024-07-27T08:17:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5299.476077 secs 2024-07-27T08:17:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5299.476077 secs 2024-07-27T08:17:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:59 smithi077 ceph-mon[37153]: pgmap v3283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:17:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:17:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:59 smithi047 ceph-mon[36835]: pgmap v3283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:17:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:17:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:59 smithi029 ceph-mon[32610]: pgmap v3283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:17:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:17:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:18:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:18:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:18:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:00 smithi077 ceph-mon[37153]: pgmap v3284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:18:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:00 smithi077 ceph-mon[37153]: pgmap v3285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:18:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:18:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:18:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:00 smithi047 ceph-mon[36835]: pgmap v3284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:18:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:00 smithi047 ceph-mon[36835]: pgmap v3285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:18:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:18:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:18:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:00 smithi029 ceph-mon[32610]: pgmap v3284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:18:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:00 smithi029 ceph-mon[32610]: pgmap v3285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:18:01.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:18:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5304.476207 secs 2024-07-27T08:18:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:03 smithi077 ceph-mon[37153]: pgmap v3286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T08:18:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5304.476207 secs 2024-07-27T08:18:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:03 smithi047 ceph-mon[36835]: pgmap v3286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T08:18:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5304.476207 secs 2024-07-27T08:18:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:03 smithi029 ceph-mon[32610]: pgmap v3286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T08:18:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:05 smithi077 ceph-mon[37153]: pgmap v3287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:18:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:05 smithi047 ceph-mon[36835]: pgmap v3287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:18:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:05 smithi029 ceph-mon[32610]: pgmap v3287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:18:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:07 smithi077 ceph-mon[37153]: pgmap v3288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:18:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:07 smithi047 ceph-mon[36835]: pgmap v3288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:18:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:07 smithi029 ceph-mon[32610]: pgmap v3288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:18:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5309.476345 secs 2024-07-27T08:18:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5309.476345 secs 2024-07-27T08:18:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5309.476345 secs 2024-07-27T08:18:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:09 smithi077 ceph-mon[37153]: pgmap v3289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:18:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:09 smithi047 ceph-mon[36835]: pgmap v3289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:18:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:09 smithi029 ceph-mon[32610]: pgmap v3289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:18:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:11 smithi077 ceph-mon[37153]: pgmap v3290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:18:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:11 smithi047 ceph-mon[36835]: pgmap v3290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:18:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:11 smithi029 ceph-mon[32610]: pgmap v3290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:18:12.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:18:12.198+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:18:12.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:18:12.198+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:18:12.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:18:12.198+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:18:12.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:18:12.202+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:18:12.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:18:12.202+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:18:12.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:18:12.202+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:18:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5314.476504 secs 2024-07-27T08:18:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:13 smithi077 ceph-mon[37153]: pgmap v3291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5314.476504 secs 2024-07-27T08:18:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:13 smithi047 ceph-mon[36835]: pgmap v3291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5314.476504 secs 2024-07-27T08:18:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:13 smithi029 ceph-mon[32610]: pgmap v3291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:15 smithi077 ceph-mon[37153]: pgmap v3292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:15 smithi047 ceph-mon[36835]: pgmap v3292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:15 smithi029 ceph-mon[32610]: pgmap v3292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:17 smithi077 ceph-mon[37153]: pgmap v3293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:17 smithi047 ceph-mon[36835]: pgmap v3293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:17 smithi029 ceph-mon[32610]: pgmap v3293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:18.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5319.476660 secs 2024-07-27T08:18:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5319.476660 secs 2024-07-27T08:18:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5319.476660 secs 2024-07-27T08:18:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:19 smithi077 ceph-mon[37153]: pgmap v3294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:19 smithi047 ceph-mon[36835]: pgmap v3294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:19 smithi029 ceph-mon[32610]: pgmap v3294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:21 smithi077 ceph-mon[37153]: pgmap v3295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:18:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:21 smithi047 ceph-mon[36835]: pgmap v3295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:18:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:21 smithi029 ceph-mon[32610]: pgmap v3295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:18:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5324.476823 secs 2024-07-27T08:18:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:23 smithi077 ceph-mon[37153]: pgmap v3296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5324.476823 secs 2024-07-27T08:18:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:23 smithi047 ceph-mon[36835]: pgmap v3296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5324.476823 secs 2024-07-27T08:18:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:23 smithi029 ceph-mon[32610]: pgmap v3296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:25 smithi077 ceph-mon[37153]: pgmap v3297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:25 smithi047 ceph-mon[36835]: pgmap v3297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:25 smithi029 ceph-mon[32610]: pgmap v3297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:18:27.209+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:18:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:18:27.209+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:18:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:18:27.209+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:18:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:18:27.211+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:18:27.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:18:27.212+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:18:27.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:18:27.212+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:18:27.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:27 smithi029 ceph-mon[32610]: pgmap v3298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:27 smithi077 ceph-mon[37153]: pgmap v3298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:27 smithi047 ceph-mon[36835]: pgmap v3298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5329.476924 secs 2024-07-27T08:18:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5329.476924 secs 2024-07-27T08:18:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5329.476924 secs 2024-07-27T08:18:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:29 smithi077 ceph-mon[37153]: pgmap v3299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:29 smithi047 ceph-mon[36835]: pgmap v3299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:29.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:29 smithi029 ceph-mon[32610]: pgmap v3299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:31 smithi077 ceph-mon[37153]: pgmap v3300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:18:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:31 smithi047 ceph-mon[36835]: pgmap v3300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:18:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:31 smithi029 ceph-mon[32610]: pgmap v3300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:18:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5334.477025 secs 2024-07-27T08:18:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:33 smithi077 ceph-mon[37153]: pgmap v3301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5334.477025 secs 2024-07-27T08:18:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:33 smithi047 ceph-mon[36835]: pgmap v3301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5334.477025 secs 2024-07-27T08:18:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:33 smithi029 ceph-mon[32610]: pgmap v3301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:35 smithi077 ceph-mon[37153]: pgmap v3302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:35 smithi047 ceph-mon[36835]: pgmap v3302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:35 smithi029 ceph-mon[32610]: pgmap v3302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:37 smithi077 ceph-mon[37153]: pgmap v3303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:37 smithi047 ceph-mon[36835]: pgmap v3303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:37 smithi029 ceph-mon[32610]: pgmap v3303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5339.477159 secs 2024-07-27T08:18:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5339.477159 secs 2024-07-27T08:18:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5339.477159 secs 2024-07-27T08:18:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:39 smithi077 ceph-mon[37153]: pgmap v3304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:39 smithi047 ceph-mon[36835]: pgmap v3304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:39 smithi029 ceph-mon[32610]: pgmap v3304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:41 smithi077 ceph-mon[37153]: pgmap v3305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:18:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:41 smithi047 ceph-mon[36835]: pgmap v3305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:18:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:41 smithi029 ceph-mon[32610]: pgmap v3305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:18:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:18:42.224+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:18:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:18:42.224+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:18:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:18:42.224+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:18:42.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:18:42.227+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:18:42.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:18:42.227+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:18:42.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:18:42.227+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:18:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5344.477255 secs 2024-07-27T08:18:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:43 smithi047 ceph-mon[36835]: pgmap v3306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5344.477255 secs 2024-07-27T08:18:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:43 smithi029 ceph-mon[32610]: pgmap v3306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5344.477255 secs 2024-07-27T08:18:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:43 smithi077 ceph-mon[37153]: pgmap v3306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:45 smithi047 ceph-mon[36835]: pgmap v3307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:45 smithi029 ceph-mon[32610]: pgmap v3307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:45 smithi077 ceph-mon[37153]: pgmap v3307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:47 smithi047 ceph-mon[36835]: pgmap v3308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:47 smithi029 ceph-mon[32610]: pgmap v3308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:47 smithi077 ceph-mon[37153]: pgmap v3308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5349.477372 secs 2024-07-27T08:18:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5349.477372 secs 2024-07-27T08:18:48.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5349.477372 secs 2024-07-27T08:18:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:49 smithi047 ceph-mon[36835]: pgmap v3309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:49 smithi029 ceph-mon[32610]: pgmap v3309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:49 smithi077 ceph-mon[37153]: pgmap v3309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:51 smithi047 ceph-mon[36835]: pgmap v3310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:18:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:51 smithi029 ceph-mon[32610]: pgmap v3310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:18:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:51 smithi077 ceph-mon[37153]: pgmap v3310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:18:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5354.477455 secs 2024-07-27T08:18:53.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:53 smithi047 ceph-mon[36835]: pgmap v3311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5354.477455 secs 2024-07-27T08:18:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:53 smithi029 ceph-mon[32610]: pgmap v3311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5354.477455 secs 2024-07-27T08:18:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:53 smithi077 ceph-mon[37153]: pgmap v3311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:55 smithi047 ceph-mon[36835]: pgmap v3312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:55 smithi029 ceph-mon[32610]: pgmap v3312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:55 smithi077 ceph-mon[37153]: pgmap v3312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:18:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:18:57.238+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:18:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:18:57.238+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:18:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:18:57.238+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:18:57.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:18:57.242+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:18:57.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:18:57.242+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:18:57.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:18:57.242+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:18:57.515 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:57 smithi029 ceph-mon[32610]: pgmap v3313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:57 smithi047 ceph-mon[36835]: pgmap v3313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:57 smithi077 ceph-mon[37153]: pgmap v3313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:18:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5359.477518 secs 2024-07-27T08:18:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5359.477518 secs 2024-07-27T08:18:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5359.477518 secs 2024-07-27T08:18:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:18:59 smithi047 ceph-mon[36835]: pgmap v3314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:18:59 smithi029 ceph-mon[32610]: pgmap v3314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:18:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:18:59 smithi077 ceph-mon[37153]: pgmap v3314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:19:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:19:00.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:19:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:01 smithi047 ceph-mon[36835]: pgmap v3315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:01.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:19:01.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:19:01.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:19:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:01 smithi029 ceph-mon[32610]: pgmap v3315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:19:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:19:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:19:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:01 smithi077 ceph-mon[37153]: pgmap v3315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:19:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:19:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:19:02.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:02 smithi047 ceph-mon[36835]: pgmap v3316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:19:02.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:02 smithi047 ceph-mon[36835]: pgmap v3317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:19:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:02 smithi029 ceph-mon[32610]: pgmap v3316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:19:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:02 smithi029 ceph-mon[32610]: pgmap v3317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:19:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:02 smithi077 ceph-mon[37153]: pgmap v3316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:19:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:02 smithi077 ceph-mon[37153]: pgmap v3317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:19:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5364.477636 secs 2024-07-27T08:19:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5364.477636 secs 2024-07-27T08:19:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5364.477636 secs 2024-07-27T08:19:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:04 smithi047 ceph-mon[36835]: pgmap v3318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:19:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:04 smithi029 ceph-mon[32610]: pgmap v3318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:19:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:04 smithi077 ceph-mon[37153]: pgmap v3318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:19:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:06 smithi047 ceph-mon[36835]: pgmap v3319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:19:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:06 smithi029 ceph-mon[32610]: pgmap v3319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:19:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:06 smithi077 ceph-mon[37153]: pgmap v3319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:19:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5369.477780 secs 2024-07-27T08:19:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:08 smithi047 ceph-mon[36835]: pgmap v3320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:19:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5369.477780 secs 2024-07-27T08:19:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:08 smithi029 ceph-mon[32610]: pgmap v3320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:19:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5369.477780 secs 2024-07-27T08:19:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:08 smithi077 ceph-mon[37153]: pgmap v3320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:19:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:10 smithi047 ceph-mon[36835]: pgmap v3321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:19:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:10 smithi029 ceph-mon[32610]: pgmap v3321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:19:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:10 smithi077 ceph-mon[37153]: pgmap v3321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:19:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:12.251+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:19:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:12.251+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:19:12.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:12.251+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:19:12.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:12.258+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:19:12.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:12.258+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:19:12.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:12.258+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:19:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:12 smithi047 ceph-mon[36835]: pgmap v3322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:19:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:12 smithi029 ceph-mon[32610]: pgmap v3322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:19:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:12 smithi077 ceph-mon[37153]: pgmap v3322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:19:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5374.477885 secs 2024-07-27T08:19:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5374.477885 secs 2024-07-27T08:19:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5374.477885 secs 2024-07-27T08:19:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:14 smithi047 ceph-mon[36835]: pgmap v3323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:14 smithi029 ceph-mon[32610]: pgmap v3323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:14 smithi077 ceph-mon[37153]: pgmap v3323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:16 smithi047 ceph-mon[36835]: pgmap v3324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:16 smithi029 ceph-mon[32610]: pgmap v3324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:16 smithi077 ceph-mon[37153]: pgmap v3324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5379.478015 secs 2024-07-27T08:19:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:18 smithi047 ceph-mon[36835]: pgmap v3325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5379.478015 secs 2024-07-27T08:19:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:18 smithi029 ceph-mon[32610]: pgmap v3325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5379.478015 secs 2024-07-27T08:19:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:18 smithi077 ceph-mon[37153]: pgmap v3325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:20 smithi047 ceph-mon[36835]: pgmap v3326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:20 smithi029 ceph-mon[32610]: pgmap v3326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:20 smithi077 ceph-mon[37153]: pgmap v3326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:21 smithi077 ceph-mon[37153]: pgmap v3327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:21 smithi047 ceph-mon[36835]: pgmap v3327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:21 smithi029 ceph-mon[32610]: pgmap v3327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5384.478111 secs 2024-07-27T08:19:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5384.478111 secs 2024-07-27T08:19:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5384.478111 secs 2024-07-27T08:19:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:23 smithi077 ceph-mon[37153]: pgmap v3328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:23 smithi047 ceph-mon[36835]: pgmap v3328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:23 smithi029 ceph-mon[32610]: pgmap v3328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:25 smithi077 ceph-mon[37153]: pgmap v3329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:25 smithi047 ceph-mon[36835]: pgmap v3329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:25 smithi029 ceph-mon[32610]: pgmap v3329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:27.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:27.253+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:19:27.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:27.253+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:19:27.255 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:27.253+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:19:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:27.269+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:19:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:27.269+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:19:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:27.269+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:19:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5389.478253 secs 2024-07-27T08:19:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:27 smithi077 ceph-mon[37153]: pgmap v3330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5389.478253 secs 2024-07-27T08:19:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:27 smithi047 ceph-mon[36835]: pgmap v3330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5389.478253 secs 2024-07-27T08:19:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:27 smithi029 ceph-mon[32610]: pgmap v3330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:29 smithi077 ceph-mon[37153]: pgmap v3331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:29 smithi047 ceph-mon[36835]: pgmap v3331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:29 smithi029 ceph-mon[32610]: pgmap v3331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:31 smithi077 ceph-mon[37153]: pgmap v3332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:31 smithi047 ceph-mon[36835]: pgmap v3332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:31 smithi029 ceph-mon[32610]: pgmap v3332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5394.478376 secs 2024-07-27T08:19:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5394.478376 secs 2024-07-27T08:19:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5394.478376 secs 2024-07-27T08:19:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:33 smithi077 ceph-mon[37153]: pgmap v3333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:33 smithi047 ceph-mon[36835]: pgmap v3333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:33 smithi029 ceph-mon[32610]: pgmap v3333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:35 smithi077 ceph-mon[37153]: pgmap v3334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:35 smithi047 ceph-mon[36835]: pgmap v3334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:35 smithi029 ceph-mon[32610]: pgmap v3334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5399.478534 secs 2024-07-27T08:19:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:37 smithi047 ceph-mon[36835]: pgmap v3335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5399.478534 secs 2024-07-27T08:19:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:37 smithi029 ceph-mon[32610]: pgmap v3335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5399.478534 secs 2024-07-27T08:19:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:37 smithi077 ceph-mon[37153]: pgmap v3335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:39 smithi047 ceph-mon[36835]: pgmap v3336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:39 smithi029 ceph-mon[32610]: pgmap v3336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:39 smithi077 ceph-mon[37153]: pgmap v3336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:41 smithi047 ceph-mon[36835]: pgmap v3337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:41 smithi029 ceph-mon[32610]: pgmap v3337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.261+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:19:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.261+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:19:42.263 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.261+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:19:42.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:42.284+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:19:42.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:42.284+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:19:42.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:42.284+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:19:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:41 smithi077 ceph-mon[37153]: pgmap v3337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.637+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-27T08:19:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.637+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T08:19:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.637+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f733c00c750 tx=0x7f733c03f000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T08:19:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.637+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f733c00c750 tx=0x7f733c03f000 comp rx=0 tx=0).stop 2024-07-27T08:19:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.637+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T08:19:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.637+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T08:19:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.637+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T08:19:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.637+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T08:19:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.637+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T08:19:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:42.638+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4e80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f733c0045a0 tx=0x7f733c073000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T08:19:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5404.478658 secs 2024-07-27T08:19:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5404.478658 secs 2024-07-27T08:19:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5404.478658 secs 2024-07-27T08:19:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:43.447+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:19:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:43.447+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:19:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:43.447+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:19:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:43.447+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:19:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:43 smithi047 ceph-mon[36835]: pgmap v3338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:43 smithi029 ceph-mon[32610]: pgmap v3338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:43 smithi077 ceph-mon[37153]: pgmap v3338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:45 smithi047 ceph-mon[36835]: pgmap v3339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:45 smithi029 ceph-mon[32610]: pgmap v3339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:45 smithi077 ceph-mon[37153]: pgmap v3339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:48.157 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5409.478780 secs 2024-07-27T08:19:48.158 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:47 smithi077 ceph-mon[37153]: pgmap v3340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5409.478780 secs 2024-07-27T08:19:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:47 smithi047 ceph-mon[36835]: pgmap v3340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5409.478780 secs 2024-07-27T08:19:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:47 smithi029 ceph-mon[32610]: pgmap v3340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:49 smithi047 ceph-mon[36835]: pgmap v3341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:49 smithi029 ceph-mon[32610]: pgmap v3341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:49 smithi077 ceph-mon[37153]: pgmap v3341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:19:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:51 smithi047 ceph-mon[36835]: pgmap v3342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:51 smithi029 ceph-mon[32610]: pgmap v3342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:51 smithi077 ceph-mon[37153]: pgmap v3342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:19:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5414.478929 secs 2024-07-27T08:19:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5414.478929 secs 2024-07-27T08:19:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5414.478929 secs 2024-07-27T08:19:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:53 smithi047 ceph-mon[36835]: pgmap v3343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:53 smithi029 ceph-mon[32610]: pgmap v3343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:53 smithi077 ceph-mon[37153]: pgmap v3343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:19:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:56 smithi077 ceph-mon[37153]: pgmap v3344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:56 smithi047 ceph-mon[36835]: pgmap v3344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:56 smithi029 ceph-mon[32610]: pgmap v3344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:19:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:57.277+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:19:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:57.277+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:19:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:19:57.277+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:19:57.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.293+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:19:57.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.293+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:19:57.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.293+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:19:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.653+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T08:19:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.653+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T08:19:57.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.653+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7eff40009780 tx=0x7eff4004f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T08:19:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.653+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7eff40009780 tx=0x7eff4004f040 comp rx=0 tx=0).stop 2024-07-27T08:19:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.653+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T08:19:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.653+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T08:19:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.653+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T08:19:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.653+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T08:19:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.653+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T08:19:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:57.654+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b4ef0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7eff400048c0 tx=0x7eff40038040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T08:19:58.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:58.449+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:19:58.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:58.449+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:19:58.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:58.449+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:19:58.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:19:58.449+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:19:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5419.479018 secs 2024-07-27T08:19:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:19:58 smithi077 ceph-mon[37153]: pgmap v3345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5419.479018 secs 2024-07-27T08:19:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:19:58 smithi047 ceph-mon[36835]: pgmap v3345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:19:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5419.479018 secs 2024-07-27T08:19:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:19:58 smithi029 ceph-mon[32610]: pgmap v3345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:00 smithi077 ceph-mon[37153]: pgmap v3346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:20:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:00 smithi047 ceph-mon[36835]: pgmap v3346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:20:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:00 smithi029 ceph-mon[32610]: pgmap v3346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:20:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:20:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:20:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:20:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:02 smithi077 ceph-mon[37153]: pgmap v3347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:20:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:20:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:02 smithi077 ceph-mon[37153]: pgmap v3348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:20:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:02 smithi077 ceph-mon[37153]: pgmap v3349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:20:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:20:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:02 smithi047 ceph-mon[36835]: pgmap v3347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:20:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:20:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:02 smithi047 ceph-mon[36835]: pgmap v3348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:20:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:02 smithi047 ceph-mon[36835]: pgmap v3349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:20:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:20:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:02 smithi029 ceph-mon[32610]: pgmap v3347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:20:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:20:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:02 smithi029 ceph-mon[32610]: pgmap v3348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:20:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:02 smithi029 ceph-mon[32610]: pgmap v3349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:20:02.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:20:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5424.479100 secs 2024-07-27T08:20:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5424.479100 secs 2024-07-27T08:20:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5424.479100 secs 2024-07-27T08:20:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:04 smithi077 ceph-mon[37153]: pgmap v3350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:20:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:04 smithi047 ceph-mon[36835]: pgmap v3350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:20:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:04 smithi029 ceph-mon[32610]: pgmap v3350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:20:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:06 smithi077 ceph-mon[37153]: pgmap v3351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:20:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:06 smithi047 ceph-mon[36835]: pgmap v3351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:20:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:06 smithi029 ceph-mon[32610]: pgmap v3351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:20:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5429.479173 secs 2024-07-27T08:20:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:08 smithi077 ceph-mon[37153]: pgmap v3352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:20:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5429.479173 secs 2024-07-27T08:20:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:08 smithi047 ceph-mon[36835]: pgmap v3352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:20:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5429.479173 secs 2024-07-27T08:20:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:08 smithi029 ceph-mon[32610]: pgmap v3352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:20:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:10 smithi077 ceph-mon[37153]: pgmap v3353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:20:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:10 smithi047 ceph-mon[36835]: pgmap v3353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:20:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:10 smithi029 ceph-mon[32610]: pgmap v3353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:20:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:12.293+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:20:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:12.293+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:20:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:12.293+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:20:12.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:12.303+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:20:12.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:12.303+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:20:12.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:12.303+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:20:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:12 smithi077 ceph-mon[37153]: pgmap v3354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:20:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:12 smithi047 ceph-mon[36835]: pgmap v3354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:20:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:12 smithi029 ceph-mon[32610]: pgmap v3354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:20:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5434.479298 secs 2024-07-27T08:20:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5434.479298 secs 2024-07-27T08:20:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5434.479298 secs 2024-07-27T08:20:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:14 smithi047 ceph-mon[36835]: pgmap v3355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:14 smithi029 ceph-mon[32610]: pgmap v3355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:14 smithi077 ceph-mon[37153]: pgmap v3355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:16 smithi047 ceph-mon[36835]: pgmap v3356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:16 smithi029 ceph-mon[32610]: pgmap v3356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:16 smithi077 ceph-mon[37153]: pgmap v3356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:18.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5439.479441 secs 2024-07-27T08:20:18.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:18 smithi077 ceph-mon[37153]: pgmap v3357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5439.479441 secs 2024-07-27T08:20:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:18 smithi047 ceph-mon[36835]: pgmap v3357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5439.479441 secs 2024-07-27T08:20:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:18 smithi029 ceph-mon[32610]: pgmap v3357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:20 smithi047 ceph-mon[36835]: pgmap v3358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:20:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:20 smithi029 ceph-mon[32610]: pgmap v3358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:20:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:20 smithi077 ceph-mon[37153]: pgmap v3358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:20:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:22 smithi047 ceph-mon[36835]: pgmap v3359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:22 smithi029 ceph-mon[32610]: pgmap v3359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:22 smithi077 ceph-mon[37153]: pgmap v3359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5444.479565 secs 2024-07-27T08:20:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5444.479565 secs 2024-07-27T08:20:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5444.479565 secs 2024-07-27T08:20:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:24 smithi047 ceph-mon[36835]: pgmap v3360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:24 smithi029 ceph-mon[32610]: pgmap v3360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:24 smithi077 ceph-mon[37153]: pgmap v3360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:26 smithi047 ceph-mon[36835]: pgmap v3361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:20:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:26 smithi029 ceph-mon[32610]: pgmap v3361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:20:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:26 smithi077 ceph-mon[37153]: pgmap v3361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:20:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:27.300+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:20:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:27.300+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:20:27.302 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:27.300+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:20:27.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:27.317+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:20:27.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:27.317+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:20:27.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:27.317+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:20:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:28.454+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:20:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:28.454+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:20:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:28.454+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:20:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:28.454+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:20:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5449.479718 secs 2024-07-27T08:20:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:28 smithi047 ceph-mon[36835]: pgmap v3362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:28.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5449.479718 secs 2024-07-27T08:20:28.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:28 smithi029 ceph-mon[32610]: pgmap v3362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5449.479718 secs 2024-07-27T08:20:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:28 smithi077 ceph-mon[37153]: pgmap v3362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:30 smithi047 ceph-mon[36835]: pgmap v3363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:20:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:30 smithi029 ceph-mon[32610]: pgmap v3363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:20:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:30 smithi077 ceph-mon[37153]: pgmap v3363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:20:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:32 smithi047 ceph-mon[36835]: pgmap v3364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:32 smithi029 ceph-mon[32610]: pgmap v3364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:32 smithi077 ceph-mon[37153]: pgmap v3364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5454.479842 secs 2024-07-27T08:20:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5454.479842 secs 2024-07-27T08:20:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5454.479842 secs 2024-07-27T08:20:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:34 smithi047 ceph-mon[36835]: pgmap v3365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:34 smithi029 ceph-mon[32610]: pgmap v3365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:34 smithi077 ceph-mon[37153]: pgmap v3365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:36 smithi047 ceph-mon[36835]: pgmap v3366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:36 smithi029 ceph-mon[32610]: pgmap v3366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:36 smithi077 ceph-mon[37153]: pgmap v3366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5459.479971 secs 2024-07-27T08:20:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:38 smithi047 ceph-mon[36835]: pgmap v3367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5459.479971 secs 2024-07-27T08:20:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:38 smithi029 ceph-mon[32610]: pgmap v3367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5459.479971 secs 2024-07-27T08:20:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:38 smithi077 ceph-mon[37153]: pgmap v3367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:40 smithi047 ceph-mon[36835]: pgmap v3368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:20:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:40 smithi029 ceph-mon[32610]: pgmap v3368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:20:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:40 smithi077 ceph-mon[37153]: pgmap v3368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:20:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:42.315+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:20:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:42.315+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:20:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:42.315+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:20:42.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:42.328+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:20:42.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:42.328+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:20:42.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:42.328+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:20:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:42 smithi047 ceph-mon[36835]: pgmap v3369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:42 smithi029 ceph-mon[32610]: pgmap v3369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:42 smithi077 ceph-mon[37153]: pgmap v3369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:43.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:43.456+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:20:43.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:43.456+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:20:43.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:43.456+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:20:43.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:43.456+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:20:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5464.480100 secs 2024-07-27T08:20:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5464.480100 secs 2024-07-27T08:20:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5464.480100 secs 2024-07-27T08:20:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:44 smithi047 ceph-mon[36835]: pgmap v3370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:44 smithi029 ceph-mon[32610]: pgmap v3370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:44 smithi077 ceph-mon[37153]: pgmap v3370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:46 smithi047 ceph-mon[36835]: pgmap v3371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:46 smithi029 ceph-mon[32610]: pgmap v3371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:46 smithi077 ceph-mon[37153]: pgmap v3371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:48.677 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5469.480237 secs 2024-07-27T08:20:48.678 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:48 smithi077 ceph-mon[37153]: pgmap v3372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5469.480237 secs 2024-07-27T08:20:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:48 smithi047 ceph-mon[36835]: pgmap v3372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5469.480237 secs 2024-07-27T08:20:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:48 smithi029 ceph-mon[32610]: pgmap v3372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:50 smithi047 ceph-mon[36835]: pgmap v3373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:20:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:50 smithi029 ceph-mon[32610]: pgmap v3373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:20:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:50 smithi077 ceph-mon[37153]: pgmap v3373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:20:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:52 smithi047 ceph-mon[36835]: pgmap v3374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:52 smithi029 ceph-mon[32610]: pgmap v3374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:52 smithi077 ceph-mon[37153]: pgmap v3374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5474.480311 secs 2024-07-27T08:20:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5474.480311 secs 2024-07-27T08:20:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5474.480311 secs 2024-07-27T08:20:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:54 smithi047 ceph-mon[36835]: pgmap v3375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:54 smithi029 ceph-mon[32610]: pgmap v3375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:54 smithi077 ceph-mon[37153]: pgmap v3375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:20:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:56 smithi047 ceph-mon[36835]: pgmap v3376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:56 smithi029 ceph-mon[32610]: pgmap v3376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:56 smithi077 ceph-mon[37153]: pgmap v3376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:20:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:57.331+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:20:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:57.331+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:20:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:20:57.331+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:20:57.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:57.331+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:20:57.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:57.331+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:20:57.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:20:57.331+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:20:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5479.480437 secs 2024-07-27T08:20:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:20:58 smithi047 ceph-mon[36835]: pgmap v3377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5479.480437 secs 2024-07-27T08:20:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:20:58 smithi029 ceph-mon[32610]: pgmap v3377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:20:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5479.480437 secs 2024-07-27T08:20:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:20:58 smithi077 ceph-mon[37153]: pgmap v3377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:00 smithi047 ceph-mon[36835]: pgmap v3378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:21:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:00 smithi029 ceph-mon[32610]: pgmap v3378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:21:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:00 smithi077 ceph-mon[37153]: pgmap v3378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:21:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:21:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:21:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:21:02.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:02 smithi029 ceph-mon[32610]: pgmap v3379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:02.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:21:02.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:21:02.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:21:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:02 smithi077 ceph-mon[37153]: pgmap v3379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:21:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:21:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:21:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:02 smithi047 ceph-mon[36835]: pgmap v3379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:21:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:21:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:21:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5484.480528 secs 2024-07-27T08:21:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:03 smithi029 ceph-mon[32610]: pgmap v3380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:21:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:03 smithi029 ceph-mon[32610]: pgmap v3381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:21:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5484.480528 secs 2024-07-27T08:21:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:03 smithi077 ceph-mon[37153]: pgmap v3380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:21:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:03 smithi077 ceph-mon[37153]: pgmap v3381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:21:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5484.480528 secs 2024-07-27T08:21:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:03 smithi047 ceph-mon[36835]: pgmap v3380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:21:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:03 smithi047 ceph-mon[36835]: pgmap v3381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:21:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:05 smithi077 ceph-mon[37153]: pgmap v3382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:21:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:05 smithi047 ceph-mon[36835]: pgmap v3382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:21:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:05 smithi029 ceph-mon[32610]: pgmap v3382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:21:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:06 smithi077 ceph-mon[37153]: pgmap v3383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:21:06.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:06 smithi047 ceph-mon[36835]: pgmap v3383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:21:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:06 smithi029 ceph-mon[32610]: pgmap v3383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:21:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5489.480673 secs 2024-07-27T08:21:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5489.480673 secs 2024-07-27T08:21:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5489.480673 secs 2024-07-27T08:21:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:08 smithi077 ceph-mon[37153]: pgmap v3384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:21:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:08 smithi047 ceph-mon[36835]: pgmap v3384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:21:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:08 smithi029 ceph-mon[32610]: pgmap v3384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:21:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:10 smithi029 ceph-mon[32610]: pgmap v3385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s 2024-07-27T08:21:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:10 smithi077 ceph-mon[37153]: pgmap v3385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s 2024-07-27T08:21:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:10 smithi047 ceph-mon[36835]: pgmap v3385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s 2024-07-27T08:21:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:21:12.338+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:21:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:21:12.338+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:21:12.339 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:21:12.338+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:21:12.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:21:12.338+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:21:12.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:21:12.338+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:21:12.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:21:12.338+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:21:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5494.480811 secs 2024-07-27T08:21:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:12 smithi029 ceph-mon[32610]: pgmap v3386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:21:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5494.480811 secs 2024-07-27T08:21:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:12 smithi077 ceph-mon[37153]: pgmap v3386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:21:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5494.480811 secs 2024-07-27T08:21:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:12 smithi047 ceph-mon[36835]: pgmap v3386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:21:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:14 smithi029 ceph-mon[32610]: pgmap v3387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:14 smithi077 ceph-mon[37153]: pgmap v3387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:14 smithi047 ceph-mon[36835]: pgmap v3387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:16 smithi029 ceph-mon[32610]: pgmap v3388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:16 smithi077 ceph-mon[37153]: pgmap v3388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:16 smithi047 ceph-mon[36835]: pgmap v3388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5499.480930 secs 2024-07-27T08:21:18.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5499.480930 secs 2024-07-27T08:21:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5499.480930 secs 2024-07-27T08:21:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:18 smithi077 ceph-mon[37153]: pgmap v3389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:18 smithi047 ceph-mon[36835]: pgmap v3389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:18 smithi029 ceph-mon[32610]: pgmap v3389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:21.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:20 smithi047 ceph-mon[36835]: pgmap v3390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:21:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:20 smithi077 ceph-mon[37153]: pgmap v3390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:21:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:20 smithi029 ceph-mon[32610]: pgmap v3390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:21:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5504.481001 secs 2024-07-27T08:21:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:22 smithi077 ceph-mon[37153]: pgmap v3391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5504.481001 secs 2024-07-27T08:21:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:22 smithi047 ceph-mon[36835]: pgmap v3391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5504.481001 secs 2024-07-27T08:21:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:22 smithi029 ceph-mon[32610]: pgmap v3391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:24 smithi077 ceph-mon[37153]: pgmap v3392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:24 smithi047 ceph-mon[36835]: pgmap v3392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:24 smithi029 ceph-mon[32610]: pgmap v3392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:27.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:26 smithi029 ceph-mon[32610]: pgmap v3393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:27.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:21:27.343+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:21:27.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:21:27.343+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:21:27.345 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:21:27.343+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:21:27.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:21:27.352+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:21:27.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:21:27.352+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:21:27.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:21:27.352+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:21:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:26 smithi077 ceph-mon[37153]: pgmap v3393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:26 smithi047 ceph-mon[36835]: pgmap v3393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5509.481080 secs 2024-07-27T08:21:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5509.481080 secs 2024-07-27T08:21:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5509.481080 secs 2024-07-27T08:21:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:28 smithi077 ceph-mon[37153]: pgmap v3394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:28 smithi047 ceph-mon[36835]: pgmap v3394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:28 smithi029 ceph-mon[32610]: pgmap v3394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:30 smithi077 ceph-mon[37153]: pgmap v3395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:21:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:30 smithi047 ceph-mon[36835]: pgmap v3395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:21:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:30 smithi029 ceph-mon[32610]: pgmap v3395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:21:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5514.481159 secs 2024-07-27T08:21:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:32 smithi077 ceph-mon[37153]: pgmap v3396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5514.481159 secs 2024-07-27T08:21:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:32 smithi047 ceph-mon[36835]: pgmap v3396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5514.481159 secs 2024-07-27T08:21:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:32 smithi029 ceph-mon[32610]: pgmap v3396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:34 smithi077 ceph-mon[37153]: pgmap v3397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:34 smithi047 ceph-mon[36835]: pgmap v3397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:34 smithi029 ceph-mon[32610]: pgmap v3397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:37 smithi077 ceph-mon[37153]: pgmap v3398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:37 smithi047 ceph-mon[36835]: pgmap v3398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:37 smithi029 ceph-mon[32610]: pgmap v3398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5519.481286 secs 2024-07-27T08:21:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5519.481286 secs 2024-07-27T08:21:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5519.481286 secs 2024-07-27T08:21:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:39 smithi077 ceph-mon[37153]: pgmap v3399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:39 smithi047 ceph-mon[36835]: pgmap v3399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:39 smithi029 ceph-mon[32610]: pgmap v3399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:41 smithi077 ceph-mon[37153]: pgmap v3400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:21:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:41 smithi047 ceph-mon[36835]: pgmap v3400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:21:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:41 smithi029 ceph-mon[32610]: pgmap v3400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:21:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:21:42.346+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:21:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:21:42.346+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:21:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:21:42.346+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:21:42.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:21:42.368+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:21:42.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:21:42.368+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:21:42.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:21:42.368+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:21:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5524.481415 secs 2024-07-27T08:21:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:43 smithi077 ceph-mon[37153]: pgmap v3401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5524.481415 secs 2024-07-27T08:21:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:43 smithi047 ceph-mon[36835]: pgmap v3401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5524.481415 secs 2024-07-27T08:21:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:43 smithi029 ceph-mon[32610]: pgmap v3401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:45 smithi077 ceph-mon[37153]: pgmap v3402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:45 smithi047 ceph-mon[36835]: pgmap v3402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:45 smithi029 ceph-mon[32610]: pgmap v3402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:47 smithi077 ceph-mon[37153]: pgmap v3403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:47 smithi047 ceph-mon[36835]: pgmap v3403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:47 smithi029 ceph-mon[32610]: pgmap v3403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5529.481573 secs 2024-07-27T08:21:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5529.481573 secs 2024-07-27T08:21:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5529.481573 secs 2024-07-27T08:21:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:49 smithi077 ceph-mon[37153]: pgmap v3404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:49 smithi047 ceph-mon[36835]: pgmap v3404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:49 smithi029 ceph-mon[32610]: pgmap v3404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:51 smithi077 ceph-mon[37153]: pgmap v3405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:21:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:51 smithi047 ceph-mon[36835]: pgmap v3405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:21:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:51 smithi029 ceph-mon[32610]: pgmap v3405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:21:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5534.481694 secs 2024-07-27T08:21:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:53 smithi077 ceph-mon[37153]: pgmap v3406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5534.481694 secs 2024-07-27T08:21:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:53 smithi047 ceph-mon[36835]: pgmap v3406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5534.481694 secs 2024-07-27T08:21:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:53 smithi029 ceph-mon[32610]: pgmap v3406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:21:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:55 smithi077 ceph-mon[37153]: pgmap v3407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:55 smithi047 ceph-mon[36835]: pgmap v3407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:55 smithi029 ceph-mon[32610]: pgmap v3407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:21:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:57 smithi077 ceph-mon[37153]: pgmap v3408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:21:57.361+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:21:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:21:57.361+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:21:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:21:57.361+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:21:57.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:21:57.383+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:21:57.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:21:57.383+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:21:57.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:21:57.383+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:21:57.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:57 smithi029 ceph-mon[32610]: pgmap v3408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:57 smithi047 ceph-mon[36835]: pgmap v3408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:21:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5539.481841 secs 2024-07-27T08:21:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5539.481841 secs 2024-07-27T08:21:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5539.481841 secs 2024-07-27T08:21:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:21:59 smithi077 ceph-mon[37153]: pgmap v3409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:21:59 smithi047 ceph-mon[36835]: pgmap v3409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:21:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:21:59 smithi029 ceph-mon[32610]: pgmap v3409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:22:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:01 smithi077 ceph-mon[37153]: pgmap v3410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:01 smithi047 ceph-mon[36835]: pgmap v3410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:01 smithi029 ceph-mon[32610]: pgmap v3410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:22:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:22:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:22:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5544.481962 secs 2024-07-27T08:22:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:03 smithi077 ceph-mon[37153]: pgmap v3411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:22:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:22:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:03 smithi077 ceph-mon[37153]: pgmap v3412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T08:22:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:03 smithi077 ceph-mon[37153]: pgmap v3413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:22:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:22:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5544.481962 secs 2024-07-27T08:22:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:03 smithi047 ceph-mon[36835]: pgmap v3411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:22:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:22:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:03 smithi047 ceph-mon[36835]: pgmap v3412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T08:22:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:03 smithi047 ceph-mon[36835]: pgmap v3413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:22:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:22:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5544.481962 secs 2024-07-27T08:22:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:03 smithi029 ceph-mon[32610]: pgmap v3411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:22:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:22:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:03 smithi029 ceph-mon[32610]: pgmap v3412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T08:22:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:03 smithi029 ceph-mon[32610]: pgmap v3413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:22:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:22:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:04 smithi077 ceph-mon[37153]: pgmap v3414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:22:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:04 smithi047 ceph-mon[36835]: pgmap v3414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:22:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:04 smithi029 ceph-mon[32610]: pgmap v3414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:22:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:06 smithi077 ceph-mon[37153]: pgmap v3415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:22:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:06 smithi047 ceph-mon[36835]: pgmap v3415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:22:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:06 smithi029 ceph-mon[32610]: pgmap v3415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:22:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5549.482086 secs 2024-07-27T08:22:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5549.482086 secs 2024-07-27T08:22:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5549.482086 secs 2024-07-27T08:22:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:08 smithi077 ceph-mon[37153]: pgmap v3416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:22:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:08 smithi047 ceph-mon[36835]: pgmap v3416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:22:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:08 smithi029 ceph-mon[32610]: pgmap v3416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:22:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:10 smithi077 ceph-mon[37153]: pgmap v3417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:22:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:10 smithi047 ceph-mon[36835]: pgmap v3417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:22:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:10 smithi029 ceph-mon[32610]: pgmap v3417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:22:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:12.377+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:22:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:12.377+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:22:12.378 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:12.377+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:22:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:12.394+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:22:12.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:12.394+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:22:12.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:12.394+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:22:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:12 smithi029 ceph-mon[32610]: pgmap v3418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:22:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:12 smithi077 ceph-mon[37153]: pgmap v3418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:22:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:12 smithi047 ceph-mon[36835]: pgmap v3418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:22:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5554.482209 secs 2024-07-27T08:22:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5554.482209 secs 2024-07-27T08:22:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5554.482209 secs 2024-07-27T08:22:14.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:14 smithi029 ceph-mon[32610]: pgmap v3419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:14 smithi077 ceph-mon[37153]: pgmap v3419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:14 smithi047 ceph-mon[36835]: pgmap v3419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:16 smithi077 ceph-mon[37153]: pgmap v3420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:16 smithi047 ceph-mon[36835]: pgmap v3420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:16 smithi029 ceph-mon[32610]: pgmap v3420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5559.482360 secs 2024-07-27T08:22:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5559.482360 secs 2024-07-27T08:22:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5559.482360 secs 2024-07-27T08:22:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:18 smithi077 ceph-mon[37153]: pgmap v3421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:18 smithi047 ceph-mon[36835]: pgmap v3421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:18 smithi029 ceph-mon[32610]: pgmap v3421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:20 smithi077 ceph-mon[37153]: pgmap v3422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:22:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:20 smithi047 ceph-mon[36835]: pgmap v3422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:22:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:20 smithi029 ceph-mon[32610]: pgmap v3422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:22:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:22 smithi077 ceph-mon[37153]: pgmap v3423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:22:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:22 smithi047 ceph-mon[36835]: pgmap v3423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:22:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:22 smithi029 ceph-mon[32610]: pgmap v3423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:22:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5564.482483 secs 2024-07-27T08:22:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5564.482483 secs 2024-07-27T08:22:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5564.482483 secs 2024-07-27T08:22:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:24 smithi077 ceph-mon[37153]: pgmap v3424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:24 smithi047 ceph-mon[36835]: pgmap v3424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:24 smithi029 ceph-mon[32610]: pgmap v3424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:26 smithi077 ceph-mon[37153]: pgmap v3425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:26 smithi047 ceph-mon[36835]: pgmap v3425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:26 smithi029 ceph-mon[32610]: pgmap v3425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:27.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:27.392+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:22:27.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:27.392+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:22:27.394 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:27.392+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:22:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:27.409+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:22:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:27.409+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:22:27.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:27.409+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:22:27.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5569.482623 secs 2024-07-27T08:22:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5569.482623 secs 2024-07-27T08:22:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5569.482623 secs 2024-07-27T08:22:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:28 smithi077 ceph-mon[37153]: pgmap v3426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:28.951 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:28 smithi047 ceph-mon[36835]: pgmap v3426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:28 smithi029 ceph-mon[32610]: pgmap v3426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:30 smithi077 ceph-mon[37153]: pgmap v3427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:22:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:30 smithi047 ceph-mon[36835]: pgmap v3427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:22:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:30 smithi029 ceph-mon[32610]: pgmap v3427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:22:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:32 smithi077 ceph-mon[37153]: pgmap v3428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:22:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:32 smithi047 ceph-mon[36835]: pgmap v3428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:22:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:32 smithi029 ceph-mon[32610]: pgmap v3428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:22:33.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5574.482748 secs 2024-07-27T08:22:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5574.482748 secs 2024-07-27T08:22:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5574.482748 secs 2024-07-27T08:22:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:34 smithi077 ceph-mon[37153]: pgmap v3429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:34 smithi047 ceph-mon[36835]: pgmap v3429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:34 smithi029 ceph-mon[32610]: pgmap v3429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:36 smithi077 ceph-mon[37153]: pgmap v3430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:36 smithi047 ceph-mon[36835]: pgmap v3430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:36 smithi029 ceph-mon[32610]: pgmap v3430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5579.482847 secs 2024-07-27T08:22:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5579.482847 secs 2024-07-27T08:22:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5579.482847 secs 2024-07-27T08:22:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:38 smithi077 ceph-mon[37153]: pgmap v3431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:38 smithi047 ceph-mon[36835]: pgmap v3431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:38 smithi029 ceph-mon[32610]: pgmap v3431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:40 smithi077 ceph-mon[37153]: pgmap v3432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:22:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:40 smithi047 ceph-mon[36835]: pgmap v3432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:22:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:40 smithi029 ceph-mon[32610]: pgmap v3432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:22:42.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:42.402+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:22:42.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:42.402+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:22:42.404 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:42.402+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:22:42.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:42.410+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:22:42.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:42.410+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:22:42.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:42.410+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:22:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:42 smithi029 ceph-mon[32610]: pgmap v3433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:22:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:42 smithi077 ceph-mon[37153]: pgmap v3433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:22:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:42 smithi047 ceph-mon[36835]: pgmap v3433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:22:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:43.473+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:22:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:43.473+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:22:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:43.474+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:22:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:43.474+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:22:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5584.482969 secs 2024-07-27T08:22:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5584.482969 secs 2024-07-27T08:22:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5584.482969 secs 2024-07-27T08:22:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:44 smithi077 ceph-mon[37153]: pgmap v3434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:44 smithi047 ceph-mon[36835]: pgmap v3434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:44 smithi029 ceph-mon[32610]: pgmap v3434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:46 smithi077 ceph-mon[37153]: pgmap v3435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:46 smithi047 ceph-mon[36835]: pgmap v3435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:46 smithi029 ceph-mon[32610]: pgmap v3435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5589.483090 secs 2024-07-27T08:22:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5589.483090 secs 2024-07-27T08:22:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5589.483090 secs 2024-07-27T08:22:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:48 smithi077 ceph-mon[37153]: pgmap v3436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:48 smithi047 ceph-mon[36835]: pgmap v3436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:48 smithi029 ceph-mon[32610]: pgmap v3436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:50 smithi077 ceph-mon[37153]: pgmap v3437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:22:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:50 smithi047 ceph-mon[36835]: pgmap v3437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:22:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:50 smithi029 ceph-mon[32610]: pgmap v3437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:22:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:52 smithi077 ceph-mon[37153]: pgmap v3438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:22:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:52 smithi047 ceph-mon[36835]: pgmap v3438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:22:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:52 smithi029 ceph-mon[32610]: pgmap v3438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:22:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5594.483214 secs 2024-07-27T08:22:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5594.483214 secs 2024-07-27T08:22:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5594.483214 secs 2024-07-27T08:22:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:54 smithi077 ceph-mon[37153]: pgmap v3439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:54 smithi047 ceph-mon[36835]: pgmap v3439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:54 smithi029 ceph-mon[32610]: pgmap v3439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:22:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:56 smithi077 ceph-mon[37153]: pgmap v3440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:56 smithi047 ceph-mon[36835]: pgmap v3440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:56 smithi029 ceph-mon[32610]: pgmap v3440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:22:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:57.418+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:22:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:57.418+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:22:57.419 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:22:57.418+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:22:57.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:57.418+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:22:57.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:57.418+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:22:57.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:57.418+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:22:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5599.483335 secs 2024-07-27T08:22:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5599.483335 secs 2024-07-27T08:22:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5599.483335 secs 2024-07-27T08:22:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:58.476+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:22:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:58.476+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:22:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:58.476+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:22:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:22:58.476+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:22:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:22:58 smithi047 ceph-mon[36835]: pgmap v3441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:22:58 smithi029 ceph-mon[32610]: pgmap v3441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:22:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:22:58 smithi077 ceph-mon[37153]: pgmap v3441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:00 smithi077 ceph-mon[37153]: pgmap v3442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:00 smithi047 ceph-mon[36835]: pgmap v3442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:00 smithi029 ceph-mon[32610]: pgmap v3442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:02 smithi077 ceph-mon[37153]: pgmap v3443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:23:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:02 smithi047 ceph-mon[36835]: pgmap v3443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:23:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:02 smithi029 ceph-mon[32610]: pgmap v3443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:23:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5604.483457 secs 2024-07-27T08:23:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:23:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:23:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:23:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:23:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:23:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:23:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:23:03.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:23:03.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:23:03.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: pgmap v3444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:23:03.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: pgmap v3445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:23:03.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:23:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5604.483457 secs 2024-07-27T08:23:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:23:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:23:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:23:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:23:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:23:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:23:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:23:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:23:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:23:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: pgmap v3444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:23:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: pgmap v3445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:23:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:23:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5604.483457 secs 2024-07-27T08:23:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:23:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:23:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:23:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:23:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:23:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:23:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:23:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:23:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:23:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: pgmap v3444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:23:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: pgmap v3445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:23:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:23:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:06 smithi077 ceph-mon[37153]: pgmap v3446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:23:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:06 smithi047 ceph-mon[36835]: pgmap v3446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:23:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:06 smithi029 ceph-mon[32610]: pgmap v3446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:23:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5609.483577 secs 2024-07-27T08:23:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:08 smithi077 ceph-mon[37153]: pgmap v3447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:23:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5609.483577 secs 2024-07-27T08:23:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:08 smithi047 ceph-mon[36835]: pgmap v3447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:23:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5609.483577 secs 2024-07-27T08:23:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:08 smithi029 ceph-mon[32610]: pgmap v3447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:23:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:10 smithi077 ceph-mon[37153]: pgmap v3448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:23:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:10 smithi047 ceph-mon[36835]: pgmap v3448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:23:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:10 smithi029 ceph-mon[32610]: pgmap v3448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:23:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:12 smithi077 ceph-mon[37153]: pgmap v3449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:23:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:23:12.430+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:23:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:23:12.430+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:23:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:23:12.430+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:23:12.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:23:12.433+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:23:12.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:23:12.433+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:23:12.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:23:12.433+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:23:12.434 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:12 smithi029 ceph-mon[32610]: pgmap v3449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:23:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:12 smithi047 ceph-mon[36835]: pgmap v3449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:23:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5614.483676 secs 2024-07-27T08:23:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5614.483676 secs 2024-07-27T08:23:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5614.483676 secs 2024-07-27T08:23:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:14 smithi077 ceph-mon[37153]: pgmap v3450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:23:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:14 smithi047 ceph-mon[36835]: pgmap v3450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:23:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:14 smithi029 ceph-mon[32610]: pgmap v3450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:23:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:16 smithi047 ceph-mon[36835]: pgmap v3451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:16 smithi029 ceph-mon[32610]: pgmap v3451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:16 smithi077 ceph-mon[37153]: pgmap v3451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:18.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5619.483780 secs 2024-07-27T08:23:18.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:18 smithi077 ceph-mon[37153]: pgmap v3452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5619.483780 secs 2024-07-27T08:23:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:18 smithi047 ceph-mon[36835]: pgmap v3452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5619.483780 secs 2024-07-27T08:23:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:18 smithi029 ceph-mon[32610]: pgmap v3452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:20.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:20 smithi047 ceph-mon[36835]: pgmap v3453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:20 smithi029 ceph-mon[32610]: pgmap v3453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:20 smithi077 ceph-mon[37153]: pgmap v3453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:22 smithi047 ceph-mon[36835]: pgmap v3454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:22 smithi029 ceph-mon[32610]: pgmap v3454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:22 smithi077 ceph-mon[37153]: pgmap v3454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5624.483907 secs 2024-07-27T08:23:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5624.483907 secs 2024-07-27T08:23:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5624.483907 secs 2024-07-27T08:23:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:24 smithi047 ceph-mon[36835]: pgmap v3455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:23:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:24 smithi029 ceph-mon[32610]: pgmap v3455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:23:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:24 smithi077 ceph-mon[37153]: pgmap v3455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:23:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:26 smithi047 ceph-mon[36835]: pgmap v3456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:26 smithi029 ceph-mon[32610]: pgmap v3456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:26 smithi077 ceph-mon[37153]: pgmap v3456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:23:27.434+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:23:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:23:27.434+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:23:27.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:23:27.434+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:23:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:23:27.444+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:23:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:23:27.444+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:23:27.446 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:23:27.444+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:23:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5629.484027 secs 2024-07-27T08:23:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:28 smithi047 ceph-mon[36835]: pgmap v3457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5629.484027 secs 2024-07-27T08:23:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:28 smithi029 ceph-mon[32610]: pgmap v3457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5629.484027 secs 2024-07-27T08:23:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:28 smithi077 ceph-mon[37153]: pgmap v3457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:30 smithi047 ceph-mon[36835]: pgmap v3458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:30 smithi029 ceph-mon[32610]: pgmap v3458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:30 smithi077 ceph-mon[37153]: pgmap v3458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:32.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:32 smithi047 ceph-mon[36835]: pgmap v3459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:32 smithi029 ceph-mon[32610]: pgmap v3459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:32 smithi077 ceph-mon[37153]: pgmap v3459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5634.484151 secs 2024-07-27T08:23:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5634.484151 secs 2024-07-27T08:23:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5634.484151 secs 2024-07-27T08:23:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:34 smithi047 ceph-mon[36835]: pgmap v3460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:23:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:34 smithi029 ceph-mon[32610]: pgmap v3460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:23:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:34 smithi077 ceph-mon[37153]: pgmap v3460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:23:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:36 smithi047 ceph-mon[36835]: pgmap v3461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:36 smithi029 ceph-mon[32610]: pgmap v3461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:36 smithi077 ceph-mon[37153]: pgmap v3461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5639.484278 secs 2024-07-27T08:23:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:38 smithi047 ceph-mon[36835]: pgmap v3462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5639.484278 secs 2024-07-27T08:23:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:38 smithi029 ceph-mon[32610]: pgmap v3462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5639.484278 secs 2024-07-27T08:23:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:38 smithi077 ceph-mon[37153]: pgmap v3462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:40.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:40 smithi047 ceph-mon[36835]: pgmap v3463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:40 smithi029 ceph-mon[32610]: pgmap v3463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:40 smithi077 ceph-mon[37153]: pgmap v3463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:42.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:23:42.449+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:23:42.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:23:42.449+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:23:42.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:23:42.449+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:23:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:23:42.459+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:23:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:23:42.459+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:23:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:23:42.459+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:23:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:42 smithi047 ceph-mon[36835]: pgmap v3464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:42 smithi029 ceph-mon[32610]: pgmap v3464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:42 smithi077 ceph-mon[37153]: pgmap v3464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5644.484381 secs 2024-07-27T08:23:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5644.484381 secs 2024-07-27T08:23:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5644.484381 secs 2024-07-27T08:23:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:44 smithi047 ceph-mon[36835]: pgmap v3465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:23:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:44 smithi029 ceph-mon[32610]: pgmap v3465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:23:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:44 smithi077 ceph-mon[37153]: pgmap v3465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:23:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:46 smithi047 ceph-mon[36835]: pgmap v3466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:46 smithi029 ceph-mon[32610]: pgmap v3466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:46 smithi077 ceph-mon[37153]: pgmap v3466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5649.484522 secs 2024-07-27T08:23:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:48 smithi047 ceph-mon[36835]: pgmap v3467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5649.484522 secs 2024-07-27T08:23:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:48 smithi029 ceph-mon[32610]: pgmap v3467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5649.484522 secs 2024-07-27T08:23:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:48 smithi077 ceph-mon[37153]: pgmap v3467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:50 smithi047 ceph-mon[36835]: pgmap v3468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:50 smithi029 ceph-mon[32610]: pgmap v3468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:50 smithi077 ceph-mon[37153]: pgmap v3468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:23:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:52 smithi047 ceph-mon[36835]: pgmap v3469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:52 smithi029 ceph-mon[32610]: pgmap v3469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:52 smithi077 ceph-mon[37153]: pgmap v3469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5654.484599 secs 2024-07-27T08:23:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5654.484599 secs 2024-07-27T08:23:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5654.484599 secs 2024-07-27T08:23:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:54 smithi029 ceph-mon[32610]: pgmap v3470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:23:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:54 smithi077 ceph-mon[37153]: pgmap v3470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:23:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:54 smithi047 ceph-mon[36835]: pgmap v3470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:23:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:56 smithi029 ceph-mon[32610]: pgmap v3471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:56 smithi077 ceph-mon[37153]: pgmap v3471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:56 smithi047 ceph-mon[36835]: pgmap v3471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:23:57.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:23:57.465+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:23:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:23:57.465+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:23:57.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:23:57.465+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:23:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:23:57.474+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:23:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:23:57.474+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:23:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:23:57.474+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:23:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5659.484664 secs 2024-07-27T08:23:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:23:58 smithi029 ceph-mon[32610]: pgmap v3472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5659.484664 secs 2024-07-27T08:23:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:23:58 smithi077 ceph-mon[37153]: pgmap v3472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:23:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5659.484664 secs 2024-07-27T08:23:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:23:58 smithi047 ceph-mon[36835]: pgmap v3472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:24:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:00 smithi029 ceph-mon[32610]: pgmap v3473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:00 smithi077 ceph-mon[37153]: pgmap v3473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:00 smithi047 ceph-mon[36835]: pgmap v3473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:02 smithi029 ceph-mon[32610]: pgmap v3474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:02 smithi077 ceph-mon[37153]: pgmap v3474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:02 smithi047 ceph-mon[36835]: pgmap v3474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5664.484800 secs 2024-07-27T08:24:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:24:03.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5664.484800 secs 2024-07-27T08:24:03.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:24:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5664.484800 secs 2024-07-27T08:24:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:24:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:04 smithi077 ceph-mon[37153]: pgmap v3475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:24:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:24:04.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:24:04.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:24:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:04 smithi047 ceph-mon[36835]: pgmap v3475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:24:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:24:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:24:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:24:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:04 smithi029 ceph-mon[32610]: pgmap v3475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:24:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:24:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:24:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:24:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:24:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:05 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:24:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:05 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:24:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:05 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:24:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:05 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:24:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:05 smithi077 ceph-mon[37153]: pgmap v3476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:24:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:05 smithi077 ceph-mon[37153]: pgmap v3477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:24:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:05 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:24:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:05 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:24:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:05 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:24:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:05 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:24:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:05 smithi047 ceph-mon[36835]: pgmap v3476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:24:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:05 smithi047 ceph-mon[36835]: pgmap v3477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:24:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:05 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:24:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:05 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:24:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:05 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:24:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:05 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:24:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:05 smithi029 ceph-mon[32610]: pgmap v3476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:24:05.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:05 smithi029 ceph-mon[32610]: pgmap v3477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:24:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:07 smithi077 ceph-mon[37153]: pgmap v3478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:24:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:07 smithi047 ceph-mon[36835]: pgmap v3478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:24:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:07 smithi029 ceph-mon[32610]: pgmap v3478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:24:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5669.484864 secs 2024-07-27T08:24:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5669.484864 secs 2024-07-27T08:24:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5669.484864 secs 2024-07-27T08:24:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:09 smithi077 ceph-mon[37153]: pgmap v3479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:24:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:09 smithi047 ceph-mon[36835]: pgmap v3479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:24:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:09 smithi029 ceph-mon[32610]: pgmap v3479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:24:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:11 smithi077 ceph-mon[37153]: pgmap v3480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:24:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:11 smithi047 ceph-mon[36835]: pgmap v3480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:24:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:11 smithi029 ceph-mon[32610]: pgmap v3480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:24:12.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:24:12.476+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:24:12.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:24:12.476+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:24:12.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:24:12.476+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:24:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:24:12.490+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:24:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:24:12.490+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:24:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:24:12.490+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:24:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:13 smithi077 ceph-mon[37153]: pgmap v3481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:24:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5674.484989 secs 2024-07-27T08:24:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:13 smithi047 ceph-mon[36835]: pgmap v3481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:24:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5674.484989 secs 2024-07-27T08:24:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:13 smithi029 ceph-mon[32610]: pgmap v3481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:24:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5674.484989 secs 2024-07-27T08:24:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:14 smithi077 ceph-mon[37153]: pgmap v3482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:24:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:14 smithi047 ceph-mon[36835]: pgmap v3482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:24:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:14 smithi029 ceph-mon[32610]: pgmap v3482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:24:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:16 smithi047 ceph-mon[36835]: pgmap v3483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:16 smithi029 ceph-mon[32610]: pgmap v3483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:16 smithi077 ceph-mon[37153]: pgmap v3483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5679.485076 secs 2024-07-27T08:24:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5679.485076 secs 2024-07-27T08:24:18.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5679.485076 secs 2024-07-27T08:24:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:18 smithi047 ceph-mon[36835]: pgmap v3484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:18 smithi029 ceph-mon[32610]: pgmap v3484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:18 smithi077 ceph-mon[37153]: pgmap v3484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:20 smithi047 ceph-mon[36835]: pgmap v3485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:24:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:20 smithi029 ceph-mon[32610]: pgmap v3485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:24:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:20 smithi077 ceph-mon[37153]: pgmap v3485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:24:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:22 smithi047 ceph-mon[36835]: pgmap v3486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:24:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5684.485206 secs 2024-07-27T08:24:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:22 smithi029 ceph-mon[32610]: pgmap v3486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:24:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5684.485206 secs 2024-07-27T08:24:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:22 smithi077 ceph-mon[37153]: pgmap v3486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:24:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5684.485206 secs 2024-07-27T08:24:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:24 smithi047 ceph-mon[36835]: pgmap v3487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:24 smithi029 ceph-mon[32610]: pgmap v3487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:24 smithi077 ceph-mon[37153]: pgmap v3487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:26 smithi047 ceph-mon[36835]: pgmap v3488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:26 smithi029 ceph-mon[32610]: pgmap v3488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:26 smithi077 ceph-mon[37153]: pgmap v3488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:27.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:24:27.490+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:24:27.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:24:27.490+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:24:27.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:24:27.490+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:24:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:24:27.495+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:24:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:24:27.495+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:24:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:24:27.495+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:24:27.864 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5689.485364 secs 2024-07-27T08:24:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5689.485364 secs 2024-07-27T08:24:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5689.485364 secs 2024-07-27T08:24:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:28 smithi047 ceph-mon[36835]: pgmap v3489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:28 smithi029 ceph-mon[32610]: pgmap v3489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:28 smithi077 ceph-mon[37153]: pgmap v3489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:30 smithi047 ceph-mon[36835]: pgmap v3490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:24:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:30 smithi029 ceph-mon[32610]: pgmap v3490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:24:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:30 smithi077 ceph-mon[37153]: pgmap v3490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:24:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:32 smithi047 ceph-mon[36835]: pgmap v3491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:24:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5694.485493 secs 2024-07-27T08:24:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:32 smithi029 ceph-mon[32610]: pgmap v3491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:24:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5694.485493 secs 2024-07-27T08:24:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:32 smithi077 ceph-mon[37153]: pgmap v3491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:24:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5694.485493 secs 2024-07-27T08:24:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:34 smithi047 ceph-mon[36835]: pgmap v3492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:34 smithi029 ceph-mon[32610]: pgmap v3492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:34 smithi077 ceph-mon[37153]: pgmap v3492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:36 smithi047 ceph-mon[36835]: pgmap v3493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:36 smithi029 ceph-mon[32610]: pgmap v3493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:36 smithi077 ceph-mon[37153]: pgmap v3493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5699.485618 secs 2024-07-27T08:24:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5699.485618 secs 2024-07-27T08:24:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5699.485618 secs 2024-07-27T08:24:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:38 smithi047 ceph-mon[36835]: pgmap v3494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:38 smithi029 ceph-mon[32610]: pgmap v3494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:38 smithi077 ceph-mon[37153]: pgmap v3494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:40 smithi047 ceph-mon[36835]: pgmap v3495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:24:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:40 smithi029 ceph-mon[32610]: pgmap v3495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:24:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:40 smithi077 ceph-mon[37153]: pgmap v3495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:24:42.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:24:42.505+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:24:42.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:24:42.505+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:24:42.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:24:42.505+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:24:42.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:24:42.510+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:24:42.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:24:42.510+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:24:42.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:24:42.510+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:24:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:42 smithi047 ceph-mon[36835]: pgmap v3496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:24:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5704.485695 secs 2024-07-27T08:24:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:42 smithi029 ceph-mon[32610]: pgmap v3496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:24:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5704.485695 secs 2024-07-27T08:24:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:42 smithi077 ceph-mon[37153]: pgmap v3496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:24:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5704.485695 secs 2024-07-27T08:24:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:44 smithi047 ceph-mon[36835]: pgmap v3497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:44 smithi029 ceph-mon[32610]: pgmap v3497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:44 smithi077 ceph-mon[37153]: pgmap v3497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:46 smithi047 ceph-mon[36835]: pgmap v3498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:46 smithi029 ceph-mon[32610]: pgmap v3498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:46 smithi077 ceph-mon[37153]: pgmap v3498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5709.485852 secs 2024-07-27T08:24:48.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5709.485852 secs 2024-07-27T08:24:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5709.485852 secs 2024-07-27T08:24:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:48 smithi029 ceph-mon[32610]: pgmap v3499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:48 smithi077 ceph-mon[37153]: pgmap v3499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:48 smithi047 ceph-mon[36835]: pgmap v3499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:50 smithi029 ceph-mon[32610]: pgmap v3500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:24:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:50 smithi077 ceph-mon[37153]: pgmap v3500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:24:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:50 smithi047 ceph-mon[36835]: pgmap v3500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:24:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:52 smithi029 ceph-mon[32610]: pgmap v3501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:24:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5714.485974 secs 2024-07-27T08:24:52.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:52 smithi047 ceph-mon[36835]: pgmap v3501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:24:52.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5714.485974 secs 2024-07-27T08:24:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:52 smithi077 ceph-mon[37153]: pgmap v3501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:24:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5714.485974 secs 2024-07-27T08:24:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:54 smithi029 ceph-mon[32610]: pgmap v3502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:54 smithi077 ceph-mon[37153]: pgmap v3502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:54 smithi047 ceph-mon[36835]: pgmap v3502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:24:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:56 smithi029 ceph-mon[32610]: pgmap v3503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:56 smithi077 ceph-mon[37153]: pgmap v3503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:56 smithi047 ceph-mon[36835]: pgmap v3503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:24:57.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:24:57.519+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:24:57.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:24:57.519+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:24:57.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:24:57.519+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:24:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:24:57.525+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:24:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:24:57.525+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:24:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:24:57.525+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:24:57.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5719.486093 secs 2024-07-27T08:24:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5719.486093 secs 2024-07-27T08:24:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5719.486093 secs 2024-07-27T08:24:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:24:58 smithi077 ceph-mon[37153]: pgmap v3504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:59.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:24:58 smithi047 ceph-mon[36835]: pgmap v3504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:24:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:24:58 smithi029 ceph-mon[32610]: pgmap v3504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:25:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:00 smithi077 ceph-mon[37153]: pgmap v3505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:25:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:00 smithi047 ceph-mon[36835]: pgmap v3505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:25:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:00 smithi029 ceph-mon[32610]: pgmap v3505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:25:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:02 smithi077 ceph-mon[37153]: pgmap v3506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5724.486194 secs 2024-07-27T08:25:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:02 smithi047 ceph-mon[36835]: pgmap v3506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5724.486194 secs 2024-07-27T08:25:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:02 smithi029 ceph-mon[32610]: pgmap v3506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5724.486194 secs 2024-07-27T08:25:04.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:25:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:25:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:25:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:04 smithi077 ceph-mon[37153]: pgmap v3507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:25:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:25:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:25:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:04 smithi077 ceph-mon[37153]: pgmap v3508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:25:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:04 smithi077 ceph-mon[37153]: pgmap v3509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:25:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:04 smithi047 ceph-mon[36835]: pgmap v3507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:25:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:25:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:25:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:04 smithi047 ceph-mon[36835]: pgmap v3508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:25:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:04 smithi047 ceph-mon[36835]: pgmap v3509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:25:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:04 smithi029 ceph-mon[32610]: pgmap v3507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:25:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:25:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:25:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:04 smithi029 ceph-mon[32610]: pgmap v3508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:25:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:04 smithi029 ceph-mon[32610]: pgmap v3509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:25:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:07 smithi047 ceph-mon[36835]: pgmap v3510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:25:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:07 smithi029 ceph-mon[32610]: pgmap v3510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:25:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:07 smithi077 ceph-mon[37153]: pgmap v3510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:25:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5729.486321 secs 2024-07-27T08:25:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5729.486321 secs 2024-07-27T08:25:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5729.486321 secs 2024-07-27T08:25:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:09 smithi047 ceph-mon[36835]: pgmap v3511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:25:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:09 smithi029 ceph-mon[32610]: pgmap v3511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:25:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:09 smithi077 ceph-mon[37153]: pgmap v3511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:25:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:11 smithi029 ceph-mon[32610]: pgmap v3512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:25:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:11 smithi077 ceph-mon[37153]: pgmap v3512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:25:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:11 smithi047 ceph-mon[36835]: pgmap v3512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:25:12.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:12.535+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:25:12.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:12.535+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:25:12.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:12.535+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:25:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:12.541+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:25:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:12.541+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:25:12.542 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:12.541+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:25:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5734.486385 secs 2024-07-27T08:25:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:13 smithi029 ceph-mon[32610]: pgmap v3513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:25:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5734.486385 secs 2024-07-27T08:25:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:13 smithi077 ceph-mon[37153]: pgmap v3513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:25:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5734.486385 secs 2024-07-27T08:25:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:13 smithi047 ceph-mon[36835]: pgmap v3513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:25:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:15 smithi029 ceph-mon[32610]: pgmap v3514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:25:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:15 smithi077 ceph-mon[37153]: pgmap v3514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:25:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:15 smithi047 ceph-mon[36835]: pgmap v3514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:25:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:17 smithi029 ceph-mon[32610]: pgmap v3515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:17 smithi077 ceph-mon[37153]: pgmap v3515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:17 smithi047 ceph-mon[36835]: pgmap v3515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:18.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5739.486507 secs 2024-07-27T08:25:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5739.486507 secs 2024-07-27T08:25:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5739.486507 secs 2024-07-27T08:25:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:19 smithi077 ceph-mon[37153]: pgmap v3516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:19 smithi047 ceph-mon[36835]: pgmap v3516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:19 smithi029 ceph-mon[32610]: pgmap v3516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:21 smithi029 ceph-mon[32610]: pgmap v3517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:25:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:21 smithi077 ceph-mon[37153]: pgmap v3517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:25:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:21 smithi047 ceph-mon[36835]: pgmap v3517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:25:23.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5744.486631 secs 2024-07-27T08:25:23.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:23 smithi047 ceph-mon[36835]: pgmap v3518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:25:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5744.486631 secs 2024-07-27T08:25:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:23 smithi077 ceph-mon[37153]: pgmap v3518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:25:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5744.486631 secs 2024-07-27T08:25:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:23 smithi029 ceph-mon[32610]: pgmap v3518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:25:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:25 smithi077 ceph-mon[37153]: pgmap v3519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:25 smithi047 ceph-mon[36835]: pgmap v3519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:25 smithi029 ceph-mon[32610]: pgmap v3519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:27.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:27 smithi029 ceph-mon[32610]: pgmap v3520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:27.547+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:25:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:27.547+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:25:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:27.547+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:25:27.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:27.547+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:25:27.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:27.547+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:25:27.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:27.547+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:25:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:27 smithi077 ceph-mon[37153]: pgmap v3520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:27 smithi047 ceph-mon[36835]: pgmap v3520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5749.486769 secs 2024-07-27T08:25:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5749.486769 secs 2024-07-27T08:25:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5749.486769 secs 2024-07-27T08:25:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:29 smithi077 ceph-mon[37153]: pgmap v3521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:29 smithi047 ceph-mon[36835]: pgmap v3521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:29 smithi029 ceph-mon[32610]: pgmap v3521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:31 smithi077 ceph-mon[37153]: pgmap v3522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:25:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:31 smithi047 ceph-mon[36835]: pgmap v3522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:25:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:31 smithi029 ceph-mon[32610]: pgmap v3522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:25:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5754.486890 secs 2024-07-27T08:25:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:33 smithi077 ceph-mon[37153]: pgmap v3523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5754.486890 secs 2024-07-27T08:25:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:33 smithi047 ceph-mon[36835]: pgmap v3523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5754.486890 secs 2024-07-27T08:25:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:33 smithi029 ceph-mon[32610]: pgmap v3523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:35 smithi077 ceph-mon[37153]: pgmap v3524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:35 smithi047 ceph-mon[36835]: pgmap v3524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:35 smithi029 ceph-mon[32610]: pgmap v3524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:37 smithi029 ceph-mon[32610]: pgmap v3525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:37 smithi077 ceph-mon[37153]: pgmap v3525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:37 smithi047 ceph-mon[36835]: pgmap v3525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5759.487009 secs 2024-07-27T08:25:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5759.487009 secs 2024-07-27T08:25:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5759.487009 secs 2024-07-27T08:25:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:39 smithi077 ceph-mon[37153]: pgmap v3526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:39 smithi047 ceph-mon[36835]: pgmap v3526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:39 smithi029 ceph-mon[32610]: pgmap v3526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:41 smithi077 ceph-mon[37153]: pgmap v3527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:25:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:41 smithi047 ceph-mon[36835]: pgmap v3527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:25:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:41 smithi029 ceph-mon[32610]: pgmap v3527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:25:42.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:42.563+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:25:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:42.563+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:25:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:42.563+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:25:42.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:42.563+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:25:42.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:42.563+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:25:42.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:42.563+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:25:43.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:43.500+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:25:43.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:43.500+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:25:43.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:43.500+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:25:43.502 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:43.500+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:25:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:43 smithi077 ceph-mon[37153]: 1 slow requests, 1 included below; oldest blocked for > 5764.487131 secs 2024-07-27T08:25:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:43 smithi077 ceph-mon[37153]: slow request 5764.487131 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T08:25:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:43 smithi077 ceph-mon[37153]: pgmap v3528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:43 smithi047 ceph-mon[36835]: 1 slow requests, 1 included below; oldest blocked for > 5764.487131 secs 2024-07-27T08:25:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:43 smithi047 ceph-mon[36835]: slow request 5764.487131 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T08:25:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:43 smithi047 ceph-mon[36835]: pgmap v3528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:43 smithi029 ceph-mon[32610]: 1 slow requests, 1 included below; oldest blocked for > 5764.487131 secs 2024-07-27T08:25:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:43 smithi029 ceph-mon[32610]: slow request 5764.487131 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T08:25:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:43 smithi029 ceph-mon[32610]: pgmap v3528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:45 smithi077 ceph-mon[37153]: pgmap v3529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:25:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:45 smithi047 ceph-mon[36835]: pgmap v3529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:25:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:45 smithi029 ceph-mon[32610]: pgmap v3529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:25:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:47 smithi077 ceph-mon[37153]: pgmap v3530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:47 smithi047 ceph-mon[36835]: pgmap v3530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:47 smithi029 ceph-mon[32610]: pgmap v3530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5769.487256 secs 2024-07-27T08:25:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5769.487256 secs 2024-07-27T08:25:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5769.487256 secs 2024-07-27T08:25:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:49 smithi077 ceph-mon[37153]: pgmap v3531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:49 smithi047 ceph-mon[36835]: pgmap v3531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:49 smithi029 ceph-mon[32610]: pgmap v3531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:51 smithi077 ceph-mon[37153]: pgmap v3532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:25:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:51 smithi047 ceph-mon[36835]: pgmap v3532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:25:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:51 smithi029 ceph-mon[32610]: pgmap v3532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:25:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5774.487383 secs 2024-07-27T08:25:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:53 smithi077 ceph-mon[37153]: pgmap v3533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5774.487383 secs 2024-07-27T08:25:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:53 smithi047 ceph-mon[36835]: pgmap v3533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5774.487383 secs 2024-07-27T08:25:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:53 smithi029 ceph-mon[32610]: pgmap v3533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:55 smithi077 ceph-mon[37153]: pgmap v3534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:25:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:55 smithi047 ceph-mon[36835]: pgmap v3534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:25:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:55 smithi029 ceph-mon[32610]: pgmap v3534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:25:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:57.566+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:25:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:57.566+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:25:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:25:57.566+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:25:57.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:57 smithi029 ceph-mon[32610]: pgmap v3535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:57.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:57.570+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:25:57.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:57.570+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:25:57.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:57.570+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:25:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:57 smithi077 ceph-mon[37153]: pgmap v3535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:57 smithi047 ceph-mon[36835]: pgmap v3535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:25:58.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:58.502+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:25:58.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:58.502+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:25:58.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:58.502+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:25:58.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:25:58.502+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:25:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5779.487501 secs 2024-07-27T08:25:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5779.487501 secs 2024-07-27T08:25:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5779.487501 secs 2024-07-27T08:25:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:25:59 smithi077 ceph-mon[37153]: pgmap v3536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:25:59 smithi047 ceph-mon[36835]: pgmap v3536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:25:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:25:59 smithi029 ceph-mon[32610]: pgmap v3536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:26:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:01 smithi077 ceph-mon[37153]: pgmap v3537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:26:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:01 smithi047 ceph-mon[36835]: pgmap v3537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:26:01.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:01 smithi029 ceph-mon[32610]: pgmap v3537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:26:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5784.487648 secs 2024-07-27T08:26:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:03 smithi077 ceph-mon[37153]: pgmap v3538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5784.487648 secs 2024-07-27T08:26:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:03 smithi047 ceph-mon[36835]: pgmap v3538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5784.487648 secs 2024-07-27T08:26:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:03 smithi029 ceph-mon[32610]: pgmap v3538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:26:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:26:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:05 smithi077 ceph-mon[37153]: pgmap v3539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:26:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:26:05.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:26:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:05 smithi047 ceph-mon[36835]: pgmap v3539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:26:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:26:05.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:26:05.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:26:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:05 smithi029 ceph-mon[32610]: pgmap v3539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:26:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:26:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:26:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:26:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:06 smithi047 ceph-mon[36835]: pgmap v3540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:26:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:06 smithi047 ceph-mon[36835]: pgmap v3541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:26:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:06 smithi029 ceph-mon[32610]: pgmap v3540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:26:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:06 smithi029 ceph-mon[32610]: pgmap v3541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:26:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:06 smithi077 ceph-mon[37153]: pgmap v3540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:26:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:06 smithi077 ceph-mon[37153]: pgmap v3541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:26:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:08 smithi047 ceph-mon[36835]: pgmap v3542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:26:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5789.487748 secs 2024-07-27T08:26:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:08 smithi029 ceph-mon[32610]: pgmap v3542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:26:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5789.487748 secs 2024-07-27T08:26:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:08 smithi077 ceph-mon[37153]: pgmap v3542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:26:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5789.487748 secs 2024-07-27T08:26:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:10 smithi047 ceph-mon[36835]: pgmap v3543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:26:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:10 smithi029 ceph-mon[32610]: pgmap v3543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:26:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:10 smithi077 ceph-mon[37153]: pgmap v3543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:26:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:26:12.577+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:26:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:26:12.577+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:26:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:26:12.577+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:26:12.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:26:12.582+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:26:12.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:26:12.582+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:26:12.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:26:12.582+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:26:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:12 smithi047 ceph-mon[36835]: pgmap v3544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:26:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:12 smithi029 ceph-mon[32610]: pgmap v3544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:26:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:12 smithi077 ceph-mon[37153]: pgmap v3544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:26:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5794.487880 secs 2024-07-27T08:26:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5794.487880 secs 2024-07-27T08:26:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5794.487880 secs 2024-07-27T08:26:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:14 smithi047 ceph-mon[36835]: pgmap v3545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:26:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:14 smithi029 ceph-mon[32610]: pgmap v3545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:26:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:14 smithi077 ceph-mon[37153]: pgmap v3545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:26:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:16 smithi047 ceph-mon[36835]: pgmap v3546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:26:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:16 smithi029 ceph-mon[32610]: pgmap v3546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:26:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:16 smithi077 ceph-mon[37153]: pgmap v3546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:26:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:18 smithi047 ceph-mon[36835]: pgmap v3547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5799.487948 secs 2024-07-27T08:26:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:18 smithi029 ceph-mon[32610]: pgmap v3547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5799.487948 secs 2024-07-27T08:26:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:18 smithi077 ceph-mon[37153]: pgmap v3547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5799.487948 secs 2024-07-27T08:26:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:20 smithi047 ceph-mon[36835]: pgmap v3548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:26:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:20 smithi029 ceph-mon[32610]: pgmap v3548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:26:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:20 smithi077 ceph-mon[37153]: pgmap v3548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:26:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:22 smithi047 ceph-mon[36835]: pgmap v3549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:26:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:22 smithi029 ceph-mon[32610]: pgmap v3549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:26:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:22 smithi077 ceph-mon[37153]: pgmap v3549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:26:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5804.488073 secs 2024-07-27T08:26:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5804.488073 secs 2024-07-27T08:26:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5804.488073 secs 2024-07-27T08:26:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:24 smithi029 ceph-mon[32610]: pgmap v3550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:26:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:24 smithi077 ceph-mon[37153]: pgmap v3550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:26:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:24 smithi047 ceph-mon[36835]: pgmap v3550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:26:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:26 smithi029 ceph-mon[32610]: pgmap v3551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:26:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:26 smithi077 ceph-mon[37153]: pgmap v3551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:26:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:26 smithi047 ceph-mon[36835]: pgmap v3551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:26:27.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:26:27.584+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:26:27.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:26:27.584+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:26:27.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:26:27.584+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:26:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:26:27.591+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:26:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:26:27.592+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:26:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:26:27.592+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:26:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:28 smithi029 ceph-mon[32610]: pgmap v3552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5809.488179 secs 2024-07-27T08:26:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:28 smithi077 ceph-mon[37153]: pgmap v3552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5809.488179 secs 2024-07-27T08:26:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:28 smithi047 ceph-mon[36835]: pgmap v3552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5809.488179 secs 2024-07-27T08:26:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:30 smithi077 ceph-mon[37153]: pgmap v3553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:26:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:30 smithi047 ceph-mon[36835]: pgmap v3553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:26:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:30 smithi029 ceph-mon[32610]: pgmap v3553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:26:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:32 smithi077 ceph-mon[37153]: pgmap v3554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:26:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:32 smithi047 ceph-mon[36835]: pgmap v3554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:26:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:32 smithi029 ceph-mon[32610]: pgmap v3554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:26:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5814.488251 secs 2024-07-27T08:26:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5814.488251 secs 2024-07-27T08:26:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5814.488251 secs 2024-07-27T08:26:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:34 smithi077 ceph-mon[37153]: pgmap v3555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:26:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:34 smithi047 ceph-mon[36835]: pgmap v3555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:26:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:34 smithi029 ceph-mon[32610]: pgmap v3555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:26:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:36 smithi077 ceph-mon[37153]: pgmap v3556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:26:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:36 smithi047 ceph-mon[36835]: pgmap v3556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:26:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:36 smithi029 ceph-mon[32610]: pgmap v3556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:26:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:38 smithi077 ceph-mon[37153]: pgmap v3557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5819.488400 secs 2024-07-27T08:26:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:38 smithi047 ceph-mon[36835]: pgmap v3557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5819.488400 secs 2024-07-27T08:26:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:38 smithi029 ceph-mon[32610]: pgmap v3557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5819.488400 secs 2024-07-27T08:26:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:40 smithi077 ceph-mon[37153]: pgmap v3558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:26:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:40 smithi047 ceph-mon[36835]: pgmap v3558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:26:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:40 smithi029 ceph-mon[32610]: pgmap v3558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:26:42.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:26:42.600+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:26:42.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:26:42.600+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:26:42.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:26:42.600+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:26:42.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:26:42.607+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:26:42.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:26:42.607+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:26:42.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:26:42.607+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:26:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:42 smithi077 ceph-mon[37153]: pgmap v3559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:26:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:42 smithi047 ceph-mon[36835]: pgmap v3559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:26:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:42 smithi029 ceph-mon[32610]: pgmap v3559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:26:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5824.488546 secs 2024-07-27T08:26:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5824.488546 secs 2024-07-27T08:26:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5824.488546 secs 2024-07-27T08:26:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:44 smithi077 ceph-mon[37153]: pgmap v3560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:26:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:44 smithi047 ceph-mon[36835]: pgmap v3560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:26:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:44 smithi029 ceph-mon[32610]: pgmap v3560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:26:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:46 smithi077 ceph-mon[37153]: pgmap v3561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:26:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:46 smithi047 ceph-mon[36835]: pgmap v3561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:26:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:46 smithi029 ceph-mon[32610]: pgmap v3561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:26:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:48 smithi077 ceph-mon[37153]: pgmap v3562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5829.488635 secs 2024-07-27T08:26:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:48 smithi047 ceph-mon[36835]: pgmap v3562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:48.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5829.488635 secs 2024-07-27T08:26:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:48 smithi029 ceph-mon[32610]: pgmap v3562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5829.488635 secs 2024-07-27T08:26:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:50 smithi077 ceph-mon[37153]: pgmap v3563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:26:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:50 smithi047 ceph-mon[36835]: pgmap v3563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:26:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:50 smithi029 ceph-mon[32610]: pgmap v3563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:26:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:52 smithi077 ceph-mon[37153]: pgmap v3564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:26:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:52 smithi047 ceph-mon[36835]: pgmap v3564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:26:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:52 smithi029 ceph-mon[32610]: pgmap v3564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:26:53.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5834.488754 secs 2024-07-27T08:26:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5834.488754 secs 2024-07-27T08:26:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5834.488754 secs 2024-07-27T08:26:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:54 smithi077 ceph-mon[37153]: pgmap v3565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:26:54.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:54 smithi029 ceph-mon[32610]: pgmap v3565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:26:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:54 smithi047 ceph-mon[36835]: pgmap v3565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:26:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:55 smithi077 ceph-mon[37153]: pgmap v3566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:26:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:55 smithi047 ceph-mon[36835]: pgmap v3566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:26:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:55 smithi029 ceph-mon[32610]: pgmap v3566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:26:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:26:57.603+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:26:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:26:57.603+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:26:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:26:57.603+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:26:57.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:26:57.609+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:26:57.611 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:26:57.609+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:26:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:26:57.609+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:26:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:57 smithi029 ceph-mon[32610]: pgmap v3567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5839.488885 secs 2024-07-27T08:26:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:57 smithi077 ceph-mon[37153]: pgmap v3567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5839.488885 secs 2024-07-27T08:26:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:57 smithi047 ceph-mon[36835]: pgmap v3567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:26:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5839.488885 secs 2024-07-27T08:27:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:26:59 smithi077 ceph-mon[37153]: pgmap v3568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:27:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:26:59 smithi047 ceph-mon[36835]: pgmap v3568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:27:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:26:59 smithi029 ceph-mon[32610]: pgmap v3568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:27:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:01 smithi077 ceph-mon[37153]: pgmap v3569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:27:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:01 smithi047 ceph-mon[36835]: pgmap v3569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:27:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:01 smithi029 ceph-mon[32610]: pgmap v3569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:27:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5844.489015 secs 2024-07-27T08:27:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5844.489015 secs 2024-07-27T08:27:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5844.489015 secs 2024-07-27T08:27:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:03 smithi077 ceph-mon[37153]: pgmap v3570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:27:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:03 smithi047 ceph-mon[36835]: pgmap v3570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:27:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:03 smithi029 ceph-mon[32610]: pgmap v3570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:27:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:05 smithi077 ceph-mon[37153]: pgmap v3571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:27:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:27:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:05 smithi047 ceph-mon[36835]: pgmap v3571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:27:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:27:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:05 smithi029 ceph-mon[32610]: pgmap v3571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:27:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:27:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:08 smithi077 ceph-mon[37153]: pgmap v3572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5849.489136 secs 2024-07-27T08:27:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:27:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:27:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:08 smithi077 ceph-mon[37153]: pgmap v3573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:27:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:08 smithi077 ceph-mon[37153]: pgmap v3574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:27:08.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:08 smithi047 ceph-mon[36835]: pgmap v3572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5849.489136 secs 2024-07-27T08:27:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:27:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:27:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:08 smithi047 ceph-mon[36835]: pgmap v3573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:27:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:08 smithi047 ceph-mon[36835]: pgmap v3574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:27:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:08 smithi029 ceph-mon[32610]: pgmap v3572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5849.489136 secs 2024-07-27T08:27:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:27:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:27:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:08 smithi029 ceph-mon[32610]: pgmap v3573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:27:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:08 smithi029 ceph-mon[32610]: pgmap v3574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:27:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:27:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:10 smithi047 ceph-mon[36835]: pgmap v3575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:27:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:10 smithi029 ceph-mon[32610]: pgmap v3575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:27:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:10 smithi077 ceph-mon[37153]: pgmap v3575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:27:12.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:27:12.607+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:27:12.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:27:12.607+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:27:12.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:27:12.607+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:27:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:27:12.617+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:27:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:27:12.618+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:27:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:27:12.618+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:27:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:12 smithi047 ceph-mon[36835]: pgmap v3576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:27:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:12 smithi029 ceph-mon[32610]: pgmap v3576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:27:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:12 smithi077 ceph-mon[37153]: pgmap v3576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:27:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5854.489226 secs 2024-07-27T08:27:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5854.489226 secs 2024-07-27T08:27:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5854.489226 secs 2024-07-27T08:27:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:14 smithi047 ceph-mon[36835]: pgmap v3577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:27:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:14 smithi029 ceph-mon[32610]: pgmap v3577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:27:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:14 smithi077 ceph-mon[37153]: pgmap v3577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:27:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:16 smithi047 ceph-mon[36835]: pgmap v3578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:27:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:16 smithi029 ceph-mon[32610]: pgmap v3578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:27:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:16 smithi077 ceph-mon[37153]: pgmap v3578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:27:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5859.489358 secs 2024-07-27T08:27:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:18 smithi047 ceph-mon[36835]: pgmap v3579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:27:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5859.489358 secs 2024-07-27T08:27:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:18 smithi029 ceph-mon[32610]: pgmap v3579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:27:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5859.489358 secs 2024-07-27T08:27:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:18 smithi077 ceph-mon[37153]: pgmap v3579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:27:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:20 smithi047 ceph-mon[36835]: pgmap v3580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:27:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:20 smithi029 ceph-mon[32610]: pgmap v3580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:27:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:20 smithi077 ceph-mon[37153]: pgmap v3580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:27:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:22 smithi047 ceph-mon[36835]: pgmap v3581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:22 smithi029 ceph-mon[32610]: pgmap v3581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:22 smithi077 ceph-mon[37153]: pgmap v3581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5864.489485 secs 2024-07-27T08:27:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5864.489485 secs 2024-07-27T08:27:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5864.489485 secs 2024-07-27T08:27:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:24 smithi047 ceph-mon[36835]: pgmap v3582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:27:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:24 smithi029 ceph-mon[32610]: pgmap v3582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:27:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:24 smithi077 ceph-mon[37153]: pgmap v3582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:27:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:26 smithi029 ceph-mon[32610]: pgmap v3583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:27:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:26 smithi077 ceph-mon[37153]: pgmap v3583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:27:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:26 smithi047 ceph-mon[36835]: pgmap v3583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:27:27.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:27:27.621+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:27:27.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:27:27.621+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:27:27.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:27:27.621+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:27:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:27:27.624+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:27:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:27:27.624+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:27:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:27:27.624+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:27:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5869.489609 secs 2024-07-27T08:27:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:28 smithi029 ceph-mon[32610]: pgmap v3584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:27:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5869.489609 secs 2024-07-27T08:27:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:28 smithi077 ceph-mon[37153]: pgmap v3584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:27:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5869.489609 secs 2024-07-27T08:27:28.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:28 smithi047 ceph-mon[36835]: pgmap v3584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:27:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:30 smithi029 ceph-mon[32610]: pgmap v3585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:27:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:30 smithi077 ceph-mon[37153]: pgmap v3585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:27:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:30 smithi047 ceph-mon[36835]: pgmap v3585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:27:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:32 smithi029 ceph-mon[32610]: pgmap v3586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:32 smithi077 ceph-mon[37153]: pgmap v3586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:32 smithi047 ceph-mon[36835]: pgmap v3586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5874.489732 secs 2024-07-27T08:27:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5874.489732 secs 2024-07-27T08:27:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5874.489732 secs 2024-07-27T08:27:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:34 smithi077 ceph-mon[37153]: pgmap v3587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:27:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:34 smithi047 ceph-mon[36835]: pgmap v3587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:27:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:34 smithi029 ceph-mon[32610]: pgmap v3587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:27:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:36 smithi077 ceph-mon[37153]: pgmap v3588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:27:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:36 smithi047 ceph-mon[36835]: pgmap v3588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:27:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:36 smithi029 ceph-mon[32610]: pgmap v3588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:27:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5879.489857 secs 2024-07-27T08:27:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:38 smithi077 ceph-mon[37153]: pgmap v3589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:27:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5879.489857 secs 2024-07-27T08:27:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:38 smithi047 ceph-mon[36835]: pgmap v3589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:27:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5879.489857 secs 2024-07-27T08:27:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:38 smithi029 ceph-mon[32610]: pgmap v3589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:27:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:40 smithi077 ceph-mon[37153]: pgmap v3590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:27:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:40 smithi047 ceph-mon[36835]: pgmap v3590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:27:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:40 smithi029 ceph-mon[32610]: pgmap v3590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:27:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:27:42.636+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:27:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:27:42.636+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:27:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:27:42.636+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:27:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:27:42.639+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:27:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:27:42.639+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:27:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:27:42.639+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:27:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:42 smithi077 ceph-mon[37153]: pgmap v3591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:42 smithi047 ceph-mon[36835]: pgmap v3591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:42 smithi029 ceph-mon[32610]: pgmap v3591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5884.489980 secs 2024-07-27T08:27:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5884.489980 secs 2024-07-27T08:27:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5884.489980 secs 2024-07-27T08:27:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:44 smithi077 ceph-mon[37153]: pgmap v3592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:27:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:44 smithi047 ceph-mon[36835]: pgmap v3592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:27:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:44 smithi029 ceph-mon[32610]: pgmap v3592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:27:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:46 smithi077 ceph-mon[37153]: pgmap v3593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:27:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:46 smithi047 ceph-mon[36835]: pgmap v3593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:27:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:46 smithi029 ceph-mon[32610]: pgmap v3593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:27:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5889.490099 secs 2024-07-27T08:27:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:48 smithi077 ceph-mon[37153]: pgmap v3594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:27:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5889.490099 secs 2024-07-27T08:27:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:48 smithi047 ceph-mon[36835]: pgmap v3594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:27:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5889.490099 secs 2024-07-27T08:27:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:48 smithi029 ceph-mon[32610]: pgmap v3594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:27:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:50 smithi077 ceph-mon[37153]: pgmap v3595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:27:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:50 smithi047 ceph-mon[36835]: pgmap v3595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:27:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:50 smithi029 ceph-mon[32610]: pgmap v3595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:27:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:52 smithi077 ceph-mon[37153]: pgmap v3596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:52 smithi047 ceph-mon[36835]: pgmap v3596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:52 smithi029 ceph-mon[32610]: pgmap v3596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:27:53.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5894.490224 secs 2024-07-27T08:27:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5894.490224 secs 2024-07-27T08:27:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5894.490224 secs 2024-07-27T08:27:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:54 smithi077 ceph-mon[37153]: pgmap v3597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:27:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:54 smithi047 ceph-mon[36835]: pgmap v3597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:27:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:54 smithi029 ceph-mon[32610]: pgmap v3597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:27:56.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:56 smithi029 ceph-mon[32610]: pgmap v3598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:27:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:56 smithi077 ceph-mon[37153]: pgmap v3598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:27:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:56 smithi047 ceph-mon[36835]: pgmap v3598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:27:57.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:27:57.645+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:27:57.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:27:57.645+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:27:57.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:27:57.645+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:27:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:27:57.654+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:27:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:27:57.654+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:27:57.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:27:57.654+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:27:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5899.490366 secs 2024-07-27T08:27:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:27:58 smithi077 ceph-mon[37153]: pgmap v3599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:27:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5899.490366 secs 2024-07-27T08:27:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:27:58 smithi047 ceph-mon[36835]: pgmap v3599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:27:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5899.490366 secs 2024-07-27T08:27:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:27:58 smithi029 ceph-mon[32610]: pgmap v3599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:28:00.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:00 smithi077 ceph-mon[37153]: pgmap v3600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:28:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:00 smithi047 ceph-mon[36835]: pgmap v3600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:28:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:00 smithi029 ceph-mon[32610]: pgmap v3600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:28:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:02 smithi077 ceph-mon[37153]: pgmap v3601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:02 smithi047 ceph-mon[36835]: pgmap v3601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:02 smithi029 ceph-mon[32610]: pgmap v3601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5904.490488 secs 2024-07-27T08:28:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5904.490488 secs 2024-07-27T08:28:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5904.490488 secs 2024-07-27T08:28:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:04 smithi077 ceph-mon[37153]: pgmap v3602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:04.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:04 smithi047 ceph-mon[36835]: pgmap v3602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:04 smithi029 ceph-mon[32610]: pgmap v3602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:05 smithi077 ceph-mon[37153]: pgmap v3603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:05 smithi047 ceph-mon[36835]: pgmap v3603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:05 smithi029 ceph-mon[32610]: pgmap v3603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5909.490648 secs 2024-07-27T08:28:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:08 smithi047 ceph-mon[36835]: pgmap v3604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:28:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:28:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:28:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:28:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5909.490648 secs 2024-07-27T08:28:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:08 smithi029 ceph-mon[32610]: pgmap v3604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:28:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:28:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:28:08.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:28:08.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5909.490648 secs 2024-07-27T08:28:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:08 smithi077 ceph-mon[37153]: pgmap v3604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:28:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:28:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:08.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:28:08.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:28:08.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:28:09.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:09 smithi047 ceph-mon[36835]: pgmap v3605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:09.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:09 smithi047 ceph-mon[36835]: pgmap v3606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:28:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:09 smithi029 ceph-mon[32610]: pgmap v3605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:09 smithi029 ceph-mon[32610]: pgmap v3606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:28:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:09 smithi077 ceph-mon[37153]: pgmap v3605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:09 smithi077 ceph-mon[37153]: pgmap v3606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:28:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:11 smithi047 ceph-mon[36835]: pgmap v3607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:28:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:11 smithi029 ceph-mon[32610]: pgmap v3607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:28:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:11 smithi077 ceph-mon[37153]: pgmap v3607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:28:12.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:12.654+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:28:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:12.654+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:28:12.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:12.654+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:28:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:12.671+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:28:12.671 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:12.671+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:28:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:12.671+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:28:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5914.490772 secs 2024-07-27T08:28:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:13 smithi047 ceph-mon[36835]: pgmap v3608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T08:28:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5914.490772 secs 2024-07-27T08:28:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:13 smithi029 ceph-mon[32610]: pgmap v3608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T08:28:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5914.490772 secs 2024-07-27T08:28:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:13 smithi077 ceph-mon[37153]: pgmap v3608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T08:28:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:15 smithi047 ceph-mon[36835]: pgmap v3609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:28:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:15 smithi029 ceph-mon[32610]: pgmap v3609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:28:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:15 smithi077 ceph-mon[37153]: pgmap v3609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:28:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:17 smithi029 ceph-mon[32610]: pgmap v3610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:28:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:17 smithi077 ceph-mon[37153]: pgmap v3610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:28:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:17 smithi047 ceph-mon[36835]: pgmap v3610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:28:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5919.490893 secs 2024-07-27T08:28:18.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5919.490893 secs 2024-07-27T08:28:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5919.490893 secs 2024-07-27T08:28:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:19 smithi029 ceph-mon[32610]: pgmap v3611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:28:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:19 smithi077 ceph-mon[37153]: pgmap v3611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:28:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:19 smithi047 ceph-mon[36835]: pgmap v3611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:28:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:21 smithi029 ceph-mon[32610]: pgmap v3612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:21 smithi077 ceph-mon[37153]: pgmap v3612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:21 smithi047 ceph-mon[36835]: pgmap v3612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5924.490986 secs 2024-07-27T08:28:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:23 smithi029 ceph-mon[32610]: pgmap v3613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:23.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5924.490986 secs 2024-07-27T08:28:23.749 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:23 smithi047 ceph-mon[36835]: pgmap v3613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5924.490986 secs 2024-07-27T08:28:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:23 smithi077 ceph-mon[37153]: pgmap v3613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:25 smithi077 ceph-mon[37153]: pgmap v3614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:28:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:25 smithi047 ceph-mon[36835]: pgmap v3614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:28:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:25 smithi029 ceph-mon[32610]: pgmap v3614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:28:27.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:27.669+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:28:27.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:27.669+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:28:27.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:27.669+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:28:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:27.685+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:28:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:27.685+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:28:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:27.685+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:28:27.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:27 smithi029 ceph-mon[32610]: pgmap v3615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:27 smithi077 ceph-mon[37153]: pgmap v3615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:27 smithi047 ceph-mon[36835]: pgmap v3615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5929.491108 secs 2024-07-27T08:28:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5929.491108 secs 2024-07-27T08:28:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5929.491108 secs 2024-07-27T08:28:29.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:29 smithi077 ceph-mon[37153]: pgmap v3616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:29 smithi047 ceph-mon[36835]: pgmap v3616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:29 smithi029 ceph-mon[32610]: pgmap v3616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:31 smithi077 ceph-mon[37153]: pgmap v3617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:31 smithi047 ceph-mon[36835]: pgmap v3617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:31 smithi029 ceph-mon[32610]: pgmap v3617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5934.491232 secs 2024-07-27T08:28:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:33 smithi077 ceph-mon[37153]: pgmap v3618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5934.491232 secs 2024-07-27T08:28:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:33 smithi047 ceph-mon[36835]: pgmap v3618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5934.491232 secs 2024-07-27T08:28:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:33 smithi029 ceph-mon[32610]: pgmap v3618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:35 smithi077 ceph-mon[37153]: pgmap v3619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:28:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:35 smithi047 ceph-mon[36835]: pgmap v3619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:28:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:35 smithi029 ceph-mon[32610]: pgmap v3619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:28:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:37 smithi077 ceph-mon[37153]: pgmap v3620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:37 smithi047 ceph-mon[36835]: pgmap v3620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:37 smithi029 ceph-mon[32610]: pgmap v3620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5939.491295 secs 2024-07-27T08:28:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5939.491295 secs 2024-07-27T08:28:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5939.491295 secs 2024-07-27T08:28:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:39 smithi077 ceph-mon[37153]: pgmap v3621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:39 smithi047 ceph-mon[36835]: pgmap v3621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:39 smithi029 ceph-mon[32610]: pgmap v3621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:41 smithi077 ceph-mon[37153]: pgmap v3622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:41 smithi047 ceph-mon[36835]: pgmap v3622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:41 smithi029 ceph-mon[32610]: pgmap v3622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:42.673+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:28:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:42.673+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:28:42.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:42.673+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:28:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:42.701+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:28:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:42.701+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:28:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:42.701+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:28:43.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:43.527+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:28:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:43.527+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:28:43.529 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:43.527+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T08:28:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:43.527+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T08:28:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:43.527+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T08:28:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:43.527+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:28:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:43.527+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T08:28:43.530 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:43.527+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:28:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5944.491416 secs 2024-07-27T08:28:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:43 smithi077 ceph-mon[37153]: pgmap v3623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:43.944 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5944.491416 secs 2024-07-27T08:28:43.944 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:43 smithi029 ceph-mon[32610]: pgmap v3623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5944.491416 secs 2024-07-27T08:28:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:43 smithi047 ceph-mon[36835]: pgmap v3623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:45 smithi077 ceph-mon[37153]: pgmap v3624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:28:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:45 smithi047 ceph-mon[36835]: pgmap v3624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:28:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:45 smithi029 ceph-mon[32610]: pgmap v3624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:28:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:47 smithi077 ceph-mon[37153]: pgmap v3625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:47 smithi047 ceph-mon[36835]: pgmap v3625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:47 smithi029 ceph-mon[32610]: pgmap v3625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5949.491537 secs 2024-07-27T08:28:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5949.491537 secs 2024-07-27T08:28:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5949.491537 secs 2024-07-27T08:28:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:49 smithi077 ceph-mon[37153]: pgmap v3626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:49 smithi047 ceph-mon[36835]: pgmap v3626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:49 smithi029 ceph-mon[32610]: pgmap v3626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:51 smithi077 ceph-mon[37153]: pgmap v3627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:51 smithi047 ceph-mon[36835]: pgmap v3627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:51 smithi029 ceph-mon[32610]: pgmap v3627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:28:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5954.491660 secs 2024-07-27T08:28:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:53 smithi077 ceph-mon[37153]: pgmap v3628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:28:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5954.491660 secs 2024-07-27T08:28:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:53 smithi047 ceph-mon[36835]: pgmap v3628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:28:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5954.491660 secs 2024-07-27T08:28:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:53 smithi029 ceph-mon[32610]: pgmap v3628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:28:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:55 smithi077 ceph-mon[37153]: pgmap v3629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:28:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:55 smithi047 ceph-mon[36835]: pgmap v3629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:28:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:55 smithi029 ceph-mon[32610]: pgmap v3629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:28:57.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:57.674+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:28:57.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:57.674+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:28:57.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:57.674+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:28:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:57.716+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:28:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:57.716+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:28:57.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:28:57.716+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:28:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:57 smithi077 ceph-mon[37153]: pgmap v3630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:57 smithi047 ceph-mon[36835]: pgmap v3630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:57 smithi029 ceph-mon[32610]: pgmap v3630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:28:58.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:58.528+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:28:58.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:58.528+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:28:58.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:58.532+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T08:28:58.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:58.532+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:28:58.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:58.532+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T08:28:58.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:58.532+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:28:58.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:58.532+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:28:58.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:28:58.536+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:28:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5959.491816 secs 2024-07-27T08:28:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5959.491816 secs 2024-07-27T08:28:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5959.491816 secs 2024-07-27T08:28:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:28:59 smithi029 ceph-mon[32610]: pgmap v3631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:28:59 smithi077 ceph-mon[37153]: pgmap v3631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:28:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:28:59 smithi047 ceph-mon[36835]: pgmap v3631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:00 smithi077 ceph-mon[37153]: pgmap v3632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:00 smithi047 ceph-mon[36835]: pgmap v3632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:00 smithi029 ceph-mon[32610]: pgmap v3632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5964.491959 secs 2024-07-27T08:29:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:03 smithi077 ceph-mon[37153]: pgmap v3633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:29:03.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5964.491959 secs 2024-07-27T08:29:03.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:03 smithi047 ceph-mon[36835]: pgmap v3633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:29:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5964.491959 secs 2024-07-27T08:29:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:03 smithi029 ceph-mon[32610]: pgmap v3633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:29:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:05 smithi077 ceph-mon[37153]: pgmap v3634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:29:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:05 smithi047 ceph-mon[36835]: pgmap v3634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:29:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:05 smithi029 ceph-mon[32610]: pgmap v3634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:29:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:07 smithi077 ceph-mon[37153]: pgmap v3635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:29:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:07 smithi047 ceph-mon[36835]: pgmap v3635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:29:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:07 smithi029 ceph-mon[32610]: pgmap v3635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:29:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5969.492069 secs 2024-07-27T08:29:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5969.492069 secs 2024-07-27T08:29:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5969.492069 secs 2024-07-27T08:29:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:09 smithi077 ceph-mon[37153]: pgmap v3636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:29:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:29:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:29:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:29:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:09 smithi047 ceph-mon[36835]: pgmap v3636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:29:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:29:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:29:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:29:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:09 smithi029 ceph-mon[32610]: pgmap v3636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:29:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:29:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:29:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:29:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:10 smithi077 ceph-mon[37153]: pgmap v3637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:29:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:10 smithi077 ceph-mon[37153]: pgmap v3638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:29:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:10 smithi047 ceph-mon[36835]: pgmap v3637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:29:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:10 smithi047 ceph-mon[36835]: pgmap v3638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:29:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:10 smithi029 ceph-mon[32610]: pgmap v3637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:29:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:10 smithi029 ceph-mon[32610]: pgmap v3638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:29:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:12 smithi077 ceph-mon[37153]: pgmap v3639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:29:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:29:12.690+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:29:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:29:12.690+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:29:12.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:29:12.690+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:29:12.692 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:12 smithi029 ceph-mon[32610]: pgmap v3639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:29:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:12 smithi047 ceph-mon[36835]: pgmap v3639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:29:12.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:29:12.718+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:29:12.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:29:12.718+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:29:12.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:29:12.718+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:29:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5974.492191 secs 2024-07-27T08:29:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5974.492191 secs 2024-07-27T08:29:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5974.492191 secs 2024-07-27T08:29:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:14 smithi077 ceph-mon[37153]: pgmap v3640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:29:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:14 smithi047 ceph-mon[36835]: pgmap v3640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:29:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:14 smithi029 ceph-mon[32610]: pgmap v3640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:29:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:16 smithi077 ceph-mon[37153]: pgmap v3641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:29:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:16 smithi047 ceph-mon[36835]: pgmap v3641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:29:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:16 smithi029 ceph-mon[32610]: pgmap v3641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:29:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:18 smithi077 ceph-mon[37153]: pgmap v3642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:29:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5979.492312 secs 2024-07-27T08:29:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:18 smithi047 ceph-mon[36835]: pgmap v3642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:29:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5979.492312 secs 2024-07-27T08:29:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:18 smithi029 ceph-mon[32610]: pgmap v3642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:29:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5979.492312 secs 2024-07-27T08:29:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:20 smithi077 ceph-mon[37153]: pgmap v3643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:29:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:20 smithi047 ceph-mon[36835]: pgmap v3643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:29:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:20 smithi029 ceph-mon[32610]: pgmap v3643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:29:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:22 smithi077 ceph-mon[37153]: pgmap v3644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:22 smithi047 ceph-mon[36835]: pgmap v3644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:22 smithi029 ceph-mon[32610]: pgmap v3644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:23.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5984.492395 secs 2024-07-27T08:29:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5984.492395 secs 2024-07-27T08:29:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5984.492395 secs 2024-07-27T08:29:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:24 smithi077 ceph-mon[37153]: pgmap v3645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:29:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:24 smithi047 ceph-mon[36835]: pgmap v3645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:29:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:24 smithi029 ceph-mon[32610]: pgmap v3645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:29:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:26 smithi077 ceph-mon[37153]: pgmap v3646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:29:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:26 smithi047 ceph-mon[36835]: pgmap v3646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:29:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:26 smithi029 ceph-mon[32610]: pgmap v3646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:29:27.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:29:27.696+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:29:27.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:29:27.696+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:29:27.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:29:27.696+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:29:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:29:27.732+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:29:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:29:27.732+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:29:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:29:27.732+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:29:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:28 smithi077 ceph-mon[37153]: pgmap v3647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:29:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5989.492472 secs 2024-07-27T08:29:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:28 smithi047 ceph-mon[36835]: pgmap v3647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:29:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5989.492472 secs 2024-07-27T08:29:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:28 smithi029 ceph-mon[32610]: pgmap v3647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:29:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5989.492472 secs 2024-07-27T08:29:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:30 smithi077 ceph-mon[37153]: pgmap v3648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:30 smithi047 ceph-mon[36835]: pgmap v3648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:30 smithi029 ceph-mon[32610]: pgmap v3648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:32 smithi077 ceph-mon[37153]: pgmap v3649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:32 smithi047 ceph-mon[36835]: pgmap v3649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:32 smithi029 ceph-mon[32610]: pgmap v3649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5994.492593 secs 2024-07-27T08:29:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5994.492593 secs 2024-07-27T08:29:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5994.492593 secs 2024-07-27T08:29:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:34 smithi077 ceph-mon[37153]: pgmap v3650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:29:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:34 smithi047 ceph-mon[36835]: pgmap v3650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:29:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:34 smithi029 ceph-mon[32610]: pgmap v3650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:29:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:36 smithi047 ceph-mon[36835]: pgmap v3651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:36 smithi029 ceph-mon[32610]: pgmap v3651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:36 smithi077 ceph-mon[37153]: pgmap v3651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:38 smithi047 ceph-mon[36835]: pgmap v3652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:29:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 5999.492686 secs 2024-07-27T08:29:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:38 smithi029 ceph-mon[32610]: pgmap v3652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:29:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 5999.492686 secs 2024-07-27T08:29:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:38 smithi077 ceph-mon[37153]: pgmap v3652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:29:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 5999.492686 secs 2024-07-27T08:29:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:40 smithi047 ceph-mon[36835]: pgmap v3653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:40 smithi029 ceph-mon[32610]: pgmap v3653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:40 smithi077 ceph-mon[37153]: pgmap v3653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:42 smithi047 ceph-mon[36835]: pgmap v3654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:42 smithi029 ceph-mon[32610]: pgmap v3654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:42.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:29:42.711+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:29:42.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:29:42.711+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:29:42.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:29:42.711+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:29:42.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:29:42.747+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:29:42.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:29:42.747+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:29:42.749 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:29:42.747+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:29:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:42 smithi077 ceph-mon[37153]: pgmap v3654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6004.492814 secs 2024-07-27T08:29:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6004.492814 secs 2024-07-27T08:29:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6004.492814 secs 2024-07-27T08:29:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:44 smithi047 ceph-mon[36835]: pgmap v3655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:29:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:44 smithi029 ceph-mon[32610]: pgmap v3655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:29:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:44 smithi077 ceph-mon[37153]: pgmap v3655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:29:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:46 smithi047 ceph-mon[36835]: pgmap v3656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:46 smithi029 ceph-mon[32610]: pgmap v3656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:46 smithi077 ceph-mon[37153]: pgmap v3656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:48.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:48 smithi077 ceph-mon[37153]: pgmap v3657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:29:48.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6009.492933 secs 2024-07-27T08:29:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:48 smithi047 ceph-mon[36835]: pgmap v3657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:29:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6009.492933 secs 2024-07-27T08:29:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:48 smithi029 ceph-mon[32610]: pgmap v3657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:29:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6009.492933 secs 2024-07-27T08:29:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:50 smithi047 ceph-mon[36835]: pgmap v3658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:50 smithi029 ceph-mon[32610]: pgmap v3658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:50 smithi077 ceph-mon[37153]: pgmap v3658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:29:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:52 smithi047 ceph-mon[36835]: pgmap v3659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:52 smithi029 ceph-mon[32610]: pgmap v3659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:52 smithi077 ceph-mon[37153]: pgmap v3659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6014.493062 secs 2024-07-27T08:29:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6014.493062 secs 2024-07-27T08:29:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6014.493062 secs 2024-07-27T08:29:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:54 smithi047 ceph-mon[36835]: pgmap v3660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:29:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:54 smithi029 ceph-mon[32610]: pgmap v3660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:29:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:54 smithi077 ceph-mon[37153]: pgmap v3660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:29:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:56 smithi047 ceph-mon[36835]: pgmap v3661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:56 smithi029 ceph-mon[32610]: pgmap v3661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:56 smithi077 ceph-mon[37153]: pgmap v3661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:29:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:29:57.727+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:29:57.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:29:57.727+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:29:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:29:57.727+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:29:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:29:57.762+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:29:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:29:57.762+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:29:57.764 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:29:57.762+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:29:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:58 smithi047 ceph-mon[36835]: pgmap v3662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:29:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:29:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6019.493183 secs 2024-07-27T08:29:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:58 smithi029 ceph-mon[32610]: pgmap v3662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:29:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:29:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6019.493183 secs 2024-07-27T08:29:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:58 smithi077 ceph-mon[37153]: pgmap v3662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:29:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:29:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6019.493183 secs 2024-07-27T08:30:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:00 smithi047 ceph-mon[36835]: pgmap v3663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:00.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:30:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:00 smithi029 ceph-mon[32610]: pgmap v3663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:30:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:00 smithi077 ceph-mon[37153]: pgmap v3663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:30:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:02 smithi047 ceph-mon[36835]: pgmap v3664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:02 smithi029 ceph-mon[32610]: pgmap v3664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:02 smithi077 ceph-mon[37153]: pgmap v3664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6024.493284 secs 2024-07-27T08:30:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6024.493284 secs 2024-07-27T08:30:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6024.493284 secs 2024-07-27T08:30:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:04 smithi047 ceph-mon[36835]: pgmap v3665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:30:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:04 smithi029 ceph-mon[32610]: pgmap v3665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:30:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:04 smithi077 ceph-mon[37153]: pgmap v3665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:30:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:06 smithi077 ceph-mon[37153]: pgmap v3666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:06 smithi047 ceph-mon[36835]: pgmap v3666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:06 smithi029 ceph-mon[32610]: pgmap v3666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:08 smithi077 ceph-mon[37153]: pgmap v3667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:30:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6029.493406 secs 2024-07-27T08:30:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:08 smithi047 ceph-mon[36835]: pgmap v3667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:30:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6029.493406 secs 2024-07-27T08:30:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:08 smithi029 ceph-mon[32610]: pgmap v3667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:30:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6029.493406 secs 2024-07-27T08:30:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:30:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:30:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:30:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:30:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:30:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:30:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:30:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:30:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:30:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:30:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:30:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:30:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:10 smithi077 ceph-mon[37153]: pgmap v3668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:10 smithi077 ceph-mon[37153]: pgmap v3669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:30:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:10 smithi077 ceph-mon[37153]: pgmap v3670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:30:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:10 smithi047 ceph-mon[36835]: pgmap v3668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:10 smithi047 ceph-mon[36835]: pgmap v3669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:30:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:10 smithi047 ceph-mon[36835]: pgmap v3670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:30:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:10 smithi029 ceph-mon[32610]: pgmap v3668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:10 smithi029 ceph-mon[32610]: pgmap v3669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:30:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:10 smithi029 ceph-mon[32610]: pgmap v3670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:30:12.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:30:12.742+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:30:12.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:30:12.743+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:30:12.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:30:12.743+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:30:12.744 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:12 smithi029 ceph-mon[32610]: pgmap v3671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:30:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:30:12.768+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:30:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:30:12.768+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:30:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:30:12.768+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:30:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:12 smithi077 ceph-mon[37153]: pgmap v3671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:30:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:12 smithi047 ceph-mon[36835]: pgmap v3671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:30:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6034.493502 secs 2024-07-27T08:30:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6034.493502 secs 2024-07-27T08:30:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6034.493502 secs 2024-07-27T08:30:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:14 smithi077 ceph-mon[37153]: pgmap v3672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:30:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:14 smithi047 ceph-mon[36835]: pgmap v3672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:30:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:14 smithi029 ceph-mon[32610]: pgmap v3672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:30:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:16 smithi077 ceph-mon[37153]: pgmap v3673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:30:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:16 smithi047 ceph-mon[36835]: pgmap v3673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:30:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:16 smithi029 ceph-mon[32610]: pgmap v3673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:30:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6039.493625 secs 2024-07-27T08:30:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:18 smithi077 ceph-mon[37153]: pgmap v3674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:30:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6039.493625 secs 2024-07-27T08:30:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:18 smithi047 ceph-mon[36835]: pgmap v3674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:30:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6039.493625 secs 2024-07-27T08:30:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:18 smithi029 ceph-mon[32610]: pgmap v3674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:30:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:20 smithi077 ceph-mon[37153]: pgmap v3675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:30:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:20 smithi047 ceph-mon[36835]: pgmap v3675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:30:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:20 smithi029 ceph-mon[32610]: pgmap v3675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:30:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:22 smithi077 ceph-mon[37153]: pgmap v3676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:30:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:22 smithi047 ceph-mon[36835]: pgmap v3676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:30:22.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:22 smithi029 ceph-mon[32610]: pgmap v3676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:30:23.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6044.493762 secs 2024-07-27T08:30:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6044.493762 secs 2024-07-27T08:30:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6044.493762 secs 2024-07-27T08:30:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:24 smithi077 ceph-mon[37153]: pgmap v3677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:24.864 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:24 smithi047 ceph-mon[36835]: pgmap v3677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:24 smithi029 ceph-mon[32610]: pgmap v3677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:26 smithi077 ceph-mon[37153]: pgmap v3678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:30:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:26 smithi047 ceph-mon[36835]: pgmap v3678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:30:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:26 smithi029 ceph-mon[32610]: pgmap v3678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:30:27.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:30:27.745+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:30:27.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:30:27.745+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:30:27.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:30:27.745+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:30:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:30:27.783+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:30:27.784 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:30:27.783+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:30:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:30:27.783+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:30:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6049.493912 secs 2024-07-27T08:30:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:28 smithi077 ceph-mon[37153]: pgmap v3679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:30:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6049.493912 secs 2024-07-27T08:30:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:28 smithi047 ceph-mon[36835]: pgmap v3679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:30:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6049.493912 secs 2024-07-27T08:30:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:28 smithi029 ceph-mon[32610]: pgmap v3679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:30:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:30 smithi077 ceph-mon[37153]: pgmap v3680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:30:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:30 smithi047 ceph-mon[36835]: pgmap v3680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:30:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:30 smithi029 ceph-mon[32610]: pgmap v3680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:30:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:32 smithi077 ceph-mon[37153]: pgmap v3681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:30:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:32 smithi047 ceph-mon[36835]: pgmap v3681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:30:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:32 smithi029 ceph-mon[32610]: pgmap v3681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:30:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6054.494033 secs 2024-07-27T08:30:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6054.494033 secs 2024-07-27T08:30:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6054.494033 secs 2024-07-27T08:30:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:34 smithi077 ceph-mon[37153]: pgmap v3682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:34 smithi047 ceph-mon[36835]: pgmap v3682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:34 smithi029 ceph-mon[32610]: pgmap v3682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:36 smithi077 ceph-mon[37153]: pgmap v3683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:36 smithi047 ceph-mon[36835]: pgmap v3683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:36 smithi029 ceph-mon[32610]: pgmap v3683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:37 smithi077 ceph-mon[37153]: pgmap v3684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:30:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:37 smithi047 ceph-mon[36835]: pgmap v3684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:30:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:37 smithi029 ceph-mon[32610]: pgmap v3684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:30:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6059.494154 secs 2024-07-27T08:30:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6059.494154 secs 2024-07-27T08:30:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6059.494154 secs 2024-07-27T08:30:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:39 smithi077 ceph-mon[37153]: pgmap v3685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:30:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:39 smithi047 ceph-mon[36835]: pgmap v3685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:30:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:39 smithi029 ceph-mon[32610]: pgmap v3685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:30:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:42 smithi047 ceph-mon[36835]: pgmap v3686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:30:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:42 smithi029 ceph-mon[32610]: pgmap v3686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:30:42.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:30:42.761+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:30:42.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:30:42.761+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:30:42.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:30:42.761+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:30:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:30:42.794+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:30:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:30:42.794+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:30:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:30:42.794+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:30:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:42 smithi077 ceph-mon[37153]: pgmap v3686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:30:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6064.494296 secs 2024-07-27T08:30:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6064.494296 secs 2024-07-27T08:30:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6064.494296 secs 2024-07-27T08:30:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:44 smithi047 ceph-mon[36835]: pgmap v3687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:44 smithi029 ceph-mon[32610]: pgmap v3687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:44 smithi077 ceph-mon[37153]: pgmap v3687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:46 smithi047 ceph-mon[36835]: pgmap v3688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:46 smithi029 ceph-mon[32610]: pgmap v3688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:46 smithi077 ceph-mon[37153]: pgmap v3688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:48.658 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6069.494420 secs 2024-07-27T08:30:48.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:48 smithi077 ceph-mon[37153]: pgmap v3689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:30:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6069.494420 secs 2024-07-27T08:30:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:48 smithi047 ceph-mon[36835]: pgmap v3689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:30:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6069.494420 secs 2024-07-27T08:30:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:48 smithi029 ceph-mon[32610]: pgmap v3689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:30:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:50 smithi047 ceph-mon[36835]: pgmap v3690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:30:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:50 smithi029 ceph-mon[32610]: pgmap v3690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:30:50.851 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:50 smithi077 ceph-mon[37153]: pgmap v3690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:30:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:52 smithi047 ceph-mon[36835]: pgmap v3691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:30:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:52 smithi029 ceph-mon[32610]: pgmap v3691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:30:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:52 smithi077 ceph-mon[37153]: pgmap v3691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:30:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6074.494545 secs 2024-07-27T08:30:53.756 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6074.494545 secs 2024-07-27T08:30:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6074.494545 secs 2024-07-27T08:30:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:54 smithi047 ceph-mon[36835]: pgmap v3692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:54 smithi029 ceph-mon[32610]: pgmap v3692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:54.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:54 smithi077 ceph-mon[37153]: pgmap v3692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:30:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:56 smithi047 ceph-mon[36835]: pgmap v3693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:56 smithi029 ceph-mon[32610]: pgmap v3693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:56 smithi077 ceph-mon[37153]: pgmap v3693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:30:57.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:30:57.776+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:30:57.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:30:57.776+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:30:57.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:30:57.776+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:30:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:30:57.805+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:30:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:30:57.805+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:30:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:30:57.805+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:30:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6079.494673 secs 2024-07-27T08:30:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:30:58 smithi047 ceph-mon[36835]: pgmap v3694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:30:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6079.494673 secs 2024-07-27T08:30:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:30:58 smithi029 ceph-mon[32610]: pgmap v3694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:30:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6079.494673 secs 2024-07-27T08:30:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:30:58 smithi077 ceph-mon[37153]: pgmap v3694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:31:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:00 smithi047 ceph-mon[36835]: pgmap v3695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:31:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:00 smithi029 ceph-mon[32610]: pgmap v3695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:31:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:00 smithi077 ceph-mon[37153]: pgmap v3695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:31:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:02 smithi047 ceph-mon[36835]: pgmap v3696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:31:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:02 smithi029 ceph-mon[32610]: pgmap v3696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:31:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:02 smithi077 ceph-mon[37153]: pgmap v3696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:31:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6084.494769 secs 2024-07-27T08:31:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6084.494769 secs 2024-07-27T08:31:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6084.494769 secs 2024-07-27T08:31:04.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:04 smithi029 ceph-mon[32610]: pgmap v3697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:04 smithi047 ceph-mon[36835]: pgmap v3697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:04 smithi077 ceph-mon[37153]: pgmap v3697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:06 smithi047 ceph-mon[36835]: pgmap v3698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:06 smithi029 ceph-mon[32610]: pgmap v3698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:06 smithi077 ceph-mon[37153]: pgmap v3698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6089.494870 secs 2024-07-27T08:31:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:08 smithi047 ceph-mon[36835]: pgmap v3699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:31:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6089.494870 secs 2024-07-27T08:31:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:08 smithi029 ceph-mon[32610]: pgmap v3699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:31:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6089.494870 secs 2024-07-27T08:31:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:08 smithi077 ceph-mon[37153]: pgmap v3699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:31:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:31:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:31:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:31:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:10 smithi029 ceph-mon[32610]: pgmap v3700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:31:10.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:31:10.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:31:10.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:31:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:10 smithi077 ceph-mon[37153]: pgmap v3700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:31:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:31:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:31:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:31:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:10 smithi047 ceph-mon[36835]: pgmap v3700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:31:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:31:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:31:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:31:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:11 smithi029 ceph-mon[32610]: pgmap v3701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:31:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:11 smithi029 ceph-mon[32610]: pgmap v3702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:31:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:11 smithi077 ceph-mon[37153]: pgmap v3701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:31:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:11 smithi077 ceph-mon[37153]: pgmap v3702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:31:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:11 smithi047 ceph-mon[36835]: pgmap v3701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:31:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:11 smithi047 ceph-mon[36835]: pgmap v3702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:31:12.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:12.792+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:31:12.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:12.792+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:31:12.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:12.792+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:31:12.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:12.821+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:31:12.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:12.821+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:31:12.822 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:12.821+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:31:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6094.494973 secs 2024-07-27T08:31:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:13 smithi029 ceph-mon[32610]: pgmap v3703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:31:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6094.494973 secs 2024-07-27T08:31:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:13 smithi077 ceph-mon[37153]: pgmap v3703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:31:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6094.494973 secs 2024-07-27T08:31:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:13 smithi047 ceph-mon[36835]: pgmap v3703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:31:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:15 smithi077 ceph-mon[37153]: pgmap v3704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:31:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:15 smithi047 ceph-mon[36835]: pgmap v3704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:31:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:15 smithi029 ceph-mon[32610]: pgmap v3704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:31:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:17 smithi077 ceph-mon[37153]: pgmap v3705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:31:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:17 smithi047 ceph-mon[36835]: pgmap v3705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:31:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:17 smithi029 ceph-mon[32610]: pgmap v3705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:31:18.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6099.495079 secs 2024-07-27T08:31:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6099.495079 secs 2024-07-27T08:31:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6099.495079 secs 2024-07-27T08:31:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:19 smithi077 ceph-mon[37153]: pgmap v3706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:31:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:19 smithi047 ceph-mon[36835]: pgmap v3706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:31:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:19 smithi029 ceph-mon[32610]: pgmap v3706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:31:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:21 smithi077 ceph-mon[37153]: pgmap v3707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:31:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:21 smithi047 ceph-mon[36835]: pgmap v3707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:31:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:21 smithi029 ceph-mon[32610]: pgmap v3707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:31:23.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6104.495205 secs 2024-07-27T08:31:23.749 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:23 smithi047 ceph-mon[36835]: pgmap v3708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:31:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6104.495205 secs 2024-07-27T08:31:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:23 smithi077 ceph-mon[37153]: pgmap v3708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:31:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6104.495205 secs 2024-07-27T08:31:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:23 smithi029 ceph-mon[32610]: pgmap v3708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:31:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:25 smithi077 ceph-mon[37153]: pgmap v3709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:25 smithi047 ceph-mon[36835]: pgmap v3709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:25 smithi029 ceph-mon[32610]: pgmap v3709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:27.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:27.806+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:31:27.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:27.806+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:31:27.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:27.806+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:31:27.809 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:27 smithi029 ceph-mon[32610]: pgmap v3710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:27.836+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:31:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:27.836+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:31:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:27.836+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:31:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:27 smithi077 ceph-mon[37153]: pgmap v3710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:27 smithi047 ceph-mon[36835]: pgmap v3710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6109.495326 secs 2024-07-27T08:31:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6109.495326 secs 2024-07-27T08:31:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6109.495326 secs 2024-07-27T08:31:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:29 smithi077 ceph-mon[37153]: pgmap v3711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:31:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:29 smithi047 ceph-mon[36835]: pgmap v3711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:31:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:29 smithi029 ceph-mon[32610]: pgmap v3711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:31:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:31 smithi077 ceph-mon[37153]: pgmap v3712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:31 smithi047 ceph-mon[36835]: pgmap v3712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:31 smithi029 ceph-mon[32610]: pgmap v3712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6114.495448 secs 2024-07-27T08:31:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:33 smithi077 ceph-mon[37153]: pgmap v3713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:31:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6114.495448 secs 2024-07-27T08:31:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:33 smithi047 ceph-mon[36835]: pgmap v3713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:31:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6114.495448 secs 2024-07-27T08:31:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:33 smithi029 ceph-mon[32610]: pgmap v3713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:31:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:35 smithi077 ceph-mon[37153]: pgmap v3714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:35 smithi047 ceph-mon[36835]: pgmap v3714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:35 smithi029 ceph-mon[32610]: pgmap v3714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:37 smithi077 ceph-mon[37153]: pgmap v3715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:37 smithi047 ceph-mon[36835]: pgmap v3715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:37 smithi029 ceph-mon[32610]: pgmap v3715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6119.495579 secs 2024-07-27T08:31:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6119.495579 secs 2024-07-27T08:31:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6119.495579 secs 2024-07-27T08:31:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:39 smithi077 ceph-mon[37153]: pgmap v3716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:31:39.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:39 smithi047 ceph-mon[36835]: pgmap v3716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:31:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:39 smithi029 ceph-mon[32610]: pgmap v3716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:31:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:41 smithi077 ceph-mon[37153]: pgmap v3717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:41 smithi047 ceph-mon[36835]: pgmap v3717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:41 smithi029 ceph-mon[32610]: pgmap v3717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:42.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:42.809+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:31:42.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:42.809+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:31:42.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:42.809+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:31:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:42.851+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:31:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:42.851+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:31:42.853 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:42.851+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:31:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:43.554+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T08:31:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:43.554+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:31:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:43.554+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T08:31:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:43.554+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:31:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6124.495701 secs 2024-07-27T08:31:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:43 smithi077 ceph-mon[37153]: pgmap v3718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:31:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6124.495701 secs 2024-07-27T08:31:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:43 smithi047 ceph-mon[36835]: pgmap v3718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:31:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6124.495701 secs 2024-07-27T08:31:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:43 smithi029 ceph-mon[32610]: pgmap v3718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:31:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:45 smithi077 ceph-mon[37153]: pgmap v3719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:45 smithi047 ceph-mon[36835]: pgmap v3719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:45 smithi029 ceph-mon[32610]: pgmap v3719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:47 smithi077 ceph-mon[37153]: pgmap v3720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:47 smithi047 ceph-mon[36835]: pgmap v3720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:47 smithi029 ceph-mon[32610]: pgmap v3720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6129.495801 secs 2024-07-27T08:31:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6129.495801 secs 2024-07-27T08:31:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6129.495801 secs 2024-07-27T08:31:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:49 smithi077 ceph-mon[37153]: pgmap v3721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:31:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:49 smithi047 ceph-mon[36835]: pgmap v3721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:31:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:49 smithi029 ceph-mon[32610]: pgmap v3721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:31:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:51 smithi077 ceph-mon[37153]: pgmap v3722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:51 smithi047 ceph-mon[36835]: pgmap v3722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:51 smithi029 ceph-mon[32610]: pgmap v3722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6134.495899 secs 2024-07-27T08:31:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:53 smithi077 ceph-mon[37153]: pgmap v3723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:31:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6134.495899 secs 2024-07-27T08:31:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:53 smithi047 ceph-mon[36835]: pgmap v3723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:31:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6134.495899 secs 2024-07-27T08:31:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:53 smithi029 ceph-mon[32610]: pgmap v3723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:31:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:55 smithi077 ceph-mon[37153]: pgmap v3724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:55 smithi047 ceph-mon[36835]: pgmap v3724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:55 smithi029 ceph-mon[32610]: pgmap v3724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:31:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:56 smithi077 ceph-mon[37153]: pgmap v3725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:56 smithi047 ceph-mon[36835]: pgmap v3725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:56 smithi029 ceph-mon[32610]: pgmap v3725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:31:57.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:57.825+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:31:57.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:57.825+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:31:57.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:57.825+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:31:57.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:57.853+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:31:57.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:57.853+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:31:57.855 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:31:57.853+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:31:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6139.496019 secs 2024-07-27T08:31:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6139.496019 secs 2024-07-27T08:31:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6139.496019 secs 2024-07-27T08:31:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:58.557+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:31:58.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:58.557+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:31:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:58.557+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:31:58.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:31:58.557+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:31:58.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:31:58 smithi029 ceph-mon[32610]: pgmap v3726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:31:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:31:58 smithi047 ceph-mon[36835]: pgmap v3726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:31:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:31:58 smithi077 ceph-mon[37153]: pgmap v3726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:32:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:00 smithi029 ceph-mon[32610]: pgmap v3727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:00 smithi077 ceph-mon[37153]: pgmap v3727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:00 smithi047 ceph-mon[36835]: pgmap v3727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6144.496162 secs 2024-07-27T08:32:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:02 smithi029 ceph-mon[32610]: pgmap v3728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:32:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6144.496162 secs 2024-07-27T08:32:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:02 smithi077 ceph-mon[37153]: pgmap v3728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:32:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6144.496162 secs 2024-07-27T08:32:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:02 smithi047 ceph-mon[36835]: pgmap v3728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:32:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:04 smithi029 ceph-mon[32610]: pgmap v3729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:04 smithi077 ceph-mon[37153]: pgmap v3729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:04 smithi047 ceph-mon[36835]: pgmap v3729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:06 smithi029 ceph-mon[32610]: pgmap v3730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:06 smithi077 ceph-mon[37153]: pgmap v3730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:06 smithi047 ceph-mon[36835]: pgmap v3730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6149.496300 secs 2024-07-27T08:32:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6149.496300 secs 2024-07-27T08:32:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6149.496300 secs 2024-07-27T08:32:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:08 smithi077 ceph-mon[37153]: pgmap v3731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:32:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:08 smithi047 ceph-mon[36835]: pgmap v3731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:32:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:08 smithi029 ceph-mon[32610]: pgmap v3731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:32:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:10 smithi077 ceph-mon[37153]: pgmap v3732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:32:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:32:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:32:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:10 smithi077 ceph-mon[37153]: pgmap v3733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:32:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:10 smithi077 ceph-mon[37153]: pgmap v3734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:32:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:32:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:10 smithi047 ceph-mon[36835]: pgmap v3732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:32:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:32:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:32:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:10 smithi047 ceph-mon[36835]: pgmap v3733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:32:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:10 smithi047 ceph-mon[36835]: pgmap v3734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:32:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:32:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:10 smithi029 ceph-mon[32610]: pgmap v3732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:32:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:32:11.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:32:11.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:10 smithi029 ceph-mon[32610]: pgmap v3733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:32:11.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:10 smithi029 ceph-mon[32610]: pgmap v3734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:32:11.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:32:12.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:32:12.830+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:32:12.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:32:12.830+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:32:12.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:32:12.830+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:32:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:32:12.868+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:32:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:32:12.868+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:32:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:32:12.868+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:32:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6154.496428 secs 2024-07-27T08:32:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:13 smithi077 ceph-mon[37153]: pgmap v3735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:32:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6154.496428 secs 2024-07-27T08:32:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:13 smithi047 ceph-mon[36835]: pgmap v3735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:32:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6154.496428 secs 2024-07-27T08:32:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:13 smithi029 ceph-mon[32610]: pgmap v3735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:32:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:15 smithi077 ceph-mon[37153]: pgmap v3736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:32:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:15 smithi047 ceph-mon[36835]: pgmap v3736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:32:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:15 smithi029 ceph-mon[32610]: pgmap v3736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:32:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:17 smithi077 ceph-mon[37153]: pgmap v3737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:32:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:17 smithi047 ceph-mon[36835]: pgmap v3737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:32:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:17 smithi029 ceph-mon[32610]: pgmap v3737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:32:18.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6159.496528 secs 2024-07-27T08:32:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6159.496528 secs 2024-07-27T08:32:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6159.496528 secs 2024-07-27T08:32:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:19 smithi077 ceph-mon[37153]: pgmap v3738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:32:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:19 smithi047 ceph-mon[36835]: pgmap v3738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:32:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:19 smithi029 ceph-mon[32610]: pgmap v3738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:32:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:21 smithi077 ceph-mon[37153]: pgmap v3739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:32:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:21 smithi047 ceph-mon[36835]: pgmap v3739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:32:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:21 smithi029 ceph-mon[32610]: pgmap v3739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:32:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6164.496629 secs 2024-07-27T08:32:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:23 smithi077 ceph-mon[37153]: pgmap v3740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:32:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6164.496629 secs 2024-07-27T08:32:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:23 smithi047 ceph-mon[36835]: pgmap v3740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:32:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6164.496629 secs 2024-07-27T08:32:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:23 smithi029 ceph-mon[32610]: pgmap v3740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:32:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:25 smithi077 ceph-mon[37153]: pgmap v3741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:32:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:25 smithi047 ceph-mon[36835]: pgmap v3741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:32:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:25 smithi029 ceph-mon[32610]: pgmap v3741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:32:27.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:32:27.844+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:32:27.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:32:27.844+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:32:27.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:32:27.844+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:32:27.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:27 smithi029 ceph-mon[32610]: pgmap v3742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:32:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:27 smithi077 ceph-mon[37153]: pgmap v3742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:32:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:32:27.882+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:32:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:32:27.882+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:32:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:32:27.882+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:32:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:27 smithi047 ceph-mon[36835]: pgmap v3742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:32:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6169.496770 secs 2024-07-27T08:32:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6169.496770 secs 2024-07-27T08:32:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6169.496770 secs 2024-07-27T08:32:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:29 smithi077 ceph-mon[37153]: pgmap v3743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:29 smithi047 ceph-mon[36835]: pgmap v3743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:29 smithi029 ceph-mon[32610]: pgmap v3743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:31 smithi077 ceph-mon[37153]: pgmap v3744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:31 smithi047 ceph-mon[36835]: pgmap v3744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:31 smithi029 ceph-mon[32610]: pgmap v3744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6174.496896 secs 2024-07-27T08:32:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:33 smithi077 ceph-mon[37153]: pgmap v3745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:32:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6174.496896 secs 2024-07-27T08:32:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:33 smithi047 ceph-mon[36835]: pgmap v3745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:32:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6174.496896 secs 2024-07-27T08:32:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:33 smithi029 ceph-mon[32610]: pgmap v3745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:32:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:35 smithi077 ceph-mon[37153]: pgmap v3746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:32:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:35 smithi047 ceph-mon[36835]: pgmap v3746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:32:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:35 smithi029 ceph-mon[32610]: pgmap v3746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:32:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:37 smithi077 ceph-mon[37153]: pgmap v3747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:32:37.928 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:37 smithi029 ceph-mon[32610]: pgmap v3747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:32:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:37 smithi047 ceph-mon[36835]: pgmap v3747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:32:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6179.496962 secs 2024-07-27T08:32:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6179.496962 secs 2024-07-27T08:32:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6179.496962 secs 2024-07-27T08:32:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:39 smithi077 ceph-mon[37153]: pgmap v3748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:39 smithi047 ceph-mon[36835]: pgmap v3748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:39 smithi029 ceph-mon[32610]: pgmap v3748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:41 smithi077 ceph-mon[37153]: pgmap v3749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:41 smithi047 ceph-mon[36835]: pgmap v3749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:41 smithi029 ceph-mon[32610]: pgmap v3749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:32:42.860+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:32:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:32:42.860+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:32:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:32:42.860+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:32:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:32:42.885+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:32:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:32:42.885+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:32:42.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:32:42.885+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:32:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6184.497065 secs 2024-07-27T08:32:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:43 smithi077 ceph-mon[37153]: pgmap v3750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:32:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6184.497065 secs 2024-07-27T08:32:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:43 smithi047 ceph-mon[36835]: pgmap v3750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:32:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6184.497065 secs 2024-07-27T08:32:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:43 smithi029 ceph-mon[32610]: pgmap v3750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:32:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:45 smithi077 ceph-mon[37153]: pgmap v3751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:32:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:45 smithi047 ceph-mon[36835]: pgmap v3751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:32:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:45 smithi029 ceph-mon[32610]: pgmap v3751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:32:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:47 smithi077 ceph-mon[37153]: pgmap v3752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:32:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:47 smithi047 ceph-mon[36835]: pgmap v3752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:32:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:47 smithi029 ceph-mon[32610]: pgmap v3752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:32:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6189.497165 secs 2024-07-27T08:32:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6189.497165 secs 2024-07-27T08:32:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6189.497165 secs 2024-07-27T08:32:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:49 smithi077 ceph-mon[37153]: pgmap v3753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:49 smithi047 ceph-mon[36835]: pgmap v3753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:49 smithi029 ceph-mon[32610]: pgmap v3753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:51 smithi077 ceph-mon[37153]: pgmap v3754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:51 smithi047 ceph-mon[36835]: pgmap v3754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:51 smithi029 ceph-mon[32610]: pgmap v3754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:32:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6194.497290 secs 2024-07-27T08:32:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:53 smithi077 ceph-mon[37153]: pgmap v3755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:32:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6194.497290 secs 2024-07-27T08:32:53.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:53 smithi047 ceph-mon[36835]: pgmap v3755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:32:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6194.497290 secs 2024-07-27T08:32:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:53 smithi029 ceph-mon[32610]: pgmap v3755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:32:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:54 smithi077 ceph-mon[37153]: pgmap v3756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:32:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:54 smithi047 ceph-mon[36835]: pgmap v3756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:32:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:54 smithi029 ceph-mon[32610]: pgmap v3756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:32:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:57 smithi077 ceph-mon[37153]: pgmap v3757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:32:57.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:32:57.876+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:32:57.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:32:57.876+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:32:57.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:32:57.876+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:32:57.877 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:57 smithi029 ceph-mon[32610]: pgmap v3757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:32:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:32:57.901+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:32:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:32:57.901+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:32:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:32:57.901+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:32:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:57 smithi047 ceph-mon[36835]: pgmap v3757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:32:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6199.497411 secs 2024-07-27T08:32:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6199.497411 secs 2024-07-27T08:32:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6199.497411 secs 2024-07-27T08:32:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:32:59 smithi077 ceph-mon[37153]: pgmap v3758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:32:59 smithi047 ceph-mon[36835]: pgmap v3758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:32:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:32:59 smithi029 ceph-mon[32610]: pgmap v3758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:01 smithi077 ceph-mon[37153]: pgmap v3759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:01 smithi047 ceph-mon[36835]: pgmap v3759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:01 smithi029 ceph-mon[32610]: pgmap v3759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6204.497496 secs 2024-07-27T08:33:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:03 smithi077 ceph-mon[37153]: pgmap v3760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:33:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6204.497496 secs 2024-07-27T08:33:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:03 smithi047 ceph-mon[36835]: pgmap v3760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:33:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6204.497496 secs 2024-07-27T08:33:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:03 smithi029 ceph-mon[32610]: pgmap v3760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:33:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:05 smithi077 ceph-mon[37153]: pgmap v3761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:33:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:05 smithi047 ceph-mon[36835]: pgmap v3761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:33:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:05 smithi029 ceph-mon[32610]: pgmap v3761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:33:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:07 smithi077 ceph-mon[37153]: pgmap v3762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:33:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:07 smithi047 ceph-mon[36835]: pgmap v3762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:33:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:07 smithi029 ceph-mon[32610]: pgmap v3762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:33:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6209.497636 secs 2024-07-27T08:33:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6209.497636 secs 2024-07-27T08:33:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6209.497636 secs 2024-07-27T08:33:09.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:09 smithi077 ceph-mon[37153]: pgmap v3763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:09 smithi047 ceph-mon[36835]: pgmap v3763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:09 smithi029 ceph-mon[32610]: pgmap v3763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:11 smithi077 ceph-mon[37153]: pgmap v3764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:33:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:33:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:33:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:33:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:33:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:33:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:33:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:33:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:11 smithi047 ceph-mon[36835]: pgmap v3764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:33:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:33:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:33:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:33:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:33:11.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:33:11.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:33:11.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:33:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:11 smithi029 ceph-mon[32610]: pgmap v3764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:33:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:33:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:33:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:33:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:33:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:33:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:33:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:33:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:12 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:33:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:12 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:33:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:12 smithi077 ceph-mon[37153]: pgmap v3765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:33:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:12 smithi077 ceph-mon[37153]: pgmap v3766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:33:12.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:33:12.891+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:33:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:33:12.891+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:33:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:33:12.891+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:33:12.893 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:12 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:33:12.893 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:12 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:33:12.893 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:12 smithi029 ceph-mon[32610]: pgmap v3765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:33:12.893 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:12 smithi029 ceph-mon[32610]: pgmap v3766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:33:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:33:12.916+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:33:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:33:12.916+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:33:12.917 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:33:12.916+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:33:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:12 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:33:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:12 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:33:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:12 smithi047 ceph-mon[36835]: pgmap v3765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:33:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:12 smithi047 ceph-mon[36835]: pgmap v3766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:33:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6214.497763 secs 2024-07-27T08:33:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6214.497763 secs 2024-07-27T08:33:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6214.497763 secs 2024-07-27T08:33:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:14 smithi077 ceph-mon[37153]: pgmap v3767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:33:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:14 smithi047 ceph-mon[36835]: pgmap v3767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:33:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:14 smithi029 ceph-mon[32610]: pgmap v3767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:33:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:16 smithi077 ceph-mon[37153]: pgmap v3768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:33:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:16 smithi047 ceph-mon[36835]: pgmap v3768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:33:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:16 smithi029 ceph-mon[32610]: pgmap v3768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:33:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6219.497865 secs 2024-07-27T08:33:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:18 smithi077 ceph-mon[37153]: pgmap v3769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:33:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6219.497865 secs 2024-07-27T08:33:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:18 smithi047 ceph-mon[36835]: pgmap v3769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:33:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6219.497865 secs 2024-07-27T08:33:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:18 smithi029 ceph-mon[32610]: pgmap v3769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:33:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:20 smithi077 ceph-mon[37153]: pgmap v3770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:33:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:20 smithi047 ceph-mon[36835]: pgmap v3770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:33:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:20 smithi029 ceph-mon[32610]: pgmap v3770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:33:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:22 smithi077 ceph-mon[37153]: pgmap v3771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:33:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:22 smithi047 ceph-mon[36835]: pgmap v3771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:33:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:22 smithi029 ceph-mon[32610]: pgmap v3771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:33:23.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6224.498017 secs 2024-07-27T08:33:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6224.498017 secs 2024-07-27T08:33:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6224.498017 secs 2024-07-27T08:33:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:24 smithi077 ceph-mon[37153]: pgmap v3772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:33:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:24 smithi047 ceph-mon[36835]: pgmap v3772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:33:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:24 smithi029 ceph-mon[32610]: pgmap v3772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:33:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:26 smithi077 ceph-mon[37153]: pgmap v3773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:26 smithi047 ceph-mon[36835]: pgmap v3773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:26 smithi029 ceph-mon[32610]: pgmap v3773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:27.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:33:27.902+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:33:27.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:33:27.902+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:33:27.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:33:27.902+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:33:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:33:27.923+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:33:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:33:27.923+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:33:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:33:27.923+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:33:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6229.498159 secs 2024-07-27T08:33:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:28 smithi077 ceph-mon[37153]: pgmap v3774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:33:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6229.498159 secs 2024-07-27T08:33:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:28 smithi047 ceph-mon[36835]: pgmap v3774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:33:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6229.498159 secs 2024-07-27T08:33:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:28 smithi029 ceph-mon[32610]: pgmap v3774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:33:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:30 smithi077 ceph-mon[37153]: pgmap v3775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:30 smithi047 ceph-mon[36835]: pgmap v3775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:30 smithi029 ceph-mon[32610]: pgmap v3775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:32 smithi077 ceph-mon[37153]: pgmap v3776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:32 smithi047 ceph-mon[36835]: pgmap v3776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:32 smithi029 ceph-mon[32610]: pgmap v3776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6234.498283 secs 2024-07-27T08:33:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6234.498283 secs 2024-07-27T08:33:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6234.498283 secs 2024-07-27T08:33:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:34 smithi077 ceph-mon[37153]: pgmap v3777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:33:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:34 smithi047 ceph-mon[36835]: pgmap v3777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:33:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:34 smithi029 ceph-mon[32610]: pgmap v3777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:33:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:36 smithi077 ceph-mon[37153]: pgmap v3778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:36 smithi047 ceph-mon[36835]: pgmap v3778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:36 smithi029 ceph-mon[32610]: pgmap v3778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6239.498412 secs 2024-07-27T08:33:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:38 smithi077 ceph-mon[37153]: pgmap v3779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:33:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6239.498412 secs 2024-07-27T08:33:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:38 smithi047 ceph-mon[36835]: pgmap v3779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:33:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6239.498412 secs 2024-07-27T08:33:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:38 smithi029 ceph-mon[32610]: pgmap v3779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:33:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:40 smithi077 ceph-mon[37153]: pgmap v3780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:40 smithi047 ceph-mon[36835]: pgmap v3780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:40 smithi029 ceph-mon[32610]: pgmap v3780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:41 smithi047 ceph-mon[36835]: pgmap v3781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:41 smithi029 ceph-mon[32610]: pgmap v3781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:41 smithi077 ceph-mon[37153]: pgmap v3781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:42.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:33:42.907+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:33:42.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:33:42.907+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:33:42.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:33:42.907+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:33:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:33:42.929+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:33:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:33:42.929+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:33:42.930 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:33:42.929+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:33:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6244.498538 secs 2024-07-27T08:33:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6244.498538 secs 2024-07-27T08:33:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6244.498538 secs 2024-07-27T08:33:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:43 smithi029 ceph-mon[32610]: pgmap v3782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:33:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:43 smithi077 ceph-mon[37153]: pgmap v3782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:33:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:43 smithi047 ceph-mon[36835]: pgmap v3782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:33:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:46 smithi077 ceph-mon[37153]: pgmap v3783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:46 smithi047 ceph-mon[36835]: pgmap v3783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:46 smithi029 ceph-mon[32610]: pgmap v3783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6249.498675 secs 2024-07-27T08:33:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:48 smithi077 ceph-mon[37153]: pgmap v3784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:33:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6249.498675 secs 2024-07-27T08:33:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:48 smithi047 ceph-mon[36835]: pgmap v3784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:33:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6249.498675 secs 2024-07-27T08:33:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:48 smithi029 ceph-mon[32610]: pgmap v3784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:33:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:50 smithi077 ceph-mon[37153]: pgmap v3785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:50 smithi047 ceph-mon[36835]: pgmap v3785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:50 smithi029 ceph-mon[32610]: pgmap v3785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:33:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:52 smithi077 ceph-mon[37153]: pgmap v3786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:52 smithi047 ceph-mon[36835]: pgmap v3786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:52.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:52 smithi029 ceph-mon[32610]: pgmap v3786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:33:53.351 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6254.498805 secs 2024-07-27T08:33:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6254.498805 secs 2024-07-27T08:33:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6254.498805 secs 2024-07-27T08:33:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:54 smithi077 ceph-mon[37153]: pgmap v3787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:33:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:54 smithi047 ceph-mon[36835]: pgmap v3787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:33:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:54 smithi029 ceph-mon[32610]: pgmap v3787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:33:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:56 smithi077 ceph-mon[37153]: pgmap v3788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:33:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:56 smithi047 ceph-mon[36835]: pgmap v3788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:33:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:56 smithi029 ceph-mon[32610]: pgmap v3788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:33:57.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:33:57.922+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:33:57.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:33:57.922+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:33:57.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:33:57.922+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:33:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:33:57.944+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:33:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:33:57.944+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:33:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:33:57.944+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:33:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6259.498927 secs 2024-07-27T08:33:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:33:58 smithi029 ceph-mon[32610]: pgmap v3789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:33:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6259.498927 secs 2024-07-27T08:33:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:33:58 smithi077 ceph-mon[37153]: pgmap v3789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:33:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6259.498927 secs 2024-07-27T08:33:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:33:58 smithi047 ceph-mon[36835]: pgmap v3789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:34:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:00 smithi077 ceph-mon[37153]: pgmap v3790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:00 smithi047 ceph-mon[36835]: pgmap v3790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:00 smithi029 ceph-mon[32610]: pgmap v3790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:02 smithi077 ceph-mon[37153]: pgmap v3791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:02 smithi047 ceph-mon[36835]: pgmap v3791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:02 smithi029 ceph-mon[32610]: pgmap v3791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6264.499049 secs 2024-07-27T08:34:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6264.499049 secs 2024-07-27T08:34:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6264.499049 secs 2024-07-27T08:34:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:04 smithi077 ceph-mon[37153]: pgmap v3792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:34:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:04 smithi047 ceph-mon[36835]: pgmap v3792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:34:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:04 smithi029 ceph-mon[32610]: pgmap v3792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:34:06.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:06 smithi077 ceph-mon[37153]: pgmap v3793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:06 smithi047 ceph-mon[36835]: pgmap v3793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:06 smithi029 ceph-mon[32610]: pgmap v3793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6269.499174 secs 2024-07-27T08:34:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:08 smithi077 ceph-mon[37153]: pgmap v3794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:34:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6269.499174 secs 2024-07-27T08:34:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:08 smithi047 ceph-mon[36835]: pgmap v3794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:34:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6269.499174 secs 2024-07-27T08:34:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:08 smithi029 ceph-mon[32610]: pgmap v3794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:34:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:10 smithi077 ceph-mon[37153]: pgmap v3795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:10 smithi047 ceph-mon[36835]: pgmap v3795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:10 smithi029 ceph-mon[32610]: pgmap v3795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:34:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:34:11.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:34:11.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:34:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:34:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:34:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:34:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:34:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:34:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:34:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:34:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:34:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:34:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:12 smithi077 ceph-mon[37153]: pgmap v3796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:12 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:34:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:12 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:34:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:12 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:34:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:12 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:34:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:12 smithi077 ceph-mon[37153]: pgmap v3797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:34:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:12 smithi077 ceph-mon[37153]: pgmap v3798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:34:12.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:12.938+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:34:12.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:12.938+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:34:12.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:12.938+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:34:12.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:12 smithi029 ceph-mon[32610]: pgmap v3796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:12.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:12 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:34:12.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:12 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:34:12.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:12 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:34:12.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:12 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:34:12.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:12 smithi029 ceph-mon[32610]: pgmap v3797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:34:12.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:12 smithi029 ceph-mon[32610]: pgmap v3798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:34:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:12 smithi047 ceph-mon[36835]: pgmap v3796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:12 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:34:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:12 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:34:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:12 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:34:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:12 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:34:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:12 smithi047 ceph-mon[36835]: pgmap v3797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:34:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:12 smithi047 ceph-mon[36835]: pgmap v3798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:34:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:12.960+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:34:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:12.960+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:34:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:12.960+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:34:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6274.499270 secs 2024-07-27T08:34:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6274.499270 secs 2024-07-27T08:34:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6274.499270 secs 2024-07-27T08:34:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:14 smithi077 ceph-mon[37153]: pgmap v3799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:34:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:14 smithi047 ceph-mon[36835]: pgmap v3799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:34:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:14 smithi029 ceph-mon[32610]: pgmap v3799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:34:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:15 smithi047 ceph-mon[36835]: pgmap v3800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:34:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:15 smithi029 ceph-mon[32610]: pgmap v3800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:34:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:15 smithi077 ceph-mon[37153]: pgmap v3800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:34:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:17 smithi047 ceph-mon[36835]: pgmap v3801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:34:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:17 smithi029 ceph-mon[32610]: pgmap v3801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:34:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:17 smithi077 ceph-mon[37153]: pgmap v3801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:34:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6279.499391 secs 2024-07-27T08:34:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6279.499391 secs 2024-07-27T08:34:19.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6279.499391 secs 2024-07-27T08:34:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:19 smithi047 ceph-mon[36835]: pgmap v3802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:34:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:19 smithi029 ceph-mon[32610]: pgmap v3802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:34:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:19 smithi077 ceph-mon[37153]: pgmap v3802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:34:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:21 smithi047 ceph-mon[36835]: pgmap v3803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:34:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:21 smithi029 ceph-mon[32610]: pgmap v3803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:34:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:21 smithi077 ceph-mon[37153]: pgmap v3803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:34:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6284.499514 secs 2024-07-27T08:34:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6284.499514 secs 2024-07-27T08:34:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6284.499514 secs 2024-07-27T08:34:24.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:23 smithi047 ceph-mon[36835]: pgmap v3804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:23 smithi077 ceph-mon[37153]: pgmap v3804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:23 smithi029 ceph-mon[32610]: pgmap v3804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:25 smithi047 ceph-mon[36835]: pgmap v3805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:25 smithi029 ceph-mon[32610]: pgmap v3805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:25 smithi077 ceph-mon[37153]: pgmap v3805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:27 smithi047 ceph-mon[36835]: pgmap v3806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:34:27.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:27.952+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:34:27.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:27.952+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:34:27.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:27.952+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:34:27.955 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:27 smithi029 ceph-mon[32610]: pgmap v3806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:34:27.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:27.974+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:34:27.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:27.974+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:34:27.976 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:27.974+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:34:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:27 smithi077 ceph-mon[37153]: pgmap v3806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:34:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6289.499650 secs 2024-07-27T08:34:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6289.499650 secs 2024-07-27T08:34:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6289.499650 secs 2024-07-27T08:34:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:29 smithi047 ceph-mon[36835]: pgmap v3807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:34:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:29 smithi029 ceph-mon[32610]: pgmap v3807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:34:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:29 smithi077 ceph-mon[37153]: pgmap v3807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:34:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:31 smithi047 ceph-mon[36835]: pgmap v3808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:34:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:31 smithi029 ceph-mon[32610]: pgmap v3808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:34:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:31 smithi077 ceph-mon[37153]: pgmap v3808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:34:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6294.499825 secs 2024-07-27T08:34:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6294.499825 secs 2024-07-27T08:34:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6294.499825 secs 2024-07-27T08:34:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:33 smithi077 ceph-mon[37153]: pgmap v3809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:33 smithi047 ceph-mon[36835]: pgmap v3809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:33 smithi029 ceph-mon[32610]: pgmap v3809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:35 smithi047 ceph-mon[36835]: pgmap v3810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:35 smithi029 ceph-mon[32610]: pgmap v3810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:35 smithi077 ceph-mon[37153]: pgmap v3810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:37 smithi047 ceph-mon[36835]: pgmap v3811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:34:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:37 smithi029 ceph-mon[32610]: pgmap v3811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:34:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:37 smithi077 ceph-mon[37153]: pgmap v3811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:34:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6299.499929 secs 2024-07-27T08:34:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6299.499929 secs 2024-07-27T08:34:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6299.499929 secs 2024-07-27T08:34:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:39 smithi047 ceph-mon[36835]: pgmap v3812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:34:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:39 smithi029 ceph-mon[32610]: pgmap v3812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:34:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:39 smithi077 ceph-mon[37153]: pgmap v3812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:34:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:41 smithi047 ceph-mon[36835]: pgmap v3813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:34:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:41 smithi029 ceph-mon[32610]: pgmap v3813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:34:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:41 smithi077 ceph-mon[37153]: pgmap v3813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:34:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:42.640+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900001942) for more than 900000000 us, fault. 2024-07-27T08:34:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:42.640+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4e80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f733c0045a0 tx=0x7f733c073000 comp rx=0 tx=0).stop 2024-07-27T08:34:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:42.640+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T08:34:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:42.640+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4e80 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T08:34:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:42.640+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T08:34:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:42.640+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T08:34:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:42.640+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f733c02b4a0 tx=0x7f733c05d000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T08:34:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6304.500063 secs 2024-07-27T08:34:42.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:42.967+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:34:42.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:42.967+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:34:42.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:42.967+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:34:42.991 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:42.990+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:34:42.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:42.990+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:34:42.992 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:42.990+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:34:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6304.500063 secs 2024-07-27T08:34:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6304.500063 secs 2024-07-27T08:34:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:43.581+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:34:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:43.581+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065c0 con 0x7f7348108670 2024-07-27T08:34:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:43.581+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065c0 con 0x7f7348108670 2024-07-27T08:34:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:43.581+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:34:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:43 smithi077 ceph-mon[37153]: pgmap v3814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:34:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:43 smithi047 ceph-mon[36835]: pgmap v3814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:34:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:43 smithi029 ceph-mon[32610]: pgmap v3814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:34:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:45 smithi047 ceph-mon[36835]: pgmap v3815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:34:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:45 smithi029 ceph-mon[32610]: pgmap v3815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:34:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:45 smithi077 ceph-mon[37153]: pgmap v3815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:34:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:47 smithi047 ceph-mon[36835]: pgmap v3816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:34:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:47 smithi029 ceph-mon[32610]: pgmap v3816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:34:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:47 smithi077 ceph-mon[37153]: pgmap v3816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:34:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6309.500187 secs 2024-07-27T08:34:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6309.500187 secs 2024-07-27T08:34:49.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6309.500187 secs 2024-07-27T08:34:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:49 smithi047 ceph-mon[36835]: pgmap v3817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:34:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:49 smithi029 ceph-mon[32610]: pgmap v3817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:34:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:49 smithi077 ceph-mon[37153]: pgmap v3817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:34:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:51 smithi047 ceph-mon[36835]: pgmap v3818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:34:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:51 smithi029 ceph-mon[32610]: pgmap v3818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:34:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:51 smithi077 ceph-mon[37153]: pgmap v3818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:34:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6314.500313 secs 2024-07-27T08:34:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6314.500313 secs 2024-07-27T08:34:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6314.500313 secs 2024-07-27T08:34:54.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:53 smithi047 ceph-mon[36835]: pgmap v3819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:53 smithi077 ceph-mon[37153]: pgmap v3819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:53 smithi029 ceph-mon[32610]: pgmap v3819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:34:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:55 smithi047 ceph-mon[36835]: pgmap v3820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:55 smithi029 ceph-mon[32610]: pgmap v3820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:55 smithi077 ceph-mon[37153]: pgmap v3820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:34:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.655+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T08:34:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.655+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T08:34:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.655+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b4ef0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7eff400048c0 tx=0x7eff40038040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T08:34:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.655+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b4ef0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7eff400048c0 tx=0x7eff40038040 comp rx=0 tx=0).stop 2024-07-27T08:34:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.655+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b4ef0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T08:34:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.655+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b4ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T08:34:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.655+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b4ef0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T08:34:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.656+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T08:34:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.656+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T08:34:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.657+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7eff40004480 tx=0x7eff4004d040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T08:34:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:57 smithi047 ceph-mon[36835]: pgmap v3821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:34:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:57 smithi029 ceph-mon[32610]: pgmap v3821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:34:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.968+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:34:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.968+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:34:57.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:57.968+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:34:58.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:58.006+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:34:58.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:58.006+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:34:58.007 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:34:58.006+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:34:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:57 smithi077 ceph-mon[37153]: pgmap v3821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:34:58.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:58.583+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:34:58.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:58.583+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:34:58.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:58.583+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:34:58.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:34:58.583+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:34:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6319.500435 secs 2024-07-27T08:34:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6319.500435 secs 2024-07-27T08:34:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6319.500435 secs 2024-07-27T08:34:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:34:59 smithi047 ceph-mon[36835]: pgmap v3822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:34:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:34:59 smithi029 ceph-mon[32610]: pgmap v3822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:35:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:34:59 smithi077 ceph-mon[37153]: pgmap v3822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:35:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:01 smithi047 ceph-mon[36835]: pgmap v3823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:01 smithi029 ceph-mon[32610]: pgmap v3823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:01 smithi077 ceph-mon[37153]: pgmap v3823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6324.500568 secs 2024-07-27T08:35:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6324.500568 secs 2024-07-27T08:35:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6324.500568 secs 2024-07-27T08:35:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:03 smithi077 ceph-mon[37153]: pgmap v3824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:35:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:03 smithi047 ceph-mon[36835]: pgmap v3824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:35:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:03 smithi029 ceph-mon[32610]: pgmap v3824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:35:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:06 smithi077 ceph-mon[37153]: pgmap v3825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:35:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:06 smithi047 ceph-mon[36835]: pgmap v3825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:35:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:06 smithi029 ceph-mon[32610]: pgmap v3825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:35:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6329.500697 secs 2024-07-27T08:35:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:08 smithi077 ceph-mon[37153]: pgmap v3826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:35:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6329.500697 secs 2024-07-27T08:35:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:08 smithi047 ceph-mon[36835]: pgmap v3826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:35:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6329.500697 secs 2024-07-27T08:35:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:08 smithi029 ceph-mon[32610]: pgmap v3826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:35:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:10 smithi077 ceph-mon[37153]: pgmap v3827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:35:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:10 smithi047 ceph-mon[36835]: pgmap v3827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:35:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:10 smithi029 ceph-mon[32610]: pgmap v3827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:35:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:12 smithi077 ceph-mon[37153]: pgmap v3828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:35:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:35:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:35:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:35:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:12 smithi047 ceph-mon[36835]: pgmap v3828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:35:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:35:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:35:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:35:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:12 smithi029 ceph-mon[32610]: pgmap v3828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:35:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:35:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:35:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:35:12.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:35:12.972+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:35:12.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:35:12.972+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:35:12.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:35:12.972+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:35:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:35:13.021+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:35:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:35:13.021+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:35:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:35:13.021+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:35:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6334.500814 secs 2024-07-27T08:35:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:13 smithi077 ceph-mon[37153]: pgmap v3829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:35:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:13 smithi077 ceph-mon[37153]: pgmap v3830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:35:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6334.500814 secs 2024-07-27T08:35:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:13 smithi047 ceph-mon[36835]: pgmap v3829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:35:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:13 smithi047 ceph-mon[36835]: pgmap v3830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:35:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6334.500814 secs 2024-07-27T08:35:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:13 smithi029 ceph-mon[32610]: pgmap v3829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:35:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:13 smithi029 ceph-mon[32610]: pgmap v3830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:35:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:14 smithi077 ceph-mon[37153]: pgmap v3831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:35:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:14 smithi047 ceph-mon[36835]: pgmap v3831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:35:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:14 smithi029 ceph-mon[32610]: pgmap v3831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:35:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:17 smithi047 ceph-mon[36835]: pgmap v3832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:35:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:17 smithi029 ceph-mon[32610]: pgmap v3832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:35:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:17 smithi077 ceph-mon[37153]: pgmap v3832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:35:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6339.500903 secs 2024-07-27T08:35:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6339.500903 secs 2024-07-27T08:35:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6339.500903 secs 2024-07-27T08:35:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:19 smithi047 ceph-mon[36835]: pgmap v3833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:35:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:19 smithi029 ceph-mon[32610]: pgmap v3833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:35:19.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:19 smithi077 ceph-mon[37153]: pgmap v3833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:35:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:21 smithi047 ceph-mon[36835]: pgmap v3834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:35:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:21 smithi029 ceph-mon[32610]: pgmap v3834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:35:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:21 smithi077 ceph-mon[37153]: pgmap v3834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:35:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6344.501038 secs 2024-07-27T08:35:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:23 smithi029 ceph-mon[32610]: pgmap v3835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:35:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6344.501038 secs 2024-07-27T08:35:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:23 smithi077 ceph-mon[37153]: pgmap v3835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:35:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6344.501038 secs 2024-07-27T08:35:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:23 smithi047 ceph-mon[36835]: pgmap v3835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:35:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:25 smithi029 ceph-mon[32610]: pgmap v3836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:35:25.498 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:25 smithi047 ceph-mon[36835]: pgmap v3836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:35:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:25 smithi077 ceph-mon[37153]: pgmap v3836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:35:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:27 smithi029 ceph-mon[32610]: pgmap v3837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:27.612 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:27 smithi077 ceph-mon[37153]: pgmap v3837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:27 smithi047 ceph-mon[36835]: pgmap v3837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:27.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:35:27.982+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:35:27.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:35:27.982+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:35:27.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:35:27.982+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:35:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:35:28.035+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:35:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:35:28.036+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:35:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:35:28.036+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:35:28.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6349.501166 secs 2024-07-27T08:35:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6349.501166 secs 2024-07-27T08:35:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6349.501166 secs 2024-07-27T08:35:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:29 smithi029 ceph-mon[32610]: pgmap v3838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:35:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:29 smithi077 ceph-mon[37153]: pgmap v3838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:35:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:29 smithi047 ceph-mon[36835]: pgmap v3838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:35:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:31 smithi077 ceph-mon[37153]: pgmap v3839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:35:31.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:31 smithi047 ceph-mon[36835]: pgmap v3839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:35:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:31 smithi029 ceph-mon[32610]: pgmap v3839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:35:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6354.501311 secs 2024-07-27T08:35:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:33 smithi077 ceph-mon[37153]: pgmap v3840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6354.501311 secs 2024-07-27T08:35:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:33 smithi047 ceph-mon[36835]: pgmap v3840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6354.501311 secs 2024-07-27T08:35:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:33 smithi029 ceph-mon[32610]: pgmap v3840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:35 smithi077 ceph-mon[37153]: pgmap v3841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:35:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:35 smithi047 ceph-mon[36835]: pgmap v3841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:35:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:35 smithi029 ceph-mon[32610]: pgmap v3841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:35:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:37 smithi077 ceph-mon[37153]: pgmap v3842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:37 smithi047 ceph-mon[36835]: pgmap v3842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:37 smithi029 ceph-mon[32610]: pgmap v3842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6359.501458 secs 2024-07-27T08:35:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6359.501458 secs 2024-07-27T08:35:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6359.501458 secs 2024-07-27T08:35:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:39 smithi077 ceph-mon[37153]: pgmap v3843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:35:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:39 smithi047 ceph-mon[36835]: pgmap v3843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:35:39.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:39 smithi029 ceph-mon[32610]: pgmap v3843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:35:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:41 smithi077 ceph-mon[37153]: pgmap v3844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:35:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:41 smithi047 ceph-mon[36835]: pgmap v3844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:35:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:41 smithi029 ceph-mon[32610]: pgmap v3844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:35:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:35:42.998+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:35:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:35:42.998+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:35:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:35:42.998+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:35:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:35:43.037+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:35:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:35:43.037+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:35:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:35:43.037+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:35:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6364.501605 secs 2024-07-27T08:35:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:43 smithi029 ceph-mon[32610]: pgmap v3845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6364.501605 secs 2024-07-27T08:35:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:43 smithi077 ceph-mon[37153]: pgmap v3845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6364.501605 secs 2024-07-27T08:35:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:43 smithi047 ceph-mon[36835]: pgmap v3845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:45 smithi077 ceph-mon[37153]: pgmap v3846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:35:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:45 smithi047 ceph-mon[36835]: pgmap v3846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:35:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:45 smithi029 ceph-mon[32610]: pgmap v3846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:35:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:47 smithi077 ceph-mon[37153]: pgmap v3847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:47 smithi047 ceph-mon[36835]: pgmap v3847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:47 smithi029 ceph-mon[32610]: pgmap v3847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6369.501748 secs 2024-07-27T08:35:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6369.501748 secs 2024-07-27T08:35:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6369.501748 secs 2024-07-27T08:35:49.507 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:49 smithi077 ceph-mon[37153]: pgmap v3848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:35:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:49 smithi047 ceph-mon[36835]: pgmap v3848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:35:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:49 smithi029 ceph-mon[32610]: pgmap v3848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:35:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:51 smithi077 ceph-mon[37153]: pgmap v3849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:35:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:51 smithi047 ceph-mon[36835]: pgmap v3849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:35:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:51 smithi029 ceph-mon[32610]: pgmap v3849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:35:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6374.501878 secs 2024-07-27T08:35:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:53 smithi077 ceph-mon[37153]: pgmap v3850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6374.501878 secs 2024-07-27T08:35:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:53 smithi047 ceph-mon[36835]: pgmap v3850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6374.501878 secs 2024-07-27T08:35:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:53 smithi029 ceph-mon[32610]: pgmap v3850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:55 smithi077 ceph-mon[37153]: pgmap v3851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:35:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:55 smithi047 ceph-mon[36835]: pgmap v3851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:35:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:55 smithi029 ceph-mon[32610]: pgmap v3851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:35:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:57 smithi077 ceph-mon[37153]: pgmap v3852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:57 smithi047 ceph-mon[36835]: pgmap v3852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:57 smithi029 ceph-mon[32610]: pgmap v3852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:35:58.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:35:58.013+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:35:58.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:35:58.013+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:35:58.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:35:58.013+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:35:58.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:35:58.053+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:35:58.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:35:58.053+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:35:58.054 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:35:58.053+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:35:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6379.502005 secs 2024-07-27T08:35:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6379.502005 secs 2024-07-27T08:35:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6379.502005 secs 2024-07-27T08:35:59.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:35:59 smithi077 ceph-mon[37153]: pgmap v3853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:35:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:35:59 smithi047 ceph-mon[36835]: pgmap v3853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:35:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:35:59 smithi029 ceph-mon[32610]: pgmap v3853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:01 smithi077 ceph-mon[37153]: pgmap v3854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:01 smithi047 ceph-mon[36835]: pgmap v3854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:01 smithi029 ceph-mon[32610]: pgmap v3854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6384.502130 secs 2024-07-27T08:36:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:03 smithi077 ceph-mon[37153]: pgmap v3855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6384.502130 secs 2024-07-27T08:36:03.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:03 smithi047 ceph-mon[36835]: pgmap v3855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6384.502130 secs 2024-07-27T08:36:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:03 smithi029 ceph-mon[32610]: pgmap v3855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:05 smithi077 ceph-mon[37153]: pgmap v3856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:36:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:05 smithi047 ceph-mon[36835]: pgmap v3856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:36:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:05 smithi029 ceph-mon[32610]: pgmap v3856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:36:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:07 smithi077 ceph-mon[37153]: pgmap v3857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:07 smithi047 ceph-mon[36835]: pgmap v3857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:07 smithi029 ceph-mon[32610]: pgmap v3857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6389.502258 secs 2024-07-27T08:36:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6389.502258 secs 2024-07-27T08:36:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6389.502258 secs 2024-07-27T08:36:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:09 smithi077 ceph-mon[37153]: pgmap v3858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:09 smithi047 ceph-mon[36835]: pgmap v3858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:09 smithi029 ceph-mon[32610]: pgmap v3858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:11 smithi077 ceph-mon[37153]: pgmap v3859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:11 smithi047 ceph-mon[36835]: pgmap v3859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:11 smithi029 ceph-mon[32610]: pgmap v3859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:36:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:36:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:36:13.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:36:13.029+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:36:13.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:36:13.029+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:36:13.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:36:13.029+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:36:13.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:13.056+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:36:13.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:13.056+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:36:13.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:13.056+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:36:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6394.502388 secs 2024-07-27T08:36:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:13 smithi077 ceph-mon[37153]: pgmap v3860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:36:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:36:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:36:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6394.502388 secs 2024-07-27T08:36:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:13 smithi047 ceph-mon[36835]: pgmap v3860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:36:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:36:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:36:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6394.502388 secs 2024-07-27T08:36:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:13 smithi029 ceph-mon[32610]: pgmap v3860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:36:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:36:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:36:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:14 smithi077 ceph-mon[37153]: pgmap v3861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T08:36:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:14 smithi077 ceph-mon[37153]: pgmap v3862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:36:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:14 smithi047 ceph-mon[36835]: pgmap v3861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T08:36:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:14 smithi047 ceph-mon[36835]: pgmap v3862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:36:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:14 smithi029 ceph-mon[32610]: pgmap v3861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T08:36:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:14 smithi029 ceph-mon[32610]: pgmap v3862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:36:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:16 smithi077 ceph-mon[37153]: pgmap v3863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-27T08:36:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:16 smithi047 ceph-mon[36835]: pgmap v3863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-27T08:36:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:16 smithi029 ceph-mon[32610]: pgmap v3863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-27T08:36:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:18 smithi077 ceph-mon[37153]: pgmap v3864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:36:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6399.502515 secs 2024-07-27T08:36:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:18 smithi047 ceph-mon[36835]: pgmap v3864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:36:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6399.502515 secs 2024-07-27T08:36:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:18 smithi029 ceph-mon[32610]: pgmap v3864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:36:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6399.502515 secs 2024-07-27T08:36:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:20 smithi077 ceph-mon[37153]: pgmap v3865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:20 smithi047 ceph-mon[36835]: pgmap v3865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:20 smithi029 ceph-mon[32610]: pgmap v3865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:22 smithi077 ceph-mon[37153]: pgmap v3866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-27T08:36:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:22 smithi047 ceph-mon[36835]: pgmap v3866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-27T08:36:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:22 smithi029 ceph-mon[32610]: pgmap v3866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-27T08:36:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6404.502581 secs 2024-07-27T08:36:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6404.502581 secs 2024-07-27T08:36:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6404.502581 secs 2024-07-27T08:36:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:24 smithi077 ceph-mon[37153]: pgmap v3867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:36:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:24 smithi047 ceph-mon[36835]: pgmap v3867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:36:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:24 smithi029 ceph-mon[32610]: pgmap v3867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:36:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:26 smithi077 ceph-mon[37153]: pgmap v3868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:36:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:26 smithi047 ceph-mon[36835]: pgmap v3868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:36:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:26 smithi029 ceph-mon[32610]: pgmap v3868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:36:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:36:28.044+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:36:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:36:28.044+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:36:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:36:28.044+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:36:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:28.071+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:36:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:28.071+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:36:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:28.071+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:36:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:28 smithi077 ceph-mon[37153]: pgmap v3869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6409.502710 secs 2024-07-27T08:36:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:28 smithi047 ceph-mon[36835]: pgmap v3869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6409.502710 secs 2024-07-27T08:36:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:28 smithi029 ceph-mon[32610]: pgmap v3869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6409.502710 secs 2024-07-27T08:36:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:30 smithi077 ceph-mon[37153]: pgmap v3870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:30 smithi047 ceph-mon[36835]: pgmap v3870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:30 smithi029 ceph-mon[32610]: pgmap v3870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:32 smithi077 ceph-mon[37153]: pgmap v3871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:32 smithi047 ceph-mon[36835]: pgmap v3871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:32 smithi029 ceph-mon[32610]: pgmap v3871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6414.502812 secs 2024-07-27T08:36:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6414.502812 secs 2024-07-27T08:36:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6414.502812 secs 2024-07-27T08:36:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:34 smithi077 ceph-mon[37153]: pgmap v3872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:36:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:34 smithi047 ceph-mon[36835]: pgmap v3872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:36:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:34 smithi029 ceph-mon[32610]: pgmap v3872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:36:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:36 smithi077 ceph-mon[37153]: pgmap v3873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:36:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:36 smithi047 ceph-mon[36835]: pgmap v3873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:36:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:36 smithi029 ceph-mon[32610]: pgmap v3873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:36:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:38 smithi047 ceph-mon[36835]: pgmap v3874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6419.502862 secs 2024-07-27T08:36:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:38 smithi029 ceph-mon[32610]: pgmap v3874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6419.502862 secs 2024-07-27T08:36:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:38 smithi077 ceph-mon[37153]: pgmap v3874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6419.502862 secs 2024-07-27T08:36:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:40 smithi047 ceph-mon[36835]: pgmap v3875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:40 smithi029 ceph-mon[32610]: pgmap v3875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:40 smithi077 ceph-mon[37153]: pgmap v3875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:42 smithi047 ceph-mon[36835]: pgmap v3876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:42 smithi029 ceph-mon[32610]: pgmap v3876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:42.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:42 smithi077 ceph-mon[37153]: pgmap v3876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:43.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:36:43.052+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:36:43.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:36:43.052+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:36:43.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:36:43.052+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:36:43.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:43.086+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:36:43.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:43.086+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:36:43.088 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:43.086+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:36:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6424.502943 secs 2024-07-27T08:36:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6424.502943 secs 2024-07-27T08:36:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6424.502943 secs 2024-07-27T08:36:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:44 smithi047 ceph-mon[36835]: pgmap v3877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:36:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:44 smithi029 ceph-mon[32610]: pgmap v3877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:36:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:44 smithi077 ceph-mon[37153]: pgmap v3877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:36:46.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:46 smithi029 ceph-mon[32610]: pgmap v3878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:36:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:46 smithi047 ceph-mon[36835]: pgmap v3878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:36:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:46 smithi077 ceph-mon[37153]: pgmap v3878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:36:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:48 smithi047 ceph-mon[36835]: pgmap v3879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6429.503063 secs 2024-07-27T08:36:48.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:48 smithi029 ceph-mon[32610]: pgmap v3879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:48.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6429.503063 secs 2024-07-27T08:36:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:48 smithi077 ceph-mon[37153]: pgmap v3879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6429.503063 secs 2024-07-27T08:36:50.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:50 smithi077 ceph-mon[37153]: pgmap v3880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:50 smithi047 ceph-mon[36835]: pgmap v3880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:50 smithi029 ceph-mon[32610]: pgmap v3880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:36:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:52 smithi047 ceph-mon[36835]: pgmap v3881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:52 smithi029 ceph-mon[32610]: pgmap v3881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:52 smithi077 ceph-mon[37153]: pgmap v3881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:36:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6434.503127 secs 2024-07-27T08:36:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6434.503127 secs 2024-07-27T08:36:53.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6434.503127 secs 2024-07-27T08:36:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:54 smithi047 ceph-mon[36835]: pgmap v3882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:36:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:54 smithi029 ceph-mon[32610]: pgmap v3882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:36:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:54 smithi077 ceph-mon[37153]: pgmap v3882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:36:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:56 smithi047 ceph-mon[36835]: pgmap v3883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:36:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:56 smithi029 ceph-mon[32610]: pgmap v3883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:36:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:56 smithi077 ceph-mon[37153]: pgmap v3883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:36:58.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:36:58.067+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:36:58.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:36:58.067+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:36:58.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:36:58.067+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:36:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:58.102+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:36:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:58.102+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:36:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:58.102+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:36:58.602 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:58.601+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:36:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:58.601+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:36:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:58.601+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:36:58.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:36:58.601+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:36:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:58 smithi047 ceph-mon[36835]: pgmap v3884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:36:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6439.503254 secs 2024-07-27T08:36:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:58 smithi029 ceph-mon[32610]: pgmap v3884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:36:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6439.503254 secs 2024-07-27T08:36:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:58 smithi077 ceph-mon[37153]: pgmap v3884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:36:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:36:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6439.503254 secs 2024-07-27T08:37:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:00 smithi047 ceph-mon[36835]: pgmap v3885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:00 smithi029 ceph-mon[32610]: pgmap v3885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:00 smithi077 ceph-mon[37153]: pgmap v3885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:02 smithi047 ceph-mon[36835]: pgmap v3886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:37:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:02 smithi029 ceph-mon[32610]: pgmap v3886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:37:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:02 smithi077 ceph-mon[37153]: pgmap v3886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:37:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6444.503400 secs 2024-07-27T08:37:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6444.503400 secs 2024-07-27T08:37:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6444.503400 secs 2024-07-27T08:37:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:04 smithi047 ceph-mon[36835]: pgmap v3887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:37:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:04 smithi029 ceph-mon[32610]: pgmap v3887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:37:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:04 smithi077 ceph-mon[37153]: pgmap v3887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:37:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:06 smithi047 ceph-mon[36835]: pgmap v3888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:37:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:06 smithi029 ceph-mon[32610]: pgmap v3888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:37:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:06 smithi077 ceph-mon[37153]: pgmap v3888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:37:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:08 smithi047 ceph-mon[36835]: pgmap v3889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:37:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6449.503525 secs 2024-07-27T08:37:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:08 smithi029 ceph-mon[32610]: pgmap v3889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:37:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6449.503525 secs 2024-07-27T08:37:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:08 smithi077 ceph-mon[37153]: pgmap v3889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:37:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6449.503525 secs 2024-07-27T08:37:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:10 smithi047 ceph-mon[36835]: pgmap v3890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:10 smithi029 ceph-mon[32610]: pgmap v3890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:10 smithi077 ceph-mon[37153]: pgmap v3890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:12 smithi047 ceph-mon[36835]: pgmap v3891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:37:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:12 smithi029 ceph-mon[32610]: pgmap v3891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:37:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:12 smithi077 ceph-mon[37153]: pgmap v3891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:37:13.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:13.083+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:37:13.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:13.083+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:37:13.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:13.083+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:37:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:13.117+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:37:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:13.117+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:37:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:13.117+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:37:13.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:13.604+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:37:13.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:13.604+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:37:13.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:13.604+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:37:13.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:13.604+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:37:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6454.503654 secs 2024-07-27T08:37:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:37:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6454.503654 secs 2024-07-27T08:37:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:37:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6454.503654 secs 2024-07-27T08:37:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:37:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:14 smithi047 ceph-mon[36835]: pgmap v3892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:37:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:14 smithi029 ceph-mon[32610]: pgmap v3892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:37:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:14 smithi077 ceph-mon[37153]: pgmap v3892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:37:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:37:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:37:15.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:37:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:37:15.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:37:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:37:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:37:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:16 smithi029 ceph-mon[32610]: pgmap v3893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:37:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:16 smithi029 ceph-mon[32610]: pgmap v3894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:37:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:16 smithi029 ceph-mon[32610]: pgmap v3895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:37:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:16 smithi077 ceph-mon[37153]: pgmap v3893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:37:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:16 smithi077 ceph-mon[37153]: pgmap v3894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:37:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:16 smithi077 ceph-mon[37153]: pgmap v3895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:37:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:16 smithi047 ceph-mon[36835]: pgmap v3893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:37:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:16 smithi047 ceph-mon[36835]: pgmap v3894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:37:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:16 smithi047 ceph-mon[36835]: pgmap v3895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:37:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6459.503794 secs 2024-07-27T08:37:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:18 smithi029 ceph-mon[32610]: pgmap v3896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:37:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6459.503794 secs 2024-07-27T08:37:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:18 smithi077 ceph-mon[37153]: pgmap v3896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:37:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6459.503794 secs 2024-07-27T08:37:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:18 smithi047 ceph-mon[36835]: pgmap v3896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:37:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:20 smithi029 ceph-mon[32610]: pgmap v3897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:20 smithi077 ceph-mon[37153]: pgmap v3897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:20 smithi047 ceph-mon[36835]: pgmap v3897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:22 smithi029 ceph-mon[32610]: pgmap v3898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:37:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:22 smithi077 ceph-mon[37153]: pgmap v3898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:37:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:22 smithi047 ceph-mon[36835]: pgmap v3898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:37:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6464.503915 secs 2024-07-27T08:37:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6464.503915 secs 2024-07-27T08:37:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6464.503915 secs 2024-07-27T08:37:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:24 smithi077 ceph-mon[37153]: pgmap v3899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:37:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:24 smithi047 ceph-mon[36835]: pgmap v3899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:37:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:24 smithi029 ceph-mon[32610]: pgmap v3899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:37:26.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:26 smithi047 ceph-mon[36835]: pgmap v3900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:37:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:26 smithi077 ceph-mon[37153]: pgmap v3900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:37:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:26 smithi029 ceph-mon[32610]: pgmap v3900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:37:28.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:28.097+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:37:28.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:28.097+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:37:28.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:28.097+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:37:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:28.131+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:37:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:28.131+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:37:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:28.131+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:37:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6469.503997 secs 2024-07-27T08:37:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:28 smithi077 ceph-mon[37153]: pgmap v3901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:37:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6469.503997 secs 2024-07-27T08:37:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:28 smithi047 ceph-mon[36835]: pgmap v3901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:37:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6469.503997 secs 2024-07-27T08:37:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:28 smithi029 ceph-mon[32610]: pgmap v3901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:37:30.808 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:30 smithi029 ceph-mon[32610]: pgmap v3902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:37:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:30 smithi077 ceph-mon[37153]: pgmap v3902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:37:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:30 smithi047 ceph-mon[36835]: pgmap v3902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:37:32.842 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:32 smithi029 ceph-mon[32610]: pgmap v3903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:37:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:32 smithi077 ceph-mon[37153]: pgmap v3903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:37:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:32 smithi047 ceph-mon[36835]: pgmap v3903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:37:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6474.504130 secs 2024-07-27T08:37:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6474.504130 secs 2024-07-27T08:37:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6474.504130 secs 2024-07-27T08:37:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:34 smithi077 ceph-mon[37153]: pgmap v3904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:37:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:34 smithi047 ceph-mon[36835]: pgmap v3904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:37:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:34 smithi029 ceph-mon[32610]: pgmap v3904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:37:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:36 smithi077 ceph-mon[37153]: pgmap v3905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:37:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:36 smithi047 ceph-mon[36835]: pgmap v3905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:37:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:36 smithi029 ceph-mon[32610]: pgmap v3905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:37:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6479.504261 secs 2024-07-27T08:37:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:38 smithi077 ceph-mon[37153]: pgmap v3906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:37:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6479.504261 secs 2024-07-27T08:37:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:38 smithi047 ceph-mon[36835]: pgmap v3906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:37:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6479.504261 secs 2024-07-27T08:37:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:38 smithi029 ceph-mon[32610]: pgmap v3906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:37:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:40 smithi077 ceph-mon[37153]: pgmap v3907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:37:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:40 smithi047 ceph-mon[36835]: pgmap v3907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:37:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:40 smithi029 ceph-mon[32610]: pgmap v3907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:37:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:42 smithi077 ceph-mon[37153]: pgmap v3908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:37:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:42 smithi047 ceph-mon[36835]: pgmap v3908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:37:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:42 smithi029 ceph-mon[32610]: pgmap v3908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:37:43.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:43.099+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:37:43.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:43.099+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:37:43.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:43.099+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:37:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:43.147+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:37:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:43.147+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:37:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:43.147+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:37:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:43.607+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:37:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:43.607+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:37:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:43.608+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:37:43.610 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:43.608+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:37:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6484.504391 secs 2024-07-27T08:37:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6484.504391 secs 2024-07-27T08:37:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6484.504391 secs 2024-07-27T08:37:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:44 smithi077 ceph-mon[37153]: pgmap v3909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:44 smithi047 ceph-mon[36835]: pgmap v3909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:44 smithi029 ceph-mon[32610]: pgmap v3909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:46 smithi077 ceph-mon[37153]: pgmap v3910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:37:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:46 smithi047 ceph-mon[36835]: pgmap v3910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:37:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:46 smithi029 ceph-mon[32610]: pgmap v3910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:37:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6489.504491 secs 2024-07-27T08:37:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:48 smithi077 ceph-mon[37153]: pgmap v3911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:37:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6489.504491 secs 2024-07-27T08:37:48.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:48 smithi047 ceph-mon[36835]: pgmap v3911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:37:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6489.504491 secs 2024-07-27T08:37:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:48 smithi029 ceph-mon[32610]: pgmap v3911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:37:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:50 smithi077 ceph-mon[37153]: pgmap v3912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:37:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:50 smithi047 ceph-mon[36835]: pgmap v3912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:37:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:50 smithi029 ceph-mon[32610]: pgmap v3912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:37:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:52 smithi077 ceph-mon[37153]: pgmap v3913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:37:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:52 smithi047 ceph-mon[36835]: pgmap v3913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:37:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:52 smithi029 ceph-mon[32610]: pgmap v3913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:37:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6494.504639 secs 2024-07-27T08:37:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6494.504639 secs 2024-07-27T08:37:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6494.504639 secs 2024-07-27T08:37:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:54 smithi077 ceph-mon[37153]: pgmap v3914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:54 smithi047 ceph-mon[36835]: pgmap v3914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:54 smithi029 ceph-mon[32610]: pgmap v3914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:37:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:56 smithi077 ceph-mon[37153]: pgmap v3915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:37:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:56 smithi047 ceph-mon[36835]: pgmap v3915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:37:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:56 smithi029 ceph-mon[32610]: pgmap v3915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:37:58.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:58.109+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:37:58.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:58.109+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:37:58.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:58.109+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:37:58.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:58.163+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:37:58.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:58.163+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:37:58.164 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:37:58.163+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:37:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:58.609+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:37:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:58.609+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:37:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:58.610+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:37:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:37:58.610+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:37:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6499.504809 secs 2024-07-27T08:37:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:37:58 smithi077 ceph-mon[37153]: pgmap v3916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:37:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6499.504809 secs 2024-07-27T08:37:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:37:58 smithi047 ceph-mon[36835]: pgmap v3916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:37:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6499.504809 secs 2024-07-27T08:37:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:37:58 smithi029 ceph-mon[32610]: pgmap v3916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:38:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:00 smithi077 ceph-mon[37153]: pgmap v3917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:38:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:00 smithi047 ceph-mon[36835]: pgmap v3917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:38:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:00 smithi029 ceph-mon[32610]: pgmap v3917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:38:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:02 smithi077 ceph-mon[37153]: pgmap v3918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:02 smithi047 ceph-mon[36835]: pgmap v3918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:02 smithi029 ceph-mon[32610]: pgmap v3918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6504.504915 secs 2024-07-27T08:38:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6504.504915 secs 2024-07-27T08:38:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6504.504915 secs 2024-07-27T08:38:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:04 smithi077 ceph-mon[37153]: pgmap v3919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:38:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:04 smithi047 ceph-mon[36835]: pgmap v3919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:38:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:04 smithi029 ceph-mon[32610]: pgmap v3919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:38:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:06 smithi077 ceph-mon[37153]: pgmap v3920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:06 smithi047 ceph-mon[36835]: pgmap v3920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:06 smithi029 ceph-mon[32610]: pgmap v3920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6509.505012 secs 2024-07-27T08:38:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:08 smithi077 ceph-mon[37153]: pgmap v3921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:38:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6509.505012 secs 2024-07-27T08:38:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:08 smithi047 ceph-mon[36835]: pgmap v3921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:38:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6509.505012 secs 2024-07-27T08:38:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:08 smithi029 ceph-mon[32610]: pgmap v3921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:38:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:10 smithi077 ceph-mon[37153]: pgmap v3922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:38:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:10 smithi047 ceph-mon[36835]: pgmap v3922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:38:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:10 smithi029 ceph-mon[32610]: pgmap v3922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:38:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:12 smithi077 ceph-mon[37153]: pgmap v3923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:12 smithi047 ceph-mon[36835]: pgmap v3923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:12 smithi029 ceph-mon[32610]: pgmap v3923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:13.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:38:13.124+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:38:13.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:38:13.124+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:38:13.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:38:13.124+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:38:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:38:13.178+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:38:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:38:13.178+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:38:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:38:13.178+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:38:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6514.505134 secs 2024-07-27T08:38:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6514.505134 secs 2024-07-27T08:38:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6514.505134 secs 2024-07-27T08:38:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:14 smithi077 ceph-mon[37153]: pgmap v3924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:38:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:14 smithi047 ceph-mon[36835]: pgmap v3924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:38:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:14 smithi029 ceph-mon[32610]: pgmap v3924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:38:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:38:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:38:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: pgmap v3925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:38:16.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:38:16.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: pgmap v3925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:38:16.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:38:16.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: pgmap v3925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:16.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:38:16.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:38:16.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:38:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:17 smithi077 ceph-mon[37153]: pgmap v3926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:38:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:17 smithi077 ceph-mon[37153]: pgmap v3927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:38:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:17 smithi047 ceph-mon[36835]: pgmap v3926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:38:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:17 smithi047 ceph-mon[36835]: pgmap v3927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:38:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:17 smithi029 ceph-mon[32610]: pgmap v3926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:38:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:17 smithi029 ceph-mon[32610]: pgmap v3927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:38:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6519.505255 secs 2024-07-27T08:38:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6519.505255 secs 2024-07-27T08:38:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6519.505255 secs 2024-07-27T08:38:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:19 smithi077 ceph-mon[37153]: pgmap v3928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:38:19.943 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:19 smithi029 ceph-mon[32610]: pgmap v3928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:38:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:19 smithi047 ceph-mon[36835]: pgmap v3928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:38:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:21 smithi077 ceph-mon[37153]: pgmap v3929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:38:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:21 smithi047 ceph-mon[36835]: pgmap v3929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:38:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:21 smithi029 ceph-mon[32610]: pgmap v3929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:38:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6524.505377 secs 2024-07-27T08:38:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:23 smithi077 ceph-mon[37153]: pgmap v3930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6524.505377 secs 2024-07-27T08:38:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:23 smithi047 ceph-mon[36835]: pgmap v3930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6524.505377 secs 2024-07-27T08:38:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:23 smithi029 ceph-mon[32610]: pgmap v3930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:25 smithi077 ceph-mon[37153]: pgmap v3931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:38:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:25 smithi047 ceph-mon[36835]: pgmap v3931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:38:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:25 smithi029 ceph-mon[32610]: pgmap v3931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:38:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:27 smithi077 ceph-mon[37153]: pgmap v3932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:38:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:27 smithi047 ceph-mon[36835]: pgmap v3932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:38:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:27 smithi029 ceph-mon[32610]: pgmap v3932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:38:28.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:38:28.133+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:38:28.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:38:28.133+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:38:28.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:38:28.133+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:38:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:38:28.186+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:38:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:38:28.186+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:38:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:38:28.186+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:38:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6529.505487 secs 2024-07-27T08:38:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6529.505487 secs 2024-07-27T08:38:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6529.505487 secs 2024-07-27T08:38:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:29 smithi047 ceph-mon[36835]: pgmap v3933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:38:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:29 smithi029 ceph-mon[32610]: pgmap v3933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:38:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:29 smithi077 ceph-mon[37153]: pgmap v3933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:38:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:31 smithi047 ceph-mon[36835]: pgmap v3934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:31 smithi029 ceph-mon[32610]: pgmap v3934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:31 smithi077 ceph-mon[37153]: pgmap v3934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6534.505642 secs 2024-07-27T08:38:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:33 smithi047 ceph-mon[36835]: pgmap v3935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:38:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6534.505642 secs 2024-07-27T08:38:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:33 smithi029 ceph-mon[32610]: pgmap v3935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:38:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6534.505642 secs 2024-07-27T08:38:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:33 smithi077 ceph-mon[37153]: pgmap v3935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:38:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:34 smithi047 ceph-mon[36835]: pgmap v3936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:38:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:34 smithi029 ceph-mon[32610]: pgmap v3936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:38:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:34 smithi077 ceph-mon[37153]: pgmap v3936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:38:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:37 smithi077 ceph-mon[37153]: pgmap v3937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:37 smithi047 ceph-mon[36835]: pgmap v3937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:37 smithi029 ceph-mon[32610]: pgmap v3937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6539.505795 secs 2024-07-27T08:38:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6539.505795 secs 2024-07-27T08:38:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6539.505795 secs 2024-07-27T08:38:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:39 smithi077 ceph-mon[37153]: pgmap v3938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:38:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:39 smithi047 ceph-mon[36835]: pgmap v3938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:38:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:39 smithi029 ceph-mon[32610]: pgmap v3938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:38:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:41 smithi077 ceph-mon[37153]: pgmap v3939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:38:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:41 smithi047 ceph-mon[36835]: pgmap v3939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:38:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:41 smithi029 ceph-mon[32610]: pgmap v3939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:38:43.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:38:43.148+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:38:43.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:38:43.148+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:38:43.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:38:43.148+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:38:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:38:43.202+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:38:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:38:43.202+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:38:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:38:43.202+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:38:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6544.505891 secs 2024-07-27T08:38:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:43 smithi077 ceph-mon[37153]: pgmap v3940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:38:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6544.505891 secs 2024-07-27T08:38:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:43 smithi047 ceph-mon[36835]: pgmap v3940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:38:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6544.505891 secs 2024-07-27T08:38:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:43 smithi029 ceph-mon[32610]: pgmap v3940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:38:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:45 smithi077 ceph-mon[37153]: pgmap v3941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:38:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:45 smithi047 ceph-mon[36835]: pgmap v3941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:38:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:45 smithi029 ceph-mon[32610]: pgmap v3941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:38:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:47 smithi077 ceph-mon[37153]: pgmap v3942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:47 smithi047 ceph-mon[36835]: pgmap v3942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:47 smithi029 ceph-mon[32610]: pgmap v3942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6549.505989 secs 2024-07-27T08:38:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6549.505989 secs 2024-07-27T08:38:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6549.505989 secs 2024-07-27T08:38:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:49 smithi077 ceph-mon[37153]: pgmap v3943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:38:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:49 smithi047 ceph-mon[36835]: pgmap v3943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:38:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:49 smithi029 ceph-mon[32610]: pgmap v3943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:38:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:51 smithi077 ceph-mon[37153]: pgmap v3944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:51 smithi047 ceph-mon[36835]: pgmap v3944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:51 smithi029 ceph-mon[32610]: pgmap v3944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:38:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6554.506113 secs 2024-07-27T08:38:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:53 smithi077 ceph-mon[37153]: pgmap v3945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:38:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6554.506113 secs 2024-07-27T08:38:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:53 smithi047 ceph-mon[36835]: pgmap v3945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:38:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6554.506113 secs 2024-07-27T08:38:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:53 smithi029 ceph-mon[32610]: pgmap v3945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:38:55.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:55 smithi077 ceph-mon[37153]: pgmap v3946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:38:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:55 smithi047 ceph-mon[36835]: pgmap v3946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:38:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:55 smithi029 ceph-mon[32610]: pgmap v3946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:38:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:57 smithi077 ceph-mon[37153]: pgmap v3947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:57 smithi047 ceph-mon[36835]: pgmap v3947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:57 smithi029 ceph-mon[32610]: pgmap v3947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:38:58.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:38:58.153+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:38:58.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:38:58.153+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:38:58.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:38:58.153+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:38:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:38:58.217+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:38:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:38:58.217+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:38:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:38:58.217+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:38:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6559.506239 secs 2024-07-27T08:38:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6559.506239 secs 2024-07-27T08:38:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6559.506239 secs 2024-07-27T08:38:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:38:59 smithi077 ceph-mon[37153]: pgmap v3948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:38:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:38:59 smithi047 ceph-mon[36835]: pgmap v3948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:38:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:38:59 smithi029 ceph-mon[32610]: pgmap v3948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:01 smithi077 ceph-mon[37153]: pgmap v3949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:01 smithi047 ceph-mon[36835]: pgmap v3949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:01 smithi029 ceph-mon[32610]: pgmap v3949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6564.506367 secs 2024-07-27T08:39:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:03 smithi077 ceph-mon[37153]: pgmap v3950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:39:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6564.506367 secs 2024-07-27T08:39:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:03 smithi047 ceph-mon[36835]: pgmap v3950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:39:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6564.506367 secs 2024-07-27T08:39:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:03 smithi029 ceph-mon[32610]: pgmap v3950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:39:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:05 smithi077 ceph-mon[37153]: pgmap v3951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:39:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:05 smithi047 ceph-mon[36835]: pgmap v3951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:39:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:05 smithi029 ceph-mon[32610]: pgmap v3951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:39:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:07 smithi077 ceph-mon[37153]: pgmap v3952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:39:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:07 smithi047 ceph-mon[36835]: pgmap v3952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:39:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:07 smithi029 ceph-mon[32610]: pgmap v3952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:39:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6569.506492 secs 2024-07-27T08:39:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6569.506492 secs 2024-07-27T08:39:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6569.506492 secs 2024-07-27T08:39:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:09 smithi077 ceph-mon[37153]: pgmap v3953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:09 smithi047 ceph-mon[36835]: pgmap v3953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:09 smithi029 ceph-mon[32610]: pgmap v3953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:11 smithi077 ceph-mon[37153]: pgmap v3954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:11 smithi047 ceph-mon[36835]: pgmap v3954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:11 smithi029 ceph-mon[32610]: pgmap v3954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:13.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:39:13.158+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:39:13.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:39:13.158+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:39:13.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:39:13.158+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:39:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:39:13.233+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:39:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:39:13.233+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:39:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:39:13.233+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:39:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6574.506621 secs 2024-07-27T08:39:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:13 smithi077 ceph-mon[37153]: pgmap v3955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:39:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6574.506621 secs 2024-07-27T08:39:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:13 smithi047 ceph-mon[36835]: pgmap v3955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:39:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6574.506621 secs 2024-07-27T08:39:13.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:13 smithi029 ceph-mon[32610]: pgmap v3955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:39:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:15 smithi077 ceph-mon[37153]: pgmap v3956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:39:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:15 smithi047 ceph-mon[36835]: pgmap v3956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:39:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:15 smithi029 ceph-mon[32610]: pgmap v3956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:39:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:39:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:39:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:39:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:17 smithi077 ceph-mon[37153]: pgmap v3957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:39:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:39:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:39:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:39:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:17 smithi047 ceph-mon[36835]: pgmap v3957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:39:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:39:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:39:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:39:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:17 smithi029 ceph-mon[32610]: pgmap v3957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:39:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:39:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:39:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:39:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6579.506717 secs 2024-07-27T08:39:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:18 smithi077 ceph-mon[37153]: pgmap v3958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:39:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:18 smithi077 ceph-mon[37153]: pgmap v3959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:39:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6579.506717 secs 2024-07-27T08:39:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:18 smithi047 ceph-mon[36835]: pgmap v3958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:39:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:18 smithi047 ceph-mon[36835]: pgmap v3959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:39:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6579.506717 secs 2024-07-27T08:39:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:18 smithi029 ceph-mon[32610]: pgmap v3958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:39:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:18 smithi029 ceph-mon[32610]: pgmap v3959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:39:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:20 smithi077 ceph-mon[37153]: pgmap v3960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:39:20.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:20 smithi047 ceph-mon[36835]: pgmap v3960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:39:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:20 smithi029 ceph-mon[32610]: pgmap v3960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:39:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:22 smithi077 ceph-mon[37153]: pgmap v3961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:39:22.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:22 smithi029 ceph-mon[32610]: pgmap v3961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:39:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:22 smithi047 ceph-mon[36835]: pgmap v3961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:39:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6584.506848 secs 2024-07-27T08:39:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6584.506848 secs 2024-07-27T08:39:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6584.506848 secs 2024-07-27T08:39:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:24 smithi047 ceph-mon[36835]: pgmap v3962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:39:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:24 smithi029 ceph-mon[32610]: pgmap v3962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:39:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:24 smithi077 ceph-mon[37153]: pgmap v3962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:39:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:26 smithi047 ceph-mon[36835]: pgmap v3963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:39:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:26 smithi029 ceph-mon[32610]: pgmap v3963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:39:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:26 smithi077 ceph-mon[37153]: pgmap v3963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:39:28.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:39:28.172+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:39:28.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:39:28.172+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:39:28.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:39:28.172+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:39:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:39:28.247+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:39:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:39:28.247+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:39:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:39:28.247+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:39:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6589.506984 secs 2024-07-27T08:39:28.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:28 smithi047 ceph-mon[36835]: pgmap v3964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:39:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6589.506984 secs 2024-07-27T08:39:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:28 smithi029 ceph-mon[32610]: pgmap v3964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:39:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6589.506984 secs 2024-07-27T08:39:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:28 smithi077 ceph-mon[37153]: pgmap v3964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:39:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:30 smithi047 ceph-mon[36835]: pgmap v3965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:30 smithi029 ceph-mon[32610]: pgmap v3965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:30 smithi077 ceph-mon[37153]: pgmap v3965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:32 smithi047 ceph-mon[36835]: pgmap v3966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:32 smithi029 ceph-mon[32610]: pgmap v3966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:32 smithi077 ceph-mon[37153]: pgmap v3966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6594.507126 secs 2024-07-27T08:39:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6594.507126 secs 2024-07-27T08:39:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6594.507126 secs 2024-07-27T08:39:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:34 smithi047 ceph-mon[36835]: pgmap v3967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:39:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:34 smithi029 ceph-mon[32610]: pgmap v3967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:39:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:34 smithi077 ceph-mon[37153]: pgmap v3967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:39:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:35 smithi047 ceph-mon[36835]: pgmap v3968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:35 smithi029 ceph-mon[32610]: pgmap v3968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:35 smithi077 ceph-mon[37153]: pgmap v3968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6599.507253 secs 2024-07-27T08:39:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:38 smithi077 ceph-mon[37153]: pgmap v3969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:39:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6599.507253 secs 2024-07-27T08:39:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:38 smithi047 ceph-mon[36835]: pgmap v3969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:39:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6599.507253 secs 2024-07-27T08:39:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:38 smithi029 ceph-mon[32610]: pgmap v3969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:39:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:40 smithi077 ceph-mon[37153]: pgmap v3970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:40 smithi047 ceph-mon[36835]: pgmap v3970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:40 smithi029 ceph-mon[32610]: pgmap v3970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:42 smithi077 ceph-mon[37153]: pgmap v3971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:42 smithi047 ceph-mon[36835]: pgmap v3971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:42 smithi029 ceph-mon[32610]: pgmap v3971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:43.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:39:43.180+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:39:43.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:39:43.180+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:39:43.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:39:43.180+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:39:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:39:43.263+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:39:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:39:43.263+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:39:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:39:43.263+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:39:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6604.507408 secs 2024-07-27T08:39:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6604.507408 secs 2024-07-27T08:39:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6604.507408 secs 2024-07-27T08:39:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:44 smithi077 ceph-mon[37153]: pgmap v3972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:39:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:44 smithi047 ceph-mon[36835]: pgmap v3972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:39:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:44 smithi029 ceph-mon[32610]: pgmap v3972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:39:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:46 smithi077 ceph-mon[37153]: pgmap v3973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:46 smithi047 ceph-mon[36835]: pgmap v3973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:46 smithi029 ceph-mon[32610]: pgmap v3973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6609.507551 secs 2024-07-27T08:39:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:48 smithi077 ceph-mon[37153]: pgmap v3974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:39:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6609.507551 secs 2024-07-27T08:39:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:48 smithi047 ceph-mon[36835]: pgmap v3974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:39:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6609.507551 secs 2024-07-27T08:39:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:48 smithi029 ceph-mon[32610]: pgmap v3974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:39:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:50 smithi077 ceph-mon[37153]: pgmap v3975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:50 smithi047 ceph-mon[36835]: pgmap v3975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:50 smithi029 ceph-mon[32610]: pgmap v3975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:39:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:52 smithi077 ceph-mon[37153]: pgmap v3976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:52 smithi047 ceph-mon[36835]: pgmap v3976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:52 smithi029 ceph-mon[32610]: pgmap v3976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6614.507685 secs 2024-07-27T08:39:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6614.507685 secs 2024-07-27T08:39:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6614.507685 secs 2024-07-27T08:39:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:54 smithi077 ceph-mon[37153]: pgmap v3977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:39:54.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:54 smithi029 ceph-mon[32610]: pgmap v3977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:39:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:54 smithi047 ceph-mon[36835]: pgmap v3977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:39:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:56 smithi077 ceph-mon[37153]: pgmap v3978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:56 smithi047 ceph-mon[36835]: pgmap v3978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:56 smithi029 ceph-mon[32610]: pgmap v3978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:39:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:39:58.185+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:39:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:39:58.185+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:39:58.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:39:58.185+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:39:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:39:58.279+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:39:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:39:58.279+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:39:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:39:58.279+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:39:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6619.507820 secs 2024-07-27T08:39:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:39:58 smithi077 ceph-mon[37153]: pgmap v3979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:39:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6619.507820 secs 2024-07-27T08:39:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:39:58 smithi047 ceph-mon[36835]: pgmap v3979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:39:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6619.507820 secs 2024-07-27T08:39:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:39:58 smithi029 ceph-mon[32610]: pgmap v3979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:40:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:00 smithi047 ceph-mon[36835]: pgmap v3980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:00.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:40:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:00 smithi029 ceph-mon[32610]: pgmap v3980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:40:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:00 smithi077 ceph-mon[37153]: pgmap v3980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:40:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:02 smithi047 ceph-mon[36835]: pgmap v3981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:02 smithi029 ceph-mon[32610]: pgmap v3981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:02 smithi077 ceph-mon[37153]: pgmap v3981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6624.507958 secs 2024-07-27T08:40:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6624.507958 secs 2024-07-27T08:40:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6624.507958 secs 2024-07-27T08:40:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:04 smithi047 ceph-mon[36835]: pgmap v3982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:40:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:04 smithi029 ceph-mon[32610]: pgmap v3982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:40:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:04 smithi077 ceph-mon[37153]: pgmap v3982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:40:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:06 smithi047 ceph-mon[36835]: pgmap v3983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:06 smithi029 ceph-mon[32610]: pgmap v3983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:06 smithi077 ceph-mon[37153]: pgmap v3983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6629.508096 secs 2024-07-27T08:40:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:08 smithi047 ceph-mon[36835]: pgmap v3984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:40:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6629.508096 secs 2024-07-27T08:40:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:08 smithi029 ceph-mon[32610]: pgmap v3984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:40:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6629.508096 secs 2024-07-27T08:40:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:08 smithi077 ceph-mon[37153]: pgmap v3984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:40:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:10 smithi047 ceph-mon[36835]: pgmap v3985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:10 smithi029 ceph-mon[32610]: pgmap v3985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:10 smithi077 ceph-mon[37153]: pgmap v3985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:12 smithi047 ceph-mon[36835]: pgmap v3986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:12 smithi029 ceph-mon[32610]: pgmap v3986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:12 smithi077 ceph-mon[37153]: pgmap v3986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:13.195+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:40:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:13.195+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:40:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:13.195+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:40:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:13.293+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:40:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:13.293+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:40:13.294 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:13.293+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:40:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6634.508218 secs 2024-07-27T08:40:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6634.508218 secs 2024-07-27T08:40:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6634.508218 secs 2024-07-27T08:40:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:14 smithi047 ceph-mon[36835]: pgmap v3987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:40:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:14 smithi029 ceph-mon[32610]: pgmap v3987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:40:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:14 smithi077 ceph-mon[37153]: pgmap v3987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:40:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:16 smithi047 ceph-mon[36835]: pgmap v3988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:16 smithi029 ceph-mon[32610]: pgmap v3988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:16 smithi077 ceph-mon[37153]: pgmap v3988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:40:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:40:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:40:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6639.508294 secs 2024-07-27T08:40:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:18 smithi047 ceph-mon[36835]: pgmap v3989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:40:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:40:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:40:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:18 smithi047 ceph-mon[36835]: pgmap v3990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T08:40:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:18 smithi047 ceph-mon[36835]: pgmap v3991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:40:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:40:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6639.508294 secs 2024-07-27T08:40:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:18 smithi029 ceph-mon[32610]: pgmap v3989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:40:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:40:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:40:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:18 smithi029 ceph-mon[32610]: pgmap v3990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T08:40:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:18 smithi029 ceph-mon[32610]: pgmap v3991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:40:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:40:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6639.508294 secs 2024-07-27T08:40:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:18 smithi077 ceph-mon[37153]: pgmap v3989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:40:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:40:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:40:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:18 smithi077 ceph-mon[37153]: pgmap v3990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T08:40:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:18 smithi077 ceph-mon[37153]: pgmap v3991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:40:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:40:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:20 smithi047 ceph-mon[36835]: pgmap v3992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:40:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:20 smithi029 ceph-mon[32610]: pgmap v3992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:40:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:20 smithi077 ceph-mon[37153]: pgmap v3992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:40:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:22 smithi047 ceph-mon[36835]: pgmap v3993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:40:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:22 smithi029 ceph-mon[32610]: pgmap v3993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:40:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:22 smithi077 ceph-mon[37153]: pgmap v3993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:40:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6644.508364 secs 2024-07-27T08:40:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6644.508364 secs 2024-07-27T08:40:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6644.508364 secs 2024-07-27T08:40:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:24 smithi077 ceph-mon[37153]: pgmap v3994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:40:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:24 smithi047 ceph-mon[36835]: pgmap v3994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:40:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:24 smithi029 ceph-mon[32610]: pgmap v3994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:40:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:26 smithi047 ceph-mon[36835]: pgmap v3995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:40:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:26 smithi029 ceph-mon[32610]: pgmap v3995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:40:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:26 smithi077 ceph-mon[37153]: pgmap v3995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:40:28.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:28.209+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:40:28.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:28.209+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:40:28.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:28.209+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:40:28.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:28.307+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:40:28.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:28.307+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:40:28.309 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:28.307+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:40:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6649.508505 secs 2024-07-27T08:40:28.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:28 smithi047 ceph-mon[36835]: pgmap v3996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:40:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6649.508505 secs 2024-07-27T08:40:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:28 smithi029 ceph-mon[32610]: pgmap v3996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:40:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6649.508505 secs 2024-07-27T08:40:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:28 smithi077 ceph-mon[37153]: pgmap v3996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:40:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:30 smithi047 ceph-mon[36835]: pgmap v3997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:40:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:30 smithi029 ceph-mon[32610]: pgmap v3997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:40:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:30 smithi077 ceph-mon[37153]: pgmap v3997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:40:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:32 smithi047 ceph-mon[36835]: pgmap v3998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:40:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:32 smithi029 ceph-mon[32610]: pgmap v3998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:40:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:32 smithi077 ceph-mon[37153]: pgmap v3998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:40:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6654.508661 secs 2024-07-27T08:40:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6654.508661 secs 2024-07-27T08:40:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6654.508661 secs 2024-07-27T08:40:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:34 smithi047 ceph-mon[36835]: pgmap v3999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:34 smithi029 ceph-mon[32610]: pgmap v3999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:34 smithi077 ceph-mon[37153]: pgmap v3999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:35 smithi047 ceph-mon[36835]: pgmap v4000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:35 smithi029 ceph-mon[32610]: pgmap v4000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:35 smithi077 ceph-mon[37153]: pgmap v4000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6659.508780 secs 2024-07-27T08:40:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:38 smithi047 ceph-mon[36835]: pgmap v4001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:40:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6659.508780 secs 2024-07-27T08:40:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:38 smithi029 ceph-mon[32610]: pgmap v4001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:40:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6659.508780 secs 2024-07-27T08:40:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:38 smithi077 ceph-mon[37153]: pgmap v4001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:40:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:40 smithi047 ceph-mon[36835]: pgmap v4002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:40:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:40 smithi029 ceph-mon[32610]: pgmap v4002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:40:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:40 smithi077 ceph-mon[37153]: pgmap v4002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:40:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:42 smithi047 ceph-mon[36835]: pgmap v4003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:40:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:42 smithi029 ceph-mon[32610]: pgmap v4003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:40:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:42 smithi077 ceph-mon[37153]: pgmap v4003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:40:43.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:43.225+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:40:43.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:43.225+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:40:43.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:43.225+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:40:43.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:43.309+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:40:43.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:43.309+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:40:43.311 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:43.309+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:40:43.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:43.634+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:40:43.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:43.634+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:40:43.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:43.635+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:40:43.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:43.635+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:40:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6664.508906 secs 2024-07-27T08:40:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6664.508906 secs 2024-07-27T08:40:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6664.508906 secs 2024-07-27T08:40:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:44 smithi047 ceph-mon[36835]: pgmap v4004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:44 smithi029 ceph-mon[32610]: pgmap v4004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:44 smithi077 ceph-mon[37153]: pgmap v4004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:46 smithi047 ceph-mon[36835]: pgmap v4005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:46 smithi029 ceph-mon[32610]: pgmap v4005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:46 smithi077 ceph-mon[37153]: pgmap v4005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:47 smithi047 ceph-mon[36835]: pgmap v4006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:40:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:47 smithi029 ceph-mon[32610]: pgmap v4006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:40:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:47 smithi077 ceph-mon[37153]: pgmap v4006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:40:48.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6669.509053 secs 2024-07-27T08:40:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6669.509053 secs 2024-07-27T08:40:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6669.509053 secs 2024-07-27T08:40:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:49 smithi029 ceph-mon[32610]: pgmap v4007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:40:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:49 smithi077 ceph-mon[37153]: pgmap v4007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:40:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:49 smithi047 ceph-mon[36835]: pgmap v4007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:40:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:52 smithi047 ceph-mon[36835]: pgmap v4008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:40:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:52 smithi029 ceph-mon[32610]: pgmap v4008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:40:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:52 smithi077 ceph-mon[37153]: pgmap v4008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:40:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6674.509147 secs 2024-07-27T08:40:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6674.509147 secs 2024-07-27T08:40:54.007 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6674.509147 secs 2024-07-27T08:40:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:54 smithi047 ceph-mon[36835]: pgmap v4009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:54 smithi029 ceph-mon[32610]: pgmap v4009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:54 smithi077 ceph-mon[37153]: pgmap v4009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:40:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:56 smithi047 ceph-mon[36835]: pgmap v4010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:56 smithi029 ceph-mon[32610]: pgmap v4010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:56 smithi077 ceph-mon[37153]: pgmap v4010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:40:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:58.241+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:40:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:58.241+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:40:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:58.241+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:58.324+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:58.324+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:40:58.326 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:40:58.324+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:40:58.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:58.637+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:40:58.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:58.637+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:40:58.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:58.642+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:40:58.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:40:58.642+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:40:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6679.509268 secs 2024-07-27T08:40:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:58 smithi047 ceph-mon[36835]: pgmap v4011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:40:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6679.509268 secs 2024-07-27T08:40:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:58 smithi029 ceph-mon[32610]: pgmap v4011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:40:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6679.509268 secs 2024-07-27T08:40:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:58 smithi077 ceph-mon[37153]: pgmap v4011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:40:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:40:59 smithi047 ceph-mon[36835]: pgmap v4012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:40:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:40:59 smithi029 ceph-mon[32610]: pgmap v4012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:41:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:40:59 smithi077 ceph-mon[37153]: pgmap v4012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:41:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:02 smithi047 ceph-mon[36835]: pgmap v4013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:02 smithi029 ceph-mon[32610]: pgmap v4013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:02 smithi077 ceph-mon[37153]: pgmap v4013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6684.509381 secs 2024-07-27T08:41:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6684.509381 secs 2024-07-27T08:41:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6684.509381 secs 2024-07-27T08:41:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:04 smithi047 ceph-mon[36835]: pgmap v4014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:04 smithi029 ceph-mon[32610]: pgmap v4014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:04 smithi077 ceph-mon[37153]: pgmap v4014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:06 smithi047 ceph-mon[36835]: pgmap v4015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:41:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:06 smithi029 ceph-mon[32610]: pgmap v4015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:41:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:06 smithi077 ceph-mon[37153]: pgmap v4015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:41:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6689.509505 secs 2024-07-27T08:41:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:08 smithi047 ceph-mon[36835]: pgmap v4016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:41:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6689.509505 secs 2024-07-27T08:41:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:08 smithi029 ceph-mon[32610]: pgmap v4016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:41:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6689.509505 secs 2024-07-27T08:41:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:08 smithi077 ceph-mon[37153]: pgmap v4016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:41:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:09 smithi047 ceph-mon[36835]: pgmap v4017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:41:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:09 smithi029 ceph-mon[32610]: pgmap v4017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:41:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:09 smithi077 ceph-mon[37153]: pgmap v4017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:41:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:12 smithi047 ceph-mon[36835]: pgmap v4018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:12 smithi029 ceph-mon[32610]: pgmap v4018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:12 smithi077 ceph-mon[37153]: pgmap v4018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:13.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:41:13.256+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:41:13.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:41:13.256+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:41:13.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:41:13.256+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:41:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:41:13.334+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:41:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:41:13.334+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:41:13.335 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:41:13.334+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:41:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6694.509631 secs 2024-07-27T08:41:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6694.509631 secs 2024-07-27T08:41:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6694.509631 secs 2024-07-27T08:41:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:14 smithi047 ceph-mon[36835]: pgmap v4019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:14 smithi029 ceph-mon[32610]: pgmap v4019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:14 smithi077 ceph-mon[37153]: pgmap v4019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:16 smithi047 ceph-mon[36835]: pgmap v4020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:41:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:16 smithi029 ceph-mon[32610]: pgmap v4020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:41:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:16 smithi077 ceph-mon[37153]: pgmap v4020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:41:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:17 smithi047 ceph-mon[36835]: pgmap v4021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:41:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:17 smithi029 ceph-mon[32610]: pgmap v4021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:41:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:17 smithi077 ceph-mon[37153]: pgmap v4021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:41:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6699.509756 secs 2024-07-27T08:41:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:41:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:41:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:41:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:18 smithi047 ceph-mon[36835]: pgmap v4022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:41:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:18 smithi047 ceph-mon[36835]: pgmap v4023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:41:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:41:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6699.509756 secs 2024-07-27T08:41:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:41:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:41:18.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:41:18.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:18 smithi029 ceph-mon[32610]: pgmap v4022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:41:18.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:18 smithi029 ceph-mon[32610]: pgmap v4023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:41:18.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:41:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6699.509756 secs 2024-07-27T08:41:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:41:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:41:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:41:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:18 smithi077 ceph-mon[37153]: pgmap v4022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:41:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:18 smithi077 ceph-mon[37153]: pgmap v4023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:41:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:41:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:21 smithi047 ceph-mon[36835]: pgmap v4024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:41:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:21 smithi029 ceph-mon[32610]: pgmap v4024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:41:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:21 smithi077 ceph-mon[37153]: pgmap v4024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:41:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6704.509882 secs 2024-07-27T08:41:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:23 smithi047 ceph-mon[36835]: pgmap v4025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:41:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6704.509882 secs 2024-07-27T08:41:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:23 smithi029 ceph-mon[32610]: pgmap v4025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:41:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6704.509882 secs 2024-07-27T08:41:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:23 smithi077 ceph-mon[37153]: pgmap v4025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:41:25.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:25 smithi047 ceph-mon[36835]: pgmap v4026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:41:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:25 smithi029 ceph-mon[32610]: pgmap v4026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:41:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:25 smithi077 ceph-mon[37153]: pgmap v4026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:41:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:27 smithi047 ceph-mon[36835]: pgmap v4027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:41:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:27 smithi029 ceph-mon[32610]: pgmap v4027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:41:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:27 smithi077 ceph-mon[37153]: pgmap v4027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:41:28.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:41:28.270+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:41:28.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:41:28.270+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:41:28.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:41:28.270+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:41:28.340 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:41:28.339+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:41:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:41:28.339+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:41:28.341 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:41:28.339+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:41:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6709.509961 secs 2024-07-27T08:41:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6709.509961 secs 2024-07-27T08:41:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6709.509961 secs 2024-07-27T08:41:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:29 smithi047 ceph-mon[36835]: pgmap v4028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:41:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:29 smithi029 ceph-mon[32610]: pgmap v4028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:41:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:29 smithi077 ceph-mon[37153]: pgmap v4028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:41:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:31 smithi047 ceph-mon[36835]: pgmap v4029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:41:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:31 smithi029 ceph-mon[32610]: pgmap v4029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:41:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:31 smithi077 ceph-mon[37153]: pgmap v4029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:41:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6714.510088 secs 2024-07-27T08:41:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:33 smithi029 ceph-mon[32610]: pgmap v4030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6714.510088 secs 2024-07-27T08:41:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:33 smithi077 ceph-mon[37153]: pgmap v4030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6714.510088 secs 2024-07-27T08:41:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:33 smithi047 ceph-mon[36835]: pgmap v4030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:35 smithi029 ceph-mon[32610]: pgmap v4031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:41:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:35 smithi077 ceph-mon[37153]: pgmap v4031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:41:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:35 smithi047 ceph-mon[36835]: pgmap v4031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:41:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:37 smithi029 ceph-mon[32610]: pgmap v4032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:37 smithi077 ceph-mon[37153]: pgmap v4032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:37 smithi047 ceph-mon[36835]: pgmap v4032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6719.510212 secs 2024-07-27T08:41:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6719.510212 secs 2024-07-27T08:41:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6719.510212 secs 2024-07-27T08:41:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:39 smithi029 ceph-mon[32610]: pgmap v4033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:39 smithi077 ceph-mon[37153]: pgmap v4033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:39 smithi047 ceph-mon[36835]: pgmap v4033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:41 smithi077 ceph-mon[37153]: pgmap v4034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:41:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:41 smithi047 ceph-mon[36835]: pgmap v4034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:41:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:41 smithi029 ceph-mon[32610]: pgmap v4034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:41:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:41:43.286+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:41:43.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:41:43.286+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:41:43.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:41:43.286+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:41:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:41:43.354+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:41:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:41:43.354+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:41:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:41:43.354+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:41:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6724.510281 secs 2024-07-27T08:41:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:43 smithi077 ceph-mon[37153]: pgmap v4035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6724.510281 secs 2024-07-27T08:41:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:43 smithi047 ceph-mon[36835]: pgmap v4035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6724.510281 secs 2024-07-27T08:41:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:43 smithi029 ceph-mon[32610]: pgmap v4035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:45 smithi077 ceph-mon[37153]: pgmap v4036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:41:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:45 smithi047 ceph-mon[36835]: pgmap v4036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:41:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:45 smithi029 ceph-mon[32610]: pgmap v4036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:41:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:47 smithi077 ceph-mon[37153]: pgmap v4037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:47 smithi047 ceph-mon[36835]: pgmap v4037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:47 smithi029 ceph-mon[32610]: pgmap v4037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6729.510379 secs 2024-07-27T08:41:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6729.510379 secs 2024-07-27T08:41:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6729.510379 secs 2024-07-27T08:41:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:49 smithi077 ceph-mon[37153]: pgmap v4038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:49 smithi047 ceph-mon[36835]: pgmap v4038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:49 smithi029 ceph-mon[32610]: pgmap v4038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:51 smithi077 ceph-mon[37153]: pgmap v4039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:41:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:51 smithi047 ceph-mon[36835]: pgmap v4039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:41:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:51 smithi029 ceph-mon[32610]: pgmap v4039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:41:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6734.510501 secs 2024-07-27T08:41:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:53 smithi077 ceph-mon[37153]: pgmap v4040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6734.510501 secs 2024-07-27T08:41:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:53 smithi047 ceph-mon[36835]: pgmap v4040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6734.510501 secs 2024-07-27T08:41:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:53 smithi029 ceph-mon[32610]: pgmap v4040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:55 smithi077 ceph-mon[37153]: pgmap v4041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:41:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:55 smithi047 ceph-mon[36835]: pgmap v4041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:41:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:55 smithi029 ceph-mon[32610]: pgmap v4041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:41:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:57 smithi077 ceph-mon[37153]: pgmap v4042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:57 smithi047 ceph-mon[36835]: pgmap v4042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:57 smithi029 ceph-mon[32610]: pgmap v4042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:41:58.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:41:58.294+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:41:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:41:58.294+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:41:58.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:41:58.294+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:41:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:41:58.370+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:41:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:41:58.370+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:41:58.371 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:41:58.370+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:41:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6739.510626 secs 2024-07-27T08:41:58.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6739.510626 secs 2024-07-27T08:41:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6739.510626 secs 2024-07-27T08:41:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:41:59 smithi077 ceph-mon[37153]: pgmap v4043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:41:59 smithi047 ceph-mon[36835]: pgmap v4043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:41:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:41:59 smithi029 ceph-mon[32610]: pgmap v4043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:01 smithi077 ceph-mon[37153]: pgmap v4044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:01 smithi047 ceph-mon[36835]: pgmap v4044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:01 smithi029 ceph-mon[32610]: pgmap v4044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6744.510746 secs 2024-07-27T08:42:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:03 smithi077 ceph-mon[37153]: pgmap v4045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6744.510746 secs 2024-07-27T08:42:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:03 smithi047 ceph-mon[36835]: pgmap v4045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6744.510746 secs 2024-07-27T08:42:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:03 smithi029 ceph-mon[32610]: pgmap v4045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:05 smithi077 ceph-mon[37153]: pgmap v4046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:42:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:05 smithi047 ceph-mon[36835]: pgmap v4046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:42:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:05 smithi029 ceph-mon[32610]: pgmap v4046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:42:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:06 smithi047 ceph-mon[36835]: pgmap v4047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:06 smithi029 ceph-mon[32610]: pgmap v4047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:06 smithi077 ceph-mon[37153]: pgmap v4047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6749.510890 secs 2024-07-27T08:42:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6749.510890 secs 2024-07-27T08:42:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6749.510890 secs 2024-07-27T08:42:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:08 smithi077 ceph-mon[37153]: pgmap v4048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:08 smithi047 ceph-mon[36835]: pgmap v4048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:08 smithi029 ceph-mon[32610]: pgmap v4048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:11 smithi047 ceph-mon[36835]: pgmap v4049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:11 smithi029 ceph-mon[32610]: pgmap v4049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:11 smithi077 ceph-mon[37153]: pgmap v4049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:13.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:42:13.309+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:42:13.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:42:13.309+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:42:13.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:42:13.309+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:42:13.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:42:13.385+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:42:13.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:42:13.385+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:42:13.386 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:42:13.385+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:42:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6754.510989 secs 2024-07-27T08:42:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:13 smithi047 ceph-mon[36835]: pgmap v4050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6754.510989 secs 2024-07-27T08:42:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:13 smithi029 ceph-mon[32610]: pgmap v4050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6754.510989 secs 2024-07-27T08:42:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:13 smithi077 ceph-mon[37153]: pgmap v4050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:15 smithi029 ceph-mon[32610]: pgmap v4051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:42:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:15 smithi077 ceph-mon[37153]: pgmap v4051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:42:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:15 smithi047 ceph-mon[36835]: pgmap v4051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:42:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:17 smithi029 ceph-mon[32610]: pgmap v4052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:17 smithi077 ceph-mon[37153]: pgmap v4052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:17 smithi047 ceph-mon[36835]: pgmap v4052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6759.511095 secs 2024-07-27T08:42:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6759.511095 secs 2024-07-27T08:42:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6759.511095 secs 2024-07-27T08:42:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:19 smithi029 ceph-mon[32610]: pgmap v4053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:42:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:42:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:42:19.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:42:19.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:19 smithi077 ceph-mon[37153]: pgmap v4053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:19.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:42:19.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:42:19.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:42:19.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:42:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:19 smithi047 ceph-mon[36835]: pgmap v4053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:42:19.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:42:19.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:42:19.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:42:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:20 smithi077 ceph-mon[37153]: pgmap v4054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:42:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:20 smithi077 ceph-mon[37153]: pgmap v4055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:42:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:20 smithi047 ceph-mon[36835]: pgmap v4054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:42:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:20 smithi047 ceph-mon[36835]: pgmap v4055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:42:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:20 smithi029 ceph-mon[32610]: pgmap v4054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:42:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:20 smithi029 ceph-mon[32610]: pgmap v4055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:42:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:22 smithi077 ceph-mon[37153]: pgmap v4056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:42:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:22 smithi047 ceph-mon[36835]: pgmap v4056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:42:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:22 smithi029 ceph-mon[32610]: pgmap v4056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:42:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6764.511160 secs 2024-07-27T08:42:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6764.511160 secs 2024-07-27T08:42:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6764.511160 secs 2024-07-27T08:42:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:24 smithi077 ceph-mon[37153]: pgmap v4057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:42:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:24 smithi047 ceph-mon[36835]: pgmap v4057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:42:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:24 smithi029 ceph-mon[32610]: pgmap v4057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:42:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:26 smithi077 ceph-mon[37153]: pgmap v4058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:42:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:26 smithi047 ceph-mon[36835]: pgmap v4058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:42:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:26 smithi029 ceph-mon[32610]: pgmap v4058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:42:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:42:28.323+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:42:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:42:28.323+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:42:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:42:28.323+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:42:28.401 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:42:28.400+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:42:28.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:42:28.400+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:42:28.402 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:42:28.400+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:42:28.498 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:28 smithi047 ceph-mon[36835]: pgmap v4059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:42:28.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6769.511282 secs 2024-07-27T08:42:28.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:28 smithi029 ceph-mon[32610]: pgmap v4059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:42:28.515 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6769.511282 secs 2024-07-27T08:42:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:28 smithi077 ceph-mon[37153]: pgmap v4059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:42:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6769.511282 secs 2024-07-27T08:42:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:30 smithi077 ceph-mon[37153]: pgmap v4060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:42:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:30 smithi047 ceph-mon[36835]: pgmap v4060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:42:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:30 smithi029 ceph-mon[32610]: pgmap v4060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:42:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:32 smithi077 ceph-mon[37153]: pgmap v4061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:32 smithi047 ceph-mon[36835]: pgmap v4061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:32 smithi029 ceph-mon[32610]: pgmap v4061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6774.511425 secs 2024-07-27T08:42:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6774.511425 secs 2024-07-27T08:42:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6774.511425 secs 2024-07-27T08:42:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:34 smithi077 ceph-mon[37153]: pgmap v4062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:42:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:34 smithi047 ceph-mon[36835]: pgmap v4062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:42:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:34 smithi029 ceph-mon[32610]: pgmap v4062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:42:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:36 smithi077 ceph-mon[37153]: pgmap v4063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:42:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:36 smithi047 ceph-mon[36835]: pgmap v4063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:42:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:36 smithi029 ceph-mon[32610]: pgmap v4063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:42:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:38 smithi077 ceph-mon[37153]: pgmap v4064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6779.511545 secs 2024-07-27T08:42:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:38 smithi047 ceph-mon[36835]: pgmap v4064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6779.511545 secs 2024-07-27T08:42:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:38 smithi029 ceph-mon[32610]: pgmap v4064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6779.511545 secs 2024-07-27T08:42:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:40 smithi077 ceph-mon[37153]: pgmap v4065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:40 smithi047 ceph-mon[36835]: pgmap v4065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:40 smithi029 ceph-mon[32610]: pgmap v4065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:42 smithi077 ceph-mon[37153]: pgmap v4066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:42 smithi047 ceph-mon[36835]: pgmap v4066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:42 smithi029 ceph-mon[32610]: pgmap v4066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:43.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:42:43.339+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:42:43.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:42:43.339+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:42:43.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:42:43.339+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:42:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:42:43.415+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:42:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:42:43.415+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:42:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:42:43.415+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:42:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6784.511651 secs 2024-07-27T08:42:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6784.511651 secs 2024-07-27T08:42:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6784.511651 secs 2024-07-27T08:42:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:44 smithi077 ceph-mon[37153]: pgmap v4067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:42:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:44 smithi047 ceph-mon[36835]: pgmap v4067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:42:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:44 smithi029 ceph-mon[32610]: pgmap v4067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:42:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:46 smithi077 ceph-mon[37153]: pgmap v4068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:42:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:46 smithi047 ceph-mon[36835]: pgmap v4068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:42:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:46 smithi029 ceph-mon[32610]: pgmap v4068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:42:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:48 smithi077 ceph-mon[37153]: pgmap v4069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6789.511815 secs 2024-07-27T08:42:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:48 smithi047 ceph-mon[36835]: pgmap v4069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6789.511815 secs 2024-07-27T08:42:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:48 smithi029 ceph-mon[32610]: pgmap v4069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6789.511815 secs 2024-07-27T08:42:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:50 smithi077 ceph-mon[37153]: pgmap v4070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:50 smithi047 ceph-mon[36835]: pgmap v4070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:50 smithi029 ceph-mon[32610]: pgmap v4070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:42:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:52 smithi077 ceph-mon[37153]: pgmap v4071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:52 smithi047 ceph-mon[36835]: pgmap v4071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:52 smithi029 ceph-mon[32610]: pgmap v4071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:42:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6794.511936 secs 2024-07-27T08:42:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6794.511936 secs 2024-07-27T08:42:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6794.511936 secs 2024-07-27T08:42:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:54 smithi077 ceph-mon[37153]: pgmap v4072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:42:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:54 smithi047 ceph-mon[36835]: pgmap v4072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:42:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:54 smithi029 ceph-mon[32610]: pgmap v4072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:42:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:56 smithi077 ceph-mon[37153]: pgmap v4073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:42:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:56 smithi047 ceph-mon[36835]: pgmap v4073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:42:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:56 smithi029 ceph-mon[32610]: pgmap v4073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:42:58.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:42:58.355+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:42:58.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:42:58.355+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:42:58.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:42:58.355+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:42:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:42:58.431+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:42:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:42:58.431+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:42:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:42:58.431+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:42:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:58 smithi077 ceph-mon[37153]: pgmap v4074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:42:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6799.512073 secs 2024-07-27T08:42:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:58 smithi047 ceph-mon[36835]: pgmap v4074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:42:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6799.512073 secs 2024-07-27T08:42:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:58 smithi029 ceph-mon[32610]: pgmap v4074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:42:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:42:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6799.512073 secs 2024-07-27T08:43:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:00 smithi077 ceph-mon[37153]: pgmap v4075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:00 smithi047 ceph-mon[36835]: pgmap v4075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:00 smithi029 ceph-mon[32610]: pgmap v4075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:02 smithi077 ceph-mon[37153]: pgmap v4076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:02 smithi047 ceph-mon[36835]: pgmap v4076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:02 smithi029 ceph-mon[32610]: pgmap v4076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6804.512137 secs 2024-07-27T08:43:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6804.512137 secs 2024-07-27T08:43:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6804.512137 secs 2024-07-27T08:43:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:04 smithi077 ceph-mon[37153]: pgmap v4077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:43:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:04 smithi047 ceph-mon[36835]: pgmap v4077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:43:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:04 smithi029 ceph-mon[32610]: pgmap v4077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:43:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:06 smithi077 ceph-mon[37153]: pgmap v4078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:43:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:06 smithi047 ceph-mon[36835]: pgmap v4078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:43:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:06 smithi029 ceph-mon[32610]: pgmap v4078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:43:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:08 smithi077 ceph-mon[37153]: pgmap v4079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:43:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6809.512266 secs 2024-07-27T08:43:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:08 smithi047 ceph-mon[36835]: pgmap v4079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:43:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6809.512266 secs 2024-07-27T08:43:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:08 smithi029 ceph-mon[32610]: pgmap v4079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:43:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6809.512266 secs 2024-07-27T08:43:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:10 smithi077 ceph-mon[37153]: pgmap v4080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:10 smithi047 ceph-mon[36835]: pgmap v4080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:10 smithi029 ceph-mon[32610]: pgmap v4080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:12 smithi077 ceph-mon[37153]: pgmap v4081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:12 smithi047 ceph-mon[36835]: pgmap v4081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:12 smithi029 ceph-mon[32610]: pgmap v4081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:13.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:13.370+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:43:13.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:13.370+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:43:13.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:13.370+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:43:13.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:13.440+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:43:13.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:13.440+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:43:13.441 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:13.440+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:43:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6814.512416 secs 2024-07-27T08:43:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6814.512416 secs 2024-07-27T08:43:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6814.512416 secs 2024-07-27T08:43:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:14 smithi077 ceph-mon[37153]: pgmap v4082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:43:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:14 smithi047 ceph-mon[36835]: pgmap v4082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:43:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:14 smithi029 ceph-mon[32610]: pgmap v4082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:43:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:16 smithi077 ceph-mon[37153]: pgmap v4083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:43:16.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:16 smithi047 ceph-mon[36835]: pgmap v4083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:43:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:16 smithi029 ceph-mon[32610]: pgmap v4083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:43:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:18 smithi077 ceph-mon[37153]: pgmap v4084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:43:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6819.512562 secs 2024-07-27T08:43:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:18 smithi047 ceph-mon[36835]: pgmap v4084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:43:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6819.512562 secs 2024-07-27T08:43:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:18 smithi029 ceph-mon[32610]: pgmap v4084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:43:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6819.512562 secs 2024-07-27T08:43:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:43:19.653 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:43:19.657 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:43:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:20 smithi077 ceph-mon[37153]: pgmap v4085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:20.674 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:20 smithi029 ceph-mon[32610]: pgmap v4085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:20 smithi047 ceph-mon[36835]: pgmap v4085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:22 smithi029 ceph-mon[32610]: pgmap v4086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:22.593 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:22 smithi047 ceph-mon[36835]: pgmap v4086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:22 smithi077 ceph-mon[37153]: pgmap v4086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6824.512720 secs 2024-07-27T08:43:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:43:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:43:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:43:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:43:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:43:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:43:23.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6824.512720 secs 2024-07-27T08:43:23.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:23.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:23.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:23.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:23.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:43:23.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:43:23.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:43:23.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:43:23.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:43:23.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:43:23.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6824.512720 secs 2024-07-27T08:43:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:43:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:43:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:43:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:43:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:43:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:43:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:43:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:24 smithi047 ceph-mon[36835]: pgmap v4087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:43:24.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:24 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:43:24.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:24 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:43:24.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:24 smithi047 ceph-mon[36835]: pgmap v4088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T08:43:24.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:24 smithi047 ceph-mon[36835]: pgmap v4089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:43:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:24 smithi029 ceph-mon[32610]: pgmap v4087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:43:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:24 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:43:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:24 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:43:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:24 smithi029 ceph-mon[32610]: pgmap v4088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T08:43:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:24 smithi029 ceph-mon[32610]: pgmap v4089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:43:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:24 smithi077 ceph-mon[37153]: pgmap v4087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:43:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:24 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:43:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:24 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:43:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:24 smithi077 ceph-mon[37153]: pgmap v4088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T08:43:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:24 smithi077 ceph-mon[37153]: pgmap v4089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:43:26.908 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:26 smithi077 ceph-mon[37153]: pgmap v4090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T08:43:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:26 smithi047 ceph-mon[36835]: pgmap v4090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T08:43:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:26 smithi029 ceph-mon[32610]: pgmap v4090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T08:43:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:28.385+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:43:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:28.385+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:43:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:28.385+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:43:28.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:28.448+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:43:28.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:28.448+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:43:28.450 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:28.448+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:43:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6829.512821 secs 2024-07-27T08:43:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:28 smithi047 ceph-mon[36835]: pgmap v4091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T08:43:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6829.512821 secs 2024-07-27T08:43:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:28 smithi029 ceph-mon[32610]: pgmap v4091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T08:43:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6829.512821 secs 2024-07-27T08:43:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:28 smithi077 ceph-mon[37153]: pgmap v4091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T08:43:30.907 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:30 smithi029 ceph-mon[32610]: pgmap v4092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:43:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:30 smithi047 ceph-mon[36835]: pgmap v4092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:43:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:30 smithi077 ceph-mon[37153]: pgmap v4092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T08:43:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:32 smithi047 ceph-mon[36835]: pgmap v4093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-27T08:43:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:32 smithi029 ceph-mon[32610]: pgmap v4093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-27T08:43:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:32 smithi077 ceph-mon[37153]: pgmap v4093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-27T08:43:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6834.512946 secs 2024-07-27T08:43:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6834.512946 secs 2024-07-27T08:43:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6834.512946 secs 2024-07-27T08:43:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:34 smithi047 ceph-mon[36835]: pgmap v4094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:43:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:34 smithi029 ceph-mon[32610]: pgmap v4094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:43:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:34 smithi077 ceph-mon[37153]: pgmap v4094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:43:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:35 smithi077 ceph-mon[37153]: pgmap v4095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:35 smithi047 ceph-mon[36835]: pgmap v4095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:35 smithi029 ceph-mon[32610]: pgmap v4095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6839.513033 secs 2024-07-27T08:43:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:37 smithi029 ceph-mon[32610]: pgmap v4096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:43:38.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6839.513033 secs 2024-07-27T08:43:38.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:37 smithi077 ceph-mon[37153]: pgmap v4096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:43:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6839.513033 secs 2024-07-27T08:43:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:37 smithi047 ceph-mon[36835]: pgmap v4096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:43:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:39 smithi029 ceph-mon[32610]: pgmap v4097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:39 smithi077 ceph-mon[37153]: pgmap v4097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:39 smithi047 ceph-mon[36835]: pgmap v4097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:41 smithi029 ceph-mon[32610]: pgmap v4098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:41 smithi077 ceph-mon[37153]: pgmap v4098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:41 smithi047 ceph-mon[36835]: pgmap v4098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6844.513102 secs 2024-07-27T08:43:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6844.513102 secs 2024-07-27T08:43:43.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:43.400+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:43:43.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:43.400+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:43:43.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:43.400+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:43:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6844.513102 secs 2024-07-27T08:43:43.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:43.450+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:43:43.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:43.450+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:43:43.451 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:43.450+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:43:43.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:43.661+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:43:43.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:43.661+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:43:43.663 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:43.661+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:43:43.664 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:43.661+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:43:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:43 smithi077 ceph-mon[37153]: pgmap v4099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:43:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:43 smithi047 ceph-mon[36835]: pgmap v4099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:43:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:43 smithi029 ceph-mon[32610]: pgmap v4099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:43:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:45 smithi077 ceph-mon[37153]: pgmap v4100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:45 smithi047 ceph-mon[36835]: pgmap v4100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:45 smithi029 ceph-mon[32610]: pgmap v4100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6849.513263 secs 2024-07-27T08:43:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:47 smithi077 ceph-mon[37153]: pgmap v4101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:43:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6849.513263 secs 2024-07-27T08:43:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:47 smithi047 ceph-mon[36835]: pgmap v4101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:43:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6849.513263 secs 2024-07-27T08:43:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:47 smithi029 ceph-mon[32610]: pgmap v4101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:43:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:49 smithi077 ceph-mon[37153]: pgmap v4102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:49 smithi047 ceph-mon[36835]: pgmap v4102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:49 smithi029 ceph-mon[32610]: pgmap v4102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:43:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:51 smithi077 ceph-mon[37153]: pgmap v4103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:51 smithi047 ceph-mon[36835]: pgmap v4103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:51 smithi029 ceph-mon[32610]: pgmap v4103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6854.513412 secs 2024-07-27T08:43:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6854.513412 secs 2024-07-27T08:43:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6854.513412 secs 2024-07-27T08:43:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:53 smithi077 ceph-mon[37153]: pgmap v4104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:43:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:53 smithi047 ceph-mon[36835]: pgmap v4104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:43:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:53 smithi029 ceph-mon[32610]: pgmap v4104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:43:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:55 smithi077 ceph-mon[37153]: pgmap v4105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:55 smithi047 ceph-mon[36835]: pgmap v4105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:55 smithi029 ceph-mon[32610]: pgmap v4105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:43:58.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6859.513540 secs 2024-07-27T08:43:58.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:57 smithi047 ceph-mon[36835]: pgmap v4106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:43:58.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6859.513540 secs 2024-07-27T08:43:58.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:57 smithi029 ceph-mon[32610]: pgmap v4106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:43:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6859.513540 secs 2024-07-27T08:43:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:57 smithi077 ceph-mon[37153]: pgmap v4106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:43:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:58.416+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:43:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:58.416+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:43:58.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:58.416+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:43:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:58.466+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:43:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:58.466+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:43:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:43:58.466+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:43:58.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:58.664+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:43:58.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:58.664+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:43:58.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:58.665+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:43:58.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:43:58.665+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:44:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:43:59 smithi077 ceph-mon[37153]: pgmap v4107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:43:59 smithi047 ceph-mon[36835]: pgmap v4107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:43:59 smithi029 ceph-mon[32610]: pgmap v4107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:01 smithi077 ceph-mon[37153]: pgmap v4108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:01 smithi047 ceph-mon[36835]: pgmap v4108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:01 smithi029 ceph-mon[32610]: pgmap v4108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6864.513681 secs 2024-07-27T08:44:03.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6864.513681 secs 2024-07-27T08:44:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6864.513681 secs 2024-07-27T08:44:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:04 smithi077 ceph-mon[37153]: pgmap v4109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:44:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:04 smithi047 ceph-mon[36835]: pgmap v4109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:44:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:04 smithi029 ceph-mon[32610]: pgmap v4109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:44:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:06 smithi077 ceph-mon[37153]: pgmap v4110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:06 smithi047 ceph-mon[36835]: pgmap v4110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:06 smithi029 ceph-mon[32610]: pgmap v4110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6869.513828 secs 2024-07-27T08:44:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:08 smithi077 ceph-mon[37153]: pgmap v4111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:44:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6869.513828 secs 2024-07-27T08:44:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:08 smithi047 ceph-mon[36835]: pgmap v4111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:44:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6869.513828 secs 2024-07-27T08:44:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:08 smithi029 ceph-mon[32610]: pgmap v4111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:44:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:10 smithi077 ceph-mon[37153]: pgmap v4112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:10 smithi047 ceph-mon[36835]: pgmap v4112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:10 smithi029 ceph-mon[32610]: pgmap v4112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:12 smithi077 ceph-mon[37153]: pgmap v4113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:12 smithi047 ceph-mon[36835]: pgmap v4113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:12 smithi029 ceph-mon[32610]: pgmap v4113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6874.513961 secs 2024-07-27T08:44:13.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:44:13.431+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:44:13.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:44:13.431+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:44:13.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:44:13.431+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:44:13.432 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6874.513961 secs 2024-07-27T08:44:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6874.513961 secs 2024-07-27T08:44:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:44:13.481+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:44:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:44:13.481+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:44:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:44:13.481+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:44:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:14 smithi077 ceph-mon[37153]: pgmap v4114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:44:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:14 smithi047 ceph-mon[36835]: pgmap v4114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:44:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:14 smithi029 ceph-mon[32610]: pgmap v4114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:44:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:16 smithi077 ceph-mon[37153]: pgmap v4115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:16 smithi047 ceph-mon[36835]: pgmap v4115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:16 smithi029 ceph-mon[32610]: pgmap v4115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6879.514092 secs 2024-07-27T08:44:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:18 smithi029 ceph-mon[32610]: pgmap v4116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:44:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6879.514092 secs 2024-07-27T08:44:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:18 smithi077 ceph-mon[37153]: pgmap v4116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:44:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6879.514092 secs 2024-07-27T08:44:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:18 smithi047 ceph-mon[36835]: pgmap v4116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:44:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:20 smithi077 ceph-mon[37153]: pgmap v4117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:20 smithi047 ceph-mon[36835]: pgmap v4117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:20 smithi029 ceph-mon[32610]: pgmap v4117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:22 smithi077 ceph-mon[37153]: pgmap v4118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:22 smithi047 ceph-mon[36835]: pgmap v4118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:22 smithi029 ceph-mon[32610]: pgmap v4118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6884.514223 secs 2024-07-27T08:44:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:44:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6884.514223 secs 2024-07-27T08:44:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:44:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6884.514223 secs 2024-07-27T08:44:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:44:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:24 smithi077 ceph-mon[37153]: pgmap v4119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:44:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:24 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:44:24.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:24 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:44:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:24 smithi047 ceph-mon[36835]: pgmap v4119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:44:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:24.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:24.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:24.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:24 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:44:24.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:24 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:44:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:24 smithi029 ceph-mon[32610]: pgmap v4119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:44:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:24.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:24 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:44:24.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:24 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:44:26.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:26 smithi077 ceph-mon[37153]: pgmap v4120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:26 smithi047 ceph-mon[36835]: pgmap v4120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:26 smithi029 ceph-mon[32610]: pgmap v4120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:44:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:44:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:44:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:44:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:44:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:44:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: pgmap v4121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:44:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: pgmap v4122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T08:44:27.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:44:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:44:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:44:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:27.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:44:27.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:44:27.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:44:27.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:44:27.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: pgmap v4121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:44:27.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: pgmap v4122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T08:44:27.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:44:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:44:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:44:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:44:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:44:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:44:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:44:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:44:28.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: pgmap v4121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:44:28.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: pgmap v4122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T08:44:28.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:44:28.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:44:28.445+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:44:28.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:44:28.445+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:44:28.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:44:28.445+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:44:28.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:44:28.495+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:44:28.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:44:28.495+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:44:28.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:44:28.495+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:44:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6889.514286 secs 2024-07-27T08:44:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6889.514286 secs 2024-07-27T08:44:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6889.514286 secs 2024-07-27T08:44:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:29 smithi047 ceph-mon[36835]: pgmap v4123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T08:44:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:29 smithi029 ceph-mon[32610]: pgmap v4123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T08:44:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:29 smithi077 ceph-mon[37153]: pgmap v4123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T08:44:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:30 smithi077 ceph-mon[37153]: pgmap v4124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:44:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:30 smithi047 ceph-mon[36835]: pgmap v4124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:44:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:30 smithi029 ceph-mon[32610]: pgmap v4124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:44:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:32 smithi077 ceph-mon[37153]: pgmap v4125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T08:44:33.191 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:32 smithi029 ceph-mon[32610]: pgmap v4125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T08:44:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:32 smithi047 ceph-mon[36835]: pgmap v4125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T08:44:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6894.514389 secs 2024-07-27T08:44:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6894.514389 secs 2024-07-27T08:44:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6894.514389 secs 2024-07-27T08:44:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:34 smithi077 ceph-mon[37153]: pgmap v4126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:44:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:34 smithi047 ceph-mon[36835]: pgmap v4126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:44:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:34 smithi029 ceph-mon[32610]: pgmap v4126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:44:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:36 smithi077 ceph-mon[37153]: pgmap v4127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:44:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:36 smithi047 ceph-mon[36835]: pgmap v4127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:44:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:36 smithi029 ceph-mon[32610]: pgmap v4127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:44:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6899.514529 secs 2024-07-27T08:44:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6899.514529 secs 2024-07-27T08:44:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6899.514529 secs 2024-07-27T08:44:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:38 smithi077 ceph-mon[37153]: pgmap v4128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:38 smithi047 ceph-mon[36835]: pgmap v4128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:38 smithi029 ceph-mon[32610]: pgmap v4128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:40 smithi077 ceph-mon[37153]: pgmap v4129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:40 smithi047 ceph-mon[36835]: pgmap v4129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:40 smithi029 ceph-mon[32610]: pgmap v4129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:44:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:42 smithi077 ceph-mon[37153]: pgmap v4130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:44:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:42 smithi047 ceph-mon[36835]: pgmap v4130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:44:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:42 smithi029 ceph-mon[32610]: pgmap v4130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:44:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:44:43.461+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:44:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:44:43.461+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:44:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:44:43.461+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:44:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:44:43.510+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:44:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:44:43.510+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:44:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:44:43.510+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:44:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6904.514650 secs 2024-07-27T08:44:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6904.514650 secs 2024-07-27T08:44:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6904.514650 secs 2024-07-27T08:44:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:44 smithi077 ceph-mon[37153]: pgmap v4131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:44:45.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:44 smithi047 ceph-mon[36835]: pgmap v4131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:44:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:44 smithi029 ceph-mon[32610]: pgmap v4131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:44:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:46 smithi077 ceph-mon[37153]: pgmap v4132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:44:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:46 smithi047 ceph-mon[36835]: pgmap v4132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:44:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:46 smithi029 ceph-mon[32610]: pgmap v4132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:44:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6909.514775 secs 2024-07-27T08:44:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6909.514775 secs 2024-07-27T08:44:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6909.514775 secs 2024-07-27T08:44:49.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:48 smithi029 ceph-mon[32610]: pgmap v4133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:48 smithi077 ceph-mon[37153]: pgmap v4133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:48 smithi047 ceph-mon[36835]: pgmap v4133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:44:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:50 smithi077 ceph-mon[37153]: pgmap v4134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:44:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:50 smithi047 ceph-mon[36835]: pgmap v4134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:44:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:50 smithi029 ceph-mon[32610]: pgmap v4134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:44:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6914.514900 secs 2024-07-27T08:44:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:53 smithi077 ceph-mon[37153]: pgmap v4135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:44:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6914.514900 secs 2024-07-27T08:44:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:53 smithi047 ceph-mon[36835]: pgmap v4135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:44:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6914.514900 secs 2024-07-27T08:44:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:53 smithi029 ceph-mon[32610]: pgmap v4135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:44:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:55 smithi077 ceph-mon[37153]: pgmap v4136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:44:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:55 smithi047 ceph-mon[36835]: pgmap v4136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:44:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:55 smithi029 ceph-mon[32610]: pgmap v4136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:44:58.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:57 smithi029 ceph-mon[32610]: pgmap v4137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:44:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:57 smithi077 ceph-mon[37153]: pgmap v4137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:44:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:57 smithi047 ceph-mon[36835]: pgmap v4137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:44:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:44:58.477+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:44:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:44:58.477+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:44:58.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:44:58.477+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:44:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:44:58.526+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:44:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:44:58.526+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:44:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:44:58.526+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:44:59.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6919.515021 secs 2024-07-27T08:44:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6919.515021 secs 2024-07-27T08:44:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6919.515021 secs 2024-07-27T08:45:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:44:59 smithi077 ceph-mon[37153]: pgmap v4138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:45:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:44:59 smithi047 ceph-mon[36835]: pgmap v4138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:45:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:44:59 smithi029 ceph-mon[32610]: pgmap v4138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:45:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:01 smithi077 ceph-mon[37153]: pgmap v4139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:45:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:01 smithi047 ceph-mon[36835]: pgmap v4139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:45:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:01 smithi029 ceph-mon[32610]: pgmap v4139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:45:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6924.515168 secs 2024-07-27T08:45:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:03 smithi077 ceph-mon[37153]: pgmap v4140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:45:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6924.515168 secs 2024-07-27T08:45:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:03 smithi047 ceph-mon[36835]: pgmap v4140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:45:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6924.515168 secs 2024-07-27T08:45:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:03 smithi029 ceph-mon[32610]: pgmap v4140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:45:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:05 smithi077 ceph-mon[37153]: pgmap v4141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:45:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:05 smithi047 ceph-mon[36835]: pgmap v4141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:45:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:05 smithi029 ceph-mon[32610]: pgmap v4141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:45:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:07 smithi077 ceph-mon[37153]: pgmap v4142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:08.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:07 smithi047 ceph-mon[36835]: pgmap v4142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:07 smithi029 ceph-mon[32610]: pgmap v4142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6929.515251 secs 2024-07-27T08:45:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6929.515251 secs 2024-07-27T08:45:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6929.515251 secs 2024-07-27T08:45:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:09 smithi077 ceph-mon[37153]: pgmap v4143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:45:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:09 smithi047 ceph-mon[36835]: pgmap v4143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:45:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:09 smithi029 ceph-mon[32610]: pgmap v4143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:45:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:11 smithi077 ceph-mon[37153]: pgmap v4144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:45:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:11 smithi047 ceph-mon[36835]: pgmap v4144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:45:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:11 smithi029 ceph-mon[32610]: pgmap v4144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:45:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6934.515375 secs 2024-07-27T08:45:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6934.515375 secs 2024-07-27T08:45:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6934.515375 secs 2024-07-27T08:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:13.491+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:13.491+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:45:13.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:13.491+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:45:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:13.542+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:45:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:13.542+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:45:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:13.542+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:45:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:13.676+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:45:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:13.676+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:45:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:13.676+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:45:13.677 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:13.676+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:45:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:13 smithi077 ceph-mon[37153]: pgmap v4145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:45:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:13 smithi047 ceph-mon[36835]: pgmap v4145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:45:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:13 smithi029 ceph-mon[32610]: pgmap v4145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:45:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:15 smithi077 ceph-mon[37153]: pgmap v4146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:45:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:15 smithi047 ceph-mon[36835]: pgmap v4146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:45:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:15 smithi029 ceph-mon[32610]: pgmap v4146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:45:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:17 smithi077 ceph-mon[37153]: pgmap v4147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6939.515481 secs 2024-07-27T08:45:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:17 smithi047 ceph-mon[36835]: pgmap v4147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6939.515481 secs 2024-07-27T08:45:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:17 smithi029 ceph-mon[32610]: pgmap v4147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6939.515481 secs 2024-07-27T08:45:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:19 smithi077 ceph-mon[37153]: pgmap v4148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:45:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:19 smithi047 ceph-mon[36835]: pgmap v4148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:45:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:19 smithi029 ceph-mon[32610]: pgmap v4148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:45:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:21 smithi077 ceph-mon[37153]: pgmap v4149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:45:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:21 smithi047 ceph-mon[36835]: pgmap v4149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:45:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:21 smithi029 ceph-mon[32610]: pgmap v4149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:45:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6944.515603 secs 2024-07-27T08:45:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6944.515603 secs 2024-07-27T08:45:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6944.515603 secs 2024-07-27T08:45:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:23 smithi077 ceph-mon[37153]: pgmap v4150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:45:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:23 smithi047 ceph-mon[36835]: pgmap v4150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:45:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:23 smithi029 ceph-mon[32610]: pgmap v4150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:45:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:25 smithi077 ceph-mon[37153]: pgmap v4151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:45:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:25 smithi047 ceph-mon[36835]: pgmap v4151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:45:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:25 smithi029 ceph-mon[32610]: pgmap v4151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:45:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:45:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:45:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:45:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:27 smithi077 ceph-mon[37153]: pgmap v4152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6949.515677 secs 2024-07-27T08:45:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:45:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:45:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:27 smithi077 ceph-mon[37153]: pgmap v4153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:45:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:27 smithi077 ceph-mon[37153]: pgmap v4154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:45:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:45:28.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:27 smithi029 ceph-mon[32610]: pgmap v4152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:28.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6949.515677 secs 2024-07-27T08:45:28.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:45:28.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:45:28.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:27 smithi029 ceph-mon[32610]: pgmap v4153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:45:28.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:27 smithi029 ceph-mon[32610]: pgmap v4154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:45:28.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:45:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:27 smithi047 ceph-mon[36835]: pgmap v4152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6949.515677 secs 2024-07-27T08:45:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:45:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:45:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:27 smithi047 ceph-mon[36835]: pgmap v4153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T08:45:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:27 smithi047 ceph-mon[36835]: pgmap v4154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:45:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:45:28.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:28.505+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:45:28.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:28.505+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:45:28.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:28.505+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:45:28.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:28.556+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:45:28.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:28.556+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:45:28.558 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:28.556+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:45:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:28.677+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:45:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:28.677+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:45:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:28.677+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:45:28.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:28.677+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:45:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:30 smithi077 ceph-mon[37153]: pgmap v4155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:45:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:30 smithi047 ceph-mon[36835]: pgmap v4155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:45:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:30 smithi029 ceph-mon[32610]: pgmap v4155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:45:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:32 smithi077 ceph-mon[37153]: pgmap v4156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:45:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:32 smithi047 ceph-mon[36835]: pgmap v4156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:45:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:32 smithi029 ceph-mon[32610]: pgmap v4156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:45:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6954.515819 secs 2024-07-27T08:45:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6954.515819 secs 2024-07-27T08:45:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6954.515819 secs 2024-07-27T08:45:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:34 smithi077 ceph-mon[37153]: pgmap v4157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:45:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:34 smithi047 ceph-mon[36835]: pgmap v4157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:45:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:34 smithi029 ceph-mon[32610]: pgmap v4157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:45:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:36 smithi029 ceph-mon[32610]: pgmap v4158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:45:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:36 smithi077 ceph-mon[37153]: pgmap v4158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:45:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:36 smithi047 ceph-mon[36835]: pgmap v4158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:45:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6959.515939 secs 2024-07-27T08:45:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:38 smithi077 ceph-mon[37153]: pgmap v4159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:45:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6959.515939 secs 2024-07-27T08:45:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:38 smithi047 ceph-mon[36835]: pgmap v4159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:45:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6959.515939 secs 2024-07-27T08:45:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:38 smithi029 ceph-mon[32610]: pgmap v4159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:45:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:40 smithi077 ceph-mon[37153]: pgmap v4160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:45:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:40 smithi047 ceph-mon[36835]: pgmap v4160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:45:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:40 smithi029 ceph-mon[32610]: pgmap v4160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:45:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:42 smithi077 ceph-mon[37153]: pgmap v4161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:42 smithi047 ceph-mon[36835]: pgmap v4161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:42 smithi029 ceph-mon[32610]: pgmap v4161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:43.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:43.521+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:45:43.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:43.521+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:45:43.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:43.521+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:45:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:43.572+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:45:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:43.572+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:45:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:43.572+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:45:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6964.516063 secs 2024-07-27T08:45:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6964.516063 secs 2024-07-27T08:45:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6964.516063 secs 2024-07-27T08:45:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:44 smithi077 ceph-mon[37153]: pgmap v4162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:45:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:44 smithi047 ceph-mon[36835]: pgmap v4162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:45:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:44 smithi029 ceph-mon[32610]: pgmap v4162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:45:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:46 smithi077 ceph-mon[37153]: pgmap v4163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:45:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:46 smithi047 ceph-mon[36835]: pgmap v4163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:45:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:46 smithi029 ceph-mon[32610]: pgmap v4163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:45:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6969.516178 secs 2024-07-27T08:45:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:48 smithi077 ceph-mon[37153]: pgmap v4164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:45:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6969.516178 secs 2024-07-27T08:45:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:48 smithi047 ceph-mon[36835]: pgmap v4164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:45:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6969.516178 secs 2024-07-27T08:45:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:48 smithi029 ceph-mon[32610]: pgmap v4164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:45:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:50 smithi077 ceph-mon[37153]: pgmap v4165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:45:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:50 smithi047 ceph-mon[36835]: pgmap v4165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:45:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:50 smithi029 ceph-mon[32610]: pgmap v4165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:45:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:52 smithi077 ceph-mon[37153]: pgmap v4166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:52 smithi047 ceph-mon[36835]: pgmap v4166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:52 smithi029 ceph-mon[32610]: pgmap v4166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:45:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6974.516319 secs 2024-07-27T08:45:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6974.516319 secs 2024-07-27T08:45:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6974.516319 secs 2024-07-27T08:45:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:54 smithi077 ceph-mon[37153]: pgmap v4167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:45:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:54 smithi047 ceph-mon[36835]: pgmap v4167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:45:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:54 smithi029 ceph-mon[32610]: pgmap v4167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:45:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:56 smithi077 ceph-mon[37153]: pgmap v4168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:45:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:56 smithi047 ceph-mon[36835]: pgmap v4168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:45:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:56 smithi029 ceph-mon[32610]: pgmap v4168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:45:58.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:58.524+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:45:58.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:58.524+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:45:58.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:45:58.524+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:45:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:58.588+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:45:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:58.588+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:45:58.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:45:58.588+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:45:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6979.516394 secs 2024-07-27T08:45:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:45:58 smithi077 ceph-mon[37153]: pgmap v4169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:45:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6979.516394 secs 2024-07-27T08:45:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:45:58 smithi047 ceph-mon[36835]: pgmap v4169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:45:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6979.516394 secs 2024-07-27T08:45:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:45:58 smithi029 ceph-mon[32610]: pgmap v4169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:46:00.614 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:00 smithi077 ceph-mon[37153]: pgmap v4170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:46:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:00 smithi047 ceph-mon[36835]: pgmap v4170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:46:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:00 smithi029 ceph-mon[32610]: pgmap v4170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:46:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:02 smithi047 ceph-mon[36835]: pgmap v4171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:02 smithi029 ceph-mon[32610]: pgmap v4171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:02 smithi077 ceph-mon[37153]: pgmap v4171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6984.516518 secs 2024-07-27T08:46:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6984.516518 secs 2024-07-27T08:46:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6984.516518 secs 2024-07-27T08:46:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:04 smithi047 ceph-mon[36835]: pgmap v4172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:46:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:04 smithi029 ceph-mon[32610]: pgmap v4172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:46:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:04 smithi077 ceph-mon[37153]: pgmap v4172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:46:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:06 smithi047 ceph-mon[36835]: pgmap v4173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:46:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:06 smithi029 ceph-mon[32610]: pgmap v4173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:46:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:06 smithi077 ceph-mon[37153]: pgmap v4173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:46:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6989.516618 secs 2024-07-27T08:46:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:08 smithi047 ceph-mon[36835]: pgmap v4174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:46:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6989.516618 secs 2024-07-27T08:46:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:08 smithi029 ceph-mon[32610]: pgmap v4174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:46:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6989.516618 secs 2024-07-27T08:46:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:08 smithi077 ceph-mon[37153]: pgmap v4174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:46:10.677 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:10 smithi077 ceph-mon[37153]: pgmap v4175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:46:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:10 smithi047 ceph-mon[36835]: pgmap v4175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:46:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:10 smithi029 ceph-mon[32610]: pgmap v4175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:46:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:12 smithi047 ceph-mon[36835]: pgmap v4176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:12 smithi029 ceph-mon[32610]: pgmap v4176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:12 smithi077 ceph-mon[37153]: pgmap v4176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:13.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:13.539+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:46:13.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:13.539+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:46:13.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:13.539+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:46:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:13.591+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:46:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:13.591+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:46:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:13.591+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:46:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6994.516742 secs 2024-07-27T08:46:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6994.516742 secs 2024-07-27T08:46:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6994.516742 secs 2024-07-27T08:46:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:14 smithi047 ceph-mon[36835]: pgmap v4177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:46:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:14 smithi029 ceph-mon[32610]: pgmap v4177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:46:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:14 smithi077 ceph-mon[37153]: pgmap v4177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:46:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:16 smithi047 ceph-mon[36835]: pgmap v4178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:46:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:16 smithi029 ceph-mon[32610]: pgmap v4178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:46:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:16 smithi077 ceph-mon[37153]: pgmap v4178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:46:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 6999.516863 secs 2024-07-27T08:46:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:18 smithi047 ceph-mon[36835]: pgmap v4179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:46:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 6999.516863 secs 2024-07-27T08:46:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:18 smithi029 ceph-mon[32610]: pgmap v4179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:46:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 6999.516863 secs 2024-07-27T08:46:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:18 smithi077 ceph-mon[37153]: pgmap v4179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:46:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:20 smithi047 ceph-mon[36835]: pgmap v4180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:46:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:20 smithi029 ceph-mon[32610]: pgmap v4180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:46:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:20 smithi077 ceph-mon[37153]: pgmap v4180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:46:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:22 smithi047 ceph-mon[36835]: pgmap v4181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:22 smithi029 ceph-mon[32610]: pgmap v4181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:22 smithi077 ceph-mon[37153]: pgmap v4181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7004.516982 secs 2024-07-27T08:46:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7004.516982 secs 2024-07-27T08:46:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7004.516982 secs 2024-07-27T08:46:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:24 smithi047 ceph-mon[36835]: pgmap v4182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:46:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:24 smithi029 ceph-mon[32610]: pgmap v4182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:46:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:24 smithi077 ceph-mon[37153]: pgmap v4182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:46:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:26 smithi047 ceph-mon[36835]: pgmap v4183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:46:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:26 smithi029 ceph-mon[32610]: pgmap v4183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:46:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:26 smithi077 ceph-mon[37153]: pgmap v4183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:46:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:46:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:46:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:46:28.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:28.540+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:46:28.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:28.540+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:46:28.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:28.540+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:46:28.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:28.601+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:46:28.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:28.601+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:46:28.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:28.601+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:46:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7009.517101 secs 2024-07-27T08:46:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:28 smithi047 ceph-mon[36835]: pgmap v4184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:46:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:46:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:46:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:46:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7009.517101 secs 2024-07-27T08:46:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:28 smithi029 ceph-mon[32610]: pgmap v4184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:46:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:46:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:46:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:46:28.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7009.517101 secs 2024-07-27T08:46:28.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:28 smithi077 ceph-mon[37153]: pgmap v4184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:46:28.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:46:28.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:46:28.759 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:46:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:29 smithi047 ceph-mon[36835]: pgmap v4185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:46:29.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:29 smithi047 ceph-mon[36835]: pgmap v4186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:46:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:29 smithi029 ceph-mon[32610]: pgmap v4185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:46:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:29 smithi029 ceph-mon[32610]: pgmap v4186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:46:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:29 smithi077 ceph-mon[37153]: pgmap v4185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:46:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:29 smithi077 ceph-mon[37153]: pgmap v4186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:46:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:31 smithi047 ceph-mon[36835]: pgmap v4187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:46:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:31 smithi029 ceph-mon[32610]: pgmap v4187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:46:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:31 smithi077 ceph-mon[37153]: pgmap v4187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:46:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:33 smithi047 ceph-mon[36835]: pgmap v4188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:46:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7014.517220 secs 2024-07-27T08:46:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:33 smithi029 ceph-mon[32610]: pgmap v4188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:46:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7014.517220 secs 2024-07-27T08:46:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:33 smithi077 ceph-mon[37153]: pgmap v4188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:46:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7014.517220 secs 2024-07-27T08:46:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:35 smithi047 ceph-mon[36835]: pgmap v4189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:46:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:35 smithi029 ceph-mon[32610]: pgmap v4189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:46:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:35 smithi077 ceph-mon[37153]: pgmap v4189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:46:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:37 smithi047 ceph-mon[36835]: pgmap v4190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:46:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:37 smithi029 ceph-mon[32610]: pgmap v4190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:46:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:37 smithi077 ceph-mon[37153]: pgmap v4190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:46:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7019.517335 secs 2024-07-27T08:46:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7019.517335 secs 2024-07-27T08:46:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7019.517335 secs 2024-07-27T08:46:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:39 smithi029 ceph-mon[32610]: pgmap v4191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:46:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:39 smithi077 ceph-mon[37153]: pgmap v4191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:46:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:39 smithi047 ceph-mon[36835]: pgmap v4191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:46:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:41 smithi029 ceph-mon[32610]: pgmap v4192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:46:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:41 smithi077 ceph-mon[37153]: pgmap v4192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:46:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:41 smithi047 ceph-mon[36835]: pgmap v4192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:46:43.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:43.547+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:46:43.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:43.547+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:46:43.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:43.547+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:46:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:43.613+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:46:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:43.613+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:46:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:43.613+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:46:43.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:43.689+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:46:43.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:43.689+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:46:43.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:43.689+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:46:43.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:43.689+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:46:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7024.517457 secs 2024-07-27T08:46:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:43 smithi029 ceph-mon[32610]: pgmap v4193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7024.517457 secs 2024-07-27T08:46:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:43 smithi077 ceph-mon[37153]: pgmap v4193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7024.517457 secs 2024-07-27T08:46:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:43 smithi047 ceph-mon[36835]: pgmap v4193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:45 smithi029 ceph-mon[32610]: pgmap v4194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:46:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:45 smithi077 ceph-mon[37153]: pgmap v4194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:46:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:45 smithi047 ceph-mon[36835]: pgmap v4194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:46:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:47 smithi029 ceph-mon[32610]: pgmap v4195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:46:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:47 smithi077 ceph-mon[37153]: pgmap v4195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:46:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:47 smithi047 ceph-mon[36835]: pgmap v4195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:46:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7029.517599 secs 2024-07-27T08:46:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7029.517599 secs 2024-07-27T08:46:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7029.517599 secs 2024-07-27T08:46:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:49 smithi077 ceph-mon[37153]: pgmap v4196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:46:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:49 smithi047 ceph-mon[36835]: pgmap v4196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:46:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:49 smithi029 ceph-mon[32610]: pgmap v4196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:46:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:51 smithi077 ceph-mon[37153]: pgmap v4197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:46:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:51 smithi047 ceph-mon[36835]: pgmap v4197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:46:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:51 smithi029 ceph-mon[32610]: pgmap v4197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:46:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7034.517721 secs 2024-07-27T08:46:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:53 smithi077 ceph-mon[37153]: pgmap v4198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7034.517721 secs 2024-07-27T08:46:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:53 smithi047 ceph-mon[36835]: pgmap v4198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7034.517721 secs 2024-07-27T08:46:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:53 smithi029 ceph-mon[32610]: pgmap v4198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:46:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:55 smithi077 ceph-mon[37153]: pgmap v4199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:46:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:55 smithi047 ceph-mon[36835]: pgmap v4199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:46:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:55 smithi029 ceph-mon[32610]: pgmap v4199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:46:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:57 smithi077 ceph-mon[37153]: pgmap v4200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:46:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:57 smithi047 ceph-mon[36835]: pgmap v4200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:46:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:57 smithi029 ceph-mon[32610]: pgmap v4200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:46:58.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:58.550+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:46:58.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:58.550+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:46:58.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:58.550+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:46:58.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:58.629+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:46:58.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:58.629+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:46:58.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:46:58.629+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:46:58.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:58.692+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:46:58.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:58.692+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:46:58.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:58.693+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:46:58.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:46:58.693+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:46:58.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7039.517843 secs 2024-07-27T08:46:58.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7039.517843 secs 2024-07-27T08:46:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7039.517843 secs 2024-07-27T08:46:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:46:59 smithi077 ceph-mon[37153]: pgmap v4201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:46:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:46:59 smithi047 ceph-mon[36835]: pgmap v4201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:46:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:46:59 smithi029 ceph-mon[32610]: pgmap v4201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:47:01.749 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:01 smithi047 ceph-mon[36835]: pgmap v4202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:01 smithi077 ceph-mon[37153]: pgmap v4202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:01 smithi029 ceph-mon[32610]: pgmap v4202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7044.517965 secs 2024-07-27T08:47:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:03 smithi077 ceph-mon[37153]: pgmap v4203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7044.517965 secs 2024-07-27T08:47:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:03 smithi047 ceph-mon[36835]: pgmap v4203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7044.517965 secs 2024-07-27T08:47:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:03 smithi029 ceph-mon[32610]: pgmap v4203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:05 smithi077 ceph-mon[37153]: pgmap v4204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:47:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:05 smithi047 ceph-mon[36835]: pgmap v4204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:47:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:05 smithi029 ceph-mon[32610]: pgmap v4204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:47:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:07 smithi077 ceph-mon[37153]: pgmap v4205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:47:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:07 smithi047 ceph-mon[36835]: pgmap v4205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:47:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:07 smithi029 ceph-mon[32610]: pgmap v4205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:47:08.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7049.518085 secs 2024-07-27T08:47:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7049.518085 secs 2024-07-27T08:47:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7049.518085 secs 2024-07-27T08:47:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:09 smithi077 ceph-mon[37153]: pgmap v4206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:47:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:09 smithi047 ceph-mon[36835]: pgmap v4206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:47:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:09 smithi029 ceph-mon[32610]: pgmap v4206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:47:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:11 smithi077 ceph-mon[37153]: pgmap v4207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:47:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:11 smithi047 ceph-mon[36835]: pgmap v4207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:47:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:11 smithi029 ceph-mon[32610]: pgmap v4207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:47:13.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:47:13.565+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:47:13.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:47:13.565+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:47:13.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:47:13.565+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:47:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:47:13.645+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:47:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:47:13.645+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:47:13.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:47:13.645+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:47:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7054.518224 secs 2024-07-27T08:47:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:13 smithi077 ceph-mon[37153]: pgmap v4208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7054.518224 secs 2024-07-27T08:47:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:13 smithi047 ceph-mon[36835]: pgmap v4208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7054.518224 secs 2024-07-27T08:47:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:13 smithi029 ceph-mon[32610]: pgmap v4208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:15.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:15 smithi077 ceph-mon[37153]: pgmap v4209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:47:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:15 smithi047 ceph-mon[36835]: pgmap v4209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:47:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:15 smithi029 ceph-mon[32610]: pgmap v4209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:47:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:17 smithi077 ceph-mon[37153]: pgmap v4210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:47:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:17 smithi047 ceph-mon[36835]: pgmap v4210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:47:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:17 smithi029 ceph-mon[32610]: pgmap v4210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:47:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7059.518332 secs 2024-07-27T08:47:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7059.518332 secs 2024-07-27T08:47:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7059.518332 secs 2024-07-27T08:47:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:19 smithi077 ceph-mon[37153]: pgmap v4211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:47:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:19 smithi047 ceph-mon[36835]: pgmap v4211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:47:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:19 smithi029 ceph-mon[32610]: pgmap v4211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:47:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:21 smithi077 ceph-mon[37153]: pgmap v4212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:21 smithi047 ceph-mon[36835]: pgmap v4212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:21 smithi029 ceph-mon[32610]: pgmap v4212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7064.518456 secs 2024-07-27T08:47:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:23 smithi077 ceph-mon[37153]: pgmap v4213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7064.518456 secs 2024-07-27T08:47:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:23 smithi047 ceph-mon[36835]: pgmap v4213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7064.518456 secs 2024-07-27T08:47:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:23 smithi029 ceph-mon[32610]: pgmap v4213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:25 smithi077 ceph-mon[37153]: pgmap v4214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:47:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:25 smithi047 ceph-mon[36835]: pgmap v4214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:47:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:25 smithi029 ceph-mon[32610]: pgmap v4214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:47:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:27 smithi077 ceph-mon[37153]: pgmap v4215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:47:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:27 smithi047 ceph-mon[36835]: pgmap v4215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:47:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:27 smithi029 ceph-mon[32610]: pgmap v4215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:47:28.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:47:28.579+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:47:28.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:47:28.579+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:47:28.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:47:28.579+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:47:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:47:28.659+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:47:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:47:28.659+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:47:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:47:28.659+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:47:28.797 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7069.518600 secs 2024-07-27T08:47:28.797 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:47:28.797 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:28.797 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:28.813 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7069.518600 secs 2024-07-27T08:47:28.814 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:47:28.814 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:28.814 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:28.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7069.518600 secs 2024-07-27T08:47:28.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:47:28.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:28.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:29 smithi077 ceph-mon[37153]: pgmap v4216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:47:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:29 smithi047 ceph-mon[36835]: pgmap v4216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:47:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:29 smithi029 ceph-mon[32610]: pgmap v4216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:47:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:47:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:47:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:47:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:47:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:47:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:47:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:47:31.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:31 smithi047 ceph-mon[36835]: pgmap v4217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:31.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:31 smithi047 ceph-mon[36835]: pgmap v4218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:47:31.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:31 smithi047 ceph-mon[36835]: pgmap v4219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:47:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:31 smithi077 ceph-mon[37153]: pgmap v4217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:31 smithi077 ceph-mon[37153]: pgmap v4218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:47:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:31 smithi077 ceph-mon[37153]: pgmap v4219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:47:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:31 smithi029 ceph-mon[32610]: pgmap v4217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:31 smithi029 ceph-mon[32610]: pgmap v4218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:47:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:31 smithi029 ceph-mon[32610]: pgmap v4219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:47:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7074.518727 secs 2024-07-27T08:47:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:33 smithi077 ceph-mon[37153]: pgmap v4220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:47:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7074.518727 secs 2024-07-27T08:47:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:33 smithi047 ceph-mon[36835]: pgmap v4220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:47:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7074.518727 secs 2024-07-27T08:47:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:33 smithi029 ceph-mon[32610]: pgmap v4220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:47:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:35 smithi077 ceph-mon[37153]: pgmap v4221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:47:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:35 smithi047 ceph-mon[36835]: pgmap v4221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:47:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:35 smithi029 ceph-mon[32610]: pgmap v4221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:47:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:37 smithi077 ceph-mon[37153]: pgmap v4222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:47:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:37 smithi047 ceph-mon[36835]: pgmap v4222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:47:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:37 smithi029 ceph-mon[32610]: pgmap v4222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:47:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7079.518844 secs 2024-07-27T08:47:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7079.518844 secs 2024-07-27T08:47:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7079.518844 secs 2024-07-27T08:47:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:39 smithi077 ceph-mon[37153]: pgmap v4223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:47:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:39 smithi047 ceph-mon[36835]: pgmap v4223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:47:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:39 smithi029 ceph-mon[32610]: pgmap v4223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:47:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:41 smithi077 ceph-mon[37153]: pgmap v4224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:47:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:41 smithi047 ceph-mon[36835]: pgmap v4224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:47:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:41 smithi029 ceph-mon[32610]: pgmap v4224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:47:43.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:47:43.595+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:47:43.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:47:43.595+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:47:43.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:47:43.595+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:47:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:47:43.673+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:47:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:47:43.673+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:47:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:47:43.673+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:47:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7084.518921 secs 2024-07-27T08:47:43.889 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:43 smithi077 ceph-mon[37153]: pgmap v4225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:47:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7084.518921 secs 2024-07-27T08:47:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:43 smithi047 ceph-mon[36835]: pgmap v4225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:47:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7084.518921 secs 2024-07-27T08:47:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:43 smithi029 ceph-mon[32610]: pgmap v4225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:47:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:45 smithi047 ceph-mon[36835]: pgmap v4226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:45 smithi029 ceph-mon[32610]: pgmap v4226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:45 smithi077 ceph-mon[37153]: pgmap v4226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:47 smithi047 ceph-mon[36835]: pgmap v4227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:47 smithi029 ceph-mon[32610]: pgmap v4227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:47 smithi077 ceph-mon[37153]: pgmap v4227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7089.519047 secs 2024-07-27T08:47:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7089.519047 secs 2024-07-27T08:47:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7089.519047 secs 2024-07-27T08:47:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:49 smithi047 ceph-mon[36835]: pgmap v4228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:47:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:49 smithi029 ceph-mon[32610]: pgmap v4228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:47:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:49 smithi077 ceph-mon[37153]: pgmap v4228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:47:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:51 smithi047 ceph-mon[36835]: pgmap v4229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:47:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:51 smithi029 ceph-mon[32610]: pgmap v4229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:47:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:51 smithi077 ceph-mon[37153]: pgmap v4229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:47:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7094.519194 secs 2024-07-27T08:47:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:53 smithi047 ceph-mon[36835]: pgmap v4230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:47:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7094.519194 secs 2024-07-27T08:47:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:53 smithi029 ceph-mon[32610]: pgmap v4230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:47:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7094.519194 secs 2024-07-27T08:47:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:53 smithi077 ceph-mon[37153]: pgmap v4230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:47:55.943 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:55 smithi029 ceph-mon[32610]: pgmap v4231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:55 smithi047 ceph-mon[36835]: pgmap v4231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:55 smithi077 ceph-mon[37153]: pgmap v4231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:47:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:57 smithi047 ceph-mon[36835]: pgmap v4232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:57 smithi029 ceph-mon[32610]: pgmap v4232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:57 smithi077 ceph-mon[37153]: pgmap v4232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:47:58.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:47:58.611+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:47:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:47:58.611+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:47:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:47:58.611+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:47:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:47:58.689+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:47:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:47:58.689+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:47:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:47:58.689+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:47:58.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7099.519296 secs 2024-07-27T08:47:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7099.519296 secs 2024-07-27T08:47:59.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7099.519296 secs 2024-07-27T08:47:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:47:59 smithi047 ceph-mon[36835]: pgmap v4233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:47:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:47:59 smithi029 ceph-mon[32610]: pgmap v4233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:47:59 smithi077 ceph-mon[37153]: pgmap v4233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:01 smithi047 ceph-mon[36835]: pgmap v4234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:01 smithi029 ceph-mon[32610]: pgmap v4234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:01 smithi077 ceph-mon[37153]: pgmap v4234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7104.519423 secs 2024-07-27T08:48:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:03 smithi047 ceph-mon[36835]: pgmap v4235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7104.519423 secs 2024-07-27T08:48:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:03 smithi029 ceph-mon[32610]: pgmap v4235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7104.519423 secs 2024-07-27T08:48:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:03 smithi077 ceph-mon[37153]: pgmap v4235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:05 smithi047 ceph-mon[36835]: pgmap v4236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:48:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:05 smithi029 ceph-mon[32610]: pgmap v4236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:48:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:05 smithi077 ceph-mon[37153]: pgmap v4236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:48:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:07 smithi047 ceph-mon[36835]: pgmap v4237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:48:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:07 smithi029 ceph-mon[32610]: pgmap v4237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:48:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:07 smithi077 ceph-mon[37153]: pgmap v4237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:48:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7109.519553 secs 2024-07-27T08:48:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7109.519553 secs 2024-07-27T08:48:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7109.519553 secs 2024-07-27T08:48:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:09 smithi047 ceph-mon[36835]: pgmap v4238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:09 smithi029 ceph-mon[32610]: pgmap v4238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:09 smithi077 ceph-mon[37153]: pgmap v4238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:11 smithi047 ceph-mon[36835]: pgmap v4239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:11 smithi029 ceph-mon[32610]: pgmap v4239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:11 smithi077 ceph-mon[37153]: pgmap v4239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:48:13.626+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:48:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:48:13.626+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:48:13.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:48:13.626+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:48:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:48:13.690+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:48:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:48:13.690+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:48:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:48:13.690+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:48:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7114.519682 secs 2024-07-27T08:48:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:13 smithi047 ceph-mon[36835]: pgmap v4240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7114.519682 secs 2024-07-27T08:48:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:13 smithi029 ceph-mon[32610]: pgmap v4240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7114.519682 secs 2024-07-27T08:48:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:13 smithi077 ceph-mon[37153]: pgmap v4240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:15 smithi047 ceph-mon[36835]: pgmap v4241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:48:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:15 smithi029 ceph-mon[32610]: pgmap v4241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:48:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:15 smithi077 ceph-mon[37153]: pgmap v4241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:48:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:17 smithi047 ceph-mon[36835]: pgmap v4242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:48:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:17 smithi029 ceph-mon[32610]: pgmap v4242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:48:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:17 smithi077 ceph-mon[37153]: pgmap v4242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:48:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7119.519776 secs 2024-07-27T08:48:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7119.519776 secs 2024-07-27T08:48:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7119.519776 secs 2024-07-27T08:48:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:19 smithi047 ceph-mon[36835]: pgmap v4243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:19 smithi029 ceph-mon[32610]: pgmap v4243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:19 smithi077 ceph-mon[37153]: pgmap v4243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:21 smithi047 ceph-mon[36835]: pgmap v4244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:21 smithi029 ceph-mon[32610]: pgmap v4244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:21 smithi077 ceph-mon[37153]: pgmap v4244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7124.519905 secs 2024-07-27T08:48:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:23 smithi047 ceph-mon[36835]: pgmap v4245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7124.519905 secs 2024-07-27T08:48:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:23 smithi029 ceph-mon[32610]: pgmap v4245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7124.519905 secs 2024-07-27T08:48:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:23 smithi077 ceph-mon[37153]: pgmap v4245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:25 smithi047 ceph-mon[36835]: pgmap v4246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:48:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:25 smithi029 ceph-mon[32610]: pgmap v4246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:48:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:25 smithi077 ceph-mon[37153]: pgmap v4246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:48:27.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:27 smithi047 ceph-mon[36835]: pgmap v4247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:48:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:27 smithi029 ceph-mon[32610]: pgmap v4247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:48:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:27 smithi077 ceph-mon[37153]: pgmap v4247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:48:28.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:48:28.633+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:48:28.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:48:28.633+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:48:28.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:48:28.633+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:48:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:48:28.705+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:48:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:48:28.705+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:48:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:48:28.705+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:48:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7129.520033 secs 2024-07-27T08:48:29.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7129.520033 secs 2024-07-27T08:48:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7129.520033 secs 2024-07-27T08:48:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:29 smithi029 ceph-mon[32610]: pgmap v4248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:29 smithi077 ceph-mon[37153]: pgmap v4248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:29 smithi047 ceph-mon[36835]: pgmap v4248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:48:31.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:48:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:48:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:31 smithi029 ceph-mon[32610]: pgmap v4249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:31.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:48:31.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:48:31.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:31 smithi029 ceph-mon[32610]: pgmap v4250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:48:31.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:31 smithi029 ceph-mon[32610]: pgmap v4251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:48:31.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:31 smithi077 ceph-mon[37153]: pgmap v4249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:48:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:48:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:31 smithi077 ceph-mon[37153]: pgmap v4250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:48:32.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:31 smithi077 ceph-mon[37153]: pgmap v4251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:48:32.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:31 smithi047 ceph-mon[36835]: pgmap v4249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:48:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:48:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:31 smithi047 ceph-mon[36835]: pgmap v4250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:48:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:31 smithi047 ceph-mon[36835]: pgmap v4251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:48:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:48:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7134.520160 secs 2024-07-27T08:48:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7134.520160 secs 2024-07-27T08:48:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7134.520160 secs 2024-07-27T08:48:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:34 smithi077 ceph-mon[37153]: pgmap v4252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:48:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:34 smithi047 ceph-mon[36835]: pgmap v4252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:48:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:34 smithi029 ceph-mon[32610]: pgmap v4252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:48:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:36 smithi077 ceph-mon[37153]: pgmap v4253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:48:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:36 smithi047 ceph-mon[36835]: pgmap v4253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:48:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:36 smithi029 ceph-mon[32610]: pgmap v4253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:48:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7139.520290 secs 2024-07-27T08:48:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:38 smithi077 ceph-mon[37153]: pgmap v4254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:48:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7139.520290 secs 2024-07-27T08:48:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:38 smithi047 ceph-mon[36835]: pgmap v4254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:48:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7139.520290 secs 2024-07-27T08:48:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:38 smithi029 ceph-mon[32610]: pgmap v4254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:48:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:40 smithi077 ceph-mon[37153]: pgmap v4255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:48:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:40 smithi047 ceph-mon[36835]: pgmap v4255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:48:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:40 smithi029 ceph-mon[32610]: pgmap v4255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:48:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:42 smithi077 ceph-mon[37153]: pgmap v4256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:48:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:42 smithi047 ceph-mon[36835]: pgmap v4256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:48:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:42 smithi029 ceph-mon[32610]: pgmap v4256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:48:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:48:43.649+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:48:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:48:43.649+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:48:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:48:43.649+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:48:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:48:43.721+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:48:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:48:43.721+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:48:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:48:43.721+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:48:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7144.520430 secs 2024-07-27T08:48:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7144.520430 secs 2024-07-27T08:48:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7144.520430 secs 2024-07-27T08:48:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:44 smithi077 ceph-mon[37153]: pgmap v4257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:44 smithi047 ceph-mon[36835]: pgmap v4257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:44 smithi029 ceph-mon[32610]: pgmap v4257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:46 smithi077 ceph-mon[37153]: pgmap v4258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:46 smithi047 ceph-mon[36835]: pgmap v4258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:46 smithi029 ceph-mon[32610]: pgmap v4258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7149.520569 secs 2024-07-27T08:48:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:48 smithi077 ceph-mon[37153]: pgmap v4259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7149.520569 secs 2024-07-27T08:48:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:48 smithi047 ceph-mon[36835]: pgmap v4259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7149.520569 secs 2024-07-27T08:48:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:48 smithi029 ceph-mon[32610]: pgmap v4259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:50 smithi077 ceph-mon[37153]: pgmap v4260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:48:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:50 smithi047 ceph-mon[36835]: pgmap v4260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:48:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:50 smithi029 ceph-mon[32610]: pgmap v4260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:48:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:52 smithi077 ceph-mon[37153]: pgmap v4261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:48:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:52 smithi047 ceph-mon[36835]: pgmap v4261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:48:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:52 smithi029 ceph-mon[32610]: pgmap v4261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:48:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7154.520690 secs 2024-07-27T08:48:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7154.520690 secs 2024-07-27T08:48:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7154.520690 secs 2024-07-27T08:48:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:54 smithi077 ceph-mon[37153]: pgmap v4262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:54 smithi047 ceph-mon[36835]: pgmap v4262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:54 smithi029 ceph-mon[32610]: pgmap v4262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:48:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:56 smithi077 ceph-mon[37153]: pgmap v4263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:56 smithi047 ceph-mon[36835]: pgmap v4263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:56 smithi029 ceph-mon[32610]: pgmap v4263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:48:58.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:48:58.659+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:48:58.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:48:58.659+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:48:58.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:48:58.659+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:48:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:48:58.727+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:48:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:48:58.727+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:48:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:48:58.727+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:48:58.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7159.520816 secs 2024-07-27T08:48:58.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:48:58 smithi029 ceph-mon[32610]: pgmap v4264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7159.520816 secs 2024-07-27T08:48:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:48:58 smithi077 ceph-mon[37153]: pgmap v4264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:48:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7159.520816 secs 2024-07-27T08:48:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:48:58 smithi047 ceph-mon[36835]: pgmap v4264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:49:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:00 smithi077 ceph-mon[37153]: pgmap v4265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:49:00.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:00 smithi047 ceph-mon[36835]: pgmap v4265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:49:00.965 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:00 smithi029 ceph-mon[32610]: pgmap v4265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:49:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:02 smithi077 ceph-mon[37153]: pgmap v4266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:49:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:02 smithi047 ceph-mon[36835]: pgmap v4266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:49:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:02 smithi029 ceph-mon[32610]: pgmap v4266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:49:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7164.520853 secs 2024-07-27T08:49:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7164.520853 secs 2024-07-27T08:49:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7164.520853 secs 2024-07-27T08:49:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:04 smithi077 ceph-mon[37153]: pgmap v4267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:04 smithi047 ceph-mon[36835]: pgmap v4267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:04 smithi029 ceph-mon[32610]: pgmap v4267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:06 smithi077 ceph-mon[37153]: pgmap v4268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:06 smithi047 ceph-mon[36835]: pgmap v4268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:06 smithi029 ceph-mon[32610]: pgmap v4268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7169.520954 secs 2024-07-27T08:49:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:08 smithi077 ceph-mon[37153]: pgmap v4269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:49:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7169.520954 secs 2024-07-27T08:49:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:08 smithi047 ceph-mon[36835]: pgmap v4269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:49:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7169.520954 secs 2024-07-27T08:49:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:08 smithi029 ceph-mon[32610]: pgmap v4269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:49:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:10 smithi077 ceph-mon[37153]: pgmap v4270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:49:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:10 smithi047 ceph-mon[36835]: pgmap v4270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:49:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:10 smithi029 ceph-mon[32610]: pgmap v4270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:49:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:12 smithi077 ceph-mon[37153]: pgmap v4271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:49:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:12 smithi047 ceph-mon[36835]: pgmap v4271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:49:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:12 smithi029 ceph-mon[32610]: pgmap v4271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:49:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:13.674+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:49:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:13.674+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:49:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:13.674+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:49:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:13.734+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:49:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:13.734+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:49:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:13.734+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:49:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7174.521074 secs 2024-07-27T08:49:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7174.521074 secs 2024-07-27T08:49:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7174.521074 secs 2024-07-27T08:49:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:14 smithi077 ceph-mon[37153]: pgmap v4272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:14 smithi047 ceph-mon[36835]: pgmap v4272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:14 smithi029 ceph-mon[32610]: pgmap v4272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:16 smithi077 ceph-mon[37153]: pgmap v4273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:16 smithi047 ceph-mon[36835]: pgmap v4273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:16 smithi029 ceph-mon[32610]: pgmap v4273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7179.521175 secs 2024-07-27T08:49:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:18 smithi077 ceph-mon[37153]: pgmap v4274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:49:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7179.521175 secs 2024-07-27T08:49:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:18 smithi047 ceph-mon[36835]: pgmap v4274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:49:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7179.521175 secs 2024-07-27T08:49:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:18 smithi029 ceph-mon[32610]: pgmap v4274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:49:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:20 smithi077 ceph-mon[37153]: pgmap v4275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:49:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:20 smithi047 ceph-mon[36835]: pgmap v4275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:49:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:20 smithi029 ceph-mon[32610]: pgmap v4275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:49:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:22 smithi077 ceph-mon[37153]: pgmap v4276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:49:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:22 smithi047 ceph-mon[36835]: pgmap v4276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:49:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:22 smithi029 ceph-mon[32610]: pgmap v4276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:49:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7184.521297 secs 2024-07-27T08:49:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7184.521297 secs 2024-07-27T08:49:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7184.521297 secs 2024-07-27T08:49:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:24 smithi077 ceph-mon[37153]: pgmap v4277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:24 smithi047 ceph-mon[36835]: pgmap v4277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:24 smithi029 ceph-mon[32610]: pgmap v4277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:26 smithi077 ceph-mon[37153]: pgmap v4278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:26 smithi047 ceph-mon[36835]: pgmap v4278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:26 smithi029 ceph-mon[32610]: pgmap v4278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:28.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:28.688+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:49:28.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:28.688+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:49:28.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:28.688+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:49:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:28.748+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:49:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:28.748+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:49:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:28.748+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:49:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7189.521427 secs 2024-07-27T08:49:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:28 smithi077 ceph-mon[37153]: pgmap v4279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:49:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7189.521427 secs 2024-07-27T08:49:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:28 smithi047 ceph-mon[36835]: pgmap v4279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:49:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7189.521427 secs 2024-07-27T08:49:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:28 smithi029 ceph-mon[32610]: pgmap v4279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:49:30.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:30 smithi077 ceph-mon[37153]: pgmap v4280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:49:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:30 smithi047 ceph-mon[36835]: pgmap v4280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:49:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:30 smithi029 ceph-mon[32610]: pgmap v4280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:49:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:49:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:49:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:49:32.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:32 smithi047 ceph-mon[36835]: pgmap v4281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:49:32.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:49:32.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:49:32.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:49:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:32 smithi077 ceph-mon[37153]: pgmap v4281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:49:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:49:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:49:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:49:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:32 smithi029 ceph-mon[32610]: pgmap v4281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:49:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:49:32.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:49:32.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:49:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7194.521515 secs 2024-07-27T08:49:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:33 smithi077 ceph-mon[37153]: pgmap v4282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:49:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:33 smithi077 ceph-mon[37153]: pgmap v4283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:49:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7194.521515 secs 2024-07-27T08:49:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:33 smithi047 ceph-mon[36835]: pgmap v4282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:49:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:33 smithi047 ceph-mon[36835]: pgmap v4283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:49:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7194.521515 secs 2024-07-27T08:49:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:33 smithi029 ceph-mon[32610]: pgmap v4282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:49:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:33 smithi029 ceph-mon[32610]: pgmap v4283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:49:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:35 smithi077 ceph-mon[37153]: pgmap v4284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:49:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:35 smithi047 ceph-mon[36835]: pgmap v4284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:49:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:35 smithi029 ceph-mon[32610]: pgmap v4284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:49:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:37 smithi077 ceph-mon[37153]: pgmap v4285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:49:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:37 smithi047 ceph-mon[36835]: pgmap v4285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:49:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:37 smithi029 ceph-mon[32610]: pgmap v4285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:49:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7199.521638 secs 2024-07-27T08:49:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7199.521638 secs 2024-07-27T08:49:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7199.521638 secs 2024-07-27T08:49:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:39 smithi077 ceph-mon[37153]: pgmap v4286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:49:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:39 smithi047 ceph-mon[36835]: pgmap v4286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:49:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:39 smithi029 ceph-mon[32610]: pgmap v4286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:49:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:41 smithi077 ceph-mon[37153]: pgmap v4287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:49:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:41 smithi047 ceph-mon[36835]: pgmap v4287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:49:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:41 smithi029 ceph-mon[32610]: pgmap v4287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:42.642+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-27T08:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:42.642+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T08:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:42.642+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f733c02b4a0 tx=0x7f733c05d000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T08:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:42.642+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f733c02b4a0 tx=0x7f733c05d000 comp rx=0 tx=0).stop 2024-07-27T08:49:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:42.642+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T08:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:42.642+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T08:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:42.642+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T08:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:42.642+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T08:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:42.642+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T08:49:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:42.642+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4bc0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f733c019550 tx=0x7f733c07a000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T08:49:43.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:43.704+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:49:43.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:43.704+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:49:43.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:43.704+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:49:43.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:43.717+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:49:43.718 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:43.717+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:49:43.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:43.717+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:49:43.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:43.717+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:49:43.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:43.757+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:49:43.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:43.757+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:49:43.759 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:43.757+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:49:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7204.521764 secs 2024-07-27T08:49:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:43 smithi077 ceph-mon[37153]: pgmap v4288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:49:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7204.521764 secs 2024-07-27T08:49:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:43 smithi047 ceph-mon[36835]: pgmap v4288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:49:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7204.521764 secs 2024-07-27T08:49:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:43 smithi029 ceph-mon[32610]: pgmap v4288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:49:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:45 smithi047 ceph-mon[36835]: pgmap v4289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:45 smithi029 ceph-mon[32610]: pgmap v4289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:45 smithi077 ceph-mon[37153]: pgmap v4289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:47 smithi047 ceph-mon[36835]: pgmap v4290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:47 smithi029 ceph-mon[32610]: pgmap v4290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:47 smithi077 ceph-mon[37153]: pgmap v4290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7209.521885 secs 2024-07-27T08:49:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7209.521885 secs 2024-07-27T08:49:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7209.521885 secs 2024-07-27T08:49:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:49 smithi047 ceph-mon[36835]: pgmap v4291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:49:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:49 smithi029 ceph-mon[32610]: pgmap v4291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:49:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:49 smithi077 ceph-mon[37153]: pgmap v4291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:49:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:51 smithi047 ceph-mon[36835]: pgmap v4292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:51 smithi029 ceph-mon[32610]: pgmap v4292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:51 smithi077 ceph-mon[37153]: pgmap v4292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7214.522002 secs 2024-07-27T08:49:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:53 smithi047 ceph-mon[36835]: pgmap v4293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:49:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7214.522002 secs 2024-07-27T08:49:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:53 smithi029 ceph-mon[32610]: pgmap v4293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:49:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7214.522002 secs 2024-07-27T08:49:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:53 smithi077 ceph-mon[37153]: pgmap v4293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:49:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:55 smithi047 ceph-mon[36835]: pgmap v4294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:55 smithi029 ceph-mon[32610]: pgmap v4294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:55 smithi077 ceph-mon[37153]: pgmap v4294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:49:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:57.659+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T08:49:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:57.659+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T08:49:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:57.659+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7eff40004480 tx=0x7eff4004d040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T08:49:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:57.659+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7eff40004480 tx=0x7eff4004d040 comp rx=0 tx=0).stop 2024-07-27T08:49:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:57.659+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T08:49:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:57.659+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T08:49:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:57.659+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T08:49:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:57.659+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T08:49:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:57.659+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T08:49:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:57.659+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b49b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7eff40004480 tx=0x7eff40041000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T08:49:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:57 smithi047 ceph-mon[36835]: pgmap v4295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:57 smithi029 ceph-mon[32610]: pgmap v4295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:57 smithi077 ceph-mon[37153]: pgmap v4295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:49:58.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:58.706+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:49:58.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:58.706+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:49:58.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:58.706+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:49:58.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:58.720+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:49:58.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:58.720+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:49:58.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:58.720+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:49:58.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:49:58.720+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:49:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:58.773+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:49:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:58.773+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:49:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:49:58.773+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:49:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7219.522064 secs 2024-07-27T08:49:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7219.522064 secs 2024-07-27T08:49:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7219.522064 secs 2024-07-27T08:49:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:49:59 smithi047 ceph-mon[36835]: pgmap v4296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:49:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:49:59 smithi029 ceph-mon[32610]: pgmap v4296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:50:00.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:49:59 smithi077 ceph-mon[37153]: pgmap v4296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:50:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:50:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:50:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T08:50:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:01 smithi047 ceph-mon[36835]: pgmap v4297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:01 smithi029 ceph-mon[32610]: pgmap v4297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:01 smithi077 ceph-mon[37153]: pgmap v4297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7224.522182 secs 2024-07-27T08:50:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:03 smithi047 ceph-mon[36835]: pgmap v4298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:50:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7224.522182 secs 2024-07-27T08:50:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:03 smithi029 ceph-mon[32610]: pgmap v4298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:50:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7224.522182 secs 2024-07-27T08:50:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:03 smithi077 ceph-mon[37153]: pgmap v4298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:50:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:05 smithi047 ceph-mon[36835]: pgmap v4299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:05 smithi029 ceph-mon[32610]: pgmap v4299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:05 smithi077 ceph-mon[37153]: pgmap v4299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:06 smithi077 ceph-mon[37153]: pgmap v4300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:06 smithi047 ceph-mon[36835]: pgmap v4300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:06 smithi029 ceph-mon[32610]: pgmap v4300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7229.522307 secs 2024-07-27T08:50:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7229.522307 secs 2024-07-27T08:50:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7229.522307 secs 2024-07-27T08:50:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:08 smithi077 ceph-mon[37153]: pgmap v4301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:50:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:08 smithi047 ceph-mon[36835]: pgmap v4301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:50:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:08 smithi029 ceph-mon[32610]: pgmap v4301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:50:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:11 smithi077 ceph-mon[37153]: pgmap v4302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:11 smithi047 ceph-mon[36835]: pgmap v4302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:11 smithi029 ceph-mon[32610]: pgmap v4302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7234.522427 secs 2024-07-27T08:50:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:13 smithi077 ceph-mon[37153]: pgmap v4303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:50:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7234.522427 secs 2024-07-27T08:50:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:13 smithi047 ceph-mon[36835]: pgmap v4303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:50:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7234.522427 secs 2024-07-27T08:50:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:13 smithi029 ceph-mon[32610]: pgmap v4303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:50:13.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:50:13.710+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:50:13.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:50:13.710+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:50:13.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:50:13.710+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:50:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:50:13.775+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:50:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:50:13.775+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:50:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:50:13.775+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:50:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:15 smithi077 ceph-mon[37153]: pgmap v4304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:15 smithi047 ceph-mon[36835]: pgmap v4304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:15 smithi029 ceph-mon[32610]: pgmap v4304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:17 smithi077 ceph-mon[37153]: pgmap v4305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:17 smithi047 ceph-mon[36835]: pgmap v4305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:17 smithi029 ceph-mon[32610]: pgmap v4305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7239.522526 secs 2024-07-27T08:50:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7239.522526 secs 2024-07-27T08:50:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7239.522526 secs 2024-07-27T08:50:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:19 smithi077 ceph-mon[37153]: pgmap v4306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:50:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:19 smithi047 ceph-mon[36835]: pgmap v4306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:50:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:19 smithi029 ceph-mon[32610]: pgmap v4306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:50:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:21 smithi077 ceph-mon[37153]: pgmap v4307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:21 smithi047 ceph-mon[36835]: pgmap v4307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:21 smithi029 ceph-mon[32610]: pgmap v4307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7244.522628 secs 2024-07-27T08:50:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:23 smithi077 ceph-mon[37153]: pgmap v4308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:50:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7244.522628 secs 2024-07-27T08:50:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:23 smithi047 ceph-mon[36835]: pgmap v4308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:50:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7244.522628 secs 2024-07-27T08:50:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:23 smithi029 ceph-mon[32610]: pgmap v4308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:50:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:25 smithi077 ceph-mon[37153]: pgmap v4309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:25 smithi047 ceph-mon[36835]: pgmap v4309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:25 smithi029 ceph-mon[32610]: pgmap v4309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:27 smithi077 ceph-mon[37153]: pgmap v4310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:27 smithi047 ceph-mon[36835]: pgmap v4310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:27 smithi029 ceph-mon[32610]: pgmap v4310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7249.522772 secs 2024-07-27T08:50:28.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7249.522772 secs 2024-07-27T08:50:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7249.522772 secs 2024-07-27T08:50:28.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:50:28.724+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:50:28.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:50:28.724+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:50:28.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:50:28.724+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:50:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:50:28.789+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:50:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:50:28.789+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:50:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:50:28.789+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:50:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:29 smithi077 ceph-mon[37153]: pgmap v4311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:50:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:29 smithi047 ceph-mon[36835]: pgmap v4311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:50:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:29 smithi029 ceph-mon[32610]: pgmap v4311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:50:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:31 smithi077 ceph-mon[37153]: pgmap v4312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:31 smithi047 ceph-mon[36835]: pgmap v4312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:31 smithi029 ceph-mon[32610]: pgmap v4312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:50:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:50:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:50:33.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7254.522887 secs 2024-07-27T08:50:33.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:33 smithi047 ceph-mon[36835]: pgmap v4313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:50:33.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:50:33.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:50:33.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:33 smithi047 ceph-mon[36835]: pgmap v4314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T08:50:33.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:33 smithi047 ceph-mon[36835]: pgmap v4315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:50:33.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:50:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7254.522887 secs 2024-07-27T08:50:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:33 smithi077 ceph-mon[37153]: pgmap v4313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:50:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:50:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:50:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:33 smithi077 ceph-mon[37153]: pgmap v4314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T08:50:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:33 smithi077 ceph-mon[37153]: pgmap v4315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:50:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:50:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7254.522887 secs 2024-07-27T08:50:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:33 smithi029 ceph-mon[32610]: pgmap v4313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:50:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:50:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:50:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:33 smithi029 ceph-mon[32610]: pgmap v4314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T08:50:33.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:33 smithi029 ceph-mon[32610]: pgmap v4315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:50:33.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:50:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:35 smithi077 ceph-mon[37153]: pgmap v4316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:50:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:35 smithi047 ceph-mon[36835]: pgmap v4316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:50:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:35 smithi029 ceph-mon[32610]: pgmap v4316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:50:37.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:37 smithi077 ceph-mon[37153]: pgmap v4317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:50:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:37 smithi047 ceph-mon[36835]: pgmap v4317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:50:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:37 smithi029 ceph-mon[32610]: pgmap v4317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:50:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7259.522955 secs 2024-07-27T08:50:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7259.522955 secs 2024-07-27T08:50:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7259.522955 secs 2024-07-27T08:50:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:39 smithi077 ceph-mon[37153]: pgmap v4318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:50:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:39 smithi047 ceph-mon[36835]: pgmap v4318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:50:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:39 smithi029 ceph-mon[32610]: pgmap v4318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:50:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:41 smithi077 ceph-mon[37153]: pgmap v4319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:50:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:41 smithi047 ceph-mon[36835]: pgmap v4319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:50:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:41 smithi029 ceph-mon[32610]: pgmap v4319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:50:43.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:50:43.726+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:50:43.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:50:43.726+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:50:43.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:50:43.726+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:50:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:50:43.805+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:50:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:50:43.805+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:50:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:50:43.805+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:50:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7264.523024 secs 2024-07-27T08:50:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:43 smithi077 ceph-mon[37153]: pgmap v4320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:50:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7264.523024 secs 2024-07-27T08:50:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:43 smithi047 ceph-mon[36835]: pgmap v4320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:50:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7264.523024 secs 2024-07-27T08:50:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:43 smithi029 ceph-mon[32610]: pgmap v4320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:50:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:45 smithi047 ceph-mon[36835]: pgmap v4321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:50:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:45 smithi029 ceph-mon[32610]: pgmap v4321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:50:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:45 smithi077 ceph-mon[37153]: pgmap v4321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:50:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:47 smithi047 ceph-mon[36835]: pgmap v4322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:50:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:47 smithi029 ceph-mon[32610]: pgmap v4322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:50:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:47 smithi077 ceph-mon[37153]: pgmap v4322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:50:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7269.523113 secs 2024-07-27T08:50:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7269.523113 secs 2024-07-27T08:50:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7269.523113 secs 2024-07-27T08:50:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:49 smithi047 ceph-mon[36835]: pgmap v4323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:49 smithi029 ceph-mon[32610]: pgmap v4323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:49 smithi077 ceph-mon[37153]: pgmap v4323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:51 smithi047 ceph-mon[36835]: pgmap v4324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:51 smithi029 ceph-mon[32610]: pgmap v4324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:51 smithi077 ceph-mon[37153]: pgmap v4324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:50:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7274.523236 secs 2024-07-27T08:50:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:53 smithi047 ceph-mon[36835]: pgmap v4325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:50:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7274.523236 secs 2024-07-27T08:50:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:53 smithi029 ceph-mon[32610]: pgmap v4325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:50:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7274.523236 secs 2024-07-27T08:50:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:53 smithi077 ceph-mon[37153]: pgmap v4325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:50:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:55 smithi047 ceph-mon[36835]: pgmap v4326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:50:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:55 smithi029 ceph-mon[32610]: pgmap v4326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:50:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:55 smithi077 ceph-mon[37153]: pgmap v4326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:50:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:57 smithi047 ceph-mon[36835]: pgmap v4327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:50:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:57 smithi029 ceph-mon[32610]: pgmap v4327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:50:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:57 smithi077 ceph-mon[37153]: pgmap v4327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:50:58.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:50:58.742+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:50:58.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:50:58.742+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:50:58.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:50:58.742+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:50:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:50:58.810+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:50:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:50:58.810+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:50:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:50:58.810+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:50:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7279.523360 secs 2024-07-27T08:50:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7279.523360 secs 2024-07-27T08:50:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7279.523360 secs 2024-07-27T08:50:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:50:59 smithi047 ceph-mon[36835]: pgmap v4328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:50:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:50:59 smithi029 ceph-mon[32610]: pgmap v4328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:50:59 smithi077 ceph-mon[37153]: pgmap v4328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:01.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:01 smithi077 ceph-mon[37153]: pgmap v4329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:01 smithi047 ceph-mon[36835]: pgmap v4329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:01 smithi029 ceph-mon[32610]: pgmap v4329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7284.523497 secs 2024-07-27T08:51:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:03 smithi047 ceph-mon[36835]: pgmap v4330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:51:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7284.523497 secs 2024-07-27T08:51:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:03 smithi029 ceph-mon[32610]: pgmap v4330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:51:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7284.523497 secs 2024-07-27T08:51:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:03 smithi077 ceph-mon[37153]: pgmap v4330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:51:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:05 smithi047 ceph-mon[36835]: pgmap v4331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:51:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:05 smithi029 ceph-mon[32610]: pgmap v4331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:51:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:05 smithi077 ceph-mon[37153]: pgmap v4331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:51:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:07 smithi047 ceph-mon[36835]: pgmap v4332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:51:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:07 smithi029 ceph-mon[32610]: pgmap v4332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:51:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:07 smithi077 ceph-mon[37153]: pgmap v4332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:51:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7289.523615 secs 2024-07-27T08:51:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7289.523615 secs 2024-07-27T08:51:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7289.523615 secs 2024-07-27T08:51:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:09 smithi047 ceph-mon[36835]: pgmap v4333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:09 smithi029 ceph-mon[32610]: pgmap v4333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:09 smithi077 ceph-mon[37153]: pgmap v4333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:11 smithi047 ceph-mon[36835]: pgmap v4334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:11 smithi029 ceph-mon[32610]: pgmap v4334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:11 smithi077 ceph-mon[37153]: pgmap v4334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:13.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:51:13.745+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:51:13.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:51:13.745+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:51:13.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:51:13.745+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:51:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:51:13.825+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:51:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:51:13.825+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:51:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:51:13.825+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:51:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7294.523715 secs 2024-07-27T08:51:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:13 smithi047 ceph-mon[36835]: pgmap v4335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:51:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7294.523715 secs 2024-07-27T08:51:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:13 smithi029 ceph-mon[32610]: pgmap v4335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:51:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7294.523715 secs 2024-07-27T08:51:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:13 smithi077 ceph-mon[37153]: pgmap v4335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:51:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:15 smithi047 ceph-mon[36835]: pgmap v4336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:51:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:15 smithi029 ceph-mon[32610]: pgmap v4336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:51:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:15 smithi077 ceph-mon[37153]: pgmap v4336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:51:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:17 smithi047 ceph-mon[36835]: pgmap v4337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:51:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7299.523813 secs 2024-07-27T08:51:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:17 smithi029 ceph-mon[32610]: pgmap v4337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:51:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7299.523813 secs 2024-07-27T08:51:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:17 smithi077 ceph-mon[37153]: pgmap v4337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:51:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7299.523813 secs 2024-07-27T08:51:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:19 smithi047 ceph-mon[36835]: pgmap v4338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:19 smithi029 ceph-mon[32610]: pgmap v4338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:20.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:19 smithi077 ceph-mon[37153]: pgmap v4338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:21 smithi047 ceph-mon[36835]: pgmap v4339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:21 smithi029 ceph-mon[32610]: pgmap v4339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:21 smithi077 ceph-mon[37153]: pgmap v4339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7304.523941 secs 2024-07-27T08:51:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7304.523941 secs 2024-07-27T08:51:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7304.523941 secs 2024-07-27T08:51:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:23 smithi047 ceph-mon[36835]: pgmap v4340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:51:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:23 smithi029 ceph-mon[32610]: pgmap v4340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:51:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:23 smithi077 ceph-mon[37153]: pgmap v4340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:51:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:25 smithi047 ceph-mon[36835]: pgmap v4341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:51:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:25 smithi029 ceph-mon[32610]: pgmap v4341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:51:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:25 smithi077 ceph-mon[37153]: pgmap v4341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:51:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:27 smithi047 ceph-mon[36835]: pgmap v4342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:51:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7309.524030 secs 2024-07-27T08:51:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:27 smithi029 ceph-mon[32610]: pgmap v4342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:51:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7309.524030 secs 2024-07-27T08:51:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:27 smithi077 ceph-mon[37153]: pgmap v4342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:51:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7309.524030 secs 2024-07-27T08:51:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:51:28.759+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:51:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:51:28.759+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:51:28.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:51:28.759+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:51:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:51:28.840+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:51:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:51:28.840+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:51:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:51:28.840+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:51:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:29 smithi047 ceph-mon[36835]: pgmap v4343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:29 smithi029 ceph-mon[32610]: pgmap v4343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:29 smithi077 ceph-mon[37153]: pgmap v4343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:31 smithi029 ceph-mon[32610]: pgmap v4344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:31 smithi077 ceph-mon[37153]: pgmap v4344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:31 smithi047 ceph-mon[36835]: pgmap v4344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7314.524135 secs 2024-07-27T08:51:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:51:33.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7314.524135 secs 2024-07-27T08:51:33.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:51:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7314.524135 secs 2024-07-27T08:51:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:51:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:33 smithi029 ceph-mon[32610]: pgmap v4345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:51:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:51:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:51:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:51:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:33 smithi077 ceph-mon[37153]: pgmap v4345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:51:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:51:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:51:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:51:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:33 smithi047 ceph-mon[36835]: pgmap v4345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:51:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:51:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:51:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:51:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:34 smithi029 ceph-mon[32610]: pgmap v4346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:51:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:34 smithi029 ceph-mon[32610]: pgmap v4347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:51:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:34 smithi077 ceph-mon[37153]: pgmap v4346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:51:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:34 smithi077 ceph-mon[37153]: pgmap v4347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:51:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:34 smithi047 ceph-mon[36835]: pgmap v4346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:51:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:34 smithi047 ceph-mon[36835]: pgmap v4347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:51:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:36 smithi077 ceph-mon[37153]: pgmap v4348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:51:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:36 smithi047 ceph-mon[36835]: pgmap v4348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:51:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:36 smithi029 ceph-mon[32610]: pgmap v4348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:51:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7319.524292 secs 2024-07-27T08:51:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7319.524292 secs 2024-07-27T08:51:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7319.524292 secs 2024-07-27T08:51:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:38 smithi077 ceph-mon[37153]: pgmap v4349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:51:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:38 smithi047 ceph-mon[36835]: pgmap v4349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:51:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:38 smithi029 ceph-mon[32610]: pgmap v4349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:51:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:40 smithi077 ceph-mon[37153]: pgmap v4350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:51:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:40 smithi047 ceph-mon[36835]: pgmap v4350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:51:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:40 smithi029 ceph-mon[32610]: pgmap v4350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:51:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:42 smithi077 ceph-mon[37153]: pgmap v4351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:51:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7324.524430 secs 2024-07-27T08:51:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:42 smithi047 ceph-mon[36835]: pgmap v4351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:51:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7324.524430 secs 2024-07-27T08:51:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:42 smithi029 ceph-mon[32610]: pgmap v4351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:51:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7324.524430 secs 2024-07-27T08:51:43.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:51:43.775+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:51:43.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:51:43.775+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:51:43.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:51:43.775+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:51:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:51:43.856+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:51:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:51:43.856+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:51:43.858 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:51:43.856+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:51:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:44 smithi077 ceph-mon[37153]: pgmap v4352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:51:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:44 smithi047 ceph-mon[36835]: pgmap v4352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:51:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:44 smithi029 ceph-mon[32610]: pgmap v4352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:51:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:46 smithi077 ceph-mon[37153]: pgmap v4353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:46 smithi047 ceph-mon[36835]: pgmap v4353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:46 smithi029 ceph-mon[32610]: pgmap v4353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7329.524547 secs 2024-07-27T08:51:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7329.524547 secs 2024-07-27T08:51:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7329.524547 secs 2024-07-27T08:51:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:48 smithi077 ceph-mon[37153]: pgmap v4354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:51:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:48 smithi047 ceph-mon[36835]: pgmap v4354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:51:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:48 smithi029 ceph-mon[32610]: pgmap v4354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:51:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:49 smithi077 ceph-mon[37153]: pgmap v4355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:50.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:49 smithi029 ceph-mon[32610]: pgmap v4355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:49 smithi047 ceph-mon[36835]: pgmap v4355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:51:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:52 smithi047 ceph-mon[36835]: pgmap v4356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7334.524672 secs 2024-07-27T08:51:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:52 smithi029 ceph-mon[32610]: pgmap v4356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7334.524672 secs 2024-07-27T08:51:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:52 smithi077 ceph-mon[37153]: pgmap v4356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:51:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7334.524672 secs 2024-07-27T08:51:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:54 smithi047 ceph-mon[36835]: pgmap v4357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:51:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:54 smithi029 ceph-mon[32610]: pgmap v4357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:51:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:54 smithi077 ceph-mon[37153]: pgmap v4357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:51:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:56 smithi047 ceph-mon[36835]: pgmap v4358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:51:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:56 smithi029 ceph-mon[32610]: pgmap v4358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:51:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:56 smithi077 ceph-mon[37153]: pgmap v4358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:51:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7339.524809 secs 2024-07-27T08:51:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7339.524809 secs 2024-07-27T08:51:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7339.524809 secs 2024-07-27T08:51:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:51:58 smithi047 ceph-mon[36835]: pgmap v4359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:51:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:51:58 smithi029 ceph-mon[32610]: pgmap v4359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:51:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:51:58 smithi077 ceph-mon[37153]: pgmap v4359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:51:58.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:51:58.790+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:51:58.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:51:58.790+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:51:58.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:51:58.790+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:51:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:51:58.871+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:51:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:51:58.871+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:51:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:51:58.871+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:52:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:00 smithi047 ceph-mon[36835]: pgmap v4360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:00 smithi029 ceph-mon[32610]: pgmap v4360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:00 smithi077 ceph-mon[37153]: pgmap v4360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:02.427 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:02 smithi077 ceph-mon[37153]: pgmap v4361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:02.427 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7344.524930 secs 2024-07-27T08:52:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:02 smithi047 ceph-mon[36835]: pgmap v4361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7344.524930 secs 2024-07-27T08:52:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:02 smithi029 ceph-mon[32610]: pgmap v4361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7344.524930 secs 2024-07-27T08:52:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:04 smithi047 ceph-mon[36835]: pgmap v4362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:52:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:04 smithi029 ceph-mon[32610]: pgmap v4362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:52:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:04 smithi077 ceph-mon[37153]: pgmap v4362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:52:06.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:06 smithi047 ceph-mon[36835]: pgmap v4363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:06 smithi029 ceph-mon[32610]: pgmap v4363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:06 smithi077 ceph-mon[37153]: pgmap v4363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7349.525063 secs 2024-07-27T08:52:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7349.525063 secs 2024-07-27T08:52:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7349.525063 secs 2024-07-27T08:52:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:08 smithi047 ceph-mon[36835]: pgmap v4364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:52:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:08 smithi029 ceph-mon[32610]: pgmap v4364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:52:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:08 smithi077 ceph-mon[37153]: pgmap v4364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:52:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:10 smithi047 ceph-mon[36835]: pgmap v4365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:10 smithi029 ceph-mon[32610]: pgmap v4365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:10 smithi077 ceph-mon[37153]: pgmap v4365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:12.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:12 smithi047 ceph-mon[36835]: pgmap v4366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:12.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7354.525186 secs 2024-07-27T08:52:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:12 smithi029 ceph-mon[32610]: pgmap v4366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7354.525186 secs 2024-07-27T08:52:12.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:12 smithi077 ceph-mon[37153]: pgmap v4366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:12.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7354.525186 secs 2024-07-27T08:52:13.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:13.803+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:52:13.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:13.804+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:52:13.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:13.804+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:52:13.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:13.886+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:52:13.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:13.886+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:52:13.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:13.886+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:52:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:14 smithi047 ceph-mon[36835]: pgmap v4367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:52:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:14 smithi029 ceph-mon[32610]: pgmap v4367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:52:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:14 smithi077 ceph-mon[37153]: pgmap v4367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:52:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:16 smithi029 ceph-mon[32610]: pgmap v4368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:16 smithi077 ceph-mon[37153]: pgmap v4368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:16 smithi047 ceph-mon[36835]: pgmap v4368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7359.525308 secs 2024-07-27T08:52:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7359.525308 secs 2024-07-27T08:52:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7359.525308 secs 2024-07-27T08:52:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:18 smithi029 ceph-mon[32610]: pgmap v4369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:52:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:18 smithi077 ceph-mon[37153]: pgmap v4369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:52:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:18 smithi047 ceph-mon[36835]: pgmap v4369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:52:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:20 smithi029 ceph-mon[32610]: pgmap v4370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:20.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:20 smithi077 ceph-mon[37153]: pgmap v4370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:20 smithi047 ceph-mon[36835]: pgmap v4370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:22 smithi077 ceph-mon[37153]: pgmap v4371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7364.525430 secs 2024-07-27T08:52:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:22 smithi047 ceph-mon[36835]: pgmap v4371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7364.525430 secs 2024-07-27T08:52:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:22 smithi029 ceph-mon[32610]: pgmap v4371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7364.525430 secs 2024-07-27T08:52:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:24 smithi077 ceph-mon[37153]: pgmap v4372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:52:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:24 smithi047 ceph-mon[36835]: pgmap v4372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:52:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:24 smithi029 ceph-mon[32610]: pgmap v4372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:52:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:26 smithi077 ceph-mon[37153]: pgmap v4373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:26 smithi047 ceph-mon[36835]: pgmap v4373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:26 smithi029 ceph-mon[32610]: pgmap v4373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7369.525549 secs 2024-07-27T08:52:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7369.525549 secs 2024-07-27T08:52:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7369.525549 secs 2024-07-27T08:52:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:28 smithi077 ceph-mon[37153]: pgmap v4374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:52:28.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:28 smithi029 ceph-mon[32610]: pgmap v4374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:52:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:28 smithi047 ceph-mon[36835]: pgmap v4374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:52:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:28.818+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:52:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:28.818+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:52:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:28.818+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:52:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:28.901+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:52:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:28.901+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:52:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:28.901+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:52:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:30 smithi077 ceph-mon[37153]: pgmap v4375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:30 smithi047 ceph-mon[36835]: pgmap v4375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:30 smithi029 ceph-mon[32610]: pgmap v4375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:32 smithi077 ceph-mon[37153]: pgmap v4376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7374.525654 secs 2024-07-27T08:52:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:32 smithi047 ceph-mon[36835]: pgmap v4376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7374.525654 secs 2024-07-27T08:52:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:32 smithi029 ceph-mon[32610]: pgmap v4376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7374.525654 secs 2024-07-27T08:52:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:52:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:52:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:52:34.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:34 smithi047 ceph-mon[36835]: pgmap v4377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:52:34.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:52:34.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:52:34.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:34 smithi047 ceph-mon[36835]: pgmap v4378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:52:34.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:34 smithi047 ceph-mon[36835]: pgmap v4379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:52:34.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:52:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:34 smithi077 ceph-mon[37153]: pgmap v4377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:52:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:52:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:52:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:34 smithi077 ceph-mon[37153]: pgmap v4378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:52:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:34 smithi077 ceph-mon[37153]: pgmap v4379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:52:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:52:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:34 smithi029 ceph-mon[32610]: pgmap v4377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:52:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:52:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:52:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:34 smithi029 ceph-mon[32610]: pgmap v4378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:52:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:34 smithi029 ceph-mon[32610]: pgmap v4379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:52:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:52:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:36 smithi029 ceph-mon[32610]: pgmap v4380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:52:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:36 smithi077 ceph-mon[37153]: pgmap v4380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:52:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:36 smithi047 ceph-mon[36835]: pgmap v4380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:52:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7379.525779 secs 2024-07-27T08:52:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7379.525779 secs 2024-07-27T08:52:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7379.525779 secs 2024-07-27T08:52:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:38 smithi077 ceph-mon[37153]: pgmap v4381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:52:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:38 smithi047 ceph-mon[36835]: pgmap v4381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:52:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:38 smithi029 ceph-mon[32610]: pgmap v4381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:52:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:40 smithi077 ceph-mon[37153]: pgmap v4382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:52:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:40 smithi047 ceph-mon[36835]: pgmap v4382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:52:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:40 smithi029 ceph-mon[32610]: pgmap v4382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:52:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:42 smithi077 ceph-mon[37153]: pgmap v4383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:52:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7384.525872 secs 2024-07-27T08:52:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:42 smithi047 ceph-mon[36835]: pgmap v4383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:52:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7384.525872 secs 2024-07-27T08:52:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:42 smithi029 ceph-mon[32610]: pgmap v4383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:52:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7384.525872 secs 2024-07-27T08:52:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:43.744+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:52:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:43.744+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:52:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:43.745+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:52:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:43.745+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:52:43.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:43.834+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:52:43.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:43.834+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:52:43.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:43.834+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:52:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:43.916+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:52:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:43.916+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:52:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:43.916+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:52:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:44 smithi077 ceph-mon[37153]: pgmap v4384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:52:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:44 smithi047 ceph-mon[36835]: pgmap v4384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:52:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:44 smithi029 ceph-mon[32610]: pgmap v4384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:52:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:46 smithi077 ceph-mon[37153]: pgmap v4385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:46 smithi047 ceph-mon[36835]: pgmap v4385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:46 smithi029 ceph-mon[32610]: pgmap v4385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7389.526022 secs 2024-07-27T08:52:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7389.526022 secs 2024-07-27T08:52:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7389.526022 secs 2024-07-27T08:52:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:48 smithi077 ceph-mon[37153]: pgmap v4386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:52:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:48 smithi047 ceph-mon[36835]: pgmap v4386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:52:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:48 smithi029 ceph-mon[32610]: pgmap v4386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:52:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:50 smithi077 ceph-mon[37153]: pgmap v4387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:52:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:50 smithi047 ceph-mon[36835]: pgmap v4387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:52:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:50 smithi029 ceph-mon[32610]: pgmap v4387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:52:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:52 smithi077 ceph-mon[37153]: pgmap v4388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:52:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7394.526147 secs 2024-07-27T08:52:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:52 smithi047 ceph-mon[36835]: pgmap v4388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:52:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7394.526147 secs 2024-07-27T08:52:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:52 smithi029 ceph-mon[32610]: pgmap v4388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:52:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7394.526147 secs 2024-07-27T08:52:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:54 smithi077 ceph-mon[37153]: pgmap v4389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:54 smithi047 ceph-mon[36835]: pgmap v4389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:54 smithi029 ceph-mon[32610]: pgmap v4389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:52:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:56 smithi077 ceph-mon[37153]: pgmap v4390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:56 smithi047 ceph-mon[36835]: pgmap v4390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:56 smithi029 ceph-mon[32610]: pgmap v4390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:52:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7399.526276 secs 2024-07-27T08:52:58.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7399.526276 secs 2024-07-27T08:52:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7399.526276 secs 2024-07-27T08:52:58.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:58.747+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:52:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:58.747+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:52:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:58.748+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:52:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:58.748+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:52:58.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:58.850+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:52:58.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:58.850+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:52:58.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:52:58.850+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:52:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:58.932+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:52:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:58.932+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:52:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:52:58.932+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:52:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:58 smithi077 ceph-mon[37153]: pgmap v4391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:52:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:58 smithi047 ceph-mon[36835]: pgmap v4391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:52:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:58 smithi029 ceph-mon[32610]: pgmap v4391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:53:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:52:59 smithi077 ceph-mon[37153]: pgmap v4392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:53:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:52:59 smithi047 ceph-mon[36835]: pgmap v4392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:53:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:52:59 smithi029 ceph-mon[32610]: pgmap v4392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:53:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:02 smithi077 ceph-mon[37153]: pgmap v4393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7404.526395 secs 2024-07-27T08:53:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:02 smithi047 ceph-mon[36835]: pgmap v4393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7404.526395 secs 2024-07-27T08:53:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:02 smithi029 ceph-mon[32610]: pgmap v4393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7404.526395 secs 2024-07-27T08:53:04.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:04 smithi047 ceph-mon[36835]: pgmap v4394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:04 smithi077 ceph-mon[37153]: pgmap v4394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:04 smithi029 ceph-mon[32610]: pgmap v4394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:06 smithi077 ceph-mon[37153]: pgmap v4395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:53:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:06 smithi047 ceph-mon[36835]: pgmap v4395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:53:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:06 smithi029 ceph-mon[32610]: pgmap v4395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:53:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7409.526516 secs 2024-07-27T08:53:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7409.526516 secs 2024-07-27T08:53:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7409.526516 secs 2024-07-27T08:53:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:08 smithi077 ceph-mon[37153]: pgmap v4396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:53:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:08 smithi047 ceph-mon[36835]: pgmap v4396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:53:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:08 smithi029 ceph-mon[32610]: pgmap v4396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:53:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:10 smithi077 ceph-mon[37153]: pgmap v4397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:53:11.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:10 smithi047 ceph-mon[36835]: pgmap v4397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:53:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:10 smithi029 ceph-mon[32610]: pgmap v4397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:53:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:12 smithi077 ceph-mon[37153]: pgmap v4398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7414.526639 secs 2024-07-27T08:53:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:12 smithi047 ceph-mon[36835]: pgmap v4398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7414.526639 secs 2024-07-27T08:53:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:12 smithi029 ceph-mon[32610]: pgmap v4398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7414.526639 secs 2024-07-27T08:53:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:13.863+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:53:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:13.863+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:53:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:13.863+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:53:13.948 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:13.948+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:53:13.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:13.948+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:53:13.949 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:13.948+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:53:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:14 smithi077 ceph-mon[37153]: pgmap v4399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:14 smithi047 ceph-mon[36835]: pgmap v4399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:14 smithi029 ceph-mon[32610]: pgmap v4399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:16 smithi077 ceph-mon[37153]: pgmap v4400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:53:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:16 smithi047 ceph-mon[36835]: pgmap v4400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:53:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:16 smithi029 ceph-mon[32610]: pgmap v4400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:53:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7419.526774 secs 2024-07-27T08:53:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7419.526774 secs 2024-07-27T08:53:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7419.526774 secs 2024-07-27T08:53:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:18 smithi077 ceph-mon[37153]: pgmap v4401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:53:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:18 smithi047 ceph-mon[36835]: pgmap v4401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:53:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:18 smithi029 ceph-mon[32610]: pgmap v4401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:53:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:20 smithi077 ceph-mon[37153]: pgmap v4402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:53:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:20 smithi047 ceph-mon[36835]: pgmap v4402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:53:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:20 smithi029 ceph-mon[32610]: pgmap v4402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:53:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:21 smithi077 ceph-mon[37153]: pgmap v4403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:21 smithi047 ceph-mon[36835]: pgmap v4403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:21 smithi029 ceph-mon[32610]: pgmap v4403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7424.526899 secs 2024-07-27T08:53:23.112 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7424.526899 secs 2024-07-27T08:53:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7424.526899 secs 2024-07-27T08:53:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:23 smithi077 ceph-mon[37153]: pgmap v4404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:23 smithi047 ceph-mon[36835]: pgmap v4404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:23 smithi029 ceph-mon[32610]: pgmap v4404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:26 smithi077 ceph-mon[37153]: pgmap v4405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:53:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:26 smithi047 ceph-mon[36835]: pgmap v4405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:53:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:26 smithi029 ceph-mon[32610]: pgmap v4405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:53:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7429.527021 secs 2024-07-27T08:53:28.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7429.527021 secs 2024-07-27T08:53:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7429.527021 secs 2024-07-27T08:53:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:28.751+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:53:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:28.751+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:53:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:28.751+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:53:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:28.751+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:53:28.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:28.877+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:53:28.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:28.877+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:53:28.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:28.877+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:53:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:28.957+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:53:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:28.957+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:53:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:28.957+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:53:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:28 smithi077 ceph-mon[37153]: pgmap v4406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:53:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:28 smithi047 ceph-mon[36835]: pgmap v4406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:53:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:28 smithi029 ceph-mon[32610]: pgmap v4406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:53:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:30 smithi077 ceph-mon[37153]: pgmap v4407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:53:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:30 smithi047 ceph-mon[36835]: pgmap v4407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:53:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:30 smithi029 ceph-mon[32610]: pgmap v4407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:53:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:32 smithi077 ceph-mon[37153]: pgmap v4408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7434.527148 secs 2024-07-27T08:53:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:32 smithi047 ceph-mon[36835]: pgmap v4408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7434.527148 secs 2024-07-27T08:53:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:32 smithi029 ceph-mon[32610]: pgmap v4408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7434.527148 secs 2024-07-27T08:53:34.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:34 smithi047 ceph-mon[36835]: pgmap v4409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:34.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:53:34.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:53:34.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:53:34.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:53:34.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:53:34.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:53:35.000 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:53:35.000 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:53:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:34 smithi077 ceph-mon[37153]: pgmap v4409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:53:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:53:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:53:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:53:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:53:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:53:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:53:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:53:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:34 smithi029 ceph-mon[32610]: pgmap v4409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:53:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:53:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:53:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:53:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:53:35.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:53:35.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:53:35.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:53:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:35 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:53:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:35 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:53:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:35 smithi077 ceph-mon[37153]: pgmap v4410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:53:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:35 smithi077 ceph-mon[37153]: pgmap v4411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:53:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:35 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:53:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:35 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:53:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:35 smithi047 ceph-mon[36835]: pgmap v4410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:53:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:35 smithi047 ceph-mon[36835]: pgmap v4411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:53:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:35 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T08:53:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:35 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T08:53:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:35 smithi029 ceph-mon[32610]: pgmap v4410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:53:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:35 smithi029 ceph-mon[32610]: pgmap v4411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:53:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:36 smithi047 ceph-mon[36835]: pgmap v4412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:53:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:36 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7439.527267 secs 2024-07-27T08:53:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:36 smithi029 ceph-mon[32610]: pgmap v4412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:53:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:36 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7439.527267 secs 2024-07-27T08:53:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:36 smithi077 ceph-mon[37153]: pgmap v4412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:53:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:36 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7439.527267 secs 2024-07-27T08:53:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:39 smithi077 ceph-mon[37153]: pgmap v4413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:53:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:39 smithi047 ceph-mon[36835]: pgmap v4413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:53:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:39 smithi029 ceph-mon[32610]: pgmap v4413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:53:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:41 smithi077 ceph-mon[37153]: pgmap v4414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:53:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:41 smithi047 ceph-mon[36835]: pgmap v4414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:53:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:41 smithi029 ceph-mon[32610]: pgmap v4414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:53:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7444.527368 secs 2024-07-27T08:53:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7444.527368 secs 2024-07-27T08:53:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7444.527368 secs 2024-07-27T08:53:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:43 smithi077 ceph-mon[37153]: pgmap v4415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:53:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:43 smithi047 ceph-mon[36835]: pgmap v4415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:53:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:43 smithi029 ceph-mon[32610]: pgmap v4415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:53:43.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:43.754+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:53:43.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:43.754+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:53:43.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:43.754+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:53:43.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:43.754+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:53:43.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:43.893+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:53:43.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:43.893+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:53:43.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:43.893+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:53:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:43.973+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:53:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:43.973+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:53:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:43.973+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:53:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:45 smithi077 ceph-mon[37153]: pgmap v4416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:53:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:45 smithi047 ceph-mon[36835]: pgmap v4416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:53:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:45 smithi029 ceph-mon[32610]: pgmap v4416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:53:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:47 smithi077 ceph-mon[37153]: pgmap v4417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7449.527489 secs 2024-07-27T08:53:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:47 smithi047 ceph-mon[36835]: pgmap v4417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7449.527489 secs 2024-07-27T08:53:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:47 smithi029 ceph-mon[32610]: pgmap v4417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7449.527489 secs 2024-07-27T08:53:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:49 smithi077 ceph-mon[37153]: pgmap v4418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:49 smithi047 ceph-mon[36835]: pgmap v4418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:49 smithi029 ceph-mon[32610]: pgmap v4418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:51 smithi077 ceph-mon[37153]: pgmap v4419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:53:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:51 smithi047 ceph-mon[36835]: pgmap v4419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:53:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:51 smithi029 ceph-mon[32610]: pgmap v4419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:53:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7454.527590 secs 2024-07-27T08:53:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7454.527590 secs 2024-07-27T08:53:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7454.527590 secs 2024-07-27T08:53:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:53 smithi077 ceph-mon[37153]: pgmap v4420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:53 smithi047 ceph-mon[36835]: pgmap v4420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:53 smithi029 ceph-mon[32610]: pgmap v4420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:55 smithi077 ceph-mon[37153]: pgmap v4421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:53:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:55 smithi047 ceph-mon[36835]: pgmap v4421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:53:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:55 smithi029 ceph-mon[32610]: pgmap v4421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:53:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:57 smithi077 ceph-mon[37153]: pgmap v4422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7459.527706 secs 2024-07-27T08:53:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:57 smithi047 ceph-mon[36835]: pgmap v4422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7459.527706 secs 2024-07-27T08:53:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:57 smithi029 ceph-mon[32610]: pgmap v4422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:53:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7459.527706 secs 2024-07-27T08:53:58.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:58.909+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:53:58.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:58.909+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:53:58.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:53:58.909+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:53:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:58.984+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:53:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:58.984+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:53:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:53:58.984+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:53:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:53:59 smithi077 ceph-mon[37153]: pgmap v4423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:53:59 smithi047 ceph-mon[36835]: pgmap v4423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:53:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:53:59 smithi029 ceph-mon[32610]: pgmap v4423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:01 smithi077 ceph-mon[37153]: pgmap v4424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:01 smithi047 ceph-mon[36835]: pgmap v4424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:01 smithi029 ceph-mon[32610]: pgmap v4424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7464.527829 secs 2024-07-27T08:54:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7464.527829 secs 2024-07-27T08:54:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7464.527829 secs 2024-07-27T08:54:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:03 smithi077 ceph-mon[37153]: pgmap v4425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:03 smithi047 ceph-mon[36835]: pgmap v4425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:03 smithi029 ceph-mon[32610]: pgmap v4425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:05 smithi077 ceph-mon[37153]: pgmap v4426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:54:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:05 smithi047 ceph-mon[36835]: pgmap v4426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:54:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:05 smithi029 ceph-mon[32610]: pgmap v4426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:54:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:06 smithi047 ceph-mon[36835]: pgmap v4427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:06 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7469.527968 secs 2024-07-27T08:54:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:06 smithi029 ceph-mon[32610]: pgmap v4427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:06 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7469.527968 secs 2024-07-27T08:54:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:06 smithi077 ceph-mon[37153]: pgmap v4427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:06 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7469.527968 secs 2024-07-27T08:54:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:09 smithi029 ceph-mon[32610]: pgmap v4428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:09 smithi077 ceph-mon[37153]: pgmap v4428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:09 smithi047 ceph-mon[36835]: pgmap v4428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:11 smithi077 ceph-mon[37153]: pgmap v4429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:11 smithi047 ceph-mon[36835]: pgmap v4429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:11 smithi029 ceph-mon[32610]: pgmap v4429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7474.528052 secs 2024-07-27T08:54:12.782 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7474.528052 secs 2024-07-27T08:54:12.783 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7474.528052 secs 2024-07-27T08:54:13.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:13 smithi077 ceph-mon[37153]: pgmap v4430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:13 smithi047 ceph-mon[36835]: pgmap v4430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:13 smithi029 ceph-mon[32610]: pgmap v4430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:13.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:54:13.920+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:54:13.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:54:13.921+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:54:13.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:54:13.921+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:54:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:54:13.993+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:54:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:54:13.993+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:54:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:54:13.993+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:54:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:15 smithi077 ceph-mon[37153]: pgmap v4431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:54:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:15 smithi047 ceph-mon[36835]: pgmap v4431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:54:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:15 smithi029 ceph-mon[32610]: pgmap v4431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:54:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:17 smithi077 ceph-mon[37153]: pgmap v4432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7479.528152 secs 2024-07-27T08:54:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:17 smithi047 ceph-mon[36835]: pgmap v4432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7479.528152 secs 2024-07-27T08:54:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:17 smithi029 ceph-mon[32610]: pgmap v4432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7479.528152 secs 2024-07-27T08:54:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:19 smithi077 ceph-mon[37153]: pgmap v4433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:19 smithi047 ceph-mon[36835]: pgmap v4433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:19 smithi029 ceph-mon[32610]: pgmap v4433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:21 smithi077 ceph-mon[37153]: pgmap v4434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:21 smithi047 ceph-mon[36835]: pgmap v4434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:21 smithi029 ceph-mon[32610]: pgmap v4434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7484.528294 secs 2024-07-27T08:54:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7484.528294 secs 2024-07-27T08:54:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7484.528294 secs 2024-07-27T08:54:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:23 smithi077 ceph-mon[37153]: pgmap v4435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:23 smithi047 ceph-mon[36835]: pgmap v4435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:23 smithi029 ceph-mon[32610]: pgmap v4435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:25 smithi077 ceph-mon[37153]: pgmap v4436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:54:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:25 smithi047 ceph-mon[36835]: pgmap v4436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:54:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:25 smithi029 ceph-mon[32610]: pgmap v4436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:54:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:27 smithi077 ceph-mon[37153]: pgmap v4437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7489.528394 secs 2024-07-27T08:54:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:27 smithi047 ceph-mon[36835]: pgmap v4437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7489.528394 secs 2024-07-27T08:54:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:27 smithi029 ceph-mon[32610]: pgmap v4437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7489.528394 secs 2024-07-27T08:54:28.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:54:28.935+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:54:28.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:54:28.935+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:54:28.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:54:28.935+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:54:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:54:29.006+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:54:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:54:29.006+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:54:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:54:29.006+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:54:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:29 smithi077 ceph-mon[37153]: pgmap v4438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:29 smithi047 ceph-mon[36835]: pgmap v4438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:29 smithi029 ceph-mon[32610]: pgmap v4438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:31 smithi077 ceph-mon[37153]: pgmap v4439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:31 smithi047 ceph-mon[36835]: pgmap v4439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:31 smithi029 ceph-mon[32610]: pgmap v4439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7494.528534 secs 2024-07-27T08:54:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7494.528534 secs 2024-07-27T08:54:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7494.528534 secs 2024-07-27T08:54:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:33 smithi077 ceph-mon[37153]: pgmap v4440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:33 smithi047 ceph-mon[36835]: pgmap v4440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:33 smithi029 ceph-mon[32610]: pgmap v4440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:54:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:54:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:54:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:54:35.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:35 smithi047 ceph-mon[36835]: pgmap v4441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:54:35.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:54:35.599 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:54:35.599 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:54:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:35 smithi077 ceph-mon[37153]: pgmap v4441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:54:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:54:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:54:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:54:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:35 smithi029 ceph-mon[32610]: pgmap v4441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:54:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T08:54:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:54:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:54:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:54:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:36 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:54:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:36 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:54:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:36 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:54:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:36 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:54:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:36 smithi077 ceph-mon[37153]: pgmap v4442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:54:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:36 smithi077 ceph-mon[37153]: pgmap v4443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:54:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:36 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:54:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:36 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:54:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:36 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:54:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:36 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:54:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:36 smithi047 ceph-mon[36835]: pgmap v4442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:54:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:36 smithi047 ceph-mon[36835]: pgmap v4443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:54:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:36 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T08:54:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:36 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T08:54:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:36 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T08:54:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:36 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T08:54:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:36 smithi029 ceph-mon[32610]: pgmap v4442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:54:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:36 smithi029 ceph-mon[32610]: pgmap v4443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:54:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7499.528677 secs 2024-07-27T08:54:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7499.528677 secs 2024-07-27T08:54:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7499.528677 secs 2024-07-27T08:54:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:38 smithi077 ceph-mon[37153]: pgmap v4444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:54:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:38 smithi047 ceph-mon[36835]: pgmap v4444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:54:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:38 smithi029 ceph-mon[32610]: pgmap v4444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:54:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:40 smithi047 ceph-mon[36835]: pgmap v4445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:54:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:40 smithi029 ceph-mon[32610]: pgmap v4445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:54:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:40 smithi077 ceph-mon[37153]: pgmap v4445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:54:42.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:42 smithi047 ceph-mon[36835]: pgmap v4446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:54:42.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7504.528819 secs 2024-07-27T08:54:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:42 smithi029 ceph-mon[32610]: pgmap v4446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:54:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7504.528819 secs 2024-07-27T08:54:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:42 smithi077 ceph-mon[37153]: pgmap v4446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:54:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7504.528819 secs 2024-07-27T08:54:43.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:54:43.951+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:54:43.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:54:43.951+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:54:43.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:54:43.951+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:54:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:54:44.012+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:54:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:54:44.012+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:54:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:54:44.012+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:54:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:44 smithi047 ceph-mon[36835]: pgmap v4447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:54:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:44 smithi029 ceph-mon[32610]: pgmap v4447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:54:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:44 smithi077 ceph-mon[37153]: pgmap v4447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:54:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:46 smithi047 ceph-mon[36835]: pgmap v4448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:54:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:46 smithi029 ceph-mon[32610]: pgmap v4448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:54:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:46 smithi077 ceph-mon[37153]: pgmap v4448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:54:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7509.528942 secs 2024-07-27T08:54:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7509.528942 secs 2024-07-27T08:54:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7509.528942 secs 2024-07-27T08:54:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:48 smithi047 ceph-mon[36835]: pgmap v4449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:54:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:48 smithi029 ceph-mon[32610]: pgmap v4449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:54:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:48 smithi077 ceph-mon[37153]: pgmap v4449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:54:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:50 smithi047 ceph-mon[36835]: pgmap v4450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:50 smithi029 ceph-mon[32610]: pgmap v4450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:50 smithi077 ceph-mon[37153]: pgmap v4450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:54:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:52 smithi047 ceph-mon[36835]: pgmap v4451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7514.529064 secs 2024-07-27T08:54:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:52 smithi029 ceph-mon[32610]: pgmap v4451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7514.529064 secs 2024-07-27T08:54:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:52 smithi077 ceph-mon[37153]: pgmap v4451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7514.529064 secs 2024-07-27T08:54:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:54 smithi047 ceph-mon[36835]: pgmap v4452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:54:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:54 smithi029 ceph-mon[32610]: pgmap v4452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:54:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:54 smithi077 ceph-mon[37153]: pgmap v4452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:54:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:56 smithi047 ceph-mon[36835]: pgmap v4453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:56 smithi029 ceph-mon[32610]: pgmap v4453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:56 smithi077 ceph-mon[37153]: pgmap v4453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:54:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7519.529187 secs 2024-07-27T08:54:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7519.529187 secs 2024-07-27T08:54:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7519.529187 secs 2024-07-27T08:54:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:54:58 smithi047 ceph-mon[36835]: pgmap v4454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:54:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:54:58 smithi029 ceph-mon[32610]: pgmap v4454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:54:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:54:58 smithi077 ceph-mon[37153]: pgmap v4454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:54:58.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:54:58.955+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:54:58.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:54:58.955+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:54:58.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:54:58.955+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:54:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:54:59.016+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:54:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:54:59.016+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:54:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:54:59.016+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:55:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:00 smithi047 ceph-mon[36835]: pgmap v4455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:55:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:00 smithi029 ceph-mon[32610]: pgmap v4455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:55:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:00 smithi077 ceph-mon[37153]: pgmap v4455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:55:02.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:02 smithi077 ceph-mon[37153]: pgmap v4456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:02.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7524.529333 secs 2024-07-27T08:55:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:02 smithi047 ceph-mon[36835]: pgmap v4456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7524.529333 secs 2024-07-27T08:55:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:02 smithi029 ceph-mon[32610]: pgmap v4456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7524.529333 secs 2024-07-27T08:55:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:04 smithi047 ceph-mon[36835]: pgmap v4457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:55:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:04 smithi029 ceph-mon[32610]: pgmap v4457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:55:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:04 smithi077 ceph-mon[37153]: pgmap v4457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:55:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:06 smithi047 ceph-mon[36835]: pgmap v4458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:55:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:06 smithi029 ceph-mon[32610]: pgmap v4458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:55:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:06 smithi077 ceph-mon[37153]: pgmap v4458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:55:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7529.529454 secs 2024-07-27T08:55:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7529.529454 secs 2024-07-27T08:55:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7529.529454 secs 2024-07-27T08:55:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:08 smithi047 ceph-mon[36835]: pgmap v4459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:55:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:08 smithi029 ceph-mon[32610]: pgmap v4459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:55:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:08 smithi077 ceph-mon[37153]: pgmap v4459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:55:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:10 smithi047 ceph-mon[36835]: pgmap v4460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:55:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:10 smithi029 ceph-mon[32610]: pgmap v4460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:55:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:10 smithi077 ceph-mon[37153]: pgmap v4460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:55:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:12 smithi047 ceph-mon[36835]: pgmap v4461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:55:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7534.529577 secs 2024-07-27T08:55:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:12 smithi029 ceph-mon[32610]: pgmap v4461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:55:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7534.529577 secs 2024-07-27T08:55:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:12 smithi077 ceph-mon[37153]: pgmap v4461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:55:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7534.529577 secs 2024-07-27T08:55:13.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:13.969+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:55:13.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:13.969+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:55:13.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:13.969+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:55:14.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:14.028+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:55:14.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:14.028+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:55:14.030 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:14.028+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:55:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:14 smithi047 ceph-mon[36835]: pgmap v4462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:55:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:14 smithi029 ceph-mon[32610]: pgmap v4462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:55:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:14 smithi077 ceph-mon[37153]: pgmap v4462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:55:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:16 smithi047 ceph-mon[36835]: pgmap v4463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:16 smithi029 ceph-mon[32610]: pgmap v4463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:16 smithi077 ceph-mon[37153]: pgmap v4463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7539.529720 secs 2024-07-27T08:55:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7539.529720 secs 2024-07-27T08:55:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7539.529720 secs 2024-07-27T08:55:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:18 smithi047 ceph-mon[36835]: pgmap v4464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:55:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:18 smithi029 ceph-mon[32610]: pgmap v4464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:55:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:18 smithi077 ceph-mon[37153]: pgmap v4464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:55:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:20 smithi047 ceph-mon[36835]: pgmap v4465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:55:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:20 smithi029 ceph-mon[32610]: pgmap v4465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:55:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:20 smithi077 ceph-mon[37153]: pgmap v4465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:55:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:22 smithi047 ceph-mon[36835]: pgmap v4466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7544.529844 secs 2024-07-27T08:55:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:22 smithi029 ceph-mon[32610]: pgmap v4466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7544.529844 secs 2024-07-27T08:55:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:22 smithi077 ceph-mon[37153]: pgmap v4466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7544.529844 secs 2024-07-27T08:55:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:24 smithi029 ceph-mon[32610]: pgmap v4467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:55:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:24 smithi077 ceph-mon[37153]: pgmap v4467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:55:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:24 smithi047 ceph-mon[36835]: pgmap v4467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:55:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:26 smithi029 ceph-mon[32610]: pgmap v4468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:26 smithi077 ceph-mon[37153]: pgmap v4468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:26 smithi047 ceph-mon[36835]: pgmap v4468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7549.529968 secs 2024-07-27T08:55:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7549.529968 secs 2024-07-27T08:55:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7549.529968 secs 2024-07-27T08:55:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:28 smithi029 ceph-mon[32610]: pgmap v4469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:55:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:28 smithi077 ceph-mon[37153]: pgmap v4469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:55:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:28 smithi047 ceph-mon[36835]: pgmap v4469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:55:28.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:28.985+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:55:28.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:28.985+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:55:28.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:28.985+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:55:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:29.043+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:55:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:29.043+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:55:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:29.043+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:55:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:30 smithi029 ceph-mon[32610]: pgmap v4470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:55:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:30 smithi077 ceph-mon[37153]: pgmap v4470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:55:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:30 smithi047 ceph-mon[36835]: pgmap v4470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:55:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:32 smithi077 ceph-mon[37153]: pgmap v4471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7554.530082 secs 2024-07-27T08:55:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:32 smithi047 ceph-mon[36835]: pgmap v4471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7554.530082 secs 2024-07-27T08:55:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:32 smithi029 ceph-mon[32610]: pgmap v4471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7554.530082 secs 2024-07-27T08:55:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:34 smithi077 ceph-mon[37153]: pgmap v4472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:55:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:34 smithi047 ceph-mon[36835]: pgmap v4472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:55:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:34 smithi029 ceph-mon[32610]: pgmap v4472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:55:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:55:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:55:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:55:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:55:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:55:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:55:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:55:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:55:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:55:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:55:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:55:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:55:36.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:36 smithi047 ceph-mon[36835]: pgmap v4473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:55:36.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:36 smithi047 ceph-mon[36835]: pgmap v4474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:55:36.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:36 smithi047 ceph-mon[36835]: pgmap v4475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:55:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:36 smithi077 ceph-mon[37153]: pgmap v4473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:55:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:36 smithi077 ceph-mon[37153]: pgmap v4474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:55:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:36 smithi077 ceph-mon[37153]: pgmap v4475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:55:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:36 smithi029 ceph-mon[32610]: pgmap v4473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:55:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:36 smithi029 ceph-mon[32610]: pgmap v4474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T08:55:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:36 smithi029 ceph-mon[32610]: pgmap v4475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:55:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7559.530206 secs 2024-07-27T08:55:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7559.530206 secs 2024-07-27T08:55:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7559.530206 secs 2024-07-27T08:55:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:38 smithi077 ceph-mon[37153]: pgmap v4476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:55:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:38 smithi047 ceph-mon[36835]: pgmap v4476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:55:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:38 smithi029 ceph-mon[32610]: pgmap v4476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T08:55:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:40 smithi077 ceph-mon[37153]: pgmap v4477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:55:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:40 smithi047 ceph-mon[36835]: pgmap v4477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:55:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:40 smithi029 ceph-mon[32610]: pgmap v4477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T08:55:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:42 smithi077 ceph-mon[37153]: pgmap v4478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:55:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7564.530328 secs 2024-07-27T08:55:42.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:42 smithi029 ceph-mon[32610]: pgmap v4478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:55:42.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7564.530328 secs 2024-07-27T08:55:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:42 smithi047 ceph-mon[36835]: pgmap v4478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:55:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7564.530328 secs 2024-07-27T08:55:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:43.773+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:55:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:43.773+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:55:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:43.773+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:55:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:43.773+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:55:43.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:43.996+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:55:43.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:43.996+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:55:43.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:43.996+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:55:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:44.057+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:55:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:44.057+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:55:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:44.057+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:55:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:44 smithi077 ceph-mon[37153]: pgmap v4479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:55:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:44 smithi047 ceph-mon[36835]: pgmap v4479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:55:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:44 smithi029 ceph-mon[32610]: pgmap v4479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:55:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:46 smithi077 ceph-mon[37153]: pgmap v4480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:55:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:46 smithi047 ceph-mon[36835]: pgmap v4480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:55:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:46 smithi029 ceph-mon[32610]: pgmap v4480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:55:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7569.530471 secs 2024-07-27T08:55:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7569.530471 secs 2024-07-27T08:55:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7569.530471 secs 2024-07-27T08:55:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:48 smithi077 ceph-mon[37153]: pgmap v4481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:55:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:48 smithi047 ceph-mon[36835]: pgmap v4481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:55:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:48 smithi029 ceph-mon[32610]: pgmap v4481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:55:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:50 smithi077 ceph-mon[37153]: pgmap v4482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:55:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:50 smithi047 ceph-mon[36835]: pgmap v4482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:55:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:50 smithi029 ceph-mon[32610]: pgmap v4482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:55:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:52 smithi077 ceph-mon[37153]: pgmap v4483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:55:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7574.530626 secs 2024-07-27T08:55:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:52 smithi047 ceph-mon[36835]: pgmap v4483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:55:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7574.530626 secs 2024-07-27T08:55:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:52 smithi029 ceph-mon[32610]: pgmap v4483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:55:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7574.530626 secs 2024-07-27T08:55:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:54 smithi077 ceph-mon[37153]: pgmap v4484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:55:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:54 smithi047 ceph-mon[36835]: pgmap v4484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:55:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:54 smithi029 ceph-mon[32610]: pgmap v4484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:55:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:56 smithi077 ceph-mon[37153]: pgmap v4485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:56 smithi047 ceph-mon[36835]: pgmap v4485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:56 smithi029 ceph-mon[32610]: pgmap v4485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:55:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7579.530774 secs 2024-07-27T08:55:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7579.530774 secs 2024-07-27T08:55:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7579.530774 secs 2024-07-27T08:55:58.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:58.776+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:55:58.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:58.776+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:55:58.777 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:55:58 smithi029 ceph-mon[32610]: pgmap v4486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:55:58.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:58.776+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:55:58.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:58.776+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:55:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:55:58 smithi077 ceph-mon[37153]: pgmap v4486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:55:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:55:58 smithi047 ceph-mon[36835]: pgmap v4486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:55:59.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:59.006+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:55:59.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:59.006+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:55:59.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:55:59.006+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:59.060+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:59.060+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:55:59.060+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:56:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:00 smithi077 ceph-mon[37153]: pgmap v4487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:56:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:00 smithi047 ceph-mon[36835]: pgmap v4487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:56:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:00 smithi029 ceph-mon[32610]: pgmap v4487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:56:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:02 smithi077 ceph-mon[37153]: pgmap v4488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:56:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7584.530896 secs 2024-07-27T08:56:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:02 smithi047 ceph-mon[36835]: pgmap v4488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:56:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7584.530896 secs 2024-07-27T08:56:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:02 smithi029 ceph-mon[32610]: pgmap v4488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:56:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7584.530896 secs 2024-07-27T08:56:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:04 smithi077 ceph-mon[37153]: pgmap v4489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:56:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:04 smithi047 ceph-mon[36835]: pgmap v4489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:56:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:04 smithi029 ceph-mon[32610]: pgmap v4489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:56:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:06 smithi077 ceph-mon[37153]: pgmap v4490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:56:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:06 smithi047 ceph-mon[36835]: pgmap v4490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:56:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:06 smithi029 ceph-mon[32610]: pgmap v4490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:56:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7589.531016 secs 2024-07-27T08:56:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7589.531016 secs 2024-07-27T08:56:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7589.531016 secs 2024-07-27T08:56:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:08 smithi077 ceph-mon[37153]: pgmap v4491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:56:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:08 smithi047 ceph-mon[36835]: pgmap v4491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:56:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:08 smithi029 ceph-mon[32610]: pgmap v4491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T08:56:10.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:10 smithi077 ceph-mon[37153]: pgmap v4492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:56:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:10 smithi047 ceph-mon[36835]: pgmap v4492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:56:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:10 smithi029 ceph-mon[32610]: pgmap v4492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:56:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:12 smithi077 ceph-mon[37153]: pgmap v4493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:56:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7594.531133 secs 2024-07-27T08:56:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:12 smithi047 ceph-mon[36835]: pgmap v4493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:56:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7594.531133 secs 2024-07-27T08:56:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:12 smithi029 ceph-mon[32610]: pgmap v4493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:56:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7594.531133 secs 2024-07-27T08:56:14.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:56:14.019+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:56:14.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:56:14.019+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:56:14.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:56:14.019+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:56:14.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:56:14.074+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:56:14.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:56:14.074+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:56:14.076 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:56:14.074+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:56:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:14 smithi077 ceph-mon[37153]: pgmap v4494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:56:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:14 smithi047 ceph-mon[36835]: pgmap v4494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:56:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:14 smithi029 ceph-mon[32610]: pgmap v4494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:56:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:16 smithi077 ceph-mon[37153]: pgmap v4495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:56:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:16 smithi047 ceph-mon[36835]: pgmap v4495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:56:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:16 smithi029 ceph-mon[32610]: pgmap v4495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:56:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7599.531251 secs 2024-07-27T08:56:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7599.531251 secs 2024-07-27T08:56:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7599.531251 secs 2024-07-27T08:56:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:18 smithi077 ceph-mon[37153]: pgmap v4496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:56:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:18 smithi047 ceph-mon[36835]: pgmap v4496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:56:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:18 smithi029 ceph-mon[32610]: pgmap v4496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:56:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:20 smithi077 ceph-mon[37153]: pgmap v4497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:56:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:20 smithi047 ceph-mon[36835]: pgmap v4497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:56:20.966 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:20 smithi029 ceph-mon[32610]: pgmap v4497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:56:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:22 smithi077 ceph-mon[37153]: pgmap v4498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:56:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7604.531369 secs 2024-07-27T08:56:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:22 smithi047 ceph-mon[36835]: pgmap v4498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:56:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7604.531369 secs 2024-07-27T08:56:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:22 smithi029 ceph-mon[32610]: pgmap v4498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:56:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7604.531369 secs 2024-07-27T08:56:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:24 smithi077 ceph-mon[37153]: pgmap v4499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:56:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:24 smithi047 ceph-mon[36835]: pgmap v4499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:56:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:24 smithi029 ceph-mon[32610]: pgmap v4499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:56:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:26 smithi077 ceph-mon[37153]: pgmap v4500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:56:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:26 smithi047 ceph-mon[36835]: pgmap v4500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:56:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:26 smithi029 ceph-mon[32610]: pgmap v4500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:56:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7609.531465 secs 2024-07-27T08:56:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7609.531465 secs 2024-07-27T08:56:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7609.531465 secs 2024-07-27T08:56:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:28 smithi077 ceph-mon[37153]: pgmap v4501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:56:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:28 smithi047 ceph-mon[36835]: pgmap v4501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:56:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:28 smithi029 ceph-mon[32610]: pgmap v4501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T08:56:29.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:56:29.034+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:56:29.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:56:29.034+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:56:29.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:56:29.034+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:56:29.085+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:56:29.085+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:56:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:56:29.085+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:56:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:30 smithi077 ceph-mon[37153]: pgmap v4502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:56:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:30 smithi047 ceph-mon[36835]: pgmap v4502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:56:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:30 smithi029 ceph-mon[32610]: pgmap v4502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T08:56:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:32 smithi077 ceph-mon[37153]: pgmap v4503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:56:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7614.531571 secs 2024-07-27T08:56:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:32 smithi047 ceph-mon[36835]: pgmap v4503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:56:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7614.531571 secs 2024-07-27T08:56:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:32 smithi029 ceph-mon[32610]: pgmap v4503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:56:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7614.531571 secs 2024-07-27T08:56:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:34 smithi077 ceph-mon[37153]: pgmap v4504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:56:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:34 smithi047 ceph-mon[36835]: pgmap v4504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:56:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:34 smithi029 ceph-mon[32610]: pgmap v4504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:56:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:56:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:56:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:56:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:36 smithi047 ceph-mon[36835]: pgmap v4505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:56:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:56:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:56:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:56:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:36 smithi029 ceph-mon[32610]: pgmap v4505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:56:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:56:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:56:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:56:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:36 smithi077 ceph-mon[37153]: pgmap v4505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:56:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:56:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:56:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:56:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:37 smithi047 ceph-mon[36835]: pgmap v4506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:56:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:37 smithi047 ceph-mon[36835]: pgmap v4507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:56:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7619.531690 secs 2024-07-27T08:56:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:37 smithi029 ceph-mon[32610]: pgmap v4506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:56:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:37 smithi029 ceph-mon[32610]: pgmap v4507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:56:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7619.531690 secs 2024-07-27T08:56:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:37 smithi077 ceph-mon[37153]: pgmap v4506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:56:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:37 smithi077 ceph-mon[37153]: pgmap v4507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T08:56:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7619.531690 secs 2024-07-27T08:56:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:39 smithi047 ceph-mon[36835]: pgmap v4508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:56:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:39 smithi029 ceph-mon[32610]: pgmap v4508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:56:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:39 smithi077 ceph-mon[37153]: pgmap v4508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:56:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:41 smithi047 ceph-mon[36835]: pgmap v4509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:56:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:41 smithi029 ceph-mon[32610]: pgmap v4509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:56:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:41 smithi077 ceph-mon[37153]: pgmap v4509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:56:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7624.531818 secs 2024-07-27T08:56:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7624.531818 secs 2024-07-27T08:56:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7624.531818 secs 2024-07-27T08:56:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:43 smithi047 ceph-mon[36835]: pgmap v4510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:56:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:43 smithi029 ceph-mon[32610]: pgmap v4510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:56:44.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:56:44.050+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:56:44.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:56:44.050+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:56:44.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:56:44.050+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:56:44.102 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:56:44.100+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:56:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:56:44.100+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:56:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:56:44.100+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:56:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:43 smithi077 ceph-mon[37153]: pgmap v4510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:56:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:45 smithi047 ceph-mon[36835]: pgmap v4511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:56:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:45 smithi029 ceph-mon[32610]: pgmap v4511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:56:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:45 smithi077 ceph-mon[37153]: pgmap v4511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:56:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:47 smithi047 ceph-mon[36835]: pgmap v4512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T08:56:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7629.531952 secs 2024-07-27T08:56:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:47 smithi029 ceph-mon[32610]: pgmap v4512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T08:56:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7629.531952 secs 2024-07-27T08:56:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:47 smithi077 ceph-mon[37153]: pgmap v4512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T08:56:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7629.531952 secs 2024-07-27T08:56:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:49 smithi047 ceph-mon[36835]: pgmap v4513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:56:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:49 smithi029 ceph-mon[32610]: pgmap v4513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:56:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:49 smithi077 ceph-mon[37153]: pgmap v4513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:56:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:51 smithi047 ceph-mon[36835]: pgmap v4514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:56:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:51 smithi029 ceph-mon[32610]: pgmap v4514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:56:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:51 smithi077 ceph-mon[37153]: pgmap v4514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:56:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7634.532098 secs 2024-07-27T08:56:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7634.532098 secs 2024-07-27T08:56:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7634.532098 secs 2024-07-27T08:56:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:53 smithi047 ceph-mon[36835]: pgmap v4515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:56:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:53 smithi029 ceph-mon[32610]: pgmap v4515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:56:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:53 smithi077 ceph-mon[37153]: pgmap v4515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:56:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:55 smithi047 ceph-mon[36835]: pgmap v4516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:56:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:55 smithi029 ceph-mon[32610]: pgmap v4516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:56:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:55 smithi077 ceph-mon[37153]: pgmap v4516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T08:56:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:57 smithi047 ceph-mon[36835]: pgmap v4517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:56:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7639.532218 secs 2024-07-27T08:56:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:57 smithi029 ceph-mon[32610]: pgmap v4517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:56:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7639.532218 secs 2024-07-27T08:56:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:57 smithi077 ceph-mon[37153]: pgmap v4517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T08:56:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7639.532218 secs 2024-07-27T08:56:59.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:56:59.058+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:56:59.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:56:59.058+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:56:59.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:56:59.058+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:56:59.117 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:56:59.117+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:56:59.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:56:59.117+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:56:59.118 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:56:59.117+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:56:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:56:59 smithi047 ceph-mon[36835]: pgmap v4518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:56:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:56:59 smithi029 ceph-mon[32610]: pgmap v4518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:57:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:56:59 smithi077 ceph-mon[37153]: pgmap v4518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T08:57:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:01 smithi047 ceph-mon[36835]: pgmap v4519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:57:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:01 smithi029 ceph-mon[32610]: pgmap v4519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:57:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:01 smithi077 ceph-mon[37153]: pgmap v4519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T08:57:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7644.532344 secs 2024-07-27T08:57:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7644.532344 secs 2024-07-27T08:57:03.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7644.532344 secs 2024-07-27T08:57:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:03 smithi047 ceph-mon[36835]: pgmap v4520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:03 smithi029 ceph-mon[32610]: pgmap v4520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:03 smithi077 ceph-mon[37153]: pgmap v4520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:05 smithi047 ceph-mon[36835]: pgmap v4521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:57:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:05 smithi029 ceph-mon[32610]: pgmap v4521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:57:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:05 smithi077 ceph-mon[37153]: pgmap v4521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:57:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:07 smithi047 ceph-mon[36835]: pgmap v4522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7649.532413 secs 2024-07-27T08:57:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:07 smithi029 ceph-mon[32610]: pgmap v4522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7649.532413 secs 2024-07-27T08:57:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:07 smithi077 ceph-mon[37153]: pgmap v4522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7649.532413 secs 2024-07-27T08:57:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:09 smithi047 ceph-mon[36835]: pgmap v4523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:57:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:09 smithi029 ceph-mon[32610]: pgmap v4523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:57:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:09 smithi077 ceph-mon[37153]: pgmap v4523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:57:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:11 smithi047 ceph-mon[36835]: pgmap v4524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:11 smithi029 ceph-mon[32610]: pgmap v4524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:11 smithi077 ceph-mon[37153]: pgmap v4524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:12.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7654.532557 secs 2024-07-27T08:57:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7654.532557 secs 2024-07-27T08:57:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7654.532557 secs 2024-07-27T08:57:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:13 smithi047 ceph-mon[36835]: pgmap v4525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:13 smithi029 ceph-mon[32610]: pgmap v4525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:14.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:57:14.073+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:57:14.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:57:14.073+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:57:14.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:57:14.073+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:57:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:13 smithi077 ceph-mon[37153]: pgmap v4525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:57:14.131+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:57:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:57:14.131+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:57:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:57:14.131+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:57:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:15 smithi047 ceph-mon[36835]: pgmap v4526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:57:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:15 smithi029 ceph-mon[32610]: pgmap v4526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:57:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:15 smithi077 ceph-mon[37153]: pgmap v4526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:57:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:17 smithi047 ceph-mon[36835]: pgmap v4527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7659.532700 secs 2024-07-27T08:57:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:17 smithi029 ceph-mon[32610]: pgmap v4527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7659.532700 secs 2024-07-27T08:57:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:17 smithi077 ceph-mon[37153]: pgmap v4527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7659.532700 secs 2024-07-27T08:57:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:19 smithi047 ceph-mon[36835]: pgmap v4528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:57:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:19 smithi029 ceph-mon[32610]: pgmap v4528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:57:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:19 smithi077 ceph-mon[37153]: pgmap v4528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:57:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:21 smithi029 ceph-mon[32610]: pgmap v4529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:21 smithi077 ceph-mon[37153]: pgmap v4529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:21 smithi047 ceph-mon[36835]: pgmap v4529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7664.532802 secs 2024-07-27T08:57:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7664.532802 secs 2024-07-27T08:57:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7664.532802 secs 2024-07-27T08:57:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:23 smithi029 ceph-mon[32610]: pgmap v4530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:23 smithi077 ceph-mon[37153]: pgmap v4530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:23 smithi047 ceph-mon[36835]: pgmap v4530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:25 smithi029 ceph-mon[32610]: pgmap v4531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:57:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:25 smithi077 ceph-mon[37153]: pgmap v4531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:57:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:25 smithi047 ceph-mon[36835]: pgmap v4531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:57:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:27 smithi029 ceph-mon[32610]: pgmap v4532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7669.532909 secs 2024-07-27T08:57:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:27 smithi077 ceph-mon[37153]: pgmap v4532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7669.532909 secs 2024-07-27T08:57:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:27 smithi047 ceph-mon[36835]: pgmap v4532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7669.532909 secs 2024-07-27T08:57:29.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:57:29.077+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:57:29.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:57:29.077+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:57:29.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:57:29.077+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:57:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:57:29.146+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:57:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:57:29.146+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:57:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:57:29.146+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:57:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:29 smithi077 ceph-mon[37153]: pgmap v4533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:57:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:29 smithi047 ceph-mon[36835]: pgmap v4533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:57:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:29 smithi029 ceph-mon[32610]: pgmap v4533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:57:32.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:31 smithi029 ceph-mon[32610]: pgmap v4534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:31 smithi077 ceph-mon[37153]: pgmap v4534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:31 smithi047 ceph-mon[36835]: pgmap v4534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:33.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7674.533030 secs 2024-07-27T08:57:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7674.533030 secs 2024-07-27T08:57:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7674.533030 secs 2024-07-27T08:57:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:33 smithi077 ceph-mon[37153]: pgmap v4535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:33 smithi047 ceph-mon[36835]: pgmap v4535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:33 smithi029 ceph-mon[32610]: pgmap v4535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:35 smithi077 ceph-mon[37153]: pgmap v4536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:57:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:35 smithi047 ceph-mon[36835]: pgmap v4536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:57:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:35 smithi029 ceph-mon[32610]: pgmap v4536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:57:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:57:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:37.069 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:57:37.069 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:37.069 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:57:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:37 smithi077 ceph-mon[37153]: pgmap v4537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7679.533129 secs 2024-07-27T08:57:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:37 smithi047 ceph-mon[36835]: pgmap v4537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7679.533129 secs 2024-07-27T08:57:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:37 smithi029 ceph-mon[32610]: pgmap v4537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7679.533129 secs 2024-07-27T08:57:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:38.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:39 smithi077 ceph-mon[37153]: pgmap v4538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:39 smithi077 ceph-mon[37153]: pgmap v4539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:39 smithi077 ceph-mon[37153]: pgmap v4540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:39 smithi047 ceph-mon[36835]: pgmap v4538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:57:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:57:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:57:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:39 smithi047 ceph-mon[36835]: pgmap v4539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:39 smithi047 ceph-mon[36835]: pgmap v4540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:57:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:39 smithi029 ceph-mon[32610]: pgmap v4538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:57:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:57:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:57:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:39 smithi029 ceph-mon[32610]: pgmap v4539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:57:39.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:39 smithi029 ceph-mon[32610]: pgmap v4540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:57:39.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:57:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:41 smithi047 ceph-mon[36835]: pgmap v4541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:57:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:41 smithi029 ceph-mon[32610]: pgmap v4541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:57:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:41 smithi077 ceph-mon[37153]: pgmap v4541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:57:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7684.533192 secs 2024-07-27T08:57:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7684.533192 secs 2024-07-27T08:57:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7684.533192 secs 2024-07-27T08:57:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:43 smithi047 ceph-mon[36835]: pgmap v4542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:57:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:43 smithi029 ceph-mon[32610]: pgmap v4542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:57:44.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:57:44.078+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:57:44.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:57:44.078+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:57:44.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:57:44.078+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:57:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:43 smithi077 ceph-mon[37153]: pgmap v4542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T08:57:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:57:44.157+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:57:44.159 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:57:44.157+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:57:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:57:44.157+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:57:45.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:45 smithi047 ceph-mon[36835]: pgmap v4543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:57:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:45 smithi029 ceph-mon[32610]: pgmap v4543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:57:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:45 smithi077 ceph-mon[37153]: pgmap v4543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:57:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:47 smithi029 ceph-mon[32610]: pgmap v4544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:57:47.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7689.533315 secs 2024-07-27T08:57:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:47 smithi077 ceph-mon[37153]: pgmap v4544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:57:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7689.533315 secs 2024-07-27T08:57:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:47 smithi047 ceph-mon[36835]: pgmap v4544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T08:57:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7689.533315 secs 2024-07-27T08:57:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:49 smithi029 ceph-mon[32610]: pgmap v4545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:57:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:49 smithi077 ceph-mon[37153]: pgmap v4545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:57:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:49 smithi047 ceph-mon[36835]: pgmap v4545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:57:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:51 smithi029 ceph-mon[32610]: pgmap v4546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:51 smithi077 ceph-mon[37153]: pgmap v4546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:51 smithi047 ceph-mon[36835]: pgmap v4546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:57:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7694.533439 secs 2024-07-27T08:57:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7694.533439 secs 2024-07-27T08:57:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7694.533439 secs 2024-07-27T08:57:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:53 smithi029 ceph-mon[32610]: pgmap v4547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:57:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:53 smithi077 ceph-mon[37153]: pgmap v4547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:57:54.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:53 smithi047 ceph-mon[36835]: pgmap v4547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:57:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:55 smithi077 ceph-mon[37153]: pgmap v4548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:57:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:55 smithi047 ceph-mon[36835]: pgmap v4548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:57:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:55 smithi029 ceph-mon[32610]: pgmap v4548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:57:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:57 smithi077 ceph-mon[37153]: pgmap v4549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:57:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7699.533566 secs 2024-07-27T08:57:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:57 smithi047 ceph-mon[36835]: pgmap v4549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:57:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7699.533566 secs 2024-07-27T08:57:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:57 smithi029 ceph-mon[32610]: pgmap v4549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:57:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7699.533566 secs 2024-07-27T08:57:59.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:57:59.094+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:57:59.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:57:59.094+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:57:59.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:57:59.094+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:57:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:57:59.164+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:57:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:57:59.164+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:57:59.165 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:57:59.164+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:58:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:57:59 smithi077 ceph-mon[37153]: pgmap v4550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:57:59 smithi047 ceph-mon[36835]: pgmap v4550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:57:59 smithi029 ceph-mon[32610]: pgmap v4550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:01 smithi077 ceph-mon[37153]: pgmap v4551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:01 smithi047 ceph-mon[36835]: pgmap v4551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:01 smithi029 ceph-mon[32610]: pgmap v4551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:03.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7704.533687 secs 2024-07-27T08:58:03.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7704.533687 secs 2024-07-27T08:58:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7704.533687 secs 2024-07-27T08:58:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:03 smithi077 ceph-mon[37153]: pgmap v4552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:03 smithi047 ceph-mon[36835]: pgmap v4552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:03 smithi029 ceph-mon[32610]: pgmap v4552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:05 smithi077 ceph-mon[37153]: pgmap v4553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:58:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:05 smithi047 ceph-mon[36835]: pgmap v4553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:58:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:05 smithi029 ceph-mon[32610]: pgmap v4553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:58:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:06 smithi077 ceph-mon[37153]: pgmap v4554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:06 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7709.533822 secs 2024-07-27T08:58:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:06 smithi047 ceph-mon[36835]: pgmap v4554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:06 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7709.533822 secs 2024-07-27T08:58:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:06 smithi029 ceph-mon[32610]: pgmap v4554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:06 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7709.533822 secs 2024-07-27T08:58:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:09 smithi047 ceph-mon[36835]: pgmap v4555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:09 smithi029 ceph-mon[32610]: pgmap v4555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:09 smithi077 ceph-mon[37153]: pgmap v4555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:11 smithi029 ceph-mon[32610]: pgmap v4556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:11 smithi077 ceph-mon[37153]: pgmap v4556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:11 smithi047 ceph-mon[36835]: pgmap v4556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7714.533964 secs 2024-07-27T08:58:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7714.533964 secs 2024-07-27T08:58:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7714.533964 secs 2024-07-27T08:58:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:13 smithi029 ceph-mon[32610]: pgmap v4557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:13 smithi077 ceph-mon[37153]: pgmap v4557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:14.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:14.108+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:58:14.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:14.108+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:58:14.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:14.108+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:58:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:14.179+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:58:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:14.179+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:58:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:14.179+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:58:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:13 smithi047 ceph-mon[36835]: pgmap v4557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:15 smithi029 ceph-mon[32610]: pgmap v4558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:58:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:15 smithi077 ceph-mon[37153]: pgmap v4558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:58:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:15 smithi047 ceph-mon[36835]: pgmap v4558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:58:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:17 smithi029 ceph-mon[32610]: pgmap v4559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7719.534099 secs 2024-07-27T08:58:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:17 smithi077 ceph-mon[37153]: pgmap v4559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7719.534099 secs 2024-07-27T08:58:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:17 smithi047 ceph-mon[36835]: pgmap v4559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7719.534099 secs 2024-07-27T08:58:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:19 smithi077 ceph-mon[37153]: pgmap v4560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:19 smithi047 ceph-mon[36835]: pgmap v4560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:19 smithi029 ceph-mon[32610]: pgmap v4560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:21 smithi077 ceph-mon[37153]: pgmap v4561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:21 smithi047 ceph-mon[36835]: pgmap v4561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:21 smithi029 ceph-mon[32610]: pgmap v4561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7724.534221 secs 2024-07-27T08:58:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7724.534221 secs 2024-07-27T08:58:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7724.534221 secs 2024-07-27T08:58:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:23 smithi077 ceph-mon[37153]: pgmap v4562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:23 smithi047 ceph-mon[36835]: pgmap v4562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:23 smithi029 ceph-mon[32610]: pgmap v4562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:25 smithi077 ceph-mon[37153]: pgmap v4563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:58:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:25 smithi047 ceph-mon[36835]: pgmap v4563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:58:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:25 smithi029 ceph-mon[32610]: pgmap v4563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:58:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:27 smithi077 ceph-mon[37153]: pgmap v4564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7729.534299 secs 2024-07-27T08:58:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:27 smithi047 ceph-mon[36835]: pgmap v4564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7729.534299 secs 2024-07-27T08:58:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:27 smithi029 ceph-mon[32610]: pgmap v4564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7729.534299 secs 2024-07-27T08:58:29.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:29.124+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:58:29.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:29.124+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:58:29.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:29.124+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:58:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:29.194+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:58:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:29.194+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:58:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:29.194+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:58:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:29 smithi077 ceph-mon[37153]: pgmap v4565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:30.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:29 smithi047 ceph-mon[36835]: pgmap v4565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:29 smithi029 ceph-mon[32610]: pgmap v4565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:31 smithi077 ceph-mon[37153]: pgmap v4566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:31 smithi047 ceph-mon[36835]: pgmap v4566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:31 smithi029 ceph-mon[32610]: pgmap v4566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:33.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7734.534420 secs 2024-07-27T08:58:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7734.534420 secs 2024-07-27T08:58:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7734.534420 secs 2024-07-27T08:58:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:33 smithi077 ceph-mon[37153]: pgmap v4567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:33 smithi047 ceph-mon[36835]: pgmap v4567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:33 smithi029 ceph-mon[32610]: pgmap v4567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:35 smithi077 ceph-mon[37153]: pgmap v4568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:58:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:35 smithi047 ceph-mon[36835]: pgmap v4568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:58:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:35 smithi029 ceph-mon[32610]: pgmap v4568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:58:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:37 smithi077 ceph-mon[37153]: pgmap v4569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7739.534542 secs 2024-07-27T08:58:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:37 smithi047 ceph-mon[36835]: pgmap v4569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7739.534542 secs 2024-07-27T08:58:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:37 smithi029 ceph-mon[32610]: pgmap v4569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7739.534542 secs 2024-07-27T08:58:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:58:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:58:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:58:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:39 smithi077 ceph-mon[37153]: pgmap v4570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:58:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:58:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:39 smithi047 ceph-mon[36835]: pgmap v4570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:58:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:58:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:39 smithi029 ceph-mon[32610]: pgmap v4570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:40.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:58:40.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:58:40.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:58:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:40 smithi077 ceph-mon[37153]: pgmap v4571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:58:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:40 smithi077 ceph-mon[37153]: pgmap v4572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:58:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:40 smithi047 ceph-mon[36835]: pgmap v4571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:58:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:40 smithi047 ceph-mon[36835]: pgmap v4572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:58:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:40 smithi029 ceph-mon[32610]: pgmap v4571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T08:58:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:40 smithi029 ceph-mon[32610]: pgmap v4572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T08:58:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:42 smithi077 ceph-mon[37153]: pgmap v4573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:58:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7744.534689 secs 2024-07-27T08:58:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:42 smithi047 ceph-mon[36835]: pgmap v4573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:58:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7744.534689 secs 2024-07-27T08:58:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:42 smithi029 ceph-mon[32610]: pgmap v4573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T08:58:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7744.534689 secs 2024-07-27T08:58:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:43.802+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:58:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:43.802+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:58:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:43.802+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T08:58:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:43.802+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T08:58:44.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:44.139+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:58:44.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:44.139+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:58:44.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:44.139+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:58:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:44.209+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:58:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:44.209+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:58:44.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:44.209+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:58:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:44 smithi077 ceph-mon[37153]: pgmap v4574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:58:45.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:44 smithi047 ceph-mon[36835]: pgmap v4574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:58:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:44 smithi029 ceph-mon[32610]: pgmap v4574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:58:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:46 smithi077 ceph-mon[37153]: pgmap v4575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:58:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:46 smithi047 ceph-mon[36835]: pgmap v4575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:58:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:46 smithi029 ceph-mon[32610]: pgmap v4575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:58:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7749.534838 secs 2024-07-27T08:58:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7749.534838 secs 2024-07-27T08:58:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7749.534838 secs 2024-07-27T08:58:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:48 smithi077 ceph-mon[37153]: pgmap v4576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:58:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:48 smithi047 ceph-mon[36835]: pgmap v4576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:58:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:48 smithi029 ceph-mon[32610]: pgmap v4576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T08:58:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:50 smithi077 ceph-mon[37153]: pgmap v4577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:58:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:50 smithi047 ceph-mon[36835]: pgmap v4577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:58:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:50 smithi029 ceph-mon[32610]: pgmap v4577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:58:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:52 smithi077 ceph-mon[37153]: pgmap v4578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7754.534960 secs 2024-07-27T08:58:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:52 smithi047 ceph-mon[36835]: pgmap v4578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7754.534960 secs 2024-07-27T08:58:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:52 smithi029 ceph-mon[32610]: pgmap v4578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:58:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7754.534960 secs 2024-07-27T08:58:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:54 smithi077 ceph-mon[37153]: pgmap v4579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:54 smithi047 ceph-mon[36835]: pgmap v4579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:54 smithi029 ceph-mon[32610]: pgmap v4579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:58:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:56 smithi077 ceph-mon[37153]: pgmap v4580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:56 smithi047 ceph-mon[36835]: pgmap v4580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:56 smithi029 ceph-mon[32610]: pgmap v4580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:58:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7759.535088 secs 2024-07-27T08:58:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7759.535088 secs 2024-07-27T08:58:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7759.535088 secs 2024-07-27T08:58:58.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:58.805+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:58:58.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:58.805+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:58:58.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:58.805+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T08:58:58.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:58.805+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T08:58:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:58:58 smithi077 ceph-mon[37153]: pgmap v4581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:59.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:58:58 smithi029 ceph-mon[32610]: pgmap v4581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:59.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:59.155+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:58:59.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:59.155+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:58:59.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:58:59.155+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:58:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:58:58 smithi047 ceph-mon[36835]: pgmap v4581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:58:59.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:59.225+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:58:59.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:59.225+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:58:59.226 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:58:59.225+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:59:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:00 smithi077 ceph-mon[37153]: pgmap v4582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:00 smithi047 ceph-mon[36835]: pgmap v4582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:00 smithi029 ceph-mon[32610]: pgmap v4582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:02 smithi077 ceph-mon[37153]: pgmap v4583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7764.535213 secs 2024-07-27T08:59:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:02 smithi047 ceph-mon[36835]: pgmap v4583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7764.535213 secs 2024-07-27T08:59:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:02 smithi029 ceph-mon[32610]: pgmap v4583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7764.535213 secs 2024-07-27T08:59:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:04 smithi077 ceph-mon[37153]: pgmap v4584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:59:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:04 smithi047 ceph-mon[36835]: pgmap v4584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:59:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:04 smithi029 ceph-mon[32610]: pgmap v4584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:59:06.152 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:05 smithi029 ceph-mon[32610]: pgmap v4585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:05 smithi047 ceph-mon[36835]: pgmap v4585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:05 smithi077 ceph-mon[37153]: pgmap v4585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:06 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7769.535367 secs 2024-07-27T08:59:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:06 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7769.535367 secs 2024-07-27T08:59:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:06 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7769.535367 secs 2024-07-27T08:59:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:07 smithi047 ceph-mon[36835]: pgmap v4586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:59:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:07 smithi029 ceph-mon[32610]: pgmap v4586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:59:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:07 smithi077 ceph-mon[37153]: pgmap v4586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:59:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:10 smithi029 ceph-mon[32610]: pgmap v4587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:10 smithi077 ceph-mon[37153]: pgmap v4587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:10 smithi047 ceph-mon[36835]: pgmap v4587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:12 smithi077 ceph-mon[37153]: pgmap v4588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7774.535486 secs 2024-07-27T08:59:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:12 smithi047 ceph-mon[36835]: pgmap v4588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7774.535486 secs 2024-07-27T08:59:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:12 smithi029 ceph-mon[32610]: pgmap v4588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7774.535486 secs 2024-07-27T08:59:14.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:59:14.162+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:59:14.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:59:14.162+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:59:14.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:59:14.162+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:59:14.241 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:59:14.240+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:59:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:59:14.240+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:59:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:59:14.240+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:59:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:14 smithi077 ceph-mon[37153]: pgmap v4589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:59:14.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:14 smithi047 ceph-mon[36835]: pgmap v4589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:59:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:14 smithi029 ceph-mon[32610]: pgmap v4589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:59:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:16 smithi077 ceph-mon[37153]: pgmap v4590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:16 smithi047 ceph-mon[36835]: pgmap v4590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:16 smithi029 ceph-mon[32610]: pgmap v4590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7779.535609 secs 2024-07-27T08:59:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7779.535609 secs 2024-07-27T08:59:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7779.535609 secs 2024-07-27T08:59:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:18 smithi077 ceph-mon[37153]: pgmap v4591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:59:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:18 smithi047 ceph-mon[36835]: pgmap v4591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:59:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:18 smithi029 ceph-mon[32610]: pgmap v4591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:59:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:20 smithi077 ceph-mon[37153]: pgmap v4592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:20.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:20 smithi047 ceph-mon[36835]: pgmap v4592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:20 smithi029 ceph-mon[32610]: pgmap v4592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:22 smithi077 ceph-mon[37153]: pgmap v4593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7784.535734 secs 2024-07-27T08:59:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:22 smithi047 ceph-mon[36835]: pgmap v4593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7784.535734 secs 2024-07-27T08:59:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:22 smithi029 ceph-mon[32610]: pgmap v4593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7784.535734 secs 2024-07-27T08:59:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:24 smithi077 ceph-mon[37153]: pgmap v4594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:59:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:24 smithi047 ceph-mon[36835]: pgmap v4594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:59:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:24 smithi029 ceph-mon[32610]: pgmap v4594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:59:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:26 smithi077 ceph-mon[37153]: pgmap v4595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:26.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:26 smithi047 ceph-mon[36835]: pgmap v4595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:26 smithi029 ceph-mon[32610]: pgmap v4595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7789.535862 secs 2024-07-27T08:59:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7789.535862 secs 2024-07-27T08:59:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7789.535862 secs 2024-07-27T08:59:28.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:28 smithi077 ceph-mon[37153]: pgmap v4596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:59:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:28 smithi047 ceph-mon[36835]: pgmap v4596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:59:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:28 smithi029 ceph-mon[32610]: pgmap v4596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:59:29.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:59:29.177+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:59:29.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:59:29.177+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:59:29.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:59:29.177+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:59:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:59:29.255+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:59:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:59:29.255+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:59:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:59:29.255+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:59:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:30 smithi077 ceph-mon[37153]: pgmap v4597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:30 smithi047 ceph-mon[36835]: pgmap v4597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:30 smithi029 ceph-mon[32610]: pgmap v4597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:32 smithi077 ceph-mon[37153]: pgmap v4598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7794.535986 secs 2024-07-27T08:59:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:32 smithi047 ceph-mon[36835]: pgmap v4598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7794.535986 secs 2024-07-27T08:59:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:32 smithi029 ceph-mon[32610]: pgmap v4598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7794.535986 secs 2024-07-27T08:59:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:34 smithi077 ceph-mon[37153]: pgmap v4599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:59:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:34 smithi047 ceph-mon[36835]: pgmap v4599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:59:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:34 smithi029 ceph-mon[32610]: pgmap v4599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T08:59:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:36 smithi077 ceph-mon[37153]: pgmap v4600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:36.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:36 smithi029 ceph-mon[32610]: pgmap v4600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:36 smithi047 ceph-mon[36835]: pgmap v4600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:37.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7799.536111 secs 2024-07-27T08:59:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7799.536111 secs 2024-07-27T08:59:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7799.536111 secs 2024-07-27T08:59:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:38 smithi077 ceph-mon[37153]: pgmap v4601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:59:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:38 smithi047 ceph-mon[36835]: pgmap v4601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:59:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:38 smithi029 ceph-mon[32610]: pgmap v4601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T08:59:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:59:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:59:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T08:59:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:40 smithi047 ceph-mon[36835]: pgmap v4602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:40.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:59:40.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:59:40.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:59:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:40 smithi029 ceph-mon[32610]: pgmap v4602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:59:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:59:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:59:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:40 smithi077 ceph-mon[37153]: pgmap v4602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T08:59:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T08:59:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T08:59:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T08:59:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:41 smithi047 ceph-mon[36835]: pgmap v4603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:59:41.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:41 smithi047 ceph-mon[36835]: pgmap v4604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:59:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:41 smithi029 ceph-mon[32610]: pgmap v4603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:59:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:41 smithi029 ceph-mon[32610]: pgmap v4604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:59:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:41 smithi077 ceph-mon[37153]: pgmap v4603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T08:59:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:41 smithi077 ceph-mon[37153]: pgmap v4604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T08:59:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7804.536232 secs 2024-07-27T08:59:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7804.536232 secs 2024-07-27T08:59:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7804.536232 secs 2024-07-27T08:59:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:43 smithi047 ceph-mon[36835]: pgmap v4605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:59:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:43 smithi029 ceph-mon[32610]: pgmap v4605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:59:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:43 smithi077 ceph-mon[37153]: pgmap v4605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T08:59:44.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:59:44.193+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:59:44.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:59:44.193+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:59:44.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:59:44.193+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:59:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:59:44.271+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:59:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:59:44.271+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:59:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:59:44.271+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:59:45.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:45 smithi047 ceph-mon[36835]: pgmap v4606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:59:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:45 smithi029 ceph-mon[32610]: pgmap v4606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:59:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:45 smithi077 ceph-mon[37153]: pgmap v4606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T08:59:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:47 smithi047 ceph-mon[36835]: pgmap v4607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:59:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7809.536370 secs 2024-07-27T08:59:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:47 smithi029 ceph-mon[32610]: pgmap v4607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:59:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7809.536370 secs 2024-07-27T08:59:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:47 smithi077 ceph-mon[37153]: pgmap v4607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T08:59:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7809.536370 secs 2024-07-27T08:59:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:49 smithi047 ceph-mon[36835]: pgmap v4608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:59:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:49 smithi029 ceph-mon[32610]: pgmap v4608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:59:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:49 smithi077 ceph-mon[37153]: pgmap v4608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T08:59:51.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:51 smithi047 ceph-mon[36835]: pgmap v4609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:59:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:51 smithi029 ceph-mon[32610]: pgmap v4609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:59:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:51 smithi077 ceph-mon[37153]: pgmap v4609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T08:59:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7814.536508 secs 2024-07-27T08:59:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7814.536508 secs 2024-07-27T08:59:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7814.536508 secs 2024-07-27T08:59:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:53 smithi047 ceph-mon[36835]: pgmap v4610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:53 smithi029 ceph-mon[32610]: pgmap v4610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:53 smithi077 ceph-mon[37153]: pgmap v4610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T08:59:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:55 smithi047 ceph-mon[36835]: pgmap v4611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:59:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:55 smithi029 ceph-mon[32610]: pgmap v4611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:59:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:55 smithi077 ceph-mon[37153]: pgmap v4611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T08:59:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:57 smithi047 ceph-mon[36835]: pgmap v4612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7819.536628 secs 2024-07-27T08:59:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:57 smithi029 ceph-mon[32610]: pgmap v4612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7819.536628 secs 2024-07-27T08:59:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:57 smithi077 ceph-mon[37153]: pgmap v4612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T08:59:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7819.536628 secs 2024-07-27T08:59:59.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:59:59.198+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T08:59:59.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:59:59.198+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:59:59.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T08:59:59.198+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T08:59:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:59:59.286+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T08:59:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:59:59.286+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T08:59:59.288 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T08:59:59.286+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T08:59:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 08:59:59 smithi047 ceph-mon[36835]: pgmap v4613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T08:59:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 08:59:59 smithi029 ceph-mon[32610]: pgmap v4613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 08:59:59 smithi077 ceph-mon[37153]: pgmap v4613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:00:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:00:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:00:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:01 smithi047 ceph-mon[36835]: pgmap v4614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:01 smithi029 ceph-mon[32610]: pgmap v4614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:01 smithi077 ceph-mon[37153]: pgmap v4614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7824.536752 secs 2024-07-27T09:00:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7824.536752 secs 2024-07-27T09:00:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7824.536752 secs 2024-07-27T09:00:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:03 smithi047 ceph-mon[36835]: pgmap v4615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:03 smithi029 ceph-mon[32610]: pgmap v4615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:04.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:03 smithi077 ceph-mon[37153]: pgmap v4615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:05 smithi047 ceph-mon[36835]: pgmap v4616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:00:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:05 smithi029 ceph-mon[32610]: pgmap v4616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:00:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:05 smithi077 ceph-mon[37153]: pgmap v4616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:00:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:07 smithi047 ceph-mon[36835]: pgmap v4617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7829.536859 secs 2024-07-27T09:00:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:07 smithi029 ceph-mon[32610]: pgmap v4617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7829.536859 secs 2024-07-27T09:00:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:07 smithi077 ceph-mon[37153]: pgmap v4617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7829.536859 secs 2024-07-27T09:00:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:09 smithi047 ceph-mon[36835]: pgmap v4618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:09 smithi029 ceph-mon[32610]: pgmap v4618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:09 smithi077 ceph-mon[37153]: pgmap v4618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:11 smithi047 ceph-mon[36835]: pgmap v4619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:11 smithi029 ceph-mon[32610]: pgmap v4619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:11 smithi077 ceph-mon[37153]: pgmap v4619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7834.536956 secs 2024-07-27T09:00:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7834.536956 secs 2024-07-27T09:00:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7834.536956 secs 2024-07-27T09:00:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:13 smithi047 ceph-mon[36835]: pgmap v4620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:13 smithi029 ceph-mon[32610]: pgmap v4620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:13 smithi077 ceph-mon[37153]: pgmap v4620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:14.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:00:14.201+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:00:14.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:00:14.201+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:00:14.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:00:14.201+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:00:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:00:14.301+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:00:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:00:14.301+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:00:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:00:14.301+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:00:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:15 smithi047 ceph-mon[36835]: pgmap v4621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:00:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:15 smithi029 ceph-mon[32610]: pgmap v4621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:00:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:15 smithi077 ceph-mon[37153]: pgmap v4621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:00:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:17 smithi047 ceph-mon[36835]: pgmap v4622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7839.537079 secs 2024-07-27T09:00:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:17 smithi029 ceph-mon[32610]: pgmap v4622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7839.537079 secs 2024-07-27T09:00:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:17 smithi077 ceph-mon[37153]: pgmap v4622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7839.537079 secs 2024-07-27T09:00:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:19 smithi029 ceph-mon[32610]: pgmap v4623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:19 smithi077 ceph-mon[37153]: pgmap v4623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:19 smithi047 ceph-mon[36835]: pgmap v4623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:21 smithi029 ceph-mon[32610]: pgmap v4624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:00:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:21 smithi077 ceph-mon[37153]: pgmap v4624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:00:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:21 smithi047 ceph-mon[36835]: pgmap v4624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:00:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7844.537204 secs 2024-07-27T09:00:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7844.537204 secs 2024-07-27T09:00:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7844.537204 secs 2024-07-27T09:00:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:23 smithi029 ceph-mon[32610]: pgmap v4625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:23 smithi077 ceph-mon[37153]: pgmap v4625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:23 smithi047 ceph-mon[36835]: pgmap v4625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:25 smithi029 ceph-mon[32610]: pgmap v4626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:00:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:25 smithi077 ceph-mon[37153]: pgmap v4626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:00:26.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:25 smithi047 ceph-mon[36835]: pgmap v4626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:00:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:27 smithi077 ceph-mon[37153]: pgmap v4627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7849.537326 secs 2024-07-27T09:00:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:27 smithi047 ceph-mon[36835]: pgmap v4627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7849.537326 secs 2024-07-27T09:00:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:27 smithi029 ceph-mon[32610]: pgmap v4627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7849.537326 secs 2024-07-27T09:00:29.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:00:29.204+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:00:29.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:00:29.204+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:00:29.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:00:29.204+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:00:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:00:29.305+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:00:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:00:29.305+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:00:29.307 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:00:29.305+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:00:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:29 smithi077 ceph-mon[37153]: pgmap v4628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:29 smithi047 ceph-mon[36835]: pgmap v4628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:29 smithi029 ceph-mon[32610]: pgmap v4628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:31 smithi077 ceph-mon[37153]: pgmap v4629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:00:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:31 smithi047 ceph-mon[36835]: pgmap v4629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:00:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:31 smithi029 ceph-mon[32610]: pgmap v4629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:00:33.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7854.537451 secs 2024-07-27T09:00:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7854.537451 secs 2024-07-27T09:00:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7854.537451 secs 2024-07-27T09:00:34.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:33 smithi077 ceph-mon[37153]: pgmap v4630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:33 smithi047 ceph-mon[36835]: pgmap v4630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:33 smithi029 ceph-mon[32610]: pgmap v4630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:35 smithi077 ceph-mon[37153]: pgmap v4631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:00:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:35 smithi047 ceph-mon[36835]: pgmap v4631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:00:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:35 smithi029 ceph-mon[32610]: pgmap v4631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:00:37.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:37 smithi047 ceph-mon[36835]: pgmap v4632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:37.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7859.537574 secs 2024-07-27T09:00:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:37 smithi077 ceph-mon[37153]: pgmap v4632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7859.537574 secs 2024-07-27T09:00:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:37 smithi029 ceph-mon[32610]: pgmap v4632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7859.537574 secs 2024-07-27T09:00:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:39 smithi077 ceph-mon[37153]: pgmap v4633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:39 smithi047 ceph-mon[36835]: pgmap v4633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:39 smithi029 ceph-mon[32610]: pgmap v4633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:00:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:00:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:00:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:00:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:00:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:00:41.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:00:41.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:00:41.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:00:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:00:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:00:41.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:00:41.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:00:42.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:41 smithi077 ceph-mon[37153]: pgmap v4634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:42.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:41 smithi077 ceph-mon[37153]: pgmap v4635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:00:42.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:41 smithi077 ceph-mon[37153]: pgmap v4636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:00:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:41 smithi047 ceph-mon[36835]: pgmap v4634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:41 smithi047 ceph-mon[36835]: pgmap v4635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:00:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:41 smithi047 ceph-mon[36835]: pgmap v4636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:00:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:41 smithi029 ceph-mon[32610]: pgmap v4634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:00:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:41 smithi029 ceph-mon[32610]: pgmap v4635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:00:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:41 smithi029 ceph-mon[32610]: pgmap v4636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:00:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7864.537690 secs 2024-07-27T09:00:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7864.537690 secs 2024-07-27T09:00:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7864.537690 secs 2024-07-27T09:00:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:43 smithi077 ceph-mon[37153]: pgmap v4637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:00:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:43 smithi047 ceph-mon[36835]: pgmap v4637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:00:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:43 smithi029 ceph-mon[32610]: pgmap v4637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:00:44.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:00:44.219+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:00:44.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:00:44.219+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:00:44.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:00:44.219+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:00:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:00:44.316+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:00:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:00:44.316+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:00:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:00:44.316+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:00:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:45 smithi077 ceph-mon[37153]: pgmap v4638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:00:46.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:45 smithi047 ceph-mon[36835]: pgmap v4638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:00:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:45 smithi029 ceph-mon[32610]: pgmap v4638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:00:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:47 smithi077 ceph-mon[37153]: pgmap v4639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:00:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7869.537815 secs 2024-07-27T09:00:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:47 smithi047 ceph-mon[36835]: pgmap v4639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:00:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7869.537815 secs 2024-07-27T09:00:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:47 smithi029 ceph-mon[32610]: pgmap v4639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:00:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7869.537815 secs 2024-07-27T09:00:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:49 smithi077 ceph-mon[37153]: pgmap v4640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:00:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:49 smithi047 ceph-mon[36835]: pgmap v4640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:00:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:49 smithi029 ceph-mon[32610]: pgmap v4640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:00:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:51 smithi077 ceph-mon[37153]: pgmap v4641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:00:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:51 smithi047 ceph-mon[36835]: pgmap v4641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:00:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:51 smithi029 ceph-mon[32610]: pgmap v4641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:00:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7874.537903 secs 2024-07-27T09:00:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7874.537903 secs 2024-07-27T09:00:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7874.537903 secs 2024-07-27T09:00:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:53 smithi077 ceph-mon[37153]: pgmap v4642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:00:54.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:53 smithi029 ceph-mon[32610]: pgmap v4642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:00:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:53 smithi047 ceph-mon[36835]: pgmap v4642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:00:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:55 smithi077 ceph-mon[37153]: pgmap v4643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:00:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:55 smithi047 ceph-mon[36835]: pgmap v4643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:00:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:55 smithi029 ceph-mon[32610]: pgmap v4643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:00:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:57 smithi077 ceph-mon[37153]: pgmap v4644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7879.537981 secs 2024-07-27T09:00:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:57 smithi047 ceph-mon[36835]: pgmap v4644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7879.537981 secs 2024-07-27T09:00:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:57 smithi029 ceph-mon[32610]: pgmap v4644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:00:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7879.537981 secs 2024-07-27T09:00:59.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:00:59.235+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:00:59.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:00:59.235+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:00:59.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:00:59.235+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:00:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:00:59.329+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:00:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:00:59.330+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:00:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:00:59.330+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:01:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:00:59 smithi077 ceph-mon[37153]: pgmap v4645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:00:59 smithi047 ceph-mon[36835]: pgmap v4645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:00:59 smithi029 ceph-mon[32610]: pgmap v4645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:01 smithi077 ceph-mon[37153]: pgmap v4646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:01 smithi047 ceph-mon[36835]: pgmap v4646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:01 smithi029 ceph-mon[32610]: pgmap v4646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7884.538064 secs 2024-07-27T09:01:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7884.538064 secs 2024-07-27T09:01:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7884.538064 secs 2024-07-27T09:01:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:03 smithi077 ceph-mon[37153]: pgmap v4647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:01:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:03 smithi047 ceph-mon[36835]: pgmap v4647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:01:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:03 smithi029 ceph-mon[32610]: pgmap v4647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:01:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:05 smithi077 ceph-mon[37153]: pgmap v4648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:01:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:05 smithi047 ceph-mon[36835]: pgmap v4648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:01:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:05 smithi029 ceph-mon[32610]: pgmap v4648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:01:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:07 smithi077 ceph-mon[37153]: pgmap v4649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7889.538208 secs 2024-07-27T09:01:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:07 smithi047 ceph-mon[36835]: pgmap v4649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7889.538208 secs 2024-07-27T09:01:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:07 smithi029 ceph-mon[32610]: pgmap v4649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7889.538208 secs 2024-07-27T09:01:10.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:09 smithi029 ceph-mon[32610]: pgmap v4650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:09 smithi077 ceph-mon[37153]: pgmap v4650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:09 smithi047 ceph-mon[36835]: pgmap v4650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:11 smithi077 ceph-mon[37153]: pgmap v4651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:11 smithi047 ceph-mon[36835]: pgmap v4651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:11 smithi029 ceph-mon[32610]: pgmap v4651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7894.538293 secs 2024-07-27T09:01:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7894.538293 secs 2024-07-27T09:01:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7894.538293 secs 2024-07-27T09:01:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:13 smithi077 ceph-mon[37153]: pgmap v4652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:01:14.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:13 smithi047 ceph-mon[36835]: pgmap v4652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:01:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:13 smithi029 ceph-mon[32610]: pgmap v4652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:01:14.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:14.235+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:01:14.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:14.235+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:01:14.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:14.235+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:01:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:14.344+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:01:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:14.344+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:01:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:14.344+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:01:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:15 smithi077 ceph-mon[37153]: pgmap v4653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:01:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:15 smithi047 ceph-mon[36835]: pgmap v4653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:01:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:15 smithi029 ceph-mon[32610]: pgmap v4653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:01:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:17 smithi077 ceph-mon[37153]: pgmap v4654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7899.538358 secs 2024-07-27T09:01:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:17 smithi047 ceph-mon[36835]: pgmap v4654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7899.538358 secs 2024-07-27T09:01:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:17 smithi029 ceph-mon[32610]: pgmap v4654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7899.538358 secs 2024-07-27T09:01:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:18 smithi077 ceph-mon[37153]: pgmap v4655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:18 smithi047 ceph-mon[36835]: pgmap v4655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:18 smithi029 ceph-mon[32610]: pgmap v4655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:21 smithi047 ceph-mon[36835]: pgmap v4656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:21 smithi029 ceph-mon[32610]: pgmap v4656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:21 smithi077 ceph-mon[37153]: pgmap v4656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7904.538485 secs 2024-07-27T09:01:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7904.538485 secs 2024-07-27T09:01:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7904.538485 secs 2024-07-27T09:01:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:23 smithi047 ceph-mon[36835]: pgmap v4657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:01:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:23 smithi029 ceph-mon[32610]: pgmap v4657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:01:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:23 smithi077 ceph-mon[37153]: pgmap v4657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:01:25.928 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:25 smithi029 ceph-mon[32610]: pgmap v4658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:01:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:25 smithi047 ceph-mon[36835]: pgmap v4658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:01:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:25 smithi077 ceph-mon[37153]: pgmap v4658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:01:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:27 smithi047 ceph-mon[36835]: pgmap v4659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7909.538585 secs 2024-07-27T09:01:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:27 smithi029 ceph-mon[32610]: pgmap v4659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7909.538585 secs 2024-07-27T09:01:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:27 smithi077 ceph-mon[37153]: pgmap v4659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7909.538585 secs 2024-07-27T09:01:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:29.245+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:01:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:29.245+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:01:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:29.245+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:01:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:29.360+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:01:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:29.360+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:01:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:29.360+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:01:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:29 smithi047 ceph-mon[36835]: pgmap v4660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:29 smithi029 ceph-mon[32610]: pgmap v4660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:29 smithi077 ceph-mon[37153]: pgmap v4660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:31 smithi047 ceph-mon[36835]: pgmap v4661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:31 smithi029 ceph-mon[32610]: pgmap v4661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:31 smithi077 ceph-mon[37153]: pgmap v4661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7914.538715 secs 2024-07-27T09:01:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7914.538715 secs 2024-07-27T09:01:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7914.538715 secs 2024-07-27T09:01:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:33 smithi047 ceph-mon[36835]: pgmap v4662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:01:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:33 smithi029 ceph-mon[32610]: pgmap v4662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:01:34.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:33 smithi077 ceph-mon[37153]: pgmap v4662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:01:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:35 smithi047 ceph-mon[36835]: pgmap v4663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:01:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:35 smithi029 ceph-mon[32610]: pgmap v4663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:01:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:35 smithi077 ceph-mon[37153]: pgmap v4663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:01:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:37 smithi047 ceph-mon[36835]: pgmap v4664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7919.538841 secs 2024-07-27T09:01:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:37 smithi029 ceph-mon[32610]: pgmap v4664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7919.538841 secs 2024-07-27T09:01:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:37 smithi077 ceph-mon[37153]: pgmap v4664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7919.538841 secs 2024-07-27T09:01:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:39 smithi047 ceph-mon[36835]: pgmap v4665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:39 smithi029 ceph-mon[32610]: pgmap v4665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:39 smithi077 ceph-mon[37153]: pgmap v4665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:01:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:01:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:01:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:41 smithi077 ceph-mon[37153]: pgmap v4666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:01:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:01:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:01:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:41 smithi047 ceph-mon[36835]: pgmap v4666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:01:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:01:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:01:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:41 smithi029 ceph-mon[32610]: pgmap v4666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:01:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:01:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:01:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:42 smithi077 ceph-mon[37153]: pgmap v4667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T09:01:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:42 smithi077 ceph-mon[37153]: pgmap v4668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:01:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7924.538928 secs 2024-07-27T09:01:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:42 smithi047 ceph-mon[36835]: pgmap v4667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T09:01:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:42 smithi047 ceph-mon[36835]: pgmap v4668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:01:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7924.538928 secs 2024-07-27T09:01:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:42 smithi029 ceph-mon[32610]: pgmap v4667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T09:01:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:42 smithi029 ceph-mon[32610]: pgmap v4668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:01:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7924.538928 secs 2024-07-27T09:01:43.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:43.829+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:01:43.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:43.829+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:01:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:43.829+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:01:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:43.829+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:01:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:43.829+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:01:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:43.829+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:01:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:43.829+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:01:43.832 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:43.829+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:01:44.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:44.257+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:01:44.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:44.257+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:01:44.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:44.257+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:01:44.376 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:44.375+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:01:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:44.375+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:01:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:44.375+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:01:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:44 smithi077 ceph-mon[37153]: pgmap v4669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:01:45.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:44 smithi047 ceph-mon[36835]: pgmap v4669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:01:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:44 smithi029 ceph-mon[32610]: pgmap v4669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:01:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:46 smithi077 ceph-mon[37153]: pgmap v4670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:01:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:46 smithi047 ceph-mon[36835]: pgmap v4670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:01:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:46 smithi029 ceph-mon[32610]: pgmap v4670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:01:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7929.538999 secs 2024-07-27T09:01:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7929.538999 secs 2024-07-27T09:01:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7929.538999 secs 2024-07-27T09:01:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:48 smithi077 ceph-mon[37153]: pgmap v4671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:01:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:48 smithi047 ceph-mon[36835]: pgmap v4671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:01:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:48 smithi029 ceph-mon[32610]: pgmap v4671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:01:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:50 smithi077 ceph-mon[37153]: pgmap v4672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:01:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:50 smithi047 ceph-mon[36835]: pgmap v4672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:01:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:50 smithi029 ceph-mon[32610]: pgmap v4672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:01:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:52 smithi077 ceph-mon[37153]: pgmap v4673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:01:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7934.539077 secs 2024-07-27T09:01:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:52 smithi047 ceph-mon[36835]: pgmap v4673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:01:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7934.539077 secs 2024-07-27T09:01:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:52 smithi029 ceph-mon[32610]: pgmap v4673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:01:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7934.539077 secs 2024-07-27T09:01:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:54 smithi077 ceph-mon[37153]: pgmap v4674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:54 smithi047 ceph-mon[36835]: pgmap v4674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:54 smithi029 ceph-mon[32610]: pgmap v4674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:01:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:56 smithi077 ceph-mon[37153]: pgmap v4675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:57.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:56 smithi029 ceph-mon[32610]: pgmap v4675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:56 smithi047 ceph-mon[36835]: pgmap v4675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:01:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7939.539143 secs 2024-07-27T09:01:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7939.539143 secs 2024-07-27T09:01:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7939.539143 secs 2024-07-27T09:01:58.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:58.831+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T09:01:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:58.831+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:01:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:58.832+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T09:01:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:58.832+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:01:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:58.832+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:01:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:58.832+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:01:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:58.832+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:01:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:58.832+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:01:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:01:58 smithi077 ceph-mon[37153]: pgmap v4676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:59.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:01:58 smithi029 ceph-mon[32610]: pgmap v4676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:59.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:01:58 smithi047 ceph-mon[36835]: pgmap v4676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:01:59.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:59.273+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:01:59.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:59.273+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:01:59.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:01:59.273+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:01:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:59.382+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:01:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:59.382+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:01:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:01:59.382+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:02:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:00 smithi077 ceph-mon[37153]: pgmap v4677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:00 smithi047 ceph-mon[36835]: pgmap v4677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:00 smithi029 ceph-mon[32610]: pgmap v4677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:02 smithi077 ceph-mon[37153]: pgmap v4678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7944.539273 secs 2024-07-27T09:02:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:02 smithi047 ceph-mon[36835]: pgmap v4678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7944.539273 secs 2024-07-27T09:02:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:02 smithi029 ceph-mon[32610]: pgmap v4678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7944.539273 secs 2024-07-27T09:02:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:04 smithi077 ceph-mon[37153]: pgmap v4679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:04 smithi047 ceph-mon[36835]: pgmap v4679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:04 smithi029 ceph-mon[32610]: pgmap v4679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:06 smithi077 ceph-mon[37153]: pgmap v4680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:06 smithi047 ceph-mon[36835]: pgmap v4680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:06 smithi029 ceph-mon[32610]: pgmap v4680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:08.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7949.539393 secs 2024-07-27T09:02:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7949.539393 secs 2024-07-27T09:02:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7949.539393 secs 2024-07-27T09:02:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:08 smithi077 ceph-mon[37153]: pgmap v4681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:08 smithi047 ceph-mon[36835]: pgmap v4681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:08 smithi029 ceph-mon[32610]: pgmap v4681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:10 smithi077 ceph-mon[37153]: pgmap v4682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:10 smithi047 ceph-mon[36835]: pgmap v4682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:10 smithi029 ceph-mon[32610]: pgmap v4682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:12 smithi077 ceph-mon[37153]: pgmap v4683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7954.539521 secs 2024-07-27T09:02:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:12 smithi047 ceph-mon[36835]: pgmap v4683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7954.539521 secs 2024-07-27T09:02:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:12 smithi029 ceph-mon[32610]: pgmap v4683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7954.539521 secs 2024-07-27T09:02:14.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:02:14.288+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:02:14.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:02:14.288+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:02:14.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:02:14.288+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:02:14.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:02:14.396+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:02:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:02:14.397+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:02:14.399 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:02:14.397+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:02:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:14 smithi077 ceph-mon[37153]: pgmap v4684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:14 smithi047 ceph-mon[36835]: pgmap v4684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:14 smithi029 ceph-mon[32610]: pgmap v4684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:16 smithi077 ceph-mon[37153]: pgmap v4685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:16 smithi047 ceph-mon[36835]: pgmap v4685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:16 smithi029 ceph-mon[32610]: pgmap v4685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7959.539632 secs 2024-07-27T09:02:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7959.539632 secs 2024-07-27T09:02:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7959.539632 secs 2024-07-27T09:02:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:18 smithi077 ceph-mon[37153]: pgmap v4686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:18 smithi047 ceph-mon[36835]: pgmap v4686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:18 smithi029 ceph-mon[32610]: pgmap v4686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:20 smithi077 ceph-mon[37153]: pgmap v4687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:20 smithi047 ceph-mon[36835]: pgmap v4687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:20 smithi029 ceph-mon[32610]: pgmap v4687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:22 smithi077 ceph-mon[37153]: pgmap v4688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7964.539755 secs 2024-07-27T09:02:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:22 smithi047 ceph-mon[36835]: pgmap v4688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7964.539755 secs 2024-07-27T09:02:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:22 smithi029 ceph-mon[32610]: pgmap v4688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7964.539755 secs 2024-07-27T09:02:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:23 smithi077 ceph-mon[37153]: pgmap v4689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:24.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:23 smithi047 ceph-mon[36835]: pgmap v4689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:23 smithi029 ceph-mon[32610]: pgmap v4689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:26 smithi077 ceph-mon[37153]: pgmap v4690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:26 smithi047 ceph-mon[36835]: pgmap v4690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:26 smithi029 ceph-mon[32610]: pgmap v4690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7969.539875 secs 2024-07-27T09:02:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7969.539875 secs 2024-07-27T09:02:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7969.539875 secs 2024-07-27T09:02:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:28 smithi077 ceph-mon[37153]: pgmap v4691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:28 smithi047 ceph-mon[36835]: pgmap v4691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:28 smithi029 ceph-mon[32610]: pgmap v4691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:02:29.289+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:02:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:02:29.289+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:02:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:02:29.289+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:02:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:02:29.412+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:02:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:02:29.412+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:02:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:02:29.412+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:02:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:30 smithi077 ceph-mon[37153]: pgmap v4692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:30 smithi047 ceph-mon[36835]: pgmap v4692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:30 smithi029 ceph-mon[32610]: pgmap v4692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:32 smithi077 ceph-mon[37153]: pgmap v4693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7974.539995 secs 2024-07-27T09:02:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:32 smithi047 ceph-mon[36835]: pgmap v4693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:32.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7974.539995 secs 2024-07-27T09:02:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:32 smithi029 ceph-mon[32610]: pgmap v4693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7974.539995 secs 2024-07-27T09:02:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:34 smithi077 ceph-mon[37153]: pgmap v4694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:34 smithi047 ceph-mon[36835]: pgmap v4694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:34 smithi029 ceph-mon[32610]: pgmap v4694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:36 smithi077 ceph-mon[37153]: pgmap v4695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:36 smithi047 ceph-mon[36835]: pgmap v4695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:36 smithi029 ceph-mon[32610]: pgmap v4695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7979.540122 secs 2024-07-27T09:02:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7979.540122 secs 2024-07-27T09:02:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7979.540122 secs 2024-07-27T09:02:38.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:38 smithi047 ceph-mon[36835]: pgmap v4696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:38 smithi077 ceph-mon[37153]: pgmap v4696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:38 smithi029 ceph-mon[32610]: pgmap v4696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:40 smithi077 ceph-mon[37153]: pgmap v4697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:40 smithi047 ceph-mon[36835]: pgmap v4697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:40 smithi029 ceph-mon[32610]: pgmap v4697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:02:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:42 smithi077 ceph-mon[37153]: pgmap v4698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:02:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:02:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:02:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:42 smithi077 ceph-mon[37153]: pgmap v4699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:02:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:42 smithi077 ceph-mon[37153]: pgmap v4700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:02:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:02:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7984.540206 secs 2024-07-27T09:02:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:42 smithi047 ceph-mon[36835]: pgmap v4698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:02:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:02:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:02:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:42 smithi047 ceph-mon[36835]: pgmap v4699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:02:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:42 smithi047 ceph-mon[36835]: pgmap v4700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:02:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:02:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7984.540206 secs 2024-07-27T09:02:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:42 smithi029 ceph-mon[32610]: pgmap v4698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:02:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:02:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:02:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:02:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:42 smithi029 ceph-mon[32610]: pgmap v4699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:02:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:42 smithi029 ceph-mon[32610]: pgmap v4700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:02:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:02:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7984.540206 secs 2024-07-27T09:02:44.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:02:44.302+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:02:44.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:02:44.302+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:02:44.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:02:44.302+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:02:44.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:02:44.422+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:02:44.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:02:44.422+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:02:44.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:02:44.422+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:02:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:44 smithi077 ceph-mon[37153]: pgmap v4701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:02:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:44 smithi047 ceph-mon[36835]: pgmap v4701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:02:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:44 smithi029 ceph-mon[32610]: pgmap v4701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:02:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:46 smithi077 ceph-mon[37153]: pgmap v4702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:02:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:46 smithi047 ceph-mon[36835]: pgmap v4702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:02:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:46 smithi029 ceph-mon[32610]: pgmap v4702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:02:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7989.540344 secs 2024-07-27T09:02:48.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7989.540344 secs 2024-07-27T09:02:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7989.540344 secs 2024-07-27T09:02:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:48 smithi077 ceph-mon[37153]: pgmap v4703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:02:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:48 smithi047 ceph-mon[36835]: pgmap v4703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:02:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:48 smithi029 ceph-mon[32610]: pgmap v4703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:02:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:50 smithi077 ceph-mon[37153]: pgmap v4704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:02:51.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:50 smithi047 ceph-mon[36835]: pgmap v4704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:02:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:50 smithi029 ceph-mon[32610]: pgmap v4704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:02:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:52 smithi077 ceph-mon[37153]: pgmap v4705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:02:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7994.540468 secs 2024-07-27T09:02:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:52 smithi047 ceph-mon[36835]: pgmap v4705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:02:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7994.540468 secs 2024-07-27T09:02:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:52 smithi029 ceph-mon[32610]: pgmap v4705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:02:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7994.540468 secs 2024-07-27T09:02:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:54 smithi077 ceph-mon[37153]: pgmap v4706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:55.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:54 smithi047 ceph-mon[36835]: pgmap v4706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:54 smithi029 ceph-mon[32610]: pgmap v4706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:02:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:56 smithi077 ceph-mon[37153]: pgmap v4707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:56 smithi047 ceph-mon[36835]: pgmap v4707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:56 smithi029 ceph-mon[32610]: pgmap v4707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:02:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 7999.540587 secs 2024-07-27T09:02:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 7999.540587 secs 2024-07-27T09:02:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 7999.540587 secs 2024-07-27T09:02:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:02:58 smithi077 ceph-mon[37153]: pgmap v4708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:02:59.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:02:58 smithi029 ceph-mon[32610]: pgmap v4708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:02:59.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:02:58 smithi047 ceph-mon[36835]: pgmap v4708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:02:59.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:02:59.318+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:02:59.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:02:59.318+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:02:59.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:02:59.318+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:02:59.424 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:02:59.423+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:02:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:02:59.423+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:02:59.425 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:02:59.423+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:03:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:00 smithi077 ceph-mon[37153]: pgmap v4709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:00 smithi047 ceph-mon[36835]: pgmap v4709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:00 smithi029 ceph-mon[32610]: pgmap v4709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:02 smithi077 ceph-mon[37153]: pgmap v4710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8004.540713 secs 2024-07-27T09:03:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:02 smithi047 ceph-mon[36835]: pgmap v4710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8004.540713 secs 2024-07-27T09:03:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:02 smithi029 ceph-mon[32610]: pgmap v4710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8004.540713 secs 2024-07-27T09:03:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:04 smithi077 ceph-mon[37153]: pgmap v4711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:03:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:04 smithi047 ceph-mon[36835]: pgmap v4711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:03:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:04 smithi029 ceph-mon[32610]: pgmap v4711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:03:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:06 smithi077 ceph-mon[37153]: pgmap v4712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:03:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:06 smithi047 ceph-mon[36835]: pgmap v4712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:03:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:06 smithi029 ceph-mon[32610]: pgmap v4712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:03:08.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8009.540837 secs 2024-07-27T09:03:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8009.540837 secs 2024-07-27T09:03:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8009.540837 secs 2024-07-27T09:03:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:08 smithi077 ceph-mon[37153]: pgmap v4713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:03:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:08 smithi047 ceph-mon[36835]: pgmap v4713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:03:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:08 smithi029 ceph-mon[32610]: pgmap v4713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:03:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:10 smithi077 ceph-mon[37153]: pgmap v4714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:11.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:10 smithi047 ceph-mon[36835]: pgmap v4714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:10 smithi029 ceph-mon[32610]: pgmap v4714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:12 smithi077 ceph-mon[37153]: pgmap v4715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8014.540901 secs 2024-07-27T09:03:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:12 smithi047 ceph-mon[36835]: pgmap v4715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8014.540901 secs 2024-07-27T09:03:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:12 smithi029 ceph-mon[32610]: pgmap v4715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:13.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8014.540901 secs 2024-07-27T09:03:14.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:03:14.325+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:03:14.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:03:14.325+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:03:14.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:03:14.325+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:03:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:03:14.438+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:03:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:03:14.438+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:03:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:03:14.438+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:03:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:14 smithi077 ceph-mon[37153]: pgmap v4716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:03:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:14 smithi047 ceph-mon[36835]: pgmap v4716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:03:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:14 smithi029 ceph-mon[32610]: pgmap v4716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:03:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:16 smithi077 ceph-mon[37153]: pgmap v4717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:03:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:16 smithi047 ceph-mon[36835]: pgmap v4717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:03:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:16 smithi029 ceph-mon[32610]: pgmap v4717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:03:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8019.541027 secs 2024-07-27T09:03:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8019.541027 secs 2024-07-27T09:03:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8019.541027 secs 2024-07-27T09:03:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:18 smithi077 ceph-mon[37153]: pgmap v4718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:03:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:18 smithi047 ceph-mon[36835]: pgmap v4718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:03:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:18 smithi029 ceph-mon[32610]: pgmap v4718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:03:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:20 smithi077 ceph-mon[37153]: pgmap v4719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:20 smithi047 ceph-mon[36835]: pgmap v4719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:20 smithi029 ceph-mon[32610]: pgmap v4719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:22 smithi077 ceph-mon[37153]: pgmap v4720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8024.541111 secs 2024-07-27T09:03:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:22 smithi047 ceph-mon[36835]: pgmap v4720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8024.541111 secs 2024-07-27T09:03:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:22 smithi029 ceph-mon[32610]: pgmap v4720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8024.541111 secs 2024-07-27T09:03:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:24 smithi077 ceph-mon[37153]: pgmap v4721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:03:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:24 smithi047 ceph-mon[36835]: pgmap v4721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:03:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:24 smithi029 ceph-mon[32610]: pgmap v4721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:03:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:25 smithi077 ceph-mon[37153]: pgmap v4722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:03:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:25 smithi047 ceph-mon[36835]: pgmap v4722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:03:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:25 smithi029 ceph-mon[32610]: pgmap v4722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:03:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:26 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8029.541215 secs 2024-07-27T09:03:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:26 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8029.541215 secs 2024-07-27T09:03:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:26 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8029.541215 secs 2024-07-27T09:03:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:27 smithi077 ceph-mon[37153]: pgmap v4723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:03:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:27 smithi047 ceph-mon[36835]: pgmap v4723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:03:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:27 smithi029 ceph-mon[32610]: pgmap v4723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:03:29.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:03:29.341+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:03:29.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:03:29.341+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:03:29.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:03:29.341+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:03:29.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:03:29.454+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:03:29.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:03:29.454+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:03:29.455 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:03:29.454+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:03:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:30 smithi077 ceph-mon[37153]: pgmap v4724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:30 smithi047 ceph-mon[36835]: pgmap v4724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:30 smithi029 ceph-mon[32610]: pgmap v4724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:32 smithi077 ceph-mon[37153]: pgmap v4725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8034.541298 secs 2024-07-27T09:03:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:32 smithi047 ceph-mon[36835]: pgmap v4725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8034.541298 secs 2024-07-27T09:03:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:32 smithi029 ceph-mon[32610]: pgmap v4725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8034.541298 secs 2024-07-27T09:03:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:34 smithi077 ceph-mon[37153]: pgmap v4726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:03:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:34 smithi047 ceph-mon[36835]: pgmap v4726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:03:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:34 smithi029 ceph-mon[32610]: pgmap v4726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:03:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:36 smithi077 ceph-mon[37153]: pgmap v4727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:03:37.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:36 smithi047 ceph-mon[36835]: pgmap v4727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:03:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:36 smithi029 ceph-mon[32610]: pgmap v4727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:03:38.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8039.541398 secs 2024-07-27T09:03:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8039.541398 secs 2024-07-27T09:03:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8039.541398 secs 2024-07-27T09:03:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:38 smithi077 ceph-mon[37153]: pgmap v4728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:03:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:38 smithi047 ceph-mon[36835]: pgmap v4728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:03:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:38 smithi029 ceph-mon[32610]: pgmap v4728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:03:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:40 smithi077 ceph-mon[37153]: pgmap v4729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:40 smithi047 ceph-mon[36835]: pgmap v4729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:40 smithi029 ceph-mon[32610]: pgmap v4729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:03:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:42 smithi077 ceph-mon[37153]: pgmap v4730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:03:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8044.541479 secs 2024-07-27T09:03:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:03:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:03:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:03:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:03:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:03:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:03:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:03:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:42 smithi047 ceph-mon[36835]: pgmap v4730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:03:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8044.541479 secs 2024-07-27T09:03:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:03:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:03:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:03:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:03:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:03:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:03:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:03:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:42 smithi029 ceph-mon[32610]: pgmap v4730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:03:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8044.541479 secs 2024-07-27T09:03:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:03:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:03:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:03:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:03:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:03:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:03:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:03:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:43 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:03:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:43 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:03:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:43 smithi077 ceph-mon[37153]: pgmap v4731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:03:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:43 smithi077 ceph-mon[37153]: pgmap v4732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:03:44.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:43 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:03:44.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:43 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:03:44.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:43 smithi047 ceph-mon[36835]: pgmap v4731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:03:44.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:43 smithi047 ceph-mon[36835]: pgmap v4732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:03:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:43 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:03:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:43 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:03:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:43 smithi029 ceph-mon[32610]: pgmap v4731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:03:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:43 smithi029 ceph-mon[32610]: pgmap v4732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:03:44.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:03:44.351+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:03:44.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:03:44.351+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:03:44.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:03:44.351+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:03:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:03:44.469+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:03:44.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:03:44.469+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:03:44.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:03:44.469+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:03:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:45 smithi077 ceph-mon[37153]: pgmap v4733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:03:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:45 smithi047 ceph-mon[36835]: pgmap v4733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:03:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:45 smithi029 ceph-mon[32610]: pgmap v4733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:03:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:47 smithi077 ceph-mon[37153]: pgmap v4734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:03:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8049.541618 secs 2024-07-27T09:03:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:47 smithi047 ceph-mon[36835]: pgmap v4734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:03:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8049.541618 secs 2024-07-27T09:03:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:47 smithi029 ceph-mon[32610]: pgmap v4734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:03:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8049.541618 secs 2024-07-27T09:03:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:49 smithi077 ceph-mon[37153]: pgmap v4735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:03:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:49 smithi047 ceph-mon[36835]: pgmap v4735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:03:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:49 smithi029 ceph-mon[32610]: pgmap v4735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:03:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:51 smithi077 ceph-mon[37153]: pgmap v4736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:03:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:51 smithi047 ceph-mon[36835]: pgmap v4736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:03:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:51 smithi029 ceph-mon[32610]: pgmap v4736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:03:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8054.541759 secs 2024-07-27T09:03:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8054.541759 secs 2024-07-27T09:03:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8054.541759 secs 2024-07-27T09:03:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:53 smithi077 ceph-mon[37153]: pgmap v4737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:03:54.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:53 smithi047 ceph-mon[36835]: pgmap v4737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:03:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:53 smithi029 ceph-mon[32610]: pgmap v4737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:03:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:55 smithi077 ceph-mon[37153]: pgmap v4738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:03:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:55 smithi047 ceph-mon[36835]: pgmap v4738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:03:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:55 smithi029 ceph-mon[32610]: pgmap v4738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:03:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:57 smithi077 ceph-mon[37153]: pgmap v4739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8059.541883 secs 2024-07-27T09:03:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:57 smithi047 ceph-mon[36835]: pgmap v4739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8059.541883 secs 2024-07-27T09:03:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:57 smithi029 ceph-mon[32610]: pgmap v4739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:03:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8059.541883 secs 2024-07-27T09:03:59.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:03:59.367+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:03:59.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:03:59.367+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:03:59.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:03:59.367+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:03:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:03:59.483+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:03:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:03:59.483+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:03:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:03:59.483+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:04:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:03:59 smithi077 ceph-mon[37153]: pgmap v4740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:03:59 smithi047 ceph-mon[36835]: pgmap v4740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:03:59 smithi029 ceph-mon[32610]: pgmap v4740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:01 smithi077 ceph-mon[37153]: pgmap v4741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:01 smithi047 ceph-mon[36835]: pgmap v4741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:01 smithi029 ceph-mon[32610]: pgmap v4741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8064.542006 secs 2024-07-27T09:04:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8064.542006 secs 2024-07-27T09:04:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8064.542006 secs 2024-07-27T09:04:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:03 smithi077 ceph-mon[37153]: pgmap v4742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:03 smithi047 ceph-mon[36835]: pgmap v4742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:03 smithi029 ceph-mon[32610]: pgmap v4742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:05 smithi077 ceph-mon[37153]: pgmap v4743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:04:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:05 smithi047 ceph-mon[36835]: pgmap v4743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:04:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:05 smithi029 ceph-mon[32610]: pgmap v4743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:04:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:06 smithi047 ceph-mon[36835]: pgmap v4744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:04:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:06 smithi029 ceph-mon[32610]: pgmap v4744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:04:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:06 smithi077 ceph-mon[37153]: pgmap v4744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:04:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8069.542103 secs 2024-07-27T09:04:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8069.542103 secs 2024-07-27T09:04:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8069.542103 secs 2024-07-27T09:04:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:08 smithi047 ceph-mon[36835]: pgmap v4745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:08 smithi029 ceph-mon[32610]: pgmap v4745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:08 smithi077 ceph-mon[37153]: pgmap v4745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:11 smithi077 ceph-mon[37153]: pgmap v4746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:04:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:11 smithi047 ceph-mon[36835]: pgmap v4746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:04:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:11 smithi029 ceph-mon[32610]: pgmap v4746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:04:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8074.542225 secs 2024-07-27T09:04:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8074.542225 secs 2024-07-27T09:04:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8074.542225 secs 2024-07-27T09:04:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:13 smithi077 ceph-mon[37153]: pgmap v4747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:13 smithi047 ceph-mon[36835]: pgmap v4747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:13 smithi029 ceph-mon[32610]: pgmap v4747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:14.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:14.381+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:04:14.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:14.382+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:04:14.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:14.382+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:04:14.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:14.494+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:04:14.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:14.494+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:04:14.496 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:14.494+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:04:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:15 smithi077 ceph-mon[37153]: pgmap v4748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:04:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:15 smithi047 ceph-mon[36835]: pgmap v4748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:04:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:15 smithi029 ceph-mon[32610]: pgmap v4748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:04:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:17 smithi077 ceph-mon[37153]: pgmap v4749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:04:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8079.542347 secs 2024-07-27T09:04:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:17 smithi047 ceph-mon[36835]: pgmap v4749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:04:17.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8079.542347 secs 2024-07-27T09:04:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:17 smithi029 ceph-mon[32610]: pgmap v4749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:04:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8079.542347 secs 2024-07-27T09:04:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:19 smithi077 ceph-mon[37153]: pgmap v4750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:19 smithi047 ceph-mon[36835]: pgmap v4750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:19 smithi029 ceph-mon[32610]: pgmap v4750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:21 smithi077 ceph-mon[37153]: pgmap v4751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:21 smithi047 ceph-mon[36835]: pgmap v4751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:21 smithi029 ceph-mon[32610]: pgmap v4751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8084.542446 secs 2024-07-27T09:04:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8084.542446 secs 2024-07-27T09:04:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8084.542446 secs 2024-07-27T09:04:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:23 smithi047 ceph-mon[36835]: pgmap v4752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:23 smithi029 ceph-mon[32610]: pgmap v4752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:23 smithi077 ceph-mon[37153]: pgmap v4752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:25 smithi047 ceph-mon[36835]: pgmap v4753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:04:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:25 smithi029 ceph-mon[32610]: pgmap v4753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:04:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:25 smithi077 ceph-mon[37153]: pgmap v4753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:04:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:27 smithi047 ceph-mon[36835]: pgmap v4754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:04:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8089.542573 secs 2024-07-27T09:04:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:27 smithi029 ceph-mon[32610]: pgmap v4754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:04:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8089.542573 secs 2024-07-27T09:04:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:27 smithi077 ceph-mon[37153]: pgmap v4754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:04:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8089.542573 secs 2024-07-27T09:04:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:29.397+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:04:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:29.397+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:04:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:29.397+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:04:29.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:29.495+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:04:29.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:29.495+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:04:29.497 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:29.495+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:04:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:29 smithi047 ceph-mon[36835]: pgmap v4755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:29 smithi029 ceph-mon[32610]: pgmap v4755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:29 smithi077 ceph-mon[37153]: pgmap v4755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:31 smithi047 ceph-mon[36835]: pgmap v4756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:31 smithi029 ceph-mon[32610]: pgmap v4756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:31 smithi077 ceph-mon[37153]: pgmap v4756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8094.542711 secs 2024-07-27T09:04:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8094.542711 secs 2024-07-27T09:04:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8094.542711 secs 2024-07-27T09:04:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:33 smithi047 ceph-mon[36835]: pgmap v4757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:33 smithi029 ceph-mon[32610]: pgmap v4757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:33 smithi077 ceph-mon[37153]: pgmap v4757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:35.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:35 smithi077 ceph-mon[37153]: pgmap v4758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:04:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:35 smithi047 ceph-mon[36835]: pgmap v4758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:04:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:35 smithi029 ceph-mon[32610]: pgmap v4758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:04:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:37 smithi047 ceph-mon[36835]: pgmap v4759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:04:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8099.542830 secs 2024-07-27T09:04:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:37 smithi029 ceph-mon[32610]: pgmap v4759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:04:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8099.542830 secs 2024-07-27T09:04:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:37 smithi077 ceph-mon[37153]: pgmap v4759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:04:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8099.542830 secs 2024-07-27T09:04:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:39 smithi047 ceph-mon[36835]: pgmap v4760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:39 smithi029 ceph-mon[32610]: pgmap v4760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:39 smithi077 ceph-mon[37153]: pgmap v4760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:04:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:41 smithi047 ceph-mon[36835]: pgmap v4761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:41 smithi029 ceph-mon[32610]: pgmap v4761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:41 smithi077 ceph-mon[37153]: pgmap v4761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:42.643+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-27T09:04:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:42.643+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T09:04:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:42.643+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4bc0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f733c019550 tx=0x7f733c07a000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T09:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:42.643+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4bc0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f733c019550 tx=0x7f733c07a000 comp rx=0 tx=0).stop 2024-07-27T09:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:42.643+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4bc0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T09:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:42.643+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4bc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T09:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:42.643+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4bc0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:42.644+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:42.644+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:04:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:42.644+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f733c005890 tx=0x7f733c08a000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:04:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8104.542953 secs 2024-07-27T09:04:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:04:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8104.542953 secs 2024-07-27T09:04:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:04:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8104.542953 secs 2024-07-27T09:04:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:04:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: pgmap v4762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:04:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:04:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:04:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:04:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:04:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:04:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: pgmap v4763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:04:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: pgmap v4764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:04:43.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:04:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: pgmap v4762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:04:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:04:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:04:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:04:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:04:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:04:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: pgmap v4763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:04:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: pgmap v4764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:04:43.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:04:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:43.855+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:04:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:43.855+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:04:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:43.855+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:04:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:43.855+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:04:43.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: pgmap v4762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:04:43.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:04:43.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:04:43.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:04:43.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:04:43.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:04:43.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:04:43.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:04:43.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: pgmap v4763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:04:43.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: pgmap v4764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:04:43.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:04:44.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:44.404+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:04:44.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:44.404+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:04:44.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:44.404+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:04:44.512 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:44.511+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:04:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:44.511+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:04:44.513 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:44.511+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:04:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:44 smithi047 ceph-mon[36835]: pgmap v4765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:04:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:44 smithi029 ceph-mon[32610]: pgmap v4765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:04:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:44 smithi077 ceph-mon[37153]: pgmap v4765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:04:47.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:46 smithi047 ceph-mon[36835]: pgmap v4766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:04:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:46 smithi029 ceph-mon[32610]: pgmap v4766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:04:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:46 smithi077 ceph-mon[37153]: pgmap v4766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:04:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8109.543051 secs 2024-07-27T09:04:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8109.543051 secs 2024-07-27T09:04:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8109.543051 secs 2024-07-27T09:04:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:48 smithi047 ceph-mon[36835]: pgmap v4767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:04:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:48 smithi029 ceph-mon[32610]: pgmap v4767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:04:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:48 smithi077 ceph-mon[37153]: pgmap v4767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:04:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:50 smithi047 ceph-mon[36835]: pgmap v4768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:04:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:50 smithi029 ceph-mon[32610]: pgmap v4768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:04:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:50 smithi077 ceph-mon[37153]: pgmap v4768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:04:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:51 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8114.543135 secs 2024-07-27T09:04:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:51 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8114.543135 secs 2024-07-27T09:04:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:51 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8114.543135 secs 2024-07-27T09:04:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:52 smithi077 ceph-mon[37153]: pgmap v4769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:04:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:52 smithi047 ceph-mon[36835]: pgmap v4769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:04:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:52 smithi029 ceph-mon[32610]: pgmap v4769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:04:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:54 smithi047 ceph-mon[36835]: pgmap v4770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:54 smithi029 ceph-mon[32610]: pgmap v4770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:54 smithi077 ceph-mon[37153]: pgmap v4770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:04:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:56 smithi047 ceph-mon[36835]: pgmap v4771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:04:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:56 smithi029 ceph-mon[32610]: pgmap v4771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:04:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:56 smithi077 ceph-mon[37153]: pgmap v4771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:04:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:57.660+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T09:04:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:57.660+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T09:04:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:57.660+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b49b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7eff40004480 tx=0x7eff40041000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T09:04:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:57.660+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b49b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7eff40004480 tx=0x7eff40041000 comp rx=0 tx=0).stop 2024-07-27T09:04:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:57.660+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b49b0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T09:04:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:57.660+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b49b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T09:04:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:57.660+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b49b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:04:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:57.660+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:04:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:57.660+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:04:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:57.661+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7eff4001c6a0 tx=0x7eff4002b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:04:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8119.543274 secs 2024-07-27T09:04:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8119.543274 secs 2024-07-27T09:04:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8119.543274 secs 2024-07-27T09:04:58.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:58.858+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:04:58.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:58.858+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:04:58.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:58.858+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:04:58.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:58.858+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:04:59.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:04:58 smithi029 ceph-mon[32610]: pgmap v4772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:04:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:04:58 smithi047 ceph-mon[36835]: pgmap v4772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:04:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:04:58 smithi077 ceph-mon[37153]: pgmap v4772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:04:59.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:59.406+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:04:59.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:59.406+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:04:59.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:04:59.406+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:04:59.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:59.527+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:04:59.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:59.527+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:04:59.528 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:04:59.527+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:05:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:00 smithi047 ceph-mon[36835]: pgmap v4773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:05:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:00 smithi029 ceph-mon[32610]: pgmap v4773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:05:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:00 smithi077 ceph-mon[37153]: pgmap v4773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:05:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:01 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8124.543396 secs 2024-07-27T09:05:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:01 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8124.543396 secs 2024-07-27T09:05:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:01 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8124.543396 secs 2024-07-27T09:05:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:02 smithi077 ceph-mon[37153]: pgmap v4774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:05:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:02 smithi047 ceph-mon[36835]: pgmap v4774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:05:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:02 smithi029 ceph-mon[32610]: pgmap v4774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:05:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:04 smithi047 ceph-mon[36835]: pgmap v4775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:05:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:04 smithi029 ceph-mon[32610]: pgmap v4775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:05:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:04 smithi077 ceph-mon[37153]: pgmap v4775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:05:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:06 smithi047 ceph-mon[36835]: pgmap v4776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:05:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:06 smithi029 ceph-mon[32610]: pgmap v4776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:05:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:06 smithi077 ceph-mon[37153]: pgmap v4776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:05:08.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8129.543514 secs 2024-07-27T09:05:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8129.543514 secs 2024-07-27T09:05:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8129.543514 secs 2024-07-27T09:05:09.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:08 smithi047 ceph-mon[36835]: pgmap v4777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:05:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:08 smithi029 ceph-mon[32610]: pgmap v4777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:05:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:08 smithi077 ceph-mon[37153]: pgmap v4777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:05:11.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:10 smithi047 ceph-mon[36835]: pgmap v4778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:10 smithi029 ceph-mon[32610]: pgmap v4778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:10 smithi077 ceph-mon[37153]: pgmap v4778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:11 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8134.543636 secs 2024-07-27T09:05:12.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:11 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8134.543636 secs 2024-07-27T09:05:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:11 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8134.543636 secs 2024-07-27T09:05:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:12 smithi077 ceph-mon[37153]: pgmap v4779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:05:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:12 smithi047 ceph-mon[36835]: pgmap v4779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:05:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:12 smithi029 ceph-mon[32610]: pgmap v4779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:05:14.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:05:14.418+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:05:14.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:05:14.418+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:05:14.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:05:14.418+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:05:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:05:14.541+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:05:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:05:14.541+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:05:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:05:14.541+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:05:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:14 smithi047 ceph-mon[36835]: pgmap v4780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:14 smithi029 ceph-mon[32610]: pgmap v4780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:15.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:14 smithi077 ceph-mon[37153]: pgmap v4780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:16 smithi047 ceph-mon[36835]: pgmap v4781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:05:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:16 smithi029 ceph-mon[32610]: pgmap v4781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:05:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:16 smithi077 ceph-mon[37153]: pgmap v4781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:05:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8139.543754 secs 2024-07-27T09:05:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8139.543754 secs 2024-07-27T09:05:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8139.543754 secs 2024-07-27T09:05:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:18 smithi047 ceph-mon[36835]: pgmap v4782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:05:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:18 smithi029 ceph-mon[32610]: pgmap v4782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:05:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:18 smithi077 ceph-mon[37153]: pgmap v4782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:05:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:20 smithi047 ceph-mon[36835]: pgmap v4783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:20 smithi029 ceph-mon[32610]: pgmap v4783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:20 smithi077 ceph-mon[37153]: pgmap v4783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:21 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8144.543866 secs 2024-07-27T09:05:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:21 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8144.543866 secs 2024-07-27T09:05:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:21 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8144.543866 secs 2024-07-27T09:05:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:22 smithi077 ceph-mon[37153]: pgmap v4784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:05:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:22 smithi047 ceph-mon[36835]: pgmap v4784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:05:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:22 smithi029 ceph-mon[32610]: pgmap v4784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:05:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:24 smithi047 ceph-mon[36835]: pgmap v4785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:24 smithi029 ceph-mon[32610]: pgmap v4785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:24 smithi077 ceph-mon[37153]: pgmap v4785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:26 smithi047 ceph-mon[36835]: pgmap v4786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:05:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:26 smithi029 ceph-mon[32610]: pgmap v4786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:05:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:26 smithi077 ceph-mon[37153]: pgmap v4786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:05:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8149.543948 secs 2024-07-27T09:05:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8149.543948 secs 2024-07-27T09:05:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8149.543948 secs 2024-07-27T09:05:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:28 smithi047 ceph-mon[36835]: pgmap v4787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:05:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:28 smithi029 ceph-mon[32610]: pgmap v4787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:05:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:28 smithi077 ceph-mon[37153]: pgmap v4787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:05:29.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:05:29.421+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:05:29.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:05:29.421+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:05:29.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:05:29.421+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:05:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:05:29.556+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:05:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:05:29.557+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:05:29.559 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:05:29.557+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:05:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:30 smithi047 ceph-mon[36835]: pgmap v4788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:30 smithi029 ceph-mon[32610]: pgmap v4788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:30 smithi077 ceph-mon[37153]: pgmap v4788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:31 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8154.544070 secs 2024-07-27T09:05:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:31 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8154.544070 secs 2024-07-27T09:05:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:31 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8154.544070 secs 2024-07-27T09:05:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:33 smithi077 ceph-mon[37153]: pgmap v4789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:05:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:33 smithi047 ceph-mon[36835]: pgmap v4789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:05:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:33 smithi029 ceph-mon[32610]: pgmap v4789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:05:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:34 smithi047 ceph-mon[36835]: pgmap v4790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:34 smithi029 ceph-mon[32610]: pgmap v4790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:34 smithi077 ceph-mon[37153]: pgmap v4790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:36 smithi047 ceph-mon[36835]: pgmap v4791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:05:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:36 smithi029 ceph-mon[32610]: pgmap v4791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:05:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:36 smithi077 ceph-mon[37153]: pgmap v4791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:05:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8159.544188 secs 2024-07-27T09:05:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8159.544188 secs 2024-07-27T09:05:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8159.544188 secs 2024-07-27T09:05:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:38 smithi047 ceph-mon[36835]: pgmap v4792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:05:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:38 smithi029 ceph-mon[32610]: pgmap v4792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:05:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:38 smithi077 ceph-mon[37153]: pgmap v4792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:05:41.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:40 smithi047 ceph-mon[36835]: pgmap v4793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:40 smithi029 ceph-mon[32610]: pgmap v4793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:40 smithi077 ceph-mon[37153]: pgmap v4793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:41 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8164.544340 secs 2024-07-27T09:05:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:41 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8164.544340 secs 2024-07-27T09:05:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:41 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8164.544340 secs 2024-07-27T09:05:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:43 smithi077 ceph-mon[37153]: pgmap v4794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:05:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:05:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:43 smithi047 ceph-mon[36835]: pgmap v4794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:05:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:05:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:43 smithi029 ceph-mon[32610]: pgmap v4794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:05:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:05:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:05:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:05:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:44 smithi077 ceph-mon[37153]: pgmap v4795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:05:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:44 smithi077 ceph-mon[37153]: pgmap v4796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:05:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:05:44.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:05:44.430+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:05:44.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:05:44.430+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:05:44.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:05:44.430+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:05:44.433 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:05:44.433 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:05:44.433 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:44 smithi029 ceph-mon[32610]: pgmap v4795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:05:44.433 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:44 smithi029 ceph-mon[32610]: pgmap v4796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:05:44.433 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:05:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:05:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:05:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:44 smithi047 ceph-mon[36835]: pgmap v4795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:05:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:44 smithi047 ceph-mon[36835]: pgmap v4796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:05:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:05:44.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:05:44.572+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:05:44.574 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:05:44.572+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:05:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:05:44.572+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:05:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:46 smithi029 ceph-mon[32610]: pgmap v4797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:05:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:46 smithi077 ceph-mon[37153]: pgmap v4797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:05:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:46 smithi047 ceph-mon[36835]: pgmap v4797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:05:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8169.544478 secs 2024-07-27T09:05:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8169.544478 secs 2024-07-27T09:05:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8169.544478 secs 2024-07-27T09:05:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:48 smithi077 ceph-mon[37153]: pgmap v4798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:05:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:48 smithi047 ceph-mon[36835]: pgmap v4798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:05:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:48 smithi029 ceph-mon[32610]: pgmap v4798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:05:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:50 smithi077 ceph-mon[37153]: pgmap v4799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:05:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:50 smithi047 ceph-mon[36835]: pgmap v4799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:05:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:50 smithi029 ceph-mon[32610]: pgmap v4799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:05:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:52 smithi077 ceph-mon[37153]: pgmap v4800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:05:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8174.544599 secs 2024-07-27T09:05:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:52 smithi047 ceph-mon[36835]: pgmap v4800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:05:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8174.544599 secs 2024-07-27T09:05:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:52 smithi029 ceph-mon[32610]: pgmap v4800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:05:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8174.544599 secs 2024-07-27T09:05:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:54 smithi077 ceph-mon[37153]: pgmap v4801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:05:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:54 smithi047 ceph-mon[36835]: pgmap v4801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:05:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:54 smithi029 ceph-mon[32610]: pgmap v4801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:05:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:56 smithi077 ceph-mon[37153]: pgmap v4802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:56.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:56 smithi047 ceph-mon[36835]: pgmap v4802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:56 smithi029 ceph-mon[32610]: pgmap v4802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:05:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8179.544740 secs 2024-07-27T09:05:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8179.544740 secs 2024-07-27T09:05:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8179.544740 secs 2024-07-27T09:05:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:05:58 smithi077 ceph-mon[37153]: pgmap v4803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:05:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:05:58 smithi047 ceph-mon[36835]: pgmap v4803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:05:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:05:58 smithi029 ceph-mon[32610]: pgmap v4803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:05:59.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:05:59.446+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:05:59.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:05:59.446+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:05:59.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:05:59.446+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:05:59.589 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:05:59.588+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:05:59.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:05:59.588+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:05:59.590 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:05:59.588+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:06:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:00 smithi077 ceph-mon[37153]: pgmap v4804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:06:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:00 smithi047 ceph-mon[36835]: pgmap v4804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:06:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:00 smithi029 ceph-mon[32610]: pgmap v4804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:06:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:02 smithi077 ceph-mon[37153]: pgmap v4805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:06:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8184.544862 secs 2024-07-27T09:06:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:02 smithi047 ceph-mon[36835]: pgmap v4805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:06:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8184.544862 secs 2024-07-27T09:06:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:02 smithi029 ceph-mon[32610]: pgmap v4805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:06:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8184.544862 secs 2024-07-27T09:06:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:04 smithi077 ceph-mon[37153]: pgmap v4806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:06:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:04 smithi047 ceph-mon[36835]: pgmap v4806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:06:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:04 smithi029 ceph-mon[32610]: pgmap v4806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:06:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:06 smithi077 ceph-mon[37153]: pgmap v4807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:06:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:06 smithi047 ceph-mon[36835]: pgmap v4807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:06:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:06 smithi029 ceph-mon[32610]: pgmap v4807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:06:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8189.544943 secs 2024-07-27T09:06:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8189.544943 secs 2024-07-27T09:06:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8189.544943 secs 2024-07-27T09:06:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:08 smithi077 ceph-mon[37153]: pgmap v4808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:06:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:08 smithi047 ceph-mon[36835]: pgmap v4808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:06:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:08 smithi029 ceph-mon[32610]: pgmap v4808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:06:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:10 smithi077 ceph-mon[37153]: pgmap v4809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:06:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:10 smithi047 ceph-mon[36835]: pgmap v4809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:06:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:10 smithi029 ceph-mon[32610]: pgmap v4809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:06:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:12 smithi077 ceph-mon[37153]: pgmap v4810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:06:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8194.545063 secs 2024-07-27T09:06:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:12 smithi047 ceph-mon[36835]: pgmap v4810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:06:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8194.545063 secs 2024-07-27T09:06:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:12 smithi029 ceph-mon[32610]: pgmap v4810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:06:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8194.545063 secs 2024-07-27T09:06:14.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:06:14.461+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:06:14.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:06:14.461+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:06:14.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:06:14.461+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:06:14.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:06:14.595+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:06:14.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:06:14.595+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:06:14.597 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:06:14.595+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:06:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:14 smithi077 ceph-mon[37153]: pgmap v4811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:06:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:14 smithi047 ceph-mon[36835]: pgmap v4811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:06:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:14 smithi029 ceph-mon[32610]: pgmap v4811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:06:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:16 smithi077 ceph-mon[37153]: pgmap v4812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:06:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:16 smithi047 ceph-mon[36835]: pgmap v4812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:06:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:16 smithi029 ceph-mon[32610]: pgmap v4812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:06:17.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8199.545185 secs 2024-07-27T09:06:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8199.545185 secs 2024-07-27T09:06:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8199.545185 secs 2024-07-27T09:06:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:18 smithi077 ceph-mon[37153]: pgmap v4813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:06:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:18 smithi047 ceph-mon[36835]: pgmap v4813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:06:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:18 smithi029 ceph-mon[32610]: pgmap v4813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:06:20.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:20 smithi029 ceph-mon[32610]: pgmap v4814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:06:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:20 smithi077 ceph-mon[37153]: pgmap v4814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:06:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:20 smithi047 ceph-mon[36835]: pgmap v4814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:06:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:22 smithi077 ceph-mon[37153]: pgmap v4815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:06:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8204.545310 secs 2024-07-27T09:06:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:22 smithi047 ceph-mon[36835]: pgmap v4815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:06:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8204.545310 secs 2024-07-27T09:06:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:22 smithi029 ceph-mon[32610]: pgmap v4815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:06:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8204.545310 secs 2024-07-27T09:06:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:24 smithi077 ceph-mon[37153]: pgmap v4816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:06:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:24 smithi047 ceph-mon[36835]: pgmap v4816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:06:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:24 smithi029 ceph-mon[32610]: pgmap v4816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:06:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:26 smithi077 ceph-mon[37153]: pgmap v4817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:06:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:26 smithi047 ceph-mon[36835]: pgmap v4817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:06:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:26 smithi029 ceph-mon[32610]: pgmap v4817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:06:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8209.545392 secs 2024-07-27T09:06:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8209.545392 secs 2024-07-27T09:06:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8209.545392 secs 2024-07-27T09:06:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:28 smithi077 ceph-mon[37153]: pgmap v4818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T09:06:28.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:28 smithi047 ceph-mon[36835]: pgmap v4818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T09:06:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:28 smithi029 ceph-mon[32610]: pgmap v4818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T09:06:29.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:06:29.468+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:06:29.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:06:29.468+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:06:29.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:06:29.468+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:06:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:06:29.602+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:06:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:06:29.602+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:06:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:06:29.602+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:06:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:30 smithi077 ceph-mon[37153]: pgmap v4819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:06:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:30 smithi047 ceph-mon[36835]: pgmap v4819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:06:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:30 smithi029 ceph-mon[32610]: pgmap v4819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:06:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:32 smithi077 ceph-mon[37153]: pgmap v4820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T09:06:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8214.545517 secs 2024-07-27T09:06:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:32 smithi047 ceph-mon[36835]: pgmap v4820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T09:06:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8214.545517 secs 2024-07-27T09:06:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:32 smithi029 ceph-mon[32610]: pgmap v4820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T09:06:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8214.545517 secs 2024-07-27T09:06:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:34 smithi077 ceph-mon[37153]: pgmap v4821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:06:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:34 smithi047 ceph-mon[36835]: pgmap v4821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:06:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:34 smithi029 ceph-mon[32610]: pgmap v4821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:06:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:36 smithi077 ceph-mon[37153]: pgmap v4822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:06:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:36 smithi047 ceph-mon[36835]: pgmap v4822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:06:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:36 smithi029 ceph-mon[32610]: pgmap v4822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:06:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8219.545640 secs 2024-07-27T09:06:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8219.545640 secs 2024-07-27T09:06:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8219.545640 secs 2024-07-27T09:06:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:38 smithi077 ceph-mon[37153]: pgmap v4823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T09:06:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:38 smithi047 ceph-mon[36835]: pgmap v4823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T09:06:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:38 smithi029 ceph-mon[32610]: pgmap v4823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T09:06:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:40 smithi077 ceph-mon[37153]: pgmap v4824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:06:40.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:40 smithi047 ceph-mon[36835]: pgmap v4824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:06:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:40 smithi029 ceph-mon[32610]: pgmap v4824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:06:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:42 smithi077 ceph-mon[37153]: pgmap v4825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:06:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8224.545760 secs 2024-07-27T09:06:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:42 smithi047 ceph-mon[36835]: pgmap v4825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:06:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8224.545760 secs 2024-07-27T09:06:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:42 smithi029 ceph-mon[32610]: pgmap v4825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:06:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8224.545760 secs 2024-07-27T09:06:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:06:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:06:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:06:44.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:06:44.482+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:06:44.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:06:44.482+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:06:44.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:06:44.482+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:06:44.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:06:44.612+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:06:44.614 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:06:44.612+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:06:44.615 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:06:44.612+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:06:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:44 smithi077 ceph-mon[37153]: pgmap v4826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:06:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:06:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:06:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:06:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:44 smithi047 ceph-mon[36835]: pgmap v4826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:06:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:06:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:06:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:06:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:44 smithi029 ceph-mon[32610]: pgmap v4826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:06:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:06:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:06:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:06:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:45 smithi077 ceph-mon[37153]: pgmap v4827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:06:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:45 smithi077 ceph-mon[37153]: pgmap v4828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:06:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:45 smithi047 ceph-mon[36835]: pgmap v4827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:06:45.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:45 smithi047 ceph-mon[36835]: pgmap v4828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:06:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:45 smithi029 ceph-mon[32610]: pgmap v4827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:06:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:45 smithi029 ceph-mon[32610]: pgmap v4828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:06:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:47 smithi077 ceph-mon[37153]: pgmap v4829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:06:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8229.545893 secs 2024-07-27T09:06:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:47 smithi047 ceph-mon[36835]: pgmap v4829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:06:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8229.545893 secs 2024-07-27T09:06:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:47 smithi029 ceph-mon[32610]: pgmap v4829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:06:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8229.545893 secs 2024-07-27T09:06:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:49 smithi077 ceph-mon[37153]: pgmap v4830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:06:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:49 smithi047 ceph-mon[36835]: pgmap v4830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:06:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:49 smithi029 ceph-mon[32610]: pgmap v4830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:06:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:51 smithi077 ceph-mon[37153]: pgmap v4831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:06:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:51 smithi047 ceph-mon[36835]: pgmap v4831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:06:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:51 smithi029 ceph-mon[32610]: pgmap v4831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:06:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8234.546017 secs 2024-07-27T09:06:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8234.546017 secs 2024-07-27T09:06:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8234.546017 secs 2024-07-27T09:06:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:53 smithi077 ceph-mon[37153]: pgmap v4832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:06:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:53 smithi047 ceph-mon[36835]: pgmap v4832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:06:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:53 smithi029 ceph-mon[32610]: pgmap v4832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:06:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:55 smithi077 ceph-mon[37153]: pgmap v4833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:06:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:55 smithi047 ceph-mon[36835]: pgmap v4833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:06:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:55 smithi029 ceph-mon[32610]: pgmap v4833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:06:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:57 smithi047 ceph-mon[36835]: pgmap v4834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:06:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8239.546130 secs 2024-07-27T09:06:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:57 smithi029 ceph-mon[32610]: pgmap v4834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:06:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8239.546130 secs 2024-07-27T09:06:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:57 smithi077 ceph-mon[37153]: pgmap v4834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:06:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8239.546130 secs 2024-07-27T09:06:59.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:06:59.497+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:06:59.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:06:59.498+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:06:59.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:06:59.498+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:06:59.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:06:59.625+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:06:59.626 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:06:59.625+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:06:59.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:06:59.625+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:06:59.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:06:59 smithi029 ceph-mon[32610]: pgmap v4835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:06:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:06:59 smithi047 ceph-mon[36835]: pgmap v4835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:00.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:06:59 smithi077 ceph-mon[37153]: pgmap v4835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:01 smithi047 ceph-mon[36835]: pgmap v4836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:01 smithi029 ceph-mon[32610]: pgmap v4836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:01 smithi077 ceph-mon[37153]: pgmap v4836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8244.546236 secs 2024-07-27T09:07:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8244.546236 secs 2024-07-27T09:07:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8244.546236 secs 2024-07-27T09:07:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:03 smithi047 ceph-mon[36835]: pgmap v4837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:03 smithi029 ceph-mon[32610]: pgmap v4837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:03 smithi077 ceph-mon[37153]: pgmap v4837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:05 smithi047 ceph-mon[36835]: pgmap v4838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:07:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:05 smithi029 ceph-mon[32610]: pgmap v4838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:07:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:05 smithi077 ceph-mon[37153]: pgmap v4838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:07:07.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:07 smithi077 ceph-mon[37153]: pgmap v4839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:07.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8249.546360 secs 2024-07-27T09:07:07.944 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:07 smithi029 ceph-mon[32610]: pgmap v4839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:07.944 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8249.546360 secs 2024-07-27T09:07:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:07 smithi047 ceph-mon[36835]: pgmap v4839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8249.546360 secs 2024-07-27T09:07:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:09 smithi047 ceph-mon[36835]: pgmap v4840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:09 smithi029 ceph-mon[32610]: pgmap v4840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:09 smithi077 ceph-mon[37153]: pgmap v4840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:11 smithi047 ceph-mon[36835]: pgmap v4841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:11 smithi029 ceph-mon[32610]: pgmap v4841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:11 smithi077 ceph-mon[37153]: pgmap v4841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8254.546498 secs 2024-07-27T09:07:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8254.546498 secs 2024-07-27T09:07:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8254.546498 secs 2024-07-27T09:07:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:13 smithi047 ceph-mon[36835]: pgmap v4842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:13 smithi029 ceph-mon[32610]: pgmap v4842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:13 smithi077 ceph-mon[37153]: pgmap v4842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:14.499+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:07:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:14.499+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:07:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:14.499+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:07:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:14.640+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:07:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:14.640+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:07:14.642 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:14.640+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:07:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:15 smithi047 ceph-mon[36835]: pgmap v4843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:07:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:15 smithi029 ceph-mon[32610]: pgmap v4843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:07:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:15 smithi077 ceph-mon[37153]: pgmap v4843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:07:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:17 smithi047 ceph-mon[36835]: pgmap v4844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8259.546624 secs 2024-07-27T09:07:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:17 smithi029 ceph-mon[32610]: pgmap v4844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8259.546624 secs 2024-07-27T09:07:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:17 smithi077 ceph-mon[37153]: pgmap v4844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8259.546624 secs 2024-07-27T09:07:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:19 smithi047 ceph-mon[36835]: pgmap v4845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:19 smithi029 ceph-mon[32610]: pgmap v4845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:19 smithi077 ceph-mon[37153]: pgmap v4845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:21 smithi047 ceph-mon[36835]: pgmap v4846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:21 smithi029 ceph-mon[32610]: pgmap v4846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:21 smithi077 ceph-mon[37153]: pgmap v4846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8264.546746 secs 2024-07-27T09:07:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8264.546746 secs 2024-07-27T09:07:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8264.546746 secs 2024-07-27T09:07:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:23 smithi047 ceph-mon[36835]: pgmap v4847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:23 smithi029 ceph-mon[32610]: pgmap v4847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:23 smithi077 ceph-mon[37153]: pgmap v4847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:25.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:25 smithi047 ceph-mon[36835]: pgmap v4848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:07:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:25 smithi029 ceph-mon[32610]: pgmap v4848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:07:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:25 smithi077 ceph-mon[37153]: pgmap v4848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:07:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:27 smithi047 ceph-mon[36835]: pgmap v4849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8269.546868 secs 2024-07-27T09:07:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:27 smithi029 ceph-mon[32610]: pgmap v4849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8269.546868 secs 2024-07-27T09:07:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:27 smithi077 ceph-mon[37153]: pgmap v4849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8269.546868 secs 2024-07-27T09:07:29.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:29.514+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:07:29.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:29.514+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:07:29.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:29.514+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:07:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:29.655+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:07:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:29.655+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:07:29.657 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:29.655+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:07:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:29 smithi047 ceph-mon[36835]: pgmap v4850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:29 smithi029 ceph-mon[32610]: pgmap v4850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:29 smithi077 ceph-mon[37153]: pgmap v4850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:31 smithi047 ceph-mon[36835]: pgmap v4851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:31 smithi029 ceph-mon[32610]: pgmap v4851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:31 smithi077 ceph-mon[37153]: pgmap v4851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8274.546953 secs 2024-07-27T09:07:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8274.546953 secs 2024-07-27T09:07:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8274.546953 secs 2024-07-27T09:07:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:33 smithi047 ceph-mon[36835]: pgmap v4852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:33 smithi029 ceph-mon[32610]: pgmap v4852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:33 smithi077 ceph-mon[37153]: pgmap v4852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:35 smithi047 ceph-mon[36835]: pgmap v4853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:07:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:35 smithi029 ceph-mon[32610]: pgmap v4853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:07:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:35 smithi077 ceph-mon[37153]: pgmap v4853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:07:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:37 smithi047 ceph-mon[36835]: pgmap v4854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8279.547067 secs 2024-07-27T09:07:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:37 smithi029 ceph-mon[32610]: pgmap v4854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8279.547067 secs 2024-07-27T09:07:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:37 smithi077 ceph-mon[37153]: pgmap v4854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8279.547067 secs 2024-07-27T09:07:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:39 smithi047 ceph-mon[36835]: pgmap v4855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:39 smithi029 ceph-mon[32610]: pgmap v4855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:39 smithi077 ceph-mon[37153]: pgmap v4855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:07:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:41 smithi047 ceph-mon[36835]: pgmap v4856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:41 smithi029 ceph-mon[32610]: pgmap v4856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:41 smithi077 ceph-mon[37153]: pgmap v4856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:07:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8284.547190 secs 2024-07-27T09:07:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8284.547190 secs 2024-07-27T09:07:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8284.547190 secs 2024-07-27T09:07:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:43.883+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:07:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:43.883+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:07:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:43.883+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:07:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:43.883+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:07:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:43 smithi029 ceph-mon[32610]: pgmap v4857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:43 smithi077 ceph-mon[37153]: pgmap v4857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:43 smithi047 ceph-mon[36835]: pgmap v4857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:07:44.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:44.529+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:07:44.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:44.529+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:07:44.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:44.529+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:07:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:44.670+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:07:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:44.670+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:07:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:44.670+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:07:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:07:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:44.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:44.980 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:07:44.980 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:44.980 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:07:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:45 smithi029 ceph-mon[32610]: pgmap v4858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:07:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:45 smithi077 ceph-mon[37153]: pgmap v4858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:07:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:45 smithi047 ceph-mon[36835]: pgmap v4858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:07:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:46 smithi047 ceph-mon[36835]: pgmap v4859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:07:47.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:07:47.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:07:47.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:46 smithi047 ceph-mon[36835]: pgmap v4860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:07:47.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:46 smithi047 ceph-mon[36835]: pgmap v4861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T09:07:47.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:46 smithi029 ceph-mon[32610]: pgmap v4859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:07:47.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:07:47.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:07:47.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:46 smithi029 ceph-mon[32610]: pgmap v4860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:07:47.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:46 smithi029 ceph-mon[32610]: pgmap v4861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T09:07:47.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:46 smithi077 ceph-mon[37153]: pgmap v4859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:07:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:47.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:07:47.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:07:47.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:46 smithi077 ceph-mon[37153]: pgmap v4860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:07:47.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:46 smithi077 ceph-mon[37153]: pgmap v4861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T09:07:47.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:07:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8289.547282 secs 2024-07-27T09:07:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8289.547282 secs 2024-07-27T09:07:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8289.547282 secs 2024-07-27T09:07:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:48 smithi047 ceph-mon[36835]: pgmap v4862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:07:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:48 smithi029 ceph-mon[32610]: pgmap v4862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:07:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:48 smithi077 ceph-mon[37153]: pgmap v4862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:07:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:51 smithi077 ceph-mon[37153]: pgmap v4863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-27T09:07:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:51 smithi047 ceph-mon[36835]: pgmap v4863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-27T09:07:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:51 smithi029 ceph-mon[32610]: pgmap v4863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-27T09:07:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8294.547405 secs 2024-07-27T09:07:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8294.547405 secs 2024-07-27T09:07:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8294.547405 secs 2024-07-27T09:07:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:53 smithi077 ceph-mon[37153]: pgmap v4864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:07:54.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:53 smithi047 ceph-mon[36835]: pgmap v4864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:07:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:53 smithi029 ceph-mon[32610]: pgmap v4864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:07:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:55 smithi077 ceph-mon[37153]: pgmap v4865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:07:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:55 smithi047 ceph-mon[36835]: pgmap v4865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:07:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:55 smithi029 ceph-mon[32610]: pgmap v4865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:07:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:57 smithi077 ceph-mon[37153]: pgmap v4866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:07:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8299.547541 secs 2024-07-27T09:07:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:57 smithi047 ceph-mon[36835]: pgmap v4866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:07:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8299.547541 secs 2024-07-27T09:07:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:57 smithi029 ceph-mon[32610]: pgmap v4866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:07:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8299.547541 secs 2024-07-27T09:07:58.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:58.885+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:07:58.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:58.885+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:07:58.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:58.886+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:07:58.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:58.886+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:07:59.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:59.545+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:07:59.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:59.545+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:07:59.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:07:59.545+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:07:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:59.686+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:07:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:59.686+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:07:59.687 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:07:59.686+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:08:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:07:59 smithi077 ceph-mon[37153]: pgmap v4867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:07:59 smithi047 ceph-mon[36835]: pgmap v4867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:07:59 smithi029 ceph-mon[32610]: pgmap v4867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:01 smithi077 ceph-mon[37153]: pgmap v4868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:01 smithi047 ceph-mon[36835]: pgmap v4868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:01 smithi029 ceph-mon[32610]: pgmap v4868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8304.547666 secs 2024-07-27T09:08:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8304.547666 secs 2024-07-27T09:08:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8304.547666 secs 2024-07-27T09:08:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:03 smithi077 ceph-mon[37153]: pgmap v4869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:03 smithi047 ceph-mon[36835]: pgmap v4869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:03 smithi029 ceph-mon[32610]: pgmap v4869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:05 smithi077 ceph-mon[37153]: pgmap v4870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:05 smithi047 ceph-mon[36835]: pgmap v4870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:05 smithi029 ceph-mon[32610]: pgmap v4870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:07 smithi047 ceph-mon[36835]: pgmap v4871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8309.547812 secs 2024-07-27T09:08:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:07 smithi029 ceph-mon[32610]: pgmap v4871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8309.547812 secs 2024-07-27T09:08:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:07 smithi077 ceph-mon[37153]: pgmap v4871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8309.547812 secs 2024-07-27T09:08:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:09 smithi077 ceph-mon[37153]: pgmap v4872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:09 smithi047 ceph-mon[36835]: pgmap v4872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:09 smithi029 ceph-mon[32610]: pgmap v4872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:11 smithi077 ceph-mon[37153]: pgmap v4873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:11 smithi047 ceph-mon[36835]: pgmap v4873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:11 smithi029 ceph-mon[32610]: pgmap v4873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8314.547953 secs 2024-07-27T09:08:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8314.547953 secs 2024-07-27T09:08:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8314.547953 secs 2024-07-27T09:08:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:13 smithi077 ceph-mon[37153]: pgmap v4874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:14.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:13 smithi047 ceph-mon[36835]: pgmap v4874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:13 smithi029 ceph-mon[32610]: pgmap v4874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:14.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:08:14.560+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:08:14.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:08:14.560+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:08:14.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:08:14.560+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:08:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:08:14.689+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:08:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:08:14.689+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:08:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:08:14.689+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:08:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:15 smithi077 ceph-mon[37153]: pgmap v4875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:16.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:15 smithi047 ceph-mon[36835]: pgmap v4875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:15 smithi029 ceph-mon[32610]: pgmap v4875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:17 smithi077 ceph-mon[37153]: pgmap v4876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8319.548050 secs 2024-07-27T09:08:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:17 smithi047 ceph-mon[36835]: pgmap v4876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8319.548050 secs 2024-07-27T09:08:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:17 smithi029 ceph-mon[32610]: pgmap v4876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8319.548050 secs 2024-07-27T09:08:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:19 smithi077 ceph-mon[37153]: pgmap v4877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:19 smithi047 ceph-mon[36835]: pgmap v4877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:19 smithi029 ceph-mon[32610]: pgmap v4877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:21 smithi077 ceph-mon[37153]: pgmap v4878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:21 smithi047 ceph-mon[36835]: pgmap v4878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:21 smithi029 ceph-mon[32610]: pgmap v4878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8324.548117 secs 2024-07-27T09:08:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8324.548117 secs 2024-07-27T09:08:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8324.548117 secs 2024-07-27T09:08:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:23 smithi077 ceph-mon[37153]: pgmap v4879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:23 smithi047 ceph-mon[36835]: pgmap v4879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:23 smithi029 ceph-mon[32610]: pgmap v4879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:25 smithi077 ceph-mon[37153]: pgmap v4880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:26.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:25 smithi047 ceph-mon[36835]: pgmap v4880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:25 smithi029 ceph-mon[32610]: pgmap v4880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:27 smithi077 ceph-mon[37153]: pgmap v4881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8329.548245 secs 2024-07-27T09:08:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:27 smithi047 ceph-mon[36835]: pgmap v4881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8329.548245 secs 2024-07-27T09:08:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:27 smithi029 ceph-mon[32610]: pgmap v4881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8329.548245 secs 2024-07-27T09:08:29.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:08:29.575+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:08:29.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:08:29.575+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:08:29.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:08:29.575+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:08:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:08:29.693+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:08:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:08:29.693+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:08:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:08:29.693+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:08:30.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:29 smithi077 ceph-mon[37153]: pgmap v4882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:30.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:29 smithi047 ceph-mon[36835]: pgmap v4882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:29 smithi029 ceph-mon[32610]: pgmap v4882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:31 smithi077 ceph-mon[37153]: pgmap v4883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:31 smithi047 ceph-mon[36835]: pgmap v4883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:31 smithi029 ceph-mon[32610]: pgmap v4883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8334.548365 secs 2024-07-27T09:08:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8334.548365 secs 2024-07-27T09:08:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8334.548365 secs 2024-07-27T09:08:34.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:33 smithi047 ceph-mon[36835]: pgmap v4884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:33 smithi029 ceph-mon[32610]: pgmap v4884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:33 smithi077 ceph-mon[37153]: pgmap v4884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:35 smithi047 ceph-mon[36835]: pgmap v4885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:35 smithi029 ceph-mon[32610]: pgmap v4885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:35 smithi077 ceph-mon[37153]: pgmap v4885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:38.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:37 smithi077 ceph-mon[37153]: pgmap v4886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:38.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8339.548507 secs 2024-07-27T09:08:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:37 smithi047 ceph-mon[36835]: pgmap v4886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8339.548507 secs 2024-07-27T09:08:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:37 smithi029 ceph-mon[32610]: pgmap v4886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8339.548507 secs 2024-07-27T09:08:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:39 smithi047 ceph-mon[36835]: pgmap v4887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:39 smithi029 ceph-mon[32610]: pgmap v4887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:39 smithi077 ceph-mon[37153]: pgmap v4887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:08:42.152 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:41 smithi029 ceph-mon[32610]: pgmap v4888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:41 smithi047 ceph-mon[36835]: pgmap v4888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:41 smithi077 ceph-mon[37153]: pgmap v4888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:08:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8344.548647 secs 2024-07-27T09:08:43.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8344.548647 secs 2024-07-27T09:08:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8344.548647 secs 2024-07-27T09:08:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:43 smithi047 ceph-mon[36835]: pgmap v4889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:43 smithi029 ceph-mon[32610]: pgmap v4889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:43 smithi077 ceph-mon[37153]: pgmap v4889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:08:44.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:08:44.591+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:08:44.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:08:44.591+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:08:44.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:08:44.591+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:08:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:08:44.702+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:08:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:08:44.702+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:08:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:08:44.702+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:08:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:45 smithi047 ceph-mon[36835]: pgmap v4890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:45 smithi029 ceph-mon[32610]: pgmap v4890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:45 smithi077 ceph-mon[37153]: pgmap v4890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:08:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:08:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:08:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:08:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: pgmap v4891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8349.548746 secs 2024-07-27T09:08:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:08:48.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:08:48.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: pgmap v4891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8349.548746 secs 2024-07-27T09:08:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:08:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:08:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: pgmap v4891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:08:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8349.548746 secs 2024-07-27T09:08:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:48.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:08:48.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:08:48.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:08:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:48 smithi047 ceph-mon[36835]: pgmap v4892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:08:49.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:48 smithi047 ceph-mon[36835]: pgmap v4893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:08:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:48 smithi029 ceph-mon[32610]: pgmap v4892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:08:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:48 smithi029 ceph-mon[32610]: pgmap v4893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:08:49.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:48 smithi077 ceph-mon[37153]: pgmap v4892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:08:49.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:48 smithi077 ceph-mon[37153]: pgmap v4893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:08:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:50 smithi047 ceph-mon[36835]: pgmap v4894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T09:08:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:50 smithi029 ceph-mon[32610]: pgmap v4894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T09:08:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:50 smithi077 ceph-mon[37153]: pgmap v4894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T09:08:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:51 smithi029 ceph-mon[32610]: pgmap v4895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:08:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:51 smithi077 ceph-mon[37153]: pgmap v4895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:08:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:51 smithi047 ceph-mon[36835]: pgmap v4895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:08:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8354.548868 secs 2024-07-27T09:08:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8354.548868 secs 2024-07-27T09:08:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8354.548868 secs 2024-07-27T09:08:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:53 smithi029 ceph-mon[32610]: pgmap v4896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:08:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:53 smithi077 ceph-mon[37153]: pgmap v4896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:08:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:53 smithi047 ceph-mon[36835]: pgmap v4896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:08:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:55 smithi029 ceph-mon[32610]: pgmap v4897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-27T09:08:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:55 smithi077 ceph-mon[37153]: pgmap v4897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-27T09:08:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:55 smithi047 ceph-mon[36835]: pgmap v4897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-27T09:08:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8359.548993 secs 2024-07-27T09:08:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8359.548993 secs 2024-07-27T09:08:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8359.548993 secs 2024-07-27T09:08:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:58 smithi077 ceph-mon[37153]: pgmap v4898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:08:58.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:58 smithi029 ceph-mon[32610]: pgmap v4898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:08:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:58 smithi047 ceph-mon[36835]: pgmap v4898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:08:59.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:08:59.606+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:08:59.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:08:59.606+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:08:59.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:08:59.606+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:08:59.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:08:59.713+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:08:59.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:08:59.713+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:08:59.714 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:08:59.713+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:09:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:08:59 smithi029 ceph-mon[32610]: pgmap v4899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:08:59 smithi077 ceph-mon[37153]: pgmap v4899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:08:59 smithi047 ceph-mon[36835]: pgmap v4899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:01 smithi029 ceph-mon[32610]: pgmap v4900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:01 smithi077 ceph-mon[37153]: pgmap v4900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:01 smithi047 ceph-mon[36835]: pgmap v4900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:03.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8364.549109 secs 2024-07-27T09:09:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8364.549109 secs 2024-07-27T09:09:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8364.549109 secs 2024-07-27T09:09:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:03 smithi029 ceph-mon[32610]: pgmap v4901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:03 smithi077 ceph-mon[37153]: pgmap v4901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:03 smithi047 ceph-mon[36835]: pgmap v4901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:05 smithi029 ceph-mon[32610]: pgmap v4902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:05 smithi077 ceph-mon[37153]: pgmap v4902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:05 smithi047 ceph-mon[36835]: pgmap v4902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8369.549221 secs 2024-07-27T09:09:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8369.549221 secs 2024-07-27T09:09:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8369.549221 secs 2024-07-27T09:09:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:08 smithi077 ceph-mon[37153]: pgmap v4903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:08 smithi047 ceph-mon[36835]: pgmap v4903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:08 smithi029 ceph-mon[32610]: pgmap v4903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:09 smithi029 ceph-mon[32610]: pgmap v4904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:09 smithi077 ceph-mon[37153]: pgmap v4904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:10.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:09 smithi047 ceph-mon[36835]: pgmap v4904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:11 smithi029 ceph-mon[32610]: pgmap v4905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:12.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:11 smithi047 ceph-mon[36835]: pgmap v4905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:11 smithi077 ceph-mon[37153]: pgmap v4905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8374.549340 secs 2024-07-27T09:09:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8374.549340 secs 2024-07-27T09:09:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8374.549340 secs 2024-07-27T09:09:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:13 smithi029 ceph-mon[32610]: pgmap v4906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:13 smithi077 ceph-mon[37153]: pgmap v4906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:13 smithi047 ceph-mon[36835]: pgmap v4906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:09:14.621+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:09:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:09:14.621+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:09:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:09:14.621+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:09:14.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:14.717+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:09:14.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:14.717+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:09:14.719 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:14.717+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:09:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:15 smithi029 ceph-mon[32610]: pgmap v4907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:15 smithi077 ceph-mon[37153]: pgmap v4907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:15 smithi047 ceph-mon[36835]: pgmap v4907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8379.549461 secs 2024-07-27T09:09:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8379.549461 secs 2024-07-27T09:09:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8379.549461 secs 2024-07-27T09:09:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:18 smithi077 ceph-mon[37153]: pgmap v4908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:18 smithi047 ceph-mon[36835]: pgmap v4908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:18 smithi029 ceph-mon[32610]: pgmap v4908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:19 smithi029 ceph-mon[32610]: pgmap v4909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:19 smithi077 ceph-mon[37153]: pgmap v4909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:20.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:19 smithi047 ceph-mon[36835]: pgmap v4909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:21 smithi029 ceph-mon[32610]: pgmap v4910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:21 smithi077 ceph-mon[37153]: pgmap v4910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:21 smithi047 ceph-mon[36835]: pgmap v4910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8384.549565 secs 2024-07-27T09:09:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8384.549565 secs 2024-07-27T09:09:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8384.549565 secs 2024-07-27T09:09:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:23 smithi029 ceph-mon[32610]: pgmap v4911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:23 smithi077 ceph-mon[37153]: pgmap v4911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:24.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:23 smithi047 ceph-mon[36835]: pgmap v4911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:25 smithi029 ceph-mon[32610]: pgmap v4912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:25 smithi077 ceph-mon[37153]: pgmap v4912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:26.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:25 smithi047 ceph-mon[36835]: pgmap v4912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8389.549706 secs 2024-07-27T09:09:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8389.549706 secs 2024-07-27T09:09:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8389.549706 secs 2024-07-27T09:09:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:28 smithi077 ceph-mon[37153]: pgmap v4913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:28.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:28 smithi029 ceph-mon[32610]: pgmap v4913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:28 smithi047 ceph-mon[36835]: pgmap v4913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:29.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:09:29.637+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:09:29.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:09:29.637+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:09:29.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:09:29.637+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:09:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:29.733+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:09:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:29.733+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:09:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:29.733+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:09:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:29 smithi029 ceph-mon[32610]: pgmap v4914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:29 smithi077 ceph-mon[37153]: pgmap v4914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:29 smithi047 ceph-mon[36835]: pgmap v4914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:31 smithi029 ceph-mon[32610]: pgmap v4915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:31 smithi077 ceph-mon[37153]: pgmap v4915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:31 smithi047 ceph-mon[36835]: pgmap v4915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8394.549833 secs 2024-07-27T09:09:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8394.549833 secs 2024-07-27T09:09:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8394.549833 secs 2024-07-27T09:09:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:33 smithi029 ceph-mon[32610]: pgmap v4916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:33 smithi077 ceph-mon[37153]: pgmap v4916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:33 smithi047 ceph-mon[36835]: pgmap v4916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:35 smithi029 ceph-mon[32610]: pgmap v4917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:35 smithi077 ceph-mon[37153]: pgmap v4917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:35 smithi047 ceph-mon[36835]: pgmap v4917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8399.549957 secs 2024-07-27T09:09:37.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8399.549957 secs 2024-07-27T09:09:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8399.549957 secs 2024-07-27T09:09:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:38 smithi077 ceph-mon[37153]: pgmap v4918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:38 smithi047 ceph-mon[36835]: pgmap v4918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:38 smithi029 ceph-mon[32610]: pgmap v4918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:39 smithi029 ceph-mon[32610]: pgmap v4919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:39 smithi077 ceph-mon[37153]: pgmap v4919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:39 smithi047 ceph-mon[36835]: pgmap v4919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:41 smithi029 ceph-mon[32610]: pgmap v4920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:42.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:41 smithi047 ceph-mon[36835]: pgmap v4920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:41 smithi077 ceph-mon[37153]: pgmap v4920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:09:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8404.550039 secs 2024-07-27T09:09:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8404.550039 secs 2024-07-27T09:09:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8404.550039 secs 2024-07-27T09:09:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:43 smithi077 ceph-mon[37153]: pgmap v4921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:44.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:43 smithi047 ceph-mon[36835]: pgmap v4921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:43 smithi029 ceph-mon[32610]: pgmap v4921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:09:44.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:09:44.643+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:09:44.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:09:44.643+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:09:44.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:09:44.643+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:09:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:44.748+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:09:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:44.748+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:09:44.750 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:44.748+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:09:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:45 smithi029 ceph-mon[32610]: pgmap v4922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:45 smithi077 ceph-mon[37153]: pgmap v4922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:45 smithi047 ceph-mon[36835]: pgmap v4922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:09:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8409.550147 secs 2024-07-27T09:09:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8409.550147 secs 2024-07-27T09:09:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8409.550147 secs 2024-07-27T09:09:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:48 smithi077 ceph-mon[37153]: pgmap v4923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:09:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:48 smithi047 ceph-mon[36835]: pgmap v4923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:09:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:48 smithi029 ceph-mon[32610]: pgmap v4923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:09:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:09:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:09:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:09:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:49 smithi077 ceph-mon[37153]: pgmap v4924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:09:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:49 smithi077 ceph-mon[37153]: pgmap v4925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:09:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:09:49.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:09:49.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:09:49.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:49 smithi047 ceph-mon[36835]: pgmap v4924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:09:49.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:49 smithi047 ceph-mon[36835]: pgmap v4925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:09:49.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:09:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:09:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:09:49.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:49 smithi029 ceph-mon[32610]: pgmap v4924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:09:49.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:49 smithi029 ceph-mon[32610]: pgmap v4925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:09:49.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:09:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:51 smithi029 ceph-mon[32610]: pgmap v4926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:09:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:51 smithi077 ceph-mon[37153]: pgmap v4926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:09:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:51 smithi047 ceph-mon[36835]: pgmap v4926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:09:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8414.550579 secs 2024-07-27T09:09:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8414.550579 secs 2024-07-27T09:09:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8414.550579 secs 2024-07-27T09:09:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:53 smithi077 ceph-mon[37153]: pgmap v4927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:09:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:53 smithi047 ceph-mon[36835]: pgmap v4927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:09:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:53 smithi029 ceph-mon[32610]: pgmap v4927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:09:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:55 smithi077 ceph-mon[37153]: pgmap v4928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:09:55.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:55 smithi047 ceph-mon[36835]: pgmap v4928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:09:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:55 smithi029 ceph-mon[32610]: pgmap v4928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:09:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:57 smithi077 ceph-mon[37153]: pgmap v4929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T09:09:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8419.550700 secs 2024-07-27T09:09:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:57 smithi047 ceph-mon[36835]: pgmap v4929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T09:09:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8419.550700 secs 2024-07-27T09:09:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:57 smithi029 ceph-mon[32610]: pgmap v4929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T09:09:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8419.550700 secs 2024-07-27T09:09:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:58.904+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:09:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:58.904+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:09:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:58.904+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:09:58.906 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:58.905+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:09:59.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:09:59.659+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:09:59.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:09:59.659+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:09:59.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:09:59.659+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:09:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:59.760+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:09:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:59.760+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:09:59.761 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:09:59.760+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:09:59.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:09:59 smithi029 ceph-mon[32610]: pgmap v4930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:09:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:09:59 smithi077 ceph-mon[37153]: pgmap v4930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:09:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:09:59 smithi047 ceph-mon[36835]: pgmap v4930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:10:00.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:10:00.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:10:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:10:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:01 smithi077 ceph-mon[37153]: pgmap v4931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:10:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:01 smithi047 ceph-mon[36835]: pgmap v4931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:10:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:01 smithi029 ceph-mon[32610]: pgmap v4931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:10:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8424.550780 secs 2024-07-27T09:10:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8424.550780 secs 2024-07-27T09:10:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8424.550780 secs 2024-07-27T09:10:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:03 smithi077 ceph-mon[37153]: pgmap v4932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:03 smithi047 ceph-mon[36835]: pgmap v4932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:03 smithi029 ceph-mon[32610]: pgmap v4932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:05 smithi077 ceph-mon[37153]: pgmap v4933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:05 smithi047 ceph-mon[36835]: pgmap v4933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:05 smithi029 ceph-mon[32610]: pgmap v4933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:07 smithi047 ceph-mon[36835]: pgmap v4934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8429.550870 secs 2024-07-27T09:10:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:07 smithi029 ceph-mon[32610]: pgmap v4934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8429.550870 secs 2024-07-27T09:10:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:07 smithi077 ceph-mon[37153]: pgmap v4934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8429.550870 secs 2024-07-27T09:10:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:09 smithi029 ceph-mon[32610]: pgmap v4935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:09 smithi077 ceph-mon[37153]: pgmap v4935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:09 smithi047 ceph-mon[36835]: pgmap v4935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:11 smithi077 ceph-mon[37153]: pgmap v4936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:10:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:11 smithi047 ceph-mon[36835]: pgmap v4936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:10:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:11 smithi029 ceph-mon[32610]: pgmap v4936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:10:12.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8434.550982 secs 2024-07-27T09:10:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8434.550982 secs 2024-07-27T09:10:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8434.550982 secs 2024-07-27T09:10:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:13 smithi077 ceph-mon[37153]: pgmap v4937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:13.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:13.906+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:10:13.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:13.906+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:10:13.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:13 smithi029 ceph-mon[32610]: pgmap v4937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:13.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:13.906+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:10:13.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:13.906+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:10:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:13 smithi047 ceph-mon[36835]: pgmap v4937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:14.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:14.673+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:10:14.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:14.673+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:10:14.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:14.673+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:10:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:14.772+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:10:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:14.772+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:10:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:14.772+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:10:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:15 smithi077 ceph-mon[37153]: pgmap v4938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:15 smithi047 ceph-mon[36835]: pgmap v4938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:15 smithi029 ceph-mon[32610]: pgmap v4938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:17 smithi077 ceph-mon[37153]: pgmap v4939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8439.551098 secs 2024-07-27T09:10:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:17 smithi047 ceph-mon[36835]: pgmap v4939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8439.551098 secs 2024-07-27T09:10:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:17 smithi029 ceph-mon[32610]: pgmap v4939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8439.551098 secs 2024-07-27T09:10:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:19 smithi077 ceph-mon[37153]: pgmap v4940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:19 smithi047 ceph-mon[36835]: pgmap v4940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:19 smithi029 ceph-mon[32610]: pgmap v4940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:21 smithi077 ceph-mon[37153]: pgmap v4941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:10:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:21 smithi047 ceph-mon[36835]: pgmap v4941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:10:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:21 smithi029 ceph-mon[32610]: pgmap v4941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:10:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8444.551225 secs 2024-07-27T09:10:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8444.551225 secs 2024-07-27T09:10:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8444.551225 secs 2024-07-27T09:10:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:23 smithi077 ceph-mon[37153]: pgmap v4942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:23 smithi047 ceph-mon[36835]: pgmap v4942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:23 smithi029 ceph-mon[32610]: pgmap v4942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:25 smithi077 ceph-mon[37153]: pgmap v4943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:25 smithi047 ceph-mon[36835]: pgmap v4943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:25 smithi029 ceph-mon[32610]: pgmap v4943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:27 smithi077 ceph-mon[37153]: pgmap v4944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8449.551361 secs 2024-07-27T09:10:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:27 smithi047 ceph-mon[36835]: pgmap v4944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8449.551361 secs 2024-07-27T09:10:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:27 smithi029 ceph-mon[32610]: pgmap v4944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8449.551361 secs 2024-07-27T09:10:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:29.689+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:10:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:29.689+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:10:29.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:29.689+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:10:29.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:29 smithi029 ceph-mon[32610]: pgmap v4945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:29.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:29.778+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:10:29.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:29.778+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:10:29.780 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:29.778+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:10:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:29 smithi077 ceph-mon[37153]: pgmap v4945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:29 smithi047 ceph-mon[36835]: pgmap v4945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:31 smithi077 ceph-mon[37153]: pgmap v4946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:10:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:31 smithi047 ceph-mon[36835]: pgmap v4946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:10:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:31 smithi029 ceph-mon[32610]: pgmap v4946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:10:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8454.551468 secs 2024-07-27T09:10:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8454.551468 secs 2024-07-27T09:10:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8454.551468 secs 2024-07-27T09:10:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:33 smithi077 ceph-mon[37153]: pgmap v4947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:33 smithi047 ceph-mon[36835]: pgmap v4947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:33 smithi029 ceph-mon[32610]: pgmap v4947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:35 smithi077 ceph-mon[37153]: pgmap v4948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:35 smithi047 ceph-mon[36835]: pgmap v4948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:35 smithi029 ceph-mon[32610]: pgmap v4948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:37 smithi077 ceph-mon[37153]: pgmap v4949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8459.551623 secs 2024-07-27T09:10:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:37 smithi047 ceph-mon[36835]: pgmap v4949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8459.551623 secs 2024-07-27T09:10:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:37 smithi029 ceph-mon[32610]: pgmap v4949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8459.551623 secs 2024-07-27T09:10:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:39 smithi077 ceph-mon[37153]: pgmap v4950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:39 smithi047 ceph-mon[36835]: pgmap v4950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:39 smithi029 ceph-mon[32610]: pgmap v4950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:41 smithi077 ceph-mon[37153]: pgmap v4951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:10:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:41 smithi047 ceph-mon[36835]: pgmap v4951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:10:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:41 smithi029 ceph-mon[32610]: pgmap v4951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:10:42.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8464.551748 secs 2024-07-27T09:10:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8464.551748 secs 2024-07-27T09:10:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8464.551748 secs 2024-07-27T09:10:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:43 smithi077 ceph-mon[37153]: pgmap v4952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:43.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:43.911+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:10:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:43.911+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:10:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:43.911+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:10:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:43.911+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:10:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:43 smithi047 ceph-mon[36835]: pgmap v4952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:43 smithi029 ceph-mon[32610]: pgmap v4952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:10:44.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:44.704+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:10:44.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:44.704+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:10:44.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:44.704+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:10:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:44.794+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:10:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:44.794+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:10:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:44.794+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:10:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:45 smithi077 ceph-mon[37153]: pgmap v4953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:45 smithi047 ceph-mon[36835]: pgmap v4953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:45 smithi029 ceph-mon[32610]: pgmap v4953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:10:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:47 smithi077 ceph-mon[37153]: pgmap v4954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8469.551897 secs 2024-07-27T09:10:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:47 smithi047 ceph-mon[36835]: pgmap v4954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8469.551897 secs 2024-07-27T09:10:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:47 smithi029 ceph-mon[32610]: pgmap v4954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:10:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8469.551897 secs 2024-07-27T09:10:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:10:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:10:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:10:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:49 smithi077 ceph-mon[37153]: pgmap v4955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:10:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:10:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:10:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:49 smithi047 ceph-mon[36835]: pgmap v4955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:10:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:10:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:10:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:49 smithi029 ceph-mon[32610]: pgmap v4955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:10:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:10:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:10:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:10:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:50 smithi077 ceph-mon[37153]: pgmap v4956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:10:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:50 smithi077 ceph-mon[37153]: pgmap v4957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:10:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:50 smithi047 ceph-mon[36835]: pgmap v4956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:10:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:50 smithi047 ceph-mon[36835]: pgmap v4957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:10:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:50 smithi029 ceph-mon[32610]: pgmap v4956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:10:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:50 smithi029 ceph-mon[32610]: pgmap v4957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:10:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:52 smithi077 ceph-mon[37153]: pgmap v4958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:10:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8474.552018 secs 2024-07-27T09:10:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:52 smithi047 ceph-mon[36835]: pgmap v4958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:10:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8474.552018 secs 2024-07-27T09:10:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:52 smithi029 ceph-mon[32610]: pgmap v4958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:10:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8474.552018 secs 2024-07-27T09:10:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:54 smithi077 ceph-mon[37153]: pgmap v4959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:10:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:54 smithi047 ceph-mon[36835]: pgmap v4959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:10:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:54 smithi029 ceph-mon[32610]: pgmap v4959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:10:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:56 smithi077 ceph-mon[37153]: pgmap v4960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:10:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:56 smithi047 ceph-mon[36835]: pgmap v4960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:10:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:56 smithi029 ceph-mon[32610]: pgmap v4960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:10:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8479.552149 secs 2024-07-27T09:10:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8479.552149 secs 2024-07-27T09:10:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8479.552149 secs 2024-07-27T09:10:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:10:58 smithi077 ceph-mon[37153]: pgmap v4961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:10:58.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:58.913+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:10:58.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:58.913+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:10:58.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:10:58 smithi029 ceph-mon[32610]: pgmap v4961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:10:58.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:58.914+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:10:58.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:58.914+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:10:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:10:58 smithi047 ceph-mon[36835]: pgmap v4961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:10:59.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:59.720+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:10:59.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:59.720+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:10:59.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:10:59.720+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:10:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:59.806+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:10:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:59.806+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:10:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:10:59.806+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:11:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:00 smithi077 ceph-mon[37153]: pgmap v4962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:11:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:00 smithi047 ceph-mon[36835]: pgmap v4962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:11:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:00 smithi029 ceph-mon[32610]: pgmap v4962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:11:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:02 smithi077 ceph-mon[37153]: pgmap v4963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8484.552301 secs 2024-07-27T09:11:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:02 smithi047 ceph-mon[36835]: pgmap v4963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8484.552301 secs 2024-07-27T09:11:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:02 smithi029 ceph-mon[32610]: pgmap v4963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8484.552301 secs 2024-07-27T09:11:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:04 smithi077 ceph-mon[37153]: pgmap v4964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:04 smithi047 ceph-mon[36835]: pgmap v4964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:04 smithi029 ceph-mon[32610]: pgmap v4964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:06 smithi077 ceph-mon[37153]: pgmap v4965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:06.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:06 smithi047 ceph-mon[36835]: pgmap v4965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:06 smithi029 ceph-mon[32610]: pgmap v4965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8489.552432 secs 2024-07-27T09:11:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8489.552432 secs 2024-07-27T09:11:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8489.552432 secs 2024-07-27T09:11:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:08 smithi077 ceph-mon[37153]: pgmap v4966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:08 smithi047 ceph-mon[36835]: pgmap v4966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:08 smithi029 ceph-mon[32610]: pgmap v4966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:10 smithi077 ceph-mon[37153]: pgmap v4967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:10 smithi047 ceph-mon[36835]: pgmap v4967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:10 smithi029 ceph-mon[32610]: pgmap v4967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:12.750 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:12 smithi047 ceph-mon[36835]: pgmap v4968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:12.750 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8494.552574 secs 2024-07-27T09:11:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:12 smithi077 ceph-mon[37153]: pgmap v4968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8494.552574 secs 2024-07-27T09:11:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:12 smithi029 ceph-mon[32610]: pgmap v4968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8494.552574 secs 2024-07-27T09:11:14.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:11:14.730+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:11:14.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:11:14.730+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:11:14.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:11:14.730+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:11:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:11:14.820+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:11:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:11:14.820+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:11:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:11:14.820+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:11:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:14 smithi077 ceph-mon[37153]: pgmap v4969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:14 smithi047 ceph-mon[36835]: pgmap v4969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:14 smithi029 ceph-mon[32610]: pgmap v4969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:16 smithi047 ceph-mon[36835]: pgmap v4970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:16 smithi029 ceph-mon[32610]: pgmap v4970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:16 smithi077 ceph-mon[37153]: pgmap v4970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:17.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8499.552688 secs 2024-07-27T09:11:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8499.552688 secs 2024-07-27T09:11:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8499.552688 secs 2024-07-27T09:11:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:18 smithi047 ceph-mon[36835]: pgmap v4971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:18 smithi029 ceph-mon[32610]: pgmap v4971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:18 smithi077 ceph-mon[37153]: pgmap v4971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:20 smithi047 ceph-mon[36835]: pgmap v4972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:20 smithi029 ceph-mon[32610]: pgmap v4972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:20 smithi077 ceph-mon[37153]: pgmap v4972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:22 smithi047 ceph-mon[36835]: pgmap v4973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8504.552828 secs 2024-07-27T09:11:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:22 smithi029 ceph-mon[32610]: pgmap v4973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8504.552828 secs 2024-07-27T09:11:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:22 smithi077 ceph-mon[37153]: pgmap v4973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8504.552828 secs 2024-07-27T09:11:24.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:24 smithi047 ceph-mon[36835]: pgmap v4974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:24 smithi029 ceph-mon[32610]: pgmap v4974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:24 smithi077 ceph-mon[37153]: pgmap v4974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:26.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:26 smithi047 ceph-mon[36835]: pgmap v4975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:26 smithi029 ceph-mon[32610]: pgmap v4975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:26 smithi077 ceph-mon[37153]: pgmap v4975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8509.552934 secs 2024-07-27T09:11:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8509.552934 secs 2024-07-27T09:11:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8509.552934 secs 2024-07-27T09:11:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:28 smithi047 ceph-mon[36835]: pgmap v4976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:28 smithi029 ceph-mon[32610]: pgmap v4976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:28 smithi077 ceph-mon[37153]: pgmap v4976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:29.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:11:29.743+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:11:29.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:11:29.743+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:11:29.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:11:29.743+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:11:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:11:29.821+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:11:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:11:29.821+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:11:29.823 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:11:29.821+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:11:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:30 smithi047 ceph-mon[36835]: pgmap v4977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:30 smithi029 ceph-mon[32610]: pgmap v4977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:30 smithi077 ceph-mon[37153]: pgmap v4977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:32 smithi047 ceph-mon[36835]: pgmap v4978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8514.553064 secs 2024-07-27T09:11:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:32 smithi029 ceph-mon[32610]: pgmap v4978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8514.553064 secs 2024-07-27T09:11:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:32 smithi077 ceph-mon[37153]: pgmap v4978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8514.553064 secs 2024-07-27T09:11:34.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:34 smithi047 ceph-mon[36835]: pgmap v4979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:34 smithi029 ceph-mon[32610]: pgmap v4979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:34 smithi077 ceph-mon[37153]: pgmap v4979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:36 smithi047 ceph-mon[36835]: pgmap v4980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:36 smithi029 ceph-mon[32610]: pgmap v4980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:36 smithi077 ceph-mon[37153]: pgmap v4980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8519.553195 secs 2024-07-27T09:11:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8519.553195 secs 2024-07-27T09:11:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8519.553195 secs 2024-07-27T09:11:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:38 smithi047 ceph-mon[36835]: pgmap v4981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:38 smithi029 ceph-mon[32610]: pgmap v4981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:38 smithi077 ceph-mon[37153]: pgmap v4981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:40 smithi047 ceph-mon[36835]: pgmap v4982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:40 smithi029 ceph-mon[32610]: pgmap v4982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:40 smithi077 ceph-mon[37153]: pgmap v4982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:42 smithi047 ceph-mon[36835]: pgmap v4983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8524.553348 secs 2024-07-27T09:11:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:42 smithi029 ceph-mon[32610]: pgmap v4983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8524.553348 secs 2024-07-27T09:11:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:42 smithi077 ceph-mon[37153]: pgmap v4983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8524.553348 secs 2024-07-27T09:11:44.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:11:44.759+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:11:44.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:11:44.759+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:11:44.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:11:44.759+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:11:44.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:11:44.836+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:11:44.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:11:44.836+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:11:44.838 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:11:44.836+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:11:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:44 smithi047 ceph-mon[36835]: pgmap v4984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:44 smithi029 ceph-mon[32610]: pgmap v4984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:44 smithi077 ceph-mon[37153]: pgmap v4984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:11:46.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:46 smithi047 ceph-mon[36835]: pgmap v4985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:46 smithi029 ceph-mon[32610]: pgmap v4985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:46 smithi077 ceph-mon[37153]: pgmap v4985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:11:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8529.553447 secs 2024-07-27T09:11:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8529.553447 secs 2024-07-27T09:11:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8529.553447 secs 2024-07-27T09:11:48.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:48 smithi047 ceph-mon[36835]: pgmap v4986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:48.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:48 smithi029 ceph-mon[32610]: pgmap v4986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:48 smithi077 ceph-mon[37153]: pgmap v4986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:11:49.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:11:49.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:11:49.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:11:49.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:11:49.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:11:49.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:11:49.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:11:49.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:11:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:11:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:11:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:11:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:11:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:50 smithi047 ceph-mon[36835]: pgmap v4987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:50 smithi047 ceph-mon[36835]: pgmap v4988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:11:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:50 smithi047 ceph-mon[36835]: pgmap v4989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:11:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:50 smithi029 ceph-mon[32610]: pgmap v4987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:50 smithi029 ceph-mon[32610]: pgmap v4988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:11:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:50 smithi029 ceph-mon[32610]: pgmap v4989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:11:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:50 smithi077 ceph-mon[37153]: pgmap v4987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:11:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:50 smithi077 ceph-mon[37153]: pgmap v4988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:11:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:50 smithi077 ceph-mon[37153]: pgmap v4989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:11:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:52 smithi047 ceph-mon[36835]: pgmap v4990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:11:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8534.553514 secs 2024-07-27T09:11:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:52 smithi029 ceph-mon[32610]: pgmap v4990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:11:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8534.553514 secs 2024-07-27T09:11:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:52 smithi077 ceph-mon[37153]: pgmap v4990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:11:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8534.553514 secs 2024-07-27T09:11:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:54 smithi047 ceph-mon[36835]: pgmap v4991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:11:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:54 smithi029 ceph-mon[32610]: pgmap v4991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:11:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:54 smithi077 ceph-mon[37153]: pgmap v4991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:11:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:56 smithi029 ceph-mon[32610]: pgmap v4992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:11:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:56 smithi077 ceph-mon[37153]: pgmap v4992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:11:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:56 smithi047 ceph-mon[36835]: pgmap v4992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:11:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8539.553641 secs 2024-07-27T09:11:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8539.553641 secs 2024-07-27T09:11:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8539.553641 secs 2024-07-27T09:11:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:11:58 smithi029 ceph-mon[32610]: pgmap v4993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:11:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:11:58 smithi077 ceph-mon[37153]: pgmap v4993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:11:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:11:58 smithi047 ceph-mon[36835]: pgmap v4993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:11:59.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:11:59.775+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:11:59.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:11:59.775+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:11:59.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:11:59.775+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:11:59.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:11:59.852+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:11:59.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:11:59.852+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:11:59.854 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:11:59.852+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:12:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:00 smithi029 ceph-mon[32610]: pgmap v4994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:12:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:00 smithi077 ceph-mon[37153]: pgmap v4994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:12:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:00 smithi047 ceph-mon[36835]: pgmap v4994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:12:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:02 smithi077 ceph-mon[37153]: pgmap v4995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8544.553760 secs 2024-07-27T09:12:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:02 smithi047 ceph-mon[36835]: pgmap v4995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8544.553760 secs 2024-07-27T09:12:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:02 smithi029 ceph-mon[32610]: pgmap v4995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8544.553760 secs 2024-07-27T09:12:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:04 smithi077 ceph-mon[37153]: pgmap v4996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:04 smithi047 ceph-mon[36835]: pgmap v4996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:04 smithi029 ceph-mon[32610]: pgmap v4996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:06 smithi077 ceph-mon[37153]: pgmap v4997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:06 smithi047 ceph-mon[36835]: pgmap v4997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:06 smithi029 ceph-mon[32610]: pgmap v4997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8549.553881 secs 2024-07-27T09:12:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8549.553881 secs 2024-07-27T09:12:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8549.553881 secs 2024-07-27T09:12:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:08 smithi077 ceph-mon[37153]: pgmap v4998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:09.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:08 smithi047 ceph-mon[36835]: pgmap v4998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:08 smithi029 ceph-mon[32610]: pgmap v4998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:10 smithi077 ceph-mon[37153]: pgmap v4999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:11.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:10 smithi047 ceph-mon[36835]: pgmap v4999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:10 smithi029 ceph-mon[32610]: pgmap v4999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:12.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:12 smithi047 ceph-mon[36835]: pgmap v5000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:12.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8554.553979 secs 2024-07-27T09:12:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:12 smithi077 ceph-mon[37153]: pgmap v5000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8554.553979 secs 2024-07-27T09:12:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:12 smithi029 ceph-mon[32610]: pgmap v5000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8554.553979 secs 2024-07-27T09:12:14.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:12:14.789+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:12:14.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:12:14.789+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:12:14.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:12:14.789+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:12:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:12:14.867+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:12:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:12:14.867+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:12:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:12:14.867+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:12:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:14 smithi077 ceph-mon[37153]: pgmap v5001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:15.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:14 smithi047 ceph-mon[36835]: pgmap v5001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:14 smithi029 ceph-mon[32610]: pgmap v5001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:16 smithi077 ceph-mon[37153]: pgmap v5002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:16 smithi047 ceph-mon[36835]: pgmap v5002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:16 smithi029 ceph-mon[32610]: pgmap v5002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8559.554103 secs 2024-07-27T09:12:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8559.554103 secs 2024-07-27T09:12:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8559.554103 secs 2024-07-27T09:12:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:18 smithi077 ceph-mon[37153]: pgmap v5003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:18 smithi047 ceph-mon[36835]: pgmap v5003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:18 smithi029 ceph-mon[32610]: pgmap v5003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:20 smithi077 ceph-mon[37153]: pgmap v5004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:20 smithi047 ceph-mon[36835]: pgmap v5004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:20 smithi029 ceph-mon[32610]: pgmap v5004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:22 smithi077 ceph-mon[37153]: pgmap v5005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8564.554223 secs 2024-07-27T09:12:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:22 smithi047 ceph-mon[36835]: pgmap v5005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8564.554223 secs 2024-07-27T09:12:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:22 smithi029 ceph-mon[32610]: pgmap v5005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8564.554223 secs 2024-07-27T09:12:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:24 smithi077 ceph-mon[37153]: pgmap v5006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:24 smithi047 ceph-mon[36835]: pgmap v5006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:24 smithi029 ceph-mon[32610]: pgmap v5006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:26 smithi077 ceph-mon[37153]: pgmap v5007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:26 smithi047 ceph-mon[36835]: pgmap v5007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:26 smithi029 ceph-mon[32610]: pgmap v5007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8569.554350 secs 2024-07-27T09:12:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8569.554350 secs 2024-07-27T09:12:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8569.554350 secs 2024-07-27T09:12:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:28 smithi077 ceph-mon[37153]: pgmap v5008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:29.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:28 smithi029 ceph-mon[32610]: pgmap v5008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:28 smithi047 ceph-mon[36835]: pgmap v5008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:12:29.797+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:12:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:12:29.797+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:12:29.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:12:29.797+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:12:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:12:29.880+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:12:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:12:29.880+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:12:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:12:29.880+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:12:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:30 smithi077 ceph-mon[37153]: pgmap v5009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:30 smithi047 ceph-mon[36835]: pgmap v5009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:30 smithi029 ceph-mon[32610]: pgmap v5009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:32 smithi077 ceph-mon[37153]: pgmap v5010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8574.554489 secs 2024-07-27T09:12:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:32 smithi047 ceph-mon[36835]: pgmap v5010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8574.554489 secs 2024-07-27T09:12:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:32 smithi029 ceph-mon[32610]: pgmap v5010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8574.554489 secs 2024-07-27T09:12:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:34 smithi077 ceph-mon[37153]: pgmap v5011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:35.112 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:34 smithi029 ceph-mon[32610]: pgmap v5011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:34 smithi047 ceph-mon[36835]: pgmap v5011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:36 smithi077 ceph-mon[37153]: pgmap v5012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:37.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:36 smithi047 ceph-mon[36835]: pgmap v5012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:36 smithi029 ceph-mon[32610]: pgmap v5012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8579.554633 secs 2024-07-27T09:12:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8579.554633 secs 2024-07-27T09:12:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8579.554633 secs 2024-07-27T09:12:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:38 smithi077 ceph-mon[37153]: pgmap v5013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:38 smithi047 ceph-mon[36835]: pgmap v5013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:38 smithi029 ceph-mon[32610]: pgmap v5013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:40 smithi077 ceph-mon[37153]: pgmap v5014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:40 smithi047 ceph-mon[36835]: pgmap v5014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:40 smithi029 ceph-mon[32610]: pgmap v5014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:42 smithi077 ceph-mon[37153]: pgmap v5015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8584.554758 secs 2024-07-27T09:12:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:42 smithi047 ceph-mon[36835]: pgmap v5015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8584.554758 secs 2024-07-27T09:12:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:42 smithi029 ceph-mon[32610]: pgmap v5015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:12:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8584.554758 secs 2024-07-27T09:12:44.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:12:44.813+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:12:44.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:12:44.813+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:12:44.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:12:44.813+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:12:44.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:12:44.896+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:12:44.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:12:44.896+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:12:44.897 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:12:44.896+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:12:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:44 smithi077 ceph-mon[37153]: pgmap v5016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:44 smithi047 ceph-mon[36835]: pgmap v5016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:44 smithi029 ceph-mon[32610]: pgmap v5016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:12:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:46 smithi077 ceph-mon[37153]: pgmap v5017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:47.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:46 smithi047 ceph-mon[36835]: pgmap v5017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:46 smithi029 ceph-mon[32610]: pgmap v5017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:12:48.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8589.554865 secs 2024-07-27T09:12:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8589.554865 secs 2024-07-27T09:12:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8589.554865 secs 2024-07-27T09:12:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:48 smithi077 ceph-mon[37153]: pgmap v5018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:48 smithi047 ceph-mon[36835]: pgmap v5018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:48 smithi029 ceph-mon[32610]: pgmap v5018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:12:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:12:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:12:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:12:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:50 smithi077 ceph-mon[37153]: pgmap v5019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:12:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:12:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:12:51.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:50 smithi047 ceph-mon[36835]: pgmap v5019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:51.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:12:51.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:12:51.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:12:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:50 smithi029 ceph-mon[32610]: pgmap v5019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:12:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:12:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:12:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:12:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:51 smithi077 ceph-mon[37153]: pgmap v5020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:12:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:51 smithi077 ceph-mon[37153]: pgmap v5021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:12:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:51 smithi047 ceph-mon[36835]: pgmap v5020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:12:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:51 smithi047 ceph-mon[36835]: pgmap v5021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:12:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:51 smithi029 ceph-mon[32610]: pgmap v5020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:12:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:51 smithi029 ceph-mon[32610]: pgmap v5021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:12:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8594.554984 secs 2024-07-27T09:12:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8594.554984 secs 2024-07-27T09:12:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8594.554984 secs 2024-07-27T09:12:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:53 smithi077 ceph-mon[37153]: pgmap v5022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:12:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:53 smithi047 ceph-mon[36835]: pgmap v5022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:12:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:53 smithi029 ceph-mon[32610]: pgmap v5022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:12:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:55 smithi077 ceph-mon[37153]: pgmap v5023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:12:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:55 smithi047 ceph-mon[36835]: pgmap v5023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:12:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:55 smithi029 ceph-mon[32610]: pgmap v5023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:12:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:57 smithi077 ceph-mon[37153]: pgmap v5024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:12:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8599.555109 secs 2024-07-27T09:12:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:57 smithi047 ceph-mon[36835]: pgmap v5024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:12:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8599.555109 secs 2024-07-27T09:12:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:57 smithi029 ceph-mon[32610]: pgmap v5024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:12:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8599.555109 secs 2024-07-27T09:12:59.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:12:59.814+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:12:59.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:12:59.814+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:12:59.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:12:59.814+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:12:59.912 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:12:59.912+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:12:59.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:12:59.912+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:12:59.913 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:12:59.912+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:13:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:12:59 smithi077 ceph-mon[37153]: pgmap v5025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:13:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:12:59 smithi047 ceph-mon[36835]: pgmap v5025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:13:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:12:59 smithi029 ceph-mon[32610]: pgmap v5025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:13:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:01 smithi077 ceph-mon[37153]: pgmap v5026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:13:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:01 smithi047 ceph-mon[36835]: pgmap v5026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:13:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:01 smithi029 ceph-mon[32610]: pgmap v5026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:13:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8604.555186 secs 2024-07-27T09:13:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8604.555186 secs 2024-07-27T09:13:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8604.555186 secs 2024-07-27T09:13:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:03 smithi077 ceph-mon[37153]: pgmap v5027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:03 smithi047 ceph-mon[36835]: pgmap v5027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:03 smithi029 ceph-mon[32610]: pgmap v5027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:05 smithi077 ceph-mon[37153]: pgmap v5028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:05 smithi047 ceph-mon[36835]: pgmap v5028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:05 smithi029 ceph-mon[32610]: pgmap v5028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:07 smithi077 ceph-mon[37153]: pgmap v5029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8609.555307 secs 2024-07-27T09:13:08.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:07 smithi047 ceph-mon[36835]: pgmap v5029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:08.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8609.555307 secs 2024-07-27T09:13:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:07 smithi029 ceph-mon[32610]: pgmap v5029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8609.555307 secs 2024-07-27T09:13:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:09 smithi047 ceph-mon[36835]: pgmap v5030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:13:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:09 smithi029 ceph-mon[32610]: pgmap v5030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:13:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:09 smithi077 ceph-mon[37153]: pgmap v5030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:13:12.158 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:11 smithi077 ceph-mon[37153]: pgmap v5031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:11 smithi047 ceph-mon[36835]: pgmap v5031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:11 smithi029 ceph-mon[32610]: pgmap v5031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8614.555455 secs 2024-07-27T09:13:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8614.555455 secs 2024-07-27T09:13:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8614.555455 secs 2024-07-27T09:13:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:13 smithi047 ceph-mon[36835]: pgmap v5032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:13 smithi029 ceph-mon[32610]: pgmap v5032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:13 smithi077 ceph-mon[37153]: pgmap v5032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:14.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:14.828+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:13:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:14.828+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:13:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:14.828+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:13:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:14.926+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:13:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:14.926+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:13:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:14.926+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:13:16.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:15 smithi047 ceph-mon[36835]: pgmap v5033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:15 smithi029 ceph-mon[32610]: pgmap v5033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:15 smithi077 ceph-mon[37153]: pgmap v5033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:17 smithi047 ceph-mon[36835]: pgmap v5034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8619.555556 secs 2024-07-27T09:13:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:17 smithi029 ceph-mon[32610]: pgmap v5034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8619.555556 secs 2024-07-27T09:13:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:17 smithi077 ceph-mon[37153]: pgmap v5034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8619.555556 secs 2024-07-27T09:13:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:19 smithi047 ceph-mon[36835]: pgmap v5035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:13:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:19 smithi029 ceph-mon[32610]: pgmap v5035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:13:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:19 smithi077 ceph-mon[37153]: pgmap v5035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:13:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:21 smithi047 ceph-mon[36835]: pgmap v5036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:21 smithi029 ceph-mon[32610]: pgmap v5036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:21 smithi077 ceph-mon[37153]: pgmap v5036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8624.555702 secs 2024-07-27T09:13:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8624.555702 secs 2024-07-27T09:13:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8624.555702 secs 2024-07-27T09:13:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:23 smithi047 ceph-mon[36835]: pgmap v5037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:23 smithi029 ceph-mon[32610]: pgmap v5037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:23 smithi077 ceph-mon[37153]: pgmap v5037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:25 smithi047 ceph-mon[36835]: pgmap v5038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:25 smithi029 ceph-mon[32610]: pgmap v5038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:25 smithi077 ceph-mon[37153]: pgmap v5038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:27 smithi047 ceph-mon[36835]: pgmap v5039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8629.555826 secs 2024-07-27T09:13:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:27 smithi029 ceph-mon[32610]: pgmap v5039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8629.555826 secs 2024-07-27T09:13:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:27 smithi077 ceph-mon[37153]: pgmap v5039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8629.555826 secs 2024-07-27T09:13:29.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:29.829+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:13:29.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:29.829+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:13:29.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:29.829+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:13:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:29.936+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:13:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:29.936+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:13:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:29.936+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:13:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:29 smithi047 ceph-mon[36835]: pgmap v5040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:13:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:29 smithi029 ceph-mon[32610]: pgmap v5040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:13:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:29 smithi077 ceph-mon[37153]: pgmap v5040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:13:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:31 smithi047 ceph-mon[36835]: pgmap v5041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:31 smithi029 ceph-mon[32610]: pgmap v5041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:31 smithi077 ceph-mon[37153]: pgmap v5041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8634.555952 secs 2024-07-27T09:13:33.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8634.555952 secs 2024-07-27T09:13:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8634.555952 secs 2024-07-27T09:13:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:33 smithi047 ceph-mon[36835]: pgmap v5042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:33 smithi029 ceph-mon[32610]: pgmap v5042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:33 smithi077 ceph-mon[37153]: pgmap v5042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:35 smithi047 ceph-mon[36835]: pgmap v5043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:35 smithi029 ceph-mon[32610]: pgmap v5043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:35 smithi077 ceph-mon[37153]: pgmap v5043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:37 smithi047 ceph-mon[36835]: pgmap v5044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8639.556077 secs 2024-07-27T09:13:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:37 smithi029 ceph-mon[32610]: pgmap v5044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8639.556077 secs 2024-07-27T09:13:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:37 smithi077 ceph-mon[37153]: pgmap v5044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8639.556077 secs 2024-07-27T09:13:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:39 smithi047 ceph-mon[36835]: pgmap v5045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:13:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:39 smithi029 ceph-mon[32610]: pgmap v5045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:13:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:39 smithi077 ceph-mon[37153]: pgmap v5045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:13:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:41 smithi047 ceph-mon[36835]: pgmap v5046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:41 smithi029 ceph-mon[32610]: pgmap v5046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:41 smithi077 ceph-mon[37153]: pgmap v5046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:13:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8644.556233 secs 2024-07-27T09:13:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8644.556233 secs 2024-07-27T09:13:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8644.556233 secs 2024-07-27T09:13:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:43.937+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:13:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:43.937+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:13:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:43.938+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:13:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:43.938+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:13:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:43 smithi047 ceph-mon[36835]: pgmap v5047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:43 smithi029 ceph-mon[32610]: pgmap v5047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:43 smithi077 ceph-mon[37153]: pgmap v5047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:44.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:44.833+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:13:44.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:44.833+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:13:44.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:44.833+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:13:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:44.938+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:13:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:44.938+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:13:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:44.938+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:13:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:45 smithi047 ceph-mon[36835]: pgmap v5048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:45 smithi029 ceph-mon[32610]: pgmap v5048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:45 smithi077 ceph-mon[37153]: pgmap v5048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:47 smithi047 ceph-mon[36835]: pgmap v5049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8649.556379 secs 2024-07-27T09:13:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:47 smithi029 ceph-mon[32610]: pgmap v5049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8649.556379 secs 2024-07-27T09:13:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:47 smithi077 ceph-mon[37153]: pgmap v5049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8649.556379 secs 2024-07-27T09:13:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:49 smithi047 ceph-mon[36835]: pgmap v5050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:13:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:49 smithi029 ceph-mon[32610]: pgmap v5050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:13:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:49 smithi077 ceph-mon[37153]: pgmap v5050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:13:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:13:51.205 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:13:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:13:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:51 smithi029 ceph-mon[32610]: pgmap v5051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:13:52.296 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:51 smithi047 ceph-mon[36835]: pgmap v5051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:13:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:51 smithi077 ceph-mon[37153]: pgmap v5051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:13:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8654.556504 secs 2024-07-27T09:13:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:52 smithi029 ceph-mon[32610]: pgmap v5052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8654.556504 secs 2024-07-27T09:13:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:52 smithi077 ceph-mon[37153]: pgmap v5052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:53.388 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8654.556504 secs 2024-07-27T09:13:53.388 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:52 smithi047 ceph-mon[36835]: pgmap v5052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:13:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: pgmap v5053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:13:55.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:13:55.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:13:55.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:13:55.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:13:55.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:13:55.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:13:55.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:13:55.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: pgmap v5054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:13:55.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: pgmap v5055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T09:13:55.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: pgmap v5053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:55.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:13:55.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:13:55.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:13:55.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:13:55.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:13:55.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:13:55.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:13:55.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:13:55.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: pgmap v5054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:13:55.465 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: pgmap v5055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T09:13:55.465 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: pgmap v5053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:13:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:13:55.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:13:55.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:13:55.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:13:55.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:13:55.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:13:55.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:13:55.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:13:55.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: pgmap v5054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:13:55.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: pgmap v5055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T09:13:55.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:13:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:57 smithi047 ceph-mon[36835]: pgmap v5056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T09:13:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8659.556594 secs 2024-07-27T09:13:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:57 smithi029 ceph-mon[32610]: pgmap v5056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T09:13:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8659.556594 secs 2024-07-27T09:13:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:57 smithi077 ceph-mon[37153]: pgmap v5056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T09:13:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8659.556594 secs 2024-07-27T09:13:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:58.940+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:13:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:58.940+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:13:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:58.941+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:13:58.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:58.941+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:13:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:13:59 smithi047 ceph-mon[36835]: pgmap v5057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:13:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:13:59 smithi029 ceph-mon[32610]: pgmap v5057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:13:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:59.838+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:13:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:59.838+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:13:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:13:59.838+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:13:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:13:59 smithi077 ceph-mon[37153]: pgmap v5057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:13:59.955 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:59.954+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:13:59.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:59.954+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:13:59.956 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:13:59.954+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:14:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:01 smithi047 ceph-mon[36835]: pgmap v5058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T09:14:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:01 smithi029 ceph-mon[32610]: pgmap v5058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T09:14:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:01 smithi077 ceph-mon[37153]: pgmap v5058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T09:14:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8664.556720 secs 2024-07-27T09:14:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8664.556720 secs 2024-07-27T09:14:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8664.556720 secs 2024-07-27T09:14:03.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:03 smithi047 ceph-mon[36835]: pgmap v5059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T09:14:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:03 smithi029 ceph-mon[32610]: pgmap v5059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T09:14:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:03 smithi077 ceph-mon[37153]: pgmap v5059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T09:14:05.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:05 smithi047 ceph-mon[36835]: pgmap v5060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:14:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:05 smithi029 ceph-mon[32610]: pgmap v5060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:14:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:05 smithi077 ceph-mon[37153]: pgmap v5060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:14:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:07 smithi077 ceph-mon[37153]: pgmap v5061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8669.556868 secs 2024-07-27T09:14:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:07 smithi047 ceph-mon[36835]: pgmap v5061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8669.556868 secs 2024-07-27T09:14:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:07 smithi029 ceph-mon[32610]: pgmap v5061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8669.556868 secs 2024-07-27T09:14:09.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:09 smithi047 ceph-mon[36835]: pgmap v5062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:09 smithi029 ceph-mon[32610]: pgmap v5062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:09 smithi077 ceph-mon[37153]: pgmap v5062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:11 smithi047 ceph-mon[36835]: pgmap v5063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:14:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:11 smithi029 ceph-mon[32610]: pgmap v5063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:14:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:11 smithi077 ceph-mon[37153]: pgmap v5063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:14:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8674.556987 secs 2024-07-27T09:14:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8674.556987 secs 2024-07-27T09:14:12.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8674.556987 secs 2024-07-27T09:14:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:13 smithi047 ceph-mon[36835]: pgmap v5064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:14:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:13 smithi029 ceph-mon[32610]: pgmap v5064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:14:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:13 smithi077 ceph-mon[37153]: pgmap v5064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:14:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:14:14.852+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:14:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:14:14.852+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:14:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:14:14.852+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:14:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:14:14.969+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:14:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:14:14.969+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:14:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:14:14.969+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:14:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:15 smithi047 ceph-mon[36835]: pgmap v5065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:14:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:15 smithi029 ceph-mon[32610]: pgmap v5065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:14:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:15 smithi077 ceph-mon[37153]: pgmap v5065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:14:17.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:17 smithi047 ceph-mon[36835]: pgmap v5066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:17.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8679.557128 secs 2024-07-27T09:14:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:17 smithi029 ceph-mon[32610]: pgmap v5066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8679.557128 secs 2024-07-27T09:14:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:17 smithi077 ceph-mon[37153]: pgmap v5066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8679.557128 secs 2024-07-27T09:14:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:19 smithi047 ceph-mon[36835]: pgmap v5067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:19 smithi029 ceph-mon[32610]: pgmap v5067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:19 smithi077 ceph-mon[37153]: pgmap v5067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:21 smithi047 ceph-mon[36835]: pgmap v5068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:14:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:21 smithi029 ceph-mon[32610]: pgmap v5068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:14:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:21 smithi077 ceph-mon[37153]: pgmap v5068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:14:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8684.557244 secs 2024-07-27T09:14:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8684.557244 secs 2024-07-27T09:14:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8684.557244 secs 2024-07-27T09:14:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:23 smithi047 ceph-mon[36835]: pgmap v5069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:14:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:23 smithi029 ceph-mon[32610]: pgmap v5069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:14:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:23 smithi077 ceph-mon[37153]: pgmap v5069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:14:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:25 smithi047 ceph-mon[36835]: pgmap v5070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:14:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:25 smithi029 ceph-mon[32610]: pgmap v5070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:14:25.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:25 smithi077 ceph-mon[37153]: pgmap v5070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:14:27.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:27 smithi047 ceph-mon[36835]: pgmap v5071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:27.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8689.557378 secs 2024-07-27T09:14:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:27 smithi029 ceph-mon[32610]: pgmap v5071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8689.557378 secs 2024-07-27T09:14:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:27 smithi077 ceph-mon[37153]: pgmap v5071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8689.557378 secs 2024-07-27T09:14:29.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:29 smithi047 ceph-mon[36835]: pgmap v5072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:29 smithi029 ceph-mon[32610]: pgmap v5072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:29 smithi077 ceph-mon[37153]: pgmap v5072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:29.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:14:29.868+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:14:29.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:14:29.868+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:14:29.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:14:29.868+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:14:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:14:29.978+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:14:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:14:29.978+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:14:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:14:29.978+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:14:31.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:31 smithi047 ceph-mon[36835]: pgmap v5073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:14:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:31 smithi029 ceph-mon[32610]: pgmap v5073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:14:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:31 smithi077 ceph-mon[37153]: pgmap v5073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:14:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8694.557464 secs 2024-07-27T09:14:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8694.557464 secs 2024-07-27T09:14:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8694.557464 secs 2024-07-27T09:14:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:33 smithi047 ceph-mon[36835]: pgmap v5074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:14:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:33 smithi029 ceph-mon[32610]: pgmap v5074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:14:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:33 smithi077 ceph-mon[37153]: pgmap v5074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:14:35.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:35 smithi047 ceph-mon[36835]: pgmap v5075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:14:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:35 smithi029 ceph-mon[32610]: pgmap v5075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:14:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:35 smithi077 ceph-mon[37153]: pgmap v5075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:14:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:37 smithi047 ceph-mon[36835]: pgmap v5076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8699.557604 secs 2024-07-27T09:14:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:37 smithi029 ceph-mon[32610]: pgmap v5076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8699.557604 secs 2024-07-27T09:14:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:37 smithi077 ceph-mon[37153]: pgmap v5076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8699.557604 secs 2024-07-27T09:14:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:39 smithi047 ceph-mon[36835]: pgmap v5077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:39 smithi029 ceph-mon[32610]: pgmap v5077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:39 smithi077 ceph-mon[37153]: pgmap v5077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:41 smithi047 ceph-mon[36835]: pgmap v5078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:14:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:41 smithi029 ceph-mon[32610]: pgmap v5078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:14:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:41 smithi077 ceph-mon[37153]: pgmap v5078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:14:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8704.557710 secs 2024-07-27T09:14:42.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8704.557710 secs 2024-07-27T09:14:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8704.557710 secs 2024-07-27T09:14:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:43 smithi029 ceph-mon[32610]: pgmap v5079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:14:43.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:43 smithi047 ceph-mon[36835]: pgmap v5079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:14:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:43 smithi077 ceph-mon[37153]: pgmap v5079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:14:44.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:14:44.884+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:14:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:14:44.884+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:14:44.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:14:44.884+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:14:44.987+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:14:44.987+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:14:44.987+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:14:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:45 smithi029 ceph-mon[32610]: pgmap v5080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:14:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:45 smithi077 ceph-mon[37153]: pgmap v5080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:14:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:45 smithi047 ceph-mon[36835]: pgmap v5080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:14:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:47 smithi029 ceph-mon[32610]: pgmap v5081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8709.557837 secs 2024-07-27T09:14:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:47 smithi077 ceph-mon[37153]: pgmap v5081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8709.557837 secs 2024-07-27T09:14:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:47 smithi047 ceph-mon[36835]: pgmap v5081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8709.557837 secs 2024-07-27T09:14:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:49 smithi029 ceph-mon[32610]: pgmap v5082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:14:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:49 smithi077 ceph-mon[37153]: pgmap v5082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:14:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:49 smithi047 ceph-mon[36835]: pgmap v5082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:14:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:51 smithi077 ceph-mon[37153]: pgmap v5083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:14:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:51 smithi047 ceph-mon[36835]: pgmap v5083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:14:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:51 smithi029 ceph-mon[32610]: pgmap v5083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:14:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8714.557952 secs 2024-07-27T09:14:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8714.557952 secs 2024-07-27T09:14:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8714.557952 secs 2024-07-27T09:14:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:53 smithi077 ceph-mon[37153]: pgmap v5084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:14:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:53 smithi047 ceph-mon[36835]: pgmap v5084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:14:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:53 smithi029 ceph-mon[32610]: pgmap v5084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:14:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:14:54.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:14:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:14:55.592 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:55 smithi077 ceph-mon[37153]: pgmap v5085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:14:55.592 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:55.592 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:55.592 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:55.592 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:55.592 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:55 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:14:55.592 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:55 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:14:55.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:55 smithi047 ceph-mon[36835]: pgmap v5085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:14:55.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:55.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:55.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:55.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:55.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:55 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:14:55.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:55 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:14:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:55 smithi029 ceph-mon[32610]: pgmap v5085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:14:55.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:55.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:55.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:55.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:55.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:55 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:14:55.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:55 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:14:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:57 smithi077 ceph-mon[37153]: pgmap v5086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8719.558092 secs 2024-07-27T09:14:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:57 smithi047 ceph-mon[36835]: pgmap v5086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8719.558092 secs 2024-07-27T09:14:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:57 smithi029 ceph-mon[32610]: pgmap v5086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:14:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8719.558092 secs 2024-07-27T09:14:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:14:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:14:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:59.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:14:59.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:14:59.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:14:59.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:14:59.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: pgmap v5087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:59.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: pgmap v5088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:14:59.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:14:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:14:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:14:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:14:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:14:59.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:14:59.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:14:59.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:14:59.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: pgmap v5087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:59.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: pgmap v5088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:14:59.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:14:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:14:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:14:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:14:59.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:59.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:59.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:59.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:14:59.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:14:59.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:14:59.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:14:59.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:14:59.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: pgmap v5087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:14:59.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: pgmap v5088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:14:59.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:14:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:14:59.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:14:59.894+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:14:59.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:14:59.894+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:14:59.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:14:59.894+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:14:59.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:14:59.998+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:14:59.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:14:59.998+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:14:59.999 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:14:59.998+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:15:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:01 smithi047 ceph-mon[36835]: pgmap v5089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:15:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:01 smithi029 ceph-mon[32610]: pgmap v5089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:15:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:01 smithi077 ceph-mon[37153]: pgmap v5089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:15:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8724.558214 secs 2024-07-27T09:15:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8724.558214 secs 2024-07-27T09:15:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8724.558214 secs 2024-07-27T09:15:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:03 smithi047 ceph-mon[36835]: pgmap v5090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:15:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:03 smithi029 ceph-mon[32610]: pgmap v5090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:15:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:03 smithi077 ceph-mon[37153]: pgmap v5090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:15:05.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:05 smithi047 ceph-mon[36835]: pgmap v5091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:15:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:05 smithi029 ceph-mon[32610]: pgmap v5091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:15:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:05 smithi077 ceph-mon[37153]: pgmap v5091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:15:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:07 smithi047 ceph-mon[36835]: pgmap v5092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:15:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8729.558320 secs 2024-07-27T09:15:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:07 smithi029 ceph-mon[32610]: pgmap v5092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:15:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8729.558320 secs 2024-07-27T09:15:07.745 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:07 smithi077 ceph-mon[37153]: pgmap v5092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T09:15:07.745 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8729.558320 secs 2024-07-27T09:15:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:09 smithi047 ceph-mon[36835]: pgmap v5093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:15:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:09 smithi029 ceph-mon[32610]: pgmap v5093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:15:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:09 smithi077 ceph-mon[37153]: pgmap v5093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:15:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:11 smithi047 ceph-mon[36835]: pgmap v5094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:11 smithi029 ceph-mon[32610]: pgmap v5094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:11 smithi077 ceph-mon[37153]: pgmap v5094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8734.558429 secs 2024-07-27T09:15:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8734.558429 secs 2024-07-27T09:15:12.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8734.558429 secs 2024-07-27T09:15:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:13 smithi047 ceph-mon[36835]: pgmap v5095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:13 smithi029 ceph-mon[32610]: pgmap v5095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:13 smithi077 ceph-mon[37153]: pgmap v5095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:14.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:15:14.908+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:15:14.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:15:14.908+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:15:14.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:15:14.908+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:15:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:15:15.001+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:15:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:15:15.001+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:15:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:15:15.001+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:15:15.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:15 smithi047 ceph-mon[36835]: pgmap v5096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:15 smithi029 ceph-mon[32610]: pgmap v5096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:15 smithi077 ceph-mon[37153]: pgmap v5096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:17 smithi047 ceph-mon[36835]: pgmap v5097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:17.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8739.558567 secs 2024-07-27T09:15:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:17 smithi029 ceph-mon[32610]: pgmap v5097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8739.558567 secs 2024-07-27T09:15:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:17 smithi077 ceph-mon[37153]: pgmap v5097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8739.558567 secs 2024-07-27T09:15:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:19 smithi047 ceph-mon[36835]: pgmap v5098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:19 smithi029 ceph-mon[32610]: pgmap v5098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:19 smithi077 ceph-mon[37153]: pgmap v5098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:21 smithi047 ceph-mon[36835]: pgmap v5099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:21 smithi029 ceph-mon[32610]: pgmap v5099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:21 smithi077 ceph-mon[37153]: pgmap v5099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8744.558711 secs 2024-07-27T09:15:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8744.558711 secs 2024-07-27T09:15:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8744.558711 secs 2024-07-27T09:15:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:23 smithi047 ceph-mon[36835]: pgmap v5100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:23 smithi029 ceph-mon[32610]: pgmap v5100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:23 smithi077 ceph-mon[37153]: pgmap v5100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:25 smithi047 ceph-mon[36835]: pgmap v5101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:25 smithi029 ceph-mon[32610]: pgmap v5101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:25 smithi077 ceph-mon[37153]: pgmap v5101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:27 smithi047 ceph-mon[36835]: pgmap v5102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8749.558831 secs 2024-07-27T09:15:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:27 smithi029 ceph-mon[32610]: pgmap v5102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8749.558831 secs 2024-07-27T09:15:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:27 smithi077 ceph-mon[37153]: pgmap v5102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8749.558831 secs 2024-07-27T09:15:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:29 smithi047 ceph-mon[36835]: pgmap v5103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:29 smithi029 ceph-mon[32610]: pgmap v5103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:29 smithi077 ceph-mon[37153]: pgmap v5103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:15:29.917+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:15:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:15:29.917+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:15:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:15:29.917+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:15:30.018 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:15:30.017+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:15:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:15:30.017+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:15:30.019 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:15:30.017+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:15:31.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:31 smithi047 ceph-mon[36835]: pgmap v5104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:31 smithi029 ceph-mon[32610]: pgmap v5104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:31 smithi077 ceph-mon[37153]: pgmap v5104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8754.558949 secs 2024-07-27T09:15:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8754.558949 secs 2024-07-27T09:15:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8754.558949 secs 2024-07-27T09:15:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:33 smithi047 ceph-mon[36835]: pgmap v5105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:33 smithi029 ceph-mon[32610]: pgmap v5105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:33 smithi077 ceph-mon[37153]: pgmap v5105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:35 smithi047 ceph-mon[36835]: pgmap v5106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:35 smithi029 ceph-mon[32610]: pgmap v5106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:35 smithi077 ceph-mon[37153]: pgmap v5106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:37 smithi047 ceph-mon[36835]: pgmap v5107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8759.559047 secs 2024-07-27T09:15:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:37 smithi029 ceph-mon[32610]: pgmap v5107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8759.559047 secs 2024-07-27T09:15:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:37 smithi077 ceph-mon[37153]: pgmap v5107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8759.559047 secs 2024-07-27T09:15:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:39 smithi047 ceph-mon[36835]: pgmap v5108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:39 smithi029 ceph-mon[32610]: pgmap v5108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:39 smithi077 ceph-mon[37153]: pgmap v5108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:41 smithi047 ceph-mon[36835]: pgmap v5109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:41 smithi029 ceph-mon[32610]: pgmap v5109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:41 smithi077 ceph-mon[37153]: pgmap v5109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8764.559168 secs 2024-07-27T09:15:42.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8764.559168 secs 2024-07-27T09:15:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8764.559168 secs 2024-07-27T09:15:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:43 smithi029 ceph-mon[32610]: pgmap v5110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:43.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:43 smithi047 ceph-mon[36835]: pgmap v5110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:43 smithi077 ceph-mon[37153]: pgmap v5110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:44.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:15:44.933+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:15:44.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:15:44.933+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:15:44.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:15:44.933+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:15:45.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:15:45.032+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:15:45.033 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:15:45.032+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:15:45.034 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:15:45.032+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:15:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:45 smithi029 ceph-mon[32610]: pgmap v5111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:45 smithi077 ceph-mon[37153]: pgmap v5111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:45 smithi047 ceph-mon[36835]: pgmap v5111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:47 smithi029 ceph-mon[32610]: pgmap v5112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8769.559233 secs 2024-07-27T09:15:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:47 smithi077 ceph-mon[37153]: pgmap v5112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8769.559233 secs 2024-07-27T09:15:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:47 smithi047 ceph-mon[36835]: pgmap v5112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8769.559233 secs 2024-07-27T09:15:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:49 smithi077 ceph-mon[37153]: pgmap v5113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:49 smithi047 ceph-mon[36835]: pgmap v5113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:49 smithi029 ceph-mon[32610]: pgmap v5113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:51 smithi077 ceph-mon[37153]: pgmap v5114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:51 smithi047 ceph-mon[36835]: pgmap v5114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:51 smithi029 ceph-mon[32610]: pgmap v5114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:15:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8774.559357 secs 2024-07-27T09:15:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8774.559357 secs 2024-07-27T09:15:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8774.559357 secs 2024-07-27T09:15:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:53 smithi077 ceph-mon[37153]: pgmap v5115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:53 smithi047 ceph-mon[36835]: pgmap v5115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:53 smithi029 ceph-mon[32610]: pgmap v5115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:15:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:55 smithi077 ceph-mon[37153]: pgmap v5116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:55 smithi047 ceph-mon[36835]: pgmap v5116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:55 smithi029 ceph-mon[32610]: pgmap v5116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:15:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:57 smithi077 ceph-mon[37153]: pgmap v5117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8779.559485 secs 2024-07-27T09:15:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:57 smithi047 ceph-mon[36835]: pgmap v5117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8779.559485 secs 2024-07-27T09:15:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:57 smithi029 ceph-mon[32610]: pgmap v5117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:15:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8779.559485 secs 2024-07-27T09:15:58.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:15:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:15:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:15:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:59 smithi077 ceph-mon[37153]: pgmap v5118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:15:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:15:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:59 smithi077 ceph-mon[37153]: pgmap v5119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:15:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:59 smithi077 ceph-mon[37153]: pgmap v5120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:15:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:15:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:15:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:59 smithi047 ceph-mon[36835]: pgmap v5118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:15:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:15:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:59 smithi047 ceph-mon[36835]: pgmap v5119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:15:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:59 smithi047 ceph-mon[36835]: pgmap v5120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:15:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:15:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:15:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:15:59.949+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:15:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:15:59.949+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:15:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:15:59.949+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:15:59.950 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:59 smithi029 ceph-mon[32610]: pgmap v5118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:15:59.951 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:15:59.951 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:15:59.951 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:59 smithi029 ceph-mon[32610]: pgmap v5119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:15:59.951 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:59 smithi029 ceph-mon[32610]: pgmap v5120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:15:59.951 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:15:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:16:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:00.043+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:16:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:00.043+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:16:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:00.043+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:16:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:01 smithi047 ceph-mon[36835]: pgmap v5121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:16:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:01 smithi029 ceph-mon[32610]: pgmap v5121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:16:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:01 smithi077 ceph-mon[37153]: pgmap v5121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:16:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8784.559613 secs 2024-07-27T09:16:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8784.559613 secs 2024-07-27T09:16:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8784.559613 secs 2024-07-27T09:16:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:03 smithi029 ceph-mon[32610]: pgmap v5122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:16:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:03 smithi077 ceph-mon[37153]: pgmap v5122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:16:04.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:03 smithi047 ceph-mon[36835]: pgmap v5122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:16:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:05 smithi029 ceph-mon[32610]: pgmap v5123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:16:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:05 smithi077 ceph-mon[37153]: pgmap v5123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:16:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:05 smithi047 ceph-mon[36835]: pgmap v5123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:16:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8789.559750 secs 2024-07-27T09:16:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:07 smithi029 ceph-mon[32610]: pgmap v5124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:16:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8789.559750 secs 2024-07-27T09:16:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:07 smithi077 ceph-mon[37153]: pgmap v5124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:16:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8789.559750 secs 2024-07-27T09:16:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:07 smithi047 ceph-mon[36835]: pgmap v5124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:16:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:09 smithi029 ceph-mon[32610]: pgmap v5125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:16:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:09 smithi077 ceph-mon[37153]: pgmap v5125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:16:10.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:09 smithi047 ceph-mon[36835]: pgmap v5125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:16:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:11 smithi077 ceph-mon[37153]: pgmap v5126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:16:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:11 smithi047 ceph-mon[36835]: pgmap v5126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:16:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:11 smithi029 ceph-mon[32610]: pgmap v5126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:16:13.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8794.559891 secs 2024-07-27T09:16:13.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:12 smithi047 ceph-mon[36835]: pgmap v5127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:16:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8794.559891 secs 2024-07-27T09:16:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:12 smithi077 ceph-mon[37153]: pgmap v5127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:16:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8794.559891 secs 2024-07-27T09:16:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:12 smithi029 ceph-mon[32610]: pgmap v5127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:16:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:14.963+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:16:14.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:14.963+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:16:14.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:14.963+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:16:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:15.053+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:16:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:15.053+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:16:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:15.053+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:16:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:15 smithi047 ceph-mon[36835]: pgmap v5128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:16:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:15 smithi029 ceph-mon[32610]: pgmap v5128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:16:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:15 smithi077 ceph-mon[37153]: pgmap v5128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:16:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8799.559959 secs 2024-07-27T09:16:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:17 smithi029 ceph-mon[32610]: pgmap v5129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8799.559959 secs 2024-07-27T09:16:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:17 smithi077 ceph-mon[37153]: pgmap v5129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8799.559959 secs 2024-07-27T09:16:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:17 smithi047 ceph-mon[36835]: pgmap v5129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:19 smithi029 ceph-mon[32610]: pgmap v5130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:16:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:19 smithi077 ceph-mon[37153]: pgmap v5130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:16:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:19 smithi047 ceph-mon[36835]: pgmap v5130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:16:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:21 smithi029 ceph-mon[32610]: pgmap v5131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:21 smithi077 ceph-mon[37153]: pgmap v5131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:21 smithi047 ceph-mon[36835]: pgmap v5131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8804.560101 secs 2024-07-27T09:16:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8804.560101 secs 2024-07-27T09:16:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8804.560101 secs 2024-07-27T09:16:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:23 smithi077 ceph-mon[37153]: pgmap v5132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:16:24.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:23 smithi047 ceph-mon[36835]: pgmap v5132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:16:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:23 smithi029 ceph-mon[32610]: pgmap v5132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:16:25.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:25 smithi077 ceph-mon[37153]: pgmap v5133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:25.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:25 smithi047 ceph-mon[36835]: pgmap v5133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:25 smithi029 ceph-mon[32610]: pgmap v5133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8809.560199 secs 2024-07-27T09:16:28.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:27 smithi029 ceph-mon[32610]: pgmap v5134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8809.560199 secs 2024-07-27T09:16:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:27 smithi077 ceph-mon[37153]: pgmap v5134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8809.560199 secs 2024-07-27T09:16:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:27 smithi047 ceph-mon[36835]: pgmap v5134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:29.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:29.970+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:16:29.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:29.970+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:16:29.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:29.970+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:16:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:30.068+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:30.068+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:16:30.071 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:30.068+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:16:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:29 smithi029 ceph-mon[32610]: pgmap v5135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:16:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:29 smithi077 ceph-mon[37153]: pgmap v5135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:16:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:29 smithi047 ceph-mon[36835]: pgmap v5135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:16:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:31 smithi029 ceph-mon[32610]: pgmap v5136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:31 smithi077 ceph-mon[37153]: pgmap v5136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:31 smithi047 ceph-mon[36835]: pgmap v5136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8814.560302 secs 2024-07-27T09:16:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8814.560302 secs 2024-07-27T09:16:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8814.560302 secs 2024-07-27T09:16:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:33 smithi077 ceph-mon[37153]: pgmap v5137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:16:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:33 smithi047 ceph-mon[36835]: pgmap v5137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:16:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:33 smithi029 ceph-mon[32610]: pgmap v5137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:16:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:35 smithi077 ceph-mon[37153]: pgmap v5138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:35 smithi047 ceph-mon[36835]: pgmap v5138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:35 smithi029 ceph-mon[32610]: pgmap v5138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8819.560427 secs 2024-07-27T09:16:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:37 smithi029 ceph-mon[32610]: pgmap v5139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8819.560427 secs 2024-07-27T09:16:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:37 smithi077 ceph-mon[37153]: pgmap v5139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8819.560427 secs 2024-07-27T09:16:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:37 smithi047 ceph-mon[36835]: pgmap v5139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:39 smithi029 ceph-mon[32610]: pgmap v5140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:16:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:39 smithi077 ceph-mon[37153]: pgmap v5140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:16:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:39 smithi047 ceph-mon[36835]: pgmap v5140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:16:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:41 smithi029 ceph-mon[32610]: pgmap v5141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:41 smithi077 ceph-mon[37153]: pgmap v5141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:41 smithi047 ceph-mon[36835]: pgmap v5141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:43.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8824.560565 secs 2024-07-27T09:16:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8824.560565 secs 2024-07-27T09:16:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8824.560565 secs 2024-07-27T09:16:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:43.965+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:16:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:43.965+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:16:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:43.965+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:16:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:43.965+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:16:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:43 smithi029 ceph-mon[32610]: pgmap v5142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:16:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:43 smithi077 ceph-mon[37153]: pgmap v5142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:16:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:43 smithi047 ceph-mon[36835]: pgmap v5142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:16:44.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:44.977+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:16:44.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:44.977+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:16:44.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:44.977+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:16:45.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:45.080+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:16:45.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:45.080+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:16:45.082 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:16:45.080+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:16:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:44 smithi077 ceph-mon[37153]: pgmap v5143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:44 smithi047 ceph-mon[36835]: pgmap v5143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:44 smithi029 ceph-mon[32610]: pgmap v5143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8829.560706 secs 2024-07-27T09:16:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:47 smithi029 ceph-mon[32610]: pgmap v5144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8829.560706 secs 2024-07-27T09:16:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:47 smithi077 ceph-mon[37153]: pgmap v5144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8829.560706 secs 2024-07-27T09:16:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:47 smithi047 ceph-mon[36835]: pgmap v5144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:49 smithi029 ceph-mon[32610]: pgmap v5145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:16:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:49 smithi077 ceph-mon[37153]: pgmap v5145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:16:50.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:49 smithi047 ceph-mon[36835]: pgmap v5145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:16:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:51 smithi077 ceph-mon[37153]: pgmap v5146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:51 smithi047 ceph-mon[36835]: pgmap v5146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:51 smithi029 ceph-mon[32610]: pgmap v5146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8834.560830 secs 2024-07-27T09:16:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8834.560830 secs 2024-07-27T09:16:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8834.560830 secs 2024-07-27T09:16:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:53 smithi077 ceph-mon[37153]: pgmap v5147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:16:54.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:53 smithi047 ceph-mon[36835]: pgmap v5147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:16:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:53 smithi029 ceph-mon[32610]: pgmap v5147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:16:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:54 smithi077 ceph-mon[37153]: pgmap v5148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:54 smithi047 ceph-mon[36835]: pgmap v5148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:54 smithi029 ceph-mon[32610]: pgmap v5148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:16:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8839.560951 secs 2024-07-27T09:16:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:57 smithi029 ceph-mon[32610]: pgmap v5149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8839.560951 secs 2024-07-27T09:16:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:57 smithi077 ceph-mon[37153]: pgmap v5149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8839.560951 secs 2024-07-27T09:16:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:57 smithi047 ceph-mon[36835]: pgmap v5149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:16:58.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:58.967+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:16:58.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:58.967+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:16:58.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:58.967+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:16:58.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:58.967+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:16:59.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:59.985+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:16:59.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:59.985+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:16:59.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:16:59.985+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:17:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:17:00.086+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:17:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:17:00.086+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:17:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:17:00.086+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:17:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:59 smithi077 ceph-mon[37153]: pgmap v5150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:17:00.367 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:17:00.367 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:17:00.367 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:17:00.367 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:16:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:17:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:59 smithi047 ceph-mon[36835]: pgmap v5150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:17:00.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:17:00.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:17:00.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:17:00.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:16:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:17:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:59 smithi029 ceph-mon[32610]: pgmap v5150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:17:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:17:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:17:00.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:17:00.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:16:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:17:01.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:00 smithi077 ceph-mon[37153]: pgmap v5151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:17:01.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:00 smithi077 ceph-mon[37153]: pgmap v5152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:17:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:00 smithi047 ceph-mon[36835]: pgmap v5151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:17:01.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:00 smithi047 ceph-mon[36835]: pgmap v5152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:17:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:00 smithi029 ceph-mon[32610]: pgmap v5151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:17:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:00 smithi029 ceph-mon[32610]: pgmap v5152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:17:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:02 smithi077 ceph-mon[37153]: pgmap v5153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:17:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:02 smithi047 ceph-mon[36835]: pgmap v5153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:17:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:02 smithi029 ceph-mon[32610]: pgmap v5153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:17:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8844.561078 secs 2024-07-27T09:17:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8844.561078 secs 2024-07-27T09:17:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8844.561078 secs 2024-07-27T09:17:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:04 smithi077 ceph-mon[37153]: pgmap v5154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:17:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:04 smithi047 ceph-mon[36835]: pgmap v5154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:17:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:04 smithi029 ceph-mon[32610]: pgmap v5154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:17:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:06 smithi077 ceph-mon[37153]: pgmap v5155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:17:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:06 smithi047 ceph-mon[36835]: pgmap v5155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:17:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:06 smithi029 ceph-mon[32610]: pgmap v5155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:17:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8849.561192 secs 2024-07-27T09:17:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8849.561192 secs 2024-07-27T09:17:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8849.561192 secs 2024-07-27T09:17:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:08 smithi077 ceph-mon[37153]: pgmap v5156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:17:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:08 smithi047 ceph-mon[36835]: pgmap v5156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:17:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:08 smithi029 ceph-mon[32610]: pgmap v5156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:17:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:10 smithi077 ceph-mon[37153]: pgmap v5157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:17:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:10 smithi047 ceph-mon[36835]: pgmap v5157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:17:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:10 smithi029 ceph-mon[32610]: pgmap v5157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:17:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:12 smithi077 ceph-mon[37153]: pgmap v5158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8854.561319 secs 2024-07-27T09:17:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:12 smithi047 ceph-mon[36835]: pgmap v5158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8854.561319 secs 2024-07-27T09:17:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:12 smithi029 ceph-mon[32610]: pgmap v5158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8854.561319 secs 2024-07-27T09:17:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:14 smithi077 ceph-mon[37153]: pgmap v5159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:17:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:14 smithi047 ceph-mon[36835]: pgmap v5159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:17:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:14 smithi029 ceph-mon[32610]: pgmap v5159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:17:15.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:17:14.999+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:17:15.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:17:14.999+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:17:15.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:17:14.999+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:17:15.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:17:15.090+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:17:15.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:17:15.090+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:17:15.092 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:17:15.090+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:17:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:16 smithi077 ceph-mon[37153]: pgmap v5160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:17:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:16 smithi047 ceph-mon[36835]: pgmap v5160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:17:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:16 smithi029 ceph-mon[32610]: pgmap v5160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:17:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8859.561442 secs 2024-07-27T09:17:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8859.561442 secs 2024-07-27T09:17:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8859.561442 secs 2024-07-27T09:17:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:18 smithi077 ceph-mon[37153]: pgmap v5161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:17:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:18 smithi047 ceph-mon[36835]: pgmap v5161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:17:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:18 smithi029 ceph-mon[32610]: pgmap v5161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:17:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:20 smithi077 ceph-mon[37153]: pgmap v5162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:17:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:20 smithi047 ceph-mon[36835]: pgmap v5162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:17:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:20 smithi029 ceph-mon[32610]: pgmap v5162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:17:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:22 smithi077 ceph-mon[37153]: pgmap v5163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8864.561567 secs 2024-07-27T09:17:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:22 smithi047 ceph-mon[36835]: pgmap v5163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8864.561567 secs 2024-07-27T09:17:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:22 smithi029 ceph-mon[32610]: pgmap v5163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8864.561567 secs 2024-07-27T09:17:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:24 smithi077 ceph-mon[37153]: pgmap v5164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:17:24.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:24 smithi047 ceph-mon[36835]: pgmap v5164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:17:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:24 smithi029 ceph-mon[32610]: pgmap v5164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:17:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:26 smithi077 ceph-mon[37153]: pgmap v5165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:17:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:26 smithi047 ceph-mon[36835]: pgmap v5165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:17:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:26 smithi029 ceph-mon[32610]: pgmap v5165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:17:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8869.561713 secs 2024-07-27T09:17:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8869.561713 secs 2024-07-27T09:17:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8869.561713 secs 2024-07-27T09:17:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:28 smithi077 ceph-mon[37153]: pgmap v5166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:17:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:28 smithi047 ceph-mon[36835]: pgmap v5166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:17:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:28 smithi029 ceph-mon[32610]: pgmap v5166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:17:30.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:17:30.014+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:17:30.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:17:30.014+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:17:30.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:17:30.014+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:17:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:17:30.105+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:17:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:17:30.105+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:17:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:17:30.105+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:17:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:30 smithi077 ceph-mon[37153]: pgmap v5167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:17:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:30 smithi047 ceph-mon[36835]: pgmap v5167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:17:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:30 smithi029 ceph-mon[32610]: pgmap v5167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:17:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:32 smithi077 ceph-mon[37153]: pgmap v5168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8874.561855 secs 2024-07-27T09:17:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:32 smithi047 ceph-mon[36835]: pgmap v5168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8874.561855 secs 2024-07-27T09:17:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:32 smithi029 ceph-mon[32610]: pgmap v5168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8874.561855 secs 2024-07-27T09:17:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:34 smithi077 ceph-mon[37153]: pgmap v5169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:17:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:34 smithi047 ceph-mon[36835]: pgmap v5169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:17:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:34 smithi029 ceph-mon[32610]: pgmap v5169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:17:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:36 smithi077 ceph-mon[37153]: pgmap v5170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:17:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:36 smithi047 ceph-mon[36835]: pgmap v5170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:17:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:36 smithi029 ceph-mon[32610]: pgmap v5170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:17:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8879.561976 secs 2024-07-27T09:17:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8879.561976 secs 2024-07-27T09:17:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8879.561976 secs 2024-07-27T09:17:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:38 smithi077 ceph-mon[37153]: pgmap v5171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:17:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:38 smithi047 ceph-mon[36835]: pgmap v5171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:17:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:38 smithi029 ceph-mon[32610]: pgmap v5171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:17:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:40 smithi077 ceph-mon[37153]: pgmap v5172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:17:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:40 smithi047 ceph-mon[36835]: pgmap v5172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:17:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:40 smithi029 ceph-mon[32610]: pgmap v5172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:17:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:42 smithi077 ceph-mon[37153]: pgmap v5173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8884.562098 secs 2024-07-27T09:17:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:42 smithi047 ceph-mon[36835]: pgmap v5173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8884.562098 secs 2024-07-27T09:17:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:42 smithi029 ceph-mon[32610]: pgmap v5173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8884.562098 secs 2024-07-27T09:17:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:44 smithi077 ceph-mon[37153]: pgmap v5174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:17:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:44 smithi047 ceph-mon[36835]: pgmap v5174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:17:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:44 smithi029 ceph-mon[32610]: pgmap v5174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:17:45.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:17:45.030+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:17:45.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:17:45.030+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:17:45.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:17:45.030+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:17:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:17:45.120+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:17:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:17:45.120+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:17:45.122 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:17:45.120+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:17:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:46 smithi077 ceph-mon[37153]: pgmap v5175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:17:46.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:46 smithi029 ceph-mon[32610]: pgmap v5175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:17:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:46 smithi047 ceph-mon[36835]: pgmap v5175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:17:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8889.562221 secs 2024-07-27T09:17:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8889.562221 secs 2024-07-27T09:17:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8889.562221 secs 2024-07-27T09:17:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:48 smithi077 ceph-mon[37153]: pgmap v5176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:17:48.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:48 smithi047 ceph-mon[36835]: pgmap v5176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:17:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:48 smithi029 ceph-mon[32610]: pgmap v5176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:17:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:50 smithi077 ceph-mon[37153]: pgmap v5177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:17:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:50 smithi047 ceph-mon[36835]: pgmap v5177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:17:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:50 smithi029 ceph-mon[32610]: pgmap v5177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:17:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:52 smithi077 ceph-mon[37153]: pgmap v5178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8894.562344 secs 2024-07-27T09:17:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:52 smithi047 ceph-mon[36835]: pgmap v5178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8894.562344 secs 2024-07-27T09:17:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:52 smithi029 ceph-mon[32610]: pgmap v5178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:17:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8894.562344 secs 2024-07-27T09:17:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:54 smithi077 ceph-mon[37153]: pgmap v5179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:17:54.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:54 smithi047 ceph-mon[36835]: pgmap v5179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:17:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:54 smithi029 ceph-mon[32610]: pgmap v5179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:17:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:56 smithi077 ceph-mon[37153]: pgmap v5180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:17:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:56 smithi047 ceph-mon[36835]: pgmap v5180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:17:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:56 smithi029 ceph-mon[32610]: pgmap v5180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:17:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8899.562466 secs 2024-07-27T09:17:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8899.562466 secs 2024-07-27T09:17:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8899.562466 secs 2024-07-27T09:17:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:58 smithi047 ceph-mon[36835]: pgmap v5181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:17:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:58 smithi029 ceph-mon[32610]: pgmap v5181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:17:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:58 smithi077 ceph-mon[37153]: pgmap v5181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:17:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:17:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:17:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:17:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:18:00.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:00.046+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:18:00.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:00.046+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:18:00.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:00.046+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:18:00.096 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:17:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:18:00.137 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:00.136+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:18:00.137 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:00.136+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:18:00.137 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:00.136+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:18:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:00 smithi047 ceph-mon[36835]: pgmap v5182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:18:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:00 smithi029 ceph-mon[32610]: pgmap v5182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:18:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:00 smithi077 ceph-mon[37153]: pgmap v5182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:18:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:02 smithi077 ceph-mon[37153]: pgmap v5183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8904.562620 secs 2024-07-27T09:18:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:02 smithi047 ceph-mon[36835]: pgmap v5183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8904.562620 secs 2024-07-27T09:18:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:02 smithi029 ceph-mon[32610]: pgmap v5183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:02.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8904.562620 secs 2024-07-27T09:18:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:18:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:18:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:03 smithi077 ceph-mon[37153]: pgmap v5184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:18:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:03 smithi077 ceph-mon[37153]: pgmap v5185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:18:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:18:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:18:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:03 smithi047 ceph-mon[36835]: pgmap v5184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:18:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:03 smithi047 ceph-mon[36835]: pgmap v5185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:18:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:18:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:18:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:03 smithi029 ceph-mon[32610]: pgmap v5184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:18:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:03 smithi029 ceph-mon[32610]: pgmap v5185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:18:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:18:05.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:05 smithi047 ceph-mon[36835]: pgmap v5186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:18:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:05 smithi029 ceph-mon[32610]: pgmap v5186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:18:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:05 smithi077 ceph-mon[37153]: pgmap v5186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:18:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:07 smithi077 ceph-mon[37153]: pgmap v5187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:18:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8909.562750 secs 2024-07-27T09:18:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:07 smithi047 ceph-mon[36835]: pgmap v5187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:18:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8909.562750 secs 2024-07-27T09:18:07.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:07 smithi029 ceph-mon[32610]: pgmap v5187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:18:07.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8909.562750 secs 2024-07-27T09:18:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:09 smithi077 ceph-mon[37153]: pgmap v5188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:18:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:09 smithi047 ceph-mon[36835]: pgmap v5188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:18:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:09 smithi029 ceph-mon[32610]: pgmap v5188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:18:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:11 smithi077 ceph-mon[37153]: pgmap v5189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:18:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:11 smithi047 ceph-mon[36835]: pgmap v5189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:18:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:11 smithi029 ceph-mon[32610]: pgmap v5189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:18:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8914.562880 secs 2024-07-27T09:18:12.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8914.562880 secs 2024-07-27T09:18:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8914.562880 secs 2024-07-27T09:18:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:13 smithi077 ceph-mon[37153]: pgmap v5190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:18:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:13 smithi047 ceph-mon[36835]: pgmap v5190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:18:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:13 smithi029 ceph-mon[32610]: pgmap v5190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:18:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:13.977+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:18:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:13.977+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:18:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:13.977+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:18:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:13.977+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:18:15.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:15.060+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:18:15.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:15.060+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:18:15.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:15.060+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:18:15.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:15.150+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:18:15.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:15.150+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:18:15.152 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:15.150+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:18:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:15 smithi029 ceph-mon[32610]: pgmap v5191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:18:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:15 smithi077 ceph-mon[37153]: pgmap v5191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:18:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:15 smithi047 ceph-mon[36835]: pgmap v5191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:18:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:17 smithi077 ceph-mon[37153]: pgmap v5192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8919.562984 secs 2024-07-27T09:18:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:17 smithi047 ceph-mon[36835]: pgmap v5192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:17.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8919.562984 secs 2024-07-27T09:18:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:17 smithi029 ceph-mon[32610]: pgmap v5192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8919.562984 secs 2024-07-27T09:18:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:19 smithi077 ceph-mon[37153]: pgmap v5193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:18:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:19 smithi047 ceph-mon[36835]: pgmap v5193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:18:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:19 smithi029 ceph-mon[32610]: pgmap v5193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:18:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:21 smithi077 ceph-mon[37153]: pgmap v5194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:21.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:21 smithi047 ceph-mon[36835]: pgmap v5194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:21 smithi029 ceph-mon[32610]: pgmap v5194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8924.563113 secs 2024-07-27T09:18:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8924.563113 secs 2024-07-27T09:18:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8924.563113 secs 2024-07-27T09:18:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:23 smithi077 ceph-mon[37153]: pgmap v5195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:23 smithi047 ceph-mon[36835]: pgmap v5195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:23 smithi029 ceph-mon[32610]: pgmap v5195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:25 smithi077 ceph-mon[37153]: pgmap v5196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:18:25.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:25 smithi047 ceph-mon[36835]: pgmap v5196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:18:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:25 smithi029 ceph-mon[32610]: pgmap v5196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:18:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:27 smithi077 ceph-mon[37153]: pgmap v5197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8929.563232 secs 2024-07-27T09:18:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:27 smithi047 ceph-mon[36835]: pgmap v5197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8929.563232 secs 2024-07-27T09:18:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:27 smithi029 ceph-mon[32610]: pgmap v5197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8929.563232 secs 2024-07-27T09:18:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:28.980+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:18:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:28.980+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:18:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:28.980+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:18:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:28.980+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:18:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:29 smithi077 ceph-mon[37153]: pgmap v5198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:18:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:29 smithi047 ceph-mon[36835]: pgmap v5198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:18:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:29 smithi029 ceph-mon[32610]: pgmap v5198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:18:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:30.075+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:18:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:30.075+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:18:30.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:30.075+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:18:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:30.161+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:18:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:30.161+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:18:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:30.161+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:18:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:31 smithi077 ceph-mon[37153]: pgmap v5199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:31.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:31 smithi047 ceph-mon[36835]: pgmap v5199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:31 smithi029 ceph-mon[32610]: pgmap v5199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8934.563377 secs 2024-07-27T09:18:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8934.563377 secs 2024-07-27T09:18:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8934.563377 secs 2024-07-27T09:18:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:33 smithi077 ceph-mon[37153]: pgmap v5200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:33 smithi047 ceph-mon[36835]: pgmap v5200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:33 smithi029 ceph-mon[32610]: pgmap v5200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:35 smithi077 ceph-mon[37153]: pgmap v5201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:18:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:35 smithi047 ceph-mon[36835]: pgmap v5201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:18:35.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:35 smithi029 ceph-mon[32610]: pgmap v5201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:18:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:37 smithi077 ceph-mon[37153]: pgmap v5202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:18:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8939.563497 secs 2024-07-27T09:18:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:37 smithi047 ceph-mon[36835]: pgmap v5202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:18:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8939.563497 secs 2024-07-27T09:18:37.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:37 smithi029 ceph-mon[32610]: pgmap v5202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:18:37.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8939.563497 secs 2024-07-27T09:18:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:39 smithi047 ceph-mon[36835]: pgmap v5203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:18:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:39 smithi029 ceph-mon[32610]: pgmap v5203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:18:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:39 smithi077 ceph-mon[37153]: pgmap v5203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:18:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:41 smithi047 ceph-mon[36835]: pgmap v5204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:18:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:41 smithi029 ceph-mon[32610]: pgmap v5204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:18:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:41 smithi077 ceph-mon[37153]: pgmap v5204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:18:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8944.563641 secs 2024-07-27T09:18:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8944.563641 secs 2024-07-27T09:18:42.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8944.563641 secs 2024-07-27T09:18:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:43 smithi047 ceph-mon[36835]: pgmap v5205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:43 smithi029 ceph-mon[32610]: pgmap v5205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:43 smithi077 ceph-mon[37153]: pgmap v5205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:45.091+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:18:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:45.091+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:18:45.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:18:45.091+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:18:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:45.171+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:18:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:45.171+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:18:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:18:45.171+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:18:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:45 smithi047 ceph-mon[36835]: pgmap v5206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:18:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:45 smithi029 ceph-mon[32610]: pgmap v5206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:18:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:45 smithi077 ceph-mon[37153]: pgmap v5206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:18:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:47 smithi047 ceph-mon[36835]: pgmap v5207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8949.563773 secs 2024-07-27T09:18:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:47 smithi029 ceph-mon[32610]: pgmap v5207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8949.563773 secs 2024-07-27T09:18:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:47 smithi077 ceph-mon[37153]: pgmap v5207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:47.874 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8949.563773 secs 2024-07-27T09:18:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:49 smithi047 ceph-mon[36835]: pgmap v5208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:18:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:49 smithi029 ceph-mon[32610]: pgmap v5208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:18:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:49 smithi077 ceph-mon[37153]: pgmap v5208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:18:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:51 smithi047 ceph-mon[36835]: pgmap v5209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:51.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:51 smithi029 ceph-mon[32610]: pgmap v5209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:51 smithi077 ceph-mon[37153]: pgmap v5209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8954.563893 secs 2024-07-27T09:18:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8954.563893 secs 2024-07-27T09:18:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8954.563893 secs 2024-07-27T09:18:53.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:53 smithi047 ceph-mon[36835]: pgmap v5210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:53 smithi029 ceph-mon[32610]: pgmap v5210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:53.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:53 smithi077 ceph-mon[37153]: pgmap v5210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:18:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:55 smithi047 ceph-mon[36835]: pgmap v5211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:18:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:55 smithi029 ceph-mon[32610]: pgmap v5211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:18:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:55 smithi077 ceph-mon[37153]: pgmap v5211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:18:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:57 smithi047 ceph-mon[36835]: pgmap v5212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8959.564015 secs 2024-07-27T09:18:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:57 smithi029 ceph-mon[32610]: pgmap v5212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8959.564015 secs 2024-07-27T09:18:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:57 smithi077 ceph-mon[37153]: pgmap v5212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:18:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8959.564015 secs 2024-07-27T09:18:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:18:59 smithi047 ceph-mon[36835]: pgmap v5213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:18:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:18:59 smithi029 ceph-mon[32610]: pgmap v5213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:18:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:18:59 smithi077 ceph-mon[37153]: pgmap v5213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:00.102+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:19:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:00.102+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:19:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:00.102+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:19:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:00.175+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:19:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:00.175+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:19:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:00.175+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:19:01.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:01 smithi047 ceph-mon[36835]: pgmap v5214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:01 smithi029 ceph-mon[32610]: pgmap v5214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:01 smithi077 ceph-mon[37153]: pgmap v5214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8964.564183 secs 2024-07-27T09:19:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:19:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8964.564183 secs 2024-07-27T09:19:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:19:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8964.564183 secs 2024-07-27T09:19:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:19:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:03 smithi077 ceph-mon[37153]: pgmap v5215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:19:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:19:03.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:19:03.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:03 smithi077 ceph-mon[37153]: pgmap v5216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T09:19:03.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:03 smithi047 ceph-mon[36835]: pgmap v5215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:19:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:19:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:19:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:03 smithi047 ceph-mon[36835]: pgmap v5216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T09:19:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:03 smithi029 ceph-mon[32610]: pgmap v5215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:19:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:19:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:19:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:03 smithi029 ceph-mon[32610]: pgmap v5216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T09:19:03.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:19:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:05 smithi077 ceph-mon[37153]: pgmap v5217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:05.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:05 smithi047 ceph-mon[36835]: pgmap v5217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:05 smithi029 ceph-mon[32610]: pgmap v5217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8969.564307 secs 2024-07-27T09:19:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:07 smithi077 ceph-mon[37153]: pgmap v5218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:19:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8969.564307 secs 2024-07-27T09:19:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:07 smithi047 ceph-mon[36835]: pgmap v5218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:19:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8969.564307 secs 2024-07-27T09:19:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:07 smithi029 ceph-mon[32610]: pgmap v5218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:19:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:09 smithi077 ceph-mon[37153]: pgmap v5219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:09.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:09 smithi047 ceph-mon[36835]: pgmap v5219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:09 smithi029 ceph-mon[32610]: pgmap v5219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:11 smithi077 ceph-mon[37153]: pgmap v5220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:19:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:11 smithi047 ceph-mon[36835]: pgmap v5220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:19:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:11 smithi029 ceph-mon[32610]: pgmap v5220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:19:12.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8974.564408 secs 2024-07-27T09:19:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8974.564408 secs 2024-07-27T09:19:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8974.564408 secs 2024-07-27T09:19:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:13 smithi047 ceph-mon[36835]: pgmap v5221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T09:19:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:13 smithi029 ceph-mon[32610]: pgmap v5221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T09:19:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:13 smithi077 ceph-mon[37153]: pgmap v5221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T09:19:15.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:15.104+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:19:15.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:15.104+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:19:15.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:15.104+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:19:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:15.178+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:19:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:15.178+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:19:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:15.178+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:19:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:15 smithi077 ceph-mon[37153]: pgmap v5222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:15 smithi047 ceph-mon[36835]: pgmap v5222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:15 smithi029 ceph-mon[32610]: pgmap v5222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:17 smithi077 ceph-mon[37153]: pgmap v5223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:17 smithi047 ceph-mon[36835]: pgmap v5223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:17 smithi029 ceph-mon[32610]: pgmap v5223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8979.564533 secs 2024-07-27T09:19:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8979.564533 secs 2024-07-27T09:19:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8979.564533 secs 2024-07-27T09:19:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:19 smithi077 ceph-mon[37153]: pgmap v5224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:19 smithi047 ceph-mon[36835]: pgmap v5224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:19 smithi029 ceph-mon[32610]: pgmap v5224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:21 smithi077 ceph-mon[37153]: pgmap v5225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:21 smithi047 ceph-mon[36835]: pgmap v5225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:21 smithi029 ceph-mon[32610]: pgmap v5225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8984.564654 secs 2024-07-27T09:19:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8984.564654 secs 2024-07-27T09:19:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8984.564654 secs 2024-07-27T09:19:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:23 smithi047 ceph-mon[36835]: pgmap v5226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:23 smithi029 ceph-mon[32610]: pgmap v5226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:23 smithi077 ceph-mon[37153]: pgmap v5226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:25 smithi077 ceph-mon[37153]: pgmap v5227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:25 smithi047 ceph-mon[36835]: pgmap v5227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:25 smithi029 ceph-mon[32610]: pgmap v5227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:27 smithi077 ceph-mon[37153]: pgmap v5228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:27 smithi047 ceph-mon[36835]: pgmap v5228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:27 smithi029 ceph-mon[32610]: pgmap v5228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8989.564752 secs 2024-07-27T09:19:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8989.564752 secs 2024-07-27T09:19:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8989.564752 secs 2024-07-27T09:19:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:29 smithi077 ceph-mon[37153]: pgmap v5229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:29.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:29 smithi047 ceph-mon[36835]: pgmap v5229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:29 smithi029 ceph-mon[32610]: pgmap v5229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:30.113+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:19:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:30.113+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:19:30.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:30.113+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:19:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:30.194+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:19:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:30.194+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:19:30.196 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:30.194+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:19:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:31 smithi077 ceph-mon[37153]: pgmap v5230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:31 smithi047 ceph-mon[36835]: pgmap v5230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:31 smithi029 ceph-mon[32610]: pgmap v5230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8994.564828 secs 2024-07-27T09:19:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8994.564828 secs 2024-07-27T09:19:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8994.564828 secs 2024-07-27T09:19:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:33 smithi047 ceph-mon[36835]: pgmap v5231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:33 smithi029 ceph-mon[32610]: pgmap v5231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:33 smithi077 ceph-mon[37153]: pgmap v5231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:35 smithi077 ceph-mon[37153]: pgmap v5232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:35.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:35 smithi047 ceph-mon[36835]: pgmap v5232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:35 smithi029 ceph-mon[32610]: pgmap v5232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:37 smithi077 ceph-mon[37153]: pgmap v5233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:37 smithi047 ceph-mon[36835]: pgmap v5233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:37 smithi029 ceph-mon[32610]: pgmap v5233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 8999.564975 secs 2024-07-27T09:19:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 8999.564975 secs 2024-07-27T09:19:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 8999.564975 secs 2024-07-27T09:19:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:39 smithi077 ceph-mon[37153]: pgmap v5234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:39 smithi047 ceph-mon[36835]: pgmap v5234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:39 smithi029 ceph-mon[32610]: pgmap v5234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:41 smithi077 ceph-mon[37153]: pgmap v5235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:41 smithi047 ceph-mon[36835]: pgmap v5235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:41 smithi029 ceph-mon[32610]: pgmap v5235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9004.565091 secs 2024-07-27T09:19:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9004.565091 secs 2024-07-27T09:19:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9004.565091 secs 2024-07-27T09:19:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:42.644+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-27T09:19:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:42.644+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T09:19:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:42.644+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f733c005890 tx=0x7f733c08a000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T09:19:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:42.644+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f733c005890 tx=0x7f733c08a000 comp rx=0 tx=0).stop 2024-07-27T09:19:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:42.644+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T09:19:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:42.644+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T09:19:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:42.644+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:19:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:42.644+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:19:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:42.644+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:19:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:42.644+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4bc0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f733c019550 tx=0x7f733c05d000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:19:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:43 smithi047 ceph-mon[36835]: pgmap v5236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:43 smithi029 ceph-mon[32610]: pgmap v5236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:43 smithi077 ceph-mon[37153]: pgmap v5236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:43.991+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:19:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:43.991+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:19:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:43.992+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:19:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:43.992+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:19:45.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:45.121+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:19:45.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:45.121+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:19:45.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:45.121+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:19:45.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:45.195+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:19:45.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:45.195+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:19:45.197 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:19:45.195+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:19:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:45 smithi077 ceph-mon[37153]: pgmap v5237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:45.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:45 smithi047 ceph-mon[36835]: pgmap v5237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:45 smithi029 ceph-mon[32610]: pgmap v5237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:47 smithi077 ceph-mon[37153]: pgmap v5238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:47 smithi047 ceph-mon[36835]: pgmap v5238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:47 smithi029 ceph-mon[32610]: pgmap v5238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9009.565232 secs 2024-07-27T09:19:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9009.565232 secs 2024-07-27T09:19:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9009.565232 secs 2024-07-27T09:19:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:49 smithi077 ceph-mon[37153]: pgmap v5239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:49.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:49 smithi047 ceph-mon[36835]: pgmap v5239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:49 smithi029 ceph-mon[32610]: pgmap v5239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:51 smithi029 ceph-mon[32610]: pgmap v5240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:51 smithi077 ceph-mon[37153]: pgmap v5240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:51 smithi047 ceph-mon[36835]: pgmap v5240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9014.565334 secs 2024-07-27T09:19:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9014.565334 secs 2024-07-27T09:19:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9014.565334 secs 2024-07-27T09:19:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:53 smithi047 ceph-mon[36835]: pgmap v5241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:53 smithi029 ceph-mon[32610]: pgmap v5241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:53 smithi077 ceph-mon[37153]: pgmap v5241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:19:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:55 smithi077 ceph-mon[37153]: pgmap v5242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:55 smithi047 ceph-mon[36835]: pgmap v5242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:55 smithi029 ceph-mon[32610]: pgmap v5242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:19:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:57 smithi077 ceph-mon[37153]: pgmap v5243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:57 smithi047 ceph-mon[36835]: pgmap v5243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:57 smithi029 ceph-mon[32610]: pgmap v5243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:19:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:57.661+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000028) for more than 900000000 us, fault. 2024-07-27T09:19:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:57.661+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7eff4001c6a0 tx=0x7eff4002b040 comp rx=0 tx=0).stop 2024-07-27T09:19:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:57.661+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T09:19:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:57.661+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:19:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:57.661+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:19:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:57.662+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:19:57.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:57.662+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b49b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7eff4001cac0 tx=0x7eff4001e040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:19:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9019.565458 secs 2024-07-27T09:19:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9019.565458 secs 2024-07-27T09:19:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9019.565458 secs 2024-07-27T09:19:58.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:58.995+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:19:58.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:58.995+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:19:58.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:58.995+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:19:58.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:19:58.995+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:19:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:19:59 smithi077 ceph-mon[37153]: pgmap v5244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:19:59 smithi047 ceph-mon[36835]: pgmap v5244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:19:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:19:59 smithi029 ceph-mon[32610]: pgmap v5244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:00.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:20:00.124+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:20:00.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:20:00.124+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:20:00.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:20:00.124+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:20:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:20:00.211+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:20:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:20:00.211+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:20:00.213 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:20:00.211+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:20:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:20:00.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:20:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:20:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:01 smithi077 ceph-mon[37153]: pgmap v5245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:01 smithi047 ceph-mon[36835]: pgmap v5245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:01 smithi029 ceph-mon[32610]: pgmap v5245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9024.565550 secs 2024-07-27T09:20:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9024.565550 secs 2024-07-27T09:20:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9024.565550 secs 2024-07-27T09:20:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:03 smithi047 ceph-mon[36835]: pgmap v5246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:20:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:20:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:03 smithi029 ceph-mon[32610]: pgmap v5246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:20:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:20:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:03 smithi077 ceph-mon[37153]: pgmap v5246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:20:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:20:04.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:20:04.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:20:04.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:04 smithi047 ceph-mon[36835]: pgmap v5247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:20:04.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:04 smithi047 ceph-mon[36835]: pgmap v5248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:20:04.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:20:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:20:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:20:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:04 smithi029 ceph-mon[32610]: pgmap v5247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:20:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:04 smithi029 ceph-mon[32610]: pgmap v5248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:20:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:20:04.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:20:04.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:20:04.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:04 smithi077 ceph-mon[37153]: pgmap v5247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:20:04.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:04 smithi077 ceph-mon[37153]: pgmap v5248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:20:04.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:20:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:06 smithi077 ceph-mon[37153]: pgmap v5249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:20:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:06 smithi047 ceph-mon[36835]: pgmap v5249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:20:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:06 smithi029 ceph-mon[32610]: pgmap v5249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:20:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9029.565678 secs 2024-07-27T09:20:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9029.565678 secs 2024-07-27T09:20:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9029.565678 secs 2024-07-27T09:20:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:08 smithi077 ceph-mon[37153]: pgmap v5250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:20:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:08 smithi047 ceph-mon[36835]: pgmap v5250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:20:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:08 smithi029 ceph-mon[32610]: pgmap v5250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:20:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:10 smithi077 ceph-mon[37153]: pgmap v5251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:20:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:10 smithi047 ceph-mon[36835]: pgmap v5251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:20:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:10 smithi029 ceph-mon[32610]: pgmap v5251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:20:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:12 smithi077 ceph-mon[37153]: pgmap v5252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:20:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9034.565818 secs 2024-07-27T09:20:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:12 smithi047 ceph-mon[36835]: pgmap v5252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:20:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9034.565818 secs 2024-07-27T09:20:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:12 smithi029 ceph-mon[32610]: pgmap v5252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:20:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9034.565818 secs 2024-07-27T09:20:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:14 smithi077 ceph-mon[37153]: pgmap v5253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:20:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:14 smithi047 ceph-mon[36835]: pgmap v5253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:20:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:14 smithi029 ceph-mon[32610]: pgmap v5253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:20:15.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:20:15.138+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:20:15.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:20:15.138+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:20:15.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:20:15.138+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:20:15.222+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:20:15.222+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:20:15.222+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:20:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:16 smithi077 ceph-mon[37153]: pgmap v5254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:16 smithi047 ceph-mon[36835]: pgmap v5254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:16 smithi029 ceph-mon[32610]: pgmap v5254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9039.565934 secs 2024-07-27T09:20:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9039.565934 secs 2024-07-27T09:20:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9039.565934 secs 2024-07-27T09:20:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:18 smithi077 ceph-mon[37153]: pgmap v5255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:18 smithi047 ceph-mon[36835]: pgmap v5255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:18 smithi029 ceph-mon[32610]: pgmap v5255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:20 smithi077 ceph-mon[37153]: pgmap v5256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:20 smithi047 ceph-mon[36835]: pgmap v5256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:20 smithi029 ceph-mon[32610]: pgmap v5256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:22 smithi077 ceph-mon[37153]: pgmap v5257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:20:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9044.566024 secs 2024-07-27T09:20:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:22 smithi047 ceph-mon[36835]: pgmap v5257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:20:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9044.566024 secs 2024-07-27T09:20:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:22 smithi029 ceph-mon[32610]: pgmap v5257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:20:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9044.566024 secs 2024-07-27T09:20:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:24 smithi077 ceph-mon[37153]: pgmap v5258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:24 smithi047 ceph-mon[36835]: pgmap v5258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:24 smithi029 ceph-mon[32610]: pgmap v5258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:26 smithi077 ceph-mon[37153]: pgmap v5259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:26 smithi047 ceph-mon[36835]: pgmap v5259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:26 smithi029 ceph-mon[32610]: pgmap v5259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9049.566145 secs 2024-07-27T09:20:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9049.566145 secs 2024-07-27T09:20:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9049.566145 secs 2024-07-27T09:20:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:28 smithi077 ceph-mon[37153]: pgmap v5260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:28 smithi047 ceph-mon[36835]: pgmap v5260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:28 smithi029 ceph-mon[32610]: pgmap v5260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:30.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:20:30.153+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:20:30.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:20:30.153+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:20:30.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:20:30.153+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:20:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:20:30.238+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:20:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:20:30.238+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:20:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:20:30.238+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:20:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:30 smithi077 ceph-mon[37153]: pgmap v5261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:30 smithi047 ceph-mon[36835]: pgmap v5261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:30 smithi029 ceph-mon[32610]: pgmap v5261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:32 smithi077 ceph-mon[37153]: pgmap v5262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:20:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9054.566267 secs 2024-07-27T09:20:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:32 smithi047 ceph-mon[36835]: pgmap v5262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:20:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9054.566267 secs 2024-07-27T09:20:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:32 smithi029 ceph-mon[32610]: pgmap v5262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:20:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9054.566267 secs 2024-07-27T09:20:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:34 smithi077 ceph-mon[37153]: pgmap v5263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:34 smithi047 ceph-mon[36835]: pgmap v5263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:34 smithi029 ceph-mon[32610]: pgmap v5263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:36 smithi077 ceph-mon[37153]: pgmap v5264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:36 smithi047 ceph-mon[36835]: pgmap v5264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:36 smithi029 ceph-mon[32610]: pgmap v5264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9059.566419 secs 2024-07-27T09:20:37.928 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9059.566419 secs 2024-07-27T09:20:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9059.566419 secs 2024-07-27T09:20:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:38 smithi077 ceph-mon[37153]: pgmap v5265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:38 smithi047 ceph-mon[36835]: pgmap v5265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:38 smithi029 ceph-mon[32610]: pgmap v5265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:40 smithi077 ceph-mon[37153]: pgmap v5266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:40 smithi047 ceph-mon[36835]: pgmap v5266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:40 smithi029 ceph-mon[32610]: pgmap v5266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:42 smithi077 ceph-mon[37153]: pgmap v5267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:20:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9064.566557 secs 2024-07-27T09:20:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:42 smithi047 ceph-mon[36835]: pgmap v5267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:20:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9064.566557 secs 2024-07-27T09:20:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:42 smithi029 ceph-mon[32610]: pgmap v5267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:20:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9064.566557 secs 2024-07-27T09:20:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:44 smithi077 ceph-mon[37153]: pgmap v5268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:44 smithi047 ceph-mon[36835]: pgmap v5268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:44 smithi029 ceph-mon[32610]: pgmap v5268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:20:45.169+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:20:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:20:45.169+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:20:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:20:45.169+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:20:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:20:45.247+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:20:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:20:45.247+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:20:45.249 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:20:45.247+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:20:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:46 smithi047 ceph-mon[36835]: pgmap v5269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:46 smithi029 ceph-mon[32610]: pgmap v5269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:46 smithi077 ceph-mon[37153]: pgmap v5269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9069.566687 secs 2024-07-27T09:20:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9069.566687 secs 2024-07-27T09:20:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9069.566687 secs 2024-07-27T09:20:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:48 smithi047 ceph-mon[36835]: pgmap v5270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:48 smithi029 ceph-mon[32610]: pgmap v5270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:48 smithi077 ceph-mon[37153]: pgmap v5270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:50 smithi047 ceph-mon[36835]: pgmap v5271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:50 smithi029 ceph-mon[32610]: pgmap v5271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:50 smithi077 ceph-mon[37153]: pgmap v5271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:20:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:52 smithi047 ceph-mon[36835]: pgmap v5272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:20:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9074.566829 secs 2024-07-27T09:20:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:52 smithi029 ceph-mon[32610]: pgmap v5272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:20:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9074.566829 secs 2024-07-27T09:20:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:52 smithi077 ceph-mon[37153]: pgmap v5272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:20:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9074.566829 secs 2024-07-27T09:20:54.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:54 smithi047 ceph-mon[36835]: pgmap v5273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:54 smithi029 ceph-mon[32610]: pgmap v5273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:54 smithi077 ceph-mon[37153]: pgmap v5273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:20:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:56 smithi047 ceph-mon[36835]: pgmap v5274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:56 smithi029 ceph-mon[32610]: pgmap v5274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:56 smithi077 ceph-mon[37153]: pgmap v5274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:20:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9079.566949 secs 2024-07-27T09:20:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9079.566949 secs 2024-07-27T09:20:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9079.566949 secs 2024-07-27T09:20:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:20:58 smithi047 ceph-mon[36835]: pgmap v5275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:20:58 smithi029 ceph-mon[32610]: pgmap v5275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:20:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:20:58 smithi077 ceph-mon[37153]: pgmap v5275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:21:00.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:21:00.177+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:21:00.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:21:00.177+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:21:00.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:21:00.177+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:21:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:21:00.255+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:21:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:21:00.255+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:21:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:21:00.255+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:21:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:00 smithi047 ceph-mon[36835]: pgmap v5276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:21:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:00 smithi029 ceph-mon[32610]: pgmap v5276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:21:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:00 smithi077 ceph-mon[37153]: pgmap v5276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:21:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:02 smithi077 ceph-mon[37153]: pgmap v5277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:02.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9084.567034 secs 2024-07-27T09:21:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:02 smithi047 ceph-mon[36835]: pgmap v5277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:02.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9084.567034 secs 2024-07-27T09:21:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:02 smithi029 ceph-mon[32610]: pgmap v5277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9084.567034 secs 2024-07-27T09:21:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:04 smithi077 ceph-mon[37153]: pgmap v5278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:21:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:21:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:21:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:21:04.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:21:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:04 smithi047 ceph-mon[36835]: pgmap v5278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:21:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:21:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:21:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:21:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:21:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:04 smithi029 ceph-mon[32610]: pgmap v5278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:21:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:21:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:21:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:21:04.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:21:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:05 smithi077 ceph-mon[37153]: pgmap v5279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:21:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:05 smithi077 ceph-mon[37153]: pgmap v5280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:21:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:05 smithi047 ceph-mon[36835]: pgmap v5279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:21:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:05 smithi047 ceph-mon[36835]: pgmap v5280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:21:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:05 smithi029 ceph-mon[32610]: pgmap v5279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:21:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:05 smithi029 ceph-mon[32610]: pgmap v5280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:21:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:07 smithi077 ceph-mon[37153]: pgmap v5281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:21:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9089.567186 secs 2024-07-27T09:21:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:07 smithi047 ceph-mon[36835]: pgmap v5281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:21:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9089.567186 secs 2024-07-27T09:21:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:07 smithi029 ceph-mon[32610]: pgmap v5281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:21:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9089.567186 secs 2024-07-27T09:21:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:09 smithi077 ceph-mon[37153]: pgmap v5282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:21:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:09 smithi047 ceph-mon[36835]: pgmap v5282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:21:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:09 smithi029 ceph-mon[32610]: pgmap v5282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:21:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:11 smithi077 ceph-mon[37153]: pgmap v5283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:21:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:11 smithi047 ceph-mon[36835]: pgmap v5283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:21:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:11 smithi029 ceph-mon[32610]: pgmap v5283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:21:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9094.567310 secs 2024-07-27T09:21:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9094.567310 secs 2024-07-27T09:21:12.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9094.567310 secs 2024-07-27T09:21:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:13 smithi077 ceph-mon[37153]: pgmap v5284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:21:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:13 smithi047 ceph-mon[36835]: pgmap v5284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:21:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:13 smithi029 ceph-mon[32610]: pgmap v5284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:21:15.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:21:15.191+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:21:15.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:21:15.191+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:21:15.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:21:15.191+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:21:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:21:15.269+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:21:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:21:15.269+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:21:15.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:21:15.269+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:21:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:15 smithi077 ceph-mon[37153]: pgmap v5285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:21:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:15 smithi047 ceph-mon[36835]: pgmap v5285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:21:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:15 smithi029 ceph-mon[32610]: pgmap v5285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:21:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:17 smithi047 ceph-mon[36835]: pgmap v5286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9099.567436 secs 2024-07-27T09:21:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:17 smithi029 ceph-mon[32610]: pgmap v5286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9099.567436 secs 2024-07-27T09:21:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:17 smithi077 ceph-mon[37153]: pgmap v5286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9099.567436 secs 2024-07-27T09:21:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:19 smithi047 ceph-mon[36835]: pgmap v5287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:19 smithi029 ceph-mon[32610]: pgmap v5287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:19 smithi077 ceph-mon[37153]: pgmap v5287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:21 smithi047 ceph-mon[36835]: pgmap v5288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:21 smithi029 ceph-mon[32610]: pgmap v5288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:21 smithi077 ceph-mon[37153]: pgmap v5288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9104.567557 secs 2024-07-27T09:21:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9104.567557 secs 2024-07-27T09:21:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9104.567557 secs 2024-07-27T09:21:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:23 smithi047 ceph-mon[36835]: pgmap v5289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:23 smithi029 ceph-mon[32610]: pgmap v5289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:23 smithi077 ceph-mon[37153]: pgmap v5289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:25.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:25 smithi047 ceph-mon[36835]: pgmap v5290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:21:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:25 smithi029 ceph-mon[32610]: pgmap v5290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:21:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:25 smithi077 ceph-mon[37153]: pgmap v5290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:21:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:27 smithi047 ceph-mon[36835]: pgmap v5291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9109.567683 secs 2024-07-27T09:21:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:27 smithi029 ceph-mon[32610]: pgmap v5291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9109.567683 secs 2024-07-27T09:21:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:27 smithi077 ceph-mon[37153]: pgmap v5291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9109.567683 secs 2024-07-27T09:21:29.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:29 smithi047 ceph-mon[36835]: pgmap v5292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:29 smithi029 ceph-mon[32610]: pgmap v5292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:29 smithi077 ceph-mon[37153]: pgmap v5292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:21:30.200+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:21:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:21:30.200+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:21:30.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:21:30.200+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:21:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:21:30.285+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:21:30.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:21:30.285+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:21:30.287 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:21:30.285+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:21:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:31 smithi047 ceph-mon[36835]: pgmap v5293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:31 smithi029 ceph-mon[32610]: pgmap v5293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:31 smithi077 ceph-mon[37153]: pgmap v5293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9114.567825 secs 2024-07-27T09:21:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9114.567825 secs 2024-07-27T09:21:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9114.567825 secs 2024-07-27T09:21:33.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:33 smithi047 ceph-mon[36835]: pgmap v5294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:33 smithi029 ceph-mon[32610]: pgmap v5294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:33 smithi077 ceph-mon[37153]: pgmap v5294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:35 smithi047 ceph-mon[36835]: pgmap v5295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:21:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:35 smithi029 ceph-mon[32610]: pgmap v5295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:21:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:35 smithi077 ceph-mon[37153]: pgmap v5295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:21:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:37 smithi047 ceph-mon[36835]: pgmap v5296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9119.567909 secs 2024-07-27T09:21:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:37 smithi029 ceph-mon[32610]: pgmap v5296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9119.567909 secs 2024-07-27T09:21:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:37 smithi077 ceph-mon[37153]: pgmap v5296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9119.567909 secs 2024-07-27T09:21:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:39 smithi047 ceph-mon[36835]: pgmap v5297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:39 smithi029 ceph-mon[32610]: pgmap v5297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:39 smithi077 ceph-mon[37153]: pgmap v5297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:41.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:41 smithi047 ceph-mon[36835]: pgmap v5298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:41 smithi029 ceph-mon[32610]: pgmap v5298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:41 smithi077 ceph-mon[37153]: pgmap v5298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9124.568024 secs 2024-07-27T09:21:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9124.568024 secs 2024-07-27T09:21:43.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9124.568024 secs 2024-07-27T09:21:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:43 smithi047 ceph-mon[36835]: pgmap v5299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:43 smithi029 ceph-mon[32610]: pgmap v5299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:44.020 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:43 smithi077 ceph-mon[37153]: pgmap v5299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:45.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:21:45.202+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:21:45.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:21:45.202+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:21:45.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:21:45.202+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:21:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:21:45.293+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:21:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:21:45.293+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:21:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:21:45.293+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:21:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:45 smithi047 ceph-mon[36835]: pgmap v5300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:21:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:45 smithi029 ceph-mon[32610]: pgmap v5300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:21:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:45 smithi077 ceph-mon[37153]: pgmap v5300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:21:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:47 smithi047 ceph-mon[36835]: pgmap v5301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:47.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9129.568143 secs 2024-07-27T09:21:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:47 smithi029 ceph-mon[32610]: pgmap v5301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9129.568143 secs 2024-07-27T09:21:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:47 smithi077 ceph-mon[37153]: pgmap v5301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9129.568143 secs 2024-07-27T09:21:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:49 smithi047 ceph-mon[36835]: pgmap v5302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:49 smithi029 ceph-mon[32610]: pgmap v5302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:49 smithi077 ceph-mon[37153]: pgmap v5302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:51 smithi047 ceph-mon[36835]: pgmap v5303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:51 smithi029 ceph-mon[32610]: pgmap v5303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:51 smithi077 ceph-mon[37153]: pgmap v5303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9134.568267 secs 2024-07-27T09:21:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9134.568267 secs 2024-07-27T09:21:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9134.568267 secs 2024-07-27T09:21:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:53 smithi047 ceph-mon[36835]: pgmap v5304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:53 smithi029 ceph-mon[32610]: pgmap v5304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:53 smithi077 ceph-mon[37153]: pgmap v5304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:21:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:55 smithi047 ceph-mon[36835]: pgmap v5305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:21:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:55 smithi029 ceph-mon[32610]: pgmap v5305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:21:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:55 smithi077 ceph-mon[37153]: pgmap v5305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:21:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:57 smithi047 ceph-mon[36835]: pgmap v5306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9139.568408 secs 2024-07-27T09:21:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:57 smithi029 ceph-mon[32610]: pgmap v5306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9139.568408 secs 2024-07-27T09:21:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:57 smithi077 ceph-mon[37153]: pgmap v5306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:21:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9139.568408 secs 2024-07-27T09:21:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:21:59 smithi047 ceph-mon[36835]: pgmap v5307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:21:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:21:59 smithi029 ceph-mon[32610]: pgmap v5307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:22:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:21:59 smithi077 ceph-mon[37153]: pgmap v5307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:22:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:00.218+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:22:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:00.218+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:22:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:00.218+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:22:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:00.302+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:22:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:00.302+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:22:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:00.302+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:22:01.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:01 smithi047 ceph-mon[36835]: pgmap v5308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:01 smithi029 ceph-mon[32610]: pgmap v5308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:01 smithi077 ceph-mon[37153]: pgmap v5308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9144.568524 secs 2024-07-27T09:22:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9144.568524 secs 2024-07-27T09:22:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9144.568524 secs 2024-07-27T09:22:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:03 smithi029 ceph-mon[32610]: pgmap v5309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:03 smithi077 ceph-mon[37153]: pgmap v5309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:03 smithi047 ceph-mon[36835]: pgmap v5309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:22:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:22:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:22:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:22:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:22:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:22:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:22:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:22:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:22:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:22:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:22:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:22:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:05 smithi029 ceph-mon[32610]: pgmap v5310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:22:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:05 smithi029 ceph-mon[32610]: pgmap v5311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:22:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:05 smithi029 ceph-mon[32610]: pgmap v5312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:22:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:05 smithi077 ceph-mon[37153]: pgmap v5310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:22:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:05 smithi077 ceph-mon[37153]: pgmap v5311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:22:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:05 smithi077 ceph-mon[37153]: pgmap v5312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:22:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:05 smithi047 ceph-mon[36835]: pgmap v5310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:22:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:05 smithi047 ceph-mon[36835]: pgmap v5311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:22:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:05 smithi047 ceph-mon[36835]: pgmap v5312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:22:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:07 smithi047 ceph-mon[36835]: pgmap v5313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:22:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9149.568660 secs 2024-07-27T09:22:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:07 smithi029 ceph-mon[32610]: pgmap v5313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:22:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9149.568660 secs 2024-07-27T09:22:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:07 smithi077 ceph-mon[37153]: pgmap v5313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:22:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9149.568660 secs 2024-07-27T09:22:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:09 smithi047 ceph-mon[36835]: pgmap v5314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:22:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:09 smithi029 ceph-mon[32610]: pgmap v5314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:22:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:09 smithi077 ceph-mon[37153]: pgmap v5314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:22:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:11 smithi047 ceph-mon[36835]: pgmap v5315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:22:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:11 smithi029 ceph-mon[32610]: pgmap v5315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:22:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:11 smithi077 ceph-mon[37153]: pgmap v5315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:22:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9154.568823 secs 2024-07-27T09:22:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9154.568823 secs 2024-07-27T09:22:13.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9154.568823 secs 2024-07-27T09:22:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:13 smithi047 ceph-mon[36835]: pgmap v5316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:22:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:13 smithi029 ceph-mon[32610]: pgmap v5316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:22:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:13 smithi077 ceph-mon[37153]: pgmap v5316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:22:15.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:15.233+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:22:15.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:15.233+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:22:15.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:15.233+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:22:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:15.308+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:22:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:15.308+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:22:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:15.308+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:22:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:15 smithi047 ceph-mon[36835]: pgmap v5317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:22:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:15 smithi029 ceph-mon[32610]: pgmap v5317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:22:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:15 smithi077 ceph-mon[37153]: pgmap v5317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:22:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:17 smithi029 ceph-mon[32610]: pgmap v5318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9159.568885 secs 2024-07-27T09:22:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:17 smithi077 ceph-mon[37153]: pgmap v5318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9159.568885 secs 2024-07-27T09:22:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:17 smithi047 ceph-mon[36835]: pgmap v5318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9159.568885 secs 2024-07-27T09:22:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:19 smithi029 ceph-mon[32610]: pgmap v5319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:22:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:19 smithi077 ceph-mon[37153]: pgmap v5319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:22:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:19 smithi047 ceph-mon[36835]: pgmap v5319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:22:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:21 smithi029 ceph-mon[32610]: pgmap v5320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:21 smithi077 ceph-mon[37153]: pgmap v5320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:21 smithi047 ceph-mon[36835]: pgmap v5320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9164.568971 secs 2024-07-27T09:22:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9164.568971 secs 2024-07-27T09:22:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9164.568971 secs 2024-07-27T09:22:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:23 smithi029 ceph-mon[32610]: pgmap v5321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:22:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:23 smithi077 ceph-mon[37153]: pgmap v5321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:22:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:23 smithi047 ceph-mon[36835]: pgmap v5321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:22:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:25 smithi077 ceph-mon[37153]: pgmap v5322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:22:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:25 smithi047 ceph-mon[36835]: pgmap v5322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:22:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:25 smithi029 ceph-mon[32610]: pgmap v5322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:22:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:27 smithi077 ceph-mon[37153]: pgmap v5323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9169.569098 secs 2024-07-27T09:22:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:27 smithi047 ceph-mon[36835]: pgmap v5323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9169.569098 secs 2024-07-27T09:22:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:27 smithi029 ceph-mon[32610]: pgmap v5323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9169.569098 secs 2024-07-27T09:22:30.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:29 smithi029 ceph-mon[32610]: pgmap v5324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:22:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:29 smithi077 ceph-mon[37153]: pgmap v5324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:22:30.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:29 smithi047 ceph-mon[36835]: pgmap v5324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:22:30.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:30.235+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:22:30.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:30.235+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:22:30.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:30.235+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:22:30.312 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:30.311+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:22:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:30.311+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:22:30.313 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:30.311+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:22:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:31 smithi077 ceph-mon[37153]: pgmap v5325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:31 smithi047 ceph-mon[36835]: pgmap v5325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:31 smithi029 ceph-mon[32610]: pgmap v5325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9174.569163 secs 2024-07-27T09:22:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9174.569163 secs 2024-07-27T09:22:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9174.569163 secs 2024-07-27T09:22:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:33 smithi077 ceph-mon[37153]: pgmap v5326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:22:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:33 smithi047 ceph-mon[36835]: pgmap v5326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:22:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:33 smithi029 ceph-mon[32610]: pgmap v5326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:22:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:35 smithi077 ceph-mon[37153]: pgmap v5327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:22:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:35 smithi047 ceph-mon[36835]: pgmap v5327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:22:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:35 smithi029 ceph-mon[32610]: pgmap v5327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:22:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:37 smithi077 ceph-mon[37153]: pgmap v5328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9179.569228 secs 2024-07-27T09:22:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:37 smithi047 ceph-mon[36835]: pgmap v5328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9179.569228 secs 2024-07-27T09:22:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:37 smithi029 ceph-mon[32610]: pgmap v5328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9179.569228 secs 2024-07-27T09:22:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:39 smithi077 ceph-mon[37153]: pgmap v5329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:22:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:39 smithi047 ceph-mon[36835]: pgmap v5329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:22:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:39 smithi029 ceph-mon[32610]: pgmap v5329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:22:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:41 smithi077 ceph-mon[37153]: pgmap v5330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:41 smithi047 ceph-mon[36835]: pgmap v5330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:41 smithi029 ceph-mon[32610]: pgmap v5330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:43.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9184.569369 secs 2024-07-27T09:22:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9184.569369 secs 2024-07-27T09:22:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9184.569369 secs 2024-07-27T09:22:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:44.021+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:22:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:44.021+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:22:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:44.021+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:22:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:44.021+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:22:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:43 smithi077 ceph-mon[37153]: pgmap v5331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:22:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:43 smithi047 ceph-mon[36835]: pgmap v5331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:22:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:43 smithi029 ceph-mon[32610]: pgmap v5331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:22:45.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:45.251+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:22:45.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:45.251+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:22:45.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:45.251+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:22:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:45.327+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:22:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:45.327+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:22:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:22:45.327+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:22:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:45 smithi077 ceph-mon[37153]: pgmap v5332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:22:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:45 smithi047 ceph-mon[36835]: pgmap v5332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:22:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:45 smithi029 ceph-mon[32610]: pgmap v5332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:22:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:47 smithi077 ceph-mon[37153]: pgmap v5333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9189.569488 secs 2024-07-27T09:22:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:47 smithi047 ceph-mon[36835]: pgmap v5333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9189.569488 secs 2024-07-27T09:22:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:47 smithi029 ceph-mon[32610]: pgmap v5333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9189.569488 secs 2024-07-27T09:22:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:49 smithi077 ceph-mon[37153]: pgmap v5334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:22:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:49 smithi047 ceph-mon[36835]: pgmap v5334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:22:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:49 smithi029 ceph-mon[32610]: pgmap v5334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:22:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:51 smithi077 ceph-mon[37153]: pgmap v5335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:51 smithi047 ceph-mon[36835]: pgmap v5335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:51 smithi029 ceph-mon[32610]: pgmap v5335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:22:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9194.569614 secs 2024-07-27T09:22:53.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9194.569614 secs 2024-07-27T09:22:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9194.569614 secs 2024-07-27T09:22:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:53 smithi077 ceph-mon[37153]: pgmap v5336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:22:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:53 smithi047 ceph-mon[36835]: pgmap v5336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:22:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:53 smithi029 ceph-mon[32610]: pgmap v5336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:22:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:55 smithi077 ceph-mon[37153]: pgmap v5337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:22:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:55 smithi047 ceph-mon[36835]: pgmap v5337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:22:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:55 smithi029 ceph-mon[32610]: pgmap v5337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:22:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:57 smithi077 ceph-mon[37153]: pgmap v5338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9199.569735 secs 2024-07-27T09:22:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:57 smithi047 ceph-mon[36835]: pgmap v5338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9199.569735 secs 2024-07-27T09:22:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:57 smithi029 ceph-mon[32610]: pgmap v5338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:22:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9199.569735 secs 2024-07-27T09:22:59.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:59.023+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:22:59.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:59.023+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:22:59.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:59.024+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:22:59.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:22:59.024+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:23:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:22:59 smithi077 ceph-mon[37153]: pgmap v5339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:22:59 smithi047 ceph-mon[36835]: pgmap v5339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:22:59 smithi029 ceph-mon[32610]: pgmap v5339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:00.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:23:00.261+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:23:00.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:23:00.262+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:23:00.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:23:00.262+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:23:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:23:00.335+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:23:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:23:00.335+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:23:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:23:00.335+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:23:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:01 smithi077 ceph-mon[37153]: pgmap v5340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:01 smithi047 ceph-mon[36835]: pgmap v5340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:01 smithi029 ceph-mon[32610]: pgmap v5340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9204.569858 secs 2024-07-27T09:23:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9204.569858 secs 2024-07-27T09:23:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9204.569858 secs 2024-07-27T09:23:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:03 smithi077 ceph-mon[37153]: pgmap v5341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:23:04.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:03 smithi047 ceph-mon[36835]: pgmap v5341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:23:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:03 smithi029 ceph-mon[32610]: pgmap v5341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:23:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:23:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:23:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:23:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:05 smithi077 ceph-mon[37153]: pgmap v5342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:23:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:23:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:23:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:23:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:05 smithi047 ceph-mon[36835]: pgmap v5342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:23:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:23:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:23:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:23:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:05 smithi029 ceph-mon[32610]: pgmap v5342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:23:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:23:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:23:06.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:23:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:06 smithi077 ceph-mon[37153]: pgmap v5343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:23:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:06 smithi077 ceph-mon[37153]: pgmap v5344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:23:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:06 smithi047 ceph-mon[36835]: pgmap v5343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:23:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:06 smithi047 ceph-mon[36835]: pgmap v5344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:23:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:06 smithi029 ceph-mon[32610]: pgmap v5343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:23:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:06 smithi029 ceph-mon[32610]: pgmap v5344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:23:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9209.569998 secs 2024-07-27T09:23:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9209.569998 secs 2024-07-27T09:23:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9209.569998 secs 2024-07-27T09:23:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:08 smithi077 ceph-mon[37153]: pgmap v5345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:23:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:08 smithi047 ceph-mon[36835]: pgmap v5345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:23:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:08 smithi029 ceph-mon[32610]: pgmap v5345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:23:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:10 smithi077 ceph-mon[37153]: pgmap v5346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:23:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:10 smithi047 ceph-mon[36835]: pgmap v5346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:23:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:10 smithi029 ceph-mon[32610]: pgmap v5346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:23:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:12 smithi077 ceph-mon[37153]: pgmap v5347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:23:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9214.570064 secs 2024-07-27T09:23:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:12 smithi047 ceph-mon[36835]: pgmap v5347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:23:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9214.570064 secs 2024-07-27T09:23:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:12 smithi029 ceph-mon[32610]: pgmap v5347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:23:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9214.570064 secs 2024-07-27T09:23:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:14 smithi077 ceph-mon[37153]: pgmap v5348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:23:15.201 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:14 smithi047 ceph-mon[36835]: pgmap v5348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:23:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:14 smithi029 ceph-mon[32610]: pgmap v5348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:23:15.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:23:15.276+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:23:15.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:23:15.276+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:23:15.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:23:15.276+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:23:15.346 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:23:15.345+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:23:15.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:23:15.345+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:23:15.347 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:23:15.345+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:23:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:16 smithi077 ceph-mon[37153]: pgmap v5349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:23:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:16 smithi047 ceph-mon[36835]: pgmap v5349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:23:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:16 smithi029 ceph-mon[32610]: pgmap v5349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:23:18.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9219.570187 secs 2024-07-27T09:23:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9219.570187 secs 2024-07-27T09:23:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9219.570187 secs 2024-07-27T09:23:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:18 smithi077 ceph-mon[37153]: pgmap v5350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:18 smithi047 ceph-mon[36835]: pgmap v5350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:18 smithi029 ceph-mon[32610]: pgmap v5350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:20 smithi077 ceph-mon[37153]: pgmap v5351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:23:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:20 smithi047 ceph-mon[36835]: pgmap v5351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:23:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:20 smithi029 ceph-mon[32610]: pgmap v5351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:23:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:22 smithi077 ceph-mon[37153]: pgmap v5352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9224.570309 secs 2024-07-27T09:23:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:22 smithi047 ceph-mon[36835]: pgmap v5352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9224.570309 secs 2024-07-27T09:23:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:22 smithi029 ceph-mon[32610]: pgmap v5352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9224.570309 secs 2024-07-27T09:23:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:24 smithi077 ceph-mon[37153]: pgmap v5353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:24 smithi047 ceph-mon[36835]: pgmap v5353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:24 smithi029 ceph-mon[32610]: pgmap v5353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:26 smithi077 ceph-mon[37153]: pgmap v5354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:26 smithi047 ceph-mon[36835]: pgmap v5354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:26 smithi029 ceph-mon[32610]: pgmap v5354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9229.570450 secs 2024-07-27T09:23:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9229.570450 secs 2024-07-27T09:23:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9229.570450 secs 2024-07-27T09:23:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:28 smithi077 ceph-mon[37153]: pgmap v5355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:29.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:28 smithi029 ceph-mon[32610]: pgmap v5355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:28 smithi047 ceph-mon[36835]: pgmap v5355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:30.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:23:30.292+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:23:30.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:23:30.292+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:23:30.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:23:30.292+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:23:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:23:30.360+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:23:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:23:30.360+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:23:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:23:30.360+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:23:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:30 smithi047 ceph-mon[36835]: pgmap v5356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:23:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:30 smithi029 ceph-mon[32610]: pgmap v5356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:23:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:30 smithi077 ceph-mon[37153]: pgmap v5356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:23:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:32 smithi047 ceph-mon[36835]: pgmap v5357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9234.570569 secs 2024-07-27T09:23:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:32 smithi029 ceph-mon[32610]: pgmap v5357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9234.570569 secs 2024-07-27T09:23:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:32 smithi077 ceph-mon[37153]: pgmap v5357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9234.570569 secs 2024-07-27T09:23:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:34 smithi047 ceph-mon[36835]: pgmap v5358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:34 smithi029 ceph-mon[32610]: pgmap v5358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:34 smithi077 ceph-mon[37153]: pgmap v5358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:36 smithi047 ceph-mon[36835]: pgmap v5359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:36 smithi029 ceph-mon[32610]: pgmap v5359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:36 smithi077 ceph-mon[37153]: pgmap v5359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9239.570667 secs 2024-07-27T09:23:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9239.570667 secs 2024-07-27T09:23:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9239.570667 secs 2024-07-27T09:23:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:38 smithi047 ceph-mon[36835]: pgmap v5360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:38 smithi029 ceph-mon[32610]: pgmap v5360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:38 smithi077 ceph-mon[37153]: pgmap v5360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:41.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:40 smithi047 ceph-mon[36835]: pgmap v5361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:23:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:40 smithi029 ceph-mon[32610]: pgmap v5361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:23:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:40 smithi077 ceph-mon[37153]: pgmap v5361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:23:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:42 smithi047 ceph-mon[36835]: pgmap v5362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9244.570769 secs 2024-07-27T09:23:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:42 smithi029 ceph-mon[32610]: pgmap v5362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9244.570769 secs 2024-07-27T09:23:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:42 smithi077 ceph-mon[37153]: pgmap v5362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9244.570769 secs 2024-07-27T09:23:45.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:44 smithi077 ceph-mon[37153]: pgmap v5363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:44 smithi047 ceph-mon[36835]: pgmap v5363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:44 smithi029 ceph-mon[32610]: pgmap v5363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:45.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:23:45.307+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:23:45.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:23:45.307+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:23:45.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:23:45.307+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:23:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:23:45.366+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:23:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:23:45.366+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:23:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:23:45.366+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:23:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:46 smithi047 ceph-mon[36835]: pgmap v5364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:46 smithi029 ceph-mon[32610]: pgmap v5364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:46 smithi077 ceph-mon[37153]: pgmap v5364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:48.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9249.570841 secs 2024-07-27T09:23:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9249.570841 secs 2024-07-27T09:23:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9249.570841 secs 2024-07-27T09:23:49.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:48 smithi047 ceph-mon[36835]: pgmap v5365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:48 smithi029 ceph-mon[32610]: pgmap v5365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:48 smithi077 ceph-mon[37153]: pgmap v5365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:50 smithi047 ceph-mon[36835]: pgmap v5366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:23:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:50 smithi029 ceph-mon[32610]: pgmap v5366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:23:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:50 smithi077 ceph-mon[37153]: pgmap v5366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:23:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:52 smithi047 ceph-mon[36835]: pgmap v5367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9254.570931 secs 2024-07-27T09:23:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:52 smithi029 ceph-mon[32610]: pgmap v5367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9254.570931 secs 2024-07-27T09:23:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:52 smithi077 ceph-mon[37153]: pgmap v5367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9254.570931 secs 2024-07-27T09:23:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:54 smithi047 ceph-mon[36835]: pgmap v5368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:54 smithi029 ceph-mon[32610]: pgmap v5368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:54 smithi077 ceph-mon[37153]: pgmap v5368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:23:57.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:56 smithi047 ceph-mon[36835]: pgmap v5369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:56 smithi029 ceph-mon[32610]: pgmap v5369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:56 smithi077 ceph-mon[37153]: pgmap v5369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:23:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9259.571028 secs 2024-07-27T09:23:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9259.571028 secs 2024-07-27T09:23:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9259.571028 secs 2024-07-27T09:23:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:23:58 smithi047 ceph-mon[36835]: pgmap v5370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:23:58 smithi029 ceph-mon[32610]: pgmap v5370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:23:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:23:58 smithi077 ceph-mon[37153]: pgmap v5370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:00.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:24:00.323+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:24:00.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:24:00.323+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:24:00.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:24:00.323+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:24:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:24:00.382+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:24:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:24:00.382+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:24:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:24:00.382+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:24:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:00 smithi047 ceph-mon[36835]: pgmap v5371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:24:01.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:00 smithi029 ceph-mon[32610]: pgmap v5371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:24:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:00 smithi077 ceph-mon[37153]: pgmap v5371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:24:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:02 smithi047 ceph-mon[36835]: pgmap v5372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9264.571169 secs 2024-07-27T09:24:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:02 smithi029 ceph-mon[32610]: pgmap v5372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9264.571169 secs 2024-07-27T09:24:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:02 smithi077 ceph-mon[37153]: pgmap v5372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9264.571169 secs 2024-07-27T09:24:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:04 smithi047 ceph-mon[36835]: pgmap v5373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:04 smithi029 ceph-mon[32610]: pgmap v5373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:04 smithi077 ceph-mon[37153]: pgmap v5373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:24:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:24:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:24:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:24:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:24:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:24:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:24:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:24:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:24:06.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:24:06.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:24:06.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:24:06.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:24:06.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:24:06.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:24:06.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:24:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:24:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:24:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:24:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:24:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:24:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:24:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:24:06.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:24:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:06 smithi047 ceph-mon[36835]: pgmap v5374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:06 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:24:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:06 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:24:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:06 smithi047 ceph-mon[36835]: pgmap v5375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:24:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:06 smithi047 ceph-mon[36835]: pgmap v5376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:24:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:06 smithi029 ceph-mon[32610]: pgmap v5374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:06 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:24:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:06 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:24:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:06 smithi029 ceph-mon[32610]: pgmap v5375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:24:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:06 smithi029 ceph-mon[32610]: pgmap v5376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:24:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:06 smithi077 ceph-mon[37153]: pgmap v5374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:06 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:24:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:06 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:24:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:06 smithi077 ceph-mon[37153]: pgmap v5375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:24:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:06 smithi077 ceph-mon[37153]: pgmap v5376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:24:08.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9269.571306 secs 2024-07-27T09:24:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9269.571306 secs 2024-07-27T09:24:08.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9269.571306 secs 2024-07-27T09:24:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:08 smithi047 ceph-mon[36835]: pgmap v5377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:24:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:08 smithi029 ceph-mon[32610]: pgmap v5377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:24:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:08 smithi077 ceph-mon[37153]: pgmap v5377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:24:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:10 smithi029 ceph-mon[32610]: pgmap v5378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:24:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:10 smithi077 ceph-mon[37153]: pgmap v5378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:24:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:10 smithi047 ceph-mon[36835]: pgmap v5378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:24:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:12 smithi029 ceph-mon[32610]: pgmap v5379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:24:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9274.571434 secs 2024-07-27T09:24:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:12 smithi077 ceph-mon[37153]: pgmap v5379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:24:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9274.571434 secs 2024-07-27T09:24:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:12 smithi047 ceph-mon[36835]: pgmap v5379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:24:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9274.571434 secs 2024-07-27T09:24:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:14 smithi029 ceph-mon[32610]: pgmap v5380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:24:15.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:24:15.337+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:24:15.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:24:15.337+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:24:15.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:24:15.337+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:24:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:14 smithi077 ceph-mon[37153]: pgmap v5380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:24:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:24:15.396+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:24:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:24:15.396+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:24:15.398 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:24:15.396+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:24:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:14 smithi047 ceph-mon[36835]: pgmap v5380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:24:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:16 smithi029 ceph-mon[32610]: pgmap v5381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:24:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:16 smithi077 ceph-mon[37153]: pgmap v5381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:24:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:16 smithi047 ceph-mon[36835]: pgmap v5381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:24:18.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9279.571554 secs 2024-07-27T09:24:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9279.571554 secs 2024-07-27T09:24:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9279.571554 secs 2024-07-27T09:24:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:18 smithi077 ceph-mon[37153]: pgmap v5382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:18 smithi047 ceph-mon[36835]: pgmap v5382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:18 smithi029 ceph-mon[32610]: pgmap v5382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:20 smithi077 ceph-mon[37153]: pgmap v5383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:24:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:20 smithi047 ceph-mon[36835]: pgmap v5383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:24:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:20 smithi029 ceph-mon[32610]: pgmap v5383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:24:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:22 smithi077 ceph-mon[37153]: pgmap v5384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9284.571674 secs 2024-07-27T09:24:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:22 smithi047 ceph-mon[36835]: pgmap v5384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9284.571674 secs 2024-07-27T09:24:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:22 smithi029 ceph-mon[32610]: pgmap v5384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9284.571674 secs 2024-07-27T09:24:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:24 smithi077 ceph-mon[37153]: pgmap v5385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:24 smithi047 ceph-mon[36835]: pgmap v5385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:24 smithi029 ceph-mon[32610]: pgmap v5385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:26 smithi077 ceph-mon[37153]: pgmap v5386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:26 smithi047 ceph-mon[36835]: pgmap v5386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:26 smithi029 ceph-mon[32610]: pgmap v5386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9289.571816 secs 2024-07-27T09:24:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9289.571816 secs 2024-07-27T09:24:28.465 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9289.571816 secs 2024-07-27T09:24:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:28 smithi077 ceph-mon[37153]: pgmap v5387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:28 smithi047 ceph-mon[36835]: pgmap v5387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:28 smithi029 ceph-mon[32610]: pgmap v5387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:30.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:24:30.349+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:24:30.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:24:30.349+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:24:30.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:24:30.349+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:24:30.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:24:30.412+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:24:30.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:24:30.412+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:24:30.413 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:24:30.412+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:24:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:30 smithi029 ceph-mon[32610]: pgmap v5388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:24:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:30 smithi077 ceph-mon[37153]: pgmap v5388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:24:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:30 smithi047 ceph-mon[36835]: pgmap v5388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:24:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:32 smithi077 ceph-mon[37153]: pgmap v5389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9294.571943 secs 2024-07-27T09:24:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:32 smithi047 ceph-mon[36835]: pgmap v5389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9294.571943 secs 2024-07-27T09:24:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:32 smithi029 ceph-mon[32610]: pgmap v5389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9294.571943 secs 2024-07-27T09:24:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:34 smithi077 ceph-mon[37153]: pgmap v5390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:34 smithi047 ceph-mon[36835]: pgmap v5390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:34 smithi029 ceph-mon[32610]: pgmap v5390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:36 smithi077 ceph-mon[37153]: pgmap v5391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:36 smithi047 ceph-mon[36835]: pgmap v5391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:36 smithi029 ceph-mon[32610]: pgmap v5391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9299.572064 secs 2024-07-27T09:24:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9299.572064 secs 2024-07-27T09:24:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9299.572064 secs 2024-07-27T09:24:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:39 smithi077 ceph-mon[37153]: pgmap v5392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:39 smithi047 ceph-mon[36835]: pgmap v5392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:39 smithi029 ceph-mon[32610]: pgmap v5392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:41 smithi077 ceph-mon[37153]: pgmap v5393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:24:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:41 smithi047 ceph-mon[36835]: pgmap v5393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:24:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:41 smithi029 ceph-mon[32610]: pgmap v5393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:24:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:43 smithi077 ceph-mon[37153]: pgmap v5394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9304.572166 secs 2024-07-27T09:24:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:43 smithi047 ceph-mon[36835]: pgmap v5394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9304.572166 secs 2024-07-27T09:24:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:43 smithi029 ceph-mon[32610]: pgmap v5394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:24:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9304.572166 secs 2024-07-27T09:24:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:45 smithi077 ceph-mon[37153]: pgmap v5395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:24:45.364+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:24:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:24:45.364+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:24:45.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:24:45.364+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:24:45.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:45 smithi029 ceph-mon[32610]: pgmap v5395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:24:45.426+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:24:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:24:45.426+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:24:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:24:45.426+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:24:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:45 smithi047 ceph-mon[36835]: pgmap v5395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:47 smithi077 ceph-mon[37153]: pgmap v5396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:47 smithi047 ceph-mon[36835]: pgmap v5396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:47 smithi029 ceph-mon[32610]: pgmap v5396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:48.351 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9309.572287 secs 2024-07-27T09:24:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9309.572287 secs 2024-07-27T09:24:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9309.572287 secs 2024-07-27T09:24:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:49 smithi077 ceph-mon[37153]: pgmap v5397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:49 smithi047 ceph-mon[36835]: pgmap v5397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:49 smithi029 ceph-mon[32610]: pgmap v5397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:51 smithi077 ceph-mon[37153]: pgmap v5398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:51.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:51 smithi047 ceph-mon[36835]: pgmap v5398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:51 smithi029 ceph-mon[32610]: pgmap v5398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:53 smithi077 ceph-mon[37153]: pgmap v5399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:24:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9314.572407 secs 2024-07-27T09:24:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:53 smithi047 ceph-mon[36835]: pgmap v5399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:24:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9314.572407 secs 2024-07-27T09:24:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:53 smithi029 ceph-mon[32610]: pgmap v5399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:24:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9314.572407 secs 2024-07-27T09:24:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:55 smithi077 ceph-mon[37153]: pgmap v5400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:55 smithi047 ceph-mon[36835]: pgmap v5400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:55 smithi029 ceph-mon[32610]: pgmap v5400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:24:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:57 smithi077 ceph-mon[37153]: pgmap v5401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:57 smithi047 ceph-mon[36835]: pgmap v5401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:57 smithi029 ceph-mon[32610]: pgmap v5401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:24:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9319.572552 secs 2024-07-27T09:24:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9319.572552 secs 2024-07-27T09:24:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9319.572552 secs 2024-07-27T09:24:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:24:59 smithi077 ceph-mon[37153]: pgmap v5402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:24:59 smithi047 ceph-mon[36835]: pgmap v5402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:24:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:24:59 smithi029 ceph-mon[32610]: pgmap v5402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:25:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:00 smithi077 ceph-mon[37153]: pgmap v5403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:00.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:00 smithi029 ceph-mon[32610]: pgmap v5403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:00.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:00.379+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:25:00.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:00.379+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:25:00.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:00.379+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:25:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:00.441+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:25:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:00.442+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:25:00.443 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:00.442+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:25:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:00 smithi047 ceph-mon[36835]: pgmap v5403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:02 smithi077 ceph-mon[37153]: pgmap v5404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:25:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9324.572674 secs 2024-07-27T09:25:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:02 smithi047 ceph-mon[36835]: pgmap v5404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:25:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9324.572674 secs 2024-07-27T09:25:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:02 smithi029 ceph-mon[32610]: pgmap v5404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:25:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9324.572674 secs 2024-07-27T09:25:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:04 smithi077 ceph-mon[37153]: pgmap v5405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:25:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:04 smithi047 ceph-mon[36835]: pgmap v5405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:25:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:04 smithi029 ceph-mon[32610]: pgmap v5405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:25:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: pgmap v5406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:25:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:25:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:25:07.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:25:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: pgmap v5406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:25:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:25:07.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:25:07.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:25:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: pgmap v5406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:25:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:25:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:25:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:25:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:25:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:07 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:25:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:07 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:25:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:07 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:25:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:07 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:25:08.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:07 smithi077 ceph-mon[37153]: pgmap v5407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:25:08.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:07 smithi077 ceph-mon[37153]: pgmap v5408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:25:08.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9329.572800 secs 2024-07-27T09:25:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:07 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:25:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:07 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:25:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:07 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:25:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:07 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:25:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:07 smithi047 ceph-mon[36835]: pgmap v5407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:25:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:07 smithi047 ceph-mon[36835]: pgmap v5408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:25:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9329.572800 secs 2024-07-27T09:25:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:07 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:25:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:07 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:25:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:07 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:25:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:07 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:25:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:07 smithi029 ceph-mon[32610]: pgmap v5407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:25:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:07 smithi029 ceph-mon[32610]: pgmap v5408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:25:08.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9329.572800 secs 2024-07-27T09:25:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:09 smithi077 ceph-mon[37153]: pgmap v5409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:25:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:09 smithi047 ceph-mon[36835]: pgmap v5409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:25:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:09 smithi029 ceph-mon[32610]: pgmap v5409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:25:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:11 smithi077 ceph-mon[37153]: pgmap v5410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:25:12.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:11 smithi047 ceph-mon[36835]: pgmap v5410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:25:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:11 smithi029 ceph-mon[32610]: pgmap v5410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:25:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9334.572876 secs 2024-07-27T09:25:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9334.572876 secs 2024-07-27T09:25:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9334.572876 secs 2024-07-27T09:25:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:13 smithi077 ceph-mon[37153]: pgmap v5411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:25:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:13 smithi047 ceph-mon[36835]: pgmap v5411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:25:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:13 smithi029 ceph-mon[32610]: pgmap v5411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:25:15.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:15.389+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:25:15.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:15.389+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:25:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:15.389+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:25:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:15.456+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:25:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:15.456+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:25:15.458 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:15.456+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:25:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:15 smithi077 ceph-mon[37153]: pgmap v5412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:25:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:15 smithi047 ceph-mon[36835]: pgmap v5412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:25:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:15 smithi029 ceph-mon[32610]: pgmap v5412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:25:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:17 smithi077 ceph-mon[37153]: pgmap v5413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:25:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9339.573002 secs 2024-07-27T09:25:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:17 smithi047 ceph-mon[36835]: pgmap v5413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:25:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9339.573002 secs 2024-07-27T09:25:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:17 smithi029 ceph-mon[32610]: pgmap v5413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:25:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9339.573002 secs 2024-07-27T09:25:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:19 smithi077 ceph-mon[37153]: pgmap v5414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:25:20.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:19 smithi047 ceph-mon[36835]: pgmap v5414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:25:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:19 smithi029 ceph-mon[32610]: pgmap v5414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:25:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:21 smithi077 ceph-mon[37153]: pgmap v5415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:21 smithi047 ceph-mon[36835]: pgmap v5415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:21 smithi029 ceph-mon[32610]: pgmap v5415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9344.573067 secs 2024-07-27T09:25:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9344.573067 secs 2024-07-27T09:25:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9344.573067 secs 2024-07-27T09:25:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:23 smithi047 ceph-mon[36835]: pgmap v5416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:25:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:23 smithi029 ceph-mon[32610]: pgmap v5416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:25:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:23 smithi077 ceph-mon[37153]: pgmap v5416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:25:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:25 smithi047 ceph-mon[36835]: pgmap v5417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:25:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:25 smithi029 ceph-mon[32610]: pgmap v5417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:25:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:25 smithi077 ceph-mon[37153]: pgmap v5417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:25:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:27 smithi047 ceph-mon[36835]: pgmap v5418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:25:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9349.573189 secs 2024-07-27T09:25:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:27 smithi029 ceph-mon[32610]: pgmap v5418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:25:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9349.573189 secs 2024-07-27T09:25:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:27 smithi077 ceph-mon[37153]: pgmap v5418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:25:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9349.573189 secs 2024-07-27T09:25:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:29 smithi047 ceph-mon[36835]: pgmap v5419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:25:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:29 smithi029 ceph-mon[32610]: pgmap v5419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:25:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:29 smithi077 ceph-mon[37153]: pgmap v5419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:25:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:30.397+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:25:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:30.397+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:25:30.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:30.397+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:25:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:30.472+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:25:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:30.472+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:25:30.473 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:30.472+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:25:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:31 smithi047 ceph-mon[36835]: pgmap v5420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:25:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:31 smithi029 ceph-mon[32610]: pgmap v5420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:25:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:31 smithi077 ceph-mon[37153]: pgmap v5420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:25:33.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9354.573315 secs 2024-07-27T09:25:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9354.573315 secs 2024-07-27T09:25:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9354.573315 secs 2024-07-27T09:25:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:33 smithi047 ceph-mon[36835]: pgmap v5421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:25:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:33 smithi029 ceph-mon[32610]: pgmap v5421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:25:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:33 smithi077 ceph-mon[37153]: pgmap v5421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:25:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:35 smithi047 ceph-mon[36835]: pgmap v5422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:25:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:35 smithi029 ceph-mon[32610]: pgmap v5422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:25:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:35 smithi077 ceph-mon[37153]: pgmap v5422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:25:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:37 smithi047 ceph-mon[36835]: pgmap v5423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:25:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9359.573459 secs 2024-07-27T09:25:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:37 smithi029 ceph-mon[32610]: pgmap v5423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:25:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9359.573459 secs 2024-07-27T09:25:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:37 smithi077 ceph-mon[37153]: pgmap v5423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:25:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9359.573459 secs 2024-07-27T09:25:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:39 smithi047 ceph-mon[36835]: pgmap v5424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:25:40.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:39 smithi029 ceph-mon[32610]: pgmap v5424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:25:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:39 smithi077 ceph-mon[37153]: pgmap v5424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:25:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:41 smithi047 ceph-mon[36835]: pgmap v5425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:41 smithi029 ceph-mon[32610]: pgmap v5425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:41 smithi077 ceph-mon[37153]: pgmap v5425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9364.573605 secs 2024-07-27T09:25:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9364.573605 secs 2024-07-27T09:25:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9364.573605 secs 2024-07-27T09:25:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:44.048+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:25:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:44.048+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:25:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:44.048+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:25:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:44.048+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:25:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:43 smithi047 ceph-mon[36835]: pgmap v5426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:25:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:43 smithi029 ceph-mon[32610]: pgmap v5426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:25:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:43 smithi077 ceph-mon[37153]: pgmap v5426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:25:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:45.405+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:25:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:45.405+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:25:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:45.405+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:25:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:45.487+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:25:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:45.487+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:25:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:25:45.487+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:25:46.158 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:45 smithi077 ceph-mon[37153]: pgmap v5427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:25:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:45 smithi047 ceph-mon[36835]: pgmap v5427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:25:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:45 smithi029 ceph-mon[32610]: pgmap v5427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:25:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:47 smithi047 ceph-mon[36835]: pgmap v5428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:25:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9369.573733 secs 2024-07-27T09:25:48.200 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:47 smithi029 ceph-mon[32610]: pgmap v5428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:25:48.200 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9369.573733 secs 2024-07-27T09:25:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:47 smithi077 ceph-mon[37153]: pgmap v5428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:25:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9369.573733 secs 2024-07-27T09:25:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:49 smithi047 ceph-mon[36835]: pgmap v5429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:25:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:49 smithi029 ceph-mon[32610]: pgmap v5429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:25:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:49 smithi077 ceph-mon[37153]: pgmap v5429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:25:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:51 smithi047 ceph-mon[36835]: pgmap v5430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:51 smithi029 ceph-mon[32610]: pgmap v5430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:51 smithi077 ceph-mon[37153]: pgmap v5430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:25:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9374.573855 secs 2024-07-27T09:25:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9374.573855 secs 2024-07-27T09:25:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9374.573855 secs 2024-07-27T09:25:54.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:53 smithi047 ceph-mon[36835]: pgmap v5431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:25:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:53 smithi029 ceph-mon[32610]: pgmap v5431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:25:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:53 smithi077 ceph-mon[37153]: pgmap v5431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:25:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:55 smithi047 ceph-mon[36835]: pgmap v5432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:25:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:55 smithi029 ceph-mon[32610]: pgmap v5432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:25:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:55 smithi077 ceph-mon[37153]: pgmap v5432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:25:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:57 smithi047 ceph-mon[36835]: pgmap v5433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:25:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9379.573939 secs 2024-07-27T09:25:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:57 smithi029 ceph-mon[32610]: pgmap v5433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:25:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9379.573939 secs 2024-07-27T09:25:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:57 smithi077 ceph-mon[37153]: pgmap v5433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:25:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9379.573939 secs 2024-07-27T09:25:59.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:59.051+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:25:59.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:59.051+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:25:59.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:59.052+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:25:59.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:25:59.052+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:26:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:25:59 smithi047 ceph-mon[36835]: pgmap v5434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:26:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:25:59 smithi029 ceph-mon[32610]: pgmap v5434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:26:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:25:59 smithi077 ceph-mon[37153]: pgmap v5434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:26:00.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:00.421+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:26:00.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:00.421+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:26:00.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:00.421+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:26:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:00.489+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:26:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:00.489+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:26:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:00.489+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:26:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:01 smithi047 ceph-mon[36835]: pgmap v5435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:01 smithi029 ceph-mon[32610]: pgmap v5435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:01 smithi077 ceph-mon[37153]: pgmap v5435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9384.574029 secs 2024-07-27T09:26:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9384.574029 secs 2024-07-27T09:26:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9384.574029 secs 2024-07-27T09:26:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:03 smithi029 ceph-mon[32610]: pgmap v5436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:26:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:03 smithi077 ceph-mon[37153]: pgmap v5436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:26:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:03 smithi047 ceph-mon[36835]: pgmap v5436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:26:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:05 smithi029 ceph-mon[32610]: pgmap v5437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:26:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:05 smithi077 ceph-mon[37153]: pgmap v5437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:26:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:05 smithi047 ceph-mon[36835]: pgmap v5437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:26:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:26:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:26:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:26:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:26:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:26:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:26:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:26:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:26:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:26:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:26:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:26:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:26:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:07 smithi029 ceph-mon[32610]: pgmap v5438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:26:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:07 smithi029 ceph-mon[32610]: pgmap v5439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:26:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:07 smithi029 ceph-mon[32610]: pgmap v5440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:26:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9389.574112 secs 2024-07-27T09:26:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:07 smithi077 ceph-mon[37153]: pgmap v5438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:26:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:07 smithi077 ceph-mon[37153]: pgmap v5439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:26:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:07 smithi077 ceph-mon[37153]: pgmap v5440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:26:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9389.574112 secs 2024-07-27T09:26:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:07 smithi047 ceph-mon[36835]: pgmap v5438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:26:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:07 smithi047 ceph-mon[36835]: pgmap v5439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:26:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:07 smithi047 ceph-mon[36835]: pgmap v5440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:26:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9389.574112 secs 2024-07-27T09:26:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:09 smithi029 ceph-mon[32610]: pgmap v5441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:26:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:09 smithi077 ceph-mon[37153]: pgmap v5441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:26:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:09 smithi047 ceph-mon[36835]: pgmap v5441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:26:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:11 smithi077 ceph-mon[37153]: pgmap v5442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:26:12.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:11 smithi047 ceph-mon[36835]: pgmap v5442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:26:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:11 smithi029 ceph-mon[32610]: pgmap v5442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:26:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9394.574235 secs 2024-07-27T09:26:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9394.574235 secs 2024-07-27T09:26:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9394.574235 secs 2024-07-27T09:26:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:13 smithi077 ceph-mon[37153]: pgmap v5443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:26:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:13 smithi047 ceph-mon[36835]: pgmap v5443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:26:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:13 smithi029 ceph-mon[32610]: pgmap v5443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:26:15.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:15.436+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:26:15.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:15.436+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:26:15.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:15.436+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:26:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:15.493+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:26:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:15.493+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:26:15.495 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:15.493+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:26:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:15 smithi077 ceph-mon[37153]: pgmap v5444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:26:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:15 smithi047 ceph-mon[36835]: pgmap v5444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:26:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:15 smithi029 ceph-mon[32610]: pgmap v5444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:26:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:17 smithi077 ceph-mon[37153]: pgmap v5445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:26:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9399.574356 secs 2024-07-27T09:26:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:17 smithi047 ceph-mon[36835]: pgmap v5445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:26:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9399.574356 secs 2024-07-27T09:26:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:17 smithi029 ceph-mon[32610]: pgmap v5445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:26:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9399.574356 secs 2024-07-27T09:26:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:19 smithi029 ceph-mon[32610]: pgmap v5446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:26:20.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:19 smithi047 ceph-mon[36835]: pgmap v5446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:26:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:19 smithi077 ceph-mon[37153]: pgmap v5446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:26:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:21 smithi077 ceph-mon[37153]: pgmap v5447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:26:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:21 smithi047 ceph-mon[36835]: pgmap v5447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:26:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:21 smithi029 ceph-mon[32610]: pgmap v5447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:26:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9404.574481 secs 2024-07-27T09:26:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9404.574481 secs 2024-07-27T09:26:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9404.574481 secs 2024-07-27T09:26:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:23 smithi077 ceph-mon[37153]: pgmap v5448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:26:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:23 smithi047 ceph-mon[36835]: pgmap v5448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:26:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:23 smithi029 ceph-mon[32610]: pgmap v5448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:26:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:25 smithi077 ceph-mon[37153]: pgmap v5449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:26:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:25 smithi047 ceph-mon[36835]: pgmap v5449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:26:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:25 smithi029 ceph-mon[32610]: pgmap v5449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:26:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9409.574597 secs 2024-07-27T09:26:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:27 smithi077 ceph-mon[37153]: pgmap v5450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:26:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9409.574597 secs 2024-07-27T09:26:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:27 smithi047 ceph-mon[36835]: pgmap v5450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:26:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9409.574597 secs 2024-07-27T09:26:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:27 smithi029 ceph-mon[32610]: pgmap v5450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:26:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:29.055+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:26:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:29.055+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:26:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:29.055+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:26:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:29.055+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:26:30.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:29 smithi029 ceph-mon[32610]: pgmap v5451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:26:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:30 smithi077 ceph-mon[37153]: pgmap v5451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:26:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:30 smithi047 ceph-mon[36835]: pgmap v5451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:26:30.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:30.451+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:26:30.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:30.451+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:26:30.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:30.451+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:26:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:30.502+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:26:30.504 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:30.502+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:26:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:30.502+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:26:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:32 smithi077 ceph-mon[37153]: pgmap v5452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:32 smithi047 ceph-mon[36835]: pgmap v5452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:32 smithi029 ceph-mon[32610]: pgmap v5452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9414.574720 secs 2024-07-27T09:26:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9414.574720 secs 2024-07-27T09:26:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9414.574720 secs 2024-07-27T09:26:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:34 smithi077 ceph-mon[37153]: pgmap v5453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:26:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:34 smithi047 ceph-mon[36835]: pgmap v5453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:26:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:34 smithi029 ceph-mon[32610]: pgmap v5453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:26:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:36 smithi077 ceph-mon[37153]: pgmap v5454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:36.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:36 smithi047 ceph-mon[36835]: pgmap v5454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:36 smithi029 ceph-mon[32610]: pgmap v5454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9419.574864 secs 2024-07-27T09:26:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:38 smithi077 ceph-mon[37153]: pgmap v5455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:26:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9419.574864 secs 2024-07-27T09:26:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:38 smithi047 ceph-mon[36835]: pgmap v5455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:26:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9419.574864 secs 2024-07-27T09:26:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:38 smithi029 ceph-mon[32610]: pgmap v5455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:26:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:40 smithi077 ceph-mon[37153]: pgmap v5456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:26:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:40 smithi047 ceph-mon[36835]: pgmap v5456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:26:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:40 smithi029 ceph-mon[32610]: pgmap v5456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:26:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:42 smithi077 ceph-mon[37153]: pgmap v5457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:42 smithi047 ceph-mon[36835]: pgmap v5457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:42 smithi029 ceph-mon[32610]: pgmap v5457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9424.574991 secs 2024-07-27T09:26:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9424.574991 secs 2024-07-27T09:26:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9424.574991 secs 2024-07-27T09:26:44.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:44.058+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:26:44.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:44.058+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:26:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:44.058+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:26:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:44.058+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:26:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:44 smithi077 ceph-mon[37153]: pgmap v5458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:26:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:44 smithi047 ceph-mon[36835]: pgmap v5458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:26:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:44 smithi029 ceph-mon[32610]: pgmap v5458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:26:45.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:45.467+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:26:45.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:45.467+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:26:45.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:26:45.467+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:26:45.511 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:45.509+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:26:45.511 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:45.509+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:26:45.511 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:26:45.509+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:26:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:46 smithi077 ceph-mon[37153]: pgmap v5459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:26:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:46 smithi047 ceph-mon[36835]: pgmap v5459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:26:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:46 smithi029 ceph-mon[32610]: pgmap v5459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:26:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9429.575143 secs 2024-07-27T09:26:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:48 smithi077 ceph-mon[37153]: pgmap v5460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:26:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9429.575143 secs 2024-07-27T09:26:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:48 smithi047 ceph-mon[36835]: pgmap v5460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:26:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9429.575143 secs 2024-07-27T09:26:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:48 smithi029 ceph-mon[32610]: pgmap v5460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:26:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:50 smithi077 ceph-mon[37153]: pgmap v5461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:26:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:50 smithi047 ceph-mon[36835]: pgmap v5461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:26:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:50 smithi029 ceph-mon[32610]: pgmap v5461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:26:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:52 smithi077 ceph-mon[37153]: pgmap v5462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:26:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:52 smithi047 ceph-mon[36835]: pgmap v5462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:26:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:52 smithi029 ceph-mon[32610]: pgmap v5462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:26:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9434.575285 secs 2024-07-27T09:26:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9434.575285 secs 2024-07-27T09:26:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9434.575285 secs 2024-07-27T09:26:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:54 smithi077 ceph-mon[37153]: pgmap v5463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:26:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:54 smithi047 ceph-mon[36835]: pgmap v5463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:26:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:54 smithi029 ceph-mon[32610]: pgmap v5463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:26:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:56 smithi077 ceph-mon[37153]: pgmap v5464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:56 smithi047 ceph-mon[36835]: pgmap v5464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:56 smithi029 ceph-mon[32610]: pgmap v5464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:26:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9439.575402 secs 2024-07-27T09:26:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:26:58 smithi077 ceph-mon[37153]: pgmap v5465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:26:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9439.575402 secs 2024-07-27T09:26:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:26:58 smithi047 ceph-mon[36835]: pgmap v5465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:26:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9439.575402 secs 2024-07-27T09:26:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:26:58 smithi029 ceph-mon[32610]: pgmap v5465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:27:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:00 smithi077 ceph-mon[37153]: pgmap v5466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:27:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:00 smithi047 ceph-mon[36835]: pgmap v5466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:27:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:00 smithi029 ceph-mon[32610]: pgmap v5466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:27:00.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:27:00.470+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:27:00.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:27:00.470+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:27:00.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:27:00.470+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:27:00.526 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:27:00.525+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:27:00.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:27:00.525+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:27:00.527 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:27:00.525+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:27:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:01 smithi077 ceph-mon[37153]: pgmap v5467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:27:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:01 smithi047 ceph-mon[36835]: pgmap v5467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:27:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:01 smithi029 ceph-mon[32610]: pgmap v5467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:27:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9444.575525 secs 2024-07-27T09:27:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9444.575525 secs 2024-07-27T09:27:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9444.575525 secs 2024-07-27T09:27:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:03 smithi029 ceph-mon[32610]: pgmap v5468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:27:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:03 smithi077 ceph-mon[37153]: pgmap v5468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:27:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:03 smithi047 ceph-mon[36835]: pgmap v5468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:27:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:05 smithi047 ceph-mon[36835]: pgmap v5469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:27:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:05 smithi029 ceph-mon[32610]: pgmap v5469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:27:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:05 smithi077 ceph-mon[37153]: pgmap v5469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:27:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9449.575649 secs 2024-07-27T09:27:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:07 smithi047 ceph-mon[36835]: pgmap v5470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:27:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:27:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:27:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:27:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:27:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9449.575649 secs 2024-07-27T09:27:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:07 smithi029 ceph-mon[32610]: pgmap v5470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:27:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:27:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:27:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:27:08.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:27:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9449.575649 secs 2024-07-27T09:27:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:07 smithi077 ceph-mon[37153]: pgmap v5470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:27:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:27:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:27:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:27:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:27:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:08 smithi047 ceph-mon[36835]: pgmap v5471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T09:27:09.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:08 smithi047 ceph-mon[36835]: pgmap v5472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:27:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:08 smithi029 ceph-mon[32610]: pgmap v5471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T09:27:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:08 smithi029 ceph-mon[32610]: pgmap v5472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:27:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:08 smithi077 ceph-mon[37153]: pgmap v5471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T09:27:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:08 smithi077 ceph-mon[37153]: pgmap v5472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:27:11.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:10 smithi047 ceph-mon[36835]: pgmap v5473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:27:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:10 smithi029 ceph-mon[32610]: pgmap v5473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:27:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:10 smithi077 ceph-mon[37153]: pgmap v5473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:27:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:12 smithi047 ceph-mon[36835]: pgmap v5474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:27:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9454.575771 secs 2024-07-27T09:27:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:12 smithi029 ceph-mon[32610]: pgmap v5474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:27:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9454.575771 secs 2024-07-27T09:27:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:12 smithi077 ceph-mon[37153]: pgmap v5474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:27:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9454.575771 secs 2024-07-27T09:27:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:14 smithi029 ceph-mon[32610]: pgmap v5475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:27:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:14 smithi077 ceph-mon[37153]: pgmap v5475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:27:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:14 smithi047 ceph-mon[36835]: pgmap v5475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:27:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:27:15.481+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:27:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:27:15.481+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:27:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:27:15.481+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:27:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:27:15.529+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:27:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:27:15.529+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:27:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:27:15.529+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:27:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:16 smithi029 ceph-mon[32610]: pgmap v5476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:27:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:16 smithi077 ceph-mon[37153]: pgmap v5476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:27:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:16 smithi047 ceph-mon[36835]: pgmap v5476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:27:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9459.575891 secs 2024-07-27T09:27:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9459.575891 secs 2024-07-27T09:27:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9459.575891 secs 2024-07-27T09:27:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:18 smithi029 ceph-mon[32610]: pgmap v5477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:27:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:18 smithi077 ceph-mon[37153]: pgmap v5477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:27:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:18 smithi047 ceph-mon[36835]: pgmap v5477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:27:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:20 smithi029 ceph-mon[32610]: pgmap v5478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:27:21.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:20 smithi047 ceph-mon[36835]: pgmap v5478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:27:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:20 smithi077 ceph-mon[37153]: pgmap v5478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:27:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:22 smithi029 ceph-mon[32610]: pgmap v5479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:27:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9464.576005 secs 2024-07-27T09:27:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:22 smithi077 ceph-mon[37153]: pgmap v5479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:27:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9464.576005 secs 2024-07-27T09:27:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:22 smithi047 ceph-mon[36835]: pgmap v5479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:27:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9464.576005 secs 2024-07-27T09:27:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:24 smithi077 ceph-mon[37153]: pgmap v5480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:27:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:24 smithi047 ceph-mon[36835]: pgmap v5480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:27:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:24 smithi029 ceph-mon[32610]: pgmap v5480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:27:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:26 smithi077 ceph-mon[37153]: pgmap v5481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:27:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:26 smithi047 ceph-mon[36835]: pgmap v5481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:27:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:26 smithi029 ceph-mon[32610]: pgmap v5481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:27:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9469.576084 secs 2024-07-27T09:27:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9469.576084 secs 2024-07-27T09:27:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9469.576084 secs 2024-07-27T09:27:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:28 smithi077 ceph-mon[37153]: pgmap v5482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:27:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:28 smithi047 ceph-mon[36835]: pgmap v5482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:27:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:28 smithi029 ceph-mon[32610]: pgmap v5482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:27:30.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:27:30.496+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:27:30.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:27:30.496+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:27:30.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:27:30.496+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:27:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:27:30.530+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:27:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:27:30.530+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:27:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:27:30.530+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:27:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:30 smithi077 ceph-mon[37153]: pgmap v5483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:27:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:30 smithi047 ceph-mon[36835]: pgmap v5483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:27:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:30 smithi029 ceph-mon[32610]: pgmap v5483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:27:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:32 smithi077 ceph-mon[37153]: pgmap v5484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:27:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9474.576187 secs 2024-07-27T09:27:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:32 smithi047 ceph-mon[36835]: pgmap v5484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:27:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9474.576187 secs 2024-07-27T09:27:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:32 smithi029 ceph-mon[32610]: pgmap v5484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:27:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9474.576187 secs 2024-07-27T09:27:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:34 smithi077 ceph-mon[37153]: pgmap v5485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:27:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:34 smithi047 ceph-mon[36835]: pgmap v5485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:27:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:34 smithi029 ceph-mon[32610]: pgmap v5485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:27:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:36 smithi077 ceph-mon[37153]: pgmap v5486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:27:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:36 smithi047 ceph-mon[36835]: pgmap v5486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:27:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:36 smithi029 ceph-mon[32610]: pgmap v5486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:27:38.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9479.576331 secs 2024-07-27T09:27:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9479.576331 secs 2024-07-27T09:27:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9479.576331 secs 2024-07-27T09:27:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:38 smithi077 ceph-mon[37153]: pgmap v5487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:27:39.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:38 smithi047 ceph-mon[36835]: pgmap v5487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:27:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:38 smithi029 ceph-mon[32610]: pgmap v5487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:27:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:41 smithi077 ceph-mon[37153]: pgmap v5488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:27:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:41 smithi047 ceph-mon[36835]: pgmap v5488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:27:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:41 smithi029 ceph-mon[32610]: pgmap v5488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:27:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:43 smithi077 ceph-mon[37153]: pgmap v5489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:27:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9484.576451 secs 2024-07-27T09:27:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:43 smithi047 ceph-mon[36835]: pgmap v5489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:27:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9484.576451 secs 2024-07-27T09:27:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:43 smithi029 ceph-mon[32610]: pgmap v5489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:27:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9484.576451 secs 2024-07-27T09:27:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:45 smithi077 ceph-mon[37153]: pgmap v5490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:27:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:45 smithi047 ceph-mon[36835]: pgmap v5490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:27:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:45 smithi029 ceph-mon[32610]: pgmap v5490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:27:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:27:45.512+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:27:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:27:45.512+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:27:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:27:45.512+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:27:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:27:45.546+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:27:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:27:45.546+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:27:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:27:45.546+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:27:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:47 smithi077 ceph-mon[37153]: pgmap v5491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:27:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:47 smithi047 ceph-mon[36835]: pgmap v5491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:27:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:47 smithi029 ceph-mon[32610]: pgmap v5491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:27:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9489.576595 secs 2024-07-27T09:27:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9489.576595 secs 2024-07-27T09:27:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9489.576595 secs 2024-07-27T09:27:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:49 smithi077 ceph-mon[37153]: pgmap v5492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:27:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:49 smithi047 ceph-mon[36835]: pgmap v5492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:27:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:49 smithi029 ceph-mon[32610]: pgmap v5492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:27:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:51 smithi077 ceph-mon[37153]: pgmap v5493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:27:51.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:51 smithi047 ceph-mon[36835]: pgmap v5493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:27:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:51 smithi029 ceph-mon[32610]: pgmap v5493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:27:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:53 smithi077 ceph-mon[37153]: pgmap v5494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:27:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9494.576718 secs 2024-07-27T09:27:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:53 smithi047 ceph-mon[36835]: pgmap v5494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:27:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9494.576718 secs 2024-07-27T09:27:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:53 smithi029 ceph-mon[32610]: pgmap v5494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:27:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9494.576718 secs 2024-07-27T09:27:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:55 smithi077 ceph-mon[37153]: pgmap v5495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:27:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:55 smithi047 ceph-mon[36835]: pgmap v5495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:27:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:55 smithi029 ceph-mon[32610]: pgmap v5495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:27:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:57 smithi077 ceph-mon[37153]: pgmap v5496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:27:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:57 smithi047 ceph-mon[36835]: pgmap v5496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:27:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:57 smithi029 ceph-mon[32610]: pgmap v5496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:27:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9499.576837 secs 2024-07-27T09:27:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9499.576837 secs 2024-07-27T09:27:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9499.576837 secs 2024-07-27T09:27:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:27:59 smithi077 ceph-mon[37153]: pgmap v5497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:27:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:27:59 smithi047 ceph-mon[36835]: pgmap v5497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:27:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:27:59 smithi029 ceph-mon[32610]: pgmap v5497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:28:00.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:00.528+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:28:00.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:00.528+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:28:00.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:00.528+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:28:00.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:00.561+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:28:00.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:00.561+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:28:00.562 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:00.561+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:28:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:01 smithi077 ceph-mon[37153]: pgmap v5498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:28:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:01 smithi047 ceph-mon[36835]: pgmap v5498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:28:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:01 smithi029 ceph-mon[32610]: pgmap v5498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:28:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:03 smithi077 ceph-mon[37153]: pgmap v5499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9504.576952 secs 2024-07-27T09:28:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:03 smithi047 ceph-mon[36835]: pgmap v5499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9504.576952 secs 2024-07-27T09:28:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:03 smithi029 ceph-mon[32610]: pgmap v5499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9504.576952 secs 2024-07-27T09:28:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:05 smithi077 ceph-mon[37153]: pgmap v5500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:28:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:05 smithi047 ceph-mon[36835]: pgmap v5500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:28:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:05 smithi029 ceph-mon[32610]: pgmap v5500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:28:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:07 smithi077 ceph-mon[37153]: pgmap v5501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:07 smithi047 ceph-mon[36835]: pgmap v5501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:07 smithi029 ceph-mon[32610]: pgmap v5501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:08.337 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9509.577071 secs 2024-07-27T09:28:08.337 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:28:08.337 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:08.337 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:08.387 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9509.577071 secs 2024-07-27T09:28:08.387 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:28:08.387 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:08.387 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9509.577071 secs 2024-07-27T09:28:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:28:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:09 smithi077 ceph-mon[37153]: pgmap v5502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:28:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:09 smithi047 ceph-mon[36835]: pgmap v5502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:28:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:09 smithi029 ceph-mon[32610]: pgmap v5502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:28:10.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:10 smithi029 ceph-mon[32610]: pgmap v5503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:28:10.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:28:10.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:28:10.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:10 smithi077 ceph-mon[37153]: pgmap v5503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:28:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:28:10.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:28:10.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:10 smithi047 ceph-mon[36835]: pgmap v5503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:28:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:28:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:28:10.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:28:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:11 smithi077 ceph-mon[37153]: pgmap v5504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:28:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:11 smithi077 ceph-mon[37153]: pgmap v5505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:28:11.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:11 smithi029 ceph-mon[32610]: pgmap v5504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:28:11.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:11 smithi029 ceph-mon[32610]: pgmap v5505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:28:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:11 smithi047 ceph-mon[36835]: pgmap v5504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:28:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:11 smithi047 ceph-mon[36835]: pgmap v5505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:28:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9514.577200 secs 2024-07-27T09:28:12.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9514.577200 secs 2024-07-27T09:28:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9514.577200 secs 2024-07-27T09:28:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:13 smithi077 ceph-mon[37153]: pgmap v5506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:28:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:13 smithi047 ceph-mon[36835]: pgmap v5506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:28:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:13 smithi029 ceph-mon[32610]: pgmap v5506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:28:15.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:15.542+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:28:15.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:15.542+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:28:15.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:15.542+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:28:15.545 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:15 smithi029 ceph-mon[32610]: pgmap v5507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:28:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:15.561+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:28:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:15.561+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:28:15.563 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:15.561+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:28:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:15 smithi077 ceph-mon[37153]: pgmap v5507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:28:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:15 smithi047 ceph-mon[36835]: pgmap v5507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:28:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:17 smithi077 ceph-mon[37153]: pgmap v5508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:28:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9519.577321 secs 2024-07-27T09:28:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:17 smithi047 ceph-mon[36835]: pgmap v5508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:28:17.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9519.577321 secs 2024-07-27T09:28:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:17 smithi029 ceph-mon[32610]: pgmap v5508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:28:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9519.577321 secs 2024-07-27T09:28:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:19 smithi077 ceph-mon[37153]: pgmap v5509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:28:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:19 smithi047 ceph-mon[36835]: pgmap v5509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:28:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:19 smithi029 ceph-mon[32610]: pgmap v5509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:28:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:21 smithi077 ceph-mon[37153]: pgmap v5510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:28:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:21 smithi047 ceph-mon[36835]: pgmap v5510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:28:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:21 smithi029 ceph-mon[32610]: pgmap v5510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:28:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9524.577465 secs 2024-07-27T09:28:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9524.577465 secs 2024-07-27T09:28:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9524.577465 secs 2024-07-27T09:28:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:23 smithi077 ceph-mon[37153]: pgmap v5511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:23 smithi047 ceph-mon[36835]: pgmap v5511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:23 smithi029 ceph-mon[32610]: pgmap v5511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:25 smithi077 ceph-mon[37153]: pgmap v5512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:28:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:25 smithi047 ceph-mon[36835]: pgmap v5512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:28:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:25 smithi029 ceph-mon[32610]: pgmap v5512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:28:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:27 smithi077 ceph-mon[37153]: pgmap v5513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9529.577589 secs 2024-07-27T09:28:27.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:27 smithi047 ceph-mon[36835]: pgmap v5513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:27.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9529.577589 secs 2024-07-27T09:28:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:27 smithi029 ceph-mon[32610]: pgmap v5513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9529.577589 secs 2024-07-27T09:28:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:29 smithi077 ceph-mon[37153]: pgmap v5514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:28:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:29 smithi047 ceph-mon[36835]: pgmap v5514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:28:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:29 smithi029 ceph-mon[32610]: pgmap v5514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:28:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:30.551+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:28:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:30.551+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:28:30.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:30.551+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:28:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:30.576+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:28:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:30.577+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:28:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:30.577+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:28:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:31 smithi077 ceph-mon[37153]: pgmap v5515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:31 smithi047 ceph-mon[36835]: pgmap v5515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:31 smithi029 ceph-mon[32610]: pgmap v5515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9534.577696 secs 2024-07-27T09:28:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9534.577696 secs 2024-07-27T09:28:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9534.577696 secs 2024-07-27T09:28:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:33 smithi077 ceph-mon[37153]: pgmap v5516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:33 smithi047 ceph-mon[36835]: pgmap v5516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:33 smithi029 ceph-mon[32610]: pgmap v5516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:35 smithi077 ceph-mon[37153]: pgmap v5517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:28:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:35 smithi047 ceph-mon[36835]: pgmap v5517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:28:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:35 smithi029 ceph-mon[32610]: pgmap v5517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:28:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:37 smithi077 ceph-mon[37153]: pgmap v5518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9539.577857 secs 2024-07-27T09:28:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:37 smithi047 ceph-mon[36835]: pgmap v5518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9539.577857 secs 2024-07-27T09:28:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:37 smithi029 ceph-mon[32610]: pgmap v5518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9539.577857 secs 2024-07-27T09:28:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:39 smithi077 ceph-mon[37153]: pgmap v5519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:28:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:39 smithi047 ceph-mon[36835]: pgmap v5519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:28:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:39 smithi029 ceph-mon[32610]: pgmap v5519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:28:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:41 smithi077 ceph-mon[37153]: pgmap v5520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:41 smithi047 ceph-mon[36835]: pgmap v5520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:41 smithi029 ceph-mon[32610]: pgmap v5520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9544.577954 secs 2024-07-27T09:28:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9544.577954 secs 2024-07-27T09:28:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9544.577954 secs 2024-07-27T09:28:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:43 smithi077 ceph-mon[37153]: pgmap v5521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:43 smithi047 ceph-mon[36835]: pgmap v5521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:43 smithi029 ceph-mon[32610]: pgmap v5521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:44.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:44.077+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:28:44.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:44.077+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:28:44.078 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:44.077+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:28:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:44.077+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:28:45.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:45.565+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:28:45.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:45.565+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:28:45.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:45.565+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:28:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:45.592+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:28:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:45.592+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:28:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:28:45.592+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:28:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:45 smithi077 ceph-mon[37153]: pgmap v5522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:28:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:45 smithi047 ceph-mon[36835]: pgmap v5522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:28:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:45 smithi029 ceph-mon[32610]: pgmap v5522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:28:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:47 smithi077 ceph-mon[37153]: pgmap v5523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9549.578079 secs 2024-07-27T09:28:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:47 smithi047 ceph-mon[36835]: pgmap v5523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9549.578079 secs 2024-07-27T09:28:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:47 smithi029 ceph-mon[32610]: pgmap v5523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9549.578079 secs 2024-07-27T09:28:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:49 smithi077 ceph-mon[37153]: pgmap v5524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:28:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:49 smithi047 ceph-mon[36835]: pgmap v5524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:28:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:49 smithi029 ceph-mon[32610]: pgmap v5524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:28:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:51 smithi077 ceph-mon[37153]: pgmap v5525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:51 smithi047 ceph-mon[36835]: pgmap v5525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:51 smithi029 ceph-mon[32610]: pgmap v5525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9554.578208 secs 2024-07-27T09:28:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9554.578208 secs 2024-07-27T09:28:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9554.578208 secs 2024-07-27T09:28:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:53 smithi077 ceph-mon[37153]: pgmap v5526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:53 smithi047 ceph-mon[36835]: pgmap v5526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:53 smithi029 ceph-mon[32610]: pgmap v5526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:28:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:55 smithi077 ceph-mon[37153]: pgmap v5527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:28:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:55 smithi047 ceph-mon[36835]: pgmap v5527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:28:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:55 smithi029 ceph-mon[32610]: pgmap v5527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:28:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:57 smithi077 ceph-mon[37153]: pgmap v5528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9559.578331 secs 2024-07-27T09:28:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:57 smithi047 ceph-mon[36835]: pgmap v5528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9559.578331 secs 2024-07-27T09:28:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:57 smithi029 ceph-mon[32610]: pgmap v5528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:28:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9559.578331 secs 2024-07-27T09:28:59.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:59.079+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:28:59.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:59.079+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:28:59.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:59.080+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:28:59.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:28:59.080+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:28:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:28:59 smithi077 ceph-mon[37153]: pgmap v5529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:28:59.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:28:59 smithi047 ceph-mon[36835]: pgmap v5529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:28:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:28:59 smithi029 ceph-mon[32610]: pgmap v5529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:00.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:29:00.581+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:29:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:29:00.581+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:29:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:29:00.581+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:29:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:29:00.606+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:29:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:29:00.606+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:29:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:29:00.606+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:29:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:01 smithi077 ceph-mon[37153]: pgmap v5530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:01 smithi047 ceph-mon[36835]: pgmap v5530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:01 smithi029 ceph-mon[32610]: pgmap v5530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9564.578473 secs 2024-07-27T09:29:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9564.578473 secs 2024-07-27T09:29:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9564.578473 secs 2024-07-27T09:29:03.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:03 smithi047 ceph-mon[36835]: pgmap v5531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:29:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:03 smithi029 ceph-mon[32610]: pgmap v5531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:29:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:03 smithi077 ceph-mon[37153]: pgmap v5531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:29:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:05 smithi047 ceph-mon[36835]: pgmap v5532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:29:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:05 smithi029 ceph-mon[32610]: pgmap v5532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:29:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:05 smithi077 ceph-mon[37153]: pgmap v5532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:29:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:07 smithi047 ceph-mon[36835]: pgmap v5533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9569.578597 secs 2024-07-27T09:29:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:07 smithi029 ceph-mon[32610]: pgmap v5533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9569.578597 secs 2024-07-27T09:29:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:07 smithi077 ceph-mon[37153]: pgmap v5533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9569.578597 secs 2024-07-27T09:29:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:09 smithi047 ceph-mon[36835]: pgmap v5534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:09 smithi029 ceph-mon[32610]: pgmap v5534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:09 smithi077 ceph-mon[37153]: pgmap v5534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:29:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:29:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:29:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:11 smithi077 ceph-mon[37153]: pgmap v5535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:29:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:29:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:11 smithi077 ceph-mon[37153]: pgmap v5536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:29:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:11 smithi077 ceph-mon[37153]: pgmap v5537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:29:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:11 smithi047 ceph-mon[36835]: pgmap v5535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:29:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:29:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:11 smithi047 ceph-mon[36835]: pgmap v5536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:29:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:11 smithi047 ceph-mon[36835]: pgmap v5537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:29:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:11 smithi029 ceph-mon[32610]: pgmap v5535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:29:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:29:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:11 smithi029 ceph-mon[32610]: pgmap v5536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:29:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:11 smithi029 ceph-mon[32610]: pgmap v5537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:29:11.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:29:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9574.578744 secs 2024-07-27T09:29:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9574.578744 secs 2024-07-27T09:29:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9574.578744 secs 2024-07-27T09:29:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:13 smithi077 ceph-mon[37153]: pgmap v5538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T09:29:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:13 smithi047 ceph-mon[36835]: pgmap v5538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T09:29:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:13 smithi029 ceph-mon[32610]: pgmap v5538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T09:29:15.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:29:15.596+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:29:15.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:29:15.596+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:29:15.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:29:15.596+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:29:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:29:15.621+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:29:15.622 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:29:15.621+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:29:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:29:15.621+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:29:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:16 smithi077 ceph-mon[37153]: pgmap v5539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:29:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:16 smithi047 ceph-mon[36835]: pgmap v5539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:29:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:16 smithi029 ceph-mon[32610]: pgmap v5539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:29:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9579.578873 secs 2024-07-27T09:29:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:17 smithi077 ceph-mon[37153]: pgmap v5540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:29:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9579.578873 secs 2024-07-27T09:29:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:17 smithi047 ceph-mon[36835]: pgmap v5540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:29:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9579.578873 secs 2024-07-27T09:29:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:17 smithi029 ceph-mon[32610]: pgmap v5540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:29:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:20 smithi077 ceph-mon[37153]: pgmap v5541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:29:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:20 smithi047 ceph-mon[36835]: pgmap v5541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:29:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:20 smithi029 ceph-mon[32610]: pgmap v5541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:29:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:22 smithi077 ceph-mon[37153]: pgmap v5542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:29:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:22 smithi047 ceph-mon[36835]: pgmap v5542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:29:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:22 smithi029 ceph-mon[32610]: pgmap v5542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:29:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9584.578998 secs 2024-07-27T09:29:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9584.578998 secs 2024-07-27T09:29:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9584.578998 secs 2024-07-27T09:29:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:24 smithi077 ceph-mon[37153]: pgmap v5543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:24 smithi047 ceph-mon[36835]: pgmap v5543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:24 smithi029 ceph-mon[32610]: pgmap v5543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:26 smithi077 ceph-mon[37153]: pgmap v5544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:26 smithi047 ceph-mon[36835]: pgmap v5544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:26 smithi029 ceph-mon[32610]: pgmap v5544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9589.579087 secs 2024-07-27T09:29:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:28 smithi077 ceph-mon[37153]: pgmap v5545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:29:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9589.579087 secs 2024-07-27T09:29:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:28 smithi047 ceph-mon[36835]: pgmap v5545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:29:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9589.579087 secs 2024-07-27T09:29:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:28 smithi029 ceph-mon[32610]: pgmap v5545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:29:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:30 smithi077 ceph-mon[37153]: pgmap v5546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:29:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:30 smithi047 ceph-mon[36835]: pgmap v5546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:29:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:30 smithi029 ceph-mon[32610]: pgmap v5546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:29:30.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:29:30.611+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:29:30.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:29:30.611+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:29:30.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:29:30.611+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:29:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:29:30.622+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:29:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:29:30.622+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:29:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:29:30.622+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:29:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:32 smithi077 ceph-mon[37153]: pgmap v5547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:32 smithi047 ceph-mon[36835]: pgmap v5547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:32 smithi029 ceph-mon[32610]: pgmap v5547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9594.579217 secs 2024-07-27T09:29:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9594.579217 secs 2024-07-27T09:29:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9594.579217 secs 2024-07-27T09:29:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:34 smithi077 ceph-mon[37153]: pgmap v5548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:34 smithi047 ceph-mon[36835]: pgmap v5548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:34 smithi029 ceph-mon[32610]: pgmap v5548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:36 smithi077 ceph-mon[37153]: pgmap v5549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:36 smithi047 ceph-mon[36835]: pgmap v5549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:36 smithi029 ceph-mon[32610]: pgmap v5549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9599.579338 secs 2024-07-27T09:29:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:38 smithi077 ceph-mon[37153]: pgmap v5550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:29:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9599.579338 secs 2024-07-27T09:29:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:38 smithi047 ceph-mon[36835]: pgmap v5550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:29:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9599.579338 secs 2024-07-27T09:29:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:38 smithi029 ceph-mon[32610]: pgmap v5550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:29:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:40 smithi077 ceph-mon[37153]: pgmap v5551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:29:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:40 smithi047 ceph-mon[36835]: pgmap v5551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:29:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:40 smithi029 ceph-mon[32610]: pgmap v5551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:29:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:42 smithi077 ceph-mon[37153]: pgmap v5552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:42 smithi047 ceph-mon[36835]: pgmap v5552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:42 smithi029 ceph-mon[32610]: pgmap v5552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9604.579486 secs 2024-07-27T09:29:43.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9604.579486 secs 2024-07-27T09:29:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9604.579486 secs 2024-07-27T09:29:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:44 smithi077 ceph-mon[37153]: pgmap v5553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:44 smithi047 ceph-mon[36835]: pgmap v5553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:44 smithi029 ceph-mon[32610]: pgmap v5553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:45.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:29:45.616+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:29:45.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:29:45.616+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:29:45.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:29:45.616+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:29:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:29:45.625+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:29:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:29:45.625+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:29:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:29:45.625+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:29:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:46 smithi077 ceph-mon[37153]: pgmap v5554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:46 smithi047 ceph-mon[36835]: pgmap v5554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:46 smithi029 ceph-mon[32610]: pgmap v5554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9609.579607 secs 2024-07-27T09:29:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:48 smithi077 ceph-mon[37153]: pgmap v5555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:29:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9609.579607 secs 2024-07-27T09:29:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:48 smithi047 ceph-mon[36835]: pgmap v5555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:29:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9609.579607 secs 2024-07-27T09:29:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:48 smithi029 ceph-mon[32610]: pgmap v5555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:29:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:49 smithi077 ceph-mon[37153]: pgmap v5556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:29:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:49 smithi047 ceph-mon[36835]: pgmap v5556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:29:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:49 smithi029 ceph-mon[32610]: pgmap v5556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:29:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:52 smithi077 ceph-mon[37153]: pgmap v5557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:52 smithi047 ceph-mon[36835]: pgmap v5557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:52 smithi029 ceph-mon[32610]: pgmap v5557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9614.579741 secs 2024-07-27T09:29:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9614.579741 secs 2024-07-27T09:29:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9614.579741 secs 2024-07-27T09:29:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:54 smithi077 ceph-mon[37153]: pgmap v5558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:54 smithi047 ceph-mon[36835]: pgmap v5558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:54 smithi029 ceph-mon[32610]: pgmap v5558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:29:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:56 smithi077 ceph-mon[37153]: pgmap v5559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:56 smithi047 ceph-mon[36835]: pgmap v5559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:56 smithi029 ceph-mon[32610]: pgmap v5559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:29:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9619.579867 secs 2024-07-27T09:29:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:29:58 smithi077 ceph-mon[37153]: pgmap v5560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:29:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9619.579867 secs 2024-07-27T09:29:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:29:58 smithi047 ceph-mon[36835]: pgmap v5560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:29:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9619.579867 secs 2024-07-27T09:29:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:29:58 smithi029 ceph-mon[32610]: pgmap v5560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:30:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:00 smithi077 ceph-mon[37153]: pgmap v5561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:30:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:00 smithi047 ceph-mon[36835]: pgmap v5561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:00.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:30:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:00 smithi029 ceph-mon[32610]: pgmap v5561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:30:00.629 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:30:00.628+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:30:00.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:30:00.628+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:30:00.630 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:30:00.628+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:30:00.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:30:00.632+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:30:00.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:30:00.632+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:30:00.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:30:00.632+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:30:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:02 smithi077 ceph-mon[37153]: pgmap v5562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:02 smithi047 ceph-mon[36835]: pgmap v5562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:02 smithi029 ceph-mon[32610]: pgmap v5562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9624.579958 secs 2024-07-27T09:30:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9624.579958 secs 2024-07-27T09:30:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9624.579958 secs 2024-07-27T09:30:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:04 smithi077 ceph-mon[37153]: pgmap v5563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:30:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:04 smithi047 ceph-mon[36835]: pgmap v5563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:30:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:04 smithi029 ceph-mon[32610]: pgmap v5563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:30:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:06 smithi077 ceph-mon[37153]: pgmap v5564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:06 smithi047 ceph-mon[36835]: pgmap v5564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:06 smithi029 ceph-mon[32610]: pgmap v5564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9629.580084 secs 2024-07-27T09:30:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:07 smithi077 ceph-mon[37153]: pgmap v5565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:30:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9629.580084 secs 2024-07-27T09:30:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:07 smithi047 ceph-mon[36835]: pgmap v5565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:30:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9629.580084 secs 2024-07-27T09:30:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:07 smithi029 ceph-mon[32610]: pgmap v5565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:30:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:10 smithi077 ceph-mon[37153]: pgmap v5566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:10 smithi047 ceph-mon[36835]: pgmap v5566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:10 smithi029 ceph-mon[32610]: pgmap v5566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:12 smithi077 ceph-mon[37153]: pgmap v5567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:30:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:30:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:30:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:30:12.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:12 smithi047 ceph-mon[36835]: pgmap v5567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:12.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:30:12.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:30:12.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:30:12.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:30:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:12 smithi029 ceph-mon[32610]: pgmap v5567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:30:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:30:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:30:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:30:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:13 smithi077 ceph-mon[37153]: pgmap v5568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:30:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:13 smithi077 ceph-mon[37153]: pgmap v5569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:30:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9634.580218 secs 2024-07-27T09:30:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:13 smithi047 ceph-mon[36835]: pgmap v5568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:30:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:13 smithi047 ceph-mon[36835]: pgmap v5569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:30:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9634.580218 secs 2024-07-27T09:30:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:13 smithi029 ceph-mon[32610]: pgmap v5568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:30:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:13 smithi029 ceph-mon[32610]: pgmap v5569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:30:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9634.580218 secs 2024-07-27T09:30:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:15 smithi077 ceph-mon[37153]: pgmap v5570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:30:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:15 smithi047 ceph-mon[36835]: pgmap v5570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:30:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:15 smithi029 ceph-mon[32610]: pgmap v5570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:30:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:30:15.635+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:30:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:30:15.635+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:30:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:30:15.635+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:30:15.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:30:15.635+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:30:15.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:30:15.635+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:30:15.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:30:15.635+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:30:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:17 smithi077 ceph-mon[37153]: pgmap v5571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:30:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:17 smithi047 ceph-mon[36835]: pgmap v5571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:30:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:17 smithi029 ceph-mon[32610]: pgmap v5571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:30:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9639.580328 secs 2024-07-27T09:30:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9639.580328 secs 2024-07-27T09:30:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9639.580328 secs 2024-07-27T09:30:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:19 smithi077 ceph-mon[37153]: pgmap v5572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:30:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:19 smithi047 ceph-mon[36835]: pgmap v5572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:30:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:19 smithi029 ceph-mon[32610]: pgmap v5572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:30:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:21 smithi077 ceph-mon[37153]: pgmap v5573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:30:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:21 smithi047 ceph-mon[36835]: pgmap v5573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:30:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:21 smithi029 ceph-mon[32610]: pgmap v5573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:30:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:23 smithi077 ceph-mon[37153]: pgmap v5574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:30:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9644.580457 secs 2024-07-27T09:30:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:23 smithi047 ceph-mon[36835]: pgmap v5574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:30:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9644.580457 secs 2024-07-27T09:30:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:23 smithi029 ceph-mon[32610]: pgmap v5574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:30:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9644.580457 secs 2024-07-27T09:30:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:25 smithi077 ceph-mon[37153]: pgmap v5575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:25.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:25 smithi047 ceph-mon[36835]: pgmap v5575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:25 smithi029 ceph-mon[32610]: pgmap v5575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:27 smithi077 ceph-mon[37153]: pgmap v5576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:27 smithi047 ceph-mon[36835]: pgmap v5576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:27 smithi029 ceph-mon[32610]: pgmap v5576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9649.580581 secs 2024-07-27T09:30:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9649.580581 secs 2024-07-27T09:30:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9649.580581 secs 2024-07-27T09:30:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:29 smithi077 ceph-mon[37153]: pgmap v5577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:30:29.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:29 smithi047 ceph-mon[36835]: pgmap v5577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:30:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:29 smithi029 ceph-mon[32610]: pgmap v5577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:30:30.637+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:30:30.637+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:30:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:30:30.637+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:30:30.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:30:30.650+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:30:30.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:30:30.650+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:30:30.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:30:30.650+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:30:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:31 smithi077 ceph-mon[37153]: pgmap v5578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:31 smithi047 ceph-mon[36835]: pgmap v5578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:31 smithi029 ceph-mon[32610]: pgmap v5578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:32.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:32 smithi047 ceph-mon[36835]: pgmap v5579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:30:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:32 smithi029 ceph-mon[32610]: pgmap v5579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:30:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:32 smithi077 ceph-mon[37153]: pgmap v5579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:30:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9654.580708 secs 2024-07-27T09:30:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9654.580708 secs 2024-07-27T09:30:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9654.580708 secs 2024-07-27T09:30:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:34 smithi047 ceph-mon[36835]: pgmap v5580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:34 smithi029 ceph-mon[32610]: pgmap v5580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:34 smithi077 ceph-mon[37153]: pgmap v5580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:36 smithi077 ceph-mon[37153]: pgmap v5581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:36 smithi047 ceph-mon[36835]: pgmap v5581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:36 smithi029 ceph-mon[32610]: pgmap v5581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9659.580865 secs 2024-07-27T09:30:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9659.580865 secs 2024-07-27T09:30:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9659.580865 secs 2024-07-27T09:30:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:38 smithi077 ceph-mon[37153]: pgmap v5582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:30:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:38 smithi047 ceph-mon[36835]: pgmap v5582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:30:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:38 smithi029 ceph-mon[32610]: pgmap v5582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:30:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:40 smithi077 ceph-mon[37153]: pgmap v5583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:40.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:40 smithi047 ceph-mon[36835]: pgmap v5583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:40 smithi029 ceph-mon[32610]: pgmap v5583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:42 smithi077 ceph-mon[37153]: pgmap v5584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:30:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9664.580991 secs 2024-07-27T09:30:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:42 smithi047 ceph-mon[36835]: pgmap v5584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:30:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9664.580991 secs 2024-07-27T09:30:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:42 smithi029 ceph-mon[32610]: pgmap v5584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:30:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9664.580991 secs 2024-07-27T09:30:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:44 smithi047 ceph-mon[36835]: pgmap v5585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:44 smithi029 ceph-mon[32610]: pgmap v5585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:44 smithi077 ceph-mon[37153]: pgmap v5585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:45.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:30:45.652+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:30:45.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:30:45.652+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:30:45.654 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:30:45.652+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:30:45.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:30:45.666+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:30:45.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:30:45.666+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:30:45.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:30:45.666+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:30:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:46 smithi047 ceph-mon[36835]: pgmap v5586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:46 smithi029 ceph-mon[32610]: pgmap v5586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:46 smithi077 ceph-mon[37153]: pgmap v5586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9669.581084 secs 2024-07-27T09:30:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9669.581084 secs 2024-07-27T09:30:48.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9669.581084 secs 2024-07-27T09:30:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:48 smithi047 ceph-mon[36835]: pgmap v5587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:30:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:48 smithi029 ceph-mon[32610]: pgmap v5587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:30:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:48 smithi077 ceph-mon[37153]: pgmap v5587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:30:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:50 smithi047 ceph-mon[36835]: pgmap v5588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:50 smithi029 ceph-mon[32610]: pgmap v5588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:50 smithi077 ceph-mon[37153]: pgmap v5588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:30:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:52 smithi047 ceph-mon[36835]: pgmap v5589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:30:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9674.581151 secs 2024-07-27T09:30:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:52 smithi029 ceph-mon[32610]: pgmap v5589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:30:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9674.581151 secs 2024-07-27T09:30:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:52 smithi077 ceph-mon[37153]: pgmap v5589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:30:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9674.581151 secs 2024-07-27T09:30:54.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:54 smithi047 ceph-mon[36835]: pgmap v5590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:54 smithi029 ceph-mon[32610]: pgmap v5590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:54 smithi077 ceph-mon[37153]: pgmap v5590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:30:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:56 smithi047 ceph-mon[36835]: pgmap v5591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:56 smithi029 ceph-mon[32610]: pgmap v5591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:56 smithi077 ceph-mon[37153]: pgmap v5591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:30:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9679.581279 secs 2024-07-27T09:30:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9679.581279 secs 2024-07-27T09:30:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9679.581279 secs 2024-07-27T09:30:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:30:58 smithi047 ceph-mon[36835]: pgmap v5592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:30:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:30:58 smithi029 ceph-mon[32610]: pgmap v5592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:30:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:30:58 smithi077 ceph-mon[37153]: pgmap v5592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:31:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:00.668+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:31:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:00.668+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:31:00.669 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:00.668+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:31:00.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:00.681+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:31:00.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:00.682+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:31:00.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:00.682+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:31:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:00 smithi047 ceph-mon[36835]: pgmap v5593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:31:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:00 smithi029 ceph-mon[32610]: pgmap v5593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:31:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:00 smithi077 ceph-mon[37153]: pgmap v5593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:31:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:02 smithi047 ceph-mon[36835]: pgmap v5594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9684.581411 secs 2024-07-27T09:31:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:02 smithi029 ceph-mon[32610]: pgmap v5594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9684.581411 secs 2024-07-27T09:31:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:02 smithi077 ceph-mon[37153]: pgmap v5594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9684.581411 secs 2024-07-27T09:31:04.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:04 smithi047 ceph-mon[36835]: pgmap v5595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:31:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:04 smithi029 ceph-mon[32610]: pgmap v5595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:31:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:04 smithi077 ceph-mon[37153]: pgmap v5595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:31:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:06 smithi047 ceph-mon[36835]: pgmap v5596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:06 smithi029 ceph-mon[32610]: pgmap v5596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:06 smithi077 ceph-mon[37153]: pgmap v5596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9689.581554 secs 2024-07-27T09:31:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9689.581554 secs 2024-07-27T09:31:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9689.581554 secs 2024-07-27T09:31:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:08 smithi047 ceph-mon[36835]: pgmap v5597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:31:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:08 smithi029 ceph-mon[32610]: pgmap v5597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:31:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:08 smithi077 ceph-mon[37153]: pgmap v5597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:31:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:10 smithi047 ceph-mon[36835]: pgmap v5598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:31:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:10 smithi029 ceph-mon[32610]: pgmap v5598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:31:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:10 smithi077 ceph-mon[37153]: pgmap v5598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:31:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:31:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:31:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:31:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:12 smithi047 ceph-mon[36835]: pgmap v5599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9694.581647 secs 2024-07-27T09:31:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:31:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:31:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:12 smithi047 ceph-mon[36835]: pgmap v5600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:31:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:12 smithi047 ceph-mon[36835]: pgmap v5601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:31:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:31:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:12 smithi029 ceph-mon[32610]: pgmap v5599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9694.581647 secs 2024-07-27T09:31:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:31:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:31:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:12 smithi029 ceph-mon[32610]: pgmap v5600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:31:12.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:12 smithi029 ceph-mon[32610]: pgmap v5601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:31:12.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:31:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:12 smithi077 ceph-mon[37153]: pgmap v5599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9694.581647 secs 2024-07-27T09:31:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:31:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:31:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:12 smithi077 ceph-mon[37153]: pgmap v5600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:31:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:12 smithi077 ceph-mon[37153]: pgmap v5601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:31:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:31:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:14 smithi047 ceph-mon[36835]: pgmap v5602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:31:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:14 smithi029 ceph-mon[32610]: pgmap v5602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:31:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:14 smithi077 ceph-mon[37153]: pgmap v5602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:31:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:15.683+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:31:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:15.683+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:31:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:15.683+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:31:15.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:15.683+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:31:15.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:15.683+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:31:15.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:15.683+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:31:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:16 smithi047 ceph-mon[36835]: pgmap v5603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:31:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:16 smithi029 ceph-mon[32610]: pgmap v5603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:31:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:16 smithi077 ceph-mon[37153]: pgmap v5603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:31:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9699.581780 secs 2024-07-27T09:31:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9699.581780 secs 2024-07-27T09:31:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9699.581780 secs 2024-07-27T09:31:18.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:18 smithi077 ceph-mon[37153]: pgmap v5604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:31:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:18 smithi047 ceph-mon[36835]: pgmap v5604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:31:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:18 smithi029 ceph-mon[32610]: pgmap v5604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:31:20.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:20 smithi047 ceph-mon[36835]: pgmap v5605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:31:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:20 smithi029 ceph-mon[32610]: pgmap v5605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:31:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:20 smithi077 ceph-mon[37153]: pgmap v5605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:31:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:22 smithi047 ceph-mon[36835]: pgmap v5606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:31:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:22 smithi029 ceph-mon[32610]: pgmap v5606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:31:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:22 smithi077 ceph-mon[37153]: pgmap v5606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:31:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9704.581905 secs 2024-07-27T09:31:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9704.581905 secs 2024-07-27T09:31:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9704.581905 secs 2024-07-27T09:31:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:24 smithi047 ceph-mon[36835]: pgmap v5607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:31:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:24 smithi029 ceph-mon[32610]: pgmap v5607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:31:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:24 smithi077 ceph-mon[37153]: pgmap v5607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:31:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:26 smithi047 ceph-mon[36835]: pgmap v5608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:26 smithi029 ceph-mon[32610]: pgmap v5608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:26 smithi077 ceph-mon[37153]: pgmap v5608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9709.582032 secs 2024-07-27T09:31:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9709.582032 secs 2024-07-27T09:31:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9709.582032 secs 2024-07-27T09:31:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:28 smithi077 ceph-mon[37153]: pgmap v5609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:31:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:28 smithi047 ceph-mon[36835]: pgmap v5609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:31:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:28 smithi029 ceph-mon[32610]: pgmap v5609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:31:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:30 smithi047 ceph-mon[36835]: pgmap v5610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:30 smithi029 ceph-mon[32610]: pgmap v5610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:30 smithi077 ceph-mon[37153]: pgmap v5610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:30.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:30.685+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:31:30.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:30.685+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:31:30.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:30.685+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:31:30.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:30.698+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:31:30.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:30.698+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:31:30.700 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:30.698+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:31:32.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:32 smithi047 ceph-mon[36835]: pgmap v5611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:32.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:32 smithi029 ceph-mon[32610]: pgmap v5611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:32 smithi077 ceph-mon[37153]: pgmap v5611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9714.582177 secs 2024-07-27T09:31:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9714.582177 secs 2024-07-27T09:31:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9714.582177 secs 2024-07-27T09:31:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:34 smithi047 ceph-mon[36835]: pgmap v5612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:31:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:34 smithi029 ceph-mon[32610]: pgmap v5612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:31:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:34 smithi077 ceph-mon[37153]: pgmap v5612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:31:36.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:36 smithi047 ceph-mon[36835]: pgmap v5613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:36 smithi029 ceph-mon[32610]: pgmap v5613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:36 smithi077 ceph-mon[37153]: pgmap v5613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9719.582303 secs 2024-07-27T09:31:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9719.582303 secs 2024-07-27T09:31:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9719.582303 secs 2024-07-27T09:31:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:38 smithi077 ceph-mon[37153]: pgmap v5614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:31:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:38 smithi047 ceph-mon[36835]: pgmap v5614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:31:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:38 smithi029 ceph-mon[32610]: pgmap v5614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:31:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:40 smithi047 ceph-mon[36835]: pgmap v5615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:40 smithi029 ceph-mon[32610]: pgmap v5615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:40 smithi077 ceph-mon[37153]: pgmap v5615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:42.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:42 smithi047 ceph-mon[36835]: pgmap v5616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:42 smithi029 ceph-mon[32610]: pgmap v5616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:42 smithi077 ceph-mon[37153]: pgmap v5616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9724.582403 secs 2024-07-27T09:31:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9724.582403 secs 2024-07-27T09:31:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9724.582403 secs 2024-07-27T09:31:44.105 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:44.103+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:31:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:44.103+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:31:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:44.104+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:31:44.106 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:44.104+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:31:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:44 smithi047 ceph-mon[36835]: pgmap v5617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:31:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:44 smithi029 ceph-mon[32610]: pgmap v5617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:31:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:44 smithi077 ceph-mon[37153]: pgmap v5617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:31:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:45.699+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:31:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:45.699+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:31:45.701 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:31:45.699+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:31:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:45.700+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:31:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:45.700+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:31:45.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:45.700+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:31:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:46 smithi047 ceph-mon[36835]: pgmap v5618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:46 smithi029 ceph-mon[32610]: pgmap v5618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:46 smithi077 ceph-mon[37153]: pgmap v5618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:47.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9729.582532 secs 2024-07-27T09:31:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9729.582532 secs 2024-07-27T09:31:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9729.582532 secs 2024-07-27T09:31:48.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:48 smithi077 ceph-mon[37153]: pgmap v5619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:31:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:48 smithi047 ceph-mon[36835]: pgmap v5619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:31:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:48 smithi029 ceph-mon[32610]: pgmap v5619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:31:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:50 smithi047 ceph-mon[36835]: pgmap v5620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:50 smithi029 ceph-mon[32610]: pgmap v5620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:50 smithi077 ceph-mon[37153]: pgmap v5620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:52 smithi047 ceph-mon[36835]: pgmap v5621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:52 smithi029 ceph-mon[32610]: pgmap v5621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:52 smithi077 ceph-mon[37153]: pgmap v5621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:31:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9734.582649 secs 2024-07-27T09:31:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9734.582649 secs 2024-07-27T09:31:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9734.582649 secs 2024-07-27T09:31:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:54 smithi047 ceph-mon[36835]: pgmap v5622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:31:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:54 smithi029 ceph-mon[32610]: pgmap v5622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:31:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:54 smithi077 ceph-mon[37153]: pgmap v5622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:31:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:57 smithi047 ceph-mon[36835]: pgmap v5623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:57 smithi029 ceph-mon[32610]: pgmap v5623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:57 smithi077 ceph-mon[37153]: pgmap v5623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:31:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9739.582780 secs 2024-07-27T09:31:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9739.582780 secs 2024-07-27T09:31:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9739.582780 secs 2024-07-27T09:31:59.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:59.106+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:31:59.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:59.106+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:31:59.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:59.107+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:31:59.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:31:59.107+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:31:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:31:59 smithi047 ceph-mon[36835]: pgmap v5624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:31:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:31:59 smithi029 ceph-mon[32610]: pgmap v5624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:31:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:31:59 smithi077 ceph-mon[37153]: pgmap v5624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:32:00.716 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:32:00.715+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:32:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:32:00.715+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:32:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:32:00.715+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:32:00.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:32:00.715+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:32:00.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:32:00.715+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:32:00.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:32:00.715+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:32:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:01 smithi047 ceph-mon[36835]: pgmap v5625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:01 smithi029 ceph-mon[32610]: pgmap v5625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:01 smithi077 ceph-mon[37153]: pgmap v5625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9744.582915 secs 2024-07-27T09:32:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:03 smithi047 ceph-mon[36835]: pgmap v5626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9744.582915 secs 2024-07-27T09:32:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:03 smithi029 ceph-mon[32610]: pgmap v5626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9744.582915 secs 2024-07-27T09:32:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:03 smithi077 ceph-mon[37153]: pgmap v5626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:05 smithi047 ceph-mon[36835]: pgmap v5627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:32:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:05 smithi029 ceph-mon[32610]: pgmap v5627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:32:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:05 smithi077 ceph-mon[37153]: pgmap v5627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:32:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:07 smithi047 ceph-mon[36835]: pgmap v5628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:07 smithi029 ceph-mon[32610]: pgmap v5628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:07 smithi077 ceph-mon[37153]: pgmap v5628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9749.583040 secs 2024-07-27T09:32:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9749.583040 secs 2024-07-27T09:32:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9749.583040 secs 2024-07-27T09:32:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:09 smithi047 ceph-mon[36835]: pgmap v5629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:32:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:09 smithi029 ceph-mon[32610]: pgmap v5629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:32:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:09 smithi077 ceph-mon[37153]: pgmap v5629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:32:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:11 smithi047 ceph-mon[36835]: pgmap v5630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:11 smithi029 ceph-mon[32610]: pgmap v5630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:11 smithi077 ceph-mon[37153]: pgmap v5630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9754.583116 secs 2024-07-27T09:32:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9754.583116 secs 2024-07-27T09:32:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9754.583116 secs 2024-07-27T09:32:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:13 smithi029 ceph-mon[32610]: pgmap v5631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:32:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:32:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:32:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:13 smithi029 ceph-mon[32610]: pgmap v5632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T09:32:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:13 smithi029 ceph-mon[32610]: pgmap v5633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:32:13.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:32:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:13 smithi077 ceph-mon[37153]: pgmap v5631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:32:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:32:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:32:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:13 smithi077 ceph-mon[37153]: pgmap v5632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T09:32:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:13 smithi077 ceph-mon[37153]: pgmap v5633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:32:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:32:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:13 smithi047 ceph-mon[36835]: pgmap v5631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:32:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:32:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:32:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:13 smithi047 ceph-mon[36835]: pgmap v5632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T09:32:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:13 smithi047 ceph-mon[36835]: pgmap v5633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:32:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:32:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:32:15.730+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:32:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:32:15.730+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:32:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:32:15.730+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:32:15.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:32:15.730+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:32:15.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:32:15.730+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:32:15.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:32:15.730+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:32:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:15 smithi029 ceph-mon[32610]: pgmap v5634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:32:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:15 smithi077 ceph-mon[37153]: pgmap v5634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:32:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:15 smithi047 ceph-mon[36835]: pgmap v5634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:32:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:17 smithi029 ceph-mon[32610]: pgmap v5635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:32:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9759.583238 secs 2024-07-27T09:32:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:17 smithi077 ceph-mon[37153]: pgmap v5635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:32:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9759.583238 secs 2024-07-27T09:32:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:17 smithi047 ceph-mon[36835]: pgmap v5635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:32:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9759.583238 secs 2024-07-27T09:32:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:19 smithi077 ceph-mon[37153]: pgmap v5636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:32:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:19 smithi047 ceph-mon[36835]: pgmap v5636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:32:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:19 smithi029 ceph-mon[32610]: pgmap v5636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:32:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:21 smithi077 ceph-mon[37153]: pgmap v5637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:32:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:21 smithi047 ceph-mon[36835]: pgmap v5637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:32:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:21 smithi029 ceph-mon[32610]: pgmap v5637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:32:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9764.583353 secs 2024-07-27T09:32:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9764.583353 secs 2024-07-27T09:32:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9764.583353 secs 2024-07-27T09:32:24.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:23 smithi047 ceph-mon[36835]: pgmap v5638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:32:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:23 smithi077 ceph-mon[37153]: pgmap v5638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:32:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:23 smithi029 ceph-mon[32610]: pgmap v5638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:32:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:25 smithi077 ceph-mon[37153]: pgmap v5639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:32:26.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:25 smithi047 ceph-mon[36835]: pgmap v5639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:32:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:25 smithi029 ceph-mon[32610]: pgmap v5639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:32:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:27 smithi077 ceph-mon[37153]: pgmap v5640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9769.583483 secs 2024-07-27T09:32:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:27 smithi047 ceph-mon[36835]: pgmap v5640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9769.583483 secs 2024-07-27T09:32:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:27 smithi029 ceph-mon[32610]: pgmap v5640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9769.583483 secs 2024-07-27T09:32:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:29 smithi077 ceph-mon[37153]: pgmap v5641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:32:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:29 smithi047 ceph-mon[36835]: pgmap v5641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:32:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:29 smithi029 ceph-mon[32610]: pgmap v5641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:32:30.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:32:30.734+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:32:30.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:32:30.734+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:32:30.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:32:30.734+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:32:30.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:32:30.745+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:32:30.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:32:30.745+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:32:30.747 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:32:30.745+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:32:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:31 smithi077 ceph-mon[37153]: pgmap v5642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:31 smithi047 ceph-mon[36835]: pgmap v5642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:31 smithi029 ceph-mon[32610]: pgmap v5642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9774.583626 secs 2024-07-27T09:32:33.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9774.583626 secs 2024-07-27T09:32:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9774.583626 secs 2024-07-27T09:32:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:33 smithi077 ceph-mon[37153]: pgmap v5643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:32:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:33 smithi047 ceph-mon[36835]: pgmap v5643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:32:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:33 smithi029 ceph-mon[32610]: pgmap v5643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:32:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:35 smithi077 ceph-mon[37153]: pgmap v5644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:32:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:35 smithi047 ceph-mon[36835]: pgmap v5644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:32:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:35 smithi029 ceph-mon[32610]: pgmap v5644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:32:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:37 smithi077 ceph-mon[37153]: pgmap v5645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9779.583769 secs 2024-07-27T09:32:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:37 smithi047 ceph-mon[36835]: pgmap v5645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9779.583769 secs 2024-07-27T09:32:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:37 smithi029 ceph-mon[32610]: pgmap v5645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9779.583769 secs 2024-07-27T09:32:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:39 smithi077 ceph-mon[37153]: pgmap v5646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:32:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:39 smithi047 ceph-mon[36835]: pgmap v5646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:32:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:39 smithi029 ceph-mon[32610]: pgmap v5646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:32:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:41 smithi077 ceph-mon[37153]: pgmap v5647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:41 smithi047 ceph-mon[36835]: pgmap v5647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:41 smithi029 ceph-mon[32610]: pgmap v5647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9784.583866 secs 2024-07-27T09:32:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9784.583866 secs 2024-07-27T09:32:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9784.583866 secs 2024-07-27T09:32:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:43 smithi077 ceph-mon[37153]: pgmap v5648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:32:44.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:43 smithi047 ceph-mon[36835]: pgmap v5648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:32:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:43 smithi029 ceph-mon[32610]: pgmap v5648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:32:45.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:32:45.749+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:32:45.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:32:45.749+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:32:45.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:32:45.749+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:32:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:32:45.761+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:32:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:32:45.761+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:32:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:32:45.761+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:32:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:45 smithi077 ceph-mon[37153]: pgmap v5649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:32:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:45 smithi047 ceph-mon[36835]: pgmap v5649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:32:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:45 smithi029 ceph-mon[32610]: pgmap v5649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:32:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:47 smithi077 ceph-mon[37153]: pgmap v5650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9789.583930 secs 2024-07-27T09:32:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:47 smithi047 ceph-mon[36835]: pgmap v5650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9789.583930 secs 2024-07-27T09:32:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:47 smithi029 ceph-mon[32610]: pgmap v5650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9789.583930 secs 2024-07-27T09:32:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:49 smithi077 ceph-mon[37153]: pgmap v5651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:32:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:49 smithi047 ceph-mon[36835]: pgmap v5651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:32:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:49 smithi029 ceph-mon[32610]: pgmap v5651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:32:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:51 smithi077 ceph-mon[37153]: pgmap v5652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:51 smithi047 ceph-mon[36835]: pgmap v5652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:51 smithi029 ceph-mon[32610]: pgmap v5652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:32:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9794.584030 secs 2024-07-27T09:32:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9794.584030 secs 2024-07-27T09:32:53.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9794.584030 secs 2024-07-27T09:32:54.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:53 smithi047 ceph-mon[36835]: pgmap v5653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:32:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:53 smithi077 ceph-mon[37153]: pgmap v5653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:32:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:53 smithi029 ceph-mon[32610]: pgmap v5653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:32:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:55 smithi077 ceph-mon[37153]: pgmap v5654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:32:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:55 smithi047 ceph-mon[36835]: pgmap v5654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:32:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:55 smithi029 ceph-mon[32610]: pgmap v5654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:32:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:57 smithi077 ceph-mon[37153]: pgmap v5655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9799.584154 secs 2024-07-27T09:32:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:57 smithi047 ceph-mon[36835]: pgmap v5655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9799.584154 secs 2024-07-27T09:32:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:57 smithi029 ceph-mon[32610]: pgmap v5655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:32:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9799.584154 secs 2024-07-27T09:33:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:32:59 smithi077 ceph-mon[37153]: pgmap v5656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:32:59 smithi047 ceph-mon[36835]: pgmap v5656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:32:59 smithi029 ceph-mon[32610]: pgmap v5656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:00.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:33:00.765+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:33:00.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:33:00.765+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:33:00.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:33:00.765+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:33:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:33:00.777+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:33:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:33:00.777+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:33:00.778 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:33:00.777+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:33:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:01 smithi077 ceph-mon[37153]: pgmap v5657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:01 smithi047 ceph-mon[36835]: pgmap v5657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:01 smithi029 ceph-mon[32610]: pgmap v5657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9804.584253 secs 2024-07-27T09:33:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9804.584253 secs 2024-07-27T09:33:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9804.584253 secs 2024-07-27T09:33:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:03 smithi077 ceph-mon[37153]: pgmap v5658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:33:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:03 smithi047 ceph-mon[36835]: pgmap v5658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:33:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:03 smithi029 ceph-mon[32610]: pgmap v5658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:33:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:05 smithi077 ceph-mon[37153]: pgmap v5659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:33:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:05 smithi047 ceph-mon[36835]: pgmap v5659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:33:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:05 smithi029 ceph-mon[32610]: pgmap v5659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:33:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:07 smithi077 ceph-mon[37153]: pgmap v5660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9809.584410 secs 2024-07-27T09:33:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:07 smithi047 ceph-mon[36835]: pgmap v5660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9809.584410 secs 2024-07-27T09:33:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:07 smithi029 ceph-mon[32610]: pgmap v5660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9809.584410 secs 2024-07-27T09:33:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:09 smithi077 ceph-mon[37153]: pgmap v5661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:09 smithi047 ceph-mon[36835]: pgmap v5661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:09 smithi029 ceph-mon[32610]: pgmap v5661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:11 smithi077 ceph-mon[37153]: pgmap v5662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:11 smithi047 ceph-mon[36835]: pgmap v5662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:11 smithi029 ceph-mon[32610]: pgmap v5662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9814.584552 secs 2024-07-27T09:33:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:33:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9814.584552 secs 2024-07-27T09:33:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:33:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9814.584552 secs 2024-07-27T09:33:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:33:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:13 smithi077 ceph-mon[37153]: pgmap v5663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:33:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:33:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:33:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:33:14.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:13 smithi047 ceph-mon[36835]: pgmap v5663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:33:14.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:33:14.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:33:14.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:33:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:13 smithi029 ceph-mon[32610]: pgmap v5663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:33:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:33:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:33:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:33:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:14 smithi077 ceph-mon[37153]: pgmap v5664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:33:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:14 smithi077 ceph-mon[37153]: pgmap v5665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:33:15.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:14 smithi047 ceph-mon[36835]: pgmap v5664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:33:15.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:14 smithi047 ceph-mon[36835]: pgmap v5665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:33:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:14 smithi029 ceph-mon[32610]: pgmap v5664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:33:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:14 smithi029 ceph-mon[32610]: pgmap v5665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:33:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:33:15.779+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:33:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:33:15.779+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:33:15.782 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:33:15.779+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:33:15.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:33:15.779+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:33:15.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:33:15.779+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:33:15.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:33:15.779+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:33:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:16 smithi077 ceph-mon[37153]: pgmap v5666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:33:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:16 smithi047 ceph-mon[36835]: pgmap v5666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:33:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:16 smithi029 ceph-mon[32610]: pgmap v5666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:33:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9819.584652 secs 2024-07-27T09:33:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9819.584652 secs 2024-07-27T09:33:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9819.584652 secs 2024-07-27T09:33:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:18 smithi077 ceph-mon[37153]: pgmap v5667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:33:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:18 smithi047 ceph-mon[36835]: pgmap v5667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:33:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:18 smithi029 ceph-mon[32610]: pgmap v5667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:33:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:20 smithi077 ceph-mon[37153]: pgmap v5668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:33:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:20 smithi047 ceph-mon[36835]: pgmap v5668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:33:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:20 smithi029 ceph-mon[32610]: pgmap v5668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:33:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:22 smithi077 ceph-mon[37153]: pgmap v5669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:33:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9824.584813 secs 2024-07-27T09:33:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:22 smithi047 ceph-mon[36835]: pgmap v5669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:33:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9824.584813 secs 2024-07-27T09:33:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:22 smithi029 ceph-mon[32610]: pgmap v5669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:33:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9824.584813 secs 2024-07-27T09:33:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:24 smithi047 ceph-mon[36835]: pgmap v5670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:33:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:24 smithi029 ceph-mon[32610]: pgmap v5670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:33:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:24 smithi077 ceph-mon[37153]: pgmap v5670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:33:27.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:26 smithi047 ceph-mon[36835]: pgmap v5671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:26 smithi029 ceph-mon[32610]: pgmap v5671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:26 smithi077 ceph-mon[37153]: pgmap v5671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9829.584925 secs 2024-07-27T09:33:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9829.584925 secs 2024-07-27T09:33:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9829.584925 secs 2024-07-27T09:33:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:28 smithi047 ceph-mon[36835]: pgmap v5672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:28 smithi029 ceph-mon[32610]: pgmap v5672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:28 smithi077 ceph-mon[37153]: pgmap v5672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:33:30.789+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:33:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:33:30.789+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:33:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:33:30.789+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:33:30.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:33:30.795+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:33:30.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:33:30.795+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:33:30.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:33:30.795+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:33:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:30 smithi047 ceph-mon[36835]: pgmap v5673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:33:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:30 smithi029 ceph-mon[32610]: pgmap v5673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:33:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:30 smithi077 ceph-mon[37153]: pgmap v5673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:33:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:32 smithi047 ceph-mon[36835]: pgmap v5674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9834.585003 secs 2024-07-27T09:33:33.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:32 smithi029 ceph-mon[32610]: pgmap v5674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:33.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9834.585003 secs 2024-07-27T09:33:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:32 smithi077 ceph-mon[37153]: pgmap v5674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9834.585003 secs 2024-07-27T09:33:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:34 smithi047 ceph-mon[36835]: pgmap v5675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:34 smithi029 ceph-mon[32610]: pgmap v5675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:34 smithi077 ceph-mon[37153]: pgmap v5675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:36 smithi047 ceph-mon[36835]: pgmap v5676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:36 smithi029 ceph-mon[32610]: pgmap v5676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:36 smithi077 ceph-mon[37153]: pgmap v5676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9839.585082 secs 2024-07-27T09:33:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9839.585082 secs 2024-07-27T09:33:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9839.585082 secs 2024-07-27T09:33:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:38 smithi047 ceph-mon[36835]: pgmap v5677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:38 smithi029 ceph-mon[32610]: pgmap v5677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:38 smithi077 ceph-mon[37153]: pgmap v5677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:41.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:40 smithi047 ceph-mon[36835]: pgmap v5678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:33:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:40 smithi029 ceph-mon[32610]: pgmap v5678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:33:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:40 smithi077 ceph-mon[37153]: pgmap v5678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:33:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:42 smithi047 ceph-mon[36835]: pgmap v5679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9844.585209 secs 2024-07-27T09:33:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:42 smithi029 ceph-mon[32610]: pgmap v5679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9844.585209 secs 2024-07-27T09:33:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:42 smithi077 ceph-mon[37153]: pgmap v5679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9844.585209 secs 2024-07-27T09:33:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:44 smithi047 ceph-mon[36835]: pgmap v5680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:44 smithi029 ceph-mon[32610]: pgmap v5680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:44 smithi077 ceph-mon[37153]: pgmap v5680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:33:45.797+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:33:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:33:45.797+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:33:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:33:45.797+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:33:45.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:33:45.802+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:33:45.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:33:45.802+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:33:45.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:33:45.802+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:33:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:46 smithi047 ceph-mon[36835]: pgmap v5681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:46 smithi029 ceph-mon[32610]: pgmap v5681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:46 smithi077 ceph-mon[37153]: pgmap v5681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9849.585328 secs 2024-07-27T09:33:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9849.585328 secs 2024-07-27T09:33:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9849.585328 secs 2024-07-27T09:33:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:48 smithi047 ceph-mon[36835]: pgmap v5682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:48 smithi029 ceph-mon[32610]: pgmap v5682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:49.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:48 smithi077 ceph-mon[37153]: pgmap v5682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:50 smithi047 ceph-mon[36835]: pgmap v5683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:33:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:50 smithi029 ceph-mon[32610]: pgmap v5683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:33:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:50 smithi077 ceph-mon[37153]: pgmap v5683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:33:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:52 smithi047 ceph-mon[36835]: pgmap v5684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9854.585448 secs 2024-07-27T09:33:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:52 smithi029 ceph-mon[32610]: pgmap v5684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9854.585448 secs 2024-07-27T09:33:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:52 smithi077 ceph-mon[37153]: pgmap v5684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9854.585448 secs 2024-07-27T09:33:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:54 smithi047 ceph-mon[36835]: pgmap v5685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:54 smithi029 ceph-mon[32610]: pgmap v5685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:54 smithi077 ceph-mon[37153]: pgmap v5685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:33:57.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:56 smithi047 ceph-mon[36835]: pgmap v5686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:56 smithi029 ceph-mon[32610]: pgmap v5686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:56 smithi077 ceph-mon[37153]: pgmap v5686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:33:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9859.585576 secs 2024-07-27T09:33:58.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9859.585576 secs 2024-07-27T09:33:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9859.585576 secs 2024-07-27T09:33:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:33:58 smithi047 ceph-mon[36835]: pgmap v5687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:33:58 smithi029 ceph-mon[32610]: pgmap v5687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:33:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:33:58 smithi077 ceph-mon[37153]: pgmap v5687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:00.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:00.806+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:34:00.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:00.806+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:34:00.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:00.806+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:34:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:00.813+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:34:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:00.813+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:34:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:00.813+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:34:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:00 smithi047 ceph-mon[36835]: pgmap v5688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:34:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:00 smithi029 ceph-mon[32610]: pgmap v5688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:34:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:00 smithi077 ceph-mon[37153]: pgmap v5688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:34:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:02 smithi047 ceph-mon[36835]: pgmap v5689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9864.585706 secs 2024-07-27T09:34:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:02 smithi029 ceph-mon[32610]: pgmap v5689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9864.585706 secs 2024-07-27T09:34:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:02 smithi077 ceph-mon[37153]: pgmap v5689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9864.585706 secs 2024-07-27T09:34:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:04 smithi047 ceph-mon[36835]: pgmap v5690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:04 smithi029 ceph-mon[32610]: pgmap v5690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:04 smithi077 ceph-mon[37153]: pgmap v5690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:06 smithi047 ceph-mon[36835]: pgmap v5691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:06 smithi029 ceph-mon[32610]: pgmap v5691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:06 smithi077 ceph-mon[37153]: pgmap v5691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9869.585801 secs 2024-07-27T09:34:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9869.585801 secs 2024-07-27T09:34:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9869.585801 secs 2024-07-27T09:34:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:08 smithi029 ceph-mon[32610]: pgmap v5692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:08 smithi077 ceph-mon[37153]: pgmap v5692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:08 smithi047 ceph-mon[36835]: pgmap v5692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:10 smithi029 ceph-mon[32610]: pgmap v5693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:34:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:10 smithi077 ceph-mon[37153]: pgmap v5693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:34:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:10 smithi047 ceph-mon[36835]: pgmap v5693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:34:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:12 smithi029 ceph-mon[32610]: pgmap v5694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9874.585924 secs 2024-07-27T09:34:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:12 smithi077 ceph-mon[37153]: pgmap v5694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9874.585924 secs 2024-07-27T09:34:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:12 smithi047 ceph-mon[36835]: pgmap v5694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9874.585924 secs 2024-07-27T09:34:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:34:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:34:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:34:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:34:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:34:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:34:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:34:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:34:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:34:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:34:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:34:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:34:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:34:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:34:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:34:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:34:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:34:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:34:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:34:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:34:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:34:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:34:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:34:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:34:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:14 smithi077 ceph-mon[37153]: pgmap v5695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:14 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:34:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:14 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:34:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:14 smithi077 ceph-mon[37153]: pgmap v5696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:34:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:14 smithi077 ceph-mon[37153]: pgmap v5697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:34:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:14 smithi047 ceph-mon[36835]: pgmap v5695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:14 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:34:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:14 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:34:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:14 smithi047 ceph-mon[36835]: pgmap v5696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:34:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:14 smithi047 ceph-mon[36835]: pgmap v5697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:34:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:14 smithi029 ceph-mon[32610]: pgmap v5695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:14 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:34:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:14 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:34:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:14 smithi029 ceph-mon[32610]: pgmap v5696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:34:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:14 smithi029 ceph-mon[32610]: pgmap v5697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:34:15.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:15.820+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:34:15.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:15.820+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:34:15.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:15.820+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:15.821+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:15.821+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:34:15.826 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:15.821+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:34:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:16 smithi077 ceph-mon[37153]: pgmap v5698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:34:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:16 smithi047 ceph-mon[36835]: pgmap v5698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:34:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:16 smithi029 ceph-mon[32610]: pgmap v5698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:34:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9879.586023 secs 2024-07-27T09:34:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9879.586023 secs 2024-07-27T09:34:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9879.586023 secs 2024-07-27T09:34:19.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:18 smithi077 ceph-mon[37153]: pgmap v5699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:34:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:18 smithi047 ceph-mon[36835]: pgmap v5699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:34:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:18 smithi029 ceph-mon[32610]: pgmap v5699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:34:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:20 smithi077 ceph-mon[37153]: pgmap v5700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s 2024-07-27T09:34:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:20 smithi047 ceph-mon[36835]: pgmap v5700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s 2024-07-27T09:34:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:20 smithi029 ceph-mon[32610]: pgmap v5700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 6 op/s 2024-07-27T09:34:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:22 smithi077 ceph-mon[37153]: pgmap v5701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:34:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9884.586147 secs 2024-07-27T09:34:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:22 smithi047 ceph-mon[36835]: pgmap v5701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:34:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9884.586147 secs 2024-07-27T09:34:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:22 smithi029 ceph-mon[32610]: pgmap v5701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:34:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9884.586147 secs 2024-07-27T09:34:25.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:24 smithi047 ceph-mon[36835]: pgmap v5702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:34:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:24 smithi077 ceph-mon[37153]: pgmap v5702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:34:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:24 smithi029 ceph-mon[32610]: pgmap v5702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:34:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:26 smithi077 ceph-mon[37153]: pgmap v5703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:27.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:26 smithi047 ceph-mon[36835]: pgmap v5703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:26 smithi029 ceph-mon[32610]: pgmap v5703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9889.586247 secs 2024-07-27T09:34:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9889.586247 secs 2024-07-27T09:34:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9889.586247 secs 2024-07-27T09:34:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:28 smithi077 ceph-mon[37153]: pgmap v5704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:34:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:28 smithi047 ceph-mon[36835]: pgmap v5704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:34:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:28 smithi029 ceph-mon[32610]: pgmap v5704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:34:30.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:30.821+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:34:30.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:30.821+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:34:30.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:30.821+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:34:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:30.829+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:34:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:30.829+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:34:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:30.829+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:34:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:30 smithi029 ceph-mon[32610]: pgmap v5705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:34:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:30 smithi077 ceph-mon[37153]: pgmap v5705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:34:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:30 smithi047 ceph-mon[36835]: pgmap v5705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:34:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:33 smithi077 ceph-mon[37153]: pgmap v5706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9894.586375 secs 2024-07-27T09:34:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:33 smithi047 ceph-mon[36835]: pgmap v5706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9894.586375 secs 2024-07-27T09:34:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:33 smithi029 ceph-mon[32610]: pgmap v5706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9894.586375 secs 2024-07-27T09:34:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:35 smithi077 ceph-mon[37153]: pgmap v5707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:35 smithi047 ceph-mon[36835]: pgmap v5707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:35 smithi029 ceph-mon[32610]: pgmap v5707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:37 smithi077 ceph-mon[37153]: pgmap v5708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:37 smithi047 ceph-mon[36835]: pgmap v5708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:37 smithi029 ceph-mon[32610]: pgmap v5708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:38.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9899.586520 secs 2024-07-27T09:34:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9899.586520 secs 2024-07-27T09:34:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9899.586520 secs 2024-07-27T09:34:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:39 smithi077 ceph-mon[37153]: pgmap v5709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:34:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:39 smithi047 ceph-mon[36835]: pgmap v5709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:34:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:39 smithi029 ceph-mon[32610]: pgmap v5709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:34:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:41 smithi077 ceph-mon[37153]: pgmap v5710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:41 smithi047 ceph-mon[36835]: pgmap v5710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:41 smithi029 ceph-mon[32610]: pgmap v5710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:42.644+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-27T09:34:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:42.644+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T09:34:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:42.644+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4bc0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f733c019550 tx=0x7f733c05d000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T09:34:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:42.644+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4bc0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f733c019550 tx=0x7f733c05d000 comp rx=0 tx=0).stop 2024-07-27T09:34:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:42.644+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4bc0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T09:34:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:42.644+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 msgr2=0x7f73380b4bc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T09:34:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:42.644+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b4bc0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:34:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:42.644+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:34:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:42.644+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:34:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:42.645+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f733c004010 tx=0x7f733c07a000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:34:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:43 smithi077 ceph-mon[37153]: pgmap v5711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9904.586680 secs 2024-07-27T09:34:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:43 smithi047 ceph-mon[36835]: pgmap v5711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9904.586680 secs 2024-07-27T09:34:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:43 smithi029 ceph-mon[32610]: pgmap v5711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:34:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9904.586680 secs 2024-07-27T09:34:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:44.132+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:34:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:44.132+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:34:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:44.132+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f7320003690 con 0x7f7348108670 2024-07-27T09:34:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:44.132+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:34:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:44.132+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f7318003690 con 0x5645ed3392f0 2024-07-27T09:34:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:44.132+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:34:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:44.132+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:34:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:44.132+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:34:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:45 smithi077 ceph-mon[37153]: pgmap v5712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:45 smithi047 ceph-mon[36835]: pgmap v5712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:45 smithi029 ceph-mon[32610]: pgmap v5712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:45.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:45.823+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:34:45.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:45.823+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:34:45.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:45.823+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:34:45.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:45.845+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:34:45.846 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:45.845+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:34:45.847 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:34:45.845+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:34:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:47 smithi077 ceph-mon[37153]: pgmap v5713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:47 smithi047 ceph-mon[36835]: pgmap v5713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:47 smithi029 ceph-mon[32610]: pgmap v5713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9909.586843 secs 2024-07-27T09:34:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9909.586843 secs 2024-07-27T09:34:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9909.586843 secs 2024-07-27T09:34:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:49 smithi077 ceph-mon[37153]: pgmap v5714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:34:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:49 smithi047 ceph-mon[36835]: pgmap v5714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:34:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:49 smithi029 ceph-mon[32610]: pgmap v5714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:34:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:51 smithi077 ceph-mon[37153]: pgmap v5715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:51 smithi047 ceph-mon[36835]: pgmap v5715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:51 smithi029 ceph-mon[32610]: pgmap v5715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:53 smithi077 ceph-mon[37153]: pgmap v5716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:34:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9914.586978 secs 2024-07-27T09:34:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:53 smithi047 ceph-mon[36835]: pgmap v5716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:34:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9914.586978 secs 2024-07-27T09:34:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:53 smithi029 ceph-mon[32610]: pgmap v5716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:34:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9914.586978 secs 2024-07-27T09:34:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:55 smithi077 ceph-mon[37153]: pgmap v5717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:55 smithi047 ceph-mon[36835]: pgmap v5717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:55 smithi029 ceph-mon[32610]: pgmap v5717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:34:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:57 smithi077 ceph-mon[37153]: pgmap v5718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:57 smithi047 ceph-mon[36835]: pgmap v5718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:57 smithi029 ceph-mon[32610]: pgmap v5718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:34:57.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:57.663+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T09:34:57.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:57.663+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T09:34:57.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:57.663+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b49b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7eff4001cac0 tx=0x7eff4001e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T09:34:57.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:57.663+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b49b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7eff4001cac0 tx=0x7eff4001e040 comp rx=0 tx=0).stop 2024-07-27T09:34:57.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:57.663+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b49b0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T09:34:57.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:57.663+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 msgr2=0x7eff3c0b49b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T09:34:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:57.663+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b3d40 0x7eff3c0b49b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:34:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:57.663+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:34:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:57.663+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:34:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:57.664+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7eff40048000 tx=0x7eff40038040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:34:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9919.587066 secs 2024-07-27T09:34:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9919.587066 secs 2024-07-27T09:34:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9919.587066 secs 2024-07-27T09:34:59.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:59.134+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:34:59.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:59.134+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:34:59.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:59.135+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T09:34:59.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:59.135+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T09:34:59.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:59.135+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T09:34:59.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:59.135+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:34:59.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:59.135+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T09:34:59.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:34:59.135+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:34:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:34:59 smithi077 ceph-mon[37153]: pgmap v5719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:34:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:34:59 smithi047 ceph-mon[36835]: pgmap v5719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:34:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:34:59 smithi029 ceph-mon[32610]: pgmap v5719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:35:00.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:35:00.826+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:35:00.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:35:00.826+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:35:00.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:35:00.826+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:35:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:35:00.861+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:35:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:35:00.861+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:35:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:35:00.861+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:35:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:01 smithi029 ceph-mon[32610]: pgmap v5720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:01 smithi077 ceph-mon[37153]: pgmap v5720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:01 smithi047 ceph-mon[36835]: pgmap v5720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:03 smithi077 ceph-mon[37153]: pgmap v5721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:35:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9924.587149 secs 2024-07-27T09:35:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:03 smithi047 ceph-mon[36835]: pgmap v5721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:35:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9924.587149 secs 2024-07-27T09:35:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:03 smithi029 ceph-mon[32610]: pgmap v5721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:35:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9924.587149 secs 2024-07-27T09:35:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:05 smithi077 ceph-mon[37153]: pgmap v5722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:05 smithi047 ceph-mon[36835]: pgmap v5722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:05 smithi029 ceph-mon[32610]: pgmap v5722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:07 smithi077 ceph-mon[37153]: pgmap v5723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:07 smithi047 ceph-mon[36835]: pgmap v5723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:07 smithi029 ceph-mon[32610]: pgmap v5723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9929.587207 secs 2024-07-27T09:35:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9929.587207 secs 2024-07-27T09:35:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9929.587207 secs 2024-07-27T09:35:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:09 smithi029 ceph-mon[32610]: pgmap v5724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:35:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:09 smithi077 ceph-mon[37153]: pgmap v5724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:35:09.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:09 smithi047 ceph-mon[36835]: pgmap v5724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:35:11.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:11 smithi077 ceph-mon[37153]: pgmap v5725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:11 smithi047 ceph-mon[36835]: pgmap v5725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:11 smithi029 ceph-mon[32610]: pgmap v5725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:13 smithi077 ceph-mon[37153]: pgmap v5726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:35:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9934.587321 secs 2024-07-27T09:35:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:13 smithi047 ceph-mon[36835]: pgmap v5726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:35:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9934.587321 secs 2024-07-27T09:35:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:13 smithi029 ceph-mon[32610]: pgmap v5726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:35:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9934.587321 secs 2024-07-27T09:35:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:35:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:35:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:35:15.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:15 smithi029 ceph-mon[32610]: pgmap v5727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:15.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:15 smithi077 ceph-mon[37153]: pgmap v5727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:35:15.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:35:15.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:35:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:15 smithi047 ceph-mon[36835]: pgmap v5727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:35:15.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:35:15.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:35:15.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:35:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:35:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:35:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:35:15.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:35:15.829+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:35:15.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:35:15.830+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:35:15.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:35:15.830+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:35:15.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:35:15.861+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:35:15.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:35:15.861+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:35:15.863 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:35:15.861+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:35:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:16 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:35:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:16 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:35:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:16 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:35:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:16 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:35:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:16 smithi077 ceph-mon[37153]: pgmap v5728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:35:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:16 smithi077 ceph-mon[37153]: pgmap v5729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:35:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:16 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:35:16.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:16 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:35:16.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:16 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:35:16.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:16 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:35:16.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:16 smithi047 ceph-mon[36835]: pgmap v5728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:35:16.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:16 smithi047 ceph-mon[36835]: pgmap v5729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:35:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:16 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:35:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:16 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:35:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:16 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:35:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:16 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:35:16.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:16 smithi029 ceph-mon[32610]: pgmap v5728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:35:16.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:16 smithi029 ceph-mon[32610]: pgmap v5729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:35:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:18 smithi077 ceph-mon[37153]: pgmap v5730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:35:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9939.587457 secs 2024-07-27T09:35:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:18 smithi047 ceph-mon[36835]: pgmap v5730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:35:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9939.587457 secs 2024-07-27T09:35:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:18 smithi029 ceph-mon[32610]: pgmap v5730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:35:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9939.587457 secs 2024-07-27T09:35:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:20 smithi077 ceph-mon[37153]: pgmap v5731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:35:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:20 smithi047 ceph-mon[36835]: pgmap v5731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:35:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:20 smithi029 ceph-mon[32610]: pgmap v5731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:35:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:22 smithi077 ceph-mon[37153]: pgmap v5732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:35:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:22 smithi047 ceph-mon[36835]: pgmap v5732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:35:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:22 smithi029 ceph-mon[32610]: pgmap v5732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:35:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9944.587581 secs 2024-07-27T09:35:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9944.587581 secs 2024-07-27T09:35:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9944.587581 secs 2024-07-27T09:35:24.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:24 smithi029 ceph-mon[32610]: pgmap v5733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:35:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:24 smithi047 ceph-mon[36835]: pgmap v5733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:35:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:24 smithi077 ceph-mon[37153]: pgmap v5733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:35:26.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:26 smithi047 ceph-mon[36835]: pgmap v5734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:35:26.476 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:26 smithi029 ceph-mon[32610]: pgmap v5734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:35:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:26 smithi077 ceph-mon[37153]: pgmap v5734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:35:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:28 smithi047 ceph-mon[36835]: pgmap v5735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:35:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9949.587721 secs 2024-07-27T09:35:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:28 smithi029 ceph-mon[32610]: pgmap v5735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:35:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9949.587721 secs 2024-07-27T09:35:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:28 smithi077 ceph-mon[37153]: pgmap v5735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:35:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9949.587721 secs 2024-07-27T09:35:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:30 smithi047 ceph-mon[36835]: pgmap v5736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:30 smithi029 ceph-mon[32610]: pgmap v5736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:30 smithi077 ceph-mon[37153]: pgmap v5736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:30.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:35:30.840+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:35:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:35:30.840+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:35:30.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:35:30.840+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:35:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:35:30.877+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:35:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:35:30.877+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:35:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:35:30.877+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:35:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:32 smithi047 ceph-mon[36835]: pgmap v5737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:32 smithi029 ceph-mon[32610]: pgmap v5737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:32 smithi077 ceph-mon[37153]: pgmap v5737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9954.587851 secs 2024-07-27T09:35:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9954.587851 secs 2024-07-27T09:35:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9954.587851 secs 2024-07-27T09:35:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:34 smithi047 ceph-mon[36835]: pgmap v5738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:35:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:34 smithi029 ceph-mon[32610]: pgmap v5738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:35:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:34 smithi077 ceph-mon[37153]: pgmap v5738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:35:36.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:36 smithi047 ceph-mon[36835]: pgmap v5739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:35:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:36 smithi029 ceph-mon[32610]: pgmap v5739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:35:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:36 smithi077 ceph-mon[37153]: pgmap v5739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:35:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:38 smithi047 ceph-mon[36835]: pgmap v5740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:35:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9959.587977 secs 2024-07-27T09:35:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:38 smithi029 ceph-mon[32610]: pgmap v5740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:35:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9959.587977 secs 2024-07-27T09:35:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:38 smithi077 ceph-mon[37153]: pgmap v5740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:35:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9959.587977 secs 2024-07-27T09:35:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:40 smithi047 ceph-mon[36835]: pgmap v5741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:40 smithi029 ceph-mon[32610]: pgmap v5741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:40 smithi077 ceph-mon[37153]: pgmap v5741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:41 smithi047 ceph-mon[36835]: pgmap v5742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:41 smithi029 ceph-mon[32610]: pgmap v5742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:41 smithi077 ceph-mon[37153]: pgmap v5742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9964.588074 secs 2024-07-27T09:35:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9964.588074 secs 2024-07-27T09:35:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9964.588074 secs 2024-07-27T09:35:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:43 smithi077 ceph-mon[37153]: pgmap v5743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:35:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:43 smithi047 ceph-mon[36835]: pgmap v5743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:35:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:43 smithi029 ceph-mon[32610]: pgmap v5743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:35:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:45 smithi047 ceph-mon[36835]: pgmap v5744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:35:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:45 smithi029 ceph-mon[32610]: pgmap v5744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:35:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:45 smithi077 ceph-mon[37153]: pgmap v5744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:35:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:35:45.856+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:35:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:35:45.856+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:35:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:35:45.856+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:35:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:35:45.885+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:35:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:35:45.885+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:35:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:35:45.885+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:35:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:47 smithi047 ceph-mon[36835]: pgmap v5745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:35:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9969.588230 secs 2024-07-27T09:35:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:47 smithi029 ceph-mon[32610]: pgmap v5745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:35:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9969.588230 secs 2024-07-27T09:35:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:47 smithi077 ceph-mon[37153]: pgmap v5745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:35:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9969.588230 secs 2024-07-27T09:35:49.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:49 smithi047 ceph-mon[36835]: pgmap v5746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:49 smithi029 ceph-mon[32610]: pgmap v5746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:49 smithi077 ceph-mon[37153]: pgmap v5746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:51 smithi047 ceph-mon[36835]: pgmap v5747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:51 smithi029 ceph-mon[32610]: pgmap v5747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:51 smithi077 ceph-mon[37153]: pgmap v5747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:35:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9974.588361 secs 2024-07-27T09:35:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9974.588361 secs 2024-07-27T09:35:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9974.588361 secs 2024-07-27T09:35:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:53 smithi047 ceph-mon[36835]: pgmap v5748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:35:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:53 smithi029 ceph-mon[32610]: pgmap v5748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:35:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:53 smithi077 ceph-mon[37153]: pgmap v5748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:35:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:55 smithi029 ceph-mon[32610]: pgmap v5749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:35:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:55 smithi077 ceph-mon[37153]: pgmap v5749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:35:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:55 smithi047 ceph-mon[36835]: pgmap v5749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:35:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:57 smithi029 ceph-mon[32610]: pgmap v5750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:35:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9979.588489 secs 2024-07-27T09:35:57.749 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:57 smithi047 ceph-mon[36835]: pgmap v5750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:35:57.749 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9979.588489 secs 2024-07-27T09:35:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:57 smithi077 ceph-mon[37153]: pgmap v5750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:35:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9979.588489 secs 2024-07-27T09:35:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:35:59 smithi029 ceph-mon[32610]: pgmap v5751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:35:59 smithi077 ceph-mon[37153]: pgmap v5751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:35:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:35:59 smithi047 ceph-mon[36835]: pgmap v5751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:00.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:36:00.872+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:36:00.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:36:00.872+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:36:00.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:36:00.872+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:36:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:36:00.894+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:36:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:36:00.894+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:36:00.895 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:36:00.894+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:36:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:01 smithi029 ceph-mon[32610]: pgmap v5752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:01 smithi077 ceph-mon[37153]: pgmap v5752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:01 smithi047 ceph-mon[36835]: pgmap v5752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9984.588615 secs 2024-07-27T09:36:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9984.588615 secs 2024-07-27T09:36:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9984.588615 secs 2024-07-27T09:36:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:03 smithi077 ceph-mon[37153]: pgmap v5753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:36:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:03 smithi047 ceph-mon[36835]: pgmap v5753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:36:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:03 smithi029 ceph-mon[32610]: pgmap v5753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:36:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:05 smithi077 ceph-mon[37153]: pgmap v5754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:36:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:05 smithi047 ceph-mon[36835]: pgmap v5754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:36:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:05 smithi029 ceph-mon[32610]: pgmap v5754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:36:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:07 smithi077 ceph-mon[37153]: pgmap v5755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:36:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9989.588742 secs 2024-07-27T09:36:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:07 smithi047 ceph-mon[36835]: pgmap v5755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:36:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9989.588742 secs 2024-07-27T09:36:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:07 smithi029 ceph-mon[32610]: pgmap v5755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:36:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9989.588742 secs 2024-07-27T09:36:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:09 smithi077 ceph-mon[37153]: pgmap v5756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:09 smithi047 ceph-mon[36835]: pgmap v5756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:09 smithi029 ceph-mon[32610]: pgmap v5756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:11 smithi029 ceph-mon[32610]: pgmap v5757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:11 smithi077 ceph-mon[37153]: pgmap v5757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:11 smithi047 ceph-mon[36835]: pgmap v5757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9994.588870 secs 2024-07-27T09:36:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9994.588870 secs 2024-07-27T09:36:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9994.588870 secs 2024-07-27T09:36:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:13 smithi077 ceph-mon[37153]: pgmap v5758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:36:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:13 smithi047 ceph-mon[36835]: pgmap v5758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:36:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:13 smithi029 ceph-mon[32610]: pgmap v5758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:36:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:36:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:36:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:36:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:15 smithi077 ceph-mon[37153]: pgmap v5759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:36:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:36:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:36:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:15 smithi077 ceph-mon[37153]: pgmap v5760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:36:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:15 smithi077 ceph-mon[37153]: pgmap v5761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:36:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:36:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:36:15.886+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:36:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:36:15.886+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:36:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:36:15.886+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:36:15.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:15 smithi029 ceph-mon[32610]: pgmap v5759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:36:15.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:36:15.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:36:15.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:15 smithi029 ceph-mon[32610]: pgmap v5760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:36:15.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:15 smithi029 ceph-mon[32610]: pgmap v5761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:36:15.890 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:36:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:36:15.908+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:36:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:36:15.908+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:36:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:36:15.908+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:36:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:15 smithi047 ceph-mon[36835]: pgmap v5759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:36:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:36:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:36:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:15 smithi047 ceph-mon[36835]: pgmap v5760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:36:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:15 smithi047 ceph-mon[36835]: pgmap v5761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:36:15.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:36:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 9999.588996 secs 2024-07-27T09:36:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:17 smithi077 ceph-mon[37153]: pgmap v5762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:36:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 9999.588996 secs 2024-07-27T09:36:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:17 smithi047 ceph-mon[36835]: pgmap v5762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:36:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 9999.588996 secs 2024-07-27T09:36:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:17 smithi029 ceph-mon[32610]: pgmap v5762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:36:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:19 smithi077 ceph-mon[37153]: pgmap v5763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:19 smithi047 ceph-mon[36835]: pgmap v5763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:19 smithi029 ceph-mon[32610]: pgmap v5763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:21 smithi077 ceph-mon[37153]: pgmap v5764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:36:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:21 smithi047 ceph-mon[36835]: pgmap v5764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:36:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:21 smithi029 ceph-mon[32610]: pgmap v5764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:36:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10004.589144 secs 2024-07-27T09:36:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10004.589144 secs 2024-07-27T09:36:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10004.589144 secs 2024-07-27T09:36:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:23 smithi077 ceph-mon[37153]: pgmap v5765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:36:24.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:23 smithi047 ceph-mon[36835]: pgmap v5765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:36:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:23 smithi029 ceph-mon[32610]: pgmap v5765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:36:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:26 smithi077 ceph-mon[37153]: pgmap v5766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:36:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:26 smithi047 ceph-mon[36835]: pgmap v5766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:36:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:26 smithi029 ceph-mon[32610]: pgmap v5766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:36:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10009.589242 secs 2024-07-27T09:36:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:28 smithi077 ceph-mon[37153]: pgmap v5767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:36:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10009.589242 secs 2024-07-27T09:36:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:28 smithi047 ceph-mon[36835]: pgmap v5767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:36:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10009.589242 secs 2024-07-27T09:36:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:28 smithi029 ceph-mon[32610]: pgmap v5767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:36:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:30 smithi077 ceph-mon[37153]: pgmap v5768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:30 smithi047 ceph-mon[36835]: pgmap v5768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:30 smithi029 ceph-mon[32610]: pgmap v5768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:30.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:36:30.902+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:36:30.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:36:30.902+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:36:30.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:36:30.902+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:36:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:36:30.923+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:36:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:36:30.923+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:36:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:36:30.923+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:36:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:32 smithi077 ceph-mon[37153]: pgmap v5769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:32 smithi047 ceph-mon[36835]: pgmap v5769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:32 smithi029 ceph-mon[32610]: pgmap v5769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10014.589329 secs 2024-07-27T09:36:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10014.589329 secs 2024-07-27T09:36:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10014.589329 secs 2024-07-27T09:36:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:34 smithi077 ceph-mon[37153]: pgmap v5770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T09:36:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:34 smithi047 ceph-mon[36835]: pgmap v5770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T09:36:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:34 smithi029 ceph-mon[32610]: pgmap v5770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T09:36:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:36 smithi077 ceph-mon[37153]: pgmap v5771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:36:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:36 smithi047 ceph-mon[36835]: pgmap v5771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:36:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:36 smithi029 ceph-mon[32610]: pgmap v5771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:36:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10019.589462 secs 2024-07-27T09:36:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:38 smithi077 ceph-mon[37153]: pgmap v5772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T09:36:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10019.589462 secs 2024-07-27T09:36:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:38 smithi047 ceph-mon[36835]: pgmap v5772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T09:36:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10019.589462 secs 2024-07-27T09:36:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:38 smithi029 ceph-mon[32610]: pgmap v5772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T09:36:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:40 smithi077 ceph-mon[37153]: pgmap v5773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:36:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:40 smithi047 ceph-mon[36835]: pgmap v5773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:36:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:40 smithi029 ceph-mon[32610]: pgmap v5773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:36:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:42 smithi077 ceph-mon[37153]: pgmap v5774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:36:42.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:42 smithi047 ceph-mon[36835]: pgmap v5774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:36:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:42 smithi029 ceph-mon[32610]: pgmap v5774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:36:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10024.589566 secs 2024-07-27T09:36:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10024.589566 secs 2024-07-27T09:36:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10024.589566 secs 2024-07-27T09:36:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:44 smithi077 ceph-mon[37153]: pgmap v5775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T09:36:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:44 smithi047 ceph-mon[36835]: pgmap v5775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T09:36:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:44 smithi029 ceph-mon[32610]: pgmap v5775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T09:36:45.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:36:45.918+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:36:45.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:36:45.918+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:36:45.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:36:45.918+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:36:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:36:45.935+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:36:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:36:45.935+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:36:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:36:45.935+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:36:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:46 smithi029 ceph-mon[32610]: pgmap v5776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:46 smithi077 ceph-mon[37153]: pgmap v5776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:46 smithi047 ceph-mon[36835]: pgmap v5776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10029.589694 secs 2024-07-27T09:36:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:48 smithi077 ceph-mon[37153]: pgmap v5777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:36:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10029.589694 secs 2024-07-27T09:36:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:48 smithi047 ceph-mon[36835]: pgmap v5777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:36:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10029.589694 secs 2024-07-27T09:36:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:48 smithi029 ceph-mon[32610]: pgmap v5777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:36:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:50 smithi077 ceph-mon[37153]: pgmap v5778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:50 smithi047 ceph-mon[36835]: pgmap v5778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:50 smithi029 ceph-mon[32610]: pgmap v5778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:36:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:52 smithi077 ceph-mon[37153]: pgmap v5779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:52 smithi047 ceph-mon[36835]: pgmap v5779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:52 smithi029 ceph-mon[32610]: pgmap v5779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10034.589827 secs 2024-07-27T09:36:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10034.589827 secs 2024-07-27T09:36:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10034.589827 secs 2024-07-27T09:36:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:54 smithi077 ceph-mon[37153]: pgmap v5780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:36:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:54 smithi047 ceph-mon[36835]: pgmap v5780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:36:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:54 smithi029 ceph-mon[32610]: pgmap v5780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:36:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:56 smithi077 ceph-mon[37153]: pgmap v5781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:56 smithi047 ceph-mon[36835]: pgmap v5781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:56 smithi029 ceph-mon[32610]: pgmap v5781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:36:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10039.589950 secs 2024-07-27T09:36:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:36:58 smithi077 ceph-mon[37153]: pgmap v5782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:36:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10039.589950 secs 2024-07-27T09:36:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:36:58 smithi047 ceph-mon[36835]: pgmap v5782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:36:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10039.589950 secs 2024-07-27T09:36:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:36:58 smithi029 ceph-mon[32610]: pgmap v5782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:37:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:00 smithi077 ceph-mon[37153]: pgmap v5783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:37:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:00 smithi047 ceph-mon[36835]: pgmap v5783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:37:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:00 smithi029 ceph-mon[32610]: pgmap v5783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:37:00.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:00.933+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:37:00.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:00.934+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:37:00.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:00.934+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:37:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:00.951+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:37:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:00.951+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:37:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:00.951+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:37:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:02 smithi077 ceph-mon[37153]: pgmap v5784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:02 smithi047 ceph-mon[36835]: pgmap v5784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:02 smithi029 ceph-mon[32610]: pgmap v5784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10044.590036 secs 2024-07-27T09:37:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10044.590036 secs 2024-07-27T09:37:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10044.590036 secs 2024-07-27T09:37:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:04 smithi077 ceph-mon[37153]: pgmap v5785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:37:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:04 smithi047 ceph-mon[36835]: pgmap v5785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:37:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:04 smithi029 ceph-mon[32610]: pgmap v5785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:37:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:06 smithi077 ceph-mon[37153]: pgmap v5786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:06 smithi047 ceph-mon[36835]: pgmap v5786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:06 smithi029 ceph-mon[32610]: pgmap v5786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10049.590141 secs 2024-07-27T09:37:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:08 smithi077 ceph-mon[37153]: pgmap v5787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:37:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10049.590141 secs 2024-07-27T09:37:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:08 smithi047 ceph-mon[36835]: pgmap v5787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:37:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10049.590141 secs 2024-07-27T09:37:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:08 smithi029 ceph-mon[32610]: pgmap v5787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:37:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:10 smithi077 ceph-mon[37153]: pgmap v5788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:37:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:10 smithi047 ceph-mon[36835]: pgmap v5788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:37:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:10 smithi029 ceph-mon[32610]: pgmap v5788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:37:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:12 smithi077 ceph-mon[37153]: pgmap v5789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:12 smithi047 ceph-mon[36835]: pgmap v5789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:12 smithi029 ceph-mon[32610]: pgmap v5789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10054.590246 secs 2024-07-27T09:37:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10054.590246 secs 2024-07-27T09:37:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10054.590246 secs 2024-07-27T09:37:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:14 smithi077 ceph-mon[37153]: pgmap v5790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:37:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:14 smithi047 ceph-mon[36835]: pgmap v5790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:37:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:14 smithi029 ceph-mon[32610]: pgmap v5790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:37:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:37:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:37:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:37:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:15.948+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:37:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:15.948+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:37:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:15.948+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:37:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:15.966+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:37:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:15.966+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:37:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:15.966+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:37:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:16 smithi029 ceph-mon[32610]: pgmap v5791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:37:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:37:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:37:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:16 smithi047 ceph-mon[36835]: pgmap v5791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:16.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:37:16.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:37:16.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:37:16.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:16 smithi077 ceph-mon[37153]: pgmap v5791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:16.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:37:16.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:37:16.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:37:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:17 smithi047 ceph-mon[36835]: pgmap v5792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:37:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:17 smithi047 ceph-mon[36835]: pgmap v5793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:37:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:17 smithi029 ceph-mon[32610]: pgmap v5792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:37:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:17 smithi029 ceph-mon[32610]: pgmap v5793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:37:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:17 smithi077 ceph-mon[37153]: pgmap v5792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:37:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:17 smithi077 ceph-mon[37153]: pgmap v5793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:37:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10059.590369 secs 2024-07-27T09:37:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10059.590369 secs 2024-07-27T09:37:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10059.590369 secs 2024-07-27T09:37:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:19 smithi047 ceph-mon[36835]: pgmap v5794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:37:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:19 smithi029 ceph-mon[32610]: pgmap v5794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:37:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:19 smithi077 ceph-mon[37153]: pgmap v5794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:37:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:21 smithi047 ceph-mon[36835]: pgmap v5795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:37:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:21 smithi029 ceph-mon[32610]: pgmap v5795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:37:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:21 smithi077 ceph-mon[37153]: pgmap v5795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:37:23.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:23 smithi077 ceph-mon[37153]: pgmap v5796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:37:23.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10064.590499 secs 2024-07-27T09:37:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:23 smithi047 ceph-mon[36835]: pgmap v5796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:37:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10064.590499 secs 2024-07-27T09:37:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:23 smithi029 ceph-mon[32610]: pgmap v5796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:37:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10064.590499 secs 2024-07-27T09:37:25.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:25 smithi047 ceph-mon[36835]: pgmap v5797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:37:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:25 smithi029 ceph-mon[32610]: pgmap v5797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:37:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:25 smithi077 ceph-mon[37153]: pgmap v5797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:37:27.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:27 smithi047 ceph-mon[36835]: pgmap v5798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:37:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:27 smithi029 ceph-mon[32610]: pgmap v5798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:37:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:27 smithi077 ceph-mon[37153]: pgmap v5798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:37:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10069.590621 secs 2024-07-27T09:37:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10069.590621 secs 2024-07-27T09:37:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10069.590621 secs 2024-07-27T09:37:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:29 smithi047 ceph-mon[36835]: pgmap v5799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:37:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:29 smithi029 ceph-mon[32610]: pgmap v5799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:37:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:29 smithi077 ceph-mon[37153]: pgmap v5799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:37:30.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:30.957+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:37:30.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:30.957+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:37:30.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:30.957+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:37:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:30.981+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:37:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:30.981+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:37:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:30.981+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:37:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:31 smithi029 ceph-mon[32610]: pgmap v5800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:37:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:31 smithi047 ceph-mon[36835]: pgmap v5800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:37:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:31 smithi077 ceph-mon[37153]: pgmap v5800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:37:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:33 smithi047 ceph-mon[36835]: pgmap v5801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:37:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10074.590770 secs 2024-07-27T09:37:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:33 smithi029 ceph-mon[32610]: pgmap v5801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:37:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10074.590770 secs 2024-07-27T09:37:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:33 smithi077 ceph-mon[37153]: pgmap v5801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:37:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10074.590770 secs 2024-07-27T09:37:35.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:35 smithi047 ceph-mon[36835]: pgmap v5802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:37:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:35 smithi029 ceph-mon[32610]: pgmap v5802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:37:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:35 smithi077 ceph-mon[37153]: pgmap v5802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:37:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:37 smithi047 ceph-mon[36835]: pgmap v5803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:37 smithi029 ceph-mon[32610]: pgmap v5803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:37 smithi077 ceph-mon[37153]: pgmap v5803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10079.590894 secs 2024-07-27T09:37:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10079.590894 secs 2024-07-27T09:37:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10079.590894 secs 2024-07-27T09:37:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:39 smithi047 ceph-mon[36835]: pgmap v5804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:37:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:39 smithi029 ceph-mon[32610]: pgmap v5804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:37:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:39 smithi077 ceph-mon[37153]: pgmap v5804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:37:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:40 smithi047 ceph-mon[36835]: pgmap v5805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:37:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:40 smithi029 ceph-mon[32610]: pgmap v5805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:37:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:40 smithi077 ceph-mon[37153]: pgmap v5805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:37:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:42 smithi077 ceph-mon[37153]: pgmap v5806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:37:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10084.590994 secs 2024-07-27T09:37:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:42 smithi047 ceph-mon[36835]: pgmap v5806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:37:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10084.590994 secs 2024-07-27T09:37:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:42 smithi029 ceph-mon[32610]: pgmap v5806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:37:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10084.590994 secs 2024-07-27T09:37:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:44.159+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:37:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:44.159+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:37:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:44.159+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:37:44.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:44.159+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:37:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:44 smithi077 ceph-mon[37153]: pgmap v5807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:37:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:44 smithi047 ceph-mon[36835]: pgmap v5807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:37:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:44 smithi029 ceph-mon[32610]: pgmap v5807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:37:45.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:45.973+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:37:45.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:45.973+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:37:45.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:45.973+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:37:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:45.997+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:37:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:45.997+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:37:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:37:45.997+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:37:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:46 smithi077 ceph-mon[37153]: pgmap v5808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:37:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:46 smithi047 ceph-mon[36835]: pgmap v5808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:37:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:46 smithi029 ceph-mon[32610]: pgmap v5808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:37:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10089.591148 secs 2024-07-27T09:37:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10089.591148 secs 2024-07-27T09:37:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10089.591148 secs 2024-07-27T09:37:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:48 smithi077 ceph-mon[37153]: pgmap v5809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:37:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:48 smithi047 ceph-mon[36835]: pgmap v5809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:37:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:48 smithi029 ceph-mon[32610]: pgmap v5809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:37:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:50 smithi077 ceph-mon[37153]: pgmap v5810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:37:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:50 smithi047 ceph-mon[36835]: pgmap v5810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:37:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:50 smithi029 ceph-mon[32610]: pgmap v5810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:37:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:52 smithi077 ceph-mon[37153]: pgmap v5811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:37:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10094.591291 secs 2024-07-27T09:37:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:52 smithi047 ceph-mon[36835]: pgmap v5811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:37:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10094.591291 secs 2024-07-27T09:37:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:52 smithi029 ceph-mon[32610]: pgmap v5811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:37:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10094.591291 secs 2024-07-27T09:37:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:54 smithi077 ceph-mon[37153]: pgmap v5812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:37:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:54 smithi047 ceph-mon[36835]: pgmap v5812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:37:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:54 smithi029 ceph-mon[32610]: pgmap v5812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:37:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:56 smithi077 ceph-mon[37153]: pgmap v5813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:56 smithi047 ceph-mon[36835]: pgmap v5813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:56 smithi029 ceph-mon[32610]: pgmap v5813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:37:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10099.591430 secs 2024-07-27T09:37:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10099.591430 secs 2024-07-27T09:37:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10099.591430 secs 2024-07-27T09:37:58.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:37:58 smithi047 ceph-mon[36835]: pgmap v5814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:37:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:37:58 smithi077 ceph-mon[37153]: pgmap v5814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:37:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:37:58 smithi029 ceph-mon[32610]: pgmap v5814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:37:59.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:59.161+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:37:59.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:59.161+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T09:37:59.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:59.162+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T09:37:59.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:37:59.162+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:38:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:00 smithi077 ceph-mon[37153]: pgmap v5815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:38:00.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:00 smithi029 ceph-mon[32610]: pgmap v5815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:38:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:00 smithi047 ceph-mon[36835]: pgmap v5815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:38:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:38:00.989+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:38:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:38:00.989+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:38:00.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:38:00.989+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:38:01.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:38:01.013+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:38:01.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:38:01.013+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:38:01.014 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:38:01.013+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:38:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:02 smithi077 ceph-mon[37153]: pgmap v5816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10104.591565 secs 2024-07-27T09:38:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:02 smithi047 ceph-mon[36835]: pgmap v5816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10104.591565 secs 2024-07-27T09:38:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:02 smithi029 ceph-mon[32610]: pgmap v5816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10104.591565 secs 2024-07-27T09:38:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:04 smithi077 ceph-mon[37153]: pgmap v5817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:38:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:04 smithi047 ceph-mon[36835]: pgmap v5817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:38:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:04 smithi029 ceph-mon[32610]: pgmap v5817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:38:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:06 smithi077 ceph-mon[37153]: pgmap v5818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:06 smithi047 ceph-mon[36835]: pgmap v5818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:06 smithi029 ceph-mon[32610]: pgmap v5818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10109.591699 secs 2024-07-27T09:38:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10109.591699 secs 2024-07-27T09:38:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10109.591699 secs 2024-07-27T09:38:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:08 smithi047 ceph-mon[36835]: pgmap v5819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:38:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:08 smithi029 ceph-mon[32610]: pgmap v5819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:38:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:08 smithi077 ceph-mon[37153]: pgmap v5819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:38:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:10 smithi047 ceph-mon[36835]: pgmap v5820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:38:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:10 smithi029 ceph-mon[32610]: pgmap v5820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:38:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:10 smithi077 ceph-mon[37153]: pgmap v5820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:38:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:12 smithi047 ceph-mon[36835]: pgmap v5821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10114.591832 secs 2024-07-27T09:38:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:12 smithi029 ceph-mon[32610]: pgmap v5821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10114.591832 secs 2024-07-27T09:38:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:12 smithi077 ceph-mon[37153]: pgmap v5821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10114.591832 secs 2024-07-27T09:38:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:14 smithi047 ceph-mon[36835]: pgmap v5822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:38:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:14 smithi029 ceph-mon[32610]: pgmap v5822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:38:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:14 smithi077 ceph-mon[37153]: pgmap v5822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:38:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:38:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:38:16.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:38:16.003+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:38:16.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:38:16.003+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:38:16.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:38:16.003+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:38:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:38:16.027+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:38:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:38:16.027+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:38:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:38:16.027+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:38:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:38:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:16 smithi047 ceph-mon[36835]: pgmap v5823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:17.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:17.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:16 smithi029 ceph-mon[32610]: pgmap v5823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:16 smithi077 ceph-mon[37153]: pgmap v5823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10119.591904 secs 2024-07-27T09:38:18.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: pgmap v5824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:38:18.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:38:18.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:38:18.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: pgmap v5825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:38:18.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: pgmap v5826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:38:18.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10119.591904 secs 2024-07-27T09:38:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: pgmap v5824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:38:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:38:18.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:38:18.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: pgmap v5825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:38:18.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: pgmap v5826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:38:18.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10119.591904 secs 2024-07-27T09:38:18.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: pgmap v5824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:38:18.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:18.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:38:18.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:38:18.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: pgmap v5825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:38:18.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: pgmap v5826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:38:18.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:38:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:20 smithi077 ceph-mon[37153]: pgmap v5827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:38:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:20 smithi047 ceph-mon[36835]: pgmap v5827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:38:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:20 smithi029 ceph-mon[32610]: pgmap v5827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:38:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:22 smithi077 ceph-mon[37153]: pgmap v5828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:38:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:22 smithi047 ceph-mon[36835]: pgmap v5828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:38:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:22 smithi029 ceph-mon[32610]: pgmap v5828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:38:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10124.592042 secs 2024-07-27T09:38:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10124.592042 secs 2024-07-27T09:38:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10124.592042 secs 2024-07-27T09:38:24.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:24 smithi077 ceph-mon[37153]: pgmap v5829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:38:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:24 smithi047 ceph-mon[36835]: pgmap v5829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:38:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:24 smithi029 ceph-mon[32610]: pgmap v5829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:38:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:26 smithi077 ceph-mon[37153]: pgmap v5830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:38:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:26 smithi047 ceph-mon[36835]: pgmap v5830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:38:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:26 smithi029 ceph-mon[32610]: pgmap v5830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:38:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10129.592163 secs 2024-07-27T09:38:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10129.592163 secs 2024-07-27T09:38:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10129.592163 secs 2024-07-27T09:38:28.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:28 smithi047 ceph-mon[36835]: pgmap v5831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:38:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:28 smithi077 ceph-mon[37153]: pgmap v5831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:38:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:28 smithi029 ceph-mon[32610]: pgmap v5831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:38:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:30 smithi077 ceph-mon[37153]: pgmap v5832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:30 smithi047 ceph-mon[36835]: pgmap v5832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:30 smithi029 ceph-mon[32610]: pgmap v5832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:31.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:38:31.018+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:38:31.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:38:31.018+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:38:31.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:38:31.018+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:38:31.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:38:31.030+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:38:31.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:38:31.030+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:38:31.032 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:38:31.030+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:38:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:32 smithi029 ceph-mon[32610]: pgmap v5833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:32 smithi077 ceph-mon[37153]: pgmap v5833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:32 smithi047 ceph-mon[36835]: pgmap v5833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10134.592312 secs 2024-07-27T09:38:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10134.592312 secs 2024-07-27T09:38:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10134.592312 secs 2024-07-27T09:38:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:34 smithi077 ceph-mon[37153]: pgmap v5834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:38:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:34 smithi047 ceph-mon[36835]: pgmap v5834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:38:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:34 smithi029 ceph-mon[32610]: pgmap v5834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:38:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:36 smithi077 ceph-mon[37153]: pgmap v5835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:36 smithi047 ceph-mon[36835]: pgmap v5835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:36 smithi029 ceph-mon[32610]: pgmap v5835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10139.592462 secs 2024-07-27T09:38:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10139.592462 secs 2024-07-27T09:38:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10139.592462 secs 2024-07-27T09:38:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:38 smithi077 ceph-mon[37153]: pgmap v5836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:38:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:38 smithi047 ceph-mon[36835]: pgmap v5836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:38:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:38 smithi029 ceph-mon[32610]: pgmap v5836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:38:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:40 smithi077 ceph-mon[37153]: pgmap v5837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:40 smithi047 ceph-mon[36835]: pgmap v5837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:40 smithi029 ceph-mon[32610]: pgmap v5837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:42 smithi077 ceph-mon[37153]: pgmap v5838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:42 smithi047 ceph-mon[36835]: pgmap v5838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:42 smithi029 ceph-mon[32610]: pgmap v5838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10144.592595 secs 2024-07-27T09:38:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10144.592595 secs 2024-07-27T09:38:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10144.592595 secs 2024-07-27T09:38:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:44 smithi077 ceph-mon[37153]: pgmap v5839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:38:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:44 smithi047 ceph-mon[36835]: pgmap v5839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:38:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:44 smithi029 ceph-mon[32610]: pgmap v5839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:38:46.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:38:46.029+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:38:46.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:38:46.029+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:38:46.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:38:46.029+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:38:46.045+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:38:46.045+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:38:46.045+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:38:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:46 smithi029 ceph-mon[32610]: pgmap v5840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:46 smithi077 ceph-mon[37153]: pgmap v5840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:46 smithi047 ceph-mon[36835]: pgmap v5840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10149.592741 secs 2024-07-27T09:38:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10149.592741 secs 2024-07-27T09:38:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10149.592741 secs 2024-07-27T09:38:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:48 smithi077 ceph-mon[37153]: pgmap v5841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:38:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:48 smithi047 ceph-mon[36835]: pgmap v5841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:38:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:48 smithi029 ceph-mon[32610]: pgmap v5841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:38:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:50 smithi077 ceph-mon[37153]: pgmap v5842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:50 smithi047 ceph-mon[36835]: pgmap v5842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:50 smithi029 ceph-mon[32610]: pgmap v5842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:52 smithi077 ceph-mon[37153]: pgmap v5843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:52 smithi047 ceph-mon[36835]: pgmap v5843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:52 smithi029 ceph-mon[32610]: pgmap v5843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:38:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10154.592830 secs 2024-07-27T09:38:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10154.592830 secs 2024-07-27T09:38:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10154.592830 secs 2024-07-27T09:38:54.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:54 smithi077 ceph-mon[37153]: pgmap v5844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:38:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:54 smithi047 ceph-mon[36835]: pgmap v5844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:38:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:54 smithi029 ceph-mon[32610]: pgmap v5844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:38:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:56 smithi077 ceph-mon[37153]: pgmap v5845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:56 smithi047 ceph-mon[36835]: pgmap v5845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:56 smithi029 ceph-mon[32610]: pgmap v5845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:38:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10159.592942 secs 2024-07-27T09:38:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10159.592942 secs 2024-07-27T09:38:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10159.592942 secs 2024-07-27T09:38:58.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:38:58 smithi047 ceph-mon[36835]: pgmap v5846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:38:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:38:58 smithi077 ceph-mon[37153]: pgmap v5846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:38:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:38:58 smithi029 ceph-mon[32610]: pgmap v5846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:00 smithi077 ceph-mon[37153]: pgmap v5847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:00 smithi047 ceph-mon[36835]: pgmap v5847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:00 smithi029 ceph-mon[32610]: pgmap v5847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:01.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:39:01.034+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:39:01.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:39:01.034+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:39:01.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:39:01.034+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:39:01.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:39:01.051+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:39:01.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:39:01.051+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:39:01.052 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:39:01.051+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:39:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:02 smithi077 ceph-mon[37153]: pgmap v5848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:02 smithi047 ceph-mon[36835]: pgmap v5848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:02 smithi029 ceph-mon[32610]: pgmap v5848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10164.593042 secs 2024-07-27T09:39:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10164.593042 secs 2024-07-27T09:39:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10164.593042 secs 2024-07-27T09:39:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:04 smithi077 ceph-mon[37153]: pgmap v5849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:39:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:04 smithi047 ceph-mon[36835]: pgmap v5849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:39:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:04 smithi029 ceph-mon[32610]: pgmap v5849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:39:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:06 smithi077 ceph-mon[37153]: pgmap v5850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:06 smithi047 ceph-mon[36835]: pgmap v5850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:06 smithi029 ceph-mon[32610]: pgmap v5850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10169.593166 secs 2024-07-27T09:39:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10169.593166 secs 2024-07-27T09:39:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10169.593166 secs 2024-07-27T09:39:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:08 smithi077 ceph-mon[37153]: pgmap v5851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:08 smithi047 ceph-mon[36835]: pgmap v5851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:08 smithi029 ceph-mon[32610]: pgmap v5851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:10 smithi077 ceph-mon[37153]: pgmap v5852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:10 smithi047 ceph-mon[36835]: pgmap v5852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:10 smithi029 ceph-mon[32610]: pgmap v5852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:12 smithi077 ceph-mon[37153]: pgmap v5853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:12 smithi047 ceph-mon[36835]: pgmap v5853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:12 smithi029 ceph-mon[32610]: pgmap v5853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10174.593257 secs 2024-07-27T09:39:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10174.593257 secs 2024-07-27T09:39:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10174.593257 secs 2024-07-27T09:39:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:14 smithi077 ceph-mon[37153]: pgmap v5854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:39:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:14 smithi047 ceph-mon[36835]: pgmap v5854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:39:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:14 smithi029 ceph-mon[32610]: pgmap v5854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:39:16.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:39:16.038+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:39:16.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:39:16.038+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:39:16.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:39:16.038+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:39:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:39:16.066+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:39:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:39:16.066+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:39:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:39:16.066+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:39:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:16 smithi029 ceph-mon[32610]: pgmap v5855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:16 smithi077 ceph-mon[37153]: pgmap v5855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:16 smithi047 ceph-mon[36835]: pgmap v5855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10179.593399 secs 2024-07-27T09:39:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10179.593399 secs 2024-07-27T09:39:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10179.593399 secs 2024-07-27T09:39:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:18 smithi029 ceph-mon[32610]: pgmap v5856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:39:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:18 smithi077 ceph-mon[37153]: pgmap v5856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:39:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:18 smithi047 ceph-mon[36835]: pgmap v5856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:39:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:39:19.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:39:19.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:39:19.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:39:19.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:19.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:39:19.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:39:19.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:39:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:20 smithi077 ceph-mon[37153]: pgmap v5857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T09:39:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:20 smithi077 ceph-mon[37153]: pgmap v5858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:39:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:20 smithi047 ceph-mon[36835]: pgmap v5857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T09:39:20.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:20 smithi047 ceph-mon[36835]: pgmap v5858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:39:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:20 smithi029 ceph-mon[32610]: pgmap v5857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T09:39:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:20 smithi029 ceph-mon[32610]: pgmap v5858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:39:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:22 smithi047 ceph-mon[36835]: pgmap v5859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:39:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10184.593526 secs 2024-07-27T09:39:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:22 smithi029 ceph-mon[32610]: pgmap v5859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:39:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10184.593526 secs 2024-07-27T09:39:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:22 smithi077 ceph-mon[37153]: pgmap v5859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:39:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10184.593526 secs 2024-07-27T09:39:24.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:24 smithi077 ceph-mon[37153]: pgmap v5860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:39:24.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:24 smithi047 ceph-mon[36835]: pgmap v5860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:39:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:24 smithi029 ceph-mon[32610]: pgmap v5860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T09:39:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:26 smithi047 ceph-mon[36835]: pgmap v5861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T09:39:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:26 smithi029 ceph-mon[32610]: pgmap v5861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T09:39:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:26 smithi077 ceph-mon[37153]: pgmap v5861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T09:39:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10189.593657 secs 2024-07-27T09:39:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10189.593657 secs 2024-07-27T09:39:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10189.593657 secs 2024-07-27T09:39:28.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:28 smithi047 ceph-mon[36835]: pgmap v5862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T09:39:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:28 smithi029 ceph-mon[32610]: pgmap v5862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T09:39:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:28 smithi077 ceph-mon[37153]: pgmap v5862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T09:39:30.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:30 smithi047 ceph-mon[36835]: pgmap v5863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:39:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:30 smithi029 ceph-mon[32610]: pgmap v5863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:39:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:39:31.054+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:39:31.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:39:31.054+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:39:31.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:39:31.054+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:39:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:39:31.081+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:39:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:39:31.081+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:39:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:39:31.081+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:39:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:30 smithi077 ceph-mon[37153]: pgmap v5863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:39:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:32 smithi047 ceph-mon[36835]: pgmap v5864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10194.593725 secs 2024-07-27T09:39:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:32 smithi029 ceph-mon[32610]: pgmap v5864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10194.593725 secs 2024-07-27T09:39:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:32 smithi077 ceph-mon[37153]: pgmap v5864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10194.593725 secs 2024-07-27T09:39:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:34 smithi047 ceph-mon[36835]: pgmap v5865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:34 smithi029 ceph-mon[32610]: pgmap v5865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:34 smithi077 ceph-mon[37153]: pgmap v5865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:36 smithi047 ceph-mon[36835]: pgmap v5866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:36 smithi029 ceph-mon[32610]: pgmap v5866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:36 smithi077 ceph-mon[37153]: pgmap v5866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10199.593858 secs 2024-07-27T09:39:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10199.593858 secs 2024-07-27T09:39:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10199.593858 secs 2024-07-27T09:39:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:38 smithi047 ceph-mon[36835]: pgmap v5867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:38 smithi029 ceph-mon[32610]: pgmap v5867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:38 smithi077 ceph-mon[37153]: pgmap v5867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:40.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:40 smithi047 ceph-mon[36835]: pgmap v5868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:39:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:40 smithi029 ceph-mon[32610]: pgmap v5868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:39:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:40 smithi077 ceph-mon[37153]: pgmap v5868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:39:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:42 smithi029 ceph-mon[32610]: pgmap v5869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10204.593960 secs 2024-07-27T09:39:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:42 smithi077 ceph-mon[37153]: pgmap v5869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10204.593960 secs 2024-07-27T09:39:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:42 smithi047 ceph-mon[36835]: pgmap v5869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10204.593960 secs 2024-07-27T09:39:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:44 smithi029 ceph-mon[32610]: pgmap v5870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:44 smithi077 ceph-mon[37153]: pgmap v5870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:44 smithi047 ceph-mon[36835]: pgmap v5870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:46.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:39:46.069+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:39:46.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:39:46.069+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:39:46.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:39:46.069+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:39:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:39:46.097+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:39:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:39:46.097+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:39:46.099 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:39:46.097+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:39:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:46 smithi029 ceph-mon[32610]: pgmap v5871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:46 smithi077 ceph-mon[37153]: pgmap v5871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:46 smithi047 ceph-mon[36835]: pgmap v5871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10209.594090 secs 2024-07-27T09:39:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10209.594090 secs 2024-07-27T09:39:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10209.594090 secs 2024-07-27T09:39:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:48 smithi029 ceph-mon[32610]: pgmap v5872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:48 smithi077 ceph-mon[37153]: pgmap v5872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:48 smithi047 ceph-mon[36835]: pgmap v5872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:50 smithi077 ceph-mon[37153]: pgmap v5873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:39:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:50 smithi047 ceph-mon[36835]: pgmap v5873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:39:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:50 smithi029 ceph-mon[32610]: pgmap v5873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:39:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:52 smithi077 ceph-mon[37153]: pgmap v5874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10214.594225 secs 2024-07-27T09:39:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:52 smithi047 ceph-mon[36835]: pgmap v5874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10214.594225 secs 2024-07-27T09:39:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:52 smithi029 ceph-mon[32610]: pgmap v5874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10214.594225 secs 2024-07-27T09:39:55.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:54 smithi077 ceph-mon[37153]: pgmap v5875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:54 smithi047 ceph-mon[36835]: pgmap v5875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:54 smithi029 ceph-mon[32610]: pgmap v5875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:39:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:56 smithi077 ceph-mon[37153]: pgmap v5876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:56 smithi047 ceph-mon[36835]: pgmap v5876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:56 smithi029 ceph-mon[32610]: pgmap v5876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:39:58.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10219.594364 secs 2024-07-27T09:39:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10219.594364 secs 2024-07-27T09:39:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10219.594364 secs 2024-07-27T09:39:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:39:58 smithi077 ceph-mon[37153]: pgmap v5877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:39:58 smithi047 ceph-mon[36835]: pgmap v5877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:39:59.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:39:58 smithi029 ceph-mon[32610]: pgmap v5877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:01.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:00 smithi029 ceph-mon[32610]: pgmap v5878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:40:01.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:40:01.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:01.085+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:40:01.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:01.085+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:40:01.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:01.085+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:40:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:00 smithi077 ceph-mon[37153]: pgmap v5878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:40:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:40:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:01.110+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:40:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:01.110+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:40:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:01.110+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:40:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:00 smithi047 ceph-mon[36835]: pgmap v5878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:40:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:40:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:02 smithi077 ceph-mon[37153]: pgmap v5879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10224.594491 secs 2024-07-27T09:40:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:02 smithi047 ceph-mon[36835]: pgmap v5879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10224.594491 secs 2024-07-27T09:40:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:02 smithi029 ceph-mon[32610]: pgmap v5879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10224.594491 secs 2024-07-27T09:40:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:04 smithi077 ceph-mon[37153]: pgmap v5880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:40:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:04 smithi047 ceph-mon[36835]: pgmap v5880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:40:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:04 smithi029 ceph-mon[32610]: pgmap v5880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:40:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:06 smithi077 ceph-mon[37153]: pgmap v5881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:40:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:06 smithi047 ceph-mon[36835]: pgmap v5881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:40:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:06 smithi029 ceph-mon[32610]: pgmap v5881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:40:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10229.594631 secs 2024-07-27T09:40:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10229.594631 secs 2024-07-27T09:40:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10229.594631 secs 2024-07-27T09:40:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:08 smithi077 ceph-mon[37153]: pgmap v5882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:09.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:08 smithi047 ceph-mon[36835]: pgmap v5882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:08 smithi029 ceph-mon[32610]: pgmap v5882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:10 smithi077 ceph-mon[37153]: pgmap v5883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:40:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:10 smithi047 ceph-mon[36835]: pgmap v5883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:40:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:10 smithi029 ceph-mon[32610]: pgmap v5883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:40:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:12 smithi077 ceph-mon[37153]: pgmap v5884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10234.594713 secs 2024-07-27T09:40:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:12 smithi047 ceph-mon[36835]: pgmap v5884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10234.594713 secs 2024-07-27T09:40:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:12 smithi029 ceph-mon[32610]: pgmap v5884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10234.594713 secs 2024-07-27T09:40:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:14 smithi077 ceph-mon[37153]: pgmap v5885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:40:15.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:14 smithi047 ceph-mon[36835]: pgmap v5885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:40:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:14 smithi029 ceph-mon[32610]: pgmap v5885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:40:16.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:16.100+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:40:16.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:16.100+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:40:16.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:16.100+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:40:16.126 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:16.124+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:40:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:16.124+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:40:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:16.124+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:40:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:16 smithi077 ceph-mon[37153]: pgmap v5886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:16 smithi047 ceph-mon[36835]: pgmap v5886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:16 smithi029 ceph-mon[32610]: pgmap v5886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10239.594841 secs 2024-07-27T09:40:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10239.594841 secs 2024-07-27T09:40:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10239.594841 secs 2024-07-27T09:40:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:18 smithi077 ceph-mon[37153]: pgmap v5887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:19.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:18 smithi047 ceph-mon[36835]: pgmap v5887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:18 smithi029 ceph-mon[32610]: pgmap v5887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:40:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:40:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:40:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:40:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:20 smithi077 ceph-mon[37153]: pgmap v5888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:40:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:40:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:40:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:20 smithi077 ceph-mon[37153]: pgmap v5889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:40:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:20 smithi077 ceph-mon[37153]: pgmap v5890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:40:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:40:21.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:20 smithi029 ceph-mon[32610]: pgmap v5888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:40:21.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:40:21.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:40:21.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:20 smithi029 ceph-mon[32610]: pgmap v5889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:40:21.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:20 smithi029 ceph-mon[32610]: pgmap v5890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:40:21.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:40:21.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:20 smithi047 ceph-mon[36835]: pgmap v5888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:40:21.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:40:21.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:40:21.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:20 smithi047 ceph-mon[36835]: pgmap v5889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:40:21.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:20 smithi047 ceph-mon[36835]: pgmap v5890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:40:21.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:40:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:22 smithi077 ceph-mon[37153]: pgmap v5891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T09:40:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10244.594971 secs 2024-07-27T09:40:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:22 smithi047 ceph-mon[36835]: pgmap v5891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T09:40:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10244.594971 secs 2024-07-27T09:40:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:22 smithi029 ceph-mon[32610]: pgmap v5891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T09:40:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10244.594971 secs 2024-07-27T09:40:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:24 smithi077 ceph-mon[37153]: pgmap v5892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:40:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:24 smithi047 ceph-mon[36835]: pgmap v5892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:24 smithi029 ceph-mon[32610]: pgmap v5892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:40:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:26 smithi077 ceph-mon[37153]: pgmap v5893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:40:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:26 smithi047 ceph-mon[36835]: pgmap v5893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:40:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:26 smithi029 ceph-mon[32610]: pgmap v5893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:40:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10249.595120 secs 2024-07-27T09:40:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10249.595120 secs 2024-07-27T09:40:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10249.595120 secs 2024-07-27T09:40:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:28 smithi047 ceph-mon[36835]: pgmap v5894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:40:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:28 smithi029 ceph-mon[32610]: pgmap v5894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:40:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:28 smithi077 ceph-mon[37153]: pgmap v5894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:40:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:31.109+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:40:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:31.109+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:40:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:31.109+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:40:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:31.137+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:40:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:31.137+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:40:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:31.137+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:40:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:30 smithi047 ceph-mon[36835]: pgmap v5895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:40:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:30 smithi029 ceph-mon[32610]: pgmap v5895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:40:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:30 smithi077 ceph-mon[37153]: pgmap v5895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:40:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:32 smithi047 ceph-mon[36835]: pgmap v5896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:40:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10254.595285 secs 2024-07-27T09:40:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:32 smithi029 ceph-mon[32610]: pgmap v5896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:40:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10254.595285 secs 2024-07-27T09:40:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:32 smithi077 ceph-mon[37153]: pgmap v5896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:40:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10254.595285 secs 2024-07-27T09:40:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:34 smithi077 ceph-mon[37153]: pgmap v5897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:40:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:34 smithi047 ceph-mon[36835]: pgmap v5897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:40:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:34 smithi029 ceph-mon[32610]: pgmap v5897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:40:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:36 smithi077 ceph-mon[37153]: pgmap v5898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:36 smithi047 ceph-mon[36835]: pgmap v5898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:36 smithi029 ceph-mon[32610]: pgmap v5898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10259.595431 secs 2024-07-27T09:40:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10259.595431 secs 2024-07-27T09:40:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10259.595431 secs 2024-07-27T09:40:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:38 smithi047 ceph-mon[36835]: pgmap v5899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:40:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:38 smithi029 ceph-mon[32610]: pgmap v5899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:40:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:38 smithi077 ceph-mon[37153]: pgmap v5899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:40:41.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:40 smithi047 ceph-mon[36835]: pgmap v5900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:40 smithi029 ceph-mon[32610]: pgmap v5900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:40 smithi077 ceph-mon[37153]: pgmap v5900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:42 smithi047 ceph-mon[36835]: pgmap v5901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:40:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10264.595536 secs 2024-07-27T09:40:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:42 smithi029 ceph-mon[32610]: pgmap v5901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:40:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10264.595536 secs 2024-07-27T09:40:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:42 smithi077 ceph-mon[37153]: pgmap v5901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:40:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10264.595536 secs 2024-07-27T09:40:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:44.187+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:40:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:44.187+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:40:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:44.187+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:40:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:44.187+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:40:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:44 smithi047 ceph-mon[36835]: pgmap v5902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:40:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:44 smithi029 ceph-mon[32610]: pgmap v5902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:40:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:44 smithi077 ceph-mon[37153]: pgmap v5902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:40:46.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:46.125+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:40:46.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:46.125+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:40:46.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:46.125+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:40:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:46.141+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:40:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:46.141+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:40:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:40:46.141+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:40:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:46 smithi047 ceph-mon[36835]: pgmap v5903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:46 smithi029 ceph-mon[32610]: pgmap v5903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:46 smithi077 ceph-mon[37153]: pgmap v5903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10269.595632 secs 2024-07-27T09:40:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10269.595632 secs 2024-07-27T09:40:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10269.595632 secs 2024-07-27T09:40:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:48 smithi047 ceph-mon[36835]: pgmap v5904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:40:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:48 smithi029 ceph-mon[32610]: pgmap v5904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:40:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:48 smithi077 ceph-mon[37153]: pgmap v5904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:40:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:50 smithi047 ceph-mon[36835]: pgmap v5905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:50 smithi029 ceph-mon[32610]: pgmap v5905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:50 smithi077 ceph-mon[37153]: pgmap v5905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:52 smithi047 ceph-mon[36835]: pgmap v5906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:40:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10274.595759 secs 2024-07-27T09:40:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:52 smithi029 ceph-mon[32610]: pgmap v5906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:40:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10274.595759 secs 2024-07-27T09:40:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:52 smithi077 ceph-mon[37153]: pgmap v5906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:40:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10274.595759 secs 2024-07-27T09:40:55.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:54 smithi077 ceph-mon[37153]: pgmap v5907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:40:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:54 smithi047 ceph-mon[36835]: pgmap v5907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:40:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:54 smithi029 ceph-mon[32610]: pgmap v5907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:40:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:56 smithi047 ceph-mon[36835]: pgmap v5908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:56 smithi029 ceph-mon[32610]: pgmap v5908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:56 smithi077 ceph-mon[37153]: pgmap v5908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:40:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10279.595907 secs 2024-07-27T09:40:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10279.595907 secs 2024-07-27T09:40:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10279.595907 secs 2024-07-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:59.190+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:40:59.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:59.190+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T09:40:59.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:40:58 smithi029 ceph-mon[32610]: pgmap v5909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:40:59.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:59.190+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:40:59.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:40:59.191+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006590 con 0x7eff50104ab0 2024-07-27T09:40:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:40:58 smithi047 ceph-mon[36835]: pgmap v5909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:40:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:40:58 smithi077 ceph-mon[37153]: pgmap v5909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:41:01.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:41:01.139+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:41:01.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:41:01.139+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:41:01.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:41:01.139+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:41:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:41:01.157+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:41:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:41:01.157+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:41:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:41:01.157+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:41:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:00 smithi047 ceph-mon[36835]: pgmap v5910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:00 smithi029 ceph-mon[32610]: pgmap v5910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:00 smithi077 ceph-mon[37153]: pgmap v5910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:02 smithi047 ceph-mon[36835]: pgmap v5911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:41:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10284.596029 secs 2024-07-27T09:41:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:02 smithi029 ceph-mon[32610]: pgmap v5911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:41:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10284.596029 secs 2024-07-27T09:41:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:02 smithi077 ceph-mon[37153]: pgmap v5911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:41:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10284.596029 secs 2024-07-27T09:41:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:04 smithi047 ceph-mon[36835]: pgmap v5912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:41:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:04 smithi029 ceph-mon[32610]: pgmap v5912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:41:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:04 smithi077 ceph-mon[37153]: pgmap v5912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:41:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:06 smithi047 ceph-mon[36835]: pgmap v5913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:06 smithi029 ceph-mon[32610]: pgmap v5913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:06 smithi077 ceph-mon[37153]: pgmap v5913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10289.596160 secs 2024-07-27T09:41:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10289.596160 secs 2024-07-27T09:41:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10289.596160 secs 2024-07-27T09:41:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:08 smithi047 ceph-mon[36835]: pgmap v5914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:41:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:08 smithi029 ceph-mon[32610]: pgmap v5914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:41:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:08 smithi077 ceph-mon[37153]: pgmap v5914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:41:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:10 smithi047 ceph-mon[36835]: pgmap v5915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:10 smithi029 ceph-mon[32610]: pgmap v5915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:10 smithi077 ceph-mon[37153]: pgmap v5915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:12 smithi047 ceph-mon[36835]: pgmap v5916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:41:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10294.596267 secs 2024-07-27T09:41:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:12 smithi029 ceph-mon[32610]: pgmap v5916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:41:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10294.596267 secs 2024-07-27T09:41:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:12 smithi077 ceph-mon[37153]: pgmap v5916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:41:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10294.596267 secs 2024-07-27T09:41:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:14 smithi047 ceph-mon[36835]: pgmap v5917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:41:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:14 smithi029 ceph-mon[32610]: pgmap v5917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:41:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:14 smithi077 ceph-mon[37153]: pgmap v5917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:41:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:41:16.139+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:41:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:41:16.139+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:41:16.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:41:16.139+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:41:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:41:16.159+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:41:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:41:16.159+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:41:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:41:16.159+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:41:17.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:16 smithi047 ceph-mon[36835]: pgmap v5918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:16 smithi029 ceph-mon[32610]: pgmap v5918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:16 smithi077 ceph-mon[37153]: pgmap v5918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10299.596413 secs 2024-07-27T09:41:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10299.596413 secs 2024-07-27T09:41:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10299.596413 secs 2024-07-27T09:41:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:18 smithi047 ceph-mon[36835]: pgmap v5919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:41:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:18 smithi029 ceph-mon[32610]: pgmap v5919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:41:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:18 smithi077 ceph-mon[37153]: pgmap v5919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:41:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:41:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:41:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:41:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:20 smithi029 ceph-mon[32610]: pgmap v5920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:41:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:41:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:41:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:20 smithi077 ceph-mon[37153]: pgmap v5920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:41:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:41:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:41:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:20 smithi047 ceph-mon[36835]: pgmap v5920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:41:21.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:41:21.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:41:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:21 smithi029 ceph-mon[32610]: pgmap v5921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:41:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:21 smithi029 ceph-mon[32610]: pgmap v5922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:41:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:21 smithi077 ceph-mon[37153]: pgmap v5921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:41:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:21 smithi077 ceph-mon[37153]: pgmap v5922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:41:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:21 smithi047 ceph-mon[36835]: pgmap v5921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:41:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:21 smithi047 ceph-mon[36835]: pgmap v5922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:41:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10304.596544 secs 2024-07-27T09:41:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10304.596544 secs 2024-07-27T09:41:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10304.596544 secs 2024-07-27T09:41:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:23 smithi077 ceph-mon[37153]: pgmap v5923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:41:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:23 smithi047 ceph-mon[36835]: pgmap v5923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:41:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:23 smithi029 ceph-mon[32610]: pgmap v5923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:41:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:25 smithi077 ceph-mon[37153]: pgmap v5924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:41:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:25 smithi047 ceph-mon[36835]: pgmap v5924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:41:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:25 smithi029 ceph-mon[32610]: pgmap v5924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:41:28.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:27 smithi047 ceph-mon[36835]: pgmap v5925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:41:28.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10309.596685 secs 2024-07-27T09:41:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:27 smithi077 ceph-mon[37153]: pgmap v5925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:41:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10309.596685 secs 2024-07-27T09:41:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:27 smithi029 ceph-mon[32610]: pgmap v5925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:41:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10309.596685 secs 2024-07-27T09:41:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:29 smithi077 ceph-mon[37153]: pgmap v5926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:41:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:29 smithi047 ceph-mon[36835]: pgmap v5926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:41:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:29 smithi029 ceph-mon[32610]: pgmap v5926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:41:31.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:41:31.155+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:41:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:41:31.155+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:41:31.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:41:31.155+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:41:31.176 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:41:31.175+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:41:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:41:31.175+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:41:31.177 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:41:31.175+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:41:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:31 smithi077 ceph-mon[37153]: pgmap v5927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:41:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:31 smithi047 ceph-mon[36835]: pgmap v5927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:41:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:31 smithi029 ceph-mon[32610]: pgmap v5927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:41:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10314.596839 secs 2024-07-27T09:41:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10314.596839 secs 2024-07-27T09:41:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10314.596839 secs 2024-07-27T09:41:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:33 smithi077 ceph-mon[37153]: pgmap v5928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:41:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:33 smithi047 ceph-mon[36835]: pgmap v5928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:41:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:33 smithi029 ceph-mon[32610]: pgmap v5928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:41:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:35 smithi077 ceph-mon[37153]: pgmap v5929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:41:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:35 smithi047 ceph-mon[36835]: pgmap v5929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:41:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:35 smithi029 ceph-mon[32610]: pgmap v5929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:41:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:37 smithi077 ceph-mon[37153]: pgmap v5930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:41:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10319.596970 secs 2024-07-27T09:41:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:37 smithi047 ceph-mon[36835]: pgmap v5930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:41:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10319.596970 secs 2024-07-27T09:41:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:37 smithi029 ceph-mon[32610]: pgmap v5930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:41:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10319.596970 secs 2024-07-27T09:41:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:39 smithi077 ceph-mon[37153]: pgmap v5931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:41:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:39 smithi047 ceph-mon[36835]: pgmap v5931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:41:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:39 smithi029 ceph-mon[32610]: pgmap v5931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:41:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:41 smithi077 ceph-mon[37153]: pgmap v5932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:41 smithi047 ceph-mon[36835]: pgmap v5932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:41 smithi029 ceph-mon[32610]: pgmap v5932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10324.597094 secs 2024-07-27T09:41:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10324.597094 secs 2024-07-27T09:41:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10324.597094 secs 2024-07-27T09:41:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:44 smithi077 ceph-mon[37153]: pgmap v5933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:41:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:44 smithi047 ceph-mon[36835]: pgmap v5933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:41:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:44 smithi029 ceph-mon[32610]: pgmap v5933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:41:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:41:46.168+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:41:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:41:46.168+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:41:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:41:46.168+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:41:46.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:41:46.189+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:41:46.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:41:46.189+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:41:46.191 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:41:46.189+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:41:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:46 smithi077 ceph-mon[37153]: pgmap v5934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:41:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:46 smithi047 ceph-mon[36835]: pgmap v5934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:41:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:46 smithi029 ceph-mon[32610]: pgmap v5934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:41:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:48 smithi077 ceph-mon[37153]: pgmap v5935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:41:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10329.597240 secs 2024-07-27T09:41:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:48 smithi047 ceph-mon[36835]: pgmap v5935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:41:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10329.597240 secs 2024-07-27T09:41:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:48 smithi029 ceph-mon[32610]: pgmap v5935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:41:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10329.597240 secs 2024-07-27T09:41:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:50 smithi077 ceph-mon[37153]: pgmap v5936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:41:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:50 smithi047 ceph-mon[36835]: pgmap v5936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:41:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:50 smithi029 ceph-mon[32610]: pgmap v5936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:41:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:52 smithi077 ceph-mon[37153]: pgmap v5937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:52 smithi047 ceph-mon[36835]: pgmap v5937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:52 smithi029 ceph-mon[32610]: pgmap v5937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:41:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10334.597365 secs 2024-07-27T09:41:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10334.597365 secs 2024-07-27T09:41:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10334.597365 secs 2024-07-27T09:41:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:54 smithi029 ceph-mon[32610]: pgmap v5938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:41:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:54 smithi077 ceph-mon[37153]: pgmap v5938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:41:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:54 smithi047 ceph-mon[36835]: pgmap v5938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:41:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:56 smithi077 ceph-mon[37153]: pgmap v5939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:41:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:56 smithi047 ceph-mon[36835]: pgmap v5939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:41:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:56 smithi029 ceph-mon[32610]: pgmap v5939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:41:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:58 smithi077 ceph-mon[37153]: pgmap v5940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:41:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:41:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10339.597463 secs 2024-07-27T09:41:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:58 smithi047 ceph-mon[36835]: pgmap v5940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:41:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:41:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10339.597463 secs 2024-07-27T09:41:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:58 smithi029 ceph-mon[32610]: pgmap v5940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:41:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:41:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10339.597463 secs 2024-07-27T09:42:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:00 smithi077 ceph-mon[37153]: pgmap v5941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:00 smithi047 ceph-mon[36835]: pgmap v5941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:00 smithi029 ceph-mon[32610]: pgmap v5941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:01.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:42:01.174+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:42:01.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:42:01.174+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:42:01.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:42:01.174+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:42:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:01.200+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:42:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:01.200+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:42:01.201 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:01.200+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:42:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:02 smithi077 ceph-mon[37153]: pgmap v5942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:02 smithi047 ceph-mon[36835]: pgmap v5942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:02 smithi029 ceph-mon[32610]: pgmap v5942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10344.597595 secs 2024-07-27T09:42:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10344.597595 secs 2024-07-27T09:42:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10344.597595 secs 2024-07-27T09:42:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:04 smithi077 ceph-mon[37153]: pgmap v5943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:42:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:04 smithi047 ceph-mon[36835]: pgmap v5943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:42:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:04 smithi029 ceph-mon[32610]: pgmap v5943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:42:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:06 smithi077 ceph-mon[37153]: pgmap v5944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:42:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:06 smithi047 ceph-mon[36835]: pgmap v5944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:42:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:06 smithi029 ceph-mon[32610]: pgmap v5944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:42:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:08 smithi077 ceph-mon[37153]: pgmap v5945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:42:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10349.597723 secs 2024-07-27T09:42:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:08 smithi047 ceph-mon[36835]: pgmap v5945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:42:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10349.597723 secs 2024-07-27T09:42:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:08 smithi029 ceph-mon[32610]: pgmap v5945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:42:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10349.597723 secs 2024-07-27T09:42:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:10 smithi077 ceph-mon[37153]: pgmap v5946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:10.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:10 smithi047 ceph-mon[36835]: pgmap v5946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:10 smithi029 ceph-mon[32610]: pgmap v5946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:12 smithi077 ceph-mon[37153]: pgmap v5947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:12 smithi047 ceph-mon[36835]: pgmap v5947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:12 smithi029 ceph-mon[32610]: pgmap v5947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10354.597862 secs 2024-07-27T09:42:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10354.597862 secs 2024-07-27T09:42:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10354.597862 secs 2024-07-27T09:42:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:14 smithi077 ceph-mon[37153]: pgmap v5948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:42:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:14 smithi047 ceph-mon[36835]: pgmap v5948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:42:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:14 smithi029 ceph-mon[32610]: pgmap v5948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:42:16.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:42:16.177+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:42:16.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:42:16.177+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:42:16.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:42:16.177+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:42:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:16.206+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:42:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:16.206+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:42:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:16.206+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:42:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:16 smithi077 ceph-mon[37153]: pgmap v5949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:42:16.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:16 smithi047 ceph-mon[36835]: pgmap v5949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:42:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:16 smithi029 ceph-mon[32610]: pgmap v5949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:42:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:18 smithi077 ceph-mon[37153]: pgmap v5950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:42:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10359.598000 secs 2024-07-27T09:42:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:18 smithi047 ceph-mon[36835]: pgmap v5950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:42:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10359.598000 secs 2024-07-27T09:42:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:18 smithi029 ceph-mon[32610]: pgmap v5950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:42:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10359.598000 secs 2024-07-27T09:42:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:20 smithi077 ceph-mon[37153]: pgmap v5951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:20 smithi047 ceph-mon[36835]: pgmap v5951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:20 smithi029 ceph-mon[32610]: pgmap v5951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:42:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:42:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:42:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:42:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:42:21.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:42:21.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:42:21.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:42:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:42:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:42:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:42:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:42:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:22 smithi077 ceph-mon[37153]: pgmap v5952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:22 smithi077 ceph-mon[37153]: pgmap v5953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:42:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:22 smithi077 ceph-mon[37153]: pgmap v5954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:42:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:22 smithi047 ceph-mon[36835]: pgmap v5952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:22 smithi047 ceph-mon[36835]: pgmap v5953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:42:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:22 smithi047 ceph-mon[36835]: pgmap v5954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:42:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:22 smithi029 ceph-mon[32610]: pgmap v5952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:22 smithi029 ceph-mon[32610]: pgmap v5953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:42:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:22 smithi029 ceph-mon[32610]: pgmap v5954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:42:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10364.598127 secs 2024-07-27T09:42:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10364.598127 secs 2024-07-27T09:42:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10364.598127 secs 2024-07-27T09:42:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:24 smithi077 ceph-mon[37153]: pgmap v5955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:42:24.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:24 smithi047 ceph-mon[36835]: pgmap v5955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:42:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:24 smithi029 ceph-mon[32610]: pgmap v5955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:42:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:26 smithi077 ceph-mon[37153]: pgmap v5956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:42:26.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:26 smithi047 ceph-mon[36835]: pgmap v5956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:42:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:26 smithi029 ceph-mon[32610]: pgmap v5956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:42:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10369.598247 secs 2024-07-27T09:42:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:28 smithi077 ceph-mon[37153]: pgmap v5957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:42:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10369.598247 secs 2024-07-27T09:42:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:28 smithi047 ceph-mon[36835]: pgmap v5957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:42:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10369.598247 secs 2024-07-27T09:42:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:28 smithi029 ceph-mon[32610]: pgmap v5957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:42:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:30 smithi077 ceph-mon[37153]: pgmap v5958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:42:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:30 smithi047 ceph-mon[36835]: pgmap v5958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:42:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:30 smithi029 ceph-mon[32610]: pgmap v5958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:42:31.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:42:31.190+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:42:31.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:42:31.190+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:42:31.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:42:31.190+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:42:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:31.221+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:42:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:31.221+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:42:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:31.221+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:42:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:32 smithi077 ceph-mon[37153]: pgmap v5959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:42:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:32 smithi047 ceph-mon[36835]: pgmap v5959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:42:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:32 smithi029 ceph-mon[32610]: pgmap v5959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:42:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10374.598376 secs 2024-07-27T09:42:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10374.598376 secs 2024-07-27T09:42:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10374.598376 secs 2024-07-27T09:42:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:34 smithi047 ceph-mon[36835]: pgmap v5960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:42:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:34 smithi029 ceph-mon[32610]: pgmap v5960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:42:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:34 smithi077 ceph-mon[37153]: pgmap v5960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:42:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:36 smithi047 ceph-mon[36835]: pgmap v5961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:36 smithi029 ceph-mon[32610]: pgmap v5961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:36 smithi077 ceph-mon[37153]: pgmap v5961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10379.598451 secs 2024-07-27T09:42:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:38 smithi047 ceph-mon[36835]: pgmap v5962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:42:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10379.598451 secs 2024-07-27T09:42:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:38 smithi029 ceph-mon[32610]: pgmap v5962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:42:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10379.598451 secs 2024-07-27T09:42:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:38 smithi077 ceph-mon[37153]: pgmap v5962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:42:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:40 smithi047 ceph-mon[36835]: pgmap v5963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:40 smithi029 ceph-mon[32610]: pgmap v5963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:40 smithi077 ceph-mon[37153]: pgmap v5963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:42.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:42 smithi047 ceph-mon[36835]: pgmap v5964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:42 smithi029 ceph-mon[32610]: pgmap v5964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:42 smithi077 ceph-mon[37153]: pgmap v5964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10384.598554 secs 2024-07-27T09:42:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10384.598554 secs 2024-07-27T09:42:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10384.598554 secs 2024-07-27T09:42:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:44 smithi047 ceph-mon[36835]: pgmap v5965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:42:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:44 smithi029 ceph-mon[32610]: pgmap v5965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:42:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:44 smithi077 ceph-mon[37153]: pgmap v5965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:42:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:42:46.205+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:42:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:42:46.205+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:42:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:42:46.205+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:42:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:46.236+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:42:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:46.236+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:42:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:46.236+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:42:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:46 smithi047 ceph-mon[36835]: pgmap v5966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:46 smithi029 ceph-mon[32610]: pgmap v5966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:46 smithi077 ceph-mon[37153]: pgmap v5966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10389.598687 secs 2024-07-27T09:42:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:48 smithi047 ceph-mon[36835]: pgmap v5967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:42:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10389.598687 secs 2024-07-27T09:42:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:48 smithi029 ceph-mon[32610]: pgmap v5967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:42:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10389.598687 secs 2024-07-27T09:42:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:48 smithi077 ceph-mon[37153]: pgmap v5967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:42:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:50 smithi047 ceph-mon[36835]: pgmap v5968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:50 smithi029 ceph-mon[32610]: pgmap v5968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:50 smithi077 ceph-mon[37153]: pgmap v5968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:42:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:52 smithi047 ceph-mon[36835]: pgmap v5969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:52 smithi029 ceph-mon[32610]: pgmap v5969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:52 smithi077 ceph-mon[37153]: pgmap v5969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10394.598799 secs 2024-07-27T09:42:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10394.598799 secs 2024-07-27T09:42:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10394.598799 secs 2024-07-27T09:42:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:54 smithi047 ceph-mon[36835]: pgmap v5970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:42:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:54 smithi029 ceph-mon[32610]: pgmap v5970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:42:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:54 smithi077 ceph-mon[37153]: pgmap v5970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:42:56.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:56 smithi077 ceph-mon[37153]: pgmap v5971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:56 smithi047 ceph-mon[36835]: pgmap v5971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:56 smithi029 ceph-mon[32610]: pgmap v5971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:42:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10399.598936 secs 2024-07-27T09:42:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:42:58 smithi047 ceph-mon[36835]: pgmap v5972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:42:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10399.598936 secs 2024-07-27T09:42:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:42:58 smithi029 ceph-mon[32610]: pgmap v5972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:42:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10399.598936 secs 2024-07-27T09:42:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:42:58 smithi077 ceph-mon[37153]: pgmap v5972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:42:59.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:59.209+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:42:59.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:59.209+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:42:59.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:59.210+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:42:59.211 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:42:59.210+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:43:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:00 smithi047 ceph-mon[36835]: pgmap v5973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:00 smithi029 ceph-mon[32610]: pgmap v5973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:00 smithi077 ceph-mon[37153]: pgmap v5973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:01.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:01.209+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:43:01.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:01.209+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:43:01.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:01.209+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:43:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:01.253+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:43:01.253 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:01.253+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:43:01.254 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:01.253+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:43:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:02 smithi047 ceph-mon[36835]: pgmap v5974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:02 smithi029 ceph-mon[32610]: pgmap v5974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:02 smithi077 ceph-mon[37153]: pgmap v5974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10404.599069 secs 2024-07-27T09:43:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10404.599069 secs 2024-07-27T09:43:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10404.599069 secs 2024-07-27T09:43:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:04 smithi047 ceph-mon[36835]: pgmap v5975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:43:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:04 smithi029 ceph-mon[32610]: pgmap v5975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:43:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:04 smithi077 ceph-mon[37153]: pgmap v5975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:43:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:06 smithi047 ceph-mon[36835]: pgmap v5976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:06 smithi029 ceph-mon[32610]: pgmap v5976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:06 smithi077 ceph-mon[37153]: pgmap v5976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10409.599160 secs 2024-07-27T09:43:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:08 smithi047 ceph-mon[36835]: pgmap v5977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:43:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10409.599160 secs 2024-07-27T09:43:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:08 smithi029 ceph-mon[32610]: pgmap v5977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:43:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10409.599160 secs 2024-07-27T09:43:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:08 smithi077 ceph-mon[37153]: pgmap v5977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:43:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:10 smithi047 ceph-mon[36835]: pgmap v5978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:10 smithi029 ceph-mon[32610]: pgmap v5978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:10 smithi077 ceph-mon[37153]: pgmap v5978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:12 smithi047 ceph-mon[36835]: pgmap v5979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:12 smithi029 ceph-mon[32610]: pgmap v5979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:12 smithi077 ceph-mon[37153]: pgmap v5979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10414.599284 secs 2024-07-27T09:43:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10414.599284 secs 2024-07-27T09:43:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10414.599284 secs 2024-07-27T09:43:14.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:14.211+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:43:14.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:14.211+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:43:14.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:14.212+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:43:14.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:14.212+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:43:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:14 smithi047 ceph-mon[36835]: pgmap v5980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:43:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:14 smithi029 ceph-mon[32610]: pgmap v5980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:43:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:14 smithi077 ceph-mon[37153]: pgmap v5980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:43:16.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:16.223+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:43:16.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:16.223+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:43:16.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:16.223+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:43:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:16.267+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:43:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:16.267+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:43:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:16.267+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:43:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:16 smithi047 ceph-mon[36835]: pgmap v5981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:16 smithi029 ceph-mon[32610]: pgmap v5981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:16 smithi077 ceph-mon[37153]: pgmap v5981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10419.599414 secs 2024-07-27T09:43:18.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:18 smithi029 ceph-mon[32610]: pgmap v5982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:43:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10419.599414 secs 2024-07-27T09:43:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:18 smithi077 ceph-mon[37153]: pgmap v5982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:43:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10419.599414 secs 2024-07-27T09:43:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:18 smithi047 ceph-mon[36835]: pgmap v5982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:43:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:20 smithi029 ceph-mon[32610]: pgmap v5983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:20 smithi077 ceph-mon[37153]: pgmap v5983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:20 smithi047 ceph-mon[36835]: pgmap v5983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:43:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:43:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:43:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:22 smithi077 ceph-mon[37153]: pgmap v5984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:43:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:43:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:43:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:22 smithi047 ceph-mon[36835]: pgmap v5984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:43:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:43:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:43:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:22 smithi029 ceph-mon[32610]: pgmap v5984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:43:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:43:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:43:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:23 smithi077 ceph-mon[37153]: pgmap v5985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:43:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:23 smithi077 ceph-mon[37153]: pgmap v5986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:43:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10424.599559 secs 2024-07-27T09:43:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:23 smithi047 ceph-mon[36835]: pgmap v5985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:43:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:23 smithi047 ceph-mon[36835]: pgmap v5986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:43:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10424.599559 secs 2024-07-27T09:43:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:23 smithi029 ceph-mon[32610]: pgmap v5985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:43:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:23 smithi029 ceph-mon[32610]: pgmap v5986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:43:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10424.599559 secs 2024-07-27T09:43:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:24 smithi077 ceph-mon[37153]: pgmap v5987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:43:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:24 smithi047 ceph-mon[36835]: pgmap v5987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:43:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:24 smithi029 ceph-mon[32610]: pgmap v5987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:43:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:26 smithi077 ceph-mon[37153]: pgmap v5988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:43:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:26 smithi047 ceph-mon[36835]: pgmap v5988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:43:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:26 smithi029 ceph-mon[32610]: pgmap v5988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:43:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10429.599703 secs 2024-07-27T09:43:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10429.599703 secs 2024-07-27T09:43:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10429.599703 secs 2024-07-27T09:43:28.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:28 smithi047 ceph-mon[36835]: pgmap v5989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:43:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:28 smithi077 ceph-mon[37153]: pgmap v5989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:43:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:28 smithi029 ceph-mon[32610]: pgmap v5989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:43:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:30 smithi077 ceph-mon[37153]: pgmap v5990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:43:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:30 smithi047 ceph-mon[36835]: pgmap v5990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:43:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:30 smithi029 ceph-mon[32610]: pgmap v5990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:43:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:31.237+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:43:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:31.237+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:43:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:31.237+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:43:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:31.274+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:43:31.275 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:31.274+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:43:31.276 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:31.274+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:43:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:32 smithi077 ceph-mon[37153]: pgmap v5991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:43:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10434.599860 secs 2024-07-27T09:43:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:32 smithi047 ceph-mon[36835]: pgmap v5991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:43:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10434.599860 secs 2024-07-27T09:43:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:32 smithi029 ceph-mon[32610]: pgmap v5991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:43:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10434.599860 secs 2024-07-27T09:43:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:34 smithi077 ceph-mon[37153]: pgmap v5992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:34 smithi047 ceph-mon[36835]: pgmap v5992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:34 smithi029 ceph-mon[32610]: pgmap v5992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:36 smithi077 ceph-mon[37153]: pgmap v5993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:36 smithi047 ceph-mon[36835]: pgmap v5993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:36 smithi029 ceph-mon[32610]: pgmap v5993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10439.599926 secs 2024-07-27T09:43:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10439.599926 secs 2024-07-27T09:43:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10439.599926 secs 2024-07-27T09:43:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:38 smithi077 ceph-mon[37153]: pgmap v5994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:43:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:38 smithi047 ceph-mon[36835]: pgmap v5994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:43:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:38 smithi029 ceph-mon[32610]: pgmap v5994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:43:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:40 smithi077 ceph-mon[37153]: pgmap v5995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:43:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:40 smithi047 ceph-mon[36835]: pgmap v5995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:43:40.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:40 smithi029 ceph-mon[32610]: pgmap v5995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:43:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:42 smithi077 ceph-mon[37153]: pgmap v5996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:43:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10444.600056 secs 2024-07-27T09:43:42.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:42 smithi029 ceph-mon[32610]: pgmap v5996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:43:42.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10444.600056 secs 2024-07-27T09:43:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:42 smithi047 ceph-mon[36835]: pgmap v5996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:43:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10444.600056 secs 2024-07-27T09:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:44.216+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:44.216+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:44.216+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:43:44.218 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:44.216+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:43:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:44 smithi077 ceph-mon[37153]: pgmap v5997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:44 smithi047 ceph-mon[36835]: pgmap v5997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:44 smithi029 ceph-mon[32610]: pgmap v5997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:46.253+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:43:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:46.253+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:43:46.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:46.253+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:43:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:46.289+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:43:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:46.289+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:43:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:43:46.289+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:43:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:46 smithi077 ceph-mon[37153]: pgmap v5998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:46 smithi047 ceph-mon[36835]: pgmap v5998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:46 smithi029 ceph-mon[32610]: pgmap v5998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:43:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10449.600208 secs 2024-07-27T09:43:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10449.600208 secs 2024-07-27T09:43:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10449.600208 secs 2024-07-27T09:43:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:48 smithi077 ceph-mon[37153]: pgmap v5999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:43:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:48 smithi047 ceph-mon[36835]: pgmap v5999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:43:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:48 smithi029 ceph-mon[32610]: pgmap v5999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:43:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:50 smithi077 ceph-mon[37153]: pgmap v6000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:43:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:50 smithi047 ceph-mon[36835]: pgmap v6000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:43:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:50 smithi029 ceph-mon[32610]: pgmap v6000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:43:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:52 smithi077 ceph-mon[37153]: pgmap v6001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:43:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10454.600346 secs 2024-07-27T09:43:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:52 smithi047 ceph-mon[36835]: pgmap v6001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:43:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10454.600346 secs 2024-07-27T09:43:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:52 smithi029 ceph-mon[32610]: pgmap v6001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:43:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10454.600346 secs 2024-07-27T09:43:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:54 smithi077 ceph-mon[37153]: pgmap v6002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:54 smithi047 ceph-mon[36835]: pgmap v6002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:54 smithi029 ceph-mon[32610]: pgmap v6002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:43:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:56 smithi077 ceph-mon[37153]: pgmap v6003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:43:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:56 smithi047 ceph-mon[36835]: pgmap v6003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:43:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:56 smithi029 ceph-mon[32610]: pgmap v6003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:43:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10459.600475 secs 2024-07-27T09:43:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10459.600475 secs 2024-07-27T09:43:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10459.600475 secs 2024-07-27T09:43:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:43:58 smithi077 ceph-mon[37153]: pgmap v6004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:43:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:43:58 smithi047 ceph-mon[36835]: pgmap v6004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:43:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:43:58 smithi029 ceph-mon[32610]: pgmap v6004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:43:59.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:59.219+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:43:59.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:59.219+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:43:59.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:59.219+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:43:59.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:43:59.219+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:44:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:00 smithi077 ceph-mon[37153]: pgmap v6005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:44:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:00 smithi047 ceph-mon[36835]: pgmap v6005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:44:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:00 smithi029 ceph-mon[32610]: pgmap v6005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:44:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:44:01.269+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:44:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:44:01.269+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:44:01.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:44:01.269+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:44:01.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:44:01.305+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:44:01.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:44:01.305+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:44:01.306 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:44:01.305+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:44:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:02 smithi047 ceph-mon[36835]: pgmap v6006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10464.600607 secs 2024-07-27T09:44:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:02 smithi029 ceph-mon[32610]: pgmap v6006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10464.600607 secs 2024-07-27T09:44:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:02 smithi077 ceph-mon[37153]: pgmap v6006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10464.600607 secs 2024-07-27T09:44:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:04 smithi047 ceph-mon[36835]: pgmap v6007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:44:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:04 smithi029 ceph-mon[32610]: pgmap v6007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:44:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:04 smithi077 ceph-mon[37153]: pgmap v6007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:44:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:06 smithi047 ceph-mon[36835]: pgmap v6008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:44:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:06 smithi029 ceph-mon[32610]: pgmap v6008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:44:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:06 smithi077 ceph-mon[37153]: pgmap v6008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:44:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10469.600730 secs 2024-07-27T09:44:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10469.600730 secs 2024-07-27T09:44:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10469.600730 secs 2024-07-27T09:44:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:08 smithi047 ceph-mon[36835]: pgmap v6009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:44:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:08 smithi029 ceph-mon[32610]: pgmap v6009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:44:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:08 smithi077 ceph-mon[37153]: pgmap v6009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:44:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:10 smithi047 ceph-mon[36835]: pgmap v6010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:44:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:10 smithi029 ceph-mon[32610]: pgmap v6010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:44:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:10 smithi077 ceph-mon[37153]: pgmap v6010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:44:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:12 smithi047 ceph-mon[36835]: pgmap v6011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10474.600857 secs 2024-07-27T09:44:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:12 smithi029 ceph-mon[32610]: pgmap v6011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10474.600857 secs 2024-07-27T09:44:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:12 smithi077 ceph-mon[37153]: pgmap v6011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10474.600857 secs 2024-07-27T09:44:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:14 smithi047 ceph-mon[36835]: pgmap v6012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:44:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:14 smithi029 ceph-mon[32610]: pgmap v6012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:44:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:14 smithi077 ceph-mon[37153]: pgmap v6012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:44:16.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:44:16.269+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:44:16.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:44:16.269+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:44:16.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:44:16.269+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:44:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:44:16.320+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:44:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:44:16.320+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:44:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:44:16.320+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:44:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:16 smithi047 ceph-mon[36835]: pgmap v6013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:44:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:16 smithi029 ceph-mon[32610]: pgmap v6013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:44:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:16 smithi077 ceph-mon[37153]: pgmap v6013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:44:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10479.600990 secs 2024-07-27T09:44:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10479.600990 secs 2024-07-27T09:44:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10479.600990 secs 2024-07-27T09:44:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:18 smithi047 ceph-mon[36835]: pgmap v6014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:44:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:18 smithi029 ceph-mon[32610]: pgmap v6014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:44:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:18 smithi077 ceph-mon[37153]: pgmap v6014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:44:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:20 smithi047 ceph-mon[36835]: pgmap v6015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:44:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:20 smithi029 ceph-mon[32610]: pgmap v6015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:44:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:20 smithi077 ceph-mon[37153]: pgmap v6015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:44:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:44:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:44:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:44:22.711 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:22 smithi047 ceph-mon[36835]: pgmap v6016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:22.711 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10484.601061 secs 2024-07-27T09:44:22.725 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:22 smithi029 ceph-mon[32610]: pgmap v6016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:22.725 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10484.601061 secs 2024-07-27T09:44:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:22 smithi077 ceph-mon[37153]: pgmap v6016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10484.601061 secs 2024-07-27T09:44:24.858 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:24 smithi047 ceph-mon[36835]: pgmap v6017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:44:24.953 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:24 smithi029 ceph-mon[32610]: pgmap v6017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:44:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:24 smithi077 ceph-mon[37153]: pgmap v6017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:44:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: pgmap v6018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:44:26.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:44:26.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:44:26.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:44:26.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:44:26.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:44:26.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:44:26.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:44:26.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:44:26.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: pgmap v6019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-27T09:44:26.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: pgmap v6020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-27T09:44:26.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: pgmap v6018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:44:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:44:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:44:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:44:26.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:44:26.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:44:26.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:44:26.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:44:26.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:44:26.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: pgmap v6019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-27T09:44:26.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: pgmap v6020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-27T09:44:26.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: pgmap v6018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:44:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:44:26.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:44:26.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:44:26.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:44:26.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:44:26.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:44:26.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:44:26.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:44:26.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: pgmap v6019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-27T09:44:26.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: pgmap v6020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-27T09:44:26.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:44:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10489.601175 secs 2024-07-27T09:44:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10489.601175 secs 2024-07-27T09:44:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10489.601175 secs 2024-07-27T09:44:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:28 smithi047 ceph-mon[36835]: pgmap v6021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:44:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:28 smithi029 ceph-mon[32610]: pgmap v6021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:44:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:28 smithi077 ceph-mon[37153]: pgmap v6021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:44:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:30 smithi077 ceph-mon[37153]: pgmap v6022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T09:44:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:30 smithi047 ceph-mon[36835]: pgmap v6022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T09:44:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:30 smithi029 ceph-mon[32610]: pgmap v6022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T09:44:31.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:44:31.277+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:44:31.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:44:31.277+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:44:31.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:44:31.277+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:44:31.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:44:31.330+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:44:31.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:44:31.330+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:44:31.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:44:31.330+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:44:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:32 smithi077 ceph-mon[37153]: pgmap v6023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T09:44:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10494.601305 secs 2024-07-27T09:44:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:32 smithi047 ceph-mon[36835]: pgmap v6023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T09:44:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10494.601305 secs 2024-07-27T09:44:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:32 smithi029 ceph-mon[32610]: pgmap v6023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T09:44:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10494.601305 secs 2024-07-27T09:44:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:34 smithi077 ceph-mon[37153]: pgmap v6024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:44:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:34 smithi047 ceph-mon[36835]: pgmap v6024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:44:35.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:34 smithi029 ceph-mon[32610]: pgmap v6024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:44:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:36 smithi077 ceph-mon[37153]: pgmap v6025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:44:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:36 smithi047 ceph-mon[36835]: pgmap v6025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:44:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:36 smithi029 ceph-mon[32610]: pgmap v6025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:44:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10499.601431 secs 2024-07-27T09:44:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10499.601431 secs 2024-07-27T09:44:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10499.601431 secs 2024-07-27T09:44:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:38 smithi077 ceph-mon[37153]: pgmap v6026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:44:39.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:38 smithi047 ceph-mon[36835]: pgmap v6026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:44:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:38 smithi029 ceph-mon[32610]: pgmap v6026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:44:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:40 smithi077 ceph-mon[37153]: pgmap v6027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:44:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:40 smithi047 ceph-mon[36835]: pgmap v6027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:44:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:40 smithi029 ceph-mon[32610]: pgmap v6027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:44:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:42 smithi029 ceph-mon[32610]: pgmap v6028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10504.601555 secs 2024-07-27T09:44:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:42 smithi077 ceph-mon[37153]: pgmap v6028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10504.601555 secs 2024-07-27T09:44:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:42 smithi047 ceph-mon[36835]: pgmap v6028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10504.601555 secs 2024-07-27T09:44:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:44 smithi077 ceph-mon[37153]: pgmap v6029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:44:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:44 smithi047 ceph-mon[36835]: pgmap v6029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:44:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:44 smithi029 ceph-mon[32610]: pgmap v6029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:44:46.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:44:46.292+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:44:46.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:44:46.292+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:44:46.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:44:46.292+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:44:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:44:46.331+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:44:46.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:44:46.331+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:44:46.333 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:44:46.331+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:44:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:46 smithi077 ceph-mon[37153]: pgmap v6030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:44:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:46 smithi047 ceph-mon[36835]: pgmap v6030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:44:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:46 smithi029 ceph-mon[32610]: pgmap v6030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:44:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10509.601674 secs 2024-07-27T09:44:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10509.601674 secs 2024-07-27T09:44:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10509.601674 secs 2024-07-27T09:44:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:48 smithi077 ceph-mon[37153]: pgmap v6031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:44:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:48 smithi047 ceph-mon[36835]: pgmap v6031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:44:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:48 smithi029 ceph-mon[32610]: pgmap v6031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:44:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:50 smithi077 ceph-mon[37153]: pgmap v6032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:44:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:50 smithi047 ceph-mon[36835]: pgmap v6032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:44:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:50 smithi029 ceph-mon[32610]: pgmap v6032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:44:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:52 smithi077 ceph-mon[37153]: pgmap v6033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10514.601821 secs 2024-07-27T09:44:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:52 smithi047 ceph-mon[36835]: pgmap v6033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10514.601821 secs 2024-07-27T09:44:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:52 smithi029 ceph-mon[32610]: pgmap v6033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:44:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10514.601821 secs 2024-07-27T09:44:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:54 smithi077 ceph-mon[37153]: pgmap v6034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:44:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:54 smithi047 ceph-mon[36835]: pgmap v6034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:44:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:54 smithi029 ceph-mon[32610]: pgmap v6034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:44:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:56 smithi029 ceph-mon[32610]: pgmap v6035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:44:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:56 smithi077 ceph-mon[37153]: pgmap v6035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:44:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:56 smithi047 ceph-mon[36835]: pgmap v6035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:44:58.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10519.601922 secs 2024-07-27T09:44:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10519.601922 secs 2024-07-27T09:44:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10519.601922 secs 2024-07-27T09:44:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:44:58 smithi077 ceph-mon[37153]: pgmap v6036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:44:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:44:58 smithi047 ceph-mon[36835]: pgmap v6036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:44:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:44:58 smithi029 ceph-mon[32610]: pgmap v6036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:45:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:00 smithi077 ceph-mon[37153]: pgmap v6037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:45:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:00 smithi047 ceph-mon[36835]: pgmap v6037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:45:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:00 smithi029 ceph-mon[32610]: pgmap v6037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:45:01.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:45:01.307+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:45:01.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:45:01.307+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:45:01.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:45:01.307+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:45:01.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:45:01.337+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:45:01.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:45:01.337+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:45:01.338 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:45:01.337+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:45:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:02 smithi077 ceph-mon[37153]: pgmap v6038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10524.602053 secs 2024-07-27T09:45:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:02 smithi047 ceph-mon[36835]: pgmap v6038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10524.602053 secs 2024-07-27T09:45:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:02 smithi029 ceph-mon[32610]: pgmap v6038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10524.602053 secs 2024-07-27T09:45:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:04 smithi077 ceph-mon[37153]: pgmap v6039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:04 smithi047 ceph-mon[36835]: pgmap v6039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:04 smithi029 ceph-mon[32610]: pgmap v6039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:06 smithi077 ceph-mon[37153]: pgmap v6040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:06 smithi047 ceph-mon[36835]: pgmap v6040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:06 smithi029 ceph-mon[32610]: pgmap v6040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10529.602174 secs 2024-07-27T09:45:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10529.602174 secs 2024-07-27T09:45:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10529.602174 secs 2024-07-27T09:45:09.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:08 smithi077 ceph-mon[37153]: pgmap v6041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:45:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:08 smithi047 ceph-mon[36835]: pgmap v6041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:45:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:08 smithi029 ceph-mon[32610]: pgmap v6041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:45:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:10 smithi077 ceph-mon[37153]: pgmap v6042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:45:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:10 smithi047 ceph-mon[36835]: pgmap v6042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:45:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:10 smithi029 ceph-mon[32610]: pgmap v6042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:45:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:12 smithi077 ceph-mon[37153]: pgmap v6043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10534.602298 secs 2024-07-27T09:45:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:12 smithi047 ceph-mon[36835]: pgmap v6043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10534.602298 secs 2024-07-27T09:45:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:12 smithi029 ceph-mon[32610]: pgmap v6043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10534.602298 secs 2024-07-27T09:45:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:14 smithi077 ceph-mon[37153]: pgmap v6044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:14 smithi047 ceph-mon[36835]: pgmap v6044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:14 smithi029 ceph-mon[32610]: pgmap v6044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:16.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:45:16.317+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:45:16.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:45:16.317+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:45:16.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:45:16.317+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:45:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:45:16.351+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:45:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:45:16.351+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:45:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:45:16.351+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:45:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:16 smithi077 ceph-mon[37153]: pgmap v6045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:16 smithi047 ceph-mon[36835]: pgmap v6045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:16 smithi029 ceph-mon[32610]: pgmap v6045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10539.602425 secs 2024-07-27T09:45:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10539.602425 secs 2024-07-27T09:45:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10539.602425 secs 2024-07-27T09:45:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:18 smithi077 ceph-mon[37153]: pgmap v6046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:45:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:18 smithi047 ceph-mon[36835]: pgmap v6046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:45:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:18 smithi029 ceph-mon[32610]: pgmap v6046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:45:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:20 smithi047 ceph-mon[36835]: pgmap v6047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:45:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:20 smithi029 ceph-mon[32610]: pgmap v6047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:45:21.360 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:20 smithi077 ceph-mon[37153]: pgmap v6047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:45:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:22 smithi047 ceph-mon[36835]: pgmap v6048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10544.602558 secs 2024-07-27T09:45:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:22 smithi029 ceph-mon[32610]: pgmap v6048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10544.602558 secs 2024-07-27T09:45:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:22 smithi077 ceph-mon[37153]: pgmap v6048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10544.602558 secs 2024-07-27T09:45:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:24 smithi047 ceph-mon[36835]: pgmap v6049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:24 smithi029 ceph-mon[32610]: pgmap v6049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:24 smithi077 ceph-mon[37153]: pgmap v6049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:45:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:45:26.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:45:26.965 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:26 smithi077 ceph-mon[37153]: pgmap v6050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:26.965 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:26.965 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:26.965 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:26.965 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:26 smithi047 ceph-mon[36835]: pgmap v6050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:27.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:27.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:27.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:27.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:26 smithi029 ceph-mon[32610]: pgmap v6050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:28.010 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:27 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:45:28.011 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:27 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:45:28.011 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10549.602624 secs 2024-07-27T09:45:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:27 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:45:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:27 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:45:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10549.602624 secs 2024-07-27T09:45:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:27 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:45:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:27 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:45:28.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10549.602624 secs 2024-07-27T09:45:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:28 smithi077 ceph-mon[37153]: pgmap v6051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:45:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:28 smithi047 ceph-mon[36835]: pgmap v6051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:45:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:28 smithi029 ceph-mon[32610]: pgmap v6051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:45:30.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:45:30.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:45:30.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:30.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:30.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:30.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:30.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:45:30.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:45:30.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:45:30.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:45:30.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: pgmap v6052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-27T09:45:30.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: pgmap v6053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:45:30.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:45:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:45:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:45:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:45:30.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:45:30.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:45:30.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:45:30.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: pgmap v6052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-27T09:45:30.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: pgmap v6053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:45:30.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:45:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:45:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:45:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:45:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:45:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:45:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:45:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:45:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: pgmap v6052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-27T09:45:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: pgmap v6053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:45:30.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:45:31.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:45:31.333+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:45:31.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:45:31.333+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:45:31.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:45:31.333+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:45:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:45:31.355+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:45:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:45:31.355+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:45:31.357 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:45:31.355+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:45:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:32 smithi077 ceph-mon[37153]: pgmap v6054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:45:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10554.602763 secs 2024-07-27T09:45:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:32 smithi047 ceph-mon[36835]: pgmap v6054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:45:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10554.602763 secs 2024-07-27T09:45:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:32 smithi029 ceph-mon[32610]: pgmap v6054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:45:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10554.602763 secs 2024-07-27T09:45:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:34 smithi077 ceph-mon[37153]: pgmap v6055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:45:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:34 smithi047 ceph-mon[36835]: pgmap v6055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:45:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:34 smithi029 ceph-mon[32610]: pgmap v6055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:45:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:36 smithi077 ceph-mon[37153]: pgmap v6056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:45:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:36 smithi047 ceph-mon[36835]: pgmap v6056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:45:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:36 smithi029 ceph-mon[32610]: pgmap v6056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:45:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10559.602886 secs 2024-07-27T09:45:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10559.602886 secs 2024-07-27T09:45:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10559.602886 secs 2024-07-27T09:45:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:38 smithi077 ceph-mon[37153]: pgmap v6057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:45:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:38 smithi047 ceph-mon[36835]: pgmap v6057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:45:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:38 smithi029 ceph-mon[32610]: pgmap v6057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:45:40.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:40 smithi047 ceph-mon[36835]: pgmap v6058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:45:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:40 smithi077 ceph-mon[37153]: pgmap v6058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:45:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:40 smithi029 ceph-mon[32610]: pgmap v6058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:45:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:42 smithi077 ceph-mon[37153]: pgmap v6059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10564.603009 secs 2024-07-27T09:45:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:42 smithi047 ceph-mon[36835]: pgmap v6059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10564.603009 secs 2024-07-27T09:45:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:42 smithi029 ceph-mon[32610]: pgmap v6059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10564.603009 secs 2024-07-27T09:45:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:44 smithi077 ceph-mon[37153]: pgmap v6060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:44 smithi047 ceph-mon[36835]: pgmap v6060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:44 smithi029 ceph-mon[32610]: pgmap v6060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:46.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:45:46.349+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:45:46.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:45:46.349+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:45:46.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:45:46.349+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:45:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:45:46.370+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:45:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:45:46.370+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:45:46.373 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:45:46.370+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:45:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:46 smithi077 ceph-mon[37153]: pgmap v6061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:46 smithi047 ceph-mon[36835]: pgmap v6061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:46 smithi029 ceph-mon[32610]: pgmap v6061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10569.603130 secs 2024-07-27T09:45:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10569.603130 secs 2024-07-27T09:45:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10569.603130 secs 2024-07-27T09:45:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:48 smithi077 ceph-mon[37153]: pgmap v6062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:45:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:48 smithi047 ceph-mon[36835]: pgmap v6062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:45:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:48 smithi029 ceph-mon[32610]: pgmap v6062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:45:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:50 smithi077 ceph-mon[37153]: pgmap v6063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:45:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:50 smithi047 ceph-mon[36835]: pgmap v6063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:45:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:50 smithi029 ceph-mon[32610]: pgmap v6063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:45:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:52 smithi077 ceph-mon[37153]: pgmap v6064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10574.603210 secs 2024-07-27T09:45:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:52 smithi047 ceph-mon[36835]: pgmap v6064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10574.603210 secs 2024-07-27T09:45:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:52 smithi029 ceph-mon[32610]: pgmap v6064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:45:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10574.603210 secs 2024-07-27T09:45:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:54 smithi077 ceph-mon[37153]: pgmap v6065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:54 smithi047 ceph-mon[36835]: pgmap v6065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:54 smithi029 ceph-mon[32610]: pgmap v6065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:45:57.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:56 smithi077 ceph-mon[37153]: pgmap v6066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:56 smithi047 ceph-mon[36835]: pgmap v6066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:56 smithi029 ceph-mon[32610]: pgmap v6066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:45:58.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10579.603333 secs 2024-07-27T09:45:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10579.603333 secs 2024-07-27T09:45:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10579.603333 secs 2024-07-27T09:45:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:45:58 smithi077 ceph-mon[37153]: pgmap v6067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:45:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:45:58 smithi047 ceph-mon[36835]: pgmap v6067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:45:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:45:58 smithi029 ceph-mon[32610]: pgmap v6067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:46:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:00 smithi077 ceph-mon[37153]: pgmap v6068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:46:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:00 smithi047 ceph-mon[36835]: pgmap v6068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:46:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:00 smithi029 ceph-mon[32610]: pgmap v6068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:46:01.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:01.364+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:46:01.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:01.364+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:46:01.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:01.364+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:46:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:01.386+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:46:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:01.387+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:46:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:01.387+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:46:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:02 smithi077 ceph-mon[37153]: pgmap v6069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10584.603454 secs 2024-07-27T09:46:03.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:02 smithi029 ceph-mon[32610]: pgmap v6069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:03.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10584.603454 secs 2024-07-27T09:46:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:02 smithi047 ceph-mon[36835]: pgmap v6069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10584.603454 secs 2024-07-27T09:46:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:04 smithi077 ceph-mon[37153]: pgmap v6070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:46:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:04 smithi047 ceph-mon[36835]: pgmap v6070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:46:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:04 smithi029 ceph-mon[32610]: pgmap v6070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:46:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:06 smithi077 ceph-mon[37153]: pgmap v6071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:06 smithi047 ceph-mon[36835]: pgmap v6071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:06 smithi029 ceph-mon[32610]: pgmap v6071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10589.603553 secs 2024-07-27T09:46:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10589.603553 secs 2024-07-27T09:46:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10589.603553 secs 2024-07-27T09:46:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:08 smithi077 ceph-mon[37153]: pgmap v6072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:46:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:08 smithi047 ceph-mon[36835]: pgmap v6072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:46:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:08 smithi029 ceph-mon[32610]: pgmap v6072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:46:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:10 smithi077 ceph-mon[37153]: pgmap v6073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:46:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:10 smithi047 ceph-mon[36835]: pgmap v6073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:46:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:10 smithi029 ceph-mon[32610]: pgmap v6073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:46:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:12 smithi077 ceph-mon[37153]: pgmap v6074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10594.603678 secs 2024-07-27T09:46:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:12 smithi047 ceph-mon[36835]: pgmap v6074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10594.603678 secs 2024-07-27T09:46:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:12 smithi029 ceph-mon[32610]: pgmap v6074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10594.603678 secs 2024-07-27T09:46:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:14 smithi077 ceph-mon[37153]: pgmap v6075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:46:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:14 smithi047 ceph-mon[36835]: pgmap v6075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:46:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:14 smithi029 ceph-mon[32610]: pgmap v6075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:46:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:16.378+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:46:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:16.378+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:46:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:16.378+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:46:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:16.389+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:46:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:16.389+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:46:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:16.389+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:46:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:16 smithi077 ceph-mon[37153]: pgmap v6076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:16 smithi047 ceph-mon[36835]: pgmap v6076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:16 smithi029 ceph-mon[32610]: pgmap v6076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10599.603825 secs 2024-07-27T09:46:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10599.603825 secs 2024-07-27T09:46:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10599.603825 secs 2024-07-27T09:46:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:18 smithi077 ceph-mon[37153]: pgmap v6077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:46:19.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:18 smithi029 ceph-mon[32610]: pgmap v6077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:46:19.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:18 smithi047 ceph-mon[36835]: pgmap v6077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:46:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:20 smithi077 ceph-mon[37153]: pgmap v6078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:46:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:20 smithi047 ceph-mon[36835]: pgmap v6078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:46:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:20 smithi029 ceph-mon[32610]: pgmap v6078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:46:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:22 smithi077 ceph-mon[37153]: pgmap v6079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10604.603965 secs 2024-07-27T09:46:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:22 smithi047 ceph-mon[36835]: pgmap v6079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10604.603965 secs 2024-07-27T09:46:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:22 smithi029 ceph-mon[32610]: pgmap v6079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10604.603965 secs 2024-07-27T09:46:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:24 smithi077 ceph-mon[37153]: pgmap v6080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:46:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:24 smithi047 ceph-mon[36835]: pgmap v6080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:46:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:24 smithi029 ceph-mon[32610]: pgmap v6080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:46:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:26 smithi077 ceph-mon[37153]: pgmap v6081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:46:27.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:26 smithi047 ceph-mon[36835]: pgmap v6081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:46:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:26 smithi029 ceph-mon[32610]: pgmap v6081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:46:28.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10609.604106 secs 2024-07-27T09:46:28.100 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10609.604106 secs 2024-07-27T09:46:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10609.604106 secs 2024-07-27T09:46:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:28 smithi077 ceph-mon[37153]: pgmap v6082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:46:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:28 smithi047 ceph-mon[36835]: pgmap v6082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:46:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:28 smithi029 ceph-mon[32610]: pgmap v6082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:46:30.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:46:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:46:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:46:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:30 smithi047 ceph-mon[36835]: pgmap v6083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:46:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:46:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:46:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:30 smithi047 ceph-mon[36835]: pgmap v6084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:46:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:30 smithi047 ceph-mon[36835]: pgmap v6085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:46:31.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:46:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:30 smithi029 ceph-mon[32610]: pgmap v6083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:46:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:46:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:46:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:30 smithi029 ceph-mon[32610]: pgmap v6084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:46:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:30 smithi029 ceph-mon[32610]: pgmap v6085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:46:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:46:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:30 smithi077 ceph-mon[37153]: pgmap v6083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:46:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:46:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:46:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:30 smithi077 ceph-mon[37153]: pgmap v6084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:46:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:30 smithi077 ceph-mon[37153]: pgmap v6085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:46:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:46:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:31.394+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:46:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:31.394+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:46:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:31.394+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:46:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:31.404+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:46:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:31.404+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:46:31.406 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:31.404+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:46:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10614.604233 secs 2024-07-27T09:46:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:33 smithi047 ceph-mon[36835]: pgmap v6086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:46:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10614.604233 secs 2024-07-27T09:46:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:33 smithi029 ceph-mon[32610]: pgmap v6086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:46:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10614.604233 secs 2024-07-27T09:46:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:33 smithi077 ceph-mon[37153]: pgmap v6086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:46:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:35 smithi029 ceph-mon[32610]: pgmap v6087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:46:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:35 smithi047 ceph-mon[36835]: pgmap v6087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:46:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:35 smithi077 ceph-mon[37153]: pgmap v6087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:46:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:37 smithi047 ceph-mon[36835]: pgmap v6088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:46:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:37 smithi029 ceph-mon[32610]: pgmap v6088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:46:37.604 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:37 smithi077 ceph-mon[37153]: pgmap v6088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T09:46:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10619.604351 secs 2024-07-27T09:46:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10619.604351 secs 2024-07-27T09:46:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10619.604351 secs 2024-07-27T09:46:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:39 smithi047 ceph-mon[36835]: pgmap v6089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:46:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:39 smithi029 ceph-mon[32610]: pgmap v6089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:46:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:39 smithi077 ceph-mon[37153]: pgmap v6089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:46:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:41 smithi047 ceph-mon[36835]: pgmap v6090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:46:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:41 smithi029 ceph-mon[32610]: pgmap v6090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:46:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:41 smithi077 ceph-mon[37153]: pgmap v6090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:46:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10624.604473 secs 2024-07-27T09:46:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:43 smithi047 ceph-mon[36835]: pgmap v6091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10624.604473 secs 2024-07-27T09:46:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:43 smithi029 ceph-mon[32610]: pgmap v6091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10624.604473 secs 2024-07-27T09:46:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:43 smithi077 ceph-mon[37153]: pgmap v6091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:44.243+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:46:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:44.243+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:46:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:44.243+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:46:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:44.243+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:46:45.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:45 smithi047 ceph-mon[36835]: pgmap v6092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:46:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:45 smithi029 ceph-mon[32610]: pgmap v6092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:46:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:45 smithi077 ceph-mon[37153]: pgmap v6092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:46:46.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:46.409+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:46:46.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:46.409+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:46:46.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:46.409+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:46:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:46.420+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:46:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:46.420+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:46:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:46:46.420+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:46:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:47 smithi047 ceph-mon[36835]: pgmap v6093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:47 smithi029 ceph-mon[32610]: pgmap v6093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:47 smithi077 ceph-mon[37153]: pgmap v6093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10629.604591 secs 2024-07-27T09:46:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10629.604591 secs 2024-07-27T09:46:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10629.604591 secs 2024-07-27T09:46:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:49 smithi047 ceph-mon[36835]: pgmap v6094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:46:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:49 smithi029 ceph-mon[32610]: pgmap v6094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:46:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:49 smithi077 ceph-mon[37153]: pgmap v6094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:46:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:51 smithi047 ceph-mon[36835]: pgmap v6095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:51 smithi029 ceph-mon[32610]: pgmap v6095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:51 smithi077 ceph-mon[37153]: pgmap v6095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10634.604730 secs 2024-07-27T09:46:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:53 smithi047 ceph-mon[36835]: pgmap v6096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10634.604730 secs 2024-07-27T09:46:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:53 smithi029 ceph-mon[32610]: pgmap v6096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10634.604730 secs 2024-07-27T09:46:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:53 smithi077 ceph-mon[37153]: pgmap v6096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:46:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:55 smithi047 ceph-mon[36835]: pgmap v6097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:46:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:55 smithi029 ceph-mon[32610]: pgmap v6097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:46:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:55 smithi077 ceph-mon[37153]: pgmap v6097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:46:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:57 smithi047 ceph-mon[36835]: pgmap v6098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:57 smithi029 ceph-mon[32610]: pgmap v6098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:57 smithi077 ceph-mon[37153]: pgmap v6098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:46:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10639.604854 secs 2024-07-27T09:46:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10639.604854 secs 2024-07-27T09:46:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10639.604854 secs 2024-07-27T09:46:59.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:59.245+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:46:59.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:59.245+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:46:59.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:59.246+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:46:59.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:46:59.246+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:46:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:46:59 smithi047 ceph-mon[36835]: pgmap v6099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:46:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:46:59 smithi029 ceph-mon[32610]: pgmap v6099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:46:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:46:59 smithi077 ceph-mon[37153]: pgmap v6099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:47:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:47:01.421+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:47:01.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:47:01.421+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:47:01.423 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:47:01.421+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:47:01.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:47:01.425+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:47:01.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:47:01.425+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:47:01.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:47:01.425+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:47:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:01 smithi047 ceph-mon[36835]: pgmap v6100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:01 smithi029 ceph-mon[32610]: pgmap v6100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:01.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:01 smithi077 ceph-mon[37153]: pgmap v6100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10644.604939 secs 2024-07-27T09:47:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:03 smithi047 ceph-mon[36835]: pgmap v6101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10644.604939 secs 2024-07-27T09:47:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:03 smithi029 ceph-mon[32610]: pgmap v6101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10644.604939 secs 2024-07-27T09:47:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:03 smithi077 ceph-mon[37153]: pgmap v6101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:05 smithi047 ceph-mon[36835]: pgmap v6102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:47:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:05 smithi029 ceph-mon[32610]: pgmap v6102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:47:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:05 smithi077 ceph-mon[37153]: pgmap v6102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:47:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:07 smithi047 ceph-mon[36835]: pgmap v6103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:07 smithi029 ceph-mon[32610]: pgmap v6103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:07 smithi077 ceph-mon[37153]: pgmap v6103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10649.605048 secs 2024-07-27T09:47:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10649.605048 secs 2024-07-27T09:47:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10649.605048 secs 2024-07-27T09:47:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:09 smithi029 ceph-mon[32610]: pgmap v6104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:47:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:09 smithi077 ceph-mon[37153]: pgmap v6104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:47:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:09 smithi047 ceph-mon[36835]: pgmap v6104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:47:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:11 smithi029 ceph-mon[32610]: pgmap v6105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:11 smithi077 ceph-mon[37153]: pgmap v6105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:11 smithi047 ceph-mon[36835]: pgmap v6105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10654.605148 secs 2024-07-27T09:47:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:13 smithi029 ceph-mon[32610]: pgmap v6106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10654.605148 secs 2024-07-27T09:47:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:13 smithi077 ceph-mon[37153]: pgmap v6106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10654.605148 secs 2024-07-27T09:47:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:13 smithi047 ceph-mon[36835]: pgmap v6106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:15 smithi029 ceph-mon[32610]: pgmap v6107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:47:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:15 smithi077 ceph-mon[37153]: pgmap v6107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:47:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:15 smithi047 ceph-mon[36835]: pgmap v6107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:47:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:47:16.436+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:47:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:47:16.436+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:47:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:47:16.436+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:47:16.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:47:16.436+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:47:16.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:47:16.436+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:47:16.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:47:16.436+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:47:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:17 smithi077 ceph-mon[37153]: pgmap v6108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:17 smithi047 ceph-mon[36835]: pgmap v6108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:17 smithi029 ceph-mon[32610]: pgmap v6108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10659.605267 secs 2024-07-27T09:47:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10659.605267 secs 2024-07-27T09:47:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10659.605267 secs 2024-07-27T09:47:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:19 smithi077 ceph-mon[37153]: pgmap v6109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:47:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:19 smithi047 ceph-mon[36835]: pgmap v6109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:47:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:19 smithi029 ceph-mon[32610]: pgmap v6109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:47:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:21 smithi077 ceph-mon[37153]: pgmap v6110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:21.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:21 smithi047 ceph-mon[36835]: pgmap v6110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:21 smithi029 ceph-mon[32610]: pgmap v6110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10664.605391 secs 2024-07-27T09:47:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:22 smithi077 ceph-mon[37153]: pgmap v6111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10664.605391 secs 2024-07-27T09:47:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:22 smithi047 ceph-mon[36835]: pgmap v6111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10664.605391 secs 2024-07-27T09:47:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:22 smithi029 ceph-mon[32610]: pgmap v6111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:25 smithi047 ceph-mon[36835]: pgmap v6112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:47:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:25 smithi029 ceph-mon[32610]: pgmap v6112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:47:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:25 smithi077 ceph-mon[37153]: pgmap v6112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:47:27.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:27 smithi077 ceph-mon[37153]: pgmap v6113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:47:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:27 smithi047 ceph-mon[36835]: pgmap v6113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:47:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:27 smithi029 ceph-mon[32610]: pgmap v6113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:47:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10669.605510 secs 2024-07-27T09:47:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10669.605510 secs 2024-07-27T09:47:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10669.605510 secs 2024-07-27T09:47:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:29 smithi047 ceph-mon[36835]: pgmap v6114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:47:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:29 smithi029 ceph-mon[32610]: pgmap v6114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:47:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:29 smithi077 ceph-mon[37153]: pgmap v6114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:47:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:31 smithi047 ceph-mon[36835]: pgmap v6115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:47:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:47:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:47:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:47:31.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:47:31.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:47:31.452+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:47:31.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:47:31.452+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:47:31.453 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:47:31.452+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:47:31.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:47:31.452+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:47:31.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:47:31.452+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:47:31.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:47:31.452+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:47:31.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:31 smithi029 ceph-mon[32610]: pgmap v6115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:47:31.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:47:31.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:47:31.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:47:31.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:47:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:31 smithi077 ceph-mon[37153]: pgmap v6115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:47:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:47:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:47:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:47:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:47:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:32 smithi047 ceph-mon[36835]: pgmap v6116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:47:32.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:32 smithi047 ceph-mon[36835]: pgmap v6117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-27T09:47:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:32 smithi029 ceph-mon[32610]: pgmap v6116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:47:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:32 smithi029 ceph-mon[32610]: pgmap v6117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-27T09:47:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:32 smithi077 ceph-mon[37153]: pgmap v6116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:47:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:32 smithi077 ceph-mon[37153]: pgmap v6117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-27T09:47:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10674.605664 secs 2024-07-27T09:47:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10674.605664 secs 2024-07-27T09:47:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10674.605664 secs 2024-07-27T09:47:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:34 smithi047 ceph-mon[36835]: pgmap v6118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:47:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:34 smithi029 ceph-mon[32610]: pgmap v6118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:47:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:34 smithi077 ceph-mon[37153]: pgmap v6118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:47:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:36 smithi047 ceph-mon[36835]: pgmap v6119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:47:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:36 smithi029 ceph-mon[32610]: pgmap v6119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:47:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:36 smithi077 ceph-mon[37153]: pgmap v6119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:47:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:38 smithi047 ceph-mon[36835]: pgmap v6120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:47:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10679.605781 secs 2024-07-27T09:47:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:38 smithi029 ceph-mon[32610]: pgmap v6120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:47:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10679.605781 secs 2024-07-27T09:47:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:38 smithi077 ceph-mon[37153]: pgmap v6120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:47:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10679.605781 secs 2024-07-27T09:47:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:40 smithi047 ceph-mon[36835]: pgmap v6121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:47:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:40 smithi029 ceph-mon[32610]: pgmap v6121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:47:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:40 smithi077 ceph-mon[37153]: pgmap v6121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:47:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:42 smithi047 ceph-mon[36835]: pgmap v6122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:47:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:42 smithi029 ceph-mon[32610]: pgmap v6122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:47:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:42 smithi077 ceph-mon[37153]: pgmap v6122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:47:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10684.605859 secs 2024-07-27T09:47:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10684.605859 secs 2024-07-27T09:47:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10684.605859 secs 2024-07-27T09:47:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:44 smithi047 ceph-mon[36835]: pgmap v6123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:47:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:44 smithi029 ceph-mon[32610]: pgmap v6123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:47:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:44 smithi077 ceph-mon[37153]: pgmap v6123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:47:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:46 smithi047 ceph-mon[36835]: pgmap v6124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:47:46.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:47:46.453+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:47:46.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:47:46.453+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:47:46.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:47:46.453+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:47:46.455 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:46 smithi029 ceph-mon[32610]: pgmap v6124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:47:46.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:47:46.453+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:47:46.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:47:46.453+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:47:46.456 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:47:46.453+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:47:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:46 smithi077 ceph-mon[37153]: pgmap v6124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:47:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:48 smithi047 ceph-mon[36835]: pgmap v6125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10689.605989 secs 2024-07-27T09:47:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:48 smithi029 ceph-mon[32610]: pgmap v6125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10689.605989 secs 2024-07-27T09:47:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:48 smithi077 ceph-mon[37153]: pgmap v6125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10689.605989 secs 2024-07-27T09:47:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:50 smithi047 ceph-mon[36835]: pgmap v6126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:47:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:50 smithi029 ceph-mon[32610]: pgmap v6126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:47:50.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:50 smithi077 ceph-mon[37153]: pgmap v6126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:47:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:52 smithi029 ceph-mon[32610]: pgmap v6127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:52 smithi077 ceph-mon[37153]: pgmap v6127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:52 smithi047 ceph-mon[36835]: pgmap v6127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:47:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10694.606053 secs 2024-07-27T09:47:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10694.606053 secs 2024-07-27T09:47:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10694.606053 secs 2024-07-27T09:47:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:54 smithi029 ceph-mon[32610]: pgmap v6128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:47:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:54 smithi077 ceph-mon[37153]: pgmap v6128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:47:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:54 smithi047 ceph-mon[36835]: pgmap v6128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:47:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:56 smithi029 ceph-mon[32610]: pgmap v6129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:47:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:56 smithi077 ceph-mon[37153]: pgmap v6129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:47:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:56 smithi047 ceph-mon[36835]: pgmap v6129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:47:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:58 smithi029 ceph-mon[32610]: pgmap v6130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:47:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10699.606177 secs 2024-07-27T09:47:58.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:58 smithi047 ceph-mon[36835]: pgmap v6130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:58.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:47:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10699.606177 secs 2024-07-27T09:47:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:58 smithi077 ceph-mon[37153]: pgmap v6130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:47:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:47:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10699.606177 secs 2024-07-27T09:48:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:00 smithi077 ceph-mon[37153]: pgmap v6131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:00 smithi047 ceph-mon[36835]: pgmap v6131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:00 smithi029 ceph-mon[32610]: pgmap v6131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:01.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:48:01.455+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:48:01.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:48:01.455+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:48:01.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:48:01.455+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:48:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:48:01.469+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:48:01.470 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:48:01.469+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:48:01.471 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:48:01.469+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:48:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:02 smithi077 ceph-mon[37153]: pgmap v6132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:02.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:02 smithi047 ceph-mon[36835]: pgmap v6132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:02 smithi029 ceph-mon[32610]: pgmap v6132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10704.606300 secs 2024-07-27T09:48:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10704.606300 secs 2024-07-27T09:48:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10704.606300 secs 2024-07-27T09:48:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:04 smithi077 ceph-mon[37153]: pgmap v6133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:48:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:04 smithi047 ceph-mon[36835]: pgmap v6133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:48:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:04 smithi029 ceph-mon[32610]: pgmap v6133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:48:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:06 smithi077 ceph-mon[37153]: pgmap v6134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:48:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:06 smithi047 ceph-mon[36835]: pgmap v6134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:48:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:06 smithi029 ceph-mon[32610]: pgmap v6134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:48:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:08 smithi029 ceph-mon[32610]: pgmap v6135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:48:08.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10709.606425 secs 2024-07-27T09:48:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:08 smithi077 ceph-mon[37153]: pgmap v6135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:48:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10709.606425 secs 2024-07-27T09:48:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:08 smithi047 ceph-mon[36835]: pgmap v6135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:48:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10709.606425 secs 2024-07-27T09:48:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:10 smithi077 ceph-mon[37153]: pgmap v6136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:10 smithi047 ceph-mon[36835]: pgmap v6136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:10 smithi029 ceph-mon[32610]: pgmap v6136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:12 smithi077 ceph-mon[37153]: pgmap v6137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:12 smithi047 ceph-mon[36835]: pgmap v6137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:12 smithi029 ceph-mon[32610]: pgmap v6137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10714.606547 secs 2024-07-27T09:48:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10714.606547 secs 2024-07-27T09:48:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10714.606547 secs 2024-07-27T09:48:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:14 smithi077 ceph-mon[37153]: pgmap v6138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:48:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:14 smithi047 ceph-mon[36835]: pgmap v6138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:48:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:14 smithi029 ceph-mon[32610]: pgmap v6138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:48:16.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:48:16.469+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:48:16.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:48:16.469+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:48:16.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:48:16.469+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:48:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:48:16.483+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:48:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:48:16.483+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:48:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:48:16.483+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:48:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:16 smithi077 ceph-mon[37153]: pgmap v6139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:48:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:16 smithi047 ceph-mon[36835]: pgmap v6139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:48:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:16 smithi029 ceph-mon[32610]: pgmap v6139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:48:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:18 smithi077 ceph-mon[37153]: pgmap v6140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:48:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10719.606683 secs 2024-07-27T09:48:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:18 smithi047 ceph-mon[36835]: pgmap v6140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:48:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10719.606683 secs 2024-07-27T09:48:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:18 smithi029 ceph-mon[32610]: pgmap v6140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:48:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10719.606683 secs 2024-07-27T09:48:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:20 smithi077 ceph-mon[37153]: pgmap v6141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:20 smithi047 ceph-mon[36835]: pgmap v6141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:20 smithi029 ceph-mon[32610]: pgmap v6141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:22 smithi077 ceph-mon[37153]: pgmap v6142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:22 smithi047 ceph-mon[36835]: pgmap v6142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:22 smithi029 ceph-mon[32610]: pgmap v6142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10724.606842 secs 2024-07-27T09:48:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10724.606842 secs 2024-07-27T09:48:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10724.606842 secs 2024-07-27T09:48:24.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:24 smithi077 ceph-mon[37153]: pgmap v6143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:48:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:24 smithi047 ceph-mon[36835]: pgmap v6143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:48:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:24 smithi029 ceph-mon[32610]: pgmap v6143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:48:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:25 smithi077 ceph-mon[37153]: pgmap v6144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:48:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:25 smithi047 ceph-mon[36835]: pgmap v6144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:48:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:25 smithi029 ceph-mon[32610]: pgmap v6144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:48:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:27 smithi029 ceph-mon[32610]: pgmap v6145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:48:27.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10729.606911 secs 2024-07-27T09:48:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:27 smithi077 ceph-mon[37153]: pgmap v6145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:48:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10729.606911 secs 2024-07-27T09:48:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:27 smithi047 ceph-mon[36835]: pgmap v6145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:48:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10729.606911 secs 2024-07-27T09:48:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:29 smithi029 ceph-mon[32610]: pgmap v6146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:29 smithi077 ceph-mon[37153]: pgmap v6146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:29 smithi047 ceph-mon[36835]: pgmap v6146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:31.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:48:31.485+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:48:31.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:48:31.485+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:48:31.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:48:31.485+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:48:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:48:31.498+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:48:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:48:31.498+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:48:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:48:31.498+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:48:31.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:31 smithi029 ceph-mon[32610]: pgmap v6147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:31.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:48:31.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:31.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:31 smithi077 ceph-mon[37153]: pgmap v6147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:48:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:31 smithi047 ceph-mon[36835]: pgmap v6147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:48:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:32.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10734.606982 secs 2024-07-27T09:48:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:32.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:32.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:32 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10734.606982 secs 2024-07-27T09:48:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:32 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10734.606982 secs 2024-07-27T09:48:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:33.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:32 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:33 smithi047 ceph-mon[36835]: pgmap v6148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:48:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:48:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:48:34.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:33 smithi029 ceph-mon[32610]: pgmap v6148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:48:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:48:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:48:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:33 smithi077 ceph-mon[37153]: pgmap v6148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:48:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:48:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:48:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:48:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:34 smithi047 ceph-mon[36835]: pgmap v6149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:48:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:34 smithi047 ceph-mon[36835]: pgmap v6150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:48:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:34 smithi029 ceph-mon[32610]: pgmap v6149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:48:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:34 smithi029 ceph-mon[32610]: pgmap v6150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:48:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:34 smithi077 ceph-mon[37153]: pgmap v6149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:48:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:34 smithi077 ceph-mon[37153]: pgmap v6150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:48:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:36 smithi047 ceph-mon[36835]: pgmap v6151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:48:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:36 smithi029 ceph-mon[32610]: pgmap v6151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:48:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:36 smithi077 ceph-mon[37153]: pgmap v6151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:48:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10739.607066 secs 2024-07-27T09:48:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10739.607066 secs 2024-07-27T09:48:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10739.607066 secs 2024-07-27T09:48:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:38 smithi047 ceph-mon[36835]: pgmap v6152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:48:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:38 smithi029 ceph-mon[32610]: pgmap v6152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:48:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:38 smithi077 ceph-mon[37153]: pgmap v6152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:48:40.927 INFO:tasks.workunit.client.0.smithi029.stderr:Terminated 2024-07-27T09:48:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-07-27T09:48:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:40 smithi047 ceph-mon[36835]: pgmap v6153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:40 smithi029 ceph-mon[32610]: pgmap v6153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:40 smithi077 ceph-mon[37153]: pgmap v6153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:48:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:42 smithi047 ceph-mon[36835]: pgmap v6154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T09:48:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10744.607195 secs 2024-07-27T09:48:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:42 smithi029 ceph-mon[32610]: pgmap v6154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T09:48:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10744.607195 secs 2024-07-27T09:48:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:42 smithi077 ceph-mon[37153]: pgmap v6154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T09:48:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10744.607195 secs 2024-07-27T09:48:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:44 smithi047 ceph-mon[36835]: pgmap v6155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:48:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:44 smithi029 ceph-mon[32610]: pgmap v6155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:48:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:44 smithi077 ceph-mon[37153]: pgmap v6155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:48:46.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:48:46.491+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:48:46.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:48:46.491+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:48:46.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:48:46.491+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:48:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:48:46.501+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:48:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:48:46.501+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:48:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:48:46.501+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:48:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:46 smithi047 ceph-mon[36835]: pgmap v6156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:48:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:46 smithi029 ceph-mon[32610]: pgmap v6156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:48:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:46 smithi077 ceph-mon[37153]: pgmap v6156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:48:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10749.607270 secs 2024-07-27T09:48:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10749.607270 secs 2024-07-27T09:48:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10749.607270 secs 2024-07-27T09:48:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:48 smithi047 ceph-mon[36835]: pgmap v6157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:48:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:48 smithi029 ceph-mon[32610]: pgmap v6157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:48:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:48 smithi077 ceph-mon[37153]: pgmap v6157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:48:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:50 smithi029 ceph-mon[32610]: pgmap v6158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:48:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:50 smithi077 ceph-mon[37153]: pgmap v6158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:48:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:50 smithi047 ceph-mon[36835]: pgmap v6158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:48:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:52 smithi029 ceph-mon[32610]: pgmap v6159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:48:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10754.607400 secs 2024-07-27T09:48:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:52 smithi077 ceph-mon[37153]: pgmap v6159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:48:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10754.607400 secs 2024-07-27T09:48:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:52 smithi047 ceph-mon[36835]: pgmap v6159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:48:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10754.607400 secs 2024-07-27T09:48:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:54 smithi029 ceph-mon[32610]: pgmap v6160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:48:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:54 smithi077 ceph-mon[37153]: pgmap v6160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:48:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:54 smithi047 ceph-mon[36835]: pgmap v6160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:48:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:56 smithi029 ceph-mon[32610]: pgmap v6161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:56 smithi077 ceph-mon[37153]: pgmap v6161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:56 smithi047 ceph-mon[36835]: pgmap v6161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:48:58.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10759.607538 secs 2024-07-27T09:48:58.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10759.607538 secs 2024-07-27T09:48:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10759.607538 secs 2024-07-27T09:48:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:48:58 smithi077 ceph-mon[37153]: pgmap v6162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:48:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:48:58 smithi047 ceph-mon[36835]: pgmap v6162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:48:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:48:58 smithi029 ceph-mon[32610]: pgmap v6162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:49:01.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:00 smithi029 ceph-mon[32610]: pgmap v6163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:49:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:00 smithi077 ceph-mon[37153]: pgmap v6163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:49:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:00 smithi047 ceph-mon[36835]: pgmap v6163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:49:01.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:01.494+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:49:01.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:01.495+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:49:01.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:01.495+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:49:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:01.517+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:49:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:01.517+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:49:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:01.517+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:49:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:02 smithi077 ceph-mon[37153]: pgmap v6164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10764.607660 secs 2024-07-27T09:49:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:02 smithi047 ceph-mon[36835]: pgmap v6164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10764.607660 secs 2024-07-27T09:49:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:02 smithi029 ceph-mon[32610]: pgmap v6164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10764.607660 secs 2024-07-27T09:49:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:04 smithi077 ceph-mon[37153]: pgmap v6165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:04 smithi047 ceph-mon[36835]: pgmap v6165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:04 smithi029 ceph-mon[32610]: pgmap v6165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:06 smithi077 ceph-mon[37153]: pgmap v6166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:49:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:06 smithi047 ceph-mon[36835]: pgmap v6166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:49:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:06 smithi029 ceph-mon[32610]: pgmap v6166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:49:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10769.607771 secs 2024-07-27T09:49:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10769.607771 secs 2024-07-27T09:49:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10769.607771 secs 2024-07-27T09:49:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:08 smithi077 ceph-mon[37153]: pgmap v6167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:49:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:08 smithi047 ceph-mon[36835]: pgmap v6167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:49:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:08 smithi029 ceph-mon[32610]: pgmap v6167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:49:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:10 smithi077 ceph-mon[37153]: pgmap v6168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:49:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:10 smithi047 ceph-mon[36835]: pgmap v6168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:49:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:10 smithi029 ceph-mon[32610]: pgmap v6168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:49:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:12 smithi077 ceph-mon[37153]: pgmap v6169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10774.607892 secs 2024-07-27T09:49:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:12 smithi047 ceph-mon[36835]: pgmap v6169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10774.607892 secs 2024-07-27T09:49:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:12 smithi029 ceph-mon[32610]: pgmap v6169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10774.607892 secs 2024-07-27T09:49:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:14 smithi077 ceph-mon[37153]: pgmap v6170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:14 smithi047 ceph-mon[36835]: pgmap v6170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:14 smithi029 ceph-mon[32610]: pgmap v6170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:16.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:16.507+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:49:16.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:16.507+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:49:16.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:16.507+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:49:16.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:16.517+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:49:16.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:16.517+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:49:16.519 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:16.517+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:49:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:16 smithi077 ceph-mon[37153]: pgmap v6171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:49:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:16 smithi047 ceph-mon[36835]: pgmap v6171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:49:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:16 smithi029 ceph-mon[32610]: pgmap v6171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:49:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10779.608016 secs 2024-07-27T09:49:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10779.608016 secs 2024-07-27T09:49:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10779.608016 secs 2024-07-27T09:49:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:19 smithi077 ceph-mon[37153]: pgmap v6172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:49:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:19 smithi047 ceph-mon[36835]: pgmap v6172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:49:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:19 smithi029 ceph-mon[32610]: pgmap v6172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:49:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:21 smithi077 ceph-mon[37153]: pgmap v6173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:49:21.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:21 smithi047 ceph-mon[36835]: pgmap v6173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:49:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:21 smithi029 ceph-mon[32610]: pgmap v6173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:49:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:23 smithi077 ceph-mon[37153]: pgmap v6174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10784.608162 secs 2024-07-27T09:49:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:23 smithi047 ceph-mon[36835]: pgmap v6174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10784.608162 secs 2024-07-27T09:49:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:23 smithi029 ceph-mon[32610]: pgmap v6174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10784.608162 secs 2024-07-27T09:49:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:25 smithi077 ceph-mon[37153]: pgmap v6175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:25 smithi047 ceph-mon[36835]: pgmap v6175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:25 smithi029 ceph-mon[32610]: pgmap v6175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:27 smithi077 ceph-mon[37153]: pgmap v6176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:49:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:27 smithi047 ceph-mon[36835]: pgmap v6176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:49:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:27 smithi029 ceph-mon[32610]: pgmap v6176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:49:28.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10789.608241 secs 2024-07-27T09:49:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10789.608241 secs 2024-07-27T09:49:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10789.608241 secs 2024-07-27T09:49:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:29 smithi077 ceph-mon[37153]: pgmap v6177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:49:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:29 smithi047 ceph-mon[36835]: pgmap v6177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:49:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:29 smithi029 ceph-mon[32610]: pgmap v6177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:49:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:31 smithi077 ceph-mon[37153]: pgmap v6178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:49:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:31 smithi047 ceph-mon[36835]: pgmap v6178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:49:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:31 smithi029 ceph-mon[32610]: pgmap v6178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:49:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:31.522+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:49:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:31.522+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:49:31.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:31.522+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:49:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:31.533+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:49:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:31.533+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:49:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:31.533+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:49:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:33 smithi077 ceph-mon[37153]: pgmap v6179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10794.608383 secs 2024-07-27T09:49:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:33 smithi047 ceph-mon[36835]: pgmap v6179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10794.608383 secs 2024-07-27T09:49:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:33 smithi029 ceph-mon[32610]: pgmap v6179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10794.608383 secs 2024-07-27T09:49:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:49:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:49:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:34.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:49:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:34.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:35 smithi077 ceph-mon[37153]: pgmap v6180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:49:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:49:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:35 smithi077 ceph-mon[37153]: pgmap v6181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:49:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:35 smithi077 ceph-mon[37153]: pgmap v6182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:49:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:35 smithi047 ceph-mon[36835]: pgmap v6180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:35.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:49:35.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:49:35.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:35 smithi047 ceph-mon[36835]: pgmap v6181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:49:35.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:35 smithi047 ceph-mon[36835]: pgmap v6182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:49:35.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:35 smithi029 ceph-mon[32610]: pgmap v6180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:49:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:49:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:35 smithi029 ceph-mon[32610]: pgmap v6181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:49:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:35 smithi029 ceph-mon[32610]: pgmap v6182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:49:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:49:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:37 smithi029 ceph-mon[32610]: pgmap v6183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:49:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:37 smithi077 ceph-mon[37153]: pgmap v6183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:49:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:37 smithi047 ceph-mon[36835]: pgmap v6183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:49:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10799.608502 secs 2024-07-27T09:49:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10799.608502 secs 2024-07-27T09:49:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10799.608502 secs 2024-07-27T09:49:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:39 smithi077 ceph-mon[37153]: pgmap v6184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T09:49:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:39 smithi047 ceph-mon[36835]: pgmap v6184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T09:49:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:39 smithi029 ceph-mon[32610]: pgmap v6184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T09:49:40.930 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-07-27T09:49:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:41 smithi077 ceph-mon[37153]: pgmap v6185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:49:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:41 smithi047 ceph-mon[36835]: pgmap v6185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:49:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:41 smithi029 ceph-mon[32610]: pgmap v6185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:49:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:42.645+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-27T09:49:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:42.645+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T09:49:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:42.645+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f733c004010 tx=0x7f733c07a000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T09:49:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:42.645+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f733c004010 tx=0x7f733c07a000 comp rx=0 tx=0).stop 2024-07-27T09:49:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:42.645+0000 7f734ce53640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T09:49:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:42.645+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 msgr2=0x7f73380c0230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T09:49:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:42.645+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380bde10 0x7f73380c0230 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:49:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:42.645+0000 7f7345ffb640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:49:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:42.645+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:49:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:42.645+0000 7f734ce53640 1 --2- 172.21.15.29:0/1357579498 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f73380b3d40 0x7f73380b5310 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f733c019550 tx=0x7f733c090000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:49:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10804.608645 secs 2024-07-27T09:49:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:43 smithi077 ceph-mon[37153]: pgmap v6186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T09:49:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10804.608645 secs 2024-07-27T09:49:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:43 smithi047 ceph-mon[36835]: pgmap v6186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T09:49:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10804.608645 secs 2024-07-27T09:49:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:43 smithi029 ceph-mon[32610]: pgmap v6186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T09:49:44.271 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:44.270+0000 7f7344ff9640 1 -- 172.21.15.29:0/1357579498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:49:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:44.270+0000 7f7326ffd640 1 -- 172.21.15.29:0/940768653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:49:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:44.270+0000 7f7345ffb640 1 -- 172.21.15.29:0/1357579498 <== mon.0 v2:172.21.15.29:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f73200065a0 con 0x7f7348108670 2024-07-27T09:49:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:44.270+0000 7f7327fff640 1 -- 172.21.15.29:0/940768653 <== mon.0 v2:172.21.15.29:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f7318006490 con 0x5645ed3392f0 2024-07-27T09:49:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:45 smithi077 ceph-mon[37153]: pgmap v6187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:49:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:45 smithi047 ceph-mon[36835]: pgmap v6187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:49:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:45 smithi029 ceph-mon[32610]: pgmap v6187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:49:46.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:46.538+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:49:46.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:46.538+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:49:46.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:46.538+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:49:46.548 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:46.547+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:49:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:46.547+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:49:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:49:46.547+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:49:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:47 smithi077 ceph-mon[37153]: pgmap v6188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:47 smithi047 ceph-mon[36835]: pgmap v6188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:47 smithi029 ceph-mon[32610]: pgmap v6188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10809.608797 secs 2024-07-27T09:49:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10809.608797 secs 2024-07-27T09:49:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10809.608797 secs 2024-07-27T09:49:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:49 smithi077 ceph-mon[37153]: pgmap v6189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:49 smithi047 ceph-mon[36835]: pgmap v6189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:49 smithi029 ceph-mon[32610]: pgmap v6189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:51 smithi077 ceph-mon[37153]: pgmap v6190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:49:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:51 smithi047 ceph-mon[36835]: pgmap v6190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:49:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:51 smithi029 ceph-mon[32610]: pgmap v6190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:49:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10814.608929 secs 2024-07-27T09:49:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:53 smithi077 ceph-mon[37153]: pgmap v6191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10814.608929 secs 2024-07-27T09:49:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:53 smithi047 ceph-mon[36835]: pgmap v6191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10814.608929 secs 2024-07-27T09:49:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:53 smithi029 ceph-mon[32610]: pgmap v6191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:55 smithi077 ceph-mon[37153]: pgmap v6192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:49:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:55 smithi047 ceph-mon[36835]: pgmap v6192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:49:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:55 smithi029 ceph-mon[32610]: pgmap v6192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:49:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:57 smithi029 ceph-mon[32610]: pgmap v6193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:57 smithi077 ceph-mon[37153]: pgmap v6193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:57.664+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T09:49:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:57.664+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T09:49:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:57.664+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7eff40048000 tx=0x7eff40038040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T09:49:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:57.664+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7eff40048000 tx=0x7eff40038040 comp rx=0 tx=0).stop 2024-07-27T09:49:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:57.664+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T09:49:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:57.664+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T09:49:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:57.664+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0bdeb0 0x7eff3c0c02d0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:49:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:57.664+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 0x7eff3c0b5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:49:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:57.664+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 0x7eff3c0b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:49:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:57.665+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 0x7eff3c0b5580 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7eff400048c0 tx=0x7eff40038040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:49:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:57 smithi047 ceph-mon[36835]: pgmap v6193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:49:58.509 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10819.609085 secs 2024-07-27T09:49:58.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10819.609085 secs 2024-07-27T09:49:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10819.609085 secs 2024-07-27T09:49:59.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:59.273+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:49:59.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:59.273+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:49:59.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:59.273+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:49:59.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:49:59.273+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:49:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:49:59 smithi077 ceph-mon[37153]: pgmap v6194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:49:59 smithi047 ceph-mon[36835]: pgmap v6194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:49:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:49:59 smithi029 ceph-mon[32610]: pgmap v6194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:50:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:50:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T09:50:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:01.542+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:50:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:01.542+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:50:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:01.542+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:50:01.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:01 smithi029 ceph-mon[32610]: pgmap v6195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:50:01.563+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:50:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:50:01.563+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:50:01.564 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:50:01.563+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:50:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:01 smithi077 ceph-mon[37153]: pgmap v6195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:01 smithi047 ceph-mon[36835]: pgmap v6195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10824.609211 secs 2024-07-27T09:50:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:03 smithi077 ceph-mon[37153]: pgmap v6196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10824.609211 secs 2024-07-27T09:50:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:03 smithi047 ceph-mon[36835]: pgmap v6196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10824.609211 secs 2024-07-27T09:50:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:03 smithi029 ceph-mon[32610]: pgmap v6196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:05 smithi077 ceph-mon[37153]: pgmap v6197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:50:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:05 smithi047 ceph-mon[36835]: pgmap v6197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:50:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:05 smithi029 ceph-mon[32610]: pgmap v6197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:50:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:06 smithi077 ceph-mon[37153]: pgmap v6198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:06 smithi047 ceph-mon[36835]: pgmap v6198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:06 smithi029 ceph-mon[32610]: pgmap v6198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10829.609317 secs 2024-07-27T09:50:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10829.609317 secs 2024-07-27T09:50:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10829.609317 secs 2024-07-27T09:50:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:08 smithi077 ceph-mon[37153]: pgmap v6199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:08 smithi047 ceph-mon[36835]: pgmap v6199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:08 smithi029 ceph-mon[32610]: pgmap v6199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:11 smithi077 ceph-mon[37153]: pgmap v6200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:11 smithi047 ceph-mon[36835]: pgmap v6200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:11 smithi029 ceph-mon[32610]: pgmap v6200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10834.609416 secs 2024-07-27T09:50:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:13 smithi029 ceph-mon[32610]: pgmap v6201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10834.609416 secs 2024-07-27T09:50:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:13 smithi077 ceph-mon[37153]: pgmap v6201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10834.609416 secs 2024-07-27T09:50:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:13 smithi047 ceph-mon[36835]: pgmap v6201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:15 smithi077 ceph-mon[37153]: pgmap v6202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:50:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:15 smithi047 ceph-mon[36835]: pgmap v6202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:50:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:15 smithi029 ceph-mon[32610]: pgmap v6202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:50:16.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:16.556+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:50:16.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:16.556+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:50:16.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:16.556+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:50:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:50:16.570+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:50:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:50:16.570+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:50:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:50:16.570+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:50:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:17 smithi077 ceph-mon[37153]: pgmap v6203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:17 smithi047 ceph-mon[36835]: pgmap v6203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:17 smithi029 ceph-mon[32610]: pgmap v6203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10839.609535 secs 2024-07-27T09:50:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10839.609535 secs 2024-07-27T09:50:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10839.609535 secs 2024-07-27T09:50:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:19 smithi077 ceph-mon[37153]: pgmap v6204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:19 smithi047 ceph-mon[36835]: pgmap v6204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:19 smithi029 ceph-mon[32610]: pgmap v6204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:21 smithi077 ceph-mon[37153]: pgmap v6205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:21 smithi047 ceph-mon[36835]: pgmap v6205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:21 smithi029 ceph-mon[32610]: pgmap v6205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10844.609689 secs 2024-07-27T09:50:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:23 smithi077 ceph-mon[37153]: pgmap v6206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10844.609689 secs 2024-07-27T09:50:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:23 smithi047 ceph-mon[36835]: pgmap v6206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10844.609689 secs 2024-07-27T09:50:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:23 smithi029 ceph-mon[32610]: pgmap v6206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:25 smithi077 ceph-mon[37153]: pgmap v6207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:50:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:25 smithi047 ceph-mon[36835]: pgmap v6207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:50:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:25 smithi029 ceph-mon[32610]: pgmap v6207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:50:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:27 smithi077 ceph-mon[37153]: pgmap v6208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:27 smithi047 ceph-mon[36835]: pgmap v6208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:27 smithi029 ceph-mon[32610]: pgmap v6208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:28.503 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10849.609822 secs 2024-07-27T09:50:28.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10849.609822 secs 2024-07-27T09:50:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10849.609822 secs 2024-07-27T09:50:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:29 smithi077 ceph-mon[37153]: pgmap v6209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:29 smithi047 ceph-mon[36835]: pgmap v6209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:29.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:29 smithi029 ceph-mon[32610]: pgmap v6209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:31.572+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:50:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:31.572+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:50:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:31.572+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:50:31.575 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:31 smithi029 ceph-mon[32610]: pgmap v6210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:31.587 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:50:31.585+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:50:31.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:50:31.586+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:50:31.588 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:50:31.586+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:50:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:31 smithi077 ceph-mon[37153]: pgmap v6210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:31 smithi047 ceph-mon[36835]: pgmap v6210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10854.609942 secs 2024-07-27T09:50:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:33 smithi077 ceph-mon[37153]: pgmap v6211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10854.609942 secs 2024-07-27T09:50:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:33 smithi047 ceph-mon[36835]: pgmap v6211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10854.609942 secs 2024-07-27T09:50:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:33 smithi029 ceph-mon[32610]: pgmap v6211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:50:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:50:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:50:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:35 smithi077 ceph-mon[37153]: pgmap v6212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:50:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:50:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:50:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:50:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:35 smithi047 ceph-mon[36835]: pgmap v6212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:50:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:50:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:50:35.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:50:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:35 smithi029 ceph-mon[32610]: pgmap v6212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:50:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:50:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:50:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:50:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:36 smithi077 ceph-mon[37153]: pgmap v6213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:50:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:36 smithi077 ceph-mon[37153]: pgmap v6214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:50:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:36 smithi047 ceph-mon[36835]: pgmap v6213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:50:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:36 smithi047 ceph-mon[36835]: pgmap v6214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:50:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:36 smithi029 ceph-mon[32610]: pgmap v6213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:50:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:36 smithi029 ceph-mon[32610]: pgmap v6214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:50:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:37 smithi077 ceph-mon[37153]: pgmap v6215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:50:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:37 smithi047 ceph-mon[36835]: pgmap v6215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:50:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:37 smithi029 ceph-mon[32610]: pgmap v6215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:50:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10859.610067 secs 2024-07-27T09:50:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10859.610067 secs 2024-07-27T09:50:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10859.610067 secs 2024-07-27T09:50:39.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:39 smithi029 ceph-mon[32610]: pgmap v6216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:50:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:39 smithi077 ceph-mon[37153]: pgmap v6216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:50:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:39 smithi047 ceph-mon[36835]: pgmap v6216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:50:40.931 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-07-27T09:50:40.931 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-07-27T09:50:40.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-07-27T09:50:41.234 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi029 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-07-27T09:50:41.459 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8805102ca7804617a2e106ff440e0705 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi029 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-07-27T09:50:41.462 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-07-27T09:50:41.473 INFO:tasks.quiescer:joining Quiescers 2024-07-27T09:50:41.474 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-07-27T09:50:41.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- ceph fs dump --format=json 2024-07-27T09:50:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T09:50:42.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:41 smithi029 ceph-mon[32610]: pgmap v6217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:50:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:41 smithi077 ceph-mon[37153]: pgmap v6217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:50:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:41 smithi047 ceph-mon[36835]: pgmap v6217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:50:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10864.610195 secs 2024-07-27T09:50:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10864.610195 secs 2024-07-27T09:50:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10864.610195 secs 2024-07-27T09:50:43.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.623+0000 7f4e6ff38640 1 -- 172.21.15.29:0/2494954726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e68108720 msgr2=0x7f4e6810ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:43.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.623+0000 7f4e6ff38640 1 --2- 172.21.15.29:0/2494954726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e68108720 0x7f4e6810ef50 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4e60009ca0 tx=0x7f4e6002a8c0 comp rx=0 tx=0).stop 2024-07-27T09:50:43.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.623+0000 7f4e6ff38640 1 -- 172.21.15.29:0/2494954726 shutdown_connections 2024-07-27T09:50:43.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.623+0000 7f4e6ff38640 1 --2- 172.21.15.29:0/2494954726 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e68108720 0x7f4e6810ef50 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:43.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.623+0000 7f4e6ff38640 1 --2- 172.21.15.29:0/2494954726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e68107dd0 0x7f4e681081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:43.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.623+0000 7f4e6ff38640 1 --2- 172.21.15.29:0/2494954726 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e68102490 0x7f4e681028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:43.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.623+0000 7f4e6ff38640 1 -- 172.21.15.29:0/2494954726 >> 172.21.15.29:0/2494954726 conn(0x7f4e68079160 msgr2=0x7f4e68079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T09:50:43.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.623+0000 7f4e6ff38640 1 -- 172.21.15.29:0/2494954726 shutdown_connections 2024-07-27T09:50:43.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.623+0000 7f4e6ff38640 1 -- 172.21.15.29:0/2494954726 wait complete. 2024-07-27T09:50:43.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6ff38640 1 Processor -- start 2024-07-27T09:50:43.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6ff38640 1 -- start start 2024-07-27T09:50:43.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6ff38640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e68102490 0x7f4e6819ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:43.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6ff38640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e68107dd0 0x7f4e6819b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:43.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6ff38640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e68108720 0x7f4e6819b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:43.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6ff38640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4e68112410 con 0x7f4e68108720 2024-07-27T09:50:43.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6ff38640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7f4e68112290 con 0x7f4e68102490 2024-07-27T09:50:43.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6ff38640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f4e68112590 con 0x7f4e68107dd0 2024-07-27T09:50:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6d4d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e68107dd0 0x7f4e6819b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6d4d3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e68107dd0 0x7f4e6819b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.29:38154/0 (socket says 172.21.15.29:38154) 2024-07-27T09:50:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6d4d3640 1 -- 172.21.15.29:0/3246062613 learned_addr learned my addr 172.21.15.29:0/3246062613 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T09:50:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6dcd4640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e68102490 0x7f4e6819ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.624+0000 7f4e6e4d5640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e68108720 0x7f4e6819b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.625+0000 7f4e6e4d5640 1 -- 172.21.15.29:0/3246062613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e68107dd0 msgr2=0x7f4e6819b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.625+0000 7f4e6e4d5640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e68107dd0 0x7f4e6819b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.625+0000 7f4e6e4d5640 1 -- 172.21.15.29:0/3246062613 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e68102490 msgr2=0x7f4e6819ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.625+0000 7f4e6e4d5640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e68102490 0x7f4e6819ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.625+0000 7f4e6e4d5640 1 -- 172.21.15.29:0/3246062613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e681a5ff0 con 0x7f4e68108720 2024-07-27T09:50:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.625+0000 7f4e6d4d3640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e68107dd0 0x7f4e6819b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T09:50:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.625+0000 7f4e6e4d5640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e68108720 0x7f4e6819b7e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4e6002ade0 tx=0x7f4e600319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:50:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.625+0000 7f4e5effd640 1 -- 172.21.15.29:0/3246062613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e60003bf0 con 0x7f4e68108720 2024-07-27T09:50:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.625+0000 7f4e5effd640 1 -- 172.21.15.29:0/3246062613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e60003d90 con 0x7f4e68108720 2024-07-27T09:50:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.625+0000 7f4e6ff38640 1 -- 172.21.15.29:0/3246062613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e681a6280 con 0x7f4e68108720 2024-07-27T09:50:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.625+0000 7f4e5effd640 1 -- 172.21.15.29:0/3246062613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6002fbd0 con 0x7f4e68108720 2024-07-27T09:50:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.625+0000 7f4e6ff38640 1 -- 172.21.15.29:0/3246062613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e681a67c0 con 0x7f4e68108720 2024-07-27T09:50:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.626+0000 7f4e6ff38640 1 -- 172.21.15.29:0/3246062613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e681028a0 con 0x7f4e68108720 2024-07-27T09:50:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.628+0000 7f4e5effd640 1 -- 172.21.15.29:0/3246062613 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7f4e6002fd70 con 0x7f4e68108720 2024-07-27T09:50:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.629+0000 7f4e5effd640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4e540b3d70 0x7f4e540b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.629+0000 7f4e6dcd4640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4e540b3d70 0x7f4e540b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.629+0000 7f4e5effd640 1 -- 172.21.15.29:0/3246062613 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 10392+0+0 (secure 0 0 0) 0x7f4e60105b50 con 0x7f4e68108720 2024-07-27T09:50:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.629+0000 7f4e6dcd4640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4e540b3d70 0x7f4e540b6280 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4e6806b3f0 tx=0x7f4e6400b390 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:50:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.632+0000 7f4e5effd640 1 -- 172.21.15.29:0/3246062613 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4e600cc7c0 con 0x7f4e68108720 2024-07-27T09:50:43.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.854+0000 7f4e6ff38640 1 -- 172.21.15.29:0/3246062613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e6802e0c0 con 0x7f4e68108720 2024-07-27T09:50:43.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.854+0000 7f4e5effd640 1 -- 172.21.15.29:0/3246062613 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) ==== 95+0+9532 (secure 0 0 0) 0x7f4e600cc0c0 con 0x7f4e68108720 2024-07-27T09:50:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-07-27T09:50:43.857 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":179,"btime":"2024-07-27T06:56:02:498261+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":179,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-27T06:43:14.801207+0000","modified":"2024-07-27T06:56:02.498224+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":16512,"mds_1":24658,"mds_2":24646,"mds_3":24617,"mds_4":24613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14781":{"gid":14781,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6839/1639930501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6838","nonce":1639930501},{"type":"v1","addr":"172.21.15.29:6839","nonce":1639930501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14793":{"gid":14793,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.29:6841/2982668187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6840","nonce":2982668187},{"type":"v1","addr":"172.21.15.29:6841","nonce":2982668187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16512":{"gid":16512,"name":"j","rank":0,"incarnation":174,"state":"up:active","state_seq":5,"addr":"172.21.15.29:6835/1620428612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1620428612},{"type":"v1","addr":"172.21.15.29:6835","nonce":1620428612}]},"join_fscid":-1,"laggy_since":"2024-07-27T06:56:02.498224+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6833/2402830639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2402830639},{"type":"v1","addr":"172.21.15.47:6833","nonce":2402830639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24566":{"gid":24566,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.47:6835/2900591116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6834","nonce":2900591116},{"type":"v1","addr":"172.21.15.47:6835","nonce":2900591116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24613":{"gid":24613,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":19,"addr":"172.21.15.77:6833/1085922080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":1085922080},{"type":"v1","addr":"172.21.15.77:6833","nonce":1085922080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24617":{"gid":24617,"name":"b","rank":3,"incarnation":23,"state":"up:active","state_seq":10,"addr":"172.21.15.47:6839/670107266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6838","nonce":670107266},{"type":"v1","addr":"172.21.15.47:6839","nonce":670107266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24646":{"gid":24646,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.77:6835/439792826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6834","nonce":439792826},{"type":"v1","addr":"172.21.15.77:6835","nonce":439792826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24658":{"gid":24658,"name":"c","rank":1,"incarnation":19,"state":"up:active","state_seq":152,"addr":"172.21.15.77:6837/2959628937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2959628937},{"type":"v1","addr":"172.21.15.77:6837","nonce":2959628937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":16512,"qdb_cluster":[24658,24646,24617,24613,16512]},"id":1}]} 2024-07-27T09:50:43.857 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 179 2024-07-27T09:50:43.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.856+0000 7f4e6ff38640 1 -- 172.21.15.29:0/3246062613 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4e540b3d70 msgr2=0x7f4e540b6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:43.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.856+0000 7f4e6ff38640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4e540b3d70 0x7f4e540b6280 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4e6806b3f0 tx=0x7f4e6400b390 comp rx=0 tx=0).stop 2024-07-27T09:50:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.856+0000 7f4e6ff38640 1 -- 172.21.15.29:0/3246062613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e68108720 msgr2=0x7f4e6819b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.856+0000 7f4e6ff38640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e68108720 0x7f4e6819b7e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4e6002ade0 tx=0x7f4e600319f0 comp rx=0 tx=0).stop 2024-07-27T09:50:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.856+0000 7f4e6ff38640 1 -- 172.21.15.29:0/3246062613 shutdown_connections 2024-07-27T09:50:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.856+0000 7f4e6ff38640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e68108720 0x7f4e6819b7e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.856+0000 7f4e6ff38640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7f4e540b3d70 0x7f4e540b6280 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.856+0000 7f4e6ff38640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e68107dd0 0x7f4e6819b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.856+0000 7f4e6ff38640 1 --2- 172.21.15.29:0/3246062613 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e68102490 0x7f4e6819ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.856+0000 7f4e6ff38640 1 -- 172.21.15.29:0/3246062613 >> 172.21.15.29:0/3246062613 conn(0x7f4e68079160 msgr2=0x7f4e68104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T09:50:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.856+0000 7f4e6ff38640 1 -- 172.21.15.29:0/3246062613 shutdown_connections 2024-07-27T09:50:43.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:43.857+0000 7f4e6ff38640 1 -- 172.21.15.29:0/3246062613 wait complete. 2024-07-27T09:50:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:43 smithi077 ceph-mon[37153]: pgmap v6218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:50:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:43 smithi047 ceph-mon[36835]: pgmap v6218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:50:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:43 smithi029 ceph-mon[32610]: pgmap v6218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:50:44.706 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.16512 quiesce db --cancel --all' 2024-07-27T09:50:44.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 0a8a0a4c-4be2-11ef-bcc7-c7b262605968 -- timeout 105 ceph tell mds.16512 quiesce db --cancel --all 2024-07-27T09:50:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:44 smithi029 ceph-mon[32610]: from='client.? 172.21.15.29:0/3246062613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T09:50:44.979 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/0a8a0a4c-4be2-11ef-bcc7-c7b262605968/mon.a/config 2024-07-27T09:50:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:44 smithi077 ceph-mon[37153]: from='client.? 172.21.15.29:0/3246062613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T09:50:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:44 smithi047 ceph-mon[36835]: from='client.? 172.21.15.29:0/3246062613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-27T09:50:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:45 smithi077 ceph-mon[37153]: pgmap v6219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:50:46.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:45 smithi029 ceph-mon[32610]: pgmap v6219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:50:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:45 smithi047 ceph-mon[36835]: pgmap v6219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:50:46.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.587+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:50:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.587+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:50:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.587+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:50:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:50:46.601+0000 7f7347fff640 1 -- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 msgr2=0x5645ed7222d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.77:6838/269899139 2024-07-27T09:50:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:50:46.601+0000 7f7347fff640 1 --2- 172.21.15.29:0/940768653 >> [v2:172.21.15.77:6838/269899139,v1:172.21.15.77:6839/269899139] conn(0x5645ed320d80 0x5645ed7222d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:50:46.603 INFO:teuthology.task.background_exec.ubuntu@smithi029.front.sepia.ceph.com.smithi029.stderr:2024-07-27T09:50:46.601+0000 7f7327fff640 1 client.16296 ms_handle_refused on v2:172.21.15.77:6838/269899139 2024-07-27T09:50:46.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.670+0000 7fbafa2cd640 1 -- 172.21.15.29:0/1265317473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf4108720 msgr2=0x7fbaf410ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:46.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.670+0000 7fbafa2cd640 1 --2- 172.21.15.29:0/1265317473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf4108720 0x7fbaf410ef50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fbaec009ca0 tx=0x7fbaec02a8c0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.670+0000 7fbafa2cd640 1 -- 172.21.15.29:0/1265317473 shutdown_connections 2024-07-27T09:50:46.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.670+0000 7fbafa2cd640 1 --2- 172.21.15.29:0/1265317473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf4108720 0x7fbaf410ef50 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.670+0000 7fbafa2cd640 1 --2- 172.21.15.29:0/1265317473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaf4107dd0 0x7fbaf41081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.670+0000 7fbafa2cd640 1 --2- 172.21.15.29:0/1265317473 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbaf4102490 0x7fbaf41028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.670+0000 7fbafa2cd640 1 -- 172.21.15.29:0/1265317473 >> 172.21.15.29:0/1265317473 conn(0x7fbaf4079160 msgr2=0x7fbaf4079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T09:50:46.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.670+0000 7fbafa2cd640 1 -- 172.21.15.29:0/1265317473 shutdown_connections 2024-07-27T09:50:46.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.670+0000 7fbafa2cd640 1 -- 172.21.15.29:0/1265317473 wait complete. 2024-07-27T09:50:46.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.671+0000 7fbafa2cd640 1 Processor -- start 2024-07-27T09:50:46.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.671+0000 7fbafa2cd640 1 -- start start 2024-07-27T09:50:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.671+0000 7fbafa2cd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaf4102490 0x7fbaf4199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.671+0000 7fbafa2cd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf4107dd0 0x7fbaf419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.671+0000 7fbafa2cd640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbaf4108720 0x7fbaf419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.671+0000 7fbafa2cd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf41123a0 con 0x7fbaf4107dd0 2024-07-27T09:50:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.671+0000 7fbafa2cd640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf4112220 con 0x7fbaf4108720 2024-07-27T09:50:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.671+0000 7fbafa2cd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf4112520 con 0x7fbaf4102490 2024-07-27T09:50:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.671+0000 7fbaebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf4107dd0 0x7fbaf419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf4107dd0 0x7fbaf419a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45740/0 (socket says 172.21.15.29:45740) 2024-07-27T09:50:46.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaebfff640 1 -- 172.21.15.29:0/2907606133 learned_addr learned my addr 172.21.15.29:0/2907606133 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T09:50:46.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaf886a640 1 --2- 172.21.15.29:0/2907606133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbaf4108720 0x7fbaf419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:46.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaf3fff640 1 --2- 172.21.15.29:0/2907606133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaf4102490 0x7fbaf4199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:46.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaebfff640 1 -- 172.21.15.29:0/2907606133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaf4102490 msgr2=0x7fbaf4199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:46.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaebfff640 1 --2- 172.21.15.29:0/2907606133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbaf4102490 0x7fbaf4199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaebfff640 1 -- 172.21.15.29:0/2907606133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbaf4108720 msgr2=0x7fbaf419f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:46.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaebfff640 1 --2- 172.21.15.29:0/2907606133 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbaf4108720 0x7fbaf419f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaebfff640 1 -- 172.21.15.29:0/2907606133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf419fdf0 con 0x7fbaf4107dd0 2024-07-27T09:50:46.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaebfff640 1 --2- 172.21.15.29:0/2907606133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbaf4107dd0 0x7fbaf419a420 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00e9f0 tx=0x7fbadc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:50:46.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaf1ffb640 1 -- 172.21.15.29:0/2907606133 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc00ccc0 con 0x7fbaf4107dd0 2024-07-27T09:50:46.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaf1ffb640 1 -- 172.21.15.29:0/2907606133 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbadc0054d0 con 0x7fbaf4107dd0 2024-07-27T09:50:46.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbafa2cd640 1 -- 172.21.15.29:0/2907606133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaf41a00e0 con 0x7fbaf4107dd0 2024-07-27T09:50:46.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbaf1ffb640 1 -- 172.21.15.29:0/2907606133 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc010430 con 0x7fbaf4107dd0 2024-07-27T09:50:46.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.672+0000 7fbafa2cd640 1 -- 172.21.15.29:0/2907606133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaf41a6840 con 0x7fbaf4107dd0 2024-07-27T09:50:46.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbb00851740 1 Processor -- start 2024-07-27T09:50:46.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbb00851740 1 -- start start 2024-07-27T09:50:46.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbaf1ffb640 1 -- 172.21.15.29:0/2907606133 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fbadc0046c0 con 0x7fbaf4107dd0 2024-07-27T09:50:46.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbb00851740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 0x556346123970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:46.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbb00851740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556345d55410 0x556346123eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:46.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbb00851740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556345d5d5f0 0x5563461243f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:46.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbb00851740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556346099a80 con 0x556345d7e2d0 2024-07-27T09:50:46.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbb00851740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x556346088f80 con 0x556345d5d5f0 2024-07-27T09:50:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbb00851740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55634608cfc0 con 0x556345d55410 2024-07-27T09:50:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbaf886a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 0x556346123970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbaf886a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 0x556346123970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45754/0 (socket says 172.21.15.29:45754) 2024-07-27T09:50:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbaf886a640 1 -- 172.21.15.29:0/3163168994 learned_addr learned my addr 172.21.15.29:0/3163168994 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T09:50:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.675+0000 7fbaf3fff640 1 --2- 172.21.15.29:0/3163168994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556345d55410 0x556346123eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaebfff640 1 --2- 172.21.15.29:0/3163168994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556345d5d5f0 0x5563461243f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaf886a640 1 -- 172.21.15.29:0/3163168994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556345d55410 msgr2=0x556346123eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T09:50:46.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaf886a640 1 --2- 172.21.15.29:0/3163168994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556345d55410 0x556346123eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaf886a640 1 -- 172.21.15.29:0/3163168994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556345d5d5f0 msgr2=0x5563461243f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T09:50:46.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaf886a640 1 --2- 172.21.15.29:0/3163168994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556345d5d5f0 0x5563461243f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaf886a640 1 -- 172.21.15.29:0/3163168994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563460c3400 con 0x556345d7e2d0 2024-07-27T09:50:46.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaf3fff640 1 --2- 172.21.15.29:0/3163168994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556345d55410 0x556346123eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-27T09:50:46.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaf886a640 1 --2- 172.21.15.29:0/3163168994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 0x556346123970 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fbaec02f410 tx=0x7fbaec02f440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c5cd458d3bb5da server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:50:46.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaf1ffb640 1 --2- 172.21.15.29:0/2907606133 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbae00b3dc0 0x7fbae00b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:46.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaea7fc640 1 -- 172.21.15.29:0/3163168994 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec031cf0 con 0x556345d7e2d0 2024-07-27T09:50:46.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaf3fff640 1 --2- 172.21.15.29:0/2907606133 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbae00b3dc0 0x7fbae00b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:46.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaea7fc640 1 -- 172.21.15.29:0/3163168994 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaec036030 con 0x556345d7e2d0 2024-07-27T09:50:46.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaf1ffb640 1 -- 172.21.15.29:0/2907606133 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 10392+0+0 (secure 0 0 0) 0x7fbadc0e4ad0 con 0x7fbaf4107dd0 2024-07-27T09:50:46.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.676+0000 7fbaea7fc640 1 -- 172.21.15.29:0/3163168994 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec0042a0 con 0x556345d7e2d0 2024-07-27T09:50:46.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.677+0000 7fbb00851740 1 -- 172.21.15.29:0/3163168994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 msgr2=0x556346123970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:46.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.677+0000 7fbaf3fff640 1 --2- 172.21.15.29:0/2907606133 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fbae00b3dc0 0x7fbae00b62d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x5563461225f0 tx=0x7fbae4005f90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:50:46.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.677+0000 7fbb00851740 1 --2- 172.21.15.29:0/3163168994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 0x556346123970 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fbaec02f410 tx=0x7fbaec02f440 comp rx=0 tx=0).stop 2024-07-27T09:50:46.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.677+0000 7fbb00851740 1 -- 172.21.15.29:0/3163168994 shutdown_connections 2024-07-27T09:50:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.677+0000 7fbb00851740 1 --2- 172.21.15.29:0/3163168994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556345d5d5f0 0x5563461243f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.677+0000 7fbb00851740 1 --2- 172.21.15.29:0/3163168994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556345d55410 0x556346123eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.677+0000 7fbb00851740 1 --2- 172.21.15.29:0/3163168994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 0x556346123970 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.677+0000 7fbb00851740 1 -- 172.21.15.29:0/3163168994 >> 172.21.15.29:0/3163168994 conn(0x556345d726c0 msgr2=0x556346121860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T09:50:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.677+0000 7fbb00851740 1 -- 172.21.15.29:0/3163168994 shutdown_connections 2024-07-27T09:50:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.677+0000 7fbb00851740 1 -- 172.21.15.29:0/3163168994 wait complete. 2024-07-27T09:50:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.678+0000 7fbb00851740 1 -- start start 2024-07-27T09:50:46.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.678+0000 7fbb00851740 1 Processor -- start 2024-07-27T09:50:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.678+0000 7fbb00851740 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556345d55410 0x55634612f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.678+0000 7fbb00851740 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556345d5d5f0 0x55634612fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.678+0000 7fbb00851740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 0x55634613b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.678+0000 7fbb00851740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x556345eec810 con 0x556345d7e2d0 2024-07-27T09:50:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.678+0000 7fbb00851740 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x55634602b960 con 0x556345d5d5f0 2024-07-27T09:50:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.678+0000 7fbb00851740 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556346131f70 con 0x556345d55410 2024-07-27T09:50:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbb00851740 10 client.0 _ll_register_callbacks cb 0x556346033500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-27T09:50:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbaf886a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 0x55634613b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:46.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbaf886a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 0x55634613b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45770/0 (socket says 172.21.15.29:45770) 2024-07-27T09:50:46.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbaf886a640 1 -- 172.21.15.29:0/671423452 learned_addr learned my addr 172.21.15.29:0/671423452 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T09:50:46.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbaebfff640 1 --2- 172.21.15.29:0/671423452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556345d5d5f0 0x55634612fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:46.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbaf3fff640 1 --2- 172.21.15.29:0/671423452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556345d55410 0x55634612f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:46.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbaf886a640 1 -- 172.21.15.29:0/671423452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556345d55410 msgr2=0x55634612f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T09:50:46.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbaf886a640 1 --2- 172.21.15.29:0/671423452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556345d55410 0x55634612f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbaf886a640 1 -- 172.21.15.29:0/671423452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556345d5d5f0 msgr2=0x55634612fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-27T09:50:46.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbaf886a640 1 --2- 172.21.15.29:0/671423452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556345d5d5f0 0x55634612fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbaf886a640 1 -- 172.21.15.29:0/671423452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55634602f9b0 con 0x556345d7e2d0 2024-07-27T09:50:46.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbaf886a640 1 --2- 172.21.15.29:0/671423452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 0x55634613b290 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbaec005e30 tx=0x7fbaec003c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f524bc89a91ca9d6 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:50:46.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbad37fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-07-27T09:50:46.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbad37fe640 1 -- 172.21.15.29:0/671423452 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec04a040 con 0x556345d7e2d0 2024-07-27T09:50:46.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.679+0000 7fbad37fe640 1 -- 172.21.15.29:0/671423452 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 10392+0+0 (secure 0 0 0) 0x7fbaec031650 con 0x556345d7e2d0 2024-07-27T09:50:46.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.680+0000 7fbb00851740 1 -- 172.21.15.29:0/671423452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbaec036030 con 0x556345d7e2d0 2024-07-27T09:50:46.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.680+0000 7fbad37fe640 1 -- 172.21.15.29:0/671423452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=179) ==== 24+0+0 (secure 0 0 0) 0x7fbaec05e050 con 0x556345d7e2d0 2024-07-27T09:50:46.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.680+0000 7fbb00851740 10 client.16530 fetch_fsmap learned FSMap version 179 2024-07-27T09:50:46.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.680+0000 7fbb00851740 1 -- 172.21.15.29:0/671423452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=179,monmap=4+}) -- 0x556345de98b0 con 0x556345d7e2d0 2024-07-27T09:50:46.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.680+0000 7fbad37fe640 1 -- 172.21.15.29:0/671423452 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaec036030 con 0x556345d7e2d0 2024-07-27T09:50:46.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.680+0000 7fbad37fe640 1 -- 172.21.15.29:0/671423452 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 179) ==== 6559+0+0 (secure 0 0 0) 0x7fbaec04d030 con 0x556345d7e2d0 2024-07-27T09:50:46.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.680+0000 7fbb00851740 10 client.16530 fetch_fsmap finished waiting for FSMap version 179 2024-07-27T09:50:46.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.680+0000 7fbb00851740 10 client.16530 resolve_mds: validated gid 16512 aka daemon mds.j 2024-07-27T09:50:46.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.680+0000 7fbb00851740 1 client.16530 shutdown 2024-07-27T09:50:46.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.681+0000 7fbb00851740 1 -- 172.21.15.29:0/671423452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 msgr2=0x55634613b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:46.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.681+0000 7fbb00851740 1 --2- 172.21.15.29:0/671423452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 0x55634613b290 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbaec005e30 tx=0x7fbaec003c40 comp rx=0 tx=0).stop 2024-07-27T09:50:46.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.681+0000 7fbb00851740 1 -- 172.21.15.29:0/671423452 shutdown_connections 2024-07-27T09:50:46.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.681+0000 7fbb00851740 1 --2- 172.21.15.29:0/671423452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556345d7e2d0 0x55634613b290 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.681+0000 7fbb00851740 1 --2- 172.21.15.29:0/671423452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x556345d5d5f0 0x55634612fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.681+0000 7fbb00851740 1 --2- 172.21.15.29:0/671423452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556345d55410 0x55634612f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:46.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.681+0000 7fbb00851740 1 -- 172.21.15.29:0/671423452 >> 172.21.15.29:0/671423452 conn(0x556345d726c0 msgr2=0x556346126b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T09:50:46.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.681+0000 7fbb00851740 1 -- 172.21.15.29:0/671423452 shutdown_connections 2024-07-27T09:50:46.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.681+0000 7fbb00851740 1 -- 172.21.15.29:0/671423452 wait complete. 2024-07-27T09:50:46.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:46.681+0000 7fbb00851740 20 client.16530 trim_cache size 0 max 16384 2024-07-27T09:50:46.698 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: problem getting command descriptions from mds.16512 2024-07-27T09:50:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-07-27T09:50:47.477 ERROR:teuthology.run_tasks:Manager failed: quiescer Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 422, in task stop_all_quiescers(ctx.ceph[cluster_name].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 381, in stop_all_quiescers raise RuntimeError(f"error during quiesce thrashing: {thrasher.exception}") RuntimeError: error during quiesce thrashing: Error parsing quiesce response: Expecting value: line 1 column 1 (char 0) 2024-07-27T09:50:47.508 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-07-27T09:50:47.519 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi029.front.sepia.ceph.com 2024-07-27T09:50:47.528 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-07-27T09:50:47.540 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-07-27T09:50:47.540 INFO:teuthology.task.exec:Executing custom commands... 2024-07-27T09:50:47.540 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-07-27T09:50:47.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-07-27T09:50:47.600 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-07-27T09:50:47.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-07-27T09:50:47.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.827+0000 7fe7c8365640 1 -- 172.21.15.29:0/620149708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7c0108670 msgr2=0x7fe7c010eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:47.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.827+0000 7fe7c8365640 1 --2- 172.21.15.29:0/620149708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7c0108670 0x7fe7c010eea0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8009ca0 tx=0x7fe7b802a8c0 comp rx=0 tx=0).stop 2024-07-27T09:50:47.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.827+0000 7fe7c8365640 1 -- 172.21.15.29:0/620149708 shutdown_connections 2024-07-27T09:50:47.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.827+0000 7fe7c8365640 1 --2- 172.21.15.29:0/620149708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7c0108670 0x7fe7c010eea0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:47.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.827+0000 7fe7c8365640 1 --2- 172.21.15.29:0/620149708 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c0107d20 0x7fe7c0108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:47.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.827+0000 7fe7c8365640 1 --2- 172.21.15.29:0/620149708 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c01023f0 0x7fe7c0102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.827+0000 7fe7c8365640 1 -- 172.21.15.29:0/620149708 >> 172.21.15.29:0/620149708 conn(0x7fe7c00fddf0 msgr2=0x7fe7c0100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-27T09:50:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.827+0000 7fe7c8365640 1 -- 172.21.15.29:0/620149708 shutdown_connections 2024-07-27T09:50:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.827+0000 7fe7c8365640 1 -- 172.21.15.29:0/620149708 wait complete. 2024-07-27T09:50:47.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c8365640 1 Processor -- start 2024-07-27T09:50:47.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c8365640 1 -- start start 2024-07-27T09:50:47.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c8365640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7c01023f0 0x7fe7c019ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:47.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c8365640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c0107d20 0x7fe7c019b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:47.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c8365640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c0108670 0x7fe7c019b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:47.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c8365640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c01123e0 con 0x7fe7c01023f0 2024-07-27T09:50:47.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c8365640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c0112260 con 0x7fe7c0107d20 2024-07-27T09:50:47.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c8365640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c0112560 con 0x7fe7c0108670 2024-07-27T09:50:47.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c6101640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7c01023f0 0x7fe7c019ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:47.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c6101640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7c01023f0 0x7fe7c019ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45792/0 (socket says 172.21.15.29:45792) 2024-07-27T09:50:47.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c6101640 1 -- 172.21.15.29:0/827749814 learned_addr learned my addr 172.21.15.29:0/827749814 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-07-27T09:50:47.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c5900640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c0107d20 0x7fe7c019b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:47.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c6902640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c0108670 0x7fe7c019b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:47.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c6101640 1 -- 172.21.15.29:0/827749814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c0108670 msgr2=0x7fe7c019b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:47.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c6101640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe7c0108670 0x7fe7c019b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:47.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c6101640 1 -- 172.21.15.29:0/827749814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c0107d20 msgr2=0x7fe7c019b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-27T09:50:47.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c6101640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c0107d20 0x7fe7c019b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T09:50:47.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c6101640 1 -- 172.21.15.29:0/827749814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c01a5fd0 con 0x7fe7c01023f0 2024-07-27T09:50:47.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.828+0000 7fe7c5900640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe7c0107d20 0x7fe7c019b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-27T09:50:47.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.829+0000 7fe7c6101640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7c01023f0 0x7fe7c019ad10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc00b4a0 tx=0x7fe7bc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:50:47.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.829+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc014070 con 0x7fe7c01023f0 2024-07-27T09:50:47.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.829+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7bc010ac0 con 0x7fe7c01023f0 2024-07-27T09:50:47.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.829+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc017c30 con 0x7fe7c01023f0 2024-07-27T09:50:47.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.829+0000 7fe7c8365640 1 -- 172.21.15.29:0/827749814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7c01a62c0 con 0x7fe7c01023f0 2024-07-27T09:50:47.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.829+0000 7fe7c8365640 1 -- 172.21.15.29:0/827749814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7c01a6810 con 0x7fe7c01023f0 2024-07-27T09:50:47.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.830+0000 7fe7c8365640 1 -- 172.21.15.29:0/827749814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7c0102800 con 0x7fe7c01023f0 2024-07-27T09:50:47.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.831+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) ==== 151190+0+0 (secure 0 0 0) 0x7fe7bc017070 con 0x7fe7c01023f0 2024-07-27T09:50:47.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.832+0000 7fe7b77fe640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe7ac0b3d40 0x7fe7ac0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T09:50:47.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.832+0000 7fe7c5900640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe7ac0b3d40 0x7fe7ac0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T09:50:47.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.832+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 10392+0+0 (secure 0 0 0) 0x7fe7bc0e3e40 con 0x7fe7c01023f0 2024-07-27T09:50:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.833+0000 7fe7c5900640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe7ac0b3d40 0x7fe7ac0b6250 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0101640 tx=0x7fe7b0005e50 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T09:50:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:47.835+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe7bc0aaab0 con 0x7fe7c01023f0 2024-07-27T09:50:48.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:50:48.050+0000 7fe7c8365640 1 -- 172.21.15.29:0/827749814 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fe7c0063230 con 0x7fe7ac0b3d40 2024-07-27T09:50:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:47 smithi077 ceph-mon[37153]: pgmap v6220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10869.610309 secs 2024-07-27T09:50:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:47 smithi047 ceph-mon[36835]: pgmap v6220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10869.610309 secs 2024-07-27T09:50:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:47 smithi029 ceph-mon[32610]: pgmap v6220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10869.610309 secs 2024-07-27T09:50:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:48 smithi077 ceph-mon[37153]: from='client.16536 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T09:50:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:48 smithi047 ceph-mon[36835]: from='client.16536 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T09:50:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:48 smithi029 ceph-mon[32610]: from='client.16536 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T09:50:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:49 smithi077 ceph-mon[37153]: pgmap v6221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:49 smithi047 ceph-mon[36835]: pgmap v6221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:49 smithi029 ceph-mon[32610]: pgmap v6221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:50:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:51 smithi077 ceph-mon[37153]: pgmap v6222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:52.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:51 smithi047 ceph-mon[36835]: pgmap v6222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:51 smithi029 ceph-mon[32610]: pgmap v6222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:50:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10874.610429 secs 2024-07-27T09:50:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10874.610429 secs 2024-07-27T09:50:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10874.610429 secs 2024-07-27T09:50:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:53 smithi077 ceph-mon[37153]: pgmap v6223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:50:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:53 smithi047 ceph-mon[36835]: pgmap v6223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:50:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:53 smithi029 ceph-mon[32610]: pgmap v6223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:50:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:55 smithi077 ceph-mon[37153]: pgmap v6224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:50:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:55 smithi047 ceph-mon[36835]: pgmap v6224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:50:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:55 smithi029 ceph-mon[32610]: pgmap v6224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:50:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:57 smithi077 ceph-mon[37153]: pgmap v6225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10879.610558 secs 2024-07-27T09:50:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:57 smithi047 ceph-mon[36835]: pgmap v6225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10879.610558 secs 2024-07-27T09:50:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:57 smithi029 ceph-mon[32610]: pgmap v6225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:50:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10879.610558 secs 2024-07-27T09:51:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:50:59 smithi077 ceph-mon[37153]: pgmap v6226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:50:59 smithi047 ceph-mon[36835]: pgmap v6226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:50:59 smithi029 ceph-mon[32610]: pgmap v6226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:01.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:01.602+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:51:01.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:01.602+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:51:01.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:01.602+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:51:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:01 smithi029 ceph-mon[32610]: pgmap v6227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:02.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:01 smithi077 ceph-mon[37153]: pgmap v6227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:01 smithi047 ceph-mon[36835]: pgmap v6227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10884.610682 secs 2024-07-27T09:51:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10884.610682 secs 2024-07-27T09:51:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10884.610682 secs 2024-07-27T09:51:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:03 smithi077 ceph-mon[37153]: pgmap v6228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:51:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:03 smithi047 ceph-mon[36835]: pgmap v6228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:51:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:03 smithi029 ceph-mon[32610]: pgmap v6228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:51:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:05 smithi077 ceph-mon[37153]: pgmap v6229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:05 smithi047 ceph-mon[36835]: pgmap v6229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:05 smithi029 ceph-mon[32610]: pgmap v6229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:07 smithi077 ceph-mon[37153]: pgmap v6230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10889.610834 secs 2024-07-27T09:51:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:07 smithi047 ceph-mon[36835]: pgmap v6230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10889.610834 secs 2024-07-27T09:51:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:07 smithi029 ceph-mon[32610]: pgmap v6230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10889.610834 secs 2024-07-27T09:51:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:09 smithi077 ceph-mon[37153]: pgmap v6231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:09 smithi047 ceph-mon[36835]: pgmap v6231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:09 smithi029 ceph-mon[32610]: pgmap v6231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:11 smithi077 ceph-mon[37153]: pgmap v6232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:11 smithi047 ceph-mon[36835]: pgmap v6232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:11 smithi029 ceph-mon[32610]: pgmap v6232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10894.610962 secs 2024-07-27T09:51:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10894.610962 secs 2024-07-27T09:51:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10894.610962 secs 2024-07-27T09:51:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:13 smithi077 ceph-mon[37153]: pgmap v6233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:51:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:13 smithi047 ceph-mon[36835]: pgmap v6233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:51:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:13 smithi029 ceph-mon[32610]: pgmap v6233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:51:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:15 smithi077 ceph-mon[37153]: pgmap v6234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:15 smithi047 ceph-mon[36835]: pgmap v6234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:15 smithi029 ceph-mon[32610]: pgmap v6234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:16.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:16.616+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:51:16.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:16.616+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:51:16.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:16.616+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:51:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:17 smithi077 ceph-mon[37153]: pgmap v6235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10899.611077 secs 2024-07-27T09:51:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:17 smithi047 ceph-mon[36835]: pgmap v6235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:18.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10899.611077 secs 2024-07-27T09:51:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:17 smithi029 ceph-mon[32610]: pgmap v6235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10899.611077 secs 2024-07-27T09:51:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:19 smithi077 ceph-mon[37153]: pgmap v6236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:19 smithi047 ceph-mon[36835]: pgmap v6236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:20.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:19 smithi029 ceph-mon[32610]: pgmap v6236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:21 smithi077 ceph-mon[37153]: pgmap v6237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:51:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:21 smithi047 ceph-mon[36835]: pgmap v6237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:51:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:21 smithi029 ceph-mon[32610]: pgmap v6237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:51:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10904.611205 secs 2024-07-27T09:51:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10904.611205 secs 2024-07-27T09:51:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10904.611205 secs 2024-07-27T09:51:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:23 smithi047 ceph-mon[36835]: pgmap v6238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:51:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:23 smithi029 ceph-mon[32610]: pgmap v6238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:51:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:23 smithi077 ceph-mon[37153]: pgmap v6238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:51:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:25 smithi047 ceph-mon[36835]: pgmap v6239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:25 smithi029 ceph-mon[32610]: pgmap v6239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:25 smithi077 ceph-mon[37153]: pgmap v6239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:27 smithi047 ceph-mon[36835]: pgmap v6240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10909.611306 secs 2024-07-27T09:51:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:27 smithi029 ceph-mon[32610]: pgmap v6240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10909.611306 secs 2024-07-27T09:51:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:27 smithi077 ceph-mon[37153]: pgmap v6240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10909.611306 secs 2024-07-27T09:51:29.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:29.285+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:51:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:29.285+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:51:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:29.285+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:51:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:29.285+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:51:30.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:29 smithi077 ceph-mon[37153]: pgmap v6241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:29 smithi047 ceph-mon[36835]: pgmap v6241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:29 smithi029 ceph-mon[32610]: pgmap v6241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:31.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:31.631+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:51:31.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:31.631+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:51:31.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:31.631+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:51:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:31 smithi029 ceph-mon[32610]: pgmap v6242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:51:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:31 smithi047 ceph-mon[36835]: pgmap v6242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:51:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:31 smithi077 ceph-mon[37153]: pgmap v6242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:51:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10914.611432 secs 2024-07-27T09:51:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10914.611432 secs 2024-07-27T09:51:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10914.611432 secs 2024-07-27T09:51:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:33 smithi047 ceph-mon[36835]: pgmap v6243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:51:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:33 smithi029 ceph-mon[32610]: pgmap v6243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:51:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:33 smithi077 ceph-mon[37153]: pgmap v6243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:51:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:51:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:51:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:51:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:35 smithi047 ceph-mon[36835]: pgmap v6244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:51:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:51:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:35 smithi047 ceph-mon[36835]: pgmap v6245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:51:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:35 smithi047 ceph-mon[36835]: pgmap v6246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:51:36.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:51:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:35 smithi029 ceph-mon[32610]: pgmap v6244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:51:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:51:36.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:35 smithi029 ceph-mon[32610]: pgmap v6245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:51:36.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:35 smithi029 ceph-mon[32610]: pgmap v6246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:51:36.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:51:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:35 smithi077 ceph-mon[37153]: pgmap v6244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:51:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:51:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:51:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:35 smithi077 ceph-mon[37153]: pgmap v6245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:51:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:35 smithi077 ceph-mon[37153]: pgmap v6246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:51:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:51:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:37 smithi077 ceph-mon[37153]: pgmap v6247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:51:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:37 smithi047 ceph-mon[36835]: pgmap v6247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:51:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:37 smithi029 ceph-mon[32610]: pgmap v6247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:51:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10919.611580 secs 2024-07-27T09:51:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10919.611580 secs 2024-07-27T09:51:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10919.611580 secs 2024-07-27T09:51:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:39 smithi077 ceph-mon[37153]: pgmap v6248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:51:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:39 smithi047 ceph-mon[36835]: pgmap v6248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:51:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:39 smithi029 ceph-mon[32610]: pgmap v6248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:51:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:41 smithi077 ceph-mon[37153]: pgmap v6249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:51:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:41 smithi047 ceph-mon[36835]: pgmap v6249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:51:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:41 smithi029 ceph-mon[32610]: pgmap v6249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:51:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10924.611706 secs 2024-07-27T09:51:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10924.611706 secs 2024-07-27T09:51:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10924.611706 secs 2024-07-27T09:51:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:43 smithi047 ceph-mon[36835]: pgmap v6250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:51:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:43 smithi029 ceph-mon[32610]: pgmap v6250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:51:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:43 smithi077 ceph-mon[37153]: pgmap v6250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:51:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:45 smithi029 ceph-mon[32610]: pgmap v6251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:51:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:45 smithi077 ceph-mon[37153]: pgmap v6251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:51:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:45 smithi047 ceph-mon[36835]: pgmap v6251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:51:46.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:46.647+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:51:46.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:46.647+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:51:46.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:51:46.647+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:51:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:47 smithi077 ceph-mon[37153]: pgmap v6252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:47 smithi047 ceph-mon[36835]: pgmap v6252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:47 smithi029 ceph-mon[32610]: pgmap v6252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10929.611856 secs 2024-07-27T09:51:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10929.611856 secs 2024-07-27T09:51:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10929.611856 secs 2024-07-27T09:51:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:49 smithi077 ceph-mon[37153]: pgmap v6253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:51:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:49 smithi047 ceph-mon[36835]: pgmap v6253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:51:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:49 smithi029 ceph-mon[32610]: pgmap v6253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:51:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:51 smithi077 ceph-mon[37153]: pgmap v6254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:51 smithi047 ceph-mon[36835]: pgmap v6254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:51 smithi029 ceph-mon[32610]: pgmap v6254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10934.611978 secs 2024-07-27T09:51:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10934.611978 secs 2024-07-27T09:51:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10934.611978 secs 2024-07-27T09:51:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:53 smithi047 ceph-mon[36835]: pgmap v6255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:53 smithi029 ceph-mon[32610]: pgmap v6255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:53 smithi077 ceph-mon[37153]: pgmap v6255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:51:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:55 smithi077 ceph-mon[37153]: pgmap v6256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:51:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:55 smithi047 ceph-mon[36835]: pgmap v6256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:51:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:55 smithi029 ceph-mon[32610]: pgmap v6256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:51:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:57 smithi077 ceph-mon[37153]: pgmap v6257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:57 smithi047 ceph-mon[36835]: pgmap v6257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:57 smithi029 ceph-mon[32610]: pgmap v6257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:51:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10939.612059 secs 2024-07-27T09:51:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10939.612059 secs 2024-07-27T09:51:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10939.612059 secs 2024-07-27T09:51:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:51:59 smithi077 ceph-mon[37153]: pgmap v6258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:51:59.612 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:51:59 smithi047 ceph-mon[36835]: pgmap v6258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:51:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:51:59 smithi029 ceph-mon[32610]: pgmap v6258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:52:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:01 smithi077 ceph-mon[37153]: pgmap v6259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:01.663+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:01.663+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:52:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:01.663+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:52:01.665 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:01 smithi029 ceph-mon[32610]: pgmap v6259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:01 smithi047 ceph-mon[36835]: pgmap v6259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10944.612124 secs 2024-07-27T09:52:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10944.612124 secs 2024-07-27T09:52:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10944.612124 secs 2024-07-27T09:52:03.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:03 smithi047 ceph-mon[36835]: pgmap v6260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:03 smithi029 ceph-mon[32610]: pgmap v6260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:03 smithi077 ceph-mon[37153]: pgmap v6260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:05 smithi077 ceph-mon[37153]: pgmap v6261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:05 smithi047 ceph-mon[36835]: pgmap v6261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:05 smithi029 ceph-mon[32610]: pgmap v6261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10949.612208 secs 2024-07-27T09:52:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:07 smithi047 ceph-mon[36835]: pgmap v6262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10949.612208 secs 2024-07-27T09:52:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:07 smithi029 ceph-mon[32610]: pgmap v6262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10949.612208 secs 2024-07-27T09:52:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:07 smithi077 ceph-mon[37153]: pgmap v6262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:09 smithi077 ceph-mon[37153]: pgmap v6263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:52:09.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:09 smithi047 ceph-mon[36835]: pgmap v6263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:52:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:09 smithi029 ceph-mon[32610]: pgmap v6263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:52:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:11 smithi077 ceph-mon[37153]: pgmap v6264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:11 smithi047 ceph-mon[36835]: pgmap v6264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:11 smithi029 ceph-mon[32610]: pgmap v6264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10954.612332 secs 2024-07-27T09:52:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10954.612332 secs 2024-07-27T09:52:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10954.612332 secs 2024-07-27T09:52:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:13 smithi047 ceph-mon[36835]: pgmap v6265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:13 smithi029 ceph-mon[32610]: pgmap v6265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:13 smithi077 ceph-mon[37153]: pgmap v6265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:15 smithi077 ceph-mon[37153]: pgmap v6266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:15 smithi047 ceph-mon[36835]: pgmap v6266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:15 smithi029 ceph-mon[32610]: pgmap v6266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:16.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:16.678+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:52:16.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:16.678+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:52:16.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:16.678+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:52:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:17 smithi077 ceph-mon[37153]: pgmap v6267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:17 smithi047 ceph-mon[36835]: pgmap v6267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:17 smithi029 ceph-mon[32610]: pgmap v6267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10959.612428 secs 2024-07-27T09:52:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10959.612428 secs 2024-07-27T09:52:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10959.612428 secs 2024-07-27T09:52:19.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:19 smithi077 ceph-mon[37153]: pgmap v6268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:52:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:19 smithi047 ceph-mon[36835]: pgmap v6268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:52:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:19 smithi029 ceph-mon[32610]: pgmap v6268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:52:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:21 smithi077 ceph-mon[37153]: pgmap v6269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:21 smithi047 ceph-mon[36835]: pgmap v6269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:21 smithi029 ceph-mon[32610]: pgmap v6269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10964.612581 secs 2024-07-27T09:52:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10964.612581 secs 2024-07-27T09:52:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10964.612581 secs 2024-07-27T09:52:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:23 smithi047 ceph-mon[36835]: pgmap v6270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:23 smithi029 ceph-mon[32610]: pgmap v6270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:23 smithi077 ceph-mon[37153]: pgmap v6270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:25 smithi077 ceph-mon[37153]: pgmap v6271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:25 smithi047 ceph-mon[36835]: pgmap v6271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:25 smithi029 ceph-mon[32610]: pgmap v6271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:27 smithi077 ceph-mon[37153]: pgmap v6272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:27 smithi047 ceph-mon[36835]: pgmap v6272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:27 smithi029 ceph-mon[32610]: pgmap v6272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10969.612658 secs 2024-07-27T09:52:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10969.612658 secs 2024-07-27T09:52:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10969.612658 secs 2024-07-27T09:52:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:29 smithi077 ceph-mon[37153]: pgmap v6273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:52:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:29 smithi047 ceph-mon[36835]: pgmap v6273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:52:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:29 smithi029 ceph-mon[32610]: pgmap v6273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:52:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:31 smithi077 ceph-mon[37153]: pgmap v6274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:31.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:31.688+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:52:31.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:31.688+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:52:31.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:31.688+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:52:31.691 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:31 smithi029 ceph-mon[32610]: pgmap v6274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:31 smithi047 ceph-mon[36835]: pgmap v6274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10974.612792 secs 2024-07-27T09:52:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10974.612792 secs 2024-07-27T09:52:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10974.612792 secs 2024-07-27T09:52:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:33 smithi047 ceph-mon[36835]: pgmap v6275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:33 smithi029 ceph-mon[32610]: pgmap v6275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:33 smithi077 ceph-mon[37153]: pgmap v6275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:35 smithi077 ceph-mon[37153]: pgmap v6276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:35 smithi047 ceph-mon[36835]: pgmap v6276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:35 smithi029 ceph-mon[32610]: pgmap v6276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:52:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:52:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:52:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:36 smithi047 ceph-mon[36835]: pgmap v6277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:52:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:36 smithi047 ceph-mon[36835]: pgmap v6278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:52:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:52:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:52:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:52:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:52:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:36 smithi029 ceph-mon[32610]: pgmap v6277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:52:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:36 smithi029 ceph-mon[32610]: pgmap v6278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:52:36.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:52:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:52:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:52:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:52:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:36 smithi077 ceph-mon[37153]: pgmap v6277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T09:52:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:36 smithi077 ceph-mon[37153]: pgmap v6278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T09:52:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:52:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10979.612905 secs 2024-07-27T09:52:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10979.612905 secs 2024-07-27T09:52:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10979.612905 secs 2024-07-27T09:52:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:38 smithi047 ceph-mon[36835]: pgmap v6279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:52:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:38 smithi029 ceph-mon[32610]: pgmap v6279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:52:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:38 smithi077 ceph-mon[37153]: pgmap v6279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:52:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:40 smithi047 ceph-mon[36835]: pgmap v6280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:52:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:40 smithi029 ceph-mon[32610]: pgmap v6280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:52:41.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:40 smithi077 ceph-mon[37153]: pgmap v6280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:52:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:42 smithi047 ceph-mon[36835]: pgmap v6281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:52:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10984.613001 secs 2024-07-27T09:52:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:42 smithi029 ceph-mon[32610]: pgmap v6281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:52:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10984.613001 secs 2024-07-27T09:52:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:42 smithi077 ceph-mon[37153]: pgmap v6281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:52:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10984.613001 secs 2024-07-27T09:52:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:44 smithi047 ceph-mon[36835]: pgmap v6282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:52:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:44 smithi029 ceph-mon[32610]: pgmap v6282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:52:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:44 smithi077 ceph-mon[37153]: pgmap v6282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:52:46.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:46.704+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:52:46.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:46.704+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:52:46.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:46.704+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:52:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:46 smithi029 ceph-mon[32610]: pgmap v6283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:52:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:46 smithi047 ceph-mon[36835]: pgmap v6283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:52:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:46 smithi077 ceph-mon[37153]: pgmap v6283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:52:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10989.613125 secs 2024-07-27T09:52:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10989.613125 secs 2024-07-27T09:52:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10989.613125 secs 2024-07-27T09:52:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:48 smithi047 ceph-mon[36835]: pgmap v6284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:52:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:48 smithi029 ceph-mon[32610]: pgmap v6284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:52:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:48 smithi077 ceph-mon[37153]: pgmap v6284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:52:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:50 smithi047 ceph-mon[36835]: pgmap v6285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:52:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:50 smithi029 ceph-mon[32610]: pgmap v6285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:52:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:50 smithi077 ceph-mon[37153]: pgmap v6285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:52:53.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:52 smithi047 ceph-mon[36835]: pgmap v6286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:53.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10994.613223 secs 2024-07-27T09:52:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:52 smithi029 ceph-mon[32610]: pgmap v6286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10994.613223 secs 2024-07-27T09:52:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:52 smithi077 ceph-mon[37153]: pgmap v6286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:52:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10994.613223 secs 2024-07-27T09:52:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:54 smithi047 ceph-mon[36835]: pgmap v6287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:54 smithi029 ceph-mon[32610]: pgmap v6287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:54 smithi077 ceph-mon[37153]: pgmap v6287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:52:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:56 smithi047 ceph-mon[36835]: pgmap v6288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:56 smithi029 ceph-mon[32610]: pgmap v6288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:56 smithi077 ceph-mon[37153]: pgmap v6288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:52:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 10999.613328 secs 2024-07-27T09:52:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 10999.613328 secs 2024-07-27T09:52:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 10999.613328 secs 2024-07-27T09:52:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:52:58 smithi047 ceph-mon[36835]: pgmap v6289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:52:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:52:58 smithi029 ceph-mon[32610]: pgmap v6289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:52:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:59.300+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:52:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:59.300+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:52:59.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:59.300+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:52:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:52:59.300+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:52:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:52:58 smithi077 ceph-mon[37153]: pgmap v6289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:53:01.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:00 smithi047 ceph-mon[36835]: pgmap v6290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:53:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:00 smithi029 ceph-mon[32610]: pgmap v6290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:53:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:00 smithi077 ceph-mon[37153]: pgmap v6290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:53:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:01.719+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:53:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:01.720+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:53:01.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:01.720+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:53:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:02 smithi047 ceph-mon[36835]: pgmap v6291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:53:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11004.613446 secs 2024-07-27T09:53:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:02 smithi029 ceph-mon[32610]: pgmap v6291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:53:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11004.613446 secs 2024-07-27T09:53:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:02 smithi077 ceph-mon[37153]: pgmap v6291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:53:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11004.613446 secs 2024-07-27T09:53:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:04 smithi047 ceph-mon[36835]: pgmap v6292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:04 smithi029 ceph-mon[32610]: pgmap v6292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:04 smithi077 ceph-mon[37153]: pgmap v6292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:06 smithi047 ceph-mon[36835]: pgmap v6293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:06 smithi029 ceph-mon[32610]: pgmap v6293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:06 smithi077 ceph-mon[37153]: pgmap v6293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11009.613565 secs 2024-07-27T09:53:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11009.613565 secs 2024-07-27T09:53:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11009.613565 secs 2024-07-27T09:53:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:08 smithi047 ceph-mon[36835]: pgmap v6294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:53:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:08 smithi029 ceph-mon[32610]: pgmap v6294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:53:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:08 smithi077 ceph-mon[37153]: pgmap v6294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:53:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:10 smithi047 ceph-mon[36835]: pgmap v6295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:53:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:10 smithi029 ceph-mon[32610]: pgmap v6295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:53:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:10 smithi077 ceph-mon[37153]: pgmap v6295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:53:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:12 smithi047 ceph-mon[36835]: pgmap v6296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:53:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11014.613666 secs 2024-07-27T09:53:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:12 smithi029 ceph-mon[32610]: pgmap v6296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:53:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11014.613666 secs 2024-07-27T09:53:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:12 smithi077 ceph-mon[37153]: pgmap v6296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:53:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11014.613666 secs 2024-07-27T09:53:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:14 smithi047 ceph-mon[36835]: pgmap v6297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:14 smithi029 ceph-mon[32610]: pgmap v6297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:14 smithi077 ceph-mon[37153]: pgmap v6297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:16.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:16.735+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:53:16.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:16.735+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:53:16.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:16.735+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:53:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:16 smithi029 ceph-mon[32610]: pgmap v6298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:16 smithi077 ceph-mon[37153]: pgmap v6298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:16 smithi047 ceph-mon[36835]: pgmap v6298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11019.613814 secs 2024-07-27T09:53:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11019.613814 secs 2024-07-27T09:53:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11019.613814 secs 2024-07-27T09:53:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:18 smithi029 ceph-mon[32610]: pgmap v6299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:53:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:18 smithi077 ceph-mon[37153]: pgmap v6299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:53:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:18 smithi047 ceph-mon[36835]: pgmap v6299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:53:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:20 smithi029 ceph-mon[32610]: pgmap v6300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:20 smithi077 ceph-mon[37153]: pgmap v6300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:20 smithi047 ceph-mon[36835]: pgmap v6300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:22 smithi077 ceph-mon[37153]: pgmap v6301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:53:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11024.613887 secs 2024-07-27T09:53:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:22 smithi047 ceph-mon[36835]: pgmap v6301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:53:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11024.613887 secs 2024-07-27T09:53:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:22 smithi029 ceph-mon[32610]: pgmap v6301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:53:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11024.613887 secs 2024-07-27T09:53:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:24 smithi077 ceph-mon[37153]: pgmap v6302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:24 smithi047 ceph-mon[36835]: pgmap v6302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:24 smithi029 ceph-mon[32610]: pgmap v6302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:26 smithi077 ceph-mon[37153]: pgmap v6303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:26 smithi047 ceph-mon[36835]: pgmap v6303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:26 smithi029 ceph-mon[32610]: pgmap v6303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11029.614013 secs 2024-07-27T09:53:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11029.614013 secs 2024-07-27T09:53:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11029.614013 secs 2024-07-27T09:53:29.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:28 smithi047 ceph-mon[36835]: pgmap v6304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:53:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:28 smithi077 ceph-mon[37153]: pgmap v6304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:53:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:28 smithi029 ceph-mon[32610]: pgmap v6304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:53:31.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:30 smithi029 ceph-mon[32610]: pgmap v6305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:30 smithi077 ceph-mon[37153]: pgmap v6305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:30 smithi047 ceph-mon[36835]: pgmap v6305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:31.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:31.748+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:53:31.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:31.748+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:53:31.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:31.748+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:53:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:32 smithi077 ceph-mon[37153]: pgmap v6306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:53:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11034.614137 secs 2024-07-27T09:53:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:32 smithi047 ceph-mon[36835]: pgmap v6306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:53:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11034.614137 secs 2024-07-27T09:53:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:32 smithi029 ceph-mon[32610]: pgmap v6306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:53:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11034.614137 secs 2024-07-27T09:53:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:34 smithi077 ceph-mon[37153]: pgmap v6307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:34 smithi047 ceph-mon[36835]: pgmap v6307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:34 smithi029 ceph-mon[32610]: pgmap v6307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:53:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:53:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:53:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:36 smithi077 ceph-mon[37153]: pgmap v6308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:53:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:53:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:53:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:36 smithi047 ceph-mon[36835]: pgmap v6308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:37.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:53:37.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:53:37.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:53:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:36 smithi029 ceph-mon[32610]: pgmap v6308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:53:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:53:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:53:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:38 smithi077 ceph-mon[37153]: pgmap v6309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:53:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:38 smithi077 ceph-mon[37153]: pgmap v6310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:53:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11039.614254 secs 2024-07-27T09:53:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:38 smithi047 ceph-mon[36835]: pgmap v6309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:53:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:38 smithi047 ceph-mon[36835]: pgmap v6310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:53:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11039.614254 secs 2024-07-27T09:53:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:38 smithi029 ceph-mon[32610]: pgmap v6309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T09:53:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:38 smithi029 ceph-mon[32610]: pgmap v6310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:53:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11039.614254 secs 2024-07-27T09:53:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:40 smithi077 ceph-mon[37153]: pgmap v6311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:53:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:40 smithi047 ceph-mon[36835]: pgmap v6311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:53:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:40 smithi029 ceph-mon[32610]: pgmap v6311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:53:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:42 smithi077 ceph-mon[37153]: pgmap v6312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:53:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:42 smithi047 ceph-mon[36835]: pgmap v6312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:53:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:42 smithi029 ceph-mon[32610]: pgmap v6312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:53:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11044.614376 secs 2024-07-27T09:53:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11044.614376 secs 2024-07-27T09:53:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11044.614376 secs 2024-07-27T09:53:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:44 smithi077 ceph-mon[37153]: pgmap v6313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:53:44.450 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:44 smithi047 ceph-mon[36835]: pgmap v6313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:53:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:44 smithi029 ceph-mon[32610]: pgmap v6313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T09:53:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:46 smithi077 ceph-mon[37153]: pgmap v6314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:53:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:46 smithi047 ceph-mon[36835]: pgmap v6314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:53:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:46 smithi029 ceph-mon[32610]: pgmap v6314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:53:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:46.763+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:53:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:46.763+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:53:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:46.763+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:53:47.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:47.856+0000 7fe7b67fc640 1 -- 172.21.15.29:0/827749814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7a80034c0 con 0x7fe7c01023f0 2024-07-27T09:53:47.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:53:47.856+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 <== mon.0 v2:172.21.15.29:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fe7a80034c0 con 0x7fe7c01023f0 2024-07-27T09:53:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:48 smithi077 ceph-mon[37153]: pgmap v6315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:53:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11049.614435 secs 2024-07-27T09:53:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:48 smithi047 ceph-mon[36835]: pgmap v6315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:53:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11049.614435 secs 2024-07-27T09:53:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:48 smithi029 ceph-mon[32610]: pgmap v6315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:53:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11049.614435 secs 2024-07-27T09:53:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:50 smithi077 ceph-mon[37153]: pgmap v6316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:50 smithi047 ceph-mon[36835]: pgmap v6316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:50 smithi029 ceph-mon[32610]: pgmap v6316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:53:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:52 smithi077 ceph-mon[37153]: pgmap v6317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:52 smithi047 ceph-mon[36835]: pgmap v6317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:52 smithi029 ceph-mon[32610]: pgmap v6317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:53:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11054.614560 secs 2024-07-27T09:53:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11054.614560 secs 2024-07-27T09:53:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11054.614560 secs 2024-07-27T09:53:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:54 smithi077 ceph-mon[37153]: pgmap v6318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:53:54.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:54 smithi047 ceph-mon[36835]: pgmap v6318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:53:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:54 smithi029 ceph-mon[32610]: pgmap v6318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:53:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:56 smithi077 ceph-mon[37153]: pgmap v6319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:53:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:56 smithi047 ceph-mon[36835]: pgmap v6319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:53:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:56 smithi029 ceph-mon[32610]: pgmap v6319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:53:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:58 smithi077 ceph-mon[37153]: pgmap v6320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:53:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:53:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11059.614628 secs 2024-07-27T09:53:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:58 smithi047 ceph-mon[36835]: pgmap v6320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:53:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:53:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11059.614628 secs 2024-07-27T09:53:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:58 smithi029 ceph-mon[32610]: pgmap v6320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:53:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:53:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11059.614628 secs 2024-07-27T09:54:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:00 smithi077 ceph-mon[37153]: pgmap v6321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:00 smithi047 ceph-mon[36835]: pgmap v6321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:00 smithi029 ceph-mon[32610]: pgmap v6321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:01.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:54:01.779+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:54:01.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:54:01.779+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:54:01.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:54:01.779+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:54:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:02 smithi077 ceph-mon[37153]: pgmap v6322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:02 smithi047 ceph-mon[36835]: pgmap v6322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:02 smithi029 ceph-mon[32610]: pgmap v6322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11064.614770 secs 2024-07-27T09:54:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11064.614770 secs 2024-07-27T09:54:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11064.614770 secs 2024-07-27T09:54:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:04 smithi077 ceph-mon[37153]: pgmap v6323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:54:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:04 smithi047 ceph-mon[36835]: pgmap v6323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:54:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:04 smithi029 ceph-mon[32610]: pgmap v6323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:54:06.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:06 smithi029 ceph-mon[32610]: pgmap v6324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:54:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:06 smithi077 ceph-mon[37153]: pgmap v6324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:54:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:06 smithi047 ceph-mon[36835]: pgmap v6324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:54:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:08 smithi077 ceph-mon[37153]: pgmap v6325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:54:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11069.614877 secs 2024-07-27T09:54:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:08 smithi047 ceph-mon[36835]: pgmap v6325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:54:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11069.614877 secs 2024-07-27T09:54:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:08 smithi029 ceph-mon[32610]: pgmap v6325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:54:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11069.614877 secs 2024-07-27T09:54:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:10 smithi077 ceph-mon[37153]: pgmap v6326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:10 smithi047 ceph-mon[36835]: pgmap v6326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:10 smithi029 ceph-mon[32610]: pgmap v6326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:12 smithi077 ceph-mon[37153]: pgmap v6327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:12 smithi047 ceph-mon[36835]: pgmap v6327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:12 smithi029 ceph-mon[32610]: pgmap v6327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11074.614976 secs 2024-07-27T09:54:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11074.614976 secs 2024-07-27T09:54:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11074.614976 secs 2024-07-27T09:54:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:14 smithi077 ceph-mon[37153]: pgmap v6328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:54:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:14 smithi047 ceph-mon[36835]: pgmap v6328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:54:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:14 smithi029 ceph-mon[32610]: pgmap v6328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:54:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:16 smithi077 ceph-mon[37153]: pgmap v6329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:54:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:16 smithi047 ceph-mon[36835]: pgmap v6329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:54:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:16 smithi029 ceph-mon[32610]: pgmap v6329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:54:16.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:54:16.794+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:54:16.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:54:16.794+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:54:16.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:54:16.794+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:54:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:18 smithi077 ceph-mon[37153]: pgmap v6330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:54:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11079.615095 secs 2024-07-27T09:54:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:18 smithi047 ceph-mon[36835]: pgmap v6330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:54:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11079.615095 secs 2024-07-27T09:54:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:18 smithi029 ceph-mon[32610]: pgmap v6330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:54:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11079.615095 secs 2024-07-27T09:54:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:20 smithi077 ceph-mon[37153]: pgmap v6331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:20 smithi047 ceph-mon[36835]: pgmap v6331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:20 smithi029 ceph-mon[32610]: pgmap v6331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:22 smithi077 ceph-mon[37153]: pgmap v6332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:22 smithi047 ceph-mon[36835]: pgmap v6332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:22 smithi029 ceph-mon[32610]: pgmap v6332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11084.615195 secs 2024-07-27T09:54:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11084.615195 secs 2024-07-27T09:54:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11084.615195 secs 2024-07-27T09:54:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:24 smithi077 ceph-mon[37153]: pgmap v6333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:54:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:24 smithi047 ceph-mon[36835]: pgmap v6333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:54:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:24 smithi029 ceph-mon[32610]: pgmap v6333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:54:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:26 smithi077 ceph-mon[37153]: pgmap v6334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:54:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:26 smithi047 ceph-mon[36835]: pgmap v6334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:54:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:26 smithi029 ceph-mon[32610]: pgmap v6334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:54:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:28 smithi047 ceph-mon[36835]: pgmap v6335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:54:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11089.615344 secs 2024-07-27T09:54:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:28 smithi029 ceph-mon[32610]: pgmap v6335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:54:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11089.615344 secs 2024-07-27T09:54:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:28 smithi077 ceph-mon[37153]: pgmap v6335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:54:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11089.615344 secs 2024-07-27T09:54:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:30 smithi047 ceph-mon[36835]: pgmap v6336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:30 smithi029 ceph-mon[32610]: pgmap v6336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:30 smithi077 ceph-mon[37153]: pgmap v6336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:31.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:54:31.797+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:54:31.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:54:31.797+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:54:31.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:54:31.797+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:54:32.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:32 smithi077 ceph-mon[37153]: pgmap v6337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:32 smithi047 ceph-mon[36835]: pgmap v6337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:32 smithi029 ceph-mon[32610]: pgmap v6337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11094.615465 secs 2024-07-27T09:54:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11094.615465 secs 2024-07-27T09:54:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11094.615465 secs 2024-07-27T09:54:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:34 smithi047 ceph-mon[36835]: pgmap v6338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:54:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:34 smithi029 ceph-mon[32610]: pgmap v6338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:54:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:34 smithi077 ceph-mon[37153]: pgmap v6338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:54:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:36 smithi047 ceph-mon[36835]: pgmap v6339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:54:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:36 smithi029 ceph-mon[32610]: pgmap v6339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:54:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:36 smithi077 ceph-mon[37153]: pgmap v6339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:54:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:54:37.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:54:37.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:54:37.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:54:37.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:54:37.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:54:37.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:54:37.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:54:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:54:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:54:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:54:37.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:54:37.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:54:37.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:54:37.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:54:37.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:54:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:54:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:54:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:54:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:54:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:54:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:54:37.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:54:37.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:54:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:38 smithi047 ceph-mon[36835]: pgmap v6340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:54:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11099.615612 secs 2024-07-27T09:54:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:38 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:54:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:38 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:54:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:38 smithi047 ceph-mon[36835]: pgmap v6341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:54:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:38 smithi047 ceph-mon[36835]: pgmap v6342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:54:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:38 smithi029 ceph-mon[32610]: pgmap v6340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:54:38.465 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11099.615612 secs 2024-07-27T09:54:38.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:38 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:54:38.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:38 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:54:38.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:38 smithi029 ceph-mon[32610]: pgmap v6341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:54:38.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:38 smithi029 ceph-mon[32610]: pgmap v6342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:54:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:38 smithi077 ceph-mon[37153]: pgmap v6340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:54:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11099.615612 secs 2024-07-27T09:54:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:38 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T09:54:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:38 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T09:54:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:38 smithi077 ceph-mon[37153]: pgmap v6341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:54:38.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:38 smithi077 ceph-mon[37153]: pgmap v6342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:54:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:40 smithi047 ceph-mon[36835]: pgmap v6343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:54:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:40 smithi029 ceph-mon[32610]: pgmap v6343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:54:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:40 smithi077 ceph-mon[37153]: pgmap v6343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:54:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:42 smithi047 ceph-mon[36835]: pgmap v6344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:54:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:42 smithi029 ceph-mon[32610]: pgmap v6344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:54:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:42 smithi077 ceph-mon[37153]: pgmap v6344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T09:54:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11104.615752 secs 2024-07-27T09:54:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11104.615752 secs 2024-07-27T09:54:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11104.615752 secs 2024-07-27T09:54:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:44 smithi047 ceph-mon[36835]: pgmap v6345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:54:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:44 smithi029 ceph-mon[32610]: pgmap v6345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:54:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:44 smithi077 ceph-mon[37153]: pgmap v6345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:54:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:46 smithi047 ceph-mon[36835]: pgmap v6346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:54:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:46 smithi029 ceph-mon[32610]: pgmap v6346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:54:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:46 smithi077 ceph-mon[37153]: pgmap v6346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:54:46.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:54:46.813+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:54:46.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:54:46.813+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:54:46.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:54:46.813+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:54:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11109.615875 secs 2024-07-27T09:54:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:48 smithi047 ceph-mon[36835]: pgmap v6347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:54:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11109.615875 secs 2024-07-27T09:54:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:48 smithi029 ceph-mon[32610]: pgmap v6347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:54:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11109.615875 secs 2024-07-27T09:54:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:48 smithi077 ceph-mon[37153]: pgmap v6347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:54:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:50 smithi047 ceph-mon[36835]: pgmap v6348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:50 smithi029 ceph-mon[32610]: pgmap v6348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:50 smithi077 ceph-mon[37153]: pgmap v6348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:54:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:52 smithi047 ceph-mon[36835]: pgmap v6349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:52 smithi029 ceph-mon[32610]: pgmap v6349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:52 smithi077 ceph-mon[37153]: pgmap v6349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:54:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11114.615995 secs 2024-07-27T09:54:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11114.615995 secs 2024-07-27T09:54:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11114.615995 secs 2024-07-27T09:54:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:54 smithi047 ceph-mon[36835]: pgmap v6350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:54:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:54 smithi029 ceph-mon[32610]: pgmap v6350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:54:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:54 smithi077 ceph-mon[37153]: pgmap v6350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:54:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:56 smithi047 ceph-mon[36835]: pgmap v6351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:54:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:56 smithi029 ceph-mon[32610]: pgmap v6351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:54:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:56 smithi077 ceph-mon[37153]: pgmap v6351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:54:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11119.616120 secs 2024-07-27T09:54:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:54:58 smithi047 ceph-mon[36835]: pgmap v6352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:54:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11119.616120 secs 2024-07-27T09:54:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:54:58 smithi029 ceph-mon[32610]: pgmap v6352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:54:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11119.616120 secs 2024-07-27T09:54:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:54:58 smithi077 ceph-mon[37153]: pgmap v6352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:00 smithi047 ceph-mon[36835]: pgmap v6353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:00 smithi029 ceph-mon[32610]: pgmap v6353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:00 smithi077 ceph-mon[37153]: pgmap v6353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:01.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:01.822+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:55:01.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:01.822+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:55:01.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:01.822+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:55:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:02 smithi047 ceph-mon[36835]: pgmap v6354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:02 smithi029 ceph-mon[32610]: pgmap v6354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:02.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:02 smithi077 ceph-mon[37153]: pgmap v6354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11124.616237 secs 2024-07-27T09:55:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11124.616237 secs 2024-07-27T09:55:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11124.616237 secs 2024-07-27T09:55:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:04 smithi047 ceph-mon[36835]: pgmap v6355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:55:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:04 smithi029 ceph-mon[32610]: pgmap v6355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:55:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:04 smithi077 ceph-mon[37153]: pgmap v6355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:55:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:06 smithi047 ceph-mon[36835]: pgmap v6356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:06 smithi029 ceph-mon[32610]: pgmap v6356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:06 smithi077 ceph-mon[37153]: pgmap v6356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11129.616338 secs 2024-07-27T09:55:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:08 smithi029 ceph-mon[32610]: pgmap v6357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11129.616338 secs 2024-07-27T09:55:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:08 smithi047 ceph-mon[36835]: pgmap v6357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11129.616338 secs 2024-07-27T09:55:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:08 smithi077 ceph-mon[37153]: pgmap v6357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:10 smithi047 ceph-mon[36835]: pgmap v6358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:10 smithi029 ceph-mon[32610]: pgmap v6358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:10 smithi077 ceph-mon[37153]: pgmap v6358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:12 smithi029 ceph-mon[32610]: pgmap v6359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:12 smithi077 ceph-mon[37153]: pgmap v6359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:12 smithi047 ceph-mon[36835]: pgmap v6359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11134.616459 secs 2024-07-27T09:55:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11134.616459 secs 2024-07-27T09:55:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11134.616459 secs 2024-07-27T09:55:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:14 smithi029 ceph-mon[32610]: pgmap v6360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:55:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:14 smithi077 ceph-mon[37153]: pgmap v6360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:55:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:14 smithi047 ceph-mon[36835]: pgmap v6360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:55:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:16 smithi029 ceph-mon[32610]: pgmap v6361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:16 smithi077 ceph-mon[37153]: pgmap v6361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:16 smithi047 ceph-mon[36835]: pgmap v6361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:16.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:16.837+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:55:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:16.837+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:55:16.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:16.837+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:55:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11139.616580 secs 2024-07-27T09:55:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:18 smithi029 ceph-mon[32610]: pgmap v6362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11139.616580 secs 2024-07-27T09:55:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:18 smithi077 ceph-mon[37153]: pgmap v6362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11139.616580 secs 2024-07-27T09:55:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:18 smithi047 ceph-mon[36835]: pgmap v6362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:20 smithi077 ceph-mon[37153]: pgmap v6363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:20 smithi047 ceph-mon[36835]: pgmap v6363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:20 smithi029 ceph-mon[32610]: pgmap v6363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:22 smithi077 ceph-mon[37153]: pgmap v6364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:22 smithi047 ceph-mon[36835]: pgmap v6364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:22 smithi029 ceph-mon[32610]: pgmap v6364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11144.616660 secs 2024-07-27T09:55:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11144.616660 secs 2024-07-27T09:55:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11144.616660 secs 2024-07-27T09:55:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:24 smithi077 ceph-mon[37153]: pgmap v6365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:55:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:24 smithi047 ceph-mon[36835]: pgmap v6365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:55:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:24 smithi029 ceph-mon[32610]: pgmap v6365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:55:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:26 smithi077 ceph-mon[37153]: pgmap v6366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:26 smithi047 ceph-mon[36835]: pgmap v6366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:26 smithi029 ceph-mon[32610]: pgmap v6366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11149.616817 secs 2024-07-27T09:55:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:28 smithi077 ceph-mon[37153]: pgmap v6367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11149.616817 secs 2024-07-27T09:55:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:28 smithi047 ceph-mon[36835]: pgmap v6367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11149.616817 secs 2024-07-27T09:55:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:28 smithi029 ceph-mon[32610]: pgmap v6367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:30.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:30 smithi047 ceph-mon[36835]: pgmap v6368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:30 smithi077 ceph-mon[37153]: pgmap v6368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:30 smithi029 ceph-mon[32610]: pgmap v6368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:31.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:31.852+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:55:31.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:31.852+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:55:31.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:31.852+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:55:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:32 smithi077 ceph-mon[37153]: pgmap v6369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:32 smithi047 ceph-mon[36835]: pgmap v6369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:32 smithi029 ceph-mon[32610]: pgmap v6369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11154.616959 secs 2024-07-27T09:55:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11154.616959 secs 2024-07-27T09:55:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11154.616959 secs 2024-07-27T09:55:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:34 smithi077 ceph-mon[37153]: pgmap v6370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:55:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:34 smithi047 ceph-mon[36835]: pgmap v6370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:55:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:34 smithi029 ceph-mon[32610]: pgmap v6370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:55:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:36 smithi077 ceph-mon[37153]: pgmap v6371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:36 smithi047 ceph-mon[36835]: pgmap v6371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:36 smithi029 ceph-mon[32610]: pgmap v6371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:55:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:55:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:55:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11159.617099 secs 2024-07-27T09:55:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: pgmap v6372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:55:38.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:55:38.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:55:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11159.617099 secs 2024-07-27T09:55:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: pgmap v6372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:55:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:55:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:55:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11159.617099 secs 2024-07-27T09:55:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: pgmap v6372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T09:55:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T09:55:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:55:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:55:38.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:55:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:39 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:55:39.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:39 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:55:39.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:39 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:55:39.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:39 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:55:39.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:39 smithi029 ceph-mon[32610]: pgmap v6373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T09:55:39.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:39 smithi029 ceph-mon[32610]: pgmap v6374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:55:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:39 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:55:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:39 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:55:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:39 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:55:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:39 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:55:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:39 smithi077 ceph-mon[37153]: pgmap v6373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T09:55:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:39 smithi077 ceph-mon[37153]: pgmap v6374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:55:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:39 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T09:55:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:39 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T09:55:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:39 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T09:55:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:39 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T09:55:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:39 smithi047 ceph-mon[36835]: pgmap v6373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T09:55:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:39 smithi047 ceph-mon[36835]: pgmap v6374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:55:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:41 smithi077 ceph-mon[37153]: pgmap v6375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:55:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:41 smithi047 ceph-mon[36835]: pgmap v6375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:55:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:41 smithi029 ceph-mon[32610]: pgmap v6375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:55:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:43 smithi077 ceph-mon[37153]: pgmap v6376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:55:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11164.617225 secs 2024-07-27T09:55:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:43 smithi047 ceph-mon[36835]: pgmap v6376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:55:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11164.617225 secs 2024-07-27T09:55:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:43 smithi029 ceph-mon[32610]: pgmap v6376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:55:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11164.617225 secs 2024-07-27T09:55:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:45 smithi077 ceph-mon[37153]: pgmap v6377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:45 smithi047 ceph-mon[36835]: pgmap v6377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:45 smithi029 ceph-mon[32610]: pgmap v6377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:46.853+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:55:46.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:46.853+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:55:46.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:46.853+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:55:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:47 smithi077 ceph-mon[37153]: pgmap v6378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:55:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:47 smithi047 ceph-mon[36835]: pgmap v6378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:55:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:47 smithi029 ceph-mon[32610]: pgmap v6378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:55:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11169.617347 secs 2024-07-27T09:55:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11169.617347 secs 2024-07-27T09:55:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11169.617347 secs 2024-07-27T09:55:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:49 smithi077 ceph-mon[37153]: pgmap v6379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:55:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:49 smithi047 ceph-mon[36835]: pgmap v6379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:55:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:49 smithi029 ceph-mon[32610]: pgmap v6379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:55:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:51 smithi077 ceph-mon[37153]: pgmap v6380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:55:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:51 smithi047 ceph-mon[36835]: pgmap v6380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:55:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:51 smithi029 ceph-mon[32610]: pgmap v6380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:55:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:53 smithi077 ceph-mon[37153]: pgmap v6381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:55:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11174.617482 secs 2024-07-27T09:55:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:53 smithi047 ceph-mon[36835]: pgmap v6381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:55:53.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11174.617482 secs 2024-07-27T09:55:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:53 smithi029 ceph-mon[32610]: pgmap v6381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:55:53.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11174.617482 secs 2024-07-27T09:55:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:55 smithi029 ceph-mon[32610]: pgmap v6382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:55 smithi077 ceph-mon[37153]: pgmap v6382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:55 smithi047 ceph-mon[36835]: pgmap v6382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:55:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:57 smithi077 ceph-mon[37153]: pgmap v6383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:57 smithi047 ceph-mon[36835]: pgmap v6383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:57 smithi029 ceph-mon[32610]: pgmap v6383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:55:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11179.617619 secs 2024-07-27T09:55:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11179.617619 secs 2024-07-27T09:55:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11179.617619 secs 2024-07-27T09:55:59.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:59.327+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:55:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:59.327+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:55:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:59.327+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:55:59.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:55:59.327+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:55:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:55:59 smithi077 ceph-mon[37153]: pgmap v6384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:55:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:55:59 smithi047 ceph-mon[36835]: pgmap v6384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:55:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:55:59 smithi029 ceph-mon[32610]: pgmap v6384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:56:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:01 smithi077 ceph-mon[37153]: pgmap v6385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:56:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:01 smithi047 ceph-mon[36835]: pgmap v6385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:56:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:01 smithi029 ceph-mon[32610]: pgmap v6385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:56:01.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:01.869+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:56:01.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:01.869+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:56:01.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:01.869+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:56:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:03 smithi077 ceph-mon[37153]: pgmap v6386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11184.617745 secs 2024-07-27T09:56:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:03 smithi047 ceph-mon[36835]: pgmap v6386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:03.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11184.617745 secs 2024-07-27T09:56:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:03 smithi029 ceph-mon[32610]: pgmap v6386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11184.617745 secs 2024-07-27T09:56:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:05 smithi077 ceph-mon[37153]: pgmap v6387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:56:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:05 smithi047 ceph-mon[36835]: pgmap v6387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:56:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:05 smithi029 ceph-mon[32610]: pgmap v6387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:56:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:07 smithi077 ceph-mon[37153]: pgmap v6388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:56:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:07 smithi047 ceph-mon[36835]: pgmap v6388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:56:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:07 smithi029 ceph-mon[32610]: pgmap v6388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:56:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11189.617866 secs 2024-07-27T09:56:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11189.617866 secs 2024-07-27T09:56:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11189.617866 secs 2024-07-27T09:56:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:09 smithi077 ceph-mon[37153]: pgmap v6389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:56:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:09 smithi047 ceph-mon[36835]: pgmap v6389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:56:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:09 smithi029 ceph-mon[32610]: pgmap v6389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:56:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:11 smithi077 ceph-mon[37153]: pgmap v6390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:56:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:11 smithi047 ceph-mon[36835]: pgmap v6390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:56:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:11 smithi029 ceph-mon[32610]: pgmap v6390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:56:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:13 smithi077 ceph-mon[37153]: pgmap v6391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11194.617986 secs 2024-07-27T09:56:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:13 smithi047 ceph-mon[36835]: pgmap v6391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11194.617986 secs 2024-07-27T09:56:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:13 smithi029 ceph-mon[32610]: pgmap v6391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11194.617986 secs 2024-07-27T09:56:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:14 smithi047 ceph-mon[36835]: pgmap v6392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:56:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:14 smithi029 ceph-mon[32610]: pgmap v6392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:56:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:14 smithi077 ceph-mon[37153]: pgmap v6392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:56:16.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:16.885+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:56:16.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:16.885+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:56:16.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:16.885+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:56:16.887 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:16 smithi029 ceph-mon[32610]: pgmap v6393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:56:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:16 smithi047 ceph-mon[36835]: pgmap v6393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:56:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:16 smithi077 ceph-mon[37153]: pgmap v6393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:56:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11199.618108 secs 2024-07-27T09:56:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11199.618108 secs 2024-07-27T09:56:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11199.618108 secs 2024-07-27T09:56:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:18 smithi047 ceph-mon[36835]: pgmap v6394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:56:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:18 smithi029 ceph-mon[32610]: pgmap v6394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:56:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:18 smithi077 ceph-mon[37153]: pgmap v6394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:56:20.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:20 smithi047 ceph-mon[36835]: pgmap v6395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:56:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:20 smithi029 ceph-mon[32610]: pgmap v6395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:56:21.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:20 smithi077 ceph-mon[37153]: pgmap v6395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:56:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:22 smithi047 ceph-mon[36835]: pgmap v6396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11204.618263 secs 2024-07-27T09:56:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:22 smithi029 ceph-mon[32610]: pgmap v6396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11204.618263 secs 2024-07-27T09:56:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:22 smithi077 ceph-mon[37153]: pgmap v6396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11204.618263 secs 2024-07-27T09:56:24.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:24 smithi047 ceph-mon[36835]: pgmap v6397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:56:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:24 smithi029 ceph-mon[32610]: pgmap v6397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:56:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:24 smithi077 ceph-mon[37153]: pgmap v6397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:56:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:26 smithi047 ceph-mon[36835]: pgmap v6398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:56:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:26 smithi029 ceph-mon[32610]: pgmap v6398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:56:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:26 smithi077 ceph-mon[37153]: pgmap v6398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:56:27.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11209.618366 secs 2024-07-27T09:56:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11209.618366 secs 2024-07-27T09:56:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11209.618366 secs 2024-07-27T09:56:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:28 smithi047 ceph-mon[36835]: pgmap v6399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:56:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:28 smithi029 ceph-mon[32610]: pgmap v6399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:56:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:28 smithi077 ceph-mon[37153]: pgmap v6399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T09:56:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:30 smithi047 ceph-mon[36835]: pgmap v6400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:56:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:30 smithi029 ceph-mon[32610]: pgmap v6400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:56:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:30 smithi077 ceph-mon[37153]: pgmap v6400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:56:31.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:31.899+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:56:31.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:31.899+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:56:31.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:31.899+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:56:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:32 smithi047 ceph-mon[36835]: pgmap v6401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11214.618487 secs 2024-07-27T09:56:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:32 smithi029 ceph-mon[32610]: pgmap v6401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11214.618487 secs 2024-07-27T09:56:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:32 smithi077 ceph-mon[37153]: pgmap v6401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11214.618487 secs 2024-07-27T09:56:34.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:34 smithi077 ceph-mon[37153]: pgmap v6402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:56:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:34 smithi047 ceph-mon[36835]: pgmap v6402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:56:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:34 smithi029 ceph-mon[32610]: pgmap v6402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:56:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:36 smithi047 ceph-mon[36835]: pgmap v6403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:56:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:36 smithi029 ceph-mon[32610]: pgmap v6403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:56:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:36 smithi077 ceph-mon[37153]: pgmap v6403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:56:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11219.618615 secs 2024-07-27T09:56:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:56:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11219.618615 secs 2024-07-27T09:56:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:56:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11219.618615 secs 2024-07-27T09:56:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:56:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:38 smithi047 ceph-mon[36835]: pgmap v6404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:56:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:56:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:56:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:38 smithi047 ceph-mon[36835]: pgmap v6405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:56:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:38 smithi047 ceph-mon[36835]: pgmap v6406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:56:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:56:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:38 smithi029 ceph-mon[32610]: pgmap v6404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:56:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:56:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:56:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:38 smithi029 ceph-mon[32610]: pgmap v6405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:56:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:38 smithi029 ceph-mon[32610]: pgmap v6406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:56:38.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:56:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:38 smithi077 ceph-mon[37153]: pgmap v6404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:56:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:56:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:56:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:38 smithi077 ceph-mon[37153]: pgmap v6405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:56:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:38 smithi077 ceph-mon[37153]: pgmap v6406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:56:39.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:56:41.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:41 smithi047 ceph-mon[36835]: pgmap v6407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:56:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:41 smithi029 ceph-mon[32610]: pgmap v6407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:56:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:41 smithi077 ceph-mon[37153]: pgmap v6407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:56:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11224.618713 secs 2024-07-27T09:56:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:43 smithi047 ceph-mon[36835]: pgmap v6408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:56:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11224.618713 secs 2024-07-27T09:56:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:43 smithi029 ceph-mon[32610]: pgmap v6408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:56:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11224.618713 secs 2024-07-27T09:56:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:43 smithi077 ceph-mon[37153]: pgmap v6408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:56:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:45 smithi047 ceph-mon[36835]: pgmap v6409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:56:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:45 smithi029 ceph-mon[32610]: pgmap v6409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:56:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:45 smithi077 ceph-mon[37153]: pgmap v6409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:56:46.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:46.907+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:56:46.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:46.907+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:56:46.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:46.907+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:56:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:47 smithi047 ceph-mon[36835]: pgmap v6410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 6 op/s 2024-07-27T09:56:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:47 smithi029 ceph-mon[32610]: pgmap v6410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 6 op/s 2024-07-27T09:56:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:47 smithi077 ceph-mon[37153]: pgmap v6410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 6 op/s 2024-07-27T09:56:47.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:47.884+0000 7fe7b67fc640 1 -- 172.21.15.29:0/827749814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7a8005850 con 0x7fe7c01023f0 2024-07-27T09:56:47.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:56:47.884+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 <== mon.0 v2:172.21.15.29:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fe7a8005850 con 0x7fe7c01023f0 2024-07-27T09:56:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11229.618844 secs 2024-07-27T09:56:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11229.618844 secs 2024-07-27T09:56:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11229.618844 secs 2024-07-27T09:56:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:49 smithi047 ceph-mon[36835]: pgmap v6411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:56:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:49 smithi029 ceph-mon[32610]: pgmap v6411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:56:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:49 smithi077 ceph-mon[37153]: pgmap v6411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T09:56:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:51 smithi047 ceph-mon[36835]: pgmap v6412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:56:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:51 smithi029 ceph-mon[32610]: pgmap v6412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:56:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:51 smithi077 ceph-mon[37153]: pgmap v6412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:56:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11234.618968 secs 2024-07-27T09:56:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:53 smithi047 ceph-mon[36835]: pgmap v6413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11234.618968 secs 2024-07-27T09:56:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:53 smithi029 ceph-mon[32610]: pgmap v6413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11234.618968 secs 2024-07-27T09:56:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:53 smithi077 ceph-mon[37153]: pgmap v6413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:56:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:55 smithi047 ceph-mon[36835]: pgmap v6414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:56:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:55 smithi029 ceph-mon[32610]: pgmap v6414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:56:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:55 smithi077 ceph-mon[37153]: pgmap v6414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:56:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:57 smithi047 ceph-mon[36835]: pgmap v6415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:56:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:57 smithi029 ceph-mon[32610]: pgmap v6415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:56:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:57 smithi077 ceph-mon[37153]: pgmap v6415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:56:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11239.619089 secs 2024-07-27T09:56:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11239.619089 secs 2024-07-27T09:56:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11239.619089 secs 2024-07-27T09:56:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:56:59 smithi047 ceph-mon[36835]: pgmap v6416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:56:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:56:59 smithi029 ceph-mon[32610]: pgmap v6416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:56:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:56:59 smithi077 ceph-mon[37153]: pgmap v6416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:57:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:01 smithi047 ceph-mon[36835]: pgmap v6417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:01 smithi029 ceph-mon[32610]: pgmap v6417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:01 smithi077 ceph-mon[37153]: pgmap v6417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:01.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:57:01.923+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:57:01.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:57:01.923+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:57:01.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:57:01.923+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:57:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11244.619225 secs 2024-07-27T09:57:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:03 smithi029 ceph-mon[32610]: pgmap v6418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11244.619225 secs 2024-07-27T09:57:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:03 smithi077 ceph-mon[37153]: pgmap v6418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11244.619225 secs 2024-07-27T09:57:03.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:03 smithi047 ceph-mon[36835]: pgmap v6418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:05 smithi029 ceph-mon[32610]: pgmap v6419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:57:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:05 smithi077 ceph-mon[37153]: pgmap v6419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:57:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:05 smithi047 ceph-mon[36835]: pgmap v6419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:57:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:07 smithi029 ceph-mon[32610]: pgmap v6420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:07 smithi077 ceph-mon[37153]: pgmap v6420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:07 smithi047 ceph-mon[36835]: pgmap v6420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11249.619374 secs 2024-07-27T09:57:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11249.619374 secs 2024-07-27T09:57:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11249.619374 secs 2024-07-27T09:57:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:09 smithi077 ceph-mon[37153]: pgmap v6421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:57:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:09 smithi047 ceph-mon[36835]: pgmap v6421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:57:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:09 smithi029 ceph-mon[32610]: pgmap v6421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:57:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:11 smithi077 ceph-mon[37153]: pgmap v6422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:11.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:11 smithi047 ceph-mon[36835]: pgmap v6422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:11 smithi029 ceph-mon[32610]: pgmap v6422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11254.619477 secs 2024-07-27T09:57:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:13 smithi077 ceph-mon[37153]: pgmap v6423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11254.619477 secs 2024-07-27T09:57:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:13 smithi047 ceph-mon[36835]: pgmap v6423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11254.619477 secs 2024-07-27T09:57:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:13 smithi029 ceph-mon[32610]: pgmap v6423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:15 smithi077 ceph-mon[37153]: pgmap v6424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:57:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:15 smithi047 ceph-mon[36835]: pgmap v6424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:57:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:15 smithi029 ceph-mon[32610]: pgmap v6424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:57:16.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:57:16.938+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:57:16.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:57:16.938+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:57:16.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:57:16.938+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:57:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:17 smithi077 ceph-mon[37153]: pgmap v6425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:57:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:17 smithi047 ceph-mon[36835]: pgmap v6425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:57:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:17 smithi029 ceph-mon[32610]: pgmap v6425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T09:57:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11259.619598 secs 2024-07-27T09:57:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11259.619598 secs 2024-07-27T09:57:18.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11259.619598 secs 2024-07-27T09:57:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:19 smithi077 ceph-mon[37153]: pgmap v6426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:57:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:19 smithi047 ceph-mon[36835]: pgmap v6426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:57:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:19 smithi029 ceph-mon[32610]: pgmap v6426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T09:57:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:21 smithi077 ceph-mon[37153]: pgmap v6427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:57:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:21 smithi047 ceph-mon[36835]: pgmap v6427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:57:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:21 smithi029 ceph-mon[32610]: pgmap v6427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T09:57:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11264.619725 secs 2024-07-27T09:57:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:23 smithi077 ceph-mon[37153]: pgmap v6428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11264.619725 secs 2024-07-27T09:57:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:23 smithi047 ceph-mon[36835]: pgmap v6428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11264.619725 secs 2024-07-27T09:57:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:23 smithi029 ceph-mon[32610]: pgmap v6428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:25 smithi077 ceph-mon[37153]: pgmap v6429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:57:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:25 smithi047 ceph-mon[36835]: pgmap v6429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:57:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:25 smithi029 ceph-mon[32610]: pgmap v6429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T09:57:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:27 smithi077 ceph-mon[37153]: pgmap v6430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:27 smithi047 ceph-mon[36835]: pgmap v6430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:27 smithi029 ceph-mon[32610]: pgmap v6430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11269.619821 secs 2024-07-27T09:57:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11269.619821 secs 2024-07-27T09:57:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11269.619821 secs 2024-07-27T09:57:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:29 smithi077 ceph-mon[37153]: pgmap v6431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:57:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:29 smithi047 ceph-mon[36835]: pgmap v6431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:57:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:29 smithi029 ceph-mon[32610]: pgmap v6431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:57:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:31 smithi077 ceph-mon[37153]: pgmap v6432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:31 smithi047 ceph-mon[36835]: pgmap v6432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:31 smithi029 ceph-mon[32610]: pgmap v6432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:57:31.953+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:57:31.953+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:57:31.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:57:31.953+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:57:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11274.619941 secs 2024-07-27T09:57:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:33 smithi077 ceph-mon[37153]: pgmap v6433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11274.619941 secs 2024-07-27T09:57:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:33 smithi047 ceph-mon[36835]: pgmap v6433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11274.619941 secs 2024-07-27T09:57:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:33 smithi029 ceph-mon[32610]: pgmap v6433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:35 smithi077 ceph-mon[37153]: pgmap v6434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:57:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:35 smithi047 ceph-mon[36835]: pgmap v6434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:57:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:35 smithi029 ceph-mon[32610]: pgmap v6434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:57:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:37 smithi077 ceph-mon[37153]: pgmap v6435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:37.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:37 smithi047 ceph-mon[36835]: pgmap v6435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:37 smithi029 ceph-mon[32610]: pgmap v6435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11279.620060 secs 2024-07-27T09:57:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:57:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11279.620060 secs 2024-07-27T09:57:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:57:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11279.620060 secs 2024-07-27T09:57:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:39 smithi077 ceph-mon[37153]: pgmap v6436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:57:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:57:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:39 smithi047 ceph-mon[36835]: pgmap v6436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:57:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:57:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:57:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:57:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:39 smithi029 ceph-mon[32610]: pgmap v6436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:57:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:57:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:57:39.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:57:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:40 smithi077 ceph-mon[37153]: pgmap v6437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:57:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:40 smithi077 ceph-mon[37153]: pgmap v6438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:57:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:40 smithi047 ceph-mon[36835]: pgmap v6437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:57:40.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:40 smithi047 ceph-mon[36835]: pgmap v6438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:57:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:40 smithi029 ceph-mon[32610]: pgmap v6437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T09:57:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:40 smithi029 ceph-mon[32610]: pgmap v6438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T09:57:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:42 smithi077 ceph-mon[37153]: pgmap v6439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:57:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:42 smithi047 ceph-mon[36835]: pgmap v6439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:57:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:42 smithi029 ceph-mon[32610]: pgmap v6439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T09:57:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11284.620177 secs 2024-07-27T09:57:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11284.620177 secs 2024-07-27T09:57:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11284.620177 secs 2024-07-27T09:57:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:44 smithi029 ceph-mon[32610]: pgmap v6440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:57:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:44 smithi077 ceph-mon[37153]: pgmap v6440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:57:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:44 smithi047 ceph-mon[36835]: pgmap v6440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T09:57:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:46 smithi077 ceph-mon[37153]: pgmap v6441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:57:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:46 smithi047 ceph-mon[36835]: pgmap v6441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:57:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:46 smithi029 ceph-mon[32610]: pgmap v6441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T09:57:46.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:57:46.968+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:57:46.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:57:46.968+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:57:46.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:57:46.968+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:57:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:48 smithi077 ceph-mon[37153]: pgmap v6442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:57:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11289.620299 secs 2024-07-27T09:57:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:48 smithi047 ceph-mon[36835]: pgmap v6442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:57:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11289.620299 secs 2024-07-27T09:57:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:48 smithi029 ceph-mon[32610]: pgmap v6442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T09:57:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11289.620299 secs 2024-07-27T09:57:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:50 smithi077 ceph-mon[37153]: pgmap v6443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:57:50.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:50 smithi047 ceph-mon[36835]: pgmap v6443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:57:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:50 smithi029 ceph-mon[32610]: pgmap v6443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:57:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:52 smithi077 ceph-mon[37153]: pgmap v6444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:52 smithi047 ceph-mon[36835]: pgmap v6444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:52 smithi029 ceph-mon[32610]: pgmap v6444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:57:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11294.620435 secs 2024-07-27T09:57:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11294.620435 secs 2024-07-27T09:57:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11294.620435 secs 2024-07-27T09:57:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:54 smithi077 ceph-mon[37153]: pgmap v6445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:57:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:54 smithi047 ceph-mon[36835]: pgmap v6445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:57:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:54 smithi029 ceph-mon[32610]: pgmap v6445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:57:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:56 smithi077 ceph-mon[37153]: pgmap v6446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:57:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:56 smithi047 ceph-mon[36835]: pgmap v6446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:57:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:56 smithi029 ceph-mon[32610]: pgmap v6446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:57:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:58 smithi077 ceph-mon[37153]: pgmap v6447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:57:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11299.620560 secs 2024-07-27T09:57:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:58 smithi047 ceph-mon[36835]: pgmap v6447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:57:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11299.620560 secs 2024-07-27T09:57:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:58 smithi029 ceph-mon[32610]: pgmap v6447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:57:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:57:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11299.620560 secs 2024-07-27T09:58:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:00 smithi077 ceph-mon[37153]: pgmap v6448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:00 smithi047 ceph-mon[36835]: pgmap v6448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:00 smithi029 ceph-mon[32610]: pgmap v6448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:01.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:01.973+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:58:01.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:01.973+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:58:01.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:01.973+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:58:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:02 smithi077 ceph-mon[37153]: pgmap v6449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:02 smithi047 ceph-mon[36835]: pgmap v6449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:02 smithi029 ceph-mon[32610]: pgmap v6449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:03.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11304.620703 secs 2024-07-27T09:58:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11304.620703 secs 2024-07-27T09:58:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11304.620703 secs 2024-07-27T09:58:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:04 smithi077 ceph-mon[37153]: pgmap v6450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:58:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:04 smithi047 ceph-mon[36835]: pgmap v6450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:58:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:04 smithi029 ceph-mon[32610]: pgmap v6450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:58:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:06 smithi077 ceph-mon[37153]: pgmap v6451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:58:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:06 smithi047 ceph-mon[36835]: pgmap v6451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:58:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:06 smithi029 ceph-mon[32610]: pgmap v6451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:58:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:08 smithi077 ceph-mon[37153]: pgmap v6452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:58:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11309.620823 secs 2024-07-27T09:58:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:08 smithi047 ceph-mon[36835]: pgmap v6452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:58:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11309.620823 secs 2024-07-27T09:58:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:08 smithi029 ceph-mon[32610]: pgmap v6452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:58:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11309.620823 secs 2024-07-27T09:58:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:10 smithi077 ceph-mon[37153]: pgmap v6453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:10 smithi047 ceph-mon[36835]: pgmap v6453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:10 smithi029 ceph-mon[32610]: pgmap v6453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:12 smithi077 ceph-mon[37153]: pgmap v6454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:12 smithi047 ceph-mon[36835]: pgmap v6454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:12 smithi029 ceph-mon[32610]: pgmap v6454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11314.620944 secs 2024-07-27T09:58:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:13 smithi077 ceph-mon[37153]: pgmap v6455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:58:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11314.620944 secs 2024-07-27T09:58:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:13 smithi047 ceph-mon[36835]: pgmap v6455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:58:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11314.620944 secs 2024-07-27T09:58:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:13 smithi029 ceph-mon[32610]: pgmap v6455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:58:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:15 smithi077 ceph-mon[37153]: pgmap v6456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:58:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:15 smithi047 ceph-mon[36835]: pgmap v6456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:58:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:15 smithi029 ceph-mon[32610]: pgmap v6456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:58:16.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:16.988+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:58:16.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:16.988+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:58:16.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:16.988+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:58:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:17 smithi077 ceph-mon[37153]: pgmap v6457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:58:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11319.621100 secs 2024-07-27T09:58:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:17 smithi047 ceph-mon[36835]: pgmap v6457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:58:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11319.621100 secs 2024-07-27T09:58:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:17 smithi029 ceph-mon[32610]: pgmap v6457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:58:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11319.621100 secs 2024-07-27T09:58:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:19 smithi077 ceph-mon[37153]: pgmap v6458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:20.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:19 smithi047 ceph-mon[36835]: pgmap v6458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:19 smithi029 ceph-mon[32610]: pgmap v6458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:22.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:21 smithi077 ceph-mon[37153]: pgmap v6459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:21 smithi047 ceph-mon[36835]: pgmap v6459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:21 smithi029 ceph-mon[32610]: pgmap v6459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11324.621239 secs 2024-07-27T09:58:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11324.621239 secs 2024-07-27T09:58:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11324.621239 secs 2024-07-27T09:58:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:23 smithi077 ceph-mon[37153]: pgmap v6460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:58:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:23 smithi047 ceph-mon[36835]: pgmap v6460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:58:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:23 smithi029 ceph-mon[32610]: pgmap v6460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:58:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:25 smithi077 ceph-mon[37153]: pgmap v6461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:58:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:25 smithi047 ceph-mon[36835]: pgmap v6461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:58:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:25 smithi029 ceph-mon[32610]: pgmap v6461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:58:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:27 smithi077 ceph-mon[37153]: pgmap v6462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:58:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11329.621382 secs 2024-07-27T09:58:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:27 smithi047 ceph-mon[36835]: pgmap v6462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:58:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11329.621382 secs 2024-07-27T09:58:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:27 smithi029 ceph-mon[32610]: pgmap v6462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:58:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11329.621382 secs 2024-07-27T09:58:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:29 smithi077 ceph-mon[37153]: pgmap v6463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:30.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:29 smithi029 ceph-mon[32610]: pgmap v6463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:29 smithi047 ceph-mon[36835]: pgmap v6463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:32.000+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:58:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:32.000+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:58:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:32.000+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:58:32.002 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:31 smithi029 ceph-mon[32610]: pgmap v6464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:31 smithi077 ceph-mon[37153]: pgmap v6464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:31 smithi047 ceph-mon[36835]: pgmap v6464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:33.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11334.621499 secs 2024-07-27T09:58:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11334.621499 secs 2024-07-27T09:58:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11334.621499 secs 2024-07-27T09:58:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:33 smithi077 ceph-mon[37153]: pgmap v6465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:58:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:33 smithi047 ceph-mon[36835]: pgmap v6465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:58:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:33 smithi029 ceph-mon[32610]: pgmap v6465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T09:58:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:35 smithi077 ceph-mon[37153]: pgmap v6466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:58:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:35 smithi047 ceph-mon[36835]: pgmap v6466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:58:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:35 smithi029 ceph-mon[32610]: pgmap v6466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:58:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:37 smithi077 ceph-mon[37153]: pgmap v6467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:58:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11339.621622 secs 2024-07-27T09:58:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:37 smithi047 ceph-mon[36835]: pgmap v6467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:58:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11339.621622 secs 2024-07-27T09:58:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:37 smithi029 ceph-mon[32610]: pgmap v6467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:58:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11339.621622 secs 2024-07-27T09:58:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:58:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:58:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:58:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:40 smithi077 ceph-mon[37153]: pgmap v6468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:40 smithi047 ceph-mon[36835]: pgmap v6468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:40.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:40 smithi029 ceph-mon[32610]: pgmap v6468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:40.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:41 smithi047 ceph-mon[36835]: pgmap v6469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:58:41.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:58:41.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:41 smithi029 ceph-mon[32610]: pgmap v6469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:41.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:58:41.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:58:41.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:41 smithi077 ceph-mon[37153]: pgmap v6469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:41.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:41.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:58:41.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:58:41.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:58:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:42 smithi077 ceph-mon[37153]: pgmap v6470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:58:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:42 smithi077 ceph-mon[37153]: pgmap v6471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T09:58:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11344.621749 secs 2024-07-27T09:58:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:42 smithi047 ceph-mon[36835]: pgmap v6470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:58:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:42 smithi047 ceph-mon[36835]: pgmap v6471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T09:58:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11344.621749 secs 2024-07-27T09:58:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:42 smithi029 ceph-mon[32610]: pgmap v6470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:58:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:42 smithi029 ceph-mon[32610]: pgmap v6471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T09:58:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11344.621749 secs 2024-07-27T09:58:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:44 smithi077 ceph-mon[37153]: pgmap v6472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:58:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:44 smithi047 ceph-mon[36835]: pgmap v6472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:58:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:44 smithi029 ceph-mon[32610]: pgmap v6472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:58:46.754 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:46 smithi029 ceph-mon[32610]: pgmap v6473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:58:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:46 smithi077 ceph-mon[37153]: pgmap v6473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:58:46.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:46 smithi047 ceph-mon[36835]: pgmap v6473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:58:47.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:47.015+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:58:47.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:47.015+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:58:47.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:47.015+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:58:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11349.621876 secs 2024-07-27T09:58:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11349.621876 secs 2024-07-27T09:58:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11349.621876 secs 2024-07-27T09:58:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:48 smithi077 ceph-mon[37153]: pgmap v6474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:58:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:48 smithi047 ceph-mon[36835]: pgmap v6474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:58:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:48 smithi029 ceph-mon[32610]: pgmap v6474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T09:58:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:50 smithi077 ceph-mon[37153]: pgmap v6475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:58:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:50 smithi047 ceph-mon[36835]: pgmap v6475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:58:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:50 smithi029 ceph-mon[32610]: pgmap v6475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T09:58:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:52 smithi077 ceph-mon[37153]: pgmap v6476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:58:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11354.622002 secs 2024-07-27T09:58:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:52 smithi047 ceph-mon[36835]: pgmap v6476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:58:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11354.622002 secs 2024-07-27T09:58:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:52 smithi029 ceph-mon[32610]: pgmap v6476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:58:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11354.622002 secs 2024-07-27T09:58:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:54 smithi077 ceph-mon[37153]: pgmap v6477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:54 smithi047 ceph-mon[36835]: pgmap v6477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:54 smithi029 ceph-mon[32610]: pgmap v6477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:58:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:56 smithi077 ceph-mon[37153]: pgmap v6478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:56 smithi047 ceph-mon[36835]: pgmap v6478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:56 smithi029 ceph-mon[32610]: pgmap v6478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:58:57.816 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11359.622141 secs 2024-07-27T09:58:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11359.622141 secs 2024-07-27T09:58:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11359.622141 secs 2024-07-27T09:58:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:58:58 smithi077 ceph-mon[37153]: pgmap v6479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:58:58.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:58:58 smithi047 ceph-mon[36835]: pgmap v6479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:58:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:58:58 smithi029 ceph-mon[32610]: pgmap v6479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:58:59.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:59.353+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:58:59.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:59.353+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:58:59.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:59.354+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:58:59.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:58:59.354+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:59:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:00 smithi077 ceph-mon[37153]: pgmap v6480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:00 smithi047 ceph-mon[36835]: pgmap v6480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:00 smithi029 ceph-mon[32610]: pgmap v6480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:02.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:02.031+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:59:02.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:02.031+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:59:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:02.031+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:59:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:02 smithi077 ceph-mon[37153]: pgmap v6481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11364.622261 secs 2024-07-27T09:59:02.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:02.905+0000 7fe7b67fc640 1 -- 172.21.15.29:0/827749814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fe7a8005b70 con 0x7fe7c01023f0 2024-07-27T09:59:02.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:02.906+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 <== mon.0 v2:172.21.15.29:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fe7a8005b70 con 0x7fe7c01023f0 2024-07-27T09:59:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:02 smithi047 ceph-mon[36835]: pgmap v6481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11364.622261 secs 2024-07-27T09:59:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:02 smithi029 ceph-mon[32610]: pgmap v6481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11364.622261 secs 2024-07-27T09:59:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:04 smithi077 ceph-mon[37153]: pgmap v6482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:04 smithi047 ceph-mon[36835]: pgmap v6482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:04 smithi029 ceph-mon[32610]: pgmap v6482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:06 smithi077 ceph-mon[37153]: pgmap v6483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:59:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:06 smithi047 ceph-mon[36835]: pgmap v6483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:59:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:06 smithi029 ceph-mon[32610]: pgmap v6483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:59:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11369.622356 secs 2024-07-27T09:59:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11369.622356 secs 2024-07-27T09:59:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11369.622356 secs 2024-07-27T09:59:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:08 smithi077 ceph-mon[37153]: pgmap v6484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:59:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:08 smithi047 ceph-mon[36835]: pgmap v6484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:59:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:08 smithi029 ceph-mon[32610]: pgmap v6484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:59:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:10 smithi077 ceph-mon[37153]: pgmap v6485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:10 smithi047 ceph-mon[36835]: pgmap v6485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:10 smithi029 ceph-mon[32610]: pgmap v6485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:12 smithi077 ceph-mon[37153]: pgmap v6486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11374.622478 secs 2024-07-27T09:59:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:12 smithi047 ceph-mon[36835]: pgmap v6486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11374.622478 secs 2024-07-27T09:59:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:12 smithi029 ceph-mon[32610]: pgmap v6486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11374.622478 secs 2024-07-27T09:59:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:14 smithi077 ceph-mon[37153]: pgmap v6487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:14 smithi047 ceph-mon[36835]: pgmap v6487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:14 smithi029 ceph-mon[32610]: pgmap v6487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:16 smithi077 ceph-mon[37153]: pgmap v6488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:59:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:16 smithi047 ceph-mon[36835]: pgmap v6488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:59:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:16 smithi029 ceph-mon[32610]: pgmap v6488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:59:17.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:17.046+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:59:17.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:17.046+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:59:17.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:17.046+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:59:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11379.622576 secs 2024-07-27T09:59:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11379.622576 secs 2024-07-27T09:59:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11379.622576 secs 2024-07-27T09:59:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:18 smithi077 ceph-mon[37153]: pgmap v6489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:59:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:18 smithi047 ceph-mon[36835]: pgmap v6489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:59:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:18 smithi029 ceph-mon[32610]: pgmap v6489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:59:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:20 smithi077 ceph-mon[37153]: pgmap v6490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:20 smithi047 ceph-mon[36835]: pgmap v6490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:20 smithi029 ceph-mon[32610]: pgmap v6490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:22 smithi077 ceph-mon[37153]: pgmap v6491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11384.622677 secs 2024-07-27T09:59:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:22 smithi047 ceph-mon[36835]: pgmap v6491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11384.622677 secs 2024-07-27T09:59:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:22 smithi029 ceph-mon[32610]: pgmap v6491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11384.622677 secs 2024-07-27T09:59:24.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:24 smithi077 ceph-mon[37153]: pgmap v6492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:24 smithi047 ceph-mon[36835]: pgmap v6492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:24 smithi029 ceph-mon[32610]: pgmap v6492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:26 smithi077 ceph-mon[37153]: pgmap v6493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:59:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:26 smithi047 ceph-mon[36835]: pgmap v6493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:59:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:26 smithi029 ceph-mon[32610]: pgmap v6493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:59:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11389.622814 secs 2024-07-27T09:59:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11389.622814 secs 2024-07-27T09:59:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11389.622814 secs 2024-07-27T09:59:28.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:28 smithi077 ceph-mon[37153]: pgmap v6494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:59:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:28 smithi047 ceph-mon[36835]: pgmap v6494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:59:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:28 smithi029 ceph-mon[32610]: pgmap v6494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:59:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:30 smithi077 ceph-mon[37153]: pgmap v6495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:30 smithi047 ceph-mon[36835]: pgmap v6495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:30 smithi029 ceph-mon[32610]: pgmap v6495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:32.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:32.061+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:59:32.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:32.061+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:59:32.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:32.061+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:59:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:32 smithi077 ceph-mon[37153]: pgmap v6496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11394.622984 secs 2024-07-27T09:59:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:32 smithi047 ceph-mon[36835]: pgmap v6496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11394.622984 secs 2024-07-27T09:59:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:32 smithi029 ceph-mon[32610]: pgmap v6496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11394.622984 secs 2024-07-27T09:59:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:34 smithi077 ceph-mon[37153]: pgmap v6497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:34 smithi047 ceph-mon[36835]: pgmap v6497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:34 smithi029 ceph-mon[32610]: pgmap v6497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:36 smithi077 ceph-mon[37153]: pgmap v6498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:59:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:36 smithi047 ceph-mon[36835]: pgmap v6498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:59:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:36 smithi029 ceph-mon[32610]: pgmap v6498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T09:59:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11399.623105 secs 2024-07-27T09:59:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11399.623105 secs 2024-07-27T09:59:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11399.623105 secs 2024-07-27T09:59:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:38 smithi047 ceph-mon[36835]: pgmap v6499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:59:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:38 smithi029 ceph-mon[32610]: pgmap v6499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:59:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:38 smithi077 ceph-mon[37153]: pgmap v6499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T09:59:40.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:40 smithi047 ceph-mon[36835]: pgmap v6500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:40 smithi029 ceph-mon[32610]: pgmap v6500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:40 smithi077 ceph-mon[37153]: pgmap v6500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T09:59:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:59:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:59:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T09:59:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:42 smithi077 ceph-mon[37153]: pgmap v6501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11404.623173 secs 2024-07-27T09:59:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:59:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:59:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:42 smithi077 ceph-mon[37153]: pgmap v6502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:59:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:42 smithi077 ceph-mon[37153]: pgmap v6503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T09:59:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:42 smithi047 ceph-mon[36835]: pgmap v6501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11404.623173 secs 2024-07-27T09:59:43.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:59:43.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:59:43.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:42 smithi047 ceph-mon[36835]: pgmap v6502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:59:43.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:42 smithi047 ceph-mon[36835]: pgmap v6503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T09:59:43.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:42 smithi029 ceph-mon[32610]: pgmap v6501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11404.623173 secs 2024-07-27T09:59:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T09:59:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T09:59:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:42 smithi029 ceph-mon[32610]: pgmap v6502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T09:59:43.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:42 smithi029 ceph-mon[32610]: pgmap v6503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T09:59:43.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T09:59:44.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:44.359+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:59:44.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:44.359+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:59:44.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:44.360+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T09:59:44.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:44.360+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T09:59:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:45 smithi077 ceph-mon[37153]: pgmap v6504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:59:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:45 smithi047 ceph-mon[36835]: pgmap v6504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:59:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:45 smithi029 ceph-mon[32610]: pgmap v6504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T09:59:47.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:47.077+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T09:59:47.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:47.077+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T09:59:47.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:47.077+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T09:59:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:47 smithi029 ceph-mon[32610]: pgmap v6505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:59:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:47 smithi077 ceph-mon[37153]: pgmap v6505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:59:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:47 smithi047 ceph-mon[36835]: pgmap v6505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T09:59:47.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:47.912+0000 7fe7b67fc640 1 -- 172.21.15.29:0/827749814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7a8005ec0 con 0x7fe7c01023f0 2024-07-27T09:59:47.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T09:59:47.912+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 <== mon.0 v2:172.21.15.29:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fe7a8005ec0 con 0x7fe7c01023f0 2024-07-27T09:59:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11409.623266 secs 2024-07-27T09:59:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11409.623266 secs 2024-07-27T09:59:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11409.623266 secs 2024-07-27T09:59:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:49 smithi077 ceph-mon[37153]: pgmap v6506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T09:59:49.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:49 smithi047 ceph-mon[36835]: pgmap v6506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T09:59:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:49 smithi029 ceph-mon[32610]: pgmap v6506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T09:59:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:51 smithi077 ceph-mon[37153]: pgmap v6507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:59:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:51 smithi047 ceph-mon[36835]: pgmap v6507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:59:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:51 smithi029 ceph-mon[32610]: pgmap v6507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:59:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11414.623392 secs 2024-07-27T09:59:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:53 smithi077 ceph-mon[37153]: pgmap v6508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:59:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11414.623392 secs 2024-07-27T09:59:53.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:53 smithi047 ceph-mon[36835]: pgmap v6508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:59:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11414.623392 secs 2024-07-27T09:59:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:53 smithi029 ceph-mon[32610]: pgmap v6508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T09:59:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:55 smithi077 ceph-mon[37153]: pgmap v6509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:59:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:55 smithi047 ceph-mon[36835]: pgmap v6509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:59:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:55 smithi029 ceph-mon[32610]: pgmap v6509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T09:59:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:57 smithi077 ceph-mon[37153]: pgmap v6510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:57 smithi047 ceph-mon[36835]: pgmap v6510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:57 smithi029 ceph-mon[32610]: pgmap v6510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T09:59:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11419.623511 secs 2024-07-27T09:59:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11419.623511 secs 2024-07-27T09:59:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11419.623511 secs 2024-07-27T09:59:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 09:59:59 smithi077 ceph-mon[37153]: pgmap v6511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 09:59:59 smithi047 ceph-mon[36835]: pgmap v6511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T09:59:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 09:59:59 smithi029 ceph-mon[32610]: pgmap v6511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:00:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:00:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:00:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:01 smithi077 ceph-mon[37153]: pgmap v6512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:00:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:01 smithi047 ceph-mon[36835]: pgmap v6512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:00:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:01 smithi029 ceph-mon[32610]: pgmap v6512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:00:02.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:00:02.092+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:00:02.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:00:02.092+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:00:02.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:00:02.092+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:00:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11424.623634 secs 2024-07-27T10:00:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:03 smithi077 ceph-mon[37153]: pgmap v6513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11424.623634 secs 2024-07-27T10:00:03.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:03 smithi047 ceph-mon[36835]: pgmap v6513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11424.623634 secs 2024-07-27T10:00:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:03 smithi029 ceph-mon[32610]: pgmap v6513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:05 smithi077 ceph-mon[37153]: pgmap v6514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:00:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:05 smithi047 ceph-mon[36835]: pgmap v6514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:00:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:05 smithi029 ceph-mon[32610]: pgmap v6514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:00:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:07 smithi077 ceph-mon[37153]: pgmap v6515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:00:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:07 smithi047 ceph-mon[36835]: pgmap v6515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:00:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:07 smithi029 ceph-mon[32610]: pgmap v6515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:00:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11429.623765 secs 2024-07-27T10:00:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11429.623765 secs 2024-07-27T10:00:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11429.623765 secs 2024-07-27T10:00:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:09 smithi077 ceph-mon[37153]: pgmap v6516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:09 smithi047 ceph-mon[36835]: pgmap v6516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:09 smithi029 ceph-mon[32610]: pgmap v6516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:11 smithi047 ceph-mon[36835]: pgmap v6517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:00:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:11 smithi029 ceph-mon[32610]: pgmap v6517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:00:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:11 smithi077 ceph-mon[37153]: pgmap v6517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:00:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11434.623868 secs 2024-07-27T10:00:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:13 smithi047 ceph-mon[36835]: pgmap v6518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11434.623868 secs 2024-07-27T10:00:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:13 smithi029 ceph-mon[32610]: pgmap v6518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11434.623868 secs 2024-07-27T10:00:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:13 smithi077 ceph-mon[37153]: pgmap v6518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:15 smithi047 ceph-mon[36835]: pgmap v6519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:00:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:15 smithi029 ceph-mon[32610]: pgmap v6519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:00:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:15 smithi077 ceph-mon[37153]: pgmap v6519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:00:17.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:00:17.100+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:00:17.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:00:17.100+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:00:17.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:00:17.100+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:00:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:17 smithi047 ceph-mon[36835]: pgmap v6520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:00:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:17 smithi029 ceph-mon[32610]: pgmap v6520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:00:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:17 smithi077 ceph-mon[37153]: pgmap v6520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:00:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11439.623967 secs 2024-07-27T10:00:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:18 smithi047 ceph-mon[36835]: pgmap v6521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11439.623967 secs 2024-07-27T10:00:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:18 smithi029 ceph-mon[32610]: pgmap v6521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11439.623967 secs 2024-07-27T10:00:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:18 smithi077 ceph-mon[37153]: pgmap v6521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:21 smithi077 ceph-mon[37153]: pgmap v6522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:00:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:21 smithi047 ceph-mon[36835]: pgmap v6522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:00:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:21 smithi029 ceph-mon[32610]: pgmap v6522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:00:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11444.624032 secs 2024-07-27T10:00:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:23 smithi077 ceph-mon[37153]: pgmap v6523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11444.624032 secs 2024-07-27T10:00:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:23 smithi047 ceph-mon[36835]: pgmap v6523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11444.624032 secs 2024-07-27T10:00:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:23 smithi029 ceph-mon[32610]: pgmap v6523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:25 smithi077 ceph-mon[37153]: pgmap v6524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:00:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:25 smithi047 ceph-mon[36835]: pgmap v6524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:00:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:25 smithi029 ceph-mon[32610]: pgmap v6524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:00:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:27 smithi077 ceph-mon[37153]: pgmap v6525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:00:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:27 smithi047 ceph-mon[36835]: pgmap v6525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:00:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:27 smithi029 ceph-mon[32610]: pgmap v6525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:00:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11449.624164 secs 2024-07-27T10:00:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11449.624164 secs 2024-07-27T10:00:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11449.624164 secs 2024-07-27T10:00:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:29 smithi077 ceph-mon[37153]: pgmap v6526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:29 smithi047 ceph-mon[36835]: pgmap v6526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:29 smithi029 ceph-mon[32610]: pgmap v6526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:31 smithi077 ceph-mon[37153]: pgmap v6527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:00:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:31 smithi047 ceph-mon[36835]: pgmap v6527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:00:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:31 smithi029 ceph-mon[32610]: pgmap v6527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:00:32.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:00:32.106+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:00:32.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:00:32.106+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:00:32.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:00:32.106+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:00:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11454.624231 secs 2024-07-27T10:00:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:33 smithi077 ceph-mon[37153]: pgmap v6528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:33.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11454.624231 secs 2024-07-27T10:00:33.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:33 smithi047 ceph-mon[36835]: pgmap v6528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11454.624231 secs 2024-07-27T10:00:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:33 smithi029 ceph-mon[32610]: pgmap v6528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:35 smithi029 ceph-mon[32610]: pgmap v6529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:00:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:35 smithi077 ceph-mon[37153]: pgmap v6529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:00:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:35 smithi047 ceph-mon[36835]: pgmap v6529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:00:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:37 smithi077 ceph-mon[37153]: pgmap v6530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:00:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:37 smithi047 ceph-mon[36835]: pgmap v6530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:00:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:37 smithi029 ceph-mon[32610]: pgmap v6530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:00:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11459.624377 secs 2024-07-27T10:00:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11459.624377 secs 2024-07-27T10:00:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11459.624377 secs 2024-07-27T10:00:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:39 smithi077 ceph-mon[37153]: pgmap v6531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:39 smithi047 ceph-mon[36835]: pgmap v6531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:39 smithi029 ceph-mon[32610]: pgmap v6531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:00:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:41 smithi077 ceph-mon[37153]: pgmap v6532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:00:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:41 smithi047 ceph-mon[36835]: pgmap v6532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:00:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:41 smithi029 ceph-mon[32610]: pgmap v6532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:00:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:42 smithi047 ceph-mon[36835]: pgmap v6533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:00:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:42 smithi029 ceph-mon[32610]: pgmap v6533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:00:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:42 smithi077 ceph-mon[37153]: pgmap v6533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:00:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:00:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11464.624512 secs 2024-07-27T10:00:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:00:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:00:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:43 smithi047 ceph-mon[36835]: pgmap v6534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:00:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:43 smithi047 ceph-mon[36835]: pgmap v6535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:00:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:00:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11464.624512 secs 2024-07-27T10:00:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:00:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:00:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:43 smithi029 ceph-mon[32610]: pgmap v6534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:00:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:43 smithi029 ceph-mon[32610]: pgmap v6535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:00:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:00:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11464.624512 secs 2024-07-27T10:00:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:00:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:00:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:43 smithi077 ceph-mon[37153]: pgmap v6534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:00:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:43 smithi077 ceph-mon[37153]: pgmap v6535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:00:43.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:00:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:45 smithi047 ceph-mon[36835]: pgmap v6536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:00:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:45 smithi029 ceph-mon[32610]: pgmap v6536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:00:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:45 smithi077 ceph-mon[37153]: pgmap v6536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:00:47.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:00:47.120+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:00:47.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:00:47.120+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:00:47.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:00:47.120+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:00:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:48 smithi077 ceph-mon[37153]: pgmap v6537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:00:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11469.624637 secs 2024-07-27T10:00:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:48 smithi047 ceph-mon[36835]: pgmap v6537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:00:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11469.624637 secs 2024-07-27T10:00:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:48 smithi029 ceph-mon[32610]: pgmap v6537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:00:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11469.624637 secs 2024-07-27T10:00:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:50 smithi077 ceph-mon[37153]: pgmap v6538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:00:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:50 smithi047 ceph-mon[36835]: pgmap v6538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:00:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:50 smithi029 ceph-mon[32610]: pgmap v6538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:00:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:52 smithi077 ceph-mon[37153]: pgmap v6539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:00:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:52 smithi047 ceph-mon[36835]: pgmap v6539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:00:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:52 smithi029 ceph-mon[32610]: pgmap v6539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:00:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11474.624775 secs 2024-07-27T10:00:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11474.624775 secs 2024-07-27T10:00:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11474.624775 secs 2024-07-27T10:00:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:54 smithi077 ceph-mon[37153]: pgmap v6540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:00:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:54 smithi047 ceph-mon[36835]: pgmap v6540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:00:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:54 smithi029 ceph-mon[32610]: pgmap v6540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:00:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:56 smithi077 ceph-mon[37153]: pgmap v6541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:00:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:56 smithi047 ceph-mon[36835]: pgmap v6541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:00:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:56 smithi029 ceph-mon[32610]: pgmap v6541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:00:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:58 smithi077 ceph-mon[37153]: pgmap v6542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:00:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:00:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11479.624903 secs 2024-07-27T10:00:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:58 smithi047 ceph-mon[36835]: pgmap v6542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:00:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:00:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11479.624903 secs 2024-07-27T10:00:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:58 smithi029 ceph-mon[32610]: pgmap v6542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:00:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:00:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11479.624903 secs 2024-07-27T10:01:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:00 smithi077 ceph-mon[37153]: pgmap v6543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:00 smithi047 ceph-mon[36835]: pgmap v6543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:00 smithi029 ceph-mon[32610]: pgmap v6543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:02.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:02.131+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:01:02.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:02.131+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:01:02.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:02.131+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:01:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:02 smithi047 ceph-mon[36835]: pgmap v6544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:02 smithi077 ceph-mon[37153]: pgmap v6544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:02 smithi029 ceph-mon[32610]: pgmap v6544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11484.625018 secs 2024-07-27T10:01:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11484.625018 secs 2024-07-27T10:01:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11484.625018 secs 2024-07-27T10:01:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:04 smithi077 ceph-mon[37153]: pgmap v6545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:04 smithi047 ceph-mon[36835]: pgmap v6545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:04 smithi029 ceph-mon[32610]: pgmap v6545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:06 smithi029 ceph-mon[32610]: pgmap v6546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:06 smithi077 ceph-mon[37153]: pgmap v6546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:06 smithi047 ceph-mon[36835]: pgmap v6546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:08 smithi077 ceph-mon[37153]: pgmap v6547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:01:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11489.625084 secs 2024-07-27T10:01:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:08 smithi047 ceph-mon[36835]: pgmap v6547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:01:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11489.625084 secs 2024-07-27T10:01:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:08 smithi029 ceph-mon[32610]: pgmap v6547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:01:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11489.625084 secs 2024-07-27T10:01:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:10 smithi077 ceph-mon[37153]: pgmap v6548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:10 smithi047 ceph-mon[36835]: pgmap v6548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:10 smithi029 ceph-mon[32610]: pgmap v6548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:12 smithi077 ceph-mon[37153]: pgmap v6549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:12 smithi047 ceph-mon[36835]: pgmap v6549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:12 smithi029 ceph-mon[32610]: pgmap v6549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11494.625204 secs 2024-07-27T10:01:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11494.625204 secs 2024-07-27T10:01:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11494.625204 secs 2024-07-27T10:01:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:14 smithi077 ceph-mon[37153]: pgmap v6550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:14 smithi047 ceph-mon[36835]: pgmap v6550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:14 smithi029 ceph-mon[32610]: pgmap v6550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:16 smithi077 ceph-mon[37153]: pgmap v6551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:16 smithi047 ceph-mon[36835]: pgmap v6551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:16 smithi029 ceph-mon[32610]: pgmap v6551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:17.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:17.132+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:01:17.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:17.132+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:01:17.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:17.132+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:01:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:18 smithi077 ceph-mon[37153]: pgmap v6552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:01:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11499.625342 secs 2024-07-27T10:01:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:18 smithi047 ceph-mon[36835]: pgmap v6552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:01:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11499.625342 secs 2024-07-27T10:01:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:18 smithi029 ceph-mon[32610]: pgmap v6552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:01:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11499.625342 secs 2024-07-27T10:01:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:20 smithi077 ceph-mon[37153]: pgmap v6553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:20.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:20 smithi047 ceph-mon[36835]: pgmap v6553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:20 smithi029 ceph-mon[32610]: pgmap v6553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:22 smithi077 ceph-mon[37153]: pgmap v6554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:22 smithi047 ceph-mon[36835]: pgmap v6554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:22 smithi029 ceph-mon[32610]: pgmap v6554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11504.625494 secs 2024-07-27T10:01:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11504.625494 secs 2024-07-27T10:01:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11504.625494 secs 2024-07-27T10:01:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:24 smithi077 ceph-mon[37153]: pgmap v6555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:24 smithi047 ceph-mon[36835]: pgmap v6555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:24 smithi029 ceph-mon[32610]: pgmap v6555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:26 smithi077 ceph-mon[37153]: pgmap v6556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:26 smithi047 ceph-mon[36835]: pgmap v6556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:26 smithi029 ceph-mon[32610]: pgmap v6556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:28 smithi077 ceph-mon[37153]: pgmap v6557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:01:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11509.625587 secs 2024-07-27T10:01:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:28 smithi047 ceph-mon[36835]: pgmap v6557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:01:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11509.625587 secs 2024-07-27T10:01:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:28 smithi029 ceph-mon[32610]: pgmap v6557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:01:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11509.625587 secs 2024-07-27T10:01:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:30 smithi077 ceph-mon[37153]: pgmap v6558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:30 smithi047 ceph-mon[36835]: pgmap v6558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:30 smithi029 ceph-mon[32610]: pgmap v6558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:32.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:32.146+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:01:32.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:32.146+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:01:32.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:32.146+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:01:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:32 smithi047 ceph-mon[36835]: pgmap v6559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:32 smithi029 ceph-mon[32610]: pgmap v6559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:32 smithi077 ceph-mon[37153]: pgmap v6559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11514.625702 secs 2024-07-27T10:01:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11514.625702 secs 2024-07-27T10:01:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11514.625702 secs 2024-07-27T10:01:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:34 smithi047 ceph-mon[36835]: pgmap v6560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:34 smithi029 ceph-mon[32610]: pgmap v6560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:34 smithi077 ceph-mon[37153]: pgmap v6560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:36 smithi047 ceph-mon[36835]: pgmap v6561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:36 smithi029 ceph-mon[32610]: pgmap v6561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:36 smithi077 ceph-mon[37153]: pgmap v6561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:38.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:38 smithi077 ceph-mon[37153]: pgmap v6562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:01:38.410 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11519.625824 secs 2024-07-27T10:01:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:38 smithi047 ceph-mon[36835]: pgmap v6562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:01:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11519.625824 secs 2024-07-27T10:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:38 smithi029 ceph-mon[32610]: pgmap v6562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:01:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11519.625824 secs 2024-07-27T10:01:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:40 smithi047 ceph-mon[36835]: pgmap v6563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:40 smithi029 ceph-mon[32610]: pgmap v6563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:40 smithi077 ceph-mon[37153]: pgmap v6563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:01:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:42 smithi047 ceph-mon[36835]: pgmap v6564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:42 smithi029 ceph-mon[32610]: pgmap v6564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:42 smithi077 ceph-mon[37153]: pgmap v6564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:43 smithi047 ceph-mon[36835]: 1 slow requests, 1 included below; oldest blocked for > 11524.625949 secs 2024-07-27T10:01:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:43 smithi047 ceph-mon[36835]: slow request 11524.625948 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T10:01:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:01:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:43 smithi029 ceph-mon[32610]: 1 slow requests, 1 included below; oldest blocked for > 11524.625949 secs 2024-07-27T10:01:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:43 smithi029 ceph-mon[32610]: slow request 11524.625948 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T10:01:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:01:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:43 smithi077 ceph-mon[37153]: 1 slow requests, 1 included below; oldest blocked for > 11524.625949 secs 2024-07-27T10:01:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:43 smithi077 ceph-mon[37153]: slow request 11524.625948 seconds old, received at 2024-07-27T06:49:37.287006+0000: client_request(client.15402:11408 rename #0x10000000299/builddefs #0x100000015b7/out 2024-07-27T06:49:37.285626+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-27T10:01:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:01:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:44 smithi047 ceph-mon[36835]: pgmap v6565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:01:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:01:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:44 smithi047 ceph-mon[36835]: pgmap v6566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:01:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:44 smithi047 ceph-mon[36835]: pgmap v6567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:01:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:01:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:44 smithi029 ceph-mon[32610]: pgmap v6565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:44.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:01:44.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:01:44.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:44 smithi029 ceph-mon[32610]: pgmap v6566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:01:44.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:44 smithi029 ceph-mon[32610]: pgmap v6567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:01:44.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:01:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:44 smithi077 ceph-mon[37153]: pgmap v6565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:01:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:01:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:01:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:44 smithi077 ceph-mon[37153]: pgmap v6566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:01:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:44 smithi077 ceph-mon[37153]: pgmap v6567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:01:44.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:01:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:45 smithi047 ceph-mon[36835]: pgmap v6568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:01:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:45 smithi029 ceph-mon[32610]: pgmap v6568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:01:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:45 smithi077 ceph-mon[37153]: pgmap v6568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:01:47.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:47.162+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:01:47.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:47.162+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:01:47.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:47.162+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:01:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:47 smithi047 ceph-mon[36835]: pgmap v6569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:01:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:47 smithi029 ceph-mon[32610]: pgmap v6569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:01:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:47 smithi077 ceph-mon[37153]: pgmap v6569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:01:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11529.626077 secs 2024-07-27T10:01:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11529.626077 secs 2024-07-27T10:01:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11529.626077 secs 2024-07-27T10:01:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:49 smithi047 ceph-mon[36835]: pgmap v6570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:01:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:49 smithi029 ceph-mon[32610]: pgmap v6570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:01:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:49 smithi077 ceph-mon[37153]: pgmap v6570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:01:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:51 smithi047 ceph-mon[36835]: pgmap v6571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:01:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:51 smithi029 ceph-mon[32610]: pgmap v6571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:01:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:51 smithi077 ceph-mon[37153]: pgmap v6571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:01:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11534.626160 secs 2024-07-27T10:01:52.935 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11534.626160 secs 2024-07-27T10:01:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11534.626160 secs 2024-07-27T10:01:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:53 smithi077 ceph-mon[37153]: pgmap v6572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:01:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:53 smithi047 ceph-mon[36835]: pgmap v6572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:01:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:53 smithi029 ceph-mon[32610]: pgmap v6572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:01:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:55 smithi047 ceph-mon[36835]: pgmap v6573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:55 smithi029 ceph-mon[32610]: pgmap v6573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:55 smithi077 ceph-mon[37153]: pgmap v6573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:01:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:57 smithi047 ceph-mon[36835]: pgmap v6574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:01:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:57 smithi029 ceph-mon[32610]: pgmap v6574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:01:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:57 smithi077 ceph-mon[37153]: pgmap v6574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:01:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11539.626245 secs 2024-07-27T10:01:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11539.626245 secs 2024-07-27T10:01:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11539.626245 secs 2024-07-27T10:01:59.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:59.381+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:01:59.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:59.381+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:01:59.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:59.382+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:01:59.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:01:59.382+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:01:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:01:59 smithi047 ceph-mon[36835]: pgmap v6575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:01:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:01:59 smithi029 ceph-mon[32610]: pgmap v6575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:01:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:01:59 smithi077 ceph-mon[37153]: pgmap v6575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:01.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:01 smithi047 ceph-mon[36835]: pgmap v6576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:01 smithi029 ceph-mon[32610]: pgmap v6576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:01 smithi077 ceph-mon[37153]: pgmap v6576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:02.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:02.172+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:02:02.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:02.172+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:02:02.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:02.172+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:02:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11544.626372 secs 2024-07-27T10:02:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11544.626372 secs 2024-07-27T10:02:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11544.626372 secs 2024-07-27T10:02:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:03 smithi077 ceph-mon[37153]: pgmap v6577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:03 smithi047 ceph-mon[36835]: pgmap v6577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:03 smithi029 ceph-mon[32610]: pgmap v6577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:05 smithi047 ceph-mon[36835]: pgmap v6578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:05 smithi029 ceph-mon[32610]: pgmap v6578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:05 smithi077 ceph-mon[37153]: pgmap v6578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:08.296 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11549.626495 secs 2024-07-27T10:02:08.296 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:08 smithi029 ceph-mon[32610]: pgmap v6579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:02:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11549.626495 secs 2024-07-27T10:02:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:08 smithi077 ceph-mon[37153]: pgmap v6579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:02:08.399 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11549.626495 secs 2024-07-27T10:02:08.400 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:08 smithi047 ceph-mon[36835]: pgmap v6579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:02:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:09 smithi047 ceph-mon[36835]: pgmap v6580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:09 smithi029 ceph-mon[32610]: pgmap v6580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:09 smithi077 ceph-mon[37153]: pgmap v6580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:11 smithi047 ceph-mon[36835]: pgmap v6581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:11 smithi029 ceph-mon[32610]: pgmap v6581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:11 smithi077 ceph-mon[37153]: pgmap v6581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11554.626615 secs 2024-07-27T10:02:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11554.626615 secs 2024-07-27T10:02:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11554.626615 secs 2024-07-27T10:02:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:14 smithi077 ceph-mon[37153]: pgmap v6582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:14 smithi047 ceph-mon[36835]: pgmap v6582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:14 smithi029 ceph-mon[32610]: pgmap v6582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:15 smithi047 ceph-mon[36835]: pgmap v6583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:15 smithi029 ceph-mon[32610]: pgmap v6583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:15 smithi077 ceph-mon[37153]: pgmap v6583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:17.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:17.188+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:02:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:17.188+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:02:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:17.188+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:02:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:17 smithi029 ceph-mon[32610]: pgmap v6584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:02:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:17 smithi047 ceph-mon[36835]: pgmap v6584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:02:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:17 smithi077 ceph-mon[37153]: pgmap v6584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:02:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11559.626730 secs 2024-07-27T10:02:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11559.626730 secs 2024-07-27T10:02:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11559.626730 secs 2024-07-27T10:02:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:19 smithi047 ceph-mon[36835]: pgmap v6585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:19 smithi029 ceph-mon[32610]: pgmap v6585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:19 smithi077 ceph-mon[37153]: pgmap v6585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:21 smithi047 ceph-mon[36835]: pgmap v6586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:21 smithi029 ceph-mon[32610]: pgmap v6586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:21 smithi077 ceph-mon[37153]: pgmap v6586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11564.626879 secs 2024-07-27T10:02:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11564.626879 secs 2024-07-27T10:02:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11564.626879 secs 2024-07-27T10:02:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:23 smithi077 ceph-mon[37153]: pgmap v6587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:23 smithi047 ceph-mon[36835]: pgmap v6587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:23 smithi029 ceph-mon[32610]: pgmap v6587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:25 smithi047 ceph-mon[36835]: pgmap v6588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:25.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:25 smithi029 ceph-mon[32610]: pgmap v6588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:25 smithi077 ceph-mon[37153]: pgmap v6588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:27 smithi047 ceph-mon[36835]: pgmap v6589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:02:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:27 smithi029 ceph-mon[32610]: pgmap v6589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:02:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:27 smithi077 ceph-mon[37153]: pgmap v6589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:02:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11569.626969 secs 2024-07-27T10:02:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11569.626969 secs 2024-07-27T10:02:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11569.626969 secs 2024-07-27T10:02:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:29 smithi029 ceph-mon[32610]: pgmap v6590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:29 smithi077 ceph-mon[37153]: pgmap v6590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:29 smithi047 ceph-mon[36835]: pgmap v6590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:31 smithi047 ceph-mon[36835]: pgmap v6591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:31 smithi029 ceph-mon[32610]: pgmap v6591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:31 smithi077 ceph-mon[37153]: pgmap v6591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:32.202+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:02:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:32.202+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:02:32.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:32.202+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:02:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11574.627058 secs 2024-07-27T10:02:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11574.627058 secs 2024-07-27T10:02:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11574.627058 secs 2024-07-27T10:02:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:33 smithi077 ceph-mon[37153]: pgmap v6592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:33 smithi047 ceph-mon[36835]: pgmap v6592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:33 smithi029 ceph-mon[32610]: pgmap v6592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:35 smithi047 ceph-mon[36835]: pgmap v6593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:35 smithi029 ceph-mon[32610]: pgmap v6593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:35 smithi077 ceph-mon[37153]: pgmap v6593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:37 smithi047 ceph-mon[36835]: pgmap v6594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:02:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:37 smithi029 ceph-mon[32610]: pgmap v6594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:02:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:37 smithi077 ceph-mon[37153]: pgmap v6594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:02:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11579.627195 secs 2024-07-27T10:02:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11579.627195 secs 2024-07-27T10:02:38.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11579.627195 secs 2024-07-27T10:02:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:39 smithi029 ceph-mon[32610]: pgmap v6595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:39 smithi077 ceph-mon[37153]: pgmap v6595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:39 smithi047 ceph-mon[36835]: pgmap v6595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:02:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:41 smithi047 ceph-mon[36835]: pgmap v6596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:41 smithi029 ceph-mon[32610]: pgmap v6596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:41 smithi077 ceph-mon[37153]: pgmap v6596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:02:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11584.627325 secs 2024-07-27T10:02:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11584.627325 secs 2024-07-27T10:02:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11584.627325 secs 2024-07-27T10:02:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:43 smithi077 ceph-mon[37153]: pgmap v6597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:02:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:43 smithi047 ceph-mon[36835]: pgmap v6597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:02:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:43 smithi029 ceph-mon[32610]: pgmap v6597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:02:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:02:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:02:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:02:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:44 smithi077 ceph-mon[37153]: pgmap v6598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:02:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:44 smithi077 ceph-mon[37153]: pgmap v6599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:02:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:02:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:02:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:02:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:44 smithi047 ceph-mon[36835]: pgmap v6598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:02:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:44 smithi047 ceph-mon[36835]: pgmap v6599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:02:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:02:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:02:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:02:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:44 smithi029 ceph-mon[32610]: pgmap v6598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:02:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:44 smithi029 ceph-mon[32610]: pgmap v6599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:02:44.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:02:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:47.210+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:02:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:47.210+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:02:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:47.210+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:02:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:46 smithi077 ceph-mon[37153]: pgmap v6600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:02:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:46 smithi047 ceph-mon[36835]: pgmap v6600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:02:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:46 smithi029 ceph-mon[32610]: pgmap v6600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:02:47.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:47.940+0000 7fe7b67fc640 1 -- 172.21.15.29:0/827749814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7a8006090 con 0x7fe7c01023f0 2024-07-27T10:02:47.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:02:47.940+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 <== mon.0 v2:172.21.15.29:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fe7a8006090 con 0x7fe7c01023f0 2024-07-27T10:02:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11589.627450 secs 2024-07-27T10:02:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11589.627450 secs 2024-07-27T10:02:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11589.627450 secs 2024-07-27T10:02:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:48 smithi077 ceph-mon[37153]: pgmap v6601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:02:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:48 smithi047 ceph-mon[36835]: pgmap v6601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:02:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:48 smithi029 ceph-mon[32610]: pgmap v6601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:02:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:50 smithi077 ceph-mon[37153]: pgmap v6602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:02:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:50 smithi047 ceph-mon[36835]: pgmap v6602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:02:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:50 smithi029 ceph-mon[32610]: pgmap v6602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:02:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11594.627572 secs 2024-07-27T10:02:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:52 smithi077 ceph-mon[37153]: pgmap v6603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:02:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11594.627572 secs 2024-07-27T10:02:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:52 smithi047 ceph-mon[36835]: pgmap v6603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:02:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11594.627572 secs 2024-07-27T10:02:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:52 smithi029 ceph-mon[32610]: pgmap v6603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:02:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:54 smithi029 ceph-mon[32610]: pgmap v6604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:02:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:54 smithi077 ceph-mon[37153]: pgmap v6604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:02:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:54 smithi047 ceph-mon[36835]: pgmap v6604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:02:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:56 smithi077 ceph-mon[37153]: pgmap v6605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:56 smithi047 ceph-mon[36835]: pgmap v6605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:56 smithi029 ceph-mon[32610]: pgmap v6605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:02:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11599.627673 secs 2024-07-27T10:02:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11599.627673 secs 2024-07-27T10:02:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11599.627673 secs 2024-07-27T10:02:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:02:59 smithi077 ceph-mon[37153]: pgmap v6606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:02:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:02:59 smithi047 ceph-mon[36835]: pgmap v6606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:02:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:02:59 smithi029 ceph-mon[32610]: pgmap v6606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:01 smithi077 ceph-mon[37153]: pgmap v6607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:01 smithi047 ceph-mon[36835]: pgmap v6607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:01 smithi029 ceph-mon[32610]: pgmap v6607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:02.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:03:02.222+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:03:02.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:03:02.222+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:03:02.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:03:02.222+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:03:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11604.627806 secs 2024-07-27T10:03:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:03 smithi077 ceph-mon[37153]: pgmap v6608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11604.627806 secs 2024-07-27T10:03:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:03 smithi047 ceph-mon[36835]: pgmap v6608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11604.627806 secs 2024-07-27T10:03:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:03 smithi029 ceph-mon[32610]: pgmap v6608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:05.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:05 smithi077 ceph-mon[37153]: pgmap v6609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:05 smithi047 ceph-mon[36835]: pgmap v6609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:05 smithi029 ceph-mon[32610]: pgmap v6609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:07 smithi077 ceph-mon[37153]: pgmap v6610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:07 smithi047 ceph-mon[36835]: pgmap v6610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:07 smithi029 ceph-mon[32610]: pgmap v6610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:08.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11609.627872 secs 2024-07-27T10:03:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11609.627872 secs 2024-07-27T10:03:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11609.627872 secs 2024-07-27T10:03:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:09 smithi077 ceph-mon[37153]: pgmap v6611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:09 smithi047 ceph-mon[36835]: pgmap v6611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:09 smithi029 ceph-mon[32610]: pgmap v6611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:11 smithi077 ceph-mon[37153]: pgmap v6612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:11 smithi047 ceph-mon[36835]: pgmap v6612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:11 smithi029 ceph-mon[32610]: pgmap v6612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11614.628000 secs 2024-07-27T10:03:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:13 smithi077 ceph-mon[37153]: pgmap v6613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11614.628000 secs 2024-07-27T10:03:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:13 smithi047 ceph-mon[36835]: pgmap v6613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11614.628000 secs 2024-07-27T10:03:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:13 smithi029 ceph-mon[32610]: pgmap v6613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:15 smithi077 ceph-mon[37153]: pgmap v6614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:15 smithi047 ceph-mon[36835]: pgmap v6614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:15 smithi029 ceph-mon[32610]: pgmap v6614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:03:17.235+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:03:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:03:17.235+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:03:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:03:17.235+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:03:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:17 smithi077 ceph-mon[37153]: pgmap v6615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:17 smithi047 ceph-mon[36835]: pgmap v6615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:17 smithi029 ceph-mon[32610]: pgmap v6615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11619.628066 secs 2024-07-27T10:03:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11619.628066 secs 2024-07-27T10:03:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11619.628066 secs 2024-07-27T10:03:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:19 smithi077 ceph-mon[37153]: pgmap v6616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:19 smithi047 ceph-mon[36835]: pgmap v6616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:19 smithi029 ceph-mon[32610]: pgmap v6616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:21 smithi077 ceph-mon[37153]: pgmap v6617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:21 smithi047 ceph-mon[36835]: pgmap v6617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:21 smithi029 ceph-mon[32610]: pgmap v6617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11624.628193 secs 2024-07-27T10:03:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:23 smithi077 ceph-mon[37153]: pgmap v6618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11624.628193 secs 2024-07-27T10:03:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:23 smithi047 ceph-mon[36835]: pgmap v6618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11624.628193 secs 2024-07-27T10:03:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:23 smithi029 ceph-mon[32610]: pgmap v6618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:25.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:25 smithi077 ceph-mon[37153]: pgmap v6619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:25 smithi047 ceph-mon[36835]: pgmap v6619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:25 smithi029 ceph-mon[32610]: pgmap v6619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:27 smithi077 ceph-mon[37153]: pgmap v6620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:27 smithi047 ceph-mon[36835]: pgmap v6620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:27 smithi029 ceph-mon[32610]: pgmap v6620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11629.628293 secs 2024-07-27T10:03:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11629.628293 secs 2024-07-27T10:03:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11629.628293 secs 2024-07-27T10:03:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:29 smithi077 ceph-mon[37153]: pgmap v6621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:29 smithi047 ceph-mon[36835]: pgmap v6621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:29 smithi029 ceph-mon[32610]: pgmap v6621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:31 smithi077 ceph-mon[37153]: pgmap v6622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:31 smithi047 ceph-mon[36835]: pgmap v6622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:31 smithi029 ceph-mon[32610]: pgmap v6622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:32.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:03:32.249+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:03:32.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:03:32.249+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:03:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:03:32.249+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:03:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11634.628422 secs 2024-07-27T10:03:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:33 smithi047 ceph-mon[36835]: pgmap v6623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11634.628422 secs 2024-07-27T10:03:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:33 smithi029 ceph-mon[32610]: pgmap v6623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11634.628422 secs 2024-07-27T10:03:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:33 smithi077 ceph-mon[37153]: pgmap v6623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:35 smithi047 ceph-mon[36835]: pgmap v6624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:35 smithi029 ceph-mon[32610]: pgmap v6624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:35 smithi077 ceph-mon[37153]: pgmap v6624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:37.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:37 smithi047 ceph-mon[36835]: pgmap v6625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:37 smithi029 ceph-mon[32610]: pgmap v6625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:37 smithi077 ceph-mon[37153]: pgmap v6625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11639.628548 secs 2024-07-27T10:03:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11639.628548 secs 2024-07-27T10:03:38.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11639.628548 secs 2024-07-27T10:03:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:39 smithi047 ceph-mon[36835]: pgmap v6626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:39 smithi029 ceph-mon[32610]: pgmap v6626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:39 smithi077 ceph-mon[37153]: pgmap v6626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:03:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:41 smithi047 ceph-mon[36835]: pgmap v6627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:41 smithi029 ceph-mon[32610]: pgmap v6627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:41 smithi077 ceph-mon[37153]: pgmap v6627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:03:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11644.628677 secs 2024-07-27T10:03:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:43 smithi047 ceph-mon[36835]: pgmap v6628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11644.628677 secs 2024-07-27T10:03:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:43 smithi029 ceph-mon[32610]: pgmap v6628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11644.628677 secs 2024-07-27T10:03:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:43 smithi077 ceph-mon[37153]: pgmap v6628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:03:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:03:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:03:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:03:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:45 smithi047 ceph-mon[36835]: pgmap v6629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:45.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:03:45.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:03:45.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:03:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:45 smithi029 ceph-mon[32610]: pgmap v6629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:03:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:03:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:03:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:45 smithi077 ceph-mon[37153]: pgmap v6629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:03:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:03:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:03:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:03:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:46 smithi047 ceph-mon[36835]: pgmap v6630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:03:46.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:46 smithi047 ceph-mon[36835]: pgmap v6631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:03:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:46 smithi029 ceph-mon[32610]: pgmap v6630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:03:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:46 smithi029 ceph-mon[32610]: pgmap v6631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:03:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:46 smithi077 ceph-mon[37153]: pgmap v6630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:03:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:46 smithi077 ceph-mon[37153]: pgmap v6631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:03:47.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:03:47.265+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:03:47.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:03:47.265+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:03:47.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:03:47.265+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:03:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:48 smithi047 ceph-mon[36835]: pgmap v6632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:03:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11649.628822 secs 2024-07-27T10:03:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:48 smithi029 ceph-mon[32610]: pgmap v6632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:03:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11649.628822 secs 2024-07-27T10:03:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:48 smithi077 ceph-mon[37153]: pgmap v6632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:03:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11649.628822 secs 2024-07-27T10:03:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:50 smithi047 ceph-mon[36835]: pgmap v6633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:03:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:50 smithi029 ceph-mon[32610]: pgmap v6633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:03:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:50 smithi077 ceph-mon[37153]: pgmap v6633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:03:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:52 smithi047 ceph-mon[36835]: pgmap v6634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:03:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:52 smithi029 ceph-mon[32610]: pgmap v6634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:03:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:52 smithi077 ceph-mon[37153]: pgmap v6634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:03:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11654.628941 secs 2024-07-27T10:03:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11654.628941 secs 2024-07-27T10:03:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11654.628941 secs 2024-07-27T10:03:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:54 smithi047 ceph-mon[36835]: pgmap v6635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:03:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:54 smithi029 ceph-mon[32610]: pgmap v6635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:03:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:54 smithi077 ceph-mon[37153]: pgmap v6635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:03:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:56 smithi047 ceph-mon[36835]: pgmap v6636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:03:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:56 smithi029 ceph-mon[32610]: pgmap v6636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:03:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:56 smithi077 ceph-mon[37153]: pgmap v6636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:03:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:58 smithi047 ceph-mon[36835]: pgmap v6637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:03:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:03:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11659.629066 secs 2024-07-27T10:03:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:58 smithi029 ceph-mon[32610]: pgmap v6637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:03:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:03:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11659.629066 secs 2024-07-27T10:03:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:58 smithi077 ceph-mon[37153]: pgmap v6637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:03:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:03:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11659.629066 secs 2024-07-27T10:04:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:00 smithi047 ceph-mon[36835]: pgmap v6638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:00 smithi029 ceph-mon[32610]: pgmap v6638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:00 smithi077 ceph-mon[37153]: pgmap v6638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:02.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:02.273+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:04:02.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:02.273+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:04:02.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:02.273+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:04:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:02 smithi047 ceph-mon[36835]: pgmap v6639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:04:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:02 smithi029 ceph-mon[32610]: pgmap v6639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:04:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:02 smithi077 ceph-mon[37153]: pgmap v6639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:04:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11664.629212 secs 2024-07-27T10:04:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11664.629212 secs 2024-07-27T10:04:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11664.629212 secs 2024-07-27T10:04:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:04 smithi047 ceph-mon[36835]: pgmap v6640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:04 smithi029 ceph-mon[32610]: pgmap v6640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:04 smithi077 ceph-mon[37153]: pgmap v6640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:06 smithi047 ceph-mon[36835]: pgmap v6641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:06 smithi029 ceph-mon[32610]: pgmap v6641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:06 smithi077 ceph-mon[37153]: pgmap v6641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:08 smithi047 ceph-mon[36835]: pgmap v6642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:04:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11669.629336 secs 2024-07-27T10:04:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:08 smithi029 ceph-mon[32610]: pgmap v6642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:04:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11669.629336 secs 2024-07-27T10:04:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:08 smithi077 ceph-mon[37153]: pgmap v6642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:04:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11669.629336 secs 2024-07-27T10:04:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:10 smithi047 ceph-mon[36835]: pgmap v6643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:10 smithi029 ceph-mon[32610]: pgmap v6643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:10 smithi077 ceph-mon[37153]: pgmap v6643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:12.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:12 smithi047 ceph-mon[36835]: pgmap v6644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:04:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:12 smithi029 ceph-mon[32610]: pgmap v6644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:04:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:12 smithi077 ceph-mon[37153]: pgmap v6644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:04:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11674.629438 secs 2024-07-27T10:04:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11674.629438 secs 2024-07-27T10:04:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11674.629438 secs 2024-07-27T10:04:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:14 smithi029 ceph-mon[32610]: pgmap v6645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:14 smithi077 ceph-mon[37153]: pgmap v6645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:14 smithi047 ceph-mon[36835]: pgmap v6645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:16 smithi029 ceph-mon[32610]: pgmap v6646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:16 smithi077 ceph-mon[37153]: pgmap v6646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:16 smithi047 ceph-mon[36835]: pgmap v6646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:17.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:17.288+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:04:17.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:17.288+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:04:17.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:17.288+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:04:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:18 smithi029 ceph-mon[32610]: pgmap v6647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:04:18.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11679.629569 secs 2024-07-27T10:04:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:18 smithi077 ceph-mon[37153]: pgmap v6647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:04:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11679.629569 secs 2024-07-27T10:04:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:18 smithi047 ceph-mon[36835]: pgmap v6647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:04:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11679.629569 secs 2024-07-27T10:04:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:20 smithi029 ceph-mon[32610]: pgmap v6648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:20 smithi077 ceph-mon[37153]: pgmap v6648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:20 smithi047 ceph-mon[36835]: pgmap v6648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:22 smithi077 ceph-mon[37153]: pgmap v6649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:04:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:22 smithi047 ceph-mon[36835]: pgmap v6649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:04:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:22 smithi029 ceph-mon[32610]: pgmap v6649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:04:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11684.629716 secs 2024-07-27T10:04:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11684.629716 secs 2024-07-27T10:04:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11684.629716 secs 2024-07-27T10:04:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:24 smithi077 ceph-mon[37153]: pgmap v6650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:24 smithi047 ceph-mon[36835]: pgmap v6650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:24 smithi029 ceph-mon[32610]: pgmap v6650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:26 smithi077 ceph-mon[37153]: pgmap v6651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:26 smithi047 ceph-mon[36835]: pgmap v6651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:26 smithi029 ceph-mon[32610]: pgmap v6651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:28 smithi029 ceph-mon[32610]: pgmap v6652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:04:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11689.629861 secs 2024-07-27T10:04:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:28 smithi077 ceph-mon[37153]: pgmap v6652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:04:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11689.629861 secs 2024-07-27T10:04:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:28 smithi047 ceph-mon[36835]: pgmap v6652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:04:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11689.629861 secs 2024-07-27T10:04:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:30 smithi077 ceph-mon[37153]: pgmap v6653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:30 smithi047 ceph-mon[36835]: pgmap v6653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:30 smithi029 ceph-mon[32610]: pgmap v6653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:32.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:32.303+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:04:32.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:32.303+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:04:32.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:32.303+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:04:32.515 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:32 smithi029 ceph-mon[32610]: pgmap v6654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:32 smithi077 ceph-mon[37153]: pgmap v6654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:32 smithi047 ceph-mon[36835]: pgmap v6654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11694.629986 secs 2024-07-27T10:04:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11694.629986 secs 2024-07-27T10:04:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11694.629986 secs 2024-07-27T10:04:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:34 smithi077 ceph-mon[37153]: pgmap v6655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:34 smithi047 ceph-mon[36835]: pgmap v6655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:34 smithi029 ceph-mon[32610]: pgmap v6655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:36 smithi077 ceph-mon[37153]: pgmap v6656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:36 smithi047 ceph-mon[36835]: pgmap v6656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:36 smithi029 ceph-mon[32610]: pgmap v6656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:38 smithi077 ceph-mon[37153]: pgmap v6657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:04:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11699.630091 secs 2024-07-27T10:04:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:38 smithi047 ceph-mon[36835]: pgmap v6657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:04:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11699.630091 secs 2024-07-27T10:04:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:38 smithi029 ceph-mon[32610]: pgmap v6657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:04:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11699.630091 secs 2024-07-27T10:04:40.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:40 smithi077 ceph-mon[37153]: pgmap v6658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:40 smithi047 ceph-mon[36835]: pgmap v6658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:40 smithi029 ceph-mon[32610]: pgmap v6658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:04:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:42 smithi077 ceph-mon[37153]: pgmap v6659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:04:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:42 smithi047 ceph-mon[36835]: pgmap v6659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:04:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:42 smithi029 ceph-mon[32610]: pgmap v6659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:04:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11704.630171 secs 2024-07-27T10:04:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11704.630171 secs 2024-07-27T10:04:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11704.630171 secs 2024-07-27T10:04:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:44 smithi077 ceph-mon[37153]: pgmap v6660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:44 smithi047 ceph-mon[36835]: pgmap v6660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:44 smithi029 ceph-mon[32610]: pgmap v6660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:04:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:04:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:04:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:04:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:04:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:04:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:04:45.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:04:45.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:04:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:04:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:04:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:04:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:04:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:04:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:04:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:04:45.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:04:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:04:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:04:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:04:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:04:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:04:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:04:45.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:04:45.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:04:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:46 smithi077 ceph-mon[37153]: pgmap v6661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:46 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:04:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:46 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:04:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:46 smithi077 ceph-mon[37153]: pgmap v6662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:04:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:46 smithi077 ceph-mon[37153]: pgmap v6663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:04:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:46 smithi047 ceph-mon[36835]: pgmap v6661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:46 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:04:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:46 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:04:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:46 smithi047 ceph-mon[36835]: pgmap v6662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:04:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:46 smithi047 ceph-mon[36835]: pgmap v6663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:04:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:46 smithi029 ceph-mon[32610]: pgmap v6661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:04:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:46 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:04:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:46 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:04:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:46 smithi029 ceph-mon[32610]: pgmap v6662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:04:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:46 smithi029 ceph-mon[32610]: pgmap v6663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:04:47.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:47.317+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:04:47.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:47.317+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:04:47.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:47.317+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:04:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11709.630294 secs 2024-07-27T10:04:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:48 smithi077 ceph-mon[37153]: pgmap v6664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:04:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11709.630294 secs 2024-07-27T10:04:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:48 smithi047 ceph-mon[36835]: pgmap v6664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:04:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11709.630294 secs 2024-07-27T10:04:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:48 smithi029 ceph-mon[32610]: pgmap v6664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:04:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:50 smithi077 ceph-mon[37153]: pgmap v6665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:04:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:50 smithi047 ceph-mon[36835]: pgmap v6665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:04:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:50 smithi029 ceph-mon[32610]: pgmap v6665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:04:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:52 smithi077 ceph-mon[37153]: pgmap v6666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:04:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:52 smithi047 ceph-mon[36835]: pgmap v6666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:04:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:52 smithi029 ceph-mon[32610]: pgmap v6666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:04:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11714.630429 secs 2024-07-27T10:04:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11714.630429 secs 2024-07-27T10:04:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11714.630429 secs 2024-07-27T10:04:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:54 smithi077 ceph-mon[37153]: pgmap v6667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:04:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:54 smithi047 ceph-mon[36835]: pgmap v6667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:04:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:54 smithi029 ceph-mon[32610]: pgmap v6667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:04:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:56 smithi077 ceph-mon[37153]: pgmap v6668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:04:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:56 smithi047 ceph-mon[36835]: pgmap v6668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:04:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:56 smithi029 ceph-mon[32610]: pgmap v6668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:04:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:57.665+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 msgr2=0x7eff3c0b5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T10:04:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:57.665+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 msgr2=0x7eff3c0b5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T10:04:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:57.665+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 0x7eff3c0b5580 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7eff400048c0 tx=0x7eff40038040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T10:04:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:57.665+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 0x7eff3c0b5580 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7eff400048c0 tx=0x7eff40038040 comp rx=0 tx=0).stop 2024-07-27T10:04:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:57.665+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 msgr2=0x7eff3c0b5580 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T10:04:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:57.665+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 msgr2=0x7eff3c0b5580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T10:04:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:57.665+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 0x7eff3c0b5580 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T10:04:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:57.665+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T10:04:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:57.665+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T10:04:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:57.665+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7eff400049e0 tx=0x7eff4006c040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T10:04:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11719.630556 secs 2024-07-27T10:04:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:04:58 smithi077 ceph-mon[37153]: pgmap v6669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:04:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11719.630556 secs 2024-07-27T10:04:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:04:58 smithi047 ceph-mon[36835]: pgmap v6669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:04:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11719.630556 secs 2024-07-27T10:04:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:04:58 smithi029 ceph-mon[32610]: pgmap v6669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:04:59.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:59.410+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:04:59.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:59.410+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:04:59.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:59.410+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:04:59.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:04:59.410+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:05:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:00 smithi077 ceph-mon[37153]: pgmap v6670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:05:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:00 smithi047 ceph-mon[36835]: pgmap v6670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:05:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:00 smithi029 ceph-mon[32610]: pgmap v6670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:05:02.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:02.319+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:05:02.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:02.319+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:05:02.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:02.319+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:05:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:02 smithi077 ceph-mon[37153]: pgmap v6671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:02 smithi047 ceph-mon[36835]: pgmap v6671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:02 smithi029 ceph-mon[32610]: pgmap v6671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11724.630664 secs 2024-07-27T10:05:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11724.630664 secs 2024-07-27T10:05:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11724.630664 secs 2024-07-27T10:05:04.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:04 smithi077 ceph-mon[37153]: pgmap v6672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:04 smithi047 ceph-mon[36835]: pgmap v6672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:04 smithi029 ceph-mon[32610]: pgmap v6672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:06 smithi077 ceph-mon[37153]: pgmap v6673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:06 smithi047 ceph-mon[36835]: pgmap v6673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:06 smithi029 ceph-mon[32610]: pgmap v6673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11729.630724 secs 2024-07-27T10:05:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:08 smithi077 ceph-mon[37153]: pgmap v6674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:05:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11729.630724 secs 2024-07-27T10:05:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:08 smithi047 ceph-mon[36835]: pgmap v6674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:05:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11729.630724 secs 2024-07-27T10:05:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:08 smithi029 ceph-mon[32610]: pgmap v6674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:05:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:10 smithi077 ceph-mon[37153]: pgmap v6675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:05:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:10 smithi047 ceph-mon[36835]: pgmap v6675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:05:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:10 smithi029 ceph-mon[32610]: pgmap v6675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:05:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:12 smithi077 ceph-mon[37153]: pgmap v6676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:12 smithi047 ceph-mon[36835]: pgmap v6676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:12 smithi029 ceph-mon[32610]: pgmap v6676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11734.630854 secs 2024-07-27T10:05:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11734.630854 secs 2024-07-27T10:05:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11734.630854 secs 2024-07-27T10:05:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:14 smithi077 ceph-mon[37153]: pgmap v6677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:14 smithi047 ceph-mon[36835]: pgmap v6677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:14 smithi029 ceph-mon[32610]: pgmap v6677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:16 smithi077 ceph-mon[37153]: pgmap v6678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:16 smithi047 ceph-mon[36835]: pgmap v6678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:16 smithi029 ceph-mon[32610]: pgmap v6678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:17.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:17.334+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:05:17.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:17.334+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:05:17.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:17.334+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:05:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11739.630975 secs 2024-07-27T10:05:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:18 smithi077 ceph-mon[37153]: pgmap v6679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:05:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11739.630975 secs 2024-07-27T10:05:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:18 smithi047 ceph-mon[36835]: pgmap v6679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:05:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11739.630975 secs 2024-07-27T10:05:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:18 smithi029 ceph-mon[32610]: pgmap v6679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:05:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:20 smithi077 ceph-mon[37153]: pgmap v6680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:05:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:20 smithi047 ceph-mon[36835]: pgmap v6680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:05:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:20 smithi029 ceph-mon[32610]: pgmap v6680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:05:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:22 smithi077 ceph-mon[37153]: pgmap v6681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:22 smithi047 ceph-mon[36835]: pgmap v6681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:22 smithi029 ceph-mon[32610]: pgmap v6681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11744.631102 secs 2024-07-27T10:05:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11744.631102 secs 2024-07-27T10:05:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11744.631102 secs 2024-07-27T10:05:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:24 smithi077 ceph-mon[37153]: pgmap v6682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:24 smithi047 ceph-mon[36835]: pgmap v6682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:24 smithi029 ceph-mon[32610]: pgmap v6682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:26 smithi077 ceph-mon[37153]: pgmap v6683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:26 smithi047 ceph-mon[36835]: pgmap v6683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:26 smithi029 ceph-mon[32610]: pgmap v6683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11749.631238 secs 2024-07-27T10:05:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:28 smithi047 ceph-mon[36835]: pgmap v6684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:05:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11749.631238 secs 2024-07-27T10:05:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:28 smithi029 ceph-mon[32610]: pgmap v6684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:05:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11749.631238 secs 2024-07-27T10:05:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:28 smithi077 ceph-mon[37153]: pgmap v6684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:05:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:30 smithi047 ceph-mon[36835]: pgmap v6685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:05:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:30 smithi029 ceph-mon[32610]: pgmap v6685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:05:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:30 smithi077 ceph-mon[37153]: pgmap v6685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:05:32.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:32.346+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:05:32.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:32.346+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:05:32.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:32.346+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:05:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:32 smithi047 ceph-mon[36835]: pgmap v6686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:32 smithi029 ceph-mon[32610]: pgmap v6686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:32 smithi077 ceph-mon[37153]: pgmap v6686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11754.631381 secs 2024-07-27T10:05:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11754.631381 secs 2024-07-27T10:05:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11754.631381 secs 2024-07-27T10:05:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:34 smithi047 ceph-mon[36835]: pgmap v6687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:34 smithi029 ceph-mon[32610]: pgmap v6687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:34 smithi077 ceph-mon[37153]: pgmap v6687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:36 smithi047 ceph-mon[36835]: pgmap v6688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:36 smithi029 ceph-mon[32610]: pgmap v6688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:36 smithi077 ceph-mon[37153]: pgmap v6688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:38.658 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11759.631502 secs 2024-07-27T10:05:38.658 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:38 smithi077 ceph-mon[37153]: pgmap v6689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:05:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11759.631502 secs 2024-07-27T10:05:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:38 smithi047 ceph-mon[36835]: pgmap v6689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:05:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11759.631502 secs 2024-07-27T10:05:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:38 smithi029 ceph-mon[32610]: pgmap v6689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:05:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:40 smithi047 ceph-mon[36835]: pgmap v6690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:05:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:40 smithi029 ceph-mon[32610]: pgmap v6690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:05:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:40 smithi077 ceph-mon[37153]: pgmap v6690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:05:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:42 smithi047 ceph-mon[36835]: pgmap v6691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:42 smithi029 ceph-mon[32610]: pgmap v6691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:42 smithi077 ceph-mon[37153]: pgmap v6691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11764.631623 secs 2024-07-27T10:05:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11764.631623 secs 2024-07-27T10:05:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11764.631623 secs 2024-07-27T10:05:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:44 smithi047 ceph-mon[36835]: pgmap v6692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:44 smithi029 ceph-mon[32610]: pgmap v6692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:44 smithi077 ceph-mon[37153]: pgmap v6692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:05:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:05:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:05:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:05:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:46 smithi047 ceph-mon[36835]: pgmap v6693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:05:46.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:05:46.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:05:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:46 smithi029 ceph-mon[32610]: pgmap v6693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:05:46.716 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:05:46.716 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:05:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:46 smithi077 ceph-mon[37153]: pgmap v6693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:05:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.860 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:05:46.860 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:05:46.860 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:05:46.860 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:05:47.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.361+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:05:47.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.361+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:05:47.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.361+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:05:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:47 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:05:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:47 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:05:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:47 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:05:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:47 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:05:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:47 smithi047 ceph-mon[36835]: pgmap v6694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:05:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:47 smithi047 ceph-mon[36835]: pgmap v6695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:05:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:47 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:05:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:47 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:05:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:47 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:05:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:47 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:05:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:47 smithi029 ceph-mon[32610]: pgmap v6694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:05:47.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:47 smithi029 ceph-mon[32610]: pgmap v6695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:05:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.853+0000 7fe7c5900640 1 -- 172.21.15.29:0/827749814 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe7ac0b3d40 msgr2=0x7fe7ac0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900020041) for more than 900000000 us, fault. 2024-07-27T10:05:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.853+0000 7fe7c5900640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe7ac0b3d40 0x7fe7ac0b6250 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0101640 tx=0x7fe7b0005e50 comp rx=0 tx=0).stop 2024-07-27T10:05:47.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.853+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe7ac0b3d40 msgr2=0x7fe7ac0b6250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T10:05:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.853+0000 7fe7b77fe640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe7ac0b3d40 0x7fe7ac0b6250 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T10:05:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.853+0000 7fe7b77fe640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe7ac0bf050 0x7fe7ac0c1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T10:05:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.853+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 --> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fe7bc018ec0 con 0x7fe7ac0bf050 2024-07-27T10:05:47.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.853+0000 7fe7c5900640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe7ac0bf050 0x7fe7ac0c1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T10:05:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:47 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:05:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:47 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:05:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:47 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:05:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:47 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:05:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:47 smithi077 ceph-mon[37153]: pgmap v6694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:05:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:47 smithi077 ceph-mon[37153]: pgmap v6695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:05:47.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.854+0000 7fe7c5900640 1 --2- 172.21.15.29:0/827749814 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7fe7ac0bf050 0x7fe7ac0c1470 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009b10 tx=0x7fe7b0013000 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T10:05:47.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.968+0000 7fe7b67fc640 1 -- 172.21.15.29:0/827749814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7a8006350 con 0x7fe7c01023f0 2024-07-27T10:05:47.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:05:47.969+0000 7fe7b77fe640 1 -- 172.21.15.29:0/827749814 <== mon.0 v2:172.21.15.29:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fe7a8006350 con 0x7fe7c01023f0 2024-07-27T10:05:48.052 INFO:teuthology.orchestra.run.smithi029.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-07-27T10:05:48.059 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-07-27T10:05:48.060 ERROR:teuthology.run_tasks:Manager failed: full_sequential_finally Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/full_sequential_finally.py", line 38, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 422, in task stop_all_quiescers(ctx.ceph[cluster_name].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 381, in stop_all_quiescers raise RuntimeError(f"error during quiesce thrashing: {thrasher.exception}") RuntimeError: error during quiesce thrashing: Error parsing quiesce response: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/full_sequential_finally.py", line 45, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi029 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/'" 2024-07-27T10:05:48.115 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-07-27T10:05:48.125 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-07-27T10:05:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11769.631759 secs 2024-07-27T10:05:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11769.631759 secs 2024-07-27T10:05:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11769.631759 secs 2024-07-27T10:05:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:49 smithi047 ceph-mon[36835]: pgmap v6696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:05:49.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:49 smithi047 ceph-mon[36835]: from='client.16536 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T10:05:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:49 smithi029 ceph-mon[32610]: pgmap v6696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:05:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:49 smithi029 ceph-mon[32610]: from='client.16536 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T10:05:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:49 smithi077 ceph-mon[37153]: pgmap v6696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:05:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:49 smithi077 ceph-mon[37153]: from='client.16536 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-27T10:05:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:51 smithi047 ceph-mon[36835]: pgmap v6697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:05:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:51 smithi029 ceph-mon[32610]: pgmap v6697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:05:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:51 smithi077 ceph-mon[37153]: pgmap v6697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:05:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:53 smithi047 ceph-mon[36835]: pgmap v6698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:05:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11774.631874 secs 2024-07-27T10:05:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:53 smithi029 ceph-mon[32610]: pgmap v6698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:05:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11774.631874 secs 2024-07-27T10:05:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:53 smithi077 ceph-mon[37153]: pgmap v6698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:05:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11774.631874 secs 2024-07-27T10:05:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:54 smithi077 ceph-mon[37153]: pgmap v6699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:05:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:54 smithi047 ceph-mon[36835]: pgmap v6699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:05:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:54 smithi029 ceph-mon[32610]: pgmap v6699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:05:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:56 smithi047 ceph-mon[36835]: pgmap v6700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:05:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:56 smithi029 ceph-mon[32610]: pgmap v6700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:05:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:56 smithi077 ceph-mon[37153]: pgmap v6700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:05:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11779.631987 secs 2024-07-27T10:05:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11779.631987 secs 2024-07-27T10:05:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11779.631987 secs 2024-07-27T10:05:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:05:58 smithi047 ceph-mon[36835]: pgmap v6701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:05:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:05:58 smithi029 ceph-mon[32610]: pgmap v6701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:05:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:05:58 smithi077 ceph-mon[37153]: pgmap v6701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:06:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:00 smithi047 ceph-mon[36835]: pgmap v6702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:06:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:00 smithi029 ceph-mon[32610]: pgmap v6702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:06:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:00 smithi077 ceph-mon[37153]: pgmap v6702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:06:02.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:06:02.367+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:06:02.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:06:02.367+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:06:02.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:06:02.367+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:06:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:02 smithi047 ceph-mon[36835]: pgmap v6703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:06:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11784.632109 secs 2024-07-27T10:06:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:02 smithi029 ceph-mon[32610]: pgmap v6703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:06:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11784.632109 secs 2024-07-27T10:06:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:02 smithi077 ceph-mon[37153]: pgmap v6703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:06:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11784.632109 secs 2024-07-27T10:06:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:04 smithi047 ceph-mon[36835]: pgmap v6704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:04 smithi029 ceph-mon[32610]: pgmap v6704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:04 smithi077 ceph-mon[37153]: pgmap v6704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:06 smithi047 ceph-mon[36835]: pgmap v6705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:06:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:06 smithi029 ceph-mon[32610]: pgmap v6705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:06:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:06 smithi077 ceph-mon[37153]: pgmap v6705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:06:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11789.632194 secs 2024-07-27T10:06:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11789.632194 secs 2024-07-27T10:06:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11789.632194 secs 2024-07-27T10:06:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:09 smithi047 ceph-mon[36835]: pgmap v6706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:06:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:09 smithi029 ceph-mon[32610]: pgmap v6706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:06:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:09 smithi077 ceph-mon[37153]: pgmap v6706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:06:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:11 smithi047 ceph-mon[36835]: pgmap v6707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:06:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:11 smithi029 ceph-mon[32610]: pgmap v6707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:06:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:11 smithi077 ceph-mon[37153]: pgmap v6707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:06:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:13 smithi047 ceph-mon[36835]: pgmap v6708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:06:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11794.632269 secs 2024-07-27T10:06:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:13 smithi029 ceph-mon[32610]: pgmap v6708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:06:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11794.632269 secs 2024-07-27T10:06:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:13 smithi077 ceph-mon[37153]: pgmap v6708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:06:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11794.632269 secs 2024-07-27T10:06:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:15 smithi047 ceph-mon[36835]: pgmap v6709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:15 smithi029 ceph-mon[32610]: pgmap v6709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:15 smithi077 ceph-mon[37153]: pgmap v6709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:17.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:06:17.381+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:06:17.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:06:17.381+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:06:17.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:06:17.381+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:06:17.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:17 smithi029 ceph-mon[32610]: pgmap v6710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:06:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:17 smithi047 ceph-mon[36835]: pgmap v6710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:06:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:17 smithi077 ceph-mon[37153]: pgmap v6710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:06:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11799.632409 secs 2024-07-27T10:06:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11799.632409 secs 2024-07-27T10:06:18.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11799.632409 secs 2024-07-27T10:06:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:19 smithi047 ceph-mon[36835]: pgmap v6711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:06:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:19 smithi029 ceph-mon[32610]: pgmap v6711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:06:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:19 smithi077 ceph-mon[37153]: pgmap v6711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:06:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:21 smithi047 ceph-mon[36835]: pgmap v6712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:06:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:21 smithi029 ceph-mon[32610]: pgmap v6712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:06:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:21 smithi077 ceph-mon[37153]: pgmap v6712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:06:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:23 smithi047 ceph-mon[36835]: pgmap v6713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:06:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11804.632505 secs 2024-07-27T10:06:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:23 smithi029 ceph-mon[32610]: pgmap v6713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:06:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11804.632505 secs 2024-07-27T10:06:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:23 smithi077 ceph-mon[37153]: pgmap v6713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:06:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11804.632505 secs 2024-07-27T10:06:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:25 smithi047 ceph-mon[36835]: pgmap v6714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:25 smithi029 ceph-mon[32610]: pgmap v6714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:25 smithi077 ceph-mon[37153]: pgmap v6714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:27 smithi047 ceph-mon[36835]: pgmap v6715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:06:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:27 smithi029 ceph-mon[32610]: pgmap v6715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:06:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:27 smithi077 ceph-mon[37153]: pgmap v6715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:06:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11809.632659 secs 2024-07-27T10:06:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11809.632659 secs 2024-07-27T10:06:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11809.632659 secs 2024-07-27T10:06:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:29 smithi047 ceph-mon[36835]: pgmap v6716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:06:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:29 smithi029 ceph-mon[32610]: pgmap v6716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:06:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:29 smithi077 ceph-mon[37153]: pgmap v6716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:06:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:31 smithi047 ceph-mon[36835]: pgmap v6717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:06:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:31 smithi029 ceph-mon[32610]: pgmap v6717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:06:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:31 smithi077 ceph-mon[37153]: pgmap v6717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:06:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:06:32.395+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:06:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:06:32.395+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:06:32.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:06:32.395+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:06:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:33 smithi047 ceph-mon[36835]: pgmap v6718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T10:06:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11814.632819 secs 2024-07-27T10:06:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:33 smithi029 ceph-mon[32610]: pgmap v6718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T10:06:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11814.632819 secs 2024-07-27T10:06:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:33 smithi077 ceph-mon[37153]: pgmap v6718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T10:06:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11814.632819 secs 2024-07-27T10:06:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:35 smithi047 ceph-mon[36835]: pgmap v6719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:06:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:35 smithi029 ceph-mon[32610]: pgmap v6719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:06:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:35 smithi077 ceph-mon[37153]: pgmap v6719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:06:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:37 smithi047 ceph-mon[36835]: pgmap v6720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:06:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:37 smithi029 ceph-mon[32610]: pgmap v6720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:06:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:37 smithi077 ceph-mon[37153]: pgmap v6720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:06:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11819.632940 secs 2024-07-27T10:06:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11819.632940 secs 2024-07-27T10:06:38.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11819.632940 secs 2024-07-27T10:06:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:39 smithi047 ceph-mon[36835]: pgmap v6721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T10:06:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:39 smithi029 ceph-mon[32610]: pgmap v6721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T10:06:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:39 smithi077 ceph-mon[37153]: pgmap v6721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T10:06:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:41 smithi047 ceph-mon[36835]: pgmap v6722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:06:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:41 smithi029 ceph-mon[32610]: pgmap v6722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:06:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:41 smithi077 ceph-mon[37153]: pgmap v6722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:06:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:43 smithi047 ceph-mon[36835]: pgmap v6723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:06:43.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11824.633060 secs 2024-07-27T10:06:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:43 smithi029 ceph-mon[32610]: pgmap v6723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:06:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11824.633060 secs 2024-07-27T10:06:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:43 smithi077 ceph-mon[37153]: pgmap v6723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:06:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11824.633060 secs 2024-07-27T10:06:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:45 smithi047 ceph-mon[36835]: pgmap v6724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:45 smithi029 ceph-mon[32610]: pgmap v6724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:45 smithi077 ceph-mon[37153]: pgmap v6724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:06:46.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:06:46.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:06:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:06:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:06:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:06:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:06:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:06:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:06:47.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:06:47.410+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:06:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:06:47.410+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:06:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:06:47.410+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:06:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:47 smithi047 ceph-mon[36835]: pgmap v6725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:06:47.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:47 smithi047 ceph-mon[36835]: pgmap v6726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:06:47.450 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:47 smithi047 ceph-mon[36835]: pgmap v6727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:06:47.450 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:06:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:47 smithi029 ceph-mon[32610]: pgmap v6725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:06:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:47 smithi029 ceph-mon[32610]: pgmap v6726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:06:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:47 smithi029 ceph-mon[32610]: pgmap v6727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:06:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:06:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:47 smithi077 ceph-mon[37153]: pgmap v6725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:06:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:47 smithi077 ceph-mon[37153]: pgmap v6726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:06:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:47 smithi077 ceph-mon[37153]: pgmap v6727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:06:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:06:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11829.633153 secs 2024-07-27T10:06:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11829.633153 secs 2024-07-27T10:06:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11829.633153 secs 2024-07-27T10:06:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:49 smithi047 ceph-mon[36835]: pgmap v6728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:06:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:49 smithi029 ceph-mon[32610]: pgmap v6728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:06:49.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:49 smithi077 ceph-mon[37153]: pgmap v6728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:06:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:51 smithi047 ceph-mon[36835]: pgmap v6729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:06:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:51 smithi029 ceph-mon[32610]: pgmap v6729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:06:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:51 smithi077 ceph-mon[37153]: pgmap v6729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:06:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11834.633278 secs 2024-07-27T10:06:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:53 smithi029 ceph-mon[32610]: pgmap v6730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:06:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11834.633278 secs 2024-07-27T10:06:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:53 smithi077 ceph-mon[37153]: pgmap v6730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:06:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11834.633278 secs 2024-07-27T10:06:53.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:53 smithi047 ceph-mon[36835]: pgmap v6730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:06:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:55 smithi029 ceph-mon[32610]: pgmap v6731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:55 smithi077 ceph-mon[37153]: pgmap v6731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:55 smithi047 ceph-mon[36835]: pgmap v6731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:06:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:57 smithi029 ceph-mon[32610]: pgmap v6732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T10:06:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:57 smithi077 ceph-mon[37153]: pgmap v6732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T10:06:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:57 smithi047 ceph-mon[36835]: pgmap v6732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T10:06:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11839.633435 secs 2024-07-27T10:06:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11839.633435 secs 2024-07-27T10:06:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11839.633435 secs 2024-07-27T10:06:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:06:59 smithi029 ceph-mon[32610]: pgmap v6733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:06:59.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:06:59 smithi077 ceph-mon[37153]: pgmap v6733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:06:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:06:59 smithi047 ceph-mon[36835]: pgmap v6733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:07:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:01 smithi077 ceph-mon[37153]: pgmap v6734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:07:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:01 smithi047 ceph-mon[36835]: pgmap v6734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:07:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:01 smithi029 ceph-mon[32610]: pgmap v6734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:07:02.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:02.414+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:07:02.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:02.415+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:07:02.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:02.415+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:07:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11844.633551 secs 2024-07-27T10:07:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:03 smithi077 ceph-mon[37153]: pgmap v6735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11844.633551 secs 2024-07-27T10:07:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:03 smithi047 ceph-mon[36835]: pgmap v6735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11844.633551 secs 2024-07-27T10:07:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:03 smithi029 ceph-mon[32610]: pgmap v6735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:05.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:05 smithi077 ceph-mon[37153]: pgmap v6736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:07:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:05 smithi047 ceph-mon[36835]: pgmap v6736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:07:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:05 smithi029 ceph-mon[32610]: pgmap v6736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:07:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:07 smithi077 ceph-mon[37153]: pgmap v6737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:07.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:07 smithi047 ceph-mon[36835]: pgmap v6737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:07 smithi029 ceph-mon[32610]: pgmap v6737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:08.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11849.633671 secs 2024-07-27T10:07:08.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11849.633671 secs 2024-07-27T10:07:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11849.633671 secs 2024-07-27T10:07:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:09 smithi077 ceph-mon[37153]: pgmap v6738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:07:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:09 smithi047 ceph-mon[36835]: pgmap v6738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:07:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:09 smithi029 ceph-mon[32610]: pgmap v6738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:07:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:11 smithi077 ceph-mon[37153]: pgmap v6739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:11 smithi047 ceph-mon[36835]: pgmap v6739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:11 smithi029 ceph-mon[32610]: pgmap v6739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11854.633813 secs 2024-07-27T10:07:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:13 smithi077 ceph-mon[37153]: pgmap v6740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11854.633813 secs 2024-07-27T10:07:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:13 smithi047 ceph-mon[36835]: pgmap v6740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11854.633813 secs 2024-07-27T10:07:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:13 smithi029 ceph-mon[32610]: pgmap v6740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:15 smithi077 ceph-mon[37153]: pgmap v6741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:07:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:15 smithi047 ceph-mon[36835]: pgmap v6741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:07:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:15 smithi029 ceph-mon[32610]: pgmap v6741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:07:17.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:17.430+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:07:17.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:17.430+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:07:17.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:17.430+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:07:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:17 smithi077 ceph-mon[37153]: pgmap v6742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:17 smithi047 ceph-mon[36835]: pgmap v6742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:17 smithi029 ceph-mon[32610]: pgmap v6742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11859.633908 secs 2024-07-27T10:07:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11859.633908 secs 2024-07-27T10:07:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11859.633908 secs 2024-07-27T10:07:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:19 smithi077 ceph-mon[37153]: pgmap v6743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:07:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:19 smithi047 ceph-mon[36835]: pgmap v6743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:07:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:19 smithi029 ceph-mon[32610]: pgmap v6743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:07:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:21 smithi077 ceph-mon[37153]: pgmap v6744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:21 smithi047 ceph-mon[36835]: pgmap v6744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:21 smithi029 ceph-mon[32610]: pgmap v6744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11864.634062 secs 2024-07-27T10:07:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:23 smithi077 ceph-mon[37153]: pgmap v6745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11864.634062 secs 2024-07-27T10:07:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:23 smithi047 ceph-mon[36835]: pgmap v6745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11864.634062 secs 2024-07-27T10:07:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:23 smithi029 ceph-mon[32610]: pgmap v6745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:25.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:25 smithi077 ceph-mon[37153]: pgmap v6746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:07:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:25 smithi047 ceph-mon[36835]: pgmap v6746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:07:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:25 smithi029 ceph-mon[32610]: pgmap v6746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:07:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:27 smithi077 ceph-mon[37153]: pgmap v6747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:27 smithi047 ceph-mon[36835]: pgmap v6747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:27 smithi029 ceph-mon[32610]: pgmap v6747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11869.634121 secs 2024-07-27T10:07:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11869.634121 secs 2024-07-27T10:07:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11869.634121 secs 2024-07-27T10:07:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:29 smithi077 ceph-mon[37153]: pgmap v6748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:07:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:29 smithi047 ceph-mon[36835]: pgmap v6748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:07:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:29 smithi029 ceph-mon[32610]: pgmap v6748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:07:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:31 smithi077 ceph-mon[37153]: pgmap v6749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:31 smithi047 ceph-mon[36835]: pgmap v6749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:31 smithi029 ceph-mon[32610]: pgmap v6749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:32.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:32.432+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:07:32.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:32.432+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:07:32.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:32.432+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:07:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11874.634245 secs 2024-07-27T10:07:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:33 smithi077 ceph-mon[37153]: pgmap v6750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11874.634245 secs 2024-07-27T10:07:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:33 smithi047 ceph-mon[36835]: pgmap v6750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11874.634245 secs 2024-07-27T10:07:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:33 smithi029 ceph-mon[32610]: pgmap v6750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:35.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:35 smithi077 ceph-mon[37153]: pgmap v6751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:07:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:35 smithi047 ceph-mon[36835]: pgmap v6751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:07:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:35 smithi029 ceph-mon[32610]: pgmap v6751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:07:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:37 smithi077 ceph-mon[37153]: pgmap v6752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:37 smithi047 ceph-mon[36835]: pgmap v6752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:37 smithi029 ceph-mon[32610]: pgmap v6752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:38.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11879.634367 secs 2024-07-27T10:07:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11879.634367 secs 2024-07-27T10:07:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11879.634367 secs 2024-07-27T10:07:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:39 smithi077 ceph-mon[37153]: pgmap v6753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:07:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:39 smithi047 ceph-mon[36835]: pgmap v6753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:07:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:39 smithi029 ceph-mon[32610]: pgmap v6753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:07:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:41 smithi077 ceph-mon[37153]: pgmap v6754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:41 smithi047 ceph-mon[36835]: pgmap v6754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:41 smithi029 ceph-mon[32610]: pgmap v6754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11884.634494 secs 2024-07-27T10:07:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:43 smithi077 ceph-mon[37153]: pgmap v6755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11884.634494 secs 2024-07-27T10:07:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:43 smithi047 ceph-mon[36835]: pgmap v6755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11884.634494 secs 2024-07-27T10:07:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:43 smithi029 ceph-mon[32610]: pgmap v6755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:07:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:45 smithi077 ceph-mon[37153]: pgmap v6756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:07:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:45 smithi047 ceph-mon[36835]: pgmap v6756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:07:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:45 smithi029 ceph-mon[32610]: pgmap v6756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:07:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:07:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:07:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:07:47.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:47.435+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:07:47.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:47.435+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:07:47.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:47.435+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:07:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:47 smithi077 ceph-mon[37153]: pgmap v6757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:07:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:07:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:07:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:47 smithi047 ceph-mon[36835]: pgmap v6757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:47.723 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:07:47.723 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:07:47.724 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:07:47.724 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:47 smithi029 ceph-mon[32610]: pgmap v6757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:07:47.724 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:07:47.725 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:07:47.725 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:07:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:48 smithi077 ceph-mon[37153]: pgmap v6758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:07:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:48 smithi077 ceph-mon[37153]: pgmap v6759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:07:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11889.634615 secs 2024-07-27T10:07:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:48 smithi047 ceph-mon[36835]: pgmap v6758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:07:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:48 smithi047 ceph-mon[36835]: pgmap v6759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:07:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11889.634615 secs 2024-07-27T10:07:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:48 smithi029 ceph-mon[32610]: pgmap v6758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:07:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:48 smithi029 ceph-mon[32610]: pgmap v6759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:07:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11889.634615 secs 2024-07-27T10:07:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:50 smithi077 ceph-mon[37153]: pgmap v6760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:07:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:50 smithi047 ceph-mon[36835]: pgmap v6760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:07:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:50 smithi029 ceph-mon[32610]: pgmap v6760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:07:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:52 smithi077 ceph-mon[37153]: pgmap v6761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:07:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:52 smithi047 ceph-mon[36835]: pgmap v6761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:07:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:52 smithi029 ceph-mon[32610]: pgmap v6761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:07:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11894.634716 secs 2024-07-27T10:07:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11894.634716 secs 2024-07-27T10:07:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11894.634716 secs 2024-07-27T10:07:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:54 smithi077 ceph-mon[37153]: pgmap v6762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:07:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:54 smithi047 ceph-mon[36835]: pgmap v6762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:07:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:54 smithi029 ceph-mon[32610]: pgmap v6762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:07:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:56 smithi077 ceph-mon[37153]: pgmap v6763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:07:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:56 smithi047 ceph-mon[36835]: pgmap v6763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:07:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:56 smithi029 ceph-mon[32610]: pgmap v6763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:07:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:58 smithi077 ceph-mon[37153]: pgmap v6764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:07:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:07:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11899.634843 secs 2024-07-27T10:07:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:58 smithi047 ceph-mon[36835]: pgmap v6764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:07:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:07:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11899.634843 secs 2024-07-27T10:07:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:58 smithi029 ceph-mon[32610]: pgmap v6764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:07:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:07:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11899.634843 secs 2024-07-27T10:07:59.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:59.438+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:07:59.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:59.438+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:07:59.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:59.438+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:07:59.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:59.438+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:07:59.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:59.438+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:07:59.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:59.438+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:07:59.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:59.439+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:07:59.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:07:59.439+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:08:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:00 smithi077 ceph-mon[37153]: pgmap v6765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:00 smithi047 ceph-mon[36835]: pgmap v6765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:00 smithi029 ceph-mon[32610]: pgmap v6765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:02.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:08:02.450+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:08:02.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:08:02.450+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:08:02.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:08:02.450+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:08:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:02 smithi077 ceph-mon[37153]: pgmap v6766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:08:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:02 smithi047 ceph-mon[36835]: pgmap v6766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:08:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:02 smithi029 ceph-mon[32610]: pgmap v6766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:08:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11904.634965 secs 2024-07-27T10:08:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11904.634965 secs 2024-07-27T10:08:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11904.634965 secs 2024-07-27T10:08:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:04 smithi077 ceph-mon[37153]: pgmap v6767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:08:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:04 smithi047 ceph-mon[36835]: pgmap v6767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:08:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:04 smithi029 ceph-mon[32610]: pgmap v6767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:08:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:06 smithi029 ceph-mon[32610]: pgmap v6768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:08:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:06 smithi077 ceph-mon[37153]: pgmap v6768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:08:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:06 smithi047 ceph-mon[36835]: pgmap v6768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:08:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:08 smithi077 ceph-mon[37153]: pgmap v6769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11909.635083 secs 2024-07-27T10:08:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:08 smithi047 ceph-mon[36835]: pgmap v6769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11909.635083 secs 2024-07-27T10:08:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:08 smithi029 ceph-mon[32610]: pgmap v6769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11909.635083 secs 2024-07-27T10:08:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:10 smithi047 ceph-mon[36835]: pgmap v6770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:08:10.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:10 smithi029 ceph-mon[32610]: pgmap v6770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:08:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:10 smithi077 ceph-mon[37153]: pgmap v6770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:08:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:12 smithi047 ceph-mon[36835]: pgmap v6771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:08:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:12 smithi029 ceph-mon[32610]: pgmap v6771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:08:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:12 smithi077 ceph-mon[37153]: pgmap v6771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:08:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11914.635167 secs 2024-07-27T10:08:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11914.635167 secs 2024-07-27T10:08:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11914.635167 secs 2024-07-27T10:08:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:14 smithi047 ceph-mon[36835]: pgmap v6772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:08:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:14 smithi029 ceph-mon[32610]: pgmap v6772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:08:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:14 smithi077 ceph-mon[37153]: pgmap v6772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:08:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:16 smithi047 ceph-mon[36835]: pgmap v6773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:08:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:16 smithi029 ceph-mon[32610]: pgmap v6773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:08:16.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:16 smithi077 ceph-mon[37153]: pgmap v6773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:08:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:08:17.465+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:08:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:08:17.465+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:08:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:08:17.465+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:08:18.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:18 smithi047 ceph-mon[36835]: pgmap v6774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:18.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11919.635304 secs 2024-07-27T10:08:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:18 smithi029 ceph-mon[32610]: pgmap v6774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11919.635304 secs 2024-07-27T10:08:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:18 smithi077 ceph-mon[37153]: pgmap v6774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11919.635304 secs 2024-07-27T10:08:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:20 smithi047 ceph-mon[36835]: pgmap v6775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:20 smithi029 ceph-mon[32610]: pgmap v6775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:20 smithi077 ceph-mon[37153]: pgmap v6775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:22 smithi047 ceph-mon[36835]: pgmap v6776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:08:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:22 smithi029 ceph-mon[32610]: pgmap v6776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:08:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:22 smithi077 ceph-mon[37153]: pgmap v6776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:08:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11924.635423 secs 2024-07-27T10:08:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11924.635423 secs 2024-07-27T10:08:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11924.635423 secs 2024-07-27T10:08:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:24 smithi047 ceph-mon[36835]: pgmap v6777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:08:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:24 smithi029 ceph-mon[32610]: pgmap v6777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:08:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:24 smithi077 ceph-mon[37153]: pgmap v6777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:08:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:26 smithi047 ceph-mon[36835]: pgmap v6778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:08:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:26 smithi029 ceph-mon[32610]: pgmap v6778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:08:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:26 smithi077 ceph-mon[37153]: pgmap v6778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:08:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:28 smithi047 ceph-mon[36835]: pgmap v6779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11929.635545 secs 2024-07-27T10:08:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:28 smithi029 ceph-mon[32610]: pgmap v6779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11929.635545 secs 2024-07-27T10:08:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:28 smithi077 ceph-mon[37153]: pgmap v6779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11929.635545 secs 2024-07-27T10:08:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:30 smithi047 ceph-mon[36835]: pgmap v6780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:30 smithi029 ceph-mon[32610]: pgmap v6780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:30 smithi077 ceph-mon[37153]: pgmap v6780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:08:32.477+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:08:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:08:32.477+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:08:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:08:32.477+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:08:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:32 smithi047 ceph-mon[36835]: pgmap v6781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:08:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:32 smithi029 ceph-mon[32610]: pgmap v6781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:08:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:32 smithi077 ceph-mon[37153]: pgmap v6781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:08:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11934.635677 secs 2024-07-27T10:08:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11934.635677 secs 2024-07-27T10:08:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11934.635677 secs 2024-07-27T10:08:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:34 smithi047 ceph-mon[36835]: pgmap v6782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:08:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:34 smithi029 ceph-mon[32610]: pgmap v6782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:08:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:34 smithi077 ceph-mon[37153]: pgmap v6782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:08:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:36 smithi047 ceph-mon[36835]: pgmap v6783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:08:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:36 smithi029 ceph-mon[32610]: pgmap v6783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:08:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:36 smithi077 ceph-mon[37153]: pgmap v6783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:08:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:38 smithi047 ceph-mon[36835]: pgmap v6784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11939.635823 secs 2024-07-27T10:08:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:38 smithi029 ceph-mon[32610]: pgmap v6784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11939.635823 secs 2024-07-27T10:08:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:38 smithi077 ceph-mon[37153]: pgmap v6784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11939.635823 secs 2024-07-27T10:08:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:40 smithi047 ceph-mon[36835]: pgmap v6785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:40 smithi029 ceph-mon[32610]: pgmap v6785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:40.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:40 smithi077 ceph-mon[37153]: pgmap v6785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:42 smithi047 ceph-mon[36835]: pgmap v6786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:08:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:42 smithi029 ceph-mon[32610]: pgmap v6786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:08:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:42 smithi077 ceph-mon[37153]: pgmap v6786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:08:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11944.635947 secs 2024-07-27T10:08:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11944.635947 secs 2024-07-27T10:08:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11944.635947 secs 2024-07-27T10:08:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:44 smithi047 ceph-mon[36835]: pgmap v6787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:08:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:44 smithi029 ceph-mon[32610]: pgmap v6787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:08:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:44 smithi077 ceph-mon[37153]: pgmap v6787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:08:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:46 smithi047 ceph-mon[36835]: pgmap v6788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:08:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:46 smithi029 ceph-mon[32610]: pgmap v6788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:08:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:46 smithi077 ceph-mon[37153]: pgmap v6788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:08:47.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:08:47.492+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:08:47.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:08:47.492+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:08:47.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:08:47.492+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:08:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:08:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:47.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:08:47.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:47.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:08:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:48 smithi047 ceph-mon[36835]: pgmap v6789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11949.636025 secs 2024-07-27T10:08:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:48 smithi029 ceph-mon[32610]: pgmap v6789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11949.636025 secs 2024-07-27T10:08:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:48 smithi077 ceph-mon[37153]: pgmap v6789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:08:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11949.636025 secs 2024-07-27T10:08:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:49 smithi077 ceph-mon[37153]: pgmap v6790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:49 smithi047 ceph-mon[36835]: pgmap v6790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:49 smithi029 ceph-mon[32610]: pgmap v6790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:08:49.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:49.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:08:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:08:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:50 smithi077 ceph-mon[37153]: pgmap v6791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:08:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:50 smithi077 ceph-mon[37153]: pgmap v6792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:08:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:08:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:08:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:50 smithi047 ceph-mon[36835]: pgmap v6791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:08:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:50 smithi047 ceph-mon[36835]: pgmap v6792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:08:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:08:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:08:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:50 smithi029 ceph-mon[32610]: pgmap v6791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:08:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:50 smithi029 ceph-mon[32610]: pgmap v6792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:08:50.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:08:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:52 smithi077 ceph-mon[37153]: pgmap v6793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:08:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11954.636152 secs 2024-07-27T10:08:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:52 smithi047 ceph-mon[36835]: pgmap v6793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:08:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11954.636152 secs 2024-07-27T10:08:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:52 smithi029 ceph-mon[32610]: pgmap v6793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:08:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11954.636152 secs 2024-07-27T10:08:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:54 smithi077 ceph-mon[37153]: pgmap v6794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:08:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:54 smithi047 ceph-mon[36835]: pgmap v6794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:08:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:54 smithi029 ceph-mon[32610]: pgmap v6794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:08:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:56 smithi077 ceph-mon[37153]: pgmap v6795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:08:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:56 smithi047 ceph-mon[36835]: pgmap v6795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:08:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:56 smithi029 ceph-mon[32610]: pgmap v6795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:08:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11959.636215 secs 2024-07-27T10:08:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11959.636215 secs 2024-07-27T10:08:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11959.636215 secs 2024-07-27T10:08:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:08:58 smithi077 ceph-mon[37153]: pgmap v6796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:08:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:08:58 smithi047 ceph-mon[36835]: pgmap v6796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:08:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:08:58 smithi029 ceph-mon[32610]: pgmap v6796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:09:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:00 smithi077 ceph-mon[37153]: pgmap v6797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:09:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:00 smithi047 ceph-mon[36835]: pgmap v6797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:09:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:00 smithi029 ceph-mon[32610]: pgmap v6797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:09:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:09:02.508+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:09:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:09:02.508+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:09:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:09:02.508+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:09:02.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:02 smithi029 ceph-mon[32610]: pgmap v6798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:02.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11964.636335 secs 2024-07-27T10:09:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:02 smithi077 ceph-mon[37153]: pgmap v6798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11964.636335 secs 2024-07-27T10:09:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:02 smithi047 ceph-mon[36835]: pgmap v6798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11964.636335 secs 2024-07-27T10:09:04.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:04 smithi077 ceph-mon[37153]: pgmap v6799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:04 smithi047 ceph-mon[36835]: pgmap v6799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:04 smithi029 ceph-mon[32610]: pgmap v6799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:06 smithi077 ceph-mon[37153]: pgmap v6800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:06 smithi047 ceph-mon[36835]: pgmap v6800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:06 smithi029 ceph-mon[32610]: pgmap v6800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11969.636455 secs 2024-07-27T10:09:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11969.636455 secs 2024-07-27T10:09:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11969.636455 secs 2024-07-27T10:09:08.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:08 smithi047 ceph-mon[36835]: pgmap v6801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:08 smithi077 ceph-mon[37153]: pgmap v6801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:08 smithi029 ceph-mon[32610]: pgmap v6801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:10 smithi077 ceph-mon[37153]: pgmap v6802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:10 smithi047 ceph-mon[36835]: pgmap v6802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:10 smithi029 ceph-mon[32610]: pgmap v6802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:12 smithi077 ceph-mon[37153]: pgmap v6803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11974.636580 secs 2024-07-27T10:09:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:12 smithi047 ceph-mon[36835]: pgmap v6803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:12.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11974.636580 secs 2024-07-27T10:09:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:12 smithi029 ceph-mon[32610]: pgmap v6803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11974.636580 secs 2024-07-27T10:09:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:14 smithi077 ceph-mon[37153]: pgmap v6804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:14 smithi047 ceph-mon[36835]: pgmap v6804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:14 smithi029 ceph-mon[32610]: pgmap v6804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:16 smithi047 ceph-mon[36835]: pgmap v6805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:16 smithi029 ceph-mon[32610]: pgmap v6805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:16 smithi077 ceph-mon[37153]: pgmap v6805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:17.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:09:17.512+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:09:17.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:09:17.512+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:09:17.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:09:17.512+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:09:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11979.636696 secs 2024-07-27T10:09:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11979.636696 secs 2024-07-27T10:09:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11979.636696 secs 2024-07-27T10:09:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:18 smithi047 ceph-mon[36835]: pgmap v6806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:18 smithi029 ceph-mon[32610]: pgmap v6806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:18 smithi077 ceph-mon[37153]: pgmap v6806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:20 smithi047 ceph-mon[36835]: pgmap v6807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:20 smithi029 ceph-mon[32610]: pgmap v6807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:20 smithi077 ceph-mon[37153]: pgmap v6807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:22 smithi047 ceph-mon[36835]: pgmap v6808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:22.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11984.636848 secs 2024-07-27T10:09:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:22 smithi029 ceph-mon[32610]: pgmap v6808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11984.636848 secs 2024-07-27T10:09:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:22 smithi077 ceph-mon[37153]: pgmap v6808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11984.636848 secs 2024-07-27T10:09:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:24 smithi047 ceph-mon[36835]: pgmap v6809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:24 smithi029 ceph-mon[32610]: pgmap v6809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:24 smithi077 ceph-mon[37153]: pgmap v6809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:26 smithi047 ceph-mon[36835]: pgmap v6810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:26 smithi029 ceph-mon[32610]: pgmap v6810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:27.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:26 smithi077 ceph-mon[37153]: pgmap v6810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11989.636948 secs 2024-07-27T10:09:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11989.636948 secs 2024-07-27T10:09:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11989.636948 secs 2024-07-27T10:09:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:28 smithi047 ceph-mon[36835]: pgmap v6811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:28 smithi029 ceph-mon[32610]: pgmap v6811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:28 smithi077 ceph-mon[37153]: pgmap v6811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:30 smithi047 ceph-mon[36835]: pgmap v6812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:30 smithi029 ceph-mon[32610]: pgmap v6812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:31.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:30 smithi077 ceph-mon[37153]: pgmap v6812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:32.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:09:32.512+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:09:32.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:09:32.512+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:09:32.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:09:32.512+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:09:32.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:32 smithi029 ceph-mon[32610]: pgmap v6813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:32.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11994.637070 secs 2024-07-27T10:09:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:32 smithi047 ceph-mon[36835]: pgmap v6813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:32.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11994.637070 secs 2024-07-27T10:09:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:32 smithi077 ceph-mon[37153]: pgmap v6813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11994.637070 secs 2024-07-27T10:09:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:34 smithi047 ceph-mon[36835]: pgmap v6814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:34 smithi029 ceph-mon[32610]: pgmap v6814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:34 smithi077 ceph-mon[37153]: pgmap v6814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:36 smithi047 ceph-mon[36835]: pgmap v6815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:36 smithi029 ceph-mon[32610]: pgmap v6815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:36 smithi077 ceph-mon[37153]: pgmap v6815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 11999.637172 secs 2024-07-27T10:09:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 11999.637172 secs 2024-07-27T10:09:38.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 11999.637172 secs 2024-07-27T10:09:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:38 smithi047 ceph-mon[36835]: pgmap v6816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:38 smithi029 ceph-mon[32610]: pgmap v6816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:38 smithi077 ceph-mon[37153]: pgmap v6816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:40 smithi047 ceph-mon[36835]: pgmap v6817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:40 smithi029 ceph-mon[32610]: pgmap v6817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:40 smithi077 ceph-mon[37153]: pgmap v6817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:42 smithi047 ceph-mon[36835]: pgmap v6818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12004.637273 secs 2024-07-27T10:09:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:42 smithi029 ceph-mon[32610]: pgmap v6818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12004.637273 secs 2024-07-27T10:09:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:42 smithi077 ceph-mon[37153]: pgmap v6818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:09:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12004.637273 secs 2024-07-27T10:09:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:44 smithi047 ceph-mon[36835]: pgmap v6819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:44 smithi029 ceph-mon[32610]: pgmap v6819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:44 smithi077 ceph-mon[37153]: pgmap v6819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:09:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:46 smithi047 ceph-mon[36835]: pgmap v6820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:46 smithi029 ceph-mon[32610]: pgmap v6820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:46 smithi077 ceph-mon[37153]: pgmap v6820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:09:47.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:09:47.528+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:09:47.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:09:47.528+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:09:47.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:09:47.528+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:09:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12009.637412 secs 2024-07-27T10:09:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12009.637412 secs 2024-07-27T10:09:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12009.637412 secs 2024-07-27T10:09:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:48 smithi047 ceph-mon[36835]: pgmap v6821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:48 smithi029 ceph-mon[32610]: pgmap v6821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:48 smithi077 ceph-mon[37153]: pgmap v6821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:09:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:09:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:09:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:09:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: pgmap v6822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:09:51.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:09:51.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: pgmap v6822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:51.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:09:51.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:09:51.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: pgmap v6822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:09:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:51.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:09:51.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:09:51.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:09:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:51 smithi077 ceph-mon[37153]: pgmap v6823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T10:09:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:51 smithi077 ceph-mon[37153]: pgmap v6824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:09:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:51 smithi047 ceph-mon[36835]: pgmap v6823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T10:09:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:51 smithi047 ceph-mon[36835]: pgmap v6824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:09:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:51 smithi029 ceph-mon[32610]: pgmap v6823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T10:09:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:51 smithi029 ceph-mon[32610]: pgmap v6824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:09:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12014.637534 secs 2024-07-27T10:09:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12014.637534 secs 2024-07-27T10:09:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12014.637534 secs 2024-07-27T10:09:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:53 smithi077 ceph-mon[37153]: pgmap v6825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:09:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:53 smithi047 ceph-mon[36835]: pgmap v6825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:09:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:53 smithi029 ceph-mon[32610]: pgmap v6825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:09:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:55 smithi077 ceph-mon[37153]: pgmap v6826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:09:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:55 smithi047 ceph-mon[36835]: pgmap v6826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:09:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:55 smithi029 ceph-mon[32610]: pgmap v6826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:09:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:57 smithi077 ceph-mon[37153]: pgmap v6827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T10:09:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12019.637676 secs 2024-07-27T10:09:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:57 smithi047 ceph-mon[36835]: pgmap v6827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T10:09:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12019.637676 secs 2024-07-27T10:09:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:57 smithi029 ceph-mon[32610]: pgmap v6827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T10:09:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12019.637676 secs 2024-07-27T10:10:00.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:09:59 smithi077 ceph-mon[37153]: pgmap v6828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:10:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:09:59 smithi047 ceph-mon[36835]: pgmap v6828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:10:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:09:59 smithi029 ceph-mon[32610]: pgmap v6828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:10:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:10:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:10:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:10:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:02 smithi077 ceph-mon[37153]: pgmap v6829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:10:02.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:02 smithi029 ceph-mon[32610]: pgmap v6829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:10:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:02 smithi047 ceph-mon[36835]: pgmap v6829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:10:02.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:02.532+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:10:02.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:02.532+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:10:02.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:02.532+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:10:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12024.637816 secs 2024-07-27T10:10:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12024.637816 secs 2024-07-27T10:10:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12024.637816 secs 2024-07-27T10:10:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:04 smithi077 ceph-mon[37153]: pgmap v6830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:04 smithi047 ceph-mon[36835]: pgmap v6830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:04 smithi029 ceph-mon[32610]: pgmap v6830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:06 smithi077 ceph-mon[37153]: pgmap v6831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:06 smithi047 ceph-mon[36835]: pgmap v6831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:06 smithi029 ceph-mon[32610]: pgmap v6831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:08.451 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:08 smithi047 ceph-mon[36835]: pgmap v6832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:08.451 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12029.637933 secs 2024-07-27T10:10:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:08 smithi029 ceph-mon[32610]: pgmap v6832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12029.637933 secs 2024-07-27T10:10:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:08 smithi077 ceph-mon[37153]: pgmap v6832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12029.637933 secs 2024-07-27T10:10:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:10 smithi047 ceph-mon[36835]: pgmap v6833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:10 smithi029 ceph-mon[32610]: pgmap v6833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:10 smithi077 ceph-mon[37153]: pgmap v6833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:12 smithi047 ceph-mon[36835]: pgmap v6834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:12 smithi029 ceph-mon[32610]: pgmap v6834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:12 smithi077 ceph-mon[37153]: pgmap v6834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12034.638054 secs 2024-07-27T10:10:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12034.638054 secs 2024-07-27T10:10:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12034.638054 secs 2024-07-27T10:10:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:14 smithi047 ceph-mon[36835]: pgmap v6835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:14 smithi029 ceph-mon[32610]: pgmap v6835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:14 smithi077 ceph-mon[37153]: pgmap v6835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:16 smithi047 ceph-mon[36835]: pgmap v6836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:16 smithi029 ceph-mon[32610]: pgmap v6836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:16 smithi077 ceph-mon[37153]: pgmap v6836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:17.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:17.547+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:10:17.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:17.547+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:10:17.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:17.547+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:10:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:18 smithi047 ceph-mon[36835]: pgmap v6837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12039.638175 secs 2024-07-27T10:10:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:18 smithi029 ceph-mon[32610]: pgmap v6837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12039.638175 secs 2024-07-27T10:10:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:18 smithi077 ceph-mon[37153]: pgmap v6837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12039.638175 secs 2024-07-27T10:10:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:20 smithi047 ceph-mon[36835]: pgmap v6838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:20 smithi029 ceph-mon[32610]: pgmap v6838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:20 smithi077 ceph-mon[37153]: pgmap v6838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:22 smithi029 ceph-mon[32610]: pgmap v6839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:22 smithi077 ceph-mon[37153]: pgmap v6839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:22 smithi047 ceph-mon[36835]: pgmap v6839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12044.638324 secs 2024-07-27T10:10:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12044.638324 secs 2024-07-27T10:10:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12044.638324 secs 2024-07-27T10:10:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:24 smithi029 ceph-mon[32610]: pgmap v6840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:24 smithi077 ceph-mon[37153]: pgmap v6840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:24 smithi047 ceph-mon[36835]: pgmap v6840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:26 smithi029 ceph-mon[32610]: pgmap v6841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:26 smithi077 ceph-mon[37153]: pgmap v6841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:26 smithi047 ceph-mon[36835]: pgmap v6841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:28 smithi029 ceph-mon[32610]: pgmap v6842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12049.638464 secs 2024-07-27T10:10:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:28 smithi077 ceph-mon[37153]: pgmap v6842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12049.638464 secs 2024-07-27T10:10:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:28 smithi047 ceph-mon[36835]: pgmap v6842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12049.638464 secs 2024-07-27T10:10:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:30 smithi047 ceph-mon[36835]: pgmap v6843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:30 smithi077 ceph-mon[37153]: pgmap v6843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:30 smithi029 ceph-mon[32610]: pgmap v6843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:32.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:32 smithi029 ceph-mon[32610]: pgmap v6844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:32.561+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:10:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:32.561+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:10:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:32.561+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:10:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:32 smithi077 ceph-mon[37153]: pgmap v6844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:32 smithi047 ceph-mon[36835]: pgmap v6844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:33.534 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12054.638612 secs 2024-07-27T10:10:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12054.638612 secs 2024-07-27T10:10:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12054.638612 secs 2024-07-27T10:10:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:34 smithi077 ceph-mon[37153]: pgmap v6845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:34 smithi047 ceph-mon[36835]: pgmap v6845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:34 smithi029 ceph-mon[32610]: pgmap v6845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:36.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:36 smithi077 ceph-mon[37153]: pgmap v6846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:36 smithi047 ceph-mon[36835]: pgmap v6846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:36 smithi029 ceph-mon[32610]: pgmap v6846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:38.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:38 smithi047 ceph-mon[36835]: pgmap v6847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:38.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12059.638732 secs 2024-07-27T10:10:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:38 smithi077 ceph-mon[37153]: pgmap v6847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12059.638732 secs 2024-07-27T10:10:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:38 smithi029 ceph-mon[32610]: pgmap v6847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12059.638732 secs 2024-07-27T10:10:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:40 smithi077 ceph-mon[37153]: pgmap v6848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:40 smithi047 ceph-mon[36835]: pgmap v6848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:40 smithi029 ceph-mon[32610]: pgmap v6848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:42 smithi077 ceph-mon[37153]: pgmap v6849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:42 smithi047 ceph-mon[36835]: pgmap v6849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:42 smithi029 ceph-mon[32610]: pgmap v6849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12064.638855 secs 2024-07-27T10:10:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12064.638855 secs 2024-07-27T10:10:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12064.638855 secs 2024-07-27T10:10:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:44 smithi077 ceph-mon[37153]: pgmap v6850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:44 smithi047 ceph-mon[36835]: pgmap v6850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:44 smithi029 ceph-mon[32610]: pgmap v6850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:10:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:46 smithi077 ceph-mon[37153]: pgmap v6851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:46 smithi047 ceph-mon[36835]: pgmap v6851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:46 smithi029 ceph-mon[32610]: pgmap v6851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:10:47.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:47.570+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:10:47.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:47.570+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:10:47.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:47.570+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:10:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:48 smithi077 ceph-mon[37153]: pgmap v6852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12069.638937 secs 2024-07-27T10:10:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:48 smithi047 ceph-mon[36835]: pgmap v6852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12069.638937 secs 2024-07-27T10:10:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:48 smithi029 ceph-mon[32610]: pgmap v6852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:10:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12069.638937 secs 2024-07-27T10:10:50.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:50 smithi077 ceph-mon[37153]: pgmap v6853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:50 smithi047 ceph-mon[36835]: pgmap v6853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:50 smithi029 ceph-mon[32610]: pgmap v6853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:10:51.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:10:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:10:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:10:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:52 smithi077 ceph-mon[37153]: pgmap v6854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:10:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:10:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:52 smithi077 ceph-mon[37153]: pgmap v6855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:10:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:52 smithi077 ceph-mon[37153]: pgmap v6856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:10:52.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:10:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:52 smithi047 ceph-mon[36835]: pgmap v6854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:10:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:10:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:52 smithi047 ceph-mon[36835]: pgmap v6855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:10:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:52 smithi047 ceph-mon[36835]: pgmap v6856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:10:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:10:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:52 smithi029 ceph-mon[32610]: pgmap v6854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:10:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:10:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:10:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:52 smithi029 ceph-mon[32610]: pgmap v6855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:10:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:52 smithi029 ceph-mon[32610]: pgmap v6856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:10:52.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:10:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12074.639052 secs 2024-07-27T10:10:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12074.639052 secs 2024-07-27T10:10:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12074.639052 secs 2024-07-27T10:10:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:54 smithi077 ceph-mon[37153]: pgmap v6857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:10:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:54 smithi047 ceph-mon[36835]: pgmap v6857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:10:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:54 smithi029 ceph-mon[32610]: pgmap v6857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:10:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:56 smithi077 ceph-mon[37153]: pgmap v6858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:10:56.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:56 smithi047 ceph-mon[36835]: pgmap v6858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:10:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:56 smithi029 ceph-mon[32610]: pgmap v6858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:10:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12079.639118 secs 2024-07-27T10:10:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:10:58 smithi077 ceph-mon[37153]: pgmap v6859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:10:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12079.639118 secs 2024-07-27T10:10:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:10:58 smithi047 ceph-mon[36835]: pgmap v6859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:10:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12079.639118 secs 2024-07-27T10:10:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:10:58 smithi029 ceph-mon[32610]: pgmap v6859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:10:59.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:59.466+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:10:59.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:59.466+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:10:59.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:59.466+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:10:59.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:10:59.466+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:11:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:00 smithi077 ceph-mon[37153]: pgmap v6860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:11:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:00 smithi047 ceph-mon[36835]: pgmap v6860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:11:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:00 smithi029 ceph-mon[32610]: pgmap v6860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:11:02.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:11:02.574+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:11:02.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:11:02.574+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:11:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:11:02.574+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:11:02.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:02 smithi029 ceph-mon[32610]: pgmap v6861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:11:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:02 smithi077 ceph-mon[37153]: pgmap v6861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:11:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:02 smithi047 ceph-mon[36835]: pgmap v6861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:11:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12084.639244 secs 2024-07-27T10:11:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12084.639244 secs 2024-07-27T10:11:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12084.639244 secs 2024-07-27T10:11:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:04 smithi077 ceph-mon[37153]: pgmap v6862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:04 smithi047 ceph-mon[36835]: pgmap v6862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:04 smithi029 ceph-mon[32610]: pgmap v6862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:06 smithi077 ceph-mon[37153]: pgmap v6863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:06 smithi047 ceph-mon[36835]: pgmap v6863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:06 smithi029 ceph-mon[32610]: pgmap v6863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12089.639363 secs 2024-07-27T10:11:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:08 smithi077 ceph-mon[37153]: pgmap v6864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12089.639363 secs 2024-07-27T10:11:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:08 smithi047 ceph-mon[36835]: pgmap v6864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12089.639363 secs 2024-07-27T10:11:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:08 smithi029 ceph-mon[32610]: pgmap v6864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:10 smithi077 ceph-mon[37153]: pgmap v6865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:11:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:10 smithi047 ceph-mon[36835]: pgmap v6865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:11:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:10 smithi029 ceph-mon[32610]: pgmap v6865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:11:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:12 smithi077 ceph-mon[37153]: pgmap v6866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:12 smithi047 ceph-mon[36835]: pgmap v6866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:12 smithi029 ceph-mon[32610]: pgmap v6866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12094.639504 secs 2024-07-27T10:11:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12094.639504 secs 2024-07-27T10:11:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12094.639504 secs 2024-07-27T10:11:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:14 smithi077 ceph-mon[37153]: pgmap v6867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:14 smithi047 ceph-mon[36835]: pgmap v6867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:14 smithi029 ceph-mon[32610]: pgmap v6867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:16 smithi077 ceph-mon[37153]: pgmap v6868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:16 smithi047 ceph-mon[36835]: pgmap v6868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:16 smithi029 ceph-mon[32610]: pgmap v6868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:11:17.589+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:11:17.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:11:17.589+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:11:17.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:11:17.589+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:11:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12099.639600 secs 2024-07-27T10:11:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:18 smithi077 ceph-mon[37153]: pgmap v6869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:11:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12099.639600 secs 2024-07-27T10:11:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:18 smithi047 ceph-mon[36835]: pgmap v6869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:11:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12099.639600 secs 2024-07-27T10:11:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:18 smithi029 ceph-mon[32610]: pgmap v6869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:11:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:20 smithi077 ceph-mon[37153]: pgmap v6870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:11:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:20 smithi047 ceph-mon[36835]: pgmap v6870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:11:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:20 smithi029 ceph-mon[32610]: pgmap v6870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:11:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:22 smithi077 ceph-mon[37153]: pgmap v6871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:22 smithi047 ceph-mon[36835]: pgmap v6871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:22 smithi029 ceph-mon[32610]: pgmap v6871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12104.639727 secs 2024-07-27T10:11:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12104.639727 secs 2024-07-27T10:11:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12104.639727 secs 2024-07-27T10:11:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:24 smithi077 ceph-mon[37153]: pgmap v6872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:24 smithi047 ceph-mon[36835]: pgmap v6872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:24 smithi029 ceph-mon[32610]: pgmap v6872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:26 smithi077 ceph-mon[37153]: pgmap v6873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:26 smithi047 ceph-mon[36835]: pgmap v6873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:26 smithi029 ceph-mon[32610]: pgmap v6873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12109.639846 secs 2024-07-27T10:11:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:28 smithi077 ceph-mon[37153]: pgmap v6874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:11:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12109.639846 secs 2024-07-27T10:11:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:28 smithi047 ceph-mon[36835]: pgmap v6874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:11:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12109.639846 secs 2024-07-27T10:11:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:28 smithi029 ceph-mon[32610]: pgmap v6874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:11:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:30 smithi077 ceph-mon[37153]: pgmap v6875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:11:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:30 smithi047 ceph-mon[36835]: pgmap v6875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:11:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:30 smithi029 ceph-mon[32610]: pgmap v6875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:11:32.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:11:32.604+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:11:32.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:11:32.604+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:11:32.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:11:32.604+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:11:32.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:32 smithi029 ceph-mon[32610]: pgmap v6876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:32.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:32 smithi077 ceph-mon[37153]: pgmap v6876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:32 smithi047 ceph-mon[36835]: pgmap v6876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12114.639968 secs 2024-07-27T10:11:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12114.639968 secs 2024-07-27T10:11:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12114.639968 secs 2024-07-27T10:11:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:34 smithi047 ceph-mon[36835]: pgmap v6877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:34 smithi029 ceph-mon[32610]: pgmap v6877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:34 smithi077 ceph-mon[37153]: pgmap v6877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:36 smithi047 ceph-mon[36835]: pgmap v6878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:36 smithi029 ceph-mon[32610]: pgmap v6878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:36 smithi077 ceph-mon[37153]: pgmap v6878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:38.658 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12119.640061 secs 2024-07-27T10:11:38.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:38 smithi077 ceph-mon[37153]: pgmap v6879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:11:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12119.640061 secs 2024-07-27T10:11:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:38 smithi047 ceph-mon[36835]: pgmap v6879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:11:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12119.640061 secs 2024-07-27T10:11:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:38 smithi029 ceph-mon[32610]: pgmap v6879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:11:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:40 smithi047 ceph-mon[36835]: pgmap v6880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:11:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:40 smithi029 ceph-mon[32610]: pgmap v6880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:11:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:40 smithi077 ceph-mon[37153]: pgmap v6880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:11:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:42 smithi047 ceph-mon[36835]: pgmap v6881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:42 smithi029 ceph-mon[32610]: pgmap v6881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:42 smithi077 ceph-mon[37153]: pgmap v6881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12124.640131 secs 2024-07-27T10:11:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12124.640131 secs 2024-07-27T10:11:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12124.640131 secs 2024-07-27T10:11:44.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:44 smithi029 ceph-mon[32610]: pgmap v6882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:44 smithi047 ceph-mon[36835]: pgmap v6882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:44 smithi077 ceph-mon[37153]: pgmap v6882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:11:46.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:46 smithi047 ceph-mon[36835]: pgmap v6883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:46 smithi029 ceph-mon[32610]: pgmap v6883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:46 smithi077 ceph-mon[37153]: pgmap v6883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:11:47.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:11:47.612+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:11:47.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:11:47.612+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:11:47.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:11:47.612+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:11:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12129.640271 secs 2024-07-27T10:11:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:48 smithi047 ceph-mon[36835]: pgmap v6884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:11:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12129.640271 secs 2024-07-27T10:11:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:48 smithi029 ceph-mon[32610]: pgmap v6884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:11:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12129.640271 secs 2024-07-27T10:11:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:48 smithi077 ceph-mon[37153]: pgmap v6884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:11:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:50 smithi047 ceph-mon[36835]: pgmap v6885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:11:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:50 smithi029 ceph-mon[32610]: pgmap v6885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:11:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:50 smithi077 ceph-mon[37153]: pgmap v6885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:11:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:11:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:11:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:11:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:52 smithi047 ceph-mon[36835]: pgmap v6886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:11:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:11:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:11:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:52 smithi029 ceph-mon[32610]: pgmap v6886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:11:52.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:11:52.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:11:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:52 smithi077 ceph-mon[37153]: pgmap v6886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:11:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:11:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:11:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:11:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:53 smithi047 ceph-mon[36835]: pgmap v6887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:11:53.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:53 smithi047 ceph-mon[36835]: pgmap v6888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:11:53.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12134.640334 secs 2024-07-27T10:11:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:53 smithi029 ceph-mon[32610]: pgmap v6887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:11:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:53 smithi029 ceph-mon[32610]: pgmap v6888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:11:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12134.640334 secs 2024-07-27T10:11:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:53 smithi077 ceph-mon[37153]: pgmap v6887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:11:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:53 smithi077 ceph-mon[37153]: pgmap v6888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:11:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12134.640334 secs 2024-07-27T10:11:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:55 smithi047 ceph-mon[36835]: pgmap v6889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:11:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:55 smithi029 ceph-mon[32610]: pgmap v6889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:11:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:55 smithi077 ceph-mon[37153]: pgmap v6889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:11:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:57 smithi047 ceph-mon[36835]: pgmap v6890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:11:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:57 smithi029 ceph-mon[32610]: pgmap v6890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:11:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:57 smithi077 ceph-mon[37153]: pgmap v6890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:11:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12139.640476 secs 2024-07-27T10:11:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12139.640476 secs 2024-07-27T10:11:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12139.640476 secs 2024-07-27T10:11:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:11:59 smithi047 ceph-mon[36835]: pgmap v6891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:11:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:11:59 smithi029 ceph-mon[32610]: pgmap v6891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:11:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:11:59 smithi077 ceph-mon[37153]: pgmap v6891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:12:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:01 smithi047 ceph-mon[36835]: pgmap v6892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:12:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:01 smithi029 ceph-mon[32610]: pgmap v6892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:12:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:01 smithi077 ceph-mon[37153]: pgmap v6892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:12:02.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:12:02.628+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:12:02.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:12:02.628+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:12:02.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:12:02.628+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:12:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:03 smithi047 ceph-mon[36835]: pgmap v6893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:12:03.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12144.640600 secs 2024-07-27T10:12:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:03 smithi029 ceph-mon[32610]: pgmap v6893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:12:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12144.640600 secs 2024-07-27T10:12:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:03 smithi077 ceph-mon[37153]: pgmap v6893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:12:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12144.640600 secs 2024-07-27T10:12:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:05 smithi047 ceph-mon[36835]: pgmap v6894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:05 smithi029 ceph-mon[32610]: pgmap v6894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:05 smithi077 ceph-mon[37153]: pgmap v6894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:07 smithi047 ceph-mon[36835]: pgmap v6895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:07 smithi029 ceph-mon[32610]: pgmap v6895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:07 smithi077 ceph-mon[37153]: pgmap v6895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12149.640745 secs 2024-07-27T10:12:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12149.640745 secs 2024-07-27T10:12:08.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12149.640745 secs 2024-07-27T10:12:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:09 smithi047 ceph-mon[36835]: pgmap v6896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:09 smithi029 ceph-mon[32610]: pgmap v6896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:09 smithi077 ceph-mon[37153]: pgmap v6896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:11 smithi047 ceph-mon[36835]: pgmap v6897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:11 smithi029 ceph-mon[32610]: pgmap v6897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:11 smithi077 ceph-mon[37153]: pgmap v6897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:13 smithi047 ceph-mon[36835]: pgmap v6898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12154.640870 secs 2024-07-27T10:12:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:13 smithi029 ceph-mon[32610]: pgmap v6898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12154.640870 secs 2024-07-27T10:12:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:13 smithi077 ceph-mon[37153]: pgmap v6898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12154.640870 secs 2024-07-27T10:12:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:15 smithi029 ceph-mon[32610]: pgmap v6899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:15 smithi077 ceph-mon[37153]: pgmap v6899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:15 smithi047 ceph-mon[36835]: pgmap v6899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:17.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:12:17.643+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:12:17.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:12:17.643+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:12:17.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:12:17.643+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:12:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:17 smithi029 ceph-mon[32610]: pgmap v6900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:17 smithi077 ceph-mon[37153]: pgmap v6900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:17 smithi047 ceph-mon[36835]: pgmap v6900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12159.640949 secs 2024-07-27T10:12:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12159.640949 secs 2024-07-27T10:12:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12159.640949 secs 2024-07-27T10:12:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:19 smithi029 ceph-mon[32610]: pgmap v6901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:19 smithi077 ceph-mon[37153]: pgmap v6901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:19 smithi047 ceph-mon[36835]: pgmap v6901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:21 smithi029 ceph-mon[32610]: pgmap v6902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:21 smithi077 ceph-mon[37153]: pgmap v6902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:21 smithi047 ceph-mon[36835]: pgmap v6902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:23 smithi077 ceph-mon[37153]: pgmap v6903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12164.641074 secs 2024-07-27T10:12:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:23 smithi047 ceph-mon[36835]: pgmap v6903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12164.641074 secs 2024-07-27T10:12:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:23 smithi029 ceph-mon[32610]: pgmap v6903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12164.641074 secs 2024-07-27T10:12:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:25 smithi077 ceph-mon[37153]: pgmap v6904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:25 smithi047 ceph-mon[36835]: pgmap v6904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:25 smithi029 ceph-mon[32610]: pgmap v6904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:27 smithi077 ceph-mon[37153]: pgmap v6905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:27 smithi047 ceph-mon[36835]: pgmap v6905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:27 smithi029 ceph-mon[32610]: pgmap v6905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12169.641186 secs 2024-07-27T10:12:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12169.641186 secs 2024-07-27T10:12:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12169.641186 secs 2024-07-27T10:12:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:29 smithi077 ceph-mon[37153]: pgmap v6906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:29 smithi047 ceph-mon[36835]: pgmap v6906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:29 smithi029 ceph-mon[32610]: pgmap v6906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:31 smithi077 ceph-mon[37153]: pgmap v6907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:31 smithi047 ceph-mon[36835]: pgmap v6907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:31 smithi029 ceph-mon[32610]: pgmap v6907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:32.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:12:32.658+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:12:32.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:12:32.658+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:12:32.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:12:32.658+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:12:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:32 smithi077 ceph-mon[37153]: pgmap v6908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:32.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:32 smithi029 ceph-mon[32610]: pgmap v6908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:32 smithi047 ceph-mon[36835]: pgmap v6908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12174.641307 secs 2024-07-27T10:12:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12174.641307 secs 2024-07-27T10:12:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12174.641307 secs 2024-07-27T10:12:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:34 smithi077 ceph-mon[37153]: pgmap v6909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:34 smithi047 ceph-mon[36835]: pgmap v6909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:34 smithi029 ceph-mon[32610]: pgmap v6909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:36 smithi047 ceph-mon[36835]: pgmap v6910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:36 smithi029 ceph-mon[32610]: pgmap v6910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:37.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:36 smithi077 ceph-mon[37153]: pgmap v6910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12179.641436 secs 2024-07-27T10:12:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12179.641436 secs 2024-07-27T10:12:38.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12179.641436 secs 2024-07-27T10:12:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:38 smithi047 ceph-mon[36835]: pgmap v6911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:38 smithi029 ceph-mon[32610]: pgmap v6911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:38 smithi077 ceph-mon[37153]: pgmap v6911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:40 smithi047 ceph-mon[36835]: pgmap v6912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:40 smithi029 ceph-mon[32610]: pgmap v6912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:40 smithi077 ceph-mon[37153]: pgmap v6912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:42 smithi047 ceph-mon[36835]: pgmap v6913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12184.641582 secs 2024-07-27T10:12:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:42 smithi029 ceph-mon[32610]: pgmap v6913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12184.641582 secs 2024-07-27T10:12:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:42 smithi077 ceph-mon[37153]: pgmap v6913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12184.641582 secs 2024-07-27T10:12:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:44 smithi047 ceph-mon[36835]: pgmap v6914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:44 smithi029 ceph-mon[32610]: pgmap v6914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:44 smithi077 ceph-mon[37153]: pgmap v6914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:12:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:46 smithi047 ceph-mon[36835]: pgmap v6915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:46 smithi029 ceph-mon[32610]: pgmap v6915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:46 smithi077 ceph-mon[37153]: pgmap v6915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:12:47.673+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:12:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:12:47.673+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:12:47.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:12:47.673+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:12:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12189.641711 secs 2024-07-27T10:12:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12189.641711 secs 2024-07-27T10:12:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12189.641711 secs 2024-07-27T10:12:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:48 smithi047 ceph-mon[36835]: pgmap v6916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:48 smithi029 ceph-mon[32610]: pgmap v6916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:48 smithi077 ceph-mon[37153]: pgmap v6916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:12:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:50 smithi047 ceph-mon[36835]: pgmap v6917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:50 smithi029 ceph-mon[32610]: pgmap v6917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:51.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:50 smithi077 ceph-mon[37153]: pgmap v6917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:12:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:52 smithi047 ceph-mon[36835]: pgmap v6918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:12:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12194.641780 secs 2024-07-27T10:12:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:12:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:12:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:52 smithi047 ceph-mon[36835]: pgmap v6919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:12:53.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:52 smithi047 ceph-mon[36835]: pgmap v6920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T10:12:53.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:12:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:52 smithi029 ceph-mon[32610]: pgmap v6918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:12:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12194.641780 secs 2024-07-27T10:12:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:12:53.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:12:53.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:52 smithi029 ceph-mon[32610]: pgmap v6919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:12:53.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:52 smithi029 ceph-mon[32610]: pgmap v6920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T10:12:53.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:12:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:52 smithi077 ceph-mon[37153]: pgmap v6918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:12:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:12:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12194.641780 secs 2024-07-27T10:12:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:12:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:12:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:52 smithi077 ceph-mon[37153]: pgmap v6919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:12:53.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:52 smithi077 ceph-mon[37153]: pgmap v6920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T10:12:53.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:12:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:55 smithi047 ceph-mon[36835]: pgmap v6921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:12:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:55 smithi029 ceph-mon[32610]: pgmap v6921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:12:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:55 smithi077 ceph-mon[37153]: pgmap v6921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:12:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:57 smithi047 ceph-mon[36835]: pgmap v6922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:12:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:57 smithi029 ceph-mon[32610]: pgmap v6922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:12:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:57 smithi077 ceph-mon[37153]: pgmap v6922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:12:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12199.641906 secs 2024-07-27T10:12:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12199.641906 secs 2024-07-27T10:12:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12199.641906 secs 2024-07-27T10:12:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:12:59 smithi047 ceph-mon[36835]: pgmap v6923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:12:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:12:59 smithi029 ceph-mon[32610]: pgmap v6923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:12:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:12:59 smithi077 ceph-mon[37153]: pgmap v6923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:13:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:01 smithi047 ceph-mon[36835]: pgmap v6924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:13:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:01 smithi029 ceph-mon[32610]: pgmap v6924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:13:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:01 smithi077 ceph-mon[37153]: pgmap v6924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:13:02.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:02.689+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:13:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:02.689+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:13:02.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:02.689+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:13:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12204.642036 secs 2024-07-27T10:13:03.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:03 smithi029 ceph-mon[32610]: pgmap v6925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:13:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12204.642036 secs 2024-07-27T10:13:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:03 smithi077 ceph-mon[37153]: pgmap v6925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:13:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12204.642036 secs 2024-07-27T10:13:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:03 smithi047 ceph-mon[36835]: pgmap v6925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:13:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:05 smithi029 ceph-mon[32610]: pgmap v6926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:05 smithi077 ceph-mon[37153]: pgmap v6926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:05 smithi047 ceph-mon[36835]: pgmap v6926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:07 smithi029 ceph-mon[32610]: pgmap v6927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:07 smithi077 ceph-mon[37153]: pgmap v6927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:07 smithi047 ceph-mon[36835]: pgmap v6927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12209.642160 secs 2024-07-27T10:13:08.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12209.642160 secs 2024-07-27T10:13:08.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12209.642160 secs 2024-07-27T10:13:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:09 smithi029 ceph-mon[32610]: pgmap v6928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:09 smithi077 ceph-mon[37153]: pgmap v6928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:09 smithi047 ceph-mon[36835]: pgmap v6928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:11 smithi077 ceph-mon[37153]: pgmap v6929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:11 smithi047 ceph-mon[36835]: pgmap v6929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:11 smithi029 ceph-mon[32610]: pgmap v6929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12214.642289 secs 2024-07-27T10:13:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:13 smithi077 ceph-mon[37153]: pgmap v6930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12214.642289 secs 2024-07-27T10:13:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:13 smithi047 ceph-mon[36835]: pgmap v6930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12214.642289 secs 2024-07-27T10:13:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:13 smithi029 ceph-mon[32610]: pgmap v6930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:15 smithi077 ceph-mon[37153]: pgmap v6931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:15 smithi047 ceph-mon[36835]: pgmap v6931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:15 smithi029 ceph-mon[32610]: pgmap v6931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:17.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:17.702+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:13:17.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:17.702+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:13:17.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:17.702+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:13:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:17 smithi077 ceph-mon[37153]: pgmap v6932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:17 smithi047 ceph-mon[36835]: pgmap v6932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:17 smithi029 ceph-mon[32610]: pgmap v6932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12219.642414 secs 2024-07-27T10:13:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12219.642414 secs 2024-07-27T10:13:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12219.642414 secs 2024-07-27T10:13:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:19 smithi077 ceph-mon[37153]: pgmap v6933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:19 smithi047 ceph-mon[36835]: pgmap v6933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:19 smithi029 ceph-mon[32610]: pgmap v6933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:21 smithi077 ceph-mon[37153]: pgmap v6934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:21 smithi047 ceph-mon[36835]: pgmap v6934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:21 smithi029 ceph-mon[32610]: pgmap v6934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12224.642573 secs 2024-07-27T10:13:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:23 smithi077 ceph-mon[37153]: pgmap v6935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12224.642573 secs 2024-07-27T10:13:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:23 smithi047 ceph-mon[36835]: pgmap v6935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12224.642573 secs 2024-07-27T10:13:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:23 smithi029 ceph-mon[32610]: pgmap v6935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:25 smithi077 ceph-mon[37153]: pgmap v6936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:25 smithi047 ceph-mon[36835]: pgmap v6936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:25 smithi029 ceph-mon[32610]: pgmap v6936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:26 smithi077 ceph-mon[37153]: pgmap v6937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:26 smithi047 ceph-mon[36835]: pgmap v6937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:26 smithi029 ceph-mon[32610]: pgmap v6937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12229.642710 secs 2024-07-27T10:13:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12229.642710 secs 2024-07-27T10:13:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12229.642710 secs 2024-07-27T10:13:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:28 smithi047 ceph-mon[36835]: pgmap v6938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:28 smithi029 ceph-mon[32610]: pgmap v6938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:28 smithi077 ceph-mon[37153]: pgmap v6938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:31 smithi047 ceph-mon[36835]: pgmap v6939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:31 smithi029 ceph-mon[32610]: pgmap v6939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:31 smithi077 ceph-mon[37153]: pgmap v6939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:32.716+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:13:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:32.716+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:13:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:32.716+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:13:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12234.642850 secs 2024-07-27T10:13:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:33 smithi047 ceph-mon[36835]: pgmap v6940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12234.642850 secs 2024-07-27T10:13:33.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:33 smithi029 ceph-mon[32610]: pgmap v6940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12234.642850 secs 2024-07-27T10:13:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:33 smithi077 ceph-mon[37153]: pgmap v6940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:35 smithi029 ceph-mon[32610]: pgmap v6941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:35 smithi077 ceph-mon[37153]: pgmap v6941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:35 smithi047 ceph-mon[36835]: pgmap v6941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:37 smithi029 ceph-mon[32610]: pgmap v6942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:37 smithi077 ceph-mon[37153]: pgmap v6942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:37 smithi047 ceph-mon[36835]: pgmap v6942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12239.642911 secs 2024-07-27T10:13:38.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12239.642911 secs 2024-07-27T10:13:38.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12239.642911 secs 2024-07-27T10:13:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:39 smithi029 ceph-mon[32610]: pgmap v6943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:39 smithi077 ceph-mon[37153]: pgmap v6943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:39 smithi047 ceph-mon[36835]: pgmap v6943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:41 smithi029 ceph-mon[32610]: pgmap v6944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:41 smithi077 ceph-mon[37153]: pgmap v6944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:41 smithi047 ceph-mon[36835]: pgmap v6944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12244.643039 secs 2024-07-27T10:13:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:43 smithi077 ceph-mon[37153]: pgmap v6945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12244.643039 secs 2024-07-27T10:13:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:43 smithi047 ceph-mon[36835]: pgmap v6945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12244.643039 secs 2024-07-27T10:13:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:43 smithi029 ceph-mon[32610]: pgmap v6945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:45 smithi077 ceph-mon[37153]: pgmap v6946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:45 smithi047 ceph-mon[36835]: pgmap v6946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:45 smithi029 ceph-mon[32610]: pgmap v6946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:13:47.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:47.732+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:13:47.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:47.732+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:13:47.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:47.732+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:13:47.734 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:47 smithi029 ceph-mon[32610]: pgmap v6947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:47 smithi077 ceph-mon[37153]: pgmap v6947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:47 smithi047 ceph-mon[36835]: pgmap v6947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:13:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12249.643167 secs 2024-07-27T10:13:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12249.643167 secs 2024-07-27T10:13:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12249.643167 secs 2024-07-27T10:13:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:49 smithi077 ceph-mon[37153]: pgmap v6948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:49 smithi047 ceph-mon[36835]: pgmap v6948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:49 smithi029 ceph-mon[32610]: pgmap v6948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:13:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:51 smithi077 ceph-mon[37153]: pgmap v6949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:51 smithi047 ceph-mon[36835]: pgmap v6949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:51 smithi029 ceph-mon[32610]: pgmap v6949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:13:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:13:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:13:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:13:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12254.643293 secs 2024-07-27T10:13:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:53 smithi077 ceph-mon[37153]: pgmap v6950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:13:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:13:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:13:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12254.643293 secs 2024-07-27T10:13:53.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:53 smithi047 ceph-mon[36835]: pgmap v6950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:53.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:13:53.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:13:53.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:13:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12254.643293 secs 2024-07-27T10:13:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:53 smithi029 ceph-mon[32610]: pgmap v6950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:13:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:13:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:13:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:13:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:54 smithi077 ceph-mon[37153]: pgmap v6951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:13:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:54 smithi077 ceph-mon[37153]: pgmap v6952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:13:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:54 smithi047 ceph-mon[36835]: pgmap v6951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:13:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:54 smithi047 ceph-mon[36835]: pgmap v6952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:13:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:54 smithi029 ceph-mon[32610]: pgmap v6951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:13:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:54 smithi029 ceph-mon[32610]: pgmap v6952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:13:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:55 smithi077 ceph-mon[37153]: pgmap v6953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:13:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:55 smithi047 ceph-mon[36835]: pgmap v6953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:13:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:55 smithi029 ceph-mon[32610]: pgmap v6953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:13:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12259.643436 secs 2024-07-27T10:13:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:57 smithi077 ceph-mon[37153]: pgmap v6954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:13:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12259.643436 secs 2024-07-27T10:13:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:57 smithi047 ceph-mon[36835]: pgmap v6954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:13:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12259.643436 secs 2024-07-27T10:13:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:57 smithi029 ceph-mon[32610]: pgmap v6954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:13:59.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:59.494+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:13:59.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:59.494+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:13:59.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:59.494+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:13:59.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:13:59.494+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:14:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:13:59 smithi077 ceph-mon[37153]: pgmap v6955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:14:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:13:59 smithi047 ceph-mon[36835]: pgmap v6955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:14:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:13:59 smithi029 ceph-mon[32610]: pgmap v6955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:14:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:01 smithi077 ceph-mon[37153]: pgmap v6956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:14:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:01 smithi047 ceph-mon[36835]: pgmap v6956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:14:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:01 smithi029 ceph-mon[32610]: pgmap v6956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:14:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:14:02.748+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:14:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:14:02.748+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:14:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:14:02.748+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:14:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12264.643575 secs 2024-07-27T10:14:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12264.643575 secs 2024-07-27T10:14:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12264.643575 secs 2024-07-27T10:14:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:04 smithi077 ceph-mon[37153]: pgmap v6957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:14:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:04 smithi047 ceph-mon[36835]: pgmap v6957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:14:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:04 smithi029 ceph-mon[32610]: pgmap v6957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:14:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:06 smithi077 ceph-mon[37153]: pgmap v6958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:06 smithi047 ceph-mon[36835]: pgmap v6958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:06 smithi029 ceph-mon[32610]: pgmap v6958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12269.643695 secs 2024-07-27T10:14:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:08 smithi077 ceph-mon[37153]: pgmap v6959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12269.643695 secs 2024-07-27T10:14:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:08 smithi047 ceph-mon[36835]: pgmap v6959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12269.643695 secs 2024-07-27T10:14:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:08 smithi029 ceph-mon[32610]: pgmap v6959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:10 smithi077 ceph-mon[37153]: pgmap v6960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:10 smithi047 ceph-mon[36835]: pgmap v6960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:10 smithi029 ceph-mon[32610]: pgmap v6960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:12 smithi077 ceph-mon[37153]: pgmap v6961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:12 smithi047 ceph-mon[36835]: pgmap v6961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:12 smithi029 ceph-mon[32610]: pgmap v6961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12274.643831 secs 2024-07-27T10:14:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12274.643831 secs 2024-07-27T10:14:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12274.643831 secs 2024-07-27T10:14:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:14 smithi077 ceph-mon[37153]: pgmap v6962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:14 smithi047 ceph-mon[36835]: pgmap v6962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:14 smithi029 ceph-mon[32610]: pgmap v6962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:16 smithi077 ceph-mon[37153]: pgmap v6963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:14:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:16 smithi047 ceph-mon[36835]: pgmap v6963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:14:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:16 smithi029 ceph-mon[32610]: pgmap v6963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:14:17.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:14:17.761+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:14:17.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:14:17.761+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:14:17.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:14:17.761+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:14:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12279.643935 secs 2024-07-27T10:14:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:18 smithi077 ceph-mon[37153]: pgmap v6964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12279.643935 secs 2024-07-27T10:14:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:18 smithi047 ceph-mon[36835]: pgmap v6964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12279.643935 secs 2024-07-27T10:14:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:18 smithi029 ceph-mon[32610]: pgmap v6964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:20 smithi077 ceph-mon[37153]: pgmap v6965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:20 smithi047 ceph-mon[36835]: pgmap v6965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:20 smithi029 ceph-mon[32610]: pgmap v6965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:22 smithi077 ceph-mon[37153]: pgmap v6966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:22 smithi047 ceph-mon[36835]: pgmap v6966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:22 smithi029 ceph-mon[32610]: pgmap v6966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12284.644080 secs 2024-07-27T10:14:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12284.644080 secs 2024-07-27T10:14:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12284.644080 secs 2024-07-27T10:14:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:24 smithi077 ceph-mon[37153]: pgmap v6967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:24 smithi047 ceph-mon[36835]: pgmap v6967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:24 smithi029 ceph-mon[32610]: pgmap v6967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:26 smithi077 ceph-mon[37153]: pgmap v6968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:26.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:26 smithi047 ceph-mon[36835]: pgmap v6968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:26 smithi029 ceph-mon[32610]: pgmap v6968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12289.644222 secs 2024-07-27T10:14:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:28 smithi077 ceph-mon[37153]: pgmap v6969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12289.644222 secs 2024-07-27T10:14:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:28 smithi047 ceph-mon[36835]: pgmap v6969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12289.644222 secs 2024-07-27T10:14:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:28 smithi029 ceph-mon[32610]: pgmap v6969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:30 smithi077 ceph-mon[37153]: pgmap v6970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:30 smithi047 ceph-mon[36835]: pgmap v6970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:30 smithi029 ceph-mon[32610]: pgmap v6970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:32 smithi077 ceph-mon[37153]: pgmap v6971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:32 smithi047 ceph-mon[36835]: pgmap v6971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:32 smithi029 ceph-mon[32610]: pgmap v6971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:32.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:14:32.775+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:14:32.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:14:32.775+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:14:32.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:14:32.775+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:14:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12294.644308 secs 2024-07-27T10:14:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12294.644308 secs 2024-07-27T10:14:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12294.644308 secs 2024-07-27T10:14:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:34 smithi077 ceph-mon[37153]: pgmap v6972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:34 smithi047 ceph-mon[36835]: pgmap v6972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:34 smithi029 ceph-mon[32610]: pgmap v6972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:36 smithi077 ceph-mon[37153]: pgmap v6973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:36 smithi047 ceph-mon[36835]: pgmap v6973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:36 smithi029 ceph-mon[32610]: pgmap v6973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12299.644434 secs 2024-07-27T10:14:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:38 smithi077 ceph-mon[37153]: pgmap v6974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12299.644434 secs 2024-07-27T10:14:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:38 smithi047 ceph-mon[36835]: pgmap v6974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12299.644434 secs 2024-07-27T10:14:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:38 smithi029 ceph-mon[32610]: pgmap v6974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:40 smithi077 ceph-mon[37153]: pgmap v6975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:40 smithi047 ceph-mon[36835]: pgmap v6975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:40 smithi029 ceph-mon[32610]: pgmap v6975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:42 smithi077 ceph-mon[37153]: pgmap v6976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:42 smithi047 ceph-mon[36835]: pgmap v6976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:42 smithi029 ceph-mon[32610]: pgmap v6976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12304.644586 secs 2024-07-27T10:14:43.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12304.644586 secs 2024-07-27T10:14:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12304.644586 secs 2024-07-27T10:14:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:44 smithi077 ceph-mon[37153]: pgmap v6977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:44 smithi047 ceph-mon[36835]: pgmap v6977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:44 smithi029 ceph-mon[32610]: pgmap v6977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:46 smithi077 ceph-mon[37153]: pgmap v6978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:46 smithi047 ceph-mon[36835]: pgmap v6978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:46 smithi029 ceph-mon[32610]: pgmap v6978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:47.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:14:47.791+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:14:47.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:14:47.791+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:14:47.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:14:47.791+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:14:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12309.644728 secs 2024-07-27T10:14:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:48 smithi077 ceph-mon[37153]: pgmap v6979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12309.644728 secs 2024-07-27T10:14:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:48 smithi047 ceph-mon[36835]: pgmap v6979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12309.644728 secs 2024-07-27T10:14:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:48 smithi029 ceph-mon[32610]: pgmap v6979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:50 smithi077 ceph-mon[37153]: pgmap v6980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:50 smithi047 ceph-mon[36835]: pgmap v6980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:50 smithi029 ceph-mon[32610]: pgmap v6980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:14:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:52 smithi077 ceph-mon[37153]: pgmap v6981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:52 smithi047 ceph-mon[36835]: pgmap v6981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:52 smithi029 ceph-mon[32610]: pgmap v6981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12314.644830 secs 2024-07-27T10:14:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:14:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12314.644830 secs 2024-07-27T10:14:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:14:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12314.644830 secs 2024-07-27T10:14:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:14:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:54 smithi029 ceph-mon[32610]: pgmap v6982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:54 smithi077 ceph-mon[37153]: pgmap v6982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:54 smithi047 ceph-mon[36835]: pgmap v6982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:14:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:56 smithi047 ceph-mon[36835]: pgmap v6983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:56 smithi077 ceph-mon[37153]: pgmap v6983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:56 smithi029 ceph-mon[32610]: pgmap v6983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:14:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12319.644970 secs 2024-07-27T10:14:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: pgmap v6984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:58.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:14:58.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:14:58.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:14:58.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:14:58.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:14:58.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:14:58.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:14:58.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:14:58.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: pgmap v6985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:14:58.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: pgmap v6986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T10:14:58.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12319.644970 secs 2024-07-27T10:14:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: pgmap v6984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:14:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:14:58.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:14:58.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:14:58.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:14:58.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:14:58.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:14:58.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:14:58.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: pgmap v6985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:14:58.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: pgmap v6986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T10:14:58.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12319.644970 secs 2024-07-27T10:14:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: pgmap v6984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:14:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:14:58.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:14:58.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:14:58.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:14:58.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:14:58.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:14:58.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:14:58.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:14:58.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:14:58.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: pgmap v6985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:14:58.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: pgmap v6986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-27T10:14:58.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:15:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:00 smithi077 ceph-mon[37153]: pgmap v6987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 7 op/s 2024-07-27T10:15:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:00 smithi047 ceph-mon[36835]: pgmap v6987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 7 op/s 2024-07-27T10:15:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:00 smithi029 ceph-mon[32610]: pgmap v6987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 7 op/s 2024-07-27T10:15:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:02 smithi077 ceph-mon[37153]: pgmap v6988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:15:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:02 smithi047 ceph-mon[36835]: pgmap v6988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:15:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:02 smithi029 ceph-mon[32610]: pgmap v6988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:15:02.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:15:02.807+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:15:02.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:15:02.807+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:15:02.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:15:02.807+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:15:03.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12324.645098 secs 2024-07-27T10:15:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12324.645098 secs 2024-07-27T10:15:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12324.645098 secs 2024-07-27T10:15:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:04 smithi077 ceph-mon[37153]: pgmap v6989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:15:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:04 smithi047 ceph-mon[36835]: pgmap v6989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:15:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:04 smithi029 ceph-mon[32610]: pgmap v6989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:15:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:06 smithi077 ceph-mon[37153]: pgmap v6990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:15:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:06 smithi047 ceph-mon[36835]: pgmap v6990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:15:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:06 smithi029 ceph-mon[32610]: pgmap v6990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:15:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12329.645194 secs 2024-07-27T10:15:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:08 smithi077 ceph-mon[37153]: pgmap v6991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T10:15:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12329.645194 secs 2024-07-27T10:15:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:08 smithi047 ceph-mon[36835]: pgmap v6991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T10:15:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12329.645194 secs 2024-07-27T10:15:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:08 smithi029 ceph-mon[32610]: pgmap v6991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T10:15:10.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:10 smithi077 ceph-mon[37153]: pgmap v6992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:15:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:10 smithi047 ceph-mon[36835]: pgmap v6992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:15:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:10 smithi029 ceph-mon[32610]: pgmap v6992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:15:12.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:12 smithi077 ceph-mon[37153]: pgmap v6993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:12 smithi047 ceph-mon[36835]: pgmap v6993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:12 smithi029 ceph-mon[32610]: pgmap v6993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12334.645329 secs 2024-07-27T10:15:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12334.645329 secs 2024-07-27T10:15:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12334.645329 secs 2024-07-27T10:15:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:14 smithi077 ceph-mon[37153]: pgmap v6994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:14 smithi047 ceph-mon[36835]: pgmap v6994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:14 smithi029 ceph-mon[32610]: pgmap v6994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:16 smithi077 ceph-mon[37153]: pgmap v6995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:16 smithi047 ceph-mon[36835]: pgmap v6995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:16 smithi029 ceph-mon[32610]: pgmap v6995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:17.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:15:17.821+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:15:17.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:15:17.821+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:15:17.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:15:17.821+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:15:18.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12339.645459 secs 2024-07-27T10:15:18.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:18 smithi047 ceph-mon[36835]: pgmap v6996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12339.645459 secs 2024-07-27T10:15:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:18 smithi029 ceph-mon[32610]: pgmap v6996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12339.645459 secs 2024-07-27T10:15:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:18 smithi077 ceph-mon[37153]: pgmap v6996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:20 smithi047 ceph-mon[36835]: pgmap v6997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:15:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:20 smithi029 ceph-mon[32610]: pgmap v6997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:15:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:20 smithi077 ceph-mon[37153]: pgmap v6997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:15:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:22 smithi047 ceph-mon[36835]: pgmap v6998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:22 smithi029 ceph-mon[32610]: pgmap v6998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:22 smithi077 ceph-mon[37153]: pgmap v6998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12344.645584 secs 2024-07-27T10:15:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12344.645584 secs 2024-07-27T10:15:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12344.645584 secs 2024-07-27T10:15:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:24 smithi047 ceph-mon[36835]: pgmap v6999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:24 smithi029 ceph-mon[32610]: pgmap v6999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:24 smithi077 ceph-mon[37153]: pgmap v6999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:26 smithi047 ceph-mon[36835]: pgmap v7000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:26 smithi029 ceph-mon[32610]: pgmap v7000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:26 smithi077 ceph-mon[37153]: pgmap v7000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12349.645712 secs 2024-07-27T10:15:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:28 smithi047 ceph-mon[36835]: pgmap v7001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12349.645712 secs 2024-07-27T10:15:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:28 smithi029 ceph-mon[32610]: pgmap v7001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12349.645712 secs 2024-07-27T10:15:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:28 smithi077 ceph-mon[37153]: pgmap v7001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:30 smithi047 ceph-mon[36835]: pgmap v7002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:15:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:30 smithi029 ceph-mon[32610]: pgmap v7002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:15:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:30 smithi077 ceph-mon[37153]: pgmap v7002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:15:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:32 smithi047 ceph-mon[36835]: pgmap v7003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:32 smithi029 ceph-mon[32610]: pgmap v7003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:15:32.829+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:15:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:15:32.829+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:15:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:15:32.829+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:15:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:32 smithi077 ceph-mon[37153]: pgmap v7003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12354.645838 secs 2024-07-27T10:15:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12354.645838 secs 2024-07-27T10:15:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12354.645838 secs 2024-07-27T10:15:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:34 smithi047 ceph-mon[36835]: pgmap v7004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:34 smithi029 ceph-mon[32610]: pgmap v7004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:34 smithi077 ceph-mon[37153]: pgmap v7004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:36.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:36 smithi047 ceph-mon[36835]: pgmap v7005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:36 smithi029 ceph-mon[32610]: pgmap v7005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:36 smithi077 ceph-mon[37153]: pgmap v7005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:38.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12359.645985 secs 2024-07-27T10:15:38.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:38 smithi077 ceph-mon[37153]: pgmap v7006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12359.645985 secs 2024-07-27T10:15:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:38 smithi047 ceph-mon[36835]: pgmap v7006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12359.645985 secs 2024-07-27T10:15:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:38 smithi029 ceph-mon[32610]: pgmap v7006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:40 smithi047 ceph-mon[36835]: pgmap v7007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:15:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:40 smithi029 ceph-mon[32610]: pgmap v7007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:15:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:40 smithi077 ceph-mon[37153]: pgmap v7007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:15:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:42 smithi047 ceph-mon[36835]: pgmap v7008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:42 smithi029 ceph-mon[32610]: pgmap v7008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:42 smithi077 ceph-mon[37153]: pgmap v7008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12364.646106 secs 2024-07-27T10:15:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12364.646106 secs 2024-07-27T10:15:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12364.646106 secs 2024-07-27T10:15:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:44 smithi047 ceph-mon[36835]: pgmap v7009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:44 smithi029 ceph-mon[32610]: pgmap v7009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:44 smithi077 ceph-mon[37153]: pgmap v7009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:46 smithi047 ceph-mon[36835]: pgmap v7010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:46 smithi029 ceph-mon[32610]: pgmap v7010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:46 smithi077 ceph-mon[37153]: pgmap v7010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:15:47.840+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:15:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:15:47.840+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:15:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:15:47.840+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:15:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12369.646228 secs 2024-07-27T10:15:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:48 smithi047 ceph-mon[36835]: pgmap v7011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12369.646228 secs 2024-07-27T10:15:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:48 smithi029 ceph-mon[32610]: pgmap v7011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12369.646228 secs 2024-07-27T10:15:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:48 smithi077 ceph-mon[37153]: pgmap v7011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:50 smithi047 ceph-mon[36835]: pgmap v7012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:15:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:50 smithi029 ceph-mon[32610]: pgmap v7012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:15:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:50 smithi077 ceph-mon[37153]: pgmap v7012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:15:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:52 smithi047 ceph-mon[36835]: pgmap v7013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:52 smithi029 ceph-mon[32610]: pgmap v7013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:52 smithi077 ceph-mon[37153]: pgmap v7013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:15:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12374.646373 secs 2024-07-27T10:15:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12374.646373 secs 2024-07-27T10:15:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12374.646373 secs 2024-07-27T10:15:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:54 smithi047 ceph-mon[36835]: pgmap v7014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:54 smithi029 ceph-mon[32610]: pgmap v7014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:54 smithi077 ceph-mon[37153]: pgmap v7014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:15:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:56 smithi047 ceph-mon[36835]: pgmap v7015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:56 smithi029 ceph-mon[32610]: pgmap v7015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:56 smithi077 ceph-mon[37153]: pgmap v7015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:15:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:15:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:15:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:15:58.525 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12379.646505 secs 2024-07-27T10:15:58.525 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:58 smithi077 ceph-mon[37153]: pgmap v7016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:58.525 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:15:58.525 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:15:58.526 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:15:58.526 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:15:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12379.646505 secs 2024-07-27T10:15:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:58 smithi047 ceph-mon[36835]: pgmap v7016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:15:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:15:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:15:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:15:58.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12379.646505 secs 2024-07-27T10:15:58.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:58 smithi029 ceph-mon[32610]: pgmap v7016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:15:58.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:15:58.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:15:58.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:15:58.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:15:59.520 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:59 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:15:59.520 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:15:59 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:15:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:59 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:15:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:15:59 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:15:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:59 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:15:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:15:59 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:16:00.497 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:00 smithi077 ceph-mon[37153]: pgmap v7017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:16:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:00 smithi029 ceph-mon[32610]: pgmap v7017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:16:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:00 smithi047 ceph-mon[36835]: pgmap v7017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:16:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:16:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:16:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:16:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:16:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:16:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:16:02.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:16:02.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:16:02.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:16:02.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:16:02.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: pgmap v7018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-27T10:16:02.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: pgmap v7019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T10:16:02.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:16:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:16:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:16:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:16:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:16:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:16:02.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:16:02.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:16:02.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:16:02.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:16:02.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:16:02.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: pgmap v7018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-27T10:16:02.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: pgmap v7019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T10:16:02.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:16:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:16:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:16:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:16:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:16:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:16:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:16:02.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:16:02.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:16:02.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:16:02.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:16:02.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: pgmap v7018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-27T10:16:02.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: pgmap v7019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T10:16:02.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:16:02.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:02.854+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:16:02.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:02.854+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:16:02.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:02.854+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:16:03.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12384.646568 secs 2024-07-27T10:16:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12384.646568 secs 2024-07-27T10:16:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12384.646568 secs 2024-07-27T10:16:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:03 smithi047 ceph-mon[36835]: pgmap v7020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:16:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:03 smithi029 ceph-mon[32610]: pgmap v7020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:16:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:03 smithi077 ceph-mon[37153]: pgmap v7020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:16:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:05 smithi029 ceph-mon[32610]: pgmap v7021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:16:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:05 smithi077 ceph-mon[37153]: pgmap v7021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:16:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:05 smithi047 ceph-mon[36835]: pgmap v7021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:16:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:07 smithi029 ceph-mon[32610]: pgmap v7022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T10:16:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12389.646695 secs 2024-07-27T10:16:08.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:07 smithi047 ceph-mon[36835]: pgmap v7022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T10:16:08.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12389.646695 secs 2024-07-27T10:16:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:07 smithi077 ceph-mon[37153]: pgmap v7022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T10:16:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12389.646695 secs 2024-07-27T10:16:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:09 smithi029 ceph-mon[32610]: pgmap v7023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:16:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:09 smithi077 ceph-mon[37153]: pgmap v7023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:16:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:09 smithi047 ceph-mon[36835]: pgmap v7023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:16:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:11 smithi077 ceph-mon[37153]: pgmap v7024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:16:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:11 smithi047 ceph-mon[36835]: pgmap v7024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:16:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:11 smithi029 ceph-mon[32610]: pgmap v7024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:16:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12394.646819 secs 2024-07-27T10:16:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12394.646819 secs 2024-07-27T10:16:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12394.646819 secs 2024-07-27T10:16:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:13 smithi077 ceph-mon[37153]: pgmap v7025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:16:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:13 smithi047 ceph-mon[36835]: pgmap v7025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:16:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:13 smithi029 ceph-mon[32610]: pgmap v7025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:16:14.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:14.515+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:16:14.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:14.515+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:16:14.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:14.515+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:16:14.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:14.515+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:16:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:15 smithi077 ceph-mon[37153]: pgmap v7026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:15 smithi047 ceph-mon[36835]: pgmap v7026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:15 smithi029 ceph-mon[32610]: pgmap v7026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:17.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:17.869+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:16:17.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:17.869+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:16:17.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:17.869+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:16:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:17 smithi029 ceph-mon[32610]: pgmap v7027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12399.646900 secs 2024-07-27T10:16:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:17 smithi077 ceph-mon[37153]: pgmap v7027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12399.646900 secs 2024-07-27T10:16:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:17 smithi047 ceph-mon[36835]: pgmap v7027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12399.646900 secs 2024-07-27T10:16:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:19 smithi077 ceph-mon[37153]: pgmap v7028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:16:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:19 smithi047 ceph-mon[36835]: pgmap v7028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:16:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:19 smithi029 ceph-mon[32610]: pgmap v7028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:16:22.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:21 smithi077 ceph-mon[37153]: pgmap v7029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:21 smithi047 ceph-mon[36835]: pgmap v7029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:21 smithi029 ceph-mon[32610]: pgmap v7029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12404.647046 secs 2024-07-27T10:16:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12404.647046 secs 2024-07-27T10:16:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12404.647046 secs 2024-07-27T10:16:24.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:23 smithi077 ceph-mon[37153]: pgmap v7030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:16:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:23 smithi047 ceph-mon[36835]: pgmap v7030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:16:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:23 smithi029 ceph-mon[32610]: pgmap v7030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:16:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:25 smithi077 ceph-mon[37153]: pgmap v7031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:25 smithi047 ceph-mon[36835]: pgmap v7031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:25 smithi029 ceph-mon[32610]: pgmap v7031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:27 smithi077 ceph-mon[37153]: pgmap v7032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12409.647193 secs 2024-07-27T10:16:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:27 smithi047 ceph-mon[36835]: pgmap v7032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12409.647193 secs 2024-07-27T10:16:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:27 smithi029 ceph-mon[32610]: pgmap v7032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12409.647193 secs 2024-07-27T10:16:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:30 smithi077 ceph-mon[37153]: pgmap v7033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:16:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:30 smithi047 ceph-mon[36835]: pgmap v7033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:16:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:29 smithi029 ceph-mon[32610]: pgmap v7033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:16:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:32 smithi077 ceph-mon[37153]: pgmap v7034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:32 smithi047 ceph-mon[36835]: pgmap v7034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:32 smithi029 ceph-mon[32610]: pgmap v7034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:32.877+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:16:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:32.877+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:16:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:32.877+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:16:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12414.647294 secs 2024-07-27T10:16:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12414.647294 secs 2024-07-27T10:16:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12414.647294 secs 2024-07-27T10:16:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:34 smithi077 ceph-mon[37153]: pgmap v7035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:16:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:34 smithi047 ceph-mon[36835]: pgmap v7035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:16:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:34 smithi029 ceph-mon[32610]: pgmap v7035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:16:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:36 smithi077 ceph-mon[37153]: pgmap v7036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:16:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:36 smithi047 ceph-mon[36835]: pgmap v7036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:16:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:36 smithi029 ceph-mon[32610]: pgmap v7036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:16:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:38 smithi077 ceph-mon[37153]: pgmap v7037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12419.647454 secs 2024-07-27T10:16:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:38 smithi047 ceph-mon[36835]: pgmap v7037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12419.647454 secs 2024-07-27T10:16:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:38 smithi029 ceph-mon[32610]: pgmap v7037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12419.647454 secs 2024-07-27T10:16:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:40 smithi077 ceph-mon[37153]: pgmap v7038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:16:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:40 smithi047 ceph-mon[36835]: pgmap v7038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:16:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:40 smithi029 ceph-mon[32610]: pgmap v7038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:16:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:42 smithi077 ceph-mon[37153]: pgmap v7039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:16:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:42 smithi047 ceph-mon[36835]: pgmap v7039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:16:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:42 smithi029 ceph-mon[32610]: pgmap v7039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:16:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12424.647580 secs 2024-07-27T10:16:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12424.647580 secs 2024-07-27T10:16:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12424.647580 secs 2024-07-27T10:16:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:44 smithi077 ceph-mon[37153]: pgmap v7040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:16:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:44 smithi047 ceph-mon[36835]: pgmap v7040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:16:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:44 smithi029 ceph-mon[32610]: pgmap v7040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:16:46.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:46 smithi077 ceph-mon[37153]: pgmap v7041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:46 smithi047 ceph-mon[36835]: pgmap v7041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:46 smithi029 ceph-mon[32610]: pgmap v7041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:47.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:47.892+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:16:47.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:47.892+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:16:47.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:47.892+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:16:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:48 smithi029 ceph-mon[32610]: pgmap v7042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12429.647716 secs 2024-07-27T10:16:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:48 smithi077 ceph-mon[37153]: pgmap v7042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12429.647716 secs 2024-07-27T10:16:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:48 smithi047 ceph-mon[36835]: pgmap v7042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12429.647716 secs 2024-07-27T10:16:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:50 smithi077 ceph-mon[37153]: pgmap v7043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:16:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:50 smithi047 ceph-mon[36835]: pgmap v7043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:16:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:50 smithi029 ceph-mon[32610]: pgmap v7043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:16:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:52 smithi077 ceph-mon[37153]: pgmap v7044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:52 smithi047 ceph-mon[36835]: pgmap v7044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:52 smithi029 ceph-mon[32610]: pgmap v7044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12434.647872 secs 2024-07-27T10:16:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12434.647872 secs 2024-07-27T10:16:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12434.647872 secs 2024-07-27T10:16:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:54 smithi077 ceph-mon[37153]: pgmap v7045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:16:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:54 smithi047 ceph-mon[36835]: pgmap v7045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:16:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:54 smithi029 ceph-mon[32610]: pgmap v7045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:16:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:56 smithi077 ceph-mon[37153]: pgmap v7046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:56 smithi047 ceph-mon[36835]: pgmap v7046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:56 smithi029 ceph-mon[32610]: pgmap v7046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:16:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:58 smithi077 ceph-mon[37153]: pgmap v7047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:16:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12439.647996 secs 2024-07-27T10:16:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:58 smithi047 ceph-mon[36835]: pgmap v7047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:16:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12439.647996 secs 2024-07-27T10:16:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:58 smithi029 ceph-mon[32610]: pgmap v7047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:16:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:16:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12439.647996 secs 2024-07-27T10:16:59.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:59.522+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:16:59.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:59.522+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:16:59.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:59.522+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:16:59.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:16:59.522+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:17:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:00 smithi077 ceph-mon[37153]: pgmap v7048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:00 smithi047 ceph-mon[36835]: pgmap v7048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:00 smithi029 ceph-mon[32610]: pgmap v7048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:17:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:17:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:17:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:02 smithi077 ceph-mon[37153]: pgmap v7049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:17:02.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:17:02.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:02 smithi077 ceph-mon[37153]: pgmap v7050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:17:02.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:02 smithi077 ceph-mon[37153]: pgmap v7051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:17:02.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:17:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:02 smithi047 ceph-mon[36835]: pgmap v7049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:17:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:17:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:02 smithi047 ceph-mon[36835]: pgmap v7050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:17:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:02 smithi047 ceph-mon[36835]: pgmap v7051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:17:02.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:17:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:02 smithi029 ceph-mon[32610]: pgmap v7049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:17:02.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:17:02.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:02 smithi029 ceph-mon[32610]: pgmap v7050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:17:02.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:02 smithi029 ceph-mon[32610]: pgmap v7051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:17:02.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:17:02.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:17:02.902+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:17:02.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:17:02.902+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:17:02.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:17:02.902+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:17:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12444.648117 secs 2024-07-27T10:17:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12444.648117 secs 2024-07-27T10:17:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12444.648117 secs 2024-07-27T10:17:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:04 smithi077 ceph-mon[37153]: pgmap v7052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:17:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:04 smithi047 ceph-mon[36835]: pgmap v7052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:17:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:04 smithi029 ceph-mon[32610]: pgmap v7052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:17:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:05 smithi077 ceph-mon[37153]: pgmap v7053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:17:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:05 smithi047 ceph-mon[36835]: pgmap v7053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:17:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:05 smithi029 ceph-mon[32610]: pgmap v7053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:17:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:07 smithi077 ceph-mon[37153]: pgmap v7054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:17:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:07 smithi047 ceph-mon[36835]: pgmap v7054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:17:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:07 smithi029 ceph-mon[32610]: pgmap v7054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:17:08.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12449.648251 secs 2024-07-27T10:17:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12449.648251 secs 2024-07-27T10:17:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12449.648251 secs 2024-07-27T10:17:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:09 smithi077 ceph-mon[37153]: pgmap v7055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:17:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:09 smithi047 ceph-mon[36835]: pgmap v7055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:17:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:09 smithi029 ceph-mon[32610]: pgmap v7055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:17:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:11 smithi077 ceph-mon[37153]: pgmap v7056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T10:17:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:11 smithi047 ceph-mon[36835]: pgmap v7056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T10:17:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:11 smithi029 ceph-mon[32610]: pgmap v7056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T10:17:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12454.648379 secs 2024-07-27T10:17:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12454.648379 secs 2024-07-27T10:17:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12454.648379 secs 2024-07-27T10:17:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:13 smithi047 ceph-mon[36835]: pgmap v7057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:17:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:13 smithi029 ceph-mon[32610]: pgmap v7057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:17:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:13 smithi077 ceph-mon[37153]: pgmap v7057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:17:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:15 smithi077 ceph-mon[37153]: pgmap v7058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:17:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:15 smithi047 ceph-mon[36835]: pgmap v7058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:17:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:15 smithi029 ceph-mon[32610]: pgmap v7058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:17:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:17 smithi077 ceph-mon[37153]: pgmap v7059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:17:17.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:17:17.917+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:17:17.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:17:17.917+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:17:17.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:17:17.917+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:17:17.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:17 smithi029 ceph-mon[32610]: pgmap v7059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:17:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:17 smithi047 ceph-mon[36835]: pgmap v7059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:17:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12459.648506 secs 2024-07-27T10:17:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12459.648506 secs 2024-07-27T10:17:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12459.648506 secs 2024-07-27T10:17:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:19 smithi077 ceph-mon[37153]: pgmap v7060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:19 smithi047 ceph-mon[36835]: pgmap v7060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:19 smithi029 ceph-mon[32610]: pgmap v7060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:21 smithi077 ceph-mon[37153]: pgmap v7061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:17:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:21 smithi047 ceph-mon[36835]: pgmap v7061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:17:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:21 smithi029 ceph-mon[32610]: pgmap v7061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:17:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12464.648651 secs 2024-07-27T10:17:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12464.648651 secs 2024-07-27T10:17:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12464.648651 secs 2024-07-27T10:17:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:23 smithi047 ceph-mon[36835]: pgmap v7062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:23 smithi029 ceph-mon[32610]: pgmap v7062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:23 smithi077 ceph-mon[37153]: pgmap v7062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:25 smithi077 ceph-mon[37153]: pgmap v7063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:25 smithi047 ceph-mon[36835]: pgmap v7063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:25 smithi029 ceph-mon[32610]: pgmap v7063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:27 smithi077 ceph-mon[37153]: pgmap v7064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:17:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:27 smithi047 ceph-mon[36835]: pgmap v7064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:17:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:27 smithi029 ceph-mon[32610]: pgmap v7064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:17:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12469.648769 secs 2024-07-27T10:17:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12469.648769 secs 2024-07-27T10:17:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12469.648769 secs 2024-07-27T10:17:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:29 smithi077 ceph-mon[37153]: pgmap v7065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:29 smithi047 ceph-mon[36835]: pgmap v7065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:29 smithi029 ceph-mon[32610]: pgmap v7065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:31 smithi077 ceph-mon[37153]: pgmap v7066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:17:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:31 smithi047 ceph-mon[36835]: pgmap v7066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:17:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:31 smithi029 ceph-mon[32610]: pgmap v7066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:17:32.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:17:32.925+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:17:32.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:17:32.925+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:17:32.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:17:32.925+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:17:32.928 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12474.648887 secs 2024-07-27T10:17:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12474.648887 secs 2024-07-27T10:17:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12474.648887 secs 2024-07-27T10:17:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:33 smithi047 ceph-mon[36835]: pgmap v7067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:33 smithi029 ceph-mon[32610]: pgmap v7067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:33 smithi077 ceph-mon[37153]: pgmap v7067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:35 smithi077 ceph-mon[37153]: pgmap v7068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:35 smithi047 ceph-mon[36835]: pgmap v7068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:35 smithi029 ceph-mon[32610]: pgmap v7068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:37 smithi077 ceph-mon[37153]: pgmap v7069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:17:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:37 smithi047 ceph-mon[36835]: pgmap v7069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:17:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:37 smithi029 ceph-mon[32610]: pgmap v7069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:17:38.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12479.649006 secs 2024-07-27T10:17:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12479.649006 secs 2024-07-27T10:17:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12479.649006 secs 2024-07-27T10:17:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:39 smithi077 ceph-mon[37153]: pgmap v7070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:39 smithi047 ceph-mon[36835]: pgmap v7070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:39 smithi029 ceph-mon[32610]: pgmap v7070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:41 smithi077 ceph-mon[37153]: pgmap v7071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:17:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:41 smithi047 ceph-mon[36835]: pgmap v7071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:17:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:41 smithi029 ceph-mon[32610]: pgmap v7071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:17:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12484.649127 secs 2024-07-27T10:17:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12484.649127 secs 2024-07-27T10:17:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12484.649127 secs 2024-07-27T10:17:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:43 smithi047 ceph-mon[36835]: pgmap v7072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:43 smithi029 ceph-mon[32610]: pgmap v7072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:43 smithi077 ceph-mon[37153]: pgmap v7072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:45 smithi077 ceph-mon[37153]: pgmap v7073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:45 smithi047 ceph-mon[36835]: pgmap v7073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:45 smithi029 ceph-mon[32610]: pgmap v7073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:47 smithi077 ceph-mon[37153]: pgmap v7074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:17:47.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:17:47.941+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:17:47.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:17:47.941+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:17:47.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:17:47.941+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:17:47.944 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:47 smithi029 ceph-mon[32610]: pgmap v7074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:17:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:47 smithi047 ceph-mon[36835]: pgmap v7074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:17:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12489.649226 secs 2024-07-27T10:17:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12489.649226 secs 2024-07-27T10:17:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12489.649226 secs 2024-07-27T10:17:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:49 smithi077 ceph-mon[37153]: pgmap v7075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:49 smithi047 ceph-mon[36835]: pgmap v7075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:49 smithi029 ceph-mon[32610]: pgmap v7075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:51 smithi077 ceph-mon[37153]: pgmap v7076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:17:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:51 smithi047 ceph-mon[36835]: pgmap v7076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:17:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:51 smithi029 ceph-mon[32610]: pgmap v7076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:17:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12494.649319 secs 2024-07-27T10:17:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12494.649319 secs 2024-07-27T10:17:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12494.649319 secs 2024-07-27T10:17:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:53 smithi047 ceph-mon[36835]: pgmap v7077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:53 smithi029 ceph-mon[32610]: pgmap v7077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:53 smithi077 ceph-mon[37153]: pgmap v7077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:17:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:55 smithi077 ceph-mon[37153]: pgmap v7078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:55 smithi047 ceph-mon[36835]: pgmap v7078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:55 smithi029 ceph-mon[32610]: pgmap v7078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:17:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:57 smithi077 ceph-mon[37153]: pgmap v7079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:17:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:57 smithi047 ceph-mon[36835]: pgmap v7079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:17:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:57 smithi029 ceph-mon[32610]: pgmap v7079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:17:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12499.649439 secs 2024-07-27T10:17:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12499.649439 secs 2024-07-27T10:17:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12499.649439 secs 2024-07-27T10:17:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:17:59 smithi077 ceph-mon[37153]: pgmap v7080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:17:59 smithi047 ceph-mon[36835]: pgmap v7080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:17:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:17:59 smithi029 ceph-mon[32610]: pgmap v7080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:18:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:01 smithi077 ceph-mon[37153]: pgmap v7081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:01 smithi047 ceph-mon[36835]: pgmap v7081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:01.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:01 smithi029 ceph-mon[32610]: pgmap v7081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:18:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:18:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:18:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:02 smithi077 ceph-mon[37153]: pgmap v7082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:18:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:02 smithi077 ceph-mon[37153]: pgmap v7083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:18:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:18:02.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:18:02.942+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:18:02.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:18:02.942+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:18:02.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:18:02.942+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:18:02.944 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:18:02.944 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:18:02.944 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:18:02.944 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:02 smithi029 ceph-mon[32610]: pgmap v7082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:18:02.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:02 smithi029 ceph-mon[32610]: pgmap v7083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:18:02.945 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:18:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:18:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:18:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:18:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:02 smithi047 ceph-mon[36835]: pgmap v7082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:18:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:02 smithi047 ceph-mon[36835]: pgmap v7083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:18:02.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:18:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12504.649535 secs 2024-07-27T10:18:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12504.649535 secs 2024-07-27T10:18:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12504.649535 secs 2024-07-27T10:18:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:04 smithi077 ceph-mon[37153]: pgmap v7084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:18:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:04 smithi047 ceph-mon[36835]: pgmap v7084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:18:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:04 smithi029 ceph-mon[32610]: pgmap v7084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:18:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:07 smithi077 ceph-mon[37153]: pgmap v7085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:18:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:07 smithi047 ceph-mon[36835]: pgmap v7085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:18:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:07 smithi029 ceph-mon[32610]: pgmap v7085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:18:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12509.649682 secs 2024-07-27T10:18:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12509.649682 secs 2024-07-27T10:18:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12509.649682 secs 2024-07-27T10:18:09.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:09 smithi077 ceph-mon[37153]: pgmap v7086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:18:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:09 smithi047 ceph-mon[36835]: pgmap v7086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:18:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:09 smithi029 ceph-mon[32610]: pgmap v7086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:18:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:11 smithi077 ceph-mon[37153]: pgmap v7087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:18:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:11 smithi047 ceph-mon[36835]: pgmap v7087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:18:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:11 smithi029 ceph-mon[32610]: pgmap v7087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:18:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12514.649828 secs 2024-07-27T10:18:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:13 smithi077 ceph-mon[37153]: pgmap v7088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:18:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12514.649828 secs 2024-07-27T10:18:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:13 smithi047 ceph-mon[36835]: pgmap v7088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:18:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12514.649828 secs 2024-07-27T10:18:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:13 smithi029 ceph-mon[32610]: pgmap v7088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:18:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:15 smithi077 ceph-mon[37153]: pgmap v7089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:18:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:15 smithi047 ceph-mon[36835]: pgmap v7089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:18:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:15 smithi029 ceph-mon[32610]: pgmap v7089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:18:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:17 smithi077 ceph-mon[37153]: pgmap v7090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:17 smithi047 ceph-mon[36835]: pgmap v7090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:17 smithi029 ceph-mon[32610]: pgmap v7090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:18:17.957+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:18:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:18:17.957+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:18:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:18:17.957+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:18:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12519.649956 secs 2024-07-27T10:18:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12519.649956 secs 2024-07-27T10:18:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12519.649956 secs 2024-07-27T10:18:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:19 smithi077 ceph-mon[37153]: pgmap v7091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:18:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:19 smithi047 ceph-mon[36835]: pgmap v7091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:18:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:19 smithi029 ceph-mon[32610]: pgmap v7091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:18:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:21 smithi077 ceph-mon[37153]: pgmap v7092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:21 smithi047 ceph-mon[36835]: pgmap v7092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:21 smithi029 ceph-mon[32610]: pgmap v7092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12524.650105 secs 2024-07-27T10:18:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:23 smithi077 ceph-mon[37153]: pgmap v7093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12524.650105 secs 2024-07-27T10:18:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:23 smithi047 ceph-mon[36835]: pgmap v7093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12524.650105 secs 2024-07-27T10:18:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:23 smithi029 ceph-mon[32610]: pgmap v7093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:25 smithi077 ceph-mon[37153]: pgmap v7094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:18:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:25 smithi047 ceph-mon[36835]: pgmap v7094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:18:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:25 smithi029 ceph-mon[32610]: pgmap v7094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:18:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:27 smithi077 ceph-mon[37153]: pgmap v7095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:27 smithi047 ceph-mon[36835]: pgmap v7095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:27 smithi029 ceph-mon[32610]: pgmap v7095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12529.650222 secs 2024-07-27T10:18:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12529.650222 secs 2024-07-27T10:18:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12529.650222 secs 2024-07-27T10:18:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:29 smithi077 ceph-mon[37153]: pgmap v7096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:18:29.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:29 smithi029 ceph-mon[32610]: pgmap v7096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:18:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:29 smithi047 ceph-mon[36835]: pgmap v7096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:18:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:31 smithi047 ceph-mon[36835]: pgmap v7097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:31 smithi029 ceph-mon[32610]: pgmap v7097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:31 smithi077 ceph-mon[37153]: pgmap v7097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:18:32.959+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:18:33.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:18:32.959+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:18:33.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:18:32.959+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:18:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12534.650289 secs 2024-07-27T10:18:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:33 smithi047 ceph-mon[36835]: pgmap v7098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12534.650289 secs 2024-07-27T10:18:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:33 smithi029 ceph-mon[32610]: pgmap v7098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12534.650289 secs 2024-07-27T10:18:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:33 smithi077 ceph-mon[37153]: pgmap v7098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:35 smithi047 ceph-mon[36835]: pgmap v7099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:18:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:35 smithi029 ceph-mon[32610]: pgmap v7099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:18:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:35 smithi077 ceph-mon[37153]: pgmap v7099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:18:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:37 smithi047 ceph-mon[36835]: pgmap v7100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:37 smithi029 ceph-mon[32610]: pgmap v7100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:37 smithi077 ceph-mon[37153]: pgmap v7100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12539.650435 secs 2024-07-27T10:18:38.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12539.650435 secs 2024-07-27T10:18:38.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12539.650435 secs 2024-07-27T10:18:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:39 smithi047 ceph-mon[36835]: pgmap v7101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:18:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:39 smithi029 ceph-mon[32610]: pgmap v7101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:18:39.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:39 smithi077 ceph-mon[37153]: pgmap v7101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:18:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:41 smithi047 ceph-mon[36835]: pgmap v7102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:41 smithi029 ceph-mon[32610]: pgmap v7102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:41 smithi077 ceph-mon[37153]: pgmap v7102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12544.650557 secs 2024-07-27T10:18:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:43 smithi047 ceph-mon[36835]: pgmap v7103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12544.650557 secs 2024-07-27T10:18:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:43 smithi029 ceph-mon[32610]: pgmap v7103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12544.650557 secs 2024-07-27T10:18:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:43 smithi077 ceph-mon[37153]: pgmap v7103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:45 smithi029 ceph-mon[32610]: pgmap v7104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:18:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:45 smithi047 ceph-mon[36835]: pgmap v7104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:18:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:45 smithi077 ceph-mon[37153]: pgmap v7104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:18:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:47 smithi047 ceph-mon[36835]: pgmap v7105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:47 smithi029 ceph-mon[32610]: pgmap v7105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:47 smithi077 ceph-mon[37153]: pgmap v7105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:47.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:18:47.975+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:18:47.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:18:47.975+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:18:47.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:18:47.975+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:18:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12549.650681 secs 2024-07-27T10:18:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12549.650681 secs 2024-07-27T10:18:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12549.650681 secs 2024-07-27T10:18:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:49 smithi047 ceph-mon[36835]: pgmap v7106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:18:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:49 smithi029 ceph-mon[32610]: pgmap v7106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:18:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:49 smithi077 ceph-mon[37153]: pgmap v7106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:18:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:51 smithi047 ceph-mon[36835]: pgmap v7107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:51 smithi029 ceph-mon[32610]: pgmap v7107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:51 smithi077 ceph-mon[37153]: pgmap v7107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:18:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12554.650821 secs 2024-07-27T10:18:53.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:53 smithi047 ceph-mon[36835]: pgmap v7108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12554.650821 secs 2024-07-27T10:18:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:53 smithi029 ceph-mon[32610]: pgmap v7108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12554.650821 secs 2024-07-27T10:18:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:53 smithi077 ceph-mon[37153]: pgmap v7108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:18:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:55 smithi047 ceph-mon[36835]: pgmap v7109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:18:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:55 smithi029 ceph-mon[32610]: pgmap v7109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:18:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:55 smithi077 ceph-mon[37153]: pgmap v7109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:18:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:57 smithi047 ceph-mon[36835]: pgmap v7110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:18:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:57 smithi029 ceph-mon[32610]: pgmap v7110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:18:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:57 smithi077 ceph-mon[37153]: pgmap v7110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:18:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12559.650951 secs 2024-07-27T10:18:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12559.650951 secs 2024-07-27T10:18:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12559.650951 secs 2024-07-27T10:18:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:18:59 smithi047 ceph-mon[36835]: pgmap v7111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:18:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:18:59 smithi029 ceph-mon[32610]: pgmap v7111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:18:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:18:59 smithi077 ceph-mon[37153]: pgmap v7111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:19:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:01 smithi047 ceph-mon[36835]: pgmap v7112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:19:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:01 smithi029 ceph-mon[32610]: pgmap v7112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:19:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:01 smithi077 ceph-mon[37153]: pgmap v7112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:19:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:19:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:19:02.755 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:19:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:02.980+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:19:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:02.980+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:19:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:02.980+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:19:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12564.651072 secs 2024-07-27T10:19:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:03 smithi077 ceph-mon[37153]: pgmap v7113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12564.651072 secs 2024-07-27T10:19:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:03 smithi047 ceph-mon[36835]: pgmap v7113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12564.651072 secs 2024-07-27T10:19:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:03 smithi029 ceph-mon[32610]: pgmap v7113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:04 smithi077 ceph-mon[37153]: pgmap v7114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:19:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:19:05.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:19:05.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:04 smithi047 ceph-mon[36835]: pgmap v7114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:19:05.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:19:05.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:19:05.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:04 smithi029 ceph-mon[32610]: pgmap v7114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:19:05.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:05.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:19:05.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:19:05.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:19:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:05 smithi047 ceph-mon[36835]: pgmap v7115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:19:06.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:05 smithi047 ceph-mon[36835]: pgmap v7116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:19:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:05 smithi029 ceph-mon[32610]: pgmap v7115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:19:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:05 smithi029 ceph-mon[32610]: pgmap v7116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:19:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:05 smithi077 ceph-mon[37153]: pgmap v7115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:19:06.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:05 smithi077 ceph-mon[37153]: pgmap v7116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:19:08.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:07 smithi077 ceph-mon[37153]: pgmap v7117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T10:19:08.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12569.651200 secs 2024-07-27T10:19:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:07 smithi047 ceph-mon[36835]: pgmap v7117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T10:19:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12569.651200 secs 2024-07-27T10:19:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:07 smithi029 ceph-mon[32610]: pgmap v7117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T10:19:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12569.651200 secs 2024-07-27T10:19:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:09 smithi047 ceph-mon[36835]: pgmap v7118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:09 smithi029 ceph-mon[32610]: pgmap v7118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:09 smithi077 ceph-mon[37153]: pgmap v7118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:11 smithi047 ceph-mon[36835]: pgmap v7119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:19:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:11 smithi029 ceph-mon[32610]: pgmap v7119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:19:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:11 smithi077 ceph-mon[37153]: pgmap v7119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:19:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12574.651351 secs 2024-07-27T10:19:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12574.651351 secs 2024-07-27T10:19:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12574.651351 secs 2024-07-27T10:19:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:13 smithi047 ceph-mon[36835]: pgmap v7120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:19:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:13 smithi029 ceph-mon[32610]: pgmap v7120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:19:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:13 smithi077 ceph-mon[37153]: pgmap v7120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:19:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:15 smithi047 ceph-mon[36835]: pgmap v7121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:19:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:15 smithi029 ceph-mon[32610]: pgmap v7121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:19:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:15 smithi077 ceph-mon[37153]: pgmap v7121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:19:17.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:17.996+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:19:17.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:17.996+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:19:17.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:17.996+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:19:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:17 smithi047 ceph-mon[36835]: pgmap v7122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12579.651495 secs 2024-07-27T10:19:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:17 smithi029 ceph-mon[32610]: pgmap v7122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12579.651495 secs 2024-07-27T10:19:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:17 smithi077 ceph-mon[37153]: pgmap v7122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12579.651495 secs 2024-07-27T10:19:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:19 smithi047 ceph-mon[36835]: pgmap v7123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:19 smithi029 ceph-mon[32610]: pgmap v7123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:19 smithi077 ceph-mon[37153]: pgmap v7123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:21 smithi047 ceph-mon[36835]: pgmap v7124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:19:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:21 smithi029 ceph-mon[32610]: pgmap v7124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:19:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:21 smithi077 ceph-mon[37153]: pgmap v7124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:19:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12584.651641 secs 2024-07-27T10:19:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12584.651641 secs 2024-07-27T10:19:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12584.651641 secs 2024-07-27T10:19:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:23 smithi047 ceph-mon[36835]: pgmap v7125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:19:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:23 smithi029 ceph-mon[32610]: pgmap v7125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:19:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:23 smithi077 ceph-mon[37153]: pgmap v7125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:19:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:25 smithi047 ceph-mon[36835]: pgmap v7126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:19:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:25 smithi029 ceph-mon[32610]: pgmap v7126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:19:26.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:25 smithi077 ceph-mon[37153]: pgmap v7126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:19:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:27 smithi047 ceph-mon[36835]: pgmap v7127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:28.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12589.651764 secs 2024-07-27T10:19:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:27 smithi029 ceph-mon[32610]: pgmap v7127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12589.651764 secs 2024-07-27T10:19:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:27 smithi077 ceph-mon[37153]: pgmap v7127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12589.651764 secs 2024-07-27T10:19:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:29 smithi047 ceph-mon[36835]: pgmap v7128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:29 smithi029 ceph-mon[32610]: pgmap v7128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:29 smithi077 ceph-mon[37153]: pgmap v7128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:31 smithi047 ceph-mon[36835]: pgmap v7129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:19:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:31 smithi029 ceph-mon[32610]: pgmap v7129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:19:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:31 smithi077 ceph-mon[37153]: pgmap v7129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:19:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:33.010+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:19:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:33.010+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:19:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:33.010+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:19:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12594.651902 secs 2024-07-27T10:19:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12594.651902 secs 2024-07-27T10:19:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12594.651902 secs 2024-07-27T10:19:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:33 smithi047 ceph-mon[36835]: pgmap v7130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:19:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:33 smithi029 ceph-mon[32610]: pgmap v7130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:19:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:33 smithi077 ceph-mon[37153]: pgmap v7130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:19:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:35 smithi047 ceph-mon[36835]: pgmap v7131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:19:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:35 smithi029 ceph-mon[32610]: pgmap v7131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:19:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:35 smithi077 ceph-mon[37153]: pgmap v7131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:19:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:37 smithi047 ceph-mon[36835]: pgmap v7132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12599.652034 secs 2024-07-27T10:19:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:37 smithi029 ceph-mon[32610]: pgmap v7132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12599.652034 secs 2024-07-27T10:19:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:37 smithi077 ceph-mon[37153]: pgmap v7132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12599.652034 secs 2024-07-27T10:19:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:39 smithi047 ceph-mon[36835]: pgmap v7133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:39 smithi029 ceph-mon[32610]: pgmap v7133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:39 smithi077 ceph-mon[37153]: pgmap v7133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:41 smithi047 ceph-mon[36835]: pgmap v7134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:19:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:41 smithi029 ceph-mon[32610]: pgmap v7134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:19:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:41 smithi077 ceph-mon[37153]: pgmap v7134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:19:43.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12604.652169 secs 2024-07-27T10:19:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12604.652169 secs 2024-07-27T10:19:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12604.652169 secs 2024-07-27T10:19:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:43 smithi047 ceph-mon[36835]: pgmap v7135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:19:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:43 smithi029 ceph-mon[32610]: pgmap v7135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:19:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:43 smithi077 ceph-mon[37153]: pgmap v7135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:19:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:45 smithi047 ceph-mon[36835]: pgmap v7136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:19:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:45 smithi029 ceph-mon[32610]: pgmap v7136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:19:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:45 smithi077 ceph-mon[37153]: pgmap v7136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:19:48.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:48.026+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:19:48.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:48.026+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:19:48.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:48.026+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:19:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:47 smithi047 ceph-mon[36835]: pgmap v7137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12609.652298 secs 2024-07-27T10:19:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:47 smithi029 ceph-mon[32610]: pgmap v7137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12609.652298 secs 2024-07-27T10:19:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:47 smithi077 ceph-mon[37153]: pgmap v7137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12609.652298 secs 2024-07-27T10:19:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:49 smithi047 ceph-mon[36835]: pgmap v7138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:49 smithi029 ceph-mon[32610]: pgmap v7138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:49 smithi077 ceph-mon[37153]: pgmap v7138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:19:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:51 smithi029 ceph-mon[32610]: pgmap v7139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:19:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:51 smithi077 ceph-mon[37153]: pgmap v7139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:19:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:51 smithi047 ceph-mon[36835]: pgmap v7139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:19:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12614.652444 secs 2024-07-27T10:19:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12614.652444 secs 2024-07-27T10:19:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12614.652444 secs 2024-07-27T10:19:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:53 smithi029 ceph-mon[32610]: pgmap v7140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:19:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:53 smithi077 ceph-mon[37153]: pgmap v7140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:19:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:53 smithi047 ceph-mon[36835]: pgmap v7140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:19:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:55 smithi029 ceph-mon[32610]: pgmap v7141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:19:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:55 smithi077 ceph-mon[37153]: pgmap v7141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:19:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:55 smithi047 ceph-mon[36835]: pgmap v7141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:19:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:57.665+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T10:19:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:57.665+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T10:19:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:57.665+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7eff400049e0 tx=0x7eff4006c040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T10:19:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:57.665+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7eff400049e0 tx=0x7eff4006c040 comp rx=0 tx=0).stop 2024-07-27T10:19:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:57.665+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T10:19:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:57.665+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T10:19:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:57.665+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T10:19:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:57.665+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 0x7eff3c0b5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T10:19:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:57.666+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 0x7eff3c0b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T10:19:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:57.666+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 0x7eff3c0b5580 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7eff40009ee0 tx=0x7eff4006c040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T10:19:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:57 smithi077 ceph-mon[37153]: pgmap v7142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12619.652584 secs 2024-07-27T10:19:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:57 smithi047 ceph-mon[36835]: pgmap v7142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12619.652584 secs 2024-07-27T10:19:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:57 smithi029 ceph-mon[32610]: pgmap v7142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:19:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12619.652584 secs 2024-07-27T10:19:59.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:59.549+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:19:59.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:59.549+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:19:59.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:59.549+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:19:59.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:19:59.549+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:20:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:19:59 smithi077 ceph-mon[37153]: pgmap v7143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:19:59 smithi047 ceph-mon[36835]: pgmap v7143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:19:59 smithi029 ceph-mon[32610]: pgmap v7143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:20:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:20:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:20:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:01 smithi077 ceph-mon[37153]: pgmap v7144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:20:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:01 smithi047 ceph-mon[36835]: pgmap v7144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:20:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:01 smithi029 ceph-mon[32610]: pgmap v7144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:20:03.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:20:03.033+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:20:03.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:20:03.033+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:20:03.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:20:03.033+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:20:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12624.652734 secs 2024-07-27T10:20:03.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12624.652734 secs 2024-07-27T10:20:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12624.652734 secs 2024-07-27T10:20:04.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:03 smithi077 ceph-mon[37153]: pgmap v7145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:20:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:03 smithi047 ceph-mon[36835]: pgmap v7145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:20:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:03 smithi029 ceph-mon[32610]: pgmap v7145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:20:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:04 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:20:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:04 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:20:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:04 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:20:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:06 smithi029 ceph-mon[32610]: pgmap v7146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:20:06.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:20:06.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:06 smithi077 ceph-mon[37153]: pgmap v7146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:20:06.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:20:06.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:06 smithi047 ceph-mon[36835]: pgmap v7146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:20:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:20:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:20:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:07 smithi077 ceph-mon[37153]: pgmap v7147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:20:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:07 smithi077 ceph-mon[37153]: pgmap v7148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:20:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:07 smithi047 ceph-mon[36835]: pgmap v7147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:20:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:07 smithi047 ceph-mon[36835]: pgmap v7148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:20:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:07 smithi029 ceph-mon[32610]: pgmap v7147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:20:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:07 smithi029 ceph-mon[32610]: pgmap v7148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:20:08.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12629.652838 secs 2024-07-27T10:20:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12629.652838 secs 2024-07-27T10:20:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12629.652838 secs 2024-07-27T10:20:09.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:09 smithi047 ceph-mon[36835]: pgmap v7149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:20:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:09 smithi077 ceph-mon[37153]: pgmap v7149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:20:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:09 smithi029 ceph-mon[32610]: pgmap v7149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:20:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:11 smithi077 ceph-mon[37153]: pgmap v7150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:20:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:11 smithi047 ceph-mon[36835]: pgmap v7150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:20:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:11 smithi029 ceph-mon[32610]: pgmap v7150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:20:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:13 smithi077 ceph-mon[37153]: pgmap v7151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:20:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12634.652938 secs 2024-07-27T10:20:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:13 smithi047 ceph-mon[36835]: pgmap v7151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:20:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12634.652938 secs 2024-07-27T10:20:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:13 smithi029 ceph-mon[32610]: pgmap v7151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:20:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12634.652938 secs 2024-07-27T10:20:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:15 smithi077 ceph-mon[37153]: pgmap v7152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:20:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:15 smithi047 ceph-mon[36835]: pgmap v7152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:20:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:15 smithi029 ceph-mon[32610]: pgmap v7152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:20:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:17 smithi077 ceph-mon[37153]: pgmap v7153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:20:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:17 smithi047 ceph-mon[36835]: pgmap v7153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:20:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:17 smithi029 ceph-mon[32610]: pgmap v7153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:20:18.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:20:18.037+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:20:18.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:20:18.037+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:20:18.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:20:18.037+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:20:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12639.653056 secs 2024-07-27T10:20:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12639.653056 secs 2024-07-27T10:20:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12639.653056 secs 2024-07-27T10:20:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:19 smithi077 ceph-mon[37153]: pgmap v7154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:19 smithi047 ceph-mon[36835]: pgmap v7154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:19 smithi029 ceph-mon[32610]: pgmap v7154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:21 smithi077 ceph-mon[37153]: pgmap v7155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:21 smithi047 ceph-mon[36835]: pgmap v7155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:21 smithi029 ceph-mon[32610]: pgmap v7155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:23 smithi077 ceph-mon[37153]: pgmap v7156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:20:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12644.653180 secs 2024-07-27T10:20:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:23 smithi047 ceph-mon[36835]: pgmap v7156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:20:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12644.653180 secs 2024-07-27T10:20:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:23 smithi029 ceph-mon[32610]: pgmap v7156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:20:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12644.653180 secs 2024-07-27T10:20:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:25 smithi077 ceph-mon[37153]: pgmap v7157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:25 smithi047 ceph-mon[36835]: pgmap v7157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:25 smithi029 ceph-mon[32610]: pgmap v7157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:27 smithi077 ceph-mon[37153]: pgmap v7158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:20:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:27 smithi047 ceph-mon[36835]: pgmap v7158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:20:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:27 smithi029 ceph-mon[32610]: pgmap v7158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:20:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12649.653253 secs 2024-07-27T10:20:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12649.653253 secs 2024-07-27T10:20:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12649.653253 secs 2024-07-27T10:20:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:29 smithi077 ceph-mon[37153]: pgmap v7159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:29 smithi047 ceph-mon[36835]: pgmap v7159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:29 smithi029 ceph-mon[32610]: pgmap v7159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:31 smithi077 ceph-mon[37153]: pgmap v7160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:31 smithi047 ceph-mon[36835]: pgmap v7160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:31 smithi029 ceph-mon[32610]: pgmap v7160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:33.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:20:33.051+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:20:33.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:20:33.051+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:20:33.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:20:33.051+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:20:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:33 smithi029 ceph-mon[32610]: pgmap v7161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:20:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12654.653377 secs 2024-07-27T10:20:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:33 smithi077 ceph-mon[37153]: pgmap v7161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:20:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12654.653377 secs 2024-07-27T10:20:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:33 smithi047 ceph-mon[36835]: pgmap v7161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:20:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12654.653377 secs 2024-07-27T10:20:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:35 smithi077 ceph-mon[37153]: pgmap v7162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:35 smithi047 ceph-mon[36835]: pgmap v7162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:35 smithi029 ceph-mon[32610]: pgmap v7162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:37.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:37 smithi077 ceph-mon[37153]: pgmap v7163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:20:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:37 smithi047 ceph-mon[36835]: pgmap v7163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:20:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:37 smithi029 ceph-mon[32610]: pgmap v7163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:20:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12659.653498 secs 2024-07-27T10:20:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12659.653498 secs 2024-07-27T10:20:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12659.653498 secs 2024-07-27T10:20:39.598 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:39 smithi047 ceph-mon[36835]: pgmap v7164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:39 smithi077 ceph-mon[37153]: pgmap v7164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:39 smithi029 ceph-mon[32610]: pgmap v7164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:41 smithi077 ceph-mon[37153]: pgmap v7165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:41 smithi047 ceph-mon[36835]: pgmap v7165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:41 smithi029 ceph-mon[32610]: pgmap v7165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:43 smithi077 ceph-mon[37153]: pgmap v7166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:20:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12664.653620 secs 2024-07-27T10:20:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:43 smithi047 ceph-mon[36835]: pgmap v7166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:20:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12664.653620 secs 2024-07-27T10:20:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:43 smithi029 ceph-mon[32610]: pgmap v7166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:20:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12664.653620 secs 2024-07-27T10:20:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:45 smithi077 ceph-mon[37153]: pgmap v7167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:45 smithi047 ceph-mon[36835]: pgmap v7167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:45 smithi029 ceph-mon[32610]: pgmap v7167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:47 smithi047 ceph-mon[36835]: pgmap v7168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:47 smithi029 ceph-mon[32610]: pgmap v7168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:47 smithi077 ceph-mon[37153]: pgmap v7168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:48.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:20:48.066+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:20:48.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:20:48.066+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:20:48.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:20:48.066+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:20:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12669.653716 secs 2024-07-27T10:20:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12669.653716 secs 2024-07-27T10:20:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12669.653716 secs 2024-07-27T10:20:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:49 smithi047 ceph-mon[36835]: pgmap v7169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:49 smithi029 ceph-mon[32610]: pgmap v7169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:49 smithi077 ceph-mon[37153]: pgmap v7169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:51 smithi047 ceph-mon[36835]: pgmap v7170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:51 smithi029 ceph-mon[32610]: pgmap v7170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:51 smithi077 ceph-mon[37153]: pgmap v7170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:20:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:53 smithi047 ceph-mon[36835]: pgmap v7171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:20:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12674.653840 secs 2024-07-27T10:20:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:53 smithi029 ceph-mon[32610]: pgmap v7171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:20:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12674.653840 secs 2024-07-27T10:20:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:53 smithi077 ceph-mon[37153]: pgmap v7171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:20:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12674.653840 secs 2024-07-27T10:20:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:55 smithi047 ceph-mon[36835]: pgmap v7172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:55 smithi029 ceph-mon[32610]: pgmap v7172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:55 smithi077 ceph-mon[37153]: pgmap v7172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:20:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:57 smithi047 ceph-mon[36835]: pgmap v7173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:20:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:57 smithi029 ceph-mon[32610]: pgmap v7173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:20:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:57 smithi077 ceph-mon[37153]: pgmap v7173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:20:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12679.653962 secs 2024-07-27T10:20:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12679.653962 secs 2024-07-27T10:20:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12679.653962 secs 2024-07-27T10:20:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:20:59 smithi047 ceph-mon[36835]: pgmap v7174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:20:59 smithi029 ceph-mon[32610]: pgmap v7174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:20:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:20:59 smithi077 ceph-mon[37153]: pgmap v7174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:21:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:01 smithi047 ceph-mon[36835]: pgmap v7175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:21:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:01 smithi029 ceph-mon[32610]: pgmap v7175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:21:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:01 smithi077 ceph-mon[37153]: pgmap v7175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:21:03.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:21:03.082+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:21:03.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:21:03.082+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:21:03.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:21:03.082+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:21:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:03 smithi047 ceph-mon[36835]: pgmap v7176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12684.654083 secs 2024-07-27T10:21:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:03 smithi029 ceph-mon[32610]: pgmap v7176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12684.654083 secs 2024-07-27T10:21:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:03 smithi077 ceph-mon[37153]: pgmap v7176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12684.654083 secs 2024-07-27T10:21:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:05 smithi047 ceph-mon[36835]: pgmap v7177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:05 smithi029 ceph-mon[32610]: pgmap v7177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:05 smithi077 ceph-mon[37153]: pgmap v7177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:21:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:21:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:21:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:21:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:21:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:21:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:21:06.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:21:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:21:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:21:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:21:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:21:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:07 smithi047 ceph-mon[36835]: pgmap v7178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:21:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:07 smithi047 ceph-mon[36835]: pgmap v7179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T10:21:07.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:07 smithi047 ceph-mon[36835]: pgmap v7180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:21:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:07 smithi029 ceph-mon[32610]: pgmap v7178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:21:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:07 smithi029 ceph-mon[32610]: pgmap v7179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T10:21:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:07 smithi029 ceph-mon[32610]: pgmap v7180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:21:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:07 smithi077 ceph-mon[37153]: pgmap v7178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:21:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:07 smithi077 ceph-mon[37153]: pgmap v7179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T10:21:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:07 smithi077 ceph-mon[37153]: pgmap v7180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:21:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12689.654175 secs 2024-07-27T10:21:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12689.654175 secs 2024-07-27T10:21:08.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12689.654175 secs 2024-07-27T10:21:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:09 smithi047 ceph-mon[36835]: pgmap v7181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:21:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:09 smithi029 ceph-mon[32610]: pgmap v7181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:21:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:09 smithi077 ceph-mon[37153]: pgmap v7181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:21:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:11 smithi047 ceph-mon[36835]: pgmap v7182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:21:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:11 smithi029 ceph-mon[32610]: pgmap v7182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:21:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:11 smithi077 ceph-mon[37153]: pgmap v7182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:21:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12694.654333 secs 2024-07-27T10:21:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:13 smithi047 ceph-mon[36835]: pgmap v7183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:21:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12694.654333 secs 2024-07-27T10:21:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:13 smithi029 ceph-mon[32610]: pgmap v7183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:21:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12694.654333 secs 2024-07-27T10:21:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:13 smithi077 ceph-mon[37153]: pgmap v7183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:21:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:15 smithi047 ceph-mon[36835]: pgmap v7184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:21:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:15 smithi029 ceph-mon[32610]: pgmap v7184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:21:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:15 smithi077 ceph-mon[37153]: pgmap v7184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:21:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:17 smithi047 ceph-mon[36835]: pgmap v7185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:21:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:17 smithi029 ceph-mon[32610]: pgmap v7185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:21:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:17 smithi077 ceph-mon[37153]: pgmap v7185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:21:18.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:21:18.097+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:21:18.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:21:18.097+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:21:18.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:21:18.097+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:21:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12699.654451 secs 2024-07-27T10:21:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12699.654451 secs 2024-07-27T10:21:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12699.654451 secs 2024-07-27T10:21:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:19 smithi047 ceph-mon[36835]: pgmap v7186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:19 smithi029 ceph-mon[32610]: pgmap v7186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:19 smithi077 ceph-mon[37153]: pgmap v7186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:21 smithi047 ceph-mon[36835]: pgmap v7187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:21:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:21 smithi029 ceph-mon[32610]: pgmap v7187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:21:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:21 smithi077 ceph-mon[37153]: pgmap v7187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:21:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12704.654571 secs 2024-07-27T10:21:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:23 smithi047 ceph-mon[36835]: pgmap v7188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12704.654571 secs 2024-07-27T10:21:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:23 smithi029 ceph-mon[32610]: pgmap v7188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12704.654571 secs 2024-07-27T10:21:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:23 smithi077 ceph-mon[37153]: pgmap v7188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:25 smithi047 ceph-mon[36835]: pgmap v7189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:21:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:25 smithi029 ceph-mon[32610]: pgmap v7189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:21:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:25 smithi077 ceph-mon[37153]: pgmap v7189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:21:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:27 smithi029 ceph-mon[32610]: pgmap v7190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:27 smithi077 ceph-mon[37153]: pgmap v7190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:27 smithi047 ceph-mon[36835]: pgmap v7190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12709.654708 secs 2024-07-27T10:21:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12709.654708 secs 2024-07-27T10:21:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12709.654708 secs 2024-07-27T10:21:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:29 smithi029 ceph-mon[32610]: pgmap v7191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:29 smithi077 ceph-mon[37153]: pgmap v7191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:29 smithi047 ceph-mon[36835]: pgmap v7191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:31 smithi029 ceph-mon[32610]: pgmap v7192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:21:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:31 smithi077 ceph-mon[37153]: pgmap v7192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:21:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:31 smithi047 ceph-mon[36835]: pgmap v7192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:21:33.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:21:33.112+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:21:33.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:21:33.112+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:21:33.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:21:33.112+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:21:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12714.654833 secs 2024-07-27T10:21:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:33 smithi029 ceph-mon[32610]: pgmap v7193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12714.654833 secs 2024-07-27T10:21:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:33 smithi077 ceph-mon[37153]: pgmap v7193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12714.654833 secs 2024-07-27T10:21:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:33 smithi047 ceph-mon[36835]: pgmap v7193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:35 smithi077 ceph-mon[37153]: pgmap v7194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:21:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:35 smithi047 ceph-mon[36835]: pgmap v7194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:21:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:35 smithi029 ceph-mon[32610]: pgmap v7194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:21:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:37 smithi077 ceph-mon[37153]: pgmap v7195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:37 smithi047 ceph-mon[36835]: pgmap v7195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:37 smithi029 ceph-mon[32610]: pgmap v7195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:38.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12719.654951 secs 2024-07-27T10:21:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12719.654951 secs 2024-07-27T10:21:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12719.654951 secs 2024-07-27T10:21:39.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:39 smithi047 ceph-mon[36835]: pgmap v7196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:39 smithi077 ceph-mon[37153]: pgmap v7196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:39 smithi029 ceph-mon[32610]: pgmap v7196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:41 smithi077 ceph-mon[37153]: pgmap v7197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:21:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:41 smithi047 ceph-mon[36835]: pgmap v7197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:21:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:41 smithi029 ceph-mon[32610]: pgmap v7197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:21:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12724.655071 secs 2024-07-27T10:21:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:43 smithi077 ceph-mon[37153]: pgmap v7198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12724.655071 secs 2024-07-27T10:21:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:43 smithi047 ceph-mon[36835]: pgmap v7198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12724.655071 secs 2024-07-27T10:21:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:43 smithi029 ceph-mon[32610]: pgmap v7198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:45 smithi077 ceph-mon[37153]: pgmap v7199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:21:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:45 smithi047 ceph-mon[36835]: pgmap v7199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:21:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:45 smithi029 ceph-mon[32610]: pgmap v7199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:21:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:47 smithi077 ceph-mon[37153]: pgmap v7200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:47 smithi047 ceph-mon[36835]: pgmap v7200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:47 smithi029 ceph-mon[32610]: pgmap v7200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:48.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:21:48.127+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:21:48.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:21:48.127+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:21:48.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:21:48.127+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:21:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12729.655188 secs 2024-07-27T10:21:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12729.655188 secs 2024-07-27T10:21:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12729.655188 secs 2024-07-27T10:21:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:49 smithi077 ceph-mon[37153]: pgmap v7201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:49 smithi047 ceph-mon[36835]: pgmap v7201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:49 smithi029 ceph-mon[32610]: pgmap v7201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:51 smithi077 ceph-mon[37153]: pgmap v7202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:21:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:51 smithi047 ceph-mon[36835]: pgmap v7202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:21:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:51 smithi029 ceph-mon[32610]: pgmap v7202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:21:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12734.655263 secs 2024-07-27T10:21:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:53 smithi077 ceph-mon[37153]: pgmap v7203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12734.655263 secs 2024-07-27T10:21:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:53 smithi047 ceph-mon[36835]: pgmap v7203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12734.655263 secs 2024-07-27T10:21:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:53 smithi029 ceph-mon[32610]: pgmap v7203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:55 smithi077 ceph-mon[37153]: pgmap v7204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:21:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:55 smithi047 ceph-mon[36835]: pgmap v7204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:21:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:55 smithi029 ceph-mon[32610]: pgmap v7204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:21:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:57 smithi077 ceph-mon[37153]: pgmap v7205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:57 smithi047 ceph-mon[36835]: pgmap v7205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:57 smithi029 ceph-mon[32610]: pgmap v7205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:21:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12739.655400 secs 2024-07-27T10:21:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12739.655400 secs 2024-07-27T10:21:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12739.655400 secs 2024-07-27T10:21:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:21:59 smithi077 ceph-mon[37153]: pgmap v7206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:21:59 smithi047 ceph-mon[36835]: pgmap v7206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:21:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:21:59 smithi029 ceph-mon[32610]: pgmap v7206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:01 smithi077 ceph-mon[37153]: pgmap v7207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:01 smithi047 ceph-mon[36835]: pgmap v7207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:01 smithi029 ceph-mon[32610]: pgmap v7207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:03.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:03.135+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:22:03.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:03.135+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:22:03.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:03.135+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:22:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12744.655525 secs 2024-07-27T10:22:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:03 smithi077 ceph-mon[37153]: pgmap v7208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12744.655525 secs 2024-07-27T10:22:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:03 smithi047 ceph-mon[36835]: pgmap v7208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12744.655525 secs 2024-07-27T10:22:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:03 smithi029 ceph-mon[32610]: pgmap v7208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:05 smithi077 ceph-mon[37153]: pgmap v7209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:22:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:05 smithi047 ceph-mon[36835]: pgmap v7209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:22:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:05 smithi029 ceph-mon[32610]: pgmap v7209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:22:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:22:06.887 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:22:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:22:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:07 smithi077 ceph-mon[37153]: pgmap v7210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:22:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:22:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:22:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:07 smithi047 ceph-mon[36835]: pgmap v7210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:22:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:22:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:22:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:07 smithi029 ceph-mon[32610]: pgmap v7210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:22:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:22:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:22:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:08 smithi077 ceph-mon[37153]: pgmap v7211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:22:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:08 smithi077 ceph-mon[37153]: pgmap v7212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:22:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12749.655649 secs 2024-07-27T10:22:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:08 smithi047 ceph-mon[36835]: pgmap v7211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:22:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:08 smithi047 ceph-mon[36835]: pgmap v7212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:22:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12749.655649 secs 2024-07-27T10:22:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:08 smithi029 ceph-mon[32610]: pgmap v7211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:22:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:08 smithi029 ceph-mon[32610]: pgmap v7212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:22:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12749.655649 secs 2024-07-27T10:22:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:10 smithi077 ceph-mon[37153]: pgmap v7213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:22:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:10 smithi047 ceph-mon[36835]: pgmap v7213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:22:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:10 smithi029 ceph-mon[32610]: pgmap v7213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:22:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:12 smithi077 ceph-mon[37153]: pgmap v7214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:22:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:12 smithi047 ceph-mon[36835]: pgmap v7214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:22:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:12 smithi029 ceph-mon[32610]: pgmap v7214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:22:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12754.655809 secs 2024-07-27T10:22:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12754.655809 secs 2024-07-27T10:22:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12754.655809 secs 2024-07-27T10:22:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:14 smithi077 ceph-mon[37153]: pgmap v7215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:22:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:14 smithi047 ceph-mon[36835]: pgmap v7215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:22:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:14 smithi029 ceph-mon[32610]: pgmap v7215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:22:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:16 smithi077 ceph-mon[37153]: pgmap v7216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:22:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:16 smithi047 ceph-mon[36835]: pgmap v7216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:22:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:16 smithi029 ceph-mon[32610]: pgmap v7216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:22:18.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:18.137+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:22:18.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:18.137+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:22:18.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:18.138+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:22:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:18 smithi077 ceph-mon[37153]: pgmap v7217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:22:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12759.655905 secs 2024-07-27T10:22:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:18 smithi047 ceph-mon[36835]: pgmap v7217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:22:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12759.655905 secs 2024-07-27T10:22:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:18 smithi029 ceph-mon[32610]: pgmap v7217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:22:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12759.655905 secs 2024-07-27T10:22:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:20 smithi077 ceph-mon[37153]: pgmap v7218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:20 smithi047 ceph-mon[36835]: pgmap v7218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:20 smithi029 ceph-mon[32610]: pgmap v7218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:22 smithi077 ceph-mon[37153]: pgmap v7219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:22 smithi047 ceph-mon[36835]: pgmap v7219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:22 smithi029 ceph-mon[32610]: pgmap v7219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12764.656000 secs 2024-07-27T10:22:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12764.656000 secs 2024-07-27T10:22:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12764.656000 secs 2024-07-27T10:22:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:24 smithi077 ceph-mon[37153]: pgmap v7220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:22:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:24 smithi047 ceph-mon[36835]: pgmap v7220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:22:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:24 smithi029 ceph-mon[32610]: pgmap v7220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:22:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:25 smithi077 ceph-mon[37153]: pgmap v7221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:22:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:25 smithi047 ceph-mon[36835]: pgmap v7221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:22:25.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:25 smithi029 ceph-mon[32610]: pgmap v7221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:22:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:27 smithi077 ceph-mon[37153]: pgmap v7222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12769.656122 secs 2024-07-27T10:22:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:27 smithi047 ceph-mon[36835]: pgmap v7222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12769.656122 secs 2024-07-27T10:22:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:27 smithi029 ceph-mon[32610]: pgmap v7222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12769.656122 secs 2024-07-27T10:22:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:29 smithi077 ceph-mon[37153]: pgmap v7223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:29 smithi047 ceph-mon[36835]: pgmap v7223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:29 smithi029 ceph-mon[32610]: pgmap v7223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:32.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:31 smithi077 ceph-mon[37153]: pgmap v7224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:31 smithi047 ceph-mon[36835]: pgmap v7224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:31 smithi029 ceph-mon[32610]: pgmap v7224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12774.656271 secs 2024-07-27T10:22:33.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12774.656271 secs 2024-07-27T10:22:33.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:33.139+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:22:33.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:33.139+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:22:33.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:33.139+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:22:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12774.656271 secs 2024-07-27T10:22:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:33 smithi077 ceph-mon[37153]: pgmap v7225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:22:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:33 smithi047 ceph-mon[36835]: pgmap v7225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:22:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:33 smithi029 ceph-mon[32610]: pgmap v7225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:22:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:35 smithi077 ceph-mon[37153]: pgmap v7226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:22:36.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:35 smithi047 ceph-mon[36835]: pgmap v7226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:22:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:35 smithi029 ceph-mon[32610]: pgmap v7226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:22:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:37 smithi077 ceph-mon[37153]: pgmap v7227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12779.656414 secs 2024-07-27T10:22:38.119 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:37 smithi029 ceph-mon[32610]: pgmap v7227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:38.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12779.656414 secs 2024-07-27T10:22:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:37 smithi047 ceph-mon[36835]: pgmap v7227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12779.656414 secs 2024-07-27T10:22:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:39 smithi077 ceph-mon[37153]: pgmap v7228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:39 smithi047 ceph-mon[36835]: pgmap v7228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:39 smithi029 ceph-mon[32610]: pgmap v7228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:41 smithi077 ceph-mon[37153]: pgmap v7229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:41 smithi047 ceph-mon[36835]: pgmap v7229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:41 smithi029 ceph-mon[32610]: pgmap v7229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12784.656580 secs 2024-07-27T10:22:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12784.656580 secs 2024-07-27T10:22:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12784.656580 secs 2024-07-27T10:22:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:43 smithi077 ceph-mon[37153]: pgmap v7230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:22:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:43 smithi047 ceph-mon[36835]: pgmap v7230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:22:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:43 smithi029 ceph-mon[32610]: pgmap v7230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:22:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:45 smithi077 ceph-mon[37153]: pgmap v7231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:22:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:45 smithi047 ceph-mon[36835]: pgmap v7231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:22:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:45 smithi029 ceph-mon[32610]: pgmap v7231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:22:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:47 smithi077 ceph-mon[37153]: pgmap v7232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12789.656710 secs 2024-07-27T10:22:48.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:48.146+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:22:48.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:48.146+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:22:48.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:48.146+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:22:48.148 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:47 smithi029 ceph-mon[32610]: pgmap v7232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:48.148 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12789.656710 secs 2024-07-27T10:22:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:47 smithi047 ceph-mon[36835]: pgmap v7232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12789.656710 secs 2024-07-27T10:22:50.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:49 smithi077 ceph-mon[37153]: pgmap v7233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:49 smithi047 ceph-mon[36835]: pgmap v7233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:49 smithi029 ceph-mon[32610]: pgmap v7233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:22:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:51 smithi077 ceph-mon[37153]: pgmap v7234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:51 smithi047 ceph-mon[36835]: pgmap v7234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:51 smithi029 ceph-mon[32610]: pgmap v7234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:22:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12794.656860 secs 2024-07-27T10:22:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12794.656860 secs 2024-07-27T10:22:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12794.656860 secs 2024-07-27T10:22:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:53 smithi047 ceph-mon[36835]: pgmap v7235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:22:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:53 smithi029 ceph-mon[32610]: pgmap v7235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:22:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:53 smithi077 ceph-mon[37153]: pgmap v7235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:22:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:55 smithi047 ceph-mon[36835]: pgmap v7236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:22:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:55 smithi029 ceph-mon[32610]: pgmap v7236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:22:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:55 smithi077 ceph-mon[37153]: pgmap v7236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:22:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:57 smithi047 ceph-mon[36835]: pgmap v7237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12799.656991 secs 2024-07-27T10:22:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:57 smithi029 ceph-mon[32610]: pgmap v7237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12799.656991 secs 2024-07-27T10:22:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:57 smithi077 ceph-mon[37153]: pgmap v7237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:22:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12799.656991 secs 2024-07-27T10:22:59.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:59.577+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:22:59.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:59.577+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:22:59.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:59.577+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:22:59.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:22:59.577+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:23:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:22:59 smithi047 ceph-mon[36835]: pgmap v7238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:22:59 smithi029 ceph-mon[32610]: pgmap v7238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:22:59 smithi077 ceph-mon[37153]: pgmap v7238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:01 smithi047 ceph-mon[36835]: pgmap v7239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:01 smithi029 ceph-mon[32610]: pgmap v7239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:01 smithi077 ceph-mon[37153]: pgmap v7239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:03.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:23:03.159+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:23:03.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:23:03.159+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:23:03.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:23:03.159+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:23:03.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12804.657124 secs 2024-07-27T10:23:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12804.657124 secs 2024-07-27T10:23:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12804.657124 secs 2024-07-27T10:23:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:03 smithi047 ceph-mon[36835]: pgmap v7240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:23:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:03 smithi029 ceph-mon[32610]: pgmap v7240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:23:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:03 smithi077 ceph-mon[37153]: pgmap v7240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:23:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:05 smithi047 ceph-mon[36835]: pgmap v7241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:23:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:05 smithi029 ceph-mon[32610]: pgmap v7241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:23:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:05 smithi077 ceph-mon[37153]: pgmap v7241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:23:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:23:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:23:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:23:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:07 smithi047 ceph-mon[36835]: pgmap v7242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12809.657202 secs 2024-07-27T10:23:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:23:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:23:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:07 smithi047 ceph-mon[36835]: pgmap v7243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:23:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:07 smithi047 ceph-mon[36835]: pgmap v7244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T10:23:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:23:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:07 smithi029 ceph-mon[32610]: pgmap v7242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12809.657202 secs 2024-07-27T10:23:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:23:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:23:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:07 smithi029 ceph-mon[32610]: pgmap v7243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:23:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:07 smithi029 ceph-mon[32610]: pgmap v7244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T10:23:08.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:23:08.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:07 smithi077 ceph-mon[37153]: pgmap v7242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:08.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12809.657202 secs 2024-07-27T10:23:08.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:23:08.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:23:08.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:07 smithi077 ceph-mon[37153]: pgmap v7243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:23:08.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:07 smithi077 ceph-mon[37153]: pgmap v7244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T10:23:08.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:23:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:10 smithi077 ceph-mon[37153]: pgmap v7245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:23:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:10 smithi047 ceph-mon[36835]: pgmap v7245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:23:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:10 smithi029 ceph-mon[32610]: pgmap v7245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:23:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:12 smithi047 ceph-mon[36835]: pgmap v7246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:23:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:12 smithi029 ceph-mon[32610]: pgmap v7246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:23:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:12 smithi077 ceph-mon[37153]: pgmap v7246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:23:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12814.657316 secs 2024-07-27T10:23:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12814.657316 secs 2024-07-27T10:23:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12814.657316 secs 2024-07-27T10:23:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:14 smithi047 ceph-mon[36835]: pgmap v7247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:23:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:14 smithi029 ceph-mon[32610]: pgmap v7247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:23:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:14 smithi077 ceph-mon[37153]: pgmap v7247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:23:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:16 smithi047 ceph-mon[36835]: pgmap v7248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:23:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:16 smithi029 ceph-mon[32610]: pgmap v7248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:23:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:16 smithi077 ceph-mon[37153]: pgmap v7248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:23:18.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:23:18.172+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:23:18.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:23:18.172+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:23:18.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:23:18.172+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:23:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12819.657436 secs 2024-07-27T10:23:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:18 smithi029 ceph-mon[32610]: pgmap v7249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:23:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12819.657436 secs 2024-07-27T10:23:18.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:18 smithi047 ceph-mon[36835]: pgmap v7249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:23:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12819.657436 secs 2024-07-27T10:23:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:18 smithi077 ceph-mon[37153]: pgmap v7249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:23:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:20 smithi047 ceph-mon[36835]: pgmap v7250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:23:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:20 smithi029 ceph-mon[32610]: pgmap v7250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:23:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:20 smithi077 ceph-mon[37153]: pgmap v7250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:23:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:22 smithi047 ceph-mon[36835]: pgmap v7251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:22 smithi029 ceph-mon[32610]: pgmap v7251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:22 smithi077 ceph-mon[37153]: pgmap v7251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12824.657578 secs 2024-07-27T10:23:23.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12824.657578 secs 2024-07-27T10:23:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12824.657578 secs 2024-07-27T10:23:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:24 smithi029 ceph-mon[32610]: pgmap v7252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:24 smithi047 ceph-mon[36835]: pgmap v7252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:24 smithi077 ceph-mon[37153]: pgmap v7252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:26 smithi047 ceph-mon[36835]: pgmap v7253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:26 smithi029 ceph-mon[32610]: pgmap v7253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:26 smithi077 ceph-mon[37153]: pgmap v7253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12829.657710 secs 2024-07-27T10:23:28.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:28 smithi047 ceph-mon[36835]: pgmap v7254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:23:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12829.657710 secs 2024-07-27T10:23:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:28 smithi029 ceph-mon[32610]: pgmap v7254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:23:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12829.657710 secs 2024-07-27T10:23:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:28 smithi077 ceph-mon[37153]: pgmap v7254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:23:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:30 smithi047 ceph-mon[36835]: pgmap v7255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:23:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:30 smithi029 ceph-mon[32610]: pgmap v7255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:23:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:30 smithi077 ceph-mon[37153]: pgmap v7255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:23:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:32 smithi047 ceph-mon[36835]: pgmap v7256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:32 smithi029 ceph-mon[32610]: pgmap v7256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:32 smithi077 ceph-mon[37153]: pgmap v7256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:33.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:23:33.182+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:23:33.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:23:33.182+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:23:33.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:23:33.182+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:23:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12834.657831 secs 2024-07-27T10:23:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12834.657831 secs 2024-07-27T10:23:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12834.657831 secs 2024-07-27T10:23:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:34 smithi047 ceph-mon[36835]: pgmap v7257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:34 smithi029 ceph-mon[32610]: pgmap v7257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:34 smithi077 ceph-mon[37153]: pgmap v7257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:36 smithi047 ceph-mon[36835]: pgmap v7258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:36 smithi029 ceph-mon[32610]: pgmap v7258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:36 smithi077 ceph-mon[37153]: pgmap v7258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12839.657962 secs 2024-07-27T10:23:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:38 smithi047 ceph-mon[36835]: pgmap v7259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:23:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12839.657962 secs 2024-07-27T10:23:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:38 smithi029 ceph-mon[32610]: pgmap v7259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:23:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12839.657962 secs 2024-07-27T10:23:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:38 smithi077 ceph-mon[37153]: pgmap v7259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:23:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:40 smithi047 ceph-mon[36835]: pgmap v7260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:23:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:40 smithi029 ceph-mon[32610]: pgmap v7260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:23:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:40 smithi077 ceph-mon[37153]: pgmap v7260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:23:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:42 smithi047 ceph-mon[36835]: pgmap v7261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:42 smithi029 ceph-mon[32610]: pgmap v7261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:42 smithi077 ceph-mon[37153]: pgmap v7261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12844.658084 secs 2024-07-27T10:23:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12844.658084 secs 2024-07-27T10:23:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12844.658084 secs 2024-07-27T10:23:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:44 smithi047 ceph-mon[36835]: pgmap v7262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:44 smithi029 ceph-mon[32610]: pgmap v7262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:44 smithi077 ceph-mon[37153]: pgmap v7262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:46 smithi047 ceph-mon[36835]: pgmap v7263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:46 smithi029 ceph-mon[32610]: pgmap v7263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:46 smithi077 ceph-mon[37153]: pgmap v7263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:23:48.192+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:23:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:23:48.192+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:23:48.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:23:48.192+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:23:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12849.658242 secs 2024-07-27T10:23:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:48 smithi029 ceph-mon[32610]: pgmap v7264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:23:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12849.658242 secs 2024-07-27T10:23:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:48 smithi047 ceph-mon[36835]: pgmap v7264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:23:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12849.658242 secs 2024-07-27T10:23:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:48 smithi077 ceph-mon[37153]: pgmap v7264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:23:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:50 smithi047 ceph-mon[36835]: pgmap v7265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:23:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:50 smithi029 ceph-mon[32610]: pgmap v7265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:23:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:50 smithi077 ceph-mon[37153]: pgmap v7265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:23:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:52 smithi047 ceph-mon[36835]: pgmap v7266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:52 smithi029 ceph-mon[32610]: pgmap v7266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:52 smithi077 ceph-mon[37153]: pgmap v7266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:23:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12854.658375 secs 2024-07-27T10:23:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12854.658375 secs 2024-07-27T10:23:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12854.658375 secs 2024-07-27T10:23:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:54 smithi029 ceph-mon[32610]: pgmap v7267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:54 smithi077 ceph-mon[37153]: pgmap v7267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:54 smithi047 ceph-mon[36835]: pgmap v7267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:23:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:56 smithi029 ceph-mon[32610]: pgmap v7268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:56 smithi077 ceph-mon[37153]: pgmap v7268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:56 smithi047 ceph-mon[36835]: pgmap v7268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:23:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12859.658503 secs 2024-07-27T10:23:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:23:58 smithi029 ceph-mon[32610]: pgmap v7269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:23:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12859.658503 secs 2024-07-27T10:23:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:23:58 smithi077 ceph-mon[37153]: pgmap v7269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:23:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12859.658503 secs 2024-07-27T10:23:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:23:58 smithi047 ceph-mon[36835]: pgmap v7269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:24:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:00 smithi029 ceph-mon[32610]: pgmap v7270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:24:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:00 smithi077 ceph-mon[37153]: pgmap v7270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:24:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:00 smithi047 ceph-mon[36835]: pgmap v7270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:24:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:02 smithi029 ceph-mon[32610]: pgmap v7271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:24:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:02 smithi077 ceph-mon[37153]: pgmap v7271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:24:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:02 smithi047 ceph-mon[36835]: pgmap v7271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:24:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:03.205+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:24:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:03.205+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:24:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:03.205+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:24:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12864.658635 secs 2024-07-27T10:24:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12864.658635 secs 2024-07-27T10:24:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12864.658635 secs 2024-07-27T10:24:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:04 smithi077 ceph-mon[37153]: pgmap v7272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:04 smithi047 ceph-mon[36835]: pgmap v7272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:04 smithi029 ceph-mon[32610]: pgmap v7272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:06.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:06 smithi077 ceph-mon[37153]: pgmap v7273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:06 smithi047 ceph-mon[36835]: pgmap v7273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:06 smithi029 ceph-mon[32610]: pgmap v7273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:24:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:24:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:24:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12869.658768 secs 2024-07-27T10:24:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:08 smithi077 ceph-mon[37153]: pgmap v7274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:24:08.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:24:08.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:24:08.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:24:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12869.658768 secs 2024-07-27T10:24:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:08 smithi047 ceph-mon[36835]: pgmap v7274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:24:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:24:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:24:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:24:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12869.658768 secs 2024-07-27T10:24:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:08 smithi029 ceph-mon[32610]: pgmap v7274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:24:08.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:24:08.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:24:08.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:24:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:09 smithi077 ceph-mon[37153]: pgmap v7275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:24:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:09 smithi077 ceph-mon[37153]: pgmap v7276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:24:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:09 smithi047 ceph-mon[36835]: pgmap v7275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:24:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:09 smithi047 ceph-mon[36835]: pgmap v7276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:24:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:09 smithi029 ceph-mon[32610]: pgmap v7275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:24:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:09 smithi029 ceph-mon[32610]: pgmap v7276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:24:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:11 smithi029 ceph-mon[32610]: pgmap v7277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:24:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:11 smithi077 ceph-mon[37153]: pgmap v7277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:24:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:11 smithi047 ceph-mon[36835]: pgmap v7277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:24:13.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:13 smithi047 ceph-mon[36835]: pgmap v7278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:24:13.749 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12874.658868 secs 2024-07-27T10:24:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:13 smithi077 ceph-mon[37153]: pgmap v7278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:24:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12874.658868 secs 2024-07-27T10:24:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:13 smithi029 ceph-mon[32610]: pgmap v7278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:24:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12874.658868 secs 2024-07-27T10:24:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:15 smithi077 ceph-mon[37153]: pgmap v7279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:24:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:15 smithi047 ceph-mon[36835]: pgmap v7279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:24:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:15 smithi029 ceph-mon[32610]: pgmap v7279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:24:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:17 smithi077 ceph-mon[37153]: pgmap v7280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:24:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:17 smithi047 ceph-mon[36835]: pgmap v7280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:24:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:17 smithi029 ceph-mon[32610]: pgmap v7280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:24:18.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:18.217+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:24:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:18.217+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:24:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:18.217+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:24:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12879.658987 secs 2024-07-27T10:24:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12879.658987 secs 2024-07-27T10:24:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12879.658987 secs 2024-07-27T10:24:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:19 smithi077 ceph-mon[37153]: pgmap v7281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:24:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:19 smithi047 ceph-mon[36835]: pgmap v7281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:24:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:19 smithi029 ceph-mon[32610]: pgmap v7281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:24:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:21 smithi077 ceph-mon[37153]: pgmap v7282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:21 smithi047 ceph-mon[36835]: pgmap v7282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:21 smithi029 ceph-mon[32610]: pgmap v7282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:23 smithi077 ceph-mon[37153]: pgmap v7283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:24:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12884.659088 secs 2024-07-27T10:24:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:23 smithi047 ceph-mon[36835]: pgmap v7283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:24:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12884.659088 secs 2024-07-27T10:24:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:23 smithi029 ceph-mon[32610]: pgmap v7283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:24:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12884.659088 secs 2024-07-27T10:24:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:25 smithi077 ceph-mon[37153]: pgmap v7284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:25 smithi047 ceph-mon[36835]: pgmap v7284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:25 smithi029 ceph-mon[32610]: pgmap v7284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:27 smithi077 ceph-mon[37153]: pgmap v7285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:27 smithi047 ceph-mon[36835]: pgmap v7285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:27 smithi029 ceph-mon[32610]: pgmap v7285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12889.659217 secs 2024-07-27T10:24:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12889.659217 secs 2024-07-27T10:24:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12889.659217 secs 2024-07-27T10:24:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:29.591+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:24:29.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:29.591+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:24:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:29.592+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:24:29.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:29.592+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:24:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:29 smithi077 ceph-mon[37153]: pgmap v7286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:24:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:29 smithi047 ceph-mon[36835]: pgmap v7286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:24:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:29 smithi029 ceph-mon[32610]: pgmap v7286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:24:31.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:31 smithi077 ceph-mon[37153]: pgmap v7287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:31 smithi047 ceph-mon[36835]: pgmap v7287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:31 smithi029 ceph-mon[32610]: pgmap v7287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:33.232+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:24:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:33.232+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:24:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:33.232+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:24:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:33 smithi077 ceph-mon[37153]: pgmap v7288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:24:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12894.659335 secs 2024-07-27T10:24:33.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:33 smithi047 ceph-mon[36835]: pgmap v7288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:24:33.950 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12894.659335 secs 2024-07-27T10:24:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:33 smithi029 ceph-mon[32610]: pgmap v7288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:24:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12894.659335 secs 2024-07-27T10:24:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:35 smithi077 ceph-mon[37153]: pgmap v7289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:35 smithi047 ceph-mon[36835]: pgmap v7289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:35 smithi029 ceph-mon[32610]: pgmap v7289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:37 smithi077 ceph-mon[37153]: pgmap v7290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:37.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:37 smithi047 ceph-mon[36835]: pgmap v7290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:37 smithi029 ceph-mon[32610]: pgmap v7290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12899.659462 secs 2024-07-27T10:24:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12899.659462 secs 2024-07-27T10:24:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12899.659462 secs 2024-07-27T10:24:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:39 smithi077 ceph-mon[37153]: pgmap v7291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:24:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:39 smithi047 ceph-mon[36835]: pgmap v7291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:24:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:39 smithi029 ceph-mon[32610]: pgmap v7291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:24:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:41 smithi077 ceph-mon[37153]: pgmap v7292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:41 smithi047 ceph-mon[36835]: pgmap v7292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:41 smithi029 ceph-mon[32610]: pgmap v7292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:43 smithi077 ceph-mon[37153]: pgmap v7293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:24:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12904.659583 secs 2024-07-27T10:24:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:43 smithi047 ceph-mon[36835]: pgmap v7293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:24:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12904.659583 secs 2024-07-27T10:24:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:43 smithi029 ceph-mon[32610]: pgmap v7293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:24:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12904.659583 secs 2024-07-27T10:24:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:45 smithi077 ceph-mon[37153]: pgmap v7294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:45 smithi047 ceph-mon[36835]: pgmap v7294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:45 smithi029 ceph-mon[32610]: pgmap v7294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:47 smithi077 ceph-mon[37153]: pgmap v7295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:47 smithi047 ceph-mon[36835]: pgmap v7295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:47 smithi029 ceph-mon[32610]: pgmap v7295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:48.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:48.248+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:24:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:48.248+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:24:48.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:24:48.248+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:24:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12909.659721 secs 2024-07-27T10:24:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12909.659721 secs 2024-07-27T10:24:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12909.659721 secs 2024-07-27T10:24:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:49 smithi077 ceph-mon[37153]: pgmap v7296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:24:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:49 smithi047 ceph-mon[36835]: pgmap v7296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:24:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:49 smithi029 ceph-mon[32610]: pgmap v7296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:24:51.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:51 smithi077 ceph-mon[37153]: pgmap v7297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:51 smithi047 ceph-mon[36835]: pgmap v7297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:51 smithi029 ceph-mon[32610]: pgmap v7297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:53 smithi077 ceph-mon[37153]: pgmap v7298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:24:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12914.659853 secs 2024-07-27T10:24:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:53 smithi047 ceph-mon[36835]: pgmap v7298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:24:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12914.659853 secs 2024-07-27T10:24:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:53 smithi029 ceph-mon[32610]: pgmap v7298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:24:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12914.659853 secs 2024-07-27T10:24:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:55 smithi077 ceph-mon[37153]: pgmap v7299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:55 smithi047 ceph-mon[36835]: pgmap v7299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:55.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:55 smithi029 ceph-mon[32610]: pgmap v7299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:24:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:57 smithi077 ceph-mon[37153]: pgmap v7300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:57 smithi047 ceph-mon[36835]: pgmap v7300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:57 smithi029 ceph-mon[32610]: pgmap v7300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:24:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12919.659974 secs 2024-07-27T10:24:58.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12919.659974 secs 2024-07-27T10:24:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12919.659974 secs 2024-07-27T10:24:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:24:59 smithi077 ceph-mon[37153]: pgmap v7301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:24:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:24:59 smithi047 ceph-mon[36835]: pgmap v7301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:24:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:24:59 smithi029 ceph-mon[32610]: pgmap v7301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:25:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:01 smithi077 ceph-mon[37153]: pgmap v7302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:01 smithi047 ceph-mon[36835]: pgmap v7302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:01 smithi029 ceph-mon[32610]: pgmap v7302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:03.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:03.263+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:25:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:03.263+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:25:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:03.263+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:25:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:03 smithi077 ceph-mon[37153]: pgmap v7303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:25:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12924.660077 secs 2024-07-27T10:25:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:03 smithi047 ceph-mon[36835]: pgmap v7303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:25:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12924.660077 secs 2024-07-27T10:25:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:03 smithi029 ceph-mon[32610]: pgmap v7303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:25:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12924.660077 secs 2024-07-27T10:25:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:05 smithi077 ceph-mon[37153]: pgmap v7304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:25:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:05 smithi047 ceph-mon[36835]: pgmap v7304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:25:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:05 smithi029 ceph-mon[32610]: pgmap v7304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:25:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:07 smithi077 ceph-mon[37153]: pgmap v7305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:25:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:07 smithi047 ceph-mon[36835]: pgmap v7305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:25:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:07 smithi029 ceph-mon[32610]: pgmap v7305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:25:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12929.660189 secs 2024-07-27T10:25:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:25:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:25:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:25:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:25:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:25:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:25:08.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:25:08.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:25:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12929.660189 secs 2024-07-27T10:25:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:25:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:25:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:25:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:25:08.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:25:08.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:25:08.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:25:08.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:25:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12929.660189 secs 2024-07-27T10:25:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:25:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:25:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:25:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:25:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:25:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:25:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:25:09.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:25:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:09 smithi047 ceph-mon[36835]: pgmap v7306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:25:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:09 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:25:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:09 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:25:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:09 smithi047 ceph-mon[36835]: pgmap v7307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:25:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:09 smithi047 ceph-mon[36835]: pgmap v7308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:09 smithi029 ceph-mon[32610]: pgmap v7306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:25:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:09 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:25:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:09 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:25:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:09 smithi029 ceph-mon[32610]: pgmap v7307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:25:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:09 smithi029 ceph-mon[32610]: pgmap v7308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:09 smithi077 ceph-mon[37153]: pgmap v7306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:25:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:09 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:25:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:09 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:25:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:09 smithi077 ceph-mon[37153]: pgmap v7307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:25:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:09 smithi077 ceph-mon[37153]: pgmap v7308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:10 smithi047 ceph-mon[36835]: pgmap v7309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:25:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:10 smithi029 ceph-mon[32610]: pgmap v7309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:25:11.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:10 smithi077 ceph-mon[37153]: pgmap v7309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:25:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12934.660286 secs 2024-07-27T10:25:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:13 smithi077 ceph-mon[37153]: pgmap v7310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T10:25:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12934.660286 secs 2024-07-27T10:25:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:13 smithi047 ceph-mon[36835]: pgmap v7310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T10:25:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12934.660286 secs 2024-07-27T10:25:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:13 smithi029 ceph-mon[32610]: pgmap v7310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T10:25:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:15 smithi077 ceph-mon[37153]: pgmap v7311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:25:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:15 smithi047 ceph-mon[36835]: pgmap v7311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:25:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:15 smithi029 ceph-mon[32610]: pgmap v7311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:25:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:17 smithi077 ceph-mon[37153]: pgmap v7312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:25:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:17 smithi047 ceph-mon[36835]: pgmap v7312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:25:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:17 smithi029 ceph-mon[32610]: pgmap v7312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:25:18.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:18.279+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:25:18.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:18.279+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:25:18.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:18.279+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:25:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12939.660424 secs 2024-07-27T10:25:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12939.660424 secs 2024-07-27T10:25:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12939.660424 secs 2024-07-27T10:25:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:19 smithi077 ceph-mon[37153]: pgmap v7313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:25:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:19 smithi047 ceph-mon[36835]: pgmap v7313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:25:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:19 smithi029 ceph-mon[32610]: pgmap v7313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:25:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:21 smithi077 ceph-mon[37153]: pgmap v7314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:21 smithi047 ceph-mon[36835]: pgmap v7314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:21 smithi029 ceph-mon[32610]: pgmap v7314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12944.660550 secs 2024-07-27T10:25:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:23 smithi077 ceph-mon[37153]: pgmap v7315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:25:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12944.660550 secs 2024-07-27T10:25:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:23 smithi047 ceph-mon[36835]: pgmap v7315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:25:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12944.660550 secs 2024-07-27T10:25:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:23 smithi029 ceph-mon[32610]: pgmap v7315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:25:25.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:25 smithi077 ceph-mon[37153]: pgmap v7316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:25:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:25 smithi047 ceph-mon[36835]: pgmap v7316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:25:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:25 smithi029 ceph-mon[32610]: pgmap v7316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:25:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:27 smithi077 ceph-mon[37153]: pgmap v7317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:25:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:27 smithi047 ceph-mon[36835]: pgmap v7317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:25:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:27 smithi029 ceph-mon[32610]: pgmap v7317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:25:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12949.660668 secs 2024-07-27T10:25:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12949.660668 secs 2024-07-27T10:25:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12949.660668 secs 2024-07-27T10:25:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:29 smithi077 ceph-mon[37153]: pgmap v7318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:29 smithi047 ceph-mon[36835]: pgmap v7318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:29 smithi029 ceph-mon[32610]: pgmap v7318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:31 smithi077 ceph-mon[37153]: pgmap v7319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:31 smithi047 ceph-mon[36835]: pgmap v7319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:31 smithi029 ceph-mon[32610]: pgmap v7319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:33.293+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:25:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:33.293+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:25:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:33.293+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:25:33.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12954.660795 secs 2024-07-27T10:25:33.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:33 smithi029 ceph-mon[32610]: pgmap v7320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:25:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12954.660795 secs 2024-07-27T10:25:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:33 smithi077 ceph-mon[37153]: pgmap v7320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:25:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12954.660795 secs 2024-07-27T10:25:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:33 smithi047 ceph-mon[36835]: pgmap v7320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:25:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:35 smithi077 ceph-mon[37153]: pgmap v7321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:25:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:35 smithi047 ceph-mon[36835]: pgmap v7321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:25:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:35 smithi029 ceph-mon[32610]: pgmap v7321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:25:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:37 smithi077 ceph-mon[37153]: pgmap v7322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:25:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:37 smithi047 ceph-mon[36835]: pgmap v7322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:25:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:37 smithi029 ceph-mon[32610]: pgmap v7322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:25:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12959.660895 secs 2024-07-27T10:25:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12959.660895 secs 2024-07-27T10:25:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12959.660895 secs 2024-07-27T10:25:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:39 smithi077 ceph-mon[37153]: pgmap v7323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:39 smithi047 ceph-mon[36835]: pgmap v7323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:39 smithi029 ceph-mon[32610]: pgmap v7323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:41 smithi077 ceph-mon[37153]: pgmap v7324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:41 smithi047 ceph-mon[36835]: pgmap v7324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:41 smithi029 ceph-mon[32610]: pgmap v7324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12964.661012 secs 2024-07-27T10:25:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:43 smithi077 ceph-mon[37153]: pgmap v7325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:25:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12964.661012 secs 2024-07-27T10:25:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:43 smithi047 ceph-mon[36835]: pgmap v7325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:25:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12964.661012 secs 2024-07-27T10:25:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:43 smithi029 ceph-mon[32610]: pgmap v7325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:25:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:45 smithi077 ceph-mon[37153]: pgmap v7326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:25:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:45 smithi047 ceph-mon[36835]: pgmap v7326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:25:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:45 smithi029 ceph-mon[32610]: pgmap v7326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:25:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:47 smithi077 ceph-mon[37153]: pgmap v7327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:25:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:47 smithi047 ceph-mon[36835]: pgmap v7327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:25:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:47 smithi029 ceph-mon[32610]: pgmap v7327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:25:48.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:48.300+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:25:48.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:48.300+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:25:48.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:48.300+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:25:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12969.661133 secs 2024-07-27T10:25:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12969.661133 secs 2024-07-27T10:25:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12969.661133 secs 2024-07-27T10:25:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:49 smithi077 ceph-mon[37153]: pgmap v7328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:49 smithi047 ceph-mon[36835]: pgmap v7328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:49 smithi029 ceph-mon[32610]: pgmap v7328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:51 smithi077 ceph-mon[37153]: pgmap v7329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:51 smithi047 ceph-mon[36835]: pgmap v7329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:51 smithi029 ceph-mon[32610]: pgmap v7329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:25:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12974.661252 secs 2024-07-27T10:25:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:53 smithi077 ceph-mon[37153]: pgmap v7330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:25:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12974.661252 secs 2024-07-27T10:25:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:53 smithi047 ceph-mon[36835]: pgmap v7330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:25:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12974.661252 secs 2024-07-27T10:25:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:53 smithi029 ceph-mon[32610]: pgmap v7330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:25:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:55 smithi077 ceph-mon[37153]: pgmap v7331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:25:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:55 smithi047 ceph-mon[36835]: pgmap v7331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:25:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:55 smithi029 ceph-mon[32610]: pgmap v7331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:25:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:57 smithi077 ceph-mon[37153]: pgmap v7332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:25:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:57 smithi047 ceph-mon[36835]: pgmap v7332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:25:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:57 smithi029 ceph-mon[32610]: pgmap v7332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:25:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12979.661373 secs 2024-07-27T10:25:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12979.661373 secs 2024-07-27T10:25:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12979.661373 secs 2024-07-27T10:25:59.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:59.604+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:25:59.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:59.604+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:25:59.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:59.604+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:25:59.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:25:59.604+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:25:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:25:59 smithi077 ceph-mon[37153]: pgmap v7333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:25:59 smithi047 ceph-mon[36835]: pgmap v7333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:25:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:25:59 smithi029 ceph-mon[32610]: pgmap v7333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:26:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:01 smithi077 ceph-mon[37153]: pgmap v7334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:01 smithi047 ceph-mon[36835]: pgmap v7334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:01 smithi029 ceph-mon[32610]: pgmap v7334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:03.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:26:03.315+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:26:03.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:26:03.315+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:26:03.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:26:03.315+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:26:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12984.661498 secs 2024-07-27T10:26:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:03 smithi077 ceph-mon[37153]: pgmap v7335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:26:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12984.661498 secs 2024-07-27T10:26:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:03 smithi047 ceph-mon[36835]: pgmap v7335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:26:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12984.661498 secs 2024-07-27T10:26:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:03 smithi029 ceph-mon[32610]: pgmap v7335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:26:05.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:05 smithi077 ceph-mon[37153]: pgmap v7336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:26:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:05 smithi047 ceph-mon[36835]: pgmap v7336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:26:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:05 smithi029 ceph-mon[32610]: pgmap v7336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:26:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:07 smithi077 ceph-mon[37153]: pgmap v7337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:07 smithi047 ceph-mon[36835]: pgmap v7337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:07 smithi029 ceph-mon[32610]: pgmap v7337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12989.661617 secs 2024-07-27T10:26:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:26:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12989.661617 secs 2024-07-27T10:26:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:26:08.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12989.661617 secs 2024-07-27T10:26:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:26:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:09 smithi077 ceph-mon[37153]: pgmap v7338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:26:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:26:09.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:26:09.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:26:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:09 smithi047 ceph-mon[36835]: pgmap v7338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:26:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:26:09.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:26:09.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:26:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:09 smithi029 ceph-mon[32610]: pgmap v7338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:26:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:26:09.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:26:09.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:26:09.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:26:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:10 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:26:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:10 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:26:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:10 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:26:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:10 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:26:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:10 smithi077 ceph-mon[37153]: pgmap v7339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:26:10.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:10 smithi077 ceph-mon[37153]: pgmap v7340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:26:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:10 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:26:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:10 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:26:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:10 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:26:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:10 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:26:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:10 smithi047 ceph-mon[36835]: pgmap v7339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:26:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:10 smithi047 ceph-mon[36835]: pgmap v7340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:26:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:10 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:26:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:10 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:26:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:10 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:26:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:10 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:26:10.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:10 smithi029 ceph-mon[32610]: pgmap v7339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:26:10.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:10 smithi029 ceph-mon[32610]: pgmap v7340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:26:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:12 smithi047 ceph-mon[36835]: pgmap v7341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:26:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:12 smithi029 ceph-mon[32610]: pgmap v7341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:26:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:12 smithi077 ceph-mon[37153]: pgmap v7341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:26:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12994.661743 secs 2024-07-27T10:26:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:13 smithi047 ceph-mon[36835]: pgmap v7342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:26:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12994.661743 secs 2024-07-27T10:26:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:13 smithi029 ceph-mon[32610]: pgmap v7342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:26:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12994.661743 secs 2024-07-27T10:26:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:13 smithi077 ceph-mon[37153]: pgmap v7342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:26:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:16 smithi077 ceph-mon[37153]: pgmap v7343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:26:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:16 smithi047 ceph-mon[36835]: pgmap v7343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:26:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:16 smithi029 ceph-mon[32610]: pgmap v7343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:26:18.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:26:18.331+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:26:18.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:26:18.331+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:26:18.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:26:18.331+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:26:18.333 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 12999.661869 secs 2024-07-27T10:26:18.333 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:18 smithi029 ceph-mon[32610]: pgmap v7344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:26:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 12999.661869 secs 2024-07-27T10:26:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:18 smithi077 ceph-mon[37153]: pgmap v7344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:26:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 12999.661869 secs 2024-07-27T10:26:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:18 smithi047 ceph-mon[36835]: pgmap v7344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:26:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:20 smithi077 ceph-mon[37153]: pgmap v7345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T10:26:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:20 smithi047 ceph-mon[36835]: pgmap v7345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T10:26:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:20 smithi029 ceph-mon[32610]: pgmap v7345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T10:26:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:22 smithi077 ceph-mon[37153]: pgmap v7346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:26:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:22 smithi047 ceph-mon[36835]: pgmap v7346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:26:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:22 smithi029 ceph-mon[32610]: pgmap v7346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:26:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13004.661991 secs 2024-07-27T10:26:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13004.661991 secs 2024-07-27T10:26:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13004.661991 secs 2024-07-27T10:26:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:24 smithi077 ceph-mon[37153]: pgmap v7347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:26:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:24 smithi047 ceph-mon[36835]: pgmap v7347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:26:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:24 smithi029 ceph-mon[32610]: pgmap v7347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:26:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:26 smithi077 ceph-mon[37153]: pgmap v7348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:26:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:26 smithi047 ceph-mon[36835]: pgmap v7348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:26:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:26 smithi029 ceph-mon[32610]: pgmap v7348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:26:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13009.662108 secs 2024-07-27T10:26:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:28 smithi077 ceph-mon[37153]: pgmap v7349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13009.662108 secs 2024-07-27T10:26:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:28 smithi047 ceph-mon[36835]: pgmap v7349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13009.662108 secs 2024-07-27T10:26:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:28 smithi029 ceph-mon[32610]: pgmap v7349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:30 smithi077 ceph-mon[37153]: pgmap v7350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:26:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:30 smithi047 ceph-mon[36835]: pgmap v7350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:26:30.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:30 smithi029 ceph-mon[32610]: pgmap v7350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:26:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:32 smithi077 ceph-mon[37153]: pgmap v7351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:32 smithi047 ceph-mon[36835]: pgmap v7351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:32 smithi029 ceph-mon[32610]: pgmap v7351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:26:33.333+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:26:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:26:33.333+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:26:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:26:33.333+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:26:33.335 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13014.662228 secs 2024-07-27T10:26:33.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13014.662228 secs 2024-07-27T10:26:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13014.662228 secs 2024-07-27T10:26:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:34 smithi077 ceph-mon[37153]: pgmap v7352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:26:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:34 smithi047 ceph-mon[36835]: pgmap v7352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:26:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:34 smithi029 ceph-mon[32610]: pgmap v7352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:26:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:36 smithi077 ceph-mon[37153]: pgmap v7353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:36 smithi047 ceph-mon[36835]: pgmap v7353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:36 smithi029 ceph-mon[32610]: pgmap v7353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13019.662350 secs 2024-07-27T10:26:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:38 smithi077 ceph-mon[37153]: pgmap v7354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13019.662350 secs 2024-07-27T10:26:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:38 smithi047 ceph-mon[36835]: pgmap v7354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13019.662350 secs 2024-07-27T10:26:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:38 smithi029 ceph-mon[32610]: pgmap v7354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:40 smithi077 ceph-mon[37153]: pgmap v7355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:26:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:40 smithi047 ceph-mon[36835]: pgmap v7355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:26:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:40 smithi029 ceph-mon[32610]: pgmap v7355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:26:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:42 smithi077 ceph-mon[37153]: pgmap v7356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:42 smithi047 ceph-mon[36835]: pgmap v7356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:42 smithi029 ceph-mon[32610]: pgmap v7356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:43.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13024.662473 secs 2024-07-27T10:26:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13024.662473 secs 2024-07-27T10:26:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13024.662473 secs 2024-07-27T10:26:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:44 smithi077 ceph-mon[37153]: pgmap v7357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:26:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:44 smithi047 ceph-mon[36835]: pgmap v7357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:26:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:44 smithi029 ceph-mon[32610]: pgmap v7357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:26:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:46 smithi077 ceph-mon[37153]: pgmap v7358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:26:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:46 smithi047 ceph-mon[36835]: pgmap v7358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:26:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:46 smithi029 ceph-mon[32610]: pgmap v7358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:26:48.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13029.662591 secs 2024-07-27T10:26:48.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:48 smithi029 ceph-mon[32610]: pgmap v7359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:48.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:26:48.349+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:26:48.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:26:48.349+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:26:48.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:26:48.349+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:26:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13029.662591 secs 2024-07-27T10:26:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:48 smithi077 ceph-mon[37153]: pgmap v7359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13029.662591 secs 2024-07-27T10:26:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:48 smithi047 ceph-mon[36835]: pgmap v7359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:50 smithi047 ceph-mon[36835]: pgmap v7360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:26:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:50 smithi029 ceph-mon[32610]: pgmap v7360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:26:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:50 smithi077 ceph-mon[37153]: pgmap v7360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:26:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:52 smithi047 ceph-mon[36835]: pgmap v7361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:26:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:52 smithi029 ceph-mon[32610]: pgmap v7361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:26:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:52 smithi077 ceph-mon[37153]: pgmap v7361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:26:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13034.662695 secs 2024-07-27T10:26:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13034.662695 secs 2024-07-27T10:26:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13034.662695 secs 2024-07-27T10:26:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:54 smithi077 ceph-mon[37153]: pgmap v7362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:26:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:54 smithi047 ceph-mon[36835]: pgmap v7362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:26:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:54 smithi029 ceph-mon[32610]: pgmap v7362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:26:56.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:56 smithi077 ceph-mon[37153]: pgmap v7363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:56 smithi047 ceph-mon[36835]: pgmap v7363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:56 smithi029 ceph-mon[32610]: pgmap v7363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:26:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13039.662762 secs 2024-07-27T10:26:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:26:58 smithi077 ceph-mon[37153]: pgmap v7364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13039.662762 secs 2024-07-27T10:26:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:26:58 smithi047 ceph-mon[36835]: pgmap v7364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:26:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13039.662762 secs 2024-07-27T10:26:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:26:58 smithi029 ceph-mon[32610]: pgmap v7364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:00 smithi077 ceph-mon[37153]: pgmap v7365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:27:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:00 smithi047 ceph-mon[36835]: pgmap v7365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:27:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:00 smithi029 ceph-mon[32610]: pgmap v7365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:27:02.823 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:02 smithi029 ceph-mon[32610]: pgmap v7366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:02 smithi077 ceph-mon[37153]: pgmap v7366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:02 smithi047 ceph-mon[36835]: pgmap v7366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:03.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:27:03.364+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:27:03.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:27:03.364+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:27:03.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:27:03.364+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:27:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13044.662889 secs 2024-07-27T10:27:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13044.662889 secs 2024-07-27T10:27:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13044.662889 secs 2024-07-27T10:27:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:04 smithi077 ceph-mon[37153]: pgmap v7367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:27:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:04 smithi047 ceph-mon[36835]: pgmap v7367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:27:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:04 smithi029 ceph-mon[32610]: pgmap v7367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:27:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:06 smithi077 ceph-mon[37153]: pgmap v7368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:06 smithi047 ceph-mon[36835]: pgmap v7368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:06 smithi029 ceph-mon[32610]: pgmap v7368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13049.662966 secs 2024-07-27T10:27:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:08 smithi077 ceph-mon[37153]: pgmap v7369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13049.662966 secs 2024-07-27T10:27:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:08 smithi047 ceph-mon[36835]: pgmap v7369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13049.662966 secs 2024-07-27T10:27:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:08 smithi029 ceph-mon[32610]: pgmap v7369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:27:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:27:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:27:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:10 smithi077 ceph-mon[37153]: pgmap v7370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:27:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:27:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:27:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:10 smithi077 ceph-mon[37153]: pgmap v7371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:27:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:10 smithi077 ceph-mon[37153]: pgmap v7372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:27:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:27:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:10 smithi047 ceph-mon[36835]: pgmap v7370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:27:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:27:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:27:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:10 smithi047 ceph-mon[36835]: pgmap v7371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:27:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:10 smithi047 ceph-mon[36835]: pgmap v7372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:27:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:27:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:10 smithi029 ceph-mon[32610]: pgmap v7370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:27:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:27:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:27:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:10 smithi029 ceph-mon[32610]: pgmap v7371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:27:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:10 smithi029 ceph-mon[32610]: pgmap v7372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:27:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:27:12.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:12 smithi077 ceph-mon[37153]: pgmap v7373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:27:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:12 smithi047 ceph-mon[36835]: pgmap v7373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:27:12.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:12 smithi029 ceph-mon[32610]: pgmap v7373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:27:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13054.663041 secs 2024-07-27T10:27:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13054.663041 secs 2024-07-27T10:27:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13054.663041 secs 2024-07-27T10:27:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:14 smithi077 ceph-mon[37153]: pgmap v7374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:27:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:14 smithi047 ceph-mon[36835]: pgmap v7374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:27:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:14 smithi029 ceph-mon[32610]: pgmap v7374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:27:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:16 smithi047 ceph-mon[36835]: pgmap v7375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:27:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:16 smithi029 ceph-mon[32610]: pgmap v7375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:27:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:16 smithi077 ceph-mon[37153]: pgmap v7375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:27:18.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:27:18.377+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:27:18.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:27:18.377+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:27:18.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:27:18.377+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:27:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13059.663141 secs 2024-07-27T10:27:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:18 smithi047 ceph-mon[36835]: pgmap v7376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:27:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13059.663141 secs 2024-07-27T10:27:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:18 smithi029 ceph-mon[32610]: pgmap v7376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:27:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13059.663141 secs 2024-07-27T10:27:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:18 smithi077 ceph-mon[37153]: pgmap v7376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:27:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:20 smithi047 ceph-mon[36835]: pgmap v7377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:27:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:20 smithi029 ceph-mon[32610]: pgmap v7377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:27:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:20 smithi077 ceph-mon[37153]: pgmap v7377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:27:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:22 smithi047 ceph-mon[36835]: pgmap v7378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:22 smithi029 ceph-mon[32610]: pgmap v7378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:22 smithi077 ceph-mon[37153]: pgmap v7378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13064.663270 secs 2024-07-27T10:27:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13064.663270 secs 2024-07-27T10:27:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13064.663270 secs 2024-07-27T10:27:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:24 smithi047 ceph-mon[36835]: pgmap v7379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:27:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:24 smithi029 ceph-mon[32610]: pgmap v7379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:27:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:24 smithi077 ceph-mon[37153]: pgmap v7379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:27:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:25 smithi047 ceph-mon[36835]: pgmap v7380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:27:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:25 smithi029 ceph-mon[32610]: pgmap v7380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:27:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:25 smithi077 ceph-mon[37153]: pgmap v7380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:27:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13069.663411 secs 2024-07-27T10:27:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:28 smithi077 ceph-mon[37153]: pgmap v7381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:27:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13069.663411 secs 2024-07-27T10:27:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:28 smithi047 ceph-mon[36835]: pgmap v7381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:27:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13069.663411 secs 2024-07-27T10:27:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:28 smithi029 ceph-mon[32610]: pgmap v7381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:27:30.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:30 smithi077 ceph-mon[37153]: pgmap v7382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:27:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:30 smithi047 ceph-mon[36835]: pgmap v7382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:27:30.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:30 smithi029 ceph-mon[32610]: pgmap v7382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:27:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:32 smithi047 ceph-mon[36835]: pgmap v7383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:32 smithi029 ceph-mon[32610]: pgmap v7383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:32 smithi077 ceph-mon[37153]: pgmap v7383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:33.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:27:33.389+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:27:33.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:27:33.389+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:27:33.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:27:33.389+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:27:33.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13074.663537 secs 2024-07-27T10:27:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13074.663537 secs 2024-07-27T10:27:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13074.663537 secs 2024-07-27T10:27:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:34 smithi047 ceph-mon[36835]: pgmap v7384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:27:34.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:34 smithi029 ceph-mon[32610]: pgmap v7384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:27:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:34 smithi077 ceph-mon[37153]: pgmap v7384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:27:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:36 smithi047 ceph-mon[36835]: pgmap v7385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:36 smithi029 ceph-mon[32610]: pgmap v7385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:36 smithi077 ceph-mon[37153]: pgmap v7385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13079.663660 secs 2024-07-27T10:27:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:38 smithi047 ceph-mon[36835]: pgmap v7386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:27:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13079.663660 secs 2024-07-27T10:27:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:38 smithi029 ceph-mon[32610]: pgmap v7386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:27:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13079.663660 secs 2024-07-27T10:27:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:38 smithi077 ceph-mon[37153]: pgmap v7386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:27:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:39 smithi047 ceph-mon[36835]: pgmap v7387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:27:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:39 smithi029 ceph-mon[32610]: pgmap v7387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:27:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:39 smithi077 ceph-mon[37153]: pgmap v7387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:27:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:42 smithi047 ceph-mon[36835]: pgmap v7388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:42 smithi029 ceph-mon[32610]: pgmap v7388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:42 smithi077 ceph-mon[37153]: pgmap v7388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13084.663758 secs 2024-07-27T10:27:43.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13084.663758 secs 2024-07-27T10:27:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13084.663758 secs 2024-07-27T10:27:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:44 smithi047 ceph-mon[36835]: pgmap v7389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:27:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:44 smithi029 ceph-mon[32610]: pgmap v7389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:27:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:44 smithi077 ceph-mon[37153]: pgmap v7389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:27:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:46 smithi047 ceph-mon[36835]: pgmap v7390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:46 smithi029 ceph-mon[32610]: pgmap v7390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:46 smithi077 ceph-mon[37153]: pgmap v7390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:48.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:27:48.391+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:27:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:27:48.391+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:27:48.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:27:48.391+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:27:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13089.663877 secs 2024-07-27T10:27:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:48 smithi029 ceph-mon[32610]: pgmap v7391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:27:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13089.663877 secs 2024-07-27T10:27:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:48 smithi047 ceph-mon[36835]: pgmap v7391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:27:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13089.663877 secs 2024-07-27T10:27:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:48 smithi077 ceph-mon[37153]: pgmap v7391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:27:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:50 smithi047 ceph-mon[36835]: pgmap v7392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:27:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:50 smithi029 ceph-mon[32610]: pgmap v7392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:27:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:50 smithi077 ceph-mon[37153]: pgmap v7392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:27:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:51 smithi047 ceph-mon[36835]: pgmap v7393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:51.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:51 smithi029 ceph-mon[32610]: pgmap v7393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:51 smithi077 ceph-mon[37153]: pgmap v7393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:27:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13094.664002 secs 2024-07-27T10:27:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13094.664002 secs 2024-07-27T10:27:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13094.664002 secs 2024-07-27T10:27:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:53 smithi077 ceph-mon[37153]: pgmap v7394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:27:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:53 smithi047 ceph-mon[36835]: pgmap v7394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:27:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:53 smithi029 ceph-mon[32610]: pgmap v7394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:27:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:56 smithi047 ceph-mon[36835]: pgmap v7395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:56 smithi029 ceph-mon[32610]: pgmap v7395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:56 smithi077 ceph-mon[37153]: pgmap v7395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:27:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13099.664119 secs 2024-07-27T10:27:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:27:58 smithi047 ceph-mon[36835]: pgmap v7396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:27:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13099.664119 secs 2024-07-27T10:27:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:27:58 smithi029 ceph-mon[32610]: pgmap v7396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:27:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13099.664119 secs 2024-07-27T10:27:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:27:58 smithi077 ceph-mon[37153]: pgmap v7396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:28:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:00 smithi047 ceph-mon[36835]: pgmap v7397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:28:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:00 smithi029 ceph-mon[32610]: pgmap v7397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:28:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:00 smithi077 ceph-mon[37153]: pgmap v7397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:28:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:02 smithi047 ceph-mon[36835]: pgmap v7398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:02 smithi029 ceph-mon[32610]: pgmap v7398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:02 smithi077 ceph-mon[37153]: pgmap v7398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:03.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:03.394+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:28:03.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:03.394+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:28:03.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:03.394+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:28:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13104.664240 secs 2024-07-27T10:28:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13104.664240 secs 2024-07-27T10:28:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13104.664240 secs 2024-07-27T10:28:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:04 smithi047 ceph-mon[36835]: pgmap v7399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:04.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:04 smithi029 ceph-mon[32610]: pgmap v7399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:04 smithi077 ceph-mon[37153]: pgmap v7399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:05 smithi047 ceph-mon[36835]: pgmap v7400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:28:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:05 smithi029 ceph-mon[32610]: pgmap v7400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:28:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:05 smithi077 ceph-mon[37153]: pgmap v7400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:28:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13109.664350 secs 2024-07-27T10:28:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:08 smithi047 ceph-mon[36835]: pgmap v7401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:28:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13109.664350 secs 2024-07-27T10:28:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:08 smithi029 ceph-mon[32610]: pgmap v7401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:28:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13109.664350 secs 2024-07-27T10:28:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:08 smithi077 ceph-mon[37153]: pgmap v7401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:28:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:10 smithi047 ceph-mon[36835]: pgmap v7402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:28:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:28:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:28:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:28:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:28:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:10 smithi029 ceph-mon[32610]: pgmap v7402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:28:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:28:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:28:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:28:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:28:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:10 smithi077 ceph-mon[37153]: pgmap v7402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:28:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:28:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:28:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:28:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:28:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:11 smithi047 ceph-mon[36835]: pgmap v7403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:28:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:11 smithi047 ceph-mon[36835]: pgmap v7404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:28:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:11 smithi029 ceph-mon[32610]: pgmap v7403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:28:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:11 smithi029 ceph-mon[32610]: pgmap v7404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:28:12.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:11 smithi077 ceph-mon[37153]: pgmap v7403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:28:12.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:11 smithi077 ceph-mon[37153]: pgmap v7404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:28:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:12 smithi047 ceph-mon[36835]: pgmap v7405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:28:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:12 smithi029 ceph-mon[32610]: pgmap v7405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:28:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:12 smithi077 ceph-mon[37153]: pgmap v7405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:28:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13114.664491 secs 2024-07-27T10:28:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13114.664491 secs 2024-07-27T10:28:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13114.664491 secs 2024-07-27T10:28:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:14 smithi047 ceph-mon[36835]: pgmap v7406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:28:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:14 smithi029 ceph-mon[32610]: pgmap v7406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:28:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:14 smithi077 ceph-mon[37153]: pgmap v7406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:28:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:17 smithi047 ceph-mon[36835]: pgmap v7407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:28:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:17 smithi029 ceph-mon[32610]: pgmap v7407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:28:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:17 smithi077 ceph-mon[37153]: pgmap v7407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:28:18.402 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:18.401+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:28:18.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:18.401+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:28:18.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:18.401+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:28:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13119.664632 secs 2024-07-27T10:28:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13119.664632 secs 2024-07-27T10:28:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13119.664632 secs 2024-07-27T10:28:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:19 smithi047 ceph-mon[36835]: pgmap v7408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:28:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:19 smithi029 ceph-mon[32610]: pgmap v7408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:28:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:19 smithi077 ceph-mon[37153]: pgmap v7408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:28:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:21 smithi047 ceph-mon[36835]: pgmap v7409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:28:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:21 smithi029 ceph-mon[32610]: pgmap v7409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:28:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:21 smithi077 ceph-mon[37153]: pgmap v7409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:28:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13124.664767 secs 2024-07-27T10:28:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:23 smithi047 ceph-mon[36835]: pgmap v7410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13124.664767 secs 2024-07-27T10:28:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:23 smithi029 ceph-mon[32610]: pgmap v7410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13124.664767 secs 2024-07-27T10:28:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:23 smithi077 ceph-mon[37153]: pgmap v7410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:25 smithi047 ceph-mon[36835]: pgmap v7411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:28:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:25 smithi029 ceph-mon[32610]: pgmap v7411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:28:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:25 smithi077 ceph-mon[37153]: pgmap v7411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:28:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:27 smithi047 ceph-mon[36835]: pgmap v7412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:27.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:27 smithi029 ceph-mon[32610]: pgmap v7412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:27 smithi077 ceph-mon[37153]: pgmap v7412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13129.664890 secs 2024-07-27T10:28:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13129.664890 secs 2024-07-27T10:28:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13129.664890 secs 2024-07-27T10:28:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:29 smithi047 ceph-mon[36835]: pgmap v7413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:29.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:29 smithi029 ceph-mon[32610]: pgmap v7413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:29 smithi077 ceph-mon[37153]: pgmap v7413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:31 smithi047 ceph-mon[36835]: pgmap v7414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:28:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:31 smithi029 ceph-mon[32610]: pgmap v7414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:28:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:31 smithi077 ceph-mon[37153]: pgmap v7414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:28:33.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:33.412+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:28:33.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:33.412+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:28:33.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:33.412+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:28:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13134.664975 secs 2024-07-27T10:28:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:33 smithi047 ceph-mon[36835]: pgmap v7415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13134.664975 secs 2024-07-27T10:28:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:33 smithi029 ceph-mon[32610]: pgmap v7415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13134.664975 secs 2024-07-27T10:28:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:33 smithi077 ceph-mon[37153]: pgmap v7415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:35 smithi047 ceph-mon[36835]: pgmap v7416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:28:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:35 smithi029 ceph-mon[32610]: pgmap v7416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:28:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:35 smithi077 ceph-mon[37153]: pgmap v7416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:28:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:37 smithi029 ceph-mon[32610]: pgmap v7417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:37 smithi077 ceph-mon[37153]: pgmap v7417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:37 smithi047 ceph-mon[36835]: pgmap v7417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13139.665060 secs 2024-07-27T10:28:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13139.665060 secs 2024-07-27T10:28:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13139.665060 secs 2024-07-27T10:28:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:39 smithi029 ceph-mon[32610]: pgmap v7418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:39 smithi077 ceph-mon[37153]: pgmap v7418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:39 smithi047 ceph-mon[36835]: pgmap v7418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:41 smithi029 ceph-mon[32610]: pgmap v7419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:28:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:41 smithi077 ceph-mon[37153]: pgmap v7419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:28:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:41 smithi047 ceph-mon[36835]: pgmap v7419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:28:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13144.665182 secs 2024-07-27T10:28:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:43 smithi029 ceph-mon[32610]: pgmap v7420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:43.498 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13144.665182 secs 2024-07-27T10:28:43.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:43 smithi047 ceph-mon[36835]: pgmap v7420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13144.665182 secs 2024-07-27T10:28:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:43 smithi077 ceph-mon[37153]: pgmap v7420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:45 smithi077 ceph-mon[37153]: pgmap v7421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:28:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:45 smithi047 ceph-mon[36835]: pgmap v7421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:28:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:45 smithi029 ceph-mon[32610]: pgmap v7421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:28:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:47 smithi077 ceph-mon[37153]: pgmap v7422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:47 smithi047 ceph-mon[36835]: pgmap v7422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:47 smithi029 ceph-mon[32610]: pgmap v7422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:48.427+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:28:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:48.427+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:28:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:48.427+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:28:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13149.665303 secs 2024-07-27T10:28:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13149.665303 secs 2024-07-27T10:28:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13149.665303 secs 2024-07-27T10:28:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:49 smithi077 ceph-mon[37153]: pgmap v7423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:49 smithi047 ceph-mon[36835]: pgmap v7423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:49 smithi029 ceph-mon[32610]: pgmap v7423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:51 smithi077 ceph-mon[37153]: pgmap v7424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:28:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:51 smithi047 ceph-mon[36835]: pgmap v7424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:28:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:51 smithi029 ceph-mon[32610]: pgmap v7424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:28:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13154.665424 secs 2024-07-27T10:28:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:53 smithi077 ceph-mon[37153]: pgmap v7425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13154.665424 secs 2024-07-27T10:28:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:53 smithi047 ceph-mon[36835]: pgmap v7425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13154.665424 secs 2024-07-27T10:28:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:53 smithi029 ceph-mon[32610]: pgmap v7425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:55 smithi077 ceph-mon[37153]: pgmap v7426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:28:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:55 smithi047 ceph-mon[36835]: pgmap v7426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:28:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:55 smithi029 ceph-mon[32610]: pgmap v7426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:28:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:57 smithi077 ceph-mon[37153]: pgmap v7427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:57 smithi047 ceph-mon[36835]: pgmap v7427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:57 smithi029 ceph-mon[32610]: pgmap v7427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:28:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13159.665546 secs 2024-07-27T10:28:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13159.665546 secs 2024-07-27T10:28:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13159.665546 secs 2024-07-27T10:28:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:28:59 smithi077 ceph-mon[37153]: pgmap v7428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:59.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:59.631+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:28:59.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:59.631+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:28:59.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:28:59 smithi029 ceph-mon[32610]: pgmap v7428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:28:59.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:59.631+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:28:59.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:28:59.631+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:28:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:28:59 smithi047 ceph-mon[36835]: pgmap v7428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:01.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:01 smithi077 ceph-mon[37153]: pgmap v7429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:01 smithi047 ceph-mon[36835]: pgmap v7429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:01 smithi029 ceph-mon[32610]: pgmap v7429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:03.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:29:03.443+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:29:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:29:03.443+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:29:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:29:03.443+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:29:03.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13164.665665 secs 2024-07-27T10:29:03.515 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:03 smithi029 ceph-mon[32610]: pgmap v7430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13164.665665 secs 2024-07-27T10:29:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:03 smithi077 ceph-mon[37153]: pgmap v7430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13164.665665 secs 2024-07-27T10:29:03.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:03 smithi047 ceph-mon[36835]: pgmap v7430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:05 smithi077 ceph-mon[37153]: pgmap v7431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:29:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:05 smithi047 ceph-mon[36835]: pgmap v7431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:29:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:05 smithi029 ceph-mon[32610]: pgmap v7431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:29:07.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:07 smithi077 ceph-mon[37153]: pgmap v7432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:07 smithi047 ceph-mon[36835]: pgmap v7432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:07 smithi029 ceph-mon[32610]: pgmap v7432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13169.665814 secs 2024-07-27T10:29:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13169.665814 secs 2024-07-27T10:29:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13169.665814 secs 2024-07-27T10:29:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:09 smithi077 ceph-mon[37153]: pgmap v7433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:09 smithi047 ceph-mon[36835]: pgmap v7433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:09 smithi029 ceph-mon[32610]: pgmap v7433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:29:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:29:10.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:29:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:11 smithi077 ceph-mon[37153]: pgmap v7434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:11 smithi047 ceph-mon[36835]: pgmap v7434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:11.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:11 smithi029 ceph-mon[32610]: pgmap v7434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13174.665882 secs 2024-07-27T10:29:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:12 smithi077 ceph-mon[37153]: pgmap v7435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13174.665882 secs 2024-07-27T10:29:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:12 smithi047 ceph-mon[36835]: pgmap v7435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13174.665882 secs 2024-07-27T10:29:13.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:12 smithi029 ceph-mon[32610]: pgmap v7435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:13.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:13.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:14.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:29:14.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:29:14.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:13 smithi047 ceph-mon[36835]: pgmap v7436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T10:29:14.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:13 smithi047 ceph-mon[36835]: pgmap v7437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:29:14.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:29:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:29:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:13 smithi077 ceph-mon[37153]: pgmap v7436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T10:29:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:13 smithi077 ceph-mon[37153]: pgmap v7437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:29:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:29:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:29:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:13 smithi029 ceph-mon[32610]: pgmap v7436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T10:29:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:13 smithi029 ceph-mon[32610]: pgmap v7437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:29:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:29:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:15 smithi047 ceph-mon[36835]: pgmap v7438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-27T10:29:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:15 smithi029 ceph-mon[32610]: pgmap v7438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-27T10:29:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:15 smithi077 ceph-mon[37153]: pgmap v7438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-07-27T10:29:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:17 smithi047 ceph-mon[36835]: pgmap v7439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T10:29:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13179.666023 secs 2024-07-27T10:29:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:17 smithi029 ceph-mon[32610]: pgmap v7439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T10:29:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13179.666023 secs 2024-07-27T10:29:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:17 smithi077 ceph-mon[37153]: pgmap v7439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T10:29:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13179.666023 secs 2024-07-27T10:29:18.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:29:18.453+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:29:18.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:29:18.453+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:29:18.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:29:18.453+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:29:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:19 smithi047 ceph-mon[36835]: pgmap v7440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:19 smithi029 ceph-mon[32610]: pgmap v7440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:19 smithi077 ceph-mon[37153]: pgmap v7440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:21 smithi047 ceph-mon[36835]: pgmap v7441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:29:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:21 smithi029 ceph-mon[32610]: pgmap v7441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:29:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:21 smithi077 ceph-mon[37153]: pgmap v7441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:29:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13184.666142 secs 2024-07-27T10:29:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13184.666142 secs 2024-07-27T10:29:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13184.666142 secs 2024-07-27T10:29:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:23 smithi047 ceph-mon[36835]: pgmap v7442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:29:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:23 smithi029 ceph-mon[32610]: pgmap v7442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:29:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:23 smithi077 ceph-mon[37153]: pgmap v7442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:29:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:25 smithi047 ceph-mon[36835]: pgmap v7443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:29:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:25 smithi029 ceph-mon[32610]: pgmap v7443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:29:26.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:25 smithi077 ceph-mon[37153]: pgmap v7443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:29:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:27 smithi047 ceph-mon[36835]: pgmap v7444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13189.666265 secs 2024-07-27T10:29:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:27 smithi029 ceph-mon[32610]: pgmap v7444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13189.666265 secs 2024-07-27T10:29:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:27 smithi077 ceph-mon[37153]: pgmap v7444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13189.666265 secs 2024-07-27T10:29:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:29 smithi047 ceph-mon[36835]: pgmap v7445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:29 smithi029 ceph-mon[32610]: pgmap v7445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:29 smithi077 ceph-mon[37153]: pgmap v7445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:31 smithi047 ceph-mon[36835]: pgmap v7446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:31 smithi029 ceph-mon[32610]: pgmap v7446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:31 smithi077 ceph-mon[37153]: pgmap v7446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13194.666387 secs 2024-07-27T10:29:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13194.666387 secs 2024-07-27T10:29:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13194.666387 secs 2024-07-27T10:29:33.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:29:33.468+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:29:33.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:29:33.468+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:29:33.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:29:33.468+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:29:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:33 smithi047 ceph-mon[36835]: pgmap v7447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:29:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:33 smithi029 ceph-mon[32610]: pgmap v7447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:29:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:33 smithi077 ceph-mon[37153]: pgmap v7447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:29:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:35 smithi047 ceph-mon[36835]: pgmap v7448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:29:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:35 smithi029 ceph-mon[32610]: pgmap v7448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:29:36.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:35 smithi077 ceph-mon[37153]: pgmap v7448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:29:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:37 smithi047 ceph-mon[36835]: pgmap v7449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13199.666514 secs 2024-07-27T10:29:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:37 smithi029 ceph-mon[32610]: pgmap v7449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13199.666514 secs 2024-07-27T10:29:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:37 smithi077 ceph-mon[37153]: pgmap v7449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13199.666514 secs 2024-07-27T10:29:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:39 smithi047 ceph-mon[36835]: pgmap v7450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:39 smithi029 ceph-mon[32610]: pgmap v7450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:39 smithi077 ceph-mon[37153]: pgmap v7450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:41 smithi047 ceph-mon[36835]: pgmap v7451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:42.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:41 smithi029 ceph-mon[32610]: pgmap v7451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:41 smithi077 ceph-mon[37153]: pgmap v7451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13204.666665 secs 2024-07-27T10:29:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13204.666665 secs 2024-07-27T10:29:43.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13204.666665 secs 2024-07-27T10:29:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:43 smithi047 ceph-mon[36835]: pgmap v7452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:29:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:43 smithi029 ceph-mon[32610]: pgmap v7452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:29:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:43 smithi077 ceph-mon[37153]: pgmap v7452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:29:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:45 smithi047 ceph-mon[36835]: pgmap v7453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:29:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:45 smithi029 ceph-mon[32610]: pgmap v7453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:29:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:45 smithi077 ceph-mon[37153]: pgmap v7453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:29:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:47 smithi029 ceph-mon[32610]: pgmap v7454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13209.666810 secs 2024-07-27T10:29:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:47 smithi077 ceph-mon[37153]: pgmap v7454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13209.666810 secs 2024-07-27T10:29:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:47 smithi047 ceph-mon[36835]: pgmap v7454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13209.666810 secs 2024-07-27T10:29:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:29:48.483+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:29:48.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:29:48.483+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:29:48.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:29:48.483+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:29:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:49 smithi029 ceph-mon[32610]: pgmap v7455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:49 smithi077 ceph-mon[37153]: pgmap v7455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:49 smithi047 ceph-mon[36835]: pgmap v7455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:29:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:51 smithi029 ceph-mon[32610]: pgmap v7456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:51 smithi077 ceph-mon[37153]: pgmap v7456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:51 smithi047 ceph-mon[36835]: pgmap v7456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:29:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13214.666840 secs 2024-07-27T10:29:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13214.666840 secs 2024-07-27T10:29:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13214.666840 secs 2024-07-27T10:29:54.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:53 smithi029 ceph-mon[32610]: pgmap v7457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:29:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:53 smithi077 ceph-mon[37153]: pgmap v7457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:29:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:53 smithi047 ceph-mon[36835]: pgmap v7457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:29:56.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:55 smithi077 ceph-mon[37153]: pgmap v7458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:29:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:55 smithi047 ceph-mon[36835]: pgmap v7458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:29:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:55 smithi029 ceph-mon[32610]: pgmap v7458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:29:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:57 smithi077 ceph-mon[37153]: pgmap v7459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13219.666964 secs 2024-07-27T10:29:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:57 smithi047 ceph-mon[36835]: pgmap v7459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13219.666964 secs 2024-07-27T10:29:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:57 smithi029 ceph-mon[32610]: pgmap v7459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:29:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13219.666964 secs 2024-07-27T10:30:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:29:59 smithi077 ceph-mon[37153]: pgmap v7460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:29:59 smithi047 ceph-mon[36835]: pgmap v7460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:29:59 smithi029 ceph-mon[32610]: pgmap v7460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:30:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:30:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:30:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:01 smithi077 ceph-mon[37153]: pgmap v7461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:01 smithi047 ceph-mon[36835]: pgmap v7461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:01 smithi029 ceph-mon[32610]: pgmap v7461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13224.667094 secs 2024-07-27T10:30:03.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13224.667094 secs 2024-07-27T10:30:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13224.667094 secs 2024-07-27T10:30:03.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:30:03.493+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:30:03.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:30:03.493+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:30:03.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:30:03.493+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:30:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:03 smithi077 ceph-mon[37153]: pgmap v7462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:30:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:03 smithi047 ceph-mon[36835]: pgmap v7462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:30:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:03 smithi029 ceph-mon[32610]: pgmap v7462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:30:06.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:05 smithi077 ceph-mon[37153]: pgmap v7463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:30:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:05 smithi047 ceph-mon[36835]: pgmap v7463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:30:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:05 smithi029 ceph-mon[32610]: pgmap v7463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:30:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:08 smithi077 ceph-mon[37153]: pgmap v7464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:30:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13229.667154 secs 2024-07-27T10:30:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:08 smithi047 ceph-mon[36835]: pgmap v7464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:30:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13229.667154 secs 2024-07-27T10:30:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:08 smithi029 ceph-mon[32610]: pgmap v7464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:30:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13229.667154 secs 2024-07-27T10:30:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:10 smithi077 ceph-mon[37153]: pgmap v7465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:10 smithi047 ceph-mon[36835]: pgmap v7465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:10 smithi029 ceph-mon[32610]: pgmap v7465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:12 smithi077 ceph-mon[37153]: pgmap v7466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:12 smithi047 ceph-mon[36835]: pgmap v7466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:12.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:12 smithi029 ceph-mon[32610]: pgmap v7466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13234.667283 secs 2024-07-27T10:30:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:30:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13234.667283 secs 2024-07-27T10:30:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:30:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:13.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13234.667283 secs 2024-07-27T10:30:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:30:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:13.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:14.797 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:14 smithi077 ceph-mon[37153]: pgmap v7467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:30:14.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:30:14.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:30:14.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:14.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:14 smithi047 ceph-mon[36835]: pgmap v7467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:30:14.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:30:14.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:30:14.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:14 smithi029 ceph-mon[32610]: pgmap v7467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:30:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:30:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:30:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:30:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:15 smithi077 ceph-mon[37153]: pgmap v7468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:30:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:15 smithi077 ceph-mon[37153]: pgmap v7469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:30:15.862 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:15 smithi047 ceph-mon[36835]: pgmap v7468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:30:15.862 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:15 smithi047 ceph-mon[36835]: pgmap v7469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:30:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:15 smithi029 ceph-mon[32610]: pgmap v7468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:30:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:15 smithi029 ceph-mon[32610]: pgmap v7469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:30:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:17 smithi077 ceph-mon[37153]: pgmap v7470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:30:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:17 smithi047 ceph-mon[36835]: pgmap v7470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:30:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:17 smithi029 ceph-mon[32610]: pgmap v7470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:30:18.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:30:18.509+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:30:18.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:30:18.509+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:30:18.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:30:18.509+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:30:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13239.667407 secs 2024-07-27T10:30:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13239.667407 secs 2024-07-27T10:30:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13239.667407 secs 2024-07-27T10:30:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:19 smithi077 ceph-mon[37153]: pgmap v7471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:30:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:19 smithi047 ceph-mon[36835]: pgmap v7471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:30:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:19 smithi029 ceph-mon[32610]: pgmap v7471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:30:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:21 smithi077 ceph-mon[37153]: pgmap v7472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:30:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:21 smithi047 ceph-mon[36835]: pgmap v7472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:30:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:21 smithi029 ceph-mon[32610]: pgmap v7472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:30:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:23 smithi077 ceph-mon[37153]: pgmap v7473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:30:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13244.667535 secs 2024-07-27T10:30:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:23 smithi047 ceph-mon[36835]: pgmap v7473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:30:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13244.667535 secs 2024-07-27T10:30:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:23 smithi029 ceph-mon[32610]: pgmap v7473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:30:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13244.667535 secs 2024-07-27T10:30:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:25 smithi077 ceph-mon[37153]: pgmap v7474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:30:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:25 smithi047 ceph-mon[36835]: pgmap v7474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:30:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:25 smithi029 ceph-mon[32610]: pgmap v7474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:30:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:27 smithi077 ceph-mon[37153]: pgmap v7475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:27 smithi047 ceph-mon[36835]: pgmap v7475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:27.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:27 smithi029 ceph-mon[32610]: pgmap v7475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13249.667688 secs 2024-07-27T10:30:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13249.667688 secs 2024-07-27T10:30:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13249.667688 secs 2024-07-27T10:30:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:29 smithi077 ceph-mon[37153]: pgmap v7476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:30:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:29 smithi047 ceph-mon[36835]: pgmap v7476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:30:29.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:29 smithi029 ceph-mon[32610]: pgmap v7476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:30:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:31 smithi077 ceph-mon[37153]: pgmap v7477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:30:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:31 smithi047 ceph-mon[36835]: pgmap v7477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:30:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:31 smithi029 ceph-mon[32610]: pgmap v7477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:30:33.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:30:33.521+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:30:33.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:30:33.521+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:30:33.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:30:33.521+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:30:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:33 smithi077 ceph-mon[37153]: pgmap v7478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:30:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13254.667837 secs 2024-07-27T10:30:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:33 smithi047 ceph-mon[36835]: pgmap v7478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:30:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13254.667837 secs 2024-07-27T10:30:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:33 smithi029 ceph-mon[32610]: pgmap v7478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:30:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13254.667837 secs 2024-07-27T10:30:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:35 smithi077 ceph-mon[37153]: pgmap v7479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:35 smithi047 ceph-mon[36835]: pgmap v7479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:35 smithi029 ceph-mon[32610]: pgmap v7479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:37 smithi077 ceph-mon[37153]: pgmap v7480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:37 smithi047 ceph-mon[36835]: pgmap v7480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:37 smithi029 ceph-mon[32610]: pgmap v7480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13259.667960 secs 2024-07-27T10:30:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13259.667960 secs 2024-07-27T10:30:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13259.667960 secs 2024-07-27T10:30:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:39 smithi077 ceph-mon[37153]: pgmap v7481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:30:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:39 smithi047 ceph-mon[36835]: pgmap v7481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:30:39.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:39 smithi029 ceph-mon[32610]: pgmap v7481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:30:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:41 smithi077 ceph-mon[37153]: pgmap v7482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:30:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:41 smithi047 ceph-mon[36835]: pgmap v7482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:30:41.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:41 smithi029 ceph-mon[32610]: pgmap v7482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:30:43.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:43 smithi077 ceph-mon[37153]: pgmap v7483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:30:43.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13264.668085 secs 2024-07-27T10:30:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:43 smithi047 ceph-mon[36835]: pgmap v7483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:30:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13264.668085 secs 2024-07-27T10:30:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:43 smithi029 ceph-mon[32610]: pgmap v7483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:30:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13264.668085 secs 2024-07-27T10:30:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:45 smithi077 ceph-mon[37153]: pgmap v7484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:45 smithi047 ceph-mon[36835]: pgmap v7484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:45 smithi029 ceph-mon[32610]: pgmap v7484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:47 smithi077 ceph-mon[37153]: pgmap v7485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:47 smithi047 ceph-mon[36835]: pgmap v7485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:47 smithi029 ceph-mon[32610]: pgmap v7485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:48.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:30:48.536+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:30:48.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:30:48.536+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:30:48.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:30:48.536+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:30:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13269.668217 secs 2024-07-27T10:30:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13269.668217 secs 2024-07-27T10:30:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13269.668217 secs 2024-07-27T10:30:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:49 smithi047 ceph-mon[36835]: pgmap v7486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:30:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:49 smithi029 ceph-mon[32610]: pgmap v7486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:30:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:49 smithi077 ceph-mon[37153]: pgmap v7486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:30:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:51 smithi047 ceph-mon[36835]: pgmap v7487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:30:51.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:51 smithi029 ceph-mon[32610]: pgmap v7487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:30:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:51 smithi077 ceph-mon[37153]: pgmap v7487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:30:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:53 smithi047 ceph-mon[36835]: pgmap v7488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:30:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13274.668344 secs 2024-07-27T10:30:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:53 smithi029 ceph-mon[32610]: pgmap v7488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:30:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13274.668344 secs 2024-07-27T10:30:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:53 smithi077 ceph-mon[37153]: pgmap v7488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:30:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13274.668344 secs 2024-07-27T10:30:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:55 smithi047 ceph-mon[36835]: pgmap v7489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:55 smithi029 ceph-mon[32610]: pgmap v7489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:55 smithi077 ceph-mon[37153]: pgmap v7489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:30:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:57 smithi047 ceph-mon[36835]: pgmap v7490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:57 smithi029 ceph-mon[32610]: pgmap v7490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:57 smithi077 ceph-mon[37153]: pgmap v7490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:30:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13279.668469 secs 2024-07-27T10:30:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13279.668469 secs 2024-07-27T10:30:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13279.668469 secs 2024-07-27T10:30:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:30:59 smithi047 ceph-mon[36835]: pgmap v7491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:30:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:30:59 smithi029 ceph-mon[32610]: pgmap v7491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:31:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:30:59 smithi077 ceph-mon[37153]: pgmap v7491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:31:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:01 smithi047 ceph-mon[36835]: pgmap v7492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:31:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:01 smithi029 ceph-mon[32610]: pgmap v7492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:31:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:01 smithi077 ceph-mon[37153]: pgmap v7492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:31:03.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:03.551+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:31:03.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:03.551+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:31:03.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:03.551+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:31:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:03 smithi047 ceph-mon[36835]: pgmap v7493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13284.668595 secs 2024-07-27T10:31:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:03 smithi029 ceph-mon[32610]: pgmap v7493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13284.668595 secs 2024-07-27T10:31:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:03 smithi077 ceph-mon[37153]: pgmap v7493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13284.668595 secs 2024-07-27T10:31:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:05 smithi047 ceph-mon[36835]: pgmap v7494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:05 smithi029 ceph-mon[32610]: pgmap v7494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:05 smithi077 ceph-mon[37153]: pgmap v7494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:07 smithi047 ceph-mon[36835]: pgmap v7495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:31:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:07 smithi029 ceph-mon[32610]: pgmap v7495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:31:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:07 smithi077 ceph-mon[37153]: pgmap v7495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:31:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13289.668739 secs 2024-07-27T10:31:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13289.668739 secs 2024-07-27T10:31:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13289.668739 secs 2024-07-27T10:31:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:09 smithi047 ceph-mon[36835]: pgmap v7496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:31:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:09 smithi029 ceph-mon[32610]: pgmap v7496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:31:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:09 smithi077 ceph-mon[37153]: pgmap v7496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:31:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:11 smithi047 ceph-mon[36835]: pgmap v7497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:31:11.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:11 smithi029 ceph-mon[32610]: pgmap v7497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:31:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:11 smithi077 ceph-mon[37153]: pgmap v7497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:31:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:12 smithi029 ceph-mon[32610]: pgmap v7498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:13.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:12 smithi077 ceph-mon[37153]: pgmap v7498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:12 smithi047 ceph-mon[36835]: pgmap v7498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13294.668891 secs 2024-07-27T10:31:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13294.668891 secs 2024-07-27T10:31:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13294.668891 secs 2024-07-27T10:31:15.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:14 smithi047 ceph-mon[36835]: pgmap v7499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:15.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:31:15.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:31:15.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:31:15.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:14 smithi047 ceph-mon[36835]: pgmap v7500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:31:15.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:14 smithi047 ceph-mon[36835]: pgmap v7501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:31:15.099 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:31:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:14 smithi077 ceph-mon[37153]: pgmap v7499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:31:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:31:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:31:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:14 smithi077 ceph-mon[37153]: pgmap v7500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:31:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:14 smithi077 ceph-mon[37153]: pgmap v7501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:31:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:31:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:14 smithi029 ceph-mon[32610]: pgmap v7499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:31:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:31:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:31:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:14 smithi029 ceph-mon[32610]: pgmap v7500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:31:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:14 smithi029 ceph-mon[32610]: pgmap v7501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:31:15.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:31:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:17 smithi077 ceph-mon[37153]: pgmap v7502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:31:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:17 smithi047 ceph-mon[36835]: pgmap v7502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:31:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:17 smithi029 ceph-mon[32610]: pgmap v7502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:31:18.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:18.567+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:31:18.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:18.567+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:31:18.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:18.567+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:31:18.569 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13299.668983 secs 2024-07-27T10:31:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13299.668983 secs 2024-07-27T10:31:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13299.668983 secs 2024-07-27T10:31:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:19 smithi077 ceph-mon[37153]: pgmap v7503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:31:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:19 smithi047 ceph-mon[36835]: pgmap v7503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:31:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:19 smithi029 ceph-mon[32610]: pgmap v7503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:31:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:21 smithi077 ceph-mon[37153]: pgmap v7504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:31:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:21 smithi047 ceph-mon[36835]: pgmap v7504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:31:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:21 smithi029 ceph-mon[32610]: pgmap v7504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:31:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13304.669116 secs 2024-07-27T10:31:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:23 smithi077 ceph-mon[37153]: pgmap v7505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:31:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13304.669116 secs 2024-07-27T10:31:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:23 smithi047 ceph-mon[36835]: pgmap v7505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:31:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13304.669116 secs 2024-07-27T10:31:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:23 smithi029 ceph-mon[32610]: pgmap v7505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:31:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:25 smithi077 ceph-mon[37153]: pgmap v7506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:31:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:25 smithi047 ceph-mon[36835]: pgmap v7506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:31:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:25 smithi029 ceph-mon[32610]: pgmap v7506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:31:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:27 smithi077 ceph-mon[37153]: pgmap v7507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:27 smithi047 ceph-mon[36835]: pgmap v7507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:27 smithi029 ceph-mon[32610]: pgmap v7507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13309.669235 secs 2024-07-27T10:31:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13309.669235 secs 2024-07-27T10:31:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13309.669235 secs 2024-07-27T10:31:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:29 smithi077 ceph-mon[37153]: pgmap v7508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:29 smithi047 ceph-mon[36835]: pgmap v7508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:29 smithi029 ceph-mon[32610]: pgmap v7508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:31 smithi077 ceph-mon[37153]: pgmap v7509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:31:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:31 smithi047 ceph-mon[36835]: pgmap v7509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:31:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:31 smithi029 ceph-mon[32610]: pgmap v7509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:31:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:33.583+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:31:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:33.583+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:31:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:33.583+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:31:33.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13314.669359 secs 2024-07-27T10:31:33.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:33 smithi029 ceph-mon[32610]: pgmap v7510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:31:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13314.669359 secs 2024-07-27T10:31:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:33 smithi077 ceph-mon[37153]: pgmap v7510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:31:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13314.669359 secs 2024-07-27T10:31:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:33 smithi047 ceph-mon[36835]: pgmap v7510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:31:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:35 smithi077 ceph-mon[37153]: pgmap v7511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:31:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:35 smithi047 ceph-mon[36835]: pgmap v7511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:31:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:35 smithi029 ceph-mon[32610]: pgmap v7511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:31:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:37 smithi077 ceph-mon[37153]: pgmap v7512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:37 smithi047 ceph-mon[36835]: pgmap v7512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:37.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:37 smithi029 ceph-mon[32610]: pgmap v7512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13319.669503 secs 2024-07-27T10:31:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13319.669503 secs 2024-07-27T10:31:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13319.669503 secs 2024-07-27T10:31:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:39 smithi077 ceph-mon[37153]: pgmap v7513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:39 smithi047 ceph-mon[36835]: pgmap v7513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:39 smithi029 ceph-mon[32610]: pgmap v7513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:41 smithi077 ceph-mon[37153]: pgmap v7514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:31:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:41 smithi047 ceph-mon[36835]: pgmap v7514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:31:41.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:41 smithi029 ceph-mon[32610]: pgmap v7514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:31:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13324.669585 secs 2024-07-27T10:31:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:43 smithi077 ceph-mon[37153]: pgmap v7515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:31:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13324.669585 secs 2024-07-27T10:31:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:43 smithi047 ceph-mon[36835]: pgmap v7515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:31:43.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13324.669585 secs 2024-07-27T10:31:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:43 smithi029 ceph-mon[32610]: pgmap v7515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:31:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:45 smithi047 ceph-mon[36835]: pgmap v7516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:31:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:45 smithi029 ceph-mon[32610]: pgmap v7516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:31:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:45 smithi077 ceph-mon[37153]: pgmap v7516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:31:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:47 smithi047 ceph-mon[36835]: pgmap v7517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:47 smithi029 ceph-mon[32610]: pgmap v7517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:47 smithi077 ceph-mon[37153]: pgmap v7517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:48.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:48.597+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:31:48.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:48.597+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:31:48.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:48.597+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:31:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13329.669727 secs 2024-07-27T10:31:48.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13329.669727 secs 2024-07-27T10:31:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13329.669727 secs 2024-07-27T10:31:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:49 smithi047 ceph-mon[36835]: pgmap v7518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:49 smithi029 ceph-mon[32610]: pgmap v7518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:49 smithi077 ceph-mon[37153]: pgmap v7518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:51 smithi047 ceph-mon[36835]: pgmap v7519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:31:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:51 smithi029 ceph-mon[32610]: pgmap v7519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:31:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:51 smithi077 ceph-mon[37153]: pgmap v7519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:31:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13334.669858 secs 2024-07-27T10:31:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:53 smithi047 ceph-mon[36835]: pgmap v7520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:31:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13334.669858 secs 2024-07-27T10:31:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:53 smithi029 ceph-mon[32610]: pgmap v7520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:31:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13334.669858 secs 2024-07-27T10:31:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:53 smithi077 ceph-mon[37153]: pgmap v7520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:31:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:55 smithi047 ceph-mon[36835]: pgmap v7521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:31:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:55 smithi029 ceph-mon[32610]: pgmap v7521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:31:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:55 smithi077 ceph-mon[37153]: pgmap v7521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:31:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:57 smithi047 ceph-mon[36835]: pgmap v7522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:57 smithi029 ceph-mon[32610]: pgmap v7522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:57 smithi077 ceph-mon[37153]: pgmap v7522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:31:58.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13339.669988 secs 2024-07-27T10:31:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13339.669988 secs 2024-07-27T10:31:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13339.669988 secs 2024-07-27T10:31:59.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:59.659+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:31:59.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:59.659+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:31:59.661 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:31:59 smithi029 ceph-mon[32610]: pgmap v7523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:59.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:59.659+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:31:59.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:31:59.659+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:31:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:31:59 smithi047 ceph-mon[36835]: pgmap v7523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:31:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:31:59 smithi077 ceph-mon[37153]: pgmap v7523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:01 smithi047 ceph-mon[36835]: pgmap v7524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:01 smithi029 ceph-mon[32610]: pgmap v7524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:01 smithi077 ceph-mon[37153]: pgmap v7524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:03.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:03.613+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:32:03.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:03.613+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:32:03.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:03.613+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:32:03.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13344.670107 secs 2024-07-27T10:32:03.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:03 smithi047 ceph-mon[36835]: pgmap v7525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:32:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13344.670107 secs 2024-07-27T10:32:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:03 smithi029 ceph-mon[32610]: pgmap v7525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:32:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13344.670107 secs 2024-07-27T10:32:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:03 smithi077 ceph-mon[37153]: pgmap v7525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:32:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:05 smithi047 ceph-mon[36835]: pgmap v7526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:32:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:05 smithi029 ceph-mon[32610]: pgmap v7526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:32:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:05 smithi077 ceph-mon[37153]: pgmap v7526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:32:07.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:07 smithi047 ceph-mon[36835]: pgmap v7527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:07 smithi029 ceph-mon[32610]: pgmap v7527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:07 smithi077 ceph-mon[37153]: pgmap v7527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13349.670238 secs 2024-07-27T10:32:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13349.670238 secs 2024-07-27T10:32:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13349.670238 secs 2024-07-27T10:32:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:09 smithi047 ceph-mon[36835]: pgmap v7528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:09 smithi029 ceph-mon[32610]: pgmap v7528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:09 smithi077 ceph-mon[37153]: pgmap v7528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:11 smithi047 ceph-mon[36835]: pgmap v7529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:11.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:11 smithi029 ceph-mon[32610]: pgmap v7529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:11 smithi077 ceph-mon[37153]: pgmap v7529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13354.670386 secs 2024-07-27T10:32:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:13 smithi047 ceph-mon[36835]: pgmap v7530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:32:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13354.670386 secs 2024-07-27T10:32:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:13 smithi029 ceph-mon[32610]: pgmap v7530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:32:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13354.670386 secs 2024-07-27T10:32:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:13 smithi077 ceph-mon[37153]: pgmap v7530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:32:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:32:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:32:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:32:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:15 smithi047 ceph-mon[36835]: pgmap v7531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:32:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:32:15.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:32:15.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:32:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:15 smithi029 ceph-mon[32610]: pgmap v7531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:32:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:32:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:32:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:32:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:15 smithi077 ceph-mon[37153]: pgmap v7531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:32:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:32:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:32:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:32:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:16 smithi047 ceph-mon[36835]: pgmap v7532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:32:16.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:16 smithi047 ceph-mon[36835]: pgmap v7533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:32:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:16 smithi029 ceph-mon[32610]: pgmap v7532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:32:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:16 smithi029 ceph-mon[32610]: pgmap v7533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:32:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:16 smithi077 ceph-mon[37153]: pgmap v7532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:32:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:16 smithi077 ceph-mon[37153]: pgmap v7533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:32:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:18.629+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:32:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:18.629+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:32:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:18.629+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:32:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:18 smithi047 ceph-mon[36835]: pgmap v7534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:32:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13359.670510 secs 2024-07-27T10:32:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:18 smithi029 ceph-mon[32610]: pgmap v7534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:32:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13359.670510 secs 2024-07-27T10:32:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:18 smithi077 ceph-mon[37153]: pgmap v7534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:32:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13359.670510 secs 2024-07-27T10:32:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:20 smithi047 ceph-mon[36835]: pgmap v7535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:32:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:20 smithi029 ceph-mon[32610]: pgmap v7535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:32:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:20 smithi077 ceph-mon[37153]: pgmap v7535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:32:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:22 smithi047 ceph-mon[36835]: pgmap v7536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:32:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:22 smithi029 ceph-mon[32610]: pgmap v7536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:32:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:22 smithi077 ceph-mon[37153]: pgmap v7536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:32:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13364.670630 secs 2024-07-27T10:32:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13364.670630 secs 2024-07-27T10:32:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13364.670630 secs 2024-07-27T10:32:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:24 smithi029 ceph-mon[32610]: pgmap v7537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:32:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:24 smithi077 ceph-mon[37153]: pgmap v7537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:32:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:24 smithi047 ceph-mon[36835]: pgmap v7537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:32:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:26 smithi029 ceph-mon[32610]: pgmap v7538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:32:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:26 smithi077 ceph-mon[37153]: pgmap v7538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:32:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:26 smithi047 ceph-mon[36835]: pgmap v7538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:32:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:28 smithi029 ceph-mon[32610]: pgmap v7539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13369.670815 secs 2024-07-27T10:32:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:28 smithi077 ceph-mon[37153]: pgmap v7539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13369.670815 secs 2024-07-27T10:32:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:28 smithi047 ceph-mon[36835]: pgmap v7539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13369.670815 secs 2024-07-27T10:32:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:30 smithi029 ceph-mon[32610]: pgmap v7540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:30 smithi077 ceph-mon[37153]: pgmap v7540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:30 smithi047 ceph-mon[36835]: pgmap v7540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:32 smithi029 ceph-mon[32610]: pgmap v7541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:32 smithi077 ceph-mon[37153]: pgmap v7541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:32 smithi047 ceph-mon[36835]: pgmap v7541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:33.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:33.630+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:32:33.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:33.630+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:32:33.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:33.630+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:32:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13374.670892 secs 2024-07-27T10:32:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13374.670892 secs 2024-07-27T10:32:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13374.670892 secs 2024-07-27T10:32:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:34 smithi077 ceph-mon[37153]: pgmap v7542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:32:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:34 smithi047 ceph-mon[36835]: pgmap v7542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:32:34.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:34 smithi029 ceph-mon[32610]: pgmap v7542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:32:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:36 smithi077 ceph-mon[37153]: pgmap v7543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:32:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:36 smithi047 ceph-mon[36835]: pgmap v7543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:32:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:36 smithi029 ceph-mon[32610]: pgmap v7543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:32:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:38 smithi077 ceph-mon[37153]: pgmap v7544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13379.670987 secs 2024-07-27T10:32:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:38 smithi047 ceph-mon[36835]: pgmap v7544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13379.670987 secs 2024-07-27T10:32:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:38 smithi029 ceph-mon[32610]: pgmap v7544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13379.670987 secs 2024-07-27T10:32:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:40 smithi077 ceph-mon[37153]: pgmap v7545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:40 smithi047 ceph-mon[36835]: pgmap v7545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:40 smithi029 ceph-mon[32610]: pgmap v7545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:42 smithi077 ceph-mon[37153]: pgmap v7546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:42 smithi047 ceph-mon[36835]: pgmap v7546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:42.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:42 smithi029 ceph-mon[32610]: pgmap v7546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:43.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13384.671074 secs 2024-07-27T10:32:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13384.671074 secs 2024-07-27T10:32:43.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13384.671074 secs 2024-07-27T10:32:44.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:44.666+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:32:44.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:44.666+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:32:44.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:44.666+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:32:44.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:44.666+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:32:44.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:44 smithi029 ceph-mon[32610]: pgmap v7547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:32:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:44 smithi077 ceph-mon[37153]: pgmap v7547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:32:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:44 smithi047 ceph-mon[36835]: pgmap v7547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:32:46.749 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:46 smithi047 ceph-mon[36835]: pgmap v7548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:32:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:46 smithi077 ceph-mon[37153]: pgmap v7548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:32:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:46 smithi029 ceph-mon[32610]: pgmap v7548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:32:48.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:48.644+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:32:48.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:48.644+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:32:48.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:32:48.644+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:32:48.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:48 smithi077 ceph-mon[37153]: pgmap v7549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:48.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13389.671203 secs 2024-07-27T10:32:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:48 smithi047 ceph-mon[36835]: pgmap v7549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:48.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13389.671203 secs 2024-07-27T10:32:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:48 smithi029 ceph-mon[32610]: pgmap v7549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13389.671203 secs 2024-07-27T10:32:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:50 smithi077 ceph-mon[37153]: pgmap v7550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:50 smithi047 ceph-mon[36835]: pgmap v7550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:50 smithi029 ceph-mon[32610]: pgmap v7550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:32:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:52 smithi077 ceph-mon[37153]: pgmap v7551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:52 smithi047 ceph-mon[36835]: pgmap v7551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:52 smithi029 ceph-mon[32610]: pgmap v7551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:32:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13394.671290 secs 2024-07-27T10:32:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13394.671290 secs 2024-07-27T10:32:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13394.671290 secs 2024-07-27T10:32:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:54 smithi077 ceph-mon[37153]: pgmap v7552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:32:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:54 smithi047 ceph-mon[36835]: pgmap v7552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:32:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:54 smithi029 ceph-mon[32610]: pgmap v7552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:32:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:56 smithi077 ceph-mon[37153]: pgmap v7553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:32:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:56 smithi047 ceph-mon[36835]: pgmap v7553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:32:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:56 smithi029 ceph-mon[32610]: pgmap v7553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:32:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:58 smithi077 ceph-mon[37153]: pgmap v7554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:32:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13399.671355 secs 2024-07-27T10:32:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:58 smithi047 ceph-mon[36835]: pgmap v7554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:32:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13399.671355 secs 2024-07-27T10:32:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:58 smithi029 ceph-mon[32610]: pgmap v7554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:32:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:32:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13399.671355 secs 2024-07-27T10:33:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:00 smithi077 ceph-mon[37153]: pgmap v7555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:00 smithi047 ceph-mon[36835]: pgmap v7555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:00 smithi029 ceph-mon[32610]: pgmap v7555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:02 smithi077 ceph-mon[37153]: pgmap v7556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:02 smithi047 ceph-mon[36835]: pgmap v7556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:02 smithi029 ceph-mon[32610]: pgmap v7556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:33:03.658+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:33:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:33:03.658+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:33:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:33:03.658+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:33:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13404.671423 secs 2024-07-27T10:33:03.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13404.671423 secs 2024-07-27T10:33:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13404.671423 secs 2024-07-27T10:33:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:04 smithi077 ceph-mon[37153]: pgmap v7557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:33:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:04 smithi047 ceph-mon[36835]: pgmap v7557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:33:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:04 smithi029 ceph-mon[32610]: pgmap v7557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:33:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:06 smithi077 ceph-mon[37153]: pgmap v7558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:33:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:06 smithi047 ceph-mon[36835]: pgmap v7558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:33:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:06 smithi029 ceph-mon[32610]: pgmap v7558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:33:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:08 smithi077 ceph-mon[37153]: pgmap v7559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:33:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13409.671541 secs 2024-07-27T10:33:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:08 smithi047 ceph-mon[36835]: pgmap v7559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:33:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13409.671541 secs 2024-07-27T10:33:08.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:08 smithi029 ceph-mon[32610]: pgmap v7559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:33:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13409.671541 secs 2024-07-27T10:33:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:10 smithi077 ceph-mon[37153]: pgmap v7560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:10 smithi047 ceph-mon[36835]: pgmap v7560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:10 smithi029 ceph-mon[32610]: pgmap v7560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:12 smithi077 ceph-mon[37153]: pgmap v7561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:12 smithi047 ceph-mon[36835]: pgmap v7561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:12.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:12 smithi029 ceph-mon[32610]: pgmap v7561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13414.671642 secs 2024-07-27T10:33:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13414.671642 secs 2024-07-27T10:33:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13414.671642 secs 2024-07-27T10:33:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:14 smithi077 ceph-mon[37153]: pgmap v7562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:33:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:14 smithi047 ceph-mon[36835]: pgmap v7562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:33:14.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:14 smithi029 ceph-mon[32610]: pgmap v7562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:33:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:33:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:33:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:33:15.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:33:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:33:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:33:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:33:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:33:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:33:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:33:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:33:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:33:16.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:16 smithi047 ceph-mon[36835]: pgmap v7563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:33:16.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:16 smithi047 ceph-mon[36835]: pgmap v7564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:33:16.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:16 smithi047 ceph-mon[36835]: pgmap v7565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:33:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:16 smithi077 ceph-mon[37153]: pgmap v7563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:33:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:16 smithi077 ceph-mon[37153]: pgmap v7564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:33:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:16 smithi077 ceph-mon[37153]: pgmap v7565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:33:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:16 smithi029 ceph-mon[32610]: pgmap v7563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:33:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:16 smithi029 ceph-mon[32610]: pgmap v7564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:33:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:16 smithi029 ceph-mon[32610]: pgmap v7565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:33:18.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:33:18.674+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:33:18.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:33:18.674+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:33:18.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:33:18.674+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:33:18.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13419.671762 secs 2024-07-27T10:33:18.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:18 smithi077 ceph-mon[37153]: pgmap v7566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T10:33:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13419.671762 secs 2024-07-27T10:33:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:18 smithi047 ceph-mon[36835]: pgmap v7566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T10:33:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13419.671762 secs 2024-07-27T10:33:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:18 smithi029 ceph-mon[32610]: pgmap v7566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T10:33:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:20 smithi077 ceph-mon[37153]: pgmap v7567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:33:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:20 smithi047 ceph-mon[36835]: pgmap v7567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:33:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:20 smithi029 ceph-mon[32610]: pgmap v7567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:33:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:22 smithi077 ceph-mon[37153]: pgmap v7568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:33:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:22 smithi047 ceph-mon[36835]: pgmap v7568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:33:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:22 smithi029 ceph-mon[32610]: pgmap v7568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:33:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13424.671894 secs 2024-07-27T10:33:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13424.671894 secs 2024-07-27T10:33:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13424.671894 secs 2024-07-27T10:33:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:24 smithi077 ceph-mon[37153]: pgmap v7569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:33:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:24 smithi047 ceph-mon[36835]: pgmap v7569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:33:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:24 smithi029 ceph-mon[32610]: pgmap v7569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:33:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:26 smithi077 ceph-mon[37153]: pgmap v7570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:33:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:26 smithi047 ceph-mon[36835]: pgmap v7570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:33:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:26 smithi029 ceph-mon[32610]: pgmap v7570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:33:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13429.672016 secs 2024-07-27T10:33:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:28 smithi077 ceph-mon[37153]: pgmap v7571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:33:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13429.672016 secs 2024-07-27T10:33:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:28 smithi047 ceph-mon[36835]: pgmap v7571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:33:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13429.672016 secs 2024-07-27T10:33:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:28 smithi029 ceph-mon[32610]: pgmap v7571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:33:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:30 smithi077 ceph-mon[37153]: pgmap v7572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:33:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:30 smithi047 ceph-mon[36835]: pgmap v7572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:33:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:30 smithi029 ceph-mon[32610]: pgmap v7572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:33:32.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:32 smithi077 ceph-mon[37153]: pgmap v7573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:33:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:32 smithi047 ceph-mon[36835]: pgmap v7573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:33:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:32 smithi029 ceph-mon[32610]: pgmap v7573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:33:33.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:33:33.689+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:33:33.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:33:33.689+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:33:33.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:33:33.689+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:33:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13434.672102 secs 2024-07-27T10:33:33.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13434.672102 secs 2024-07-27T10:33:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13434.672102 secs 2024-07-27T10:33:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:34 smithi077 ceph-mon[37153]: pgmap v7574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:34 smithi047 ceph-mon[36835]: pgmap v7574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:34 smithi029 ceph-mon[32610]: pgmap v7574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:36 smithi077 ceph-mon[37153]: pgmap v7575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:36 smithi047 ceph-mon[36835]: pgmap v7575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:36 smithi029 ceph-mon[32610]: pgmap v7575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13439.672186 secs 2024-07-27T10:33:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:38 smithi047 ceph-mon[36835]: pgmap v7576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:33:38.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13439.672186 secs 2024-07-27T10:33:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:38 smithi029 ceph-mon[32610]: pgmap v7576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:33:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13439.672186 secs 2024-07-27T10:33:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:38 smithi077 ceph-mon[37153]: pgmap v7576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:33:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:40 smithi047 ceph-mon[36835]: pgmap v7577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:33:40.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:40 smithi029 ceph-mon[32610]: pgmap v7577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:33:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:40 smithi077 ceph-mon[37153]: pgmap v7577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:33:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:42 smithi047 ceph-mon[36835]: pgmap v7578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:33:42.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:42 smithi029 ceph-mon[32610]: pgmap v7578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:33:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:42 smithi077 ceph-mon[37153]: pgmap v7578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:33:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13444.672254 secs 2024-07-27T10:33:43.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13444.672254 secs 2024-07-27T10:33:44.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13444.672254 secs 2024-07-27T10:33:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:44 smithi047 ceph-mon[36835]: pgmap v7579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:44.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:44 smithi029 ceph-mon[32610]: pgmap v7579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:44 smithi077 ceph-mon[37153]: pgmap v7579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:46 smithi047 ceph-mon[36835]: pgmap v7580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:46 smithi029 ceph-mon[32610]: pgmap v7580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:46 smithi077 ceph-mon[37153]: pgmap v7580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:48.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:33:48.704+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:33:48.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:33:48.704+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:33:48.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:33:48.704+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:33:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13449.672411 secs 2024-07-27T10:33:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:48 smithi047 ceph-mon[36835]: pgmap v7581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:33:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13449.672411 secs 2024-07-27T10:33:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:48 smithi029 ceph-mon[32610]: pgmap v7581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:33:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13449.672411 secs 2024-07-27T10:33:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:48 smithi077 ceph-mon[37153]: pgmap v7581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:33:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:50 smithi047 ceph-mon[36835]: pgmap v7582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:33:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:50 smithi029 ceph-mon[32610]: pgmap v7582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:33:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:50 smithi077 ceph-mon[37153]: pgmap v7582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:33:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:52 smithi047 ceph-mon[36835]: pgmap v7583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:33:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:52 smithi029 ceph-mon[32610]: pgmap v7583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:33:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:52 smithi077 ceph-mon[37153]: pgmap v7583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:33:53.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13454.672532 secs 2024-07-27T10:33:53.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13454.672532 secs 2024-07-27T10:33:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13454.672532 secs 2024-07-27T10:33:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:54 smithi047 ceph-mon[36835]: pgmap v7584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:54.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:54 smithi029 ceph-mon[32610]: pgmap v7584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:54 smithi077 ceph-mon[37153]: pgmap v7584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:33:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:56 smithi047 ceph-mon[36835]: pgmap v7585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:56 smithi029 ceph-mon[32610]: pgmap v7585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:56 smithi077 ceph-mon[37153]: pgmap v7585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:33:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13459.672649 secs 2024-07-27T10:33:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:33:58 smithi047 ceph-mon[36835]: pgmap v7586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:33:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13459.672649 secs 2024-07-27T10:33:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:33:58 smithi029 ceph-mon[32610]: pgmap v7586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:33:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13459.672649 secs 2024-07-27T10:33:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:33:58 smithi077 ceph-mon[37153]: pgmap v7586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:34:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:00 smithi047 ceph-mon[36835]: pgmap v7587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:34:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:00 smithi029 ceph-mon[32610]: pgmap v7587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:34:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:00 smithi077 ceph-mon[37153]: pgmap v7587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:34:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:02 smithi047 ceph-mon[36835]: pgmap v7588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:34:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:02 smithi029 ceph-mon[32610]: pgmap v7588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:34:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:02 smithi077 ceph-mon[37153]: pgmap v7588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:34:03.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:03.717+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:34:03.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:03.717+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:34:03.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:03.717+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:34:03.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13464.672747 secs 2024-07-27T10:34:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13464.672747 secs 2024-07-27T10:34:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13464.672747 secs 2024-07-27T10:34:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:04 smithi047 ceph-mon[36835]: pgmap v7589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:04 smithi029 ceph-mon[32610]: pgmap v7589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:04 smithi077 ceph-mon[37153]: pgmap v7589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:06 smithi047 ceph-mon[36835]: pgmap v7590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:06 smithi029 ceph-mon[32610]: pgmap v7590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:06 smithi077 ceph-mon[37153]: pgmap v7590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13469.672867 secs 2024-07-27T10:34:08.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:08 smithi047 ceph-mon[36835]: pgmap v7591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:34:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13469.672867 secs 2024-07-27T10:34:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:08 smithi029 ceph-mon[32610]: pgmap v7591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:34:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13469.672867 secs 2024-07-27T10:34:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:08 smithi077 ceph-mon[37153]: pgmap v7591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:34:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:10 smithi047 ceph-mon[36835]: pgmap v7592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:34:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:10 smithi029 ceph-mon[32610]: pgmap v7592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:34:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:10 smithi077 ceph-mon[37153]: pgmap v7592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T10:34:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:12 smithi047 ceph-mon[36835]: pgmap v7593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:34:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:12 smithi029 ceph-mon[32610]: pgmap v7593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:34:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:12 smithi077 ceph-mon[37153]: pgmap v7593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:34:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13474.673007 secs 2024-07-27T10:34:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13474.673007 secs 2024-07-27T10:34:14.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13474.673007 secs 2024-07-27T10:34:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:14 smithi047 ceph-mon[36835]: pgmap v7594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:14 smithi029 ceph-mon[32610]: pgmap v7594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:14 smithi077 ceph-mon[37153]: pgmap v7594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:15 smithi047 ceph-mon[36835]: pgmap v7595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:34:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:15 smithi029 ceph-mon[32610]: pgmap v7595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:34:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:15 smithi077 ceph-mon[37153]: pgmap v7595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:34:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:34:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:34:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:16 smithi047 ceph-mon[36835]: pgmap v7596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:34:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:16 smithi047 ceph-mon[36835]: pgmap v7597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:34:16.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:34:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:34:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:34:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:16 smithi029 ceph-mon[32610]: pgmap v7596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:34:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:16 smithi029 ceph-mon[32610]: pgmap v7597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:34:16.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:34:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:34:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:34:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:16 smithi077 ceph-mon[37153]: pgmap v7596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:34:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:16 smithi077 ceph-mon[37153]: pgmap v7597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T10:34:17.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:34:18.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:18.732+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:34:18.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:18.732+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:34:18.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:18.732+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:34:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13479.673120 secs 2024-07-27T10:34:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13479.673120 secs 2024-07-27T10:34:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13479.673120 secs 2024-07-27T10:34:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:19 smithi077 ceph-mon[37153]: pgmap v7598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:34:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:19 smithi047 ceph-mon[36835]: pgmap v7598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:34:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:19 smithi029 ceph-mon[32610]: pgmap v7598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:34:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:21 smithi077 ceph-mon[37153]: pgmap v7599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:34:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:21 smithi047 ceph-mon[36835]: pgmap v7599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:34:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:21 smithi029 ceph-mon[32610]: pgmap v7599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:34:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:23 smithi077 ceph-mon[37153]: pgmap v7600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:34:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13484.673241 secs 2024-07-27T10:34:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:23 smithi047 ceph-mon[36835]: pgmap v7600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:34:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13484.673241 secs 2024-07-27T10:34:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:23 smithi029 ceph-mon[32610]: pgmap v7600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:34:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13484.673241 secs 2024-07-27T10:34:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:25 smithi077 ceph-mon[37153]: pgmap v7601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:34:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:25 smithi047 ceph-mon[36835]: pgmap v7601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:34:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:25 smithi029 ceph-mon[32610]: pgmap v7601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:34:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:27 smithi077 ceph-mon[37153]: pgmap v7602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:34:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:27 smithi047 ceph-mon[36835]: pgmap v7602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:34:27.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:27 smithi029 ceph-mon[32610]: pgmap v7602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:34:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13489.673379 secs 2024-07-27T10:34:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13489.673379 secs 2024-07-27T10:34:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13489.673379 secs 2024-07-27T10:34:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:29 smithi047 ceph-mon[36835]: pgmap v7603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:34:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:29 smithi029 ceph-mon[32610]: pgmap v7603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:34:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:29 smithi077 ceph-mon[37153]: pgmap v7603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:34:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:31 smithi047 ceph-mon[36835]: pgmap v7604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:34:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:31 smithi029 ceph-mon[32610]: pgmap v7604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:34:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:31 smithi077 ceph-mon[37153]: pgmap v7604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:34:33.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:33.739+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:34:33.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:33.739+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:34:33.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:33.739+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:34:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:33 smithi047 ceph-mon[36835]: pgmap v7605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:34:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13494.673510 secs 2024-07-27T10:34:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:33 smithi029 ceph-mon[32610]: pgmap v7605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:34:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13494.673510 secs 2024-07-27T10:34:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:33 smithi077 ceph-mon[37153]: pgmap v7605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:34:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13494.673510 secs 2024-07-27T10:34:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:35 smithi047 ceph-mon[36835]: pgmap v7606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:35 smithi029 ceph-mon[32610]: pgmap v7606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:35 smithi077 ceph-mon[37153]: pgmap v7606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:37 smithi047 ceph-mon[36835]: pgmap v7607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:37 smithi029 ceph-mon[32610]: pgmap v7607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:37 smithi077 ceph-mon[37153]: pgmap v7607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13499.673567 secs 2024-07-27T10:34:38.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13499.673567 secs 2024-07-27T10:34:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13499.673567 secs 2024-07-27T10:34:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:39 smithi047 ceph-mon[36835]: pgmap v7608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:34:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:39 smithi029 ceph-mon[32610]: pgmap v7608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:34:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:39 smithi077 ceph-mon[37153]: pgmap v7608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:34:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:41 smithi047 ceph-mon[36835]: pgmap v7609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:41 smithi029 ceph-mon[32610]: pgmap v7609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:41 smithi077 ceph-mon[37153]: pgmap v7609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:43 smithi047 ceph-mon[36835]: pgmap v7610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:34:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13504.673690 secs 2024-07-27T10:34:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:43 smithi029 ceph-mon[32610]: pgmap v7610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:34:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13504.673690 secs 2024-07-27T10:34:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:43 smithi077 ceph-mon[37153]: pgmap v7610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:34:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13504.673690 secs 2024-07-27T10:34:45.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:45 smithi077 ceph-mon[37153]: pgmap v7611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:45 smithi047 ceph-mon[36835]: pgmap v7611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:45.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:45 smithi029 ceph-mon[32610]: pgmap v7611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:47 smithi047 ceph-mon[36835]: pgmap v7612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:47.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:47 smithi029 ceph-mon[32610]: pgmap v7612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:47 smithi077 ceph-mon[37153]: pgmap v7612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:48.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:48.747+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:34:48.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:48.747+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:34:48.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:48.747+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:34:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13509.673832 secs 2024-07-27T10:34:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13509.673832 secs 2024-07-27T10:34:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13509.673832 secs 2024-07-27T10:34:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:49 smithi047 ceph-mon[36835]: pgmap v7613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:34:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:49 smithi029 ceph-mon[32610]: pgmap v7613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:34:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:49 smithi077 ceph-mon[37153]: pgmap v7613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:34:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:51 smithi047 ceph-mon[36835]: pgmap v7614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:34:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:51 smithi029 ceph-mon[32610]: pgmap v7614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:34:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:51 smithi077 ceph-mon[37153]: pgmap v7614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:34:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:53 smithi047 ceph-mon[36835]: pgmap v7615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:34:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13514.673963 secs 2024-07-27T10:34:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:53 smithi029 ceph-mon[32610]: pgmap v7615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:34:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13514.673963 secs 2024-07-27T10:34:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:53 smithi077 ceph-mon[37153]: pgmap v7615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:34:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13514.673963 secs 2024-07-27T10:34:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:55 smithi047 ceph-mon[36835]: pgmap v7616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:55 smithi029 ceph-mon[32610]: pgmap v7616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:55 smithi077 ceph-mon[37153]: pgmap v7616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:34:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:57.666+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 msgr2=0x7eff3c0b5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T10:34:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:57.666+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 msgr2=0x7eff3c0b5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T10:34:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:57.666+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 0x7eff3c0b5580 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7eff40009ee0 tx=0x7eff4006c040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T10:34:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:57.666+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 0x7eff3c0b5580 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7eff40009ee0 tx=0x7eff4006c040 comp rx=0 tx=0).stop 2024-07-27T10:34:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:57.666+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 msgr2=0x7eff3c0b5580 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T10:34:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:57.666+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 msgr2=0x7eff3c0b5580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T10:34:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:57.666+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5170 0x7eff3c0b5580 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T10:34:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:57.666+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T10:34:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:57.666+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T10:34:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:57.667+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7eff400049e0 tx=0x7eff40038040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T10:34:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:57 smithi047 ceph-mon[36835]: pgmap v7617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:57 smithi029 ceph-mon[32610]: pgmap v7617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:57 smithi077 ceph-mon[37153]: pgmap v7617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:34:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13519.674089 secs 2024-07-27T10:34:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13519.674089 secs 2024-07-27T10:34:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13519.674089 secs 2024-07-27T10:34:59.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:59.687+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:34:59.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:59.687+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:34:59.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:59.688+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:34:59.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:34:59.688+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:34:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:34:59 smithi047 ceph-mon[36835]: pgmap v7618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:34:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:34:59 smithi029 ceph-mon[32610]: pgmap v7618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:35:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:34:59 smithi077 ceph-mon[37153]: pgmap v7618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:35:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:01 smithi047 ceph-mon[36835]: pgmap v7619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:01.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:01 smithi029 ceph-mon[32610]: pgmap v7619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:01 smithi077 ceph-mon[37153]: pgmap v7619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:35:03.754+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:35:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:35:03.754+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:35:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:35:03.754+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:35:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:03 smithi047 ceph-mon[36835]: pgmap v7620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:35:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13524.674190 secs 2024-07-27T10:35:03.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:03 smithi029 ceph-mon[32610]: pgmap v7620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:35:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13524.674190 secs 2024-07-27T10:35:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:03 smithi077 ceph-mon[37153]: pgmap v7620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:35:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13524.674190 secs 2024-07-27T10:35:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:05 smithi047 ceph-mon[36835]: pgmap v7621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:05.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:05 smithi029 ceph-mon[32610]: pgmap v7621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:05 smithi077 ceph-mon[37153]: pgmap v7621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:07 smithi047 ceph-mon[36835]: pgmap v7622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:07 smithi029 ceph-mon[32610]: pgmap v7622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:07 smithi077 ceph-mon[37153]: pgmap v7622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13529.674291 secs 2024-07-27T10:35:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13529.674291 secs 2024-07-27T10:35:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13529.674291 secs 2024-07-27T10:35:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:09 smithi047 ceph-mon[36835]: pgmap v7623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:35:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:09 smithi029 ceph-mon[32610]: pgmap v7623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:35:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:09 smithi077 ceph-mon[37153]: pgmap v7623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:35:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:10 smithi029 ceph-mon[32610]: pgmap v7624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:10 smithi077 ceph-mon[37153]: pgmap v7624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:10 smithi047 ceph-mon[36835]: pgmap v7624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:12 smithi077 ceph-mon[37153]: pgmap v7625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:35:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13534.674435 secs 2024-07-27T10:35:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:12 smithi047 ceph-mon[36835]: pgmap v7625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:35:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13534.674435 secs 2024-07-27T10:35:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:12 smithi029 ceph-mon[32610]: pgmap v7625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:35:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13534.674435 secs 2024-07-27T10:35:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:14 smithi077 ceph-mon[37153]: pgmap v7626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:14 smithi047 ceph-mon[36835]: pgmap v7626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:14 smithi029 ceph-mon[32610]: pgmap v7626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: pgmap v7627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:35:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:35:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:35:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:35:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:35:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:35:17.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:35:17.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:35:17.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:35:17.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: pgmap v7628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:35:17.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: pgmap v7629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:35:17.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:35:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: pgmap v7627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:35:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:35:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:35:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:35:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:35:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:35:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:35:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:35:17.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:35:17.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: pgmap v7628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:35:17.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: pgmap v7629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:35:17.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:35:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: pgmap v7627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:35:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:35:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:35:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:35:17.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:35:17.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:35:17.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:35:17.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:35:17.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:35:17.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: pgmap v7628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:35:17.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: pgmap v7629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:35:17.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:35:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13539.674580 secs 2024-07-27T10:35:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13539.674580 secs 2024-07-27T10:35:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13539.674580 secs 2024-07-27T10:35:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:35:18.770+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:35:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:35:18.770+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:35:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:35:18.770+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:35:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:19 smithi029 ceph-mon[32610]: pgmap v7630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:35:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:19 smithi077 ceph-mon[37153]: pgmap v7630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:35:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:19 smithi047 ceph-mon[36835]: pgmap v7630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:35:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:21 smithi077 ceph-mon[37153]: pgmap v7631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:35:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:21 smithi047 ceph-mon[36835]: pgmap v7631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:35:21.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:21 smithi029 ceph-mon[32610]: pgmap v7631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:35:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13544.674722 secs 2024-07-27T10:35:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:23 smithi077 ceph-mon[37153]: pgmap v7632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:35:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13544.674722 secs 2024-07-27T10:35:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:23 smithi047 ceph-mon[36835]: pgmap v7632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:35:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13544.674722 secs 2024-07-27T10:35:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:23 smithi029 ceph-mon[32610]: pgmap v7632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:35:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:25 smithi077 ceph-mon[37153]: pgmap v7633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:35:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:25 smithi047 ceph-mon[36835]: pgmap v7633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:35:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:25 smithi029 ceph-mon[32610]: pgmap v7633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:35:27.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:27 smithi077 ceph-mon[37153]: pgmap v7634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:35:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:27 smithi047 ceph-mon[36835]: pgmap v7634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:35:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:27 smithi029 ceph-mon[32610]: pgmap v7634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:35:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13549.674854 secs 2024-07-27T10:35:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13549.674854 secs 2024-07-27T10:35:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13549.674854 secs 2024-07-27T10:35:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:29 smithi077 ceph-mon[37153]: pgmap v7635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:29 smithi047 ceph-mon[36835]: pgmap v7635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:29 smithi029 ceph-mon[32610]: pgmap v7635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:31 smithi077 ceph-mon[37153]: pgmap v7636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:31 smithi047 ceph-mon[36835]: pgmap v7636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:31 smithi029 ceph-mon[32610]: pgmap v7636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:33.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:35:33.782+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:35:33.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:35:33.782+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:35:33.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:35:33.782+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:35:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13554.674980 secs 2024-07-27T10:35:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:33 smithi077 ceph-mon[37153]: pgmap v7637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:35:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13554.674980 secs 2024-07-27T10:35:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:33 smithi047 ceph-mon[36835]: pgmap v7637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:35:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13554.674980 secs 2024-07-27T10:35:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:33 smithi029 ceph-mon[32610]: pgmap v7637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:35:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:35 smithi077 ceph-mon[37153]: pgmap v7638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:35:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:35 smithi047 ceph-mon[36835]: pgmap v7638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:35:35.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:35 smithi029 ceph-mon[32610]: pgmap v7638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:35:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:37 smithi077 ceph-mon[37153]: pgmap v7639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:35:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:37 smithi047 ceph-mon[36835]: pgmap v7639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:35:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:37 smithi029 ceph-mon[32610]: pgmap v7639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:35:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13559.675093 secs 2024-07-27T10:35:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13559.675093 secs 2024-07-27T10:35:38.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13559.675093 secs 2024-07-27T10:35:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:39 smithi077 ceph-mon[37153]: pgmap v7640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:35:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:39 smithi047 ceph-mon[36835]: pgmap v7640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:35:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:39 smithi029 ceph-mon[32610]: pgmap v7640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:35:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:41 smithi077 ceph-mon[37153]: pgmap v7641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:35:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:41 smithi047 ceph-mon[36835]: pgmap v7641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:35:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:41 smithi029 ceph-mon[32610]: pgmap v7641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:35:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13564.675218 secs 2024-07-27T10:35:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:43 smithi077 ceph-mon[37153]: pgmap v7642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:35:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13564.675218 secs 2024-07-27T10:35:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:43 smithi047 ceph-mon[36835]: pgmap v7642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:35:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13564.675218 secs 2024-07-27T10:35:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:43 smithi029 ceph-mon[32610]: pgmap v7642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:35:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:45 smithi077 ceph-mon[37153]: pgmap v7643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:35:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:45 smithi047 ceph-mon[36835]: pgmap v7643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:35:45.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:45 smithi029 ceph-mon[32610]: pgmap v7643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:35:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:47 smithi077 ceph-mon[37153]: pgmap v7644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:35:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:47 smithi047 ceph-mon[36835]: pgmap v7644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:35:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:47 smithi029 ceph-mon[32610]: pgmap v7644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:35:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:35:48.782+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:35:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:35:48.782+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:35:48.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:35:48.782+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:35:48.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13569.675344 secs 2024-07-27T10:35:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13569.675344 secs 2024-07-27T10:35:48.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13569.675344 secs 2024-07-27T10:35:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:49 smithi077 ceph-mon[37153]: pgmap v7645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:49 smithi047 ceph-mon[36835]: pgmap v7645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:49 smithi029 ceph-mon[32610]: pgmap v7645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:51 smithi077 ceph-mon[37153]: pgmap v7646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:51 smithi047 ceph-mon[36835]: pgmap v7646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:51 smithi029 ceph-mon[32610]: pgmap v7646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:35:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13574.675440 secs 2024-07-27T10:35:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:53 smithi077 ceph-mon[37153]: pgmap v7647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:35:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13574.675440 secs 2024-07-27T10:35:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:53 smithi047 ceph-mon[36835]: pgmap v7647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:35:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13574.675440 secs 2024-07-27T10:35:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:53 smithi029 ceph-mon[32610]: pgmap v7647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:35:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:55 smithi077 ceph-mon[37153]: pgmap v7648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:35:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:55 smithi047 ceph-mon[36835]: pgmap v7648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:35:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:55 smithi029 ceph-mon[32610]: pgmap v7648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:35:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:57 smithi077 ceph-mon[37153]: pgmap v7649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:35:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:57 smithi047 ceph-mon[36835]: pgmap v7649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:35:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:57 smithi029 ceph-mon[32610]: pgmap v7649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:35:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13579.675503 secs 2024-07-27T10:35:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13579.675503 secs 2024-07-27T10:35:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13579.675503 secs 2024-07-27T10:35:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:35:59 smithi047 ceph-mon[36835]: pgmap v7650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:35:59.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:35:59 smithi029 ceph-mon[32610]: pgmap v7650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:36:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:35:59 smithi077 ceph-mon[37153]: pgmap v7650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:36:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:01 smithi047 ceph-mon[36835]: pgmap v7651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:01.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:01 smithi029 ceph-mon[32610]: pgmap v7651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:01 smithi077 ceph-mon[37153]: pgmap v7651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:36:03.791+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:36:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:36:03.791+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:36:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:36:03.791+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:36:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13584.675590 secs 2024-07-27T10:36:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:03 smithi047 ceph-mon[36835]: pgmap v7652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:36:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13584.675590 secs 2024-07-27T10:36:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:03 smithi029 ceph-mon[32610]: pgmap v7652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:36:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13584.675590 secs 2024-07-27T10:36:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:03 smithi077 ceph-mon[37153]: pgmap v7652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:36:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:05 smithi047 ceph-mon[36835]: pgmap v7653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:36:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:05 smithi029 ceph-mon[32610]: pgmap v7653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:36:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:05 smithi077 ceph-mon[37153]: pgmap v7653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:36:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:07 smithi047 ceph-mon[36835]: pgmap v7654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:36:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:07 smithi029 ceph-mon[32610]: pgmap v7654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:36:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:07 smithi077 ceph-mon[37153]: pgmap v7654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:36:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13589.675733 secs 2024-07-27T10:36:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13589.675733 secs 2024-07-27T10:36:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13589.675733 secs 2024-07-27T10:36:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:09 smithi047 ceph-mon[36835]: pgmap v7655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:36:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:09 smithi029 ceph-mon[32610]: pgmap v7655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:36:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:09 smithi077 ceph-mon[37153]: pgmap v7655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:36:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:11 smithi047 ceph-mon[36835]: pgmap v7656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:11.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:11 smithi029 ceph-mon[32610]: pgmap v7656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:11 smithi077 ceph-mon[37153]: pgmap v7656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13594.675855 secs 2024-07-27T10:36:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:13 smithi047 ceph-mon[36835]: pgmap v7657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:36:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13594.675855 secs 2024-07-27T10:36:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:13 smithi029 ceph-mon[32610]: pgmap v7657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:36:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13594.675855 secs 2024-07-27T10:36:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:13 smithi077 ceph-mon[37153]: pgmap v7657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:36:15.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:15 smithi077 ceph-mon[37153]: pgmap v7658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:36:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:15 smithi047 ceph-mon[36835]: pgmap v7658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:36:15.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:15 smithi029 ceph-mon[32610]: pgmap v7658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:36:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:16 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:36:16.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:16 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:36:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:16 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:36:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:17 smithi047 ceph-mon[36835]: pgmap v7659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:36:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:36:17.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:36:17.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:36:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:17 smithi029 ceph-mon[32610]: pgmap v7659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:36:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:17.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:36:17.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:36:17.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:36:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:17 smithi077 ceph-mon[37153]: pgmap v7659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:36:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:18.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:18.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:18.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:36:18.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:36:18.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:36:18.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:36:18.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:36:18.807+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:36:18.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:36:18.807+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:36:18.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:36:18.807+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:36:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13599.675932 secs 2024-07-27T10:36:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:18 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:36:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:18 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:36:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:18 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:36:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:18 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:36:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:18 smithi047 ceph-mon[36835]: pgmap v7660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:36:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:18 smithi047 ceph-mon[36835]: pgmap v7661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:36:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13599.675932 secs 2024-07-27T10:36:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:18 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:36:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:18 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:36:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:18 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:36:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:18 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:36:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:18 smithi029 ceph-mon[32610]: pgmap v7660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:36:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:18 smithi029 ceph-mon[32610]: pgmap v7661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:36:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13599.675932 secs 2024-07-27T10:36:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:18 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:36:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:18 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:36:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:18 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:36:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:18 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:36:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:18 smithi077 ceph-mon[37153]: pgmap v7660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:36:19.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:18 smithi077 ceph-mon[37153]: pgmap v7661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:36:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:20 smithi047 ceph-mon[36835]: pgmap v7662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:36:20.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:20 smithi029 ceph-mon[32610]: pgmap v7662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:36:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:20 smithi077 ceph-mon[37153]: pgmap v7662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:36:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:22 smithi047 ceph-mon[36835]: pgmap v7663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:36:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:22 smithi029 ceph-mon[32610]: pgmap v7663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:36:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:22 smithi077 ceph-mon[37153]: pgmap v7663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:36:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13604.676061 secs 2024-07-27T10:36:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13604.676061 secs 2024-07-27T10:36:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13604.676061 secs 2024-07-27T10:36:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:24 smithi047 ceph-mon[36835]: pgmap v7664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:36:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:24 smithi029 ceph-mon[32610]: pgmap v7664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:36:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:24 smithi077 ceph-mon[37153]: pgmap v7664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:36:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:26 smithi047 ceph-mon[36835]: pgmap v7665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:36:26.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:26 smithi029 ceph-mon[32610]: pgmap v7665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:36:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:26 smithi077 ceph-mon[37153]: pgmap v7665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:36:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13609.676179 secs 2024-07-27T10:36:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:28 smithi047 ceph-mon[36835]: pgmap v7666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:36:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13609.676179 secs 2024-07-27T10:36:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:28 smithi029 ceph-mon[32610]: pgmap v7666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:36:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13609.676179 secs 2024-07-27T10:36:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:28 smithi077 ceph-mon[37153]: pgmap v7666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:36:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:30 smithi047 ceph-mon[36835]: pgmap v7667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:36:30.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:30 smithi029 ceph-mon[32610]: pgmap v7667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:36:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:30 smithi077 ceph-mon[37153]: pgmap v7667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:36:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:32 smithi047 ceph-mon[36835]: pgmap v7668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:36:32.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:32 smithi029 ceph-mon[32610]: pgmap v7668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:36:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:32 smithi077 ceph-mon[37153]: pgmap v7668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:36:33.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:36:33.818+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:36:33.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:36:33.818+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:36:33.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:36:33.818+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:36:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13614.676333 secs 2024-07-27T10:36:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13614.676333 secs 2024-07-27T10:36:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13614.676333 secs 2024-07-27T10:36:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:34 smithi047 ceph-mon[36835]: pgmap v7669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:36:34.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:34 smithi029 ceph-mon[32610]: pgmap v7669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:36:35.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:34 smithi077 ceph-mon[37153]: pgmap v7669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:36:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:36 smithi029 ceph-mon[32610]: pgmap v7670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:36:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:36 smithi077 ceph-mon[37153]: pgmap v7670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:36:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:36 smithi047 ceph-mon[36835]: pgmap v7670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:36:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13619.676466 secs 2024-07-27T10:36:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:38 smithi029 ceph-mon[32610]: pgmap v7671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:36:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13619.676466 secs 2024-07-27T10:36:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:38 smithi077 ceph-mon[37153]: pgmap v7671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:36:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13619.676466 secs 2024-07-27T10:36:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:38 smithi047 ceph-mon[36835]: pgmap v7671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:36:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:40 smithi029 ceph-mon[32610]: pgmap v7672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:36:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:40 smithi077 ceph-mon[37153]: pgmap v7672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:36:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:40 smithi047 ceph-mon[36835]: pgmap v7672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:36:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:41 smithi077 ceph-mon[37153]: pgmap v7673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:41 smithi047 ceph-mon[36835]: pgmap v7673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:41 smithi029 ceph-mon[32610]: pgmap v7673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13624.676597 secs 2024-07-27T10:36:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13624.676597 secs 2024-07-27T10:36:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13624.676597 secs 2024-07-27T10:36:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:43 smithi047 ceph-mon[36835]: pgmap v7674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:36:44.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:43 smithi029 ceph-mon[32610]: pgmap v7674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:36:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:43 smithi077 ceph-mon[37153]: pgmap v7674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:36:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:46 smithi047 ceph-mon[36835]: pgmap v7675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:46.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:46 smithi029 ceph-mon[32610]: pgmap v7675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:46 smithi077 ceph-mon[37153]: pgmap v7675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13629.676727 secs 2024-07-27T10:36:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:48 smithi047 ceph-mon[36835]: pgmap v7676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:36:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13629.676727 secs 2024-07-27T10:36:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:48 smithi029 ceph-mon[32610]: pgmap v7676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:36:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13629.676727 secs 2024-07-27T10:36:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:48 smithi077 ceph-mon[37153]: pgmap v7676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:36:48.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:36:48.832+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:36:48.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:36:48.832+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:36:48.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:36:48.832+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:36:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:50 smithi047 ceph-mon[36835]: pgmap v7677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:36:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:50 smithi029 ceph-mon[32610]: pgmap v7677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:36:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:50 smithi077 ceph-mon[37153]: pgmap v7677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:36:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:52 smithi047 ceph-mon[36835]: pgmap v7678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:52.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:52 smithi029 ceph-mon[32610]: pgmap v7678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:52 smithi077 ceph-mon[37153]: pgmap v7678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13634.676859 secs 2024-07-27T10:36:53.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13634.676859 secs 2024-07-27T10:36:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13634.676859 secs 2024-07-27T10:36:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:54 smithi047 ceph-mon[36835]: pgmap v7679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:36:54.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:54 smithi029 ceph-mon[32610]: pgmap v7679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:36:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:54 smithi077 ceph-mon[37153]: pgmap v7679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:36:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:56 smithi047 ceph-mon[36835]: pgmap v7680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:56.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:56 smithi029 ceph-mon[32610]: pgmap v7680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:56 smithi077 ceph-mon[37153]: pgmap v7680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:36:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13639.676925 secs 2024-07-27T10:36:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:36:58 smithi047 ceph-mon[36835]: pgmap v7681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:36:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13639.676925 secs 2024-07-27T10:36:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:36:58 smithi029 ceph-mon[32610]: pgmap v7681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:36:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13639.676925 secs 2024-07-27T10:36:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:36:58 smithi077 ceph-mon[37153]: pgmap v7681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:37:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:00 smithi047 ceph-mon[36835]: pgmap v7682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:37:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:00 smithi029 ceph-mon[32610]: pgmap v7682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:37:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:00 smithi077 ceph-mon[37153]: pgmap v7682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:37:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:02 smithi047 ceph-mon[36835]: pgmap v7683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:02.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:02 smithi029 ceph-mon[32610]: pgmap v7683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:02 smithi077 ceph-mon[37153]: pgmap v7683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13644.677048 secs 2024-07-27T10:37:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13644.677048 secs 2024-07-27T10:37:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13644.677048 secs 2024-07-27T10:37:03.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:03.845+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:37:03.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:03.845+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:37:03.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:03.845+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:37:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:04 smithi047 ceph-mon[36835]: pgmap v7684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:37:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:04 smithi029 ceph-mon[32610]: pgmap v7684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:37:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:04 smithi077 ceph-mon[37153]: pgmap v7684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:37:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:06 smithi047 ceph-mon[36835]: pgmap v7685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:37:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:06 smithi029 ceph-mon[32610]: pgmap v7685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:37:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:06 smithi077 ceph-mon[37153]: pgmap v7685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:37:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13649.677163 secs 2024-07-27T10:37:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:08 smithi047 ceph-mon[36835]: pgmap v7686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:37:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13649.677163 secs 2024-07-27T10:37:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:08 smithi029 ceph-mon[32610]: pgmap v7686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:37:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13649.677163 secs 2024-07-27T10:37:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:08 smithi077 ceph-mon[37153]: pgmap v7686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:37:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:10 smithi047 ceph-mon[36835]: pgmap v7687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:37:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:10 smithi029 ceph-mon[32610]: pgmap v7687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:37:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:10 smithi077 ceph-mon[37153]: pgmap v7687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:37:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:12 smithi047 ceph-mon[36835]: pgmap v7688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:37:12.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:12 smithi029 ceph-mon[32610]: pgmap v7688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:37:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:12 smithi077 ceph-mon[37153]: pgmap v7688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:37:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13654.677307 secs 2024-07-27T10:37:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13654.677307 secs 2024-07-27T10:37:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13654.677307 secs 2024-07-27T10:37:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:14 smithi047 ceph-mon[36835]: pgmap v7689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:37:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:14 smithi029 ceph-mon[32610]: pgmap v7689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:37:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:14 smithi077 ceph-mon[37153]: pgmap v7689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:37:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:16 smithi047 ceph-mon[36835]: pgmap v7690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:16 smithi029 ceph-mon[32610]: pgmap v7690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:16 smithi077 ceph-mon[37153]: pgmap v7690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:37:17.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:37:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:37:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13659.677451 secs 2024-07-27T10:37:18.450 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:18 smithi047 ceph-mon[36835]: pgmap v7691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:37:18.450 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:37:18.450 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:37:18.450 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:18 smithi047 ceph-mon[36835]: pgmap v7692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T10:37:18.450 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:18 smithi047 ceph-mon[36835]: pgmap v7693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:37:18.450 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:37:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13659.677451 secs 2024-07-27T10:37:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:18 smithi029 ceph-mon[32610]: pgmap v7691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:37:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:37:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:37:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:18 smithi029 ceph-mon[32610]: pgmap v7692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T10:37:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:18 smithi029 ceph-mon[32610]: pgmap v7693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:37:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:37:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13659.677451 secs 2024-07-27T10:37:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:18 smithi077 ceph-mon[37153]: pgmap v7691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:37:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:37:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:37:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:18 smithi077 ceph-mon[37153]: pgmap v7692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T10:37:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:18 smithi077 ceph-mon[37153]: pgmap v7693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:37:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:37:18.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:18.853+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:37:18.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:18.853+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:37:18.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:18.853+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:37:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:20 smithi047 ceph-mon[36835]: pgmap v7694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:37:20.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:20 smithi029 ceph-mon[32610]: pgmap v7694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:37:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:20 smithi077 ceph-mon[37153]: pgmap v7694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:37:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:22 smithi047 ceph-mon[36835]: pgmap v7695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:37:22.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:22 smithi029 ceph-mon[32610]: pgmap v7695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:37:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:22 smithi077 ceph-mon[37153]: pgmap v7695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:37:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13664.677584 secs 2024-07-27T10:37:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13664.677584 secs 2024-07-27T10:37:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13664.677584 secs 2024-07-27T10:37:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:24 smithi047 ceph-mon[36835]: pgmap v7696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:37:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:24 smithi029 ceph-mon[32610]: pgmap v7696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:37:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:24 smithi077 ceph-mon[37153]: pgmap v7696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:37:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:26 smithi047 ceph-mon[36835]: pgmap v7697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:37:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:26 smithi029 ceph-mon[32610]: pgmap v7697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:37:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:26 smithi077 ceph-mon[37153]: pgmap v7697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:37:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13669.677712 secs 2024-07-27T10:37:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:28 smithi047 ceph-mon[36835]: pgmap v7698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:37:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13669.677712 secs 2024-07-27T10:37:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:28 smithi029 ceph-mon[32610]: pgmap v7698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:37:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13669.677712 secs 2024-07-27T10:37:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:28 smithi077 ceph-mon[37153]: pgmap v7698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:37:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:30 smithi047 ceph-mon[36835]: pgmap v7699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:37:30.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:30 smithi029 ceph-mon[32610]: pgmap v7699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:37:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:30 smithi077 ceph-mon[37153]: pgmap v7699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:37:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:32 smithi047 ceph-mon[36835]: pgmap v7700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:37:32.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:32 smithi029 ceph-mon[32610]: pgmap v7700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:37:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:32 smithi077 ceph-mon[37153]: pgmap v7700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:37:33.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:33.868+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:37:33.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:33.868+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:37:33.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:33.868+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:37:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13674.677812 secs 2024-07-27T10:37:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13674.677812 secs 2024-07-27T10:37:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13674.677812 secs 2024-07-27T10:37:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:34 smithi047 ceph-mon[36835]: pgmap v7701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:37:34.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:34 smithi029 ceph-mon[32610]: pgmap v7701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:37:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:34 smithi077 ceph-mon[37153]: pgmap v7701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:37:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:36 smithi047 ceph-mon[36835]: pgmap v7702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:36.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:36 smithi029 ceph-mon[32610]: pgmap v7702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:36 smithi077 ceph-mon[37153]: pgmap v7702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13679.677942 secs 2024-07-27T10:37:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:38 smithi047 ceph-mon[36835]: pgmap v7703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:37:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13679.677942 secs 2024-07-27T10:37:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:38 smithi029 ceph-mon[32610]: pgmap v7703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:37:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13679.677942 secs 2024-07-27T10:37:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:38 smithi077 ceph-mon[37153]: pgmap v7703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:37:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:40 smithi029 ceph-mon[32610]: pgmap v7704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:37:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:40 smithi077 ceph-mon[37153]: pgmap v7704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:37:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:40 smithi047 ceph-mon[36835]: pgmap v7704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:37:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:42 smithi029 ceph-mon[32610]: pgmap v7705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:37:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:42 smithi077 ceph-mon[37153]: pgmap v7705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:37:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:42 smithi047 ceph-mon[36835]: pgmap v7705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:37:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13684.678077 secs 2024-07-27T10:37:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13684.678077 secs 2024-07-27T10:37:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13684.678077 secs 2024-07-27T10:37:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:44 smithi029 ceph-mon[32610]: pgmap v7706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:37:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:44 smithi077 ceph-mon[37153]: pgmap v7706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:37:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:44 smithi047 ceph-mon[36835]: pgmap v7706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:37:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:46 smithi029 ceph-mon[32610]: pgmap v7707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:46 smithi077 ceph-mon[37153]: pgmap v7707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:46 smithi047 ceph-mon[36835]: pgmap v7707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:48.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:48.883+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:37:48.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:48.883+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:37:48.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:48.883+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:37:48.998 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13689.678199 secs 2024-07-27T10:37:48.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:48 smithi047 ceph-mon[36835]: pgmap v7708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:37:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13689.678199 secs 2024-07-27T10:37:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:48 smithi077 ceph-mon[37153]: pgmap v7708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:37:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13689.678199 secs 2024-07-27T10:37:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:48 smithi029 ceph-mon[32610]: pgmap v7708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:37:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:50 smithi077 ceph-mon[37153]: pgmap v7709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:37:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:50 smithi047 ceph-mon[36835]: pgmap v7709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:37:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:50 smithi029 ceph-mon[32610]: pgmap v7709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:37:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:51 smithi077 ceph-mon[37153]: pgmap v7710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:37:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:51 smithi047 ceph-mon[36835]: pgmap v7710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:37:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:51 smithi029 ceph-mon[32610]: pgmap v7710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:37:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13694.678321 secs 2024-07-27T10:37:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13694.678321 secs 2024-07-27T10:37:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13694.678321 secs 2024-07-27T10:37:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:53 smithi047 ceph-mon[36835]: pgmap v7711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:37:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:53 smithi029 ceph-mon[32610]: pgmap v7711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:37:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:53 smithi077 ceph-mon[37153]: pgmap v7711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:37:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:56 smithi047 ceph-mon[36835]: pgmap v7712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:56 smithi029 ceph-mon[32610]: pgmap v7712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:56 smithi077 ceph-mon[37153]: pgmap v7712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:37:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13699.678430 secs 2024-07-27T10:37:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:37:58 smithi047 ceph-mon[36835]: pgmap v7713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:37:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13699.678430 secs 2024-07-27T10:37:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:37:58 smithi029 ceph-mon[32610]: pgmap v7713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:37:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13699.678430 secs 2024-07-27T10:37:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:37:58 smithi077 ceph-mon[37153]: pgmap v7713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:37:59.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:59.715+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:37:59.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:59.715+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:37:59.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:59.716+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:37:59.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:37:59.716+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:38:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:00 smithi047 ceph-mon[36835]: pgmap v7714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:38:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:00 smithi029 ceph-mon[32610]: pgmap v7714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:38:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:00 smithi077 ceph-mon[37153]: pgmap v7714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:38:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:02 smithi047 ceph-mon[36835]: pgmap v7715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:02 smithi029 ceph-mon[32610]: pgmap v7715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:02 smithi077 ceph-mon[37153]: pgmap v7715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:03.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:38:03.893+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:38:03.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:38:03.893+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:38:03.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:38:03.893+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:38:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13704.678551 secs 2024-07-27T10:38:03.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13704.678551 secs 2024-07-27T10:38:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13704.678551 secs 2024-07-27T10:38:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:04 smithi047 ceph-mon[36835]: pgmap v7716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:38:04.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:04 smithi029 ceph-mon[32610]: pgmap v7716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:38:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:04 smithi077 ceph-mon[37153]: pgmap v7716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:38:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:06 smithi047 ceph-mon[36835]: pgmap v7717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:38:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:06 smithi029 ceph-mon[32610]: pgmap v7717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:38:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:06 smithi077 ceph-mon[37153]: pgmap v7717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:38:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13709.678668 secs 2024-07-27T10:38:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:08 smithi047 ceph-mon[36835]: pgmap v7718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:38:08.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13709.678668 secs 2024-07-27T10:38:08.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:08 smithi029 ceph-mon[32610]: pgmap v7718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:38:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13709.678668 secs 2024-07-27T10:38:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:08 smithi077 ceph-mon[37153]: pgmap v7718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:38:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:10 smithi047 ceph-mon[36835]: pgmap v7719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:38:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:10 smithi029 ceph-mon[32610]: pgmap v7719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:38:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:10 smithi077 ceph-mon[37153]: pgmap v7719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:38:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:12 smithi029 ceph-mon[32610]: pgmap v7720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:12 smithi077 ceph-mon[37153]: pgmap v7720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:12 smithi047 ceph-mon[36835]: pgmap v7720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13714.678809 secs 2024-07-27T10:38:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13714.678809 secs 2024-07-27T10:38:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13714.678809 secs 2024-07-27T10:38:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:14 smithi029 ceph-mon[32610]: pgmap v7721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:38:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:14 smithi077 ceph-mon[37153]: pgmap v7721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:38:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:14 smithi047 ceph-mon[36835]: pgmap v7721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:38:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:16 smithi029 ceph-mon[32610]: pgmap v7722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:38:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:16 smithi077 ceph-mon[37153]: pgmap v7722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:38:17.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:16 smithi047 ceph-mon[36835]: pgmap v7722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:38:18.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:17 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:38:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:17 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:38:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:17 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:38:18.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:38:18.909+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:38:18.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:38:18.909+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:38:18.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:38:18.909+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:38:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13719.678929 secs 2024-07-27T10:38:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:18 smithi077 ceph-mon[37153]: pgmap v7723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:38:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:38:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:38:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:38:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13719.678929 secs 2024-07-27T10:38:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:18 smithi047 ceph-mon[36835]: pgmap v7723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:38:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:38:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:38:19.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:38:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13719.678929 secs 2024-07-27T10:38:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:18 smithi029 ceph-mon[32610]: pgmap v7723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:38:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:38:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:38:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:38:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:19 smithi077 ceph-mon[37153]: pgmap v7724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:38:20.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:19 smithi077 ceph-mon[37153]: pgmap v7725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:38:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:19 smithi047 ceph-mon[36835]: pgmap v7724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:38:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:19 smithi047 ceph-mon[36835]: pgmap v7725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:38:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:19 smithi029 ceph-mon[32610]: pgmap v7724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:38:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:19 smithi029 ceph-mon[32610]: pgmap v7725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:38:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:20 smithi077 ceph-mon[37153]: pgmap v7726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:38:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:20 smithi047 ceph-mon[36835]: pgmap v7726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:38:21.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:20 smithi029 ceph-mon[32610]: pgmap v7726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:38:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13724.679048 secs 2024-07-27T10:38:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:23 smithi077 ceph-mon[37153]: pgmap v7727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:38:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13724.679048 secs 2024-07-27T10:38:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:23 smithi047 ceph-mon[36835]: pgmap v7727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:38:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13724.679048 secs 2024-07-27T10:38:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:23 smithi029 ceph-mon[32610]: pgmap v7727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:38:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:25 smithi077 ceph-mon[37153]: pgmap v7728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:38:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:25 smithi047 ceph-mon[36835]: pgmap v7728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:38:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:25 smithi029 ceph-mon[32610]: pgmap v7728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:38:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:27 smithi077 ceph-mon[37153]: pgmap v7729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:38:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:27 smithi047 ceph-mon[36835]: pgmap v7729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:38:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:27 smithi029 ceph-mon[32610]: pgmap v7729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:38:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13729.679149 secs 2024-07-27T10:38:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13729.679149 secs 2024-07-27T10:38:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13729.679149 secs 2024-07-27T10:38:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:29 smithi077 ceph-mon[37153]: pgmap v7730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:38:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:29 smithi047 ceph-mon[36835]: pgmap v7730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:38:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:29 smithi029 ceph-mon[32610]: pgmap v7730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:38:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:31 smithi077 ceph-mon[37153]: pgmap v7731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:38:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:31 smithi047 ceph-mon[36835]: pgmap v7731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:38:31.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:31 smithi029 ceph-mon[32610]: pgmap v7731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:38:33.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13734.679267 secs 2024-07-27T10:38:33.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:33 smithi029 ceph-mon[32610]: pgmap v7732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13734.679267 secs 2024-07-27T10:38:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:33 smithi077 ceph-mon[37153]: pgmap v7732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13734.679267 secs 2024-07-27T10:38:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:33 smithi047 ceph-mon[36835]: pgmap v7732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:38:33.925+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:38:33.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:38:33.925+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:38:33.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:38:33.925+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:38:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:35 smithi077 ceph-mon[37153]: pgmap v7733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:38:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:35 smithi047 ceph-mon[36835]: pgmap v7733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:38:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:35 smithi029 ceph-mon[32610]: pgmap v7733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:38:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:37 smithi077 ceph-mon[37153]: pgmap v7734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:37 smithi047 ceph-mon[36835]: pgmap v7734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:37 smithi029 ceph-mon[32610]: pgmap v7734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13739.679412 secs 2024-07-27T10:38:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13739.679412 secs 2024-07-27T10:38:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13739.679412 secs 2024-07-27T10:38:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:39 smithi077 ceph-mon[37153]: pgmap v7735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:38:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:39 smithi047 ceph-mon[36835]: pgmap v7735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:38:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:39 smithi029 ceph-mon[32610]: pgmap v7735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:38:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:41 smithi077 ceph-mon[37153]: pgmap v7736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:38:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:41 smithi047 ceph-mon[36835]: pgmap v7736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:38:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:41 smithi029 ceph-mon[32610]: pgmap v7736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:38:43.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13744.679546 secs 2024-07-27T10:38:43.515 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:43 smithi029 ceph-mon[32610]: pgmap v7737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13744.679546 secs 2024-07-27T10:38:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:43 smithi077 ceph-mon[37153]: pgmap v7737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13744.679546 secs 2024-07-27T10:38:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:43 smithi047 ceph-mon[36835]: pgmap v7737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:45 smithi077 ceph-mon[37153]: pgmap v7738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:38:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:45 smithi047 ceph-mon[36835]: pgmap v7738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:38:45.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:45 smithi029 ceph-mon[32610]: pgmap v7738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:38:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:47 smithi077 ceph-mon[37153]: pgmap v7739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:47 smithi047 ceph-mon[36835]: pgmap v7739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:47 smithi029 ceph-mon[32610]: pgmap v7739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:48.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13749.679677 secs 2024-07-27T10:38:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13749.679677 secs 2024-07-27T10:38:48.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13749.679677 secs 2024-07-27T10:38:48.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:38:48.925+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:38:48.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:38:48.925+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:38:48.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:38:48.925+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:38:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:49 smithi077 ceph-mon[37153]: pgmap v7740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:38:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:49 smithi047 ceph-mon[36835]: pgmap v7740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:38:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:49 smithi029 ceph-mon[32610]: pgmap v7740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:38:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:51 smithi077 ceph-mon[37153]: pgmap v7741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:38:51.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:51 smithi047 ceph-mon[36835]: pgmap v7741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:38:51.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:51 smithi029 ceph-mon[32610]: pgmap v7741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:38:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13754.679781 secs 2024-07-27T10:38:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:53 smithi077 ceph-mon[37153]: pgmap v7742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13754.679781 secs 2024-07-27T10:38:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:53 smithi047 ceph-mon[36835]: pgmap v7742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13754.679781 secs 2024-07-27T10:38:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:53 smithi029 ceph-mon[32610]: pgmap v7742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:55 smithi077 ceph-mon[37153]: pgmap v7743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:38:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:55 smithi047 ceph-mon[36835]: pgmap v7743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:38:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:55 smithi029 ceph-mon[32610]: pgmap v7743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:38:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:57 smithi077 ceph-mon[37153]: pgmap v7744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:57 smithi047 ceph-mon[36835]: pgmap v7744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:57.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:57 smithi029 ceph-mon[32610]: pgmap v7744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:38:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13759.679893 secs 2024-07-27T10:38:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13759.679893 secs 2024-07-27T10:38:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13759.679893 secs 2024-07-27T10:38:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:38:59 smithi029 ceph-mon[32610]: pgmap v7745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:38:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:38:59 smithi077 ceph-mon[37153]: pgmap v7745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:38:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:38:59 smithi047 ceph-mon[36835]: pgmap v7745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:01 smithi077 ceph-mon[37153]: pgmap v7746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:01 smithi047 ceph-mon[36835]: pgmap v7746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:01.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:01 smithi029 ceph-mon[32610]: pgmap v7746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13764.680012 secs 2024-07-27T10:39:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:03 smithi077 ceph-mon[37153]: pgmap v7747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13764.680012 secs 2024-07-27T10:39:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:03 smithi047 ceph-mon[36835]: pgmap v7747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13764.680012 secs 2024-07-27T10:39:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:03 smithi029 ceph-mon[32610]: pgmap v7747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:03.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:39:03.940+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:39:03.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:39:03.940+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:39:03.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:39:03.940+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:39:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:05 smithi077 ceph-mon[37153]: pgmap v7748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:39:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:05 smithi047 ceph-mon[36835]: pgmap v7748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:39:05.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:05 smithi029 ceph-mon[32610]: pgmap v7748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:39:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:07 smithi077 ceph-mon[37153]: pgmap v7749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:07 smithi047 ceph-mon[36835]: pgmap v7749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:07.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:07 smithi029 ceph-mon[32610]: pgmap v7749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13769.680123 secs 2024-07-27T10:39:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13769.680123 secs 2024-07-27T10:39:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13769.680123 secs 2024-07-27T10:39:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:09 smithi077 ceph-mon[37153]: pgmap v7750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:09 smithi047 ceph-mon[36835]: pgmap v7750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:09 smithi029 ceph-mon[32610]: pgmap v7750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:11 smithi077 ceph-mon[37153]: pgmap v7751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:11 smithi047 ceph-mon[36835]: pgmap v7751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:11 smithi029 ceph-mon[32610]: pgmap v7751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13774.680244 secs 2024-07-27T10:39:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:13 smithi077 ceph-mon[37153]: pgmap v7752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13774.680244 secs 2024-07-27T10:39:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:13 smithi047 ceph-mon[36835]: pgmap v7752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13774.680244 secs 2024-07-27T10:39:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:13 smithi029 ceph-mon[32610]: pgmap v7752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:15 smithi077 ceph-mon[37153]: pgmap v7753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:39:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:15 smithi047 ceph-mon[36835]: pgmap v7753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:39:15.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:15 smithi029 ceph-mon[32610]: pgmap v7753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:39:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:17 smithi077 ceph-mon[37153]: pgmap v7754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:17 smithi047 ceph-mon[36835]: pgmap v7754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:17.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:17 smithi029 ceph-mon[32610]: pgmap v7754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:18.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13779.680382 secs 2024-07-27T10:39:18.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:39:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13779.680382 secs 2024-07-27T10:39:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:39:18.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13779.680382 secs 2024-07-27T10:39:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:39:18.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:39:18.956+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:39:18.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:39:18.956+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:39:18.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:39:18.956+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:39:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:19 smithi077 ceph-mon[37153]: pgmap v7755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:19.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:19 smithi047 ceph-mon[36835]: pgmap v7755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:19.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:19 smithi029 ceph-mon[32610]: pgmap v7755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:20 smithi029 ceph-mon[32610]: pgmap v7756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:20 smithi077 ceph-mon[37153]: pgmap v7756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:21.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:20 smithi047 ceph-mon[36835]: pgmap v7756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13784.680481 secs 2024-07-27T10:39:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:23 smithi047 ceph-mon[36835]: pgmap v7757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:39:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:39:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13784.680481 secs 2024-07-27T10:39:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:23 smithi029 ceph-mon[32610]: pgmap v7757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:39:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:39:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13784.680481 secs 2024-07-27T10:39:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:23 smithi077 ceph-mon[37153]: pgmap v7757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:39:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:39:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:39:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:39:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:24 smithi047 ceph-mon[36835]: pgmap v7758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T10:39:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:24 smithi047 ceph-mon[36835]: pgmap v7759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:39:24.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:24 smithi029 ceph-mon[32610]: pgmap v7758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T10:39:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:24 smithi029 ceph-mon[32610]: pgmap v7759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:39:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:24 smithi077 ceph-mon[37153]: pgmap v7758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T10:39:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:24 smithi077 ceph-mon[37153]: pgmap v7759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:39:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:26 smithi047 ceph-mon[36835]: pgmap v7760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:39:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:26 smithi029 ceph-mon[32610]: pgmap v7760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:39:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:26 smithi077 ceph-mon[37153]: pgmap v7760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:39:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13789.680601 secs 2024-07-27T10:39:28.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:28 smithi047 ceph-mon[36835]: pgmap v7761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:39:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13789.680601 secs 2024-07-27T10:39:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:28 smithi029 ceph-mon[32610]: pgmap v7761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:39:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13789.680601 secs 2024-07-27T10:39:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:28 smithi077 ceph-mon[37153]: pgmap v7761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:39:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:30 smithi047 ceph-mon[36835]: pgmap v7762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:39:30.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:30 smithi029 ceph-mon[32610]: pgmap v7762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:39:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:30 smithi077 ceph-mon[37153]: pgmap v7762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:39:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:32 smithi047 ceph-mon[36835]: pgmap v7763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:39:32.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:32 smithi029 ceph-mon[32610]: pgmap v7763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:39:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:32 smithi077 ceph-mon[37153]: pgmap v7763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:39:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13794.680719 secs 2024-07-27T10:39:33.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:39:33.958+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:39:33.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:39:33.958+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:39:33.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:39:33.958+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:39:33.959 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13794.680719 secs 2024-07-27T10:39:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13794.680719 secs 2024-07-27T10:39:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:34 smithi047 ceph-mon[36835]: pgmap v7764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:39:34.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:34 smithi029 ceph-mon[32610]: pgmap v7764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:39:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:34 smithi077 ceph-mon[37153]: pgmap v7764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:39:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:36 smithi047 ceph-mon[36835]: pgmap v7765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:36.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:36 smithi029 ceph-mon[32610]: pgmap v7765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:36 smithi077 ceph-mon[37153]: pgmap v7765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13799.680841 secs 2024-07-27T10:39:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:38 smithi047 ceph-mon[36835]: pgmap v7766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:39:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13799.680841 secs 2024-07-27T10:39:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:38 smithi029 ceph-mon[32610]: pgmap v7766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:39:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13799.680841 secs 2024-07-27T10:39:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:38 smithi077 ceph-mon[37153]: pgmap v7766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:39:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:40 smithi047 ceph-mon[36835]: pgmap v7767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:40.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:40 smithi029 ceph-mon[32610]: pgmap v7767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:40 smithi077 ceph-mon[37153]: pgmap v7767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:42 smithi047 ceph-mon[36835]: pgmap v7768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:42 smithi029 ceph-mon[32610]: pgmap v7768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:42 smithi077 ceph-mon[37153]: pgmap v7768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13804.680955 secs 2024-07-27T10:39:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13804.680955 secs 2024-07-27T10:39:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13804.680955 secs 2024-07-27T10:39:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:44 smithi047 ceph-mon[36835]: pgmap v7769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:39:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:44 smithi029 ceph-mon[32610]: pgmap v7769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:39:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:44 smithi077 ceph-mon[37153]: pgmap v7769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:39:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:46 smithi047 ceph-mon[36835]: pgmap v7770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:46 smithi029 ceph-mon[32610]: pgmap v7770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:46 smithi077 ceph-mon[37153]: pgmap v7770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13809.681072 secs 2024-07-27T10:39:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:48 smithi047 ceph-mon[36835]: pgmap v7771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:39:48.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13809.681072 secs 2024-07-27T10:39:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:48 smithi029 ceph-mon[32610]: pgmap v7771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:39:48.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:39:48.960+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:39:48.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:39:48.960+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:39:48.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:39:48.960+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:39:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13809.681072 secs 2024-07-27T10:39:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:48 smithi077 ceph-mon[37153]: pgmap v7771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:39:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:50 smithi047 ceph-mon[36835]: pgmap v7772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:50 smithi029 ceph-mon[32610]: pgmap v7772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:50 smithi077 ceph-mon[37153]: pgmap v7772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:39:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:52 smithi047 ceph-mon[36835]: pgmap v7773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:52 smithi029 ceph-mon[32610]: pgmap v7773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:52 smithi077 ceph-mon[37153]: pgmap v7773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:39:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13814.681196 secs 2024-07-27T10:39:53.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13814.681196 secs 2024-07-27T10:39:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13814.681196 secs 2024-07-27T10:39:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:54 smithi047 ceph-mon[36835]: pgmap v7774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:39:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:54 smithi029 ceph-mon[32610]: pgmap v7774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:39:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:54 smithi077 ceph-mon[37153]: pgmap v7774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:39:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:56 smithi047 ceph-mon[36835]: pgmap v7775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:39:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:56 smithi029 ceph-mon[32610]: pgmap v7775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:39:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:56 smithi077 ceph-mon[37153]: pgmap v7775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:39:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13819.681314 secs 2024-07-27T10:39:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:39:58 smithi047 ceph-mon[36835]: pgmap v7776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:39:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13819.681314 secs 2024-07-27T10:39:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:39:58 smithi029 ceph-mon[32610]: pgmap v7776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:39:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13819.681314 secs 2024-07-27T10:39:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:39:58 smithi077 ceph-mon[37153]: pgmap v7776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:40:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:00 smithi047 ceph-mon[36835]: pgmap v7777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:40:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:00 smithi029 ceph-mon[32610]: pgmap v7777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:40:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:00 smithi077 ceph-mon[37153]: pgmap v7777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:40:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:02 smithi029 ceph-mon[32610]: pgmap v7778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:02 smithi077 ceph-mon[37153]: pgmap v7778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:02 smithi047 ceph-mon[36835]: pgmap v7778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13824.681433 secs 2024-07-27T10:40:03.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:03.976+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:40:03.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:03.976+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:40:03.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:03.976+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:40:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13824.681433 secs 2024-07-27T10:40:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13824.681433 secs 2024-07-27T10:40:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:04 smithi029 ceph-mon[32610]: pgmap v7779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:40:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:04 smithi077 ceph-mon[37153]: pgmap v7779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:40:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:04 smithi047 ceph-mon[36835]: pgmap v7779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:40:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:06 smithi029 ceph-mon[32610]: pgmap v7780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:40:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:06 smithi077 ceph-mon[37153]: pgmap v7780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:40:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:06 smithi047 ceph-mon[36835]: pgmap v7780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:40:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13829.681554 secs 2024-07-27T10:40:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:08 smithi077 ceph-mon[37153]: pgmap v7781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:40:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13829.681554 secs 2024-07-27T10:40:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:08 smithi047 ceph-mon[36835]: pgmap v7781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:40:09.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13829.681554 secs 2024-07-27T10:40:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:08 smithi029 ceph-mon[32610]: pgmap v7781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:40:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:10 smithi077 ceph-mon[37153]: pgmap v7782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:10 smithi047 ceph-mon[36835]: pgmap v7782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:10 smithi029 ceph-mon[32610]: pgmap v7782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:12 smithi077 ceph-mon[37153]: pgmap v7783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:12 smithi047 ceph-mon[36835]: pgmap v7783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:12 smithi029 ceph-mon[32610]: pgmap v7783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13834.681618 secs 2024-07-27T10:40:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13834.681618 secs 2024-07-27T10:40:14.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13834.681618 secs 2024-07-27T10:40:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:14 smithi077 ceph-mon[37153]: pgmap v7784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:40:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:14 smithi047 ceph-mon[36835]: pgmap v7784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:40:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:14 smithi029 ceph-mon[32610]: pgmap v7784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:40:17.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:16 smithi029 ceph-mon[32610]: pgmap v7785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:16 smithi077 ceph-mon[37153]: pgmap v7785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:16 smithi047 ceph-mon[36835]: pgmap v7785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:18.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:18.992+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:40:18.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:18.992+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:40:18.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:18.992+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:40:18.994 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13839.681756 secs 2024-07-27T10:40:18.994 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:18 smithi029 ceph-mon[32610]: pgmap v7786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:40:18.998 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13839.681756 secs 2024-07-27T10:40:18.998 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:18 smithi047 ceph-mon[36835]: pgmap v7786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:40:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13839.681756 secs 2024-07-27T10:40:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:18 smithi077 ceph-mon[37153]: pgmap v7786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:40:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:20 smithi077 ceph-mon[37153]: pgmap v7787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:20 smithi047 ceph-mon[36835]: pgmap v7787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:21.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:20 smithi029 ceph-mon[32610]: pgmap v7787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:22 smithi077 ceph-mon[37153]: pgmap v7788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:22 smithi047 ceph-mon[36835]: pgmap v7788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:22 smithi029 ceph-mon[32610]: pgmap v7788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13844.681881 secs 2024-07-27T10:40:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:40:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13844.681881 secs 2024-07-27T10:40:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:40:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:24.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:24.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:24.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13844.681881 secs 2024-07-27T10:40:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:40:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:24 smithi077 ceph-mon[37153]: pgmap v7789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:40:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:40:25.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:40:25.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:24 smithi077 ceph-mon[37153]: pgmap v7790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T10:40:25.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:24 smithi077 ceph-mon[37153]: pgmap v7791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:40:25.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:24 smithi047 ceph-mon[36835]: pgmap v7789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:40:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:40:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:40:25.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:24 smithi047 ceph-mon[36835]: pgmap v7790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T10:40:25.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:24 smithi047 ceph-mon[36835]: pgmap v7791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:40:25.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:24 smithi029 ceph-mon[32610]: pgmap v7789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:25.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:40:25.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:40:25.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:24 smithi029 ceph-mon[32610]: pgmap v7790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T10:40:25.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:24 smithi029 ceph-mon[32610]: pgmap v7791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:40:25.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:27 smithi029 ceph-mon[32610]: pgmap v7792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:40:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:27 smithi077 ceph-mon[37153]: pgmap v7792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:40:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:27 smithi047 ceph-mon[36835]: pgmap v7792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:40:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13849.681976 secs 2024-07-27T10:40:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13849.681976 secs 2024-07-27T10:40:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13849.681976 secs 2024-07-27T10:40:29.611 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:29 smithi077 ceph-mon[37153]: pgmap v7793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:40:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:29 smithi047 ceph-mon[36835]: pgmap v7793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:40:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:29 smithi029 ceph-mon[32610]: pgmap v7793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:40:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:31 smithi077 ceph-mon[37153]: pgmap v7794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T10:40:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:31 smithi047 ceph-mon[36835]: pgmap v7794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T10:40:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:31 smithi029 ceph-mon[32610]: pgmap v7794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T10:40:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13854.682093 secs 2024-07-27T10:40:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:33 smithi077 ceph-mon[37153]: pgmap v7795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-27T10:40:33.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13854.682093 secs 2024-07-27T10:40:33.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:33 smithi029 ceph-mon[32610]: pgmap v7795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-27T10:40:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13854.682093 secs 2024-07-27T10:40:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:33 smithi047 ceph-mon[36835]: pgmap v7795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-27T10:40:33.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:33.998+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:40:33.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:33.998+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:40:33.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:33.998+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:40:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:35 smithi077 ceph-mon[37153]: pgmap v7796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:40:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:35 smithi047 ceph-mon[36835]: pgmap v7796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:40:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:35 smithi029 ceph-mon[32610]: pgmap v7796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:40:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:37 smithi077 ceph-mon[37153]: pgmap v7797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:37 smithi047 ceph-mon[36835]: pgmap v7797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:37 smithi029 ceph-mon[32610]: pgmap v7797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:38.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13859.682214 secs 2024-07-27T10:40:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13859.682214 secs 2024-07-27T10:40:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13859.682214 secs 2024-07-27T10:40:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:39 smithi077 ceph-mon[37153]: pgmap v7798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:39 smithi047 ceph-mon[36835]: pgmap v7798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:39.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:39 smithi029 ceph-mon[32610]: pgmap v7798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:41 smithi077 ceph-mon[37153]: pgmap v7799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:41 smithi047 ceph-mon[36835]: pgmap v7799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:41 smithi029 ceph-mon[32610]: pgmap v7799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:40:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13864.682313 secs 2024-07-27T10:40:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:43 smithi077 ceph-mon[37153]: pgmap v7800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13864.682313 secs 2024-07-27T10:40:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:43 smithi047 ceph-mon[36835]: pgmap v7800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13864.682313 secs 2024-07-27T10:40:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:43 smithi029 ceph-mon[32610]: pgmap v7800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:45 smithi077 ceph-mon[37153]: pgmap v7801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:40:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:45 smithi047 ceph-mon[36835]: pgmap v7801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:40:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:45 smithi029 ceph-mon[32610]: pgmap v7801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:40:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:47 smithi077 ceph-mon[37153]: pgmap v7802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:47 smithi047 ceph-mon[36835]: pgmap v7802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:47 smithi029 ceph-mon[32610]: pgmap v7802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13869.682438 secs 2024-07-27T10:40:48.507 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13869.682438 secs 2024-07-27T10:40:48.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13869.682438 secs 2024-07-27T10:40:49.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:49.010+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:40:49.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:49.010+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:40:49.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:49.010+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:40:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:49 smithi029 ceph-mon[32610]: pgmap v7803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:49 smithi077 ceph-mon[37153]: pgmap v7803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:49 smithi047 ceph-mon[36835]: pgmap v7803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:51 smithi077 ceph-mon[37153]: pgmap v7804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:40:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:51 smithi047 ceph-mon[36835]: pgmap v7804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:40:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:51 smithi029 ceph-mon[32610]: pgmap v7804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:40:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13874.682525 secs 2024-07-27T10:40:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:53 smithi077 ceph-mon[37153]: pgmap v7805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13874.682525 secs 2024-07-27T10:40:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:53 smithi047 ceph-mon[36835]: pgmap v7805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13874.682525 secs 2024-07-27T10:40:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:53 smithi029 ceph-mon[32610]: pgmap v7805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:55 smithi077 ceph-mon[37153]: pgmap v7806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:40:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:55 smithi047 ceph-mon[36835]: pgmap v7806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:40:55.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:55 smithi029 ceph-mon[32610]: pgmap v7806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:40:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:57 smithi077 ceph-mon[37153]: pgmap v7807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:57 smithi047 ceph-mon[36835]: pgmap v7807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:57.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:57 smithi029 ceph-mon[32610]: pgmap v7807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:40:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13879.682662 secs 2024-07-27T10:40:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13879.682662 secs 2024-07-27T10:40:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13879.682662 secs 2024-07-27T10:40:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:40:59 smithi077 ceph-mon[37153]: pgmap v7808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:40:59 smithi047 ceph-mon[36835]: pgmap v7808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:59.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:40:59 smithi029 ceph-mon[32610]: pgmap v7808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:40:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:59.744+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:40:59.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:59.744+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:40:59.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:59.744+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T10:40:59.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:59.744+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:40:59.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:59.744+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T10:40:59.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:59.744+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:40:59.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:59.744+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:40:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:40:59.744+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:41:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:01 smithi077 ceph-mon[37153]: pgmap v7809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:01 smithi047 ceph-mon[36835]: pgmap v7809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:01.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:01 smithi029 ceph-mon[32610]: pgmap v7809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13884.682808 secs 2024-07-27T10:41:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:03 smithi077 ceph-mon[37153]: pgmap v7810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13884.682808 secs 2024-07-27T10:41:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:03 smithi047 ceph-mon[36835]: pgmap v7810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13884.682808 secs 2024-07-27T10:41:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:03 smithi029 ceph-mon[32610]: pgmap v7810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:04.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:41:04.026+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:41:04.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:41:04.026+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:41:04.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:41:04.026+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:41:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:05 smithi077 ceph-mon[37153]: pgmap v7811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:41:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:05 smithi047 ceph-mon[36835]: pgmap v7811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:41:05.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:05 smithi029 ceph-mon[32610]: pgmap v7811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:41:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:07 smithi077 ceph-mon[37153]: pgmap v7812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:07 smithi047 ceph-mon[36835]: pgmap v7812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:07 smithi029 ceph-mon[32610]: pgmap v7812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13889.682927 secs 2024-07-27T10:41:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13889.682927 secs 2024-07-27T10:41:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13889.682927 secs 2024-07-27T10:41:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:09 smithi077 ceph-mon[37153]: pgmap v7813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:09 smithi047 ceph-mon[36835]: pgmap v7813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:09 smithi029 ceph-mon[32610]: pgmap v7813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:11 smithi077 ceph-mon[37153]: pgmap v7814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:11 smithi047 ceph-mon[36835]: pgmap v7814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:11 smithi029 ceph-mon[32610]: pgmap v7814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13894.683027 secs 2024-07-27T10:41:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:13 smithi077 ceph-mon[37153]: pgmap v7815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13894.683027 secs 2024-07-27T10:41:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:13 smithi047 ceph-mon[36835]: pgmap v7815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13894.683027 secs 2024-07-27T10:41:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:13 smithi029 ceph-mon[32610]: pgmap v7815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:15 smithi077 ceph-mon[37153]: pgmap v7816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:41:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:15 smithi047 ceph-mon[36835]: pgmap v7816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:41:16.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:15 smithi029 ceph-mon[32610]: pgmap v7816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:41:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:17 smithi077 ceph-mon[37153]: pgmap v7817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:17 smithi047 ceph-mon[36835]: pgmap v7817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:17 smithi029 ceph-mon[32610]: pgmap v7817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:19.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13899.683152 secs 2024-07-27T10:41:19.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:41:19.040+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:41:19.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:41:19.040+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:41:19.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:41:19.040+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:41:19.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13899.683152 secs 2024-07-27T10:41:19.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13899.683152 secs 2024-07-27T10:41:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:19 smithi077 ceph-mon[37153]: pgmap v7818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:19 smithi047 ceph-mon[36835]: pgmap v7818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:19 smithi029 ceph-mon[32610]: pgmap v7818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:21 smithi077 ceph-mon[37153]: pgmap v7819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:21 smithi047 ceph-mon[36835]: pgmap v7819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:21 smithi029 ceph-mon[32610]: pgmap v7819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13904.683274 secs 2024-07-27T10:41:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:23 smithi077 ceph-mon[37153]: pgmap v7820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13904.683274 secs 2024-07-27T10:41:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:23 smithi047 ceph-mon[36835]: pgmap v7820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13904.683274 secs 2024-07-27T10:41:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:23 smithi029 ceph-mon[32610]: pgmap v7820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:41:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:41:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:41:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:41:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:41:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:41:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:41:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:41:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:41:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:41:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:41:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:41:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:25 smithi077 ceph-mon[37153]: pgmap v7821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:41:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:25 smithi077 ceph-mon[37153]: pgmap v7822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:41:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:25 smithi077 ceph-mon[37153]: pgmap v7823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:41:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:25 smithi047 ceph-mon[36835]: pgmap v7821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:41:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:25 smithi047 ceph-mon[36835]: pgmap v7822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:41:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:25 smithi047 ceph-mon[36835]: pgmap v7823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:41:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:25 smithi029 ceph-mon[32610]: pgmap v7821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:41:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:25 smithi029 ceph-mon[32610]: pgmap v7822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:41:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:25 smithi029 ceph-mon[32610]: pgmap v7823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:41:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:26 smithi077 ceph-mon[37153]: pgmap v7824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:41:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:26 smithi047 ceph-mon[36835]: pgmap v7824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:41:27.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:26 smithi029 ceph-mon[32610]: pgmap v7824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:41:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13909.683411 secs 2024-07-27T10:41:28.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13909.683411 secs 2024-07-27T10:41:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13909.683411 secs 2024-07-27T10:41:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:28 smithi047 ceph-mon[36835]: pgmap v7825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:41:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:28 smithi029 ceph-mon[32610]: pgmap v7825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:41:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:28 smithi077 ceph-mon[37153]: pgmap v7825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:41:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:30 smithi077 ceph-mon[37153]: pgmap v7826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:41:31.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:30 smithi047 ceph-mon[36835]: pgmap v7826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:41:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:30 smithi029 ceph-mon[32610]: pgmap v7826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:41:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:32 smithi077 ceph-mon[37153]: pgmap v7827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:41:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:32 smithi047 ceph-mon[36835]: pgmap v7827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:41:33.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:32 smithi029 ceph-mon[32610]: pgmap v7827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:41:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:41:34.056+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:41:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:41:34.056+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:41:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:41:34.056+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:41:34.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13914.683473 secs 2024-07-27T10:41:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13914.683473 secs 2024-07-27T10:41:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13914.683473 secs 2024-07-27T10:41:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:34 smithi077 ceph-mon[37153]: pgmap v7828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:41:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:34 smithi047 ceph-mon[36835]: pgmap v7828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:41:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:34 smithi029 ceph-mon[32610]: pgmap v7828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:41:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:36 smithi077 ceph-mon[37153]: pgmap v7829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:36 smithi047 ceph-mon[36835]: pgmap v7829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:37.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:36 smithi029 ceph-mon[32610]: pgmap v7829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13919.683607 secs 2024-07-27T10:41:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13919.683607 secs 2024-07-27T10:41:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13919.683607 secs 2024-07-27T10:41:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:38 smithi047 ceph-mon[36835]: pgmap v7830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:39.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:38 smithi029 ceph-mon[32610]: pgmap v7830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:38 smithi077 ceph-mon[37153]: pgmap v7830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:40 smithi077 ceph-mon[37153]: pgmap v7831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:40 smithi047 ceph-mon[36835]: pgmap v7831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:40 smithi029 ceph-mon[32610]: pgmap v7831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:42 smithi077 ceph-mon[37153]: pgmap v7832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:41:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:42 smithi047 ceph-mon[36835]: pgmap v7832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:41:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:42 smithi029 ceph-mon[32610]: pgmap v7832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:41:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13924.683698 secs 2024-07-27T10:41:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13924.683698 secs 2024-07-27T10:41:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13924.683698 secs 2024-07-27T10:41:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:44 smithi077 ceph-mon[37153]: pgmap v7833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:41:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:44 smithi047 ceph-mon[36835]: pgmap v7833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:41:45.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:44 smithi029 ceph-mon[32610]: pgmap v7833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:41:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:46 smithi077 ceph-mon[37153]: pgmap v7834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:46 smithi047 ceph-mon[36835]: pgmap v7834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:47.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:46 smithi029 ceph-mon[32610]: pgmap v7834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13929.683800 secs 2024-07-27T10:41:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13929.683800 secs 2024-07-27T10:41:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13929.683800 secs 2024-07-27T10:41:49.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:41:49.070+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:41:49.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:41:49.070+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:41:49.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:41:49.070+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:41:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:48 smithi047 ceph-mon[36835]: pgmap v7835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:48 smithi029 ceph-mon[32610]: pgmap v7835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:49.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:48 smithi077 ceph-mon[37153]: pgmap v7835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:50 smithi077 ceph-mon[37153]: pgmap v7836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:50 smithi047 ceph-mon[36835]: pgmap v7836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:51.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:50 smithi029 ceph-mon[32610]: pgmap v7836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:41:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:52 smithi077 ceph-mon[37153]: pgmap v7837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:41:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:52 smithi047 ceph-mon[36835]: pgmap v7837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:41:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:52 smithi029 ceph-mon[32610]: pgmap v7837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:41:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13934.683914 secs 2024-07-27T10:41:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13934.683914 secs 2024-07-27T10:41:54.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13934.683914 secs 2024-07-27T10:41:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:54 smithi077 ceph-mon[37153]: pgmap v7838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:41:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:54 smithi047 ceph-mon[36835]: pgmap v7838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:41:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:54 smithi029 ceph-mon[32610]: pgmap v7838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:41:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:56 smithi077 ceph-mon[37153]: pgmap v7839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:56 smithi047 ceph-mon[36835]: pgmap v7839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:57.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:56 smithi029 ceph-mon[32610]: pgmap v7839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:41:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13939.684039 secs 2024-07-27T10:41:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13939.684039 secs 2024-07-27T10:41:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13939.684039 secs 2024-07-27T10:41:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:41:58 smithi047 ceph-mon[36835]: pgmap v7840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:59.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:41:58 smithi029 ceph-mon[32610]: pgmap v7840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:41:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:41:58 smithi077 ceph-mon[37153]: pgmap v7840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:00 smithi077 ceph-mon[37153]: pgmap v7841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:00 smithi047 ceph-mon[36835]: pgmap v7841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:01.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:00 smithi029 ceph-mon[32610]: pgmap v7841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:02 smithi077 ceph-mon[37153]: pgmap v7842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:42:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:02 smithi047 ceph-mon[36835]: pgmap v7842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:42:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:02 smithi029 ceph-mon[32610]: pgmap v7842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:42:04.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:42:04.081+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:42:04.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:42:04.081+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:42:04.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:42:04.081+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:42:04.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13944.684163 secs 2024-07-27T10:42:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13944.684163 secs 2024-07-27T10:42:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13944.684163 secs 2024-07-27T10:42:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:04 smithi077 ceph-mon[37153]: pgmap v7843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:42:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:04 smithi047 ceph-mon[36835]: pgmap v7843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:42:05.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:04 smithi029 ceph-mon[32610]: pgmap v7843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:42:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:06 smithi077 ceph-mon[37153]: pgmap v7844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:06 smithi047 ceph-mon[36835]: pgmap v7844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:06 smithi029 ceph-mon[32610]: pgmap v7844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13949.684254 secs 2024-07-27T10:42:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13949.684254 secs 2024-07-27T10:42:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13949.684254 secs 2024-07-27T10:42:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:09 smithi077 ceph-mon[37153]: pgmap v7845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:09 smithi047 ceph-mon[36835]: pgmap v7845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:09 smithi029 ceph-mon[32610]: pgmap v7845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:11 smithi077 ceph-mon[37153]: pgmap v7846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:11 smithi047 ceph-mon[36835]: pgmap v7846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:11 smithi029 ceph-mon[32610]: pgmap v7846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13954.684407 secs 2024-07-27T10:42:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:13 smithi077 ceph-mon[37153]: pgmap v7847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:42:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13954.684407 secs 2024-07-27T10:42:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:13 smithi047 ceph-mon[36835]: pgmap v7847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:42:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13954.684407 secs 2024-07-27T10:42:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:13 smithi029 ceph-mon[32610]: pgmap v7847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:42:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:14 smithi077 ceph-mon[37153]: pgmap v7848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:42:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:14 smithi047 ceph-mon[36835]: pgmap v7848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:42:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:14 smithi029 ceph-mon[32610]: pgmap v7848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:42:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:16 smithi077 ceph-mon[37153]: pgmap v7849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:16 smithi047 ceph-mon[36835]: pgmap v7849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:17.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:16 smithi029 ceph-mon[32610]: pgmap v7849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13959.684529 secs 2024-07-27T10:42:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13959.684529 secs 2024-07-27T10:42:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13959.684529 secs 2024-07-27T10:42:19.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:42:19.096+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:42:19.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:42:19.096+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:42:19.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:42:19.096+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:42:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:18 smithi047 ceph-mon[36835]: pgmap v7850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:18 smithi029 ceph-mon[32610]: pgmap v7850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:19.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:18 smithi077 ceph-mon[37153]: pgmap v7850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:20 smithi077 ceph-mon[37153]: pgmap v7851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:20 smithi047 ceph-mon[36835]: pgmap v7851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:21.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:20 smithi029 ceph-mon[32610]: pgmap v7851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:22 smithi077 ceph-mon[37153]: pgmap v7852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:42:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:22 smithi047 ceph-mon[36835]: pgmap v7852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:42:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:22 smithi029 ceph-mon[32610]: pgmap v7852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:42:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13964.684627 secs 2024-07-27T10:42:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13964.684627 secs 2024-07-27T10:42:24.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13964.684627 secs 2024-07-27T10:42:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:24 smithi077 ceph-mon[37153]: pgmap v7853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:42:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:24 smithi047 ceph-mon[36835]: pgmap v7853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:42:25.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:24 smithi029 ceph-mon[32610]: pgmap v7853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:42:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:42:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:42:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:42:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:25 smithi077 ceph-mon[37153]: pgmap v7854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:42:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:25 smithi077 ceph-mon[37153]: pgmap v7855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:42:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:42:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:42:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:42:26.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:42:26.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:25 smithi047 ceph-mon[36835]: pgmap v7854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:42:26.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:25 smithi047 ceph-mon[36835]: pgmap v7855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:42:26.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:42:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:42:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:42:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:42:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:25 smithi029 ceph-mon[32610]: pgmap v7854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:42:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:25 smithi029 ceph-mon[32610]: pgmap v7855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:42:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:42:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13969.684740 secs 2024-07-27T10:42:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:28 smithi077 ceph-mon[37153]: pgmap v7856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:42:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13969.684740 secs 2024-07-27T10:42:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:28 smithi047 ceph-mon[36835]: pgmap v7856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:42:28.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13969.684740 secs 2024-07-27T10:42:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:28 smithi029 ceph-mon[32610]: pgmap v7856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:42:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:30 smithi077 ceph-mon[37153]: pgmap v7857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:42:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:30 smithi047 ceph-mon[36835]: pgmap v7857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:42:30.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:30 smithi029 ceph-mon[32610]: pgmap v7857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:42:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:32 smithi077 ceph-mon[37153]: pgmap v7858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:42:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:32 smithi047 ceph-mon[36835]: pgmap v7858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:42:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:32 smithi029 ceph-mon[32610]: pgmap v7858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:42:33.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13974.684883 secs 2024-07-27T10:42:33.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13974.684883 secs 2024-07-27T10:42:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13974.684883 secs 2024-07-27T10:42:34.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:42:34.104+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:42:34.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:42:34.104+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:42:34.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:42:34.104+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:42:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:34 smithi047 ceph-mon[36835]: pgmap v7859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:42:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:34 smithi029 ceph-mon[32610]: pgmap v7859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:42:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:34 smithi077 ceph-mon[37153]: pgmap v7859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:42:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:36 smithi029 ceph-mon[32610]: pgmap v7860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:42:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:36 smithi047 ceph-mon[36835]: pgmap v7860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:42:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:36 smithi077 ceph-mon[37153]: pgmap v7860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:42:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13979.685008 secs 2024-07-27T10:42:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:38 smithi047 ceph-mon[36835]: pgmap v7861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13979.685008 secs 2024-07-27T10:42:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:38 smithi029 ceph-mon[32610]: pgmap v7861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13979.685008 secs 2024-07-27T10:42:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:38 smithi077 ceph-mon[37153]: pgmap v7861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:40 smithi047 ceph-mon[36835]: pgmap v7862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:42:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:40 smithi029 ceph-mon[32610]: pgmap v7862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:42:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:40 smithi077 ceph-mon[37153]: pgmap v7862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:42:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:42 smithi047 ceph-mon[36835]: pgmap v7863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:42.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:42 smithi029 ceph-mon[32610]: pgmap v7863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:42 smithi077 ceph-mon[37153]: pgmap v7863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13984.685163 secs 2024-07-27T10:42:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13984.685163 secs 2024-07-27T10:42:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13984.685163 secs 2024-07-27T10:42:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:44 smithi047 ceph-mon[36835]: pgmap v7864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:44 smithi029 ceph-mon[32610]: pgmap v7864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:44 smithi077 ceph-mon[37153]: pgmap v7864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:46 smithi047 ceph-mon[36835]: pgmap v7865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:46 smithi029 ceph-mon[32610]: pgmap v7865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:46 smithi077 ceph-mon[37153]: pgmap v7865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13989.685284 secs 2024-07-27T10:42:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:48 smithi047 ceph-mon[36835]: pgmap v7866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:48.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13989.685284 secs 2024-07-27T10:42:48.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:48 smithi029 ceph-mon[32610]: pgmap v7866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13989.685284 secs 2024-07-27T10:42:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:48 smithi077 ceph-mon[37153]: pgmap v7866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:49.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:42:49.112+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:42:49.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:42:49.112+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:42:49.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:42:49.112+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:42:50.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:50 smithi077 ceph-mon[37153]: pgmap v7867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:42:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:50 smithi047 ceph-mon[36835]: pgmap v7867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:42:50.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:50 smithi029 ceph-mon[32610]: pgmap v7867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:42:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:52 smithi047 ceph-mon[36835]: pgmap v7868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:52 smithi029 ceph-mon[32610]: pgmap v7868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:52 smithi077 ceph-mon[37153]: pgmap v7868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13994.685426 secs 2024-07-27T10:42:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13994.685426 secs 2024-07-27T10:42:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13994.685426 secs 2024-07-27T10:42:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:54 smithi047 ceph-mon[36835]: pgmap v7869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:54 smithi029 ceph-mon[32610]: pgmap v7869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:54 smithi077 ceph-mon[37153]: pgmap v7869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:42:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:56 smithi047 ceph-mon[36835]: pgmap v7870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:56.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:56 smithi029 ceph-mon[32610]: pgmap v7870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:56 smithi077 ceph-mon[37153]: pgmap v7870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:42:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 13999.685523 secs 2024-07-27T10:42:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:42:58 smithi047 ceph-mon[36835]: pgmap v7871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 13999.685523 secs 2024-07-27T10:42:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:42:58 smithi029 ceph-mon[32610]: pgmap v7871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:42:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 13999.685523 secs 2024-07-27T10:42:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:42:58 smithi077 ceph-mon[37153]: pgmap v7871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:00 smithi047 ceph-mon[36835]: pgmap v7872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:43:00.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:00 smithi029 ceph-mon[32610]: pgmap v7872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:43:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:00 smithi077 ceph-mon[37153]: pgmap v7872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:43:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:02 smithi047 ceph-mon[36835]: pgmap v7873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:02 smithi029 ceph-mon[32610]: pgmap v7873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:02 smithi077 ceph-mon[37153]: pgmap v7873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14004.685643 secs 2024-07-27T10:43:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14004.685643 secs 2024-07-27T10:43:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14004.685643 secs 2024-07-27T10:43:04.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:04.128+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:43:04.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:04.128+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:43:04.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:04.128+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:43:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:04 smithi047 ceph-mon[36835]: pgmap v7874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:04.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:04 smithi029 ceph-mon[32610]: pgmap v7874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:04 smithi077 ceph-mon[37153]: pgmap v7874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:06 smithi047 ceph-mon[36835]: pgmap v7875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:06.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:06 smithi029 ceph-mon[32610]: pgmap v7875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:06 smithi077 ceph-mon[37153]: pgmap v7875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14009.685717 secs 2024-07-27T10:43:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:08 smithi047 ceph-mon[36835]: pgmap v7876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14009.685717 secs 2024-07-27T10:43:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:08 smithi029 ceph-mon[32610]: pgmap v7876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14009.685717 secs 2024-07-27T10:43:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:08 smithi077 ceph-mon[37153]: pgmap v7876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:10 smithi047 ceph-mon[36835]: pgmap v7877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:43:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:10 smithi029 ceph-mon[32610]: pgmap v7877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:43:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:10 smithi077 ceph-mon[37153]: pgmap v7877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:43:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:12 smithi047 ceph-mon[36835]: pgmap v7878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:12.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:12 smithi029 ceph-mon[32610]: pgmap v7878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:12 smithi077 ceph-mon[37153]: pgmap v7878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14014.685858 secs 2024-07-27T10:43:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14014.685858 secs 2024-07-27T10:43:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14014.685858 secs 2024-07-27T10:43:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:14 smithi047 ceph-mon[36835]: pgmap v7879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:14.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:14 smithi029 ceph-mon[32610]: pgmap v7879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:14 smithi077 ceph-mon[37153]: pgmap v7879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:16 smithi047 ceph-mon[36835]: pgmap v7880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:16.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:16 smithi029 ceph-mon[32610]: pgmap v7880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:16 smithi077 ceph-mon[37153]: pgmap v7880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14019.685997 secs 2024-07-27T10:43:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:18 smithi077 ceph-mon[37153]: pgmap v7881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:19.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:19.134+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:43:19.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:19.134+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:43:19.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:19.134+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:43:19.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14019.685997 secs 2024-07-27T10:43:19.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:18 smithi029 ceph-mon[32610]: pgmap v7881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14019.685997 secs 2024-07-27T10:43:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:18 smithi047 ceph-mon[36835]: pgmap v7881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:20.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:19 smithi047 ceph-mon[36835]: pgmap v7882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:43:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:19 smithi077 ceph-mon[37153]: pgmap v7882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:43:20.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:19 smithi029 ceph-mon[32610]: pgmap v7882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:43:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:22 smithi047 ceph-mon[36835]: pgmap v7883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:22 smithi029 ceph-mon[32610]: pgmap v7883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:22 smithi077 ceph-mon[37153]: pgmap v7883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:43:23.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14024.686116 secs 2024-07-27T10:43:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14024.686116 secs 2024-07-27T10:43:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14024.686116 secs 2024-07-27T10:43:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:24 smithi047 ceph-mon[36835]: pgmap v7884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:24 smithi029 ceph-mon[32610]: pgmap v7884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:24 smithi077 ceph-mon[37153]: pgmap v7884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:26 smithi047 ceph-mon[36835]: pgmap v7885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:43:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:43:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:43:26.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:43:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:26 smithi029 ceph-mon[32610]: pgmap v7885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:43:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:43:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:43:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:43:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:26 smithi077 ceph-mon[37153]: pgmap v7885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:43:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:43:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:43:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:43:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:27 smithi047 ceph-mon[36835]: pgmap v7886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:43:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:27 smithi047 ceph-mon[36835]: pgmap v7887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:43:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:27 smithi029 ceph-mon[32610]: pgmap v7886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:43:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:27 smithi029 ceph-mon[32610]: pgmap v7887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:43:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:27 smithi077 ceph-mon[37153]: pgmap v7886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:43:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:27 smithi077 ceph-mon[37153]: pgmap v7887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:43:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14029.686235 secs 2024-07-27T10:43:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14029.686235 secs 2024-07-27T10:43:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14029.686235 secs 2024-07-27T10:43:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:29 smithi047 ceph-mon[36835]: pgmap v7888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:43:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:29 smithi029 ceph-mon[32610]: pgmap v7888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:43:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:29 smithi077 ceph-mon[37153]: pgmap v7888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:43:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:31 smithi047 ceph-mon[36835]: pgmap v7889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:43:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:31 smithi029 ceph-mon[32610]: pgmap v7889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:43:31.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:31 smithi077 ceph-mon[37153]: pgmap v7889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:43:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:33 smithi047 ceph-mon[36835]: pgmap v7890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:43:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14034.686359 secs 2024-07-27T10:43:33.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:33 smithi029 ceph-mon[32610]: pgmap v7890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:43:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14034.686359 secs 2024-07-27T10:43:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:33 smithi077 ceph-mon[37153]: pgmap v7890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:43:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14034.686359 secs 2024-07-27T10:43:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:34.142+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:43:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:34.142+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:43:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:34.142+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:43:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:35 smithi047 ceph-mon[36835]: pgmap v7891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:43:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:35 smithi029 ceph-mon[32610]: pgmap v7891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:43:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:35 smithi077 ceph-mon[37153]: pgmap v7891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:43:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:37 smithi047 ceph-mon[36835]: pgmap v7892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:43:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:37 smithi029 ceph-mon[32610]: pgmap v7892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:43:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:37 smithi077 ceph-mon[37153]: pgmap v7892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:43:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14039.686476 secs 2024-07-27T10:43:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14039.686476 secs 2024-07-27T10:43:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14039.686476 secs 2024-07-27T10:43:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:39 smithi029 ceph-mon[32610]: pgmap v7893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:43:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:39 smithi047 ceph-mon[36835]: pgmap v7893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:43:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:39 smithi077 ceph-mon[37153]: pgmap v7893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:43:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:41 smithi047 ceph-mon[36835]: pgmap v7894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:41 smithi029 ceph-mon[32610]: pgmap v7894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:41 smithi077 ceph-mon[37153]: pgmap v7894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:43 smithi047 ceph-mon[36835]: pgmap v7895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:43:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14044.686620 secs 2024-07-27T10:43:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:43 smithi029 ceph-mon[32610]: pgmap v7895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:43:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14044.686620 secs 2024-07-27T10:43:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:43 smithi077 ceph-mon[37153]: pgmap v7895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:43:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14044.686620 secs 2024-07-27T10:43:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:45 smithi047 ceph-mon[36835]: pgmap v7896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:45 smithi029 ceph-mon[32610]: pgmap v7896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:45 smithi077 ceph-mon[37153]: pgmap v7896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:47 smithi047 ceph-mon[36835]: pgmap v7897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:47 smithi029 ceph-mon[32610]: pgmap v7897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:47 smithi077 ceph-mon[37153]: pgmap v7897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14049.686691 secs 2024-07-27T10:43:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14049.686691 secs 2024-07-27T10:43:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14049.686691 secs 2024-07-27T10:43:49.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:49.156+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:43:49.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:49.156+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:43:49.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:49.156+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:43:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:49 smithi047 ceph-mon[36835]: pgmap v7898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:43:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:49 smithi029 ceph-mon[32610]: pgmap v7898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:43:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:49 smithi077 ceph-mon[37153]: pgmap v7898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:43:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:51 smithi047 ceph-mon[36835]: pgmap v7899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:51 smithi029 ceph-mon[32610]: pgmap v7899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:51 smithi077 ceph-mon[37153]: pgmap v7899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:53 smithi047 ceph-mon[36835]: pgmap v7900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:43:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14054.686814 secs 2024-07-27T10:43:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:53 smithi029 ceph-mon[32610]: pgmap v7900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:43:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14054.686814 secs 2024-07-27T10:43:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:53 smithi077 ceph-mon[37153]: pgmap v7900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:43:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14054.686814 secs 2024-07-27T10:43:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:55 smithi047 ceph-mon[36835]: pgmap v7901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:55.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:55 smithi029 ceph-mon[32610]: pgmap v7901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:55 smithi077 ceph-mon[37153]: pgmap v7901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:43:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:57 smithi047 ceph-mon[36835]: pgmap v7902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:57.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:57 smithi029 ceph-mon[32610]: pgmap v7902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:57 smithi077 ceph-mon[37153]: pgmap v7902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:43:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14059.686934 secs 2024-07-27T10:43:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14059.686934 secs 2024-07-27T10:43:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14059.686934 secs 2024-07-27T10:43:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:43:59 smithi029 ceph-mon[32610]: pgmap v7903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:43:59.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:59.771+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:43:59.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:59.771+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:43:59.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:59.771+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:43:59.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:43:59.771+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:43:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:43:59 smithi077 ceph-mon[37153]: pgmap v7903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:43:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:43:59 smithi047 ceph-mon[36835]: pgmap v7903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:44:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:01 smithi029 ceph-mon[32610]: pgmap v7904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:01 smithi077 ceph-mon[37153]: pgmap v7904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:01 smithi047 ceph-mon[36835]: pgmap v7904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:03 smithi029 ceph-mon[32610]: pgmap v7905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:44:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14064.687054 secs 2024-07-27T10:44:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:03 smithi077 ceph-mon[37153]: pgmap v7905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:44:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14064.687054 secs 2024-07-27T10:44:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:03 smithi047 ceph-mon[36835]: pgmap v7905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:44:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14064.687054 secs 2024-07-27T10:44:04.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:44:04.172+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:44:04.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:44:04.172+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:44:04.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:44:04.172+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:44:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:05 smithi029 ceph-mon[32610]: pgmap v7906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:05 smithi077 ceph-mon[37153]: pgmap v7906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:05 smithi047 ceph-mon[36835]: pgmap v7906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:07 smithi077 ceph-mon[37153]: pgmap v7907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:07 smithi047 ceph-mon[36835]: pgmap v7907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:07 smithi029 ceph-mon[32610]: pgmap v7907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14069.687153 secs 2024-07-27T10:44:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14069.687153 secs 2024-07-27T10:44:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14069.687153 secs 2024-07-27T10:44:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:09 smithi077 ceph-mon[37153]: pgmap v7908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:44:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:09 smithi047 ceph-mon[36835]: pgmap v7908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:44:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:09 smithi029 ceph-mon[32610]: pgmap v7908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:44:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:11 smithi077 ceph-mon[37153]: pgmap v7909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:11 smithi047 ceph-mon[36835]: pgmap v7909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:11 smithi029 ceph-mon[32610]: pgmap v7909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:13 smithi077 ceph-mon[37153]: pgmap v7910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:44:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14074.687278 secs 2024-07-27T10:44:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:13 smithi047 ceph-mon[36835]: pgmap v7910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:44:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14074.687278 secs 2024-07-27T10:44:13.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:13 smithi029 ceph-mon[32610]: pgmap v7910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:44:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14074.687278 secs 2024-07-27T10:44:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:15 smithi077 ceph-mon[37153]: pgmap v7911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:15 smithi047 ceph-mon[36835]: pgmap v7911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:15 smithi029 ceph-mon[32610]: pgmap v7911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:17 smithi077 ceph-mon[37153]: pgmap v7912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:17 smithi047 ceph-mon[36835]: pgmap v7912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:17.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:17 smithi029 ceph-mon[32610]: pgmap v7912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14079.687403 secs 2024-07-27T10:44:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14079.687403 secs 2024-07-27T10:44:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14079.687403 secs 2024-07-27T10:44:19.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:44:19.188+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:44:19.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:44:19.188+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:44:19.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:44:19.188+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:44:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:19 smithi077 ceph-mon[37153]: pgmap v7913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:44:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:19 smithi047 ceph-mon[36835]: pgmap v7913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:44:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:19 smithi029 ceph-mon[32610]: pgmap v7913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:44:21.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:21 smithi047 ceph-mon[36835]: pgmap v7914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:21 smithi077 ceph-mon[37153]: pgmap v7914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:21.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:21 smithi029 ceph-mon[32610]: pgmap v7914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:23 smithi077 ceph-mon[37153]: pgmap v7915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:44:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14084.687525 secs 2024-07-27T10:44:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:23 smithi047 ceph-mon[36835]: pgmap v7915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:44:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14084.687525 secs 2024-07-27T10:44:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:23 smithi029 ceph-mon[32610]: pgmap v7915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:44:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14084.687525 secs 2024-07-27T10:44:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:25 smithi077 ceph-mon[37153]: pgmap v7916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:25 smithi047 ceph-mon[36835]: pgmap v7916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:25.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:25 smithi029 ceph-mon[32610]: pgmap v7916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:44:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:44:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:44:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:44:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:44:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:44:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:44:26.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:44:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:44:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:44:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:44:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:44:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:27 smithi077 ceph-mon[37153]: pgmap v7917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:27 smithi077 ceph-mon[37153]: pgmap v7918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:44:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:27 smithi077 ceph-mon[37153]: pgmap v7919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:44:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:27 smithi047 ceph-mon[36835]: pgmap v7917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:27 smithi047 ceph-mon[36835]: pgmap v7918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:44:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:27 smithi047 ceph-mon[36835]: pgmap v7919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:44:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:27 smithi029 ceph-mon[32610]: pgmap v7917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:27 smithi029 ceph-mon[32610]: pgmap v7918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:44:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:27 smithi029 ceph-mon[32610]: pgmap v7919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:44:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14089.687646 secs 2024-07-27T10:44:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14089.687646 secs 2024-07-27T10:44:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14089.687646 secs 2024-07-27T10:44:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:29 smithi077 ceph-mon[37153]: pgmap v7920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:44:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:29 smithi047 ceph-mon[36835]: pgmap v7920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:44:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:29 smithi029 ceph-mon[32610]: pgmap v7920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:44:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:31 smithi077 ceph-mon[37153]: pgmap v7921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:44:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:31 smithi047 ceph-mon[36835]: pgmap v7921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:44:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:31 smithi029 ceph-mon[32610]: pgmap v7921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:44:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14094.687819 secs 2024-07-27T10:44:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:33 smithi077 ceph-mon[37153]: pgmap v7922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:44:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14094.687819 secs 2024-07-27T10:44:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:33 smithi047 ceph-mon[36835]: pgmap v7922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:44:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14094.687819 secs 2024-07-27T10:44:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:33 smithi029 ceph-mon[32610]: pgmap v7922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:44:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:44:34.203+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:44:34.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:44:34.203+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:44:34.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:44:34.203+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:44:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:35 smithi077 ceph-mon[37153]: pgmap v7923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:44:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:35 smithi047 ceph-mon[36835]: pgmap v7923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:44:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:35 smithi029 ceph-mon[32610]: pgmap v7923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:44:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:37 smithi077 ceph-mon[37153]: pgmap v7924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:44:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:37 smithi047 ceph-mon[36835]: pgmap v7924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:44:37.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:37 smithi029 ceph-mon[32610]: pgmap v7924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:44:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14099.687930 secs 2024-07-27T10:44:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14099.687930 secs 2024-07-27T10:44:38.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14099.687930 secs 2024-07-27T10:44:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:39 smithi077 ceph-mon[37153]: pgmap v7925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:39 smithi047 ceph-mon[36835]: pgmap v7925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:39 smithi029 ceph-mon[32610]: pgmap v7925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:41 smithi077 ceph-mon[37153]: pgmap v7926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:41 smithi047 ceph-mon[36835]: pgmap v7926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:41.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:41 smithi029 ceph-mon[32610]: pgmap v7926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14104.688037 secs 2024-07-27T10:44:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:43 smithi077 ceph-mon[37153]: pgmap v7927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:44:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14104.688037 secs 2024-07-27T10:44:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:43 smithi047 ceph-mon[36835]: pgmap v7927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:44:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14104.688037 secs 2024-07-27T10:44:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:43 smithi029 ceph-mon[32610]: pgmap v7927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:44:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:45 smithi077 ceph-mon[37153]: pgmap v7928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:44:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:45 smithi047 ceph-mon[36835]: pgmap v7928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:44:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:45 smithi029 ceph-mon[32610]: pgmap v7928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:44:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:47 smithi077 ceph-mon[37153]: pgmap v7929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:44:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:47 smithi047 ceph-mon[36835]: pgmap v7929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:44:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:47 smithi029 ceph-mon[32610]: pgmap v7929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:44:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14109.688156 secs 2024-07-27T10:44:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14109.688156 secs 2024-07-27T10:44:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14109.688156 secs 2024-07-27T10:44:49.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:44:49.203+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:44:49.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:44:49.203+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:44:49.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:44:49.203+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:44:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:49 smithi077 ceph-mon[37153]: pgmap v7930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:49 smithi047 ceph-mon[36835]: pgmap v7930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:49 smithi029 ceph-mon[32610]: pgmap v7930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:51 smithi077 ceph-mon[37153]: pgmap v7931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:51 smithi047 ceph-mon[36835]: pgmap v7931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:51.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:51 smithi029 ceph-mon[32610]: pgmap v7931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:44:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14114.688309 secs 2024-07-27T10:44:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:53 smithi077 ceph-mon[37153]: pgmap v7932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:44:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14114.688309 secs 2024-07-27T10:44:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:53 smithi047 ceph-mon[36835]: pgmap v7932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:44:53.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14114.688309 secs 2024-07-27T10:44:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:53 smithi029 ceph-mon[32610]: pgmap v7932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:44:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:55 smithi077 ceph-mon[37153]: pgmap v7933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:44:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:55 smithi047 ceph-mon[36835]: pgmap v7933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:44:55.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:55 smithi029 ceph-mon[32610]: pgmap v7933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:44:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:57 smithi077 ceph-mon[37153]: pgmap v7934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:44:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:57 smithi047 ceph-mon[36835]: pgmap v7934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:44:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:57 smithi029 ceph-mon[32610]: pgmap v7934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:44:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14119.688454 secs 2024-07-27T10:44:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14119.688454 secs 2024-07-27T10:44:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14119.688454 secs 2024-07-27T10:44:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:44:59 smithi077 ceph-mon[37153]: pgmap v7935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:44:59 smithi047 ceph-mon[36835]: pgmap v7935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:44:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:44:59 smithi029 ceph-mon[32610]: pgmap v7935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:45:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:01 smithi077 ceph-mon[37153]: pgmap v7936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:45:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:01 smithi047 ceph-mon[36835]: pgmap v7936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:45:01.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:01 smithi029 ceph-mon[32610]: pgmap v7936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:45:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14124.688594 secs 2024-07-27T10:45:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:03 smithi077 ceph-mon[37153]: pgmap v7937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14124.688594 secs 2024-07-27T10:45:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:03 smithi047 ceph-mon[36835]: pgmap v7937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14124.688594 secs 2024-07-27T10:45:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:03 smithi029 ceph-mon[32610]: pgmap v7937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:04.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:45:04.219+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:45:04.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:45:04.219+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:45:04.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:45:04.219+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:45:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:05 smithi077 ceph-mon[37153]: pgmap v7938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:45:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:05 smithi047 ceph-mon[36835]: pgmap v7938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:45:05.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:05 smithi029 ceph-mon[32610]: pgmap v7938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:45:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:07 smithi077 ceph-mon[37153]: pgmap v7939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:07 smithi047 ceph-mon[36835]: pgmap v7939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:07 smithi029 ceph-mon[32610]: pgmap v7939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14129.688714 secs 2024-07-27T10:45:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14129.688714 secs 2024-07-27T10:45:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14129.688714 secs 2024-07-27T10:45:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:09 smithi077 ceph-mon[37153]: pgmap v7940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:45:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:09 smithi047 ceph-mon[36835]: pgmap v7940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:45:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:09 smithi029 ceph-mon[32610]: pgmap v7940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:45:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:11 smithi077 ceph-mon[37153]: pgmap v7941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:45:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:11 smithi047 ceph-mon[36835]: pgmap v7941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:45:11.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:11 smithi029 ceph-mon[32610]: pgmap v7941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:45:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14134.688876 secs 2024-07-27T10:45:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:13 smithi047 ceph-mon[36835]: pgmap v7942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14134.688876 secs 2024-07-27T10:45:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:13 smithi029 ceph-mon[32610]: pgmap v7942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14134.688876 secs 2024-07-27T10:45:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:13 smithi077 ceph-mon[37153]: pgmap v7942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:15 smithi047 ceph-mon[36835]: pgmap v7943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:45:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:15 smithi029 ceph-mon[32610]: pgmap v7943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:45:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:15 smithi077 ceph-mon[37153]: pgmap v7943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:45:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:17 smithi047 ceph-mon[36835]: pgmap v7944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:17.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:17 smithi029 ceph-mon[32610]: pgmap v7944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:17 smithi077 ceph-mon[37153]: pgmap v7944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14139.688995 secs 2024-07-27T10:45:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14139.688995 secs 2024-07-27T10:45:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14139.688995 secs 2024-07-27T10:45:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:45:19.220+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:45:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:45:19.220+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:45:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:45:19.220+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:45:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:19 smithi047 ceph-mon[36835]: pgmap v7945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:45:19.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:19 smithi029 ceph-mon[32610]: pgmap v7945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:45:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:19 smithi077 ceph-mon[37153]: pgmap v7945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:45:21.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:21 smithi077 ceph-mon[37153]: pgmap v7946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:45:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:21 smithi047 ceph-mon[36835]: pgmap v7946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:45:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:21 smithi029 ceph-mon[32610]: pgmap v7946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:45:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14144.689126 secs 2024-07-27T10:45:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:23 smithi047 ceph-mon[36835]: pgmap v7947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:45:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14144.689126 secs 2024-07-27T10:45:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:23 smithi029 ceph-mon[32610]: pgmap v7947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:45:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14144.689126 secs 2024-07-27T10:45:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:23 smithi077 ceph-mon[37153]: pgmap v7947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:45:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:25 smithi047 ceph-mon[36835]: pgmap v7948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:45:25.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:25 smithi029 ceph-mon[32610]: pgmap v7948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:45:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:25 smithi077 ceph-mon[37153]: pgmap v7948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:45:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:45:26.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:45:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:45:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:27 smithi047 ceph-mon[36835]: pgmap v7949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:27.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:27 smithi029 ceph-mon[32610]: pgmap v7949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:27 smithi077 ceph-mon[37153]: pgmap v7949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:28.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14149.689228 secs 2024-07-27T10:45:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14149.689228 secs 2024-07-27T10:45:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14149.689228 secs 2024-07-27T10:45:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:29 smithi047 ceph-mon[36835]: pgmap v7950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:45:29.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:29 smithi029 ceph-mon[32610]: pgmap v7950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:45:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:29 smithi077 ceph-mon[37153]: pgmap v7950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:45:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: pgmap v7951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:45:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:45:31.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:45:31.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:45:31.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:45:31.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:45:31.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:45:31.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:45:31.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:45:31.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: pgmap v7952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:45:31.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: pgmap v7953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-27T10:45:31.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: pgmap v7951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:45:31.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:45:31.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:45:31.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:45:31.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:45:31.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:45:31.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:45:31.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:45:31.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:45:31.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: pgmap v7952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:45:31.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: pgmap v7953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-27T10:45:31.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: pgmap v7951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:45:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:45:31.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:45:31.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:45:31.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:45:31.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:45:31.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:45:31.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:45:31.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:45:31.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: pgmap v7952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:45:31.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: pgmap v7953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-07-27T10:45:31.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:45:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14154.689329 secs 2024-07-27T10:45:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:33 smithi077 ceph-mon[37153]: pgmap v7954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14154.689329 secs 2024-07-27T10:45:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:33 smithi047 ceph-mon[36835]: pgmap v7954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14154.689329 secs 2024-07-27T10:45:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:33 smithi029 ceph-mon[32610]: pgmap v7954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:34.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:45:34.236+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:45:34.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:45:34.236+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:45:34.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:45:34.236+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:45:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:35 smithi077 ceph-mon[37153]: pgmap v7955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T10:45:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:35 smithi047 ceph-mon[36835]: pgmap v7955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T10:45:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:35 smithi029 ceph-mon[32610]: pgmap v7955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T10:45:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:37 smithi077 ceph-mon[37153]: pgmap v7956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-27T10:45:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:37 smithi047 ceph-mon[36835]: pgmap v7956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-27T10:45:37.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:37 smithi029 ceph-mon[32610]: pgmap v7956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-27T10:45:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14159.689397 secs 2024-07-27T10:45:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14159.689397 secs 2024-07-27T10:45:38.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14159.689397 secs 2024-07-27T10:45:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:39 smithi077 ceph-mon[37153]: pgmap v7957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:39 smithi047 ceph-mon[36835]: pgmap v7957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:39 smithi029 ceph-mon[32610]: pgmap v7957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:41 smithi077 ceph-mon[37153]: pgmap v7958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:45:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:41 smithi047 ceph-mon[36835]: pgmap v7958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:45:41.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:41 smithi029 ceph-mon[32610]: pgmap v7958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:45:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14164.689567 secs 2024-07-27T10:45:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:43 smithi077 ceph-mon[37153]: pgmap v7959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14164.689567 secs 2024-07-27T10:45:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:43 smithi047 ceph-mon[36835]: pgmap v7959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14164.689567 secs 2024-07-27T10:45:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:43 smithi029 ceph-mon[32610]: pgmap v7959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:45 smithi029 ceph-mon[32610]: pgmap v7960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:45:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:45 smithi077 ceph-mon[37153]: pgmap v7960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:45:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:45 smithi047 ceph-mon[36835]: pgmap v7960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:45:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:47 smithi077 ceph-mon[37153]: pgmap v7961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:47 smithi047 ceph-mon[36835]: pgmap v7961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:47.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:47 smithi029 ceph-mon[32610]: pgmap v7961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14169.689694 secs 2024-07-27T10:45:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14169.689694 secs 2024-07-27T10:45:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14169.689694 secs 2024-07-27T10:45:49.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:45:49.251+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:45:49.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:45:49.251+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:45:49.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:45:49.251+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:45:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:49 smithi077 ceph-mon[37153]: pgmap v7962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:45:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:49 smithi047 ceph-mon[36835]: pgmap v7962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:45:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:49 smithi029 ceph-mon[32610]: pgmap v7962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:45:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:51 smithi077 ceph-mon[37153]: pgmap v7963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:45:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:51 smithi047 ceph-mon[36835]: pgmap v7963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:45:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:51 smithi029 ceph-mon[32610]: pgmap v7963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:45:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14174.689829 secs 2024-07-27T10:45:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:53 smithi077 ceph-mon[37153]: pgmap v7964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14174.689829 secs 2024-07-27T10:45:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:53 smithi047 ceph-mon[36835]: pgmap v7964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14174.689829 secs 2024-07-27T10:45:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:53 smithi029 ceph-mon[32610]: pgmap v7964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:45:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:55 smithi077 ceph-mon[37153]: pgmap v7965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:45:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:55 smithi047 ceph-mon[36835]: pgmap v7965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:45:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:55 smithi029 ceph-mon[32610]: pgmap v7965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:45:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:57 smithi077 ceph-mon[37153]: pgmap v7966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:57 smithi047 ceph-mon[36835]: pgmap v7966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:57 smithi029 ceph-mon[32610]: pgmap v7966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:45:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14179.689950 secs 2024-07-27T10:45:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14179.689950 secs 2024-07-27T10:45:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14179.689950 secs 2024-07-27T10:45:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:45:59 smithi077 ceph-mon[37153]: pgmap v7967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:45:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:45:59 smithi047 ceph-mon[36835]: pgmap v7967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:45:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:45:59 smithi029 ceph-mon[32610]: pgmap v7967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:46:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:01 smithi077 ceph-mon[37153]: pgmap v7968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:46:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:01 smithi047 ceph-mon[36835]: pgmap v7968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:46:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:01 smithi029 ceph-mon[32610]: pgmap v7968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:46:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14184.690072 secs 2024-07-27T10:46:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:03 smithi077 ceph-mon[37153]: pgmap v7969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:46:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14184.690072 secs 2024-07-27T10:46:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:03 smithi047 ceph-mon[36835]: pgmap v7969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:46:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14184.690072 secs 2024-07-27T10:46:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:03 smithi029 ceph-mon[32610]: pgmap v7969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:46:04.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:04.262+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:46:04.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:04.262+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:46:04.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:04.262+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:46:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:05 smithi077 ceph-mon[37153]: pgmap v7970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:46:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:05 smithi047 ceph-mon[36835]: pgmap v7970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:46:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:05 smithi029 ceph-mon[32610]: pgmap v7970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:46:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:07 smithi077 ceph-mon[37153]: pgmap v7971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:07 smithi047 ceph-mon[36835]: pgmap v7971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:07 smithi029 ceph-mon[32610]: pgmap v7971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14189.690212 secs 2024-07-27T10:46:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:08 smithi077 ceph-mon[37153]: pgmap v7972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:46:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14189.690212 secs 2024-07-27T10:46:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:08 smithi047 ceph-mon[36835]: pgmap v7972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:46:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14189.690212 secs 2024-07-27T10:46:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:08 smithi029 ceph-mon[32610]: pgmap v7972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:46:11.811 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:11 smithi029 ceph-mon[32610]: pgmap v7973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:46:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:11 smithi077 ceph-mon[37153]: pgmap v7973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:46:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:11 smithi047 ceph-mon[36835]: pgmap v7973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:46:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14194.690312 secs 2024-07-27T10:46:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:13 smithi077 ceph-mon[37153]: pgmap v7974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:46:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14194.690312 secs 2024-07-27T10:46:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:13 smithi047 ceph-mon[36835]: pgmap v7974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:46:13.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14194.690312 secs 2024-07-27T10:46:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:13 smithi029 ceph-mon[32610]: pgmap v7974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:46:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:15 smithi077 ceph-mon[37153]: pgmap v7975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:46:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:15 smithi047 ceph-mon[36835]: pgmap v7975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:46:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:15 smithi029 ceph-mon[32610]: pgmap v7975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:46:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:17 smithi077 ceph-mon[37153]: pgmap v7976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:17 smithi047 ceph-mon[36835]: pgmap v7976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:17 smithi029 ceph-mon[32610]: pgmap v7976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14199.690452 secs 2024-07-27T10:46:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14199.690452 secs 2024-07-27T10:46:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14199.690452 secs 2024-07-27T10:46:19.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:19.275+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:46:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:19.275+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:46:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:19.275+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:46:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:19 smithi077 ceph-mon[37153]: pgmap v7977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:46:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:19 smithi047 ceph-mon[36835]: pgmap v7977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:46:19.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:19 smithi029 ceph-mon[32610]: pgmap v7977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:46:21.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:21 smithi077 ceph-mon[37153]: pgmap v7978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:46:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:21 smithi047 ceph-mon[36835]: pgmap v7978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:46:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:21 smithi029 ceph-mon[32610]: pgmap v7978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:46:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14204.690572 secs 2024-07-27T10:46:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:23 smithi077 ceph-mon[37153]: pgmap v7979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:46:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14204.690572 secs 2024-07-27T10:46:23.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:23 smithi047 ceph-mon[36835]: pgmap v7979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:46:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14204.690572 secs 2024-07-27T10:46:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:23 smithi029 ceph-mon[32610]: pgmap v7979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:46:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:25 smithi077 ceph-mon[37153]: pgmap v7980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:46:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:25 smithi047 ceph-mon[36835]: pgmap v7980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:46:25.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:25 smithi029 ceph-mon[32610]: pgmap v7980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:46:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:27 smithi077 ceph-mon[37153]: pgmap v7981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:27 smithi047 ceph-mon[36835]: pgmap v7981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:27.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:27 smithi029 ceph-mon[32610]: pgmap v7981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14209.690710 secs 2024-07-27T10:46:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14209.690710 secs 2024-07-27T10:46:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14209.690710 secs 2024-07-27T10:46:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:29 smithi077 ceph-mon[37153]: pgmap v7982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:46:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:29 smithi047 ceph-mon[36835]: pgmap v7982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:46:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:29 smithi029 ceph-mon[32610]: pgmap v7982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:46:31.702 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:31 smithi077 ceph-mon[37153]: pgmap v7983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:46:31.702 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:46:31.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:31.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:31.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:31.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:31 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:31 smithi047 ceph-mon[36835]: pgmap v7983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:46:31.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:46:31.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:31.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:31.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:31.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:31 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:31 smithi029 ceph-mon[32610]: pgmap v7983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:46:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:46:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:31.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:31 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:32.823 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:32 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:46:32.823 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:32 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:46:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:32 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:46:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:32 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:46:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:32 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:46:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:32 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:46:33.637 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14214.690833 secs 2024-07-27T10:46:33.637 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:33 smithi077 ceph-mon[37153]: pgmap v7984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:46:33.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14214.690833 secs 2024-07-27T10:46:33.864 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:33 smithi029 ceph-mon[32610]: pgmap v7984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:46:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14214.690833 secs 2024-07-27T10:46:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:33 smithi047 ceph-mon[36835]: pgmap v7984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:46:34.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:34.291+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:46:34.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:34.291+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:46:34.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:34.291+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:46:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:46:35.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:46:35.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:35.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:35.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:35.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:35.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:46:35.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:46:35.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:46:35.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:46:35.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: pgmap v7985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-27T10:46:35.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: pgmap v7986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:35.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:46:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:46:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:46:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:46:35.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:46:35.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:46:35.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:46:35.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: pgmap v7985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-27T10:46:35.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: pgmap v7986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:35.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:46:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:46:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:46:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:46:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:46:35.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:46:35.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:46:35.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:46:35.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: pgmap v7985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-27T10:46:35.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: pgmap v7986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:35.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:46:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:37 smithi029 ceph-mon[32610]: pgmap v7987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:37 smithi077 ceph-mon[37153]: pgmap v7987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:37 smithi047 ceph-mon[36835]: pgmap v7987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14219.690952 secs 2024-07-27T10:46:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14219.690952 secs 2024-07-27T10:46:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14219.690952 secs 2024-07-27T10:46:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:39 smithi029 ceph-mon[32610]: pgmap v7988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:39 smithi077 ceph-mon[37153]: pgmap v7988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:39 smithi047 ceph-mon[36835]: pgmap v7988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:41 smithi077 ceph-mon[37153]: pgmap v7989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:41 smithi047 ceph-mon[36835]: pgmap v7989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:41 smithi029 ceph-mon[32610]: pgmap v7989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:43.836 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14224.691095 secs 2024-07-27T10:46:43.837 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:43 smithi077 ceph-mon[37153]: pgmap v7990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14224.691095 secs 2024-07-27T10:46:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:43 smithi047 ceph-mon[36835]: pgmap v7990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14224.691095 secs 2024-07-27T10:46:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:43 smithi029 ceph-mon[32610]: pgmap v7990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T10:46:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:45 smithi077 ceph-mon[37153]: pgmap v7991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:46:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:45 smithi047 ceph-mon[36835]: pgmap v7991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:46:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:45 smithi029 ceph-mon[32610]: pgmap v7991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:46:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:47 smithi077 ceph-mon[37153]: pgmap v7992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:47 smithi047 ceph-mon[36835]: pgmap v7992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:47 smithi029 ceph-mon[32610]: pgmap v7992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14229.691216 secs 2024-07-27T10:46:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14229.691216 secs 2024-07-27T10:46:48.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14229.691216 secs 2024-07-27T10:46:49.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:49.305+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:46:49.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:49.305+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:46:49.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:49.305+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:46:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:49 smithi029 ceph-mon[32610]: pgmap v7993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:46:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:49 smithi077 ceph-mon[37153]: pgmap v7993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:46:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:49 smithi047 ceph-mon[36835]: pgmap v7993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:46:51.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:51 smithi047 ceph-mon[36835]: pgmap v7994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:46:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:51 smithi077 ceph-mon[37153]: pgmap v7994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:46:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:51 smithi029 ceph-mon[32610]: pgmap v7994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:46:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14234.691342 secs 2024-07-27T10:46:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:53 smithi077 ceph-mon[37153]: pgmap v7995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:46:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14234.691342 secs 2024-07-27T10:46:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:53 smithi047 ceph-mon[36835]: pgmap v7995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:46:53.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14234.691342 secs 2024-07-27T10:46:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:53 smithi029 ceph-mon[32610]: pgmap v7995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:46:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:55 smithi077 ceph-mon[37153]: pgmap v7996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:46:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:55 smithi047 ceph-mon[36835]: pgmap v7996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:46:55.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:55 smithi029 ceph-mon[32610]: pgmap v7996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:46:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:57 smithi077 ceph-mon[37153]: pgmap v7997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:57 smithi047 ceph-mon[36835]: pgmap v7997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:57 smithi029 ceph-mon[32610]: pgmap v7997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:46:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14239.691463 secs 2024-07-27T10:46:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14239.691463 secs 2024-07-27T10:46:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14239.691463 secs 2024-07-27T10:46:59.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:59.797+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:46:59.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:59.797+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:46:59.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:46:59 smithi029 ceph-mon[32610]: pgmap v7998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:46:59.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:59.797+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:46:59.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:46:59.797+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:46:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:46:59 smithi077 ceph-mon[37153]: pgmap v7998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:46:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:46:59 smithi047 ceph-mon[36835]: pgmap v7998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:01 smithi077 ceph-mon[37153]: pgmap v7999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:01.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:01 smithi029 ceph-mon[32610]: pgmap v7999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:01 smithi047 ceph-mon[36835]: pgmap v7999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14244.691632 secs 2024-07-27T10:47:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:03 smithi077 ceph-mon[37153]: pgmap v8000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:47:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14244.691632 secs 2024-07-27T10:47:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:03 smithi047 ceph-mon[36835]: pgmap v8000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:47:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14244.691632 secs 2024-07-27T10:47:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:03 smithi029 ceph-mon[32610]: pgmap v8000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:47:04.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:47:04.321+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:47:04.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:47:04.321+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:47:04.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:47:04.321+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:47:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:05 smithi077 ceph-mon[37153]: pgmap v8001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:47:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:05 smithi047 ceph-mon[36835]: pgmap v8001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:47:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:05 smithi029 ceph-mon[32610]: pgmap v8001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:47:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:07 smithi077 ceph-mon[37153]: pgmap v8002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:47:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:07 smithi047 ceph-mon[36835]: pgmap v8002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:47:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:07 smithi029 ceph-mon[32610]: pgmap v8002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:47:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14249.691747 secs 2024-07-27T10:47:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:08 smithi077 ceph-mon[37153]: pgmap v8003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14249.691747 secs 2024-07-27T10:47:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:08 smithi047 ceph-mon[36835]: pgmap v8003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14249.691747 secs 2024-07-27T10:47:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:08 smithi029 ceph-mon[32610]: pgmap v8003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:11 smithi077 ceph-mon[37153]: pgmap v8004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:11 smithi047 ceph-mon[36835]: pgmap v8004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:11 smithi029 ceph-mon[32610]: pgmap v8004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14254.691870 secs 2024-07-27T10:47:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:13 smithi077 ceph-mon[37153]: pgmap v8005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:47:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14254.691870 secs 2024-07-27T10:47:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:13 smithi047 ceph-mon[36835]: pgmap v8005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:47:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14254.691870 secs 2024-07-27T10:47:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:13 smithi029 ceph-mon[32610]: pgmap v8005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:47:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:15 smithi077 ceph-mon[37153]: pgmap v8006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:47:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:15 smithi047 ceph-mon[36835]: pgmap v8006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:47:15.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:15 smithi029 ceph-mon[32610]: pgmap v8006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:47:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:17 smithi029 ceph-mon[32610]: pgmap v8007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:47:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:17 smithi077 ceph-mon[37153]: pgmap v8007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:47:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:17 smithi047 ceph-mon[36835]: pgmap v8007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:47:18.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14259.692006 secs 2024-07-27T10:47:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14259.692006 secs 2024-07-27T10:47:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14259.692006 secs 2024-07-27T10:47:19.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:47:19.330+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:47:19.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:47:19.330+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:47:19.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:47:19.330+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:47:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:19 smithi029 ceph-mon[32610]: pgmap v8008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:19 smithi077 ceph-mon[37153]: pgmap v8008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:19 smithi047 ceph-mon[36835]: pgmap v8008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:21.749 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:21 smithi047 ceph-mon[36835]: pgmap v8009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:21 smithi077 ceph-mon[37153]: pgmap v8009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:21 smithi029 ceph-mon[32610]: pgmap v8009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14264.692101 secs 2024-07-27T10:47:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:23 smithi077 ceph-mon[37153]: pgmap v8010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:47:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14264.692101 secs 2024-07-27T10:47:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:23 smithi047 ceph-mon[36835]: pgmap v8010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:47:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14264.692101 secs 2024-07-27T10:47:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:23 smithi029 ceph-mon[32610]: pgmap v8010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:47:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:25 smithi077 ceph-mon[37153]: pgmap v8011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:47:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:25 smithi047 ceph-mon[36835]: pgmap v8011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:47:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:25 smithi029 ceph-mon[32610]: pgmap v8011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:47:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:27 smithi077 ceph-mon[37153]: pgmap v8012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:47:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:27 smithi047 ceph-mon[36835]: pgmap v8012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:47:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:27 smithi029 ceph-mon[32610]: pgmap v8012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:47:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14269.692177 secs 2024-07-27T10:47:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14269.692177 secs 2024-07-27T10:47:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14269.692177 secs 2024-07-27T10:47:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:29 smithi077 ceph-mon[37153]: pgmap v8013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:29 smithi047 ceph-mon[36835]: pgmap v8013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:29 smithi029 ceph-mon[32610]: pgmap v8013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:31 smithi077 ceph-mon[37153]: pgmap v8014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:31 smithi047 ceph-mon[36835]: pgmap v8014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:31 smithi029 ceph-mon[32610]: pgmap v8014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14274.692252 secs 2024-07-27T10:47:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:33 smithi077 ceph-mon[37153]: pgmap v8015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:47:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14274.692252 secs 2024-07-27T10:47:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:33 smithi047 ceph-mon[36835]: pgmap v8015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:47:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14274.692252 secs 2024-07-27T10:47:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:33 smithi029 ceph-mon[32610]: pgmap v8015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:47:34.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:47:34.332+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:47:34.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:47:34.332+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:47:34.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:47:34.332+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:47:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:35 smithi077 ceph-mon[37153]: pgmap v8016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:47:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:47:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:47:35.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:47:35.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:47:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:35 smithi047 ceph-mon[36835]: pgmap v8016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:47:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:47:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:47:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:47:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:47:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:35 smithi029 ceph-mon[32610]: pgmap v8016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:47:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:47:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:47:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:47:35.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:47:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:36 smithi077 ceph-mon[37153]: pgmap v8017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:47:36.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:36 smithi077 ceph-mon[37153]: pgmap v8018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:47:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:36 smithi047 ceph-mon[36835]: pgmap v8017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:47:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:36 smithi047 ceph-mon[36835]: pgmap v8018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:47:36.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:36 smithi029 ceph-mon[32610]: pgmap v8017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:47:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:36 smithi029 ceph-mon[32610]: pgmap v8018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:47:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:38 smithi077 ceph-mon[37153]: pgmap v8019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:47:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14279.692382 secs 2024-07-27T10:47:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:38 smithi047 ceph-mon[36835]: pgmap v8019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:47:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14279.692382 secs 2024-07-27T10:47:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:38 smithi029 ceph-mon[32610]: pgmap v8019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:47:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14279.692382 secs 2024-07-27T10:47:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:40 smithi077 ceph-mon[37153]: pgmap v8020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:47:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:40 smithi047 ceph-mon[36835]: pgmap v8020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:47:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:40 smithi029 ceph-mon[32610]: pgmap v8020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:47:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:42 smithi077 ceph-mon[37153]: pgmap v8021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:47:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:42 smithi047 ceph-mon[36835]: pgmap v8021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:47:42.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:42 smithi029 ceph-mon[32610]: pgmap v8021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:47:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14284.692504 secs 2024-07-27T10:47:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14284.692504 secs 2024-07-27T10:47:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14284.692504 secs 2024-07-27T10:47:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:44 smithi077 ceph-mon[37153]: pgmap v8022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:47:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:44 smithi047 ceph-mon[36835]: pgmap v8022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:47:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:44 smithi029 ceph-mon[32610]: pgmap v8022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:47:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:46 smithi077 ceph-mon[37153]: pgmap v8023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:47:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:46 smithi047 ceph-mon[36835]: pgmap v8023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:47:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:46 smithi029 ceph-mon[32610]: pgmap v8023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:47:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:48 smithi077 ceph-mon[37153]: pgmap v8024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:47:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14289.692636 secs 2024-07-27T10:47:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:48 smithi047 ceph-mon[36835]: pgmap v8024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:47:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14289.692636 secs 2024-07-27T10:47:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:48 smithi029 ceph-mon[32610]: pgmap v8024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:47:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14289.692636 secs 2024-07-27T10:47:49.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:47:49.347+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:47:49.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:47:49.347+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:47:49.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:47:49.347+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:47:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:50 smithi077 ceph-mon[37153]: pgmap v8025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:50 smithi047 ceph-mon[36835]: pgmap v8025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:50 smithi029 ceph-mon[32610]: pgmap v8025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:47:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:52 smithi077 ceph-mon[37153]: pgmap v8026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:52 smithi047 ceph-mon[36835]: pgmap v8026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:52.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:52 smithi029 ceph-mon[32610]: pgmap v8026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:47:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14294.692746 secs 2024-07-27T10:47:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14294.692746 secs 2024-07-27T10:47:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14294.692746 secs 2024-07-27T10:47:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:54 smithi077 ceph-mon[37153]: pgmap v8027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:47:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:54 smithi047 ceph-mon[36835]: pgmap v8027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:47:54.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:54 smithi029 ceph-mon[32610]: pgmap v8027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:47:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:56 smithi077 ceph-mon[37153]: pgmap v8028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:47:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:56 smithi047 ceph-mon[36835]: pgmap v8028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:47:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:56 smithi029 ceph-mon[32610]: pgmap v8028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:47:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14299.692824 secs 2024-07-27T10:47:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:47:58 smithi077 ceph-mon[37153]: pgmap v8029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:47:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14299.692824 secs 2024-07-27T10:47:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:47:58 smithi047 ceph-mon[36835]: pgmap v8029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:47:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14299.692824 secs 2024-07-27T10:47:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:47:58 smithi029 ceph-mon[32610]: pgmap v8029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:48:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:00 smithi077 ceph-mon[37153]: pgmap v8030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:48:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:00 smithi047 ceph-mon[36835]: pgmap v8030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:48:00.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:00 smithi029 ceph-mon[32610]: pgmap v8030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T10:48:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:02 smithi077 ceph-mon[37153]: pgmap v8031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:48:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:02 smithi047 ceph-mon[36835]: pgmap v8031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:48:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:02 smithi029 ceph-mon[32610]: pgmap v8031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:48:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14304.692949 secs 2024-07-27T10:48:03.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14304.692949 secs 2024-07-27T10:48:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14304.692949 secs 2024-07-27T10:48:04.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:48:04.362+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:48:04.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:48:04.362+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:48:04.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:48:04.362+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:48:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:04 smithi077 ceph-mon[37153]: pgmap v8032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T10:48:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:04 smithi047 ceph-mon[36835]: pgmap v8032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T10:48:04.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:04 smithi029 ceph-mon[32610]: pgmap v8032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-27T10:48:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:06 smithi077 ceph-mon[37153]: pgmap v8033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:48:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:06 smithi047 ceph-mon[36835]: pgmap v8033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:48:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:06 smithi029 ceph-mon[32610]: pgmap v8033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:48:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14309.693093 secs 2024-07-27T10:48:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:08 smithi077 ceph-mon[37153]: pgmap v8034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:48:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14309.693093 secs 2024-07-27T10:48:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:08 smithi047 ceph-mon[36835]: pgmap v8034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:48:08.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14309.693093 secs 2024-07-27T10:48:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:08 smithi029 ceph-mon[32610]: pgmap v8034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:48:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:10 smithi077 ceph-mon[37153]: pgmap v8035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:48:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:10 smithi047 ceph-mon[36835]: pgmap v8035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:48:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:10 smithi029 ceph-mon[32610]: pgmap v8035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:48:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:12 smithi077 ceph-mon[37153]: pgmap v8036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:12 smithi047 ceph-mon[36835]: pgmap v8036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:12.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:12 smithi029 ceph-mon[32610]: pgmap v8036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14314.693211 secs 2024-07-27T10:48:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14314.693211 secs 2024-07-27T10:48:13.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14314.693211 secs 2024-07-27T10:48:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:14 smithi077 ceph-mon[37153]: pgmap v8037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:48:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:14 smithi047 ceph-mon[36835]: pgmap v8037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:48:14.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:14 smithi029 ceph-mon[32610]: pgmap v8037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:48:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:16 smithi077 ceph-mon[37153]: pgmap v8038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:16 smithi047 ceph-mon[36835]: pgmap v8038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:16 smithi029 ceph-mon[32610]: pgmap v8038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14319.693366 secs 2024-07-27T10:48:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:18 smithi077 ceph-mon[37153]: pgmap v8039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:48:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14319.693366 secs 2024-07-27T10:48:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:18 smithi047 ceph-mon[36835]: pgmap v8039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:48:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14319.693366 secs 2024-07-27T10:48:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:18 smithi029 ceph-mon[32610]: pgmap v8039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:48:19.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:48:19.378+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:48:19.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:48:19.378+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:48:19.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:48:19.378+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:48:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:20 smithi077 ceph-mon[37153]: pgmap v8040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:48:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:20 smithi047 ceph-mon[36835]: pgmap v8040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:48:20.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:20 smithi029 ceph-mon[32610]: pgmap v8040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:48:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:22 smithi047 ceph-mon[36835]: pgmap v8041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:22 smithi029 ceph-mon[32610]: pgmap v8041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:22 smithi077 ceph-mon[37153]: pgmap v8041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14324.693470 secs 2024-07-27T10:48:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14324.693470 secs 2024-07-27T10:48:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14324.693470 secs 2024-07-27T10:48:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:24 smithi047 ceph-mon[36835]: pgmap v8042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:48:24.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:24 smithi029 ceph-mon[32610]: pgmap v8042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:48:25.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:24 smithi077 ceph-mon[37153]: pgmap v8042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:48:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:26 smithi047 ceph-mon[36835]: pgmap v8043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:26 smithi029 ceph-mon[32610]: pgmap v8043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:26 smithi077 ceph-mon[37153]: pgmap v8043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14329.693591 secs 2024-07-27T10:48:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:28 smithi047 ceph-mon[36835]: pgmap v8044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:48:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14329.693591 secs 2024-07-27T10:48:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:28 smithi029 ceph-mon[32610]: pgmap v8044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:48:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14329.693591 secs 2024-07-27T10:48:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:28 smithi077 ceph-mon[37153]: pgmap v8044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:48:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:30 smithi047 ceph-mon[36835]: pgmap v8045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:48:30.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:30 smithi029 ceph-mon[32610]: pgmap v8045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:48:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:30 smithi077 ceph-mon[37153]: pgmap v8045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:48:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:32 smithi047 ceph-mon[36835]: pgmap v8046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:32 smithi029 ceph-mon[32610]: pgmap v8046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:32 smithi077 ceph-mon[37153]: pgmap v8046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14334.693718 secs 2024-07-27T10:48:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14334.693718 secs 2024-07-27T10:48:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14334.693718 secs 2024-07-27T10:48:34.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:48:34.382+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:48:34.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:48:34.383+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:48:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:48:34.383+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:48:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:34 smithi047 ceph-mon[36835]: pgmap v8047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:48:34.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:34 smithi029 ceph-mon[32610]: pgmap v8047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:48:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:34 smithi077 ceph-mon[37153]: pgmap v8047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:48:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:48:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:48:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:48:35.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:48:35.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:48:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:48:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:48:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:48:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:48:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:48:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:48:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:48:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:36 smithi047 ceph-mon[36835]: pgmap v8048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:36 smithi047 ceph-mon[36835]: pgmap v8049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:48:36.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:36 smithi047 ceph-mon[36835]: pgmap v8050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:48:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:36 smithi029 ceph-mon[32610]: pgmap v8048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:36 smithi029 ceph-mon[32610]: pgmap v8049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:48:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:36 smithi029 ceph-mon[32610]: pgmap v8050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:48:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:36 smithi077 ceph-mon[37153]: pgmap v8048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:36 smithi077 ceph-mon[37153]: pgmap v8049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:48:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:36 smithi077 ceph-mon[37153]: pgmap v8050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:48:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14339.693839 secs 2024-07-27T10:48:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:38 smithi047 ceph-mon[36835]: pgmap v8051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:48:38.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14339.693839 secs 2024-07-27T10:48:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:38 smithi029 ceph-mon[32610]: pgmap v8051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:48:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14339.693839 secs 2024-07-27T10:48:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:38 smithi077 ceph-mon[37153]: pgmap v8051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:48:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:40 smithi047 ceph-mon[36835]: pgmap v8052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:48:40.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:40 smithi029 ceph-mon[32610]: pgmap v8052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:48:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:40 smithi077 ceph-mon[37153]: pgmap v8052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:48:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:42 smithi047 ceph-mon[36835]: pgmap v8053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:48:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:42 smithi029 ceph-mon[32610]: pgmap v8053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:48:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:42 smithi077 ceph-mon[37153]: pgmap v8053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:48:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14344.693894 secs 2024-07-27T10:48:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14344.693894 secs 2024-07-27T10:48:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14344.693894 secs 2024-07-27T10:48:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:44 smithi047 ceph-mon[36835]: pgmap v8054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:48:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:44 smithi029 ceph-mon[32610]: pgmap v8054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:48:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:44 smithi077 ceph-mon[37153]: pgmap v8054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:48:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:46 smithi047 ceph-mon[36835]: pgmap v8055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:48:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:46 smithi029 ceph-mon[32610]: pgmap v8055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:48:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:46 smithi077 ceph-mon[37153]: pgmap v8055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:48:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14349.694012 secs 2024-07-27T10:48:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:48 smithi047 ceph-mon[36835]: pgmap v8056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:48:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14349.694012 secs 2024-07-27T10:48:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:48 smithi029 ceph-mon[32610]: pgmap v8056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:48:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14349.694012 secs 2024-07-27T10:48:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:48 smithi077 ceph-mon[37153]: pgmap v8056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:48:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:48:49.393+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:48:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:48:49.393+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:48:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:48:49.393+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:48:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:50 smithi047 ceph-mon[36835]: pgmap v8057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:48:50.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:50 smithi029 ceph-mon[32610]: pgmap v8057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:48:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:50 smithi077 ceph-mon[37153]: pgmap v8057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:48:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:52 smithi047 ceph-mon[36835]: pgmap v8058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:48:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:52 smithi029 ceph-mon[32610]: pgmap v8058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:48:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:52 smithi077 ceph-mon[37153]: pgmap v8058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:48:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14354.694129 secs 2024-07-27T10:48:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14354.694129 secs 2024-07-27T10:48:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14354.694129 secs 2024-07-27T10:48:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:54 smithi047 ceph-mon[36835]: pgmap v8059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:48:54.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:54 smithi029 ceph-mon[32610]: pgmap v8059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:48:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:54 smithi077 ceph-mon[37153]: pgmap v8059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:48:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:56 smithi047 ceph-mon[36835]: pgmap v8060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:56 smithi029 ceph-mon[32610]: pgmap v8060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:56 smithi077 ceph-mon[37153]: pgmap v8060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:48:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14359.694225 secs 2024-07-27T10:48:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:48:58 smithi047 ceph-mon[36835]: pgmap v8061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:48:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14359.694225 secs 2024-07-27T10:48:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:48:58 smithi029 ceph-mon[32610]: pgmap v8061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:48:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14359.694225 secs 2024-07-27T10:48:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:48:58 smithi077 ceph-mon[37153]: pgmap v8061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:00 smithi047 ceph-mon[36835]: pgmap v8062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:49:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:00 smithi029 ceph-mon[32610]: pgmap v8062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:49:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:00 smithi077 ceph-mon[37153]: pgmap v8062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:49:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:02 smithi047 ceph-mon[36835]: pgmap v8063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:02 smithi029 ceph-mon[32610]: pgmap v8063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:02 smithi077 ceph-mon[37153]: pgmap v8063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14364.694359 secs 2024-07-27T10:49:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14364.694359 secs 2024-07-27T10:49:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14364.694359 secs 2024-07-27T10:49:04.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:04.396+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:49:04.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:04.396+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:49:04.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:04.396+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:49:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:04 smithi029 ceph-mon[32610]: pgmap v8064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:04 smithi077 ceph-mon[37153]: pgmap v8064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:04 smithi047 ceph-mon[36835]: pgmap v8064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:06 smithi029 ceph-mon[32610]: pgmap v8065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:06 smithi077 ceph-mon[37153]: pgmap v8065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:06 smithi047 ceph-mon[36835]: pgmap v8065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14369.694513 secs 2024-07-27T10:49:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:08 smithi029 ceph-mon[32610]: pgmap v8066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14369.694513 secs 2024-07-27T10:49:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:08 smithi077 ceph-mon[37153]: pgmap v8066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14369.694513 secs 2024-07-27T10:49:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:08 smithi047 ceph-mon[36835]: pgmap v8066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:10 smithi029 ceph-mon[32610]: pgmap v8067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:49:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:10 smithi077 ceph-mon[37153]: pgmap v8067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:49:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:10 smithi047 ceph-mon[36835]: pgmap v8067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:49:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:12 smithi077 ceph-mon[37153]: pgmap v8068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:12 smithi047 ceph-mon[36835]: pgmap v8068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:12 smithi029 ceph-mon[32610]: pgmap v8068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14374.694653 secs 2024-07-27T10:49:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14374.694653 secs 2024-07-27T10:49:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14374.694653 secs 2024-07-27T10:49:14.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:14.818+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:49:14.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:14.818+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:49:14.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:14.818+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:49:14.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:14.818+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:49:15.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:14 smithi077 ceph-mon[37153]: pgmap v8069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:14 smithi047 ceph-mon[36835]: pgmap v8069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:14 smithi029 ceph-mon[32610]: pgmap v8069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:16 smithi077 ceph-mon[37153]: pgmap v8070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:16 smithi047 ceph-mon[36835]: pgmap v8070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:17.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:16 smithi029 ceph-mon[32610]: pgmap v8070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14379.694821 secs 2024-07-27T10:49:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:18 smithi077 ceph-mon[37153]: pgmap v8071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14379.694821 secs 2024-07-27T10:49:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:18 smithi047 ceph-mon[36835]: pgmap v8071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14379.694821 secs 2024-07-27T10:49:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:18 smithi029 ceph-mon[32610]: pgmap v8071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:19.412+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:49:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:19.412+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:49:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:19.412+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:49:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:20 smithi077 ceph-mon[37153]: pgmap v8072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:49:21.191 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:20 smithi029 ceph-mon[32610]: pgmap v8072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:49:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:20 smithi047 ceph-mon[36835]: pgmap v8072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:49:22.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:22 smithi047 ceph-mon[36835]: pgmap v8073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:22 smithi077 ceph-mon[37153]: pgmap v8073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:22 smithi029 ceph-mon[32610]: pgmap v8073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14384.694921 secs 2024-07-27T10:49:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14384.694921 secs 2024-07-27T10:49:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14384.694921 secs 2024-07-27T10:49:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:24 smithi077 ceph-mon[37153]: pgmap v8074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:24 smithi047 ceph-mon[36835]: pgmap v8074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:24 smithi029 ceph-mon[32610]: pgmap v8074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:26 smithi077 ceph-mon[37153]: pgmap v8075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:26 smithi047 ceph-mon[36835]: pgmap v8075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:26 smithi029 ceph-mon[32610]: pgmap v8075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14389.695061 secs 2024-07-27T10:49:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:28 smithi077 ceph-mon[37153]: pgmap v8076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14389.695061 secs 2024-07-27T10:49:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:28 smithi047 ceph-mon[36835]: pgmap v8076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14389.695061 secs 2024-07-27T10:49:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:28 smithi029 ceph-mon[32610]: pgmap v8076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:30 smithi077 ceph-mon[37153]: pgmap v8077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:49:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:30 smithi047 ceph-mon[36835]: pgmap v8077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:49:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:30 smithi029 ceph-mon[32610]: pgmap v8077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:49:33.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:32 smithi029 ceph-mon[32610]: pgmap v8078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:32 smithi077 ceph-mon[37153]: pgmap v8078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:32 smithi047 ceph-mon[36835]: pgmap v8078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14394.695142 secs 2024-07-27T10:49:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14394.695142 secs 2024-07-27T10:49:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14394.695142 secs 2024-07-27T10:49:34.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:34.416+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:49:34.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:34.416+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:49:34.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:34.416+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:49:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:34 smithi077 ceph-mon[37153]: pgmap v8079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:34 smithi047 ceph-mon[36835]: pgmap v8079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:34 smithi029 ceph-mon[32610]: pgmap v8079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:36.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:35 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:49:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:35 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:49:36.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:35 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:49:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:36 smithi047 ceph-mon[36835]: pgmap v8080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:36 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:36 smithi029 ceph-mon[32610]: pgmap v8080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:36 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:36 smithi077 ceph-mon[37153]: pgmap v8080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:37.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:36 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14399.695214 secs 2024-07-27T10:49:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:38 smithi077 ceph-mon[37153]: pgmap v8081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:49:38.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:49:38.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14399.695214 secs 2024-07-27T10:49:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:38 smithi047 ceph-mon[36835]: pgmap v8081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:49:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:49:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14399.695214 secs 2024-07-27T10:49:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:38 smithi029 ceph-mon[32610]: pgmap v8081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:49:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:49:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:49:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:49:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:39 smithi077 ceph-mon[37153]: pgmap v8082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:49:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:39 smithi077 ceph-mon[37153]: pgmap v8083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:49:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:39 smithi047 ceph-mon[36835]: pgmap v8082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:49:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:39 smithi047 ceph-mon[36835]: pgmap v8083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:49:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:39 smithi029 ceph-mon[32610]: pgmap v8082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:49:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:39 smithi029 ceph-mon[32610]: pgmap v8083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:49:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:41 smithi077 ceph-mon[37153]: pgmap v8084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:49:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:41 smithi047 ceph-mon[36835]: pgmap v8084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:49:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:41 smithi029 ceph-mon[32610]: pgmap v8084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:49:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14404.695334 secs 2024-07-27T10:49:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:43 smithi077 ceph-mon[37153]: pgmap v8085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:49:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14404.695334 secs 2024-07-27T10:49:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:43 smithi047 ceph-mon[36835]: pgmap v8085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:49:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14404.695334 secs 2024-07-27T10:49:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:43 smithi029 ceph-mon[32610]: pgmap v8085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:49:45.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:45 smithi077 ceph-mon[37153]: pgmap v8086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:45 smithi047 ceph-mon[36835]: pgmap v8086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:45 smithi029 ceph-mon[32610]: pgmap v8086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:47 smithi077 ceph-mon[37153]: pgmap v8087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T10:49:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:47 smithi047 ceph-mon[36835]: pgmap v8087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T10:49:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:47 smithi029 ceph-mon[32610]: pgmap v8087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T10:49:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14409.695441 secs 2024-07-27T10:49:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14409.695441 secs 2024-07-27T10:49:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14409.695441 secs 2024-07-27T10:49:49.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:49.429+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:49:49.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:49.429+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:49:49.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:49.429+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:49:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:49 smithi047 ceph-mon[36835]: pgmap v8088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:49:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:49 smithi029 ceph-mon[32610]: pgmap v8088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:49:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:49 smithi077 ceph-mon[37153]: pgmap v8088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:49:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:51 smithi047 ceph-mon[36835]: pgmap v8089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:51 smithi029 ceph-mon[32610]: pgmap v8089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:51 smithi077 ceph-mon[37153]: pgmap v8089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:49:53.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14414.695570 secs 2024-07-27T10:49:53.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:53 smithi077 ceph-mon[37153]: pgmap v8090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14414.695570 secs 2024-07-27T10:49:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:53 smithi047 ceph-mon[36835]: pgmap v8090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14414.695570 secs 2024-07-27T10:49:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:53 smithi029 ceph-mon[32610]: pgmap v8090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:55 smithi047 ceph-mon[36835]: pgmap v8091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:49:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:55 smithi029 ceph-mon[32610]: pgmap v8091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:49:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:55 smithi077 ceph-mon[37153]: pgmap v8091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:49:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:57.667+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T10:49:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:57.667+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T10:49:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:57.667+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7eff400049e0 tx=0x7eff40038040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T10:49:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:57.667+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7eff400049e0 tx=0x7eff40038040 comp rx=0 tx=0).stop 2024-07-27T10:49:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:57.667+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T10:49:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:57.667+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T10:49:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:57.667+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T10:49:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:57.667+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T10:49:57.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:57 smithi029 ceph-mon[32610]: pgmap v8092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:57.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:57.667+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T10:49:57.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:57.667+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7eff40004a10 tx=0x7eff4002b070 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T10:49:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:57 smithi047 ceph-mon[36835]: pgmap v8092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:57 smithi077 ceph-mon[37153]: pgmap v8092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:49:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14419.695689 secs 2024-07-27T10:49:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14419.695689 secs 2024-07-27T10:49:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14419.695689 secs 2024-07-27T10:49:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:49:59 smithi047 ceph-mon[36835]: pgmap v8093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:49:59 smithi029 ceph-mon[32610]: pgmap v8093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:49:59.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:59.826+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:49:59.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:59.826+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:49:59.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:59.826+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:49:59.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:49:59.826+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:49:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:49:59 smithi077 ceph-mon[37153]: pgmap v8093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:50:00.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:50:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T10:50:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:01 smithi047 ceph-mon[36835]: pgmap v8094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:01.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:01 smithi029 ceph-mon[32610]: pgmap v8094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:01 smithi077 ceph-mon[37153]: pgmap v8094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14424.695821 secs 2024-07-27T10:50:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:03 smithi047 ceph-mon[36835]: pgmap v8095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14424.695821 secs 2024-07-27T10:50:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:03 smithi029 ceph-mon[32610]: pgmap v8095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14424.695821 secs 2024-07-27T10:50:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:03 smithi077 ceph-mon[37153]: pgmap v8095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:04.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:50:04.436+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:50:04.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:50:04.436+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:50:04.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:50:04.437+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:50:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:05 smithi047 ceph-mon[36835]: pgmap v8096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:50:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:05 smithi029 ceph-mon[32610]: pgmap v8096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:50:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:05 smithi077 ceph-mon[37153]: pgmap v8096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:50:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:07 smithi047 ceph-mon[36835]: pgmap v8097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:07 smithi029 ceph-mon[32610]: pgmap v8097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:07 smithi077 ceph-mon[37153]: pgmap v8097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14429.695858 secs 2024-07-27T10:50:09.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:08 smithi047 ceph-mon[36835]: pgmap v8098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14429.695858 secs 2024-07-27T10:50:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:08 smithi029 ceph-mon[32610]: pgmap v8098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14429.695858 secs 2024-07-27T10:50:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:08 smithi077 ceph-mon[37153]: pgmap v8098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:11 smithi077 ceph-mon[37153]: pgmap v8099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:11 smithi047 ceph-mon[36835]: pgmap v8099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:11.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:11 smithi029 ceph-mon[32610]: pgmap v8099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14434.695936 secs 2024-07-27T10:50:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:13 smithi077 ceph-mon[37153]: pgmap v8100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14434.695936 secs 2024-07-27T10:50:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:13 smithi047 ceph-mon[36835]: pgmap v8100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14434.695936 secs 2024-07-27T10:50:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:13 smithi029 ceph-mon[32610]: pgmap v8100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:15 smithi077 ceph-mon[37153]: pgmap v8101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:50:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:15 smithi047 ceph-mon[36835]: pgmap v8101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:50:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:15 smithi029 ceph-mon[32610]: pgmap v8101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:50:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:17 smithi077 ceph-mon[37153]: pgmap v8102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:17 smithi047 ceph-mon[36835]: pgmap v8102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:17.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:17 smithi029 ceph-mon[32610]: pgmap v8102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14439.696049 secs 2024-07-27T10:50:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14439.696049 secs 2024-07-27T10:50:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14439.696049 secs 2024-07-27T10:50:19.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:50:19.439+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:50:19.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:50:19.439+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:50:19.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:50:19.439+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:50:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:19 smithi077 ceph-mon[37153]: pgmap v8103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:19 smithi047 ceph-mon[36835]: pgmap v8103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:19 smithi029 ceph-mon[32610]: pgmap v8103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:21 smithi077 ceph-mon[37153]: pgmap v8104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:21 smithi047 ceph-mon[36835]: pgmap v8104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:21 smithi029 ceph-mon[32610]: pgmap v8104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14444.696172 secs 2024-07-27T10:50:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:23 smithi077 ceph-mon[37153]: pgmap v8105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14444.696172 secs 2024-07-27T10:50:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:23 smithi047 ceph-mon[36835]: pgmap v8105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14444.696172 secs 2024-07-27T10:50:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:23 smithi029 ceph-mon[32610]: pgmap v8105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:25 smithi077 ceph-mon[37153]: pgmap v8106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:50:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:25 smithi047 ceph-mon[36835]: pgmap v8106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:50:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:25 smithi029 ceph-mon[32610]: pgmap v8106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:50:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:27 smithi077 ceph-mon[37153]: pgmap v8107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:27 smithi047 ceph-mon[36835]: pgmap v8107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:27 smithi029 ceph-mon[32610]: pgmap v8107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14449.696246 secs 2024-07-27T10:50:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14449.696246 secs 2024-07-27T10:50:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14449.696246 secs 2024-07-27T10:50:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:29 smithi047 ceph-mon[36835]: pgmap v8108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:29 smithi029 ceph-mon[32610]: pgmap v8108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:29.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:29 smithi077 ceph-mon[37153]: pgmap v8108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:31 smithi047 ceph-mon[36835]: pgmap v8109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:31.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:31 smithi029 ceph-mon[32610]: pgmap v8109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:31 smithi077 ceph-mon[37153]: pgmap v8109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14454.696398 secs 2024-07-27T10:50:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:33 smithi047 ceph-mon[36835]: pgmap v8110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14454.696398 secs 2024-07-27T10:50:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:33 smithi029 ceph-mon[32610]: pgmap v8110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14454.696398 secs 2024-07-27T10:50:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:33 smithi077 ceph-mon[37153]: pgmap v8110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:50:34.455+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:50:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:50:34.455+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:50:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:50:34.455+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:50:35.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:35 smithi047 ceph-mon[36835]: pgmap v8111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:50:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:35 smithi029 ceph-mon[32610]: pgmap v8111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:50:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:35 smithi077 ceph-mon[37153]: pgmap v8111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:50:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:37 smithi047 ceph-mon[36835]: pgmap v8112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:37.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:37 smithi029 ceph-mon[32610]: pgmap v8112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:37 smithi077 ceph-mon[37153]: pgmap v8112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:50:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14459.696500 secs 2024-07-27T10:50:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:50:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14459.696500 secs 2024-07-27T10:50:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:50:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14459.696500 secs 2024-07-27T10:50:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:50:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:39 smithi047 ceph-mon[36835]: pgmap v8113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:50:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:50:39.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:39 smithi029 ceph-mon[32610]: pgmap v8113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:39.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:39.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:39.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:50:39.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:50:39.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:39 smithi077 ceph-mon[37153]: pgmap v8113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:50:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:50:40.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:50:40.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:50:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:40 smithi029 ceph-mon[32610]: pgmap v8114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:50:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:40 smithi029 ceph-mon[32610]: pgmap v8115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:50:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:40 smithi077 ceph-mon[37153]: pgmap v8114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:50:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:40 smithi077 ceph-mon[37153]: pgmap v8115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:50:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:40 smithi047 ceph-mon[36835]: pgmap v8114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:50:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:40 smithi047 ceph-mon[36835]: pgmap v8115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:50:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:42 smithi077 ceph-mon[37153]: pgmap v8116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:50:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:42 smithi047 ceph-mon[36835]: pgmap v8116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:50:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:42 smithi029 ceph-mon[32610]: pgmap v8116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:50:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14464.696641 secs 2024-07-27T10:50:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14464.696641 secs 2024-07-27T10:50:44.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14464.696641 secs 2024-07-27T10:50:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:44 smithi077 ceph-mon[37153]: pgmap v8117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:50:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:44 smithi047 ceph-mon[36835]: pgmap v8117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:50:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:44 smithi029 ceph-mon[32610]: pgmap v8117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T10:50:47.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:46 smithi077 ceph-mon[37153]: pgmap v8118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:50:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:46 smithi047 ceph-mon[36835]: pgmap v8118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:50:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:46 smithi029 ceph-mon[32610]: pgmap v8118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:50:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14469.696778 secs 2024-07-27T10:50:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:48 smithi077 ceph-mon[37153]: pgmap v8119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:50:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14469.696778 secs 2024-07-27T10:50:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:48 smithi047 ceph-mon[36835]: pgmap v8119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:50:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14469.696778 secs 2024-07-27T10:50:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:48 smithi029 ceph-mon[32610]: pgmap v8119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:50:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:50:49.469+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:50:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:50:49.469+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:50:49.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:50:49.469+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:50:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:50 smithi077 ceph-mon[37153]: pgmap v8120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:50:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:50 smithi047 ceph-mon[36835]: pgmap v8120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:50:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:50 smithi029 ceph-mon[32610]: pgmap v8120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:50:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:52 smithi077 ceph-mon[37153]: pgmap v8121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:52 smithi047 ceph-mon[36835]: pgmap v8121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:52 smithi029 ceph-mon[32610]: pgmap v8121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:50:54.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14474.696898 secs 2024-07-27T10:50:54.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14474.696898 secs 2024-07-27T10:50:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14474.696898 secs 2024-07-27T10:50:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:54 smithi077 ceph-mon[37153]: pgmap v8122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:50:55.113 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:54 smithi047 ceph-mon[36835]: pgmap v8122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:50:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:54 smithi029 ceph-mon[32610]: pgmap v8122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:50:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:56 smithi077 ceph-mon[37153]: pgmap v8123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:50:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:56 smithi047 ceph-mon[36835]: pgmap v8123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:50:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:56 smithi029 ceph-mon[32610]: pgmap v8123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:50:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14479.697018 secs 2024-07-27T10:50:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:50:58 smithi077 ceph-mon[37153]: pgmap v8124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:50:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14479.697018 secs 2024-07-27T10:50:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:50:58 smithi047 ceph-mon[36835]: pgmap v8124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:50:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14479.697018 secs 2024-07-27T10:50:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:50:58 smithi029 ceph-mon[32610]: pgmap v8124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:00 smithi077 ceph-mon[37153]: pgmap v8125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:00 smithi047 ceph-mon[36835]: pgmap v8125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:00 smithi029 ceph-mon[32610]: pgmap v8125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:03.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:02 smithi029 ceph-mon[32610]: pgmap v8126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:02 smithi077 ceph-mon[37153]: pgmap v8126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:02 smithi047 ceph-mon[36835]: pgmap v8126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14484.697137 secs 2024-07-27T10:51:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14484.697137 secs 2024-07-27T10:51:04.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14484.697137 secs 2024-07-27T10:51:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:51:04.477+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:51:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:51:04.477+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:51:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:51:04.477+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:51:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:04 smithi077 ceph-mon[37153]: pgmap v8127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:51:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:04 smithi047 ceph-mon[36835]: pgmap v8127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:51:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:04 smithi029 ceph-mon[32610]: pgmap v8127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:51:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:06 smithi077 ceph-mon[37153]: pgmap v8128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:06 smithi047 ceph-mon[36835]: pgmap v8128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:07.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:06 smithi029 ceph-mon[32610]: pgmap v8128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14489.697236 secs 2024-07-27T10:51:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:08 smithi077 ceph-mon[37153]: pgmap v8129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14489.697236 secs 2024-07-27T10:51:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:08 smithi047 ceph-mon[36835]: pgmap v8129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:09.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14489.697236 secs 2024-07-27T10:51:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:08 smithi029 ceph-mon[32610]: pgmap v8129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:10 smithi077 ceph-mon[37153]: pgmap v8130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:10 smithi047 ceph-mon[36835]: pgmap v8130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:10 smithi029 ceph-mon[32610]: pgmap v8130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:12 smithi077 ceph-mon[37153]: pgmap v8131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:12 smithi047 ceph-mon[36835]: pgmap v8131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:12 smithi029 ceph-mon[32610]: pgmap v8131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14494.697357 secs 2024-07-27T10:51:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14494.697357 secs 2024-07-27T10:51:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14494.697357 secs 2024-07-27T10:51:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:14 smithi077 ceph-mon[37153]: pgmap v8132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:51:15.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:14 smithi047 ceph-mon[36835]: pgmap v8132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:51:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:14 smithi029 ceph-mon[32610]: pgmap v8132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:51:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:16 smithi077 ceph-mon[37153]: pgmap v8133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:16 smithi047 ceph-mon[36835]: pgmap v8133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:16 smithi029 ceph-mon[32610]: pgmap v8133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14499.697476 secs 2024-07-27T10:51:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:18 smithi077 ceph-mon[37153]: pgmap v8134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14499.697476 secs 2024-07-27T10:51:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:18 smithi047 ceph-mon[36835]: pgmap v8134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:19.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14499.697476 secs 2024-07-27T10:51:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:18 smithi029 ceph-mon[32610]: pgmap v8134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:19.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:51:19.493+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:51:19.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:51:19.493+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:51:19.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:51:19.493+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:51:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:20 smithi077 ceph-mon[37153]: pgmap v8135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:20 smithi047 ceph-mon[36835]: pgmap v8135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:21.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:20 smithi029 ceph-mon[32610]: pgmap v8135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:22 smithi077 ceph-mon[37153]: pgmap v8136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:22 smithi047 ceph-mon[36835]: pgmap v8136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:22 smithi029 ceph-mon[32610]: pgmap v8136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:24.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14504.697618 secs 2024-07-27T10:51:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14504.697618 secs 2024-07-27T10:51:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14504.697618 secs 2024-07-27T10:51:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:24 smithi077 ceph-mon[37153]: pgmap v8137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:51:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:24 smithi047 ceph-mon[36835]: pgmap v8137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:51:25.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:24 smithi029 ceph-mon[32610]: pgmap v8137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:51:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:26 smithi077 ceph-mon[37153]: pgmap v8138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:26 smithi047 ceph-mon[36835]: pgmap v8138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:27.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:26 smithi029 ceph-mon[32610]: pgmap v8138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14509.697765 secs 2024-07-27T10:51:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:28 smithi077 ceph-mon[37153]: pgmap v8139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14509.697765 secs 2024-07-27T10:51:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:28 smithi047 ceph-mon[36835]: pgmap v8139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:29.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14509.697765 secs 2024-07-27T10:51:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:28 smithi029 ceph-mon[32610]: pgmap v8139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:30 smithi077 ceph-mon[37153]: pgmap v8140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:30 smithi047 ceph-mon[36835]: pgmap v8140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:31.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:30 smithi029 ceph-mon[32610]: pgmap v8140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:32 smithi077 ceph-mon[37153]: pgmap v8141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:32 smithi047 ceph-mon[36835]: pgmap v8141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:32 smithi029 ceph-mon[32610]: pgmap v8141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14514.697862 secs 2024-07-27T10:51:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14514.697862 secs 2024-07-27T10:51:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14514.697862 secs 2024-07-27T10:51:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:51:34.509+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:51:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:51:34.509+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:51:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:51:34.509+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:51:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:34 smithi077 ceph-mon[37153]: pgmap v8142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:51:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:34 smithi047 ceph-mon[36835]: pgmap v8142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:51:35.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:34 smithi029 ceph-mon[32610]: pgmap v8142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:51:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:36 smithi077 ceph-mon[37153]: pgmap v8143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:51:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:36 smithi047 ceph-mon[36835]: pgmap v8143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:51:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:36 smithi029 ceph-mon[32610]: pgmap v8143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:51:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14519.697956 secs 2024-07-27T10:51:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:38 smithi077 ceph-mon[37153]: pgmap v8144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14519.697956 secs 2024-07-27T10:51:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:38 smithi047 ceph-mon[36835]: pgmap v8144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14519.697956 secs 2024-07-27T10:51:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:38 smithi029 ceph-mon[32610]: pgmap v8144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:51:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:51:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:51:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:51:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:51:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:51:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:51:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:51:40.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:51:40.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:51:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:51:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:51:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:51:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:40 smithi077 ceph-mon[37153]: pgmap v8145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:40 smithi077 ceph-mon[37153]: pgmap v8146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:51:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:40 smithi077 ceph-mon[37153]: pgmap v8147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:51:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:40 smithi047 ceph-mon[36835]: pgmap v8145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:40 smithi047 ceph-mon[36835]: pgmap v8146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:51:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:40 smithi047 ceph-mon[36835]: pgmap v8147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:51:41.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:40 smithi029 ceph-mon[32610]: pgmap v8145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:40 smithi029 ceph-mon[32610]: pgmap v8146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:51:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:40 smithi029 ceph-mon[32610]: pgmap v8147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:51:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:42 smithi077 ceph-mon[37153]: pgmap v8148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:51:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:42 smithi047 ceph-mon[36835]: pgmap v8148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:51:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:42 smithi029 ceph-mon[32610]: pgmap v8148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:51:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14524.698043 secs 2024-07-27T10:51:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14524.698043 secs 2024-07-27T10:51:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14524.698043 secs 2024-07-27T10:51:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:44 smithi077 ceph-mon[37153]: pgmap v8149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:51:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:44 smithi047 ceph-mon[36835]: pgmap v8149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:51:45.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:44 smithi029 ceph-mon[32610]: pgmap v8149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:51:46.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:45 smithi077 ceph-mon[37153]: pgmap v8150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:51:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:45 smithi047 ceph-mon[36835]: pgmap v8150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:51:46.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:45 smithi029 ceph-mon[32610]: pgmap v8150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:51:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14529.698164 secs 2024-07-27T10:51:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:48 smithi077 ceph-mon[37153]: pgmap v8151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:51:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14529.698164 secs 2024-07-27T10:51:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:48 smithi047 ceph-mon[36835]: pgmap v8151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:51:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14529.698164 secs 2024-07-27T10:51:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:48 smithi029 ceph-mon[32610]: pgmap v8151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:51:49.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:51:49.524+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:51:49.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:51:49.524+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:51:49.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:51:49.524+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:51:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:50 smithi077 ceph-mon[37153]: pgmap v8152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:51:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:50 smithi047 ceph-mon[36835]: pgmap v8152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:51:51.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:50 smithi029 ceph-mon[32610]: pgmap v8152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:51:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:52 smithi077 ceph-mon[37153]: pgmap v8153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:51:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:52 smithi047 ceph-mon[36835]: pgmap v8153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:51:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:52 smithi029 ceph-mon[32610]: pgmap v8153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:51:54.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14534.698248 secs 2024-07-27T10:51:54.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14534.698248 secs 2024-07-27T10:51:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14534.698248 secs 2024-07-27T10:51:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:54 smithi077 ceph-mon[37153]: pgmap v8154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:54 smithi047 ceph-mon[36835]: pgmap v8154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:54 smithi029 ceph-mon[32610]: pgmap v8154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:51:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:56 smithi077 ceph-mon[37153]: pgmap v8155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:56 smithi047 ceph-mon[36835]: pgmap v8155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:57.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:56 smithi029 ceph-mon[32610]: pgmap v8155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:51:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14539.698326 secs 2024-07-27T10:51:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:51:58 smithi077 ceph-mon[37153]: pgmap v8156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:51:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14539.698326 secs 2024-07-27T10:51:59.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:51:58 smithi047 ceph-mon[36835]: pgmap v8156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:51:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14539.698326 secs 2024-07-27T10:51:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:51:58 smithi029 ceph-mon[32610]: pgmap v8156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:00 smithi077 ceph-mon[37153]: pgmap v8157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:00 smithi047 ceph-mon[36835]: pgmap v8157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:00 smithi029 ceph-mon[32610]: pgmap v8157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:03.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:02 smithi029 ceph-mon[32610]: pgmap v8158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:02 smithi077 ceph-mon[37153]: pgmap v8158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:02 smithi047 ceph-mon[36835]: pgmap v8158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14544.698447 secs 2024-07-27T10:52:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14544.698447 secs 2024-07-27T10:52:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14544.698447 secs 2024-07-27T10:52:04.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:04.539+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:52:04.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:04.539+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:52:04.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:04.539+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:52:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:04 smithi029 ceph-mon[32610]: pgmap v8159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:04 smithi077 ceph-mon[37153]: pgmap v8159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:04 smithi047 ceph-mon[36835]: pgmap v8159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:07.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:06 smithi077 ceph-mon[37153]: pgmap v8160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:52:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:06 smithi047 ceph-mon[36835]: pgmap v8160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:52:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:06 smithi029 ceph-mon[32610]: pgmap v8160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:52:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14549.698561 secs 2024-07-27T10:52:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:08 smithi077 ceph-mon[37153]: pgmap v8161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14549.698561 secs 2024-07-27T10:52:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:08 smithi047 ceph-mon[36835]: pgmap v8161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14549.698561 secs 2024-07-27T10:52:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:08 smithi029 ceph-mon[32610]: pgmap v8161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:10 smithi077 ceph-mon[37153]: pgmap v8162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:10 smithi047 ceph-mon[36835]: pgmap v8162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:10 smithi029 ceph-mon[32610]: pgmap v8162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:13.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:12 smithi077 ceph-mon[37153]: pgmap v8163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:12 smithi047 ceph-mon[36835]: pgmap v8163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:12 smithi029 ceph-mon[32610]: pgmap v8163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14554.698683 secs 2024-07-27T10:52:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14554.698683 secs 2024-07-27T10:52:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14554.698683 secs 2024-07-27T10:52:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:14 smithi077 ceph-mon[37153]: pgmap v8164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:14 smithi047 ceph-mon[36835]: pgmap v8164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:14 smithi029 ceph-mon[32610]: pgmap v8164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:16 smithi077 ceph-mon[37153]: pgmap v8165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:52:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:16 smithi047 ceph-mon[36835]: pgmap v8165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:52:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:16 smithi029 ceph-mon[32610]: pgmap v8165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:52:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14559.698775 secs 2024-07-27T10:52:19.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:18 smithi077 ceph-mon[37153]: pgmap v8166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14559.698775 secs 2024-07-27T10:52:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:18 smithi047 ceph-mon[36835]: pgmap v8166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14559.698775 secs 2024-07-27T10:52:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:18 smithi029 ceph-mon[32610]: pgmap v8166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:19.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:19.555+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:52:19.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:19.555+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:52:19.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:19.555+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:52:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:20 smithi077 ceph-mon[37153]: pgmap v8167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:20 smithi047 ceph-mon[36835]: pgmap v8167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:20 smithi029 ceph-mon[32610]: pgmap v8167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:22 smithi077 ceph-mon[37153]: pgmap v8168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:22 smithi047 ceph-mon[36835]: pgmap v8168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:22 smithi029 ceph-mon[32610]: pgmap v8168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:24.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14564.698931 secs 2024-07-27T10:52:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14564.698931 secs 2024-07-27T10:52:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14564.698931 secs 2024-07-27T10:52:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:24 smithi077 ceph-mon[37153]: pgmap v8169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:24 smithi047 ceph-mon[36835]: pgmap v8169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:24 smithi029 ceph-mon[32610]: pgmap v8169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:26 smithi077 ceph-mon[37153]: pgmap v8170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:52:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:26 smithi047 ceph-mon[36835]: pgmap v8170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:52:27.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:26 smithi029 ceph-mon[32610]: pgmap v8170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:52:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14569.699028 secs 2024-07-27T10:52:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:28 smithi077 ceph-mon[37153]: pgmap v8171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14569.699028 secs 2024-07-27T10:52:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:28 smithi047 ceph-mon[36835]: pgmap v8171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14569.699028 secs 2024-07-27T10:52:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:28 smithi029 ceph-mon[32610]: pgmap v8171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:31.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:30 smithi077 ceph-mon[37153]: pgmap v8172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:30 smithi047 ceph-mon[36835]: pgmap v8172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:31.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:30 smithi029 ceph-mon[32610]: pgmap v8172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:32 smithi077 ceph-mon[37153]: pgmap v8173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:32 smithi047 ceph-mon[36835]: pgmap v8173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:33.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:32 smithi029 ceph-mon[32610]: pgmap v8173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14574.699155 secs 2024-07-27T10:52:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14574.699155 secs 2024-07-27T10:52:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14574.699155 secs 2024-07-27T10:52:34.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:34.571+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:52:34.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:34.571+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:52:34.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:34.571+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:52:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:34 smithi029 ceph-mon[32610]: pgmap v8174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:34 smithi077 ceph-mon[37153]: pgmap v8174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:35.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:34 smithi047 ceph-mon[36835]: pgmap v8174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:36 smithi077 ceph-mon[37153]: pgmap v8175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:52:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:36 smithi047 ceph-mon[36835]: pgmap v8175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:52:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:36 smithi029 ceph-mon[32610]: pgmap v8175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:52:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14579.699274 secs 2024-07-27T10:52:39.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:38 smithi077 ceph-mon[37153]: pgmap v8176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14579.699274 secs 2024-07-27T10:52:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:38 smithi047 ceph-mon[36835]: pgmap v8176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14579.699274 secs 2024-07-27T10:52:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:38 smithi029 ceph-mon[32610]: pgmap v8176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:52:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:52:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:52:40.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:52:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:40 smithi077 ceph-mon[37153]: pgmap v8177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:41.147 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:52:41.148 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:52:41.148 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:52:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:40 smithi047 ceph-mon[36835]: pgmap v8177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:52:41.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:52:41.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:52:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:40 smithi029 ceph-mon[32610]: pgmap v8177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:52:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:52:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:52:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:52:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:41 smithi077 ceph-mon[37153]: pgmap v8178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:52:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:41 smithi077 ceph-mon[37153]: pgmap v8179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:52:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:41 smithi047 ceph-mon[36835]: pgmap v8178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:52:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:41 smithi047 ceph-mon[36835]: pgmap v8179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:52:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:41 smithi029 ceph-mon[32610]: pgmap v8178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:52:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:41 smithi029 ceph-mon[32610]: pgmap v8179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:52:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14584.699395 secs 2024-07-27T10:52:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:43 smithi077 ceph-mon[37153]: pgmap v8180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:52:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14584.699395 secs 2024-07-27T10:52:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:43 smithi047 ceph-mon[36835]: pgmap v8180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:52:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14584.699395 secs 2024-07-27T10:52:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:43 smithi029 ceph-mon[32610]: pgmap v8180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:52:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:45 smithi077 ceph-mon[37153]: pgmap v8181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:52:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:45 smithi047 ceph-mon[36835]: pgmap v8181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:52:46.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:45 smithi029 ceph-mon[32610]: pgmap v8181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:52:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:47 smithi077 ceph-mon[37153]: pgmap v8182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:52:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:47 smithi047 ceph-mon[36835]: pgmap v8182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:52:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:47 smithi029 ceph-mon[32610]: pgmap v8182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:52:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14589.699517 secs 2024-07-27T10:52:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14589.699517 secs 2024-07-27T10:52:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14589.699517 secs 2024-07-27T10:52:49.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:49.575+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:52:49.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:49.575+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:52:49.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:49.575+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:52:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:49 smithi047 ceph-mon[36835]: pgmap v8183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:52:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:49 smithi029 ceph-mon[32610]: pgmap v8183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:52:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:49 smithi077 ceph-mon[37153]: pgmap v8183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:52:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:50 smithi047 ceph-mon[36835]: pgmap v8184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:52:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:50 smithi029 ceph-mon[32610]: pgmap v8184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:52:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:50 smithi077 ceph-mon[37153]: pgmap v8184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:52:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14594.699634 secs 2024-07-27T10:52:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:53 smithi077 ceph-mon[37153]: pgmap v8185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14594.699634 secs 2024-07-27T10:52:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:53 smithi047 ceph-mon[36835]: pgmap v8185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14594.699634 secs 2024-07-27T10:52:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:53 smithi029 ceph-mon[32610]: pgmap v8185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:55 smithi077 ceph-mon[37153]: pgmap v8186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:52:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:55 smithi047 ceph-mon[36835]: pgmap v8186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:52:55.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:55 smithi029 ceph-mon[32610]: pgmap v8186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:52:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:57 smithi077 ceph-mon[37153]: pgmap v8187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:57 smithi047 ceph-mon[36835]: pgmap v8187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:57 smithi029 ceph-mon[32610]: pgmap v8187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:52:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14599.699749 secs 2024-07-27T10:52:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14599.699749 secs 2024-07-27T10:52:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14599.699749 secs 2024-07-27T10:52:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:52:59 smithi077 ceph-mon[37153]: pgmap v8188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:52:59 smithi047 ceph-mon[36835]: pgmap v8188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:59.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:52:59 smithi029 ceph-mon[32610]: pgmap v8188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:52:59.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:59.853+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:52:59.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:59.853+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:52:59.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:59.853+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:52:59.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:52:59.853+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:53:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:01 smithi077 ceph-mon[37153]: pgmap v8189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:01 smithi047 ceph-mon[36835]: pgmap v8189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:01 smithi029 ceph-mon[32610]: pgmap v8189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14604.699875 secs 2024-07-27T10:53:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:03 smithi077 ceph-mon[37153]: pgmap v8190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14604.699875 secs 2024-07-27T10:53:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:03 smithi047 ceph-mon[36835]: pgmap v8190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14604.699875 secs 2024-07-27T10:53:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:03 smithi029 ceph-mon[32610]: pgmap v8190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:04.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:53:04.591+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:53:04.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:53:04.591+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:53:04.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:53:04.591+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:53:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:05 smithi077 ceph-mon[37153]: pgmap v8191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:53:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:05 smithi047 ceph-mon[36835]: pgmap v8191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:53:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:05 smithi029 ceph-mon[32610]: pgmap v8191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:53:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:07 smithi077 ceph-mon[37153]: pgmap v8192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:07 smithi047 ceph-mon[36835]: pgmap v8192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:07.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:07 smithi029 ceph-mon[32610]: pgmap v8192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14609.699953 secs 2024-07-27T10:53:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14609.699953 secs 2024-07-27T10:53:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14609.699953 secs 2024-07-27T10:53:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:09 smithi077 ceph-mon[37153]: pgmap v8193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:09 smithi047 ceph-mon[36835]: pgmap v8193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:09 smithi029 ceph-mon[32610]: pgmap v8193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:11.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:11 smithi077 ceph-mon[37153]: pgmap v8194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:11 smithi047 ceph-mon[36835]: pgmap v8194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:11 smithi029 ceph-mon[32610]: pgmap v8194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14614.700020 secs 2024-07-27T10:53:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:13 smithi077 ceph-mon[37153]: pgmap v8195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14614.700020 secs 2024-07-27T10:53:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:13 smithi047 ceph-mon[36835]: pgmap v8195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:13.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14614.700020 secs 2024-07-27T10:53:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:13 smithi029 ceph-mon[32610]: pgmap v8195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:15 smithi029 ceph-mon[32610]: pgmap v8196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:53:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:15 smithi077 ceph-mon[37153]: pgmap v8196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:53:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:15 smithi047 ceph-mon[36835]: pgmap v8196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:53:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:17 smithi077 ceph-mon[37153]: pgmap v8197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:17 smithi047 ceph-mon[36835]: pgmap v8197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:17 smithi029 ceph-mon[32610]: pgmap v8197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14619.700112 secs 2024-07-27T10:53:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14619.700112 secs 2024-07-27T10:53:18.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14619.700112 secs 2024-07-27T10:53:19.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:53:19.594+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:53:19.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:53:19.594+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:53:19.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:53:19.594+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:53:19.598 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:19 smithi029 ceph-mon[32610]: pgmap v8198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:19 smithi077 ceph-mon[37153]: pgmap v8198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:19 smithi047 ceph-mon[36835]: pgmap v8198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:21 smithi077 ceph-mon[37153]: pgmap v8199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:21 smithi047 ceph-mon[36835]: pgmap v8199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:21 smithi029 ceph-mon[32610]: pgmap v8199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14624.700225 secs 2024-07-27T10:53:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:23 smithi077 ceph-mon[37153]: pgmap v8200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14624.700225 secs 2024-07-27T10:53:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:23 smithi047 ceph-mon[36835]: pgmap v8200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14624.700225 secs 2024-07-27T10:53:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:23 smithi029 ceph-mon[32610]: pgmap v8200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:25 smithi077 ceph-mon[37153]: pgmap v8201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:53:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:25 smithi047 ceph-mon[36835]: pgmap v8201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:53:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:25 smithi029 ceph-mon[32610]: pgmap v8201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:53:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:27 smithi077 ceph-mon[37153]: pgmap v8202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:27 smithi047 ceph-mon[36835]: pgmap v8202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:27 smithi029 ceph-mon[32610]: pgmap v8202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14629.700360 secs 2024-07-27T10:53:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14629.700360 secs 2024-07-27T10:53:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14629.700360 secs 2024-07-27T10:53:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:29 smithi047 ceph-mon[36835]: pgmap v8203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:29 smithi029 ceph-mon[32610]: pgmap v8203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:29 smithi077 ceph-mon[37153]: pgmap v8203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:31 smithi047 ceph-mon[36835]: pgmap v8204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:31 smithi029 ceph-mon[32610]: pgmap v8204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:31 smithi077 ceph-mon[37153]: pgmap v8204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14634.700505 secs 2024-07-27T10:53:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:33 smithi047 ceph-mon[36835]: pgmap v8205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14634.700505 secs 2024-07-27T10:53:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:33 smithi029 ceph-mon[32610]: pgmap v8205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14634.700505 secs 2024-07-27T10:53:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:33 smithi077 ceph-mon[37153]: pgmap v8205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:34.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:53:34.607+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:53:34.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:53:34.607+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:53:34.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:53:34.607+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:53:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:35 smithi047 ceph-mon[36835]: pgmap v8206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:53:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:35 smithi029 ceph-mon[32610]: pgmap v8206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:53:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:35 smithi077 ceph-mon[37153]: pgmap v8206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:53:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:37 smithi047 ceph-mon[36835]: pgmap v8207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:37.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:37 smithi029 ceph-mon[32610]: pgmap v8207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:37 smithi077 ceph-mon[37153]: pgmap v8207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14639.700627 secs 2024-07-27T10:53:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14639.700627 secs 2024-07-27T10:53:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14639.700627 secs 2024-07-27T10:53:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:39 smithi047 ceph-mon[36835]: pgmap v8208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:39 smithi029 ceph-mon[32610]: pgmap v8208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:39 smithi077 ceph-mon[37153]: pgmap v8208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:40 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:53:40.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:40 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:53:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:40 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:53:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:41 smithi047 ceph-mon[36835]: pgmap v8209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:53:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:53:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:41 smithi047 ceph-mon[36835]: pgmap v8210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:53:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:41 smithi047 ceph-mon[36835]: pgmap v8211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:53:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:53:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:41 smithi029 ceph-mon[32610]: pgmap v8209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:53:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:53:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:41 smithi029 ceph-mon[32610]: pgmap v8210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:53:41.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:41 smithi029 ceph-mon[32610]: pgmap v8211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:53:41.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:53:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:41 smithi077 ceph-mon[37153]: pgmap v8209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:53:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:53:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:53:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:41 smithi077 ceph-mon[37153]: pgmap v8210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T10:53:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:41 smithi077 ceph-mon[37153]: pgmap v8211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:53:41.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:53:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14644.700745 secs 2024-07-27T10:53:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:43 smithi077 ceph-mon[37153]: pgmap v8212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:53:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14644.700745 secs 2024-07-27T10:53:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:43 smithi047 ceph-mon[36835]: pgmap v8212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:53:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14644.700745 secs 2024-07-27T10:53:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:43 smithi029 ceph-mon[32610]: pgmap v8212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:53:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:45 smithi077 ceph-mon[37153]: pgmap v8213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:53:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:45 smithi047 ceph-mon[36835]: pgmap v8213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:53:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:45 smithi029 ceph-mon[32610]: pgmap v8213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:53:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:47 smithi077 ceph-mon[37153]: pgmap v8214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:53:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:47 smithi047 ceph-mon[36835]: pgmap v8214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:53:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:47 smithi029 ceph-mon[32610]: pgmap v8214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:53:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14649.700862 secs 2024-07-27T10:53:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14649.700862 secs 2024-07-27T10:53:49.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14649.700862 secs 2024-07-27T10:53:49.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:53:49.621+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:53:49.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:53:49.621+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:53:49.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:53:49.621+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:53:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:49 smithi029 ceph-mon[32610]: pgmap v8215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:53:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:49 smithi077 ceph-mon[37153]: pgmap v8215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:53:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:49 smithi047 ceph-mon[36835]: pgmap v8215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:53:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:51 smithi077 ceph-mon[37153]: pgmap v8216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:53:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:51 smithi047 ceph-mon[36835]: pgmap v8216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:53:52.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:51 smithi029 ceph-mon[32610]: pgmap v8216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:53:54.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14654.700982 secs 2024-07-27T10:53:54.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:53 smithi047 ceph-mon[36835]: pgmap v8217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:53:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14654.700982 secs 2024-07-27T10:53:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:53 smithi029 ceph-mon[32610]: pgmap v8217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:53:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14654.700982 secs 2024-07-27T10:53:54.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:53 smithi077 ceph-mon[37153]: pgmap v8217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:53:56.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:55 smithi077 ceph-mon[37153]: pgmap v8218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:53:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:55 smithi047 ceph-mon[36835]: pgmap v8218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:53:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:55 smithi029 ceph-mon[32610]: pgmap v8218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:53:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:56 smithi047 ceph-mon[36835]: pgmap v8219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:56 smithi029 ceph-mon[32610]: pgmap v8219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:56 smithi077 ceph-mon[37153]: pgmap v8219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:53:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14659.701104 secs 2024-07-27T10:53:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14659.701104 secs 2024-07-27T10:53:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14659.701104 secs 2024-07-27T10:53:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:53:59 smithi077 ceph-mon[37153]: pgmap v8220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:53:59 smithi047 ceph-mon[36835]: pgmap v8220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:53:59.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:53:59 smithi029 ceph-mon[32610]: pgmap v8220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:01 smithi077 ceph-mon[37153]: pgmap v8221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:01 smithi047 ceph-mon[36835]: pgmap v8221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:01 smithi029 ceph-mon[32610]: pgmap v8221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14664.701183 secs 2024-07-27T10:54:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:03 smithi077 ceph-mon[37153]: pgmap v8222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:54:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14664.701183 secs 2024-07-27T10:54:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:03 smithi047 ceph-mon[36835]: pgmap v8222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:54:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14664.701183 secs 2024-07-27T10:54:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:03 smithi029 ceph-mon[32610]: pgmap v8222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:54:04.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:54:04.623+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:54:04.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:54:04.623+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:54:04.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:54:04.623+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:54:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:05 smithi077 ceph-mon[37153]: pgmap v8223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:54:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:05 smithi047 ceph-mon[36835]: pgmap v8223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:54:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:05 smithi029 ceph-mon[32610]: pgmap v8223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:54:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:07 smithi047 ceph-mon[36835]: pgmap v8224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:07 smithi029 ceph-mon[32610]: pgmap v8224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:07 smithi077 ceph-mon[37153]: pgmap v8224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14669.701243 secs 2024-07-27T10:54:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:08 smithi029 ceph-mon[32610]: pgmap v8225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14669.701243 secs 2024-07-27T10:54:09.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:08 smithi077 ceph-mon[37153]: pgmap v8225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14669.701243 secs 2024-07-27T10:54:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:08 smithi047 ceph-mon[36835]: pgmap v8225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:11 smithi077 ceph-mon[37153]: pgmap v8226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:12.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:11 smithi047 ceph-mon[36835]: pgmap v8226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:12.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:11 smithi029 ceph-mon[32610]: pgmap v8226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14674.701310 secs 2024-07-27T10:54:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:13 smithi077 ceph-mon[37153]: pgmap v8227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:54:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14674.701310 secs 2024-07-27T10:54:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:13 smithi047 ceph-mon[36835]: pgmap v8227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:54:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14674.701310 secs 2024-07-27T10:54:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:13 smithi029 ceph-mon[32610]: pgmap v8227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:54:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:15 smithi077 ceph-mon[37153]: pgmap v8228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:54:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:15 smithi047 ceph-mon[36835]: pgmap v8228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:54:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:15 smithi029 ceph-mon[32610]: pgmap v8228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:54:18.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:17 smithi029 ceph-mon[32610]: pgmap v8229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:17 smithi047 ceph-mon[36835]: pgmap v8229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:17 smithi077 ceph-mon[37153]: pgmap v8229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14679.701428 secs 2024-07-27T10:54:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14679.701428 secs 2024-07-27T10:54:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14679.701428 secs 2024-07-27T10:54:19.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:54:19.638+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:54:19.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:54:19.638+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:54:19.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:54:19.638+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:54:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:19 smithi029 ceph-mon[32610]: pgmap v8230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:19 smithi047 ceph-mon[36835]: pgmap v8230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:19 smithi077 ceph-mon[37153]: pgmap v8230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:21 smithi047 ceph-mon[36835]: pgmap v8231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:21 smithi029 ceph-mon[32610]: pgmap v8231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:21 smithi077 ceph-mon[37153]: pgmap v8231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:22 smithi047 ceph-mon[36835]: pgmap v8232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:54:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:22 smithi029 ceph-mon[32610]: pgmap v8232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:54:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:22 smithi077 ceph-mon[37153]: pgmap v8232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:54:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14684.701494 secs 2024-07-27T10:54:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14684.701494 secs 2024-07-27T10:54:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14684.701494 secs 2024-07-27T10:54:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:24 smithi047 ceph-mon[36835]: pgmap v8233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:54:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:24 smithi029 ceph-mon[32610]: pgmap v8233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:54:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:24 smithi077 ceph-mon[37153]: pgmap v8233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:54:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:27 smithi077 ceph-mon[37153]: pgmap v8234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:27 smithi047 ceph-mon[36835]: pgmap v8234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:27 smithi029 ceph-mon[32610]: pgmap v8234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14689.701637 secs 2024-07-27T10:54:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14689.701637 secs 2024-07-27T10:54:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14689.701637 secs 2024-07-27T10:54:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:29 smithi047 ceph-mon[36835]: pgmap v8235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:29 smithi029 ceph-mon[32610]: pgmap v8235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:29 smithi077 ceph-mon[37153]: pgmap v8235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:31 smithi047 ceph-mon[36835]: pgmap v8236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:31 smithi029 ceph-mon[32610]: pgmap v8236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:31 smithi077 ceph-mon[37153]: pgmap v8236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14694.701740 secs 2024-07-27T10:54:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:33 smithi047 ceph-mon[36835]: pgmap v8237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:54:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14694.701740 secs 2024-07-27T10:54:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:33 smithi029 ceph-mon[32610]: pgmap v8237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:54:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14694.701740 secs 2024-07-27T10:54:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:33 smithi077 ceph-mon[37153]: pgmap v8237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:54:34.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:54:34.654+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:54:34.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:54:34.654+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:54:34.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:54:34.654+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:54:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:35 smithi047 ceph-mon[36835]: pgmap v8238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:54:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:35 smithi029 ceph-mon[32610]: pgmap v8238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:54:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:35 smithi077 ceph-mon[37153]: pgmap v8238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:54:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:36 smithi047 ceph-mon[36835]: pgmap v8239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:36 smithi029 ceph-mon[32610]: pgmap v8239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:36 smithi077 ceph-mon[37153]: pgmap v8239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14699.701867 secs 2024-07-27T10:54:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14699.701867 secs 2024-07-27T10:54:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14699.701867 secs 2024-07-27T10:54:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:39 smithi077 ceph-mon[37153]: pgmap v8240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:39 smithi047 ceph-mon[36835]: pgmap v8240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:39 smithi029 ceph-mon[32610]: pgmap v8240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:41 smithi047 ceph-mon[36835]: pgmap v8241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:54:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:54:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:54:42.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:54:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:41 smithi029 ceph-mon[32610]: pgmap v8241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:54:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:54:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:54:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:54:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:41 smithi077 ceph-mon[37153]: pgmap v8241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:54:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:54:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:54:42.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:54:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:42 smithi047 ceph-mon[36835]: pgmap v8242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:54:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:42 smithi047 ceph-mon[36835]: pgmap v8243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:54:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:42 smithi029 ceph-mon[32610]: pgmap v8242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:54:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:42 smithi029 ceph-mon[32610]: pgmap v8243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:54:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:42 smithi077 ceph-mon[37153]: pgmap v8242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T10:54:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:42 smithi077 ceph-mon[37153]: pgmap v8243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:54:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14704.701927 secs 2024-07-27T10:54:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:43 smithi029 ceph-mon[32610]: pgmap v8244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:54:44.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14704.701927 secs 2024-07-27T10:54:44.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:43 smithi077 ceph-mon[37153]: pgmap v8244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:54:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14704.701927 secs 2024-07-27T10:54:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:43 smithi047 ceph-mon[36835]: pgmap v8244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:54:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:46 smithi047 ceph-mon[36835]: pgmap v8245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:54:46.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:46 smithi029 ceph-mon[32610]: pgmap v8245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:54:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:46 smithi077 ceph-mon[37153]: pgmap v8245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:54:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14709.702055 secs 2024-07-27T10:54:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:48 smithi047 ceph-mon[36835]: pgmap v8246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:54:48.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14709.702055 secs 2024-07-27T10:54:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:48 smithi029 ceph-mon[32610]: pgmap v8246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:54:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14709.702055 secs 2024-07-27T10:54:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:48 smithi077 ceph-mon[37153]: pgmap v8246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:54:49.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:54:49.669+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:54:49.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:54:49.669+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:54:49.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:54:49.669+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:54:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:50 smithi047 ceph-mon[36835]: pgmap v8247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:54:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:50 smithi029 ceph-mon[32610]: pgmap v8247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:54:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:50 smithi077 ceph-mon[37153]: pgmap v8247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:54:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:52 smithi047 ceph-mon[36835]: pgmap v8248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:54:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:52 smithi029 ceph-mon[32610]: pgmap v8248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:54:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:52 smithi077 ceph-mon[37153]: pgmap v8248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:54:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14714.702176 secs 2024-07-27T10:54:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14714.702176 secs 2024-07-27T10:54:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14714.702176 secs 2024-07-27T10:54:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:54 smithi047 ceph-mon[36835]: pgmap v8249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:54 smithi029 ceph-mon[32610]: pgmap v8249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:54 smithi077 ceph-mon[37153]: pgmap v8249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:54:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:56 smithi047 ceph-mon[36835]: pgmap v8250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:56.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:56 smithi029 ceph-mon[32610]: pgmap v8250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:56 smithi077 ceph-mon[37153]: pgmap v8250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:54:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14719.702239 secs 2024-07-27T10:54:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:54:58 smithi047 ceph-mon[36835]: pgmap v8251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14719.702239 secs 2024-07-27T10:54:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:54:58 smithi029 ceph-mon[32610]: pgmap v8251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:54:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14719.702239 secs 2024-07-27T10:54:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:54:58 smithi077 ceph-mon[37153]: pgmap v8251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:00 smithi047 ceph-mon[36835]: pgmap v8252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:55:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:00 smithi029 ceph-mon[32610]: pgmap v8252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:55:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:00 smithi077 ceph-mon[37153]: pgmap v8252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:55:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:02 smithi047 ceph-mon[36835]: pgmap v8253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:02.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:02 smithi029 ceph-mon[32610]: pgmap v8253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:02 smithi077 ceph-mon[37153]: pgmap v8253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14724.702361 secs 2024-07-27T10:55:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14724.702361 secs 2024-07-27T10:55:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14724.702361 secs 2024-07-27T10:55:04.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:04 smithi029 ceph-mon[32610]: pgmap v8254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:55:04.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:04.679+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:55:04.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:04.679+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:55:04.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:04.679+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:55:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:04 smithi047 ceph-mon[36835]: pgmap v8254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:55:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:04 smithi077 ceph-mon[37153]: pgmap v8254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:55:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:06 smithi047 ceph-mon[36835]: pgmap v8255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:55:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:06 smithi029 ceph-mon[32610]: pgmap v8255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:55:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:06 smithi077 ceph-mon[37153]: pgmap v8255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:55:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14729.702480 secs 2024-07-27T10:55:08.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:08 smithi047 ceph-mon[36835]: pgmap v8256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14729.702480 secs 2024-07-27T10:55:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:08 smithi029 ceph-mon[32610]: pgmap v8256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14729.702480 secs 2024-07-27T10:55:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:08 smithi077 ceph-mon[37153]: pgmap v8256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:10 smithi047 ceph-mon[36835]: pgmap v8257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:55:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:10 smithi029 ceph-mon[32610]: pgmap v8257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:55:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:10 smithi077 ceph-mon[37153]: pgmap v8257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T10:55:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:12 smithi029 ceph-mon[32610]: pgmap v8258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:12 smithi077 ceph-mon[37153]: pgmap v8258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:12 smithi047 ceph-mon[36835]: pgmap v8258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14734.702598 secs 2024-07-27T10:55:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14734.702598 secs 2024-07-27T10:55:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14734.702598 secs 2024-07-27T10:55:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:14 smithi029 ceph-mon[32610]: pgmap v8259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:55:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:14 smithi077 ceph-mon[37153]: pgmap v8259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:55:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:14 smithi047 ceph-mon[36835]: pgmap v8259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:55:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:16 smithi029 ceph-mon[32610]: pgmap v8260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:55:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:16 smithi077 ceph-mon[37153]: pgmap v8260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:55:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:16 smithi047 ceph-mon[36835]: pgmap v8260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:55:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14739.702718 secs 2024-07-27T10:55:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:18 smithi029 ceph-mon[32610]: pgmap v8261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14739.702718 secs 2024-07-27T10:55:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:18 smithi077 ceph-mon[37153]: pgmap v8261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14739.702718 secs 2024-07-27T10:55:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:18 smithi047 ceph-mon[36835]: pgmap v8261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:19.693+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:55:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:19.693+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:55:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:19.693+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:55:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:20 smithi077 ceph-mon[37153]: pgmap v8262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:55:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:20 smithi047 ceph-mon[36835]: pgmap v8262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:55:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:20 smithi029 ceph-mon[32610]: pgmap v8262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:55:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:22 smithi077 ceph-mon[37153]: pgmap v8263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:22 smithi047 ceph-mon[36835]: pgmap v8263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:22 smithi029 ceph-mon[32610]: pgmap v8263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14744.702863 secs 2024-07-27T10:55:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14744.702863 secs 2024-07-27T10:55:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14744.702863 secs 2024-07-27T10:55:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:24 smithi077 ceph-mon[37153]: pgmap v8264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:55:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:24 smithi047 ceph-mon[36835]: pgmap v8264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:55:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:24 smithi029 ceph-mon[32610]: pgmap v8264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:55:26.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:26 smithi047 ceph-mon[36835]: pgmap v8265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:55:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:26 smithi077 ceph-mon[37153]: pgmap v8265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:55:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:26 smithi029 ceph-mon[32610]: pgmap v8265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:55:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14749.703001 secs 2024-07-27T10:55:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:28 smithi077 ceph-mon[37153]: pgmap v8266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14749.703001 secs 2024-07-27T10:55:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:28 smithi047 ceph-mon[36835]: pgmap v8266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14749.703001 secs 2024-07-27T10:55:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:28 smithi029 ceph-mon[32610]: pgmap v8266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:30 smithi077 ceph-mon[37153]: pgmap v8267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:55:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:30 smithi047 ceph-mon[36835]: pgmap v8267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:55:30.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:30 smithi029 ceph-mon[32610]: pgmap v8267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:55:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:32 smithi077 ceph-mon[37153]: pgmap v8268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:32 smithi047 ceph-mon[36835]: pgmap v8268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:32 smithi029 ceph-mon[32610]: pgmap v8268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14754.703125 secs 2024-07-27T10:55:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14754.703125 secs 2024-07-27T10:55:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14754.703125 secs 2024-07-27T10:55:34.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:34.702+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:55:34.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:34.702+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:55:34.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:34.702+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:55:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:34 smithi077 ceph-mon[37153]: pgmap v8269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:55:34.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:34 smithi029 ceph-mon[32610]: pgmap v8269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:55:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:34 smithi047 ceph-mon[36835]: pgmap v8269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:55:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:36 smithi077 ceph-mon[37153]: pgmap v8270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:55:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:36 smithi047 ceph-mon[36835]: pgmap v8270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:55:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:36 smithi029 ceph-mon[32610]: pgmap v8270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:55:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14759.703225 secs 2024-07-27T10:55:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:38 smithi077 ceph-mon[37153]: pgmap v8271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14759.703225 secs 2024-07-27T10:55:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:38 smithi047 ceph-mon[36835]: pgmap v8271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:38.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14759.703225 secs 2024-07-27T10:55:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:38 smithi029 ceph-mon[32610]: pgmap v8271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:40 smithi077 ceph-mon[37153]: pgmap v8272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:55:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:40 smithi047 ceph-mon[36835]: pgmap v8272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:55:40.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:40 smithi029 ceph-mon[32610]: pgmap v8272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:55:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:55:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:55:41.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:55:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:42 smithi077 ceph-mon[37153]: pgmap v8273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:55:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:55:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:55:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:55:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:55:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:55:42.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:55:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:42 smithi047 ceph-mon[36835]: pgmap v8273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:55:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:55:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:55:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:55:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:55:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:55:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:55:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:42 smithi029 ceph-mon[32610]: pgmap v8273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:55:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:55:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:55:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:55:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:55:42.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:55:42.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:55:42.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:55:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:43 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:55:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:43 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:55:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:43 smithi077 ceph-mon[37153]: pgmap v8274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:55:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:43 smithi077 ceph-mon[37153]: pgmap v8275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:55:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14764.703371 secs 2024-07-27T10:55:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:43 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:55:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:43 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:55:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:43 smithi047 ceph-mon[36835]: pgmap v8274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:55:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:43 smithi047 ceph-mon[36835]: pgmap v8275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:55:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14764.703371 secs 2024-07-27T10:55:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:43 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T10:55:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:43 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T10:55:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:43 smithi029 ceph-mon[32610]: pgmap v8274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:55:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:43 smithi029 ceph-mon[32610]: pgmap v8275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:55:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14764.703371 secs 2024-07-27T10:55:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:45 smithi077 ceph-mon[37153]: pgmap v8276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:55:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:45 smithi047 ceph-mon[36835]: pgmap v8276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:55:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:45 smithi029 ceph-mon[32610]: pgmap v8276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:55:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:47 smithi077 ceph-mon[37153]: pgmap v8277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:55:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:47 smithi047 ceph-mon[36835]: pgmap v8277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:55:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:47 smithi029 ceph-mon[32610]: pgmap v8277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T10:55:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14769.703525 secs 2024-07-27T10:55:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14769.703525 secs 2024-07-27T10:55:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14769.703525 secs 2024-07-27T10:55:49.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:49.716+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:55:49.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:49.716+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:55:49.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:49.716+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:55:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:49 smithi077 ceph-mon[37153]: pgmap v8278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:55:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:49 smithi047 ceph-mon[36835]: pgmap v8278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:55:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:49 smithi029 ceph-mon[32610]: pgmap v8278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:55:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:51 smithi077 ceph-mon[37153]: pgmap v8279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:55:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:51 smithi047 ceph-mon[36835]: pgmap v8279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:55:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:51 smithi029 ceph-mon[32610]: pgmap v8279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:55:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:53 smithi077 ceph-mon[37153]: pgmap v8280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:55:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14774.703669 secs 2024-07-27T10:55:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:53 smithi047 ceph-mon[36835]: pgmap v8280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:55:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14774.703669 secs 2024-07-27T10:55:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:53 smithi029 ceph-mon[32610]: pgmap v8280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:55:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14774.703669 secs 2024-07-27T10:55:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:55 smithi077 ceph-mon[37153]: pgmap v8281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:55:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:55 smithi047 ceph-mon[36835]: pgmap v8281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:55:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:55 smithi029 ceph-mon[32610]: pgmap v8281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:55:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:57 smithi077 ceph-mon[37153]: pgmap v8282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:55:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:57 smithi047 ceph-mon[36835]: pgmap v8282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:55:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:57 smithi029 ceph-mon[32610]: pgmap v8282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:55:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14779.703731 secs 2024-07-27T10:55:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14779.703731 secs 2024-07-27T10:55:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14779.703731 secs 2024-07-27T10:55:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:55:59 smithi077 ceph-mon[37153]: pgmap v8283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:55:59.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:59.880+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:55:59.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:59.880+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:55:59.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:55:59 smithi029 ceph-mon[32610]: pgmap v8283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:55:59.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:59.880+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:55:59.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:55:59.880+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:55:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:55:59 smithi047 ceph-mon[36835]: pgmap v8283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:56:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:01 smithi077 ceph-mon[37153]: pgmap v8284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:01 smithi047 ceph-mon[36835]: pgmap v8284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:01.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:01 smithi029 ceph-mon[32610]: pgmap v8284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:03 smithi077 ceph-mon[37153]: pgmap v8285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:03.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14784.703831 secs 2024-07-27T10:56:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:03 smithi047 ceph-mon[36835]: pgmap v8285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14784.703831 secs 2024-07-27T10:56:03.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:03 smithi029 ceph-mon[32610]: pgmap v8285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14784.703831 secs 2024-07-27T10:56:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:56:04.725+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:56:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:56:04.725+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:56:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:56:04.725+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:56:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:05 smithi077 ceph-mon[37153]: pgmap v8286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:56:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:05 smithi047 ceph-mon[36835]: pgmap v8286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:56:05.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:05 smithi029 ceph-mon[32610]: pgmap v8286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:56:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:07 smithi077 ceph-mon[37153]: pgmap v8287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:07 smithi047 ceph-mon[36835]: pgmap v8287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:07 smithi029 ceph-mon[32610]: pgmap v8287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14789.703928 secs 2024-07-27T10:56:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14789.703928 secs 2024-07-27T10:56:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14789.703928 secs 2024-07-27T10:56:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:09 smithi077 ceph-mon[37153]: pgmap v8288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:56:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:09 smithi047 ceph-mon[36835]: pgmap v8288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:56:09.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:09 smithi029 ceph-mon[32610]: pgmap v8288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:56:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:11 smithi077 ceph-mon[37153]: pgmap v8289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:11 smithi047 ceph-mon[36835]: pgmap v8289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:11.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:11 smithi029 ceph-mon[32610]: pgmap v8289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:13 smithi077 ceph-mon[37153]: pgmap v8290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14794.704050 secs 2024-07-27T10:56:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:13 smithi047 ceph-mon[36835]: pgmap v8290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14794.704050 secs 2024-07-27T10:56:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:13 smithi029 ceph-mon[32610]: pgmap v8290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14794.704050 secs 2024-07-27T10:56:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:15 smithi077 ceph-mon[37153]: pgmap v8291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:56:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:15 smithi047 ceph-mon[36835]: pgmap v8291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:56:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:15 smithi029 ceph-mon[32610]: pgmap v8291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:56:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:17 smithi077 ceph-mon[37153]: pgmap v8292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:17 smithi047 ceph-mon[36835]: pgmap v8292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:17 smithi029 ceph-mon[32610]: pgmap v8292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14799.704169 secs 2024-07-27T10:56:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14799.704169 secs 2024-07-27T10:56:18.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14799.704169 secs 2024-07-27T10:56:19.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:56:19.741+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:56:19.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:56:19.741+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:56:19.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:56:19.741+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:56:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:19 smithi077 ceph-mon[37153]: pgmap v8293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:56:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:19 smithi047 ceph-mon[36835]: pgmap v8293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:56:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:19 smithi029 ceph-mon[32610]: pgmap v8293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:56:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:21 smithi077 ceph-mon[37153]: pgmap v8294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:21 smithi047 ceph-mon[36835]: pgmap v8294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:21 smithi029 ceph-mon[32610]: pgmap v8294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:23 smithi077 ceph-mon[37153]: pgmap v8295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14804.704293 secs 2024-07-27T10:56:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:23 smithi047 ceph-mon[36835]: pgmap v8295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14804.704293 secs 2024-07-27T10:56:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:23 smithi029 ceph-mon[32610]: pgmap v8295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14804.704293 secs 2024-07-27T10:56:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:25 smithi047 ceph-mon[36835]: pgmap v8296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:56:25.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:25 smithi029 ceph-mon[32610]: pgmap v8296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:56:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:25 smithi077 ceph-mon[37153]: pgmap v8296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:56:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:27 smithi047 ceph-mon[36835]: pgmap v8297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:56:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:27 smithi029 ceph-mon[32610]: pgmap v8297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:56:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:27 smithi077 ceph-mon[37153]: pgmap v8297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T10:56:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14809.704411 secs 2024-07-27T10:56:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14809.704411 secs 2024-07-27T10:56:29.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14809.704411 secs 2024-07-27T10:56:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:29 smithi047 ceph-mon[36835]: pgmap v8298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:56:29.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:29 smithi029 ceph-mon[32610]: pgmap v8298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:56:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:29 smithi077 ceph-mon[37153]: pgmap v8298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T10:56:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:31 smithi047 ceph-mon[36835]: pgmap v8299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:56:31.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:31 smithi029 ceph-mon[32610]: pgmap v8299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:56:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:31 smithi077 ceph-mon[37153]: pgmap v8299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:56:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:33 smithi047 ceph-mon[36835]: pgmap v8300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14814.704554 secs 2024-07-27T10:56:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:33 smithi029 ceph-mon[32610]: pgmap v8300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14814.704554 secs 2024-07-27T10:56:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:33 smithi077 ceph-mon[37153]: pgmap v8300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14814.704554 secs 2024-07-27T10:56:34.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:56:34.742+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:56:34.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:56:34.742+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:56:34.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:56:34.742+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:56:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:35 smithi047 ceph-mon[36835]: pgmap v8301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:56:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:35 smithi029 ceph-mon[32610]: pgmap v8301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:56:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:35 smithi077 ceph-mon[37153]: pgmap v8301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:56:37.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:37 smithi029 ceph-mon[32610]: pgmap v8302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:37 smithi047 ceph-mon[36835]: pgmap v8302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:37 smithi077 ceph-mon[37153]: pgmap v8302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14819.704672 secs 2024-07-27T10:56:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14819.704672 secs 2024-07-27T10:56:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14819.704672 secs 2024-07-27T10:56:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:39 smithi047 ceph-mon[36835]: pgmap v8303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:56:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:39 smithi029 ceph-mon[32610]: pgmap v8303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:56:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:39 smithi077 ceph-mon[37153]: pgmap v8303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:56:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:41 smithi047 ceph-mon[36835]: pgmap v8304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:41 smithi029 ceph-mon[32610]: pgmap v8304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:41 smithi077 ceph-mon[37153]: pgmap v8304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:56:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:56:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:56:42.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:56:42.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:56:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:56:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:42.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:56:42.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:56:42.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:56:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:56:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T10:56:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:56:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:56:43.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:56:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:43 smithi047 ceph-mon[36835]: pgmap v8305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14824.704812 secs 2024-07-27T10:56:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:43 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:56:43.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:43 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:56:43.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:43 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:56:43.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:43 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:56:43.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:43 smithi047 ceph-mon[36835]: pgmap v8306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T10:56:43.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:43 smithi047 ceph-mon[36835]: pgmap v8307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:56:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:43 smithi029 ceph-mon[32610]: pgmap v8305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14824.704812 secs 2024-07-27T10:56:43.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:43 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:56:43.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:43 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:56:43.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:43 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:56:43.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:43 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:56:43.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:43 smithi029 ceph-mon[32610]: pgmap v8306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T10:56:43.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:43 smithi029 ceph-mon[32610]: pgmap v8307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:56:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:43 smithi077 ceph-mon[37153]: pgmap v8305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:56:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14824.704812 secs 2024-07-27T10:56:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:43 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T10:56:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:43 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T10:56:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:43 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T10:56:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:43 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T10:56:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:43 smithi077 ceph-mon[37153]: pgmap v8306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T10:56:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:43 smithi077 ceph-mon[37153]: pgmap v8307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:56:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:45 smithi047 ceph-mon[36835]: pgmap v8308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:56:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:45 smithi029 ceph-mon[32610]: pgmap v8308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:56:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:45 smithi077 ceph-mon[37153]: pgmap v8308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:56:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:47 smithi047 ceph-mon[36835]: pgmap v8309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:56:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:47 smithi029 ceph-mon[32610]: pgmap v8309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:56:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:47 smithi077 ceph-mon[37153]: pgmap v8309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T10:56:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14829.704953 secs 2024-07-27T10:56:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14829.704953 secs 2024-07-27T10:56:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14829.704953 secs 2024-07-27T10:56:49.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:56:49.754+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:56:49.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:56:49.754+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:56:49.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:56:49.754+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:56:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:49 smithi047 ceph-mon[36835]: pgmap v8310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:56:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:49 smithi029 ceph-mon[32610]: pgmap v8310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:56:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:49 smithi077 ceph-mon[37153]: pgmap v8310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:56:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:51 smithi047 ceph-mon[36835]: pgmap v8311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:56:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:51 smithi029 ceph-mon[32610]: pgmap v8311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:56:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:51 smithi077 ceph-mon[37153]: pgmap v8311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:56:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14834.705074 secs 2024-07-27T10:56:53.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:53 smithi047 ceph-mon[36835]: pgmap v8312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:56:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14834.705074 secs 2024-07-27T10:56:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:53 smithi029 ceph-mon[32610]: pgmap v8312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:56:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14834.705074 secs 2024-07-27T10:56:54.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:53 smithi077 ceph-mon[37153]: pgmap v8312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:56:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:55 smithi047 ceph-mon[36835]: pgmap v8313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:56:55.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:55 smithi029 ceph-mon[32610]: pgmap v8313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:56:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:55 smithi077 ceph-mon[37153]: pgmap v8313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:56:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:57 smithi047 ceph-mon[36835]: pgmap v8314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:56:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:57 smithi029 ceph-mon[32610]: pgmap v8314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:56:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:57 smithi077 ceph-mon[37153]: pgmap v8314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:56:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14839.705191 secs 2024-07-27T10:56:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14839.705191 secs 2024-07-27T10:56:59.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14839.705191 secs 2024-07-27T10:56:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:56:59 smithi047 ceph-mon[36835]: pgmap v8315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:56:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:56:59 smithi029 ceph-mon[32610]: pgmap v8315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:57:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:56:59 smithi077 ceph-mon[37153]: pgmap v8315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:57:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:01 smithi047 ceph-mon[36835]: pgmap v8316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:01 smithi029 ceph-mon[32610]: pgmap v8316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:01 smithi077 ceph-mon[37153]: pgmap v8316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14844.705314 secs 2024-07-27T10:57:03.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:03 smithi047 ceph-mon[36835]: pgmap v8317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:57:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14844.705314 secs 2024-07-27T10:57:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:03 smithi029 ceph-mon[32610]: pgmap v8317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:57:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14844.705314 secs 2024-07-27T10:57:04.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:03 smithi077 ceph-mon[37153]: pgmap v8317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:57:04.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:04.763+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:57:04.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:04.763+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:57:04.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:04.763+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:57:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:05 smithi029 ceph-mon[32610]: pgmap v8318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:57:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:05 smithi077 ceph-mon[37153]: pgmap v8318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:57:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:05 smithi047 ceph-mon[36835]: pgmap v8318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:57:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:07 smithi029 ceph-mon[32610]: pgmap v8319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:07 smithi077 ceph-mon[37153]: pgmap v8319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:07 smithi047 ceph-mon[36835]: pgmap v8319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14849.705438 secs 2024-07-27T10:57:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14849.705438 secs 2024-07-27T10:57:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14849.705438 secs 2024-07-27T10:57:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:09 smithi029 ceph-mon[32610]: pgmap v8320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:57:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:09 smithi077 ceph-mon[37153]: pgmap v8320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:57:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:09 smithi047 ceph-mon[36835]: pgmap v8320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:57:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:11 smithi029 ceph-mon[32610]: pgmap v8321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:11 smithi077 ceph-mon[37153]: pgmap v8321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:11 smithi047 ceph-mon[36835]: pgmap v8321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14854.705558 secs 2024-07-27T10:57:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:13 smithi077 ceph-mon[37153]: pgmap v8322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:57:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14854.705558 secs 2024-07-27T10:57:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:13 smithi047 ceph-mon[36835]: pgmap v8322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:57:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14854.705558 secs 2024-07-27T10:57:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:13 smithi029 ceph-mon[32610]: pgmap v8322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:57:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:15 smithi077 ceph-mon[37153]: pgmap v8323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:57:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:15 smithi047 ceph-mon[36835]: pgmap v8323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:57:16.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:15 smithi029 ceph-mon[32610]: pgmap v8323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:57:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:17 smithi077 ceph-mon[37153]: pgmap v8324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:17 smithi047 ceph-mon[36835]: pgmap v8324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:17 smithi029 ceph-mon[32610]: pgmap v8324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14859.705695 secs 2024-07-27T10:57:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14859.705695 secs 2024-07-27T10:57:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14859.705695 secs 2024-07-27T10:57:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:19.771+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:57:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:19.771+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:57:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:19.771+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:57:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:19 smithi077 ceph-mon[37153]: pgmap v8325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:57:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:19 smithi047 ceph-mon[36835]: pgmap v8325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:57:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:19 smithi029 ceph-mon[32610]: pgmap v8325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:57:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:21 smithi077 ceph-mon[37153]: pgmap v8326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:21 smithi047 ceph-mon[36835]: pgmap v8326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:21 smithi029 ceph-mon[32610]: pgmap v8326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14864.705831 secs 2024-07-27T10:57:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:23 smithi077 ceph-mon[37153]: pgmap v8327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:57:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14864.705831 secs 2024-07-27T10:57:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:23 smithi047 ceph-mon[36835]: pgmap v8327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:57:24.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14864.705831 secs 2024-07-27T10:57:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:23 smithi029 ceph-mon[32610]: pgmap v8327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:57:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:25 smithi077 ceph-mon[37153]: pgmap v8328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:57:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:25 smithi047 ceph-mon[36835]: pgmap v8328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:57:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:25 smithi029 ceph-mon[32610]: pgmap v8328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:57:27.998 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:27 smithi047 ceph-mon[36835]: pgmap v8329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:28.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:27 smithi077 ceph-mon[37153]: pgmap v8329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:27 smithi029 ceph-mon[32610]: pgmap v8329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:29.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14869.705949 secs 2024-07-27T10:57:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14869.705949 secs 2024-07-27T10:57:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14869.705949 secs 2024-07-27T10:57:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:29.895+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:57:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:29.895+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:57:29.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:29.895+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:57:29.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:29.896+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:57:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:29 smithi077 ceph-mon[37153]: pgmap v8330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:57:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:29 smithi047 ceph-mon[36835]: pgmap v8330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:57:30.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:29 smithi029 ceph-mon[32610]: pgmap v8330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:57:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:31 smithi077 ceph-mon[37153]: pgmap v8331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:32.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:31 smithi047 ceph-mon[36835]: pgmap v8331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:31 smithi029 ceph-mon[32610]: pgmap v8331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:34.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14874.706062 secs 2024-07-27T10:57:34.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:33 smithi029 ceph-mon[32610]: pgmap v8332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:57:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14874.706062 secs 2024-07-27T10:57:34.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:33 smithi077 ceph-mon[37153]: pgmap v8332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:57:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14874.706062 secs 2024-07-27T10:57:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:33 smithi047 ceph-mon[36835]: pgmap v8332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:57:34.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:34.774+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:57:34.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:34.774+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:57:34.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:34.774+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:57:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:35 smithi077 ceph-mon[37153]: pgmap v8333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:57:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:35 smithi047 ceph-mon[36835]: pgmap v8333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:57:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:35 smithi029 ceph-mon[32610]: pgmap v8333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:57:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:37 smithi077 ceph-mon[37153]: pgmap v8334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:37 smithi047 ceph-mon[36835]: pgmap v8334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:37 smithi029 ceph-mon[32610]: pgmap v8334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14879.706142 secs 2024-07-27T10:57:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14879.706142 secs 2024-07-27T10:57:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14879.706142 secs 2024-07-27T10:57:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:39 smithi077 ceph-mon[37153]: pgmap v8335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:57:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:39 smithi047 ceph-mon[36835]: pgmap v8335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:57:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:39 smithi029 ceph-mon[32610]: pgmap v8335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T10:57:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:41 smithi077 ceph-mon[37153]: pgmap v8336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:57:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:41 smithi047 ceph-mon[36835]: pgmap v8336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:57:42.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:41 smithi029 ceph-mon[32610]: pgmap v8336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T10:57:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:57:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:57:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:57:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14884.706266 secs 2024-07-27T10:57:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:43 smithi077 ceph-mon[37153]: pgmap v8337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:57:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:57:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:57:44.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:57:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14884.706266 secs 2024-07-27T10:57:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:43 smithi047 ceph-mon[36835]: pgmap v8337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:57:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:57:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:57:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:57:44.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14884.706266 secs 2024-07-27T10:57:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:43 smithi029 ceph-mon[32610]: pgmap v8337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T10:57:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:57:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:57:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:57:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:44 smithi077 ceph-mon[37153]: pgmap v8338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:57:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:44 smithi077 ceph-mon[37153]: pgmap v8339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:57:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:44 smithi047 ceph-mon[36835]: pgmap v8338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:57:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:44 smithi047 ceph-mon[36835]: pgmap v8339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:57:45.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:44 smithi029 ceph-mon[32610]: pgmap v8338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T10:57:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:44 smithi029 ceph-mon[32610]: pgmap v8339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:57:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:46 smithi077 ceph-mon[37153]: pgmap v8340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:57:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:46 smithi047 ceph-mon[36835]: pgmap v8340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:57:47.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:46 smithi029 ceph-mon[32610]: pgmap v8340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:57:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14889.706385 secs 2024-07-27T10:57:49.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:48 smithi077 ceph-mon[37153]: pgmap v8341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:57:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14889.706385 secs 2024-07-27T10:57:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:48 smithi047 ceph-mon[36835]: pgmap v8341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:57:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14889.706385 secs 2024-07-27T10:57:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:48 smithi029 ceph-mon[32610]: pgmap v8341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:57:49.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:49.784+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:57:49.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:49.784+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:57:49.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:57:49.784+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:57:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:50 smithi077 ceph-mon[37153]: pgmap v8342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:57:51.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:50 smithi047 ceph-mon[36835]: pgmap v8342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:57:51.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:50 smithi029 ceph-mon[32610]: pgmap v8342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:57:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:52 smithi077 ceph-mon[37153]: pgmap v8343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:57:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:52 smithi047 ceph-mon[36835]: pgmap v8343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:57:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:52 smithi029 ceph-mon[32610]: pgmap v8343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T10:57:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14894.706490 secs 2024-07-27T10:57:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14894.706490 secs 2024-07-27T10:57:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14894.706490 secs 2024-07-27T10:57:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:54 smithi077 ceph-mon[37153]: pgmap v8344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:57:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:54 smithi047 ceph-mon[36835]: pgmap v8344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:57:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:54 smithi029 ceph-mon[32610]: pgmap v8344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:57:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:56 smithi077 ceph-mon[37153]: pgmap v8345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:56 smithi047 ceph-mon[36835]: pgmap v8345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:56 smithi029 ceph-mon[32610]: pgmap v8345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:57:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14899.706629 secs 2024-07-27T10:57:59.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:57:58 smithi077 ceph-mon[37153]: pgmap v8346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14899.706629 secs 2024-07-27T10:57:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:57:58 smithi047 ceph-mon[36835]: pgmap v8346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:57:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14899.706629 secs 2024-07-27T10:57:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:57:58 smithi029 ceph-mon[32610]: pgmap v8346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:00 smithi077 ceph-mon[37153]: pgmap v8347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:00 smithi047 ceph-mon[36835]: pgmap v8347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:00 smithi029 ceph-mon[32610]: pgmap v8347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:02 smithi077 ceph-mon[37153]: pgmap v8348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:02 smithi047 ceph-mon[36835]: pgmap v8348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:03.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:02 smithi029 ceph-mon[32610]: pgmap v8348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14904.706748 secs 2024-07-27T10:58:04.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14904.706748 secs 2024-07-27T10:58:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14904.706748 secs 2024-07-27T10:58:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:04.800+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:58:04.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:04.800+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:58:04.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:04.800+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:58:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:04 smithi077 ceph-mon[37153]: pgmap v8349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:04 smithi047 ceph-mon[36835]: pgmap v8349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:05.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:04 smithi029 ceph-mon[32610]: pgmap v8349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:06 smithi077 ceph-mon[37153]: pgmap v8350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:06 smithi047 ceph-mon[36835]: pgmap v8350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:06 smithi029 ceph-mon[32610]: pgmap v8350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14909.706813 secs 2024-07-27T10:58:09.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:08 smithi077 ceph-mon[37153]: pgmap v8351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14909.706813 secs 2024-07-27T10:58:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:08 smithi047 ceph-mon[36835]: pgmap v8351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14909.706813 secs 2024-07-27T10:58:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:08 smithi029 ceph-mon[32610]: pgmap v8351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:11.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:10 smithi077 ceph-mon[37153]: pgmap v8352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:11.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:10 smithi029 ceph-mon[32610]: pgmap v8352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:10 smithi047 ceph-mon[36835]: pgmap v8352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:12 smithi077 ceph-mon[37153]: pgmap v8353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14914.706928 secs 2024-07-27T10:58:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:12 smithi047 ceph-mon[36835]: pgmap v8353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14914.706928 secs 2024-07-27T10:58:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:12 smithi029 ceph-mon[32610]: pgmap v8353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14914.706928 secs 2024-07-27T10:58:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:14 smithi077 ceph-mon[37153]: pgmap v8354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:14 smithi047 ceph-mon[36835]: pgmap v8354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:14 smithi029 ceph-mon[32610]: pgmap v8354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:17.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:16 smithi077 ceph-mon[37153]: pgmap v8355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:16 smithi047 ceph-mon[36835]: pgmap v8355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:16 smithi029 ceph-mon[32610]: pgmap v8355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14919.707010 secs 2024-07-27T10:58:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14919.707010 secs 2024-07-27T10:58:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14919.707010 secs 2024-07-27T10:58:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:18 smithi047 ceph-mon[36835]: pgmap v8356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:18 smithi029 ceph-mon[32610]: pgmap v8356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:18 smithi077 ceph-mon[37153]: pgmap v8356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:19.815+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:58:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:19.815+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:58:19.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:19.815+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:58:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:20 smithi047 ceph-mon[36835]: pgmap v8357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:21.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:20 smithi029 ceph-mon[32610]: pgmap v8357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:20 smithi077 ceph-mon[37153]: pgmap v8357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:22 smithi047 ceph-mon[36835]: pgmap v8358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14924.707153 secs 2024-07-27T10:58:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:22 smithi029 ceph-mon[32610]: pgmap v8358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14924.707153 secs 2024-07-27T10:58:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:22 smithi077 ceph-mon[37153]: pgmap v8358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14924.707153 secs 2024-07-27T10:58:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:24 smithi047 ceph-mon[36835]: pgmap v8359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:25.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:24 smithi029 ceph-mon[32610]: pgmap v8359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:24 smithi077 ceph-mon[37153]: pgmap v8359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:26 smithi047 ceph-mon[36835]: pgmap v8360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:27.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:26 smithi029 ceph-mon[32610]: pgmap v8360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:27.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:26 smithi077 ceph-mon[37153]: pgmap v8360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14929.707292 secs 2024-07-27T10:58:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14929.707292 secs 2024-07-27T10:58:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14929.707292 secs 2024-07-27T10:58:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:28 smithi047 ceph-mon[36835]: pgmap v8361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:28 smithi029 ceph-mon[32610]: pgmap v8361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:28 smithi077 ceph-mon[37153]: pgmap v8361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:31.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:30 smithi077 ceph-mon[37153]: pgmap v8362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:30 smithi047 ceph-mon[36835]: pgmap v8362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:30 smithi029 ceph-mon[32610]: pgmap v8362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:32 smithi047 ceph-mon[36835]: pgmap v8363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14934.707439 secs 2024-07-27T10:58:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:32 smithi029 ceph-mon[32610]: pgmap v8363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14934.707439 secs 2024-07-27T10:58:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:32 smithi077 ceph-mon[37153]: pgmap v8363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14934.707439 secs 2024-07-27T10:58:34.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:34.823+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:58:34.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:34.823+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:58:34.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:34.823+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:58:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:34 smithi047 ceph-mon[36835]: pgmap v8364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:34 smithi029 ceph-mon[32610]: pgmap v8364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:34 smithi077 ceph-mon[37153]: pgmap v8364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:36 smithi047 ceph-mon[36835]: pgmap v8365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:36 smithi029 ceph-mon[32610]: pgmap v8365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:36 smithi077 ceph-mon[37153]: pgmap v8365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14939.707564 secs 2024-07-27T10:58:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14939.707564 secs 2024-07-27T10:58:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14939.707564 secs 2024-07-27T10:58:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:38 smithi047 ceph-mon[36835]: pgmap v8366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:39.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:38 smithi029 ceph-mon[32610]: pgmap v8366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:38 smithi077 ceph-mon[37153]: pgmap v8366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:58:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:40 smithi047 ceph-mon[36835]: pgmap v8367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:41.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:40 smithi029 ceph-mon[32610]: pgmap v8367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:40 smithi077 ceph-mon[37153]: pgmap v8367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T10:58:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:42 smithi047 ceph-mon[36835]: pgmap v8368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14944.707688 secs 2024-07-27T10:58:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:42 smithi029 ceph-mon[32610]: pgmap v8368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14944.707688 secs 2024-07-27T10:58:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:42 smithi077 ceph-mon[37153]: pgmap v8368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14944.707688 secs 2024-07-27T10:58:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:58:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:58:44.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:58:44.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:58:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:58:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:58:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:58:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:58:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:58:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:58:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:58:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:58:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:44 smithi047 ceph-mon[36835]: pgmap v8369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:44 smithi047 ceph-mon[36835]: pgmap v8370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:58:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:44 smithi047 ceph-mon[36835]: pgmap v8371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:58:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:44 smithi029 ceph-mon[32610]: pgmap v8369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:44 smithi029 ceph-mon[32610]: pgmap v8370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:58:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:44 smithi029 ceph-mon[32610]: pgmap v8371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:58:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:44 smithi077 ceph-mon[37153]: pgmap v8369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T10:58:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:44 smithi077 ceph-mon[37153]: pgmap v8370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T10:58:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:44 smithi077 ceph-mon[37153]: pgmap v8371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T10:58:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:45 smithi047 ceph-mon[36835]: pgmap v8372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:58:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:45 smithi029 ceph-mon[32610]: pgmap v8372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:58:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:45 smithi077 ceph-mon[37153]: pgmap v8372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T10:58:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14949.707833 secs 2024-07-27T10:58:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14949.707833 secs 2024-07-27T10:58:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14949.707833 secs 2024-07-27T10:58:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:48 smithi077 ceph-mon[37153]: pgmap v8373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:58:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:48 smithi047 ceph-mon[36835]: pgmap v8373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:58:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:48 smithi029 ceph-mon[32610]: pgmap v8373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T10:58:49.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:49.827+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:58:49.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:49.827+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:58:49.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:49.827+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:58:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:50 smithi047 ceph-mon[36835]: pgmap v8374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:58:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:50 smithi029 ceph-mon[32610]: pgmap v8374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:58:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:50 smithi077 ceph-mon[37153]: pgmap v8374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T10:58:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:52 smithi047 ceph-mon[36835]: pgmap v8375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:58:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14954.707929 secs 2024-07-27T10:58:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:52 smithi029 ceph-mon[32610]: pgmap v8375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:58:52.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14954.707929 secs 2024-07-27T10:58:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:52 smithi077 ceph-mon[37153]: pgmap v8375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T10:58:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14954.707929 secs 2024-07-27T10:58:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:54 smithi047 ceph-mon[36835]: pgmap v8376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:58:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:54 smithi029 ceph-mon[32610]: pgmap v8376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:58:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:54 smithi077 ceph-mon[37153]: pgmap v8376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:58:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:56 smithi047 ceph-mon[36835]: pgmap v8377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:56 smithi029 ceph-mon[32610]: pgmap v8377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:56 smithi077 ceph-mon[37153]: pgmap v8377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:58:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14959.708047 secs 2024-07-27T10:58:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14959.708047 secs 2024-07-27T10:58:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14959.708047 secs 2024-07-27T10:58:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:58:58 smithi029 ceph-mon[32610]: pgmap v8378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:58:59.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:58:58 smithi047 ceph-mon[36835]: pgmap v8378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:58:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:58:58 smithi077 ceph-mon[37153]: pgmap v8378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:58:59.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:59.908+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:58:59.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:59.908+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:58:59.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:59.908+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T10:58:59.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:58:59.909+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T10:59:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:00 smithi029 ceph-mon[32610]: pgmap v8379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:00 smithi077 ceph-mon[37153]: pgmap v8379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:00 smithi047 ceph-mon[36835]: pgmap v8379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:02 smithi029 ceph-mon[32610]: pgmap v8380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14964.708176 secs 2024-07-27T10:59:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:02 smithi077 ceph-mon[37153]: pgmap v8380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14964.708176 secs 2024-07-27T10:59:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:02 smithi047 ceph-mon[36835]: pgmap v8380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14964.708176 secs 2024-07-27T10:59:04.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:59:04.840+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:59:04.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:59:04.840+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:59:04.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:59:04.840+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:59:04.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:04 smithi029 ceph-mon[32610]: pgmap v8381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:04 smithi077 ceph-mon[37153]: pgmap v8381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:04 smithi047 ceph-mon[36835]: pgmap v8381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:06 smithi077 ceph-mon[37153]: pgmap v8382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:06 smithi047 ceph-mon[36835]: pgmap v8382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:07.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:06 smithi029 ceph-mon[32610]: pgmap v8382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14969.708302 secs 2024-07-27T10:59:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14969.708302 secs 2024-07-27T10:59:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14969.708302 secs 2024-07-27T10:59:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:08 smithi077 ceph-mon[37153]: pgmap v8383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:59:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:08 smithi047 ceph-mon[36835]: pgmap v8383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:59:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:08 smithi029 ceph-mon[32610]: pgmap v8383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:59:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:10 smithi077 ceph-mon[37153]: pgmap v8384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:10 smithi047 ceph-mon[36835]: pgmap v8384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:10 smithi029 ceph-mon[32610]: pgmap v8384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:13.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:12 smithi029 ceph-mon[32610]: pgmap v8385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:13.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14974.708426 secs 2024-07-27T10:59:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:12 smithi077 ceph-mon[37153]: pgmap v8385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14974.708426 secs 2024-07-27T10:59:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:12 smithi047 ceph-mon[36835]: pgmap v8385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14974.708426 secs 2024-07-27T10:59:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:14 smithi077 ceph-mon[37153]: pgmap v8386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:14 smithi047 ceph-mon[36835]: pgmap v8386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:14 smithi029 ceph-mon[32610]: pgmap v8386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:16 smithi077 ceph-mon[37153]: pgmap v8387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:16 smithi047 ceph-mon[36835]: pgmap v8387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:16 smithi029 ceph-mon[32610]: pgmap v8387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14979.708522 secs 2024-07-27T10:59:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14979.708522 secs 2024-07-27T10:59:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14979.708522 secs 2024-07-27T10:59:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:18 smithi077 ceph-mon[37153]: pgmap v8388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:59:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:18 smithi047 ceph-mon[36835]: pgmap v8388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:59:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:18 smithi029 ceph-mon[32610]: pgmap v8388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:59:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:59:19.856+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:59:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:59:19.856+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:59:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:59:19.856+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:59:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:20 smithi077 ceph-mon[37153]: pgmap v8389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:20 smithi047 ceph-mon[36835]: pgmap v8389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:20 smithi029 ceph-mon[32610]: pgmap v8389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:22 smithi077 ceph-mon[37153]: pgmap v8390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14984.708644 secs 2024-07-27T10:59:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:22 smithi047 ceph-mon[36835]: pgmap v8390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14984.708644 secs 2024-07-27T10:59:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:22 smithi029 ceph-mon[32610]: pgmap v8390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14984.708644 secs 2024-07-27T10:59:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:24 smithi077 ceph-mon[37153]: pgmap v8391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:24 smithi047 ceph-mon[36835]: pgmap v8391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:24 smithi029 ceph-mon[32610]: pgmap v8391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:26 smithi077 ceph-mon[37153]: pgmap v8392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:26 smithi047 ceph-mon[36835]: pgmap v8392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:27.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:26 smithi029 ceph-mon[32610]: pgmap v8392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14989.708781 secs 2024-07-27T10:59:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14989.708781 secs 2024-07-27T10:59:28.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14989.708781 secs 2024-07-27T10:59:29.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:28 smithi047 ceph-mon[36835]: pgmap v8393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:59:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:28 smithi077 ceph-mon[37153]: pgmap v8393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:59:29.112 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:28 smithi029 ceph-mon[32610]: pgmap v8393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:59:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:30 smithi077 ceph-mon[37153]: pgmap v8394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:30 smithi047 ceph-mon[36835]: pgmap v8394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:30 smithi029 ceph-mon[32610]: pgmap v8394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:32 smithi077 ceph-mon[37153]: pgmap v8395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14994.708875 secs 2024-07-27T10:59:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:32 smithi047 ceph-mon[36835]: pgmap v8395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14994.708875 secs 2024-07-27T10:59:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:32 smithi029 ceph-mon[32610]: pgmap v8395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14994.708875 secs 2024-07-27T10:59:34.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:59:34.872+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:59:34.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:59:34.872+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:59:34.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:59:34.872+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:59:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:34 smithi077 ceph-mon[37153]: pgmap v8396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:34 smithi047 ceph-mon[36835]: pgmap v8396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:34 smithi029 ceph-mon[32610]: pgmap v8396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:36 smithi077 ceph-mon[37153]: pgmap v8397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:36 smithi047 ceph-mon[36835]: pgmap v8397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:37.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:36 smithi029 ceph-mon[32610]: pgmap v8397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 14999.708935 secs 2024-07-27T10:59:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 14999.708935 secs 2024-07-27T10:59:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 14999.708935 secs 2024-07-27T10:59:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:38 smithi077 ceph-mon[37153]: pgmap v8398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:59:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:38 smithi047 ceph-mon[36835]: pgmap v8398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:59:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:38 smithi029 ceph-mon[32610]: pgmap v8398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T10:59:41.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:40 smithi077 ceph-mon[37153]: pgmap v8399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:40 smithi047 ceph-mon[36835]: pgmap v8399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:41.288 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:40 smithi029 ceph-mon[32610]: pgmap v8399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T10:59:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:42 smithi077 ceph-mon[37153]: pgmap v8400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15004.709064 secs 2024-07-27T10:59:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:42 smithi047 ceph-mon[36835]: pgmap v8400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15004.709064 secs 2024-07-27T10:59:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:42 smithi029 ceph-mon[32610]: pgmap v8400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T10:59:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15004.709064 secs 2024-07-27T10:59:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:59:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:59:44.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T10:59:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:44 smithi029 ceph-mon[32610]: pgmap v8401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:44 smithi077 ceph-mon[37153]: pgmap v8401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:45.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:44 smithi047 ceph-mon[36835]: pgmap v8401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:45.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:46 smithi047 ceph-mon[36835]: pgmap v8402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:59:46.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:59:46.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:46 smithi029 ceph-mon[32610]: pgmap v8402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:59:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:59:46.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:46 smithi077 ceph-mon[37153]: pgmap v8402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T10:59:46.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:46.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T10:59:46.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T10:59:46.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T10:59:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:47 smithi077 ceph-mon[37153]: pgmap v8403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:59:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:47 smithi077 ceph-mon[37153]: pgmap v8404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T10:59:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15009.709123 secs 2024-07-27T10:59:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:47 smithi047 ceph-mon[36835]: pgmap v8403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:59:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:47 smithi047 ceph-mon[36835]: pgmap v8404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T10:59:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15009.709123 secs 2024-07-27T10:59:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:47 smithi029 ceph-mon[32610]: pgmap v8403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T10:59:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:47 smithi029 ceph-mon[32610]: pgmap v8404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T10:59:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15009.709123 secs 2024-07-27T10:59:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:49 smithi077 ceph-mon[37153]: pgmap v8405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:59:49.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:49 smithi029 ceph-mon[32610]: pgmap v8405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:59:49.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:59:49.886+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T10:59:49.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:59:49.886+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T10:59:49.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T10:59:49.886+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T10:59:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:49 smithi047 ceph-mon[36835]: pgmap v8405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T10:59:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:51 smithi077 ceph-mon[37153]: pgmap v8406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:59:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:51 smithi047 ceph-mon[36835]: pgmap v8406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:59:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:51 smithi029 ceph-mon[32610]: pgmap v8406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:59:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15014.709226 secs 2024-07-27T10:59:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15014.709226 secs 2024-07-27T10:59:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15014.709226 secs 2024-07-27T10:59:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:53 smithi077 ceph-mon[37153]: pgmap v8407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-27T10:59:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:53 smithi047 ceph-mon[36835]: pgmap v8407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-27T10:59:53.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:53 smithi029 ceph-mon[32610]: pgmap v8407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 8 op/s 2024-07-27T10:59:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:55 smithi077 ceph-mon[37153]: pgmap v8408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:55 smithi047 ceph-mon[36835]: pgmap v8408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:55.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:55 smithi029 ceph-mon[32610]: pgmap v8408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:57 smithi077 ceph-mon[37153]: pgmap v8409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:59:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15019.709366 secs 2024-07-27T10:59:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:57 smithi047 ceph-mon[36835]: pgmap v8409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:59:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15019.709366 secs 2024-07-27T10:59:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:57 smithi029 ceph-mon[32610]: pgmap v8409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T10:59:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15019.709366 secs 2024-07-27T10:59:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 10:59:59 smithi077 ceph-mon[37153]: pgmap v8410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 10:59:59 smithi047 ceph-mon[36835]: pgmap v8410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T10:59:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 10:59:59 smithi029 ceph-mon[32610]: pgmap v8410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:00:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:00:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:00:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:01 smithi077 ceph-mon[37153]: pgmap v8411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:01 smithi047 ceph-mon[36835]: pgmap v8411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:01.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:01 smithi029 ceph-mon[32610]: pgmap v8411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15024.709496 secs 2024-07-27T11:00:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15024.709496 secs 2024-07-27T11:00:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15024.709496 secs 2024-07-27T11:00:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:03 smithi077 ceph-mon[37153]: pgmap v8412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:00:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:03 smithi047 ceph-mon[36835]: pgmap v8412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:00:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:03 smithi029 ceph-mon[32610]: pgmap v8412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:00:04.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:00:04.902+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:00:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:00:04.902+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:00:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:00:04.902+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:00:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:05 smithi077 ceph-mon[37153]: pgmap v8413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:00:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:05 smithi047 ceph-mon[36835]: pgmap v8413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:00:05.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:05 smithi029 ceph-mon[32610]: pgmap v8413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:00:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:07 smithi077 ceph-mon[37153]: pgmap v8414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15029.709599 secs 2024-07-27T11:00:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:07 smithi047 ceph-mon[36835]: pgmap v8414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15029.709599 secs 2024-07-27T11:00:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:07 smithi029 ceph-mon[32610]: pgmap v8414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15029.709599 secs 2024-07-27T11:00:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:09 smithi077 ceph-mon[37153]: pgmap v8415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:09 smithi047 ceph-mon[36835]: pgmap v8415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:09 smithi029 ceph-mon[32610]: pgmap v8415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:11 smithi047 ceph-mon[36835]: pgmap v8416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:11.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:11 smithi029 ceph-mon[32610]: pgmap v8416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:11 smithi077 ceph-mon[37153]: pgmap v8416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15034.709722 secs 2024-07-27T11:00:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15034.709722 secs 2024-07-27T11:00:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15034.709722 secs 2024-07-27T11:00:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:13 smithi047 ceph-mon[36835]: pgmap v8417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:00:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:13 smithi029 ceph-mon[32610]: pgmap v8417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:00:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:13 smithi077 ceph-mon[37153]: pgmap v8417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:00:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:15 smithi047 ceph-mon[36835]: pgmap v8418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:00:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:15 smithi029 ceph-mon[32610]: pgmap v8418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:00:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:15 smithi077 ceph-mon[37153]: pgmap v8418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:00:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:17 smithi047 ceph-mon[36835]: pgmap v8419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15039.709841 secs 2024-07-27T11:00:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:17 smithi029 ceph-mon[32610]: pgmap v8419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15039.709841 secs 2024-07-27T11:00:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:17 smithi077 ceph-mon[37153]: pgmap v8419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15039.709841 secs 2024-07-27T11:00:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:00:19.918+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:00:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:00:19.918+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:00:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:00:19.918+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:00:19.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:19 smithi029 ceph-mon[32610]: pgmap v8420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:19 smithi047 ceph-mon[36835]: pgmap v8420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:19 smithi077 ceph-mon[37153]: pgmap v8420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:21 smithi047 ceph-mon[36835]: pgmap v8421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:21 smithi029 ceph-mon[32610]: pgmap v8421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:21 smithi077 ceph-mon[37153]: pgmap v8421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15044.709963 secs 2024-07-27T11:00:22.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15044.709963 secs 2024-07-27T11:00:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15044.709963 secs 2024-07-27T11:00:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:23 smithi047 ceph-mon[36835]: pgmap v8422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:00:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:23 smithi029 ceph-mon[32610]: pgmap v8422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:00:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:23 smithi077 ceph-mon[37153]: pgmap v8422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:00:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:25 smithi047 ceph-mon[36835]: pgmap v8423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:00:25.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:25 smithi029 ceph-mon[32610]: pgmap v8423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:00:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:25 smithi077 ceph-mon[37153]: pgmap v8423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:00:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:27 smithi047 ceph-mon[36835]: pgmap v8424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15049.710082 secs 2024-07-27T11:00:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:27 smithi029 ceph-mon[32610]: pgmap v8424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15049.710082 secs 2024-07-27T11:00:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:27 smithi077 ceph-mon[37153]: pgmap v8424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15049.710082 secs 2024-07-27T11:00:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:29 smithi047 ceph-mon[36835]: pgmap v8425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:29 smithi029 ceph-mon[32610]: pgmap v8425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:29 smithi077 ceph-mon[37153]: pgmap v8425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:31 smithi047 ceph-mon[36835]: pgmap v8426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:31.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:31 smithi029 ceph-mon[32610]: pgmap v8426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:31 smithi077 ceph-mon[37153]: pgmap v8426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15054.710204 secs 2024-07-27T11:00:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15054.710204 secs 2024-07-27T11:00:33.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15054.710204 secs 2024-07-27T11:00:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:33 smithi047 ceph-mon[36835]: pgmap v8427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:00:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:33 smithi029 ceph-mon[32610]: pgmap v8427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:00:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:33 smithi077 ceph-mon[37153]: pgmap v8427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:00:34.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:00:34.930+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:00:34.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:00:34.930+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:00:34.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:00:34.930+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:00:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:35 smithi047 ceph-mon[36835]: pgmap v8428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:00:35.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:35 smithi029 ceph-mon[32610]: pgmap v8428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:00:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:35 smithi077 ceph-mon[37153]: pgmap v8428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:00:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:37 smithi047 ceph-mon[36835]: pgmap v8429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15059.710302 secs 2024-07-27T11:00:37.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:37 smithi029 ceph-mon[32610]: pgmap v8429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15059.710302 secs 2024-07-27T11:00:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:37 smithi077 ceph-mon[37153]: pgmap v8429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15059.710302 secs 2024-07-27T11:00:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:39 smithi047 ceph-mon[36835]: pgmap v8430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:39 smithi029 ceph-mon[32610]: pgmap v8430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:39 smithi077 ceph-mon[37153]: pgmap v8430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:00:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:41 smithi047 ceph-mon[36835]: pgmap v8431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:41 smithi029 ceph-mon[32610]: pgmap v8431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:41 smithi077 ceph-mon[37153]: pgmap v8431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:00:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15064.710431 secs 2024-07-27T11:00:42.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15064.710431 secs 2024-07-27T11:00:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15064.710431 secs 2024-07-27T11:00:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:43 smithi047 ceph-mon[36835]: pgmap v8432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:00:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:43 smithi029 ceph-mon[32610]: pgmap v8432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:00:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:43 smithi077 ceph-mon[37153]: pgmap v8432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:00:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:45 smithi047 ceph-mon[36835]: pgmap v8433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:00:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:45 smithi029 ceph-mon[32610]: pgmap v8433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:00:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:45 smithi077 ceph-mon[37153]: pgmap v8433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:00:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:00:46.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:00:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:00:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:47 smithi077 ceph-mon[37153]: pgmap v8434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15069.710544 secs 2024-07-27T11:00:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:00:48.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:00:48.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:47 smithi047 ceph-mon[36835]: pgmap v8434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15069.710544 secs 2024-07-27T11:00:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:00:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:00:48.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:47 smithi029 ceph-mon[32610]: pgmap v8434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:00:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15069.710544 secs 2024-07-27T11:00:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:00:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:00:48.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:00:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:48 smithi077 ceph-mon[37153]: pgmap v8435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:00:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:48 smithi077 ceph-mon[37153]: pgmap v8436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T11:00:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:48 smithi047 ceph-mon[36835]: pgmap v8435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:00:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:48 smithi047 ceph-mon[36835]: pgmap v8436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T11:00:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:48 smithi029 ceph-mon[32610]: pgmap v8435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:00:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:48 smithi029 ceph-mon[32610]: pgmap v8436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T11:00:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:00:49.941+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:00:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:00:49.941+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:00:49.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:00:49.941+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:00:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:50 smithi077 ceph-mon[37153]: pgmap v8437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:00:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:50 smithi047 ceph-mon[36835]: pgmap v8437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:00:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:50 smithi029 ceph-mon[32610]: pgmap v8437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:00:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:52 smithi077 ceph-mon[37153]: pgmap v8438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:00:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15074.710667 secs 2024-07-27T11:00:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:52 smithi047 ceph-mon[36835]: pgmap v8438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:00:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15074.710667 secs 2024-07-27T11:00:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:52 smithi029 ceph-mon[32610]: pgmap v8438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:00:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15074.710667 secs 2024-07-27T11:00:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:54 smithi077 ceph-mon[37153]: pgmap v8439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T11:00:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:54 smithi047 ceph-mon[36835]: pgmap v8439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T11:00:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:54 smithi029 ceph-mon[32610]: pgmap v8439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T11:00:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:56 smithi077 ceph-mon[37153]: pgmap v8440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:00:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:56 smithi047 ceph-mon[36835]: pgmap v8440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:00:57.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:56 smithi029 ceph-mon[32610]: pgmap v8440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:00:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15079.710789 secs 2024-07-27T11:00:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15079.710789 secs 2024-07-27T11:00:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15079.710789 secs 2024-07-27T11:00:59.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:00:58 smithi047 ceph-mon[36835]: pgmap v8441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:00:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:00:58 smithi077 ceph-mon[37153]: pgmap v8441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:00:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:00:58 smithi029 ceph-mon[32610]: pgmap v8441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:01:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:00 smithi077 ceph-mon[37153]: pgmap v8442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:00 smithi047 ceph-mon[36835]: pgmap v8442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:01.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:00 smithi029 ceph-mon[32610]: pgmap v8442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:02 smithi077 ceph-mon[37153]: pgmap v8443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15084.710905 secs 2024-07-27T11:01:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:02 smithi047 ceph-mon[36835]: pgmap v8443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15084.710905 secs 2024-07-27T11:01:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:02 smithi029 ceph-mon[32610]: pgmap v8443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15084.710905 secs 2024-07-27T11:01:04.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:04.957+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:01:04.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:04.957+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:01:04.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:04.957+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:01:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:04 smithi077 ceph-mon[37153]: pgmap v8444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:04 smithi047 ceph-mon[36835]: pgmap v8444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:05.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:04 smithi029 ceph-mon[32610]: pgmap v8444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:06 smithi077 ceph-mon[37153]: pgmap v8445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:01:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:06 smithi047 ceph-mon[36835]: pgmap v8445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:01:07.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:06 smithi029 ceph-mon[32610]: pgmap v8445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:01:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15089.711039 secs 2024-07-27T11:01:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15089.711039 secs 2024-07-27T11:01:08.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15089.711039 secs 2024-07-27T11:01:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:08 smithi077 ceph-mon[37153]: pgmap v8446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:08 smithi047 ceph-mon[36835]: pgmap v8446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:08 smithi029 ceph-mon[32610]: pgmap v8446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:10 smithi077 ceph-mon[37153]: pgmap v8447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:10 smithi047 ceph-mon[36835]: pgmap v8447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:10 smithi029 ceph-mon[32610]: pgmap v8447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:12 smithi077 ceph-mon[37153]: pgmap v8448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15094.711107 secs 2024-07-27T11:01:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:12 smithi047 ceph-mon[36835]: pgmap v8448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15094.711107 secs 2024-07-27T11:01:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:12 smithi029 ceph-mon[32610]: pgmap v8448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15094.711107 secs 2024-07-27T11:01:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:14 smithi077 ceph-mon[37153]: pgmap v8449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:14 smithi047 ceph-mon[36835]: pgmap v8449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:14 smithi029 ceph-mon[32610]: pgmap v8449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:16 smithi077 ceph-mon[37153]: pgmap v8450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:01:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:16 smithi047 ceph-mon[36835]: pgmap v8450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:01:17.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:16 smithi029 ceph-mon[32610]: pgmap v8450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:01:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15099.711228 secs 2024-07-27T11:01:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15099.711228 secs 2024-07-27T11:01:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15099.711228 secs 2024-07-27T11:01:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:18 smithi047 ceph-mon[36835]: pgmap v8451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:18 smithi029 ceph-mon[32610]: pgmap v8451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:18 smithi077 ceph-mon[37153]: pgmap v8451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:19.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:19.965+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:01:19.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:19.965+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:01:19.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:19.965+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:01:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:20 smithi047 ceph-mon[36835]: pgmap v8452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:20 smithi029 ceph-mon[32610]: pgmap v8452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:20 smithi077 ceph-mon[37153]: pgmap v8452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:22 smithi047 ceph-mon[36835]: pgmap v8453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15104.711352 secs 2024-07-27T11:01:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:22 smithi029 ceph-mon[32610]: pgmap v8453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15104.711352 secs 2024-07-27T11:01:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:22 smithi077 ceph-mon[37153]: pgmap v8453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15104.711352 secs 2024-07-27T11:01:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:24 smithi047 ceph-mon[36835]: pgmap v8454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:25.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:24 smithi029 ceph-mon[32610]: pgmap v8454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:24 smithi077 ceph-mon[37153]: pgmap v8454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:26 smithi047 ceph-mon[36835]: pgmap v8455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:01:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:26 smithi029 ceph-mon[32610]: pgmap v8455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:01:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:26 smithi077 ceph-mon[37153]: pgmap v8455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:01:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15109.711490 secs 2024-07-27T11:01:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15109.711490 secs 2024-07-27T11:01:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15109.711490 secs 2024-07-27T11:01:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:28 smithi047 ceph-mon[36835]: pgmap v8456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:29.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:28 smithi029 ceph-mon[32610]: pgmap v8456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:28 smithi077 ceph-mon[37153]: pgmap v8456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:30 smithi047 ceph-mon[36835]: pgmap v8457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:30 smithi029 ceph-mon[32610]: pgmap v8457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:30 smithi077 ceph-mon[37153]: pgmap v8457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:33.158 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:32 smithi077 ceph-mon[37153]: pgmap v8458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:33.158 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15114.711634 secs 2024-07-27T11:01:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:32 smithi047 ceph-mon[36835]: pgmap v8458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15114.711634 secs 2024-07-27T11:01:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:32 smithi029 ceph-mon[32610]: pgmap v8458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15114.711634 secs 2024-07-27T11:01:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:34 smithi077 ceph-mon[37153]: pgmap v8459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:34 smithi047 ceph-mon[36835]: pgmap v8459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:34 smithi029 ceph-mon[32610]: pgmap v8459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:34.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:34.980+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:01:34.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:34.980+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:01:34.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:34.980+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:01:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:36 smithi077 ceph-mon[37153]: pgmap v8460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:01:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:36 smithi047 ceph-mon[36835]: pgmap v8460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:01:36.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:36 smithi029 ceph-mon[32610]: pgmap v8460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:01:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15119.711732 secs 2024-07-27T11:01:37.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15119.711732 secs 2024-07-27T11:01:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15119.711732 secs 2024-07-27T11:01:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:38 smithi077 ceph-mon[37153]: pgmap v8461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:38 smithi047 ceph-mon[36835]: pgmap v8461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:38 smithi029 ceph-mon[32610]: pgmap v8461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:40 smithi077 ceph-mon[37153]: pgmap v8462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:40 smithi047 ceph-mon[36835]: pgmap v8462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:40 smithi029 ceph-mon[32610]: pgmap v8462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:01:42.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:42 smithi077 ceph-mon[37153]: pgmap v8463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:42.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15124.711836 secs 2024-07-27T11:01:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:42 smithi047 ceph-mon[36835]: pgmap v8463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15124.711836 secs 2024-07-27T11:01:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:42 smithi029 ceph-mon[32610]: pgmap v8463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15124.711836 secs 2024-07-27T11:01:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:44 smithi077 ceph-mon[37153]: pgmap v8464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:44 smithi047 ceph-mon[36835]: pgmap v8464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:44.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:44 smithi029 ceph-mon[32610]: pgmap v8464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:01:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:46 smithi077 ceph-mon[37153]: pgmap v8465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:01:46.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:46 smithi047 ceph-mon[36835]: pgmap v8465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:01:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:46 smithi029 ceph-mon[32610]: pgmap v8465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:01:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15129.711930 secs 2024-07-27T11:01:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15129.711930 secs 2024-07-27T11:01:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15129.711930 secs 2024-07-27T11:01:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:48 smithi029 ceph-mon[32610]: pgmap v8466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:01:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:01:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:01:48.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:48 smithi029 ceph-mon[32610]: pgmap v8467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8 op/s 2024-07-27T11:01:48.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:48 smithi029 ceph-mon[32610]: pgmap v8468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:01:48.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:01:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:48 smithi077 ceph-mon[37153]: pgmap v8466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:01:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:01:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:01:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:48 smithi077 ceph-mon[37153]: pgmap v8467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8 op/s 2024-07-27T11:01:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:48 smithi077 ceph-mon[37153]: pgmap v8468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:01:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:01:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:48 smithi047 ceph-mon[36835]: pgmap v8466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:01:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:01:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:01:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:01:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:48 smithi047 ceph-mon[36835]: pgmap v8467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8 op/s 2024-07-27T11:01:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:48 smithi047 ceph-mon[36835]: pgmap v8468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:01:48.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:01:49.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:49.995+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:01:49.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:49.995+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:01:49.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:49.995+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:01:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:50 smithi077 ceph-mon[37153]: pgmap v8469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:01:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:50 smithi047 ceph-mon[36835]: pgmap v8469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:01:51.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:50 smithi029 ceph-mon[32610]: pgmap v8469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:01:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:52 smithi077 ceph-mon[37153]: pgmap v8470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:01:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15134.712013 secs 2024-07-27T11:01:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:52 smithi047 ceph-mon[36835]: pgmap v8470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:01:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15134.712013 secs 2024-07-27T11:01:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:52 smithi029 ceph-mon[32610]: pgmap v8470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:01:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15134.712013 secs 2024-07-27T11:01:55.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:54 smithi077 ceph-mon[37153]: pgmap v8471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:01:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:54 smithi047 ceph-mon[36835]: pgmap v8471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:01:55.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:54 smithi029 ceph-mon[32610]: pgmap v8471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:01:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:56 smithi077 ceph-mon[37153]: pgmap v8472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:01:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:56 smithi047 ceph-mon[36835]: pgmap v8472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:01:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:56 smithi029 ceph-mon[32610]: pgmap v8472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:01:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15139.712140 secs 2024-07-27T11:01:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15139.712140 secs 2024-07-27T11:01:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15139.712140 secs 2024-07-27T11:01:59.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:01:58 smithi047 ceph-mon[36835]: pgmap v8473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:01:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:01:58 smithi077 ceph-mon[37153]: pgmap v8473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:01:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:01:58 smithi029 ceph-mon[32610]: pgmap v8473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:01:59.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:59.936+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:01:59.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:59.936+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:01:59.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:59.937+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:01:59.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:01:59.937+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:02:01.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:00 smithi077 ceph-mon[37153]: pgmap v8474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:00 smithi047 ceph-mon[36835]: pgmap v8474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:01.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:00 smithi029 ceph-mon[32610]: pgmap v8474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:02 smithi077 ceph-mon[37153]: pgmap v8475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15144.712262 secs 2024-07-27T11:02:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:02 smithi047 ceph-mon[36835]: pgmap v8475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15144.712262 secs 2024-07-27T11:02:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:02 smithi029 ceph-mon[32610]: pgmap v8475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15144.712262 secs 2024-07-27T11:02:05.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:02:05.010+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:02:05.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:02:05.010+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:02:05.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:02:05.010+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:02:05.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:04 smithi077 ceph-mon[37153]: pgmap v8476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:04 smithi047 ceph-mon[36835]: pgmap v8476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:04 smithi029 ceph-mon[32610]: pgmap v8476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:06 smithi077 ceph-mon[37153]: pgmap v8477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:06 smithi047 ceph-mon[36835]: pgmap v8477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:06 smithi029 ceph-mon[32610]: pgmap v8477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15149.712382 secs 2024-07-27T11:02:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15149.712382 secs 2024-07-27T11:02:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15149.712382 secs 2024-07-27T11:02:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:08 smithi047 ceph-mon[36835]: pgmap v8478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:08 smithi029 ceph-mon[32610]: pgmap v8478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:08 smithi077 ceph-mon[37153]: pgmap v8478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:10 smithi047 ceph-mon[36835]: pgmap v8479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:10 smithi029 ceph-mon[32610]: pgmap v8479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:10 smithi077 ceph-mon[37153]: pgmap v8479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:12 smithi047 ceph-mon[36835]: pgmap v8480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15154.712531 secs 2024-07-27T11:02:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:12 smithi029 ceph-mon[32610]: pgmap v8480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15154.712531 secs 2024-07-27T11:02:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:12 smithi077 ceph-mon[37153]: pgmap v8480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15154.712531 secs 2024-07-27T11:02:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:14 smithi047 ceph-mon[36835]: pgmap v8481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:15.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:14 smithi029 ceph-mon[32610]: pgmap v8481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:14 smithi077 ceph-mon[37153]: pgmap v8481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:16 smithi047 ceph-mon[36835]: pgmap v8482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:16 smithi029 ceph-mon[32610]: pgmap v8482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:16 smithi077 ceph-mon[37153]: pgmap v8482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15159.712619 secs 2024-07-27T11:02:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15159.712619 secs 2024-07-27T11:02:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15159.712619 secs 2024-07-27T11:02:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:18 smithi047 ceph-mon[36835]: pgmap v8483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:18 smithi029 ceph-mon[32610]: pgmap v8483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:18 smithi077 ceph-mon[37153]: pgmap v8483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:20.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:02:20.026+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:02:20.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:02:20.026+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:02:20.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:02:20.026+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:02:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:20 smithi047 ceph-mon[36835]: pgmap v8484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:21.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:20 smithi029 ceph-mon[32610]: pgmap v8484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:20 smithi077 ceph-mon[37153]: pgmap v8484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:22 smithi047 ceph-mon[36835]: pgmap v8485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15164.712780 secs 2024-07-27T11:02:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:22 smithi029 ceph-mon[32610]: pgmap v8485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15164.712780 secs 2024-07-27T11:02:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:22 smithi077 ceph-mon[37153]: pgmap v8485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15164.712780 secs 2024-07-27T11:02:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:24 smithi047 ceph-mon[36835]: pgmap v8486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:24 smithi029 ceph-mon[32610]: pgmap v8486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:24 smithi077 ceph-mon[37153]: pgmap v8486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:26 smithi047 ceph-mon[36835]: pgmap v8487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:27.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:26 smithi029 ceph-mon[32610]: pgmap v8487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:26 smithi077 ceph-mon[37153]: pgmap v8487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15169.712837 secs 2024-07-27T11:02:28.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15169.712837 secs 2024-07-27T11:02:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15169.712837 secs 2024-07-27T11:02:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:28 smithi047 ceph-mon[36835]: pgmap v8488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:28 smithi029 ceph-mon[32610]: pgmap v8488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:28 smithi077 ceph-mon[37153]: pgmap v8488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:30 smithi047 ceph-mon[36835]: pgmap v8489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:31.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:30 smithi029 ceph-mon[32610]: pgmap v8489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:31.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:30 smithi077 ceph-mon[37153]: pgmap v8489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:32 smithi047 ceph-mon[36835]: pgmap v8490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15174.712907 secs 2024-07-27T11:02:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:32 smithi029 ceph-mon[32610]: pgmap v8490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15174.712907 secs 2024-07-27T11:02:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:32 smithi077 ceph-mon[37153]: pgmap v8490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15174.712907 secs 2024-07-27T11:02:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:02:35.042+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:02:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:02:35.042+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:02:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:02:35.042+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:02:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:34 smithi047 ceph-mon[36835]: pgmap v8491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:34 smithi029 ceph-mon[32610]: pgmap v8491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:34 smithi077 ceph-mon[37153]: pgmap v8491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:36 smithi047 ceph-mon[36835]: pgmap v8492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:36 smithi029 ceph-mon[32610]: pgmap v8492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:36 smithi077 ceph-mon[37153]: pgmap v8492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15179.713029 secs 2024-07-27T11:02:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15179.713029 secs 2024-07-27T11:02:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15179.713029 secs 2024-07-27T11:02:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:38 smithi047 ceph-mon[36835]: pgmap v8493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:38 smithi029 ceph-mon[32610]: pgmap v8493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:38 smithi077 ceph-mon[37153]: pgmap v8493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:40 smithi047 ceph-mon[36835]: pgmap v8494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:40 smithi029 ceph-mon[32610]: pgmap v8494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:40 smithi077 ceph-mon[37153]: pgmap v8494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:02:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:42 smithi047 ceph-mon[36835]: pgmap v8495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15184.713151 secs 2024-07-27T11:02:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:42 smithi029 ceph-mon[32610]: pgmap v8495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15184.713151 secs 2024-07-27T11:02:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:42 smithi077 ceph-mon[37153]: pgmap v8495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:02:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15184.713151 secs 2024-07-27T11:02:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:44 smithi047 ceph-mon[36835]: pgmap v8496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:44 smithi029 ceph-mon[32610]: pgmap v8496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:44 smithi077 ceph-mon[37153]: pgmap v8496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:02:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:46 smithi047 ceph-mon[36835]: pgmap v8497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:47.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:46 smithi029 ceph-mon[32610]: pgmap v8497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:46 smithi077 ceph-mon[37153]: pgmap v8497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:02:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15189.713268 secs 2024-07-27T11:02:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:02:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15189.713268 secs 2024-07-27T11:02:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:02:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15189.713268 secs 2024-07-27T11:02:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:02:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:48 smithi047 ceph-mon[36835]: pgmap v8498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:02:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:02:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:02:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:48 smithi029 ceph-mon[32610]: pgmap v8498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:02:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:02:49.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:02:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:48 smithi077 ceph-mon[37153]: pgmap v8498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:02:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:02:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:02:49.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:02:50.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:02:50.054+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:02:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:02:50.054+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:02:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:02:50.054+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:02:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:49 smithi029 ceph-mon[32610]: pgmap v8499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:02:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:49 smithi029 ceph-mon[32610]: pgmap v8500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:02:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:49 smithi077 ceph-mon[37153]: pgmap v8499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:02:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:49 smithi077 ceph-mon[37153]: pgmap v8500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:02:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:49 smithi047 ceph-mon[36835]: pgmap v8499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:02:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:49 smithi047 ceph-mon[36835]: pgmap v8500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:02:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:50 smithi029 ceph-mon[32610]: pgmap v8501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:02:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:50 smithi077 ceph-mon[37153]: pgmap v8501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:02:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:50 smithi047 ceph-mon[36835]: pgmap v8501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:02:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15194.713351 secs 2024-07-27T11:02:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15194.713351 secs 2024-07-27T11:02:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15194.713351 secs 2024-07-27T11:02:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:53 smithi077 ceph-mon[37153]: pgmap v8502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T11:02:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:53 smithi047 ceph-mon[36835]: pgmap v8502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T11:02:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:53 smithi029 ceph-mon[32610]: pgmap v8502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T11:02:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:55 smithi047 ceph-mon[36835]: pgmap v8503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:02:55.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:55 smithi029 ceph-mon[32610]: pgmap v8503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:02:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:55 smithi077 ceph-mon[37153]: pgmap v8503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:02:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:57 smithi047 ceph-mon[36835]: pgmap v8504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:02:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15199.713449 secs 2024-07-27T11:02:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:57 smithi029 ceph-mon[32610]: pgmap v8504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:02:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15199.713449 secs 2024-07-27T11:02:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:57 smithi077 ceph-mon[37153]: pgmap v8504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:02:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15199.713449 secs 2024-07-27T11:02:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:02:59 smithi047 ceph-mon[36835]: pgmap v8505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:02:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:02:59 smithi029 ceph-mon[32610]: pgmap v8505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:02:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:02:59 smithi077 ceph-mon[37153]: pgmap v8505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:03:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:01 smithi047 ceph-mon[36835]: pgmap v8506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:01 smithi029 ceph-mon[32610]: pgmap v8506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:01 smithi077 ceph-mon[37153]: pgmap v8506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15204.713589 secs 2024-07-27T11:03:02.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15204.713589 secs 2024-07-27T11:03:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15204.713589 secs 2024-07-27T11:03:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:03 smithi047 ceph-mon[36835]: pgmap v8507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:03 smithi029 ceph-mon[32610]: pgmap v8507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:03 smithi077 ceph-mon[37153]: pgmap v8507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:05.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:03:05.069+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:03:05.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:03:05.069+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:03:05.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:03:05.069+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:03:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:05 smithi047 ceph-mon[36835]: pgmap v8508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:05 smithi029 ceph-mon[32610]: pgmap v8508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:05 smithi077 ceph-mon[37153]: pgmap v8508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:07 smithi047 ceph-mon[36835]: pgmap v8509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15209.713726 secs 2024-07-27T11:03:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:07 smithi029 ceph-mon[32610]: pgmap v8509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15209.713726 secs 2024-07-27T11:03:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:07 smithi077 ceph-mon[37153]: pgmap v8509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:07.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15209.713726 secs 2024-07-27T11:03:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:09 smithi047 ceph-mon[36835]: pgmap v8510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:09 smithi029 ceph-mon[32610]: pgmap v8510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:09 smithi077 ceph-mon[37153]: pgmap v8510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:11 smithi047 ceph-mon[36835]: pgmap v8511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:11 smithi029 ceph-mon[32610]: pgmap v8511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:11 smithi077 ceph-mon[37153]: pgmap v8511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15214.713876 secs 2024-07-27T11:03:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15214.713876 secs 2024-07-27T11:03:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15214.713876 secs 2024-07-27T11:03:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:13 smithi047 ceph-mon[36835]: pgmap v8512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:13.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:13 smithi029 ceph-mon[32610]: pgmap v8512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:13 smithi077 ceph-mon[37153]: pgmap v8512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:15 smithi047 ceph-mon[36835]: pgmap v8513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:15.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:15 smithi029 ceph-mon[32610]: pgmap v8513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:15 smithi077 ceph-mon[37153]: pgmap v8513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:17 smithi047 ceph-mon[36835]: pgmap v8514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15219.713987 secs 2024-07-27T11:03:17.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:17 smithi029 ceph-mon[32610]: pgmap v8514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15219.713987 secs 2024-07-27T11:03:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:17 smithi077 ceph-mon[37153]: pgmap v8514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15219.713987 secs 2024-07-27T11:03:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:19 smithi047 ceph-mon[36835]: pgmap v8515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:19 smithi029 ceph-mon[32610]: pgmap v8515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:19 smithi077 ceph-mon[37153]: pgmap v8515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:20.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:03:20.085+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:03:20.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:03:20.085+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:03:20.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:03:20.085+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:03:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:21 smithi047 ceph-mon[36835]: pgmap v8516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:21 smithi029 ceph-mon[32610]: pgmap v8516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:21 smithi077 ceph-mon[37153]: pgmap v8516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15224.714068 secs 2024-07-27T11:03:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15224.714068 secs 2024-07-27T11:03:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15224.714068 secs 2024-07-27T11:03:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:23 smithi029 ceph-mon[32610]: pgmap v8517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:23 smithi077 ceph-mon[37153]: pgmap v8517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:23 smithi047 ceph-mon[36835]: pgmap v8517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:25 smithi029 ceph-mon[32610]: pgmap v8518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:25 smithi077 ceph-mon[37153]: pgmap v8518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:25 smithi047 ceph-mon[36835]: pgmap v8518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:27 smithi029 ceph-mon[32610]: pgmap v8519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15229.714195 secs 2024-07-27T11:03:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:27 smithi077 ceph-mon[37153]: pgmap v8519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15229.714195 secs 2024-07-27T11:03:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:27 smithi047 ceph-mon[36835]: pgmap v8519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15229.714195 secs 2024-07-27T11:03:29.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:29 smithi047 ceph-mon[36835]: pgmap v8520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:29 smithi077 ceph-mon[37153]: pgmap v8520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:29 smithi029 ceph-mon[32610]: pgmap v8520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:31 smithi077 ceph-mon[37153]: pgmap v8521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:31 smithi047 ceph-mon[36835]: pgmap v8521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:31.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:31 smithi029 ceph-mon[32610]: pgmap v8521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15234.714309 secs 2024-07-27T11:03:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15234.714309 secs 2024-07-27T11:03:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15234.714309 secs 2024-07-27T11:03:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:33 smithi077 ceph-mon[37153]: pgmap v8522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:33 smithi047 ceph-mon[36835]: pgmap v8522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:33 smithi029 ceph-mon[32610]: pgmap v8522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:03:35.101+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:03:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:03:35.101+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:03:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:03:35.101+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:03:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:35 smithi077 ceph-mon[37153]: pgmap v8523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:35 smithi047 ceph-mon[36835]: pgmap v8523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:35.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:35 smithi029 ceph-mon[32610]: pgmap v8523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:37.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:37 smithi029 ceph-mon[32610]: pgmap v8524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:37.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15239.714420 secs 2024-07-27T11:03:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:37 smithi077 ceph-mon[37153]: pgmap v8524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15239.714420 secs 2024-07-27T11:03:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:37 smithi047 ceph-mon[36835]: pgmap v8524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15239.714420 secs 2024-07-27T11:03:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:39 smithi077 ceph-mon[37153]: pgmap v8525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:39 smithi047 ceph-mon[36835]: pgmap v8525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:39.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:39 smithi029 ceph-mon[32610]: pgmap v8525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:41 smithi077 ceph-mon[37153]: pgmap v8526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:41 smithi047 ceph-mon[36835]: pgmap v8526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:41.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:41 smithi029 ceph-mon[32610]: pgmap v8526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:03:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15244.714541 secs 2024-07-27T11:03:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15244.714541 secs 2024-07-27T11:03:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15244.714541 secs 2024-07-27T11:03:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:43 smithi077 ceph-mon[37153]: pgmap v8527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:43 smithi047 ceph-mon[36835]: pgmap v8527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:43 smithi029 ceph-mon[32610]: pgmap v8527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:03:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:45 smithi077 ceph-mon[37153]: pgmap v8528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:45 smithi047 ceph-mon[36835]: pgmap v8528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:45.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:45 smithi029 ceph-mon[32610]: pgmap v8528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:03:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:47 smithi077 ceph-mon[37153]: pgmap v8529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15249.714664 secs 2024-07-27T11:03:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:47 smithi047 ceph-mon[36835]: pgmap v8529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15249.714664 secs 2024-07-27T11:03:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:47 smithi029 ceph-mon[32610]: pgmap v8529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:03:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15249.714664 secs 2024-07-27T11:03:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:03:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:03:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:03:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:49 smithi077 ceph-mon[37153]: pgmap v8530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:03:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:03:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:49 smithi077 ceph-mon[37153]: pgmap v8531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:03:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:49 smithi077 ceph-mon[37153]: pgmap v8532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:03:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:03:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:49 smithi047 ceph-mon[36835]: pgmap v8530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:03:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:03:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:49 smithi047 ceph-mon[36835]: pgmap v8531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:03:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:49 smithi047 ceph-mon[36835]: pgmap v8532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:03:49.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:03:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:49 smithi029 ceph-mon[32610]: pgmap v8530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:03:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:03:49.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:03:49.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:49 smithi029 ceph-mon[32610]: pgmap v8531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:03:49.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:49 smithi029 ceph-mon[32610]: pgmap v8532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:03:49.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:03:50.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:03:50.101+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:03:50.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:03:50.101+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:03:50.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:03:50.101+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:03:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:51 smithi029 ceph-mon[32610]: pgmap v8533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T11:03:52.363 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:51 smithi077 ceph-mon[37153]: pgmap v8533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T11:03:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:51 smithi047 ceph-mon[36835]: pgmap v8533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T11:03:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15254.714823 secs 2024-07-27T11:03:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15254.714823 secs 2024-07-27T11:03:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15254.714823 secs 2024-07-27T11:03:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:53 smithi077 ceph-mon[37153]: pgmap v8534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:03:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:53 smithi047 ceph-mon[36835]: pgmap v8534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:03:54.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:53 smithi029 ceph-mon[32610]: pgmap v8534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:03:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:54 smithi077 ceph-mon[37153]: pgmap v8535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:03:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:54 smithi047 ceph-mon[36835]: pgmap v8535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:03:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:54 smithi029 ceph-mon[32610]: pgmap v8535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:03:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:57 smithi029 ceph-mon[32610]: pgmap v8536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:03:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15259.714938 secs 2024-07-27T11:03:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:57 smithi077 ceph-mon[37153]: pgmap v8536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:03:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15259.714938 secs 2024-07-27T11:03:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:57 smithi047 ceph-mon[36835]: pgmap v8536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:03:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15259.714938 secs 2024-07-27T11:03:59.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:03:59 smithi047 ceph-mon[36835]: pgmap v8537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:03:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:03:59 smithi077 ceph-mon[37153]: pgmap v8537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:03:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:03:59 smithi029 ceph-mon[32610]: pgmap v8537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:04:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:01 smithi029 ceph-mon[32610]: pgmap v8538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:01 smithi077 ceph-mon[37153]: pgmap v8538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:01 smithi047 ceph-mon[36835]: pgmap v8538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15264.715038 secs 2024-07-27T11:04:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:02 smithi077 ceph-mon[37153]: pgmap v8539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15264.715038 secs 2024-07-27T11:04:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:02 smithi047 ceph-mon[36835]: pgmap v8539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15264.715038 secs 2024-07-27T11:04:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:02 smithi029 ceph-mon[32610]: pgmap v8539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:05.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:05.109+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:04:05.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:05.109+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:04:05.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:05.109+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:04:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:05 smithi077 ceph-mon[37153]: pgmap v8540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:05 smithi047 ceph-mon[36835]: pgmap v8540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:05 smithi029 ceph-mon[32610]: pgmap v8540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:07.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:07 smithi029 ceph-mon[32610]: pgmap v8541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:07 smithi077 ceph-mon[37153]: pgmap v8541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:07 smithi047 ceph-mon[36835]: pgmap v8541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15269.715165 secs 2024-07-27T11:04:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15269.715165 secs 2024-07-27T11:04:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15269.715165 secs 2024-07-27T11:04:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:09 smithi077 ceph-mon[37153]: pgmap v8542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:09 smithi047 ceph-mon[36835]: pgmap v8542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:09 smithi029 ceph-mon[32610]: pgmap v8542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:11 smithi077 ceph-mon[37153]: pgmap v8543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:11 smithi047 ceph-mon[36835]: pgmap v8543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:11 smithi029 ceph-mon[32610]: pgmap v8543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15274.715284 secs 2024-07-27T11:04:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15274.715284 secs 2024-07-27T11:04:12.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15274.715284 secs 2024-07-27T11:04:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:13 smithi047 ceph-mon[36835]: pgmap v8544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:13 smithi029 ceph-mon[32610]: pgmap v8544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:13 smithi077 ceph-mon[37153]: pgmap v8544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:15 smithi077 ceph-mon[37153]: pgmap v8545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:15 smithi047 ceph-mon[36835]: pgmap v8545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:15 smithi029 ceph-mon[32610]: pgmap v8545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:17 smithi077 ceph-mon[37153]: pgmap v8546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:17 smithi047 ceph-mon[36835]: pgmap v8546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:17 smithi029 ceph-mon[32610]: pgmap v8546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15279.715412 secs 2024-07-27T11:04:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15279.715412 secs 2024-07-27T11:04:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15279.715412 secs 2024-07-27T11:04:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:19 smithi077 ceph-mon[37153]: pgmap v8547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:19 smithi047 ceph-mon[36835]: pgmap v8547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:19 smithi029 ceph-mon[32610]: pgmap v8547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:20.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:20.111+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:04:20.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:20.111+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:04:20.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:20.111+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:04:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:21 smithi077 ceph-mon[37153]: pgmap v8548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:21 smithi047 ceph-mon[36835]: pgmap v8548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:21.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:21 smithi029 ceph-mon[32610]: pgmap v8548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15284.715492 secs 2024-07-27T11:04:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15284.715492 secs 2024-07-27T11:04:22.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15284.715492 secs 2024-07-27T11:04:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:23 smithi047 ceph-mon[36835]: pgmap v8549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:23 smithi029 ceph-mon[32610]: pgmap v8549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:23 smithi077 ceph-mon[37153]: pgmap v8549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:25 smithi029 ceph-mon[32610]: pgmap v8550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:25 smithi077 ceph-mon[37153]: pgmap v8550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:25 smithi047 ceph-mon[36835]: pgmap v8550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:27 smithi077 ceph-mon[37153]: pgmap v8551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:27 smithi047 ceph-mon[36835]: pgmap v8551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:27.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:27 smithi029 ceph-mon[32610]: pgmap v8551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15289.715643 secs 2024-07-27T11:04:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15289.715643 secs 2024-07-27T11:04:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15289.715643 secs 2024-07-27T11:04:29.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:29 smithi047 ceph-mon[36835]: pgmap v8552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:29 smithi077 ceph-mon[37153]: pgmap v8552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:29 smithi029 ceph-mon[32610]: pgmap v8552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:31 smithi077 ceph-mon[37153]: pgmap v8553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:31 smithi047 ceph-mon[36835]: pgmap v8553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:31 smithi029 ceph-mon[32610]: pgmap v8553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15294.715819 secs 2024-07-27T11:04:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15294.715819 secs 2024-07-27T11:04:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15294.715819 secs 2024-07-27T11:04:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:33 smithi047 ceph-mon[36835]: pgmap v8554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:33.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:33 smithi029 ceph-mon[32610]: pgmap v8554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:33 smithi077 ceph-mon[37153]: pgmap v8554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:35.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:35.127+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:04:35.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:35.127+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:04:35.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:35.127+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:04:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:35 smithi077 ceph-mon[37153]: pgmap v8555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:35 smithi047 ceph-mon[36835]: pgmap v8555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:35 smithi029 ceph-mon[32610]: pgmap v8555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:37.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:37 smithi029 ceph-mon[32610]: pgmap v8556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:37 smithi077 ceph-mon[37153]: pgmap v8556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:37 smithi047 ceph-mon[36835]: pgmap v8556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:38.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15299.715942 secs 2024-07-27T11:04:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15299.715942 secs 2024-07-27T11:04:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15299.715942 secs 2024-07-27T11:04:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:39 smithi077 ceph-mon[37153]: pgmap v8557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:39 smithi047 ceph-mon[36835]: pgmap v8557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:39.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:39 smithi029 ceph-mon[32610]: pgmap v8557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:41 smithi077 ceph-mon[37153]: pgmap v8558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:41 smithi047 ceph-mon[36835]: pgmap v8558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:41.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:41 smithi029 ceph-mon[32610]: pgmap v8558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:04:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15304.716011 secs 2024-07-27T11:04:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15304.716011 secs 2024-07-27T11:04:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15304.716011 secs 2024-07-27T11:04:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:43 smithi047 ceph-mon[36835]: pgmap v8559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:43 smithi029 ceph-mon[32610]: pgmap v8559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:43 smithi077 ceph-mon[37153]: pgmap v8559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:04:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:45 smithi077 ceph-mon[37153]: pgmap v8560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:45 smithi047 ceph-mon[36835]: pgmap v8560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:45 smithi029 ceph-mon[32610]: pgmap v8560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:04:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:47 smithi077 ceph-mon[37153]: pgmap v8561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:47 smithi047 ceph-mon[36835]: pgmap v8561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:47.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:47 smithi029 ceph-mon[32610]: pgmap v8561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:04:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15309.716114 secs 2024-07-27T11:04:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15309.716114 secs 2024-07-27T11:04:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15309.716114 secs 2024-07-27T11:04:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:49 smithi077 ceph-mon[37153]: pgmap v8562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:04:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:49 smithi047 ceph-mon[36835]: pgmap v8562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:04:49.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:49 smithi029 ceph-mon[32610]: pgmap v8562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:04:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:04:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:50.141+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:04:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:50.141+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:04:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:50.141+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:04:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:04:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:04:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:50 smithi077 ceph-mon[37153]: pgmap v8563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:04:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:50 smithi077 ceph-mon[37153]: pgmap v8564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:04:50.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:04:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:04:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:04:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:50 smithi047 ceph-mon[36835]: pgmap v8563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:04:50.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:50 smithi047 ceph-mon[36835]: pgmap v8564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:04:50.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:04:50.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:04:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:04:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:50 smithi029 ceph-mon[32610]: pgmap v8563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:04:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:50 smithi029 ceph-mon[32610]: pgmap v8564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:04:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:04:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:52 smithi077 ceph-mon[37153]: pgmap v8565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:04:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15314.716239 secs 2024-07-27T11:04:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:52 smithi047 ceph-mon[36835]: pgmap v8565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:04:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15314.716239 secs 2024-07-27T11:04:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:52 smithi029 ceph-mon[32610]: pgmap v8565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:04:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15314.716239 secs 2024-07-27T11:04:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:54 smithi077 ceph-mon[37153]: pgmap v8566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:04:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:54 smithi047 ceph-mon[36835]: pgmap v8566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:04:54.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:54 smithi029 ceph-mon[32610]: pgmap v8566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:04:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:56 smithi077 ceph-mon[37153]: pgmap v8567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:04:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:56 smithi047 ceph-mon[36835]: pgmap v8567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:04:56.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:56 smithi029 ceph-mon[32610]: pgmap v8567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:04:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:57.667+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 msgr2=0x7eff3c0b5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T11:04:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:57.667+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 msgr2=0x7eff3c0b5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T11:04:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:57.667+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7eff40004a10 tx=0x7eff4002b070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T11:04:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:57.667+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7eff40004a10 tx=0x7eff4002b070 comp rx=0 tx=0).stop 2024-07-27T11:04:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:57.667+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 msgr2=0x7eff3c0b5ed0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T11:04:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:57.667+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 msgr2=0x7eff3c0b5ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T11:04:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:57.667+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T11:04:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:57.667+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T11:04:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:57.667+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T11:04:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:57.668+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7eff40004a10 tx=0x7eff4002b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T11:04:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15319.716360 secs 2024-07-27T11:04:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15319.716360 secs 2024-07-27T11:04:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15319.716360 secs 2024-07-27T11:04:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:04:58 smithi077 ceph-mon[37153]: pgmap v8568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:04:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:04:58 smithi047 ceph-mon[36835]: pgmap v8568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:04:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:04:58 smithi029 ceph-mon[32610]: pgmap v8568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:04:59.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:59.964+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:04:59.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:59.964+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:04:59.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:59.964+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:04:59.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:04:59.964+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:05:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:00 smithi077 ceph-mon[37153]: pgmap v8569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:05:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:00 smithi047 ceph-mon[36835]: pgmap v8569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:05:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:00 smithi029 ceph-mon[32610]: pgmap v8569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:05:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:02 smithi077 ceph-mon[37153]: pgmap v8570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15324.716490 secs 2024-07-27T11:05:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:02 smithi047 ceph-mon[36835]: pgmap v8570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15324.716490 secs 2024-07-27T11:05:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:02 smithi029 ceph-mon[32610]: pgmap v8570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15324.716490 secs 2024-07-27T11:05:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:04 smithi077 ceph-mon[37153]: pgmap v8571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:04 smithi047 ceph-mon[36835]: pgmap v8571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:04.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:04 smithi029 ceph-mon[32610]: pgmap v8571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:05.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:05.149+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:05:05.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:05.149+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:05:05.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:05.149+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:05:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:06 smithi077 ceph-mon[37153]: pgmap v8572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:06 smithi047 ceph-mon[36835]: pgmap v8572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:06.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:06 smithi029 ceph-mon[32610]: pgmap v8572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15329.716614 secs 2024-07-27T11:05:07.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15329.716614 secs 2024-07-27T11:05:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15329.716614 secs 2024-07-27T11:05:08.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:08 smithi077 ceph-mon[37153]: pgmap v8573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:05:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:08 smithi047 ceph-mon[36835]: pgmap v8573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:05:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:08 smithi029 ceph-mon[32610]: pgmap v8573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:05:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:10 smithi077 ceph-mon[37153]: pgmap v8574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:10 smithi047 ceph-mon[36835]: pgmap v8574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:10 smithi029 ceph-mon[32610]: pgmap v8574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:12 smithi077 ceph-mon[37153]: pgmap v8575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15334.716741 secs 2024-07-27T11:05:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:12 smithi047 ceph-mon[36835]: pgmap v8575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15334.716741 secs 2024-07-27T11:05:12.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:12 smithi029 ceph-mon[32610]: pgmap v8575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15334.716741 secs 2024-07-27T11:05:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:14 smithi077 ceph-mon[37153]: pgmap v8576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:14 smithi047 ceph-mon[36835]: pgmap v8576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:14 smithi029 ceph-mon[32610]: pgmap v8576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:16 smithi077 ceph-mon[37153]: pgmap v8577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:05:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:16 smithi047 ceph-mon[36835]: pgmap v8577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:05:16.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:16 smithi029 ceph-mon[32610]: pgmap v8577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:05:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15339.716861 secs 2024-07-27T11:05:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15339.716861 secs 2024-07-27T11:05:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15339.716861 secs 2024-07-27T11:05:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:18 smithi077 ceph-mon[37153]: pgmap v8578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:05:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:18 smithi047 ceph-mon[36835]: pgmap v8578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:05:18.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:18 smithi029 ceph-mon[32610]: pgmap v8578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:05:20.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:20.155+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:05:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:20.155+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:05:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:20.155+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:05:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:20 smithi077 ceph-mon[37153]: pgmap v8579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:20 smithi047 ceph-mon[36835]: pgmap v8579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:20 smithi029 ceph-mon[32610]: pgmap v8579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:22 smithi077 ceph-mon[37153]: pgmap v8580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15344.716981 secs 2024-07-27T11:05:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:22 smithi047 ceph-mon[36835]: pgmap v8580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15344.716981 secs 2024-07-27T11:05:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:22 smithi029 ceph-mon[32610]: pgmap v8580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15344.716981 secs 2024-07-27T11:05:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:24 smithi077 ceph-mon[37153]: pgmap v8581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:24 smithi047 ceph-mon[36835]: pgmap v8581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:24.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:24 smithi029 ceph-mon[32610]: pgmap v8581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:26 smithi077 ceph-mon[37153]: pgmap v8582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:05:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:26 smithi047 ceph-mon[36835]: pgmap v8582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:05:26.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:26 smithi029 ceph-mon[32610]: pgmap v8582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:05:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15349.717102 secs 2024-07-27T11:05:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15349.717102 secs 2024-07-27T11:05:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15349.717102 secs 2024-07-27T11:05:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:28 smithi077 ceph-mon[37153]: pgmap v8583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:05:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:28 smithi047 ceph-mon[36835]: pgmap v8583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:05:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:28 smithi029 ceph-mon[32610]: pgmap v8583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:05:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:30 smithi077 ceph-mon[37153]: pgmap v8584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:30 smithi047 ceph-mon[36835]: pgmap v8584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:30.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:30 smithi029 ceph-mon[32610]: pgmap v8584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:32 smithi077 ceph-mon[37153]: pgmap v8585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15354.717249 secs 2024-07-27T11:05:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:32 smithi047 ceph-mon[36835]: pgmap v8585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15354.717249 secs 2024-07-27T11:05:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:32 smithi029 ceph-mon[32610]: pgmap v8585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15354.717249 secs 2024-07-27T11:05:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:34 smithi077 ceph-mon[37153]: pgmap v8586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:34 smithi047 ceph-mon[36835]: pgmap v8586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:34.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:34 smithi029 ceph-mon[32610]: pgmap v8586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:35.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:35.171+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:05:35.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:35.171+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:05:35.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:35.171+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:05:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:36 smithi077 ceph-mon[37153]: pgmap v8587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:05:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:36 smithi047 ceph-mon[36835]: pgmap v8587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:05:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:36 smithi029 ceph-mon[32610]: pgmap v8587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:05:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15359.717388 secs 2024-07-27T11:05:37.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15359.717388 secs 2024-07-27T11:05:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15359.717388 secs 2024-07-27T11:05:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:38 smithi047 ceph-mon[36835]: pgmap v8588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:05:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:38 smithi029 ceph-mon[32610]: pgmap v8588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:05:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:38 smithi077 ceph-mon[37153]: pgmap v8588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:05:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:40 smithi047 ceph-mon[36835]: pgmap v8589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:40.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:40 smithi029 ceph-mon[32610]: pgmap v8589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:40 smithi077 ceph-mon[37153]: pgmap v8589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:05:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:42 smithi047 ceph-mon[36835]: pgmap v8590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15364.717517 secs 2024-07-27T11:05:42.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:42 smithi029 ceph-mon[32610]: pgmap v8590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:42.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15364.717517 secs 2024-07-27T11:05:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:42 smithi077 ceph-mon[37153]: pgmap v8590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:05:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15364.717517 secs 2024-07-27T11:05:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:44 smithi047 ceph-mon[36835]: pgmap v8591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:44.977 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:44 smithi029 ceph-mon[32610]: pgmap v8591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:44.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:44.970+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:05:44.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:44.970+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:05:44.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:44.971+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:05:44.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:44.971+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:05:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:44 smithi077 ceph-mon[37153]: pgmap v8591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:46 smithi047 ceph-mon[36835]: pgmap v8592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:05:46.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:46 smithi029 ceph-mon[32610]: pgmap v8592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:05:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:46 smithi077 ceph-mon[37153]: pgmap v8592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:05:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15369.717657 secs 2024-07-27T11:05:47.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15369.717657 secs 2024-07-27T11:05:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15369.717657 secs 2024-07-27T11:05:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:48 smithi047 ceph-mon[36835]: pgmap v8593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:05:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:48 smithi029 ceph-mon[32610]: pgmap v8593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:05:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:48 smithi077 ceph-mon[37153]: pgmap v8593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:05:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:05:49.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:05:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:05:50.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:50.186+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:05:50.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:50.186+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:05:50.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:05:50.186+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:05:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:50 smithi047 ceph-mon[36835]: pgmap v8594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:05:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:05:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:05:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:05:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:05:50.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:05:50.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:05:50.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:05:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:50 smithi029 ceph-mon[32610]: pgmap v8594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:05:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:05:50.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:05:50.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:05:50.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:05:50.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:05:50.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:05:50.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:05:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:50 smithi077 ceph-mon[37153]: pgmap v8594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:05:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:05:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:05:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:05:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:05:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:05:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:05:51.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:05:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:51 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:05:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:51 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:05:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:51 smithi047 ceph-mon[36835]: pgmap v8595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:05:51.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:51 smithi047 ceph-mon[36835]: pgmap v8596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:05:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:51 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:05:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:51 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:05:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:51 smithi029 ceph-mon[32610]: pgmap v8595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:05:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:51 smithi029 ceph-mon[32610]: pgmap v8596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:05:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:51 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:05:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:51 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:05:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:51 smithi077 ceph-mon[37153]: pgmap v8595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:05:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:51 smithi077 ceph-mon[37153]: pgmap v8596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:05:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15374.717759 secs 2024-07-27T11:05:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15374.717759 secs 2024-07-27T11:05:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15374.717759 secs 2024-07-27T11:05:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:53 smithi047 ceph-mon[36835]: pgmap v8597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:05:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:53 smithi029 ceph-mon[32610]: pgmap v8597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:05:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:53 smithi077 ceph-mon[37153]: pgmap v8597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:05:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:55 smithi047 ceph-mon[36835]: pgmap v8598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:55.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:55 smithi029 ceph-mon[32610]: pgmap v8598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:55 smithi077 ceph-mon[37153]: pgmap v8598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:05:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:57 smithi047 ceph-mon[36835]: pgmap v8599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:05:57.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15379.717881 secs 2024-07-27T11:05:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:57 smithi029 ceph-mon[32610]: pgmap v8599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:05:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15379.717881 secs 2024-07-27T11:05:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:57 smithi077 ceph-mon[37153]: pgmap v8599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:05:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15379.717881 secs 2024-07-27T11:05:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:05:59 smithi047 ceph-mon[36835]: pgmap v8600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:05:59.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:05:59 smithi029 ceph-mon[32610]: pgmap v8600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:06:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:05:59 smithi077 ceph-mon[37153]: pgmap v8600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:06:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:01 smithi047 ceph-mon[36835]: pgmap v8601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:06:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:01 smithi029 ceph-mon[32610]: pgmap v8601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:06:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:01 smithi077 ceph-mon[37153]: pgmap v8601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:06:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15384.717984 secs 2024-07-27T11:06:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15384.717984 secs 2024-07-27T11:06:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15384.717984 secs 2024-07-27T11:06:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:03 smithi047 ceph-mon[36835]: pgmap v8602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:03 smithi029 ceph-mon[32610]: pgmap v8602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:03 smithi077 ceph-mon[37153]: pgmap v8602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:05.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:06:05.200+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:06:05.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:06:05.200+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:06:05.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:06:05.200+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:06:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:05 smithi047 ceph-mon[36835]: pgmap v8603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:06:05.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:05 smithi029 ceph-mon[32610]: pgmap v8603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:06:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:05 smithi077 ceph-mon[37153]: pgmap v8603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:06:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:07 smithi047 ceph-mon[36835]: pgmap v8604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:07.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15389.718083 secs 2024-07-27T11:06:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:07 smithi029 ceph-mon[32610]: pgmap v8604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15389.718083 secs 2024-07-27T11:06:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:07 smithi077 ceph-mon[37153]: pgmap v8604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15389.718083 secs 2024-07-27T11:06:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:09 smithi047 ceph-mon[36835]: pgmap v8605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:06:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:09 smithi029 ceph-mon[32610]: pgmap v8605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:06:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:09 smithi077 ceph-mon[37153]: pgmap v8605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:06:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:11 smithi047 ceph-mon[36835]: pgmap v8606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:11 smithi029 ceph-mon[32610]: pgmap v8606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:11 smithi077 ceph-mon[37153]: pgmap v8606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15394.718204 secs 2024-07-27T11:06:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15394.718204 secs 2024-07-27T11:06:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15394.718204 secs 2024-07-27T11:06:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:13 smithi047 ceph-mon[36835]: pgmap v8607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:13 smithi029 ceph-mon[32610]: pgmap v8607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:13 smithi077 ceph-mon[37153]: pgmap v8607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:15 smithi047 ceph-mon[36835]: pgmap v8608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:06:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:15 smithi029 ceph-mon[32610]: pgmap v8608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:06:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:15 smithi077 ceph-mon[37153]: pgmap v8608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:06:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:17 smithi047 ceph-mon[36835]: pgmap v8609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15399.718324 secs 2024-07-27T11:06:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:17 smithi029 ceph-mon[32610]: pgmap v8609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15399.718324 secs 2024-07-27T11:06:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:17 smithi077 ceph-mon[37153]: pgmap v8609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15399.718324 secs 2024-07-27T11:06:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:19 smithi029 ceph-mon[32610]: pgmap v8610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:06:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:19 smithi077 ceph-mon[37153]: pgmap v8610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:06:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:19 smithi047 ceph-mon[36835]: pgmap v8610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:06:20.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:06:20.216+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:06:20.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:06:20.216+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:06:20.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:06:20.216+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:06:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:21 smithi029 ceph-mon[32610]: pgmap v8611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:21 smithi077 ceph-mon[37153]: pgmap v8611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:21 smithi047 ceph-mon[36835]: pgmap v8611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15404.718447 secs 2024-07-27T11:06:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15404.718447 secs 2024-07-27T11:06:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15404.718447 secs 2024-07-27T11:06:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:23 smithi029 ceph-mon[32610]: pgmap v8612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:23 smithi077 ceph-mon[37153]: pgmap v8612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:23 smithi047 ceph-mon[36835]: pgmap v8612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:25 smithi077 ceph-mon[37153]: pgmap v8613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:06:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:25 smithi047 ceph-mon[36835]: pgmap v8613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:06:26.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:25 smithi029 ceph-mon[32610]: pgmap v8613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:06:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:27 smithi077 ceph-mon[37153]: pgmap v8614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:06:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15409.718565 secs 2024-07-27T11:06:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:27 smithi047 ceph-mon[36835]: pgmap v8614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:06:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15409.718565 secs 2024-07-27T11:06:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:27 smithi029 ceph-mon[32610]: pgmap v8614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:06:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15409.718565 secs 2024-07-27T11:06:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:29 smithi077 ceph-mon[37153]: pgmap v8615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:06:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:29 smithi047 ceph-mon[36835]: pgmap v8615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:06:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:29 smithi029 ceph-mon[32610]: pgmap v8615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:06:31.998 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:31 smithi047 ceph-mon[36835]: pgmap v8616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:06:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:31 smithi077 ceph-mon[37153]: pgmap v8616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:06:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:31 smithi029 ceph-mon[32610]: pgmap v8616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:06:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15414.718702 secs 2024-07-27T11:06:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15414.718702 secs 2024-07-27T11:06:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15414.718702 secs 2024-07-27T11:06:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:33 smithi077 ceph-mon[37153]: pgmap v8617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:33 smithi047 ceph-mon[36835]: pgmap v8617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:33 smithi029 ceph-mon[32610]: pgmap v8617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:06:35.231+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:06:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:06:35.231+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:06:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:06:35.231+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:06:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:35 smithi077 ceph-mon[37153]: pgmap v8618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:06:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:35 smithi047 ceph-mon[36835]: pgmap v8618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:06:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:35 smithi029 ceph-mon[32610]: pgmap v8618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:06:38.013 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:37 smithi029 ceph-mon[32610]: pgmap v8619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:38.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15419.718823 secs 2024-07-27T11:06:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:37 smithi077 ceph-mon[37153]: pgmap v8619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15419.718823 secs 2024-07-27T11:06:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:37 smithi047 ceph-mon[36835]: pgmap v8619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15419.718823 secs 2024-07-27T11:06:40.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:39 smithi077 ceph-mon[37153]: pgmap v8620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:06:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:39 smithi047 ceph-mon[36835]: pgmap v8620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:06:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:39 smithi029 ceph-mon[32610]: pgmap v8620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:06:42.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:41 smithi077 ceph-mon[37153]: pgmap v8621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:41 smithi047 ceph-mon[36835]: pgmap v8621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:42.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:41 smithi029 ceph-mon[32610]: pgmap v8621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15424.718952 secs 2024-07-27T11:06:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15424.718952 secs 2024-07-27T11:06:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15424.718952 secs 2024-07-27T11:06:44.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:43 smithi077 ceph-mon[37153]: pgmap v8622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:43 smithi047 ceph-mon[36835]: pgmap v8622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:43 smithi029 ceph-mon[32610]: pgmap v8622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:06:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:45 smithi077 ceph-mon[37153]: pgmap v8623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:06:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:45 smithi047 ceph-mon[36835]: pgmap v8623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:06:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:45 smithi029 ceph-mon[32610]: pgmap v8623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:06:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:47 smithi077 ceph-mon[37153]: pgmap v8624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15429.719071 secs 2024-07-27T11:06:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:47 smithi047 ceph-mon[36835]: pgmap v8624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15429.719071 secs 2024-07-27T11:06:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:47 smithi029 ceph-mon[32610]: pgmap v8624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15429.719071 secs 2024-07-27T11:06:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:49 smithi077 ceph-mon[37153]: pgmap v8625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:06:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:49 smithi047 ceph-mon[36835]: pgmap v8625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:06:50.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:49 smithi029 ceph-mon[32610]: pgmap v8625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:06:50.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:06:50.241+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:06:50.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:06:50.241+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:06:50.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:06:50.241+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:06:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:06:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:06:51.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:06:51.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:06:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:06:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:06:51.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:06:51.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:06:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:06:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:06:51.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:06:51.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:06:51.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:06:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:51 smithi077 ceph-mon[37153]: pgmap v8626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:51 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:06:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:51 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:06:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:51 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:06:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:51 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:06:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:51 smithi077 ceph-mon[37153]: pgmap v8627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:06:52.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:51 smithi077 ceph-mon[37153]: pgmap v8628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:06:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:51 smithi047 ceph-mon[36835]: pgmap v8626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:51 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:06:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:51 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:06:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:51 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:06:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:51 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:06:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:51 smithi047 ceph-mon[36835]: pgmap v8627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:06:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:51 smithi047 ceph-mon[36835]: pgmap v8628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:06:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:51 smithi029 ceph-mon[32610]: pgmap v8626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:06:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:51 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:06:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:51 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:06:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:51 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:06:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:51 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:06:52.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:51 smithi029 ceph-mon[32610]: pgmap v8627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:06:52.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:51 smithi029 ceph-mon[32610]: pgmap v8628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:06:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15434.719165 secs 2024-07-27T11:06:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15434.719165 secs 2024-07-27T11:06:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15434.719165 secs 2024-07-27T11:06:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:53 smithi077 ceph-mon[37153]: pgmap v8629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:06:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:53 smithi047 ceph-mon[36835]: pgmap v8629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:06:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:53 smithi029 ceph-mon[32610]: pgmap v8629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:06:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:55 smithi077 ceph-mon[37153]: pgmap v8630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:06:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:55 smithi047 ceph-mon[36835]: pgmap v8630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:06:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:55 smithi029 ceph-mon[32610]: pgmap v8630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:06:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:57 smithi077 ceph-mon[37153]: pgmap v8631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:06:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15439.719330 secs 2024-07-27T11:06:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:57 smithi047 ceph-mon[36835]: pgmap v8631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:06:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15439.719330 secs 2024-07-27T11:06:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:57 smithi029 ceph-mon[32610]: pgmap v8631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:06:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15439.719330 secs 2024-07-27T11:07:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:06:59 smithi077 ceph-mon[37153]: pgmap v8632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:07:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:06:59 smithi047 ceph-mon[36835]: pgmap v8632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:07:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:06:59 smithi029 ceph-mon[32610]: pgmap v8632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:07:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:01 smithi077 ceph-mon[37153]: pgmap v8633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:07:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:01 smithi047 ceph-mon[36835]: pgmap v8633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:07:02.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:01 smithi029 ceph-mon[32610]: pgmap v8633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:07:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15444.719461 secs 2024-07-27T11:07:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15444.719461 secs 2024-07-27T11:07:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15444.719461 secs 2024-07-27T11:07:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:03 smithi077 ceph-mon[37153]: pgmap v8634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:07:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:03 smithi047 ceph-mon[36835]: pgmap v8634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:07:04.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:03 smithi029 ceph-mon[32610]: pgmap v8634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:07:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:05.245+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:07:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:05.245+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:07:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:05.245+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:07:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:05 smithi077 ceph-mon[37153]: pgmap v8635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:07:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:05 smithi047 ceph-mon[36835]: pgmap v8635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:07:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:05 smithi029 ceph-mon[32610]: pgmap v8635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:07:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:07 smithi077 ceph-mon[37153]: pgmap v8636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15449.719580 secs 2024-07-27T11:07:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:07 smithi047 ceph-mon[36835]: pgmap v8636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15449.719580 secs 2024-07-27T11:07:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:07 smithi029 ceph-mon[32610]: pgmap v8636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15449.719580 secs 2024-07-27T11:07:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:09 smithi077 ceph-mon[37153]: pgmap v8637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:07:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:09 smithi047 ceph-mon[36835]: pgmap v8637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:07:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:09 smithi029 ceph-mon[32610]: pgmap v8637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:07:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:11 smithi077 ceph-mon[37153]: pgmap v8638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:07:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:11 smithi047 ceph-mon[36835]: pgmap v8638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:07:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:11 smithi029 ceph-mon[32610]: pgmap v8638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:07:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15454.719717 secs 2024-07-27T11:07:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15454.719717 secs 2024-07-27T11:07:13.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15454.719717 secs 2024-07-27T11:07:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:13 smithi077 ceph-mon[37153]: pgmap v8639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:07:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:13 smithi047 ceph-mon[36835]: pgmap v8639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:07:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:13 smithi029 ceph-mon[32610]: pgmap v8639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:07:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:15 smithi077 ceph-mon[37153]: pgmap v8640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:07:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:15 smithi047 ceph-mon[36835]: pgmap v8640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:07:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:15 smithi029 ceph-mon[32610]: pgmap v8640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:07:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:17 smithi077 ceph-mon[37153]: pgmap v8641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:18.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15459.719855 secs 2024-07-27T11:07:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:17 smithi047 ceph-mon[36835]: pgmap v8641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15459.719855 secs 2024-07-27T11:07:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:17 smithi029 ceph-mon[32610]: pgmap v8641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15459.719855 secs 2024-07-27T11:07:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:19 smithi077 ceph-mon[37153]: pgmap v8642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:07:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:19 smithi047 ceph-mon[36835]: pgmap v8642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:07:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:19 smithi029 ceph-mon[32610]: pgmap v8642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:07:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:20.261+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:07:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:20.261+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:07:20.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:20.261+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:07:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:21 smithi077 ceph-mon[37153]: pgmap v8643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:07:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:21 smithi047 ceph-mon[36835]: pgmap v8643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:07:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:21 smithi029 ceph-mon[32610]: pgmap v8643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:07:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15464.719973 secs 2024-07-27T11:07:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15464.719973 secs 2024-07-27T11:07:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15464.719973 secs 2024-07-27T11:07:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:23 smithi077 ceph-mon[37153]: pgmap v8644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:07:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:23 smithi047 ceph-mon[36835]: pgmap v8644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:07:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:23 smithi029 ceph-mon[32610]: pgmap v8644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:07:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:25 smithi077 ceph-mon[37153]: pgmap v8645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:07:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:25 smithi047 ceph-mon[36835]: pgmap v8645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:07:26.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:25 smithi029 ceph-mon[32610]: pgmap v8645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:07:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:27 smithi077 ceph-mon[37153]: pgmap v8646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15469.720113 secs 2024-07-27T11:07:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:27 smithi047 ceph-mon[36835]: pgmap v8646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15469.720113 secs 2024-07-27T11:07:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:27 smithi029 ceph-mon[32610]: pgmap v8646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15469.720113 secs 2024-07-27T11:07:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:29 smithi077 ceph-mon[37153]: pgmap v8647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:07:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:29 smithi047 ceph-mon[36835]: pgmap v8647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:07:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:29 smithi029 ceph-mon[32610]: pgmap v8647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:07:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:31 smithi077 ceph-mon[37153]: pgmap v8648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:07:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:31 smithi047 ceph-mon[36835]: pgmap v8648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:07:32.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:31 smithi029 ceph-mon[32610]: pgmap v8648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:07:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15474.720256 secs 2024-07-27T11:07:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15474.720256 secs 2024-07-27T11:07:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15474.720256 secs 2024-07-27T11:07:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:33 smithi047 ceph-mon[36835]: pgmap v8649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:07:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:33 smithi029 ceph-mon[32610]: pgmap v8649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:07:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:33 smithi077 ceph-mon[37153]: pgmap v8649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:07:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:35.266+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:07:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:35.266+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:07:35.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:35.266+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:07:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:35 smithi047 ceph-mon[36835]: pgmap v8650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:07:36.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:35 smithi029 ceph-mon[32610]: pgmap v8650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:07:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:35 smithi077 ceph-mon[37153]: pgmap v8650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:07:38.161 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:37 smithi077 ceph-mon[37153]: pgmap v8651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:38.162 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15479.720374 secs 2024-07-27T11:07:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:37 smithi047 ceph-mon[36835]: pgmap v8651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15479.720374 secs 2024-07-27T11:07:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:37 smithi029 ceph-mon[32610]: pgmap v8651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15479.720374 secs 2024-07-27T11:07:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:39 smithi047 ceph-mon[36835]: pgmap v8652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:07:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:39 smithi029 ceph-mon[32610]: pgmap v8652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:07:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:39 smithi077 ceph-mon[37153]: pgmap v8652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:07:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:41 smithi047 ceph-mon[36835]: pgmap v8653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:07:42.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:41 smithi029 ceph-mon[32610]: pgmap v8653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:07:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:41 smithi077 ceph-mon[37153]: pgmap v8653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:07:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15484.720521 secs 2024-07-27T11:07:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15484.720521 secs 2024-07-27T11:07:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15484.720521 secs 2024-07-27T11:07:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:43 smithi047 ceph-mon[36835]: pgmap v8654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:07:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:43 smithi029 ceph-mon[32610]: pgmap v8654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:07:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:43 smithi077 ceph-mon[37153]: pgmap v8654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:07:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:45 smithi047 ceph-mon[36835]: pgmap v8655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:07:46.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:45 smithi029 ceph-mon[32610]: pgmap v8655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:07:46.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:45 smithi077 ceph-mon[37153]: pgmap v8655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:07:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:47 smithi047 ceph-mon[36835]: pgmap v8656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15489.720638 secs 2024-07-27T11:07:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:47 smithi029 ceph-mon[32610]: pgmap v8656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15489.720638 secs 2024-07-27T11:07:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:47 smithi077 ceph-mon[37153]: pgmap v8656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:07:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15489.720638 secs 2024-07-27T11:07:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:49 smithi047 ceph-mon[36835]: pgmap v8657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:07:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:49 smithi029 ceph-mon[32610]: pgmap v8657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:07:50.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:50.281+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:07:50.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:50.281+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:07:50.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:50.281+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:07:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:49 smithi077 ceph-mon[37153]: pgmap v8657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:07:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:07:51.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:07:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:07:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:51 smithi047 ceph-mon[36835]: pgmap v8658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:07:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:07:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:07:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:07:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:51 smithi029 ceph-mon[32610]: pgmap v8658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:07:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:07:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:07:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:07:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:51 smithi077 ceph-mon[37153]: pgmap v8658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:07:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:07:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:07:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:07:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:52 smithi047 ceph-mon[36835]: pgmap v8659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:07:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:52 smithi047 ceph-mon[36835]: pgmap v8660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:07:53.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15494.720775 secs 2024-07-27T11:07:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:52 smithi029 ceph-mon[32610]: pgmap v8659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:07:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:52 smithi029 ceph-mon[32610]: pgmap v8660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:07:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15494.720775 secs 2024-07-27T11:07:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:52 smithi077 ceph-mon[37153]: pgmap v8659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:07:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:52 smithi077 ceph-mon[37153]: pgmap v8660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:07:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15494.720775 secs 2024-07-27T11:07:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:54 smithi047 ceph-mon[36835]: pgmap v8661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:07:55.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:54 smithi029 ceph-mon[32610]: pgmap v8661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:07:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:54 smithi077 ceph-mon[37153]: pgmap v8661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:07:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:56 smithi047 ceph-mon[36835]: pgmap v8662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:07:57.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:56 smithi029 ceph-mon[32610]: pgmap v8662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:07:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:56 smithi077 ceph-mon[37153]: pgmap v8662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:07:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15499.720904 secs 2024-07-27T11:07:58.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15499.720904 secs 2024-07-27T11:07:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15499.720904 secs 2024-07-27T11:07:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:07:58 smithi047 ceph-mon[36835]: pgmap v8663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:07:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:07:58 smithi029 ceph-mon[32610]: pgmap v8663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:07:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:07:58 smithi077 ceph-mon[37153]: pgmap v8663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:07:59.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:59.991+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:07:59.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:59.991+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:07:59.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:59.991+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:07:59.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:07:59.991+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:08:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:00 smithi047 ceph-mon[36835]: pgmap v8664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:08:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:00 smithi029 ceph-mon[32610]: pgmap v8664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:08:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:00 smithi077 ceph-mon[37153]: pgmap v8664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:08:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:02 smithi047 ceph-mon[36835]: pgmap v8665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:08:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15504.721028 secs 2024-07-27T11:08:03.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:02 smithi029 ceph-mon[32610]: pgmap v8665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:08:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15504.721028 secs 2024-07-27T11:08:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:02 smithi077 ceph-mon[37153]: pgmap v8665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:08:03.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15504.721028 secs 2024-07-27T11:08:05.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:04 smithi047 ceph-mon[36835]: pgmap v8666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:08:05.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:04 smithi029 ceph-mon[32610]: pgmap v8666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:08:05.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:08:05.296+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:08:05.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:08:05.296+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:08:05.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:08:05.296+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:08:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:04 smithi077 ceph-mon[37153]: pgmap v8666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:08:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:06 smithi047 ceph-mon[36835]: pgmap v8667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:07.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:06 smithi029 ceph-mon[32610]: pgmap v8667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:06 smithi077 ceph-mon[37153]: pgmap v8667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15509.721148 secs 2024-07-27T11:08:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15509.721148 secs 2024-07-27T11:08:08.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15509.721148 secs 2024-07-27T11:08:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:08 smithi047 ceph-mon[36835]: pgmap v8668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:08 smithi029 ceph-mon[32610]: pgmap v8668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:08 smithi077 ceph-mon[37153]: pgmap v8668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:10 smithi047 ceph-mon[36835]: pgmap v8669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:08:11.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:10 smithi029 ceph-mon[32610]: pgmap v8669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:08:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:10 smithi077 ceph-mon[37153]: pgmap v8669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:08:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:12 smithi047 ceph-mon[36835]: pgmap v8670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15514.721284 secs 2024-07-27T11:08:13.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:12 smithi029 ceph-mon[32610]: pgmap v8670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15514.721284 secs 2024-07-27T11:08:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:12 smithi077 ceph-mon[37153]: pgmap v8670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15514.721284 secs 2024-07-27T11:08:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:14 smithi047 ceph-mon[36835]: pgmap v8671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:08:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:14 smithi029 ceph-mon[32610]: pgmap v8671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:08:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:14 smithi077 ceph-mon[37153]: pgmap v8671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:08:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:16 smithi029 ceph-mon[32610]: pgmap v8672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:16 smithi077 ceph-mon[37153]: pgmap v8672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:16 smithi047 ceph-mon[36835]: pgmap v8672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:18.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15519.721401 secs 2024-07-27T11:08:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15519.721401 secs 2024-07-27T11:08:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15519.721401 secs 2024-07-27T11:08:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:18 smithi029 ceph-mon[32610]: pgmap v8673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:18 smithi077 ceph-mon[37153]: pgmap v8673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:18 smithi047 ceph-mon[36835]: pgmap v8673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:20.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:08:20.304+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:08:20.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:08:20.304+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:08:20.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:08:20.304+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:08:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:20 smithi029 ceph-mon[32610]: pgmap v8674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:08:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:20 smithi077 ceph-mon[37153]: pgmap v8674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:08:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:20 smithi047 ceph-mon[36835]: pgmap v8674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:08:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:22 smithi077 ceph-mon[37153]: pgmap v8675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15524.721529 secs 2024-07-27T11:08:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:22 smithi047 ceph-mon[36835]: pgmap v8675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15524.721529 secs 2024-07-27T11:08:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:22 smithi029 ceph-mon[32610]: pgmap v8675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15524.721529 secs 2024-07-27T11:08:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:24 smithi077 ceph-mon[37153]: pgmap v8676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:08:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:24 smithi047 ceph-mon[36835]: pgmap v8676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:08:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:24 smithi029 ceph-mon[32610]: pgmap v8676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:08:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:26 smithi077 ceph-mon[37153]: pgmap v8677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:26 smithi047 ceph-mon[36835]: pgmap v8677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:27.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:26 smithi029 ceph-mon[32610]: pgmap v8677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15529.721655 secs 2024-07-27T11:08:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15529.721655 secs 2024-07-27T11:08:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15529.721655 secs 2024-07-27T11:08:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:28 smithi077 ceph-mon[37153]: pgmap v8678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:28 smithi047 ceph-mon[36835]: pgmap v8678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:28 smithi029 ceph-mon[32610]: pgmap v8678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:30 smithi077 ceph-mon[37153]: pgmap v8679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:08:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:30 smithi047 ceph-mon[36835]: pgmap v8679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:08:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:30 smithi029 ceph-mon[32610]: pgmap v8679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:08:33.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:32 smithi047 ceph-mon[36835]: pgmap v8680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:08:33.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15534.721781 secs 2024-07-27T11:08:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:32 smithi077 ceph-mon[37153]: pgmap v8680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:08:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15534.721781 secs 2024-07-27T11:08:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:32 smithi029 ceph-mon[32610]: pgmap v8680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:08:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15534.721781 secs 2024-07-27T11:08:35.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:08:35.320+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:08:35.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:08:35.320+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:08:35.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:08:35.320+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:08:35.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:34 smithi029 ceph-mon[32610]: pgmap v8681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:08:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:34 smithi077 ceph-mon[37153]: pgmap v8681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:08:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:34 smithi047 ceph-mon[36835]: pgmap v8681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:08:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:36 smithi077 ceph-mon[37153]: pgmap v8682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:08:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:36 smithi047 ceph-mon[36835]: pgmap v8682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:08:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:36 smithi029 ceph-mon[32610]: pgmap v8682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:08:38.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15539.721881 secs 2024-07-27T11:08:38.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15539.721881 secs 2024-07-27T11:08:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15539.721881 secs 2024-07-27T11:08:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:39 smithi077 ceph-mon[37153]: pgmap v8683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:39 smithi047 ceph-mon[36835]: pgmap v8683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:39 smithi029 ceph-mon[32610]: pgmap v8683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:40 smithi077 ceph-mon[37153]: pgmap v8684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:08:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:40 smithi047 ceph-mon[36835]: pgmap v8684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:08:40.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:40 smithi029 ceph-mon[32610]: pgmap v8684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:08:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:42 smithi029 ceph-mon[32610]: pgmap v8685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15544.722011 secs 2024-07-27T11:08:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:42 smithi077 ceph-mon[37153]: pgmap v8685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15544.722011 secs 2024-07-27T11:08:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:42 smithi047 ceph-mon[36835]: pgmap v8685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15544.722011 secs 2024-07-27T11:08:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:44 smithi029 ceph-mon[32610]: pgmap v8686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:08:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:44 smithi077 ceph-mon[37153]: pgmap v8686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:08:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:44 smithi047 ceph-mon[36835]: pgmap v8686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:08:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:46 smithi029 ceph-mon[32610]: pgmap v8687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:46 smithi077 ceph-mon[37153]: pgmap v8687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:46 smithi047 ceph-mon[36835]: pgmap v8687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15549.722130 secs 2024-07-27T11:08:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15549.722130 secs 2024-07-27T11:08:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15549.722130 secs 2024-07-27T11:08:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:48 smithi077 ceph-mon[37153]: pgmap v8688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:48 smithi047 ceph-mon[36835]: pgmap v8688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:48 smithi029 ceph-mon[32610]: pgmap v8688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:08:50.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:08:50.325+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:08:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:08:50.325+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:08:50.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:08:50.325+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:08:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:50 smithi077 ceph-mon[37153]: pgmap v8689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:08:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:50 smithi047 ceph-mon[36835]: pgmap v8689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:08:50.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:50 smithi029 ceph-mon[32610]: pgmap v8689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:08:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:52 smithi077 ceph-mon[37153]: pgmap v8690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:08:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:08:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:08:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:52 smithi077 ceph-mon[37153]: pgmap v8691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T11:08:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:52 smithi077 ceph-mon[37153]: pgmap v8692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:08:52.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:08:52.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15554.722226 secs 2024-07-27T11:08:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:52 smithi047 ceph-mon[36835]: pgmap v8690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:08:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:08:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:08:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:52 smithi047 ceph-mon[36835]: pgmap v8691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T11:08:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:52 smithi047 ceph-mon[36835]: pgmap v8692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:08:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:08:52.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15554.722226 secs 2024-07-27T11:08:52.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:52 smithi029 ceph-mon[32610]: pgmap v8690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:08:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:08:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:08:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:08:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:52 smithi029 ceph-mon[32610]: pgmap v8691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T11:08:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:52 smithi029 ceph-mon[32610]: pgmap v8692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:08:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:08:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15554.722226 secs 2024-07-27T11:08:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:54 smithi077 ceph-mon[37153]: pgmap v8693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:08:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:54 smithi047 ceph-mon[36835]: pgmap v8693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:08:55.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:54 smithi029 ceph-mon[32610]: pgmap v8693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:08:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:56 smithi077 ceph-mon[37153]: pgmap v8694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:08:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:56 smithi047 ceph-mon[36835]: pgmap v8694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:08:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:56 smithi029 ceph-mon[32610]: pgmap v8694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:08:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15559.722346 secs 2024-07-27T11:08:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15559.722346 secs 2024-07-27T11:08:58.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15559.722346 secs 2024-07-27T11:08:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:08:58 smithi077 ceph-mon[37153]: pgmap v8695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:08:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:08:58 smithi047 ceph-mon[36835]: pgmap v8695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:08:59.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:08:58 smithi029 ceph-mon[32610]: pgmap v8695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:09:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:00 smithi077 ceph-mon[37153]: pgmap v8696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:09:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:00 smithi047 ceph-mon[36835]: pgmap v8696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:09:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:00 smithi029 ceph-mon[32610]: pgmap v8696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:09:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:02 smithi077 ceph-mon[37153]: pgmap v8697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:09:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15564.722469 secs 2024-07-27T11:09:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:02 smithi047 ceph-mon[36835]: pgmap v8697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:09:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15564.722469 secs 2024-07-27T11:09:03.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:02 smithi029 ceph-mon[32610]: pgmap v8697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:09:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15564.722469 secs 2024-07-27T11:09:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:04 smithi077 ceph-mon[37153]: pgmap v8698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:04 smithi047 ceph-mon[36835]: pgmap v8698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:04 smithi029 ceph-mon[32610]: pgmap v8698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:05.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:09:05.327+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:09:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:09:05.327+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:09:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:09:05.327+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:09:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:06 smithi077 ceph-mon[37153]: pgmap v8699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:06 smithi047 ceph-mon[36835]: pgmap v8699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:06 smithi029 ceph-mon[32610]: pgmap v8699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:08.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15569.722584 secs 2024-07-27T11:09:08.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15569.722584 secs 2024-07-27T11:09:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15569.722584 secs 2024-07-27T11:09:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:08 smithi077 ceph-mon[37153]: pgmap v8700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:09:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:08 smithi047 ceph-mon[36835]: pgmap v8700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:09:09.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:08 smithi029 ceph-mon[32610]: pgmap v8700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:09:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:10 smithi077 ceph-mon[37153]: pgmap v8701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:10 smithi047 ceph-mon[36835]: pgmap v8701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:11.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:10 smithi029 ceph-mon[32610]: pgmap v8701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:12 smithi077 ceph-mon[37153]: pgmap v8702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15574.722649 secs 2024-07-27T11:09:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:12 smithi047 ceph-mon[36835]: pgmap v8702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15574.722649 secs 2024-07-27T11:09:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:12 smithi029 ceph-mon[32610]: pgmap v8702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15574.722649 secs 2024-07-27T11:09:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:14 smithi077 ceph-mon[37153]: pgmap v8703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:14 smithi047 ceph-mon[36835]: pgmap v8703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:15.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:14 smithi029 ceph-mon[32610]: pgmap v8703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:16 smithi077 ceph-mon[37153]: pgmap v8704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:16 smithi047 ceph-mon[36835]: pgmap v8704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:17.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:16 smithi029 ceph-mon[32610]: pgmap v8704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15579.722776 secs 2024-07-27T11:09:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15579.722776 secs 2024-07-27T11:09:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15579.722776 secs 2024-07-27T11:09:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:18 smithi077 ceph-mon[37153]: pgmap v8705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:09:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:18 smithi047 ceph-mon[36835]: pgmap v8705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:09:19.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:18 smithi029 ceph-mon[32610]: pgmap v8705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:09:20.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:09:20.343+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:09:20.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:09:20.343+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:09:20.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:09:20.343+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:09:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:20 smithi077 ceph-mon[37153]: pgmap v8706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:20 smithi047 ceph-mon[36835]: pgmap v8706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:21.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:20 smithi029 ceph-mon[32610]: pgmap v8706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:22 smithi077 ceph-mon[37153]: pgmap v8707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15584.722908 secs 2024-07-27T11:09:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:22 smithi047 ceph-mon[36835]: pgmap v8707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15584.722908 secs 2024-07-27T11:09:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:22 smithi029 ceph-mon[32610]: pgmap v8707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15584.722908 secs 2024-07-27T11:09:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:24 smithi077 ceph-mon[37153]: pgmap v8708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:24 smithi047 ceph-mon[36835]: pgmap v8708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:25.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:24 smithi029 ceph-mon[32610]: pgmap v8708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:26 smithi077 ceph-mon[37153]: pgmap v8709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:26 smithi047 ceph-mon[36835]: pgmap v8709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:27.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:26 smithi029 ceph-mon[32610]: pgmap v8709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15589.722989 secs 2024-07-27T11:09:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15589.722989 secs 2024-07-27T11:09:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15589.722989 secs 2024-07-27T11:09:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:28 smithi077 ceph-mon[37153]: pgmap v8710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:09:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:28 smithi047 ceph-mon[36835]: pgmap v8710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:09:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:28 smithi029 ceph-mon[32610]: pgmap v8710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:09:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:30 smithi077 ceph-mon[37153]: pgmap v8711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:30 smithi047 ceph-mon[36835]: pgmap v8711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:31.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:30 smithi029 ceph-mon[32610]: pgmap v8711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:32 smithi077 ceph-mon[37153]: pgmap v8712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15594.723110 secs 2024-07-27T11:09:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:32 smithi047 ceph-mon[36835]: pgmap v8712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15594.723110 secs 2024-07-27T11:09:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:32 smithi029 ceph-mon[32610]: pgmap v8712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15594.723110 secs 2024-07-27T11:09:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:34 smithi077 ceph-mon[37153]: pgmap v8713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:34 smithi047 ceph-mon[36835]: pgmap v8713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:34 smithi029 ceph-mon[32610]: pgmap v8713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:09:35.358+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:09:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:09:35.358+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:09:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:09:35.358+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:09:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:36 smithi077 ceph-mon[37153]: pgmap v8714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:36 smithi047 ceph-mon[36835]: pgmap v8714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:36 smithi029 ceph-mon[32610]: pgmap v8714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15599.723175 secs 2024-07-27T11:09:38.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15599.723175 secs 2024-07-27T11:09:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15599.723175 secs 2024-07-27T11:09:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:38 smithi077 ceph-mon[37153]: pgmap v8715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:09:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:38 smithi047 ceph-mon[36835]: pgmap v8715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:09:39.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:38 smithi029 ceph-mon[32610]: pgmap v8715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:09:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:40 smithi077 ceph-mon[37153]: pgmap v8716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:40 smithi047 ceph-mon[36835]: pgmap v8716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:40 smithi029 ceph-mon[32610]: pgmap v8716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:42 smithi077 ceph-mon[37153]: pgmap v8717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15604.723301 secs 2024-07-27T11:09:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:42 smithi047 ceph-mon[36835]: pgmap v8717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15604.723301 secs 2024-07-27T11:09:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:42 smithi029 ceph-mon[32610]: pgmap v8717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15604.723301 secs 2024-07-27T11:09:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:44 smithi077 ceph-mon[37153]: pgmap v8718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:44 smithi047 ceph-mon[36835]: pgmap v8718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:44 smithi029 ceph-mon[32610]: pgmap v8718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:46 smithi077 ceph-mon[37153]: pgmap v8719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:46 smithi047 ceph-mon[36835]: pgmap v8719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:47.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:46 smithi029 ceph-mon[32610]: pgmap v8719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:09:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15609.723423 secs 2024-07-27T11:09:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15609.723423 secs 2024-07-27T11:09:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15609.723423 secs 2024-07-27T11:09:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:48 smithi077 ceph-mon[37153]: pgmap v8720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:09:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:48 smithi047 ceph-mon[36835]: pgmap v8720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:09:49.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:48 smithi029 ceph-mon[32610]: pgmap v8720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:09:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:09:50.373+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:09:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:09:50.373+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:09:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:09:50.373+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:09:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:50 smithi077 ceph-mon[37153]: pgmap v8721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:50 smithi047 ceph-mon[36835]: pgmap v8721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:50 smithi029 ceph-mon[32610]: pgmap v8721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:09:52.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:09:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:09:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:09:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:52 smithi047 ceph-mon[36835]: pgmap v8722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15614.723516 secs 2024-07-27T11:09:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:52 smithi029 ceph-mon[32610]: pgmap v8722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15614.723516 secs 2024-07-27T11:09:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:52 smithi077 ceph-mon[37153]: pgmap v8722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:09:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15614.723516 secs 2024-07-27T11:09:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:54 smithi047 ceph-mon[36835]: pgmap v8723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:54 smithi029 ceph-mon[32610]: pgmap v8723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:54.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:54 smithi077 ceph-mon[37153]: pgmap v8723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:09:54.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:54.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:09:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:09:55.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:55 smithi047 ceph-mon[36835]: pgmap v8724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:09:55.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:55 smithi047 ceph-mon[36835]: pgmap v8725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:09:55.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:09:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:09:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:55 smithi029 ceph-mon[32610]: pgmap v8724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:09:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:55 smithi029 ceph-mon[32610]: pgmap v8725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:09:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:09:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:09:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:55 smithi077 ceph-mon[37153]: pgmap v8724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:09:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:55 smithi077 ceph-mon[37153]: pgmap v8725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:09:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:09:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:57 smithi077 ceph-mon[37153]: pgmap v8726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:09:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15619.723658 secs 2024-07-27T11:09:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:57 smithi047 ceph-mon[36835]: pgmap v8726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:09:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15619.723658 secs 2024-07-27T11:09:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:57 smithi029 ceph-mon[32610]: pgmap v8726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:09:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15619.723658 secs 2024-07-27T11:09:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:09:59 smithi077 ceph-mon[37153]: pgmap v8727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:09:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:09:59 smithi047 ceph-mon[36835]: pgmap v8727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:09:59.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:09:59 smithi029 ceph-mon[32610]: pgmap v8727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:10:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:10:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:10:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:10:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:01 smithi077 ceph-mon[37153]: pgmap v8728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:10:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:01 smithi047 ceph-mon[36835]: pgmap v8728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:10:01.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:01 smithi029 ceph-mon[32610]: pgmap v8728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:10:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15624.723773 secs 2024-07-27T11:10:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15624.723773 secs 2024-07-27T11:10:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15624.723773 secs 2024-07-27T11:10:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:03 smithi077 ceph-mon[37153]: pgmap v8729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:10:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:03 smithi047 ceph-mon[36835]: pgmap v8729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:10:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:03 smithi029 ceph-mon[32610]: pgmap v8729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:10:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:10:05.388+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:10:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:10:05.388+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:10:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:10:05.388+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:10:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:05 smithi029 ceph-mon[32610]: pgmap v8730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:10:05.765 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:05 smithi047 ceph-mon[36835]: pgmap v8730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:10:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:05 smithi077 ceph-mon[37153]: pgmap v8730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:10:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:07 smithi077 ceph-mon[37153]: pgmap v8731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15629.723893 secs 2024-07-27T11:10:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:07 smithi047 ceph-mon[36835]: pgmap v8731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15629.723893 secs 2024-07-27T11:10:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:07 smithi029 ceph-mon[32610]: pgmap v8731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15629.723893 secs 2024-07-27T11:10:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:09 smithi077 ceph-mon[37153]: pgmap v8732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:09 smithi047 ceph-mon[36835]: pgmap v8732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:09 smithi029 ceph-mon[32610]: pgmap v8732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:11.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:11 smithi077 ceph-mon[37153]: pgmap v8733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:11 smithi047 ceph-mon[36835]: pgmap v8733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:11 smithi029 ceph-mon[32610]: pgmap v8733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15634.724014 secs 2024-07-27T11:10:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15634.724014 secs 2024-07-27T11:10:12.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15634.724014 secs 2024-07-27T11:10:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:13 smithi077 ceph-mon[37153]: pgmap v8734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:13 smithi047 ceph-mon[36835]: pgmap v8734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:13.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:13 smithi029 ceph-mon[32610]: pgmap v8734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:15 smithi077 ceph-mon[37153]: pgmap v8735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:15 smithi047 ceph-mon[36835]: pgmap v8735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:15 smithi029 ceph-mon[32610]: pgmap v8735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:17 smithi077 ceph-mon[37153]: pgmap v8736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15639.724133 secs 2024-07-27T11:10:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:17 smithi047 ceph-mon[36835]: pgmap v8736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15639.724133 secs 2024-07-27T11:10:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:17 smithi029 ceph-mon[32610]: pgmap v8736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15639.724133 secs 2024-07-27T11:10:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:19 smithi077 ceph-mon[37153]: pgmap v8737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:19 smithi047 ceph-mon[36835]: pgmap v8737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:19 smithi029 ceph-mon[32610]: pgmap v8737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:20.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:10:20.392+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:10:20.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:10:20.392+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:10:20.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:10:20.392+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:10:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:21 smithi077 ceph-mon[37153]: pgmap v8738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:21 smithi047 ceph-mon[36835]: pgmap v8738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:21.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:21 smithi029 ceph-mon[32610]: pgmap v8738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15644.724258 secs 2024-07-27T11:10:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15644.724258 secs 2024-07-27T11:10:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15644.724258 secs 2024-07-27T11:10:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:23 smithi077 ceph-mon[37153]: pgmap v8739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:23 smithi047 ceph-mon[36835]: pgmap v8739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:23 smithi029 ceph-mon[32610]: pgmap v8739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:25 smithi077 ceph-mon[37153]: pgmap v8740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:25 smithi047 ceph-mon[36835]: pgmap v8740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:25 smithi029 ceph-mon[32610]: pgmap v8740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:27 smithi077 ceph-mon[37153]: pgmap v8741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:27.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15649.724371 secs 2024-07-27T11:10:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:27 smithi047 ceph-mon[36835]: pgmap v8741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15649.724371 secs 2024-07-27T11:10:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:27 smithi029 ceph-mon[32610]: pgmap v8741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15649.724371 secs 2024-07-27T11:10:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:29 smithi077 ceph-mon[37153]: pgmap v8742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:29 smithi047 ceph-mon[36835]: pgmap v8742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:29 smithi029 ceph-mon[32610]: pgmap v8742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:31 smithi077 ceph-mon[37153]: pgmap v8743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:31 smithi047 ceph-mon[36835]: pgmap v8743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:31 smithi029 ceph-mon[32610]: pgmap v8743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15654.724525 secs 2024-07-27T11:10:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15654.724525 secs 2024-07-27T11:10:32.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15654.724525 secs 2024-07-27T11:10:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:33 smithi077 ceph-mon[37153]: pgmap v8744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:33 smithi047 ceph-mon[36835]: pgmap v8744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:33 smithi029 ceph-mon[32610]: pgmap v8744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:35.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:10:35.408+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:10:35.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:10:35.408+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:10:35.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:10:35.408+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:10:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:35 smithi029 ceph-mon[32610]: pgmap v8745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:35 smithi077 ceph-mon[37153]: pgmap v8745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:35 smithi047 ceph-mon[36835]: pgmap v8745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:37 smithi077 ceph-mon[37153]: pgmap v8746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15659.724645 secs 2024-07-27T11:10:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:37 smithi047 ceph-mon[36835]: pgmap v8746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15659.724645 secs 2024-07-27T11:10:37.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:37 smithi029 ceph-mon[32610]: pgmap v8746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15659.724645 secs 2024-07-27T11:10:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:39 smithi077 ceph-mon[37153]: pgmap v8747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:39 smithi047 ceph-mon[36835]: pgmap v8747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:39 smithi029 ceph-mon[32610]: pgmap v8747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:41 smithi077 ceph-mon[37153]: pgmap v8748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:41 smithi047 ceph-mon[36835]: pgmap v8748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:41 smithi029 ceph-mon[32610]: pgmap v8748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15664.724769 secs 2024-07-27T11:10:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15664.724769 secs 2024-07-27T11:10:42.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15664.724769 secs 2024-07-27T11:10:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:43 smithi077 ceph-mon[37153]: pgmap v8749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:43 smithi047 ceph-mon[36835]: pgmap v8749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:43 smithi029 ceph-mon[32610]: pgmap v8749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:45 smithi077 ceph-mon[37153]: pgmap v8750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:45 smithi047 ceph-mon[36835]: pgmap v8750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:45 smithi029 ceph-mon[32610]: pgmap v8750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:47 smithi077 ceph-mon[37153]: pgmap v8751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15669.724912 secs 2024-07-27T11:10:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:47 smithi047 ceph-mon[36835]: pgmap v8751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15669.724912 secs 2024-07-27T11:10:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:47 smithi029 ceph-mon[32610]: pgmap v8751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:10:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15669.724912 secs 2024-07-27T11:10:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:49 smithi077 ceph-mon[37153]: pgmap v8752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:49 smithi047 ceph-mon[36835]: pgmap v8752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:49 smithi029 ceph-mon[32610]: pgmap v8752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:10:50.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:10:50.414+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:10:50.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:10:50.414+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:10:50.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:10:50.414+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:10:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:51 smithi077 ceph-mon[37153]: pgmap v8753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:51 smithi047 ceph-mon[36835]: pgmap v8753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:51.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:51 smithi029 ceph-mon[32610]: pgmap v8753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:10:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15674.725039 secs 2024-07-27T11:10:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15674.725039 secs 2024-07-27T11:10:52.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15674.725039 secs 2024-07-27T11:10:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:53 smithi077 ceph-mon[37153]: pgmap v8754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:53 smithi047 ceph-mon[36835]: pgmap v8754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:53.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:53 smithi029 ceph-mon[32610]: pgmap v8754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:10:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:10:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:10:54.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:10:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: pgmap v8755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:10:56.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:10:56.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: pgmap v8755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:10:56.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:10:56.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: pgmap v8755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:10:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:56.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:10:56.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:10:56.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:10:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:56 smithi047 ceph-mon[36835]: pgmap v8756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T11:10:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:56 smithi047 ceph-mon[36835]: pgmap v8757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:10:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:56 smithi029 ceph-mon[32610]: pgmap v8756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T11:10:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:56 smithi029 ceph-mon[32610]: pgmap v8757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:10:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:56 smithi077 ceph-mon[37153]: pgmap v8756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T11:10:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:56 smithi077 ceph-mon[37153]: pgmap v8757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:10:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15679.725140 secs 2024-07-27T11:10:58.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15679.725140 secs 2024-07-27T11:10:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15679.725140 secs 2024-07-27T11:10:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:10:58 smithi047 ceph-mon[36835]: pgmap v8758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:10:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:10:58 smithi029 ceph-mon[32610]: pgmap v8758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:10:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:10:58 smithi077 ceph-mon[37153]: pgmap v8758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:11:00.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:00.018+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:11:00.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:00.018+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:11:00.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:00.019+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:11:00.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:00.019+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:11:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:00 smithi047 ceph-mon[36835]: pgmap v8759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:11:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:00 smithi029 ceph-mon[32610]: pgmap v8759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:11:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:00 smithi077 ceph-mon[37153]: pgmap v8759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:11:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:02 smithi047 ceph-mon[36835]: pgmap v8760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T11:11:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15684.725266 secs 2024-07-27T11:11:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:02 smithi029 ceph-mon[32610]: pgmap v8760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T11:11:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15684.725266 secs 2024-07-27T11:11:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:02 smithi077 ceph-mon[37153]: pgmap v8760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T11:11:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15684.725266 secs 2024-07-27T11:11:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:04 smithi047 ceph-mon[36835]: pgmap v8761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:11:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:04 smithi029 ceph-mon[32610]: pgmap v8761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:11:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:04 smithi077 ceph-mon[37153]: pgmap v8761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:11:05.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:05.428+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:11:05.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:05.428+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:11:05.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:05.428+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:11:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:06 smithi047 ceph-mon[36835]: pgmap v8762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:11:07.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:06 smithi029 ceph-mon[32610]: pgmap v8762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:11:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:06 smithi077 ceph-mon[37153]: pgmap v8762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:11:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15689.725409 secs 2024-07-27T11:11:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15689.725409 secs 2024-07-27T11:11:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15689.725409 secs 2024-07-27T11:11:09.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:08 smithi029 ceph-mon[32610]: pgmap v8763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:08 smithi047 ceph-mon[36835]: pgmap v8763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:09.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:08 smithi077 ceph-mon[37153]: pgmap v8763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:10 smithi047 ceph-mon[36835]: pgmap v8764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:10 smithi029 ceph-mon[32610]: pgmap v8764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:10 smithi077 ceph-mon[37153]: pgmap v8764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:12 smithi047 ceph-mon[36835]: pgmap v8765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15694.725531 secs 2024-07-27T11:11:13.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:12 smithi029 ceph-mon[32610]: pgmap v8765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15694.725531 secs 2024-07-27T11:11:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:12 smithi077 ceph-mon[37153]: pgmap v8765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15694.725531 secs 2024-07-27T11:11:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:14 smithi047 ceph-mon[36835]: pgmap v8766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:14 smithi029 ceph-mon[32610]: pgmap v8766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:14 smithi077 ceph-mon[37153]: pgmap v8766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:16 smithi047 ceph-mon[36835]: pgmap v8767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:16 smithi029 ceph-mon[32610]: pgmap v8767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:16 smithi077 ceph-mon[37153]: pgmap v8767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15699.725672 secs 2024-07-27T11:11:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15699.725672 secs 2024-07-27T11:11:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15699.725672 secs 2024-07-27T11:11:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:18 smithi047 ceph-mon[36835]: pgmap v8768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:19.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:18 smithi029 ceph-mon[32610]: pgmap v8768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:18 smithi077 ceph-mon[37153]: pgmap v8768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:20.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:20.429+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:11:20.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:20.429+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:11:20.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:20.429+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:11:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:20 smithi047 ceph-mon[36835]: pgmap v8769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:20 smithi029 ceph-mon[32610]: pgmap v8769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:20 smithi077 ceph-mon[37153]: pgmap v8769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:22 smithi047 ceph-mon[36835]: pgmap v8770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15704.725814 secs 2024-07-27T11:11:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:22 smithi029 ceph-mon[32610]: pgmap v8770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15704.725814 secs 2024-07-27T11:11:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:22 smithi077 ceph-mon[37153]: pgmap v8770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15704.725814 secs 2024-07-27T11:11:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:24 smithi047 ceph-mon[36835]: pgmap v8771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:25.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:24 smithi029 ceph-mon[32610]: pgmap v8771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:24 smithi077 ceph-mon[37153]: pgmap v8771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:27.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:26 smithi047 ceph-mon[36835]: pgmap v8772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:26 smithi029 ceph-mon[32610]: pgmap v8772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:26 smithi077 ceph-mon[37153]: pgmap v8772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15709.725933 secs 2024-07-27T11:11:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15709.725933 secs 2024-07-27T11:11:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15709.725933 secs 2024-07-27T11:11:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:28 smithi047 ceph-mon[36835]: pgmap v8773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:28 smithi029 ceph-mon[32610]: pgmap v8773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:28 smithi077 ceph-mon[37153]: pgmap v8773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:30 smithi047 ceph-mon[36835]: pgmap v8774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:30 smithi029 ceph-mon[32610]: pgmap v8774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:30 smithi077 ceph-mon[37153]: pgmap v8774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:32 smithi047 ceph-mon[36835]: pgmap v8775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15714.726079 secs 2024-07-27T11:11:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:32 smithi029 ceph-mon[32610]: pgmap v8775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15714.726079 secs 2024-07-27T11:11:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:32 smithi077 ceph-mon[37153]: pgmap v8775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15714.726079 secs 2024-07-27T11:11:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:34 smithi047 ceph-mon[36835]: pgmap v8776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:34 smithi029 ceph-mon[32610]: pgmap v8776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:34 smithi077 ceph-mon[37153]: pgmap v8776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:35.444+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:11:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:35.444+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:11:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:35.444+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:11:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:36 smithi047 ceph-mon[36835]: pgmap v8777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:37.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:36 smithi029 ceph-mon[32610]: pgmap v8777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:36 smithi077 ceph-mon[37153]: pgmap v8777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15719.726199 secs 2024-07-27T11:11:38.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15719.726199 secs 2024-07-27T11:11:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15719.726199 secs 2024-07-27T11:11:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:38 smithi029 ceph-mon[32610]: pgmap v8778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:39.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:38 smithi077 ceph-mon[37153]: pgmap v8778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:38 smithi047 ceph-mon[36835]: pgmap v8778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:40 smithi029 ceph-mon[32610]: pgmap v8779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:40 smithi077 ceph-mon[37153]: pgmap v8779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:40 smithi047 ceph-mon[36835]: pgmap v8779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:42 smithi029 ceph-mon[32610]: pgmap v8780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15724.726325 secs 2024-07-27T11:11:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:42 smithi077 ceph-mon[37153]: pgmap v8780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15724.726325 secs 2024-07-27T11:11:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:42 smithi047 ceph-mon[36835]: pgmap v8780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15724.726325 secs 2024-07-27T11:11:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:44 smithi077 ceph-mon[37153]: pgmap v8781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:44 smithi047 ceph-mon[36835]: pgmap v8781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:44 smithi029 ceph-mon[32610]: pgmap v8781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:46 smithi077 ceph-mon[37153]: pgmap v8782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:46 smithi047 ceph-mon[36835]: pgmap v8782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:47.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:46 smithi029 ceph-mon[32610]: pgmap v8782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15729.726455 secs 2024-07-27T11:11:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15729.726455 secs 2024-07-27T11:11:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15729.726455 secs 2024-07-27T11:11:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:48 smithi077 ceph-mon[37153]: pgmap v8783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:48 smithi047 ceph-mon[36835]: pgmap v8783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:48 smithi029 ceph-mon[32610]: pgmap v8783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:11:50.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:50.455+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:11:50.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:50.455+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:11:50.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:11:50.455+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:11:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:50 smithi077 ceph-mon[37153]: pgmap v8784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:50 smithi047 ceph-mon[36835]: pgmap v8784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:51.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:50 smithi029 ceph-mon[32610]: pgmap v8784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:11:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:52 smithi077 ceph-mon[37153]: pgmap v8785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15734.726575 secs 2024-07-27T11:11:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:52 smithi047 ceph-mon[36835]: pgmap v8785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15734.726575 secs 2024-07-27T11:11:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:52 smithi029 ceph-mon[32610]: pgmap v8785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:11:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15734.726575 secs 2024-07-27T11:11:55.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:54 smithi077 ceph-mon[37153]: pgmap v8786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:54 smithi047 ceph-mon[36835]: pgmap v8786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:54 smithi029 ceph-mon[32610]: pgmap v8786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:11:56.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:11:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:11:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:11:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:56 smithi077 ceph-mon[37153]: pgmap v8787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:11:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:11:57.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:11:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:56 smithi047 ceph-mon[36835]: pgmap v8787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:11:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:11:57.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:11:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:56 smithi029 ceph-mon[32610]: pgmap v8787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:11:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:11:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:11:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:11:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:57 smithi077 ceph-mon[37153]: pgmap v8788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T11:11:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:57 smithi077 ceph-mon[37153]: pgmap v8789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:11:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:11:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15739.726716 secs 2024-07-27T11:11:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:57 smithi047 ceph-mon[36835]: pgmap v8788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T11:11:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:57 smithi047 ceph-mon[36835]: pgmap v8789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:11:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:11:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15739.726716 secs 2024-07-27T11:11:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:57 smithi029 ceph-mon[32610]: pgmap v8788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T11:11:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:57 smithi029 ceph-mon[32610]: pgmap v8789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:11:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:11:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15739.726716 secs 2024-07-27T11:12:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:00 smithi077 ceph-mon[37153]: pgmap v8790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:12:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:00 smithi047 ceph-mon[36835]: pgmap v8790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:12:00.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:00 smithi029 ceph-mon[32610]: pgmap v8790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:12:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:02 smithi077 ceph-mon[37153]: pgmap v8791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:12:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:02 smithi047 ceph-mon[36835]: pgmap v8791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:12:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:02 smithi029 ceph-mon[32610]: pgmap v8791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:12:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15744.726814 secs 2024-07-27T11:12:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15744.726814 secs 2024-07-27T11:12:03.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15744.726814 secs 2024-07-27T11:12:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:04 smithi077 ceph-mon[37153]: pgmap v8792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:12:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:04 smithi047 ceph-mon[36835]: pgmap v8792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:12:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:04 smithi029 ceph-mon[32610]: pgmap v8792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:12:05.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:12:05.469+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:12:05.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:12:05.469+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:12:05.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:12:05.469+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:12:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:06 smithi077 ceph-mon[37153]: pgmap v8793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:12:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:06 smithi047 ceph-mon[36835]: pgmap v8793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:12:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:06 smithi029 ceph-mon[32610]: pgmap v8793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:12:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:08 smithi077 ceph-mon[37153]: pgmap v8794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:12:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15749.726855 secs 2024-07-27T11:12:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:08 smithi047 ceph-mon[36835]: pgmap v8794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:12:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15749.726855 secs 2024-07-27T11:12:08.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:08 smithi029 ceph-mon[32610]: pgmap v8794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:12:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15749.726855 secs 2024-07-27T11:12:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:10 smithi077 ceph-mon[37153]: pgmap v8795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:10 smithi047 ceph-mon[36835]: pgmap v8795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:10 smithi029 ceph-mon[32610]: pgmap v8795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:12 smithi077 ceph-mon[37153]: pgmap v8796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:12 smithi047 ceph-mon[36835]: pgmap v8796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:12.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:12 smithi029 ceph-mon[32610]: pgmap v8796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15754.726971 secs 2024-07-27T11:12:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15754.726971 secs 2024-07-27T11:12:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15754.726971 secs 2024-07-27T11:12:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:14 smithi077 ceph-mon[37153]: pgmap v8797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:14 smithi047 ceph-mon[36835]: pgmap v8797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:14.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:14 smithi029 ceph-mon[32610]: pgmap v8797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:15 smithi077 ceph-mon[37153]: pgmap v8798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:15 smithi047 ceph-mon[36835]: pgmap v8798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:15.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:15 smithi029 ceph-mon[32610]: pgmap v8798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:17 smithi047 ceph-mon[36835]: pgmap v8799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:17.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15759.727083 secs 2024-07-27T11:12:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:17 smithi029 ceph-mon[32610]: pgmap v8799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15759.727083 secs 2024-07-27T11:12:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:17 smithi077 ceph-mon[37153]: pgmap v8799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:17.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15759.727083 secs 2024-07-27T11:12:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:19 smithi047 ceph-mon[36835]: pgmap v8800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:19 smithi029 ceph-mon[32610]: pgmap v8800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:19 smithi077 ceph-mon[37153]: pgmap v8800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:12:20.485+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:12:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:12:20.485+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:12:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:12:20.485+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:12:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:21 smithi047 ceph-mon[36835]: pgmap v8801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:21 smithi029 ceph-mon[32610]: pgmap v8801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:21 smithi077 ceph-mon[37153]: pgmap v8801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15764.727177 secs 2024-07-27T11:12:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15764.727177 secs 2024-07-27T11:12:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15764.727177 secs 2024-07-27T11:12:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:23 smithi029 ceph-mon[32610]: pgmap v8802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:23 smithi077 ceph-mon[37153]: pgmap v8802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:23 smithi047 ceph-mon[36835]: pgmap v8802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:25 smithi029 ceph-mon[32610]: pgmap v8803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:25 smithi077 ceph-mon[37153]: pgmap v8803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:25 smithi047 ceph-mon[36835]: pgmap v8803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:27 smithi029 ceph-mon[32610]: pgmap v8804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15769.727270 secs 2024-07-27T11:12:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:27 smithi077 ceph-mon[37153]: pgmap v8804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15769.727270 secs 2024-07-27T11:12:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:27 smithi047 ceph-mon[36835]: pgmap v8804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15769.727270 secs 2024-07-27T11:12:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:29 smithi029 ceph-mon[32610]: pgmap v8805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:29 smithi077 ceph-mon[37153]: pgmap v8805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:29 smithi047 ceph-mon[36835]: pgmap v8805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:31 smithi077 ceph-mon[37153]: pgmap v8806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:31 smithi047 ceph-mon[36835]: pgmap v8806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:31 smithi029 ceph-mon[32610]: pgmap v8806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15774.727414 secs 2024-07-27T11:12:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15774.727414 secs 2024-07-27T11:12:32.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15774.727414 secs 2024-07-27T11:12:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:33 smithi077 ceph-mon[37153]: pgmap v8807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:33 smithi047 ceph-mon[36835]: pgmap v8807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:33 smithi029 ceph-mon[32610]: pgmap v8807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:35.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:12:35.501+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:12:35.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:12:35.501+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:12:35.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:12:35.501+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:12:35.503 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:35 smithi029 ceph-mon[32610]: pgmap v8808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:35 smithi077 ceph-mon[37153]: pgmap v8808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:35 smithi047 ceph-mon[36835]: pgmap v8808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:37.498 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:37 smithi047 ceph-mon[36835]: pgmap v8809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:37.498 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15779.727552 secs 2024-07-27T11:12:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:37 smithi077 ceph-mon[37153]: pgmap v8809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15779.727552 secs 2024-07-27T11:12:37.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:37 smithi029 ceph-mon[32610]: pgmap v8809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15779.727552 secs 2024-07-27T11:12:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:39 smithi077 ceph-mon[37153]: pgmap v8810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:39 smithi047 ceph-mon[36835]: pgmap v8810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:39.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:39 smithi029 ceph-mon[32610]: pgmap v8810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:41.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:41 smithi029 ceph-mon[32610]: pgmap v8811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:41 smithi077 ceph-mon[37153]: pgmap v8811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:41 smithi047 ceph-mon[36835]: pgmap v8811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15784.727692 secs 2024-07-27T11:12:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15784.727692 secs 2024-07-27T11:12:42.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15784.727692 secs 2024-07-27T11:12:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:43 smithi029 ceph-mon[32610]: pgmap v8812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:43 smithi077 ceph-mon[37153]: pgmap v8812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:43 smithi047 ceph-mon[36835]: pgmap v8812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:45 smithi077 ceph-mon[37153]: pgmap v8813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:45 smithi047 ceph-mon[36835]: pgmap v8813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:45.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:45 smithi029 ceph-mon[32610]: pgmap v8813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:47 smithi077 ceph-mon[37153]: pgmap v8814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15789.727817 secs 2024-07-27T11:12:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:47 smithi047 ceph-mon[36835]: pgmap v8814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15789.727817 secs 2024-07-27T11:12:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:47 smithi029 ceph-mon[32610]: pgmap v8814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15789.727817 secs 2024-07-27T11:12:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:49 smithi077 ceph-mon[37153]: pgmap v8815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:49 smithi047 ceph-mon[36835]: pgmap v8815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:49 smithi029 ceph-mon[32610]: pgmap v8815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:12:50.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:12:50.516+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:12:50.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:12:50.516+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:12:50.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:12:50.516+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:12:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:51 smithi077 ceph-mon[37153]: pgmap v8816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:51 smithi047 ceph-mon[36835]: pgmap v8816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:51 smithi029 ceph-mon[32610]: pgmap v8816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:12:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15794.727936 secs 2024-07-27T11:12:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15794.727936 secs 2024-07-27T11:12:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15794.727936 secs 2024-07-27T11:12:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:53 smithi077 ceph-mon[37153]: pgmap v8817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:53 smithi047 ceph-mon[36835]: pgmap v8817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:53 smithi029 ceph-mon[32610]: pgmap v8817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:55 smithi077 ceph-mon[37153]: pgmap v8818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:55 smithi047 ceph-mon[36835]: pgmap v8818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:55 smithi029 ceph-mon[32610]: pgmap v8818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:12:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:12:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:12:56.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:12:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:57 smithi077 ceph-mon[37153]: pgmap v8819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:12:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:12:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:57 smithi077 ceph-mon[37153]: pgmap v8820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:12:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:57 smithi077 ceph-mon[37153]: pgmap v8821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:12:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:12:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15799.728020 secs 2024-07-27T11:12:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:57 smithi047 ceph-mon[36835]: pgmap v8819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:12:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:12:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:57 smithi047 ceph-mon[36835]: pgmap v8820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:12:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:57 smithi047 ceph-mon[36835]: pgmap v8821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:12:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:12:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15799.728020 secs 2024-07-27T11:12:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:57 smithi029 ceph-mon[32610]: pgmap v8819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:12:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:12:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:12:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:57 smithi029 ceph-mon[32610]: pgmap v8820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:12:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:57 smithi029 ceph-mon[32610]: pgmap v8821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:12:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:12:57.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15799.728020 secs 2024-07-27T11:13:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:12:59 smithi077 ceph-mon[37153]: pgmap v8822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:13:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:12:59 smithi047 ceph-mon[36835]: pgmap v8822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:13:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:12:59 smithi029 ceph-mon[32610]: pgmap v8822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:13:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:01 smithi077 ceph-mon[37153]: pgmap v8823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:13:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:01 smithi047 ceph-mon[36835]: pgmap v8823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:13:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:01 smithi029 ceph-mon[32610]: pgmap v8823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:13:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15804.728146 secs 2024-07-27T11:13:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15804.728146 secs 2024-07-27T11:13:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15804.728146 secs 2024-07-27T11:13:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:03 smithi077 ceph-mon[37153]: pgmap v8824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:13:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:03 smithi047 ceph-mon[36835]: pgmap v8824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:13:04.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:03 smithi029 ceph-mon[32610]: pgmap v8824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:13:05.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:13:05.531+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:13:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:13:05.531+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:13:05.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:13:05.531+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:13:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:05 smithi029 ceph-mon[32610]: pgmap v8825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:13:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:05 smithi077 ceph-mon[37153]: pgmap v8825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:13:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:05 smithi047 ceph-mon[36835]: pgmap v8825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:13:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:07 smithi077 ceph-mon[37153]: pgmap v8826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:13:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15809.728225 secs 2024-07-27T11:13:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:07 smithi047 ceph-mon[36835]: pgmap v8826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:13:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15809.728225 secs 2024-07-27T11:13:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:07 smithi029 ceph-mon[32610]: pgmap v8826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:13:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15809.728225 secs 2024-07-27T11:13:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:09 smithi077 ceph-mon[37153]: pgmap v8827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:09 smithi047 ceph-mon[36835]: pgmap v8827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:09 smithi029 ceph-mon[32610]: pgmap v8827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:12.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:11 smithi029 ceph-mon[32610]: pgmap v8828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:11 smithi077 ceph-mon[37153]: pgmap v8828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:11 smithi047 ceph-mon[36835]: pgmap v8828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15814.728344 secs 2024-07-27T11:13:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15814.728344 secs 2024-07-27T11:13:13.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15814.728344 secs 2024-07-27T11:13:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:13 smithi077 ceph-mon[37153]: pgmap v8829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:13 smithi047 ceph-mon[36835]: pgmap v8829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:14.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:13 smithi029 ceph-mon[32610]: pgmap v8829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:15 smithi077 ceph-mon[37153]: pgmap v8830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:15 smithi047 ceph-mon[36835]: pgmap v8830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:16.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:15 smithi029 ceph-mon[32610]: pgmap v8830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:17 smithi077 ceph-mon[37153]: pgmap v8831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15819.728459 secs 2024-07-27T11:13:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:17 smithi047 ceph-mon[36835]: pgmap v8831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15819.728459 secs 2024-07-27T11:13:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:17 smithi029 ceph-mon[32610]: pgmap v8831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15819.728459 secs 2024-07-27T11:13:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:19 smithi077 ceph-mon[37153]: pgmap v8832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:19 smithi047 ceph-mon[36835]: pgmap v8832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:20.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:19 smithi029 ceph-mon[32610]: pgmap v8832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:20.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:13:20.546+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:13:20.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:13:20.546+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:13:20.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:13:20.546+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:13:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:21 smithi077 ceph-mon[37153]: pgmap v8833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:21 smithi047 ceph-mon[36835]: pgmap v8833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:22.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:21 smithi029 ceph-mon[32610]: pgmap v8833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15824.728599 secs 2024-07-27T11:13:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15824.728599 secs 2024-07-27T11:13:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15824.728599 secs 2024-07-27T11:13:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:23 smithi077 ceph-mon[37153]: pgmap v8834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:23 smithi047 ceph-mon[36835]: pgmap v8834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:24.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:23 smithi029 ceph-mon[32610]: pgmap v8834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:25 smithi077 ceph-mon[37153]: pgmap v8835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:25 smithi047 ceph-mon[36835]: pgmap v8835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:25 smithi029 ceph-mon[32610]: pgmap v8835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:27 smithi077 ceph-mon[37153]: pgmap v8836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15829.728736 secs 2024-07-27T11:13:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:27 smithi047 ceph-mon[36835]: pgmap v8836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15829.728736 secs 2024-07-27T11:13:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:27 smithi029 ceph-mon[32610]: pgmap v8836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15829.728736 secs 2024-07-27T11:13:30.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:29 smithi077 ceph-mon[37153]: pgmap v8837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:30.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:29 smithi029 ceph-mon[32610]: pgmap v8837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:29 smithi047 ceph-mon[36835]: pgmap v8837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:31 smithi077 ceph-mon[37153]: pgmap v8838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:31 smithi047 ceph-mon[36835]: pgmap v8838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:32.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:31 smithi029 ceph-mon[32610]: pgmap v8838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15834.728880 secs 2024-07-27T11:13:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15834.728880 secs 2024-07-27T11:13:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15834.728880 secs 2024-07-27T11:13:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:33 smithi077 ceph-mon[37153]: pgmap v8839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:33 smithi047 ceph-mon[36835]: pgmap v8839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:34.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:33 smithi029 ceph-mon[32610]: pgmap v8839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:35.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:13:35.562+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:13:35.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:13:35.562+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:13:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:13:35.562+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:13:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:35 smithi077 ceph-mon[37153]: pgmap v8840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:35 smithi047 ceph-mon[36835]: pgmap v8840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:35 smithi029 ceph-mon[32610]: pgmap v8840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:37 smithi077 ceph-mon[37153]: pgmap v8841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15839.728957 secs 2024-07-27T11:13:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:37 smithi047 ceph-mon[36835]: pgmap v8841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15839.728957 secs 2024-07-27T11:13:38.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:37 smithi029 ceph-mon[32610]: pgmap v8841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15839.728957 secs 2024-07-27T11:13:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:39 smithi077 ceph-mon[37153]: pgmap v8842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:39 smithi047 ceph-mon[36835]: pgmap v8842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:40.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:39 smithi029 ceph-mon[32610]: pgmap v8842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:41 smithi077 ceph-mon[37153]: pgmap v8843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:41 smithi047 ceph-mon[36835]: pgmap v8843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:42.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:41 smithi029 ceph-mon[32610]: pgmap v8843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15844.729039 secs 2024-07-27T11:13:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15844.729039 secs 2024-07-27T11:13:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15844.729039 secs 2024-07-27T11:13:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:43 smithi077 ceph-mon[37153]: pgmap v8844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:43 smithi047 ceph-mon[36835]: pgmap v8844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:44.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:43 smithi029 ceph-mon[32610]: pgmap v8844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:45 smithi077 ceph-mon[37153]: pgmap v8845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:45 smithi047 ceph-mon[36835]: pgmap v8845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:45 smithi029 ceph-mon[32610]: pgmap v8845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:47 smithi077 ceph-mon[37153]: pgmap v8846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15849.729160 secs 2024-07-27T11:13:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:47 smithi047 ceph-mon[36835]: pgmap v8846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15849.729160 secs 2024-07-27T11:13:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:47 smithi029 ceph-mon[32610]: pgmap v8846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15849.729160 secs 2024-07-27T11:13:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:49 smithi077 ceph-mon[37153]: pgmap v8847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:49 smithi047 ceph-mon[36835]: pgmap v8847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:49 smithi029 ceph-mon[32610]: pgmap v8847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:13:50.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:13:50.572+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:13:50.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:13:50.572+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:13:50.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:13:50.572+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:13:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:51 smithi077 ceph-mon[37153]: pgmap v8848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:51 smithi047 ceph-mon[36835]: pgmap v8848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:52.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:51 smithi029 ceph-mon[32610]: pgmap v8848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:13:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15854.729288 secs 2024-07-27T11:13:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15854.729288 secs 2024-07-27T11:13:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15854.729288 secs 2024-07-27T11:13:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:53 smithi077 ceph-mon[37153]: pgmap v8849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:53 smithi047 ceph-mon[36835]: pgmap v8849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:54.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:53 smithi029 ceph-mon[32610]: pgmap v8849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:13:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:55 smithi077 ceph-mon[37153]: pgmap v8850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:55 smithi047 ceph-mon[36835]: pgmap v8850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:55 smithi029 ceph-mon[32610]: pgmap v8850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:13:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:13:57.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:13:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:13:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:57 smithi047 ceph-mon[36835]: pgmap v8851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15859.729351 secs 2024-07-27T11:13:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:13:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:13:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:13:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:57 smithi029 ceph-mon[32610]: pgmap v8851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15859.729351 secs 2024-07-27T11:13:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:13:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:13:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:13:58.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:57 smithi077 ceph-mon[37153]: pgmap v8851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:13:58.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15859.729351 secs 2024-07-27T11:13:58.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:13:58.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:13:58.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:13:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:58 smithi047 ceph-mon[36835]: pgmap v8852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T11:13:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:13:58 smithi047 ceph-mon[36835]: pgmap v8853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T11:13:59.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:58 smithi029 ceph-mon[32610]: pgmap v8852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T11:13:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:13:58 smithi029 ceph-mon[32610]: pgmap v8853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T11:13:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:58 smithi077 ceph-mon[37153]: pgmap v8852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T11:13:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:13:58 smithi077 ceph-mon[37153]: pgmap v8853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T11:14:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:00.046+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:14:00.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:00.046+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:14:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:00.046+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:14:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:00.046+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:14:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:00.047+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:14:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:00.047+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:14:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:00.047+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:14:00.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:00.047+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:14:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:00 smithi047 ceph-mon[36835]: pgmap v8854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:14:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:00 smithi029 ceph-mon[32610]: pgmap v8854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:14:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:00 smithi077 ceph-mon[37153]: pgmap v8854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:14:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:02 smithi047 ceph-mon[36835]: pgmap v8855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:14:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15864.729480 secs 2024-07-27T11:14:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:02 smithi029 ceph-mon[32610]: pgmap v8855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:14:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15864.729480 secs 2024-07-27T11:14:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:02 smithi077 ceph-mon[37153]: pgmap v8855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:14:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15864.729480 secs 2024-07-27T11:14:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:04 smithi047 ceph-mon[36835]: pgmap v8856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:14:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:04 smithi029 ceph-mon[32610]: pgmap v8856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:14:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:04 smithi077 ceph-mon[37153]: pgmap v8856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:14:05.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:05.575+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:14:05.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:05.575+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:14:05.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:05.575+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:14:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:06 smithi047 ceph-mon[36835]: pgmap v8857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:14:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:06 smithi029 ceph-mon[32610]: pgmap v8857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:14:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:06 smithi077 ceph-mon[37153]: pgmap v8857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:14:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15869.729575 secs 2024-07-27T11:14:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15869.729575 secs 2024-07-27T11:14:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15869.729575 secs 2024-07-27T11:14:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:08 smithi047 ceph-mon[36835]: pgmap v8858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:14:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:08 smithi029 ceph-mon[32610]: pgmap v8858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:14:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:08 smithi077 ceph-mon[37153]: pgmap v8858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:14:11.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:10 smithi077 ceph-mon[37153]: pgmap v8859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:10 smithi047 ceph-mon[36835]: pgmap v8859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:10 smithi029 ceph-mon[32610]: pgmap v8859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:12 smithi047 ceph-mon[36835]: pgmap v8860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15874.729717 secs 2024-07-27T11:14:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:12 smithi029 ceph-mon[32610]: pgmap v8860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15874.729717 secs 2024-07-27T11:14:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:12 smithi077 ceph-mon[37153]: pgmap v8860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15874.729717 secs 2024-07-27T11:14:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:14 smithi047 ceph-mon[36835]: pgmap v8861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:14 smithi029 ceph-mon[32610]: pgmap v8861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:14 smithi077 ceph-mon[37153]: pgmap v8861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:16 smithi047 ceph-mon[36835]: pgmap v8862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:16 smithi029 ceph-mon[32610]: pgmap v8862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:16 smithi077 ceph-mon[37153]: pgmap v8862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15879.729837 secs 2024-07-27T11:14:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15879.729837 secs 2024-07-27T11:14:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15879.729837 secs 2024-07-27T11:14:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:18 smithi047 ceph-mon[36835]: pgmap v8863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:18 smithi029 ceph-mon[32610]: pgmap v8863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:18 smithi077 ceph-mon[37153]: pgmap v8863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:20.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:20.589+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:14:20.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:20.589+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:14:20.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:20.589+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:14:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:20 smithi047 ceph-mon[36835]: pgmap v8864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:20 smithi029 ceph-mon[32610]: pgmap v8864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:20 smithi077 ceph-mon[37153]: pgmap v8864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:22 smithi047 ceph-mon[36835]: pgmap v8865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15884.729959 secs 2024-07-27T11:14:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:22 smithi029 ceph-mon[32610]: pgmap v8865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15884.729959 secs 2024-07-27T11:14:23.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:22 smithi077 ceph-mon[37153]: pgmap v8865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:23.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15884.729959 secs 2024-07-27T11:14:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:24 smithi047 ceph-mon[36835]: pgmap v8866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:24 smithi029 ceph-mon[32610]: pgmap v8866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:24 smithi077 ceph-mon[37153]: pgmap v8866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:26 smithi047 ceph-mon[36835]: pgmap v8867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:26 smithi029 ceph-mon[32610]: pgmap v8867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:26 smithi077 ceph-mon[37153]: pgmap v8867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15889.730049 secs 2024-07-27T11:14:28.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15889.730049 secs 2024-07-27T11:14:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15889.730049 secs 2024-07-27T11:14:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:28 smithi047 ceph-mon[36835]: pgmap v8868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:29.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:28 smithi029 ceph-mon[32610]: pgmap v8868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:28 smithi077 ceph-mon[37153]: pgmap v8868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:30 smithi047 ceph-mon[36835]: pgmap v8869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:30 smithi029 ceph-mon[32610]: pgmap v8869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:30 smithi077 ceph-mon[37153]: pgmap v8869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:32 smithi047 ceph-mon[36835]: pgmap v8870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15894.730163 secs 2024-07-27T11:14:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:32 smithi029 ceph-mon[32610]: pgmap v8870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15894.730163 secs 2024-07-27T11:14:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:32 smithi077 ceph-mon[37153]: pgmap v8870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15894.730163 secs 2024-07-27T11:14:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:34 smithi047 ceph-mon[36835]: pgmap v8871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:34 smithi029 ceph-mon[32610]: pgmap v8871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:34 smithi077 ceph-mon[37153]: pgmap v8871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:35.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:35.605+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:14:35.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:35.605+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:14:35.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:35.605+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:14:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:36 smithi047 ceph-mon[36835]: pgmap v8872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:36 smithi029 ceph-mon[32610]: pgmap v8872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:36 smithi077 ceph-mon[37153]: pgmap v8872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15899.730279 secs 2024-07-27T11:14:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15899.730279 secs 2024-07-27T11:14:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15899.730279 secs 2024-07-27T11:14:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:38 smithi029 ceph-mon[32610]: pgmap v8873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:38 smithi077 ceph-mon[37153]: pgmap v8873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:38 smithi047 ceph-mon[36835]: pgmap v8873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:40 smithi029 ceph-mon[32610]: pgmap v8874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:40 smithi077 ceph-mon[37153]: pgmap v8874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:40 smithi047 ceph-mon[36835]: pgmap v8874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:42 smithi029 ceph-mon[32610]: pgmap v8875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15904.730406 secs 2024-07-27T11:14:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:42 smithi077 ceph-mon[37153]: pgmap v8875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15904.730406 secs 2024-07-27T11:14:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:42 smithi047 ceph-mon[36835]: pgmap v8875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15904.730406 secs 2024-07-27T11:14:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:44 smithi029 ceph-mon[32610]: pgmap v8876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:44 smithi077 ceph-mon[37153]: pgmap v8876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:44 smithi047 ceph-mon[36835]: pgmap v8876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:46 smithi077 ceph-mon[37153]: pgmap v8877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:46 smithi047 ceph-mon[36835]: pgmap v8877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:47.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:46 smithi029 ceph-mon[32610]: pgmap v8877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15909.730531 secs 2024-07-27T11:14:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15909.730531 secs 2024-07-27T11:14:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15909.730531 secs 2024-07-27T11:14:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:48 smithi077 ceph-mon[37153]: pgmap v8878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:48 smithi047 ceph-mon[36835]: pgmap v8878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:48 smithi029 ceph-mon[32610]: pgmap v8878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:50.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:50.620+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:14:50.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:50.620+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:14:50.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:14:50.620+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:14:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:50 smithi077 ceph-mon[37153]: pgmap v8879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:50 smithi047 ceph-mon[36835]: pgmap v8879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:50 smithi029 ceph-mon[32610]: pgmap v8879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:14:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:52 smithi077 ceph-mon[37153]: pgmap v8880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15914.730654 secs 2024-07-27T11:14:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:52 smithi047 ceph-mon[36835]: pgmap v8880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15914.730654 secs 2024-07-27T11:14:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:52 smithi029 ceph-mon[32610]: pgmap v8880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:14:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15914.730654 secs 2024-07-27T11:14:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:54 smithi077 ceph-mon[37153]: pgmap v8881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:54 smithi047 ceph-mon[36835]: pgmap v8881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:55.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:54 smithi029 ceph-mon[32610]: pgmap v8881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:14:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:56 smithi077 ceph-mon[37153]: pgmap v8882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:56 smithi047 ceph-mon[36835]: pgmap v8882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:57.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:56 smithi029 ceph-mon[32610]: pgmap v8882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:14:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15919.730728 secs 2024-07-27T11:14:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:14:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:14:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:14:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:14:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15919.730728 secs 2024-07-27T11:14:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:14:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:14:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:14:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:14:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15919.730728 secs 2024-07-27T11:14:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:14:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:14:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:14:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:14:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:58 smithi077 ceph-mon[37153]: pgmap v8883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:58 smithi077 ceph-mon[37153]: pgmap v8884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:14:59.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:14:58 smithi077 ceph-mon[37153]: pgmap v8885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:14:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:58 smithi047 ceph-mon[36835]: pgmap v8883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:58 smithi047 ceph-mon[36835]: pgmap v8884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:14:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:14:58 smithi047 ceph-mon[36835]: pgmap v8885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:14:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:58 smithi029 ceph-mon[32610]: pgmap v8883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:14:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:58 smithi029 ceph-mon[32610]: pgmap v8884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:14:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:14:58 smithi029 ceph-mon[32610]: pgmap v8885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:15:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:00 smithi077 ceph-mon[37153]: pgmap v8886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:15:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:00 smithi047 ceph-mon[36835]: pgmap v8886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:15:01.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:00 smithi029 ceph-mon[32610]: pgmap v8886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:15:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:02 smithi029 ceph-mon[32610]: pgmap v8887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:15:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15924.730851 secs 2024-07-27T11:15:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:02 smithi077 ceph-mon[37153]: pgmap v8887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:15:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15924.730851 secs 2024-07-27T11:15:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:02 smithi047 ceph-mon[36835]: pgmap v8887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:15:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15924.730851 secs 2024-07-27T11:15:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:05 smithi077 ceph-mon[37153]: pgmap v8888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:15:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:05 smithi047 ceph-mon[36835]: pgmap v8888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:15:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:05 smithi029 ceph-mon[32610]: pgmap v8888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:15:05.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:15:05.631+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:15:05.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:15:05.631+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:15:05.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:15:05.631+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:15:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:07 smithi077 ceph-mon[37153]: pgmap v8889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T11:15:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:07 smithi047 ceph-mon[36835]: pgmap v8889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T11:15:07.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:07 smithi029 ceph-mon[32610]: pgmap v8889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T11:15:08.012 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15929.730985 secs 2024-07-27T11:15:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15929.730985 secs 2024-07-27T11:15:08.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15929.730985 secs 2024-07-27T11:15:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:09 smithi077 ceph-mon[37153]: pgmap v8890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:15:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:09 smithi047 ceph-mon[36835]: pgmap v8890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:15:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:09 smithi029 ceph-mon[32610]: pgmap v8890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:15:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:11 smithi077 ceph-mon[37153]: pgmap v8891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:11 smithi047 ceph-mon[36835]: pgmap v8891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:11.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:11 smithi029 ceph-mon[32610]: pgmap v8891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:13 smithi077 ceph-mon[37153]: pgmap v8892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15934.731047 secs 2024-07-27T11:15:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:13 smithi047 ceph-mon[36835]: pgmap v8892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15934.731047 secs 2024-07-27T11:15:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:13 smithi029 ceph-mon[32610]: pgmap v8892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15934.731047 secs 2024-07-27T11:15:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:15 smithi077 ceph-mon[37153]: pgmap v8893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:15:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:15 smithi047 ceph-mon[36835]: pgmap v8893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:15:15.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:15 smithi029 ceph-mon[32610]: pgmap v8893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:15:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:17 smithi077 ceph-mon[37153]: pgmap v8894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:17 smithi047 ceph-mon[36835]: pgmap v8894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:17 smithi029 ceph-mon[32610]: pgmap v8894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15939.731165 secs 2024-07-27T11:15:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15939.731165 secs 2024-07-27T11:15:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15939.731165 secs 2024-07-27T11:15:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:19 smithi077 ceph-mon[37153]: pgmap v8895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:15:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:19 smithi047 ceph-mon[36835]: pgmap v8895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:15:19.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:19 smithi029 ceph-mon[32610]: pgmap v8895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:15:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:15:20.647+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:15:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:15:20.647+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:15:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:15:20.647+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:15:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:21 smithi077 ceph-mon[37153]: pgmap v8896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:21 smithi047 ceph-mon[36835]: pgmap v8896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:21 smithi029 ceph-mon[32610]: pgmap v8896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:23 smithi077 ceph-mon[37153]: pgmap v8897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15944.731288 secs 2024-07-27T11:15:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:23 smithi047 ceph-mon[36835]: pgmap v8897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15944.731288 secs 2024-07-27T11:15:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:23 smithi029 ceph-mon[32610]: pgmap v8897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15944.731288 secs 2024-07-27T11:15:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:25 smithi077 ceph-mon[37153]: pgmap v8898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:15:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:25 smithi047 ceph-mon[36835]: pgmap v8898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:15:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:25 smithi029 ceph-mon[32610]: pgmap v8898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:15:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:27 smithi077 ceph-mon[37153]: pgmap v8899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:27 smithi047 ceph-mon[36835]: pgmap v8899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:27 smithi029 ceph-mon[32610]: pgmap v8899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15949.731410 secs 2024-07-27T11:15:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15949.731410 secs 2024-07-27T11:15:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15949.731410 secs 2024-07-27T11:15:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:29 smithi077 ceph-mon[37153]: pgmap v8900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:15:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:29 smithi047 ceph-mon[36835]: pgmap v8900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:15:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:29 smithi029 ceph-mon[32610]: pgmap v8900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:15:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:31 smithi077 ceph-mon[37153]: pgmap v8901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:31 smithi047 ceph-mon[36835]: pgmap v8901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:31.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:31 smithi029 ceph-mon[32610]: pgmap v8901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:33 smithi077 ceph-mon[37153]: pgmap v8902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15954.731533 secs 2024-07-27T11:15:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:33 smithi047 ceph-mon[36835]: pgmap v8902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15954.731533 secs 2024-07-27T11:15:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:33 smithi029 ceph-mon[32610]: pgmap v8902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15954.731533 secs 2024-07-27T11:15:34.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:34 smithi077 ceph-mon[37153]: pgmap v8903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:15:34.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:34 smithi029 ceph-mon[32610]: pgmap v8903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:15:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:34 smithi047 ceph-mon[36835]: pgmap v8903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:15:35.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:15:35.655+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:15:35.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:15:35.655+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:15:35.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:15:35.655+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:15:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:36 smithi047 ceph-mon[36835]: pgmap v8904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:15:37.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:36 smithi029 ceph-mon[32610]: pgmap v8904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:15:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:36 smithi077 ceph-mon[37153]: pgmap v8904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:15:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15959.731668 secs 2024-07-27T11:15:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15959.731668 secs 2024-07-27T11:15:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15959.731668 secs 2024-07-27T11:15:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:38 smithi047 ceph-mon[36835]: pgmap v8905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:15:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:38 smithi029 ceph-mon[32610]: pgmap v8905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:15:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:38 smithi077 ceph-mon[37153]: pgmap v8905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:15:41.158 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:40 smithi077 ceph-mon[37153]: pgmap v8906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:15:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:40 smithi047 ceph-mon[36835]: pgmap v8906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:15:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:40 smithi029 ceph-mon[32610]: pgmap v8906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:15:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:42 smithi047 ceph-mon[36835]: pgmap v8907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15964.731799 secs 2024-07-27T11:15:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:42 smithi029 ceph-mon[32610]: pgmap v8907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15964.731799 secs 2024-07-27T11:15:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:42 smithi077 ceph-mon[37153]: pgmap v8907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15964.731799 secs 2024-07-27T11:15:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:44 smithi047 ceph-mon[36835]: pgmap v8908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:15:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:44 smithi029 ceph-mon[32610]: pgmap v8908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:15:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:44 smithi077 ceph-mon[37153]: pgmap v8908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:15:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:46 smithi047 ceph-mon[36835]: pgmap v8909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:46 smithi029 ceph-mon[32610]: pgmap v8909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:46 smithi077 ceph-mon[37153]: pgmap v8909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15969.731898 secs 2024-07-27T11:15:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15969.731898 secs 2024-07-27T11:15:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15969.731898 secs 2024-07-27T11:15:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:48 smithi047 ceph-mon[36835]: pgmap v8910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:15:49.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:48 smithi029 ceph-mon[32610]: pgmap v8910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:15:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:48 smithi077 ceph-mon[37153]: pgmap v8910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:15:50.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:15:50.662+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:15:50.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:15:50.662+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:15:50.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:15:50.662+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:15:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:50 smithi029 ceph-mon[32610]: pgmap v8911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:51.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:50 smithi047 ceph-mon[36835]: pgmap v8911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:50 smithi077 ceph-mon[37153]: pgmap v8911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:15:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:52 smithi047 ceph-mon[36835]: pgmap v8912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15974.732016 secs 2024-07-27T11:15:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:52 smithi029 ceph-mon[32610]: pgmap v8912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15974.732016 secs 2024-07-27T11:15:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:52 smithi077 ceph-mon[37153]: pgmap v8912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:15:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15974.732016 secs 2024-07-27T11:15:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:54 smithi047 ceph-mon[36835]: pgmap v8913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:15:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:54 smithi029 ceph-mon[32610]: pgmap v8913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:15:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:54 smithi077 ceph-mon[37153]: pgmap v8913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:15:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:56 smithi047 ceph-mon[36835]: pgmap v8914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:15:57.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:56 smithi029 ceph-mon[32610]: pgmap v8914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:15:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:56 smithi077 ceph-mon[37153]: pgmap v8914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:15:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15979.732101 secs 2024-07-27T11:15:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:15:58.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15979.732101 secs 2024-07-27T11:15:58.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:15:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15979.732101 secs 2024-07-27T11:15:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:15:59.028 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:15:58 smithi047 ceph-mon[36835]: pgmap v8915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:15:59.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:15:58 smithi029 ceph-mon[32610]: pgmap v8915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:15:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:15:58 smithi077 ceph-mon[37153]: pgmap v8915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:16:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:00 smithi047 ceph-mon[36835]: pgmap v8916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:16:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:00 smithi029 ceph-mon[32610]: pgmap v8916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:16:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:00 smithi077 ceph-mon[37153]: pgmap v8916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:16:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: pgmap v8917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15984.732170 secs 2024-07-27T11:16:03.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:16:03.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:16:03.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:16:03.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:16:03.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:16:03.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:16:03.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:16:03.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:16:03.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: pgmap v8918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:16:03.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: pgmap v8919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T11:16:03.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:02 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: pgmap v8917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15984.732170 secs 2024-07-27T11:16:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:16:03.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:16:03.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:16:03.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:16:03.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:16:03.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:16:03.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:16:03.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:16:03.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: pgmap v8918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:16:03.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: pgmap v8919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T11:16:03.200 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:02 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: pgmap v8917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:03.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15984.732170 secs 2024-07-27T11:16:03.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:16:03.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:16:03.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:16:03.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:16:03.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:16:03.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:16:03.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:16:03.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:16:03.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: pgmap v8918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:16:03.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: pgmap v8919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T11:16:03.215 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:02 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:16:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:04 smithi047 ceph-mon[36835]: pgmap v8920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 7 op/s 2024-07-27T11:16:04.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:04 smithi029 ceph-mon[32610]: pgmap v8920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 7 op/s 2024-07-27T11:16:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:04 smithi077 ceph-mon[37153]: pgmap v8920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 7 op/s 2024-07-27T11:16:05.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:16:05.676+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:16:05.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:16:05.676+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:16:05.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:16:05.676+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:16:06.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:06 smithi029 ceph-mon[32610]: pgmap v8921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-27T11:16:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:06 smithi047 ceph-mon[36835]: pgmap v8921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-27T11:16:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:06 smithi077 ceph-mon[37153]: pgmap v8921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-27T11:16:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15989.732231 secs 2024-07-27T11:16:07.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15989.732231 secs 2024-07-27T11:16:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15989.732231 secs 2024-07-27T11:16:08.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:08 smithi047 ceph-mon[36835]: pgmap v8922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:16:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:08 smithi077 ceph-mon[37153]: pgmap v8922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:16:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:08 smithi029 ceph-mon[32610]: pgmap v8922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:16:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:10 smithi047 ceph-mon[36835]: pgmap v8923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T11:16:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:10 smithi029 ceph-mon[32610]: pgmap v8923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T11:16:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:10 smithi077 ceph-mon[37153]: pgmap v8923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T11:16:12.393 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:12 smithi047 ceph-mon[36835]: pgmap v8924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:16:12.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:12 smithi077 ceph-mon[37153]: pgmap v8924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:16:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:12 smithi029 ceph-mon[32610]: pgmap v8924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:16:13.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15994.732352 secs 2024-07-27T11:16:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15994.732352 secs 2024-07-27T11:16:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15994.732352 secs 2024-07-27T11:16:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:14 smithi047 ceph-mon[36835]: pgmap v8925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:14.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:14 smithi029 ceph-mon[32610]: pgmap v8925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:14 smithi077 ceph-mon[37153]: pgmap v8925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:16 smithi047 ceph-mon[36835]: pgmap v8926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:16 smithi029 ceph-mon[32610]: pgmap v8926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:16 smithi077 ceph-mon[37153]: pgmap v8926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 15999.732470 secs 2024-07-27T11:16:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 15999.732470 secs 2024-07-27T11:16:17.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 15999.732470 secs 2024-07-27T11:16:18.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:18 smithi077 ceph-mon[37153]: pgmap v8927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:16:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:18 smithi047 ceph-mon[36835]: pgmap v8927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:16:18.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:18 smithi029 ceph-mon[32610]: pgmap v8927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:16:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:20 smithi047 ceph-mon[36835]: pgmap v8928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:20 smithi029 ceph-mon[32610]: pgmap v8928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:20 smithi077 ceph-mon[37153]: pgmap v8928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:20.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:16:20.685+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:16:20.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:16:20.685+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:16:20.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:16:20.685+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:16:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:22 smithi047 ceph-mon[36835]: pgmap v8929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:22 smithi029 ceph-mon[32610]: pgmap v8929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:22 smithi077 ceph-mon[37153]: pgmap v8929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:23.413 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16004.732592 secs 2024-07-27T11:16:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16004.732592 secs 2024-07-27T11:16:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16004.732592 secs 2024-07-27T11:16:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:24 smithi077 ceph-mon[37153]: pgmap v8930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:24 smithi047 ceph-mon[36835]: pgmap v8930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:24.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:24 smithi029 ceph-mon[32610]: pgmap v8930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:26 smithi047 ceph-mon[36835]: pgmap v8931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:26.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:26 smithi029 ceph-mon[32610]: pgmap v8931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:26 smithi077 ceph-mon[37153]: pgmap v8931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16009.732715 secs 2024-07-27T11:16:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16009.732715 secs 2024-07-27T11:16:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16009.732715 secs 2024-07-27T11:16:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:28 smithi077 ceph-mon[37153]: pgmap v8932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:16:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:28 smithi047 ceph-mon[36835]: pgmap v8932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:16:28.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:28 smithi029 ceph-mon[32610]: pgmap v8932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:16:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:30 smithi047 ceph-mon[36835]: pgmap v8933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:30.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:30 smithi029 ceph-mon[32610]: pgmap v8933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:30 smithi077 ceph-mon[37153]: pgmap v8933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:32 smithi047 ceph-mon[36835]: pgmap v8934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:32 smithi029 ceph-mon[32610]: pgmap v8934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:32 smithi077 ceph-mon[37153]: pgmap v8934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16014.732869 secs 2024-07-27T11:16:33.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16014.732869 secs 2024-07-27T11:16:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16014.732869 secs 2024-07-27T11:16:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:34 smithi047 ceph-mon[36835]: pgmap v8935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:34.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:34 smithi029 ceph-mon[32610]: pgmap v8935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:34 smithi077 ceph-mon[37153]: pgmap v8935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:16:35.691+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:16:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:16:35.691+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:16:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:16:35.691+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:16:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:36 smithi047 ceph-mon[36835]: pgmap v8936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:36.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:36 smithi029 ceph-mon[32610]: pgmap v8936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:36 smithi077 ceph-mon[37153]: pgmap v8936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:37.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16019.733017 secs 2024-07-27T11:16:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16019.733017 secs 2024-07-27T11:16:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16019.733017 secs 2024-07-27T11:16:38.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:38 smithi047 ceph-mon[36835]: pgmap v8937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:16:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:38 smithi077 ceph-mon[37153]: pgmap v8937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:16:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:38 smithi029 ceph-mon[32610]: pgmap v8937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:16:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:40 smithi047 ceph-mon[36835]: pgmap v8938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:40.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:40 smithi029 ceph-mon[32610]: pgmap v8938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:40 smithi077 ceph-mon[37153]: pgmap v8938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:42.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:42 smithi077 ceph-mon[37153]: pgmap v8939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:42 smithi047 ceph-mon[36835]: pgmap v8939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:42 smithi029 ceph-mon[32610]: pgmap v8939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16024.733160 secs 2024-07-27T11:16:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16024.733160 secs 2024-07-27T11:16:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16024.733160 secs 2024-07-27T11:16:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:44 smithi047 ceph-mon[36835]: pgmap v8940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:44 smithi029 ceph-mon[32610]: pgmap v8940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:44 smithi077 ceph-mon[37153]: pgmap v8940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:46 smithi047 ceph-mon[36835]: pgmap v8941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:46 smithi029 ceph-mon[32610]: pgmap v8941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:46 smithi077 ceph-mon[37153]: pgmap v8941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16029.733281 secs 2024-07-27T11:16:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16029.733281 secs 2024-07-27T11:16:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16029.733281 secs 2024-07-27T11:16:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:48 smithi077 ceph-mon[37153]: pgmap v8942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:16:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:48 smithi047 ceph-mon[36835]: pgmap v8942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:16:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:48 smithi029 ceph-mon[32610]: pgmap v8942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:16:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:50 smithi047 ceph-mon[36835]: pgmap v8943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:50.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:50 smithi029 ceph-mon[32610]: pgmap v8943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:50 smithi077 ceph-mon[37153]: pgmap v8943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:50.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:16:50.706+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:16:50.707 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:16:50.706+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:16:50.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:16:50.706+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:16:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:52 smithi029 ceph-mon[32610]: pgmap v8944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:52 smithi047 ceph-mon[36835]: pgmap v8944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:52 smithi077 ceph-mon[37153]: pgmap v8944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:16:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16034.733403 secs 2024-07-27T11:16:53.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16034.733403 secs 2024-07-27T11:16:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16034.733403 secs 2024-07-27T11:16:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:54 smithi047 ceph-mon[36835]: pgmap v8945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:54 smithi029 ceph-mon[32610]: pgmap v8945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:54 smithi077 ceph-mon[37153]: pgmap v8945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:16:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:56 smithi047 ceph-mon[36835]: pgmap v8946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:56.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:56 smithi029 ceph-mon[32610]: pgmap v8946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:56 smithi077 ceph-mon[37153]: pgmap v8946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:16:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16039.733523 secs 2024-07-27T11:16:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16039.733523 secs 2024-07-27T11:16:57.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16039.733523 secs 2024-07-27T11:16:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:16:58 smithi077 ceph-mon[37153]: pgmap v8947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:16:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:16:58 smithi047 ceph-mon[36835]: pgmap v8947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:16:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:16:58 smithi029 ceph-mon[32610]: pgmap v8947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:17:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:00.073+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:17:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:00.073+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:17:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:00.073+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:17:00.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:00.073+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:17:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:00 smithi047 ceph-mon[36835]: pgmap v8948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:17:00.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:00 smithi029 ceph-mon[32610]: pgmap v8948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:17:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:00 smithi077 ceph-mon[37153]: pgmap v8948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:17:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:02 smithi047 ceph-mon[36835]: pgmap v8949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:02.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:02 smithi029 ceph-mon[32610]: pgmap v8949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:02 smithi077 ceph-mon[37153]: pgmap v8949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:03.252 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16044.733649 secs 2024-07-27T11:17:03.253 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:17:03.253 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:03.253 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:03.253 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:03.253 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:03 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:03.254 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:03 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:17:03.254 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:03 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:17:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16044.733649 secs 2024-07-27T11:17:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:17:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:03 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:03 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:17:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:03 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:17:03.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16044.733649 secs 2024-07-27T11:17:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:17:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:03 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:03 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:17:03.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:03 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:17:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:04 smithi077 ceph-mon[37153]: pgmap v8950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:17:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:04 smithi047 ceph-mon[36835]: pgmap v8950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:17:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:04 smithi029 ceph-mon[32610]: pgmap v8950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:17:05.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:05.709+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:17:05.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:05.709+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:17:05.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:05.709+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:17:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:17:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:17:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:17:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:17:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:17:07.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:17:07.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: pgmap v8951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:07.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: pgmap v8952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T11:17:07.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:17:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:17:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:17:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:17:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:17:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:17:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:17:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: pgmap v8951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: pgmap v8952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T11:17:07.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:17:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:17:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:17:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:17:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:17:07.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:17:07.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:17:07.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:17:07.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: pgmap v8951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:07.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: pgmap v8952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T11:17:07.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:17:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16049.733767 secs 2024-07-27T11:17:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16049.733767 secs 2024-07-27T11:17:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16049.733767 secs 2024-07-27T11:17:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:08 smithi047 ceph-mon[36835]: pgmap v8953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:17:09.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:08 smithi029 ceph-mon[32610]: pgmap v8953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:17:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:08 smithi077 ceph-mon[37153]: pgmap v8953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:17:11.360 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:11 smithi077 ceph-mon[37153]: pgmap v8954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:17:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:11 smithi047 ceph-mon[36835]: pgmap v8954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:17:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:11 smithi029 ceph-mon[32610]: pgmap v8954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:17:13.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:13 smithi029 ceph-mon[32610]: pgmap v8955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:17:13.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16054.733892 secs 2024-07-27T11:17:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:13 smithi077 ceph-mon[37153]: pgmap v8955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:17:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16054.733892 secs 2024-07-27T11:17:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:13 smithi047 ceph-mon[36835]: pgmap v8955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:17:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16054.733892 secs 2024-07-27T11:17:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:15 smithi077 ceph-mon[37153]: pgmap v8956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:17:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:15 smithi047 ceph-mon[36835]: pgmap v8956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:17:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:15 smithi029 ceph-mon[32610]: pgmap v8956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:17:17.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:17 smithi077 ceph-mon[37153]: pgmap v8957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:17:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:17 smithi047 ceph-mon[36835]: pgmap v8957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:17:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:17 smithi029 ceph-mon[32610]: pgmap v8957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:17:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16059.733986 secs 2024-07-27T11:17:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16059.733986 secs 2024-07-27T11:17:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16059.733986 secs 2024-07-27T11:17:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:19 smithi047 ceph-mon[36835]: pgmap v8958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:19 smithi077 ceph-mon[37153]: pgmap v8958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:19 smithi029 ceph-mon[32610]: pgmap v8958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:20.721+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:17:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:20.721+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:17:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:20.721+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:17:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:21 smithi077 ceph-mon[37153]: pgmap v8959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:21 smithi047 ceph-mon[36835]: pgmap v8959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:21.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:21 smithi029 ceph-mon[32610]: pgmap v8959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:23 smithi077 ceph-mon[37153]: pgmap v8960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16064.734115 secs 2024-07-27T11:17:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:23 smithi047 ceph-mon[36835]: pgmap v8960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16064.734115 secs 2024-07-27T11:17:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:23 smithi029 ceph-mon[32610]: pgmap v8960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16064.734115 secs 2024-07-27T11:17:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:25 smithi077 ceph-mon[37153]: pgmap v8961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:17:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:25 smithi047 ceph-mon[36835]: pgmap v8961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:17:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:25 smithi029 ceph-mon[32610]: pgmap v8961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:17:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:27 smithi077 ceph-mon[37153]: pgmap v8962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:27 smithi047 ceph-mon[36835]: pgmap v8962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:27.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:27 smithi029 ceph-mon[32610]: pgmap v8962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16069.734213 secs 2024-07-27T11:17:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16069.734213 secs 2024-07-27T11:17:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16069.734213 secs 2024-07-27T11:17:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:29 smithi077 ceph-mon[37153]: pgmap v8963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:29 smithi047 ceph-mon[36835]: pgmap v8963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:29.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:29 smithi029 ceph-mon[32610]: pgmap v8963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:31 smithi077 ceph-mon[37153]: pgmap v8964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:31 smithi047 ceph-mon[36835]: pgmap v8964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:31.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:31 smithi029 ceph-mon[32610]: pgmap v8964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:33 smithi077 ceph-mon[37153]: pgmap v8965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16074.734336 secs 2024-07-27T11:17:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:33 smithi047 ceph-mon[36835]: pgmap v8965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16074.734336 secs 2024-07-27T11:17:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:33 smithi029 ceph-mon[32610]: pgmap v8965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16074.734336 secs 2024-07-27T11:17:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:35 smithi077 ceph-mon[37153]: pgmap v8966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:17:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:35 smithi047 ceph-mon[36835]: pgmap v8966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:17:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:35 smithi029 ceph-mon[32610]: pgmap v8966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:17:35.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:35.733+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:17:35.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:35.733+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:17:35.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:35.733+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:17:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:37 smithi077 ceph-mon[37153]: pgmap v8967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:37 smithi047 ceph-mon[36835]: pgmap v8967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:37 smithi029 ceph-mon[32610]: pgmap v8967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:38.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16079.734455 secs 2024-07-27T11:17:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16079.734455 secs 2024-07-27T11:17:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16079.734455 secs 2024-07-27T11:17:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:39 smithi077 ceph-mon[37153]: pgmap v8968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:39.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:39 smithi029 ceph-mon[32610]: pgmap v8968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:39 smithi047 ceph-mon[36835]: pgmap v8968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:41 smithi077 ceph-mon[37153]: pgmap v8969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:41 smithi047 ceph-mon[36835]: pgmap v8969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:41 smithi029 ceph-mon[32610]: pgmap v8969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:43 smithi077 ceph-mon[37153]: pgmap v8970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16084.734576 secs 2024-07-27T11:17:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:43 smithi047 ceph-mon[36835]: pgmap v8970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16084.734576 secs 2024-07-27T11:17:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:43 smithi029 ceph-mon[32610]: pgmap v8970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16084.734576 secs 2024-07-27T11:17:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:45 smithi077 ceph-mon[37153]: pgmap v8971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:17:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:45 smithi047 ceph-mon[36835]: pgmap v8971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:17:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:45 smithi029 ceph-mon[32610]: pgmap v8971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:17:47.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:47 smithi077 ceph-mon[37153]: pgmap v8972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:47 smithi047 ceph-mon[36835]: pgmap v8972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:47 smithi029 ceph-mon[32610]: pgmap v8972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16089.734715 secs 2024-07-27T11:17:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16089.734715 secs 2024-07-27T11:17:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16089.734715 secs 2024-07-27T11:17:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:49 smithi077 ceph-mon[37153]: pgmap v8973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:49 smithi047 ceph-mon[36835]: pgmap v8973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:49 smithi029 ceph-mon[32610]: pgmap v8973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:50.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:50.748+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:17:50.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:50.748+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:17:50.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:17:50.748+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:17:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:51 smithi077 ceph-mon[37153]: pgmap v8974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:51 smithi047 ceph-mon[36835]: pgmap v8974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:51 smithi029 ceph-mon[32610]: pgmap v8974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:52 smithi077 ceph-mon[37153]: pgmap v8975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:52 smithi047 ceph-mon[36835]: pgmap v8975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:52.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:52 smithi029 ceph-mon[32610]: pgmap v8975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:17:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16094.734859 secs 2024-07-27T11:17:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16094.734859 secs 2024-07-27T11:17:53.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16094.734859 secs 2024-07-27T11:17:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:54 smithi077 ceph-mon[37153]: pgmap v8976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:17:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:54 smithi047 ceph-mon[36835]: pgmap v8976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:17:54.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:54 smithi029 ceph-mon[32610]: pgmap v8976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:17:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:57 smithi077 ceph-mon[37153]: pgmap v8977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:57 smithi047 ceph-mon[36835]: pgmap v8977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:57.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:57 smithi029 ceph-mon[32610]: pgmap v8977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:17:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16099.734924 secs 2024-07-27T11:17:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16099.734924 secs 2024-07-27T11:17:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16099.734924 secs 2024-07-27T11:17:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:17:59 smithi077 ceph-mon[37153]: pgmap v8978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:17:59 smithi047 ceph-mon[36835]: pgmap v8978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:17:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:17:59 smithi029 ceph-mon[32610]: pgmap v8978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:18:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:01 smithi077 ceph-mon[37153]: pgmap v8979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:01 smithi047 ceph-mon[36835]: pgmap v8979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:01.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:01 smithi029 ceph-mon[32610]: pgmap v8979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:03 smithi077 ceph-mon[37153]: pgmap v8980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:18:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16104.735008 secs 2024-07-27T11:18:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:03 smithi047 ceph-mon[36835]: pgmap v8980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:18:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16104.735008 secs 2024-07-27T11:18:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:03 smithi029 ceph-mon[32610]: pgmap v8980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:18:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16104.735008 secs 2024-07-27T11:18:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:05 smithi077 ceph-mon[37153]: pgmap v8981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:05 smithi047 ceph-mon[36835]: pgmap v8981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:05 smithi029 ceph-mon[32610]: pgmap v8981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:18:05.754+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:18:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:18:05.754+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:18:05.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:18:05.754+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:18:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:07 smithi077 ceph-mon[37153]: pgmap v8982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:18:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:18:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:18:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:18:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:07 smithi047 ceph-mon[36835]: pgmap v8982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:18:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:18:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:18:07.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:18:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:07 smithi029 ceph-mon[32610]: pgmap v8982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:18:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:18:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:18:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:18:08.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:08 smithi047 ceph-mon[36835]: pgmap v8983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T11:18:08.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:08 smithi047 ceph-mon[36835]: pgmap v8984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:18:08.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16109.735125 secs 2024-07-27T11:18:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:08 smithi077 ceph-mon[37153]: pgmap v8983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T11:18:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:08 smithi077 ceph-mon[37153]: pgmap v8984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:18:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16109.735125 secs 2024-07-27T11:18:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:08 smithi029 ceph-mon[32610]: pgmap v8983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T11:18:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:08 smithi029 ceph-mon[32610]: pgmap v8984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:18:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16109.735125 secs 2024-07-27T11:18:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:09 smithi029 ceph-mon[32610]: pgmap v8985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:18:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:09 smithi077 ceph-mon[37153]: pgmap v8985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:18:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:09 smithi047 ceph-mon[36835]: pgmap v8985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:18:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:11 smithi077 ceph-mon[37153]: pgmap v8986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:18:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:11 smithi047 ceph-mon[36835]: pgmap v8986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:18:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:11 smithi029 ceph-mon[32610]: pgmap v8986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:18:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16114.735256 secs 2024-07-27T11:18:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16114.735256 secs 2024-07-27T11:18:12.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16114.735256 secs 2024-07-27T11:18:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:13 smithi077 ceph-mon[37153]: pgmap v8987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:18:13.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:13 smithi029 ceph-mon[32610]: pgmap v8987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:18:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:13 smithi047 ceph-mon[36835]: pgmap v8987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:18:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:15 smithi077 ceph-mon[37153]: pgmap v8988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:18:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:15 smithi047 ceph-mon[36835]: pgmap v8988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:18:15.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:15 smithi029 ceph-mon[32610]: pgmap v8988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:18:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:17 smithi077 ceph-mon[37153]: pgmap v8989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T11:18:17.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16119.735376 secs 2024-07-27T11:18:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:17 smithi047 ceph-mon[36835]: pgmap v8989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T11:18:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16119.735376 secs 2024-07-27T11:18:17.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:17 smithi029 ceph-mon[32610]: pgmap v8989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T11:18:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16119.735376 secs 2024-07-27T11:18:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:19 smithi077 ceph-mon[37153]: pgmap v8990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:18:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:19 smithi047 ceph-mon[36835]: pgmap v8990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:18:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:19 smithi029 ceph-mon[32610]: pgmap v8990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:18:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:18:20.770+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:18:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:18:20.770+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:18:20.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:18:20.770+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:18:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:21 smithi047 ceph-mon[36835]: pgmap v8991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:18:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:21 smithi029 ceph-mon[32610]: pgmap v8991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:18:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:21 smithi077 ceph-mon[37153]: pgmap v8991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:18:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16124.735475 secs 2024-07-27T11:18:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16124.735475 secs 2024-07-27T11:18:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16124.735475 secs 2024-07-27T11:18:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:23 smithi047 ceph-mon[36835]: pgmap v8992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:18:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:23 smithi029 ceph-mon[32610]: pgmap v8992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:18:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:23 smithi077 ceph-mon[37153]: pgmap v8992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:18:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:25 smithi047 ceph-mon[36835]: pgmap v8993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:18:25.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:25 smithi029 ceph-mon[32610]: pgmap v8993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:18:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:25 smithi077 ceph-mon[37153]: pgmap v8993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:18:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:27 smithi047 ceph-mon[36835]: pgmap v8994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:18:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16129.735597 secs 2024-07-27T11:18:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:27 smithi029 ceph-mon[32610]: pgmap v8994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:18:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16129.735597 secs 2024-07-27T11:18:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:27 smithi077 ceph-mon[37153]: pgmap v8994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:18:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16129.735597 secs 2024-07-27T11:18:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:29 smithi047 ceph-mon[36835]: pgmap v8995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:29.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:29 smithi029 ceph-mon[32610]: pgmap v8995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:29 smithi077 ceph-mon[37153]: pgmap v8995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:31 smithi047 ceph-mon[36835]: pgmap v8996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:31 smithi029 ceph-mon[32610]: pgmap v8996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:31 smithi077 ceph-mon[37153]: pgmap v8996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16134.735660 secs 2024-07-27T11:18:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16134.735660 secs 2024-07-27T11:18:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16134.735660 secs 2024-07-27T11:18:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:33 smithi047 ceph-mon[36835]: pgmap v8997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:18:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:33 smithi029 ceph-mon[32610]: pgmap v8997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:18:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:33 smithi077 ceph-mon[37153]: pgmap v8997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:18:35.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:18:35.786+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:18:35.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:18:35.786+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:18:35.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:18:35.786+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:18:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:35 smithi047 ceph-mon[36835]: pgmap v8998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:18:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:35 smithi029 ceph-mon[32610]: pgmap v8998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:18:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:35 smithi077 ceph-mon[37153]: pgmap v8998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:18:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:37 smithi047 ceph-mon[36835]: pgmap v8999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:18:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16139.735781 secs 2024-07-27T11:18:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:37 smithi029 ceph-mon[32610]: pgmap v8999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:18:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16139.735781 secs 2024-07-27T11:18:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:37 smithi077 ceph-mon[37153]: pgmap v8999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:18:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16139.735781 secs 2024-07-27T11:18:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:39 smithi047 ceph-mon[36835]: pgmap v9000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:39.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:39 smithi029 ceph-mon[32610]: pgmap v9000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:39 smithi077 ceph-mon[37153]: pgmap v9000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:41 smithi047 ceph-mon[36835]: pgmap v9001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:41 smithi029 ceph-mon[32610]: pgmap v9001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:41 smithi077 ceph-mon[37153]: pgmap v9001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16144.735903 secs 2024-07-27T11:18:42.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16144.735903 secs 2024-07-27T11:18:43.007 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16144.735903 secs 2024-07-27T11:18:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:43 smithi047 ceph-mon[36835]: pgmap v9002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:18:43.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:43 smithi029 ceph-mon[32610]: pgmap v9002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:18:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:43 smithi077 ceph-mon[37153]: pgmap v9002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:18:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:45 smithi047 ceph-mon[36835]: pgmap v9003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:18:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:45 smithi029 ceph-mon[32610]: pgmap v9003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:18:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:45 smithi077 ceph-mon[37153]: pgmap v9003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:18:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:47 smithi047 ceph-mon[36835]: pgmap v9004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:18:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16149.736022 secs 2024-07-27T11:18:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:47 smithi029 ceph-mon[32610]: pgmap v9004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:18:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16149.736022 secs 2024-07-27T11:18:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:47 smithi077 ceph-mon[37153]: pgmap v9004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:18:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16149.736022 secs 2024-07-27T11:18:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:49 smithi047 ceph-mon[36835]: pgmap v9005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:49 smithi029 ceph-mon[32610]: pgmap v9005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:49 smithi077 ceph-mon[37153]: pgmap v9005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:50.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:18:50.790+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:18:50.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:18:50.790+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:18:50.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:18:50.790+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:18:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:51 smithi047 ceph-mon[36835]: pgmap v9006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:51.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:51 smithi029 ceph-mon[32610]: pgmap v9006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:51 smithi077 ceph-mon[37153]: pgmap v9006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:18:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16154.736111 secs 2024-07-27T11:18:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16154.736111 secs 2024-07-27T11:18:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16154.736111 secs 2024-07-27T11:18:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:53 smithi047 ceph-mon[36835]: pgmap v9007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:18:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:53 smithi029 ceph-mon[32610]: pgmap v9007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:18:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:53 smithi077 ceph-mon[37153]: pgmap v9007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:18:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:55 smithi047 ceph-mon[36835]: pgmap v9008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:18:55.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:55 smithi029 ceph-mon[32610]: pgmap v9008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:18:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:55 smithi077 ceph-mon[37153]: pgmap v9008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:18:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:57 smithi047 ceph-mon[36835]: pgmap v9009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:18:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16159.736233 secs 2024-07-27T11:18:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:57 smithi029 ceph-mon[32610]: pgmap v9009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:18:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16159.736233 secs 2024-07-27T11:18:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:57 smithi077 ceph-mon[37153]: pgmap v9009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:18:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16159.736233 secs 2024-07-27T11:18:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:18:59 smithi047 ceph-mon[36835]: pgmap v9010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:18:59.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:18:59 smithi029 ceph-mon[32610]: pgmap v9010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:18:59 smithi077 ceph-mon[37153]: pgmap v9010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:01 smithi047 ceph-mon[36835]: pgmap v9011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:01.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:01 smithi029 ceph-mon[32610]: pgmap v9011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:01 smithi077 ceph-mon[37153]: pgmap v9011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16164.736356 secs 2024-07-27T11:19:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16164.736356 secs 2024-07-27T11:19:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16164.736356 secs 2024-07-27T11:19:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:03 smithi029 ceph-mon[32610]: pgmap v9012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:19:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:03 smithi077 ceph-mon[37153]: pgmap v9012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:19:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:03 smithi047 ceph-mon[36835]: pgmap v9012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:19:05.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:05.801+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:19:05.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:05.801+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:19:05.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:05.801+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:19:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:05 smithi029 ceph-mon[32610]: pgmap v9013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:19:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:05 smithi077 ceph-mon[37153]: pgmap v9013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:19:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:05 smithi047 ceph-mon[36835]: pgmap v9013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:19:06.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:19:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:19:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:19:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:07 smithi077 ceph-mon[37153]: pgmap v9014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:19:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16169.736501 secs 2024-07-27T11:19:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:19:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:19:08.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:07 smithi077 ceph-mon[37153]: pgmap v9015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:19:08.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:07 smithi077 ceph-mon[37153]: pgmap v9016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:19:08.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:19:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:07 smithi047 ceph-mon[36835]: pgmap v9014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:19:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16169.736501 secs 2024-07-27T11:19:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:19:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:19:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:07 smithi047 ceph-mon[36835]: pgmap v9015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:19:08.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:07 smithi047 ceph-mon[36835]: pgmap v9016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:19:08.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:19:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:07 smithi029 ceph-mon[32610]: pgmap v9014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:19:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16169.736501 secs 2024-07-27T11:19:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:19:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:19:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:07 smithi029 ceph-mon[32610]: pgmap v9015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:19:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:07 smithi029 ceph-mon[32610]: pgmap v9016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:19:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:19:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:09 smithi047 ceph-mon[36835]: pgmap v9017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:19:09.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:09 smithi029 ceph-mon[32610]: pgmap v9017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:19:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:09 smithi077 ceph-mon[37153]: pgmap v9017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:19:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:11 smithi047 ceph-mon[36835]: pgmap v9018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:19:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:11 smithi029 ceph-mon[32610]: pgmap v9018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:19:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:11 smithi077 ceph-mon[37153]: pgmap v9018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:19:12.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16174.736624 secs 2024-07-27T11:19:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16174.736624 secs 2024-07-27T11:19:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16174.736624 secs 2024-07-27T11:19:13.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:13 smithi077 ceph-mon[37153]: pgmap v9019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:19:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:13 smithi047 ceph-mon[36835]: pgmap v9019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:19:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:13 smithi029 ceph-mon[32610]: pgmap v9019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:19:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:15 smithi047 ceph-mon[36835]: pgmap v9020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:19:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:15 smithi029 ceph-mon[32610]: pgmap v9020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:19:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:15 smithi077 ceph-mon[37153]: pgmap v9020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:19:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:17 smithi047 ceph-mon[36835]: pgmap v9021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:19:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:17 smithi029 ceph-mon[32610]: pgmap v9021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:19:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:17 smithi077 ceph-mon[37153]: pgmap v9021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:19:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16179.736744 secs 2024-07-27T11:19:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16179.736744 secs 2024-07-27T11:19:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16179.736744 secs 2024-07-27T11:19:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:19 smithi047 ceph-mon[36835]: pgmap v9022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:19 smithi029 ceph-mon[32610]: pgmap v9022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:19 smithi077 ceph-mon[37153]: pgmap v9022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:20.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:20.805+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:19:20.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:20.805+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:19:20.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:20.805+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:19:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:21 smithi047 ceph-mon[36835]: pgmap v9023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:21 smithi029 ceph-mon[32610]: pgmap v9023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:21 smithi077 ceph-mon[37153]: pgmap v9023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16184.736865 secs 2024-07-27T11:19:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16184.736865 secs 2024-07-27T11:19:22.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16184.736865 secs 2024-07-27T11:19:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:23 smithi077 ceph-mon[37153]: pgmap v9024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:19:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:23 smithi047 ceph-mon[36835]: pgmap v9024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:19:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:23 smithi029 ceph-mon[32610]: pgmap v9024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:19:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:25 smithi047 ceph-mon[36835]: pgmap v9025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:25.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:25 smithi029 ceph-mon[32610]: pgmap v9025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:25 smithi077 ceph-mon[37153]: pgmap v9025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:27 smithi047 ceph-mon[36835]: pgmap v9026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:19:27.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:27 smithi029 ceph-mon[32610]: pgmap v9026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:19:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:27 smithi077 ceph-mon[37153]: pgmap v9026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:19:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16189.736956 secs 2024-07-27T11:19:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16189.736956 secs 2024-07-27T11:19:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16189.736956 secs 2024-07-27T11:19:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:29 smithi047 ceph-mon[36835]: pgmap v9027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:29.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:29 smithi029 ceph-mon[32610]: pgmap v9027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:29 smithi077 ceph-mon[37153]: pgmap v9027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:31 smithi047 ceph-mon[36835]: pgmap v9028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:31.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:31 smithi029 ceph-mon[32610]: pgmap v9028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:31 smithi077 ceph-mon[37153]: pgmap v9028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16194.737054 secs 2024-07-27T11:19:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16194.737054 secs 2024-07-27T11:19:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16194.737054 secs 2024-07-27T11:19:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:33 smithi077 ceph-mon[37153]: pgmap v9029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:19:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:33 smithi047 ceph-mon[36835]: pgmap v9029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:19:33.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:33 smithi029 ceph-mon[32610]: pgmap v9029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:19:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:35 smithi047 ceph-mon[36835]: pgmap v9030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:35 smithi029 ceph-mon[32610]: pgmap v9030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:35 smithi077 ceph-mon[37153]: pgmap v9030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:35.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:35.821+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:19:35.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:35.821+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:19:35.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:35.821+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:19:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:37 smithi047 ceph-mon[36835]: pgmap v9031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:19:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:37 smithi029 ceph-mon[32610]: pgmap v9031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:19:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:37 smithi077 ceph-mon[37153]: pgmap v9031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:19:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16199.737192 secs 2024-07-27T11:19:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16199.737192 secs 2024-07-27T11:19:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16199.737192 secs 2024-07-27T11:19:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:40 smithi047 ceph-mon[36835]: pgmap v9032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:40 smithi029 ceph-mon[32610]: pgmap v9032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:40 smithi077 ceph-mon[37153]: pgmap v9032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:42 smithi047 ceph-mon[36835]: pgmap v9033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:42 smithi029 ceph-mon[32610]: pgmap v9033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:42 smithi077 ceph-mon[37153]: pgmap v9033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16204.737341 secs 2024-07-27T11:19:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16204.737341 secs 2024-07-27T11:19:43.507 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16204.737341 secs 2024-07-27T11:19:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:44 smithi047 ceph-mon[36835]: pgmap v9034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:19:44.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:44 smithi029 ceph-mon[32610]: pgmap v9034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:19:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:44 smithi077 ceph-mon[37153]: pgmap v9034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:19:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:46 smithi047 ceph-mon[36835]: pgmap v9035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:46.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:46 smithi029 ceph-mon[32610]: pgmap v9035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:46 smithi077 ceph-mon[37153]: pgmap v9035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16209.737484 secs 2024-07-27T11:19:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:48 smithi047 ceph-mon[36835]: pgmap v9036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:19:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16209.737484 secs 2024-07-27T11:19:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:48 smithi029 ceph-mon[32610]: pgmap v9036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:19:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16209.737484 secs 2024-07-27T11:19:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:48 smithi077 ceph-mon[37153]: pgmap v9036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:19:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:50 smithi047 ceph-mon[36835]: pgmap v9037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:50.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:50 smithi029 ceph-mon[32610]: pgmap v9037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:50 smithi077 ceph-mon[37153]: pgmap v9037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:19:50.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:50.822+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:19:50.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:50.822+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:19:50.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:50.822+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:19:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:52 smithi047 ceph-mon[36835]: pgmap v9038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:52 smithi029 ceph-mon[32610]: pgmap v9038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:52 smithi077 ceph-mon[37153]: pgmap v9038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16214.737605 secs 2024-07-27T11:19:53.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16214.737605 secs 2024-07-27T11:19:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16214.737605 secs 2024-07-27T11:19:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:54 smithi047 ceph-mon[36835]: pgmap v9039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:19:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:54 smithi029 ceph-mon[32610]: pgmap v9039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:19:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:54 smithi077 ceph-mon[37153]: pgmap v9039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:19:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:56 smithi029 ceph-mon[32610]: pgmap v9040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:56 smithi077 ceph-mon[37153]: pgmap v9040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:56 smithi047 ceph-mon[36835]: pgmap v9040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:19:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:57.668+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T11:19:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:57.668+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T11:19:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:57.668+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7eff40004a10 tx=0x7eff4002b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T11:19:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:57.668+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7eff40004a10 tx=0x7eff4002b040 comp rx=0 tx=0).stop 2024-07-27T11:19:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:57.668+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T11:19:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:57.668+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T11:19:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:57.668+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T11:19:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:57.668+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T11:19:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:57.668+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T11:19:57.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:19:57.669+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7eff40004800 tx=0x7eff4001e040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T11:19:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16219.737723 secs 2024-07-27T11:19:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:19:58 smithi029 ceph-mon[32610]: pgmap v9041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:19:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16219.737723 secs 2024-07-27T11:19:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:19:58 smithi077 ceph-mon[37153]: pgmap v9041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:19:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16219.737723 secs 2024-07-27T11:19:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:19:58 smithi047 ceph-mon[36835]: pgmap v9041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:00.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:00.101+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:20:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:00.101+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:20:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:00.101+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:20:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:00.101+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:20:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:00 smithi029 ceph-mon[32610]: pgmap v9042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:20:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:00 smithi077 ceph-mon[37153]: pgmap v9042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:20:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:00 smithi047 ceph-mon[36835]: pgmap v9042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:20:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:02 smithi029 ceph-mon[32610]: pgmap v9043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16224.737855 secs 2024-07-27T11:20:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:02 smithi077 ceph-mon[37153]: pgmap v9043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:02.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16224.737855 secs 2024-07-27T11:20:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:02 smithi047 ceph-mon[36835]: pgmap v9043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16224.737855 secs 2024-07-27T11:20:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:04 smithi029 ceph-mon[32610]: pgmap v9044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:04 smithi077 ceph-mon[37153]: pgmap v9044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:04 smithi047 ceph-mon[36835]: pgmap v9044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:05.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:05.830+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:20:05.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:05.830+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:20:05.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:05.830+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:20:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:06 smithi029 ceph-mon[32610]: pgmap v9045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:06 smithi077 ceph-mon[37153]: pgmap v9045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:06 smithi047 ceph-mon[36835]: pgmap v9045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:07.489 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16229.737916 secs 2024-07-27T11:20:07.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:07 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:20:07.540 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16229.737916 secs 2024-07-27T11:20:07.540 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:07 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:20:07.551 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16229.737916 secs 2024-07-27T11:20:07.552 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:07 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:20:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:08 smithi077 ceph-mon[37153]: pgmap v9046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:08 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:08 smithi047 ceph-mon[36835]: pgmap v9046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:08 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:08.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:08 smithi029 ceph-mon[32610]: pgmap v9046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:08 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:20:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:20:10.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:20:10.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:20:10.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:20:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:20:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:20:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:11 smithi077 ceph-mon[37153]: pgmap v9047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:11 smithi077 ceph-mon[37153]: pgmap v9048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:11 smithi077 ceph-mon[37153]: pgmap v9049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:11 smithi047 ceph-mon[36835]: pgmap v9047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:11 smithi047 ceph-mon[36835]: pgmap v9048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:11 smithi047 ceph-mon[36835]: pgmap v9049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:11 smithi029 ceph-mon[32610]: pgmap v9047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:11 smithi029 ceph-mon[32610]: pgmap v9048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:11.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:11 smithi029 ceph-mon[32610]: pgmap v9049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:13 smithi077 ceph-mon[37153]: pgmap v9050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:20:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16234.738041 secs 2024-07-27T11:20:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:13 smithi047 ceph-mon[36835]: pgmap v9050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:20:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16234.738041 secs 2024-07-27T11:20:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:13 smithi029 ceph-mon[32610]: pgmap v9050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:20:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16234.738041 secs 2024-07-27T11:20:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:15 smithi077 ceph-mon[37153]: pgmap v9051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:15 smithi047 ceph-mon[36835]: pgmap v9051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:15.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:15 smithi029 ceph-mon[32610]: pgmap v9051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:17 smithi077 ceph-mon[37153]: pgmap v9052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:20:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:17 smithi047 ceph-mon[36835]: pgmap v9052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:20:17.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:17 smithi029 ceph-mon[32610]: pgmap v9052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:20:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16239.738169 secs 2024-07-27T11:20:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16239.738169 secs 2024-07-27T11:20:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16239.738169 secs 2024-07-27T11:20:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:19 smithi077 ceph-mon[37153]: pgmap v9053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:20:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:19 smithi047 ceph-mon[36835]: pgmap v9053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:20:19.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:19 smithi029 ceph-mon[32610]: pgmap v9053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:20:20.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:20.845+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:20:20.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:20.845+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:20:20.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:20.845+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:20:21.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:21 smithi029 ceph-mon[32610]: pgmap v9054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:20:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:21 smithi077 ceph-mon[37153]: pgmap v9054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:20:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:21 smithi047 ceph-mon[36835]: pgmap v9054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:20:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:23 smithi077 ceph-mon[37153]: pgmap v9055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16244.738296 secs 2024-07-27T11:20:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:23 smithi047 ceph-mon[36835]: pgmap v9055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16244.738296 secs 2024-07-27T11:20:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:23 smithi029 ceph-mon[32610]: pgmap v9055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16244.738296 secs 2024-07-27T11:20:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:25 smithi077 ceph-mon[37153]: pgmap v9056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:25 smithi047 ceph-mon[36835]: pgmap v9056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:25.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:25 smithi029 ceph-mon[32610]: pgmap v9056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:27 smithi077 ceph-mon[37153]: pgmap v9057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:27 smithi047 ceph-mon[36835]: pgmap v9057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:27.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:27 smithi029 ceph-mon[32610]: pgmap v9057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16249.738416 secs 2024-07-27T11:20:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16249.738416 secs 2024-07-27T11:20:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16249.738416 secs 2024-07-27T11:20:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:29 smithi047 ceph-mon[36835]: pgmap v9058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:29.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:29 smithi029 ceph-mon[32610]: pgmap v9058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:29 smithi077 ceph-mon[37153]: pgmap v9058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:30 smithi047 ceph-mon[36835]: pgmap v9059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:20:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:30 smithi029 ceph-mon[32610]: pgmap v9059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:20:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:30 smithi077 ceph-mon[37153]: pgmap v9059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:20:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:32 smithi047 ceph-mon[36835]: pgmap v9060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16254.738515 secs 2024-07-27T11:20:33.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:32 smithi029 ceph-mon[32610]: pgmap v9060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16254.738515 secs 2024-07-27T11:20:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:32 smithi077 ceph-mon[37153]: pgmap v9060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16254.738515 secs 2024-07-27T11:20:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:34 smithi047 ceph-mon[36835]: pgmap v9061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:34 smithi029 ceph-mon[32610]: pgmap v9061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:34 smithi077 ceph-mon[37153]: pgmap v9061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:35.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:35.857+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:20:35.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:35.857+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:20:35.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:35.857+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:20:37.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:36 smithi047 ceph-mon[36835]: pgmap v9062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:37.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:36 smithi029 ceph-mon[32610]: pgmap v9062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:36 smithi077 ceph-mon[37153]: pgmap v9062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16259.738634 secs 2024-07-27T11:20:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16259.738634 secs 2024-07-27T11:20:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16259.738634 secs 2024-07-27T11:20:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:38 smithi047 ceph-mon[36835]: pgmap v9063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:38 smithi029 ceph-mon[32610]: pgmap v9063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:38 smithi077 ceph-mon[37153]: pgmap v9063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:40 smithi047 ceph-mon[36835]: pgmap v9064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:20:41.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:40 smithi029 ceph-mon[32610]: pgmap v9064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:20:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:40 smithi077 ceph-mon[37153]: pgmap v9064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:20:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:42 smithi047 ceph-mon[36835]: pgmap v9065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16264.738774 secs 2024-07-27T11:20:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:42 smithi029 ceph-mon[32610]: pgmap v9065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16264.738774 secs 2024-07-27T11:20:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:42 smithi077 ceph-mon[37153]: pgmap v9065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16264.738774 secs 2024-07-27T11:20:45.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:44 smithi047 ceph-mon[36835]: pgmap v9066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:44 smithi029 ceph-mon[32610]: pgmap v9066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:44 smithi077 ceph-mon[37153]: pgmap v9066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:46 smithi047 ceph-mon[36835]: pgmap v9067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:46 smithi029 ceph-mon[32610]: pgmap v9067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:46 smithi077 ceph-mon[37153]: pgmap v9067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16269.738894 secs 2024-07-27T11:20:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16269.738894 secs 2024-07-27T11:20:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16269.738894 secs 2024-07-27T11:20:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:48 smithi047 ceph-mon[36835]: pgmap v9068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:49.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:48 smithi029 ceph-mon[32610]: pgmap v9068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:48 smithi077 ceph-mon[37153]: pgmap v9068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:50.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:50.872+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:20:50.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:50.872+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:20:50.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:20:50.872+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:20:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:50 smithi047 ceph-mon[36835]: pgmap v9069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:51.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:50 smithi029 ceph-mon[32610]: pgmap v9069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:50 smithi077 ceph-mon[37153]: pgmap v9069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:52 smithi047 ceph-mon[36835]: pgmap v9070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16274.739019 secs 2024-07-27T11:20:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:52 smithi029 ceph-mon[32610]: pgmap v9070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16274.739019 secs 2024-07-27T11:20:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:52 smithi077 ceph-mon[37153]: pgmap v9070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:20:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16274.739019 secs 2024-07-27T11:20:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:54 smithi029 ceph-mon[32610]: pgmap v9071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:54 smithi077 ceph-mon[37153]: pgmap v9071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:54 smithi047 ceph-mon[36835]: pgmap v9071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:20:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:56 smithi029 ceph-mon[32610]: pgmap v9072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:56 smithi077 ceph-mon[37153]: pgmap v9072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:56 smithi047 ceph-mon[36835]: pgmap v9072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:20:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16279.739136 secs 2024-07-27T11:20:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16279.739136 secs 2024-07-27T11:20:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16279.739136 secs 2024-07-27T11:20:59.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:20:58 smithi029 ceph-mon[32610]: pgmap v9073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:20:58 smithi077 ceph-mon[37153]: pgmap v9073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:20:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:20:58 smithi047 ceph-mon[36835]: pgmap v9073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:21:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:00 smithi029 ceph-mon[32610]: pgmap v9074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:00 smithi077 ceph-mon[37153]: pgmap v9074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:00 smithi047 ceph-mon[36835]: pgmap v9074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:02 smithi077 ceph-mon[37153]: pgmap v9075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:21:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16284.739238 secs 2024-07-27T11:21:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:02 smithi047 ceph-mon[36835]: pgmap v9075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:21:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16284.739238 secs 2024-07-27T11:21:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:02 smithi029 ceph-mon[32610]: pgmap v9075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:21:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16284.739238 secs 2024-07-27T11:21:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:04 smithi077 ceph-mon[37153]: pgmap v9076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:21:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:04 smithi047 ceph-mon[36835]: pgmap v9076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:21:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:04 smithi029 ceph-mon[32610]: pgmap v9076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:21:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:21:05.884+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:21:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:21:05.884+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:21:05.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:21:05.884+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:21:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:06 smithi077 ceph-mon[37153]: pgmap v9077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:06 smithi047 ceph-mon[36835]: pgmap v9077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:06 smithi029 ceph-mon[32610]: pgmap v9077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16289.739357 secs 2024-07-27T11:21:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16289.739357 secs 2024-07-27T11:21:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16289.739357 secs 2024-07-27T11:21:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:08 smithi077 ceph-mon[37153]: pgmap v9078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:21:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:08 smithi047 ceph-mon[36835]: pgmap v9078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:21:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:08 smithi029 ceph-mon[32610]: pgmap v9078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:21:10.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:21:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:21:10.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:21:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:10 smithi077 ceph-mon[37153]: pgmap v9079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:21:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:21:11.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:10 smithi047 ceph-mon[36835]: pgmap v9079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:21:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:21:11.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:10 smithi029 ceph-mon[32610]: pgmap v9079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:21:11.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:21:11.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:21:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:11 smithi077 ceph-mon[37153]: pgmap v9080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:21:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:11 smithi077 ceph-mon[37153]: pgmap v9081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:21:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:11 smithi047 ceph-mon[36835]: pgmap v9080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:21:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:11 smithi047 ceph-mon[36835]: pgmap v9081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:21:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:11 smithi029 ceph-mon[32610]: pgmap v9080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:21:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:11 smithi029 ceph-mon[32610]: pgmap v9081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:21:13.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16294.739420 secs 2024-07-27T11:21:13.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16294.739420 secs 2024-07-27T11:21:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16294.739420 secs 2024-07-27T11:21:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:13 smithi077 ceph-mon[37153]: pgmap v9082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T11:21:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:13 smithi047 ceph-mon[36835]: pgmap v9082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T11:21:14.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:13 smithi029 ceph-mon[32610]: pgmap v9082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T11:21:16.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:15 smithi029 ceph-mon[32610]: pgmap v9083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:21:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:15 smithi077 ceph-mon[37153]: pgmap v9083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:21:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:15 smithi047 ceph-mon[36835]: pgmap v9083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:21:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:18 smithi077 ceph-mon[37153]: pgmap v9084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:21:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16299.739546 secs 2024-07-27T11:21:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:18 smithi047 ceph-mon[36835]: pgmap v9084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:21:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16299.739546 secs 2024-07-27T11:21:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:18 smithi029 ceph-mon[32610]: pgmap v9084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:21:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16299.739546 secs 2024-07-27T11:21:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:20 smithi077 ceph-mon[37153]: pgmap v9085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:21:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:20 smithi047 ceph-mon[36835]: pgmap v9085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:21:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:20 smithi029 ceph-mon[32610]: pgmap v9085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:21:20.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:21:20.885+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:21:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:21:20.885+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:21:20.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:21:20.885+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:21:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:22 smithi077 ceph-mon[37153]: pgmap v9086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:21:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:22 smithi047 ceph-mon[36835]: pgmap v9086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:21:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:22 smithi029 ceph-mon[32610]: pgmap v9086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:21:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16304.739688 secs 2024-07-27T11:21:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16304.739688 secs 2024-07-27T11:21:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16304.739688 secs 2024-07-27T11:21:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:24 smithi077 ceph-mon[37153]: pgmap v9087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:21:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:24 smithi047 ceph-mon[36835]: pgmap v9087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:21:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:24 smithi029 ceph-mon[32610]: pgmap v9087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:21:26.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:26 smithi077 ceph-mon[37153]: pgmap v9088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:21:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:26 smithi047 ceph-mon[36835]: pgmap v9088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:21:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:26 smithi029 ceph-mon[32610]: pgmap v9088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:21:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:28 smithi077 ceph-mon[37153]: pgmap v9089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:21:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16309.739781 secs 2024-07-27T11:21:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:28 smithi047 ceph-mon[36835]: pgmap v9089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:21:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16309.739781 secs 2024-07-27T11:21:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:28 smithi029 ceph-mon[32610]: pgmap v9089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:21:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16309.739781 secs 2024-07-27T11:21:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:30 smithi077 ceph-mon[37153]: pgmap v9090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:21:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:30 smithi047 ceph-mon[36835]: pgmap v9090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:21:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:30 smithi029 ceph-mon[32610]: pgmap v9090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:21:32.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:32 smithi029 ceph-mon[32610]: pgmap v9091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:32 smithi077 ceph-mon[37153]: pgmap v9091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:32 smithi047 ceph-mon[36835]: pgmap v9091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16314.739908 secs 2024-07-27T11:21:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16314.739908 secs 2024-07-27T11:21:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16314.739908 secs 2024-07-27T11:21:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:34 smithi077 ceph-mon[37153]: pgmap v9092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:21:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:34 smithi047 ceph-mon[36835]: pgmap v9092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:21:34.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:34 smithi029 ceph-mon[32610]: pgmap v9092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:21:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:21:35.901+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:21:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:21:35.901+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:21:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:21:35.901+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:21:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:36 smithi029 ceph-mon[32610]: pgmap v9093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:21:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:36 smithi077 ceph-mon[37153]: pgmap v9093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:21:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:36 smithi047 ceph-mon[36835]: pgmap v9093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:21:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:38 smithi077 ceph-mon[37153]: pgmap v9094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:21:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16319.740041 secs 2024-07-27T11:21:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:38 smithi047 ceph-mon[36835]: pgmap v9094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:21:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16319.740041 secs 2024-07-27T11:21:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:38 smithi029 ceph-mon[32610]: pgmap v9094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:21:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16319.740041 secs 2024-07-27T11:21:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:40 smithi077 ceph-mon[37153]: pgmap v9095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:21:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:40 smithi047 ceph-mon[36835]: pgmap v9095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:21:40.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:40 smithi029 ceph-mon[32610]: pgmap v9095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:21:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:42 smithi077 ceph-mon[37153]: pgmap v9096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:42 smithi047 ceph-mon[36835]: pgmap v9096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:42 smithi029 ceph-mon[32610]: pgmap v9096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16324.740164 secs 2024-07-27T11:21:43.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16324.740164 secs 2024-07-27T11:21:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16324.740164 secs 2024-07-27T11:21:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:44 smithi077 ceph-mon[37153]: pgmap v9097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:21:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:44 smithi047 ceph-mon[36835]: pgmap v9097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:21:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:44 smithi029 ceph-mon[32610]: pgmap v9097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:21:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:46 smithi077 ceph-mon[37153]: pgmap v9098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:21:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:46 smithi047 ceph-mon[36835]: pgmap v9098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:21:46.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:46 smithi029 ceph-mon[32610]: pgmap v9098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:21:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:48 smithi077 ceph-mon[37153]: pgmap v9099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:21:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16329.740285 secs 2024-07-27T11:21:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:48 smithi047 ceph-mon[36835]: pgmap v9099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:21:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16329.740285 secs 2024-07-27T11:21:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:48 smithi029 ceph-mon[32610]: pgmap v9099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:21:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16329.740285 secs 2024-07-27T11:21:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:50 smithi077 ceph-mon[37153]: pgmap v9100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:21:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:50 smithi047 ceph-mon[36835]: pgmap v9100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:21:50.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:50 smithi029 ceph-mon[32610]: pgmap v9100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:21:50.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:21:50.910+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:21:50.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:21:50.910+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:21:50.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:21:50.910+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:21:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:52 smithi077 ceph-mon[37153]: pgmap v9101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:52 smithi047 ceph-mon[36835]: pgmap v9101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:52.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:52 smithi029 ceph-mon[32610]: pgmap v9101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:21:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16334.740443 secs 2024-07-27T11:21:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16334.740443 secs 2024-07-27T11:21:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16334.740443 secs 2024-07-27T11:21:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:54 smithi077 ceph-mon[37153]: pgmap v9102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:21:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:54 smithi047 ceph-mon[36835]: pgmap v9102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:21:54.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:54 smithi029 ceph-mon[32610]: pgmap v9102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:21:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:56 smithi077 ceph-mon[37153]: pgmap v9103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:21:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:56 smithi047 ceph-mon[36835]: pgmap v9103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:21:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:56 smithi029 ceph-mon[32610]: pgmap v9103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:21:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:58 smithi077 ceph-mon[37153]: pgmap v9104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:21:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:21:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16339.740572 secs 2024-07-27T11:21:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:58 smithi047 ceph-mon[36835]: pgmap v9104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:21:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:21:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16339.740572 secs 2024-07-27T11:21:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:58 smithi029 ceph-mon[32610]: pgmap v9104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:21:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:21:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16339.740572 secs 2024-07-27T11:22:00.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:00 smithi077 ceph-mon[37153]: pgmap v9105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:00 smithi047 ceph-mon[36835]: pgmap v9105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:00 smithi029 ceph-mon[32610]: pgmap v9105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:02 smithi077 ceph-mon[37153]: pgmap v9106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:22:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:02 smithi047 ceph-mon[36835]: pgmap v9106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:22:02.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:02 smithi029 ceph-mon[32610]: pgmap v9106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:22:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16344.740667 secs 2024-07-27T11:22:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16344.740667 secs 2024-07-27T11:22:03.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16344.740667 secs 2024-07-27T11:22:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:04 smithi077 ceph-mon[37153]: pgmap v9107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:22:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:04 smithi047 ceph-mon[36835]: pgmap v9107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:22:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:04 smithi029 ceph-mon[32610]: pgmap v9107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:22:05.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:05.912+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:22:05.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:05.912+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:22:05.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:05.912+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:22:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:06 smithi029 ceph-mon[32610]: pgmap v9108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:22:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:06 smithi077 ceph-mon[37153]: pgmap v9108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:22:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:06 smithi047 ceph-mon[36835]: pgmap v9108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:22:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:08 smithi077 ceph-mon[37153]: pgmap v9109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16349.740763 secs 2024-07-27T11:22:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:08 smithi047 ceph-mon[36835]: pgmap v9109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16349.740763 secs 2024-07-27T11:22:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:08 smithi029 ceph-mon[32610]: pgmap v9109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16349.740763 secs 2024-07-27T11:22:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:09 smithi047 ceph-mon[36835]: pgmap v9110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:09 smithi029 ceph-mon[32610]: pgmap v9110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:09 smithi077 ceph-mon[37153]: pgmap v9110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:11 smithi077 ceph-mon[37153]: pgmap v9111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:22:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:22:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:22:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:22:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:11 smithi047 ceph-mon[36835]: pgmap v9111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:22:12.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:22:12.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:22:12.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:22:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:11 smithi029 ceph-mon[32610]: pgmap v9111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:22:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:22:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:22:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:22:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:12 smithi077 ceph-mon[37153]: pgmap v9112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T11:22:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:12 smithi077 ceph-mon[37153]: pgmap v9113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:22:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16354.740921 secs 2024-07-27T11:22:13.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:12 smithi029 ceph-mon[32610]: pgmap v9112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T11:22:13.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:12 smithi029 ceph-mon[32610]: pgmap v9113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:22:13.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16354.740921 secs 2024-07-27T11:22:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:12 smithi047 ceph-mon[36835]: pgmap v9112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-27T11:22:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:12 smithi047 ceph-mon[36835]: pgmap v9113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:22:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16354.740921 secs 2024-07-27T11:22:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:14 smithi077 ceph-mon[37153]: pgmap v9114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:22:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:15.123+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:22:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:15.123+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:22:15.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:14 smithi029 ceph-mon[32610]: pgmap v9114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:22:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:15.123+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:22:15.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:15.123+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:22:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:14 smithi047 ceph-mon[36835]: pgmap v9114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:22:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:16 smithi077 ceph-mon[37153]: pgmap v9115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:22:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:16 smithi047 ceph-mon[36835]: pgmap v9115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:22:17.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:16 smithi029 ceph-mon[32610]: pgmap v9115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:22:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16359.741040 secs 2024-07-27T11:22:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16359.741040 secs 2024-07-27T11:22:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16359.741040 secs 2024-07-27T11:22:19.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:18 smithi029 ceph-mon[32610]: pgmap v9116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:22:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:18 smithi077 ceph-mon[37153]: pgmap v9116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:22:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:18 smithi047 ceph-mon[36835]: pgmap v9116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:22:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:20.928+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:22:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:20.928+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:22:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:20.928+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:22:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:20 smithi077 ceph-mon[37153]: pgmap v9117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:22:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:20 smithi047 ceph-mon[36835]: pgmap v9117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:22:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:20 smithi029 ceph-mon[32610]: pgmap v9117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:22:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:22 smithi077 ceph-mon[37153]: pgmap v9118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:22:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16364.741162 secs 2024-07-27T11:22:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:22 smithi047 ceph-mon[36835]: pgmap v9118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:22:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16364.741162 secs 2024-07-27T11:22:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:22 smithi029 ceph-mon[32610]: pgmap v9118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:22:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16364.741162 secs 2024-07-27T11:22:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:24 smithi077 ceph-mon[37153]: pgmap v9119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:24 smithi047 ceph-mon[36835]: pgmap v9119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:24 smithi029 ceph-mon[32610]: pgmap v9119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:26 smithi077 ceph-mon[37153]: pgmap v9120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:26 smithi047 ceph-mon[36835]: pgmap v9120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:26 smithi029 ceph-mon[32610]: pgmap v9120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16369.741271 secs 2024-07-27T11:22:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16369.741271 secs 2024-07-27T11:22:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16369.741271 secs 2024-07-27T11:22:29.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:28 smithi077 ceph-mon[37153]: pgmap v9121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:28 smithi047 ceph-mon[36835]: pgmap v9121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:29.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:28 smithi029 ceph-mon[32610]: pgmap v9121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:30 smithi077 ceph-mon[37153]: pgmap v9122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:22:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:30 smithi047 ceph-mon[36835]: pgmap v9122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:22:31.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:30 smithi029 ceph-mon[32610]: pgmap v9122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:22:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:32 smithi077 ceph-mon[37153]: pgmap v9123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16374.741350 secs 2024-07-27T11:22:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:32 smithi047 ceph-mon[36835]: pgmap v9123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16374.741350 secs 2024-07-27T11:22:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:32 smithi029 ceph-mon[32610]: pgmap v9123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16374.741350 secs 2024-07-27T11:22:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:34 smithi077 ceph-mon[37153]: pgmap v9124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:34 smithi047 ceph-mon[36835]: pgmap v9124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:35.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:34 smithi029 ceph-mon[32610]: pgmap v9124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:35.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:35.942+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:22:35.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:35.942+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:22:35.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:35.942+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:22:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:36 smithi077 ceph-mon[37153]: pgmap v9125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:36 smithi047 ceph-mon[36835]: pgmap v9125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:37.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:36 smithi029 ceph-mon[32610]: pgmap v9125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16379.741500 secs 2024-07-27T11:22:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16379.741500 secs 2024-07-27T11:22:38.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16379.741500 secs 2024-07-27T11:22:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:38 smithi077 ceph-mon[37153]: pgmap v9126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:38 smithi047 ceph-mon[36835]: pgmap v9126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:38 smithi029 ceph-mon[32610]: pgmap v9126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:40 smithi077 ceph-mon[37153]: pgmap v9127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:22:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:40 smithi047 ceph-mon[36835]: pgmap v9127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:22:41.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:40 smithi029 ceph-mon[32610]: pgmap v9127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:22:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:42 smithi077 ceph-mon[37153]: pgmap v9128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:43.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16384.741596 secs 2024-07-27T11:22:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:42 smithi047 ceph-mon[36835]: pgmap v9128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16384.741596 secs 2024-07-27T11:22:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:42 smithi029 ceph-mon[32610]: pgmap v9128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16384.741596 secs 2024-07-27T11:22:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:44 smithi077 ceph-mon[37153]: pgmap v9129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:45.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:44 smithi047 ceph-mon[36835]: pgmap v9129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:44 smithi029 ceph-mon[32610]: pgmap v9129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:46 smithi077 ceph-mon[37153]: pgmap v9130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:46 smithi047 ceph-mon[36835]: pgmap v9130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:47.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:46 smithi029 ceph-mon[32610]: pgmap v9130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16389.741719 secs 2024-07-27T11:22:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16389.741719 secs 2024-07-27T11:22:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16389.741719 secs 2024-07-27T11:22:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:48 smithi077 ceph-mon[37153]: pgmap v9131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:48 smithi047 ceph-mon[36835]: pgmap v9131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:49.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:48 smithi029 ceph-mon[32610]: pgmap v9131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:50.957+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:22:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:50.957+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:22:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:22:50.957+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:22:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:50 smithi077 ceph-mon[37153]: pgmap v9132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:22:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:50 smithi047 ceph-mon[36835]: pgmap v9132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:22:51.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:50 smithi029 ceph-mon[32610]: pgmap v9132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:22:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:52 smithi077 ceph-mon[37153]: pgmap v9133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:53.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16394.741867 secs 2024-07-27T11:22:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:52 smithi047 ceph-mon[36835]: pgmap v9133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16394.741867 secs 2024-07-27T11:22:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:52 smithi029 ceph-mon[32610]: pgmap v9133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16394.741867 secs 2024-07-27T11:22:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:54 smithi047 ceph-mon[36835]: pgmap v9134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:55.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:54 smithi029 ceph-mon[32610]: pgmap v9134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:54 smithi077 ceph-mon[37153]: pgmap v9134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:22:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:56 smithi047 ceph-mon[36835]: pgmap v9135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:56 smithi029 ceph-mon[32610]: pgmap v9135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:56 smithi077 ceph-mon[37153]: pgmap v9135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:22:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16399.742000 secs 2024-07-27T11:22:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16399.742000 secs 2024-07-27T11:22:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16399.742000 secs 2024-07-27T11:22:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:22:58 smithi047 ceph-mon[36835]: pgmap v9136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:59.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:22:58 smithi029 ceph-mon[32610]: pgmap v9136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:22:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:22:58 smithi077 ceph-mon[37153]: pgmap v9136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:00.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:00.130+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:23:00.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:00.130+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:23:00.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:00.130+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:23:00.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:00.130+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:23:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:00 smithi047 ceph-mon[36835]: pgmap v9137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:23:01.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:00 smithi029 ceph-mon[32610]: pgmap v9137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:23:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:00 smithi077 ceph-mon[37153]: pgmap v9137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:23:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:02 smithi047 ceph-mon[36835]: pgmap v9138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16404.742159 secs 2024-07-27T11:23:03.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:02 smithi029 ceph-mon[32610]: pgmap v9138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:03.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16404.742159 secs 2024-07-27T11:23:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:02 smithi077 ceph-mon[37153]: pgmap v9138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16404.742159 secs 2024-07-27T11:23:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:04 smithi047 ceph-mon[36835]: pgmap v9139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:04 smithi029 ceph-mon[32610]: pgmap v9139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:04 smithi077 ceph-mon[37153]: pgmap v9139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:05.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:05.965+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:23:05.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:05.965+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:23:05.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:05.965+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:23:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:06 smithi047 ceph-mon[36835]: pgmap v9140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:07.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:06 smithi029 ceph-mon[32610]: pgmap v9140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:06 smithi077 ceph-mon[37153]: pgmap v9140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16409.742262 secs 2024-07-27T11:23:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16409.742262 secs 2024-07-27T11:23:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16409.742262 secs 2024-07-27T11:23:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:08 smithi047 ceph-mon[36835]: pgmap v9141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:09.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:08 smithi029 ceph-mon[32610]: pgmap v9141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:08 smithi077 ceph-mon[37153]: pgmap v9141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:10 smithi047 ceph-mon[36835]: pgmap v9142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:23:11.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:10 smithi029 ceph-mon[32610]: pgmap v9142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:23:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:10 smithi077 ceph-mon[37153]: pgmap v9142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:23:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:23:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:23:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:23:12.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:23:12.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:23:12.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:23:12.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:23:12.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:23:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:23:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:23:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:23:12.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:23:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:12 smithi047 ceph-mon[36835]: pgmap v9143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:12 smithi047 ceph-mon[36835]: pgmap v9144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:23:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:12 smithi047 ceph-mon[36835]: pgmap v9145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:23:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16414.742396 secs 2024-07-27T11:23:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:12 smithi029 ceph-mon[32610]: pgmap v9143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:12 smithi029 ceph-mon[32610]: pgmap v9144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:23:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:12 smithi029 ceph-mon[32610]: pgmap v9145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:23:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16414.742396 secs 2024-07-27T11:23:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:12 smithi077 ceph-mon[37153]: pgmap v9143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:12 smithi077 ceph-mon[37153]: pgmap v9144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:23:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:12 smithi077 ceph-mon[37153]: pgmap v9145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:23:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16414.742396 secs 2024-07-27T11:23:15.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:14 smithi077 ceph-mon[37153]: pgmap v9146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:23:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:14 smithi047 ceph-mon[36835]: pgmap v9146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:23:15.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:14 smithi029 ceph-mon[32610]: pgmap v9146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:23:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:16 smithi047 ceph-mon[36835]: pgmap v9147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:23:17.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:16 smithi029 ceph-mon[32610]: pgmap v9147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:23:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:16 smithi077 ceph-mon[37153]: pgmap v9147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:23:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16419.742533 secs 2024-07-27T11:23:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16419.742533 secs 2024-07-27T11:23:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16419.742533 secs 2024-07-27T11:23:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:18 smithi047 ceph-mon[36835]: pgmap v9148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:23:19.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:18 smithi029 ceph-mon[32610]: pgmap v9148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:23:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:18 smithi077 ceph-mon[37153]: pgmap v9148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:23:20.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:20.981+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:23:20.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:20.981+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:23:20.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:20.981+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:23:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:20 smithi047 ceph-mon[36835]: pgmap v9149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:23:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:20 smithi029 ceph-mon[32610]: pgmap v9149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:23:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:20 smithi077 ceph-mon[37153]: pgmap v9149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:23:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:22 smithi047 ceph-mon[36835]: pgmap v9150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:23:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16424.742601 secs 2024-07-27T11:23:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:22 smithi029 ceph-mon[32610]: pgmap v9150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:23:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16424.742601 secs 2024-07-27T11:23:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:22 smithi077 ceph-mon[37153]: pgmap v9150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:23:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16424.742601 secs 2024-07-27T11:23:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:24 smithi047 ceph-mon[36835]: pgmap v9151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:25.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:24 smithi029 ceph-mon[32610]: pgmap v9151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:24 smithi077 ceph-mon[37153]: pgmap v9151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:26 smithi047 ceph-mon[36835]: pgmap v9152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:27.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:26 smithi029 ceph-mon[32610]: pgmap v9152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:26 smithi077 ceph-mon[37153]: pgmap v9152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16429.742698 secs 2024-07-27T11:23:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16429.742698 secs 2024-07-27T11:23:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16429.742698 secs 2024-07-27T11:23:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:28 smithi047 ceph-mon[36835]: pgmap v9153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:23:29.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:28 smithi029 ceph-mon[32610]: pgmap v9153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:23:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:28 smithi077 ceph-mon[37153]: pgmap v9153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:23:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:30 smithi047 ceph-mon[36835]: pgmap v9154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:23:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:30 smithi029 ceph-mon[32610]: pgmap v9154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:23:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:30 smithi077 ceph-mon[37153]: pgmap v9154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:23:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:32 smithi047 ceph-mon[36835]: pgmap v9155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16434.742868 secs 2024-07-27T11:23:33.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:32 smithi029 ceph-mon[32610]: pgmap v9155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16434.742868 secs 2024-07-27T11:23:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:32 smithi077 ceph-mon[37153]: pgmap v9155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16434.742868 secs 2024-07-27T11:23:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:34 smithi047 ceph-mon[36835]: pgmap v9156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:35.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:34 smithi029 ceph-mon[32610]: pgmap v9156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:34 smithi077 ceph-mon[37153]: pgmap v9156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:35.990+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:23:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:35.990+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:23:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:35.990+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:23:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:36 smithi029 ceph-mon[32610]: pgmap v9157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:36 smithi077 ceph-mon[37153]: pgmap v9157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:36 smithi047 ceph-mon[36835]: pgmap v9157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16439.742954 secs 2024-07-27T11:23:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16439.742954 secs 2024-07-27T11:23:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16439.742954 secs 2024-07-27T11:23:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:38 smithi029 ceph-mon[32610]: pgmap v9158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:23:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:38 smithi077 ceph-mon[37153]: pgmap v9158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:23:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:38 smithi047 ceph-mon[36835]: pgmap v9158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:23:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:40 smithi029 ceph-mon[32610]: pgmap v9159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:23:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:40 smithi077 ceph-mon[37153]: pgmap v9159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:23:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:40 smithi047 ceph-mon[36835]: pgmap v9159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:23:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:42 smithi029 ceph-mon[32610]: pgmap v9160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16444.743077 secs 2024-07-27T11:23:43.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:42 smithi047 ceph-mon[36835]: pgmap v9160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:43.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16444.743077 secs 2024-07-27T11:23:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:42 smithi077 ceph-mon[37153]: pgmap v9160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16444.743077 secs 2024-07-27T11:23:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:44 smithi077 ceph-mon[37153]: pgmap v9161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:44 smithi047 ceph-mon[36835]: pgmap v9161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:44 smithi029 ceph-mon[32610]: pgmap v9161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:46 smithi077 ceph-mon[37153]: pgmap v9162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:46 smithi047 ceph-mon[36835]: pgmap v9162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:47.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:46 smithi029 ceph-mon[32610]: pgmap v9162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16449.743193 secs 2024-07-27T11:23:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16449.743193 secs 2024-07-27T11:23:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16449.743193 secs 2024-07-27T11:23:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:48 smithi077 ceph-mon[37153]: pgmap v9163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:23:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:48 smithi047 ceph-mon[36835]: pgmap v9163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:23:49.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:48 smithi029 ceph-mon[32610]: pgmap v9163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:23:50.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:50.998+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:23:50.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:50.998+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:23:50.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:23:50.998+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:23:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:50 smithi077 ceph-mon[37153]: pgmap v9164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:23:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:50 smithi047 ceph-mon[36835]: pgmap v9164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:23:51.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:50 smithi029 ceph-mon[32610]: pgmap v9164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:23:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:52 smithi077 ceph-mon[37153]: pgmap v9165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16454.743334 secs 2024-07-27T11:23:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:52 smithi047 ceph-mon[36835]: pgmap v9165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16454.743334 secs 2024-07-27T11:23:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:52 smithi029 ceph-mon[32610]: pgmap v9165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:23:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16454.743334 secs 2024-07-27T11:23:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:54 smithi077 ceph-mon[37153]: pgmap v9166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:54 smithi047 ceph-mon[36835]: pgmap v9166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:55.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:54 smithi029 ceph-mon[32610]: pgmap v9166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:23:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:56 smithi077 ceph-mon[37153]: pgmap v9167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:56 smithi047 ceph-mon[36835]: pgmap v9167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:57.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:56 smithi029 ceph-mon[32610]: pgmap v9167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:23:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16459.743399 secs 2024-07-27T11:23:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16459.743399 secs 2024-07-27T11:23:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16459.743399 secs 2024-07-27T11:23:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:23:58 smithi077 ceph-mon[37153]: pgmap v9168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:23:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:23:58 smithi047 ceph-mon[36835]: pgmap v9168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:23:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:23:58 smithi029 ceph-mon[32610]: pgmap v9168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:24:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:01 smithi077 ceph-mon[37153]: pgmap v9169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:24:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:01 smithi047 ceph-mon[36835]: pgmap v9169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:24:01.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:01 smithi029 ceph-mon[32610]: pgmap v9169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:24:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:03 smithi077 ceph-mon[37153]: pgmap v9170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16464.743467 secs 2024-07-27T11:24:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:03 smithi047 ceph-mon[36835]: pgmap v9170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16464.743467 secs 2024-07-27T11:24:03.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:03 smithi029 ceph-mon[32610]: pgmap v9170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16464.743467 secs 2024-07-27T11:24:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:05 smithi077 ceph-mon[37153]: pgmap v9171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:24:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:05 smithi047 ceph-mon[36835]: pgmap v9171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:24:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:05 smithi029 ceph-mon[32610]: pgmap v9171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:24:06.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:24:06.002+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:24:06.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:24:06.002+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:24:06.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:24:06.002+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:24:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:07 smithi077 ceph-mon[37153]: pgmap v9172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:24:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:07 smithi047 ceph-mon[36835]: pgmap v9172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:24:07.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:07 smithi029 ceph-mon[32610]: pgmap v9172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:24:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16469.743531 secs 2024-07-27T11:24:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16469.743531 secs 2024-07-27T11:24:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16469.743531 secs 2024-07-27T11:24:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:09 smithi077 ceph-mon[37153]: pgmap v9173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:24:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:09 smithi047 ceph-mon[36835]: pgmap v9173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:24:09.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:09 smithi029 ceph-mon[32610]: pgmap v9173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:24:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:11 smithi077 ceph-mon[37153]: pgmap v9174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:24:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:11 smithi047 ceph-mon[36835]: pgmap v9174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:24:11.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:11 smithi029 ceph-mon[32610]: pgmap v9174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:24:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:24:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:24:12.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:24:13.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:13 smithi047 ceph-mon[36835]: pgmap v9175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:13.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16474.743603 secs 2024-07-27T11:24:13.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:24:13.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:24:13.348 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:24:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:13 smithi077 ceph-mon[37153]: pgmap v9175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16474.743603 secs 2024-07-27T11:24:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:24:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:24:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:24:13.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:13 smithi029 ceph-mon[32610]: pgmap v9175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:13.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16474.743603 secs 2024-07-27T11:24:13.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:24:13.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:24:13.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:24:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:14 smithi077 ceph-mon[37153]: pgmap v9176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T11:24:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:14 smithi077 ceph-mon[37153]: pgmap v9177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T11:24:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:14 smithi047 ceph-mon[36835]: pgmap v9176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T11:24:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:14 smithi047 ceph-mon[36835]: pgmap v9177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T11:24:14.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:14 smithi029 ceph-mon[32610]: pgmap v9176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T11:24:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:14 smithi029 ceph-mon[32610]: pgmap v9177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T11:24:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:16 smithi077 ceph-mon[37153]: pgmap v9178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:24:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:16 smithi047 ceph-mon[36835]: pgmap v9178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:24:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:16 smithi029 ceph-mon[32610]: pgmap v9178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:24:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:18 smithi077 ceph-mon[37153]: pgmap v9179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:24:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16479.743732 secs 2024-07-27T11:24:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:18 smithi047 ceph-mon[36835]: pgmap v9179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:24:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16479.743732 secs 2024-07-27T11:24:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:18 smithi029 ceph-mon[32610]: pgmap v9179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:24:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16479.743732 secs 2024-07-27T11:24:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:20 smithi077 ceph-mon[37153]: pgmap v9180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:24:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:20 smithi047 ceph-mon[36835]: pgmap v9180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:24:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:20 smithi029 ceph-mon[32610]: pgmap v9180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:24:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:24:21.012+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:24:21.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:24:21.012+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:24:21.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:24:21.012+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:24:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:22 smithi077 ceph-mon[37153]: pgmap v9181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:24:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:22 smithi047 ceph-mon[36835]: pgmap v9181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:24:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:22 smithi029 ceph-mon[32610]: pgmap v9181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:24:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16484.743863 secs 2024-07-27T11:24:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16484.743863 secs 2024-07-27T11:24:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16484.743863 secs 2024-07-27T11:24:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:24 smithi077 ceph-mon[37153]: pgmap v9182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:24:24.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:24 smithi029 ceph-mon[32610]: pgmap v9182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:24:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:24 smithi047 ceph-mon[36835]: pgmap v9182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:24:26.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:26 smithi077 ceph-mon[37153]: pgmap v9183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:24:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:26 smithi047 ceph-mon[36835]: pgmap v9183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:24:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:26 smithi029 ceph-mon[32610]: pgmap v9183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:24:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:28 smithi077 ceph-mon[37153]: pgmap v9184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16489.743960 secs 2024-07-27T11:24:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:28 smithi047 ceph-mon[36835]: pgmap v9184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16489.743960 secs 2024-07-27T11:24:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:28 smithi029 ceph-mon[32610]: pgmap v9184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16489.743960 secs 2024-07-27T11:24:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:30 smithi077 ceph-mon[37153]: pgmap v9185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:24:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:30 smithi047 ceph-mon[36835]: pgmap v9185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:24:30.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:30 smithi029 ceph-mon[32610]: pgmap v9185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:24:32.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:32 smithi077 ceph-mon[37153]: pgmap v9186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:24:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:32 smithi047 ceph-mon[36835]: pgmap v9186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:24:32.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:32 smithi029 ceph-mon[32610]: pgmap v9186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:24:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16494.744067 secs 2024-07-27T11:24:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16494.744067 secs 2024-07-27T11:24:33.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16494.744067 secs 2024-07-27T11:24:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:34 smithi077 ceph-mon[37153]: pgmap v9187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:24:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:34 smithi047 ceph-mon[36835]: pgmap v9187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:24:34.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:34 smithi029 ceph-mon[32610]: pgmap v9187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:24:36.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:24:36.022+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:24:36.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:24:36.022+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:24:36.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:24:36.022+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:24:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:36 smithi077 ceph-mon[37153]: pgmap v9188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:24:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:36 smithi047 ceph-mon[36835]: pgmap v9188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:24:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:36 smithi029 ceph-mon[32610]: pgmap v9188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:24:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:38 smithi077 ceph-mon[37153]: pgmap v9189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16499.744191 secs 2024-07-27T11:24:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:38 smithi047 ceph-mon[36835]: pgmap v9189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16499.744191 secs 2024-07-27T11:24:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:38 smithi029 ceph-mon[32610]: pgmap v9189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16499.744191 secs 2024-07-27T11:24:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:40 smithi077 ceph-mon[37153]: pgmap v9190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:24:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:40 smithi047 ceph-mon[36835]: pgmap v9190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:24:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:40 smithi029 ceph-mon[32610]: pgmap v9190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:24:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:42 smithi077 ceph-mon[37153]: pgmap v9191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:24:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:42 smithi047 ceph-mon[36835]: pgmap v9191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:24:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:42 smithi029 ceph-mon[32610]: pgmap v9191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:24:43.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16504.744264 secs 2024-07-27T11:24:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16504.744264 secs 2024-07-27T11:24:43.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16504.744264 secs 2024-07-27T11:24:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:44 smithi077 ceph-mon[37153]: pgmap v9192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:24:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:44 smithi047 ceph-mon[36835]: pgmap v9192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:24:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:44 smithi029 ceph-mon[32610]: pgmap v9192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:24:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:46 smithi077 ceph-mon[37153]: pgmap v9193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:24:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:46 smithi047 ceph-mon[36835]: pgmap v9193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:24:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:46 smithi029 ceph-mon[32610]: pgmap v9193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:24:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:48 smithi077 ceph-mon[37153]: pgmap v9194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16509.744406 secs 2024-07-27T11:24:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:48 smithi047 ceph-mon[36835]: pgmap v9194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16509.744406 secs 2024-07-27T11:24:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:48 smithi029 ceph-mon[32610]: pgmap v9194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16509.744406 secs 2024-07-27T11:24:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:50 smithi047 ceph-mon[36835]: pgmap v9195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:24:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:50 smithi029 ceph-mon[32610]: pgmap v9195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:24:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:50 smithi077 ceph-mon[37153]: pgmap v9195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:24:51.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:24:51.037+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:24:51.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:24:51.037+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:24:51.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:24:51.037+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:24:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:52 smithi047 ceph-mon[36835]: pgmap v9196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:24:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:52 smithi029 ceph-mon[32610]: pgmap v9196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:24:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:52 smithi077 ceph-mon[37153]: pgmap v9196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:24:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16514.744470 secs 2024-07-27T11:24:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16514.744470 secs 2024-07-27T11:24:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16514.744470 secs 2024-07-27T11:24:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:54 smithi047 ceph-mon[36835]: pgmap v9197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:24:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:54 smithi029 ceph-mon[32610]: pgmap v9197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:24:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:54 smithi077 ceph-mon[37153]: pgmap v9197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:24:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:56 smithi047 ceph-mon[36835]: pgmap v9198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:24:56.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:56 smithi029 ceph-mon[32610]: pgmap v9198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:24:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:56 smithi077 ceph-mon[37153]: pgmap v9198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:24:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:58 smithi047 ceph-mon[36835]: pgmap v9199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:58.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:24:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16519.744592 secs 2024-07-27T11:24:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:58 smithi029 ceph-mon[32610]: pgmap v9199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:24:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16519.744592 secs 2024-07-27T11:24:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:58 smithi077 ceph-mon[37153]: pgmap v9199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:24:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:24:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16519.744592 secs 2024-07-27T11:25:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:00 smithi047 ceph-mon[36835]: pgmap v9200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:00.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:00 smithi029 ceph-mon[32610]: pgmap v9200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:00 smithi077 ceph-mon[37153]: pgmap v9200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:02 smithi047 ceph-mon[36835]: pgmap v9201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:02 smithi029 ceph-mon[32610]: pgmap v9201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:02 smithi077 ceph-mon[37153]: pgmap v9201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16524.744712 secs 2024-07-27T11:25:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16524.744712 secs 2024-07-27T11:25:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16524.744712 secs 2024-07-27T11:25:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:04 smithi047 ceph-mon[36835]: pgmap v9202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:25:04.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:04 smithi029 ceph-mon[32610]: pgmap v9202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:25:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:04 smithi077 ceph-mon[37153]: pgmap v9202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:25:06.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:25:06.050+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:25:06.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:25:06.050+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:25:06.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:25:06.050+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:25:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:06 smithi047 ceph-mon[36835]: pgmap v9203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:25:06.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:06 smithi029 ceph-mon[32610]: pgmap v9203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:25:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:06 smithi077 ceph-mon[37153]: pgmap v9203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:25:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:08 smithi047 ceph-mon[36835]: pgmap v9204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:25:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16529.744837 secs 2024-07-27T11:25:08.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:08 smithi029 ceph-mon[32610]: pgmap v9204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:25:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16529.744837 secs 2024-07-27T11:25:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:08 smithi077 ceph-mon[37153]: pgmap v9204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:25:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16529.744837 secs 2024-07-27T11:25:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:10 smithi047 ceph-mon[36835]: pgmap v9205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:10.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:10 smithi029 ceph-mon[32610]: pgmap v9205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:10 smithi077 ceph-mon[37153]: pgmap v9205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:12 smithi047 ceph-mon[36835]: pgmap v9206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:12 smithi029 ceph-mon[32610]: pgmap v9206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:12 smithi077 ceph-mon[37153]: pgmap v9206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16534.744962 secs 2024-07-27T11:25:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:13 smithi047 ceph-mon[36835]: pgmap v9207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:25:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:25:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:25:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:25:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:13 smithi047 ceph-mon[36835]: pgmap v9208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:25:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:13 smithi047 ceph-mon[36835]: pgmap v9209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:25:13.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:25:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16534.744962 secs 2024-07-27T11:25:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:13 smithi029 ceph-mon[32610]: pgmap v9207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:25:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:25:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:25:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:25:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:13 smithi029 ceph-mon[32610]: pgmap v9208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:25:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:13 smithi029 ceph-mon[32610]: pgmap v9209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:25:13.465 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:25:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16534.744962 secs 2024-07-27T11:25:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:13 smithi077 ceph-mon[37153]: pgmap v9207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:25:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:25:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:25:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:25:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:13 smithi077 ceph-mon[37153]: pgmap v9208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:25:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:13 smithi077 ceph-mon[37153]: pgmap v9209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:25:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:25:16.309 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:15 smithi077 ceph-mon[37153]: pgmap v9210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:25:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:15 smithi047 ceph-mon[36835]: pgmap v9210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:25:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:15 smithi029 ceph-mon[32610]: pgmap v9210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:25:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:17 smithi077 ceph-mon[37153]: pgmap v9211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:25:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16539.745092 secs 2024-07-27T11:25:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:17 smithi047 ceph-mon[36835]: pgmap v9211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:25:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16539.745092 secs 2024-07-27T11:25:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:17 smithi029 ceph-mon[32610]: pgmap v9211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:25:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16539.745092 secs 2024-07-27T11:25:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:19 smithi077 ceph-mon[37153]: pgmap v9212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:19 smithi047 ceph-mon[36835]: pgmap v9212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:20.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:19 smithi029 ceph-mon[32610]: pgmap v9212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:21.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:25:21.066+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:25:21.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:25:21.066+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:25:21.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:25:21.066+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:25:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:21 smithi077 ceph-mon[37153]: pgmap v9213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T11:25:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:21 smithi047 ceph-mon[36835]: pgmap v9213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T11:25:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:21 smithi029 ceph-mon[32610]: pgmap v9213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-27T11:25:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16544.745224 secs 2024-07-27T11:25:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16544.745224 secs 2024-07-27T11:25:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16544.745224 secs 2024-07-27T11:25:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:23 smithi077 ceph-mon[37153]: pgmap v9214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T11:25:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:23 smithi047 ceph-mon[36835]: pgmap v9214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T11:25:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:23 smithi029 ceph-mon[32610]: pgmap v9214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T11:25:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:26 smithi029 ceph-mon[32610]: pgmap v9215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:25:26.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:26 smithi077 ceph-mon[37153]: pgmap v9215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:25:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:26 smithi047 ceph-mon[36835]: pgmap v9215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:25:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:28 smithi077 ceph-mon[37153]: pgmap v9216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:25:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16549.745344 secs 2024-07-27T11:25:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:28 smithi047 ceph-mon[36835]: pgmap v9216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:25:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16549.745344 secs 2024-07-27T11:25:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:28 smithi029 ceph-mon[32610]: pgmap v9216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:25:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16549.745344 secs 2024-07-27T11:25:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:30 smithi077 ceph-mon[37153]: pgmap v9217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:30 smithi047 ceph-mon[36835]: pgmap v9217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:30.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:30 smithi029 ceph-mon[32610]: pgmap v9217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:32 smithi077 ceph-mon[37153]: pgmap v9218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:32 smithi047 ceph-mon[36835]: pgmap v9218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:32 smithi029 ceph-mon[32610]: pgmap v9218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16554.745470 secs 2024-07-27T11:25:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16554.745470 secs 2024-07-27T11:25:33.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16554.745470 secs 2024-07-27T11:25:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:34 smithi077 ceph-mon[37153]: pgmap v9219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:25:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:34 smithi047 ceph-mon[36835]: pgmap v9219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:25:34.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:34 smithi029 ceph-mon[32610]: pgmap v9219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:25:36.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:25:36.082+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:25:36.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:25:36.082+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:25:36.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:25:36.082+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:25:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:36 smithi077 ceph-mon[37153]: pgmap v9220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:25:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:36 smithi047 ceph-mon[36835]: pgmap v9220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:25:36.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:36 smithi029 ceph-mon[32610]: pgmap v9220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:25:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:38 smithi077 ceph-mon[37153]: pgmap v9221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:25:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16559.745628 secs 2024-07-27T11:25:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:38 smithi047 ceph-mon[36835]: pgmap v9221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:25:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16559.745628 secs 2024-07-27T11:25:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:38 smithi029 ceph-mon[32610]: pgmap v9221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:25:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16559.745628 secs 2024-07-27T11:25:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:40 smithi077 ceph-mon[37153]: pgmap v9222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:40 smithi047 ceph-mon[36835]: pgmap v9222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:40 smithi029 ceph-mon[32610]: pgmap v9222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:42 smithi077 ceph-mon[37153]: pgmap v9223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:42 smithi047 ceph-mon[36835]: pgmap v9223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:42 smithi029 ceph-mon[32610]: pgmap v9223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:43.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16564.745698 secs 2024-07-27T11:25:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16564.745698 secs 2024-07-27T11:25:43.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16564.745698 secs 2024-07-27T11:25:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:44 smithi077 ceph-mon[37153]: pgmap v9224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:25:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:44 smithi047 ceph-mon[36835]: pgmap v9224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:25:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:44 smithi029 ceph-mon[32610]: pgmap v9224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:25:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:46 smithi077 ceph-mon[37153]: pgmap v9225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:46 smithi047 ceph-mon[36835]: pgmap v9225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:46 smithi029 ceph-mon[32610]: pgmap v9225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:48 smithi077 ceph-mon[37153]: pgmap v9226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:25:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16569.745807 secs 2024-07-27T11:25:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:48 smithi047 ceph-mon[36835]: pgmap v9226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:25:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16569.745807 secs 2024-07-27T11:25:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:48 smithi029 ceph-mon[32610]: pgmap v9226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:25:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16569.745807 secs 2024-07-27T11:25:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:50 smithi077 ceph-mon[37153]: pgmap v9227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:50 smithi047 ceph-mon[36835]: pgmap v9227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:50.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:50 smithi029 ceph-mon[32610]: pgmap v9227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:25:51.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:25:51.097+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:25:51.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:25:51.097+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:25:51.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:25:51.097+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:25:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:52 smithi077 ceph-mon[37153]: pgmap v9228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:52 smithi047 ceph-mon[36835]: pgmap v9228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:52.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:52 smithi029 ceph-mon[32610]: pgmap v9228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16574.745904 secs 2024-07-27T11:25:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16574.745904 secs 2024-07-27T11:25:53.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16574.745904 secs 2024-07-27T11:25:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:54 smithi077 ceph-mon[37153]: pgmap v9229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:25:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:54 smithi047 ceph-mon[36835]: pgmap v9229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:25:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:54 smithi029 ceph-mon[32610]: pgmap v9229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:25:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:56 smithi077 ceph-mon[37153]: pgmap v9230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:56 smithi047 ceph-mon[36835]: pgmap v9230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:56 smithi029 ceph-mon[32610]: pgmap v9230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:25:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:58 smithi077 ceph-mon[37153]: pgmap v9231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:25:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:25:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16579.746028 secs 2024-07-27T11:25:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:58 smithi047 ceph-mon[36835]: pgmap v9231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:25:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:25:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16579.746028 secs 2024-07-27T11:25:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:58 smithi029 ceph-mon[32610]: pgmap v9231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:25:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:25:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16579.746028 secs 2024-07-27T11:26:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:00.157+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:26:00.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:00.157+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:26:00.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:00.157+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:26:00.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:00.157+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:26:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:00 smithi077 ceph-mon[37153]: pgmap v9232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:26:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:00 smithi047 ceph-mon[36835]: pgmap v9232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:26:00.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:00 smithi029 ceph-mon[32610]: pgmap v9232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:26:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:02 smithi077 ceph-mon[37153]: pgmap v9233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:02 smithi047 ceph-mon[36835]: pgmap v9233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:02 smithi029 ceph-mon[32610]: pgmap v9233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16584.746100 secs 2024-07-27T11:26:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16584.746100 secs 2024-07-27T11:26:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16584.746100 secs 2024-07-27T11:26:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:04 smithi077 ceph-mon[37153]: pgmap v9234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:26:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:04 smithi047 ceph-mon[36835]: pgmap v9234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:26:04.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:04 smithi029 ceph-mon[32610]: pgmap v9234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:26:06.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:06.102+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:26:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:06.102+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:26:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:06.102+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:26:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:06 smithi077 ceph-mon[37153]: pgmap v9235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:06 smithi047 ceph-mon[36835]: pgmap v9235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:06.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:06 smithi029 ceph-mon[32610]: pgmap v9235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:08 smithi077 ceph-mon[37153]: pgmap v9236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:26:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16589.746225 secs 2024-07-27T11:26:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:08 smithi047 ceph-mon[36835]: pgmap v9236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:26:08.449 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16589.746225 secs 2024-07-27T11:26:08.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:08 smithi029 ceph-mon[32610]: pgmap v9236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:26:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16589.746225 secs 2024-07-27T11:26:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:09 smithi077 ceph-mon[37153]: pgmap v9237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:26:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:09 smithi047 ceph-mon[36835]: pgmap v9237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:26:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:09 smithi029 ceph-mon[32610]: pgmap v9237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:26:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:12 smithi077 ceph-mon[37153]: pgmap v9238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:12 smithi047 ceph-mon[36835]: pgmap v9238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:12.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:12 smithi029 ceph-mon[32610]: pgmap v9238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:13.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16594.746350 secs 2024-07-27T11:26:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16594.746350 secs 2024-07-27T11:26:13.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16594.746350 secs 2024-07-27T11:26:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:14 smithi077 ceph-mon[37153]: pgmap v9239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:26:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:26:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:26:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:26:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:26:14.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:26:14.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:26:14.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:26:14.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:26:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:14 smithi047 ceph-mon[36835]: pgmap v9239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:26:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:26:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:26:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:26:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:26:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:26:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:26:14.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:26:14.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:26:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:14 smithi029 ceph-mon[32610]: pgmap v9239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:26:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:26:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:26:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:26:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:26:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:26:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:26:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:26:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:26:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:15 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:26:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:15 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:26:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:15 smithi077 ceph-mon[37153]: pgmap v9240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:26:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:15 smithi077 ceph-mon[37153]: pgmap v9241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:26:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:15 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:26:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:15 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:26:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:15 smithi047 ceph-mon[36835]: pgmap v9240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:26:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:15 smithi047 ceph-mon[36835]: pgmap v9241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:26:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:15 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:26:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:15 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:26:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:15 smithi029 ceph-mon[32610]: pgmap v9240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:26:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:15 smithi029 ceph-mon[32610]: pgmap v9241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:26:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:17 smithi077 ceph-mon[37153]: pgmap v9242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:26:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:17 smithi047 ceph-mon[36835]: pgmap v9242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:26:17.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:17 smithi029 ceph-mon[32610]: pgmap v9242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:26:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16599.746494 secs 2024-07-27T11:26:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16599.746494 secs 2024-07-27T11:26:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16599.746494 secs 2024-07-27T11:26:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:19 smithi077 ceph-mon[37153]: pgmap v9243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:26:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:19 smithi047 ceph-mon[36835]: pgmap v9243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:26:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:19 smithi029 ceph-mon[32610]: pgmap v9243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:26:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:21.118+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:26:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:21.118+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:26:21.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:21.118+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:26:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:21 smithi077 ceph-mon[37153]: pgmap v9244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:26:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:21 smithi047 ceph-mon[36835]: pgmap v9244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:26:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:21 smithi029 ceph-mon[32610]: pgmap v9244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:26:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:23 smithi077 ceph-mon[37153]: pgmap v9245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:26:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16604.746589 secs 2024-07-27T11:26:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:23 smithi047 ceph-mon[36835]: pgmap v9245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:26:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16604.746589 secs 2024-07-27T11:26:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:23 smithi029 ceph-mon[32610]: pgmap v9245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:26:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16604.746589 secs 2024-07-27T11:26:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:25 smithi077 ceph-mon[37153]: pgmap v9246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:26:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:25 smithi047 ceph-mon[36835]: pgmap v9246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:26:25.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:25 smithi029 ceph-mon[32610]: pgmap v9246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:26:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:27 smithi077 ceph-mon[37153]: pgmap v9247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:27 smithi047 ceph-mon[36835]: pgmap v9247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:27 smithi029 ceph-mon[32610]: pgmap v9247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16609.746730 secs 2024-07-27T11:26:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16609.746730 secs 2024-07-27T11:26:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16609.746730 secs 2024-07-27T11:26:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:29 smithi077 ceph-mon[37153]: pgmap v9248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:26:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:29 smithi047 ceph-mon[36835]: pgmap v9248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:26:29.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:29 smithi029 ceph-mon[32610]: pgmap v9248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:26:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:31 smithi077 ceph-mon[37153]: pgmap v9249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:26:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:31 smithi047 ceph-mon[36835]: pgmap v9249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:26:31.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:31 smithi029 ceph-mon[32610]: pgmap v9249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:26:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:33 smithi077 ceph-mon[37153]: pgmap v9250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:26:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16614.746853 secs 2024-07-27T11:26:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:33 smithi047 ceph-mon[36835]: pgmap v9250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:26:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16614.746853 secs 2024-07-27T11:26:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:33 smithi029 ceph-mon[32610]: pgmap v9250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:26:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16614.746853 secs 2024-07-27T11:26:35.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:35 smithi077 ceph-mon[37153]: pgmap v9251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:26:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:35 smithi047 ceph-mon[36835]: pgmap v9251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:26:35.465 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:35 smithi029 ceph-mon[32610]: pgmap v9251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:26:36.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:36.133+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:26:36.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:36.133+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:26:36.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:36.133+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:26:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:37 smithi077 ceph-mon[37153]: pgmap v9252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:26:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:37 smithi047 ceph-mon[36835]: pgmap v9252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:26:37.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:37 smithi029 ceph-mon[32610]: pgmap v9252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:26:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16619.746970 secs 2024-07-27T11:26:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16619.746970 secs 2024-07-27T11:26:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16619.746970 secs 2024-07-27T11:26:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:39 smithi077 ceph-mon[37153]: pgmap v9253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:26:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:39 smithi047 ceph-mon[36835]: pgmap v9253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:26:39.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:39 smithi029 ceph-mon[32610]: pgmap v9253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:26:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:41 smithi077 ceph-mon[37153]: pgmap v9254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:26:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:41 smithi047 ceph-mon[36835]: pgmap v9254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:26:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:41 smithi029 ceph-mon[32610]: pgmap v9254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:26:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:43 smithi077 ceph-mon[37153]: pgmap v9255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:26:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16624.747086 secs 2024-07-27T11:26:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:43 smithi047 ceph-mon[36835]: pgmap v9255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:26:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16624.747086 secs 2024-07-27T11:26:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:43 smithi029 ceph-mon[32610]: pgmap v9255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:26:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16624.747086 secs 2024-07-27T11:26:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:45 smithi077 ceph-mon[37153]: pgmap v9256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:26:45.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:45 smithi047 ceph-mon[36835]: pgmap v9256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:26:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:45 smithi029 ceph-mon[32610]: pgmap v9256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:26:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:47 smithi077 ceph-mon[37153]: pgmap v9257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:47 smithi047 ceph-mon[36835]: pgmap v9257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:47.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:47 smithi029 ceph-mon[32610]: pgmap v9257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:26:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16629.747114 secs 2024-07-27T11:26:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16629.747114 secs 2024-07-27T11:26:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16629.747114 secs 2024-07-27T11:26:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:49 smithi077 ceph-mon[37153]: pgmap v9258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:26:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:49 smithi047 ceph-mon[36835]: pgmap v9258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:26:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:49 smithi029 ceph-mon[32610]: pgmap v9258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:26:51.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:51.149+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:26:51.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:51.149+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:26:51.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:26:51.149+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:26:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:51 smithi077 ceph-mon[37153]: pgmap v9259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:26:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:51 smithi047 ceph-mon[36835]: pgmap v9259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:26:51.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:51 smithi029 ceph-mon[32610]: pgmap v9259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:26:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:53 smithi077 ceph-mon[37153]: pgmap v9260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:26:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16634.747245 secs 2024-07-27T11:26:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:53 smithi047 ceph-mon[36835]: pgmap v9260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:26:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16634.747245 secs 2024-07-27T11:26:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:53 smithi029 ceph-mon[32610]: pgmap v9260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:26:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16634.747245 secs 2024-07-27T11:26:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:55 smithi077 ceph-mon[37153]: pgmap v9261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:26:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:55 smithi047 ceph-mon[36835]: pgmap v9261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:26:55.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:55 smithi029 ceph-mon[32610]: pgmap v9261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:26:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:57 smithi077 ceph-mon[37153]: pgmap v9262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:26:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:57 smithi047 ceph-mon[36835]: pgmap v9262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:26:57.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:57 smithi029 ceph-mon[32610]: pgmap v9262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:26:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16639.747368 secs 2024-07-27T11:26:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16639.747368 secs 2024-07-27T11:26:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16639.747368 secs 2024-07-27T11:26:59.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:26:59 smithi029 ceph-mon[32610]: pgmap v9263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:26:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:26:59 smithi047 ceph-mon[36835]: pgmap v9263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:26:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:26:59 smithi077 ceph-mon[37153]: pgmap v9263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:27:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:01 smithi047 ceph-mon[36835]: pgmap v9264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:27:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:01 smithi029 ceph-mon[32610]: pgmap v9264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:27:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:01 smithi077 ceph-mon[37153]: pgmap v9264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:27:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:03 smithi047 ceph-mon[36835]: pgmap v9265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16644.747502 secs 2024-07-27T11:27:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:03 smithi029 ceph-mon[32610]: pgmap v9265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16644.747502 secs 2024-07-27T11:27:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:03 smithi077 ceph-mon[37153]: pgmap v9265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16644.747502 secs 2024-07-27T11:27:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:05 smithi047 ceph-mon[36835]: pgmap v9266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:05 smithi029 ceph-mon[32610]: pgmap v9266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:05 smithi077 ceph-mon[37153]: pgmap v9266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:06.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:27:06.153+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:27:06.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:27:06.153+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:27:06.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:27:06.153+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:27:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:07 smithi047 ceph-mon[36835]: pgmap v9267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:27:07.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:07 smithi029 ceph-mon[32610]: pgmap v9267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:27:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:07 smithi077 ceph-mon[37153]: pgmap v9267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:27:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16649.747627 secs 2024-07-27T11:27:08.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16649.747627 secs 2024-07-27T11:27:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16649.747627 secs 2024-07-27T11:27:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:09 smithi047 ceph-mon[36835]: pgmap v9268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:27:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:09 smithi029 ceph-mon[32610]: pgmap v9268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:27:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:09 smithi077 ceph-mon[37153]: pgmap v9268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:27:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:11 smithi047 ceph-mon[36835]: pgmap v9269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:27:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:11 smithi029 ceph-mon[32610]: pgmap v9269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:27:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:11 smithi077 ceph-mon[37153]: pgmap v9269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:27:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:13 smithi047 ceph-mon[36835]: pgmap v9270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16654.747726 secs 2024-07-27T11:27:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:13 smithi029 ceph-mon[32610]: pgmap v9270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16654.747726 secs 2024-07-27T11:27:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:13 smithi077 ceph-mon[37153]: pgmap v9270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16654.747726 secs 2024-07-27T11:27:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:27:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:27:14.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:27:14.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:27:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:27:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:27:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:27:14.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:27:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:27:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:27:14.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:27:14.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:27:14.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:27:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:15 smithi047 ceph-mon[36835]: pgmap v9271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:15 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:27:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:15 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:27:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:15 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:27:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:15 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:27:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:15 smithi047 ceph-mon[36835]: pgmap v9272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:27:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:15 smithi047 ceph-mon[36835]: pgmap v9273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:27:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:15 smithi029 ceph-mon[32610]: pgmap v9271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:15 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:27:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:15 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:27:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:15 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:27:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:15 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:27:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:15 smithi029 ceph-mon[32610]: pgmap v9272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:27:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:15 smithi029 ceph-mon[32610]: pgmap v9273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:27:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:15 smithi077 ceph-mon[37153]: pgmap v9271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:15 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:27:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:15 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:27:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:15 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:27:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:15 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:27:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:15 smithi077 ceph-mon[37153]: pgmap v9272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:27:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:15 smithi077 ceph-mon[37153]: pgmap v9273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:27:17.408 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:17 smithi077 ceph-mon[37153]: pgmap v9274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:27:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:17 smithi047 ceph-mon[36835]: pgmap v9274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:27:17.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:17 smithi029 ceph-mon[32610]: pgmap v9274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:27:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16659.747834 secs 2024-07-27T11:27:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16659.747834 secs 2024-07-27T11:27:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16659.747834 secs 2024-07-27T11:27:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:19 smithi047 ceph-mon[36835]: pgmap v9275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:27:19.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:19 smithi029 ceph-mon[32610]: pgmap v9275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:27:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:19 smithi077 ceph-mon[37153]: pgmap v9275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:27:21.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:27:21.168+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:27:21.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:27:21.168+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:27:21.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:27:21.168+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:27:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:21 smithi047 ceph-mon[36835]: pgmap v9276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:27:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:21 smithi029 ceph-mon[32610]: pgmap v9276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:27:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:21 smithi077 ceph-mon[37153]: pgmap v9276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:27:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16664.747960 secs 2024-07-27T11:27:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:23 smithi047 ceph-mon[36835]: pgmap v9277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:27:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16664.747960 secs 2024-07-27T11:27:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:23 smithi029 ceph-mon[32610]: pgmap v9277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:27:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16664.747960 secs 2024-07-27T11:27:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:23 smithi077 ceph-mon[37153]: pgmap v9277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:27:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:25 smithi047 ceph-mon[36835]: pgmap v9278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:27:25.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:25 smithi029 ceph-mon[32610]: pgmap v9278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:27:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:25 smithi077 ceph-mon[37153]: pgmap v9278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:27:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:27 smithi047 ceph-mon[36835]: pgmap v9279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:27.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:27 smithi029 ceph-mon[32610]: pgmap v9279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:27 smithi077 ceph-mon[37153]: pgmap v9279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16669.748083 secs 2024-07-27T11:27:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16669.748083 secs 2024-07-27T11:27:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16669.748083 secs 2024-07-27T11:27:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:29 smithi047 ceph-mon[36835]: pgmap v9280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:29 smithi029 ceph-mon[32610]: pgmap v9280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:29 smithi077 ceph-mon[37153]: pgmap v9280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:30 smithi047 ceph-mon[36835]: pgmap v9281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:27:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:30 smithi029 ceph-mon[32610]: pgmap v9281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:27:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:30 smithi077 ceph-mon[37153]: pgmap v9281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:27:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16674.748185 secs 2024-07-27T11:27:33.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:33 smithi047 ceph-mon[36835]: pgmap v9282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16674.748185 secs 2024-07-27T11:27:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:33 smithi029 ceph-mon[32610]: pgmap v9282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16674.748185 secs 2024-07-27T11:27:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:33 smithi077 ceph-mon[37153]: pgmap v9282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:35 smithi047 ceph-mon[36835]: pgmap v9283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:27:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:35 smithi029 ceph-mon[32610]: pgmap v9283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:27:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:35 smithi077 ceph-mon[37153]: pgmap v9283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:27:36.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:27:36.172+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:27:36.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:27:36.172+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:27:36.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:27:36.172+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:27:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:37 smithi047 ceph-mon[36835]: pgmap v9284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:37.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:37 smithi029 ceph-mon[32610]: pgmap v9284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:37 smithi077 ceph-mon[37153]: pgmap v9284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16679.748334 secs 2024-07-27T11:27:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16679.748334 secs 2024-07-27T11:27:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16679.748334 secs 2024-07-27T11:27:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:39 smithi047 ceph-mon[36835]: pgmap v9285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:39 smithi029 ceph-mon[32610]: pgmap v9285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:39 smithi077 ceph-mon[37153]: pgmap v9285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:41 smithi047 ceph-mon[36835]: pgmap v9286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:27:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:41 smithi029 ceph-mon[32610]: pgmap v9286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:27:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:41 smithi077 ceph-mon[37153]: pgmap v9286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:27:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16684.748457 secs 2024-07-27T11:27:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:43 smithi047 ceph-mon[36835]: pgmap v9287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16684.748457 secs 2024-07-27T11:27:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:43 smithi029 ceph-mon[32610]: pgmap v9287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16684.748457 secs 2024-07-27T11:27:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:43 smithi077 ceph-mon[37153]: pgmap v9287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:45 smithi047 ceph-mon[36835]: pgmap v9288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:27:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:45 smithi029 ceph-mon[32610]: pgmap v9288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:27:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:45 smithi077 ceph-mon[37153]: pgmap v9288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:27:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:47 smithi047 ceph-mon[36835]: pgmap v9289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:47 smithi029 ceph-mon[32610]: pgmap v9289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:47 smithi077 ceph-mon[37153]: pgmap v9289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16689.748587 secs 2024-07-27T11:27:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16689.748587 secs 2024-07-27T11:27:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16689.748587 secs 2024-07-27T11:27:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:49 smithi047 ceph-mon[36835]: pgmap v9290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:49.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:49 smithi029 ceph-mon[32610]: pgmap v9290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:49 smithi077 ceph-mon[37153]: pgmap v9290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:27:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:50 smithi047 ceph-mon[36835]: pgmap v9291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:27:50.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:50 smithi029 ceph-mon[32610]: pgmap v9291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:27:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:50 smithi077 ceph-mon[37153]: pgmap v9291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:27:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:27:51.174+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:27:51.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:27:51.174+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:27:51.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:27:51.174+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:27:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16694.748713 secs 2024-07-27T11:27:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:53 smithi047 ceph-mon[36835]: pgmap v9292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16694.748713 secs 2024-07-27T11:27:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:53 smithi029 ceph-mon[32610]: pgmap v9292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16694.748713 secs 2024-07-27T11:27:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:53 smithi077 ceph-mon[37153]: pgmap v9292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:55 smithi047 ceph-mon[36835]: pgmap v9293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:27:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:55 smithi029 ceph-mon[32610]: pgmap v9293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:27:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:55 smithi077 ceph-mon[37153]: pgmap v9293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:27:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:57 smithi047 ceph-mon[36835]: pgmap v9294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:57 smithi029 ceph-mon[32610]: pgmap v9294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:57 smithi077 ceph-mon[37153]: pgmap v9294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:27:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16699.748860 secs 2024-07-27T11:27:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16699.748860 secs 2024-07-27T11:27:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16699.748860 secs 2024-07-27T11:27:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:27:59 smithi047 ceph-mon[36835]: pgmap v9295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:27:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:27:59 smithi029 ceph-mon[32610]: pgmap v9295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:27:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:27:59 smithi077 ceph-mon[37153]: pgmap v9295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:28:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:01 smithi047 ceph-mon[36835]: pgmap v9296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:28:01.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:01 smithi029 ceph-mon[32610]: pgmap v9296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:28:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:01 smithi077 ceph-mon[37153]: pgmap v9296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:28:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16704.748984 secs 2024-07-27T11:28:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:03 smithi047 ceph-mon[36835]: pgmap v9297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16704.748984 secs 2024-07-27T11:28:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:03 smithi029 ceph-mon[32610]: pgmap v9297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16704.748984 secs 2024-07-27T11:28:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:03 smithi077 ceph-mon[37153]: pgmap v9297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:05 smithi047 ceph-mon[36835]: pgmap v9298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:28:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:05 smithi029 ceph-mon[32610]: pgmap v9298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:28:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:05 smithi077 ceph-mon[37153]: pgmap v9298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:28:06.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:28:06.188+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:28:06.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:28:06.188+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:28:06.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:28:06.188+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:28:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:07 smithi047 ceph-mon[36835]: pgmap v9299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:07 smithi029 ceph-mon[32610]: pgmap v9299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:07 smithi077 ceph-mon[37153]: pgmap v9299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16709.749100 secs 2024-07-27T11:28:08.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16709.749100 secs 2024-07-27T11:28:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16709.749100 secs 2024-07-27T11:28:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:09 smithi047 ceph-mon[36835]: pgmap v9300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:28:09.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:09 smithi029 ceph-mon[32610]: pgmap v9300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:28:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:09 smithi077 ceph-mon[37153]: pgmap v9300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:28:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:10 smithi047 ceph-mon[36835]: pgmap v9301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:28:10.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:10 smithi029 ceph-mon[32610]: pgmap v9301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:28:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:10 smithi077 ceph-mon[37153]: pgmap v9301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:28:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16714.749170 secs 2024-07-27T11:28:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:13 smithi047 ceph-mon[36835]: pgmap v9302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16714.749170 secs 2024-07-27T11:28:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:13 smithi029 ceph-mon[32610]: pgmap v9302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16714.749170 secs 2024-07-27T11:28:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:13 smithi077 ceph-mon[37153]: pgmap v9302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:15 smithi047 ceph-mon[36835]: pgmap v9303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:28:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:28:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:28:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:28:15.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:28:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:15 smithi029 ceph-mon[32610]: pgmap v9303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:28:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:28:15.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:28:15.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:28:15.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:28:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:15 smithi077 ceph-mon[37153]: pgmap v9303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:28:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:28:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:28:15.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:28:15.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:28:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:16 smithi047 ceph-mon[36835]: pgmap v9304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:28:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:16 smithi047 ceph-mon[36835]: pgmap v9305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:28:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:16 smithi029 ceph-mon[32610]: pgmap v9304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:28:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:16 smithi029 ceph-mon[32610]: pgmap v9305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:28:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:16 smithi077 ceph-mon[37153]: pgmap v9304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:28:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:16 smithi077 ceph-mon[37153]: pgmap v9305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:28:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:18 smithi047 ceph-mon[36835]: pgmap v9306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:28:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16719.749293 secs 2024-07-27T11:28:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:18 smithi029 ceph-mon[32610]: pgmap v9306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:28:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16719.749293 secs 2024-07-27T11:28:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:18 smithi077 ceph-mon[37153]: pgmap v9306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:28:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16719.749293 secs 2024-07-27T11:28:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:20 smithi047 ceph-mon[36835]: pgmap v9307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:28:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:20 smithi029 ceph-mon[32610]: pgmap v9307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:28:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:20 smithi077 ceph-mon[37153]: pgmap v9307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:28:21.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:28:21.197+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:28:21.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:28:21.197+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:28:21.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:28:21.197+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:28:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:22 smithi047 ceph-mon[36835]: pgmap v9308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:28:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:22 smithi029 ceph-mon[32610]: pgmap v9308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:28:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:22 smithi077 ceph-mon[37153]: pgmap v9308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:28:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16724.749450 secs 2024-07-27T11:28:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16724.749450 secs 2024-07-27T11:28:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16724.749450 secs 2024-07-27T11:28:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:24 smithi047 ceph-mon[36835]: pgmap v9309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:28:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:24 smithi029 ceph-mon[32610]: pgmap v9309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:28:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:24 smithi077 ceph-mon[37153]: pgmap v9309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:28:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:25 smithi047 ceph-mon[36835]: pgmap v9310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:28:25.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:25 smithi029 ceph-mon[32610]: pgmap v9310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:28:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:25 smithi077 ceph-mon[37153]: pgmap v9310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:28:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:27 smithi029 ceph-mon[32610]: pgmap v9311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16729.749574 secs 2024-07-27T11:28:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:27 smithi077 ceph-mon[37153]: pgmap v9311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16729.749574 secs 2024-07-27T11:28:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:27 smithi047 ceph-mon[36835]: pgmap v9311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16729.749574 secs 2024-07-27T11:28:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:29 smithi077 ceph-mon[37153]: pgmap v9312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:28:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:29 smithi047 ceph-mon[36835]: pgmap v9312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:28:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:29 smithi029 ceph-mon[32610]: pgmap v9312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:28:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:31 smithi029 ceph-mon[32610]: pgmap v9313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:28:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:31 smithi077 ceph-mon[37153]: pgmap v9313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:28:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:31 smithi047 ceph-mon[36835]: pgmap v9313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:28:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16734.749705 secs 2024-07-27T11:28:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16734.749705 secs 2024-07-27T11:28:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16734.749705 secs 2024-07-27T11:28:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:33 smithi029 ceph-mon[32610]: pgmap v9314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:28:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:33 smithi077 ceph-mon[37153]: pgmap v9314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:28:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:33 smithi047 ceph-mon[36835]: pgmap v9314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:28:35.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:35 smithi029 ceph-mon[32610]: pgmap v9315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:28:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:35 smithi077 ceph-mon[37153]: pgmap v9315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:28:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:35 smithi047 ceph-mon[36835]: pgmap v9315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:28:36.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:28:36.213+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:28:36.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:28:36.213+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:28:36.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:28:36.213+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:28:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:37 smithi077 ceph-mon[37153]: pgmap v9316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16739.749851 secs 2024-07-27T11:28:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:37 smithi047 ceph-mon[36835]: pgmap v9316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16739.749851 secs 2024-07-27T11:28:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:37 smithi029 ceph-mon[32610]: pgmap v9316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16739.749851 secs 2024-07-27T11:28:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:39 smithi077 ceph-mon[37153]: pgmap v9317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:28:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:39 smithi047 ceph-mon[36835]: pgmap v9317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:28:40.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:39 smithi029 ceph-mon[32610]: pgmap v9317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:28:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:41 smithi077 ceph-mon[37153]: pgmap v9318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:28:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:41 smithi047 ceph-mon[36835]: pgmap v9318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:28:42.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:41 smithi029 ceph-mon[32610]: pgmap v9318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:28:43.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16744.749996 secs 2024-07-27T11:28:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16744.749996 secs 2024-07-27T11:28:43.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16744.749996 secs 2024-07-27T11:28:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:43 smithi077 ceph-mon[37153]: pgmap v9319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:28:44.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:43 smithi047 ceph-mon[36835]: pgmap v9319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:28:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:43 smithi029 ceph-mon[32610]: pgmap v9319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:28:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:45 smithi077 ceph-mon[37153]: pgmap v9320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:28:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:45 smithi047 ceph-mon[36835]: pgmap v9320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:28:46.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:45 smithi029 ceph-mon[32610]: pgmap v9320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:28:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:47 smithi077 ceph-mon[37153]: pgmap v9321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16749.750118 secs 2024-07-27T11:28:48.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:47 smithi029 ceph-mon[32610]: pgmap v9321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:48.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16749.750118 secs 2024-07-27T11:28:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:47 smithi047 ceph-mon[36835]: pgmap v9321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16749.750118 secs 2024-07-27T11:28:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:49 smithi077 ceph-mon[37153]: pgmap v9322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:28:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:49 smithi047 ceph-mon[36835]: pgmap v9322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:28:50.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:49 smithi029 ceph-mon[32610]: pgmap v9322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:28:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:28:51.228+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:28:51.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:28:51.228+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:28:51.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:28:51.228+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:28:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:51 smithi077 ceph-mon[37153]: pgmap v9323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:28:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:51 smithi047 ceph-mon[36835]: pgmap v9323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:28:52.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:51 smithi029 ceph-mon[32610]: pgmap v9323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:28:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16754.750247 secs 2024-07-27T11:28:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16754.750247 secs 2024-07-27T11:28:53.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16754.750247 secs 2024-07-27T11:28:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:53 smithi077 ceph-mon[37153]: pgmap v9324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:28:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:53 smithi047 ceph-mon[36835]: pgmap v9324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:28:54.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:53 smithi029 ceph-mon[32610]: pgmap v9324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:28:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:55 smithi077 ceph-mon[37153]: pgmap v9325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:28:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:55 smithi047 ceph-mon[36835]: pgmap v9325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:28:56.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:55 smithi029 ceph-mon[32610]: pgmap v9325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:28:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:57 smithi077 ceph-mon[37153]: pgmap v9326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16759.750387 secs 2024-07-27T11:28:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:57 smithi047 ceph-mon[36835]: pgmap v9326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16759.750387 secs 2024-07-27T11:28:58.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:57 smithi029 ceph-mon[32610]: pgmap v9326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:28:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16759.750387 secs 2024-07-27T11:29:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:28:59 smithi077 ceph-mon[37153]: pgmap v9327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:00.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:00.183+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:29:00.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:00.183+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:29:00.185 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:28:59 smithi029 ceph-mon[32610]: pgmap v9327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:00.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:00.183+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:29:00.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:00.183+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:29:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:28:59 smithi047 ceph-mon[36835]: pgmap v9327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:01 smithi077 ceph-mon[37153]: pgmap v9328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:01 smithi047 ceph-mon[36835]: pgmap v9328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:01 smithi029 ceph-mon[32610]: pgmap v9328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16764.750510 secs 2024-07-27T11:29:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16764.750510 secs 2024-07-27T11:29:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16764.750510 secs 2024-07-27T11:29:04.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:03 smithi029 ceph-mon[32610]: pgmap v9329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:29:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:03 smithi077 ceph-mon[37153]: pgmap v9329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:29:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:03 smithi047 ceph-mon[36835]: pgmap v9329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:29:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:05 smithi077 ceph-mon[37153]: pgmap v9330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:29:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:05 smithi047 ceph-mon[36835]: pgmap v9330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:29:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:05 smithi029 ceph-mon[32610]: pgmap v9330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:29:06.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:06.237+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:29:06.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:06.237+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:29:06.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:06.237+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:29:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:07 smithi077 ceph-mon[37153]: pgmap v9331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16769.750612 secs 2024-07-27T11:29:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:07 smithi047 ceph-mon[36835]: pgmap v9331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16769.750612 secs 2024-07-27T11:29:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:07 smithi029 ceph-mon[32610]: pgmap v9331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16769.750612 secs 2024-07-27T11:29:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:09 smithi077 ceph-mon[37153]: pgmap v9332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:09 smithi047 ceph-mon[36835]: pgmap v9332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:10.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:09 smithi029 ceph-mon[32610]: pgmap v9332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:12.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:11 smithi077 ceph-mon[37153]: pgmap v9333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:11 smithi047 ceph-mon[36835]: pgmap v9333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:11 smithi029 ceph-mon[32610]: pgmap v9333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:13.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16774.750755 secs 2024-07-27T11:29:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16774.750755 secs 2024-07-27T11:29:13.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16774.750755 secs 2024-07-27T11:29:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:13 smithi077 ceph-mon[37153]: pgmap v9334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:29:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:13 smithi047 ceph-mon[36835]: pgmap v9334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:29:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:13 smithi029 ceph-mon[32610]: pgmap v9334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:29:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:29:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:29:15.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:29:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:15 smithi077 ceph-mon[37153]: pgmap v9335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:29:16.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:29:16.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:29:16.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:29:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:15 smithi047 ceph-mon[36835]: pgmap v9335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:29:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:29:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:29:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:29:16.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:15 smithi029 ceph-mon[32610]: pgmap v9335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:29:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:29:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:29:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:29:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:16 smithi077 ceph-mon[37153]: pgmap v9336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:29:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:16 smithi077 ceph-mon[37153]: pgmap v9337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:29:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:16 smithi047 ceph-mon[36835]: pgmap v9336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:29:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:16 smithi047 ceph-mon[36835]: pgmap v9337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:29:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:16 smithi029 ceph-mon[32610]: pgmap v9336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:29:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:16 smithi029 ceph-mon[32610]: pgmap v9337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:29:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16779.750891 secs 2024-07-27T11:29:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16779.750891 secs 2024-07-27T11:29:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16779.750891 secs 2024-07-27T11:29:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:18 smithi077 ceph-mon[37153]: pgmap v9338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:29:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:18 smithi047 ceph-mon[36835]: pgmap v9338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:29:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:18 smithi029 ceph-mon[32610]: pgmap v9338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:29:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:20 smithi077 ceph-mon[37153]: pgmap v9339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:29:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:20 smithi047 ceph-mon[36835]: pgmap v9339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:29:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:20 smithi029 ceph-mon[32610]: pgmap v9339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:29:21.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:21.253+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:29:21.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:21.253+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:29:21.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:21.253+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:29:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:22 smithi077 ceph-mon[37153]: pgmap v9340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:29:23.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16784.750962 secs 2024-07-27T11:29:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:22 smithi047 ceph-mon[36835]: pgmap v9340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:29:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16784.750962 secs 2024-07-27T11:29:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:22 smithi029 ceph-mon[32610]: pgmap v9340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:29:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16784.750962 secs 2024-07-27T11:29:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:24 smithi077 ceph-mon[37153]: pgmap v9341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:29:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:24 smithi047 ceph-mon[36835]: pgmap v9341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:29:25.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:24 smithi029 ceph-mon[32610]: pgmap v9341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:29:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:26 smithi077 ceph-mon[37153]: pgmap v9342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:29:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:26 smithi047 ceph-mon[36835]: pgmap v9342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:29:27.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:26 smithi029 ceph-mon[32610]: pgmap v9342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:29:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16789.751042 secs 2024-07-27T11:29:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16789.751042 secs 2024-07-27T11:29:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16789.751042 secs 2024-07-27T11:29:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:28 smithi077 ceph-mon[37153]: pgmap v9343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:28 smithi047 ceph-mon[36835]: pgmap v9343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:28 smithi029 ceph-mon[32610]: pgmap v9343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:30 smithi077 ceph-mon[37153]: pgmap v9344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:29:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:30 smithi047 ceph-mon[36835]: pgmap v9344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:29:31.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:30 smithi029 ceph-mon[32610]: pgmap v9344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:29:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:32 smithi077 ceph-mon[37153]: pgmap v9345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16794.751171 secs 2024-07-27T11:29:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:32 smithi047 ceph-mon[36835]: pgmap v9345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16794.751171 secs 2024-07-27T11:29:33.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:32 smithi029 ceph-mon[32610]: pgmap v9345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16794.751171 secs 2024-07-27T11:29:35.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:34 smithi077 ceph-mon[37153]: pgmap v9346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:34 smithi047 ceph-mon[36835]: pgmap v9346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:34 smithi029 ceph-mon[32610]: pgmap v9346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:36.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:36.268+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:29:36.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:36.268+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:29:36.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:36.268+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:29:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:36 smithi077 ceph-mon[37153]: pgmap v9347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:36 smithi047 ceph-mon[36835]: pgmap v9347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:37.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:36 smithi029 ceph-mon[32610]: pgmap v9347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16799.751310 secs 2024-07-27T11:29:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16799.751310 secs 2024-07-27T11:29:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16799.751310 secs 2024-07-27T11:29:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:38 smithi047 ceph-mon[36835]: pgmap v9348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:39.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:38 smithi029 ceph-mon[32610]: pgmap v9348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:38 smithi077 ceph-mon[37153]: pgmap v9348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:41.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:40 smithi047 ceph-mon[36835]: pgmap v9349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:29:41.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:40 smithi029 ceph-mon[32610]: pgmap v9349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:29:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:40 smithi077 ceph-mon[37153]: pgmap v9349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:29:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:42 smithi047 ceph-mon[36835]: pgmap v9350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16804.751435 secs 2024-07-27T11:29:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:42 smithi029 ceph-mon[32610]: pgmap v9350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16804.751435 secs 2024-07-27T11:29:43.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:42 smithi077 ceph-mon[37153]: pgmap v9350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:43.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16804.751435 secs 2024-07-27T11:29:45.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:44 smithi047 ceph-mon[36835]: pgmap v9351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:44 smithi029 ceph-mon[32610]: pgmap v9351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:44 smithi077 ceph-mon[37153]: pgmap v9351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:46 smithi047 ceph-mon[36835]: pgmap v9352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:47.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:46 smithi029 ceph-mon[32610]: pgmap v9352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:46 smithi077 ceph-mon[37153]: pgmap v9352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16809.751558 secs 2024-07-27T11:29:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16809.751558 secs 2024-07-27T11:29:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16809.751558 secs 2024-07-27T11:29:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:48 smithi047 ceph-mon[36835]: pgmap v9353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:49.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:48 smithi029 ceph-mon[32610]: pgmap v9353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:49.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:48 smithi077 ceph-mon[37153]: pgmap v9353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:51.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:50 smithi047 ceph-mon[36835]: pgmap v9354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:29:51.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:50 smithi029 ceph-mon[32610]: pgmap v9354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:29:51.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:51.274+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:29:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:51.274+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:29:51.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:29:51.274+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:29:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:50 smithi077 ceph-mon[37153]: pgmap v9354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:29:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:52 smithi047 ceph-mon[36835]: pgmap v9355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16814.751692 secs 2024-07-27T11:29:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:52 smithi029 ceph-mon[32610]: pgmap v9355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16814.751692 secs 2024-07-27T11:29:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:52 smithi077 ceph-mon[37153]: pgmap v9355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16814.751692 secs 2024-07-27T11:29:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:54 smithi047 ceph-mon[36835]: pgmap v9356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:55.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:54 smithi029 ceph-mon[32610]: pgmap v9356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:54 smithi077 ceph-mon[37153]: pgmap v9356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:29:57.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:56 smithi047 ceph-mon[36835]: pgmap v9357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:56 smithi029 ceph-mon[32610]: pgmap v9357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:56 smithi077 ceph-mon[37153]: pgmap v9357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:29:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16819.751867 secs 2024-07-27T11:29:58.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16819.751867 secs 2024-07-27T11:29:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16819.751867 secs 2024-07-27T11:29:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:29:58 smithi047 ceph-mon[36835]: pgmap v9358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:59.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:29:58 smithi029 ceph-mon[32610]: pgmap v9358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:29:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:29:58 smithi077 ceph-mon[37153]: pgmap v9358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:00 smithi047 ceph-mon[36835]: pgmap v9359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:30:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:30:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:00 smithi029 ceph-mon[32610]: pgmap v9359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:30:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:30:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:00 smithi077 ceph-mon[37153]: pgmap v9359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:30:01.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:30:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:02 smithi047 ceph-mon[36835]: pgmap v9360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16824.751992 secs 2024-07-27T11:30:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:02 smithi029 ceph-mon[32610]: pgmap v9360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16824.751992 secs 2024-07-27T11:30:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:02 smithi077 ceph-mon[37153]: pgmap v9360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16824.751992 secs 2024-07-27T11:30:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:04 smithi047 ceph-mon[36835]: pgmap v9361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:05.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:04 smithi029 ceph-mon[32610]: pgmap v9361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:04 smithi077 ceph-mon[37153]: pgmap v9361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:06.288+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:30:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:06.288+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:30:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:06.288+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:30:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:06 smithi047 ceph-mon[36835]: pgmap v9362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:07.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:06 smithi029 ceph-mon[32610]: pgmap v9362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:07.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:06 smithi077 ceph-mon[37153]: pgmap v9362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16829.752117 secs 2024-07-27T11:30:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16829.752117 secs 2024-07-27T11:30:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16829.752117 secs 2024-07-27T11:30:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:08 smithi047 ceph-mon[36835]: pgmap v9363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:08 smithi029 ceph-mon[32610]: pgmap v9363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:08 smithi077 ceph-mon[37153]: pgmap v9363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:10 smithi047 ceph-mon[36835]: pgmap v9364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:30:11.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:10 smithi029 ceph-mon[32610]: pgmap v9364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:30:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:10 smithi077 ceph-mon[37153]: pgmap v9364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:30:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:12 smithi047 ceph-mon[36835]: pgmap v9365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16834.752182 secs 2024-07-27T11:30:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:12 smithi029 ceph-mon[32610]: pgmap v9365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16834.752182 secs 2024-07-27T11:30:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:12 smithi077 ceph-mon[37153]: pgmap v9365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16834.752182 secs 2024-07-27T11:30:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:14 smithi047 ceph-mon[36835]: pgmap v9366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:15.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:14 smithi029 ceph-mon[32610]: pgmap v9366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:14 smithi077 ceph-mon[37153]: pgmap v9366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:30:16.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:16.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:15 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:30:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:15 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:30:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:16.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:15 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:16 smithi047 ceph-mon[36835]: pgmap v9367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:17.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:16 smithi029 ceph-mon[32610]: pgmap v9367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:16 smithi077 ceph-mon[37153]: pgmap v9367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16839.752254 secs 2024-07-27T11:30:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:30:18.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:30:18.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:18 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16839.752254 secs 2024-07-27T11:30:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:30:18.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:30:18.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:18 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16839.752254 secs 2024-07-27T11:30:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:18.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:30:18.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:30:18.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:18 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:30:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:19 smithi077 ceph-mon[37153]: pgmap v9368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:19 smithi077 ceph-mon[37153]: pgmap v9369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T11:30:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:19 smithi077 ceph-mon[37153]: pgmap v9370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:30:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:19 smithi047 ceph-mon[36835]: pgmap v9368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:19 smithi047 ceph-mon[36835]: pgmap v9369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T11:30:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:19 smithi047 ceph-mon[36835]: pgmap v9370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:30:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:19 smithi029 ceph-mon[32610]: pgmap v9368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:30:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:19 smithi029 ceph-mon[32610]: pgmap v9369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8 op/s 2024-07-27T11:30:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:19 smithi029 ceph-mon[32610]: pgmap v9370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:30:21.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:21.304+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:30:21.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:21.304+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:30:21.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:21.304+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:30:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:21 smithi047 ceph-mon[36835]: pgmap v9371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:30:21.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:21 smithi029 ceph-mon[32610]: pgmap v9371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:30:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:21 smithi077 ceph-mon[37153]: pgmap v9371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:30:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:23 smithi047 ceph-mon[36835]: pgmap v9372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:30:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16844.752384 secs 2024-07-27T11:30:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:23 smithi029 ceph-mon[32610]: pgmap v9372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:30:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16844.752384 secs 2024-07-27T11:30:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:23 smithi077 ceph-mon[37153]: pgmap v9372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:30:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16844.752384 secs 2024-07-27T11:30:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:25 smithi047 ceph-mon[36835]: pgmap v9373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:25.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:25 smithi029 ceph-mon[32610]: pgmap v9373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:25 smithi077 ceph-mon[37153]: pgmap v9373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:27 smithi047 ceph-mon[36835]: pgmap v9374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:30:27.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:27 smithi029 ceph-mon[32610]: pgmap v9374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:30:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:27 smithi077 ceph-mon[37153]: pgmap v9374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:30:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16849.752540 secs 2024-07-27T11:30:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16849.752540 secs 2024-07-27T11:30:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16849.752540 secs 2024-07-27T11:30:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:29 smithi047 ceph-mon[36835]: pgmap v9375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:30:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:29 smithi029 ceph-mon[32610]: pgmap v9375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:30:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:29 smithi077 ceph-mon[37153]: pgmap v9375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:30:30.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:30.198+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:30:30.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:30.198+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:30:30.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:30.198+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:30:30.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:30.198+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:30:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:31 smithi047 ceph-mon[36835]: pgmap v9376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:31 smithi029 ceph-mon[32610]: pgmap v9376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:31 smithi077 ceph-mon[37153]: pgmap v9376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:33 smithi047 ceph-mon[36835]: pgmap v9377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:30:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16854.752683 secs 2024-07-27T11:30:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:33 smithi029 ceph-mon[32610]: pgmap v9377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:30:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16854.752683 secs 2024-07-27T11:30:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:33 smithi077 ceph-mon[37153]: pgmap v9377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:30:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16854.752683 secs 2024-07-27T11:30:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:35 smithi047 ceph-mon[36835]: pgmap v9378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:35 smithi029 ceph-mon[32610]: pgmap v9378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:35 smithi077 ceph-mon[37153]: pgmap v9378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:36.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:36.318+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:30:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:36.318+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:30:36.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:36.318+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:30:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:37 smithi047 ceph-mon[36835]: pgmap v9379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:37 smithi029 ceph-mon[32610]: pgmap v9379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:37 smithi077 ceph-mon[37153]: pgmap v9379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16859.752819 secs 2024-07-27T11:30:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16859.752819 secs 2024-07-27T11:30:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16859.752819 secs 2024-07-27T11:30:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:39 smithi047 ceph-mon[36835]: pgmap v9380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:30:39.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:39 smithi029 ceph-mon[32610]: pgmap v9380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:30:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:39 smithi077 ceph-mon[37153]: pgmap v9380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:30:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:41 smithi047 ceph-mon[36835]: pgmap v9381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:41.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:41 smithi029 ceph-mon[32610]: pgmap v9381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:41 smithi077 ceph-mon[37153]: pgmap v9381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:43 smithi047 ceph-mon[36835]: pgmap v9382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:30:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16864.752944 secs 2024-07-27T11:30:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:43 smithi029 ceph-mon[32610]: pgmap v9382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:30:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16864.752944 secs 2024-07-27T11:30:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:43 smithi077 ceph-mon[37153]: pgmap v9382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:30:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16864.752944 secs 2024-07-27T11:30:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:45 smithi047 ceph-mon[36835]: pgmap v9383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:45 smithi029 ceph-mon[32610]: pgmap v9383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:45 smithi077 ceph-mon[37153]: pgmap v9383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:47 smithi047 ceph-mon[36835]: pgmap v9384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:47 smithi029 ceph-mon[32610]: pgmap v9384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:47 smithi077 ceph-mon[37153]: pgmap v9384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16869.753037 secs 2024-07-27T11:30:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16869.753037 secs 2024-07-27T11:30:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16869.753037 secs 2024-07-27T11:30:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:49 smithi047 ceph-mon[36835]: pgmap v9385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:30:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:49 smithi029 ceph-mon[32610]: pgmap v9385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:30:49.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:49 smithi077 ceph-mon[37153]: pgmap v9385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:30:51.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:51.333+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:30:51.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:51.333+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:30:51.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:30:51.333+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:30:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:51 smithi047 ceph-mon[36835]: pgmap v9386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:51 smithi029 ceph-mon[32610]: pgmap v9386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:51 smithi077 ceph-mon[37153]: pgmap v9386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:53 smithi047 ceph-mon[36835]: pgmap v9387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:30:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16874.753165 secs 2024-07-27T11:30:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:53 smithi029 ceph-mon[32610]: pgmap v9387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:30:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16874.753165 secs 2024-07-27T11:30:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:53 smithi077 ceph-mon[37153]: pgmap v9387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:30:53.609 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16874.753165 secs 2024-07-27T11:30:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:55 smithi047 ceph-mon[36835]: pgmap v9388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:55 smithi029 ceph-mon[32610]: pgmap v9388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:55 smithi077 ceph-mon[37153]: pgmap v9388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:30:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:57 smithi047 ceph-mon[36835]: pgmap v9389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:57 smithi029 ceph-mon[32610]: pgmap v9389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:57 smithi077 ceph-mon[37153]: pgmap v9389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:30:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16879.753315 secs 2024-07-27T11:30:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16879.753315 secs 2024-07-27T11:30:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16879.753315 secs 2024-07-27T11:30:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:30:59 smithi047 ceph-mon[36835]: pgmap v9390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:30:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:30:59 smithi029 ceph-mon[32610]: pgmap v9390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:30:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:30:59 smithi077 ceph-mon[37153]: pgmap v9390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:31:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:01 smithi047 ceph-mon[36835]: pgmap v9391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:01 smithi029 ceph-mon[32610]: pgmap v9391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:01 smithi077 ceph-mon[37153]: pgmap v9391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:03 smithi047 ceph-mon[36835]: pgmap v9392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:31:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16884.753448 secs 2024-07-27T11:31:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:03 smithi029 ceph-mon[32610]: pgmap v9392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:31:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16884.753448 secs 2024-07-27T11:31:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:03 smithi077 ceph-mon[37153]: pgmap v9392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:31:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16884.753448 secs 2024-07-27T11:31:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:05 smithi029 ceph-mon[32610]: pgmap v9393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:31:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:05 smithi077 ceph-mon[37153]: pgmap v9393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:31:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:05 smithi047 ceph-mon[36835]: pgmap v9393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:31:06.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:31:06.348+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:31:06.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:31:06.348+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:31:06.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:31:06.348+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:31:07.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:07 smithi029 ceph-mon[32610]: pgmap v9394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:07 smithi077 ceph-mon[37153]: pgmap v9394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:07 smithi047 ceph-mon[36835]: pgmap v9394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:08.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16889.753589 secs 2024-07-27T11:31:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16889.753589 secs 2024-07-27T11:31:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16889.753589 secs 2024-07-27T11:31:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:09 smithi029 ceph-mon[32610]: pgmap v9395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:31:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:09 smithi077 ceph-mon[37153]: pgmap v9395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:31:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:09 smithi047 ceph-mon[36835]: pgmap v9395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:31:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:10 smithi077 ceph-mon[37153]: pgmap v9396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:31:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:10 smithi047 ceph-mon[36835]: pgmap v9396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:31:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:10 smithi029 ceph-mon[32610]: pgmap v9396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:31:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:12 smithi077 ceph-mon[37153]: pgmap v9397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:31:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16894.753697 secs 2024-07-27T11:31:13.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:12 smithi029 ceph-mon[32610]: pgmap v9397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:31:13.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16894.753697 secs 2024-07-27T11:31:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:12 smithi047 ceph-mon[36835]: pgmap v9397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:31:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16894.753697 secs 2024-07-27T11:31:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:15 smithi077 ceph-mon[37153]: pgmap v9398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:31:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:14 smithi047 ceph-mon[36835]: pgmap v9398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:31:15.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:14 smithi029 ceph-mon[32610]: pgmap v9398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:31:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:17 smithi077 ceph-mon[37153]: pgmap v9399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:17 smithi047 ceph-mon[36835]: pgmap v9399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:17.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:17 smithi029 ceph-mon[32610]: pgmap v9399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16899.753827 secs 2024-07-27T11:31:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16899.753827 secs 2024-07-27T11:31:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16899.753827 secs 2024-07-27T11:31:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:19 smithi077 ceph-mon[37153]: pgmap v9400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:31:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:31:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:31:19.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:31:19.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:19 smithi047 ceph-mon[36835]: pgmap v9400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:31:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:31:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:31:19.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:31:19.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:19 smithi029 ceph-mon[32610]: pgmap v9400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:31:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:31:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:19.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:31:19.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:31:19.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:31:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:20 smithi077 ceph-mon[37153]: pgmap v9401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:31:20.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:20 smithi077 ceph-mon[37153]: pgmap v9402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:31:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:20 smithi047 ceph-mon[36835]: pgmap v9401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:31:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:20 smithi047 ceph-mon[36835]: pgmap v9402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:31:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:20 smithi029 ceph-mon[32610]: pgmap v9401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:31:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:20 smithi029 ceph-mon[32610]: pgmap v9402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:31:21.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:31:21.363+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:31:21.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:31:21.363+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:31:21.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:31:21.363+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:31:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:22 smithi077 ceph-mon[37153]: pgmap v9403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:31:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:22 smithi047 ceph-mon[36835]: pgmap v9403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:31:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:22 smithi029 ceph-mon[32610]: pgmap v9403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:31:23.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16904.753965 secs 2024-07-27T11:31:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16904.753965 secs 2024-07-27T11:31:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16904.753965 secs 2024-07-27T11:31:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:24 smithi077 ceph-mon[37153]: pgmap v9404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T11:31:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:24 smithi047 ceph-mon[36835]: pgmap v9404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T11:31:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:24 smithi029 ceph-mon[32610]: pgmap v9404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T11:31:26.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:26 smithi077 ceph-mon[37153]: pgmap v9405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:31:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:26 smithi047 ceph-mon[36835]: pgmap v9405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:31:26.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:26 smithi029 ceph-mon[32610]: pgmap v9405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:31:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:28 smithi077 ceph-mon[37153]: pgmap v9406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:31:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16909.754112 secs 2024-07-27T11:31:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:28 smithi047 ceph-mon[36835]: pgmap v9406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:31:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16909.754112 secs 2024-07-27T11:31:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:28 smithi029 ceph-mon[32610]: pgmap v9406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T11:31:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16909.754112 secs 2024-07-27T11:31:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:30 smithi077 ceph-mon[37153]: pgmap v9407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:31:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:30 smithi047 ceph-mon[36835]: pgmap v9407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:31:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:30 smithi029 ceph-mon[32610]: pgmap v9407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:31:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:32 smithi077 ceph-mon[37153]: pgmap v9408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:32 smithi047 ceph-mon[36835]: pgmap v9408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:32.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:32 smithi029 ceph-mon[32610]: pgmap v9408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16914.754185 secs 2024-07-27T11:31:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16914.754185 secs 2024-07-27T11:31:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16914.754185 secs 2024-07-27T11:31:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:34 smithi077 ceph-mon[37153]: pgmap v9409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:31:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:34 smithi047 ceph-mon[36835]: pgmap v9409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:31:34.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:34 smithi029 ceph-mon[32610]: pgmap v9409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:31:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:36 smithi077 ceph-mon[37153]: pgmap v9410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:31:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:31:36.379+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:31:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:31:36.379+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:31:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:31:36.379+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:31:36.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:36 smithi029 ceph-mon[32610]: pgmap v9410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:31:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:36 smithi047 ceph-mon[36835]: pgmap v9410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:31:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:38 smithi077 ceph-mon[37153]: pgmap v9411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:31:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16919.754311 secs 2024-07-27T11:31:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:38 smithi047 ceph-mon[36835]: pgmap v9411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:31:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16919.754311 secs 2024-07-27T11:31:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:38 smithi029 ceph-mon[32610]: pgmap v9411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:31:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16919.754311 secs 2024-07-27T11:31:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:40 smithi077 ceph-mon[37153]: pgmap v9412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:31:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:40 smithi047 ceph-mon[36835]: pgmap v9412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:31:40.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:40 smithi029 ceph-mon[32610]: pgmap v9412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:31:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:42 smithi077 ceph-mon[37153]: pgmap v9413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:42 smithi047 ceph-mon[36835]: pgmap v9413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:42 smithi029 ceph-mon[32610]: pgmap v9413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16924.754455 secs 2024-07-27T11:31:43.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16924.754455 secs 2024-07-27T11:31:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16924.754455 secs 2024-07-27T11:31:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:44 smithi077 ceph-mon[37153]: pgmap v9414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:31:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:44 smithi047 ceph-mon[36835]: pgmap v9414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:31:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:44 smithi029 ceph-mon[32610]: pgmap v9414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:31:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:46 smithi077 ceph-mon[37153]: pgmap v9415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:31:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:46 smithi047 ceph-mon[36835]: pgmap v9415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:31:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:46 smithi029 ceph-mon[32610]: pgmap v9415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:31:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:48 smithi077 ceph-mon[37153]: pgmap v9416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:31:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16929.754591 secs 2024-07-27T11:31:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:48 smithi047 ceph-mon[36835]: pgmap v9416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:31:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16929.754591 secs 2024-07-27T11:31:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:48 smithi029 ceph-mon[32610]: pgmap v9416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:31:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16929.754591 secs 2024-07-27T11:31:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:50 smithi077 ceph-mon[37153]: pgmap v9417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:31:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:50 smithi047 ceph-mon[36835]: pgmap v9417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:31:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:50 smithi029 ceph-mon[32610]: pgmap v9417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:31:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:31:51.395+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:31:51.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:31:51.395+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:31:51.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:31:51.395+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:31:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:52 smithi077 ceph-mon[37153]: pgmap v9418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:52 smithi047 ceph-mon[36835]: pgmap v9418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:52 smithi029 ceph-mon[32610]: pgmap v9418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:31:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16934.754684 secs 2024-07-27T11:31:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16934.754684 secs 2024-07-27T11:31:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16934.754684 secs 2024-07-27T11:31:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:54 smithi077 ceph-mon[37153]: pgmap v9419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:31:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:54 smithi047 ceph-mon[36835]: pgmap v9419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:31:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:54 smithi029 ceph-mon[32610]: pgmap v9419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:31:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:56 smithi077 ceph-mon[37153]: pgmap v9420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:31:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:56 smithi047 ceph-mon[36835]: pgmap v9420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:31:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:56 smithi029 ceph-mon[32610]: pgmap v9420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:31:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:58 smithi077 ceph-mon[37153]: pgmap v9421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:31:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:31:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16939.754834 secs 2024-07-27T11:31:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:58 smithi047 ceph-mon[36835]: pgmap v9421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:31:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:31:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16939.754834 secs 2024-07-27T11:31:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:58 smithi029 ceph-mon[32610]: pgmap v9421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:31:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:31:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16939.754834 secs 2024-07-27T11:32:00.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:00.211+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:32:00.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:00.211+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:32:00.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:00.215+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:32:00.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:00.216+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:32:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:00 smithi077 ceph-mon[37153]: pgmap v9422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:00 smithi047 ceph-mon[36835]: pgmap v9422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:00 smithi029 ceph-mon[32610]: pgmap v9422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:02 smithi077 ceph-mon[37153]: pgmap v9423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:32:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:02 smithi047 ceph-mon[36835]: pgmap v9423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:32:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:02 smithi029 ceph-mon[32610]: pgmap v9423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:32:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16944.754982 secs 2024-07-27T11:32:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16944.754982 secs 2024-07-27T11:32:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16944.754982 secs 2024-07-27T11:32:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:04 smithi047 ceph-mon[36835]: pgmap v9424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:32:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:04 smithi029 ceph-mon[32610]: pgmap v9424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:32:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:04 smithi077 ceph-mon[37153]: pgmap v9424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:32:06.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:06.403+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:32:06.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:06.403+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:32:06.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:06.403+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:32:06.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:06 smithi029 ceph-mon[32610]: pgmap v9425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:32:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:06 smithi047 ceph-mon[36835]: pgmap v9425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:32:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:06 smithi077 ceph-mon[37153]: pgmap v9425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:32:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:08 smithi047 ceph-mon[36835]: pgmap v9426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16949.755113 secs 2024-07-27T11:32:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:08 smithi029 ceph-mon[32610]: pgmap v9426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16949.755113 secs 2024-07-27T11:32:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:08 smithi077 ceph-mon[37153]: pgmap v9426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16949.755113 secs 2024-07-27T11:32:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:10 smithi047 ceph-mon[36835]: pgmap v9427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:10 smithi029 ceph-mon[32610]: pgmap v9427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:10 smithi077 ceph-mon[37153]: pgmap v9427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:12 smithi047 ceph-mon[36835]: pgmap v9428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:32:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:12 smithi029 ceph-mon[32610]: pgmap v9428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:32:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:12 smithi077 ceph-mon[37153]: pgmap v9428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:32:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16954.755239 secs 2024-07-27T11:32:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16954.755239 secs 2024-07-27T11:32:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16954.755239 secs 2024-07-27T11:32:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:14 smithi047 ceph-mon[36835]: pgmap v9429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:32:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:14 smithi029 ceph-mon[32610]: pgmap v9429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:32:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:14 smithi077 ceph-mon[37153]: pgmap v9429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:32:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:16 smithi047 ceph-mon[36835]: pgmap v9430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:32:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:16 smithi029 ceph-mon[32610]: pgmap v9430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:32:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:16 smithi077 ceph-mon[37153]: pgmap v9430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:32:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:18 smithi047 ceph-mon[36835]: pgmap v9431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16959.755373 secs 2024-07-27T11:32:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:18 smithi029 ceph-mon[32610]: pgmap v9431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16959.755373 secs 2024-07-27T11:32:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:18 smithi077 ceph-mon[37153]: pgmap v9431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16959.755373 secs 2024-07-27T11:32:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:19 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:32:19.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:19 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:32:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:19 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:32:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:20 smithi047 ceph-mon[36835]: pgmap v9432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:32:20.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:32:20.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:32:20.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:20 smithi029 ceph-mon[32610]: pgmap v9432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:32:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:32:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:32:20.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:20 smithi077 ceph-mon[37153]: pgmap v9432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:20.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:32:20.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:32:20.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:32:21.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:21.418+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:32:21.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:21.418+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:32:21.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:21.418+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:32:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:21 smithi029 ceph-mon[32610]: pgmap v9433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:32:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:21 smithi029 ceph-mon[32610]: pgmap v9434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:32:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:21 smithi047 ceph-mon[36835]: pgmap v9433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:32:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:21 smithi047 ceph-mon[36835]: pgmap v9434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:32:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:21 smithi077 ceph-mon[37153]: pgmap v9433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:32:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:21 smithi077 ceph-mon[37153]: pgmap v9434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:32:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:23 smithi047 ceph-mon[36835]: pgmap v9435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:32:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16964.755476 secs 2024-07-27T11:32:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:23 smithi029 ceph-mon[32610]: pgmap v9435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:32:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16964.755476 secs 2024-07-27T11:32:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:23 smithi077 ceph-mon[37153]: pgmap v9435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:32:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16964.755476 secs 2024-07-27T11:32:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:25 smithi047 ceph-mon[36835]: pgmap v9436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:32:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:25 smithi029 ceph-mon[32610]: pgmap v9436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:32:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:25 smithi077 ceph-mon[37153]: pgmap v9436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:32:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:27 smithi047 ceph-mon[36835]: pgmap v9437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:32:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:27 smithi029 ceph-mon[32610]: pgmap v9437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:32:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:27 smithi077 ceph-mon[37153]: pgmap v9437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:32:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16969.755587 secs 2024-07-27T11:32:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16969.755587 secs 2024-07-27T11:32:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16969.755587 secs 2024-07-27T11:32:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:29 smithi047 ceph-mon[36835]: pgmap v9438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T11:32:29.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:29 smithi029 ceph-mon[32610]: pgmap v9438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T11:32:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:29 smithi077 ceph-mon[37153]: pgmap v9438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T11:32:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:31 smithi047 ceph-mon[36835]: pgmap v9439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:32:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:31 smithi029 ceph-mon[32610]: pgmap v9439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:32:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:31 smithi077 ceph-mon[37153]: pgmap v9439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:32:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:33 smithi047 ceph-mon[36835]: pgmap v9440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16974.755718 secs 2024-07-27T11:32:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:33 smithi029 ceph-mon[32610]: pgmap v9440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16974.755718 secs 2024-07-27T11:32:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:33 smithi077 ceph-mon[37153]: pgmap v9440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16974.755718 secs 2024-07-27T11:32:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:35 smithi047 ceph-mon[36835]: pgmap v9441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:35.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:35 smithi029 ceph-mon[32610]: pgmap v9441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:35 smithi077 ceph-mon[37153]: pgmap v9441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:36.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:36.434+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:32:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:36.434+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:32:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:36.434+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:32:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:37 smithi047 ceph-mon[36835]: pgmap v9442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:32:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:37 smithi029 ceph-mon[32610]: pgmap v9442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:32:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:37 smithi077 ceph-mon[37153]: pgmap v9442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:32:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16979.755866 secs 2024-07-27T11:32:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16979.755866 secs 2024-07-27T11:32:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16979.755866 secs 2024-07-27T11:32:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:39 smithi047 ceph-mon[36835]: pgmap v9443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:32:39.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:39 smithi029 ceph-mon[32610]: pgmap v9443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:32:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:39 smithi077 ceph-mon[37153]: pgmap v9443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:32:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:41 smithi047 ceph-mon[36835]: pgmap v9444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:32:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:41 smithi029 ceph-mon[32610]: pgmap v9444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:32:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:41 smithi077 ceph-mon[37153]: pgmap v9444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:32:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:43 smithi047 ceph-mon[36835]: pgmap v9445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16984.755998 secs 2024-07-27T11:32:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:43 smithi029 ceph-mon[32610]: pgmap v9445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16984.755998 secs 2024-07-27T11:32:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:43 smithi077 ceph-mon[37153]: pgmap v9445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16984.755998 secs 2024-07-27T11:32:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:45 smithi029 ceph-mon[32610]: pgmap v9446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:45 smithi077 ceph-mon[37153]: pgmap v9446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:45 smithi047 ceph-mon[36835]: pgmap v9446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:47 smithi029 ceph-mon[32610]: pgmap v9447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:32:47.498 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:47 smithi047 ceph-mon[36835]: pgmap v9447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:32:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:47 smithi077 ceph-mon[37153]: pgmap v9447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:32:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16989.756085 secs 2024-07-27T11:32:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16989.756085 secs 2024-07-27T11:32:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16989.756085 secs 2024-07-27T11:32:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:49 smithi029 ceph-mon[32610]: pgmap v9448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:32:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:49 smithi077 ceph-mon[37153]: pgmap v9448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:32:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:49 smithi047 ceph-mon[36835]: pgmap v9448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:32:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:51.445+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:32:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:51.445+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:32:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:32:51.445+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:32:51.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:51 smithi029 ceph-mon[32610]: pgmap v9449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:32:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:51 smithi077 ceph-mon[37153]: pgmap v9449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:32:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:51 smithi047 ceph-mon[36835]: pgmap v9449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:32:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:53 smithi029 ceph-mon[32610]: pgmap v9450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16994.756170 secs 2024-07-27T11:32:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:53 smithi077 ceph-mon[37153]: pgmap v9450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16994.756170 secs 2024-07-27T11:32:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:53 smithi047 ceph-mon[36835]: pgmap v9450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:32:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16994.756170 secs 2024-07-27T11:32:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:55 smithi077 ceph-mon[37153]: pgmap v9451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:55 smithi047 ceph-mon[36835]: pgmap v9451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:55 smithi029 ceph-mon[32610]: pgmap v9451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:32:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:57 smithi077 ceph-mon[37153]: pgmap v9452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:32:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:57 smithi047 ceph-mon[36835]: pgmap v9452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:32:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:57 smithi029 ceph-mon[32610]: pgmap v9452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:32:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 16999.756295 secs 2024-07-27T11:32:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 16999.756295 secs 2024-07-27T11:32:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 16999.756295 secs 2024-07-27T11:32:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:32:59 smithi077 ceph-mon[37153]: pgmap v9453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:32:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:32:59 smithi047 ceph-mon[36835]: pgmap v9453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:32:59.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:32:59 smithi029 ceph-mon[32610]: pgmap v9453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:33:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:00 smithi077 ceph-mon[37153]: pgmap v9454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:33:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:00 smithi047 ceph-mon[36835]: pgmap v9454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:33:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:00 smithi029 ceph-mon[32610]: pgmap v9454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:33:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:02 smithi047 ceph-mon[36835]: pgmap v9455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17004.756439 secs 2024-07-27T11:33:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:02 smithi029 ceph-mon[32610]: pgmap v9455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17004.756439 secs 2024-07-27T11:33:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:02 smithi077 ceph-mon[37153]: pgmap v9455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17004.756439 secs 2024-07-27T11:33:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:04 smithi047 ceph-mon[36835]: pgmap v9456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:04 smithi029 ceph-mon[32610]: pgmap v9456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:04 smithi077 ceph-mon[37153]: pgmap v9456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:33:06.455+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:33:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:33:06.455+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:33:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:33:06.455+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:33:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:06 smithi047 ceph-mon[36835]: pgmap v9457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:06 smithi029 ceph-mon[32610]: pgmap v9457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:06 smithi077 ceph-mon[37153]: pgmap v9457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17009.756584 secs 2024-07-27T11:33:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17009.756584 secs 2024-07-27T11:33:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17009.756584 secs 2024-07-27T11:33:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:08 smithi047 ceph-mon[36835]: pgmap v9458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:33:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:08 smithi029 ceph-mon[32610]: pgmap v9458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:33:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:08 smithi077 ceph-mon[37153]: pgmap v9458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:33:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:10 smithi047 ceph-mon[36835]: pgmap v9459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:33:10.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:10 smithi029 ceph-mon[32610]: pgmap v9459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:33:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:10 smithi077 ceph-mon[37153]: pgmap v9459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:33:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:12 smithi047 ceph-mon[36835]: pgmap v9460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17014.756725 secs 2024-07-27T11:33:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:12 smithi029 ceph-mon[32610]: pgmap v9460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17014.756725 secs 2024-07-27T11:33:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:12 smithi077 ceph-mon[37153]: pgmap v9460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17014.756725 secs 2024-07-27T11:33:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:14 smithi047 ceph-mon[36835]: pgmap v9461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:14 smithi029 ceph-mon[32610]: pgmap v9461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:14 smithi077 ceph-mon[37153]: pgmap v9461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:16 smithi047 ceph-mon[36835]: pgmap v9462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:16 smithi029 ceph-mon[32610]: pgmap v9462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:16 smithi077 ceph-mon[37153]: pgmap v9462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17019.756841 secs 2024-07-27T11:33:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17019.756841 secs 2024-07-27T11:33:17.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17019.756841 secs 2024-07-27T11:33:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:18 smithi047 ceph-mon[36835]: pgmap v9463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:33:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:18 smithi029 ceph-mon[32610]: pgmap v9463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:33:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:18 smithi077 ceph-mon[37153]: pgmap v9463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:33:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:20 smithi047 ceph-mon[36835]: pgmap v9464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:33:20.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:33:20.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:33:20.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:33:20.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:20 smithi047 ceph-mon[36835]: pgmap v9465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:33:20.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:20 smithi047 ceph-mon[36835]: pgmap v9466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:33:20.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:33:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:20 smithi029 ceph-mon[32610]: pgmap v9464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:33:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:33:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:33:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:33:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:20 smithi029 ceph-mon[32610]: pgmap v9465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:33:20.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:20 smithi029 ceph-mon[32610]: pgmap v9466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:33:20.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:33:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:20 smithi077 ceph-mon[37153]: pgmap v9464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:33:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:33:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:33:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:33:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:20 smithi077 ceph-mon[37153]: pgmap v9465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:33:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:20 smithi077 ceph-mon[37153]: pgmap v9466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:33:20.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:33:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:33:21.461+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:33:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:33:21.461+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:33:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:33:21.461+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:33:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:22 smithi047 ceph-mon[36835]: pgmap v9467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:33:23.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17024.756964 secs 2024-07-27T11:33:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:22 smithi029 ceph-mon[32610]: pgmap v9467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:33:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17024.756964 secs 2024-07-27T11:33:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:22 smithi077 ceph-mon[37153]: pgmap v9467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:33:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17024.756964 secs 2024-07-27T11:33:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:24 smithi047 ceph-mon[36835]: pgmap v9468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:33:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:24 smithi029 ceph-mon[32610]: pgmap v9468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:33:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:24 smithi077 ceph-mon[37153]: pgmap v9468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:33:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:26 smithi047 ceph-mon[36835]: pgmap v9469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T11:33:27.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:26 smithi029 ceph-mon[32610]: pgmap v9469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T11:33:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:26 smithi077 ceph-mon[37153]: pgmap v9469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T11:33:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17029.757035 secs 2024-07-27T11:33:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17029.757035 secs 2024-07-27T11:33:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17029.757035 secs 2024-07-27T11:33:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:28 smithi047 ceph-mon[36835]: pgmap v9470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:33:29.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:28 smithi029 ceph-mon[32610]: pgmap v9470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:33:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:28 smithi077 ceph-mon[37153]: pgmap v9470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:33:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:30 smithi047 ceph-mon[36835]: pgmap v9471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:33:31.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:30 smithi029 ceph-mon[32610]: pgmap v9471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:33:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:30 smithi077 ceph-mon[37153]: pgmap v9471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:33:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:32 smithi029 ceph-mon[32610]: pgmap v9472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17034.757166 secs 2024-07-27T11:33:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:32 smithi077 ceph-mon[37153]: pgmap v9472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17034.757166 secs 2024-07-27T11:33:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:32 smithi047 ceph-mon[36835]: pgmap v9472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17034.757166 secs 2024-07-27T11:33:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:34 smithi029 ceph-mon[32610]: pgmap v9473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:34 smithi077 ceph-mon[37153]: pgmap v9473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:34 smithi047 ceph-mon[36835]: pgmap v9473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:33:36.477+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:33:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:33:36.477+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:33:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:33:36.477+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:33:37.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:36 smithi029 ceph-mon[32610]: pgmap v9474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:36 smithi077 ceph-mon[37153]: pgmap v9474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:36 smithi047 ceph-mon[36835]: pgmap v9474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17039.757293 secs 2024-07-27T11:33:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17039.757293 secs 2024-07-27T11:33:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17039.757293 secs 2024-07-27T11:33:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:38 smithi077 ceph-mon[37153]: pgmap v9475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:33:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:38 smithi047 ceph-mon[36835]: pgmap v9475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:33:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:38 smithi029 ceph-mon[32610]: pgmap v9475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:33:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:40 smithi077 ceph-mon[37153]: pgmap v9476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:33:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:40 smithi047 ceph-mon[36835]: pgmap v9476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:33:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:40 smithi029 ceph-mon[32610]: pgmap v9476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:33:43.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:42 smithi029 ceph-mon[32610]: pgmap v9477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:43.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17044.757417 secs 2024-07-27T11:33:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:42 smithi077 ceph-mon[37153]: pgmap v9477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17044.757417 secs 2024-07-27T11:33:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:42 smithi047 ceph-mon[36835]: pgmap v9477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17044.757417 secs 2024-07-27T11:33:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:44 smithi077 ceph-mon[37153]: pgmap v9478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:44 smithi047 ceph-mon[36835]: pgmap v9478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:44 smithi029 ceph-mon[32610]: pgmap v9478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:47.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:46 smithi047 ceph-mon[36835]: pgmap v9479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:46 smithi077 ceph-mon[37153]: pgmap v9479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:46 smithi029 ceph-mon[32610]: pgmap v9479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17049.757563 secs 2024-07-27T11:33:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17049.757563 secs 2024-07-27T11:33:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17049.757563 secs 2024-07-27T11:33:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:48 smithi077 ceph-mon[37153]: pgmap v9480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:33:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:48 smithi047 ceph-mon[36835]: pgmap v9480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:33:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:48 smithi029 ceph-mon[32610]: pgmap v9480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:33:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:50 smithi077 ceph-mon[37153]: pgmap v9481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:33:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:50 smithi047 ceph-mon[36835]: pgmap v9481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:33:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:50 smithi029 ceph-mon[32610]: pgmap v9481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:33:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:33:51.486+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:33:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:33:51.486+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:33:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:33:51.486+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:33:53.359 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:52 smithi077 ceph-mon[37153]: pgmap v9482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:53.360 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17054.757685 secs 2024-07-27T11:33:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:52 smithi047 ceph-mon[36835]: pgmap v9482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17054.757685 secs 2024-07-27T11:33:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:52 smithi029 ceph-mon[32610]: pgmap v9482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:33:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17054.757685 secs 2024-07-27T11:33:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:54 smithi077 ceph-mon[37153]: pgmap v9483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:54 smithi047 ceph-mon[36835]: pgmap v9483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:55.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:54 smithi029 ceph-mon[32610]: pgmap v9483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:33:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:56 smithi077 ceph-mon[37153]: pgmap v9484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:56 smithi047 ceph-mon[36835]: pgmap v9484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:56 smithi029 ceph-mon[32610]: pgmap v9484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:33:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17059.757835 secs 2024-07-27T11:33:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17059.757835 secs 2024-07-27T11:33:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17059.757835 secs 2024-07-27T11:33:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:33:59 smithi077 ceph-mon[37153]: pgmap v9485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:33:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:33:59 smithi047 ceph-mon[36835]: pgmap v9485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:33:59.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:33:59 smithi029 ceph-mon[32610]: pgmap v9485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:34:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:01 smithi077 ceph-mon[37153]: pgmap v9486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:01 smithi047 ceph-mon[36835]: pgmap v9486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:01 smithi029 ceph-mon[32610]: pgmap v9486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:03 smithi077 ceph-mon[37153]: pgmap v9487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17064.757911 secs 2024-07-27T11:34:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:03 smithi047 ceph-mon[36835]: pgmap v9487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17064.757911 secs 2024-07-27T11:34:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:03 smithi029 ceph-mon[32610]: pgmap v9487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17064.757911 secs 2024-07-27T11:34:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:05 smithi077 ceph-mon[37153]: pgmap v9488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:34:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:05 smithi047 ceph-mon[36835]: pgmap v9488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:34:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:05 smithi029 ceph-mon[32610]: pgmap v9488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:34:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:06.500+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:34:06.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:06.500+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:34:06.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:06.500+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:34:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:07 smithi077 ceph-mon[37153]: pgmap v9489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:34:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:07 smithi047 ceph-mon[36835]: pgmap v9489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:34:07.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:07 smithi029 ceph-mon[32610]: pgmap v9489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:34:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17069.757976 secs 2024-07-27T11:34:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17069.757976 secs 2024-07-27T11:34:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17069.757976 secs 2024-07-27T11:34:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:09 smithi077 ceph-mon[37153]: pgmap v9490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:34:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:09 smithi047 ceph-mon[36835]: pgmap v9490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:34:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:09 smithi029 ceph-mon[32610]: pgmap v9490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:34:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:11 smithi077 ceph-mon[37153]: pgmap v9491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:11 smithi047 ceph-mon[36835]: pgmap v9491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:11.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:11 smithi029 ceph-mon[32610]: pgmap v9491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:13 smithi077 ceph-mon[37153]: pgmap v9492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17074.758043 secs 2024-07-27T11:34:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:13 smithi047 ceph-mon[36835]: pgmap v9492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17074.758043 secs 2024-07-27T11:34:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:13 smithi029 ceph-mon[32610]: pgmap v9492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17074.758043 secs 2024-07-27T11:34:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:15 smithi029 ceph-mon[32610]: pgmap v9493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:34:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:15 smithi077 ceph-mon[37153]: pgmap v9493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:34:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:15 smithi047 ceph-mon[36835]: pgmap v9493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:34:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:17 smithi077 ceph-mon[37153]: pgmap v9494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:34:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:17 smithi047 ceph-mon[36835]: pgmap v9494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:34:17.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:17 smithi029 ceph-mon[32610]: pgmap v9494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:34:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17079.758148 secs 2024-07-27T11:34:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17079.758148 secs 2024-07-27T11:34:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17079.758148 secs 2024-07-27T11:34:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:19 smithi077 ceph-mon[37153]: pgmap v9495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:34:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:19 smithi047 ceph-mon[36835]: pgmap v9495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:34:19.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:19 smithi029 ceph-mon[32610]: pgmap v9495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:34:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:20 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:34:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:20 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:34:20.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:20 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:34:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:21 smithi077 ceph-mon[37153]: pgmap v9496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:34:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:34:21.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:34:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:21 smithi047 ceph-mon[36835]: pgmap v9496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:34:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:34:21.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:34:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:21 smithi029 ceph-mon[32610]: pgmap v9496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:34:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:34:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:34:21.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:21.509+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:34:21.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:21.509+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:34:21.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:21.509+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:34:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:22 smithi077 ceph-mon[37153]: pgmap v9497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:34:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:22 smithi077 ceph-mon[37153]: pgmap v9498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:34:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:22 smithi047 ceph-mon[36835]: pgmap v9497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:34:22.477 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:22 smithi047 ceph-mon[36835]: pgmap v9498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:34:22.477 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:22 smithi029 ceph-mon[32610]: pgmap v9497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:34:22.478 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:22 smithi029 ceph-mon[32610]: pgmap v9498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:34:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17084.758246 secs 2024-07-27T11:34:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17084.758246 secs 2024-07-27T11:34:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17084.758246 secs 2024-07-27T11:34:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:24 smithi077 ceph-mon[37153]: pgmap v9499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:34:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:24 smithi047 ceph-mon[36835]: pgmap v9499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:34:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:24 smithi029 ceph-mon[32610]: pgmap v9499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:34:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:26 smithi077 ceph-mon[37153]: pgmap v9500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:34:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:26 smithi047 ceph-mon[36835]: pgmap v9500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:34:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:26 smithi029 ceph-mon[32610]: pgmap v9500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:34:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:28 smithi077 ceph-mon[37153]: pgmap v9501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:34:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17089.758368 secs 2024-07-27T11:34:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:28 smithi047 ceph-mon[36835]: pgmap v9501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:34:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17089.758368 secs 2024-07-27T11:34:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:28 smithi029 ceph-mon[32610]: pgmap v9501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:34:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17089.758368 secs 2024-07-27T11:34:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:30 smithi077 ceph-mon[37153]: pgmap v9502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:34:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:30 smithi047 ceph-mon[36835]: pgmap v9502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:34:30.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:30 smithi029 ceph-mon[32610]: pgmap v9502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:34:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:32 smithi077 ceph-mon[37153]: pgmap v9503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:34:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:32 smithi047 ceph-mon[36835]: pgmap v9503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:34:32.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:32 smithi029 ceph-mon[32610]: pgmap v9503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:34:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17094.758510 secs 2024-07-27T11:34:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17094.758510 secs 2024-07-27T11:34:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17094.758510 secs 2024-07-27T11:34:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:34 smithi077 ceph-mon[37153]: pgmap v9504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:34:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:34 smithi047 ceph-mon[36835]: pgmap v9504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:34:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:34 smithi029 ceph-mon[32610]: pgmap v9504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:34:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:36 smithi077 ceph-mon[37153]: pgmap v9505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:36 smithi047 ceph-mon[36835]: pgmap v9505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:36.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:36 smithi029 ceph-mon[32610]: pgmap v9505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:36.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:36.525+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:34:36.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:36.525+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:34:36.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:36.525+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:34:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:38 smithi077 ceph-mon[37153]: pgmap v9506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17099.758655 secs 2024-07-27T11:34:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:38 smithi047 ceph-mon[36835]: pgmap v9506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:38.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17099.758655 secs 2024-07-27T11:34:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:38 smithi029 ceph-mon[32610]: pgmap v9506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17099.758655 secs 2024-07-27T11:34:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:40 smithi077 ceph-mon[37153]: pgmap v9507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:34:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:40 smithi047 ceph-mon[36835]: pgmap v9507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:34:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:40 smithi029 ceph-mon[32610]: pgmap v9507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:34:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:42 smithi077 ceph-mon[37153]: pgmap v9508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:34:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:42 smithi047 ceph-mon[36835]: pgmap v9508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:34:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:42 smithi029 ceph-mon[32610]: pgmap v9508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:34:43.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17104.758719 secs 2024-07-27T11:34:43.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17104.758719 secs 2024-07-27T11:34:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17104.758719 secs 2024-07-27T11:34:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:44 smithi077 ceph-mon[37153]: pgmap v9509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:34:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:44 smithi047 ceph-mon[36835]: pgmap v9509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:34:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:44 smithi029 ceph-mon[32610]: pgmap v9509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:34:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:46 smithi047 ceph-mon[36835]: pgmap v9510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:46.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:46 smithi029 ceph-mon[32610]: pgmap v9510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:46 smithi077 ceph-mon[37153]: pgmap v9510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:48 smithi047 ceph-mon[36835]: pgmap v9511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:48.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17109.758863 secs 2024-07-27T11:34:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:48 smithi029 ceph-mon[32610]: pgmap v9511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17109.758863 secs 2024-07-27T11:34:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:48 smithi077 ceph-mon[37153]: pgmap v9511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17109.758863 secs 2024-07-27T11:34:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:50 smithi047 ceph-mon[36835]: pgmap v9512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:34:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:50 smithi029 ceph-mon[32610]: pgmap v9512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:34:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:50 smithi077 ceph-mon[37153]: pgmap v9512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:34:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:51.534+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:34:51.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:51.534+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:34:51.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:51.534+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:34:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:52 smithi047 ceph-mon[36835]: pgmap v9513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:34:52.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:52 smithi029 ceph-mon[32610]: pgmap v9513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:34:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:52 smithi077 ceph-mon[37153]: pgmap v9513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:34:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17114.758986 secs 2024-07-27T11:34:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17114.758986 secs 2024-07-27T11:34:53.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17114.758986 secs 2024-07-27T11:34:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:54 smithi047 ceph-mon[36835]: pgmap v9514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:34:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:54 smithi029 ceph-mon[32610]: pgmap v9514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:34:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:54 smithi077 ceph-mon[37153]: pgmap v9514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:34:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:56 smithi047 ceph-mon[36835]: pgmap v9515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:56 smithi029 ceph-mon[32610]: pgmap v9515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:56 smithi077 ceph-mon[37153]: pgmap v9515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:34:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:57.669+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 msgr2=0x7eff3c0b5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T11:34:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:57.669+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 msgr2=0x7eff3c0b5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T11:34:57.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:57.669+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7eff40004800 tx=0x7eff4001e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T11:34:57.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:57.669+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7eff40004800 tx=0x7eff4001e040 comp rx=0 tx=0).stop 2024-07-27T11:34:57.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:57.669+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 msgr2=0x7eff3c0b5ed0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T11:34:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:57.669+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 msgr2=0x7eff3c0b5ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T11:34:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:57.669+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T11:34:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:57.669+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T11:34:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:57.669+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T11:34:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:34:57.670+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7eff40004800 tx=0x7eff4001e040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T11:34:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:58 smithi047 ceph-mon[36835]: pgmap v9516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:34:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17119.759110 secs 2024-07-27T11:34:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:58 smithi029 ceph-mon[32610]: pgmap v9516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:34:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17119.759110 secs 2024-07-27T11:34:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:58 smithi077 ceph-mon[37153]: pgmap v9516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:34:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:34:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17119.759110 secs 2024-07-27T11:35:00.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:00.237+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:35:00.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:00.238+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:35:00.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:00.241+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:35:00.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:00.241+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:35:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:00 smithi047 ceph-mon[36835]: pgmap v9517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:00 smithi029 ceph-mon[32610]: pgmap v9517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:00 smithi077 ceph-mon[37153]: pgmap v9517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:02 smithi029 ceph-mon[32610]: pgmap v9518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:02 smithi047 ceph-mon[36835]: pgmap v9518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:02 smithi077 ceph-mon[37153]: pgmap v9518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17124.759233 secs 2024-07-27T11:35:03.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17124.759233 secs 2024-07-27T11:35:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17124.759233 secs 2024-07-27T11:35:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:04 smithi047 ceph-mon[36835]: pgmap v9519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:35:04.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:04 smithi029 ceph-mon[32610]: pgmap v9519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:35:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:04 smithi077 ceph-mon[37153]: pgmap v9519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:35:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:06 smithi047 ceph-mon[36835]: pgmap v9520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:35:06.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:06 smithi029 ceph-mon[32610]: pgmap v9520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:35:06.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:06.541+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:35:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:06.541+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:35:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:06.541+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:35:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:06 smithi077 ceph-mon[37153]: pgmap v9520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:35:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:08 smithi047 ceph-mon[36835]: pgmap v9521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:35:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17129.759363 secs 2024-07-27T11:35:08.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:08 smithi029 ceph-mon[32610]: pgmap v9521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:35:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17129.759363 secs 2024-07-27T11:35:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:08 smithi077 ceph-mon[37153]: pgmap v9521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:35:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17129.759363 secs 2024-07-27T11:35:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:10 smithi047 ceph-mon[36835]: pgmap v9522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:10 smithi029 ceph-mon[32610]: pgmap v9522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:10 smithi077 ceph-mon[37153]: pgmap v9522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:12 smithi047 ceph-mon[36835]: pgmap v9523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:12.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:12 smithi029 ceph-mon[32610]: pgmap v9523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:12 smithi077 ceph-mon[37153]: pgmap v9523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17134.759485 secs 2024-07-27T11:35:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17134.759485 secs 2024-07-27T11:35:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17134.759485 secs 2024-07-27T11:35:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:14 smithi047 ceph-mon[36835]: pgmap v9524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:35:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:14 smithi029 ceph-mon[32610]: pgmap v9524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:35:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:14 smithi077 ceph-mon[37153]: pgmap v9524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:35:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:16 smithi047 ceph-mon[36835]: pgmap v9525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:35:16.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:16 smithi029 ceph-mon[32610]: pgmap v9525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:35:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:16 smithi077 ceph-mon[37153]: pgmap v9525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:35:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:18 smithi047 ceph-mon[36835]: pgmap v9526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:35:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17139.759583 secs 2024-07-27T11:35:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:18 smithi029 ceph-mon[32610]: pgmap v9526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:35:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17139.759583 secs 2024-07-27T11:35:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:18 smithi077 ceph-mon[37153]: pgmap v9526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:35:18.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17139.759583 secs 2024-07-27T11:35:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:20 smithi047 ceph-mon[36835]: pgmap v9527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:20.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:20 smithi029 ceph-mon[32610]: pgmap v9527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:20 smithi077 ceph-mon[37153]: pgmap v9527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:35:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:35:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:35:21.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:35:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:35:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:35:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:35:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:35:21.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:21.556+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:35:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:21.556+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:35:21.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:21.556+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:35:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:35:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:35:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:35:21.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:35:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:22 smithi047 ceph-mon[36835]: pgmap v9528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:35:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:22 smithi047 ceph-mon[36835]: pgmap v9529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:35:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:22 smithi047 ceph-mon[36835]: pgmap v9530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:35:22.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:22 smithi029 ceph-mon[32610]: pgmap v9528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:35:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:22 smithi029 ceph-mon[32610]: pgmap v9529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:35:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:22 smithi029 ceph-mon[32610]: pgmap v9530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:35:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:22 smithi077 ceph-mon[37153]: pgmap v9528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:35:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:22 smithi077 ceph-mon[37153]: pgmap v9529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:35:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:22 smithi077 ceph-mon[37153]: pgmap v9530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:35:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17144.759739 secs 2024-07-27T11:35:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17144.759739 secs 2024-07-27T11:35:23.507 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17144.759739 secs 2024-07-27T11:35:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:24 smithi047 ceph-mon[36835]: pgmap v9531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:35:24.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:24 smithi029 ceph-mon[32610]: pgmap v9531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:35:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:24 smithi077 ceph-mon[37153]: pgmap v9531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:35:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:26 smithi029 ceph-mon[32610]: pgmap v9532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:35:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:26 smithi077 ceph-mon[37153]: pgmap v9532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:35:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:26 smithi047 ceph-mon[36835]: pgmap v9532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:35:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:28 smithi029 ceph-mon[32610]: pgmap v9533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:35:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17149.759863 secs 2024-07-27T11:35:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:28 smithi077 ceph-mon[37153]: pgmap v9533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:35:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17149.759863 secs 2024-07-27T11:35:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:28 smithi047 ceph-mon[36835]: pgmap v9533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:35:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17149.759863 secs 2024-07-27T11:35:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:30 smithi029 ceph-mon[32610]: pgmap v9534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:35:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:30 smithi077 ceph-mon[37153]: pgmap v9534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:35:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:30 smithi047 ceph-mon[36835]: pgmap v9534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:35:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:32 smithi029 ceph-mon[32610]: pgmap v9535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:35:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:32 smithi077 ceph-mon[37153]: pgmap v9535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:35:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:32 smithi047 ceph-mon[36835]: pgmap v9535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:35:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17154.759946 secs 2024-07-27T11:35:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17154.759946 secs 2024-07-27T11:35:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17154.759946 secs 2024-07-27T11:35:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:34 smithi077 ceph-mon[37153]: pgmap v9536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:35:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:34 smithi047 ceph-mon[36835]: pgmap v9536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:35:34.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:34 smithi029 ceph-mon[32610]: pgmap v9536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:35:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:36.563+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:35:36.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:36.563+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:35:36.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:36.563+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:35:36.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:36 smithi029 ceph-mon[32610]: pgmap v9537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:36 smithi077 ceph-mon[37153]: pgmap v9537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:36 smithi047 ceph-mon[36835]: pgmap v9537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:38 smithi077 ceph-mon[37153]: pgmap v9538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:35:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17159.760044 secs 2024-07-27T11:35:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:38 smithi047 ceph-mon[36835]: pgmap v9538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:35:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17159.760044 secs 2024-07-27T11:35:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:38 smithi029 ceph-mon[32610]: pgmap v9538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:35:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17159.760044 secs 2024-07-27T11:35:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:40 smithi077 ceph-mon[37153]: pgmap v9539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:40 smithi047 ceph-mon[36835]: pgmap v9539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:40 smithi029 ceph-mon[32610]: pgmap v9539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:42 smithi077 ceph-mon[37153]: pgmap v9540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:42 smithi047 ceph-mon[36835]: pgmap v9540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:42.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:42 smithi029 ceph-mon[32610]: pgmap v9540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17164.760126 secs 2024-07-27T11:35:43.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17164.760126 secs 2024-07-27T11:35:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17164.760126 secs 2024-07-27T11:35:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:44 smithi077 ceph-mon[37153]: pgmap v9541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:35:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:44 smithi047 ceph-mon[36835]: pgmap v9541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:35:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:44 smithi029 ceph-mon[32610]: pgmap v9541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:35:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:46 smithi077 ceph-mon[37153]: pgmap v9542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:46 smithi047 ceph-mon[36835]: pgmap v9542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:46 smithi029 ceph-mon[32610]: pgmap v9542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:48.498 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:48 smithi047 ceph-mon[36835]: pgmap v9543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:35:48.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17169.760250 secs 2024-07-27T11:35:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:48 smithi077 ceph-mon[37153]: pgmap v9543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:35:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17169.760250 secs 2024-07-27T11:35:48.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:48 smithi029 ceph-mon[32610]: pgmap v9543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:35:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17169.760250 secs 2024-07-27T11:35:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:50 smithi077 ceph-mon[37153]: pgmap v9544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:50 smithi047 ceph-mon[36835]: pgmap v9544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:50.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:50 smithi029 ceph-mon[32610]: pgmap v9544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:35:51.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:51.579+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:35:51.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:51.579+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:35:51.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:35:51.579+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:35:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:52 smithi077 ceph-mon[37153]: pgmap v9545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:52 smithi047 ceph-mon[36835]: pgmap v9545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:52.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:52 smithi029 ceph-mon[32610]: pgmap v9545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:53.507 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17174.760373 secs 2024-07-27T11:35:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17174.760373 secs 2024-07-27T11:35:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17174.760373 secs 2024-07-27T11:35:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:54 smithi077 ceph-mon[37153]: pgmap v9546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:35:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:54 smithi047 ceph-mon[36835]: pgmap v9546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:35:54.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:54 smithi029 ceph-mon[32610]: pgmap v9546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:35:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:55 smithi077 ceph-mon[37153]: pgmap v9547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:55 smithi047 ceph-mon[36835]: pgmap v9547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:55.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:55 smithi029 ceph-mon[32610]: pgmap v9547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:35:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:58 smithi077 ceph-mon[37153]: pgmap v9548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:35:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:35:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17179.760523 secs 2024-07-27T11:35:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:58 smithi047 ceph-mon[36835]: pgmap v9548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:35:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:35:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17179.760523 secs 2024-07-27T11:35:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:58 smithi029 ceph-mon[32610]: pgmap v9548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:35:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:35:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17179.760523 secs 2024-07-27T11:36:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:00 smithi077 ceph-mon[37153]: pgmap v9549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:36:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:00 smithi047 ceph-mon[36835]: pgmap v9549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:36:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:00 smithi029 ceph-mon[32610]: pgmap v9549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:36:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:02 smithi077 ceph-mon[37153]: pgmap v9550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:36:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:02 smithi047 ceph-mon[36835]: pgmap v9550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:36:02.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:02 smithi029 ceph-mon[32610]: pgmap v9550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:36:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17184.760645 secs 2024-07-27T11:36:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17184.760645 secs 2024-07-27T11:36:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17184.760645 secs 2024-07-27T11:36:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:04 smithi029 ceph-mon[32610]: pgmap v9551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:36:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:04 smithi077 ceph-mon[37153]: pgmap v9551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:36:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:04 smithi047 ceph-mon[36835]: pgmap v9551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:36:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:06 smithi077 ceph-mon[37153]: pgmap v9552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:36:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:06 smithi047 ceph-mon[36835]: pgmap v9552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:36:06.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:06 smithi029 ceph-mon[32610]: pgmap v9552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:36:06.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:36:06.593+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:36:06.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:36:06.593+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:36:06.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:36:06.593+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:36:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:08 smithi077 ceph-mon[37153]: pgmap v9553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:36:08.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17189.760803 secs 2024-07-27T11:36:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:08 smithi047 ceph-mon[36835]: pgmap v9553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:36:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17189.760803 secs 2024-07-27T11:36:08.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:08 smithi029 ceph-mon[32610]: pgmap v9553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:36:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17189.760803 secs 2024-07-27T11:36:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:10 smithi077 ceph-mon[37153]: pgmap v9554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:36:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:10 smithi047 ceph-mon[36835]: pgmap v9554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:36:10.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:10 smithi029 ceph-mon[32610]: pgmap v9554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:36:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:12 smithi077 ceph-mon[37153]: pgmap v9555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:36:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:12 smithi047 ceph-mon[36835]: pgmap v9555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:36:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:12 smithi029 ceph-mon[32610]: pgmap v9555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:36:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17194.760924 secs 2024-07-27T11:36:13.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17194.760924 secs 2024-07-27T11:36:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17194.760924 secs 2024-07-27T11:36:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:14 smithi077 ceph-mon[37153]: pgmap v9556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:36:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:14 smithi047 ceph-mon[36835]: pgmap v9556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:36:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:14 smithi029 ceph-mon[32610]: pgmap v9556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:36:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:16 smithi077 ceph-mon[37153]: pgmap v9557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:36:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:16 smithi047 ceph-mon[36835]: pgmap v9557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:36:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:16 smithi029 ceph-mon[32610]: pgmap v9557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:36:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:18 smithi077 ceph-mon[37153]: pgmap v9558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:36:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17199.761047 secs 2024-07-27T11:36:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:18 smithi047 ceph-mon[36835]: pgmap v9558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:36:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17199.761047 secs 2024-07-27T11:36:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:18 smithi029 ceph-mon[32610]: pgmap v9558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:36:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17199.761047 secs 2024-07-27T11:36:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:20 smithi077 ceph-mon[37153]: pgmap v9559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:36:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:20 smithi047 ceph-mon[36835]: pgmap v9559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:36:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:20 smithi029 ceph-mon[32610]: pgmap v9559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:36:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:21 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:36:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:21 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:36:21.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:21 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:36:21.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:36:21.608+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:36:21.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:36:21.608+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:36:21.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:36:21.609+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:36:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:22 smithi047 ceph-mon[36835]: pgmap v9560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:36:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:36:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:36:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:36:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:36:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:36:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:36:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:36:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:22 smithi029 ceph-mon[32610]: pgmap v9560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:36:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:36:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:36:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:36:22.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:36:22.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:36:22.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:36:22.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:36:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:22 smithi077 ceph-mon[37153]: pgmap v9560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:36:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:36:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:36:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:36:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:36:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:36:22.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:36:22.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:36:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:23 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:36:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:23 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:36:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:23 smithi047 ceph-mon[36835]: pgmap v9561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:36:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:23 smithi047 ceph-mon[36835]: pgmap v9562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:36:23.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17204.761197 secs 2024-07-27T11:36:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:23 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:36:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:23 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:36:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:23 smithi029 ceph-mon[32610]: pgmap v9561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:36:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:23 smithi029 ceph-mon[32610]: pgmap v9562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:36:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17204.761197 secs 2024-07-27T11:36:23.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:23 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:36:23.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:23 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:36:23.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:23 smithi077 ceph-mon[37153]: pgmap v9561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:36:23.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:23 smithi077 ceph-mon[37153]: pgmap v9562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:36:23.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17204.761197 secs 2024-07-27T11:36:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:25 smithi047 ceph-mon[36835]: pgmap v9563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:36:25.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:25 smithi029 ceph-mon[32610]: pgmap v9563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:36:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:25 smithi077 ceph-mon[37153]: pgmap v9563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:36:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:27 smithi047 ceph-mon[36835]: pgmap v9564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:36:27.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:27 smithi029 ceph-mon[32610]: pgmap v9564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:36:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:27 smithi077 ceph-mon[37153]: pgmap v9564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:36:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17209.761319 secs 2024-07-27T11:36:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17209.761319 secs 2024-07-27T11:36:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17209.761319 secs 2024-07-27T11:36:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:29 smithi047 ceph-mon[36835]: pgmap v9565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:36:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:29 smithi029 ceph-mon[32610]: pgmap v9565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:36:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:29 smithi077 ceph-mon[37153]: pgmap v9565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:36:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:31 smithi047 ceph-mon[36835]: pgmap v9566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:36:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:31 smithi029 ceph-mon[32610]: pgmap v9566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:36:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:31 smithi077 ceph-mon[37153]: pgmap v9566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:36:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:33 smithi047 ceph-mon[36835]: pgmap v9567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:36:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17214.761396 secs 2024-07-27T11:36:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:33 smithi029 ceph-mon[32610]: pgmap v9567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:36:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17214.761396 secs 2024-07-27T11:36:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:33 smithi077 ceph-mon[37153]: pgmap v9567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:36:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17214.761396 secs 2024-07-27T11:36:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:35 smithi047 ceph-mon[36835]: pgmap v9568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:36:35.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:35 smithi029 ceph-mon[32610]: pgmap v9568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:36:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:35 smithi077 ceph-mon[37153]: pgmap v9568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:36:36.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:36:36.625+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:36:36.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:36:36.625+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:36:36.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:36:36.625+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:36:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:37 smithi047 ceph-mon[36835]: pgmap v9569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:36:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:37 smithi029 ceph-mon[32610]: pgmap v9569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:36:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:37 smithi077 ceph-mon[37153]: pgmap v9569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:36:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17219.761520 secs 2024-07-27T11:36:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17219.761520 secs 2024-07-27T11:36:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17219.761520 secs 2024-07-27T11:36:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:39 smithi047 ceph-mon[36835]: pgmap v9570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:36:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:39 smithi029 ceph-mon[32610]: pgmap v9570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:36:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:39 smithi077 ceph-mon[37153]: pgmap v9570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:36:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:41 smithi047 ceph-mon[36835]: pgmap v9571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:36:41.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:41 smithi029 ceph-mon[32610]: pgmap v9571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:36:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:41 smithi077 ceph-mon[37153]: pgmap v9571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:36:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:43 smithi047 ceph-mon[36835]: pgmap v9572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:36:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17224.761643 secs 2024-07-27T11:36:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:43 smithi029 ceph-mon[32610]: pgmap v9572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:36:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17224.761643 secs 2024-07-27T11:36:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:43 smithi077 ceph-mon[37153]: pgmap v9572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:36:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17224.761643 secs 2024-07-27T11:36:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:45 smithi047 ceph-mon[36835]: pgmap v9573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:36:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:45 smithi029 ceph-mon[32610]: pgmap v9573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:36:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:45 smithi077 ceph-mon[37153]: pgmap v9573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:36:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:47 smithi047 ceph-mon[36835]: pgmap v9574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:36:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:47 smithi029 ceph-mon[32610]: pgmap v9574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:36:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:47 smithi077 ceph-mon[37153]: pgmap v9574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:36:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17229.761765 secs 2024-07-27T11:36:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17229.761765 secs 2024-07-27T11:36:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17229.761765 secs 2024-07-27T11:36:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:49 smithi047 ceph-mon[36835]: pgmap v9575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:36:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:49 smithi029 ceph-mon[32610]: pgmap v9575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:36:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:49 smithi077 ceph-mon[37153]: pgmap v9575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:36:51.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:51 smithi029 ceph-mon[32610]: pgmap v9576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:36:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:51 smithi047 ceph-mon[36835]: pgmap v9576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:36:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:51 smithi077 ceph-mon[37153]: pgmap v9576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:36:51.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:36:51.630+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:36:51.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:36:51.630+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:36:51.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:36:51.630+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:36:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:53 smithi047 ceph-mon[36835]: pgmap v9577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:36:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17234.761885 secs 2024-07-27T11:36:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:53 smithi029 ceph-mon[32610]: pgmap v9577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:36:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17234.761885 secs 2024-07-27T11:36:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:53 smithi077 ceph-mon[37153]: pgmap v9577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:36:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17234.761885 secs 2024-07-27T11:36:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:55 smithi047 ceph-mon[36835]: pgmap v9578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:36:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:55 smithi029 ceph-mon[32610]: pgmap v9578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:36:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:55 smithi077 ceph-mon[37153]: pgmap v9578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:36:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:57 smithi047 ceph-mon[36835]: pgmap v9579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:36:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:57 smithi029 ceph-mon[32610]: pgmap v9579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:36:57.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:57 smithi077 ceph-mon[37153]: pgmap v9579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:36:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17239.762011 secs 2024-07-27T11:36:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17239.762011 secs 2024-07-27T11:36:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17239.762011 secs 2024-07-27T11:36:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:36:59 smithi047 ceph-mon[36835]: pgmap v9580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:36:59.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:36:59 smithi029 ceph-mon[32610]: pgmap v9580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:36:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:36:59 smithi077 ceph-mon[37153]: pgmap v9580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:37:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:01 smithi047 ceph-mon[36835]: pgmap v9581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:37:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:01 smithi029 ceph-mon[32610]: pgmap v9581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:37:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:01 smithi077 ceph-mon[37153]: pgmap v9581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:37:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:03 smithi029 ceph-mon[32610]: pgmap v9582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17244.762144 secs 2024-07-27T11:37:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:03 smithi077 ceph-mon[37153]: pgmap v9582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17244.762144 secs 2024-07-27T11:37:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:03 smithi047 ceph-mon[36835]: pgmap v9582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17244.762144 secs 2024-07-27T11:37:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:05 smithi029 ceph-mon[32610]: pgmap v9583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:37:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:05 smithi077 ceph-mon[37153]: pgmap v9583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:37:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:05 smithi047 ceph-mon[36835]: pgmap v9583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:37:06.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:37:06.633+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:37:06.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:37:06.633+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:37:06.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:37:06.633+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:37:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:07 smithi029 ceph-mon[32610]: pgmap v9584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:07 smithi077 ceph-mon[37153]: pgmap v9584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:07 smithi047 ceph-mon[36835]: pgmap v9584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17249.762266 secs 2024-07-27T11:37:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17249.762266 secs 2024-07-27T11:37:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17249.762266 secs 2024-07-27T11:37:09.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:09 smithi029 ceph-mon[32610]: pgmap v9585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:37:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:09 smithi077 ceph-mon[37153]: pgmap v9585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:37:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:09 smithi047 ceph-mon[36835]: pgmap v9585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:37:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:11 smithi077 ceph-mon[37153]: pgmap v9586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:37:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:11 smithi047 ceph-mon[36835]: pgmap v9586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:37:11.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:11 smithi029 ceph-mon[32610]: pgmap v9586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:37:13.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:13 smithi029 ceph-mon[32610]: pgmap v9587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:13.515 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17254.762394 secs 2024-07-27T11:37:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:13 smithi077 ceph-mon[37153]: pgmap v9587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17254.762394 secs 2024-07-27T11:37:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:13 smithi047 ceph-mon[36835]: pgmap v9587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17254.762394 secs 2024-07-27T11:37:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:15 smithi077 ceph-mon[37153]: pgmap v9588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:37:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:15 smithi047 ceph-mon[36835]: pgmap v9588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:37:15.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:15 smithi029 ceph-mon[32610]: pgmap v9588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:37:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:17 smithi077 ceph-mon[37153]: pgmap v9589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:17 smithi047 ceph-mon[36835]: pgmap v9589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:17 smithi029 ceph-mon[32610]: pgmap v9589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:18.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17259.762496 secs 2024-07-27T11:37:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17259.762496 secs 2024-07-27T11:37:18.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17259.762496 secs 2024-07-27T11:37:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:19 smithi077 ceph-mon[37153]: pgmap v9590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:37:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:19 smithi047 ceph-mon[36835]: pgmap v9590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:37:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:19 smithi029 ceph-mon[32610]: pgmap v9590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:37:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:21 smithi029 ceph-mon[32610]: pgmap v9591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:37:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:21 smithi077 ceph-mon[37153]: pgmap v9591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:37:21.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:37:21.648+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:37:21.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:37:21.648+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:37:21.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:37:21.648+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:37:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:21 smithi047 ceph-mon[36835]: pgmap v9591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:37:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:37:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:37:22.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:37:22.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:37:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:37:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:37:22.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:37:22.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:37:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:37:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:37:22.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:37:22.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:37:22.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:37:23.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:23 smithi077 ceph-mon[37153]: pgmap v9592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:23.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17264.762623 secs 2024-07-27T11:37:23.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:23 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:37:23.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:23 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:37:23.509 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:23 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:37:23.509 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:23 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:37:23.509 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:23 smithi077 ceph-mon[37153]: pgmap v9593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:37:23.509 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:23 smithi077 ceph-mon[37153]: pgmap v9594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:37:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:23 smithi047 ceph-mon[36835]: pgmap v9592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17264.762623 secs 2024-07-27T11:37:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:23 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:37:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:23 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:37:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:23 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:37:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:23 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:37:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:23 smithi047 ceph-mon[36835]: pgmap v9593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:37:23.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:23 smithi047 ceph-mon[36835]: pgmap v9594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:37:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:23 smithi029 ceph-mon[32610]: pgmap v9592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17264.762623 secs 2024-07-27T11:37:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:23 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:37:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:23 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:37:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:23 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:37:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:23 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:37:23.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:23 smithi029 ceph-mon[32610]: pgmap v9593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:37:23.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:23 smithi029 ceph-mon[32610]: pgmap v9594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:37:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:25 smithi077 ceph-mon[37153]: pgmap v9595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:37:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:25 smithi047 ceph-mon[36835]: pgmap v9595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:37:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:25 smithi029 ceph-mon[32610]: pgmap v9595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:37:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:27 smithi077 ceph-mon[37153]: pgmap v9596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:37:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:27 smithi047 ceph-mon[36835]: pgmap v9596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:37:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:27 smithi029 ceph-mon[32610]: pgmap v9596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:37:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17269.762746 secs 2024-07-27T11:37:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17269.762746 secs 2024-07-27T11:37:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17269.762746 secs 2024-07-27T11:37:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:29 smithi077 ceph-mon[37153]: pgmap v9597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:37:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:29 smithi047 ceph-mon[36835]: pgmap v9597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:37:29.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:29 smithi029 ceph-mon[32610]: pgmap v9597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:37:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:31 smithi077 ceph-mon[37153]: pgmap v9598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:37:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:31 smithi047 ceph-mon[36835]: pgmap v9598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:37:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:31 smithi029 ceph-mon[32610]: pgmap v9598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:37:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:32 smithi077 ceph-mon[37153]: pgmap v9599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:37:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:32 smithi047 ceph-mon[36835]: pgmap v9599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:37:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:32 smithi029 ceph-mon[32610]: pgmap v9599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:37:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17274.762900 secs 2024-07-27T11:37:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17274.762900 secs 2024-07-27T11:37:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17274.762900 secs 2024-07-27T11:37:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:34 smithi077 ceph-mon[37153]: pgmap v9600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:37:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:34 smithi047 ceph-mon[36835]: pgmap v9600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:37:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:34 smithi029 ceph-mon[32610]: pgmap v9600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:37:36.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:37:36.664+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:37:36.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:37:36.664+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:37:36.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:37:36.664+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:37:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:37 smithi047 ceph-mon[36835]: pgmap v9601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:37 smithi029 ceph-mon[32610]: pgmap v9601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:37 smithi077 ceph-mon[37153]: pgmap v9601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17279.763039 secs 2024-07-27T11:37:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17279.763039 secs 2024-07-27T11:37:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17279.763039 secs 2024-07-27T11:37:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:39 smithi047 ceph-mon[36835]: pgmap v9602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:37:39.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:39 smithi029 ceph-mon[32610]: pgmap v9602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:37:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:39 smithi077 ceph-mon[37153]: pgmap v9602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:37:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:41 smithi047 ceph-mon[36835]: pgmap v9603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:37:41.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:41 smithi029 ceph-mon[32610]: pgmap v9603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:37:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:41 smithi077 ceph-mon[37153]: pgmap v9603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:37:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17284.763175 secs 2024-07-27T11:37:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:43 smithi047 ceph-mon[36835]: pgmap v9604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17284.763175 secs 2024-07-27T11:37:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:43 smithi029 ceph-mon[32610]: pgmap v9604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17284.763175 secs 2024-07-27T11:37:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:43 smithi077 ceph-mon[37153]: pgmap v9604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:45 smithi047 ceph-mon[36835]: pgmap v9605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:37:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:45 smithi029 ceph-mon[32610]: pgmap v9605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:37:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:45 smithi077 ceph-mon[37153]: pgmap v9605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:37:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:47 smithi047 ceph-mon[36835]: pgmap v9606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:37:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:47 smithi029 ceph-mon[32610]: pgmap v9606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:37:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:47 smithi077 ceph-mon[37153]: pgmap v9606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:37:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17289.763301 secs 2024-07-27T11:37:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17289.763301 secs 2024-07-27T11:37:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17289.763301 secs 2024-07-27T11:37:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:49 smithi047 ceph-mon[36835]: pgmap v9607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:37:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:49 smithi029 ceph-mon[32610]: pgmap v9607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:37:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:49 smithi077 ceph-mon[37153]: pgmap v9607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:37:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:51 smithi029 ceph-mon[32610]: pgmap v9608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:51 smithi077 ceph-mon[37153]: pgmap v9608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:51.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:37:51.678+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:37:51.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:37:51.678+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:37:51.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:37:51.678+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:37:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:51 smithi047 ceph-mon[36835]: pgmap v9608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17294.763424 secs 2024-07-27T11:37:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:53 smithi029 ceph-mon[32610]: pgmap v9609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17294.763424 secs 2024-07-27T11:37:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:53 smithi077 ceph-mon[37153]: pgmap v9609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17294.763424 secs 2024-07-27T11:37:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:53 smithi047 ceph-mon[36835]: pgmap v9609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:37:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:55 smithi029 ceph-mon[32610]: pgmap v9610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:37:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:55 smithi077 ceph-mon[37153]: pgmap v9610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:37:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:55 smithi047 ceph-mon[36835]: pgmap v9610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:37:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:57 smithi029 ceph-mon[32610]: pgmap v9611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:57 smithi077 ceph-mon[37153]: pgmap v9611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:57 smithi047 ceph-mon[36835]: pgmap v9611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:37:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17299.763547 secs 2024-07-27T11:37:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17299.763547 secs 2024-07-27T11:37:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17299.763547 secs 2024-07-27T11:37:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:37:59 smithi077 ceph-mon[37153]: pgmap v9612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:37:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:37:59 smithi047 ceph-mon[36835]: pgmap v9612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:37:59.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:37:59 smithi029 ceph-mon[32610]: pgmap v9612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:38:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:00.264+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:38:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:00.264+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:38:00.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:00.265+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:38:00.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:00.265+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:38:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:01 smithi077 ceph-mon[37153]: pgmap v9613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:01 smithi047 ceph-mon[36835]: pgmap v9613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:01.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:01 smithi029 ceph-mon[32610]: pgmap v9613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17304.763669 secs 2024-07-27T11:38:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:03 smithi077 ceph-mon[37153]: pgmap v9614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17304.763669 secs 2024-07-27T11:38:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:03 smithi047 ceph-mon[36835]: pgmap v9614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17304.763669 secs 2024-07-27T11:38:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:03 smithi029 ceph-mon[32610]: pgmap v9614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:05 smithi077 ceph-mon[37153]: pgmap v9615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:38:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:05 smithi047 ceph-mon[36835]: pgmap v9615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:38:05.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:05 smithi029 ceph-mon[32610]: pgmap v9615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:38:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:06.679+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:38:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:06.679+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:38:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:06.679+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:38:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:07 smithi077 ceph-mon[37153]: pgmap v9616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:07 smithi047 ceph-mon[36835]: pgmap v9616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:07 smithi029 ceph-mon[32610]: pgmap v9616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17309.763811 secs 2024-07-27T11:38:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17309.763811 secs 2024-07-27T11:38:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17309.763811 secs 2024-07-27T11:38:09.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:09 smithi077 ceph-mon[37153]: pgmap v9617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:38:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:09 smithi047 ceph-mon[36835]: pgmap v9617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:38:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:09 smithi029 ceph-mon[32610]: pgmap v9617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:38:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:11 smithi077 ceph-mon[37153]: pgmap v9618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:11 smithi047 ceph-mon[36835]: pgmap v9618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:11.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:11 smithi029 ceph-mon[32610]: pgmap v9618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:13.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17314.763937 secs 2024-07-27T11:38:13.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:13 smithi029 ceph-mon[32610]: pgmap v9619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17314.763937 secs 2024-07-27T11:38:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:13 smithi077 ceph-mon[37153]: pgmap v9619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17314.763937 secs 2024-07-27T11:38:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:13 smithi047 ceph-mon[36835]: pgmap v9619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:15 smithi077 ceph-mon[37153]: pgmap v9620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:38:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:15 smithi047 ceph-mon[36835]: pgmap v9620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:38:15.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:15 smithi029 ceph-mon[32610]: pgmap v9620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:38:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:17 smithi077 ceph-mon[37153]: pgmap v9621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:17 smithi047 ceph-mon[36835]: pgmap v9621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:17 smithi029 ceph-mon[32610]: pgmap v9621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:18.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17319.764062 secs 2024-07-27T11:38:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17319.764062 secs 2024-07-27T11:38:18.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17319.764062 secs 2024-07-27T11:38:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:19 smithi077 ceph-mon[37153]: pgmap v9622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:38:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:19 smithi047 ceph-mon[36835]: pgmap v9622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:38:19.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:19 smithi029 ceph-mon[32610]: pgmap v9622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:38:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:21 smithi077 ceph-mon[37153]: pgmap v9623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:21.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:21.681+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:38:21.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:21.681+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:38:21.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:21.681+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:38:21.683 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:21 smithi029 ceph-mon[32610]: pgmap v9623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:21 smithi047 ceph-mon[36835]: pgmap v9623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:38:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:38:22.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:38:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17324.764187 secs 2024-07-27T11:38:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:23 smithi077 ceph-mon[37153]: pgmap v9624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:38:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:38:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:38:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17324.764187 secs 2024-07-27T11:38:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:23 smithi047 ceph-mon[36835]: pgmap v9624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:38:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:38:23.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:38:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17324.764187 secs 2024-07-27T11:38:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:23 smithi029 ceph-mon[32610]: pgmap v9624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:38:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:38:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:38:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:24 smithi029 ceph-mon[32610]: pgmap v9625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T11:38:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:24 smithi029 ceph-mon[32610]: pgmap v9626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:38:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:24 smithi077 ceph-mon[37153]: pgmap v9625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T11:38:24.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:24 smithi077 ceph-mon[37153]: pgmap v9626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:38:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:24 smithi047 ceph-mon[36835]: pgmap v9625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T11:38:24.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:24 smithi047 ceph-mon[36835]: pgmap v9626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:38:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:25 smithi077 ceph-mon[37153]: pgmap v9627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:38:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:25 smithi047 ceph-mon[36835]: pgmap v9627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:38:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:25 smithi029 ceph-mon[32610]: pgmap v9627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:38:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:27 smithi077 ceph-mon[37153]: pgmap v9628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:38:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17329.764328 secs 2024-07-27T11:38:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:27 smithi047 ceph-mon[36835]: pgmap v9628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:38:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17329.764328 secs 2024-07-27T11:38:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:27 smithi029 ceph-mon[32610]: pgmap v9628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:38:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17329.764328 secs 2024-07-27T11:38:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:29 smithi077 ceph-mon[37153]: pgmap v9629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:38:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:29 smithi047 ceph-mon[36835]: pgmap v9629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:38:30.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:29 smithi029 ceph-mon[32610]: pgmap v9629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:38:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:31 smithi077 ceph-mon[37153]: pgmap v9630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:38:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:31 smithi047 ceph-mon[36835]: pgmap v9630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:38:32.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:31 smithi029 ceph-mon[32610]: pgmap v9630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:38:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17334.764467 secs 2024-07-27T11:38:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17334.764467 secs 2024-07-27T11:38:33.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17334.764467 secs 2024-07-27T11:38:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:33 smithi077 ceph-mon[37153]: pgmap v9631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:38:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:33 smithi047 ceph-mon[36835]: pgmap v9631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:38:34.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:33 smithi029 ceph-mon[32610]: pgmap v9631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:38:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:35 smithi077 ceph-mon[37153]: pgmap v9632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:38:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:35 smithi047 ceph-mon[36835]: pgmap v9632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:38:36.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:35 smithi029 ceph-mon[32610]: pgmap v9632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:38:36.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:36.696+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:38:36.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:36.696+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:38:36.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:36.696+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:38:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:37 smithi077 ceph-mon[37153]: pgmap v9633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:38.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17339.764603 secs 2024-07-27T11:38:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:37 smithi047 ceph-mon[36835]: pgmap v9633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17339.764603 secs 2024-07-27T11:38:38.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:37 smithi029 ceph-mon[32610]: pgmap v9633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17339.764603 secs 2024-07-27T11:38:40.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:39 smithi029 ceph-mon[32610]: pgmap v9634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:38:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:39 smithi077 ceph-mon[37153]: pgmap v9634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:38:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:39 smithi047 ceph-mon[36835]: pgmap v9634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:38:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:41 smithi077 ceph-mon[37153]: pgmap v9635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:41 smithi047 ceph-mon[36835]: pgmap v9635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:42.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:41 smithi029 ceph-mon[32610]: pgmap v9635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17344.764730 secs 2024-07-27T11:38:43.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17344.764730 secs 2024-07-27T11:38:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17344.764730 secs 2024-07-27T11:38:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:43 smithi077 ceph-mon[37153]: pgmap v9636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:38:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:43 smithi047 ceph-mon[36835]: pgmap v9636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:38:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:43 smithi029 ceph-mon[32610]: pgmap v9636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:38:45.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:45.272+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:38:45.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:45.272+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:38:45.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:45.272+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:38:45.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:45.272+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:38:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:45 smithi077 ceph-mon[37153]: pgmap v9637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:38:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:45 smithi047 ceph-mon[36835]: pgmap v9637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:38:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:45 smithi029 ceph-mon[32610]: pgmap v9637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:38:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:47 smithi077 ceph-mon[37153]: pgmap v9638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17349.764817 secs 2024-07-27T11:38:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:47 smithi047 ceph-mon[36835]: pgmap v9638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17349.764817 secs 2024-07-27T11:38:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:47 smithi029 ceph-mon[32610]: pgmap v9638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17349.764817 secs 2024-07-27T11:38:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:49 smithi077 ceph-mon[37153]: pgmap v9639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:38:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:49 smithi047 ceph-mon[36835]: pgmap v9639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:38:50.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:49 smithi029 ceph-mon[32610]: pgmap v9639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:38:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:51.712+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:38:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:51.712+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:38:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:38:51.712+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:38:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:51 smithi077 ceph-mon[37153]: pgmap v9640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:51 smithi047 ceph-mon[36835]: pgmap v9640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:52.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:51 smithi029 ceph-mon[32610]: pgmap v9640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:38:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17354.764938 secs 2024-07-27T11:38:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17354.764938 secs 2024-07-27T11:38:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17354.764938 secs 2024-07-27T11:38:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:53 smithi077 ceph-mon[37153]: pgmap v9641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:38:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:53 smithi047 ceph-mon[36835]: pgmap v9641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:38:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:53 smithi029 ceph-mon[32610]: pgmap v9641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:38:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:55 smithi077 ceph-mon[37153]: pgmap v9642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:38:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:55 smithi047 ceph-mon[36835]: pgmap v9642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:38:56.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:55 smithi029 ceph-mon[32610]: pgmap v9642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:38:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:57 smithi077 ceph-mon[37153]: pgmap v9643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:58.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17359.765060 secs 2024-07-27T11:38:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:57 smithi047 ceph-mon[36835]: pgmap v9643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17359.765060 secs 2024-07-27T11:38:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:57 smithi029 ceph-mon[32610]: pgmap v9643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:38:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17359.765060 secs 2024-07-27T11:39:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:38:59 smithi077 ceph-mon[37153]: pgmap v9644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:39:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:38:59 smithi047 ceph-mon[36835]: pgmap v9644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:39:00.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:38:59 smithi029 ceph-mon[32610]: pgmap v9644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:39:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:01 smithi077 ceph-mon[37153]: pgmap v9645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:39:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:01 smithi047 ceph-mon[36835]: pgmap v9645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:39:02.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:01 smithi029 ceph-mon[32610]: pgmap v9645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:39:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17364.765186 secs 2024-07-27T11:39:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17364.765186 secs 2024-07-27T11:39:03.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17364.765186 secs 2024-07-27T11:39:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:03 smithi077 ceph-mon[37153]: pgmap v9646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:39:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:03 smithi047 ceph-mon[36835]: pgmap v9646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:39:04.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:03 smithi029 ceph-mon[32610]: pgmap v9646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:39:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:05 smithi077 ceph-mon[37153]: pgmap v9647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:39:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:05 smithi047 ceph-mon[36835]: pgmap v9647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:39:06.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:05 smithi029 ceph-mon[32610]: pgmap v9647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:39:06.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:39:06.727+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:39:06.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:39:06.727+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:39:06.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:39:06.727+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:39:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:07 smithi077 ceph-mon[37153]: pgmap v9648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17369.765305 secs 2024-07-27T11:39:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:07 smithi047 ceph-mon[36835]: pgmap v9648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17369.765305 secs 2024-07-27T11:39:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:07 smithi029 ceph-mon[32610]: pgmap v9648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17369.765305 secs 2024-07-27T11:39:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:09 smithi077 ceph-mon[37153]: pgmap v9649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:39:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:09 smithi047 ceph-mon[36835]: pgmap v9649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:39:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:09 smithi029 ceph-mon[32610]: pgmap v9649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:39:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:11 smithi077 ceph-mon[37153]: pgmap v9650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:39:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:11 smithi047 ceph-mon[36835]: pgmap v9650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:39:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:11 smithi029 ceph-mon[32610]: pgmap v9650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:39:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17374.765428 secs 2024-07-27T11:39:13.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17374.765428 secs 2024-07-27T11:39:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17374.765428 secs 2024-07-27T11:39:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:13 smithi077 ceph-mon[37153]: pgmap v9651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:39:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:13 smithi047 ceph-mon[36835]: pgmap v9651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:39:14.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:13 smithi029 ceph-mon[32610]: pgmap v9651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:39:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:15 smithi077 ceph-mon[37153]: pgmap v9652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:39:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:15 smithi047 ceph-mon[36835]: pgmap v9652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:39:16.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:15 smithi029 ceph-mon[32610]: pgmap v9652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:39:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:17 smithi077 ceph-mon[37153]: pgmap v9653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17379.765559 secs 2024-07-27T11:39:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:17 smithi047 ceph-mon[36835]: pgmap v9653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17379.765559 secs 2024-07-27T11:39:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:17 smithi029 ceph-mon[32610]: pgmap v9653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17379.765559 secs 2024-07-27T11:39:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:19 smithi077 ceph-mon[37153]: pgmap v9654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:39:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:19 smithi047 ceph-mon[36835]: pgmap v9654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:39:20.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:19 smithi029 ceph-mon[32610]: pgmap v9654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:39:21.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:39:21.742+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:39:21.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:39:21.742+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:39:21.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:39:21.742+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:39:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:21 smithi077 ceph-mon[37153]: pgmap v9655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:39:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:21 smithi047 ceph-mon[36835]: pgmap v9655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:39:22.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:21 smithi029 ceph-mon[32610]: pgmap v9655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:39:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17384.765705 secs 2024-07-27T11:39:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:22 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:39:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17384.765705 secs 2024-07-27T11:39:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:22 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:39:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17384.765705 secs 2024-07-27T11:39:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:22 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:39:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:23 smithi077 ceph-mon[37153]: pgmap v9656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:39:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:39:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:39:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:23 smithi077 ceph-mon[37153]: pgmap v9657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:39:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:23 smithi077 ceph-mon[37153]: pgmap v9658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:39:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:39:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:23 smithi047 ceph-mon[36835]: pgmap v9656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:39:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:39:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:39:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:23 smithi047 ceph-mon[36835]: pgmap v9657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:39:24.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:23 smithi047 ceph-mon[36835]: pgmap v9658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:39:24.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:39:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:23 smithi029 ceph-mon[32610]: pgmap v9656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:39:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:39:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:39:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:23 smithi029 ceph-mon[32610]: pgmap v9657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:39:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:23 smithi029 ceph-mon[32610]: pgmap v9658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:39:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:39:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:26 smithi029 ceph-mon[32610]: pgmap v9659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:39:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:26 smithi077 ceph-mon[37153]: pgmap v9659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:39:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:26 smithi047 ceph-mon[36835]: pgmap v9659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:39:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:27 smithi077 ceph-mon[37153]: pgmap v9660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:39:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:27 smithi047 ceph-mon[36835]: pgmap v9660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:39:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:27 smithi029 ceph-mon[32610]: pgmap v9660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:39:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17389.765830 secs 2024-07-27T11:39:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17389.765830 secs 2024-07-27T11:39:28.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17389.765830 secs 2024-07-27T11:39:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:29 smithi077 ceph-mon[37153]: pgmap v9661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:39:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:29 smithi047 ceph-mon[36835]: pgmap v9661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:39:29.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:29 smithi029 ceph-mon[32610]: pgmap v9661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:39:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:31 smithi077 ceph-mon[37153]: pgmap v9662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:39:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:31 smithi047 ceph-mon[36835]: pgmap v9662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:39:31.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:31 smithi029 ceph-mon[32610]: pgmap v9662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:39:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17394.765953 secs 2024-07-27T11:39:32.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17394.765953 secs 2024-07-27T11:39:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17394.765953 secs 2024-07-27T11:39:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:33 smithi047 ceph-mon[36835]: pgmap v9663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:39:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:33 smithi029 ceph-mon[32610]: pgmap v9663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:39:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:33 smithi077 ceph-mon[37153]: pgmap v9663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:39:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:35 smithi077 ceph-mon[37153]: pgmap v9664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:39:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:35 smithi047 ceph-mon[36835]: pgmap v9664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:39:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:35 smithi029 ceph-mon[32610]: pgmap v9664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:39:36.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:39:36.758+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:39:36.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:39:36.758+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:39:36.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:39:36.758+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:39:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:37 smithi077 ceph-mon[37153]: pgmap v9665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:37 smithi047 ceph-mon[36835]: pgmap v9665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:37.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:37 smithi029 ceph-mon[32610]: pgmap v9665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17399.766070 secs 2024-07-27T11:39:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17399.766070 secs 2024-07-27T11:39:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17399.766070 secs 2024-07-27T11:39:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:39 smithi077 ceph-mon[37153]: pgmap v9666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:39:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:39 smithi047 ceph-mon[36835]: pgmap v9666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:39:39.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:39 smithi029 ceph-mon[32610]: pgmap v9666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:39:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:41 smithi077 ceph-mon[37153]: pgmap v9667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:41 smithi047 ceph-mon[36835]: pgmap v9667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:41 smithi029 ceph-mon[32610]: pgmap v9667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17404.766137 secs 2024-07-27T11:39:42.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17404.766137 secs 2024-07-27T11:39:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17404.766137 secs 2024-07-27T11:39:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:43 smithi047 ceph-mon[36835]: pgmap v9668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:39:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:43 smithi029 ceph-mon[32610]: pgmap v9668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:39:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:43 smithi077 ceph-mon[37153]: pgmap v9668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:39:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:45 smithi077 ceph-mon[37153]: pgmap v9669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:39:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:45 smithi047 ceph-mon[36835]: pgmap v9669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:39:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:45 smithi029 ceph-mon[32610]: pgmap v9669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:39:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:47 smithi077 ceph-mon[37153]: pgmap v9670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:47 smithi047 ceph-mon[36835]: pgmap v9670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:47 smithi029 ceph-mon[32610]: pgmap v9670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17409.766260 secs 2024-07-27T11:39:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17409.766260 secs 2024-07-27T11:39:48.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17409.766260 secs 2024-07-27T11:39:49.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:49 smithi047 ceph-mon[36835]: pgmap v9671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:39:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:49 smithi077 ceph-mon[37153]: pgmap v9671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:39:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:49 smithi029 ceph-mon[32610]: pgmap v9671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:39:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:51 smithi077 ceph-mon[37153]: pgmap v9672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:51 smithi047 ceph-mon[36835]: pgmap v9672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:51 smithi029 ceph-mon[32610]: pgmap v9672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:51.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:39:51.774+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:39:51.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:39:51.774+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:39:51.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:39:51.774+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:39:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17414.766385 secs 2024-07-27T11:39:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17414.766385 secs 2024-07-27T11:39:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17414.766385 secs 2024-07-27T11:39:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:53 smithi047 ceph-mon[36835]: pgmap v9673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:39:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:53 smithi029 ceph-mon[32610]: pgmap v9673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:39:53.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:53 smithi077 ceph-mon[37153]: pgmap v9673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:39:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:55 smithi077 ceph-mon[37153]: pgmap v9674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:39:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:55 smithi047 ceph-mon[36835]: pgmap v9674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:39:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:55 smithi029 ceph-mon[32610]: pgmap v9674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:39:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:57 smithi077 ceph-mon[37153]: pgmap v9675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:57 smithi047 ceph-mon[36835]: pgmap v9675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:57.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:57 smithi029 ceph-mon[32610]: pgmap v9675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:39:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17419.766515 secs 2024-07-27T11:39:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17419.766515 secs 2024-07-27T11:39:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17419.766515 secs 2024-07-27T11:39:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:39:59 smithi077 ceph-mon[37153]: pgmap v9676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:39:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:39:59 smithi047 ceph-mon[36835]: pgmap v9676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:39:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:39:59 smithi029 ceph-mon[32610]: pgmap v9676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:40:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:40:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:40:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:40:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:01 smithi077 ceph-mon[37153]: pgmap v9677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:01 smithi047 ceph-mon[36835]: pgmap v9677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:01 smithi029 ceph-mon[32610]: pgmap v9677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17424.766640 secs 2024-07-27T11:40:02.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17424.766640 secs 2024-07-27T11:40:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17424.766640 secs 2024-07-27T11:40:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:03 smithi047 ceph-mon[36835]: pgmap v9678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:03 smithi029 ceph-mon[32610]: pgmap v9678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:03 smithi077 ceph-mon[37153]: pgmap v9678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:05 smithi077 ceph-mon[37153]: pgmap v9679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:05 smithi047 ceph-mon[36835]: pgmap v9679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:05 smithi029 ceph-mon[32610]: pgmap v9679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:06.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:40:06.788+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:40:06.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:40:06.788+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:40:06.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:40:06.788+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:40:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:07 smithi077 ceph-mon[37153]: pgmap v9680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:07 smithi047 ceph-mon[36835]: pgmap v9680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:07 smithi029 ceph-mon[32610]: pgmap v9680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17429.766678 secs 2024-07-27T11:40:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17429.766678 secs 2024-07-27T11:40:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17429.766678 secs 2024-07-27T11:40:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:09 smithi077 ceph-mon[37153]: pgmap v9681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:40:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:09 smithi047 ceph-mon[36835]: pgmap v9681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:40:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:09 smithi029 ceph-mon[32610]: pgmap v9681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:40:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:11 smithi077 ceph-mon[37153]: pgmap v9682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:11 smithi047 ceph-mon[36835]: pgmap v9682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:11.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:11 smithi029 ceph-mon[32610]: pgmap v9682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17434.766821 secs 2024-07-27T11:40:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17434.766821 secs 2024-07-27T11:40:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17434.766821 secs 2024-07-27T11:40:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:13 smithi029 ceph-mon[32610]: pgmap v9683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:13 smithi077 ceph-mon[37153]: pgmap v9683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:13 smithi047 ceph-mon[36835]: pgmap v9683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:15 smithi077 ceph-mon[37153]: pgmap v9684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:15 smithi047 ceph-mon[36835]: pgmap v9684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:15.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:15 smithi029 ceph-mon[32610]: pgmap v9684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:17 smithi077 ceph-mon[37153]: pgmap v9685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:17 smithi047 ceph-mon[36835]: pgmap v9685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:17 smithi029 ceph-mon[32610]: pgmap v9685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17439.766905 secs 2024-07-27T11:40:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17439.766905 secs 2024-07-27T11:40:18.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17439.766905 secs 2024-07-27T11:40:19.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:19 smithi047 ceph-mon[36835]: pgmap v9686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:40:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:19 smithi077 ceph-mon[37153]: pgmap v9686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:40:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:19 smithi029 ceph-mon[32610]: pgmap v9686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:40:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:21 smithi077 ceph-mon[37153]: pgmap v9687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:21 smithi047 ceph-mon[36835]: pgmap v9687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:21 smithi029 ceph-mon[32610]: pgmap v9687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:21.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:40:21.797+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:40:21.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:40:21.797+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:40:21.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:40:21.797+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:40:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17444.767007 secs 2024-07-27T11:40:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17444.767007 secs 2024-07-27T11:40:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17444.767007 secs 2024-07-27T11:40:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:23 smithi029 ceph-mon[32610]: pgmap v9688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:23 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:40:23.757 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:23 smithi077 ceph-mon[37153]: pgmap v9688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:23.757 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:23 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:40:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:23 smithi047 ceph-mon[36835]: pgmap v9688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:23 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:40:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:24 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:24.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:24.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:24 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:25.041 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:25.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:24 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:25 smithi077 ceph-mon[37153]: pgmap v9689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:26.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:40:26.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:40:26.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:25 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:25 smithi047 ceph-mon[36835]: pgmap v9689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:26.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:40:26.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:40:26.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:25 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:25 smithi029 ceph-mon[32610]: pgmap v9689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:26.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:26.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:40:26.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:40:26.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:25 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:40:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:27 smithi047 ceph-mon[36835]: pgmap v9690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:27 smithi047 ceph-mon[36835]: pgmap v9691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:27 smithi029 ceph-mon[32610]: pgmap v9690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:27 smithi029 ceph-mon[32610]: pgmap v9691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:40:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:27 smithi077 ceph-mon[37153]: pgmap v9690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:40:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:27 smithi077 ceph-mon[37153]: pgmap v9691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T11:40:28.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17449.767150 secs 2024-07-27T11:40:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17449.767150 secs 2024-07-27T11:40:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17449.767150 secs 2024-07-27T11:40:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:29 smithi047 ceph-mon[36835]: pgmap v9692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:40:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:29 smithi029 ceph-mon[32610]: pgmap v9692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:40:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:29 smithi077 ceph-mon[37153]: pgmap v9692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:40:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:31 smithi047 ceph-mon[36835]: pgmap v9693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:40:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:31 smithi029 ceph-mon[32610]: pgmap v9693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:40:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:31 smithi077 ceph-mon[37153]: pgmap v9693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:40:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:33 smithi047 ceph-mon[36835]: pgmap v9694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:40:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17454.767273 secs 2024-07-27T11:40:33.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:33 smithi029 ceph-mon[32610]: pgmap v9694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:40:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17454.767273 secs 2024-07-27T11:40:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:33 smithi077 ceph-mon[37153]: pgmap v9694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:40:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17454.767273 secs 2024-07-27T11:40:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:35 smithi047 ceph-mon[36835]: pgmap v9695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:35.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:35 smithi029 ceph-mon[32610]: pgmap v9695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:35.486 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:35 smithi077 ceph-mon[37153]: pgmap v9695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:40:36.813+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:40:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:40:36.813+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:40:36.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:40:36.813+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:40:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:37 smithi047 ceph-mon[36835]: pgmap v9696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:40:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:37 smithi029 ceph-mon[32610]: pgmap v9696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:40:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:37 smithi077 ceph-mon[37153]: pgmap v9696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:40:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17459.767421 secs 2024-07-27T11:40:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17459.767421 secs 2024-07-27T11:40:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17459.767421 secs 2024-07-27T11:40:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:39 smithi047 ceph-mon[36835]: pgmap v9697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:40:39.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:39 smithi029 ceph-mon[32610]: pgmap v9697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:40:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:39 smithi077 ceph-mon[37153]: pgmap v9697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:40:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:41 smithi047 ceph-mon[36835]: pgmap v9698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:41.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:41 smithi029 ceph-mon[32610]: pgmap v9698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:41 smithi077 ceph-mon[37153]: pgmap v9698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:43 smithi047 ceph-mon[36835]: pgmap v9699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:40:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17464.767560 secs 2024-07-27T11:40:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:43 smithi029 ceph-mon[32610]: pgmap v9699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:40:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17464.767560 secs 2024-07-27T11:40:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:43 smithi077 ceph-mon[37153]: pgmap v9699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:40:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17464.767560 secs 2024-07-27T11:40:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:45 smithi047 ceph-mon[36835]: pgmap v9700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:45 smithi029 ceph-mon[32610]: pgmap v9700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:45 smithi077 ceph-mon[37153]: pgmap v9700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:47 smithi047 ceph-mon[36835]: pgmap v9701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:47.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:47 smithi029 ceph-mon[32610]: pgmap v9701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:47 smithi077 ceph-mon[37153]: pgmap v9701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17469.767689 secs 2024-07-27T11:40:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17469.767689 secs 2024-07-27T11:40:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17469.767689 secs 2024-07-27T11:40:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:49 smithi047 ceph-mon[36835]: pgmap v9702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:40:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:49 smithi029 ceph-mon[32610]: pgmap v9702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:40:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:49 smithi077 ceph-mon[37153]: pgmap v9702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:40:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:51 smithi047 ceph-mon[36835]: pgmap v9703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:51.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:51 smithi029 ceph-mon[32610]: pgmap v9703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:51 smithi077 ceph-mon[37153]: pgmap v9703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:51.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:40:51.822+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:40:51.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:40:51.822+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:40:51.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:40:51.822+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:40:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:53 smithi047 ceph-mon[36835]: pgmap v9704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:40:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17474.767824 secs 2024-07-27T11:40:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:53 smithi029 ceph-mon[32610]: pgmap v9704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:40:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17474.767824 secs 2024-07-27T11:40:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:53 smithi077 ceph-mon[37153]: pgmap v9704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:40:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17474.767824 secs 2024-07-27T11:40:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:55 smithi029 ceph-mon[32610]: pgmap v9705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:55 smithi077 ceph-mon[37153]: pgmap v9705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:55 smithi047 ceph-mon[36835]: pgmap v9705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:40:57.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:57 smithi029 ceph-mon[32610]: pgmap v9706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:57 smithi077 ceph-mon[37153]: pgmap v9706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:57 smithi047 ceph-mon[36835]: pgmap v9706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:40:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17479.767949 secs 2024-07-27T11:40:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17479.767949 secs 2024-07-27T11:40:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17479.767949 secs 2024-07-27T11:40:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:40:59 smithi029 ceph-mon[32610]: pgmap v9707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:40:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:40:59 smithi077 ceph-mon[37153]: pgmap v9707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:40:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:40:59 smithi047 ceph-mon[36835]: pgmap v9707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:00.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:00.291+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:41:00.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:00.291+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:41:00.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:00.292+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:41:00.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:00.292+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:41:01.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:01 smithi029 ceph-mon[32610]: pgmap v9708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:01 smithi077 ceph-mon[37153]: pgmap v9708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:01 smithi047 ceph-mon[36835]: pgmap v9708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:03 smithi077 ceph-mon[37153]: pgmap v9709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:41:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17484.768049 secs 2024-07-27T11:41:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:03 smithi047 ceph-mon[36835]: pgmap v9709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:41:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17484.768049 secs 2024-07-27T11:41:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:03 smithi029 ceph-mon[32610]: pgmap v9709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:41:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17484.768049 secs 2024-07-27T11:41:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:05 smithi077 ceph-mon[37153]: pgmap v9710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:41:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:05 smithi047 ceph-mon[36835]: pgmap v9710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:41:05.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:05 smithi029 ceph-mon[32610]: pgmap v9710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:41:06.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:06.830+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:41:06.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:06.830+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:41:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:06.830+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:41:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:07 smithi077 ceph-mon[37153]: pgmap v9711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:07 smithi047 ceph-mon[36835]: pgmap v9711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:07 smithi029 ceph-mon[32610]: pgmap v9711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17489.768121 secs 2024-07-27T11:41:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17489.768121 secs 2024-07-27T11:41:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17489.768121 secs 2024-07-27T11:41:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:09 smithi077 ceph-mon[37153]: pgmap v9712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:09 smithi047 ceph-mon[36835]: pgmap v9712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:09 smithi029 ceph-mon[32610]: pgmap v9712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:11 smithi077 ceph-mon[37153]: pgmap v9713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:11 smithi047 ceph-mon[36835]: pgmap v9713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:11.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:11 smithi029 ceph-mon[32610]: pgmap v9713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:13.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:13 smithi029 ceph-mon[32610]: pgmap v9714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:41:13.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17494.768244 secs 2024-07-27T11:41:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:13 smithi077 ceph-mon[37153]: pgmap v9714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:41:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17494.768244 secs 2024-07-27T11:41:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:13 smithi047 ceph-mon[36835]: pgmap v9714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:41:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17494.768244 secs 2024-07-27T11:41:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:15 smithi029 ceph-mon[32610]: pgmap v9715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:41:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:15 smithi077 ceph-mon[37153]: pgmap v9715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:41:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:15 smithi047 ceph-mon[36835]: pgmap v9715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:41:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:17 smithi077 ceph-mon[37153]: pgmap v9716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:17 smithi047 ceph-mon[36835]: pgmap v9716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:17.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:17 smithi029 ceph-mon[32610]: pgmap v9716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17499.768373 secs 2024-07-27T11:41:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17499.768373 secs 2024-07-27T11:41:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17499.768373 secs 2024-07-27T11:41:19.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:19 smithi047 ceph-mon[36835]: pgmap v9717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:19 smithi077 ceph-mon[37153]: pgmap v9717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:19 smithi029 ceph-mon[32610]: pgmap v9717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:21 smithi077 ceph-mon[37153]: pgmap v9718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:21 smithi047 ceph-mon[36835]: pgmap v9718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:21.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:21 smithi029 ceph-mon[32610]: pgmap v9718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:21.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:21.845+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:41:21.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:21.845+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:41:21.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:21.845+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:41:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:23 smithi077 ceph-mon[37153]: pgmap v9719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:41:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17504.768511 secs 2024-07-27T11:41:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:23 smithi047 ceph-mon[36835]: pgmap v9719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:41:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17504.768511 secs 2024-07-27T11:41:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:23 smithi029 ceph-mon[32610]: pgmap v9719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:41:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17504.768511 secs 2024-07-27T11:41:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:25 smithi077 ceph-mon[37153]: pgmap v9720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:41:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:25 smithi047 ceph-mon[36835]: pgmap v9720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:41:25.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:25 smithi029 ceph-mon[32610]: pgmap v9720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:41:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:26 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:41:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:26 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:41:26.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:26 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:41:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: pgmap v9721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:41:27.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:41:27.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:27 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: pgmap v9721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:41:27.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:41:27.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:27 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: pgmap v9721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:27.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:41:27.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:41:27.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:27 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:41:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17509.768627 secs 2024-07-27T11:41:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:28 smithi077 ceph-mon[37153]: pgmap v9722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:41:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:28 smithi077 ceph-mon[37153]: pgmap v9723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T11:41:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17509.768627 secs 2024-07-27T11:41:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:28 smithi047 ceph-mon[36835]: pgmap v9722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:41:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:28 smithi047 ceph-mon[36835]: pgmap v9723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T11:41:28.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17509.768627 secs 2024-07-27T11:41:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:28 smithi029 ceph-mon[32610]: pgmap v9722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:41:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:28 smithi029 ceph-mon[32610]: pgmap v9723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T11:41:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:30 smithi077 ceph-mon[37153]: pgmap v9724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T11:41:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:30 smithi047 ceph-mon[36835]: pgmap v9724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T11:41:30.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:30 smithi029 ceph-mon[32610]: pgmap v9724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T11:41:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:32 smithi077 ceph-mon[37153]: pgmap v9725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:41:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:32 smithi047 ceph-mon[36835]: pgmap v9725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:41:32.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:32 smithi029 ceph-mon[32610]: pgmap v9725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:41:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17514.768760 secs 2024-07-27T11:41:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17514.768760 secs 2024-07-27T11:41:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17514.768760 secs 2024-07-27T11:41:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:34 smithi077 ceph-mon[37153]: pgmap v9726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:41:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:34 smithi047 ceph-mon[36835]: pgmap v9726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:41:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:34 smithi029 ceph-mon[32610]: pgmap v9726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:41:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:36 smithi077 ceph-mon[37153]: pgmap v9727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T11:41:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:36 smithi047 ceph-mon[36835]: pgmap v9727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T11:41:36.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:36 smithi029 ceph-mon[32610]: pgmap v9727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-27T11:41:36.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:36.861+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:41:36.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:36.861+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:41:36.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:36.861+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:41:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17519.768888 secs 2024-07-27T11:41:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:38 smithi077 ceph-mon[37153]: pgmap v9728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:41:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17519.768888 secs 2024-07-27T11:41:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:38 smithi047 ceph-mon[36835]: pgmap v9728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:41:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17519.768888 secs 2024-07-27T11:41:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:38 smithi029 ceph-mon[32610]: pgmap v9728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:41:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:40 smithi077 ceph-mon[37153]: pgmap v9729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:41:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:40 smithi047 ceph-mon[36835]: pgmap v9729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:41:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:40 smithi029 ceph-mon[32610]: pgmap v9729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:41:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:42 smithi077 ceph-mon[37153]: pgmap v9730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:42 smithi047 ceph-mon[36835]: pgmap v9730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:42.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:42 smithi029 ceph-mon[32610]: pgmap v9730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17524.768973 secs 2024-07-27T11:41:43.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17524.768973 secs 2024-07-27T11:41:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17524.768973 secs 2024-07-27T11:41:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:44 smithi077 ceph-mon[37153]: pgmap v9731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:44 smithi047 ceph-mon[36835]: pgmap v9731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:44 smithi029 ceph-mon[32610]: pgmap v9731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:45.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:45 smithi077 ceph-mon[37153]: pgmap v9732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:45 smithi047 ceph-mon[36835]: pgmap v9732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:45.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:45 smithi029 ceph-mon[32610]: pgmap v9732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17529.769098 secs 2024-07-27T11:41:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:48 smithi047 ceph-mon[36835]: pgmap v9733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:41:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17529.769098 secs 2024-07-27T11:41:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:48 smithi029 ceph-mon[32610]: pgmap v9733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:41:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17529.769098 secs 2024-07-27T11:41:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:48 smithi077 ceph-mon[37153]: pgmap v9733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:41:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:49 smithi077 ceph-mon[37153]: pgmap v9734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:41:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:49 smithi047 ceph-mon[36835]: pgmap v9734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:41:50.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:49 smithi029 ceph-mon[32610]: pgmap v9734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:41:51.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:51.870+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:41:51.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:51.870+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:41:51.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:41:51.870+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:41:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:52 smithi047 ceph-mon[36835]: pgmap v9735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:52 smithi029 ceph-mon[32610]: pgmap v9735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:52 smithi077 ceph-mon[37153]: pgmap v9735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17534.769220 secs 2024-07-27T11:41:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17534.769220 secs 2024-07-27T11:41:53.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17534.769220 secs 2024-07-27T11:41:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:54 smithi047 ceph-mon[36835]: pgmap v9736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:54.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:54 smithi029 ceph-mon[32610]: pgmap v9736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:54 smithi077 ceph-mon[37153]: pgmap v9736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:41:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:56 smithi047 ceph-mon[36835]: pgmap v9737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:56 smithi029 ceph-mon[32610]: pgmap v9737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:56 smithi077 ceph-mon[37153]: pgmap v9737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:41:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17539.769343 secs 2024-07-27T11:41:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:41:58 smithi047 ceph-mon[36835]: pgmap v9738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:41:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17539.769343 secs 2024-07-27T11:41:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:41:58 smithi029 ceph-mon[32610]: pgmap v9738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:41:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17539.769343 secs 2024-07-27T11:41:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:41:58 smithi077 ceph-mon[37153]: pgmap v9738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:00 smithi047 ceph-mon[36835]: pgmap v9739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:42:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:00 smithi029 ceph-mon[32610]: pgmap v9739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:42:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:00 smithi077 ceph-mon[37153]: pgmap v9739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:42:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:02 smithi047 ceph-mon[36835]: pgmap v9740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:02 smithi029 ceph-mon[32610]: pgmap v9740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:02 smithi077 ceph-mon[37153]: pgmap v9740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17544.769468 secs 2024-07-27T11:42:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17544.769468 secs 2024-07-27T11:42:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17544.769468 secs 2024-07-27T11:42:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:04 smithi047 ceph-mon[36835]: pgmap v9741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:42:04.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:04 smithi029 ceph-mon[32610]: pgmap v9741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:42:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:04 smithi077 ceph-mon[37153]: pgmap v9741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:42:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:06 smithi047 ceph-mon[36835]: pgmap v9742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:06 smithi029 ceph-mon[32610]: pgmap v9742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:06 smithi077 ceph-mon[37153]: pgmap v9742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:06.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:42:06.884+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:42:06.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:42:06.884+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:42:06.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:42:06.884+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:42:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17549.769593 secs 2024-07-27T11:42:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:08 smithi047 ceph-mon[36835]: pgmap v9743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17549.769593 secs 2024-07-27T11:42:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:08 smithi029 ceph-mon[32610]: pgmap v9743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17549.769593 secs 2024-07-27T11:42:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:08 smithi077 ceph-mon[37153]: pgmap v9743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:09 smithi077 ceph-mon[37153]: pgmap v9744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:42:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:09 smithi047 ceph-mon[36835]: pgmap v9744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:42:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:09 smithi029 ceph-mon[32610]: pgmap v9744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:42:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:12 smithi047 ceph-mon[36835]: pgmap v9745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:12.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:12 smithi029 ceph-mon[32610]: pgmap v9745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:12 smithi077 ceph-mon[37153]: pgmap v9745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17554.769715 secs 2024-07-27T11:42:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17554.769715 secs 2024-07-27T11:42:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17554.769715 secs 2024-07-27T11:42:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:14 smithi047 ceph-mon[36835]: pgmap v9746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:42:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:14 smithi029 ceph-mon[32610]: pgmap v9746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:42:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:14 smithi077 ceph-mon[37153]: pgmap v9746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:42:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:16 smithi047 ceph-mon[36835]: pgmap v9747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:16 smithi029 ceph-mon[32610]: pgmap v9747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:16 smithi077 ceph-mon[37153]: pgmap v9747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17559.769837 secs 2024-07-27T11:42:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:18 smithi047 ceph-mon[36835]: pgmap v9748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17559.769837 secs 2024-07-27T11:42:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:18 smithi029 ceph-mon[32610]: pgmap v9748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17559.769837 secs 2024-07-27T11:42:18.652 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:18 smithi077 ceph-mon[37153]: pgmap v9748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:20 smithi047 ceph-mon[36835]: pgmap v9749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:42:20.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:20 smithi029 ceph-mon[32610]: pgmap v9749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:42:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:20 smithi077 ceph-mon[37153]: pgmap v9749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:42:21.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:42:21.893+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:42:21.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:42:21.893+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:42:21.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:42:21.893+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:42:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:22 smithi047 ceph-mon[36835]: pgmap v9750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:22.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:22 smithi029 ceph-mon[32610]: pgmap v9750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:22 smithi077 ceph-mon[37153]: pgmap v9750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17564.769906 secs 2024-07-27T11:42:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17564.769906 secs 2024-07-27T11:42:23.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17564.769906 secs 2024-07-27T11:42:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:24 smithi047 ceph-mon[36835]: pgmap v9751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:42:24.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:24 smithi029 ceph-mon[32610]: pgmap v9751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:42:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:24 smithi077 ceph-mon[37153]: pgmap v9751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:42:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:26 smithi047 ceph-mon[36835]: pgmap v9752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:26.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:26 smithi029 ceph-mon[32610]: pgmap v9752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:26 smithi077 ceph-mon[37153]: pgmap v9752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17569.770033 secs 2024-07-27T11:42:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:28 smithi047 ceph-mon[36835]: pgmap v9753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:42:28.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:42:28.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:42:28.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:42:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17569.770033 secs 2024-07-27T11:42:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:28 smithi029 ceph-mon[32610]: pgmap v9753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:42:28.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:42:28.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:42:28.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:42:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17569.770033 secs 2024-07-27T11:42:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:28 smithi077 ceph-mon[37153]: pgmap v9753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:42:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:42:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:42:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:42:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:29 smithi047 ceph-mon[36835]: pgmap v9754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T11:42:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:29 smithi047 ceph-mon[36835]: pgmap v9755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:42:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:29 smithi029 ceph-mon[32610]: pgmap v9754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T11:42:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:29 smithi029 ceph-mon[32610]: pgmap v9755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:42:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:29 smithi077 ceph-mon[37153]: pgmap v9754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T11:42:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:29 smithi077 ceph-mon[37153]: pgmap v9755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:42:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:31 smithi029 ceph-mon[32610]: pgmap v9756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:42:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:31 smithi077 ceph-mon[37153]: pgmap v9756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:42:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:31 smithi047 ceph-mon[36835]: pgmap v9756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:42:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:33 smithi029 ceph-mon[32610]: pgmap v9757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:42:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17574.770163 secs 2024-07-27T11:42:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:33 smithi077 ceph-mon[37153]: pgmap v9757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:42:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17574.770163 secs 2024-07-27T11:42:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:33 smithi047 ceph-mon[36835]: pgmap v9757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:42:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17574.770163 secs 2024-07-27T11:42:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:35 smithi029 ceph-mon[32610]: pgmap v9758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:42:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:35 smithi077 ceph-mon[37153]: pgmap v9758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:42:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:35 smithi047 ceph-mon[36835]: pgmap v9758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:42:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:42:36.909+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:42:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:42:36.909+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:42:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:42:36.909+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:42:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:37 smithi029 ceph-mon[32610]: pgmap v9759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:42:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:37 smithi077 ceph-mon[37153]: pgmap v9759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:42:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:37 smithi047 ceph-mon[36835]: pgmap v9759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:42:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17579.770292 secs 2024-07-27T11:42:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17579.770292 secs 2024-07-27T11:42:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17579.770292 secs 2024-07-27T11:42:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:39 smithi077 ceph-mon[37153]: pgmap v9760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:42:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:39 smithi047 ceph-mon[36835]: pgmap v9760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:42:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:39 smithi029 ceph-mon[32610]: pgmap v9760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:42:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:41 smithi077 ceph-mon[37153]: pgmap v9761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:42:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:41 smithi047 ceph-mon[36835]: pgmap v9761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:42:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:41 smithi029 ceph-mon[32610]: pgmap v9761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:42:43.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:43 smithi029 ceph-mon[32610]: pgmap v9762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:43.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17584.770416 secs 2024-07-27T11:42:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:43 smithi077 ceph-mon[37153]: pgmap v9762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17584.770416 secs 2024-07-27T11:42:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:43 smithi047 ceph-mon[36835]: pgmap v9762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17584.770416 secs 2024-07-27T11:42:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:45 smithi077 ceph-mon[37153]: pgmap v9763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:42:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:45 smithi047 ceph-mon[36835]: pgmap v9763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:42:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:45 smithi029 ceph-mon[32610]: pgmap v9763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:42:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:47 smithi077 ceph-mon[37153]: pgmap v9764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:47 smithi047 ceph-mon[36835]: pgmap v9764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:47 smithi029 ceph-mon[32610]: pgmap v9764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17589.770547 secs 2024-07-27T11:42:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17589.770547 secs 2024-07-27T11:42:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17589.770547 secs 2024-07-27T11:42:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:49 smithi077 ceph-mon[37153]: pgmap v9765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:42:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:49 smithi047 ceph-mon[36835]: pgmap v9765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:42:49.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:49 smithi029 ceph-mon[32610]: pgmap v9765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:42:51.498 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:51 smithi047 ceph-mon[36835]: pgmap v9766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:42:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:51 smithi077 ceph-mon[37153]: pgmap v9766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:42:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:51 smithi029 ceph-mon[32610]: pgmap v9766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:42:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:42:51.918+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:42:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:42:51.918+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:42:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:42:51.918+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:42:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:53 smithi077 ceph-mon[37153]: pgmap v9767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17594.770685 secs 2024-07-27T11:42:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:53 smithi047 ceph-mon[36835]: pgmap v9767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17594.770685 secs 2024-07-27T11:42:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:53 smithi029 ceph-mon[32610]: pgmap v9767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:42:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17594.770685 secs 2024-07-27T11:42:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:55 smithi077 ceph-mon[37153]: pgmap v9768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:42:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:55 smithi047 ceph-mon[36835]: pgmap v9768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:42:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:55 smithi029 ceph-mon[32610]: pgmap v9768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:42:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:57 smithi077 ceph-mon[37153]: pgmap v9769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:57 smithi047 ceph-mon[36835]: pgmap v9769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:57 smithi029 ceph-mon[32610]: pgmap v9769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:42:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17599.770812 secs 2024-07-27T11:42:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17599.770812 secs 2024-07-27T11:42:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17599.770812 secs 2024-07-27T11:42:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:42:59 smithi077 ceph-mon[37153]: pgmap v9770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:42:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:42:59 smithi047 ceph-mon[36835]: pgmap v9770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:42:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:42:59 smithi029 ceph-mon[32610]: pgmap v9770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:43:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:01 smithi077 ceph-mon[37153]: pgmap v9771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:43:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:01 smithi047 ceph-mon[36835]: pgmap v9771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:43:01.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:01 smithi029 ceph-mon[32610]: pgmap v9771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:43:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:03 smithi077 ceph-mon[37153]: pgmap v9772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17604.770927 secs 2024-07-27T11:43:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:03 smithi047 ceph-mon[36835]: pgmap v9772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17604.770927 secs 2024-07-27T11:43:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:03 smithi029 ceph-mon[32610]: pgmap v9772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17604.770927 secs 2024-07-27T11:43:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:05 smithi077 ceph-mon[37153]: pgmap v9773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:05 smithi047 ceph-mon[36835]: pgmap v9773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:05.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:05 smithi029 ceph-mon[32610]: pgmap v9773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:06.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:43:06.928+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:43:06.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:43:06.928+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:43:06.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:43:06.928+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:43:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:07 smithi077 ceph-mon[37153]: pgmap v9774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:07 smithi047 ceph-mon[36835]: pgmap v9774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:07 smithi029 ceph-mon[32610]: pgmap v9774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17609.771014 secs 2024-07-27T11:43:08.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17609.771014 secs 2024-07-27T11:43:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17609.771014 secs 2024-07-27T11:43:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:09 smithi077 ceph-mon[37153]: pgmap v9775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:43:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:09 smithi047 ceph-mon[36835]: pgmap v9775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:43:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:09 smithi029 ceph-mon[32610]: pgmap v9775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:43:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:11 smithi077 ceph-mon[37153]: pgmap v9776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:43:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:11 smithi047 ceph-mon[36835]: pgmap v9776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:43:11.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:11 smithi029 ceph-mon[32610]: pgmap v9776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:43:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:13 smithi077 ceph-mon[37153]: pgmap v9777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17614.771142 secs 2024-07-27T11:43:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:13 smithi047 ceph-mon[36835]: pgmap v9777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17614.771142 secs 2024-07-27T11:43:13.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:13 smithi029 ceph-mon[32610]: pgmap v9777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17614.771142 secs 2024-07-27T11:43:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:15 smithi077 ceph-mon[37153]: pgmap v9778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:15 smithi047 ceph-mon[36835]: pgmap v9778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:15.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:15 smithi029 ceph-mon[32610]: pgmap v9778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:17 smithi077 ceph-mon[37153]: pgmap v9779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:17 smithi047 ceph-mon[36835]: pgmap v9779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:17.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:17 smithi029 ceph-mon[32610]: pgmap v9779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17619.771240 secs 2024-07-27T11:43:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17619.771240 secs 2024-07-27T11:43:18.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17619.771240 secs 2024-07-27T11:43:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:19 smithi077 ceph-mon[37153]: pgmap v9780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:43:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:19 smithi047 ceph-mon[36835]: pgmap v9780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:43:19.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:19 smithi029 ceph-mon[32610]: pgmap v9780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:43:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:21 smithi077 ceph-mon[37153]: pgmap v9781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:43:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:21 smithi047 ceph-mon[36835]: pgmap v9781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:43:21.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:21 smithi029 ceph-mon[32610]: pgmap v9781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:43:21.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:43:21.941+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:43:21.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:43:21.941+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:43:21.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:43:21.941+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:43:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:23 smithi077 ceph-mon[37153]: pgmap v9782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17624.771365 secs 2024-07-27T11:43:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:23 smithi047 ceph-mon[36835]: pgmap v9782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17624.771365 secs 2024-07-27T11:43:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:23 smithi029 ceph-mon[32610]: pgmap v9782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17624.771365 secs 2024-07-27T11:43:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:25 smithi077 ceph-mon[37153]: pgmap v9783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:25 smithi047 ceph-mon[36835]: pgmap v9783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:25 smithi029 ceph-mon[32610]: pgmap v9783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:27 smithi077 ceph-mon[37153]: pgmap v9784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:27 smithi047 ceph-mon[36835]: pgmap v9784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:27 smithi029 ceph-mon[32610]: pgmap v9784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17629.771507 secs 2024-07-27T11:43:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:43:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:43:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:43:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:43:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17629.771507 secs 2024-07-27T11:43:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:43:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:43:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:43:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:43:28.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17629.771507 secs 2024-07-27T11:43:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:43:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:43:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:43:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:43:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:29 smithi077 ceph-mon[37153]: pgmap v9785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:43:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:29 smithi077 ceph-mon[37153]: pgmap v9786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:43:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:29 smithi077 ceph-mon[37153]: pgmap v9787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:43:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:29 smithi047 ceph-mon[36835]: pgmap v9785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:43:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:29 smithi047 ceph-mon[36835]: pgmap v9786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:43:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:29 smithi047 ceph-mon[36835]: pgmap v9787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:43:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:29 smithi029 ceph-mon[32610]: pgmap v9785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:43:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:29 smithi029 ceph-mon[32610]: pgmap v9786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:43:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:29 smithi029 ceph-mon[32610]: pgmap v9787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:43:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:30 smithi077 ceph-mon[37153]: pgmap v9788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:43:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:30 smithi047 ceph-mon[36835]: pgmap v9788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:43:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:30 smithi029 ceph-mon[32610]: pgmap v9788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:43:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17634.771655 secs 2024-07-27T11:43:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:33 smithi077 ceph-mon[37153]: pgmap v9789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:43:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17634.771655 secs 2024-07-27T11:43:33.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:33 smithi047 ceph-mon[36835]: pgmap v9789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:43:33.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17634.771655 secs 2024-07-27T11:43:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:33 smithi029 ceph-mon[32610]: pgmap v9789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:43:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:35 smithi077 ceph-mon[37153]: pgmap v9790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:43:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:35 smithi047 ceph-mon[36835]: pgmap v9790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:43:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:35 smithi029 ceph-mon[32610]: pgmap v9790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:43:36.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:43:36.957+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:43:36.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:43:36.957+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:43:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:43:36.957+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:43:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:37 smithi077 ceph-mon[37153]: pgmap v9791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:43:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:37 smithi047 ceph-mon[36835]: pgmap v9791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:43:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:37 smithi029 ceph-mon[32610]: pgmap v9791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:43:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17639.771802 secs 2024-07-27T11:43:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17639.771802 secs 2024-07-27T11:43:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17639.771802 secs 2024-07-27T11:43:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:39 smithi077 ceph-mon[37153]: pgmap v9792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:43:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:39 smithi047 ceph-mon[36835]: pgmap v9792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:43:39.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:39 smithi029 ceph-mon[32610]: pgmap v9792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:43:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:41 smithi077 ceph-mon[37153]: pgmap v9793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:41 smithi047 ceph-mon[36835]: pgmap v9793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:41 smithi029 ceph-mon[32610]: pgmap v9793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17644.771914 secs 2024-07-27T11:43:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:43 smithi077 ceph-mon[37153]: pgmap v9794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17644.771914 secs 2024-07-27T11:43:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:43 smithi047 ceph-mon[36835]: pgmap v9794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17644.771914 secs 2024-07-27T11:43:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:43 smithi029 ceph-mon[32610]: pgmap v9794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:45 smithi077 ceph-mon[37153]: pgmap v9795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:43:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:45 smithi047 ceph-mon[36835]: pgmap v9795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:43:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:45 smithi029 ceph-mon[32610]: pgmap v9795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:43:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:47 smithi077 ceph-mon[37153]: pgmap v9796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:47 smithi047 ceph-mon[36835]: pgmap v9796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:47 smithi029 ceph-mon[32610]: pgmap v9796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17649.772038 secs 2024-07-27T11:43:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:48 smithi077 ceph-mon[37153]: pgmap v9797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17649.772038 secs 2024-07-27T11:43:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:48 smithi047 ceph-mon[36835]: pgmap v9797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:48.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17649.772038 secs 2024-07-27T11:43:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:48 smithi029 ceph-mon[32610]: pgmap v9797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:51 smithi077 ceph-mon[37153]: pgmap v9798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:51 smithi047 ceph-mon[36835]: pgmap v9798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:51 smithi029 ceph-mon[32610]: pgmap v9798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:43:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:43:51.966+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:43:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:43:51.966+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:43:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:43:51.966+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:43:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17654.772173 secs 2024-07-27T11:43:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:53 smithi077 ceph-mon[37153]: pgmap v9799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17654.772173 secs 2024-07-27T11:43:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:53 smithi047 ceph-mon[36835]: pgmap v9799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17654.772173 secs 2024-07-27T11:43:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:53 smithi029 ceph-mon[32610]: pgmap v9799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:55 smithi077 ceph-mon[37153]: pgmap v9800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:43:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:55 smithi047 ceph-mon[36835]: pgmap v9800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:43:55.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:55 smithi029 ceph-mon[32610]: pgmap v9800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:43:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:57 smithi077 ceph-mon[37153]: pgmap v9801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:57 smithi047 ceph-mon[36835]: pgmap v9801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:57 smithi029 ceph-mon[32610]: pgmap v9801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:43:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17659.772294 secs 2024-07-27T11:43:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17659.772294 secs 2024-07-27T11:43:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17659.772294 secs 2024-07-27T11:43:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:43:59 smithi077 ceph-mon[37153]: pgmap v9802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:43:59 smithi047 ceph-mon[36835]: pgmap v9802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:43:59.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:43:59 smithi029 ceph-mon[32610]: pgmap v9802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:00.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:00.315+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:44:00.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:00.315+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:44:00.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:00.318+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:44:00.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:00.318+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:44:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:01 smithi077 ceph-mon[37153]: pgmap v9803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:01 smithi047 ceph-mon[36835]: pgmap v9803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:01 smithi029 ceph-mon[32610]: pgmap v9803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:02 smithi077 ceph-mon[37153]: pgmap v9804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:02 smithi047 ceph-mon[36835]: pgmap v9804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:02 smithi029 ceph-mon[32610]: pgmap v9804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17664.772397 secs 2024-07-27T11:44:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17664.772397 secs 2024-07-27T11:44:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17664.772397 secs 2024-07-27T11:44:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:04 smithi077 ceph-mon[37153]: pgmap v9805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:44:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:04 smithi047 ceph-mon[36835]: pgmap v9805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:44:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:04 smithi029 ceph-mon[32610]: pgmap v9805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:44:06.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:06.980+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:44:06.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:06.980+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:44:06.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:06.980+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:44:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:07 smithi077 ceph-mon[37153]: pgmap v9806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:07 smithi047 ceph-mon[36835]: pgmap v9806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:07 smithi029 ceph-mon[32610]: pgmap v9806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17669.772524 secs 2024-07-27T11:44:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17669.772524 secs 2024-07-27T11:44:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17669.772524 secs 2024-07-27T11:44:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:09 smithi077 ceph-mon[37153]: pgmap v9807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:09 smithi047 ceph-mon[36835]: pgmap v9807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:09 smithi029 ceph-mon[32610]: pgmap v9807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:11 smithi077 ceph-mon[37153]: pgmap v9808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:11 smithi047 ceph-mon[36835]: pgmap v9808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:11.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:11 smithi029 ceph-mon[32610]: pgmap v9808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17674.772647 secs 2024-07-27T11:44:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:13 smithi077 ceph-mon[37153]: pgmap v9809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17674.772647 secs 2024-07-27T11:44:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:13 smithi047 ceph-mon[36835]: pgmap v9809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17674.772647 secs 2024-07-27T11:44:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:13 smithi029 ceph-mon[32610]: pgmap v9809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:15 smithi077 ceph-mon[37153]: pgmap v9810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:44:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:15 smithi047 ceph-mon[36835]: pgmap v9810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:44:15.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:15 smithi029 ceph-mon[32610]: pgmap v9810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:44:16.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:16 smithi077 ceph-mon[37153]: pgmap v9811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:16 smithi047 ceph-mon[36835]: pgmap v9811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:16.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:16 smithi029 ceph-mon[32610]: pgmap v9811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17679.772805 secs 2024-07-27T11:44:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17679.772805 secs 2024-07-27T11:44:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17679.772805 secs 2024-07-27T11:44:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:18 smithi077 ceph-mon[37153]: pgmap v9812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:18 smithi047 ceph-mon[36835]: pgmap v9812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:18 smithi029 ceph-mon[32610]: pgmap v9812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:21 smithi077 ceph-mon[37153]: pgmap v9813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:21.649 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:21 smithi047 ceph-mon[36835]: pgmap v9813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:21 smithi029 ceph-mon[32610]: pgmap v9813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:21.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:21.989+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:44:21.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:21.989+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:44:21.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:21.989+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:44:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17684.772921 secs 2024-07-27T11:44:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:23 smithi077 ceph-mon[37153]: pgmap v9814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17684.772921 secs 2024-07-27T11:44:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:23 smithi047 ceph-mon[36835]: pgmap v9814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17684.772921 secs 2024-07-27T11:44:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:23 smithi029 ceph-mon[32610]: pgmap v9814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:25 smithi077 ceph-mon[37153]: pgmap v9815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:44:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:25 smithi047 ceph-mon[36835]: pgmap v9815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:44:25.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:25 smithi029 ceph-mon[32610]: pgmap v9815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:44:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:27 smithi077 ceph-mon[37153]: pgmap v9816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:27 smithi047 ceph-mon[36835]: pgmap v9816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:27 smithi029 ceph-mon[32610]: pgmap v9816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:44:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17689.773044 secs 2024-07-27T11:44:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17689.773044 secs 2024-07-27T11:44:28.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17689.773044 secs 2024-07-27T11:44:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:29 smithi077 ceph-mon[37153]: pgmap v9817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:44:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:44:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:44:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:29 smithi077 ceph-mon[37153]: pgmap v9818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:44:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:29 smithi077 ceph-mon[37153]: pgmap v9819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:44:29.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:44:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:29 smithi047 ceph-mon[36835]: pgmap v9817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:44:29.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:44:29.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:44:29.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:29 smithi047 ceph-mon[36835]: pgmap v9818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:44:29.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:29 smithi047 ceph-mon[36835]: pgmap v9819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:44:29.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:44:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:29 smithi029 ceph-mon[32610]: pgmap v9817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:44:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:44:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:44:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:29 smithi029 ceph-mon[32610]: pgmap v9818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:44:29.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:29 smithi029 ceph-mon[32610]: pgmap v9819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:44:29.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:44:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:31 smithi077 ceph-mon[37153]: pgmap v9820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:44:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:31 smithi047 ceph-mon[36835]: pgmap v9820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:44:32.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:31 smithi029 ceph-mon[32610]: pgmap v9820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:44:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17694.773162 secs 2024-07-27T11:44:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17694.773162 secs 2024-07-27T11:44:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17694.773162 secs 2024-07-27T11:44:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:33 smithi077 ceph-mon[37153]: pgmap v9821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:44:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:33 smithi047 ceph-mon[36835]: pgmap v9821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:44:34.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:33 smithi029 ceph-mon[32610]: pgmap v9821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:44:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:35 smithi047 ceph-mon[36835]: pgmap v9822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:44:36.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:35 smithi029 ceph-mon[32610]: pgmap v9822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:44:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:35 smithi077 ceph-mon[37153]: pgmap v9822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:44:37.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:37.005+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:44:37.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:37.005+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:44:37.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:37.005+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:44:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:37 smithi047 ceph-mon[36835]: pgmap v9823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:44:38.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17699.773286 secs 2024-07-27T11:44:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:37 smithi029 ceph-mon[32610]: pgmap v9823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:44:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17699.773286 secs 2024-07-27T11:44:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:37 smithi077 ceph-mon[37153]: pgmap v9823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:44:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17699.773286 secs 2024-07-27T11:44:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:39 smithi047 ceph-mon[36835]: pgmap v9824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:44:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:39 smithi029 ceph-mon[32610]: pgmap v9824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:44:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:39 smithi077 ceph-mon[37153]: pgmap v9824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:44:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:41 smithi047 ceph-mon[36835]: pgmap v9825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:41 smithi029 ceph-mon[32610]: pgmap v9825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:41 smithi077 ceph-mon[37153]: pgmap v9825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17704.773410 secs 2024-07-27T11:44:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17704.773410 secs 2024-07-27T11:44:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17704.773410 secs 2024-07-27T11:44:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:43 smithi047 ceph-mon[36835]: pgmap v9826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:44:44.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:43 smithi029 ceph-mon[32610]: pgmap v9826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:44:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:43 smithi077 ceph-mon[37153]: pgmap v9826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:44:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:45 smithi047 ceph-mon[36835]: pgmap v9827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:46.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:45 smithi029 ceph-mon[32610]: pgmap v9827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:45 smithi077 ceph-mon[37153]: pgmap v9827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:47 smithi047 ceph-mon[36835]: pgmap v9828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:44:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17709.773516 secs 2024-07-27T11:44:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:47 smithi029 ceph-mon[32610]: pgmap v9828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:44:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17709.773516 secs 2024-07-27T11:44:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:47 smithi077 ceph-mon[37153]: pgmap v9828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:44:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17709.773516 secs 2024-07-27T11:44:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:49 smithi047 ceph-mon[36835]: pgmap v9829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:49 smithi029 ceph-mon[32610]: pgmap v9829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:49 smithi077 ceph-mon[37153]: pgmap v9829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:44:52.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:52.014+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:44:52.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:52.014+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:44:52.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:44:52.014+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:44:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:51 smithi047 ceph-mon[36835]: pgmap v9830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:52.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:51 smithi029 ceph-mon[32610]: pgmap v9830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:51 smithi077 ceph-mon[37153]: pgmap v9830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:53.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17714.773655 secs 2024-07-27T11:44:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17714.773655 secs 2024-07-27T11:44:53.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17714.773655 secs 2024-07-27T11:44:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:53 smithi047 ceph-mon[36835]: pgmap v9831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:44:54.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:53 smithi029 ceph-mon[32610]: pgmap v9831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:44:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:53 smithi077 ceph-mon[37153]: pgmap v9831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:44:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:55 smithi047 ceph-mon[36835]: pgmap v9832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:56.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:55 smithi029 ceph-mon[32610]: pgmap v9832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:55 smithi077 ceph-mon[37153]: pgmap v9832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:44:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:57 smithi047 ceph-mon[36835]: pgmap v9833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:44:58.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17719.773836 secs 2024-07-27T11:44:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:57 smithi029 ceph-mon[32610]: pgmap v9833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:44:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17719.773836 secs 2024-07-27T11:44:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:57 smithi077 ceph-mon[37153]: pgmap v9833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:44:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17719.773836 secs 2024-07-27T11:45:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:44:59 smithi047 ceph-mon[36835]: pgmap v9834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:00.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:44:59 smithi029 ceph-mon[32610]: pgmap v9834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:44:59 smithi077 ceph-mon[37153]: pgmap v9834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:01 smithi047 ceph-mon[36835]: pgmap v9835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:02.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:01 smithi029 ceph-mon[32610]: pgmap v9835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:01 smithi077 ceph-mon[37153]: pgmap v9835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17724.773944 secs 2024-07-27T11:45:03.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17724.773944 secs 2024-07-27T11:45:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17724.773944 secs 2024-07-27T11:45:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:03 smithi047 ceph-mon[36835]: pgmap v9836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:45:04.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:03 smithi029 ceph-mon[32610]: pgmap v9836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:45:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:03 smithi077 ceph-mon[37153]: pgmap v9836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:45:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:05 smithi047 ceph-mon[36835]: pgmap v9837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:45:06.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:05 smithi029 ceph-mon[32610]: pgmap v9837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:45:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:05 smithi077 ceph-mon[37153]: pgmap v9837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:45:07.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:45:07.028+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:45:07.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:45:07.028+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:45:07.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:45:07.028+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:45:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:07 smithi047 ceph-mon[36835]: pgmap v9838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:45:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17729.774050 secs 2024-07-27T11:45:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:07 smithi029 ceph-mon[32610]: pgmap v9838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:45:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17729.774050 secs 2024-07-27T11:45:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:07 smithi077 ceph-mon[37153]: pgmap v9838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:45:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17729.774050 secs 2024-07-27T11:45:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:09 smithi047 ceph-mon[36835]: pgmap v9839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:10.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:09 smithi029 ceph-mon[32610]: pgmap v9839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:09 smithi077 ceph-mon[37153]: pgmap v9839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:11 smithi047 ceph-mon[36835]: pgmap v9840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:11 smithi029 ceph-mon[32610]: pgmap v9840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:11 smithi077 ceph-mon[37153]: pgmap v9840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17734.774172 secs 2024-07-27T11:45:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17734.774172 secs 2024-07-27T11:45:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17734.774172 secs 2024-07-27T11:45:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:13 smithi047 ceph-mon[36835]: pgmap v9841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:45:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:13 smithi029 ceph-mon[32610]: pgmap v9841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:45:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:13 smithi077 ceph-mon[37153]: pgmap v9841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:45:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:15 smithi047 ceph-mon[36835]: pgmap v9842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:15 smithi029 ceph-mon[32610]: pgmap v9842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:15 smithi077 ceph-mon[37153]: pgmap v9842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:17 smithi047 ceph-mon[36835]: pgmap v9843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:45:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17739.774313 secs 2024-07-27T11:45:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:17 smithi029 ceph-mon[32610]: pgmap v9843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:45:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17739.774313 secs 2024-07-27T11:45:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:17 smithi077 ceph-mon[37153]: pgmap v9843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:45:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17739.774313 secs 2024-07-27T11:45:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:19 smithi047 ceph-mon[36835]: pgmap v9844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:19 smithi029 ceph-mon[32610]: pgmap v9844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:19 smithi077 ceph-mon[37153]: pgmap v9844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:22.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:45:22.044+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:45:22.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:45:22.044+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:45:22.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:45:22.044+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:45:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:21 smithi029 ceph-mon[32610]: pgmap v9845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:21 smithi077 ceph-mon[37153]: pgmap v9845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:21 smithi047 ceph-mon[36835]: pgmap v9845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17744.774439 secs 2024-07-27T11:45:23.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17744.774439 secs 2024-07-27T11:45:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17744.774439 secs 2024-07-27T11:45:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:23 smithi029 ceph-mon[32610]: pgmap v9846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:45:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:23 smithi077 ceph-mon[37153]: pgmap v9846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:45:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:23 smithi047 ceph-mon[36835]: pgmap v9846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:45:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:25 smithi029 ceph-mon[32610]: pgmap v9847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:45:26.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:25 smithi077 ceph-mon[37153]: pgmap v9847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:45:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:25 smithi047 ceph-mon[36835]: pgmap v9847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:45:28.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:27 smithi029 ceph-mon[32610]: pgmap v9848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:45:28.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17749.774577 secs 2024-07-27T11:45:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:27 smithi077 ceph-mon[37153]: pgmap v9848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:45:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17749.774577 secs 2024-07-27T11:45:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:27 smithi047 ceph-mon[36835]: pgmap v9848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:45:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17749.774577 secs 2024-07-27T11:45:29.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:28 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:45:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:28 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:45:29.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:28 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:45:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:29 smithi077 ceph-mon[37153]: pgmap v9849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:45:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:45:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:45:30.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:29 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:45:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:29 smithi047 ceph-mon[36835]: pgmap v9849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:45:30.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:45:30.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:45:30.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:29 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:45:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:29 smithi029 ceph-mon[32610]: pgmap v9849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:45:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:45:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:45:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:29 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:45:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:30 smithi077 ceph-mon[37153]: pgmap v9850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:45:31.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:30 smithi077 ceph-mon[37153]: pgmap v9851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:45:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:30 smithi047 ceph-mon[36835]: pgmap v9850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:45:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:30 smithi047 ceph-mon[36835]: pgmap v9851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:45:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:30 smithi029 ceph-mon[32610]: pgmap v9850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:45:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:30 smithi029 ceph-mon[32610]: pgmap v9851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:45:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:32 smithi077 ceph-mon[37153]: pgmap v9852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:45:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17754.774701 secs 2024-07-27T11:45:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:32 smithi047 ceph-mon[36835]: pgmap v9852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:45:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17754.774701 secs 2024-07-27T11:45:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:32 smithi029 ceph-mon[32610]: pgmap v9852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:45:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17754.774701 secs 2024-07-27T11:45:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:34 smithi077 ceph-mon[37153]: pgmap v9853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:45:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:34 smithi047 ceph-mon[36835]: pgmap v9853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:45:35.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:34 smithi029 ceph-mon[32610]: pgmap v9853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:45:37.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:45:37.060+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:45:37.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:45:37.060+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:45:37.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:45:37.060+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:45:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:36 smithi077 ceph-mon[37153]: pgmap v9854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:45:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:36 smithi047 ceph-mon[36835]: pgmap v9854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:45:37.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:36 smithi029 ceph-mon[32610]: pgmap v9854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:45:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17759.774830 secs 2024-07-27T11:45:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17759.774830 secs 2024-07-27T11:45:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17759.774830 secs 2024-07-27T11:45:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:38 smithi077 ceph-mon[37153]: pgmap v9855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:45:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:38 smithi047 ceph-mon[36835]: pgmap v9855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:45:39.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:38 smithi029 ceph-mon[32610]: pgmap v9855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:45:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:40 smithi077 ceph-mon[37153]: pgmap v9856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:45:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:40 smithi047 ceph-mon[36835]: pgmap v9856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:45:41.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:40 smithi029 ceph-mon[32610]: pgmap v9856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:45:43.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:42 smithi029 ceph-mon[32610]: pgmap v9857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:45:43.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17764.774908 secs 2024-07-27T11:45:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:42 smithi077 ceph-mon[37153]: pgmap v9857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:45:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17764.774908 secs 2024-07-27T11:45:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:42 smithi047 ceph-mon[36835]: pgmap v9857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:45:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17764.774908 secs 2024-07-27T11:45:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:44 smithi077 ceph-mon[37153]: pgmap v9858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:44 smithi047 ceph-mon[36835]: pgmap v9858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:44 smithi029 ceph-mon[32610]: pgmap v9858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:47 smithi077 ceph-mon[37153]: pgmap v9859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:47 smithi047 ceph-mon[36835]: pgmap v9859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:47 smithi029 ceph-mon[32610]: pgmap v9859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17769.775034 secs 2024-07-27T11:45:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17769.775034 secs 2024-07-27T11:45:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17769.775034 secs 2024-07-27T11:45:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:49 smithi077 ceph-mon[37153]: pgmap v9860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:45:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:49 smithi047 ceph-mon[36835]: pgmap v9860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:45:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:49 smithi029 ceph-mon[32610]: pgmap v9860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:45:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:51 smithi077 ceph-mon[37153]: pgmap v9861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:45:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:51 smithi047 ceph-mon[36835]: pgmap v9861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:45:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:51 smithi029 ceph-mon[32610]: pgmap v9861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:45:52.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:45:52.075+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:45:52.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:45:52.075+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:45:52.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:45:52.075+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:45:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:53 smithi077 ceph-mon[37153]: pgmap v9862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:45:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17774.775158 secs 2024-07-27T11:45:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:53 smithi047 ceph-mon[36835]: pgmap v9862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:45:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17774.775158 secs 2024-07-27T11:45:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:53 smithi029 ceph-mon[32610]: pgmap v9862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:45:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17774.775158 secs 2024-07-27T11:45:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:55 smithi077 ceph-mon[37153]: pgmap v9863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:55 smithi047 ceph-mon[36835]: pgmap v9863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:55 smithi029 ceph-mon[32610]: pgmap v9863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:45:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:57 smithi077 ceph-mon[37153]: pgmap v9864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:57 smithi047 ceph-mon[36835]: pgmap v9864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:57.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:57 smithi029 ceph-mon[32610]: pgmap v9864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:45:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17779.775287 secs 2024-07-27T11:45:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17779.775287 secs 2024-07-27T11:45:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17779.775287 secs 2024-07-27T11:45:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:45:59 smithi077 ceph-mon[37153]: pgmap v9865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:45:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:45:59 smithi047 ceph-mon[36835]: pgmap v9865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:45:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:45:59 smithi029 ceph-mon[32610]: pgmap v9865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:46:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:01 smithi077 ceph-mon[37153]: pgmap v9866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:46:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:01 smithi047 ceph-mon[36835]: pgmap v9866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:46:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:01 smithi029 ceph-mon[32610]: pgmap v9866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:46:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:03 smithi077 ceph-mon[37153]: pgmap v9867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:03.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17784.775420 secs 2024-07-27T11:46:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:03 smithi047 ceph-mon[36835]: pgmap v9867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:03.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17784.775420 secs 2024-07-27T11:46:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:03 smithi029 ceph-mon[32610]: pgmap v9867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17784.775420 secs 2024-07-27T11:46:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:05 smithi077 ceph-mon[37153]: pgmap v9868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:05 smithi047 ceph-mon[36835]: pgmap v9868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:05 smithi029 ceph-mon[32610]: pgmap v9868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:07.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:46:07.088+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:46:07.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:46:07.088+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:46:07.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:46:07.088+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:46:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:07 smithi077 ceph-mon[37153]: pgmap v9869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:07 smithi047 ceph-mon[36835]: pgmap v9869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:07.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:07 smithi029 ceph-mon[32610]: pgmap v9869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17789.775525 secs 2024-07-27T11:46:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17789.775525 secs 2024-07-27T11:46:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17789.775525 secs 2024-07-27T11:46:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:09 smithi077 ceph-mon[37153]: pgmap v9870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:09 smithi047 ceph-mon[36835]: pgmap v9870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:09.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:09 smithi029 ceph-mon[32610]: pgmap v9870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:11 smithi077 ceph-mon[37153]: pgmap v9871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:46:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:11 smithi047 ceph-mon[36835]: pgmap v9871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:46:11.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:11 smithi029 ceph-mon[32610]: pgmap v9871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:46:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:13 smithi077 ceph-mon[37153]: pgmap v9872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:13.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17794.775627 secs 2024-07-27T11:46:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:13 smithi047 ceph-mon[36835]: pgmap v9872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17794.775627 secs 2024-07-27T11:46:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:13 smithi029 ceph-mon[32610]: pgmap v9872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17794.775627 secs 2024-07-27T11:46:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:15 smithi077 ceph-mon[37153]: pgmap v9873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:15 smithi047 ceph-mon[36835]: pgmap v9873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:15 smithi029 ceph-mon[32610]: pgmap v9873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:17 smithi077 ceph-mon[37153]: pgmap v9874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:17 smithi047 ceph-mon[36835]: pgmap v9874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:17.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:17 smithi029 ceph-mon[32610]: pgmap v9874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17799.775754 secs 2024-07-27T11:46:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17799.775754 secs 2024-07-27T11:46:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17799.775754 secs 2024-07-27T11:46:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:19 smithi077 ceph-mon[37153]: pgmap v9875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:19 smithi047 ceph-mon[36835]: pgmap v9875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:19.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:19 smithi029 ceph-mon[32610]: pgmap v9875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:21 smithi077 ceph-mon[37153]: pgmap v9876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:46:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:21 smithi047 ceph-mon[36835]: pgmap v9876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:46:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:21 smithi029 ceph-mon[32610]: pgmap v9876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:46:22.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:46:22.094+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:46:22.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:46:22.094+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:46:22.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:46:22.094+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:46:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:23 smithi077 ceph-mon[37153]: pgmap v9877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:23.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17804.775909 secs 2024-07-27T11:46:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:23 smithi047 ceph-mon[36835]: pgmap v9877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17804.775909 secs 2024-07-27T11:46:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:23 smithi029 ceph-mon[32610]: pgmap v9877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17804.775909 secs 2024-07-27T11:46:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:25 smithi077 ceph-mon[37153]: pgmap v9878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:25 smithi047 ceph-mon[36835]: pgmap v9878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:25.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:25 smithi029 ceph-mon[32610]: pgmap v9878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:27 smithi077 ceph-mon[37153]: pgmap v9879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:27 smithi047 ceph-mon[36835]: pgmap v9879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:27 smithi029 ceph-mon[32610]: pgmap v9879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17809.775978 secs 2024-07-27T11:46:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17809.775978 secs 2024-07-27T11:46:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17809.775978 secs 2024-07-27T11:46:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:29 smithi077 ceph-mon[37153]: pgmap v9880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:29 smithi047 ceph-mon[36835]: pgmap v9880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:29.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:29 smithi029 ceph-mon[32610]: pgmap v9880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:30.348 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:30 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:46:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:30 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:46:30.444 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:30 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:46:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:31 smithi077 ceph-mon[37153]: pgmap v9881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:46:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:31 smithi047 ceph-mon[36835]: pgmap v9881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:46:31.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:31 smithi029 ceph-mon[32610]: pgmap v9881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:46:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:33 smithi077 ceph-mon[37153]: pgmap v9882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17814.776108 secs 2024-07-27T11:46:33.369 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:33 smithi047 ceph-mon[36835]: pgmap v9882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:33.369 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17814.776108 secs 2024-07-27T11:46:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:33 smithi029 ceph-mon[32610]: pgmap v9882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17814.776108 secs 2024-07-27T11:46:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: pgmap v9883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:46:34.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:46:34.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:46:34.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:46:34.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:46:34.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:46:34.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:46:34.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:46:34.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: pgmap v9884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T11:46:34.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: pgmap v9885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:46:34.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: pgmap v9883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:46:34.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:46:34.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:46:34.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:46:34.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:46:34.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:46:34.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:46:34.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:46:34.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: pgmap v9884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T11:46:34.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: pgmap v9885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:46:34.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: pgmap v9883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:34.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:46:34.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:46:34.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:46:34.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:46:34.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:46:34.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:46:34.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:46:34.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:46:34.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: pgmap v9884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T11:46:34.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: pgmap v9885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:46:34.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:46:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:36 smithi047 ceph-mon[36835]: pgmap v9886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-27T11:46:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:36 smithi029 ceph-mon[32610]: pgmap v9886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-27T11:46:37.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:36 smithi077 ceph-mon[37153]: pgmap v9886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-27T11:46:37.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:46:37.109+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:46:37.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:46:37.109+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:46:37.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:46:37.109+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:46:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17819.776236 secs 2024-07-27T11:46:37.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17819.776236 secs 2024-07-27T11:46:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17819.776236 secs 2024-07-27T11:46:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:38 smithi047 ceph-mon[36835]: pgmap v9887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:46:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:38 smithi029 ceph-mon[32610]: pgmap v9887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:46:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:38 smithi077 ceph-mon[37153]: pgmap v9887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:46:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:40 smithi047 ceph-mon[36835]: pgmap v9888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T11:46:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:40 smithi029 ceph-mon[32610]: pgmap v9888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T11:46:41.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:40 smithi077 ceph-mon[37153]: pgmap v9888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T11:46:42.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:42 smithi029 ceph-mon[32610]: pgmap v9889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T11:46:42.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17824.776370 secs 2024-07-27T11:46:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:42 smithi047 ceph-mon[36835]: pgmap v9889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T11:46:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17824.776370 secs 2024-07-27T11:46:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:42 smithi077 ceph-mon[37153]: pgmap v9889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-27T11:46:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17824.776370 secs 2024-07-27T11:46:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:44 smithi047 ceph-mon[36835]: pgmap v9890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:46:44.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:44 smithi029 ceph-mon[32610]: pgmap v9890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:46:45.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:44 smithi077 ceph-mon[37153]: pgmap v9890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:46:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:46 smithi047 ceph-mon[36835]: pgmap v9891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:46 smithi029 ceph-mon[32610]: pgmap v9891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:46 smithi077 ceph-mon[37153]: pgmap v9891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17829.776502 secs 2024-07-27T11:46:47.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17829.776502 secs 2024-07-27T11:46:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17829.776502 secs 2024-07-27T11:46:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:48 smithi047 ceph-mon[36835]: pgmap v9892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:48.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:48 smithi029 ceph-mon[32610]: pgmap v9892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:48 smithi077 ceph-mon[37153]: pgmap v9892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:50 smithi047 ceph-mon[36835]: pgmap v9893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:46:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:50 smithi029 ceph-mon[32610]: pgmap v9893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:46:51.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:50 smithi077 ceph-mon[37153]: pgmap v9893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:46:52.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:46:52.125+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:46:52.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:46:52.125+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:46:52.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:46:52.125+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:46:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:52 smithi047 ceph-mon[36835]: pgmap v9894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17834.776632 secs 2024-07-27T11:46:52.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:52 smithi029 ceph-mon[32610]: pgmap v9894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17834.776632 secs 2024-07-27T11:46:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:52 smithi077 ceph-mon[37153]: pgmap v9894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:46:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17834.776632 secs 2024-07-27T11:46:54.908 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:54 smithi077 ceph-mon[37153]: pgmap v9895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:54 smithi047 ceph-mon[36835]: pgmap v9895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:54.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:54 smithi029 ceph-mon[32610]: pgmap v9895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:46:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:56 smithi047 ceph-mon[36835]: pgmap v9896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:56 smithi029 ceph-mon[32610]: pgmap v9896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:57.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:56 smithi077 ceph-mon[37153]: pgmap v9896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:46:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17839.776764 secs 2024-07-27T11:46:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17839.776764 secs 2024-07-27T11:46:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17839.776764 secs 2024-07-27T11:46:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:46:58 smithi047 ceph-mon[36835]: pgmap v9897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:46:58 smithi029 ceph-mon[32610]: pgmap v9897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:46:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:46:58 smithi077 ceph-mon[37153]: pgmap v9897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:47:00.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:00.339+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:47:00.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:00.339+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:47:00.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:00.340+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T11:47:00.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:00.340+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:47:00.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:00.340+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T11:47:00.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:00.340+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:47:00.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:00.340+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:47:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:00.340+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:47:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:00 smithi047 ceph-mon[36835]: pgmap v9898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:47:00.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:00 smithi029 ceph-mon[32610]: pgmap v9898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:47:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:00 smithi077 ceph-mon[37153]: pgmap v9898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:47:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:02 smithi047 ceph-mon[36835]: pgmap v9899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17844.776843 secs 2024-07-27T11:47:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:02 smithi029 ceph-mon[32610]: pgmap v9899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17844.776843 secs 2024-07-27T11:47:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:02 smithi077 ceph-mon[37153]: pgmap v9899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:03.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17844.776843 secs 2024-07-27T11:47:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:04 smithi047 ceph-mon[36835]: pgmap v9900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:47:04.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:04 smithi029 ceph-mon[32610]: pgmap v9900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:47:05.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:04 smithi077 ceph-mon[37153]: pgmap v9900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:47:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:06 smithi047 ceph-mon[36835]: pgmap v9901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:47:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:06 smithi029 ceph-mon[32610]: pgmap v9901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:47:07.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:06 smithi077 ceph-mon[37153]: pgmap v9901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:47:07.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:07.130+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:47:07.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:07.130+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:47:07.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:07.130+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:47:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17849.776951 secs 2024-07-27T11:47:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17849.776951 secs 2024-07-27T11:47:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17849.776951 secs 2024-07-27T11:47:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:08 smithi047 ceph-mon[36835]: pgmap v9902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:47:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:08 smithi029 ceph-mon[32610]: pgmap v9902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:47:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:08 smithi077 ceph-mon[37153]: pgmap v9902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:47:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:10 smithi029 ceph-mon[32610]: pgmap v9903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:47:11.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:10 smithi077 ceph-mon[37153]: pgmap v9903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:47:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:10 smithi047 ceph-mon[36835]: pgmap v9903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:47:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:12 smithi029 ceph-mon[32610]: pgmap v9904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17854.777041 secs 2024-07-27T11:47:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:12 smithi077 ceph-mon[37153]: pgmap v9904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17854.777041 secs 2024-07-27T11:47:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:12 smithi047 ceph-mon[36835]: pgmap v9904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17854.777041 secs 2024-07-27T11:47:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:14 smithi029 ceph-mon[32610]: pgmap v9905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:47:15.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:14 smithi077 ceph-mon[37153]: pgmap v9905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:47:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:14 smithi047 ceph-mon[36835]: pgmap v9905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:47:16.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:16 smithi029 ceph-mon[32610]: pgmap v9906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:47:17.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:16 smithi077 ceph-mon[37153]: pgmap v9906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:47:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:16 smithi047 ceph-mon[36835]: pgmap v9906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:47:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17859.777147 secs 2024-07-27T11:47:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17859.777147 secs 2024-07-27T11:47:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17859.777147 secs 2024-07-27T11:47:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:18 smithi077 ceph-mon[37153]: pgmap v9907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:47:19.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:18 smithi047 ceph-mon[36835]: pgmap v9907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:47:19.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:18 smithi029 ceph-mon[32610]: pgmap v9907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:47:21.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:20 smithi077 ceph-mon[37153]: pgmap v9908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:47:21.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:20 smithi047 ceph-mon[36835]: pgmap v9908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:47:21.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:20 smithi029 ceph-mon[32610]: pgmap v9908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:47:22.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:22.136+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:47:22.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:22.136+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:47:22.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:22.136+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:47:22.998 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:22 smithi047 ceph-mon[36835]: pgmap v9909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:22.999 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17864.777275 secs 2024-07-27T11:47:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:22 smithi077 ceph-mon[37153]: pgmap v9909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17864.777275 secs 2024-07-27T11:47:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:22 smithi029 ceph-mon[32610]: pgmap v9909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17864.777275 secs 2024-07-27T11:47:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:24 smithi077 ceph-mon[37153]: pgmap v9910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:47:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:24 smithi047 ceph-mon[36835]: pgmap v9910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:47:25.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:24 smithi029 ceph-mon[32610]: pgmap v9910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:47:27.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:26 smithi077 ceph-mon[37153]: pgmap v9911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:47:27.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:26 smithi047 ceph-mon[36835]: pgmap v9911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:47:27.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:26 smithi029 ceph-mon[32610]: pgmap v9911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:47:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17869.777397 secs 2024-07-27T11:47:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17869.777397 secs 2024-07-27T11:47:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17869.777397 secs 2024-07-27T11:47:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:28 smithi077 ceph-mon[37153]: pgmap v9912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:47:29.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:28 smithi047 ceph-mon[36835]: pgmap v9912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:47:29.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:28 smithi029 ceph-mon[32610]: pgmap v9912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:47:31.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:30 smithi077 ceph-mon[37153]: pgmap v9913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:47:31.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:30 smithi047 ceph-mon[36835]: pgmap v9913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:47:31.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:30 smithi029 ceph-mon[32610]: pgmap v9913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:47:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:32 smithi077 ceph-mon[37153]: pgmap v9914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:33.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17874.777549 secs 2024-07-27T11:47:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:32 smithi047 ceph-mon[36835]: pgmap v9914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17874.777549 secs 2024-07-27T11:47:33.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:32 smithi029 ceph-mon[32610]: pgmap v9914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17874.777549 secs 2024-07-27T11:47:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:33 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:47:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:33 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:47:34.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:33 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:47:34.850 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:34 smithi077 ceph-mon[37153]: pgmap v9915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:47:34.850 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:34.851 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:34.851 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:34.851 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:34 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:34.851 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:34 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:47:34.851 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:34 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:47:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:34 smithi047 ceph-mon[36835]: pgmap v9915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:47:35.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:34 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:34 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:47:35.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:34 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:47:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:34 smithi029 ceph-mon[32610]: pgmap v9915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:47:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:34 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:35.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:34 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:47:35.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:34 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:47:36.907 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:36 smithi077 ceph-mon[37153]: pgmap v9916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:47:37.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:37.151+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:47:37.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:37.151+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:47:37.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:37.151+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:47:37.153 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:36 smithi029 ceph-mon[32610]: pgmap v9916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:47:37.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:36 smithi047 ceph-mon[36835]: pgmap v9916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:47:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17879.777660 secs 2024-07-27T11:47:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:47:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:47:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:38.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:47:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:47:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:47:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:47:38.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: pgmap v9917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T11:47:38.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: pgmap v9918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:38.699 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:47:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17879.777660 secs 2024-07-27T11:47:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:47:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:47:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:47:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:47:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:47:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:47:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: pgmap v9917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T11:47:38.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: pgmap v9918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:38.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:47:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17879.777660 secs 2024-07-27T11:47:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:47:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:47:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:47:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:47:38.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:47:38.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:47:38.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:47:38.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: pgmap v9917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T11:47:38.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: pgmap v9918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:38.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:47:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:40 smithi029 ceph-mon[32610]: pgmap v9919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:40 smithi077 ceph-mon[37153]: pgmap v9919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:40 smithi047 ceph-mon[36835]: pgmap v9919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:42 smithi029 ceph-mon[32610]: pgmap v9920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:42 smithi077 ceph-mon[37153]: pgmap v9920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:42 smithi047 ceph-mon[36835]: pgmap v9920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17884.777806 secs 2024-07-27T11:47:43.901 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17884.777806 secs 2024-07-27T11:47:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17884.777806 secs 2024-07-27T11:47:44.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:44 smithi029 ceph-mon[32610]: pgmap v9921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:44 smithi077 ceph-mon[37153]: pgmap v9921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:44 smithi047 ceph-mon[36835]: pgmap v9921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:46 smithi077 ceph-mon[37153]: pgmap v9922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:46 smithi047 ceph-mon[36835]: pgmap v9922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:46.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:46 smithi029 ceph-mon[32610]: pgmap v9922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:47:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17889.777895 secs 2024-07-27T11:47:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17889.777895 secs 2024-07-27T11:47:47.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17889.777895 secs 2024-07-27T11:47:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:48 smithi077 ceph-mon[37153]: pgmap v9923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:47:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:48 smithi047 ceph-mon[36835]: pgmap v9923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:47:48.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:48 smithi029 ceph-mon[32610]: pgmap v9923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:47:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:50 smithi077 ceph-mon[37153]: pgmap v9924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:47:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:50 smithi047 ceph-mon[36835]: pgmap v9924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:47:50.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:50 smithi029 ceph-mon[32610]: pgmap v9924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:47:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:52.158+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:47:52.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:52.158+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:47:52.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:47:52.158+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:47:52.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:52 smithi047 ceph-mon[36835]: pgmap v9925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:52.749 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17894.778015 secs 2024-07-27T11:47:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:52 smithi077 ceph-mon[37153]: pgmap v9925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17894.778015 secs 2024-07-27T11:47:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:52 smithi029 ceph-mon[32610]: pgmap v9925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:47:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17894.778015 secs 2024-07-27T11:47:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:54 smithi029 ceph-mon[32610]: pgmap v9926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:47:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:54 smithi077 ceph-mon[37153]: pgmap v9926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:47:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:54 smithi047 ceph-mon[36835]: pgmap v9926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:47:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:56 smithi077 ceph-mon[37153]: pgmap v9927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:47:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:56 smithi047 ceph-mon[36835]: pgmap v9927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:47:56.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:56 smithi029 ceph-mon[32610]: pgmap v9927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:47:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17899.778139 secs 2024-07-27T11:47:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17899.778139 secs 2024-07-27T11:47:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17899.778139 secs 2024-07-27T11:47:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:47:58 smithi077 ceph-mon[37153]: pgmap v9928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:47:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:47:58 smithi047 ceph-mon[36835]: pgmap v9928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:47:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:47:58 smithi029 ceph-mon[32610]: pgmap v9928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:48:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:00 smithi077 ceph-mon[37153]: pgmap v9929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:48:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:00 smithi047 ceph-mon[36835]: pgmap v9929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:48:00.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:00 smithi029 ceph-mon[32610]: pgmap v9929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:48:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:02 smithi077 ceph-mon[37153]: pgmap v9930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:48:02.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17904.778237 secs 2024-07-27T11:48:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:02 smithi047 ceph-mon[36835]: pgmap v9930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:48:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17904.778237 secs 2024-07-27T11:48:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:02 smithi029 ceph-mon[32610]: pgmap v9930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:48:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17904.778237 secs 2024-07-27T11:48:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:04 smithi077 ceph-mon[37153]: pgmap v9931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:48:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:04 smithi047 ceph-mon[36835]: pgmap v9931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:48:04.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:04 smithi029 ceph-mon[32610]: pgmap v9931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:48:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:06 smithi077 ceph-mon[37153]: pgmap v9932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:06 smithi047 ceph-mon[36835]: pgmap v9932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:06 smithi029 ceph-mon[32610]: pgmap v9932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:07.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:48:07.167+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:48:07.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:48:07.167+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:48:07.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:48:07.167+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:48:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17909.778364 secs 2024-07-27T11:48:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17909.778364 secs 2024-07-27T11:48:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17909.778364 secs 2024-07-27T11:48:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:08 smithi077 ceph-mon[37153]: pgmap v9933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:48:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:08 smithi047 ceph-mon[36835]: pgmap v9933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:48:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:08 smithi029 ceph-mon[32610]: pgmap v9933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:48:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:10 smithi077 ceph-mon[37153]: pgmap v9934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:48:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:10 smithi047 ceph-mon[36835]: pgmap v9934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:48:10.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:10 smithi029 ceph-mon[32610]: pgmap v9934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:48:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:12 smithi077 ceph-mon[37153]: pgmap v9935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:48:12.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17914.778459 secs 2024-07-27T11:48:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:12 smithi047 ceph-mon[36835]: pgmap v9935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:48:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17914.778459 secs 2024-07-27T11:48:12.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:12 smithi029 ceph-mon[32610]: pgmap v9935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:48:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17914.778459 secs 2024-07-27T11:48:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:14 smithi077 ceph-mon[37153]: pgmap v9936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:48:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:14 smithi047 ceph-mon[36835]: pgmap v9936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:48:14.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:14 smithi029 ceph-mon[32610]: pgmap v9936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:48:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:16 smithi077 ceph-mon[37153]: pgmap v9937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:16 smithi047 ceph-mon[36835]: pgmap v9937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:16.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:16 smithi029 ceph-mon[32610]: pgmap v9937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17919.778596 secs 2024-07-27T11:48:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17919.778596 secs 2024-07-27T11:48:17.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17919.778596 secs 2024-07-27T11:48:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:18 smithi077 ceph-mon[37153]: pgmap v9938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:48:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:18 smithi047 ceph-mon[36835]: pgmap v9938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:48:18.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:18 smithi029 ceph-mon[32610]: pgmap v9938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:48:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:20 smithi077 ceph-mon[37153]: pgmap v9939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:48:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:20 smithi047 ceph-mon[36835]: pgmap v9939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:48:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:20 smithi029 ceph-mon[32610]: pgmap v9939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:48:22.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:48:22.168+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:48:22.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:48:22.168+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:48:22.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:48:22.168+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:48:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:22 smithi077 ceph-mon[37153]: pgmap v9940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:48:22.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17924.778698 secs 2024-07-27T11:48:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:22 smithi047 ceph-mon[36835]: pgmap v9940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:48:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17924.778698 secs 2024-07-27T11:48:22.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:22 smithi029 ceph-mon[32610]: pgmap v9940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:48:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17924.778698 secs 2024-07-27T11:48:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:24 smithi077 ceph-mon[37153]: pgmap v9941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:48:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:24 smithi047 ceph-mon[36835]: pgmap v9941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:48:24.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:24 smithi029 ceph-mon[32610]: pgmap v9941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:48:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:26 smithi077 ceph-mon[37153]: pgmap v9942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:26 smithi047 ceph-mon[36835]: pgmap v9942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:26.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:26 smithi029 ceph-mon[32610]: pgmap v9942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17929.778827 secs 2024-07-27T11:48:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17929.778827 secs 2024-07-27T11:48:27.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17929.778827 secs 2024-07-27T11:48:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:28 smithi077 ceph-mon[37153]: pgmap v9943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:48:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:28 smithi047 ceph-mon[36835]: pgmap v9943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:48:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:28 smithi029 ceph-mon[32610]: pgmap v9943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:48:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:30 smithi077 ceph-mon[37153]: pgmap v9944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:48:30.887 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:30 smithi029 ceph-mon[32610]: pgmap v9944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:48:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:30 smithi047 ceph-mon[36835]: pgmap v9944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:48:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:32 smithi077 ceph-mon[37153]: pgmap v9945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:48:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17934.778936 secs 2024-07-27T11:48:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:32 smithi047 ceph-mon[36835]: pgmap v9945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:48:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17934.778936 secs 2024-07-27T11:48:32.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:32 smithi029 ceph-mon[32610]: pgmap v9945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:48:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17934.778936 secs 2024-07-27T11:48:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:34 smithi077 ceph-mon[37153]: pgmap v9946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:48:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:34 smithi047 ceph-mon[36835]: pgmap v9946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:48:34.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:34 smithi029 ceph-mon[32610]: pgmap v9946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T11:48:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:36 smithi077 ceph-mon[37153]: pgmap v9947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:48:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:36 smithi047 ceph-mon[36835]: pgmap v9947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:48:36.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:36 smithi029 ceph-mon[32610]: pgmap v9947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:48:37.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:48:37.184+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:48:37.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:48:37.184+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:48:37.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:48:37.184+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:48:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17939.779082 secs 2024-07-27T11:48:37.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:37 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:48:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17939.779082 secs 2024-07-27T11:48:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:37 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:48:37.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17939.779082 secs 2024-07-27T11:48:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:37 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:48:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:38 smithi077 ceph-mon[37153]: pgmap v9948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T11:48:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:48:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:48:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:38 smithi077 ceph-mon[37153]: pgmap v9949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T11:48:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:38 smithi077 ceph-mon[37153]: pgmap v9950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-07-27T11:48:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:38 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:48:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:38 smithi047 ceph-mon[36835]: pgmap v9948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T11:48:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:48:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:48:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:38 smithi047 ceph-mon[36835]: pgmap v9949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T11:48:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:38 smithi047 ceph-mon[36835]: pgmap v9950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-07-27T11:48:38.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:38 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:48:38.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:38 smithi029 ceph-mon[32610]: pgmap v9948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T11:48:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:48:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:48:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:38 smithi029 ceph-mon[32610]: pgmap v9949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T11:48:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:38 smithi029 ceph-mon[32610]: pgmap v9950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-07-27T11:48:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:38 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:48:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:41 smithi077 ceph-mon[37153]: pgmap v9951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:48:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:41 smithi047 ceph-mon[36835]: pgmap v9951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:48:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:41 smithi029 ceph-mon[32610]: pgmap v9951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:48:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:43 smithi077 ceph-mon[37153]: pgmap v9952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:48:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17944.779156 secs 2024-07-27T11:48:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:43 smithi047 ceph-mon[36835]: pgmap v9952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:48:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17944.779156 secs 2024-07-27T11:48:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:43 smithi029 ceph-mon[32610]: pgmap v9952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:48:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17944.779156 secs 2024-07-27T11:48:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:45 smithi077 ceph-mon[37153]: pgmap v9953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:48:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:45 smithi047 ceph-mon[36835]: pgmap v9953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:48:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:45 smithi029 ceph-mon[32610]: pgmap v9953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:48:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:47 smithi077 ceph-mon[37153]: pgmap v9954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:48:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:47 smithi047 ceph-mon[36835]: pgmap v9954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:48:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:47 smithi029 ceph-mon[32610]: pgmap v9954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:48:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17949.779230 secs 2024-07-27T11:48:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17949.779230 secs 2024-07-27T11:48:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17949.779230 secs 2024-07-27T11:48:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:49 smithi077 ceph-mon[37153]: pgmap v9955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:48:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:49 smithi047 ceph-mon[36835]: pgmap v9955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:48:49.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:49 smithi029 ceph-mon[32610]: pgmap v9955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:48:51.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:51 smithi077 ceph-mon[37153]: pgmap v9956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:51 smithi047 ceph-mon[36835]: pgmap v9956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:51.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:51 smithi029 ceph-mon[32610]: pgmap v9956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:52.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:48:52.200+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:48:52.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:48:52.200+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:48:52.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:48:52.200+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:48:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:53 smithi077 ceph-mon[37153]: pgmap v9957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:48:53.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17954.779359 secs 2024-07-27T11:48:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:53 smithi047 ceph-mon[36835]: pgmap v9957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:48:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17954.779359 secs 2024-07-27T11:48:53.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:53 smithi029 ceph-mon[32610]: pgmap v9957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:48:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17954.779359 secs 2024-07-27T11:48:55.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:55 smithi077 ceph-mon[37153]: pgmap v9958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:48:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:55 smithi047 ceph-mon[36835]: pgmap v9958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:48:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:55 smithi029 ceph-mon[32610]: pgmap v9958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:48:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:57 smithi077 ceph-mon[37153]: pgmap v9959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:57 smithi047 ceph-mon[36835]: pgmap v9959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:57 smithi029 ceph-mon[32610]: pgmap v9959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:48:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17959.779462 secs 2024-07-27T11:48:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17959.779462 secs 2024-07-27T11:48:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17959.779462 secs 2024-07-27T11:48:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:48:59 smithi077 ceph-mon[37153]: pgmap v9960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:48:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:48:59 smithi047 ceph-mon[36835]: pgmap v9960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:48:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:48:59 smithi029 ceph-mon[32610]: pgmap v9960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:01 smithi077 ceph-mon[37153]: pgmap v9961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:01 smithi047 ceph-mon[36835]: pgmap v9961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:01.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:01 smithi029 ceph-mon[32610]: pgmap v9961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:03 smithi077 ceph-mon[37153]: pgmap v9962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:49:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17964.779611 secs 2024-07-27T11:49:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:03 smithi047 ceph-mon[36835]: pgmap v9962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:49:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17964.779611 secs 2024-07-27T11:49:03.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:03 smithi029 ceph-mon[32610]: pgmap v9962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:49:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17964.779611 secs 2024-07-27T11:49:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:05 smithi077 ceph-mon[37153]: pgmap v9963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:49:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:05 smithi047 ceph-mon[36835]: pgmap v9963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:49:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:05 smithi029 ceph-mon[32610]: pgmap v9963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:49:07.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:07.214+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:49:07.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:07.214+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:49:07.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:07.214+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:49:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:07 smithi077 ceph-mon[37153]: pgmap v9964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:07 smithi047 ceph-mon[36835]: pgmap v9964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:07.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:07 smithi029 ceph-mon[32610]: pgmap v9964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17969.779745 secs 2024-07-27T11:49:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17969.779745 secs 2024-07-27T11:49:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17969.779745 secs 2024-07-27T11:49:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:09 smithi047 ceph-mon[36835]: pgmap v9965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:09.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:09 smithi029 ceph-mon[32610]: pgmap v9965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:09 smithi077 ceph-mon[37153]: pgmap v9965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:11 smithi047 ceph-mon[36835]: pgmap v9966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:11.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:11 smithi029 ceph-mon[32610]: pgmap v9966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:11 smithi077 ceph-mon[37153]: pgmap v9966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:13 smithi047 ceph-mon[36835]: pgmap v9967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:49:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17974.779870 secs 2024-07-27T11:49:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:13 smithi029 ceph-mon[32610]: pgmap v9967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:49:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17974.779870 secs 2024-07-27T11:49:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:13 smithi077 ceph-mon[37153]: pgmap v9967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:49:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17974.779870 secs 2024-07-27T11:49:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:15 smithi047 ceph-mon[36835]: pgmap v9968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:49:15.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:15 smithi029 ceph-mon[32610]: pgmap v9968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:49:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:15 smithi077 ceph-mon[37153]: pgmap v9968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:49:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:17 smithi047 ceph-mon[36835]: pgmap v9969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:17 smithi029 ceph-mon[32610]: pgmap v9969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:17 smithi077 ceph-mon[37153]: pgmap v9969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:18.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17979.780005 secs 2024-07-27T11:49:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17979.780005 secs 2024-07-27T11:49:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17979.780005 secs 2024-07-27T11:49:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:19 smithi047 ceph-mon[36835]: pgmap v9970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:19.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:19 smithi029 ceph-mon[32610]: pgmap v9970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:19 smithi077 ceph-mon[37153]: pgmap v9970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:21 smithi047 ceph-mon[36835]: pgmap v9971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:21.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:21 smithi029 ceph-mon[32610]: pgmap v9971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:21 smithi077 ceph-mon[37153]: pgmap v9971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:22.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:22.226+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:49:22.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:22.226+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:49:22.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:22.226+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:49:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:23 smithi047 ceph-mon[36835]: pgmap v9972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:49:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17984.780100 secs 2024-07-27T11:49:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:23 smithi029 ceph-mon[32610]: pgmap v9972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:49:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17984.780100 secs 2024-07-27T11:49:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:23 smithi077 ceph-mon[37153]: pgmap v9972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:49:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17984.780100 secs 2024-07-27T11:49:25.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:24 smithi047 ceph-mon[36835]: pgmap v9973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:49:25.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:24 smithi029 ceph-mon[32610]: pgmap v9973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:49:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:24 smithi077 ceph-mon[37153]: pgmap v9973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:49:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:27 smithi077 ceph-mon[37153]: pgmap v9974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:27 smithi047 ceph-mon[36835]: pgmap v9974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:27 smithi029 ceph-mon[32610]: pgmap v9974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17989.780175 secs 2024-07-27T11:49:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17989.780175 secs 2024-07-27T11:49:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17989.780175 secs 2024-07-27T11:49:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:29 smithi077 ceph-mon[37153]: pgmap v9975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:29 smithi047 ceph-mon[36835]: pgmap v9975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:29.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:29 smithi029 ceph-mon[32610]: pgmap v9975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:31 smithi077 ceph-mon[37153]: pgmap v9976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:31 smithi047 ceph-mon[36835]: pgmap v9976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:31.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:31 smithi029 ceph-mon[32610]: pgmap v9976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:33 smithi029 ceph-mon[32610]: pgmap v9977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:49:33.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17994.780258 secs 2024-07-27T11:49:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:33 smithi077 ceph-mon[37153]: pgmap v9977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:49:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17994.780258 secs 2024-07-27T11:49:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:33 smithi047 ceph-mon[36835]: pgmap v9977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:49:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17994.780258 secs 2024-07-27T11:49:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:35 smithi077 ceph-mon[37153]: pgmap v9978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:49:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:35 smithi047 ceph-mon[36835]: pgmap v9978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:49:35.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:35 smithi029 ceph-mon[32610]: pgmap v9978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:49:37.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:37.242+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:49:37.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:37.242+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:49:37.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:37.242+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:49:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:37 smithi077 ceph-mon[37153]: pgmap v9979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:37 smithi047 ceph-mon[36835]: pgmap v9979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:37.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:37 smithi029 ceph-mon[32610]: pgmap v9979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 17999.780408 secs 2024-07-27T11:49:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 17999.780408 secs 2024-07-27T11:49:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 17999.780408 secs 2024-07-27T11:49:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:39 smithi077 ceph-mon[37153]: pgmap v9980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:49:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:49:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:49:39.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:49:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:39 smithi047 ceph-mon[36835]: pgmap v9980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:49:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:49:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:49:39.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:49:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:39 smithi029 ceph-mon[32610]: pgmap v9980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:49:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:49:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:49:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:49:39.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:49:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:40 smithi077 ceph-mon[37153]: pgmap v9981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:49:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:40 smithi077 ceph-mon[37153]: pgmap v9982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:49:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:40 smithi047 ceph-mon[36835]: pgmap v9981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:49:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:40 smithi047 ceph-mon[36835]: pgmap v9982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:49:40.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:40 smithi029 ceph-mon[32610]: pgmap v9981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:49:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:40 smithi029 ceph-mon[32610]: pgmap v9982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:49:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:42 smithi077 ceph-mon[37153]: pgmap v9983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:49:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:42 smithi047 ceph-mon[36835]: pgmap v9983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:49:42.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:42 smithi029 ceph-mon[32610]: pgmap v9983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:49:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18004.780542 secs 2024-07-27T11:49:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18004.780542 secs 2024-07-27T11:49:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18004.780542 secs 2024-07-27T11:49:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:44 smithi047 ceph-mon[36835]: pgmap v9984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:49:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:44 smithi029 ceph-mon[32610]: pgmap v9984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:49:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:44 smithi077 ceph-mon[37153]: pgmap v9984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:49:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:46 smithi047 ceph-mon[36835]: pgmap v9985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:49:46.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:46 smithi029 ceph-mon[32610]: pgmap v9985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:49:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:46 smithi077 ceph-mon[37153]: pgmap v9985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:49:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:48 smithi047 ceph-mon[36835]: pgmap v9986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:49:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18009.780688 secs 2024-07-27T11:49:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:48 smithi029 ceph-mon[32610]: pgmap v9986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:49:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18009.780688 secs 2024-07-27T11:49:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:48 smithi077 ceph-mon[37153]: pgmap v9986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:49:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18009.780688 secs 2024-07-27T11:49:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:50 smithi047 ceph-mon[36835]: pgmap v9987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:49:50.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:50 smithi029 ceph-mon[32610]: pgmap v9987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:49:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:50 smithi077 ceph-mon[37153]: pgmap v9987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:49:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:52.247+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:49:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:52.247+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:49:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:52.247+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:49:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:52 smithi047 ceph-mon[36835]: pgmap v9988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:52.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:52 smithi029 ceph-mon[32610]: pgmap v9988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:52 smithi077 ceph-mon[37153]: pgmap v9988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:49:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18014.780821 secs 2024-07-27T11:49:53.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18014.780821 secs 2024-07-27T11:49:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18014.780821 secs 2024-07-27T11:49:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:54 smithi047 ceph-mon[36835]: pgmap v9989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:49:54.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:54 smithi029 ceph-mon[32610]: pgmap v9989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:49:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:54 smithi077 ceph-mon[37153]: pgmap v9989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:49:56.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:56 smithi077 ceph-mon[37153]: pgmap v9990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:49:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:56 smithi047 ceph-mon[36835]: pgmap v9990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:49:56.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:56 smithi029 ceph-mon[32610]: pgmap v9990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:49:57.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:57.671+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T11:49:57.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:57.671+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T11:49:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:57.671+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7eff40004800 tx=0x7eff4001e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T11:49:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:57.671+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7eff40004800 tx=0x7eff4001e040 comp rx=0 tx=0).stop 2024-07-27T11:49:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:57.671+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T11:49:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:57.671+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 msgr2=0x7eff3c0c02d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T11:49:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:57.671+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5ac0 0x7eff3c0c02d0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T11:49:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:57.671+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T11:49:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:57.672+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T11:49:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:49:57.672+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7eff40004480 tx=0x7eff40043050 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T11:49:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:58 smithi047 ceph-mon[36835]: pgmap v9991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:49:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:49:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18019.780914 secs 2024-07-27T11:49:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:58 smithi029 ceph-mon[32610]: pgmap v9991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:49:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:49:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18019.780914 secs 2024-07-27T11:49:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:58 smithi077 ceph-mon[37153]: pgmap v9991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:49:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:49:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18019.780914 secs 2024-07-27T11:50:00.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:00.365+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:50:00.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:00.365+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:50:00.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:00.365+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:50:00.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:00.365+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:50:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:00 smithi047 ceph-mon[36835]: pgmap v9992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:50:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:00 smithi029 ceph-mon[32610]: pgmap v9992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:00.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:50:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:00 smithi077 ceph-mon[37153]: pgmap v9992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T11:50:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:02 smithi047 ceph-mon[36835]: pgmap v9993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:02.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:02 smithi029 ceph-mon[32610]: pgmap v9993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:02 smithi077 ceph-mon[37153]: pgmap v9993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18024.781052 secs 2024-07-27T11:50:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18024.781052 secs 2024-07-27T11:50:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18024.781052 secs 2024-07-27T11:50:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:04 smithi047 ceph-mon[36835]: pgmap v9994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:50:04.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:04 smithi029 ceph-mon[32610]: pgmap v9994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:50:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:04 smithi077 ceph-mon[37153]: pgmap v9994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:50:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:06 smithi047 ceph-mon[36835]: pgmap v9995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:06.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:06 smithi029 ceph-mon[32610]: pgmap v9995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:06 smithi077 ceph-mon[37153]: pgmap v9995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:07.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:07.253+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:50:07.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:07.253+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:50:07.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:07.253+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:50:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:08 smithi047 ceph-mon[36835]: pgmap v9996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18029.781182 secs 2024-07-27T11:50:08.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:08 smithi029 ceph-mon[32610]: pgmap v9996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18029.781182 secs 2024-07-27T11:50:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:08 smithi077 ceph-mon[37153]: pgmap v9996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18029.781182 secs 2024-07-27T11:50:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:10 smithi047 ceph-mon[36835]: pgmap v9997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:10.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:10 smithi029 ceph-mon[32610]: pgmap v9997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:10 smithi077 ceph-mon[37153]: pgmap v9997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:12 smithi047 ceph-mon[36835]: pgmap v9998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:12.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:12 smithi029 ceph-mon[32610]: pgmap v9998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:12 smithi077 ceph-mon[37153]: pgmap v9998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18034.781309 secs 2024-07-27T11:50:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18034.781309 secs 2024-07-27T11:50:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18034.781309 secs 2024-07-27T11:50:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:14 smithi047 ceph-mon[36835]: pgmap v9999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:50:14.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:14 smithi029 ceph-mon[32610]: pgmap v9999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:50:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:14 smithi077 ceph-mon[37153]: pgmap v9999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:50:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:16 smithi047 ceph-mon[36835]: pgmap v10000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:16 smithi029 ceph-mon[32610]: pgmap v10000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:16 smithi077 ceph-mon[37153]: pgmap v10000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:18 smithi047 ceph-mon[36835]: pgmap v10001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18039.781414 secs 2024-07-27T11:50:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:18 smithi029 ceph-mon[32610]: pgmap v10001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18039.781414 secs 2024-07-27T11:50:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:18 smithi077 ceph-mon[37153]: pgmap v10001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18039.781414 secs 2024-07-27T11:50:20.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:20 smithi029 ceph-mon[32610]: pgmap v10002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:20 smithi047 ceph-mon[36835]: pgmap v10002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:20 smithi077 ceph-mon[37153]: pgmap v10002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:22.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:22.269+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:50:22.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:22.269+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:50:22.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:22.269+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:50:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:22 smithi047 ceph-mon[36835]: pgmap v10003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:22.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:22 smithi029 ceph-mon[32610]: pgmap v10003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:22 smithi077 ceph-mon[37153]: pgmap v10003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18044.781550 secs 2024-07-27T11:50:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18044.781550 secs 2024-07-27T11:50:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18044.781550 secs 2024-07-27T11:50:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:24 smithi029 ceph-mon[32610]: pgmap v10004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:50:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:24 smithi077 ceph-mon[37153]: pgmap v10004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:50:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:24 smithi047 ceph-mon[36835]: pgmap v10004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:50:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:26 smithi029 ceph-mon[32610]: pgmap v10005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:26 smithi077 ceph-mon[37153]: pgmap v10005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:26 smithi047 ceph-mon[36835]: pgmap v10005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:28 smithi029 ceph-mon[32610]: pgmap v10006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18049.781681 secs 2024-07-27T11:50:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:28 smithi077 ceph-mon[37153]: pgmap v10006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18049.781681 secs 2024-07-27T11:50:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:28 smithi047 ceph-mon[36835]: pgmap v10006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18049.781681 secs 2024-07-27T11:50:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:30 smithi047 ceph-mon[36835]: pgmap v10007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:30 smithi029 ceph-mon[32610]: pgmap v10007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:30 smithi077 ceph-mon[37153]: pgmap v10007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:32.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:32 smithi029 ceph-mon[32610]: pgmap v10008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:32 smithi077 ceph-mon[37153]: pgmap v10008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:32 smithi047 ceph-mon[36835]: pgmap v10008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18054.781832 secs 2024-07-27T11:50:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18054.781832 secs 2024-07-27T11:50:33.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18054.781832 secs 2024-07-27T11:50:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:34 smithi077 ceph-mon[37153]: pgmap v10009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:50:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:34 smithi047 ceph-mon[36835]: pgmap v10009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:50:34.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:34 smithi029 ceph-mon[32610]: pgmap v10009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:50:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:36 smithi077 ceph-mon[37153]: pgmap v10010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:36 smithi047 ceph-mon[36835]: pgmap v10010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:36.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:36 smithi029 ceph-mon[32610]: pgmap v10010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:50:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:37.278+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:50:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:37.278+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:50:37.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:37.278+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:50:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:38 smithi077 ceph-mon[37153]: pgmap v10011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18059.781962 secs 2024-07-27T11:50:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:38 smithi047 ceph-mon[36835]: pgmap v10011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18059.781962 secs 2024-07-27T11:50:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:38 smithi029 ceph-mon[32610]: pgmap v10011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18059.781962 secs 2024-07-27T11:50:39.573 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:50:39.573 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:39.574 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:39 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:50:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:39.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:39 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:39.627 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:50:39.627 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:39.628 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:39 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:40 smithi077 ceph-mon[37153]: pgmap v10012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:40 smithi047 ceph-mon[36835]: pgmap v10012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:40.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:40 smithi029 ceph-mon[32610]: pgmap v10012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:41 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:41 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:41 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:42 smithi077 ceph-mon[37153]: pgmap v10013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:50:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:50:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:42 smithi077 ceph-mon[37153]: pgmap v10014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:50:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:42 smithi077 ceph-mon[37153]: pgmap v10015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:50:42.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:42 smithi047 ceph-mon[36835]: pgmap v10013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:50:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:50:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:42 smithi047 ceph-mon[36835]: pgmap v10014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:50:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:42 smithi047 ceph-mon[36835]: pgmap v10015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:50:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:42 smithi029 ceph-mon[32610]: pgmap v10013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:50:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:50:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:42 smithi029 ceph-mon[32610]: pgmap v10014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:50:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:42 smithi029 ceph-mon[32610]: pgmap v10015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T11:50:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:50:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18064.782095 secs 2024-07-27T11:50:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18064.782095 secs 2024-07-27T11:50:43.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18064.782095 secs 2024-07-27T11:50:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:44 smithi077 ceph-mon[37153]: pgmap v10016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:50:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:44 smithi047 ceph-mon[36835]: pgmap v10016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:50:44.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:44 smithi029 ceph-mon[32610]: pgmap v10016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T11:50:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:45 smithi047 ceph-mon[36835]: pgmap v10017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:50:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:45 smithi029 ceph-mon[32610]: pgmap v10017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:50:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:45 smithi077 ceph-mon[37153]: pgmap v10017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:50:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:47 smithi047 ceph-mon[36835]: pgmap v10018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:50:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:47 smithi029 ceph-mon[32610]: pgmap v10018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:50:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:47 smithi077 ceph-mon[37153]: pgmap v10018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:50:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18069.782222 secs 2024-07-27T11:50:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18069.782222 secs 2024-07-27T11:50:48.861 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18069.782222 secs 2024-07-27T11:50:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:49 smithi047 ceph-mon[36835]: pgmap v10019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:49 smithi029 ceph-mon[32610]: pgmap v10019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:49 smithi077 ceph-mon[37153]: pgmap v10019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:51.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:51 smithi029 ceph-mon[32610]: pgmap v10020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:50:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:51 smithi047 ceph-mon[36835]: pgmap v10020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:50:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:51 smithi077 ceph-mon[37153]: pgmap v10020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:50:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:52.286+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:50:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:52.286+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:50:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:50:52.286+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:50:52.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18074.782361 secs 2024-07-27T11:50:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18074.782361 secs 2024-07-27T11:50:52.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18074.782361 secs 2024-07-27T11:50:53.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:53 smithi047 ceph-mon[36835]: pgmap v10021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:53 smithi077 ceph-mon[37153]: pgmap v10021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:53 smithi029 ceph-mon[32610]: pgmap v10021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:50:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:55 smithi047 ceph-mon[36835]: pgmap v10022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:55 smithi029 ceph-mon[32610]: pgmap v10022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:55 smithi077 ceph-mon[37153]: pgmap v10022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:50:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:57 smithi047 ceph-mon[36835]: pgmap v10023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:57.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:57 smithi029 ceph-mon[32610]: pgmap v10023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:57 smithi077 ceph-mon[37153]: pgmap v10023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:50:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18079.782466 secs 2024-07-27T11:50:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18079.782466 secs 2024-07-27T11:50:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18079.782466 secs 2024-07-27T11:50:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:50:59 smithi047 ceph-mon[36835]: pgmap v10024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:50:59.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:50:59 smithi029 ceph-mon[32610]: pgmap v10024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:50:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:50:59 smithi077 ceph-mon[37153]: pgmap v10024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:01 smithi047 ceph-mon[36835]: pgmap v10025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:01 smithi029 ceph-mon[32610]: pgmap v10025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:01 smithi077 ceph-mon[37153]: pgmap v10025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18084.782600 secs 2024-07-27T11:51:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18084.782600 secs 2024-07-27T11:51:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18084.782600 secs 2024-07-27T11:51:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:03 smithi077 ceph-mon[37153]: pgmap v10026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:51:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:03 smithi047 ceph-mon[36835]: pgmap v10026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:51:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:03 smithi029 ceph-mon[32610]: pgmap v10026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:51:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:05 smithi047 ceph-mon[36835]: pgmap v10027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:05.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:05 smithi029 ceph-mon[32610]: pgmap v10027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:05 smithi077 ceph-mon[37153]: pgmap v10027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:07.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:51:07.300+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:51:07.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:51:07.300+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:51:07.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:51:07.300+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:51:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:07 smithi047 ceph-mon[36835]: pgmap v10028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:07.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:07 smithi029 ceph-mon[32610]: pgmap v10028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:07 smithi077 ceph-mon[37153]: pgmap v10028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18089.782733 secs 2024-07-27T11:51:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18089.782733 secs 2024-07-27T11:51:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18089.782733 secs 2024-07-27T11:51:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:09 smithi047 ceph-mon[36835]: pgmap v10029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:09 smithi029 ceph-mon[32610]: pgmap v10029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:09 smithi077 ceph-mon[37153]: pgmap v10029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:11 smithi047 ceph-mon[36835]: pgmap v10030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:11.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:11 smithi029 ceph-mon[32610]: pgmap v10030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:11 smithi077 ceph-mon[37153]: pgmap v10030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18094.782863 secs 2024-07-27T11:51:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18094.782863 secs 2024-07-27T11:51:12.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18094.782863 secs 2024-07-27T11:51:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:13 smithi077 ceph-mon[37153]: pgmap v10031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:51:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:13 smithi047 ceph-mon[36835]: pgmap v10031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:51:13.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:13 smithi029 ceph-mon[32610]: pgmap v10031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:51:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:15 smithi047 ceph-mon[36835]: pgmap v10032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:15 smithi029 ceph-mon[32610]: pgmap v10032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:15 smithi077 ceph-mon[37153]: pgmap v10032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:17 smithi047 ceph-mon[36835]: pgmap v10033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:17.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:17 smithi029 ceph-mon[32610]: pgmap v10033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:17 smithi077 ceph-mon[37153]: pgmap v10033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18099.783005 secs 2024-07-27T11:51:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18099.783005 secs 2024-07-27T11:51:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18099.783005 secs 2024-07-27T11:51:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:19 smithi047 ceph-mon[36835]: pgmap v10034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:19.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:19 smithi029 ceph-mon[32610]: pgmap v10034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:19 smithi077 ceph-mon[37153]: pgmap v10034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:21 smithi047 ceph-mon[36835]: pgmap v10035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:21.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:21 smithi029 ceph-mon[32610]: pgmap v10035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:21 smithi077 ceph-mon[37153]: pgmap v10035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:22.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:51:22.316+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:51:22.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:51:22.316+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:51:22.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:51:22.316+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:51:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18104.783122 secs 2024-07-27T11:51:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18104.783122 secs 2024-07-27T11:51:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18104.783122 secs 2024-07-27T11:51:23.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:23 smithi047 ceph-mon[36835]: pgmap v10036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:51:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:23 smithi077 ceph-mon[37153]: pgmap v10036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:51:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:23 smithi029 ceph-mon[32610]: pgmap v10036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:51:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:25 smithi047 ceph-mon[36835]: pgmap v10037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:25 smithi029 ceph-mon[32610]: pgmap v10037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:25 smithi077 ceph-mon[37153]: pgmap v10037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:27 smithi047 ceph-mon[36835]: pgmap v10038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:27 smithi029 ceph-mon[32610]: pgmap v10038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:27 smithi077 ceph-mon[37153]: pgmap v10038: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18109.783257 secs 2024-07-27T11:51:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18109.783257 secs 2024-07-27T11:51:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18109.783257 secs 2024-07-27T11:51:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:29 smithi047 ceph-mon[36835]: pgmap v10039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:29.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:29 smithi029 ceph-mon[32610]: pgmap v10039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:29 smithi077 ceph-mon[37153]: pgmap v10039: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:31 smithi047 ceph-mon[36835]: pgmap v10040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:31.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:31 smithi029 ceph-mon[32610]: pgmap v10040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:31 smithi077 ceph-mon[37153]: pgmap v10040: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18114.783401 secs 2024-07-27T11:51:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18114.783401 secs 2024-07-27T11:51:32.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18114.783401 secs 2024-07-27T11:51:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:33 smithi077 ceph-mon[37153]: pgmap v10041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:51:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:33 smithi047 ceph-mon[36835]: pgmap v10041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:51:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:33 smithi029 ceph-mon[32610]: pgmap v10041: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:51:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:35 smithi047 ceph-mon[36835]: pgmap v10042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:35 smithi029 ceph-mon[32610]: pgmap v10042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:35 smithi077 ceph-mon[37153]: pgmap v10042: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:37.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:51:37.324+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:51:37.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:51:37.324+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:51:37.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:51:37.324+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:51:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:37 smithi047 ceph-mon[36835]: pgmap v10043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:37 smithi029 ceph-mon[32610]: pgmap v10043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:37 smithi077 ceph-mon[37153]: pgmap v10043: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18119.783486 secs 2024-07-27T11:51:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18119.783486 secs 2024-07-27T11:51:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18119.783486 secs 2024-07-27T11:51:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:39 smithi047 ceph-mon[36835]: pgmap v10044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:39.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:39 smithi029 ceph-mon[32610]: pgmap v10044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:39 smithi077 ceph-mon[37153]: pgmap v10044: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:41 smithi047 ceph-mon[36835]: pgmap v10045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:41.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:41 smithi029 ceph-mon[32610]: pgmap v10045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:41 smithi077 ceph-mon[37153]: pgmap v10045: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:51:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:51:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:51:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:51:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:42 smithi047 ceph-mon[36835]: pgmap v10046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:51:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:42 smithi047 ceph-mon[36835]: pgmap v10047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T11:51:42.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:51:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:51:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:51:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:42 smithi029 ceph-mon[32610]: pgmap v10046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:51:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:42 smithi029 ceph-mon[32610]: pgmap v10047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T11:51:42.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:51:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:51:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:51:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:42 smithi077 ceph-mon[37153]: pgmap v10046: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:51:42.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:42 smithi077 ceph-mon[37153]: pgmap v10047: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T11:51:42.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:51:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18124.783584 secs 2024-07-27T11:51:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18124.783584 secs 2024-07-27T11:51:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18124.783584 secs 2024-07-27T11:51:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:44 smithi047 ceph-mon[36835]: pgmap v10048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:51:44.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:44 smithi029 ceph-mon[32610]: pgmap v10048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:51:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:44 smithi077 ceph-mon[37153]: pgmap v10048: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:51:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:47 smithi077 ceph-mon[37153]: pgmap v10049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:51:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:47 smithi047 ceph-mon[36835]: pgmap v10049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:51:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:47 smithi029 ceph-mon[32610]: pgmap v10049: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:51:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18129.783721 secs 2024-07-27T11:51:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18129.783721 secs 2024-07-27T11:51:48.699 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18129.783721 secs 2024-07-27T11:51:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:49 smithi077 ceph-mon[37153]: pgmap v10050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T11:51:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:49 smithi047 ceph-mon[36835]: pgmap v10050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T11:51:49.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:49 smithi029 ceph-mon[32610]: pgmap v10050: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T11:51:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:51 smithi077 ceph-mon[37153]: pgmap v10051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:51:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:51 smithi047 ceph-mon[36835]: pgmap v10051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:51:51.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:51 smithi029 ceph-mon[32610]: pgmap v10051: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:51:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:51:52.329+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:51:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:51:52.329+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:51:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:51:52.329+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:51:53.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18134.783819 secs 2024-07-27T11:51:53.499 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:53 smithi047 ceph-mon[36835]: pgmap v10052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:51:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18134.783819 secs 2024-07-27T11:51:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:53 smithi077 ceph-mon[37153]: pgmap v10052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:51:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18134.783819 secs 2024-07-27T11:51:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:53 smithi029 ceph-mon[32610]: pgmap v10052: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:51:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:55 smithi077 ceph-mon[37153]: pgmap v10053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:55 smithi047 ceph-mon[36835]: pgmap v10053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:55.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:55 smithi029 ceph-mon[32610]: pgmap v10053: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:51:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:57 smithi077 ceph-mon[37153]: pgmap v10054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:57 smithi047 ceph-mon[36835]: pgmap v10054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:57 smithi029 ceph-mon[32610]: pgmap v10054: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:51:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18139.783935 secs 2024-07-27T11:51:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18139.783935 secs 2024-07-27T11:51:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18139.783935 secs 2024-07-27T11:51:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:51:59 smithi077 ceph-mon[37153]: pgmap v10055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:51:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:51:59 smithi047 ceph-mon[36835]: pgmap v10055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:51:59.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:51:59 smithi029 ceph-mon[32610]: pgmap v10055: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:01 smithi077 ceph-mon[37153]: pgmap v10056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:01 smithi047 ceph-mon[36835]: pgmap v10056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:01.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:01 smithi029 ceph-mon[32610]: pgmap v10056: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18144.784078 secs 2024-07-27T11:52:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:03 smithi077 ceph-mon[37153]: pgmap v10057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18144.784078 secs 2024-07-27T11:52:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:03 smithi047 ceph-mon[36835]: pgmap v10057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18144.784078 secs 2024-07-27T11:52:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:03 smithi029 ceph-mon[32610]: pgmap v10057: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:05 smithi077 ceph-mon[37153]: pgmap v10058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:52:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:05 smithi047 ceph-mon[36835]: pgmap v10058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:52:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:05 smithi029 ceph-mon[32610]: pgmap v10058: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:52:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:52:07.343+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:52:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:52:07.343+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:52:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:52:07.343+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:52:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:07 smithi077 ceph-mon[37153]: pgmap v10059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:07 smithi047 ceph-mon[36835]: pgmap v10059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:07 smithi029 ceph-mon[32610]: pgmap v10059: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18149.784138 secs 2024-07-27T11:52:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18149.784138 secs 2024-07-27T11:52:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18149.784138 secs 2024-07-27T11:52:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:09 smithi077 ceph-mon[37153]: pgmap v10060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:09 smithi047 ceph-mon[36835]: pgmap v10060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:09 smithi029 ceph-mon[32610]: pgmap v10060: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:11 smithi077 ceph-mon[37153]: pgmap v10061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:11 smithi047 ceph-mon[36835]: pgmap v10061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:11.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:11 smithi029 ceph-mon[32610]: pgmap v10061: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18154.784205 secs 2024-07-27T11:52:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:13 smithi077 ceph-mon[37153]: pgmap v10062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18154.784205 secs 2024-07-27T11:52:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:13 smithi047 ceph-mon[36835]: pgmap v10062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:13.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18154.784205 secs 2024-07-27T11:52:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:13 smithi029 ceph-mon[32610]: pgmap v10062: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:15 smithi077 ceph-mon[37153]: pgmap v10063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:52:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:15 smithi047 ceph-mon[36835]: pgmap v10063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:52:15.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:15 smithi029 ceph-mon[32610]: pgmap v10063: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:52:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:17 smithi077 ceph-mon[37153]: pgmap v10064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:17 smithi047 ceph-mon[36835]: pgmap v10064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:17 smithi029 ceph-mon[32610]: pgmap v10064: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18159.784333 secs 2024-07-27T11:52:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18159.784333 secs 2024-07-27T11:52:18.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18159.784333 secs 2024-07-27T11:52:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:19 smithi077 ceph-mon[37153]: pgmap v10065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:19 smithi047 ceph-mon[36835]: pgmap v10065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:19.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:19 smithi029 ceph-mon[32610]: pgmap v10065: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:21 smithi077 ceph-mon[37153]: pgmap v10066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:52:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:21 smithi047 ceph-mon[36835]: pgmap v10066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:52:21.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:21 smithi029 ceph-mon[32610]: pgmap v10066: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:52:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:52:22.349+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:52:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:52:22.349+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:52:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:52:22.349+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:52:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18164.784482 secs 2024-07-27T11:52:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:23 smithi077 ceph-mon[37153]: pgmap v10067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18164.784482 secs 2024-07-27T11:52:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:23 smithi047 ceph-mon[36835]: pgmap v10067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18164.784482 secs 2024-07-27T11:52:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:23 smithi029 ceph-mon[32610]: pgmap v10067: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:25 smithi077 ceph-mon[37153]: pgmap v10068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:52:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:25 smithi047 ceph-mon[36835]: pgmap v10068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:52:25.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:25 smithi029 ceph-mon[32610]: pgmap v10068: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:52:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:27 smithi077 ceph-mon[37153]: pgmap v10069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:27 smithi047 ceph-mon[36835]: pgmap v10069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:27 smithi029 ceph-mon[32610]: pgmap v10069: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18169.784629 secs 2024-07-27T11:52:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18169.784629 secs 2024-07-27T11:52:28.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18169.784629 secs 2024-07-27T11:52:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:29 smithi077 ceph-mon[37153]: pgmap v10070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:29 smithi047 ceph-mon[36835]: pgmap v10070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:29.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:29 smithi029 ceph-mon[32610]: pgmap v10070: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:31 smithi077 ceph-mon[37153]: pgmap v10071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:31 smithi047 ceph-mon[36835]: pgmap v10071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:31.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:31 smithi029 ceph-mon[32610]: pgmap v10071: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18174.784757 secs 2024-07-27T11:52:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:33 smithi077 ceph-mon[37153]: pgmap v10072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18174.784757 secs 2024-07-27T11:52:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:33 smithi047 ceph-mon[36835]: pgmap v10072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18174.784757 secs 2024-07-27T11:52:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:33 smithi029 ceph-mon[32610]: pgmap v10072: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:35 smithi077 ceph-mon[37153]: pgmap v10073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:52:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:35 smithi047 ceph-mon[36835]: pgmap v10073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:52:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:35 smithi029 ceph-mon[32610]: pgmap v10073: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:52:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:52:37.351+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:52:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:52:37.351+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:52:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:52:37.351+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:52:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:37 smithi077 ceph-mon[37153]: pgmap v10074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:37 smithi047 ceph-mon[36835]: pgmap v10074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:37.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:37 smithi029 ceph-mon[32610]: pgmap v10074: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18179.784887 secs 2024-07-27T11:52:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18179.784887 secs 2024-07-27T11:52:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18179.784887 secs 2024-07-27T11:52:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:39 smithi077 ceph-mon[37153]: pgmap v10075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:39 smithi047 ceph-mon[36835]: pgmap v10075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:39.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:39 smithi029 ceph-mon[32610]: pgmap v10075: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:52:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:41 smithi077 ceph-mon[37153]: pgmap v10076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:41 smithi047 ceph-mon[36835]: pgmap v10076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:41.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:41 smithi029 ceph-mon[32610]: pgmap v10076: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:52:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:52:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:52:42.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:52:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18184.784992 secs 2024-07-27T11:52:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:43 smithi077 ceph-mon[37153]: pgmap v10077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:52:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:52:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:52:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18184.784992 secs 2024-07-27T11:52:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:43 smithi047 ceph-mon[36835]: pgmap v10077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:52:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:52:43.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:52:43.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18184.784992 secs 2024-07-27T11:52:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:43 smithi029 ceph-mon[32610]: pgmap v10077: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:52:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:52:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:52:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:44 smithi077 ceph-mon[37153]: pgmap v10078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T11:52:44.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:44 smithi077 ceph-mon[37153]: pgmap v10079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:52:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:44 smithi047 ceph-mon[36835]: pgmap v10078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T11:52:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:44 smithi047 ceph-mon[36835]: pgmap v10079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:52:44.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:44 smithi029 ceph-mon[32610]: pgmap v10078: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T11:52:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:44 smithi029 ceph-mon[32610]: pgmap v10079: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:52:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:46 smithi077 ceph-mon[37153]: pgmap v10080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:52:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:46 smithi047 ceph-mon[36835]: pgmap v10080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:52:46.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:46 smithi029 ceph-mon[32610]: pgmap v10080: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:52:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:48 smithi047 ceph-mon[36835]: pgmap v10081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:52:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18189.785121 secs 2024-07-27T11:52:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:48 smithi029 ceph-mon[32610]: pgmap v10081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:52:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18189.785121 secs 2024-07-27T11:52:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:48 smithi077 ceph-mon[37153]: pgmap v10081: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:52:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18189.785121 secs 2024-07-27T11:52:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:50 smithi047 ceph-mon[36835]: pgmap v10082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:52:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:50 smithi029 ceph-mon[32610]: pgmap v10082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:52:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:50 smithi077 ceph-mon[37153]: pgmap v10082: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:52:52.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:52:52.366+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:52:52.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:52:52.366+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:52:52.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:52:52.366+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:52:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:52 smithi047 ceph-mon[36835]: pgmap v10083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:52:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:52 smithi029 ceph-mon[32610]: pgmap v10083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:52:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:52 smithi077 ceph-mon[37153]: pgmap v10083: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:52:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18194.785248 secs 2024-07-27T11:52:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18194.785248 secs 2024-07-27T11:52:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18194.785248 secs 2024-07-27T11:52:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:54 smithi047 ceph-mon[36835]: pgmap v10084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:52:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:54 smithi029 ceph-mon[32610]: pgmap v10084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:52:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:54 smithi077 ceph-mon[37153]: pgmap v10084: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:52:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:56 smithi047 ceph-mon[36835]: pgmap v10085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:52:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:56 smithi029 ceph-mon[32610]: pgmap v10085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:52:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:56 smithi077 ceph-mon[37153]: pgmap v10085: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:52:58.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:58 smithi077 ceph-mon[37153]: pgmap v10086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:58.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:52:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18199.785414 secs 2024-07-27T11:52:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:58 smithi047 ceph-mon[36835]: pgmap v10086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:52:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18199.785414 secs 2024-07-27T11:52:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:58 smithi029 ceph-mon[32610]: pgmap v10086: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:52:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:52:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18199.785414 secs 2024-07-27T11:53:00.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:00.390+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:53:00.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:00.390+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:53:00.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:00.390+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:53:00.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:00.391+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:53:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:00 smithi047 ceph-mon[36835]: pgmap v10087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:00.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:00 smithi029 ceph-mon[32610]: pgmap v10087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:00 smithi077 ceph-mon[37153]: pgmap v10087: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:02 smithi047 ceph-mon[36835]: pgmap v10088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:02 smithi029 ceph-mon[32610]: pgmap v10088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:02 smithi077 ceph-mon[37153]: pgmap v10088: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18204.785546 secs 2024-07-27T11:53:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18204.785546 secs 2024-07-27T11:53:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18204.785546 secs 2024-07-27T11:53:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:04 smithi047 ceph-mon[36835]: pgmap v10089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:53:04.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:04 smithi029 ceph-mon[32610]: pgmap v10089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:53:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:04 smithi077 ceph-mon[37153]: pgmap v10089: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:53:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:05 smithi047 ceph-mon[36835]: pgmap v10090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:53:05.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:05 smithi029 ceph-mon[32610]: pgmap v10090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:53:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:05 smithi077 ceph-mon[37153]: pgmap v10090: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:53:07.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:07.380+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:53:07.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:07.380+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:53:07.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:07.380+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:53:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:07 smithi077 ceph-mon[37153]: pgmap v10091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18209.785637 secs 2024-07-27T11:53:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:07 smithi047 ceph-mon[36835]: pgmap v10091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18209.785637 secs 2024-07-27T11:53:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:07 smithi029 ceph-mon[32610]: pgmap v10091: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18209.785637 secs 2024-07-27T11:53:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:09 smithi047 ceph-mon[36835]: pgmap v10092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:10.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:09 smithi029 ceph-mon[32610]: pgmap v10092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:09 smithi077 ceph-mon[37153]: pgmap v10092: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:11 smithi047 ceph-mon[36835]: pgmap v10093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:12.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:11 smithi029 ceph-mon[32610]: pgmap v10093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:11 smithi077 ceph-mon[37153]: pgmap v10093: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18214.785729 secs 2024-07-27T11:53:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18214.785729 secs 2024-07-27T11:53:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18214.785729 secs 2024-07-27T11:53:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:13 smithi047 ceph-mon[36835]: pgmap v10094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:53:14.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:13 smithi029 ceph-mon[32610]: pgmap v10094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:53:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:13 smithi077 ceph-mon[37153]: pgmap v10094: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:53:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:15 smithi047 ceph-mon[36835]: pgmap v10095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:53:16.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:15 smithi029 ceph-mon[32610]: pgmap v10095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:53:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:15 smithi077 ceph-mon[37153]: pgmap v10095: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:53:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:17 smithi047 ceph-mon[36835]: pgmap v10096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18219.785861 secs 2024-07-27T11:53:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:17 smithi029 ceph-mon[32610]: pgmap v10096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18219.785861 secs 2024-07-27T11:53:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:17 smithi077 ceph-mon[37153]: pgmap v10096: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18219.785861 secs 2024-07-27T11:53:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:19 smithi047 ceph-mon[36835]: pgmap v10097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:20.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:19 smithi029 ceph-mon[32610]: pgmap v10097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:19 smithi077 ceph-mon[37153]: pgmap v10097: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:22.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:21 smithi029 ceph-mon[32610]: pgmap v10098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:21 smithi077 ceph-mon[37153]: pgmap v10098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:22.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:22.392+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:53:22.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:22.392+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:53:22.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:22.392+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:53:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:21 smithi047 ceph-mon[36835]: pgmap v10098: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18224.785995 secs 2024-07-27T11:53:23.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18224.785995 secs 2024-07-27T11:53:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18224.785995 secs 2024-07-27T11:53:24.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:23 smithi029 ceph-mon[32610]: pgmap v10099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:53:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:23 smithi077 ceph-mon[37153]: pgmap v10099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:53:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:23 smithi047 ceph-mon[36835]: pgmap v10099: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:53:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:25 smithi029 ceph-mon[32610]: pgmap v10100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:53:26.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:25 smithi077 ceph-mon[37153]: pgmap v10100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:53:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:25 smithi047 ceph-mon[36835]: pgmap v10100: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:53:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:27 smithi029 ceph-mon[32610]: pgmap v10101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18229.786085 secs 2024-07-27T11:53:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:27 smithi077 ceph-mon[37153]: pgmap v10101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18229.786085 secs 2024-07-27T11:53:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:27 smithi047 ceph-mon[36835]: pgmap v10101: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18229.786085 secs 2024-07-27T11:53:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:29 smithi077 ceph-mon[37153]: pgmap v10102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:29 smithi047 ceph-mon[36835]: pgmap v10102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:30.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:29 smithi029 ceph-mon[32610]: pgmap v10102: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:31 smithi077 ceph-mon[37153]: pgmap v10103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:31 smithi047 ceph-mon[36835]: pgmap v10103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:32.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:31 smithi029 ceph-mon[32610]: pgmap v10103: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18234.786184 secs 2024-07-27T11:53:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18234.786184 secs 2024-07-27T11:53:33.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18234.786184 secs 2024-07-27T11:53:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:33 smithi077 ceph-mon[37153]: pgmap v10104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:53:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:33 smithi047 ceph-mon[36835]: pgmap v10104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:53:34.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:33 smithi029 ceph-mon[32610]: pgmap v10104: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:53:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:35 smithi077 ceph-mon[37153]: pgmap v10105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:53:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:35 smithi047 ceph-mon[36835]: pgmap v10105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:53:36.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:35 smithi029 ceph-mon[32610]: pgmap v10105: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:53:37.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:37.394+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:53:37.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:37.394+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:53:37.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:37.394+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:53:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:37 smithi077 ceph-mon[37153]: pgmap v10106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18239.786320 secs 2024-07-27T11:53:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:37 smithi047 ceph-mon[36835]: pgmap v10106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18239.786320 secs 2024-07-27T11:53:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:37 smithi029 ceph-mon[32610]: pgmap v10106: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18239.786320 secs 2024-07-27T11:53:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:39 smithi077 ceph-mon[37153]: pgmap v10107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:39 smithi047 ceph-mon[36835]: pgmap v10107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:39 smithi029 ceph-mon[32610]: pgmap v10107: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:53:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:41 smithi077 ceph-mon[37153]: pgmap v10108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:41 smithi047 ceph-mon[36835]: pgmap v10108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:42.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:41 smithi029 ceph-mon[32610]: pgmap v10108: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18244.786472 secs 2024-07-27T11:53:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:42 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:53:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18244.786472 secs 2024-07-27T11:53:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:42 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:53:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18244.786472 secs 2024-07-27T11:53:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:42 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:53:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:43 smithi077 ceph-mon[37153]: pgmap v10109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:53:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:53:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:53:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:43 smithi077 ceph-mon[37153]: pgmap v10110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:53:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:43 smithi077 ceph-mon[37153]: pgmap v10111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:53:44.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:43 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:53:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:43 smithi047 ceph-mon[36835]: pgmap v10109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:53:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:53:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:53:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:43 smithi047 ceph-mon[36835]: pgmap v10110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:53:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:43 smithi047 ceph-mon[36835]: pgmap v10111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:53:44.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:43 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:53:44.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:43 smithi029 ceph-mon[32610]: pgmap v10109: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:53:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:53:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:53:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:43 smithi029 ceph-mon[32610]: pgmap v10110: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:53:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:43 smithi029 ceph-mon[32610]: pgmap v10111: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:53:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:43 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:53:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:46 smithi077 ceph-mon[37153]: pgmap v10112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:53:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:46 smithi047 ceph-mon[36835]: pgmap v10112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:53:46.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:46 smithi029 ceph-mon[32610]: pgmap v10112: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:53:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18249.786603 secs 2024-07-27T11:53:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:48 smithi077 ceph-mon[37153]: pgmap v10113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:53:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18249.786603 secs 2024-07-27T11:53:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:48 smithi047 ceph-mon[36835]: pgmap v10113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:53:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18249.786603 secs 2024-07-27T11:53:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:48 smithi029 ceph-mon[32610]: pgmap v10113: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:53:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:50 smithi077 ceph-mon[37153]: pgmap v10114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:53:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:50 smithi047 ceph-mon[36835]: pgmap v10114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:53:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:50 smithi029 ceph-mon[32610]: pgmap v10114: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:53:52.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:52.397+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:53:52.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:52.397+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:53:52.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:53:52.397+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:53:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:52 smithi047 ceph-mon[36835]: pgmap v10115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:53:52.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:52 smithi029 ceph-mon[32610]: pgmap v10115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:53:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:52 smithi077 ceph-mon[37153]: pgmap v10115: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:53:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18254.786735 secs 2024-07-27T11:53:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18254.786735 secs 2024-07-27T11:53:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18254.786735 secs 2024-07-27T11:53:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:54 smithi047 ceph-mon[36835]: pgmap v10116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:53:54.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:54 smithi029 ceph-mon[32610]: pgmap v10116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:53:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:54 smithi077 ceph-mon[37153]: pgmap v10116: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:53:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:56 smithi047 ceph-mon[36835]: pgmap v10117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:56.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:56 smithi029 ceph-mon[32610]: pgmap v10117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:56 smithi077 ceph-mon[37153]: pgmap v10117: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:53:58.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18259.786878 secs 2024-07-27T11:53:58.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:53:58 smithi077 ceph-mon[37153]: pgmap v10118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18259.786878 secs 2024-07-27T11:53:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:53:58 smithi047 ceph-mon[36835]: pgmap v10118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:53:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18259.786878 secs 2024-07-27T11:53:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:53:58 smithi029 ceph-mon[32610]: pgmap v10118: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:00 smithi047 ceph-mon[36835]: pgmap v10119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:00 smithi029 ceph-mon[32610]: pgmap v10119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:00 smithi077 ceph-mon[37153]: pgmap v10119: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:02 smithi047 ceph-mon[36835]: pgmap v10120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:02.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:02 smithi029 ceph-mon[32610]: pgmap v10120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:02 smithi077 ceph-mon[37153]: pgmap v10120: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18264.786994 secs 2024-07-27T11:54:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18264.786994 secs 2024-07-27T11:54:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18264.786994 secs 2024-07-27T11:54:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:04 smithi047 ceph-mon[36835]: pgmap v10121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:04.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:04 smithi029 ceph-mon[32610]: pgmap v10121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:04 smithi077 ceph-mon[37153]: pgmap v10121: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:06 smithi047 ceph-mon[36835]: pgmap v10122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:06 smithi029 ceph-mon[32610]: pgmap v10122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:06 smithi077 ceph-mon[37153]: pgmap v10122: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:07.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:54:07.408+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:54:07.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:54:07.408+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:54:07.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:54:07.408+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:54:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18269.787123 secs 2024-07-27T11:54:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:08 smithi047 ceph-mon[36835]: pgmap v10123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18269.787123 secs 2024-07-27T11:54:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:08 smithi029 ceph-mon[32610]: pgmap v10123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18269.787123 secs 2024-07-27T11:54:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:08 smithi077 ceph-mon[37153]: pgmap v10123: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:09 smithi047 ceph-mon[36835]: pgmap v10124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:10.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:09 smithi029 ceph-mon[32610]: pgmap v10124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:09 smithi077 ceph-mon[37153]: pgmap v10124: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:12 smithi047 ceph-mon[36835]: pgmap v10125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:12 smithi029 ceph-mon[32610]: pgmap v10125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:12 smithi077 ceph-mon[37153]: pgmap v10125: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18274.787258 secs 2024-07-27T11:54:13.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18274.787258 secs 2024-07-27T11:54:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18274.787258 secs 2024-07-27T11:54:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:14 smithi047 ceph-mon[36835]: pgmap v10126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:14.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:14 smithi029 ceph-mon[32610]: pgmap v10126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:14 smithi077 ceph-mon[37153]: pgmap v10126: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:16 smithi047 ceph-mon[36835]: pgmap v10127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:16.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:16 smithi029 ceph-mon[32610]: pgmap v10127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:16 smithi077 ceph-mon[37153]: pgmap v10127: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18279.787388 secs 2024-07-27T11:54:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:18 smithi047 ceph-mon[36835]: pgmap v10128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18279.787388 secs 2024-07-27T11:54:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:18 smithi029 ceph-mon[32610]: pgmap v10128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18279.787388 secs 2024-07-27T11:54:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:18 smithi077 ceph-mon[37153]: pgmap v10128: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:20 smithi047 ceph-mon[36835]: pgmap v10129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:20.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:20 smithi029 ceph-mon[32610]: pgmap v10129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:20 smithi077 ceph-mon[37153]: pgmap v10129: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:22.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:54:22.422+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:54:22.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:54:22.422+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:54:22.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:54:22.422+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:54:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:22 smithi047 ceph-mon[36835]: pgmap v10130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:22 smithi029 ceph-mon[32610]: pgmap v10130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:22 smithi077 ceph-mon[37153]: pgmap v10130: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18284.787521 secs 2024-07-27T11:54:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18284.787521 secs 2024-07-27T11:54:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18284.787521 secs 2024-07-27T11:54:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:24 smithi047 ceph-mon[36835]: pgmap v10131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:24.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:24 smithi029 ceph-mon[32610]: pgmap v10131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:24 smithi077 ceph-mon[37153]: pgmap v10131: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:26 smithi047 ceph-mon[36835]: pgmap v10132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:26.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:26 smithi029 ceph-mon[32610]: pgmap v10132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:26 smithi077 ceph-mon[37153]: pgmap v10132: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:28.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18289.787673 secs 2024-07-27T11:54:28.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:28 smithi077 ceph-mon[37153]: pgmap v10133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18289.787673 secs 2024-07-27T11:54:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:28 smithi047 ceph-mon[36835]: pgmap v10133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18289.787673 secs 2024-07-27T11:54:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:28 smithi029 ceph-mon[32610]: pgmap v10133: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:30 smithi047 ceph-mon[36835]: pgmap v10134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:30.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:30 smithi029 ceph-mon[32610]: pgmap v10134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:30 smithi077 ceph-mon[37153]: pgmap v10134: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:32 smithi047 ceph-mon[36835]: pgmap v10135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:32.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:32 smithi029 ceph-mon[32610]: pgmap v10135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:32 smithi077 ceph-mon[37153]: pgmap v10135: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18294.787823 secs 2024-07-27T11:54:33.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18294.787823 secs 2024-07-27T11:54:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18294.787823 secs 2024-07-27T11:54:34.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:34 smithi047 ceph-mon[36835]: pgmap v10136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:34.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:34 smithi029 ceph-mon[32610]: pgmap v10136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:34 smithi077 ceph-mon[37153]: pgmap v10136: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:35 smithi047 ceph-mon[36835]: pgmap v10137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:35 smithi029 ceph-mon[32610]: pgmap v10137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:35 smithi077 ceph-mon[37153]: pgmap v10137: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:37.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:54:37.438+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:54:37.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:54:37.438+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:54:37.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:54:37.438+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:54:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18299.787971 secs 2024-07-27T11:54:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:38 smithi047 ceph-mon[36835]: pgmap v10138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18299.787971 secs 2024-07-27T11:54:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:38 smithi029 ceph-mon[32610]: pgmap v10138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18299.787971 secs 2024-07-27T11:54:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:38 smithi077 ceph-mon[37153]: pgmap v10138: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:40 smithi047 ceph-mon[36835]: pgmap v10139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:40 smithi029 ceph-mon[32610]: pgmap v10139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:40 smithi077 ceph-mon[37153]: pgmap v10139: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:54:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:42 smithi047 ceph-mon[36835]: pgmap v10140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:42.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:42 smithi029 ceph-mon[32610]: pgmap v10140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:42.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:42 smithi077 ceph-mon[37153]: pgmap v10140: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:54:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18304.788111 secs 2024-07-27T11:54:43.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18304.788111 secs 2024-07-27T11:54:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18304.788111 secs 2024-07-27T11:54:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:44 smithi047 ceph-mon[36835]: pgmap v10141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:54:44.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:54:44.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:54:44.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:54:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:44 smithi029 ceph-mon[32610]: pgmap v10141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:54:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:54:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:54:44.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:54:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:44 smithi077 ceph-mon[37153]: pgmap v10141: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:54:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:54:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:54:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:54:44.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:54:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:45 smithi029 ceph-mon[32610]: pgmap v10142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:54:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:45 smithi029 ceph-mon[32610]: pgmap v10143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:54:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:45 smithi047 ceph-mon[36835]: pgmap v10142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:54:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:45 smithi047 ceph-mon[36835]: pgmap v10143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:54:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:45 smithi077 ceph-mon[37153]: pgmap v10142: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:54:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:45 smithi077 ceph-mon[37153]: pgmap v10143: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:54:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:47 smithi047 ceph-mon[36835]: pgmap v10144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:54:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:47 smithi029 ceph-mon[32610]: pgmap v10144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:54:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:47 smithi077 ceph-mon[37153]: pgmap v10144: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T11:54:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18309.788218 secs 2024-07-27T11:54:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18309.788218 secs 2024-07-27T11:54:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18309.788218 secs 2024-07-27T11:54:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:49 smithi047 ceph-mon[36835]: pgmap v10145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:54:49.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:49 smithi029 ceph-mon[32610]: pgmap v10145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:54:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:49 smithi077 ceph-mon[37153]: pgmap v10145: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:54:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:51 smithi047 ceph-mon[36835]: pgmap v10146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:54:51.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:51 smithi029 ceph-mon[32610]: pgmap v10146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:54:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:51 smithi077 ceph-mon[37153]: pgmap v10146: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:54:52.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:54:52.446+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:54:52.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:54:52.446+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:54:52.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:54:52.446+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:54:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:53 smithi047 ceph-mon[36835]: pgmap v10147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:54:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18314.788346 secs 2024-07-27T11:54:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:53 smithi029 ceph-mon[32610]: pgmap v10147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:54:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18314.788346 secs 2024-07-27T11:54:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:53 smithi077 ceph-mon[37153]: pgmap v10147: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:54:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18314.788346 secs 2024-07-27T11:54:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:55 smithi047 ceph-mon[36835]: pgmap v10148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:54:55.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:55 smithi029 ceph-mon[32610]: pgmap v10148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:54:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:55 smithi077 ceph-mon[37153]: pgmap v10148: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:54:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:56 smithi047 ceph-mon[36835]: pgmap v10149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:56.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:56 smithi029 ceph-mon[32610]: pgmap v10149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:56 smithi077 ceph-mon[37153]: pgmap v10149: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:54:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18319.788486 secs 2024-07-27T11:54:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18319.788486 secs 2024-07-27T11:54:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18319.788486 secs 2024-07-27T11:54:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:54:58 smithi077 ceph-mon[37153]: pgmap v10150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:54:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:54:58 smithi047 ceph-mon[36835]: pgmap v10150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:54:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:54:58 smithi029 ceph-mon[32610]: pgmap v10150: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:55:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:00 smithi047 ceph-mon[36835]: pgmap v10151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:55:01.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:00 smithi029 ceph-mon[32610]: pgmap v10151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:55:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:00 smithi077 ceph-mon[37153]: pgmap v10151: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:55:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:02 smithi047 ceph-mon[36835]: pgmap v10152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:55:03.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18324.788609 secs 2024-07-27T11:55:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:02 smithi029 ceph-mon[32610]: pgmap v10152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:55:03.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18324.788609 secs 2024-07-27T11:55:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:02 smithi077 ceph-mon[37153]: pgmap v10152: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:55:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18324.788609 secs 2024-07-27T11:55:05.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:04 smithi047 ceph-mon[36835]: pgmap v10153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:55:05.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:04 smithi029 ceph-mon[32610]: pgmap v10153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:55:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:04 smithi077 ceph-mon[37153]: pgmap v10153: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:55:07.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:06 smithi047 ceph-mon[36835]: pgmap v10154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:07.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:06 smithi029 ceph-mon[32610]: pgmap v10154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:06 smithi077 ceph-mon[37153]: pgmap v10154: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:07.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:07.459+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:55:07.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:07.459+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:55:07.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:07.459+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:55:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18329.788736 secs 2024-07-27T11:55:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18329.788736 secs 2024-07-27T11:55:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18329.788736 secs 2024-07-27T11:55:09.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:08 smithi047 ceph-mon[36835]: pgmap v10155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:55:09.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:08 smithi029 ceph-mon[32610]: pgmap v10155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:55:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:08 smithi077 ceph-mon[37153]: pgmap v10155: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:55:11.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:10 smithi047 ceph-mon[36835]: pgmap v10156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:11.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:10 smithi029 ceph-mon[32610]: pgmap v10156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:10 smithi077 ceph-mon[37153]: pgmap v10156: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:12 smithi047 ceph-mon[36835]: pgmap v10157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:55:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18334.788890 secs 2024-07-27T11:55:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:12 smithi029 ceph-mon[32610]: pgmap v10157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:55:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18334.788890 secs 2024-07-27T11:55:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:12 smithi077 ceph-mon[37153]: pgmap v10157: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:55:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18334.788890 secs 2024-07-27T11:55:15.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:14 smithi047 ceph-mon[36835]: pgmap v10158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:55:15.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:14 smithi029 ceph-mon[32610]: pgmap v10158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:55:15.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:14 smithi077 ceph-mon[37153]: pgmap v10158: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:55:15.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:15.407+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:55:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:15.407+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:55:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:15.407+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:55:15.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:15.407+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:55:17.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:16 smithi047 ceph-mon[36835]: pgmap v10159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:17.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:16 smithi029 ceph-mon[32610]: pgmap v10159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:16 smithi077 ceph-mon[37153]: pgmap v10159: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18339.789008 secs 2024-07-27T11:55:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18339.789008 secs 2024-07-27T11:55:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18339.789008 secs 2024-07-27T11:55:19.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:18 smithi029 ceph-mon[32610]: pgmap v10160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:55:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:18 smithi077 ceph-mon[37153]: pgmap v10160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:55:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:18 smithi047 ceph-mon[36835]: pgmap v10160: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:55:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:20 smithi077 ceph-mon[37153]: pgmap v10161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:20 smithi047 ceph-mon[36835]: pgmap v10161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:21.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:20 smithi029 ceph-mon[32610]: pgmap v10161: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:22.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:22.474+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:55:22.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:22.474+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:55:22.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:22.474+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:55:23.248 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:22 smithi047 ceph-mon[36835]: pgmap v10162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:55:23.249 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18344.789135 secs 2024-07-27T11:55:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:22 smithi077 ceph-mon[37153]: pgmap v10162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:55:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18344.789135 secs 2024-07-27T11:55:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:22 smithi029 ceph-mon[32610]: pgmap v10162: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:55:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18344.789135 secs 2024-07-27T11:55:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:24 smithi077 ceph-mon[37153]: pgmap v10163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:55:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:24 smithi047 ceph-mon[36835]: pgmap v10163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:55:25.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:24 smithi029 ceph-mon[32610]: pgmap v10163: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:55:27.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:26 smithi077 ceph-mon[37153]: pgmap v10164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:26 smithi047 ceph-mon[36835]: pgmap v10164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:26 smithi029 ceph-mon[32610]: pgmap v10164: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:28.257 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18349.789252 secs 2024-07-27T11:55:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18349.789252 secs 2024-07-27T11:55:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18349.789252 secs 2024-07-27T11:55:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:28 smithi077 ceph-mon[37153]: pgmap v10165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:55:29.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:28 smithi047 ceph-mon[36835]: pgmap v10165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:55:29.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:28 smithi029 ceph-mon[32610]: pgmap v10165: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:55:31.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:30 smithi029 ceph-mon[32610]: pgmap v10166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:30 smithi077 ceph-mon[37153]: pgmap v10166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:30 smithi047 ceph-mon[36835]: pgmap v10166: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:55:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:32 smithi077 ceph-mon[37153]: pgmap v10167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:55:33.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18354.789347 secs 2024-07-27T11:55:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:32 smithi047 ceph-mon[36835]: pgmap v10167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:55:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18354.789347 secs 2024-07-27T11:55:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:32 smithi029 ceph-mon[32610]: pgmap v10167: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:55:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18354.789347 secs 2024-07-27T11:55:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:34 smithi077 ceph-mon[37153]: pgmap v10168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:55:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:34 smithi047 ceph-mon[36835]: pgmap v10168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:55:35.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:34 smithi029 ceph-mon[32610]: pgmap v10168: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:55:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:36 smithi077 ceph-mon[37153]: pgmap v10169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:55:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:36 smithi047 ceph-mon[36835]: pgmap v10169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:55:37.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:36 smithi029 ceph-mon[32610]: pgmap v10169: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:55:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:37.490+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:55:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:37.490+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:55:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:37.490+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:55:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18359.789492 secs 2024-07-27T11:55:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18359.789492 secs 2024-07-27T11:55:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18359.789492 secs 2024-07-27T11:55:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:38 smithi077 ceph-mon[37153]: pgmap v10170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:55:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:38 smithi047 ceph-mon[36835]: pgmap v10170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:55:39.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:38 smithi029 ceph-mon[32610]: pgmap v10170: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:55:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:41 smithi077 ceph-mon[37153]: pgmap v10171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:55:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:41 smithi047 ceph-mon[36835]: pgmap v10171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:55:41.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:41 smithi029 ceph-mon[32610]: pgmap v10171: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:55:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:43 smithi077 ceph-mon[37153]: pgmap v10172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:55:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18364.789621 secs 2024-07-27T11:55:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:43 smithi047 ceph-mon[36835]: pgmap v10172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:55:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18364.789621 secs 2024-07-27T11:55:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:43 smithi029 ceph-mon[32610]: pgmap v10172: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:55:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18364.789621 secs 2024-07-27T11:55:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:55:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:55:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:55:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:45 smithi077 ceph-mon[37153]: pgmap v10173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:55:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:55:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:55:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:45 smithi077 ceph-mon[37153]: pgmap v10174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:55:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:45 smithi077 ceph-mon[37153]: pgmap v10175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:55:45.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:55:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:45 smithi047 ceph-mon[36835]: pgmap v10173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:55:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:55:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:55:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:45 smithi047 ceph-mon[36835]: pgmap v10174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:55:45.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:45 smithi047 ceph-mon[36835]: pgmap v10175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:55:45.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:55:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:45 smithi029 ceph-mon[32610]: pgmap v10173: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:55:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:55:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:55:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:45 smithi029 ceph-mon[32610]: pgmap v10174: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:55:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:45 smithi029 ceph-mon[32610]: pgmap v10175: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:55:45.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:55:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:46 smithi077 ceph-mon[37153]: pgmap v10176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:55:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:46 smithi047 ceph-mon[36835]: pgmap v10176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:55:46.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:46 smithi029 ceph-mon[32610]: pgmap v10176: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:55:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18369.789750 secs 2024-07-27T11:55:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18369.789750 secs 2024-07-27T11:55:47.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18369.789750 secs 2024-07-27T11:55:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:48 smithi077 ceph-mon[37153]: pgmap v10177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:55:48.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:48 smithi029 ceph-mon[32610]: pgmap v10177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:55:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:48 smithi047 ceph-mon[36835]: pgmap v10177: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:55:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:50 smithi077 ceph-mon[37153]: pgmap v10178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:55:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:50 smithi047 ceph-mon[36835]: pgmap v10178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:55:50.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:50 smithi029 ceph-mon[32610]: pgmap v10178: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:55:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:52.505+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:55:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:52.505+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:55:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:55:52.505+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:55:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:52 smithi077 ceph-mon[37153]: pgmap v10179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:55:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:52 smithi047 ceph-mon[36835]: pgmap v10179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:55:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:52 smithi029 ceph-mon[32610]: pgmap v10179: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:55:53.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18374.789880 secs 2024-07-27T11:55:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18374.789880 secs 2024-07-27T11:55:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18374.789880 secs 2024-07-27T11:55:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:54 smithi077 ceph-mon[37153]: pgmap v10180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:55:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:54 smithi047 ceph-mon[36835]: pgmap v10180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:55:54.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:54 smithi029 ceph-mon[32610]: pgmap v10180: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:55:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:56 smithi077 ceph-mon[37153]: pgmap v10181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:55:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:56 smithi047 ceph-mon[36835]: pgmap v10181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:55:56.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:56 smithi029 ceph-mon[32610]: pgmap v10181: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:55:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18379.790013 secs 2024-07-27T11:55:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18379.790013 secs 2024-07-27T11:55:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18379.790013 secs 2024-07-27T11:55:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:55:58 smithi077 ceph-mon[37153]: pgmap v10182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:55:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:55:58 smithi047 ceph-mon[36835]: pgmap v10182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:55:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:55:58 smithi029 ceph-mon[32610]: pgmap v10182: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:56:00.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:00.414+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:56:00.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:00.414+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:56:00.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:00.414+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:56:00.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:00.414+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:56:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:00 smithi029 ceph-mon[32610]: pgmap v10183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:56:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:00 smithi077 ceph-mon[37153]: pgmap v10183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:56:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:00 smithi047 ceph-mon[36835]: pgmap v10183: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:56:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:02 smithi077 ceph-mon[37153]: pgmap v10184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:02 smithi047 ceph-mon[36835]: pgmap v10184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:02.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:02 smithi029 ceph-mon[32610]: pgmap v10184: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18384.790156 secs 2024-07-27T11:56:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18384.790156 secs 2024-07-27T11:56:03.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18384.790156 secs 2024-07-27T11:56:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:04 smithi077 ceph-mon[37153]: pgmap v10185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:56:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:04 smithi047 ceph-mon[36835]: pgmap v10185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:56:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:04 smithi029 ceph-mon[32610]: pgmap v10185: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-27T11:56:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:06 smithi077 ceph-mon[37153]: pgmap v10186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:06 smithi047 ceph-mon[36835]: pgmap v10186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:06 smithi029 ceph-mon[32610]: pgmap v10186: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:07.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:07.505+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:56:07.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:07.505+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:56:07.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:07.505+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:56:07.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18389.790288 secs 2024-07-27T11:56:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18389.790288 secs 2024-07-27T11:56:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18389.790288 secs 2024-07-27T11:56:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:08 smithi077 ceph-mon[37153]: pgmap v10187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:56:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:08 smithi047 ceph-mon[36835]: pgmap v10187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:56:08.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:08 smithi029 ceph-mon[32610]: pgmap v10187: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:56:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:10 smithi077 ceph-mon[37153]: pgmap v10188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:56:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:10 smithi047 ceph-mon[36835]: pgmap v10188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:56:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:10 smithi029 ceph-mon[32610]: pgmap v10188: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:56:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:12 smithi077 ceph-mon[37153]: pgmap v10189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:12 smithi047 ceph-mon[36835]: pgmap v10189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:12 smithi029 ceph-mon[32610]: pgmap v10189: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18394.790362 secs 2024-07-27T11:56:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18394.790362 secs 2024-07-27T11:56:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18394.790362 secs 2024-07-27T11:56:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:14 smithi077 ceph-mon[37153]: pgmap v10190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:56:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:14 smithi047 ceph-mon[36835]: pgmap v10190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:56:14.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:14 smithi029 ceph-mon[32610]: pgmap v10190: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:56:17.831 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:17 smithi029 ceph-mon[32610]: pgmap v10191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:17 smithi077 ceph-mon[37153]: pgmap v10191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:17 smithi047 ceph-mon[36835]: pgmap v10191: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:18.733 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18399.790488 secs 2024-07-27T11:56:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18399.790488 secs 2024-07-27T11:56:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18399.790488 secs 2024-07-27T11:56:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:19 smithi077 ceph-mon[37153]: pgmap v10192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:56:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:19 smithi047 ceph-mon[36835]: pgmap v10192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:56:19.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:19 smithi029 ceph-mon[32610]: pgmap v10192: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:56:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:21 smithi077 ceph-mon[37153]: pgmap v10193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:56:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:21 smithi047 ceph-mon[36835]: pgmap v10193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:56:21.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:21 smithi029 ceph-mon[32610]: pgmap v10193: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:56:22.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:22.520+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:56:22.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:22.521+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:56:22.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:22.521+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:56:23.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18404.790592 secs 2024-07-27T11:56:23.749 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:23 smithi047 ceph-mon[36835]: pgmap v10194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:56:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18404.790592 secs 2024-07-27T11:56:23.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:23 smithi077 ceph-mon[37153]: pgmap v10194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:56:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18404.790592 secs 2024-07-27T11:56:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:23 smithi029 ceph-mon[32610]: pgmap v10194: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:56:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:25 smithi077 ceph-mon[37153]: pgmap v10195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:56:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:25 smithi047 ceph-mon[36835]: pgmap v10195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:56:25.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:25 smithi029 ceph-mon[32610]: pgmap v10195: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T11:56:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:27 smithi077 ceph-mon[37153]: pgmap v10196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:27 smithi047 ceph-mon[36835]: pgmap v10196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:27 smithi029 ceph-mon[32610]: pgmap v10196: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:28.757 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18409.790728 secs 2024-07-27T11:56:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18409.790728 secs 2024-07-27T11:56:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18409.790728 secs 2024-07-27T11:56:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:29 smithi077 ceph-mon[37153]: pgmap v10197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:56:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:29 smithi047 ceph-mon[36835]: pgmap v10197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:56:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:29 smithi029 ceph-mon[32610]: pgmap v10197: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:56:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:31 smithi077 ceph-mon[37153]: pgmap v10198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:56:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:31 smithi047 ceph-mon[36835]: pgmap v10198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:56:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:31 smithi029 ceph-mon[32610]: pgmap v10198: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:56:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18414.790864 secs 2024-07-27T11:56:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:33 smithi077 ceph-mon[37153]: pgmap v10199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:56:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18414.790864 secs 2024-07-27T11:56:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:33 smithi047 ceph-mon[36835]: pgmap v10199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:56:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18414.790864 secs 2024-07-27T11:56:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:33 smithi029 ceph-mon[32610]: pgmap v10199: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T11:56:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:35 smithi077 ceph-mon[37153]: pgmap v10200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:56:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:35 smithi047 ceph-mon[36835]: pgmap v10200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:56:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:35 smithi029 ceph-mon[32610]: pgmap v10200: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:56:37.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:37.536+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:56:37.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:37.536+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:56:37.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:37.536+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:56:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:37 smithi077 ceph-mon[37153]: pgmap v10201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:37 smithi047 ceph-mon[36835]: pgmap v10201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:37.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:37 smithi029 ceph-mon[32610]: pgmap v10201: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:56:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18419.790996 secs 2024-07-27T11:56:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18419.790996 secs 2024-07-27T11:56:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18419.790996 secs 2024-07-27T11:56:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:39 smithi077 ceph-mon[37153]: pgmap v10202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:56:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:39 smithi047 ceph-mon[36835]: pgmap v10202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:56:39.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:39 smithi029 ceph-mon[32610]: pgmap v10202: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:56:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:41 smithi077 ceph-mon[37153]: pgmap v10203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:56:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:41 smithi047 ceph-mon[36835]: pgmap v10203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:56:41.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:41 smithi029 ceph-mon[32610]: pgmap v10203: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:56:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18424.791079 secs 2024-07-27T11:56:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:43 smithi077 ceph-mon[37153]: pgmap v10204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:56:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18424.791079 secs 2024-07-27T11:56:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:43 smithi047 ceph-mon[36835]: pgmap v10204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:56:43.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18424.791079 secs 2024-07-27T11:56:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:43 smithi029 ceph-mon[32610]: pgmap v10204: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:56:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:44 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:56:44.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:44 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:56:44.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:44 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:56:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:45 smithi077 ceph-mon[37153]: pgmap v10205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:56:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:56:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:56:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:56:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:56:45.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:45 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:56:45.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:45 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:56:45.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:56:45.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:56:45.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:45 smithi077 ceph-mon[37153]: pgmap v10206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:56:45.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:45 smithi077 ceph-mon[37153]: pgmap v10207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:56:45.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:56:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:45 smithi047 ceph-mon[36835]: pgmap v10205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:56:45.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:56:45.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:56:45.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:56:45.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:56:45.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:45 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:56:45.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:45 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:56:45.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:56:45.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:56:45.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:45 smithi047 ceph-mon[36835]: pgmap v10206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:56:45.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:45 smithi047 ceph-mon[36835]: pgmap v10207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:56:45.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:56:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:45 smithi029 ceph-mon[32610]: pgmap v10205: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T11:56:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:56:45.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:56:45.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:56:45.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:56:45.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:45 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T11:56:45.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:45 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T11:56:45.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:56:45.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:56:45.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:45 smithi029 ceph-mon[32610]: pgmap v10206: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:56:45.965 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:45 smithi029 ceph-mon[32610]: pgmap v10207: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:56:45.965 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:56:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:48 smithi077 ceph-mon[37153]: pgmap v10208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:56:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18429.791202 secs 2024-07-27T11:56:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:48 smithi047 ceph-mon[36835]: pgmap v10208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:56:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18429.791202 secs 2024-07-27T11:56:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:48 smithi029 ceph-mon[32610]: pgmap v10208: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:56:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18429.791202 secs 2024-07-27T11:56:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:50 smithi077 ceph-mon[37153]: pgmap v10209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:56:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:50 smithi047 ceph-mon[36835]: pgmap v10209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:56:50.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:50 smithi029 ceph-mon[32610]: pgmap v10209: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:56:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:52 smithi077 ceph-mon[37153]: pgmap v10210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:56:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:52 smithi047 ceph-mon[36835]: pgmap v10210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:56:52.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:52 smithi029 ceph-mon[32610]: pgmap v10210: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:56:52.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:52.542+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:56:52.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:52.542+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:56:52.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:56:52.542+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:56:53.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18434.791329 secs 2024-07-27T11:56:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18434.791329 secs 2024-07-27T11:56:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18434.791329 secs 2024-07-27T11:56:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:54 smithi077 ceph-mon[37153]: pgmap v10211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:56:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:54 smithi047 ceph-mon[36835]: pgmap v10211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:56:54.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:54 smithi029 ceph-mon[32610]: pgmap v10211: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:56:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:56 smithi077 ceph-mon[37153]: pgmap v10212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:56:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:56 smithi047 ceph-mon[36835]: pgmap v10212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:56:56.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:56 smithi029 ceph-mon[32610]: pgmap v10212: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:56:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:58 smithi077 ceph-mon[37153]: pgmap v10213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:56:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:56:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18439.791487 secs 2024-07-27T11:56:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:58 smithi047 ceph-mon[36835]: pgmap v10213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:56:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:56:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18439.791487 secs 2024-07-27T11:56:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:58 smithi029 ceph-mon[32610]: pgmap v10213: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:56:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:56:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18439.791487 secs 2024-07-27T11:57:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:00 smithi077 ceph-mon[37153]: pgmap v10214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:57:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:00 smithi047 ceph-mon[36835]: pgmap v10214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:57:00.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:00 smithi029 ceph-mon[32610]: pgmap v10214: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:57:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:02 smithi077 ceph-mon[37153]: pgmap v10215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:02 smithi047 ceph-mon[36835]: pgmap v10215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:02.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:02 smithi029 ceph-mon[32610]: pgmap v10215: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18444.791610 secs 2024-07-27T11:57:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18444.791610 secs 2024-07-27T11:57:03.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18444.791610 secs 2024-07-27T11:57:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:04 smithi077 ceph-mon[37153]: pgmap v10216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:57:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:04 smithi047 ceph-mon[36835]: pgmap v10216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:57:04.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:04 smithi029 ceph-mon[32610]: pgmap v10216: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:57:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:06 smithi077 ceph-mon[37153]: pgmap v10217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:57:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:06 smithi047 ceph-mon[36835]: pgmap v10217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:57:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:06 smithi029 ceph-mon[32610]: pgmap v10217: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:57:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:57:07.553+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:57:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:57:07.553+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:57:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:57:07.553+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:57:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:08 smithi077 ceph-mon[37153]: pgmap v10218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:57:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18449.791707 secs 2024-07-27T11:57:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:08 smithi047 ceph-mon[36835]: pgmap v10218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:57:08.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18449.791707 secs 2024-07-27T11:57:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:08 smithi029 ceph-mon[32610]: pgmap v10218: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:57:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18449.791707 secs 2024-07-27T11:57:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:10 smithi077 ceph-mon[37153]: pgmap v10219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:57:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:10 smithi047 ceph-mon[36835]: pgmap v10219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:57:10.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:10 smithi029 ceph-mon[32610]: pgmap v10219: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:57:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:12 smithi077 ceph-mon[37153]: pgmap v10220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:57:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:12 smithi047 ceph-mon[36835]: pgmap v10220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:57:12.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:12 smithi029 ceph-mon[32610]: pgmap v10220: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T11:57:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18454.791830 secs 2024-07-27T11:57:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18454.791830 secs 2024-07-27T11:57:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18454.791830 secs 2024-07-27T11:57:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:14 smithi077 ceph-mon[37153]: pgmap v10221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:57:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:14 smithi047 ceph-mon[36835]: pgmap v10221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:57:14.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:14 smithi029 ceph-mon[32610]: pgmap v10221: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T11:57:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:16 smithi077 ceph-mon[37153]: pgmap v10222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:16 smithi047 ceph-mon[36835]: pgmap v10222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:16 smithi029 ceph-mon[32610]: pgmap v10222: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:18 smithi077 ceph-mon[37153]: pgmap v10223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:57:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18459.791955 secs 2024-07-27T11:57:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:18 smithi047 ceph-mon[36835]: pgmap v10223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:57:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18459.791955 secs 2024-07-27T11:57:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:18 smithi029 ceph-mon[32610]: pgmap v10223: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:57:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18459.791955 secs 2024-07-27T11:57:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:20 smithi077 ceph-mon[37153]: pgmap v10224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:57:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:20 smithi047 ceph-mon[36835]: pgmap v10224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:57:20.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:20 smithi029 ceph-mon[32610]: pgmap v10224: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:57:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:22 smithi047 ceph-mon[36835]: pgmap v10225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:22.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:22 smithi029 ceph-mon[32610]: pgmap v10225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:57:22.554+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:57:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:57:22.554+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:57:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:57:22.554+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:57:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:22 smithi077 ceph-mon[37153]: pgmap v10225: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18464.792025 secs 2024-07-27T11:57:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18464.792025 secs 2024-07-27T11:57:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18464.792025 secs 2024-07-27T11:57:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:24 smithi047 ceph-mon[36835]: pgmap v10226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:57:24.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:24 smithi029 ceph-mon[32610]: pgmap v10226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:57:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:24 smithi077 ceph-mon[37153]: pgmap v10226: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:57:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:26 smithi047 ceph-mon[36835]: pgmap v10227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:26.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:26 smithi029 ceph-mon[32610]: pgmap v10227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:26 smithi077 ceph-mon[37153]: pgmap v10227: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:28.408 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:28 smithi077 ceph-mon[37153]: pgmap v10228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:57:28.408 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18469.792146 secs 2024-07-27T11:57:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:28 smithi047 ceph-mon[36835]: pgmap v10228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:57:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18469.792146 secs 2024-07-27T11:57:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:28 smithi029 ceph-mon[32610]: pgmap v10228: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:57:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18469.792146 secs 2024-07-27T11:57:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:30 smithi047 ceph-mon[36835]: pgmap v10229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:57:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:30 smithi029 ceph-mon[32610]: pgmap v10229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:57:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:30 smithi077 ceph-mon[37153]: pgmap v10229: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:57:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:32 smithi047 ceph-mon[36835]: pgmap v10230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:32.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:32 smithi029 ceph-mon[32610]: pgmap v10230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:32 smithi077 ceph-mon[37153]: pgmap v10230: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18474.792291 secs 2024-07-27T11:57:33.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18474.792291 secs 2024-07-27T11:57:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18474.792291 secs 2024-07-27T11:57:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:34 smithi047 ceph-mon[36835]: pgmap v10231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:57:34.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:34 smithi029 ceph-mon[32610]: pgmap v10231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:57:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:34 smithi077 ceph-mon[37153]: pgmap v10231: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:57:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:36 smithi047 ceph-mon[36835]: pgmap v10232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:36 smithi029 ceph-mon[32610]: pgmap v10232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:36 smithi077 ceph-mon[37153]: pgmap v10232: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:37.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:57:37.559+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:57:37.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:57:37.559+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:57:37.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:57:37.559+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:57:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:38 smithi047 ceph-mon[36835]: pgmap v10233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:57:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18479.792420 secs 2024-07-27T11:57:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:38 smithi029 ceph-mon[32610]: pgmap v10233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:57:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18479.792420 secs 2024-07-27T11:57:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:38 smithi077 ceph-mon[37153]: pgmap v10233: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:57:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18479.792420 secs 2024-07-27T11:57:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:40 smithi047 ceph-mon[36835]: pgmap v10234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:57:40.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:40 smithi029 ceph-mon[32610]: pgmap v10234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:57:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:40 smithi077 ceph-mon[37153]: pgmap v10234: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:57:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:42 smithi047 ceph-mon[36835]: pgmap v10235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:42.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:42 smithi029 ceph-mon[32610]: pgmap v10235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:42 smithi077 ceph-mon[37153]: pgmap v10235: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18484.792512 secs 2024-07-27T11:57:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18484.792512 secs 2024-07-27T11:57:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18484.792512 secs 2024-07-27T11:57:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:44 smithi047 ceph-mon[36835]: pgmap v10236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:57:44.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:44 smithi029 ceph-mon[32610]: pgmap v10236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:57:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:44 smithi077 ceph-mon[37153]: pgmap v10236: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:57:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:45 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:57:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:45 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:57:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:45 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:57:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:46 smithi047 ceph-mon[36835]: pgmap v10237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:57:46.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:57:46.448 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:57:46.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:46 smithi029 ceph-mon[32610]: pgmap v10237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:57:46.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:57:46.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:57:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:46 smithi077 ceph-mon[37153]: pgmap v10237: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:57:46.661 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.662 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.662 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.662 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.662 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.662 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.662 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.663 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T11:57:46.663 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:57:46.663 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:57:46.663 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:57:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:47 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:57:47.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:47 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:57:47.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:47 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:57:47.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:47 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:57:47.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:47 smithi047 ceph-mon[36835]: pgmap v10238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:57:47.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:47 smithi047 ceph-mon[36835]: pgmap v10239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:57:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:47 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:57:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:47 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:57:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:47 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:57:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:47 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:57:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:47 smithi029 ceph-mon[32610]: pgmap v10238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:57:47.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:47 smithi029 ceph-mon[32610]: pgmap v10239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:57:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:47 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T11:57:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:47 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T11:57:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:47 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T11:57:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:47 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T11:57:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:47 smithi077 ceph-mon[37153]: pgmap v10238: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:57:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:47 smithi077 ceph-mon[37153]: pgmap v10239: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:57:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18489.792653 secs 2024-07-27T11:57:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18489.792653 secs 2024-07-27T11:57:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18489.792653 secs 2024-07-27T11:57:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:49 smithi047 ceph-mon[36835]: pgmap v10240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:57:49.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:49 smithi029 ceph-mon[32610]: pgmap v10240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:57:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:49 smithi077 ceph-mon[37153]: pgmap v10240: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:57:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:51 smithi047 ceph-mon[36835]: pgmap v10241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:57:51.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:51 smithi029 ceph-mon[32610]: pgmap v10241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:57:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:51 smithi077 ceph-mon[37153]: pgmap v10241: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T11:57:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:57:52.574+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:57:52.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:57:52.574+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:57:52.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:57:52.574+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:57:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:53 smithi047 ceph-mon[36835]: pgmap v10242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:57:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18494.792751 secs 2024-07-27T11:57:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:53 smithi029 ceph-mon[32610]: pgmap v10242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:57:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18494.792751 secs 2024-07-27T11:57:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:53 smithi077 ceph-mon[37153]: pgmap v10242: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:57:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18494.792751 secs 2024-07-27T11:57:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:55 smithi047 ceph-mon[36835]: pgmap v10243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:57:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:55 smithi029 ceph-mon[32610]: pgmap v10243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:57:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:55 smithi077 ceph-mon[37153]: pgmap v10243: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:57:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:57 smithi029 ceph-mon[32610]: pgmap v10244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:57:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:57 smithi077 ceph-mon[37153]: pgmap v10244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:57:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:57 smithi047 ceph-mon[36835]: pgmap v10244: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:57:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18499.792873 secs 2024-07-27T11:57:58.507 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18499.792873 secs 2024-07-27T11:57:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18499.792873 secs 2024-07-27T11:57:59.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:57:59 smithi029 ceph-mon[32610]: pgmap v10245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:57:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:57:59 smithi077 ceph-mon[37153]: pgmap v10245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:57:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:57:59 smithi047 ceph-mon[36835]: pgmap v10245: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:58:01.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:01 smithi029 ceph-mon[32610]: pgmap v10246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:58:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:01 smithi077 ceph-mon[37153]: pgmap v10246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:58:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:01 smithi047 ceph-mon[36835]: pgmap v10246: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:58:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:03 smithi029 ceph-mon[32610]: pgmap v10247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18504.792996 secs 2024-07-27T11:58:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:03 smithi077 ceph-mon[37153]: pgmap v10247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18504.792996 secs 2024-07-27T11:58:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:03 smithi047 ceph-mon[36835]: pgmap v10247: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18504.792996 secs 2024-07-27T11:58:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:05 smithi077 ceph-mon[37153]: pgmap v10248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:58:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:05 smithi047 ceph-mon[36835]: pgmap v10248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:58:05.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:05 smithi029 ceph-mon[32610]: pgmap v10248: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:58:07.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:58:07.588+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:58:07.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:58:07.588+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:58:07.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:58:07.588+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:58:07.591 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:07 smithi029 ceph-mon[32610]: pgmap v10249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:58:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:07 smithi077 ceph-mon[37153]: pgmap v10249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:58:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:07 smithi047 ceph-mon[36835]: pgmap v10249: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:58:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18509.793108 secs 2024-07-27T11:58:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18509.793108 secs 2024-07-27T11:58:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18509.793108 secs 2024-07-27T11:58:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:09 smithi077 ceph-mon[37153]: pgmap v10250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:58:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:09 smithi047 ceph-mon[36835]: pgmap v10250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:58:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:09 smithi029 ceph-mon[32610]: pgmap v10250: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:58:11.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:11 smithi077 ceph-mon[37153]: pgmap v10251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:58:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:11 smithi047 ceph-mon[36835]: pgmap v10251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:58:11.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:11 smithi029 ceph-mon[32610]: pgmap v10251: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:58:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:13 smithi077 ceph-mon[37153]: pgmap v10252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18514.793226 secs 2024-07-27T11:58:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:13 smithi047 ceph-mon[36835]: pgmap v10252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18514.793226 secs 2024-07-27T11:58:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:13 smithi029 ceph-mon[32610]: pgmap v10252: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18514.793226 secs 2024-07-27T11:58:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:15 smithi077 ceph-mon[37153]: pgmap v10253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:58:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:15 smithi047 ceph-mon[36835]: pgmap v10253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:58:15.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:15 smithi029 ceph-mon[32610]: pgmap v10253: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:58:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:17 smithi077 ceph-mon[37153]: pgmap v10254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:58:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:17 smithi047 ceph-mon[36835]: pgmap v10254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:58:17.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:17 smithi029 ceph-mon[32610]: pgmap v10254: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:58:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18519.793373 secs 2024-07-27T11:58:18.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18519.793373 secs 2024-07-27T11:58:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18519.793373 secs 2024-07-27T11:58:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:19 smithi077 ceph-mon[37153]: pgmap v10255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:58:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:19 smithi047 ceph-mon[36835]: pgmap v10255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:58:19.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:19 smithi029 ceph-mon[32610]: pgmap v10255: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T11:58:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:21 smithi077 ceph-mon[37153]: pgmap v10256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:58:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:21 smithi047 ceph-mon[36835]: pgmap v10256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:58:21.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:21 smithi029 ceph-mon[32610]: pgmap v10256: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T11:58:22.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:58:22.604+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:58:22.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:58:22.604+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:58:22.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:58:22.604+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:58:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:23 smithi077 ceph-mon[37153]: pgmap v10257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18524.793495 secs 2024-07-27T11:58:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:23 smithi047 ceph-mon[36835]: pgmap v10257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18524.793495 secs 2024-07-27T11:58:23.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:23 smithi029 ceph-mon[32610]: pgmap v10257: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:23.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18524.793495 secs 2024-07-27T11:58:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:25 smithi077 ceph-mon[37153]: pgmap v10258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:58:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:25 smithi047 ceph-mon[36835]: pgmap v10258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:58:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:25 smithi029 ceph-mon[32610]: pgmap v10258: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:58:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:27 smithi077 ceph-mon[37153]: pgmap v10259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:58:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:27 smithi047 ceph-mon[36835]: pgmap v10259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:58:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:27 smithi029 ceph-mon[32610]: pgmap v10259: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:58:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18529.793619 secs 2024-07-27T11:58:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18529.793619 secs 2024-07-27T11:58:28.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18529.793619 secs 2024-07-27T11:58:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:29 smithi077 ceph-mon[37153]: pgmap v10260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:58:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:29 smithi047 ceph-mon[36835]: pgmap v10260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:58:29.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:29 smithi029 ceph-mon[32610]: pgmap v10260: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:58:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:31 smithi077 ceph-mon[37153]: pgmap v10261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:58:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:31 smithi047 ceph-mon[36835]: pgmap v10261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:58:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:31 smithi029 ceph-mon[32610]: pgmap v10261: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:58:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:33 smithi077 ceph-mon[37153]: pgmap v10262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18534.793743 secs 2024-07-27T11:58:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:33 smithi047 ceph-mon[36835]: pgmap v10262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18534.793743 secs 2024-07-27T11:58:33.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:33 smithi029 ceph-mon[32610]: pgmap v10262: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:33.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18534.793743 secs 2024-07-27T11:58:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:35 smithi077 ceph-mon[37153]: pgmap v10263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:58:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:35 smithi047 ceph-mon[36835]: pgmap v10263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:58:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:35 smithi029 ceph-mon[32610]: pgmap v10263: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:58:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:37 smithi077 ceph-mon[37153]: pgmap v10264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:58:37.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:58:37.620+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:58:37.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:58:37.620+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:58:37.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:58:37.620+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:58:37.621 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:37 smithi029 ceph-mon[32610]: pgmap v10264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:58:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:37 smithi047 ceph-mon[36835]: pgmap v10264: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:58:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18539.793875 secs 2024-07-27T11:58:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18539.793875 secs 2024-07-27T11:58:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18539.793875 secs 2024-07-27T11:58:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:39 smithi077 ceph-mon[37153]: pgmap v10265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:58:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:39 smithi047 ceph-mon[36835]: pgmap v10265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:58:39.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:39 smithi029 ceph-mon[32610]: pgmap v10265: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T11:58:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:41 smithi077 ceph-mon[37153]: pgmap v10266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:58:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:41 smithi047 ceph-mon[36835]: pgmap v10266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:58:41.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:41 smithi029 ceph-mon[32610]: pgmap v10266: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T11:58:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:43 smithi077 ceph-mon[37153]: pgmap v10267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18544.794000 secs 2024-07-27T11:58:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:43 smithi047 ceph-mon[36835]: pgmap v10267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18544.794000 secs 2024-07-27T11:58:43.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:43 smithi029 ceph-mon[32610]: pgmap v10267: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:58:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18544.794000 secs 2024-07-27T11:58:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:45 smithi077 ceph-mon[37153]: pgmap v10268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:58:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:45 smithi047 ceph-mon[36835]: pgmap v10268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:58:45.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:45 smithi029 ceph-mon[32610]: pgmap v10268: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:58:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:58:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:58:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:58:46.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:58:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:58:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:58:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:58:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:58:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:58:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:58:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:58:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:58:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:47 smithi077 ceph-mon[37153]: pgmap v10269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:58:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:47 smithi077 ceph-mon[37153]: pgmap v10270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:58:47.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:47 smithi077 ceph-mon[37153]: pgmap v10271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:58:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:47 smithi047 ceph-mon[36835]: pgmap v10269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:58:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:47 smithi047 ceph-mon[36835]: pgmap v10270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:58:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:47 smithi047 ceph-mon[36835]: pgmap v10271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:58:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:47 smithi029 ceph-mon[32610]: pgmap v10269: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:58:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:47 smithi029 ceph-mon[32610]: pgmap v10270: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T11:58:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:47 smithi029 ceph-mon[32610]: pgmap v10271: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T11:58:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18549.794121 secs 2024-07-27T11:58:48.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18549.794121 secs 2024-07-27T11:58:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18549.794121 secs 2024-07-27T11:58:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:49 smithi077 ceph-mon[37153]: pgmap v10272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:58:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:49 smithi047 ceph-mon[36835]: pgmap v10272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:58:49.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:49 smithi029 ceph-mon[32610]: pgmap v10272: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:58:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:51 smithi077 ceph-mon[37153]: pgmap v10273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:58:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:51 smithi047 ceph-mon[36835]: pgmap v10273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:58:51.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:51 smithi029 ceph-mon[32610]: pgmap v10273: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:58:52.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:58:52.621+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:58:52.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:58:52.621+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:58:52.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:58:52.621+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:58:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18554.794250 secs 2024-07-27T11:58:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:53 smithi077 ceph-mon[37153]: pgmap v10274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:58:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18554.794250 secs 2024-07-27T11:58:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:53 smithi047 ceph-mon[36835]: pgmap v10274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:58:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18554.794250 secs 2024-07-27T11:58:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:53 smithi029 ceph-mon[32610]: pgmap v10274: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:58:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:55 smithi077 ceph-mon[37153]: pgmap v10275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:58:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:55 smithi047 ceph-mon[36835]: pgmap v10275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:58:55.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:55 smithi029 ceph-mon[32610]: pgmap v10275: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:58:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:57 smithi077 ceph-mon[37153]: pgmap v10276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:58:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:57 smithi047 ceph-mon[36835]: pgmap v10276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:58:57.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:57 smithi029 ceph-mon[32610]: pgmap v10276: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T11:58:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18559.794379 secs 2024-07-27T11:58:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18559.794379 secs 2024-07-27T11:58:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18559.794379 secs 2024-07-27T11:58:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:58:59 smithi077 ceph-mon[37153]: pgmap v10277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:58:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:58:59 smithi047 ceph-mon[36835]: pgmap v10277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:58:59.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:58:59 smithi029 ceph-mon[32610]: pgmap v10277: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T11:59:00.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:00.438+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:59:00.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:00.438+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:59:00.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:00.438+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T11:59:00.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:00.438+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T11:59:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:01 smithi077 ceph-mon[37153]: pgmap v10278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:01 smithi047 ceph-mon[36835]: pgmap v10278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:01.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:01 smithi029 ceph-mon[32610]: pgmap v10278: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18564.794524 secs 2024-07-27T11:59:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:03 smithi077 ceph-mon[37153]: pgmap v10279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18564.794524 secs 2024-07-27T11:59:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:03 smithi047 ceph-mon[36835]: pgmap v10279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18564.794524 secs 2024-07-27T11:59:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:03 smithi029 ceph-mon[32610]: pgmap v10279: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:05 smithi077 ceph-mon[37153]: pgmap v10280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:05 smithi047 ceph-mon[36835]: pgmap v10280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:05.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:05 smithi029 ceph-mon[32610]: pgmap v10280: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:07 smithi077 ceph-mon[37153]: pgmap v10281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:07.630+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:59:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:07.630+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:59:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:07.630+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:59:07.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:07 smithi029 ceph-mon[32610]: pgmap v10281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:07 smithi047 ceph-mon[36835]: pgmap v10281: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18569.794656 secs 2024-07-27T11:59:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18569.794656 secs 2024-07-27T11:59:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18569.794656 secs 2024-07-27T11:59:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:09 smithi077 ceph-mon[37153]: pgmap v10282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:09 smithi047 ceph-mon[36835]: pgmap v10282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:09.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:09 smithi029 ceph-mon[32610]: pgmap v10282: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:11 smithi047 ceph-mon[36835]: pgmap v10283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:11.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:11 smithi029 ceph-mon[32610]: pgmap v10283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:11 smithi077 ceph-mon[37153]: pgmap v10283: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18574.794777 secs 2024-07-27T11:59:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:13 smithi047 ceph-mon[36835]: pgmap v10284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:13.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18574.794777 secs 2024-07-27T11:59:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:13 smithi029 ceph-mon[32610]: pgmap v10284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18574.794777 secs 2024-07-27T11:59:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:13 smithi077 ceph-mon[37153]: pgmap v10284: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:15 smithi047 ceph-mon[36835]: pgmap v10285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:15.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:15 smithi029 ceph-mon[32610]: pgmap v10285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:15 smithi077 ceph-mon[37153]: pgmap v10285: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:17 smithi047 ceph-mon[36835]: pgmap v10286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:17.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:17 smithi029 ceph-mon[32610]: pgmap v10286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:17 smithi077 ceph-mon[37153]: pgmap v10286: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18579.794901 secs 2024-07-27T11:59:18.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18579.794901 secs 2024-07-27T11:59:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18579.794901 secs 2024-07-27T11:59:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:19 smithi047 ceph-mon[36835]: pgmap v10287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:19.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:19 smithi029 ceph-mon[32610]: pgmap v10287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:19.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:19 smithi077 ceph-mon[37153]: pgmap v10287: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:21 smithi047 ceph-mon[36835]: pgmap v10288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:21 smithi029 ceph-mon[32610]: pgmap v10288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:21 smithi077 ceph-mon[37153]: pgmap v10288: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:22.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:22.644+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:59:22.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:22.644+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:59:22.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:22.644+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:59:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18584.795029 secs 2024-07-27T11:59:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:23 smithi047 ceph-mon[36835]: pgmap v10289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18584.795029 secs 2024-07-27T11:59:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:23 smithi029 ceph-mon[32610]: pgmap v10289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18584.795029 secs 2024-07-27T11:59:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:23 smithi077 ceph-mon[37153]: pgmap v10289: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:25 smithi047 ceph-mon[36835]: pgmap v10290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:25 smithi029 ceph-mon[32610]: pgmap v10290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:25 smithi077 ceph-mon[37153]: pgmap v10290: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:27 smithi047 ceph-mon[36835]: pgmap v10291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:27 smithi029 ceph-mon[32610]: pgmap v10291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:27 smithi077 ceph-mon[37153]: pgmap v10291: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18589.795121 secs 2024-07-27T11:59:28.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18589.795121 secs 2024-07-27T11:59:28.757 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18589.795121 secs 2024-07-27T11:59:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:29 smithi047 ceph-mon[36835]: pgmap v10292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:29.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:29 smithi029 ceph-mon[32610]: pgmap v10292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:29 smithi077 ceph-mon[37153]: pgmap v10292: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:31 smithi047 ceph-mon[36835]: pgmap v10293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:31 smithi029 ceph-mon[32610]: pgmap v10293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:31 smithi077 ceph-mon[37153]: pgmap v10293: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18594.795248 secs 2024-07-27T11:59:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:33 smithi047 ceph-mon[36835]: pgmap v10294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:33.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18594.795248 secs 2024-07-27T11:59:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:33 smithi029 ceph-mon[32610]: pgmap v10294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18594.795248 secs 2024-07-27T11:59:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:33 smithi077 ceph-mon[37153]: pgmap v10294: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:35 smithi047 ceph-mon[36835]: pgmap v10295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:35 smithi029 ceph-mon[32610]: pgmap v10295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:35 smithi077 ceph-mon[37153]: pgmap v10295: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:37.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:37.659+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:59:37.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:37.659+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:59:37.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:37.659+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:59:37.661 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:37 smithi029 ceph-mon[32610]: pgmap v10296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:37 smithi047 ceph-mon[36835]: pgmap v10296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:37 smithi077 ceph-mon[37153]: pgmap v10296: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18599.795375 secs 2024-07-27T11:59:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18599.795375 secs 2024-07-27T11:59:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18599.795375 secs 2024-07-27T11:59:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:39 smithi047 ceph-mon[36835]: pgmap v10297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:39.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:39 smithi029 ceph-mon[32610]: pgmap v10297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:39 smithi077 ceph-mon[37153]: pgmap v10297: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:41 smithi047 ceph-mon[36835]: pgmap v10298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:41.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:41 smithi029 ceph-mon[32610]: pgmap v10298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:41 smithi077 ceph-mon[37153]: pgmap v10298: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T11:59:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18604.795504 secs 2024-07-27T11:59:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:43 smithi047 ceph-mon[36835]: pgmap v10299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:43.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18604.795504 secs 2024-07-27T11:59:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:43 smithi029 ceph-mon[32610]: pgmap v10299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18604.795504 secs 2024-07-27T11:59:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:43 smithi077 ceph-mon[37153]: pgmap v10299: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T11:59:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:45 smithi047 ceph-mon[36835]: pgmap v10300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:45.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:45 smithi029 ceph-mon[32610]: pgmap v10300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:45 smithi077 ceph-mon[37153]: pgmap v10300: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T11:59:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:59:46.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:59:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T11:59:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:47 smithi047 ceph-mon[36835]: pgmap v10301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:59:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:59:47.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:47 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:59:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:47 smithi029 ceph-mon[32610]: pgmap v10301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:59:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:59:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:47 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:59:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:47 smithi077 ceph-mon[37153]: pgmap v10301: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T11:59:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T11:59:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T11:59:47.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:47 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T11:59:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:48 smithi047 ceph-mon[36835]: pgmap v10302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:59:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:48 smithi047 ceph-mon[36835]: pgmap v10303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:59:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18609.795646 secs 2024-07-27T11:59:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:48 smithi029 ceph-mon[32610]: pgmap v10302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:59:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:48 smithi029 ceph-mon[32610]: pgmap v10303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:59:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18609.795646 secs 2024-07-27T11:59:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:48 smithi077 ceph-mon[37153]: pgmap v10302: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T11:59:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:48 smithi077 ceph-mon[37153]: pgmap v10303: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T11:59:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18609.795646 secs 2024-07-27T11:59:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:50 smithi047 ceph-mon[36835]: pgmap v10304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:59:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:50 smithi029 ceph-mon[32610]: pgmap v10304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:59:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:50 smithi077 ceph-mon[37153]: pgmap v10304: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T11:59:52.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:52.664+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T11:59:52.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:52.664+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T11:59:52.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T11:59:52.664+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T11:59:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:52 smithi047 ceph-mon[36835]: pgmap v10305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:59:52.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:52 smithi029 ceph-mon[32610]: pgmap v10305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:59:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:52 smithi077 ceph-mon[37153]: pgmap v10305: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:59:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18614.795776 secs 2024-07-27T11:59:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18614.795776 secs 2024-07-27T11:59:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18614.795776 secs 2024-07-27T11:59:54.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:54 smithi029 ceph-mon[32610]: pgmap v10306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:59:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:54 smithi077 ceph-mon[37153]: pgmap v10306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:59:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:54 smithi047 ceph-mon[36835]: pgmap v10306: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T11:59:55.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:55 smithi029 ceph-mon[32610]: pgmap v10307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:59:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:55 smithi077 ceph-mon[37153]: pgmap v10307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:59:55.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:55 smithi047 ceph-mon[36835]: pgmap v10307: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T11:59:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:57 smithi029 ceph-mon[32610]: pgmap v10308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:59:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18619.795897 secs 2024-07-27T11:59:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:57 smithi077 ceph-mon[37153]: pgmap v10308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:59:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18619.795897 secs 2024-07-27T11:59:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:57 smithi047 ceph-mon[36835]: pgmap v10308: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T11:59:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18619.795897 secs 2024-07-27T11:59:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 11:59:59 smithi029 ceph-mon[32610]: pgmap v10309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 11:59:59 smithi077 ceph-mon[37153]: pgmap v10309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 11:59:59 smithi047 ceph-mon[36835]: pgmap v10309: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:01.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T12:00:01.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T12:00:01.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T12:00:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:01 smithi077 ceph-mon[37153]: pgmap v10310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:00:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:01 smithi047 ceph-mon[36835]: pgmap v10310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:00:02.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:01 smithi029 ceph-mon[32610]: pgmap v10310: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:00:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18624.796015 secs 2024-07-27T12:00:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18624.796015 secs 2024-07-27T12:00:03.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18624.796015 secs 2024-07-27T12:00:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:03 smithi077 ceph-mon[37153]: pgmap v10311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:03 smithi047 ceph-mon[36835]: pgmap v10311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:04.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:03 smithi029 ceph-mon[32610]: pgmap v10311: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:05 smithi077 ceph-mon[37153]: pgmap v10312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:05 smithi047 ceph-mon[36835]: pgmap v10312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:06.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:05 smithi029 ceph-mon[32610]: pgmap v10312: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:00:07.672+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:00:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:00:07.672+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:00:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:00:07.672+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:00:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:07 smithi029 ceph-mon[32610]: pgmap v10313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18629.796137 secs 2024-07-27T12:00:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:07 smithi077 ceph-mon[37153]: pgmap v10313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:08.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18629.796137 secs 2024-07-27T12:00:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:07 smithi047 ceph-mon[36835]: pgmap v10313: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18629.796137 secs 2024-07-27T12:00:10.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:09 smithi077 ceph-mon[37153]: pgmap v10314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:09 smithi047 ceph-mon[36835]: pgmap v10314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:10.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:09 smithi029 ceph-mon[32610]: pgmap v10314: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:11 smithi077 ceph-mon[37153]: pgmap v10315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:00:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:11 smithi047 ceph-mon[36835]: pgmap v10315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:00:12.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:11 smithi029 ceph-mon[32610]: pgmap v10315: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:00:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18634.796270 secs 2024-07-27T12:00:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18634.796270 secs 2024-07-27T12:00:13.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18634.796270 secs 2024-07-27T12:00:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:13 smithi077 ceph-mon[37153]: pgmap v10316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:14.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:13 smithi047 ceph-mon[36835]: pgmap v10316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:14.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:13 smithi029 ceph-mon[32610]: pgmap v10316: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:15 smithi077 ceph-mon[37153]: pgmap v10317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:15 smithi047 ceph-mon[36835]: pgmap v10317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:16.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:15 smithi029 ceph-mon[32610]: pgmap v10317: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:18.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:17 smithi029 ceph-mon[32610]: pgmap v10318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:18.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18639.796401 secs 2024-07-27T12:00:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:17 smithi077 ceph-mon[37153]: pgmap v10318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:18.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18639.796401 secs 2024-07-27T12:00:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:17 smithi047 ceph-mon[36835]: pgmap v10318: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18639.796401 secs 2024-07-27T12:00:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:19 smithi077 ceph-mon[37153]: pgmap v10319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:19 smithi047 ceph-mon[36835]: pgmap v10319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:20.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:19 smithi029 ceph-mon[32610]: pgmap v10319: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:21 smithi077 ceph-mon[37153]: pgmap v10320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:00:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:21 smithi047 ceph-mon[36835]: pgmap v10320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:00:22.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:21 smithi029 ceph-mon[32610]: pgmap v10320: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:00:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:00:22.688+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:00:22.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:00:22.688+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:00:22.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:00:22.688+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:00:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18644.796522 secs 2024-07-27T12:00:23.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18644.796522 secs 2024-07-27T12:00:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18644.796522 secs 2024-07-27T12:00:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:23 smithi077 ceph-mon[37153]: pgmap v10321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:23 smithi047 ceph-mon[36835]: pgmap v10321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:24.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:23 smithi029 ceph-mon[32610]: pgmap v10321: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:26.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:25 smithi077 ceph-mon[37153]: pgmap v10322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:25 smithi047 ceph-mon[36835]: pgmap v10322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:26.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:25 smithi029 ceph-mon[32610]: pgmap v10322: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:27 smithi077 ceph-mon[37153]: pgmap v10323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18649.796667 secs 2024-07-27T12:00:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:27 smithi047 ceph-mon[36835]: pgmap v10323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18649.796667 secs 2024-07-27T12:00:28.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:27 smithi029 ceph-mon[32610]: pgmap v10323: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18649.796667 secs 2024-07-27T12:00:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:29 smithi077 ceph-mon[37153]: pgmap v10324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:29 smithi047 ceph-mon[36835]: pgmap v10324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:30.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:29 smithi029 ceph-mon[32610]: pgmap v10324: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:31 smithi077 ceph-mon[37153]: pgmap v10325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:31 smithi047 ceph-mon[36835]: pgmap v10325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:32.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:31 smithi029 ceph-mon[32610]: pgmap v10325: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18654.796818 secs 2024-07-27T12:00:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18654.796818 secs 2024-07-27T12:00:33.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18654.796818 secs 2024-07-27T12:00:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:33 smithi077 ceph-mon[37153]: pgmap v10326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:33 smithi047 ceph-mon[36835]: pgmap v10326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:34.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:33 smithi029 ceph-mon[32610]: pgmap v10326: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:35 smithi077 ceph-mon[37153]: pgmap v10327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:35 smithi047 ceph-mon[36835]: pgmap v10327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:36.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:35 smithi029 ceph-mon[32610]: pgmap v10327: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:37.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:00:37.697+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:00:37.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:00:37.697+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:00:37.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:00:37.697+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:00:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:37 smithi029 ceph-mon[32610]: pgmap v10328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18659.796952 secs 2024-07-27T12:00:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:37 smithi077 ceph-mon[37153]: pgmap v10328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18659.796952 secs 2024-07-27T12:00:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:37 smithi047 ceph-mon[36835]: pgmap v10328: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18659.796952 secs 2024-07-27T12:00:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:39 smithi077 ceph-mon[37153]: pgmap v10329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:39 smithi047 ceph-mon[36835]: pgmap v10329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:39 smithi029 ceph-mon[32610]: pgmap v10329: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:41 smithi077 ceph-mon[37153]: pgmap v10330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:41 smithi047 ceph-mon[36835]: pgmap v10330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:42.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:41 smithi029 ceph-mon[32610]: pgmap v10330: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18664.797068 secs 2024-07-27T12:00:43.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18664.797068 secs 2024-07-27T12:00:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18664.797068 secs 2024-07-27T12:00:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:43 smithi077 ceph-mon[37153]: pgmap v10331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:43 smithi047 ceph-mon[36835]: pgmap v10331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:44.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:43 smithi029 ceph-mon[32610]: pgmap v10331: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:00:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:45 smithi077 ceph-mon[37153]: pgmap v10332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:45 smithi047 ceph-mon[36835]: pgmap v10332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:46.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:45 smithi029 ceph-mon[32610]: pgmap v10332: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:00:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:46 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:00:47.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:46 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:00:47.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:46 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:00:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:48 smithi077 ceph-mon[37153]: pgmap v10333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18669.797144 secs 2024-07-27T12:00:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:48 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:48 smithi047 ceph-mon[36835]: pgmap v10333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18669.797144 secs 2024-07-27T12:00:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:48 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:48 smithi029 ceph-mon[32610]: pgmap v10333: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:00:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18669.797144 secs 2024-07-27T12:00:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:48 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:00:49.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:00:49.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:49 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:00:49.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:00:49.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:49 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:49.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:00:49.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:00:49.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:49 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:00:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:50 smithi047 ceph-mon[36835]: pgmap v10334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:50 smithi047 ceph-mon[36835]: pgmap v10335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:00:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:50 smithi047 ceph-mon[36835]: pgmap v10336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:50 smithi029 ceph-mon[32610]: pgmap v10334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:50 smithi029 ceph-mon[32610]: pgmap v10335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:00:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:50 smithi029 ceph-mon[32610]: pgmap v10336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:50 smithi077 ceph-mon[37153]: pgmap v10334: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:50 smithi077 ceph-mon[37153]: pgmap v10335: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:00:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:50 smithi077 ceph-mon[37153]: pgmap v10336: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:00:52.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:52 smithi047 ceph-mon[36835]: pgmap v10337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T12:00:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:52 smithi029 ceph-mon[32610]: pgmap v10337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T12:00:52.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:00:52.712+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:00:52.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:00:52.712+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:00:52.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:00:52.712+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:00:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:52 smithi077 ceph-mon[37153]: pgmap v10337: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T12:00:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18674.797273 secs 2024-07-27T12:00:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18674.797273 secs 2024-07-27T12:00:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18674.797273 secs 2024-07-27T12:00:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:54 smithi047 ceph-mon[36835]: pgmap v10338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:00:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:54 smithi029 ceph-mon[32610]: pgmap v10338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:00:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:54 smithi077 ceph-mon[37153]: pgmap v10338: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:00:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:56 smithi047 ceph-mon[36835]: pgmap v10339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:00:56.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:56 smithi029 ceph-mon[32610]: pgmap v10339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:00:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:56 smithi077 ceph-mon[37153]: pgmap v10339: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:00:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18679.797396 secs 2024-07-27T12:00:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:00:58 smithi047 ceph-mon[36835]: pgmap v10340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:00:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18679.797396 secs 2024-07-27T12:00:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:00:58 smithi029 ceph-mon[32610]: pgmap v10340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:00:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18679.797396 secs 2024-07-27T12:00:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:00:58 smithi077 ceph-mon[37153]: pgmap v10340: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:01:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:00 smithi047 ceph-mon[36835]: pgmap v10341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:01:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:00 smithi029 ceph-mon[32610]: pgmap v10341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:01:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:00 smithi077 ceph-mon[37153]: pgmap v10341: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:01:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:02 smithi047 ceph-mon[36835]: pgmap v10342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:02.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:02 smithi029 ceph-mon[32610]: pgmap v10342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:02 smithi077 ceph-mon[37153]: pgmap v10342: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18684.797538 secs 2024-07-27T12:01:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18684.797538 secs 2024-07-27T12:01:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18684.797538 secs 2024-07-27T12:01:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:04 smithi047 ceph-mon[36835]: pgmap v10343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:04.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:04 smithi029 ceph-mon[32610]: pgmap v10343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:04 smithi077 ceph-mon[37153]: pgmap v10343: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:06 smithi047 ceph-mon[36835]: pgmap v10344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:06.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:06 smithi029 ceph-mon[32610]: pgmap v10344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:06 smithi077 ceph-mon[37153]: pgmap v10344: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:07.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:01:07.726+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:01:07.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:01:07.726+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:01:07.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:01:07.726+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:01:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18689.797690 secs 2024-07-27T12:01:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:08 smithi047 ceph-mon[36835]: pgmap v10345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18689.797690 secs 2024-07-27T12:01:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:08 smithi029 ceph-mon[32610]: pgmap v10345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18689.797690 secs 2024-07-27T12:01:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:08 smithi077 ceph-mon[37153]: pgmap v10345: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:10 smithi047 ceph-mon[36835]: pgmap v10346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:10.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:10 smithi029 ceph-mon[32610]: pgmap v10346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:10 smithi077 ceph-mon[37153]: pgmap v10346: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:12 smithi047 ceph-mon[36835]: pgmap v10347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:12.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:12 smithi029 ceph-mon[32610]: pgmap v10347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:12 smithi077 ceph-mon[37153]: pgmap v10347: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18694.797843 secs 2024-07-27T12:01:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18694.797843 secs 2024-07-27T12:01:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18694.797843 secs 2024-07-27T12:01:14.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:14 smithi029 ceph-mon[32610]: pgmap v10348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:14 smithi077 ceph-mon[37153]: pgmap v10348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:14 smithi047 ceph-mon[36835]: pgmap v10348: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:16 smithi029 ceph-mon[32610]: pgmap v10349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:16 smithi077 ceph-mon[37153]: pgmap v10349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:16 smithi047 ceph-mon[36835]: pgmap v10349: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18699.797969 secs 2024-07-27T12:01:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:18 smithi029 ceph-mon[32610]: pgmap v10350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:18.860 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18699.797969 secs 2024-07-27T12:01:18.861 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:18 smithi077 ceph-mon[37153]: pgmap v10350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18699.797969 secs 2024-07-27T12:01:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:18 smithi047 ceph-mon[36835]: pgmap v10350: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:20 smithi029 ceph-mon[32610]: pgmap v10351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:20 smithi077 ceph-mon[37153]: pgmap v10351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:20 smithi047 ceph-mon[36835]: pgmap v10351: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:21 smithi077 ceph-mon[37153]: pgmap v10352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:21 smithi047 ceph-mon[36835]: pgmap v10352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:21.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:21 smithi029 ceph-mon[32610]: pgmap v10352: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:22.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:01:22.742+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:01:22.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:01:22.742+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:01:22.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:01:22.742+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:01:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18704.798121 secs 2024-07-27T12:01:22.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18704.798121 secs 2024-07-27T12:01:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18704.798121 secs 2024-07-27T12:01:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:23 smithi047 ceph-mon[36835]: pgmap v10353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:23 smithi029 ceph-mon[32610]: pgmap v10353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:23 smithi077 ceph-mon[37153]: pgmap v10353: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:26 smithi047 ceph-mon[36835]: pgmap v10354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:26.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:26 smithi029 ceph-mon[32610]: pgmap v10354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:26 smithi077 ceph-mon[37153]: pgmap v10354: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18709.798239 secs 2024-07-27T12:01:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:28 smithi029 ceph-mon[32610]: pgmap v10355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18709.798239 secs 2024-07-27T12:01:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:28 smithi077 ceph-mon[37153]: pgmap v10355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18709.798239 secs 2024-07-27T12:01:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:28 smithi047 ceph-mon[36835]: pgmap v10355: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:30.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:30 smithi029 ceph-mon[32610]: pgmap v10356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:30 smithi077 ceph-mon[37153]: pgmap v10356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:30 smithi047 ceph-mon[36835]: pgmap v10356: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:32 smithi029 ceph-mon[32610]: pgmap v10357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:32.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:32 smithi077 ceph-mon[37153]: pgmap v10357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:32 smithi047 ceph-mon[36835]: pgmap v10357: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18714.798339 secs 2024-07-27T12:01:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18714.798339 secs 2024-07-27T12:01:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18714.798339 secs 2024-07-27T12:01:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:34 smithi029 ceph-mon[32610]: pgmap v10358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:34 smithi077 ceph-mon[37153]: pgmap v10358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:34 smithi047 ceph-mon[36835]: pgmap v10358: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:35 smithi077 ceph-mon[37153]: pgmap v10359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:35 smithi047 ceph-mon[36835]: pgmap v10359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:35.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:35 smithi029 ceph-mon[32610]: pgmap v10359: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:37.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:01:37.752+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:01:37.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:01:37.752+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:01:37.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:01:37.752+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:01:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18719.798468 secs 2024-07-27T12:01:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:38 smithi029 ceph-mon[32610]: pgmap v10360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18719.798468 secs 2024-07-27T12:01:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:38 smithi077 ceph-mon[37153]: pgmap v10360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18719.798468 secs 2024-07-27T12:01:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:38 smithi047 ceph-mon[36835]: pgmap v10360: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:40.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:40 smithi029 ceph-mon[32610]: pgmap v10361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:40 smithi077 ceph-mon[37153]: pgmap v10361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:40 smithi047 ceph-mon[36835]: pgmap v10361: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:42 smithi029 ceph-mon[32610]: pgmap v10362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:42 smithi077 ceph-mon[37153]: pgmap v10362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:42 smithi047 ceph-mon[36835]: pgmap v10362: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:01:43.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18724.798616 secs 2024-07-27T12:01:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18724.798616 secs 2024-07-27T12:01:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18724.798616 secs 2024-07-27T12:01:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:44 smithi077 ceph-mon[37153]: pgmap v10363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:44 smithi047 ceph-mon[36835]: pgmap v10363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:44.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:44 smithi029 ceph-mon[32610]: pgmap v10363: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:01:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:45 smithi077 ceph-mon[37153]: pgmap v10364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:45 smithi047 ceph-mon[36835]: pgmap v10364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:45.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:45 smithi029 ceph-mon[32610]: pgmap v10364: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:01:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18729.798722 secs 2024-07-27T12:01:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:48 smithi029 ceph-mon[32610]: pgmap v10365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:48.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18729.798722 secs 2024-07-27T12:01:48.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:48 smithi077 ceph-mon[37153]: pgmap v10365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18729.798722 secs 2024-07-27T12:01:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:48 smithi047 ceph-mon[36835]: pgmap v10365: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:01:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:50 smithi029 ceph-mon[32610]: pgmap v10366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:01:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:01:50.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:01:50.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:50 smithi077 ceph-mon[37153]: pgmap v10366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:01:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:01:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:01:50.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:50 smithi047 ceph-mon[36835]: pgmap v10366: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:01:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:01:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:01:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:01:50.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:01:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:51 smithi029 ceph-mon[32610]: pgmap v10367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:01:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:51 smithi029 ceph-mon[32610]: pgmap v10368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:01:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:51 smithi077 ceph-mon[37153]: pgmap v10367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:01:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:51 smithi077 ceph-mon[37153]: pgmap v10368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:01:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:51 smithi047 ceph-mon[36835]: pgmap v10367: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:01:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:51 smithi047 ceph-mon[36835]: pgmap v10368: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:01:52.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:01:52.768+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:01:52.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:01:52.768+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:01:52.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:01:52.768+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:01:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18734.798859 secs 2024-07-27T12:01:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:53 smithi029 ceph-mon[32610]: pgmap v10369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T12:01:53.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18734.798859 secs 2024-07-27T12:01:53.748 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:53 smithi047 ceph-mon[36835]: pgmap v10369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T12:01:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18734.798859 secs 2024-07-27T12:01:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:53 smithi077 ceph-mon[37153]: pgmap v10369: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-27T12:01:55.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:55 smithi029 ceph-mon[32610]: pgmap v10370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:01:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:55 smithi077 ceph-mon[37153]: pgmap v10370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:01:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:55 smithi047 ceph-mon[36835]: pgmap v10370: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:01:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:56 smithi077 ceph-mon[37153]: pgmap v10371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T12:01:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:56 smithi047 ceph-mon[36835]: pgmap v10371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T12:01:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:56 smithi029 ceph-mon[32610]: pgmap v10371: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T12:01:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18739.799007 secs 2024-07-27T12:01:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18739.799007 secs 2024-07-27T12:01:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18739.799007 secs 2024-07-27T12:01:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:01:58 smithi047 ceph-mon[36835]: pgmap v10372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:01:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:01:58 smithi029 ceph-mon[32610]: pgmap v10372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:01:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:01:58 smithi077 ceph-mon[37153]: pgmap v10372: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:02:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:00.462+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:02:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:00.462+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:02:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:00.462+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:02:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:00.462+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:02:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:01 smithi077 ceph-mon[37153]: pgmap v10373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:02:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:01 smithi047 ceph-mon[36835]: pgmap v10373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:02:01.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:01 smithi029 ceph-mon[32610]: pgmap v10373: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:02:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18744.799137 secs 2024-07-27T12:02:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:03 smithi077 ceph-mon[37153]: pgmap v10374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18744.799137 secs 2024-07-27T12:02:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:03 smithi047 ceph-mon[36835]: pgmap v10374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18744.799137 secs 2024-07-27T12:02:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:03 smithi029 ceph-mon[32610]: pgmap v10374: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:05 smithi077 ceph-mon[37153]: pgmap v10375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:05 smithi047 ceph-mon[36835]: pgmap v10375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:05.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:05 smithi029 ceph-mon[32610]: pgmap v10375: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:07 smithi077 ceph-mon[37153]: pgmap v10376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:07 smithi047 ceph-mon[36835]: pgmap v10376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:07 smithi029 ceph-mon[32610]: pgmap v10376: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:07.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:07.770+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:02:07.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:07.770+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:02:07.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:07.770+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:02:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18749.799262 secs 2024-07-27T12:02:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18749.799262 secs 2024-07-27T12:02:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18749.799262 secs 2024-07-27T12:02:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:09 smithi077 ceph-mon[37153]: pgmap v10377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:09 smithi047 ceph-mon[36835]: pgmap v10377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:09 smithi029 ceph-mon[32610]: pgmap v10377: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:11 smithi077 ceph-mon[37153]: pgmap v10378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:11 smithi047 ceph-mon[36835]: pgmap v10378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:11.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:11 smithi029 ceph-mon[32610]: pgmap v10378: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18754.799389 secs 2024-07-27T12:02:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:13 smithi077 ceph-mon[37153]: pgmap v10379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18754.799389 secs 2024-07-27T12:02:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:13 smithi047 ceph-mon[36835]: pgmap v10379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:13.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18754.799389 secs 2024-07-27T12:02:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:13 smithi029 ceph-mon[32610]: pgmap v10379: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:15 smithi077 ceph-mon[37153]: pgmap v10380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:15 smithi047 ceph-mon[36835]: pgmap v10380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:15.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:15 smithi029 ceph-mon[32610]: pgmap v10380: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:17 smithi077 ceph-mon[37153]: pgmap v10381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:17 smithi047 ceph-mon[36835]: pgmap v10381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:17.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:17 smithi029 ceph-mon[32610]: pgmap v10381: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18759.799537 secs 2024-07-27T12:02:18.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18759.799537 secs 2024-07-27T12:02:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18759.799537 secs 2024-07-27T12:02:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:19 smithi077 ceph-mon[37153]: pgmap v10382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:19 smithi047 ceph-mon[36835]: pgmap v10382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:19.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:19 smithi029 ceph-mon[32610]: pgmap v10382: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:21 smithi077 ceph-mon[37153]: pgmap v10383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:21 smithi047 ceph-mon[36835]: pgmap v10383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:21 smithi029 ceph-mon[32610]: pgmap v10383: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:22.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:22.781+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:02:22.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:22.781+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:02:22.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:22.781+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:02:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18764.799639 secs 2024-07-27T12:02:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:23 smithi077 ceph-mon[37153]: pgmap v10384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:23.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18764.799639 secs 2024-07-27T12:02:23.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:23 smithi047 ceph-mon[36835]: pgmap v10384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18764.799639 secs 2024-07-27T12:02:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:23 smithi029 ceph-mon[32610]: pgmap v10384: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:25 smithi077 ceph-mon[37153]: pgmap v10385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:25 smithi047 ceph-mon[36835]: pgmap v10385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:25.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:25 smithi029 ceph-mon[32610]: pgmap v10385: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:27 smithi077 ceph-mon[37153]: pgmap v10386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:27 smithi047 ceph-mon[36835]: pgmap v10386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:27 smithi029 ceph-mon[32610]: pgmap v10386: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18769.799773 secs 2024-07-27T12:02:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18769.799773 secs 2024-07-27T12:02:28.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18769.799773 secs 2024-07-27T12:02:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:29 smithi077 ceph-mon[37153]: pgmap v10387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:29 smithi047 ceph-mon[36835]: pgmap v10387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:29.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:29 smithi029 ceph-mon[32610]: pgmap v10387: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:31 smithi077 ceph-mon[37153]: pgmap v10388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:31 smithi047 ceph-mon[36835]: pgmap v10388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:31.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:31 smithi029 ceph-mon[32610]: pgmap v10388: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18774.799849 secs 2024-07-27T12:02:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:33 smithi077 ceph-mon[37153]: pgmap v10389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18774.799849 secs 2024-07-27T12:02:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:33 smithi047 ceph-mon[36835]: pgmap v10389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:33.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18774.799849 secs 2024-07-27T12:02:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:33 smithi029 ceph-mon[32610]: pgmap v10389: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:35 smithi077 ceph-mon[37153]: pgmap v10390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:35 smithi047 ceph-mon[36835]: pgmap v10390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:35 smithi029 ceph-mon[32610]: pgmap v10390: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:37 smithi047 ceph-mon[36835]: pgmap v10391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:37 smithi029 ceph-mon[32610]: pgmap v10391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:37.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:37.796+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:02:37.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:37.796+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:02:37.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:37.796+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:02:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:37 smithi077 ceph-mon[37153]: pgmap v10391: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18779.799984 secs 2024-07-27T12:02:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18779.799984 secs 2024-07-27T12:02:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18779.799984 secs 2024-07-27T12:02:39.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:39 smithi047 ceph-mon[36835]: pgmap v10392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:39.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:39 smithi029 ceph-mon[32610]: pgmap v10392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:39 smithi077 ceph-mon[37153]: pgmap v10392: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:41 smithi047 ceph-mon[36835]: pgmap v10393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:41.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:41 smithi029 ceph-mon[32610]: pgmap v10393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:41.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:41 smithi077 ceph-mon[37153]: pgmap v10393: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18784.800111 secs 2024-07-27T12:02:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:43 smithi047 ceph-mon[36835]: pgmap v10394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18784.800111 secs 2024-07-27T12:02:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:43 smithi029 ceph-mon[32610]: pgmap v10394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18784.800111 secs 2024-07-27T12:02:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:43 smithi077 ceph-mon[37153]: pgmap v10394: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:45 smithi047 ceph-mon[36835]: pgmap v10395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:45 smithi029 ceph-mon[32610]: pgmap v10395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:45 smithi077 ceph-mon[37153]: pgmap v10395: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:02:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:47 smithi047 ceph-mon[36835]: pgmap v10396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:47 smithi029 ceph-mon[32610]: pgmap v10396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:47 smithi077 ceph-mon[37153]: pgmap v10396: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:02:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18789.800201 secs 2024-07-27T12:02:48.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18789.800201 secs 2024-07-27T12:02:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18789.800201 secs 2024-07-27T12:02:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:49 smithi047 ceph-mon[36835]: pgmap v10397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:49.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:49 smithi029 ceph-mon[32610]: pgmap v10397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:49 smithi077 ceph-mon[37153]: pgmap v10397: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:02:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:50 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:02:50.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:50 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:02:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:50 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:02:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:51 smithi047 ceph-mon[36835]: pgmap v10398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:02:51.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:02:51.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:02:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:51 smithi029 ceph-mon[32610]: pgmap v10398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:02:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:02:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:02:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:51 smithi077 ceph-mon[37153]: pgmap v10398: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:02:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:02:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:02:51.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:02:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:52 smithi047 ceph-mon[36835]: pgmap v10399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T12:02:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:52 smithi047 ceph-mon[36835]: pgmap v10400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T12:02:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:52 smithi029 ceph-mon[32610]: pgmap v10399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T12:02:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:52 smithi029 ceph-mon[32610]: pgmap v10400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T12:02:52.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:52.798+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:02:52.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:52.798+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:02:52.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:02:52.798+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:02:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:52 smithi077 ceph-mon[37153]: pgmap v10399: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T12:02:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:52 smithi077 ceph-mon[37153]: pgmap v10400: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T12:02:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18794.800296 secs 2024-07-27T12:02:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18794.800296 secs 2024-07-27T12:02:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18794.800296 secs 2024-07-27T12:02:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:54 smithi047 ceph-mon[36835]: pgmap v10401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:02:54.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:54 smithi029 ceph-mon[32610]: pgmap v10401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:02:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:54 smithi077 ceph-mon[37153]: pgmap v10401: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:02:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:56 smithi047 ceph-mon[36835]: pgmap v10402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:02:56.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:56 smithi029 ceph-mon[32610]: pgmap v10402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:02:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:56 smithi077 ceph-mon[37153]: pgmap v10402: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:02:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:58 smithi047 ceph-mon[36835]: pgmap v10403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:02:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:02:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18799.800433 secs 2024-07-27T12:02:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:58 smithi029 ceph-mon[32610]: pgmap v10403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:02:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:02:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18799.800433 secs 2024-07-27T12:02:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:58 smithi077 ceph-mon[37153]: pgmap v10403: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:02:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:02:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18799.800433 secs 2024-07-27T12:03:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:00 smithi047 ceph-mon[36835]: pgmap v10404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:03:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:00 smithi029 ceph-mon[32610]: pgmap v10404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:03:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:00 smithi077 ceph-mon[37153]: pgmap v10404: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:03:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:02 smithi047 ceph-mon[36835]: pgmap v10405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:03:02.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:02 smithi029 ceph-mon[32610]: pgmap v10405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:03:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:02 smithi077 ceph-mon[37153]: pgmap v10405: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:03:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18804.800560 secs 2024-07-27T12:03:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18804.800560 secs 2024-07-27T12:03:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18804.800560 secs 2024-07-27T12:03:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:04 smithi047 ceph-mon[36835]: pgmap v10406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:04 smithi029 ceph-mon[32610]: pgmap v10406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:04 smithi077 ceph-mon[37153]: pgmap v10406: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:06 smithi047 ceph-mon[36835]: pgmap v10407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:06.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:06 smithi029 ceph-mon[32610]: pgmap v10407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:06 smithi077 ceph-mon[37153]: pgmap v10407: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:07.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:07.812+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:03:07.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:07.812+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:03:07.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:07.812+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:03:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:08 smithi047 ceph-mon[36835]: pgmap v10408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18809.800704 secs 2024-07-27T12:03:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:08 smithi029 ceph-mon[32610]: pgmap v10408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18809.800704 secs 2024-07-27T12:03:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:08 smithi077 ceph-mon[37153]: pgmap v10408: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18809.800704 secs 2024-07-27T12:03:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:10 smithi047 ceph-mon[36835]: pgmap v10409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:10.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:10 smithi029 ceph-mon[32610]: pgmap v10409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:10 smithi077 ceph-mon[37153]: pgmap v10409: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:12 smithi047 ceph-mon[36835]: pgmap v10410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:12 smithi029 ceph-mon[32610]: pgmap v10410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:12 smithi077 ceph-mon[37153]: pgmap v10410: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18814.800831 secs 2024-07-27T12:03:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18814.800831 secs 2024-07-27T12:03:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18814.800831 secs 2024-07-27T12:03:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:14 smithi047 ceph-mon[36835]: pgmap v10411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:14.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:14 smithi029 ceph-mon[32610]: pgmap v10411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:14 smithi077 ceph-mon[37153]: pgmap v10411: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:16.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:16 smithi029 ceph-mon[32610]: pgmap v10412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:16 smithi077 ceph-mon[37153]: pgmap v10412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:16 smithi047 ceph-mon[36835]: pgmap v10412: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:18 smithi029 ceph-mon[32610]: pgmap v10413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18819.800901 secs 2024-07-27T12:03:18.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:18 smithi077 ceph-mon[37153]: pgmap v10413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:18.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18819.800901 secs 2024-07-27T12:03:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:18 smithi047 ceph-mon[36835]: pgmap v10413: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18819.800901 secs 2024-07-27T12:03:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:20 smithi029 ceph-mon[32610]: pgmap v10414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:20 smithi077 ceph-mon[37153]: pgmap v10414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:20 smithi047 ceph-mon[36835]: pgmap v10414: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:22.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:22 smithi029 ceph-mon[32610]: pgmap v10415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:22.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:22.828+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:03:22.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:22.828+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:03:22.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:22.828+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:03:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:22 smithi077 ceph-mon[37153]: pgmap v10415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:22 smithi047 ceph-mon[36835]: pgmap v10415: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:23.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18824.800988 secs 2024-07-27T12:03:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18824.800988 secs 2024-07-27T12:03:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18824.800988 secs 2024-07-27T12:03:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:24 smithi077 ceph-mon[37153]: pgmap v10416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:24 smithi047 ceph-mon[36835]: pgmap v10416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:24.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:24 smithi029 ceph-mon[32610]: pgmap v10416: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:26.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:26 smithi077 ceph-mon[37153]: pgmap v10417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:26 smithi047 ceph-mon[36835]: pgmap v10417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:26.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:26 smithi029 ceph-mon[32610]: pgmap v10417: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:28 smithi077 ceph-mon[37153]: pgmap v10418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18829.801110 secs 2024-07-27T12:03:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:28 smithi047 ceph-mon[36835]: pgmap v10418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:28.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18829.801110 secs 2024-07-27T12:03:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:28 smithi029 ceph-mon[32610]: pgmap v10418: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18829.801110 secs 2024-07-27T12:03:30.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:30.474+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:03:30.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:30.474+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:03:30.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:30.475+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:03:30.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:30.475+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:03:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:30 smithi077 ceph-mon[37153]: pgmap v10419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:30 smithi047 ceph-mon[36835]: pgmap v10419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:30.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:30 smithi029 ceph-mon[32610]: pgmap v10419: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:32 smithi077 ceph-mon[37153]: pgmap v10420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:32 smithi047 ceph-mon[36835]: pgmap v10420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:32 smithi029 ceph-mon[32610]: pgmap v10420: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18834.801266 secs 2024-07-27T12:03:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18834.801266 secs 2024-07-27T12:03:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18834.801266 secs 2024-07-27T12:03:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:34 smithi077 ceph-mon[37153]: pgmap v10421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:34 smithi047 ceph-mon[36835]: pgmap v10421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:34.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:34 smithi029 ceph-mon[32610]: pgmap v10421: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:35 smithi077 ceph-mon[37153]: pgmap v10422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:35 smithi047 ceph-mon[36835]: pgmap v10422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:35.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:35 smithi029 ceph-mon[32610]: pgmap v10422: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:37.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:37.844+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:03:37.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:37.844+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:03:37.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:37.844+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:03:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:37 smithi077 ceph-mon[37153]: pgmap v10423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18839.801390 secs 2024-07-27T12:03:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:37 smithi047 ceph-mon[36835]: pgmap v10423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18839.801390 secs 2024-07-27T12:03:38.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:37 smithi029 ceph-mon[32610]: pgmap v10423: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18839.801390 secs 2024-07-27T12:03:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:39 smithi047 ceph-mon[36835]: pgmap v10424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:39 smithi029 ceph-mon[32610]: pgmap v10424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:39 smithi077 ceph-mon[37153]: pgmap v10424: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:41 smithi047 ceph-mon[36835]: pgmap v10425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:41 smithi029 ceph-mon[32610]: pgmap v10425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:41 smithi077 ceph-mon[37153]: pgmap v10425: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18844.801514 secs 2024-07-27T12:03:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18844.801514 secs 2024-07-27T12:03:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18844.801514 secs 2024-07-27T12:03:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:43 smithi047 ceph-mon[36835]: pgmap v10426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:44.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:43 smithi029 ceph-mon[32610]: pgmap v10426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:43 smithi077 ceph-mon[37153]: pgmap v10426: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:03:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:45 smithi047 ceph-mon[36835]: pgmap v10427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:46.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:45 smithi029 ceph-mon[32610]: pgmap v10427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:45 smithi077 ceph-mon[37153]: pgmap v10427: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:03:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:47 smithi047 ceph-mon[36835]: pgmap v10428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18849.801637 secs 2024-07-27T12:03:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:47 smithi029 ceph-mon[32610]: pgmap v10428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18849.801637 secs 2024-07-27T12:03:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:47 smithi077 ceph-mon[37153]: pgmap v10428: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:03:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18849.801637 secs 2024-07-27T12:03:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:49 smithi047 ceph-mon[36835]: pgmap v10429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:49 smithi029 ceph-mon[32610]: pgmap v10429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:49 smithi077 ceph-mon[37153]: pgmap v10429: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:03:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:51 smithi047 ceph-mon[36835]: pgmap v10430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:03:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:03:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:03:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:51 smithi047 ceph-mon[36835]: pgmap v10431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T12:03:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:51 smithi047 ceph-mon[36835]: pgmap v10432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:03:52.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:03:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:51 smithi029 ceph-mon[32610]: pgmap v10430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:03:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:03:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:03:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:51 smithi029 ceph-mon[32610]: pgmap v10431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T12:03:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:51 smithi029 ceph-mon[32610]: pgmap v10432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:03:52.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:03:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:51 smithi077 ceph-mon[37153]: pgmap v10430: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:03:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:03:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:03:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:03:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:51 smithi077 ceph-mon[37153]: pgmap v10431: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-27T12:03:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:51 smithi077 ceph-mon[37153]: pgmap v10432: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:03:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:03:52.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:52.859+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:03:52.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:52.859+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:03:52.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:03:52.859+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:03:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18854.801731 secs 2024-07-27T12:03:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18854.801731 secs 2024-07-27T12:03:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18854.801731 secs 2024-07-27T12:03:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:53 smithi047 ceph-mon[36835]: pgmap v10433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:03:54.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:53 smithi029 ceph-mon[32610]: pgmap v10433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:03:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:53 smithi077 ceph-mon[37153]: pgmap v10433: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:03:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:56 smithi047 ceph-mon[36835]: pgmap v10434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:03:56.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:56 smithi029 ceph-mon[32610]: pgmap v10434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:03:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:56 smithi077 ceph-mon[37153]: pgmap v10434: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:03:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18859.801857 secs 2024-07-27T12:03:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:03:58 smithi047 ceph-mon[36835]: pgmap v10435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T12:03:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18859.801857 secs 2024-07-27T12:03:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:03:58 smithi029 ceph-mon[32610]: pgmap v10435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T12:03:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18859.801857 secs 2024-07-27T12:03:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:03:58 smithi077 ceph-mon[37153]: pgmap v10435: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T12:04:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:00 smithi047 ceph-mon[36835]: pgmap v10436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:04:00.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:00 smithi029 ceph-mon[32610]: pgmap v10436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:04:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:00 smithi077 ceph-mon[37153]: pgmap v10436: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:04:02.658 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:02 smithi077 ceph-mon[37153]: pgmap v10437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:04:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:02 smithi047 ceph-mon[36835]: pgmap v10437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:04:02.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:02 smithi029 ceph-mon[32610]: pgmap v10437: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:04:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18864.802005 secs 2024-07-27T12:04:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18864.802005 secs 2024-07-27T12:04:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18864.802005 secs 2024-07-27T12:04:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:04 smithi047 ceph-mon[36835]: pgmap v10438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:04.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:04 smithi029 ceph-mon[32610]: pgmap v10438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:04 smithi077 ceph-mon[37153]: pgmap v10438: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:06 smithi047 ceph-mon[36835]: pgmap v10439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:06.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:06 smithi029 ceph-mon[32610]: pgmap v10439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:06 smithi077 ceph-mon[37153]: pgmap v10439: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:07.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:07.874+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:04:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:07.874+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:04:07.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:07.874+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:04:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18869.802083 secs 2024-07-27T12:04:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:08 smithi047 ceph-mon[36835]: pgmap v10440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18869.802083 secs 2024-07-27T12:04:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:08 smithi029 ceph-mon[32610]: pgmap v10440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18869.802083 secs 2024-07-27T12:04:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:08 smithi077 ceph-mon[37153]: pgmap v10440: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:10 smithi047 ceph-mon[36835]: pgmap v10441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:10.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:10 smithi029 ceph-mon[32610]: pgmap v10441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:10 smithi077 ceph-mon[37153]: pgmap v10441: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:12 smithi047 ceph-mon[36835]: pgmap v10442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:12.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:12 smithi029 ceph-mon[32610]: pgmap v10442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:12 smithi077 ceph-mon[37153]: pgmap v10442: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18874.802185 secs 2024-07-27T12:04:13.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18874.802185 secs 2024-07-27T12:04:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18874.802185 secs 2024-07-27T12:04:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:14 smithi047 ceph-mon[36835]: pgmap v10443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:14.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:14 smithi029 ceph-mon[32610]: pgmap v10443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:14 smithi077 ceph-mon[37153]: pgmap v10443: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:16 smithi047 ceph-mon[36835]: pgmap v10444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:16.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:16 smithi029 ceph-mon[32610]: pgmap v10444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:16 smithi077 ceph-mon[37153]: pgmap v10444: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18879.802310 secs 2024-07-27T12:04:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:18 smithi047 ceph-mon[36835]: pgmap v10445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:18.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18879.802310 secs 2024-07-27T12:04:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:18 smithi029 ceph-mon[32610]: pgmap v10445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18879.802310 secs 2024-07-27T12:04:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:18 smithi077 ceph-mon[37153]: pgmap v10445: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:20 smithi047 ceph-mon[36835]: pgmap v10446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:20 smithi029 ceph-mon[32610]: pgmap v10446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:20 smithi077 ceph-mon[37153]: pgmap v10446: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:22.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:22 smithi047 ceph-mon[36835]: pgmap v10447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:22.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:22 smithi029 ceph-mon[32610]: pgmap v10447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:22 smithi077 ceph-mon[37153]: pgmap v10447: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:22.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:22.877+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:04:22.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:22.877+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:04:22.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:22.877+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:04:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18884.802449 secs 2024-07-27T12:04:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18884.802449 secs 2024-07-27T12:04:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18884.802449 secs 2024-07-27T12:04:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:24 smithi047 ceph-mon[36835]: pgmap v10448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:24.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:24 smithi029 ceph-mon[32610]: pgmap v10448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:24 smithi077 ceph-mon[37153]: pgmap v10448: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:26 smithi047 ceph-mon[36835]: pgmap v10449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:26.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:26 smithi029 ceph-mon[32610]: pgmap v10449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:26 smithi077 ceph-mon[37153]: pgmap v10449: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18889.802567 secs 2024-07-27T12:04:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:28 smithi047 ceph-mon[36835]: pgmap v10450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18889.802567 secs 2024-07-27T12:04:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:28 smithi029 ceph-mon[32610]: pgmap v10450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18889.802567 secs 2024-07-27T12:04:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:28 smithi077 ceph-mon[37153]: pgmap v10450: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:30 smithi047 ceph-mon[36835]: pgmap v10451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:30.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:30 smithi029 ceph-mon[32610]: pgmap v10451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:30 smithi077 ceph-mon[37153]: pgmap v10451: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:32 smithi047 ceph-mon[36835]: pgmap v10452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:32.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:32 smithi029 ceph-mon[32610]: pgmap v10452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:32 smithi077 ceph-mon[37153]: pgmap v10452: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18894.802708 secs 2024-07-27T12:04:33.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18894.802708 secs 2024-07-27T12:04:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18894.802708 secs 2024-07-27T12:04:34.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:34 smithi029 ceph-mon[32610]: pgmap v10453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:34 smithi077 ceph-mon[37153]: pgmap v10453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:34 smithi047 ceph-mon[36835]: pgmap v10453: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:36.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:36 smithi029 ceph-mon[32610]: pgmap v10454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:36 smithi077 ceph-mon[37153]: pgmap v10454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:36 smithi047 ceph-mon[36835]: pgmap v10454: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:37.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:37.892+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:04:37.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:37.892+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:04:37.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:37.892+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:04:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18899.802829 secs 2024-07-27T12:04:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:38 smithi029 ceph-mon[32610]: pgmap v10455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18899.802829 secs 2024-07-27T12:04:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:38 smithi077 ceph-mon[37153]: pgmap v10455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18899.802829 secs 2024-07-27T12:04:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:38 smithi047 ceph-mon[36835]: pgmap v10455: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:40.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:40 smithi029 ceph-mon[32610]: pgmap v10456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:40 smithi077 ceph-mon[37153]: pgmap v10456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:40 smithi047 ceph-mon[36835]: pgmap v10456: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:42.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:42 smithi029 ceph-mon[32610]: pgmap v10457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:42 smithi077 ceph-mon[37153]: pgmap v10457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:42 smithi047 ceph-mon[36835]: pgmap v10457: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18904.802944 secs 2024-07-27T12:04:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18904.802944 secs 2024-07-27T12:04:43.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18904.802944 secs 2024-07-27T12:04:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:44 smithi077 ceph-mon[37153]: pgmap v10458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:44 smithi047 ceph-mon[36835]: pgmap v10458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:44.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:44 smithi029 ceph-mon[32610]: pgmap v10458: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:04:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:46 smithi077 ceph-mon[37153]: pgmap v10459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:46 smithi047 ceph-mon[36835]: pgmap v10459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:46.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:46 smithi029 ceph-mon[32610]: pgmap v10459: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:04:48.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18909.803067 secs 2024-07-27T12:04:48.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:48 smithi029 ceph-mon[32610]: pgmap v10460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18909.803067 secs 2024-07-27T12:04:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:48 smithi077 ceph-mon[37153]: pgmap v10460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18909.803067 secs 2024-07-27T12:04:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:48 smithi047 ceph-mon[36835]: pgmap v10460: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:04:50.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:50 smithi077 ceph-mon[37153]: pgmap v10461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:50 smithi047 ceph-mon[36835]: pgmap v10461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:50 smithi029 ceph-mon[32610]: pgmap v10461: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:04:51.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:51 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:04:51.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:51 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:04:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:51 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:04:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:52 smithi077 ceph-mon[37153]: pgmap v10462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:04:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:04:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:52 smithi077 ceph-mon[37153]: pgmap v10463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:04:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:52 smithi077 ceph-mon[37153]: pgmap v10464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:04:52.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:04:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:52.893+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:04:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:52.893+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:04:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:52.893+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:04:52.894 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:52 smithi029 ceph-mon[32610]: pgmap v10462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:52.895 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:04:52.895 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:04:52.895 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:52 smithi029 ceph-mon[32610]: pgmap v10463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:04:52.895 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:52 smithi029 ceph-mon[32610]: pgmap v10464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:04:52.895 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:04:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:52 smithi047 ceph-mon[36835]: pgmap v10462: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:04:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:04:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:04:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:52 smithi047 ceph-mon[36835]: pgmap v10463: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:04:52.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:52 smithi047 ceph-mon[36835]: pgmap v10464: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:04:52.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:04:53.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18914.803192 secs 2024-07-27T12:04:53.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18914.803192 secs 2024-07-27T12:04:53.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18914.803192 secs 2024-07-27T12:04:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:54 smithi077 ceph-mon[37153]: pgmap v10465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:04:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:54 smithi047 ceph-mon[36835]: pgmap v10465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:04:54.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:54 smithi029 ceph-mon[32610]: pgmap v10465: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:04:57.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:56 smithi029 ceph-mon[32610]: pgmap v10466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T12:04:57.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:56 smithi077 ceph-mon[37153]: pgmap v10466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T12:04:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:56 smithi047 ceph-mon[36835]: pgmap v10466: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-27T12:04:57.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:57.672+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 msgr2=0x7eff3c0b5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T12:04:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:57.672+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 msgr2=0x7eff3c0b5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T12:04:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:57.672+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7eff40004480 tx=0x7eff40043050 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T12:04:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:57.672+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7eff40004480 tx=0x7eff40043050 comp rx=0 tx=0).stop 2024-07-27T12:04:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:57.672+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 msgr2=0x7eff3c0b5ed0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T12:04:57.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:57.672+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 msgr2=0x7eff3c0b5ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T12:04:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:57.672+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T12:04:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:57.672+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0be2c0 0x7eff3c0c2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T12:04:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:57.672+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0be2c0 0x7eff3c0c2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T12:04:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:04:57.673+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0be2c0 0x7eff3c0c2820 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7eff400047b0 tx=0x7eff4002b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T12:04:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18919.803315 secs 2024-07-27T12:04:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18919.803315 secs 2024-07-27T12:04:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18919.803315 secs 2024-07-27T12:04:59.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:04:58 smithi077 ceph-mon[37153]: pgmap v10467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:04:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:04:58 smithi047 ceph-mon[36835]: pgmap v10467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:04:59.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:04:58 smithi029 ceph-mon[32610]: pgmap v10467: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:05:00.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:00.487+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:05:00.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:00.487+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:05:00.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:00.487+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:05:00.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:00.487+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:05:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:00 smithi077 ceph-mon[37153]: pgmap v10468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:05:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:00 smithi047 ceph-mon[36835]: pgmap v10468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:05:01.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:00 smithi029 ceph-mon[32610]: pgmap v10468: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:05:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:02 smithi077 ceph-mon[37153]: pgmap v10469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:05:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18924.803419 secs 2024-07-27T12:05:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:02 smithi047 ceph-mon[36835]: pgmap v10469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:05:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18924.803419 secs 2024-07-27T12:05:03.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:02 smithi029 ceph-mon[32610]: pgmap v10469: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:05:03.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18924.803419 secs 2024-07-27T12:05:05.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:04 smithi077 ceph-mon[37153]: pgmap v10470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:05:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:04 smithi047 ceph-mon[36835]: pgmap v10470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:05:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:04 smithi029 ceph-mon[32610]: pgmap v10470: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:05:07.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:06 smithi029 ceph-mon[32610]: pgmap v10471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:05:07.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:06 smithi077 ceph-mon[37153]: pgmap v10471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:05:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:06 smithi047 ceph-mon[36835]: pgmap v10471: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:05:07.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:07.894+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:05:07.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:07.894+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:05:07.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:07.894+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:05:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18929.803552 secs 2024-07-27T12:05:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18929.803552 secs 2024-07-27T12:05:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18929.803552 secs 2024-07-27T12:05:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:08 smithi077 ceph-mon[37153]: pgmap v10472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:05:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:08 smithi047 ceph-mon[36835]: pgmap v10472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:05:09.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:08 smithi029 ceph-mon[32610]: pgmap v10472: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:05:11.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:10 smithi077 ceph-mon[37153]: pgmap v10473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:05:11.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:10 smithi047 ceph-mon[36835]: pgmap v10473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:05:11.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:10 smithi029 ceph-mon[32610]: pgmap v10473: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:05:13.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:12 smithi077 ceph-mon[37153]: pgmap v10474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:13.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18934.803698 secs 2024-07-27T12:05:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:12 smithi047 ceph-mon[36835]: pgmap v10474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18934.803698 secs 2024-07-27T12:05:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:12 smithi029 ceph-mon[32610]: pgmap v10474: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18934.803698 secs 2024-07-27T12:05:15.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:14 smithi077 ceph-mon[37153]: pgmap v10475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:05:15.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:14 smithi047 ceph-mon[36835]: pgmap v10475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:05:15.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:14 smithi029 ceph-mon[32610]: pgmap v10475: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:05:17.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:16 smithi077 ceph-mon[37153]: pgmap v10476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:05:17.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:16 smithi047 ceph-mon[36835]: pgmap v10476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:05:17.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:16 smithi029 ceph-mon[32610]: pgmap v10476: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:05:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18939.803828 secs 2024-07-27T12:05:18.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18939.803828 secs 2024-07-27T12:05:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18939.803828 secs 2024-07-27T12:05:19.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:19 smithi077 ceph-mon[37153]: pgmap v10477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:05:19.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:19 smithi047 ceph-mon[36835]: pgmap v10477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:05:19.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:19 smithi029 ceph-mon[32610]: pgmap v10477: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:05:21.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:21 smithi077 ceph-mon[37153]: pgmap v10478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:05:21.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:21 smithi047 ceph-mon[36835]: pgmap v10478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:05:21.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:21 smithi029 ceph-mon[32610]: pgmap v10478: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:05:22.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:22.909+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:05:22.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:22.909+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:05:22.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:22.909+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:05:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:23 smithi029 ceph-mon[32610]: pgmap v10479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18944.803965 secs 2024-07-27T12:05:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:23 smithi077 ceph-mon[37153]: pgmap v10479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18944.803965 secs 2024-07-27T12:05:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:23 smithi047 ceph-mon[36835]: pgmap v10479: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18944.803965 secs 2024-07-27T12:05:25.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:25 smithi077 ceph-mon[37153]: pgmap v10480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:05:25.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:25 smithi047 ceph-mon[36835]: pgmap v10480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:05:25.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:25 smithi029 ceph-mon[32610]: pgmap v10480: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:05:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:27 smithi077 ceph-mon[37153]: pgmap v10481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T12:05:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:27 smithi047 ceph-mon[36835]: pgmap v10481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T12:05:27.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:27 smithi029 ceph-mon[32610]: pgmap v10481: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T12:05:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18949.804059 secs 2024-07-27T12:05:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18949.804059 secs 2024-07-27T12:05:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18949.804059 secs 2024-07-27T12:05:29.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:29 smithi077 ceph-mon[37153]: pgmap v10482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:05:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:29 smithi047 ceph-mon[36835]: pgmap v10482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:05:29.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:29 smithi029 ceph-mon[32610]: pgmap v10482: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:05:31.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:31 smithi077 ceph-mon[37153]: pgmap v10483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:05:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:31 smithi047 ceph-mon[36835]: pgmap v10483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:05:31.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:31 smithi029 ceph-mon[32610]: pgmap v10483: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:05:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:33 smithi077 ceph-mon[37153]: pgmap v10484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:33.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18954.804193 secs 2024-07-27T12:05:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:33 smithi047 ceph-mon[36835]: pgmap v10484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18954.804193 secs 2024-07-27T12:05:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:33 smithi029 ceph-mon[32610]: pgmap v10484: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18954.804193 secs 2024-07-27T12:05:35.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:35 smithi077 ceph-mon[37153]: pgmap v10485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:05:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:35 smithi047 ceph-mon[36835]: pgmap v10485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:05:35.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:35 smithi029 ceph-mon[32610]: pgmap v10485: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:05:37.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:37 smithi077 ceph-mon[37153]: pgmap v10486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:05:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:37 smithi047 ceph-mon[36835]: pgmap v10486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:05:37.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:37 smithi029 ceph-mon[32610]: pgmap v10486: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:05:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:37.916+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:05:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:37.916+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:05:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:37.916+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:05:38.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18959.804257 secs 2024-07-27T12:05:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18959.804257 secs 2024-07-27T12:05:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18959.804257 secs 2024-07-27T12:05:39.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:39 smithi077 ceph-mon[37153]: pgmap v10487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:05:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:39 smithi047 ceph-mon[36835]: pgmap v10487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:05:39.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:39 smithi029 ceph-mon[32610]: pgmap v10487: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:05:41.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:41 smithi077 ceph-mon[37153]: pgmap v10488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:05:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:41 smithi047 ceph-mon[36835]: pgmap v10488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:05:41.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:41 smithi029 ceph-mon[32610]: pgmap v10488: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:05:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:43 smithi077 ceph-mon[37153]: pgmap v10489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:43.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18964.804341 secs 2024-07-27T12:05:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:43 smithi047 ceph-mon[36835]: pgmap v10489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18964.804341 secs 2024-07-27T12:05:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:43 smithi029 ceph-mon[32610]: pgmap v10489: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18964.804341 secs 2024-07-27T12:05:45.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:45 smithi077 ceph-mon[37153]: pgmap v10490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:05:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:45 smithi047 ceph-mon[36835]: pgmap v10490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:05:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:45 smithi029 ceph-mon[32610]: pgmap v10490: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:05:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:47 smithi077 ceph-mon[37153]: pgmap v10491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:05:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:47 smithi047 ceph-mon[36835]: pgmap v10491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:05:47.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:47 smithi029 ceph-mon[32610]: pgmap v10491: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:05:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18969.804471 secs 2024-07-27T12:05:48.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18969.804471 secs 2024-07-27T12:05:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18969.804471 secs 2024-07-27T12:05:49.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:49 smithi077 ceph-mon[37153]: pgmap v10492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:05:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:49 smithi047 ceph-mon[36835]: pgmap v10492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:05:49.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:49 smithi029 ceph-mon[32610]: pgmap v10492: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:05:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:51 smithi047 ceph-mon[36835]: pgmap v10493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:05:51.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:51 smithi029 ceph-mon[32610]: pgmap v10493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:05:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:51 smithi077 ceph-mon[37153]: pgmap v10493: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:05:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:52 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:05:52.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:52 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:05:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:52 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:05:52.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:52.926+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:05:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:52.926+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:05:52.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:05:52.926+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:05:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:53 smithi029 ceph-mon[32610]: pgmap v10494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18974.804569 secs 2024-07-27T12:05:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:05:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:05:53.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:53 smithi029 ceph-mon[32610]: pgmap v10495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T12:05:53.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:53 smithi029 ceph-mon[32610]: pgmap v10496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:05:53.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:05:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:53 smithi047 ceph-mon[36835]: pgmap v10494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18974.804569 secs 2024-07-27T12:05:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:05:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:05:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:53 smithi047 ceph-mon[36835]: pgmap v10495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T12:05:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:53 smithi047 ceph-mon[36835]: pgmap v10496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:05:53.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:05:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:53 smithi077 ceph-mon[37153]: pgmap v10494: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:05:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18974.804569 secs 2024-07-27T12:05:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:05:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:05:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:53 smithi077 ceph-mon[37153]: pgmap v10495: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-27T12:05:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:53 smithi077 ceph-mon[37153]: pgmap v10496: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:05:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:05:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:54 smithi077 ceph-mon[37153]: pgmap v10497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:05:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:54 smithi047 ceph-mon[36835]: pgmap v10497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:05:54.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:54 smithi029 ceph-mon[32610]: pgmap v10497: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:05:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:56 smithi077 ceph-mon[37153]: pgmap v10498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:05:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:56 smithi047 ceph-mon[36835]: pgmap v10498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:05:56.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:56 smithi029 ceph-mon[32610]: pgmap v10498: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:05:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18979.804704 secs 2024-07-27T12:05:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18979.804704 secs 2024-07-27T12:05:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18979.804704 secs 2024-07-27T12:05:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:05:58 smithi047 ceph-mon[36835]: pgmap v10499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:05:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:05:58 smithi029 ceph-mon[32610]: pgmap v10499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:05:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:05:58 smithi077 ceph-mon[37153]: pgmap v10499: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:06:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:00 smithi077 ceph-mon[37153]: pgmap v10500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:06:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:00 smithi047 ceph-mon[36835]: pgmap v10500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:06:00.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:00 smithi029 ceph-mon[32610]: pgmap v10500: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:06:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:02 smithi077 ceph-mon[37153]: pgmap v10501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:06:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:02 smithi047 ceph-mon[36835]: pgmap v10501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:06:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:02 smithi029 ceph-mon[32610]: pgmap v10501: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:06:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18984.804815 secs 2024-07-27T12:06:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18984.804815 secs 2024-07-27T12:06:03.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18984.804815 secs 2024-07-27T12:06:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:04 smithi077 ceph-mon[37153]: pgmap v10502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:04 smithi047 ceph-mon[36835]: pgmap v10502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:04 smithi029 ceph-mon[32610]: pgmap v10502: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:06 smithi077 ceph-mon[37153]: pgmap v10503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:06 smithi047 ceph-mon[36835]: pgmap v10503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:06.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:06 smithi029 ceph-mon[32610]: pgmap v10503: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:07.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:06:07.940+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:06:07.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:06:07.940+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:06:07.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:06:07.940+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:06:07.943 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18989.804959 secs 2024-07-27T12:06:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18989.804959 secs 2024-07-27T12:06:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18989.804959 secs 2024-07-27T12:06:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:08 smithi047 ceph-mon[36835]: pgmap v10504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:06:08.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:08 smithi029 ceph-mon[32610]: pgmap v10504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:06:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:08 smithi077 ceph-mon[37153]: pgmap v10504: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:06:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:10 smithi077 ceph-mon[37153]: pgmap v10505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:06:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:10 smithi047 ceph-mon[36835]: pgmap v10505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:06:10.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:10 smithi029 ceph-mon[32610]: pgmap v10505: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:06:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:12 smithi077 ceph-mon[37153]: pgmap v10506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T12:06:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:12 smithi047 ceph-mon[36835]: pgmap v10506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T12:06:12.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:12 smithi029 ceph-mon[32610]: pgmap v10506: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T12:06:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18994.805009 secs 2024-07-27T12:06:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18994.805009 secs 2024-07-27T12:06:13.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18994.805009 secs 2024-07-27T12:06:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:14 smithi077 ceph-mon[37153]: pgmap v10507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:06:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:14 smithi047 ceph-mon[36835]: pgmap v10507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:06:14.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:14 smithi029 ceph-mon[32610]: pgmap v10507: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:06:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:16 smithi077 ceph-mon[37153]: pgmap v10508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:16 smithi047 ceph-mon[36835]: pgmap v10508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:16.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:16 smithi029 ceph-mon[32610]: pgmap v10508: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 18999.805140 secs 2024-07-27T12:06:17.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 18999.805140 secs 2024-07-27T12:06:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 18999.805140 secs 2024-07-27T12:06:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:18 smithi047 ceph-mon[36835]: pgmap v10509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:06:18.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:18 smithi029 ceph-mon[32610]: pgmap v10509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:06:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:18 smithi077 ceph-mon[37153]: pgmap v10509: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:06:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:20 smithi077 ceph-mon[37153]: pgmap v10510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:06:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:20 smithi047 ceph-mon[36835]: pgmap v10510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:06:20.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:20 smithi029 ceph-mon[32610]: pgmap v10510: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:06:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:22 smithi077 ceph-mon[37153]: pgmap v10511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T12:06:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:22 smithi047 ceph-mon[36835]: pgmap v10511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T12:06:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:06:22.956+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:06:22.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:06:22.956+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:06:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:06:22.956+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:06:22.958 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:22 smithi029 ceph-mon[32610]: pgmap v10511: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T12:06:23.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19004.805267 secs 2024-07-27T12:06:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19004.805267 secs 2024-07-27T12:06:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19004.805267 secs 2024-07-27T12:06:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:24 smithi077 ceph-mon[37153]: pgmap v10512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:24 smithi047 ceph-mon[36835]: pgmap v10512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:24.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:24 smithi029 ceph-mon[32610]: pgmap v10512: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:26 smithi077 ceph-mon[37153]: pgmap v10513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:26 smithi047 ceph-mon[36835]: pgmap v10513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:26.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:26 smithi029 ceph-mon[32610]: pgmap v10513: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19009.805371 secs 2024-07-27T12:06:27.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19009.805371 secs 2024-07-27T12:06:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19009.805371 secs 2024-07-27T12:06:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:28 smithi047 ceph-mon[36835]: pgmap v10514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:06:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:28 smithi029 ceph-mon[32610]: pgmap v10514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:06:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:28 smithi077 ceph-mon[37153]: pgmap v10514: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:06:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:30 smithi077 ceph-mon[37153]: pgmap v10515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:06:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:30 smithi047 ceph-mon[36835]: pgmap v10515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:06:30.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:30 smithi029 ceph-mon[32610]: pgmap v10515: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:06:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:32 smithi077 ceph-mon[37153]: pgmap v10516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:06:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:32 smithi047 ceph-mon[36835]: pgmap v10516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:06:32.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:32 smithi029 ceph-mon[32610]: pgmap v10516: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:06:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19014.805477 secs 2024-07-27T12:06:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19014.805477 secs 2024-07-27T12:06:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19014.805477 secs 2024-07-27T12:06:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:34 smithi077 ceph-mon[37153]: pgmap v10517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:34 smithi047 ceph-mon[36835]: pgmap v10517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:34.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:34 smithi029 ceph-mon[32610]: pgmap v10517: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:36 smithi077 ceph-mon[37153]: pgmap v10518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:36 smithi047 ceph-mon[36835]: pgmap v10518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:36.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:36 smithi029 ceph-mon[32610]: pgmap v10518: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19019.805606 secs 2024-07-27T12:06:37.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19019.805606 secs 2024-07-27T12:06:37.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:06:37.971+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:06:37.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:06:37.971+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:06:37.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:06:37.971+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:06:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19019.805606 secs 2024-07-27T12:06:38.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:38 smithi029 ceph-mon[32610]: pgmap v10519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:06:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:38 smithi077 ceph-mon[37153]: pgmap v10519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:06:39.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:38 smithi047 ceph-mon[36835]: pgmap v10519: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:06:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:40 smithi077 ceph-mon[37153]: pgmap v10520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:40.935 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:40 smithi029 ceph-mon[32610]: pgmap v10520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:40 smithi047 ceph-mon[36835]: pgmap v10520: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:42 smithi077 ceph-mon[37153]: pgmap v10521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:06:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:42 smithi047 ceph-mon[36835]: pgmap v10521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:06:42.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:42 smithi029 ceph-mon[32610]: pgmap v10521: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:06:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19024.805732 secs 2024-07-27T12:06:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19024.805732 secs 2024-07-27T12:06:43.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19024.805732 secs 2024-07-27T12:06:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:44 smithi077 ceph-mon[37153]: pgmap v10522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:44 smithi047 ceph-mon[36835]: pgmap v10522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:44.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:44 smithi029 ceph-mon[32610]: pgmap v10522: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:06:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:46 smithi077 ceph-mon[37153]: pgmap v10523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:46 smithi047 ceph-mon[36835]: pgmap v10523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:46.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:46 smithi029 ceph-mon[32610]: pgmap v10523: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:06:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19029.805895 secs 2024-07-27T12:06:47.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19029.805895 secs 2024-07-27T12:06:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19029.805895 secs 2024-07-27T12:06:48.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:48 smithi029 ceph-mon[32610]: pgmap v10524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:06:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:48 smithi077 ceph-mon[37153]: pgmap v10524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:06:49.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:48 smithi047 ceph-mon[36835]: pgmap v10524: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:06:50.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:50 smithi077 ceph-mon[37153]: pgmap v10525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:06:50.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:50 smithi047 ceph-mon[36835]: pgmap v10525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:06:50.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:50 smithi029 ceph-mon[32610]: pgmap v10525: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:06:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:52 smithi077 ceph-mon[37153]: pgmap v10526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:06:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:52 smithi047 ceph-mon[36835]: pgmap v10526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:06:52.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:52 smithi029 ceph-mon[32610]: pgmap v10526: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:06:52.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:06:52.987+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:06:52.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:06:52.987+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:06:52.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:06:52.987+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:06:53.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19034.805992 secs 2024-07-27T12:06:53.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:06:53.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:06:53.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:06:53.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:06:53.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:06:53.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T12:06:53.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T12:06:53.848 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:06:53.849 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:06:53.849 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: pgmap v10527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:06:53.849 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: pgmap v10528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:06:53.849 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:06:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19034.805992 secs 2024-07-27T12:06:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:06:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:06:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:06:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:06:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:06:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T12:06:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T12:06:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:06:53.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:06:53.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: pgmap v10527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:06:53.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: pgmap v10528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:06:53.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:06:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19034.805992 secs 2024-07-27T12:06:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:06:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:06:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:06:53.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:06:53.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:06:53.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T12:06:53.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T12:06:53.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:06:53.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:06:53.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: pgmap v10527: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:06:53.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: pgmap v10528: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:06:53.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:06:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:56 smithi077 ceph-mon[37153]: pgmap v10529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:06:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:56 smithi047 ceph-mon[36835]: pgmap v10529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:06:56.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:56 smithi029 ceph-mon[32610]: pgmap v10529: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:06:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:58 smithi077 ceph-mon[37153]: pgmap v10530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:06:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:06:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19039.806124 secs 2024-07-27T12:06:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:58 smithi047 ceph-mon[36835]: pgmap v10530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:06:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:06:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19039.806124 secs 2024-07-27T12:06:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:58 smithi029 ceph-mon[32610]: pgmap v10530: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:06:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:06:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19039.806124 secs 2024-07-27T12:07:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:00 smithi077 ceph-mon[37153]: pgmap v10531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:07:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:00 smithi047 ceph-mon[36835]: pgmap v10531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:07:00.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:00 smithi029 ceph-mon[32610]: pgmap v10531: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:07:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:02 smithi077 ceph-mon[37153]: pgmap v10532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T12:07:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:02 smithi047 ceph-mon[36835]: pgmap v10532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T12:07:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:02 smithi029 ceph-mon[32610]: pgmap v10532: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T12:07:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19044.806210 secs 2024-07-27T12:07:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19044.806210 secs 2024-07-27T12:07:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19044.806210 secs 2024-07-27T12:07:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:04 smithi077 ceph-mon[37153]: pgmap v10533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:07:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:04 smithi047 ceph-mon[36835]: pgmap v10533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:07:04.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:04 smithi029 ceph-mon[32610]: pgmap v10533: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:07:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:06 smithi077 ceph-mon[37153]: pgmap v10534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:06 smithi047 ceph-mon[36835]: pgmap v10534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:06.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:06 smithi029 ceph-mon[32610]: pgmap v10534: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:08.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:07:08.001+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:07:08.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:07:08.001+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:07:08.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:07:08.001+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:07:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:08 smithi077 ceph-mon[37153]: pgmap v10535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19049.806358 secs 2024-07-27T12:07:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:08 smithi047 ceph-mon[36835]: pgmap v10535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19049.806358 secs 2024-07-27T12:07:08.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:08 smithi029 ceph-mon[32610]: pgmap v10535: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19049.806358 secs 2024-07-27T12:07:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:10 smithi077 ceph-mon[37153]: pgmap v10536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:07:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:10 smithi047 ceph-mon[36835]: pgmap v10536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:07:10.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:10 smithi029 ceph-mon[32610]: pgmap v10536: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:07:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:12 smithi047 ceph-mon[36835]: pgmap v10537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:12.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:12 smithi029 ceph-mon[32610]: pgmap v10537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:12 smithi077 ceph-mon[37153]: pgmap v10537: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19054.806486 secs 2024-07-27T12:07:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19054.806486 secs 2024-07-27T12:07:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19054.806486 secs 2024-07-27T12:07:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:14 smithi047 ceph-mon[36835]: pgmap v10538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:07:14.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:14 smithi029 ceph-mon[32610]: pgmap v10538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:07:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:14 smithi077 ceph-mon[37153]: pgmap v10538: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:07:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:16 smithi047 ceph-mon[36835]: pgmap v10539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:16.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:16 smithi029 ceph-mon[32610]: pgmap v10539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:16 smithi077 ceph-mon[37153]: pgmap v10539: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:18 smithi047 ceph-mon[36835]: pgmap v10540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19059.806628 secs 2024-07-27T12:07:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:18 smithi029 ceph-mon[32610]: pgmap v10540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19059.806628 secs 2024-07-27T12:07:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:18 smithi077 ceph-mon[37153]: pgmap v10540: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19059.806628 secs 2024-07-27T12:07:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:20 smithi047 ceph-mon[36835]: pgmap v10541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:07:20.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:20 smithi029 ceph-mon[32610]: pgmap v10541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:07:20.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:20 smithi077 ceph-mon[37153]: pgmap v10541: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:07:22.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:22 smithi047 ceph-mon[36835]: pgmap v10542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:22.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:22 smithi029 ceph-mon[32610]: pgmap v10542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:22 smithi077 ceph-mon[37153]: pgmap v10542: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:23.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:07:23.017+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:07:23.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:07:23.017+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:07:23.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:07:23.017+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:07:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19064.806776 secs 2024-07-27T12:07:23.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19064.806776 secs 2024-07-27T12:07:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19064.806776 secs 2024-07-27T12:07:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:24 smithi047 ceph-mon[36835]: pgmap v10543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:07:24.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:24 smithi029 ceph-mon[32610]: pgmap v10543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:07:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:24 smithi077 ceph-mon[37153]: pgmap v10543: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:07:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:26 smithi047 ceph-mon[36835]: pgmap v10544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:26 smithi029 ceph-mon[32610]: pgmap v10544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:26.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:26 smithi077 ceph-mon[37153]: pgmap v10544: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:28 smithi047 ceph-mon[36835]: pgmap v10545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19069.806908 secs 2024-07-27T12:07:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:28 smithi029 ceph-mon[32610]: pgmap v10545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19069.806908 secs 2024-07-27T12:07:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:28 smithi077 ceph-mon[37153]: pgmap v10545: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19069.806908 secs 2024-07-27T12:07:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:30 smithi047 ceph-mon[36835]: pgmap v10546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:07:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:30 smithi029 ceph-mon[32610]: pgmap v10546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:07:30.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:30 smithi077 ceph-mon[37153]: pgmap v10546: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:07:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:32 smithi047 ceph-mon[36835]: pgmap v10547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:32.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:32 smithi029 ceph-mon[32610]: pgmap v10547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:32 smithi077 ceph-mon[37153]: pgmap v10547: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19074.807038 secs 2024-07-27T12:07:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19074.807038 secs 2024-07-27T12:07:33.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19074.807038 secs 2024-07-27T12:07:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:34 smithi047 ceph-mon[36835]: pgmap v10548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:07:34.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:34 smithi029 ceph-mon[32610]: pgmap v10548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:07:34.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:34 smithi077 ceph-mon[37153]: pgmap v10548: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:07:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:36 smithi047 ceph-mon[36835]: pgmap v10549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:36.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:36 smithi029 ceph-mon[32610]: pgmap v10549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:36 smithi077 ceph-mon[37153]: pgmap v10549: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:38.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:07:38.033+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:07:38.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:07:38.033+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:07:38.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:07:38.033+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:07:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:38 smithi047 ceph-mon[36835]: pgmap v10550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19079.807168 secs 2024-07-27T12:07:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:38 smithi029 ceph-mon[32610]: pgmap v10550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19079.807168 secs 2024-07-27T12:07:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:38 smithi077 ceph-mon[37153]: pgmap v10550: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19079.807168 secs 2024-07-27T12:07:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:40 smithi047 ceph-mon[36835]: pgmap v10551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:07:40.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:40 smithi029 ceph-mon[32610]: pgmap v10551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:07:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:40 smithi077 ceph-mon[37153]: pgmap v10551: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:07:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:42 smithi047 ceph-mon[36835]: pgmap v10552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T12:07:42.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:42 smithi029 ceph-mon[32610]: pgmap v10552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T12:07:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:42 smithi077 ceph-mon[37153]: pgmap v10552: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T12:07:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19084.807299 secs 2024-07-27T12:07:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19084.807299 secs 2024-07-27T12:07:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19084.807299 secs 2024-07-27T12:07:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:44 smithi047 ceph-mon[36835]: pgmap v10553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:07:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:44 smithi029 ceph-mon[32610]: pgmap v10553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:07:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:44 smithi077 ceph-mon[37153]: pgmap v10553: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:07:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:46 smithi047 ceph-mon[36835]: pgmap v10554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:07:46.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:46 smithi029 ceph-mon[32610]: pgmap v10554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:07:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:46 smithi077 ceph-mon[37153]: pgmap v10554: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:07:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:48 smithi047 ceph-mon[36835]: pgmap v10555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19089.807403 secs 2024-07-27T12:07:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:48 smithi029 ceph-mon[32610]: pgmap v10555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19089.807403 secs 2024-07-27T12:07:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:48 smithi077 ceph-mon[37153]: pgmap v10555: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:07:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19089.807403 secs 2024-07-27T12:07:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:50 smithi047 ceph-mon[36835]: pgmap v10556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:07:50.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:50 smithi029 ceph-mon[32610]: pgmap v10556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:07:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:50 smithi077 ceph-mon[37153]: pgmap v10556: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:07:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:52 smithi047 ceph-mon[36835]: pgmap v10557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:52.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:52 smithi029 ceph-mon[32610]: pgmap v10557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:52 smithi077 ceph-mon[37153]: pgmap v10557: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:07:53.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:07:53.047+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:07:53.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:07:53.047+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:07:53.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:07:53.047+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:07:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19094.807533 secs 2024-07-27T12:07:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:53 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:07:53.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19094.807533 secs 2024-07-27T12:07:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:53 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:07:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19094.807533 secs 2024-07-27T12:07:53.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:53 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:07:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:54 smithi029 ceph-mon[32610]: pgmap v10558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:07:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:07:54.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:07:54.464 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:07:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:54 smithi077 ceph-mon[37153]: pgmap v10558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:07:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:07:54.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:07:54.608 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:07:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:54 smithi047 ceph-mon[36835]: pgmap v10558: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:07:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:07:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:07:54.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:07:54.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:07:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:55 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T12:07:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:55 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T12:07:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:55 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T12:07:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:55 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T12:07:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:55 smithi029 ceph-mon[32610]: pgmap v10559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:07:55.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:55 smithi029 ceph-mon[32610]: pgmap v10560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:07:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:55 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T12:07:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:55 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T12:07:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:55 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T12:07:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:55 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T12:07:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:55 smithi077 ceph-mon[37153]: pgmap v10559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:07:55.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:55 smithi077 ceph-mon[37153]: pgmap v10560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:07:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:55 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T12:07:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:55 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T12:07:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:55 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T12:07:55.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:55 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T12:07:55.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:55 smithi047 ceph-mon[36835]: pgmap v10559: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:07:55.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:55 smithi047 ceph-mon[36835]: pgmap v10560: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:07:57.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:57 smithi029 ceph-mon[32610]: pgmap v10561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:07:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:57 smithi077 ceph-mon[37153]: pgmap v10561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:07:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:57 smithi047 ceph-mon[36835]: pgmap v10561: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:07:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19099.807643 secs 2024-07-27T12:07:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19099.807643 secs 2024-07-27T12:07:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19099.807643 secs 2024-07-27T12:07:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:07:59 smithi029 ceph-mon[32610]: pgmap v10562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:07:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:07:59 smithi077 ceph-mon[37153]: pgmap v10562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:07:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:07:59 smithi047 ceph-mon[36835]: pgmap v10562: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:08:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:00.512+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:08:00.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:00.512+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:08:00.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:00.512+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:08:00.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:00.512+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:08:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:01 smithi077 ceph-mon[37153]: pgmap v10563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:08:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:01 smithi047 ceph-mon[36835]: pgmap v10563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:08:01.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:01 smithi029 ceph-mon[32610]: pgmap v10563: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:08:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:03 smithi077 ceph-mon[37153]: pgmap v10564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:08:03.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19104.807774 secs 2024-07-27T12:08:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:03 smithi047 ceph-mon[36835]: pgmap v10564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:08:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19104.807774 secs 2024-07-27T12:08:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:03 smithi029 ceph-mon[32610]: pgmap v10564: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:08:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19104.807774 secs 2024-07-27T12:08:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:05 smithi077 ceph-mon[37153]: pgmap v10565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:08:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:05 smithi047 ceph-mon[36835]: pgmap v10565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:08:05.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:05 smithi029 ceph-mon[32610]: pgmap v10565: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:08:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:07 smithi077 ceph-mon[37153]: pgmap v10566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:08:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:07 smithi047 ceph-mon[36835]: pgmap v10566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:08:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:07 smithi029 ceph-mon[32610]: pgmap v10566: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:08:08.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:08.049+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:08:08.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:08.049+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:08:08.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:08.049+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:08:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19109.807901 secs 2024-07-27T12:08:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19109.807901 secs 2024-07-27T12:08:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19109.807901 secs 2024-07-27T12:08:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:09 smithi077 ceph-mon[37153]: pgmap v10567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T12:08:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:09 smithi047 ceph-mon[36835]: pgmap v10567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T12:08:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:09 smithi029 ceph-mon[32610]: pgmap v10567: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T12:08:11.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:11 smithi077 ceph-mon[37153]: pgmap v10568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:08:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:11 smithi047 ceph-mon[36835]: pgmap v10568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:08:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:11 smithi029 ceph-mon[32610]: pgmap v10568: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:08:13.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:13 smithi077 ceph-mon[37153]: pgmap v10569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19114.808004 secs 2024-07-27T12:08:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:13 smithi047 ceph-mon[36835]: pgmap v10569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19114.808004 secs 2024-07-27T12:08:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:13 smithi029 ceph-mon[32610]: pgmap v10569: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19114.808004 secs 2024-07-27T12:08:15.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:15 smithi077 ceph-mon[37153]: pgmap v10570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:08:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:15 smithi047 ceph-mon[36835]: pgmap v10570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:08:15.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:15 smithi029 ceph-mon[32610]: pgmap v10570: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:08:17.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:17 smithi077 ceph-mon[37153]: pgmap v10571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:08:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:17 smithi047 ceph-mon[36835]: pgmap v10571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:08:17.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:17 smithi029 ceph-mon[32610]: pgmap v10571: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:08:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19119.808119 secs 2024-07-27T12:08:18.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19119.808119 secs 2024-07-27T12:08:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19119.808119 secs 2024-07-27T12:08:19.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:19 smithi077 ceph-mon[37153]: pgmap v10572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T12:08:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:19 smithi047 ceph-mon[36835]: pgmap v10572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T12:08:19.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:19 smithi029 ceph-mon[32610]: pgmap v10572: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-27T12:08:21.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:21 smithi077 ceph-mon[37153]: pgmap v10573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:08:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:21 smithi047 ceph-mon[36835]: pgmap v10573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:08:21.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:21 smithi029 ceph-mon[32610]: pgmap v10573: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:08:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:23.053+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:08:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:23.053+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:08:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:23.053+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:08:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:23 smithi029 ceph-mon[32610]: pgmap v10574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:23.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19124.808225 secs 2024-07-27T12:08:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:23 smithi077 ceph-mon[37153]: pgmap v10574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:23.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19124.808225 secs 2024-07-27T12:08:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:23 smithi047 ceph-mon[36835]: pgmap v10574: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19124.808225 secs 2024-07-27T12:08:25.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:25 smithi077 ceph-mon[37153]: pgmap v10575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:08:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:25 smithi047 ceph-mon[36835]: pgmap v10575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:08:25.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:25 smithi029 ceph-mon[32610]: pgmap v10575: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:08:27.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:27 smithi077 ceph-mon[37153]: pgmap v10576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:08:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:27 smithi047 ceph-mon[36835]: pgmap v10576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:08:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:27 smithi029 ceph-mon[32610]: pgmap v10576: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:08:28.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19129.808315 secs 2024-07-27T12:08:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19129.808315 secs 2024-07-27T12:08:28.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19129.808315 secs 2024-07-27T12:08:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:29 smithi077 ceph-mon[37153]: pgmap v10577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:08:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:29 smithi047 ceph-mon[36835]: pgmap v10577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:08:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:29 smithi029 ceph-mon[32610]: pgmap v10577: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:08:31.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:31 smithi077 ceph-mon[37153]: pgmap v10578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:08:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:31 smithi047 ceph-mon[36835]: pgmap v10578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:08:31.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:31 smithi029 ceph-mon[32610]: pgmap v10578: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:08:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:33 smithi077 ceph-mon[37153]: pgmap v10579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:33.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19134.808447 secs 2024-07-27T12:08:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:33 smithi047 ceph-mon[36835]: pgmap v10579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19134.808447 secs 2024-07-27T12:08:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:33 smithi029 ceph-mon[32610]: pgmap v10579: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19134.808447 secs 2024-07-27T12:08:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:35 smithi077 ceph-mon[37153]: pgmap v10580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:08:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:35 smithi047 ceph-mon[36835]: pgmap v10580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:08:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:35 smithi029 ceph-mon[32610]: pgmap v10580: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:08:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:37 smithi077 ceph-mon[37153]: pgmap v10581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:08:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:37 smithi047 ceph-mon[36835]: pgmap v10581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:08:37.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:37 smithi029 ceph-mon[32610]: pgmap v10581: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:08:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:38.069+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:08:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:38.069+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:08:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:38.069+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:08:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19139.808604 secs 2024-07-27T12:08:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19139.808604 secs 2024-07-27T12:08:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19139.808604 secs 2024-07-27T12:08:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:39 smithi077 ceph-mon[37153]: pgmap v10582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:08:39.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:39 smithi047 ceph-mon[36835]: pgmap v10582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:08:39.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:39 smithi029 ceph-mon[32610]: pgmap v10582: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:08:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:41 smithi077 ceph-mon[37153]: pgmap v10583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:08:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:41 smithi047 ceph-mon[36835]: pgmap v10583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:08:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:41 smithi029 ceph-mon[32610]: pgmap v10583: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:08:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:43 smithi077 ceph-mon[37153]: pgmap v10584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19144.808690 secs 2024-07-27T12:08:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:43 smithi047 ceph-mon[36835]: pgmap v10584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19144.808690 secs 2024-07-27T12:08:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:43 smithi029 ceph-mon[32610]: pgmap v10584: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19144.808690 secs 2024-07-27T12:08:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:45 smithi077 ceph-mon[37153]: pgmap v10585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:08:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:45 smithi047 ceph-mon[36835]: pgmap v10585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:08:45.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:45 smithi029 ceph-mon[32610]: pgmap v10585: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:08:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:47 smithi077 ceph-mon[37153]: pgmap v10586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:08:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:47 smithi047 ceph-mon[36835]: pgmap v10586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:08:47.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:47 smithi029 ceph-mon[32610]: pgmap v10586: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:08:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19149.808818 secs 2024-07-27T12:08:48.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19149.808818 secs 2024-07-27T12:08:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19149.808818 secs 2024-07-27T12:08:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:49 smithi077 ceph-mon[37153]: pgmap v10587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:08:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:49 smithi047 ceph-mon[36835]: pgmap v10587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:08:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:49 smithi029 ceph-mon[32610]: pgmap v10587: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:08:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:51 smithi077 ceph-mon[37153]: pgmap v10588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:08:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:51 smithi047 ceph-mon[36835]: pgmap v10588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:08:51.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:51 smithi029 ceph-mon[32610]: pgmap v10588: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:08:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:53.070+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:08:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:53.070+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:08:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:08:53.070+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:08:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:53 smithi077 ceph-mon[37153]: pgmap v10589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19154.808884 secs 2024-07-27T12:08:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:53 smithi047 ceph-mon[36835]: pgmap v10589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19154.808884 secs 2024-07-27T12:08:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:53 smithi029 ceph-mon[32610]: pgmap v10589: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:08:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19154.808884 secs 2024-07-27T12:08:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:08:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:08:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:08:54.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:08:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:08:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:08:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:08:54.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:08:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:08:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:08:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:08:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:08:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:55 smithi077 ceph-mon[37153]: pgmap v10590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:08:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:55 smithi077 ceph-mon[37153]: pgmap v10591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:08:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:55 smithi077 ceph-mon[37153]: pgmap v10592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:08:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:55 smithi047 ceph-mon[36835]: pgmap v10590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:08:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:55 smithi047 ceph-mon[36835]: pgmap v10591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:08:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:55 smithi047 ceph-mon[36835]: pgmap v10592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:08:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:55 smithi029 ceph-mon[32610]: pgmap v10590: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:08:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:55 smithi029 ceph-mon[32610]: pgmap v10591: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:08:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:55 smithi029 ceph-mon[32610]: pgmap v10592: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:08:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:57 smithi077 ceph-mon[37153]: pgmap v10593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T12:08:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:57 smithi047 ceph-mon[36835]: pgmap v10593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T12:08:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:57 smithi029 ceph-mon[32610]: pgmap v10593: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T12:08:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19159.809014 secs 2024-07-27T12:08:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19159.809014 secs 2024-07-27T12:08:58.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19159.809014 secs 2024-07-27T12:08:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:08:59 smithi077 ceph-mon[37153]: pgmap v10594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:08:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:08:59 smithi047 ceph-mon[36835]: pgmap v10594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:08:59.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:08:59 smithi029 ceph-mon[32610]: pgmap v10594: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:09:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:01 smithi077 ceph-mon[37153]: pgmap v10595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:09:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:01 smithi047 ceph-mon[36835]: pgmap v10595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:09:01.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:01 smithi029 ceph-mon[32610]: pgmap v10595: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:09:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19164.809146 secs 2024-07-27T12:09:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:03 smithi077 ceph-mon[37153]: pgmap v10596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:09:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19164.809146 secs 2024-07-27T12:09:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:03 smithi047 ceph-mon[36835]: pgmap v10596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:09:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19164.809146 secs 2024-07-27T12:09:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:03 smithi029 ceph-mon[32610]: pgmap v10596: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:09:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:05 smithi077 ceph-mon[37153]: pgmap v10597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:09:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:05 smithi047 ceph-mon[36835]: pgmap v10597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:09:05.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:05 smithi029 ceph-mon[32610]: pgmap v10597: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:09:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:07 smithi077 ceph-mon[37153]: pgmap v10598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:07 smithi047 ceph-mon[36835]: pgmap v10598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:07.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:07 smithi029 ceph-mon[32610]: pgmap v10598: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:08.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:09:08.079+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:09:08.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:09:08.079+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:09:08.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:09:08.079+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:09:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19169.809281 secs 2024-07-27T12:09:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19169.809281 secs 2024-07-27T12:09:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19169.809281 secs 2024-07-27T12:09:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:09 smithi047 ceph-mon[36835]: pgmap v10599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:09 smithi029 ceph-mon[32610]: pgmap v10599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:09 smithi077 ceph-mon[37153]: pgmap v10599: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:11 smithi047 ceph-mon[36835]: pgmap v10600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:11.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:11 smithi029 ceph-mon[32610]: pgmap v10600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:11 smithi077 ceph-mon[37153]: pgmap v10600: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19174.809379 secs 2024-07-27T12:09:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:13 smithi047 ceph-mon[36835]: pgmap v10601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:13.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19174.809379 secs 2024-07-27T12:09:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:13 smithi029 ceph-mon[32610]: pgmap v10601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19174.809379 secs 2024-07-27T12:09:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:13 smithi077 ceph-mon[37153]: pgmap v10601: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:15 smithi047 ceph-mon[36835]: pgmap v10602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:15.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:15 smithi029 ceph-mon[32610]: pgmap v10602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:15 smithi077 ceph-mon[37153]: pgmap v10602: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:17 smithi047 ceph-mon[36835]: pgmap v10603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:17.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:17 smithi029 ceph-mon[32610]: pgmap v10603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:17 smithi077 ceph-mon[37153]: pgmap v10603: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19179.809448 secs 2024-07-27T12:09:18.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19179.809448 secs 2024-07-27T12:09:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19179.809448 secs 2024-07-27T12:09:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:19 smithi047 ceph-mon[36835]: pgmap v10604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:19.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:19 smithi029 ceph-mon[32610]: pgmap v10604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:19 smithi077 ceph-mon[37153]: pgmap v10604: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:21 smithi047 ceph-mon[36835]: pgmap v10605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:21.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:21 smithi029 ceph-mon[32610]: pgmap v10605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:21 smithi077 ceph-mon[37153]: pgmap v10605: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:09:23.094+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:09:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:09:23.094+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:09:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:09:23.094+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:09:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19184.809576 secs 2024-07-27T12:09:23.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:23 smithi047 ceph-mon[36835]: pgmap v10606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19184.809576 secs 2024-07-27T12:09:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:23 smithi029 ceph-mon[32610]: pgmap v10606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19184.809576 secs 2024-07-27T12:09:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:23 smithi077 ceph-mon[37153]: pgmap v10606: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:25 smithi047 ceph-mon[36835]: pgmap v10607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:25.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:25 smithi029 ceph-mon[32610]: pgmap v10607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:25 smithi077 ceph-mon[37153]: pgmap v10607: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:27 smithi047 ceph-mon[36835]: pgmap v10608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:27.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:27 smithi029 ceph-mon[32610]: pgmap v10608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:27 smithi077 ceph-mon[37153]: pgmap v10608: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19189.809674 secs 2024-07-27T12:09:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19189.809674 secs 2024-07-27T12:09:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19189.809674 secs 2024-07-27T12:09:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:29 smithi047 ceph-mon[36835]: pgmap v10609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:29.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:29 smithi029 ceph-mon[32610]: pgmap v10609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:29 smithi077 ceph-mon[37153]: pgmap v10609: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:31 smithi047 ceph-mon[36835]: pgmap v10610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:31.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:31 smithi029 ceph-mon[32610]: pgmap v10610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:31 smithi077 ceph-mon[37153]: pgmap v10610: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:33.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19194.809804 secs 2024-07-27T12:09:33.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:33 smithi077 ceph-mon[37153]: pgmap v10611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19194.809804 secs 2024-07-27T12:09:33.783 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:33 smithi047 ceph-mon[36835]: pgmap v10611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:33.784 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19194.809804 secs 2024-07-27T12:09:33.784 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:33 smithi029 ceph-mon[32610]: pgmap v10611: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:35 smithi047 ceph-mon[36835]: pgmap v10612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:35.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:35 smithi029 ceph-mon[32610]: pgmap v10612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:35 smithi077 ceph-mon[37153]: pgmap v10612: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:37 smithi047 ceph-mon[36835]: pgmap v10613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:37.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:37 smithi029 ceph-mon[32610]: pgmap v10613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:37 smithi077 ceph-mon[37153]: pgmap v10613: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:38.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:09:38.110+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:09:38.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:09:38.110+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:09:38.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:09:38.110+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:09:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19199.809925 secs 2024-07-27T12:09:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19199.809925 secs 2024-07-27T12:09:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19199.809925 secs 2024-07-27T12:09:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:39 smithi047 ceph-mon[36835]: pgmap v10614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:39.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:39 smithi029 ceph-mon[32610]: pgmap v10614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:39 smithi077 ceph-mon[37153]: pgmap v10614: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:41 smithi047 ceph-mon[36835]: pgmap v10615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:09:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:41 smithi029 ceph-mon[32610]: pgmap v10615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:09:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:41 smithi077 ceph-mon[37153]: pgmap v10615: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:09:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19204.810047 secs 2024-07-27T12:09:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:43 smithi047 ceph-mon[36835]: pgmap v10616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19204.810047 secs 2024-07-27T12:09:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:43 smithi029 ceph-mon[32610]: pgmap v10616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19204.810047 secs 2024-07-27T12:09:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:43 smithi077 ceph-mon[37153]: pgmap v10616: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:45 smithi047 ceph-mon[36835]: pgmap v10617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:45.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:45 smithi029 ceph-mon[32610]: pgmap v10617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:45 smithi077 ceph-mon[37153]: pgmap v10617: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:47 smithi047 ceph-mon[36835]: pgmap v10618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:47 smithi029 ceph-mon[32610]: pgmap v10618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:47 smithi077 ceph-mon[37153]: pgmap v10618: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19209.810170 secs 2024-07-27T12:09:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19209.810170 secs 2024-07-27T12:09:48.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19209.810170 secs 2024-07-27T12:09:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:49 smithi047 ceph-mon[36835]: pgmap v10619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:49.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:49 smithi029 ceph-mon[32610]: pgmap v10619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:49 smithi077 ceph-mon[37153]: pgmap v10619: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:09:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:51 smithi047 ceph-mon[36835]: pgmap v10620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:51.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:51 smithi029 ceph-mon[32610]: pgmap v10620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:51 smithi077 ceph-mon[37153]: pgmap v10620: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:09:53.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:09:53.114+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:09:53.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:09:53.114+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:09:53.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:09:53.114+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:09:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19214.810292 secs 2024-07-27T12:09:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:53 smithi029 ceph-mon[32610]: pgmap v10621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19214.810292 secs 2024-07-27T12:09:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:53 smithi077 ceph-mon[37153]: pgmap v10621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19214.810292 secs 2024-07-27T12:09:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:53 smithi047 ceph-mon[36835]: pgmap v10621: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:09:54.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:09:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:09:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:09:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:55 smithi029 ceph-mon[32610]: pgmap v10622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:09:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:09:55.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:55 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:09:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:55 smithi077 ceph-mon[37153]: pgmap v10622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:09:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:09:55.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:55 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:09:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:55 smithi047 ceph-mon[36835]: pgmap v10622: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:09:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:09:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:09:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:55 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:09:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:56 smithi077 ceph-mon[37153]: pgmap v10623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:09:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:56 smithi077 ceph-mon[37153]: pgmap v10624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:09:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:56 smithi047 ceph-mon[36835]: pgmap v10623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:09:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:56 smithi047 ceph-mon[36835]: pgmap v10624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:09:56.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:56 smithi029 ceph-mon[32610]: pgmap v10623: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:09:56.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:56 smithi029 ceph-mon[32610]: pgmap v10624: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:09:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:58 smithi077 ceph-mon[37153]: pgmap v10625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:09:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:09:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19219.810415 secs 2024-07-27T12:09:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:58 smithi047 ceph-mon[36835]: pgmap v10625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:09:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:09:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19219.810415 secs 2024-07-27T12:09:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:58 smithi029 ceph-mon[32610]: pgmap v10625: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:09:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:09:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19219.810415 secs 2024-07-27T12:10:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:00 smithi077 ceph-mon[37153]: pgmap v10626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:10:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T12:10:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:00 smithi047 ceph-mon[36835]: pgmap v10626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:10:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T12:10:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:00 smithi029 ceph-mon[32610]: pgmap v10626: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:10:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T12:10:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:02 smithi077 ceph-mon[37153]: pgmap v10627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:10:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:02 smithi047 ceph-mon[36835]: pgmap v10627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:10:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:02 smithi029 ceph-mon[32610]: pgmap v10627: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:10:03.757 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19224.810573 secs 2024-07-27T12:10:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19224.810573 secs 2024-07-27T12:10:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19224.810573 secs 2024-07-27T12:10:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:04 smithi077 ceph-mon[37153]: pgmap v10628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:10:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:04 smithi047 ceph-mon[36835]: pgmap v10628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:10:04.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:04 smithi029 ceph-mon[32610]: pgmap v10628: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:10:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:06 smithi077 ceph-mon[37153]: pgmap v10629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:10:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:06 smithi047 ceph-mon[36835]: pgmap v10629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:10:06.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:06 smithi029 ceph-mon[32610]: pgmap v10629: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:10:08.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:10:08.127+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:10:08.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:10:08.127+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:10:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:10:08.127+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:10:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:08 smithi077 ceph-mon[37153]: pgmap v10630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:08.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19229.810692 secs 2024-07-27T12:10:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:08 smithi047 ceph-mon[36835]: pgmap v10630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19229.810692 secs 2024-07-27T12:10:08.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:08 smithi029 ceph-mon[32610]: pgmap v10630: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19229.810692 secs 2024-07-27T12:10:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:10 smithi077 ceph-mon[37153]: pgmap v10631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:10 smithi047 ceph-mon[36835]: pgmap v10631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:10.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:10 smithi029 ceph-mon[32610]: pgmap v10631: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:12 smithi077 ceph-mon[37153]: pgmap v10632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:12 smithi047 ceph-mon[36835]: pgmap v10632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:12 smithi029 ceph-mon[32610]: pgmap v10632: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19234.810863 secs 2024-07-27T12:10:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19234.810863 secs 2024-07-27T12:10:13.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19234.810863 secs 2024-07-27T12:10:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:14 smithi077 ceph-mon[37153]: pgmap v10633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:14 smithi047 ceph-mon[36835]: pgmap v10633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:14 smithi029 ceph-mon[32610]: pgmap v10633: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:16.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:16 smithi077 ceph-mon[37153]: pgmap v10634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:16 smithi047 ceph-mon[36835]: pgmap v10634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:16.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:16 smithi029 ceph-mon[32610]: pgmap v10634: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:18.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:18 smithi029 ceph-mon[32610]: pgmap v10635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:18.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19239.810941 secs 2024-07-27T12:10:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:18 smithi077 ceph-mon[37153]: pgmap v10635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19239.810941 secs 2024-07-27T12:10:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:18 smithi047 ceph-mon[36835]: pgmap v10635: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19239.810941 secs 2024-07-27T12:10:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:20 smithi077 ceph-mon[37153]: pgmap v10636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:20 smithi047 ceph-mon[36835]: pgmap v10636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:20.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:20 smithi029 ceph-mon[32610]: pgmap v10636: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:22 smithi077 ceph-mon[37153]: pgmap v10637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:22 smithi047 ceph-mon[36835]: pgmap v10637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:22 smithi029 ceph-mon[32610]: pgmap v10637: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:23.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:10:23.141+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:10:23.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:10:23.141+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:10:23.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:10:23.141+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:10:23.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19244.811069 secs 2024-07-27T12:10:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19244.811069 secs 2024-07-27T12:10:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19244.811069 secs 2024-07-27T12:10:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:24 smithi077 ceph-mon[37153]: pgmap v10638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:24 smithi047 ceph-mon[36835]: pgmap v10638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:24.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:24 smithi029 ceph-mon[32610]: pgmap v10638: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:26 smithi077 ceph-mon[37153]: pgmap v10639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:26 smithi047 ceph-mon[36835]: pgmap v10639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:26.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:26 smithi029 ceph-mon[32610]: pgmap v10639: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:28 smithi077 ceph-mon[37153]: pgmap v10640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19249.811146 secs 2024-07-27T12:10:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:28 smithi047 ceph-mon[36835]: pgmap v10640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19249.811146 secs 2024-07-27T12:10:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:28 smithi029 ceph-mon[32610]: pgmap v10640: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19249.811146 secs 2024-07-27T12:10:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:30 smithi077 ceph-mon[37153]: pgmap v10641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:30 smithi047 ceph-mon[36835]: pgmap v10641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:30.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:30 smithi029 ceph-mon[32610]: pgmap v10641: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:32 smithi077 ceph-mon[37153]: pgmap v10642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:32 smithi047 ceph-mon[36835]: pgmap v10642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:32.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:32 smithi029 ceph-mon[32610]: pgmap v10642: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19254.811261 secs 2024-07-27T12:10:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:33 smithi077 ceph-mon[37153]: pgmap v10643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19254.811261 secs 2024-07-27T12:10:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:33 smithi047 ceph-mon[36835]: pgmap v10643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19254.811261 secs 2024-07-27T12:10:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:33 smithi029 ceph-mon[32610]: pgmap v10643: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:35 smithi077 ceph-mon[37153]: pgmap v10644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:35 smithi047 ceph-mon[36835]: pgmap v10644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:36.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:35 smithi029 ceph-mon[32610]: pgmap v10644: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:37 smithi077 ceph-mon[37153]: pgmap v10645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19259.811383 secs 2024-07-27T12:10:38.152 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:37 smithi029 ceph-mon[32610]: pgmap v10645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:38.152 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19259.811383 secs 2024-07-27T12:10:38.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:10:38.151+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:10:38.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:10:38.151+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:10:38.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:10:38.151+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:10:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:37 smithi047 ceph-mon[36835]: pgmap v10645: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19259.811383 secs 2024-07-27T12:10:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:39 smithi077 ceph-mon[37153]: pgmap v10646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:39 smithi047 ceph-mon[36835]: pgmap v10646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:40.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:39 smithi029 ceph-mon[32610]: pgmap v10646: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:41 smithi077 ceph-mon[37153]: pgmap v10647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:41 smithi047 ceph-mon[36835]: pgmap v10647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:41 smithi029 ceph-mon[32610]: pgmap v10647: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19264.811525 secs 2024-07-27T12:10:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19264.811525 secs 2024-07-27T12:10:43.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19264.811525 secs 2024-07-27T12:10:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:43 smithi077 ceph-mon[37153]: pgmap v10648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:43 smithi047 ceph-mon[36835]: pgmap v10648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:43 smithi029 ceph-mon[32610]: pgmap v10648: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:45 smithi077 ceph-mon[37153]: pgmap v10649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:45 smithi047 ceph-mon[36835]: pgmap v10649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:46.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:45 smithi029 ceph-mon[32610]: pgmap v10649: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:47 smithi077 ceph-mon[37153]: pgmap v10650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19269.811631 secs 2024-07-27T12:10:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:47 smithi047 ceph-mon[36835]: pgmap v10650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19269.811631 secs 2024-07-27T12:10:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:47 smithi029 ceph-mon[32610]: pgmap v10650: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19269.811631 secs 2024-07-27T12:10:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:49 smithi077 ceph-mon[37153]: pgmap v10651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:49 smithi047 ceph-mon[36835]: pgmap v10651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:50.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:49 smithi029 ceph-mon[32610]: pgmap v10651: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:10:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:51 smithi077 ceph-mon[37153]: pgmap v10652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:51 smithi047 ceph-mon[36835]: pgmap v10652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:52.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:51 smithi029 ceph-mon[32610]: pgmap v10652: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:10:53.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19274.811759 secs 2024-07-27T12:10:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19274.811759 secs 2024-07-27T12:10:53.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:10:53.166+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:10:53.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:10:53.166+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:10:53.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:10:53.166+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:10:53.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19274.811759 secs 2024-07-27T12:10:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:53 smithi077 ceph-mon[37153]: pgmap v10653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:53 smithi047 ceph-mon[36835]: pgmap v10653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:54.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:53 smithi029 ceph-mon[32610]: pgmap v10653: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:10:55.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:54 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:10:55.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:54 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:10:55.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:54 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:10:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:56 smithi047 ceph-mon[36835]: pgmap v10654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:56 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:56.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:56 smithi029 ceph-mon[32610]: pgmap v10654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:56 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:56 smithi077 ceph-mon[37153]: pgmap v10654: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:10:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:56.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:56 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:57 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:57 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:57.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:57 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:58 smithi047 ceph-mon[36835]: pgmap v10655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19279.811874 secs 2024-07-27T12:10:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:10:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:10:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:58 smithi047 ceph-mon[36835]: pgmap v10656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T12:10:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:58 smithi047 ceph-mon[36835]: pgmap v10657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:10:58.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:10:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:58 smithi029 ceph-mon[32610]: pgmap v10655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19279.811874 secs 2024-07-27T12:10:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:10:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:10:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:58 smithi029 ceph-mon[32610]: pgmap v10656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T12:10:58.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:58 smithi029 ceph-mon[32610]: pgmap v10657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:10:58.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:10:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:10:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:58 smithi077 ceph-mon[37153]: pgmap v10655: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:10:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19279.811874 secs 2024-07-27T12:10:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:10:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:10:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:58 smithi077 ceph-mon[37153]: pgmap v10656: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T12:10:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:58 smithi077 ceph-mon[37153]: pgmap v10657: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:10:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:10:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:00.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:00.536+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:11:00.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:00.536+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:11:00.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:00.536+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:11:00.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:00.536+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:11:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:00 smithi077 ceph-mon[37153]: pgmap v10658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:00 smithi047 ceph-mon[36835]: pgmap v10658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:00.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:00 smithi029 ceph-mon[32610]: pgmap v10658: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:02 smithi077 ceph-mon[37153]: pgmap v10659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T12:11:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:02 smithi047 ceph-mon[36835]: pgmap v10659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T12:11:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:02 smithi029 ceph-mon[32610]: pgmap v10659: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T12:11:03.757 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19284.811998 secs 2024-07-27T12:11:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19284.811998 secs 2024-07-27T12:11:03.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19284.811998 secs 2024-07-27T12:11:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:04 smithi077 ceph-mon[37153]: pgmap v10660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:11:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:04 smithi047 ceph-mon[36835]: pgmap v10660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:11:04.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:04 smithi029 ceph-mon[32610]: pgmap v10660: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:11:06.754 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:06 smithi029 ceph-mon[32610]: pgmap v10661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:11:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:06 smithi077 ceph-mon[37153]: pgmap v10661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:11:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:06 smithi047 ceph-mon[36835]: pgmap v10661: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:11:08.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:08.177+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:11:08.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:08.177+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:11:08.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:08.177+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:11:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19289.812114 secs 2024-07-27T12:11:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:08 smithi077 ceph-mon[37153]: pgmap v10662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:11:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19289.812114 secs 2024-07-27T12:11:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:08 smithi047 ceph-mon[36835]: pgmap v10662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:11:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19289.812114 secs 2024-07-27T12:11:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:08 smithi029 ceph-mon[32610]: pgmap v10662: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:11:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:10 smithi077 ceph-mon[37153]: pgmap v10663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:10 smithi047 ceph-mon[36835]: pgmap v10663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:10.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:10 smithi029 ceph-mon[32610]: pgmap v10663: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:12 smithi077 ceph-mon[37153]: pgmap v10664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:12 smithi047 ceph-mon[36835]: pgmap v10664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:12.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:12 smithi029 ceph-mon[32610]: pgmap v10664: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19294.812239 secs 2024-07-27T12:11:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19294.812239 secs 2024-07-27T12:11:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19294.812239 secs 2024-07-27T12:11:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:14 smithi077 ceph-mon[37153]: pgmap v10665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:14 smithi047 ceph-mon[36835]: pgmap v10665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:14.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:14 smithi029 ceph-mon[32610]: pgmap v10665: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:16 smithi077 ceph-mon[37153]: pgmap v10666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:16 smithi047 ceph-mon[36835]: pgmap v10666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:16.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:16 smithi029 ceph-mon[32610]: pgmap v10666: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19299.812362 secs 2024-07-27T12:11:18.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:18 smithi077 ceph-mon[37153]: pgmap v10667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19299.812362 secs 2024-07-27T12:11:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:18 smithi047 ceph-mon[36835]: pgmap v10667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19299.812362 secs 2024-07-27T12:11:18.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:18 smithi029 ceph-mon[32610]: pgmap v10667: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:20 smithi077 ceph-mon[37153]: pgmap v10668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:20 smithi047 ceph-mon[36835]: pgmap v10668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:20.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:20 smithi029 ceph-mon[32610]: pgmap v10668: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:22 smithi077 ceph-mon[37153]: pgmap v10669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:22 smithi047 ceph-mon[36835]: pgmap v10669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:22.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:22 smithi029 ceph-mon[32610]: pgmap v10669: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:23.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:23.178+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:11:23.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:23.178+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:11:23.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:23.178+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:11:23.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19304.812483 secs 2024-07-27T12:11:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19304.812483 secs 2024-07-27T12:11:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19304.812483 secs 2024-07-27T12:11:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:24 smithi077 ceph-mon[37153]: pgmap v10670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:24.859 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:24 smithi047 ceph-mon[36835]: pgmap v10670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:24.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:24 smithi029 ceph-mon[32610]: pgmap v10670: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:25 smithi077 ceph-mon[37153]: pgmap v10671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:11:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:25 smithi047 ceph-mon[36835]: pgmap v10671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:11:25.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:25 smithi029 ceph-mon[32610]: pgmap v10671: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:11:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19309.812623 secs 2024-07-27T12:11:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:28 smithi077 ceph-mon[37153]: pgmap v10672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19309.812623 secs 2024-07-27T12:11:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:28 smithi047 ceph-mon[36835]: pgmap v10672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19309.812623 secs 2024-07-27T12:11:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:28 smithi029 ceph-mon[32610]: pgmap v10672: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:30 smithi077 ceph-mon[37153]: pgmap v10673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:30 smithi047 ceph-mon[36835]: pgmap v10673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:30.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:30 smithi029 ceph-mon[32610]: pgmap v10673: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:32 smithi077 ceph-mon[37153]: pgmap v10674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:32 smithi047 ceph-mon[36835]: pgmap v10674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:32.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:32 smithi029 ceph-mon[32610]: pgmap v10674: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19314.812767 secs 2024-07-27T12:11:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19314.812767 secs 2024-07-27T12:11:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19314.812767 secs 2024-07-27T12:11:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:34 smithi077 ceph-mon[37153]: pgmap v10675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:34 smithi047 ceph-mon[36835]: pgmap v10675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:34.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:34 smithi029 ceph-mon[32610]: pgmap v10675: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:36 smithi077 ceph-mon[37153]: pgmap v10676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:11:36.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:36 smithi029 ceph-mon[32610]: pgmap v10676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:11:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:36 smithi047 ceph-mon[36835]: pgmap v10676: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:11:38.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:38.190+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:11:38.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:38.190+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:11:38.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:38.190+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:11:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19319.812887 secs 2024-07-27T12:11:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:38 smithi077 ceph-mon[37153]: pgmap v10677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19319.812887 secs 2024-07-27T12:11:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:38 smithi047 ceph-mon[36835]: pgmap v10677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19319.812887 secs 2024-07-27T12:11:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:38 smithi029 ceph-mon[32610]: pgmap v10677: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:40 smithi077 ceph-mon[37153]: pgmap v10678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:40 smithi047 ceph-mon[36835]: pgmap v10678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:40.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:40 smithi029 ceph-mon[32610]: pgmap v10678: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:42 smithi077 ceph-mon[37153]: pgmap v10679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:42 smithi047 ceph-mon[36835]: pgmap v10679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:42.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:42 smithi029 ceph-mon[32610]: pgmap v10679: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19324.812989 secs 2024-07-27T12:11:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19324.812989 secs 2024-07-27T12:11:43.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19324.812989 secs 2024-07-27T12:11:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:44 smithi077 ceph-mon[37153]: pgmap v10680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:44 smithi047 ceph-mon[36835]: pgmap v10680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:44.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:44 smithi029 ceph-mon[32610]: pgmap v10680: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:45.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:45.541+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:11:45.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:45.541+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:11:45.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:45.544+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:11:45.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:45.544+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:11:46.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:46 smithi077 ceph-mon[37153]: pgmap v10681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:11:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:46 smithi047 ceph-mon[36835]: pgmap v10681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:11:46.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:46 smithi029 ceph-mon[32610]: pgmap v10681: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:11:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:47 smithi077 ceph-mon[37153]: pgmap v10682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:47 smithi047 ceph-mon[36835]: pgmap v10682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:47.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:47 smithi029 ceph-mon[32610]: pgmap v10682: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19329.813111 secs 2024-07-27T12:11:48.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19329.813111 secs 2024-07-27T12:11:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19329.813111 secs 2024-07-27T12:11:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:49 smithi047 ceph-mon[36835]: pgmap v10683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:49.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:49 smithi029 ceph-mon[32610]: pgmap v10683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:49 smithi077 ceph-mon[37153]: pgmap v10683: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:11:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:52 smithi077 ceph-mon[37153]: pgmap v10684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:52 smithi047 ceph-mon[36835]: pgmap v10684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:52.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:52 smithi029 ceph-mon[32610]: pgmap v10684: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:11:53.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:53.198+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:11:53.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:53.198+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:11:53.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:11:53.198+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:11:53.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19334.813230 secs 2024-07-27T12:11:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19334.813230 secs 2024-07-27T12:11:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19334.813230 secs 2024-07-27T12:11:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:54 smithi077 ceph-mon[37153]: pgmap v10685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:54 smithi047 ceph-mon[36835]: pgmap v10685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:54.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:54 smithi029 ceph-mon[32610]: pgmap v10685: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:11:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:56 smithi077 ceph-mon[37153]: pgmap v10686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:11:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:56 smithi047 ceph-mon[36835]: pgmap v10686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:11:56.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:56 smithi029 ceph-mon[32610]: pgmap v10686: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:11:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19339.813354 secs 2024-07-27T12:11:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:58 smithi077 ceph-mon[37153]: pgmap v10687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:11:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19339.813354 secs 2024-07-27T12:11:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:58 smithi047 ceph-mon[36835]: pgmap v10687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:11:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19339.813354 secs 2024-07-27T12:11:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:58 smithi029 ceph-mon[32610]: pgmap v10687: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:11:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:11:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:58.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:11:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:11:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:59 smithi077 ceph-mon[37153]: pgmap v10688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T12:11:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:59 smithi077 ceph-mon[37153]: pgmap v10689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T12:11:59.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:11:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:11:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:11:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:59 smithi047 ceph-mon[36835]: pgmap v10688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T12:11:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:59 smithi047 ceph-mon[36835]: pgmap v10689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T12:11:59.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:11:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:11:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:11:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:11:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:59 smithi029 ceph-mon[32610]: pgmap v10688: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-27T12:11:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:59 smithi029 ceph-mon[32610]: pgmap v10689: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T12:11:59.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:11:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:12:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:01 smithi047 ceph-mon[36835]: pgmap v10690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:12:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:01 smithi029 ceph-mon[32610]: pgmap v10690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:12:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:01 smithi077 ceph-mon[37153]: pgmap v10690: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:12:03.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19344.813496 secs 2024-07-27T12:12:03.909 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:03 smithi077 ceph-mon[37153]: pgmap v10691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T12:12:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19344.813496 secs 2024-07-27T12:12:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:03 smithi047 ceph-mon[36835]: pgmap v10691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T12:12:03.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19344.813496 secs 2024-07-27T12:12:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:03 smithi029 ceph-mon[32610]: pgmap v10691: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T12:12:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:05 smithi047 ceph-mon[36835]: pgmap v10692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T12:12:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:05 smithi029 ceph-mon[32610]: pgmap v10692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T12:12:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:05 smithi077 ceph-mon[37153]: pgmap v10692: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-27T12:12:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:07 smithi047 ceph-mon[36835]: pgmap v10693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T12:12:07.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:07 smithi029 ceph-mon[32610]: pgmap v10693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T12:12:08.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:07 smithi077 ceph-mon[37153]: pgmap v10693: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-27T12:12:08.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:12:08.212+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:12:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:12:08.212+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:12:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:12:08.212+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:12:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19349.813619 secs 2024-07-27T12:12:08.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19349.813619 secs 2024-07-27T12:12:09.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19349.813619 secs 2024-07-27T12:12:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:09 smithi047 ceph-mon[36835]: pgmap v10694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:12:09.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:09 smithi029 ceph-mon[32610]: pgmap v10694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:12:10.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:09 smithi077 ceph-mon[37153]: pgmap v10694: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:12:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:11 smithi047 ceph-mon[36835]: pgmap v10695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:12:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:11 smithi029 ceph-mon[32610]: pgmap v10695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:12:12.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:11 smithi077 ceph-mon[37153]: pgmap v10695: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:12:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19354.813736 secs 2024-07-27T12:12:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:13 smithi047 ceph-mon[36835]: pgmap v10696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19354.813736 secs 2024-07-27T12:12:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:13 smithi029 ceph-mon[32610]: pgmap v10696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19354.813736 secs 2024-07-27T12:12:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:13 smithi077 ceph-mon[37153]: pgmap v10696: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:15 smithi047 ceph-mon[36835]: pgmap v10697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:15.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:15 smithi029 ceph-mon[32610]: pgmap v10697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:16.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:15 smithi077 ceph-mon[37153]: pgmap v10697: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:17 smithi047 ceph-mon[36835]: pgmap v10698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:17.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:17 smithi029 ceph-mon[32610]: pgmap v10698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:17 smithi077 ceph-mon[37153]: pgmap v10698: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19359.813801 secs 2024-07-27T12:12:18.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19359.813801 secs 2024-07-27T12:12:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19359.813801 secs 2024-07-27T12:12:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:19 smithi047 ceph-mon[36835]: pgmap v10699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:12:19.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:19 smithi029 ceph-mon[32610]: pgmap v10699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:12:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:19 smithi077 ceph-mon[37153]: pgmap v10699: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:12:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:21 smithi047 ceph-mon[36835]: pgmap v10700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:12:21.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:21 smithi029 ceph-mon[32610]: pgmap v10700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:12:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:21 smithi077 ceph-mon[37153]: pgmap v10700: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:12:23.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:12:23.228+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:12:23.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:12:23.228+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:12:23.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:12:23.228+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:12:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19364.813916 secs 2024-07-27T12:12:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:23 smithi047 ceph-mon[36835]: pgmap v10701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19364.813916 secs 2024-07-27T12:12:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:23 smithi029 ceph-mon[32610]: pgmap v10701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19364.813916 secs 2024-07-27T12:12:24.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:23 smithi077 ceph-mon[37153]: pgmap v10701: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:25 smithi047 ceph-mon[36835]: pgmap v10702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:25.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:25 smithi029 ceph-mon[32610]: pgmap v10702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:25 smithi077 ceph-mon[37153]: pgmap v10702: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:27 smithi029 ceph-mon[32610]: pgmap v10703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19369.814036 secs 2024-07-27T12:12:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:27 smithi077 ceph-mon[37153]: pgmap v10703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:28.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19369.814036 secs 2024-07-27T12:12:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:27 smithi047 ceph-mon[36835]: pgmap v10703: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19369.814036 secs 2024-07-27T12:12:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:29 smithi029 ceph-mon[32610]: pgmap v10704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:12:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:29 smithi077 ceph-mon[37153]: pgmap v10704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:12:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:29 smithi047 ceph-mon[36835]: pgmap v10704: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:12:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:31 smithi029 ceph-mon[32610]: pgmap v10705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:31 smithi077 ceph-mon[37153]: pgmap v10705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:31 smithi047 ceph-mon[36835]: pgmap v10705: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:32.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19374.814153 secs 2024-07-27T12:12:33.007 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19374.814153 secs 2024-07-27T12:12:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19374.814153 secs 2024-07-27T12:12:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:33 smithi077 ceph-mon[37153]: pgmap v10706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:33 smithi047 ceph-mon[36835]: pgmap v10706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:33 smithi029 ceph-mon[32610]: pgmap v10706: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:35 smithi077 ceph-mon[37153]: pgmap v10707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:35 smithi047 ceph-mon[36835]: pgmap v10707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:36.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:35 smithi029 ceph-mon[32610]: pgmap v10707: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:37 smithi077 ceph-mon[37153]: pgmap v10708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:38.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19379.814277 secs 2024-07-27T12:12:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:37 smithi047 ceph-mon[36835]: pgmap v10708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19379.814277 secs 2024-07-27T12:12:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:37 smithi029 ceph-mon[32610]: pgmap v10708: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19379.814277 secs 2024-07-27T12:12:38.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:12:38.243+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:12:38.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:12:38.243+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:12:38.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:12:38.243+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:12:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:39 smithi077 ceph-mon[37153]: pgmap v10709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:12:40.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:39 smithi047 ceph-mon[36835]: pgmap v10709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:12:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:39 smithi029 ceph-mon[32610]: pgmap v10709: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:12:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:41 smithi077 ceph-mon[37153]: pgmap v10710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:12:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:41 smithi047 ceph-mon[36835]: pgmap v10710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:12:42.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:41 smithi029 ceph-mon[32610]: pgmap v10710: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:12:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19384.814420 secs 2024-07-27T12:12:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19384.814420 secs 2024-07-27T12:12:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19384.814420 secs 2024-07-27T12:12:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:43 smithi077 ceph-mon[37153]: pgmap v10711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:43 smithi047 ceph-mon[36835]: pgmap v10711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:43 smithi029 ceph-mon[32610]: pgmap v10711: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:45 smithi077 ceph-mon[37153]: pgmap v10712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:45 smithi047 ceph-mon[36835]: pgmap v10712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:46.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:45 smithi029 ceph-mon[32610]: pgmap v10712: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:48.014 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:47 smithi029 ceph-mon[32610]: pgmap v10713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:48.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19389.814542 secs 2024-07-27T12:12:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:47 smithi077 ceph-mon[37153]: pgmap v10713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:48.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19389.814542 secs 2024-07-27T12:12:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:47 smithi047 ceph-mon[36835]: pgmap v10713: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19389.814542 secs 2024-07-27T12:12:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:49 smithi077 ceph-mon[37153]: pgmap v10714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:12:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:49 smithi047 ceph-mon[36835]: pgmap v10714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:12:50.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:49 smithi029 ceph-mon[32610]: pgmap v10714: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:12:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:51 smithi077 ceph-mon[37153]: pgmap v10715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:12:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:51 smithi047 ceph-mon[36835]: pgmap v10715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:12:52.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:51 smithi029 ceph-mon[32610]: pgmap v10715: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:12:53.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19394.814658 secs 2024-07-27T12:12:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19394.814658 secs 2024-07-27T12:12:53.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19394.814658 secs 2024-07-27T12:12:53.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:12:53.247+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:12:53.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:12:53.247+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:12:53.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:12:53.247+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:12:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:53 smithi077 ceph-mon[37153]: pgmap v10716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:54.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:53 smithi029 ceph-mon[32610]: pgmap v10716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:53 smithi047 ceph-mon[36835]: pgmap v10716: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:12:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:55 smithi077 ceph-mon[37153]: pgmap v10717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:55 smithi047 ceph-mon[36835]: pgmap v10717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:56.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:55 smithi029 ceph-mon[32610]: pgmap v10717: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:12:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:57 smithi077 ceph-mon[37153]: pgmap v10718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19399.814837 secs 2024-07-27T12:12:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:57 smithi047 ceph-mon[36835]: pgmap v10718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19399.814837 secs 2024-07-27T12:12:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:57 smithi029 ceph-mon[32610]: pgmap v10718: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:12:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19399.814837 secs 2024-07-27T12:12:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:58 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:12:59.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:58 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:12:59.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:58 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:13:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:59 smithi077 ceph-mon[37153]: pgmap v10719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:13:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:13:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:13:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:59 smithi077 ceph-mon[37153]: pgmap v10720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:13:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:59 smithi077 ceph-mon[37153]: pgmap v10721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:13:00.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:12:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:13:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:59 smithi047 ceph-mon[36835]: pgmap v10719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:13:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:13:00.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:13:00.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:59 smithi047 ceph-mon[36835]: pgmap v10720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:13:00.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:59 smithi047 ceph-mon[36835]: pgmap v10721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:13:00.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:12:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:13:00.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:59 smithi029 ceph-mon[32610]: pgmap v10719: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:13:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:13:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:13:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:59 smithi029 ceph-mon[32610]: pgmap v10720: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:13:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:59 smithi029 ceph-mon[32610]: pgmap v10721: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:13:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:12:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:13:02.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:02 smithi029 ceph-mon[32610]: pgmap v10722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T12:13:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:02 smithi077 ceph-mon[37153]: pgmap v10722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T12:13:02.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:02 smithi047 ceph-mon[36835]: pgmap v10722: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T12:13:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19404.814954 secs 2024-07-27T12:13:03.508 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19404.814954 secs 2024-07-27T12:13:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19404.814954 secs 2024-07-27T12:13:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:04 smithi077 ceph-mon[37153]: pgmap v10723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:13:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:04 smithi047 ceph-mon[36835]: pgmap v10723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:13:04.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:04 smithi029 ceph-mon[32610]: pgmap v10723: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:13:06.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:06 smithi077 ceph-mon[37153]: pgmap v10724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:13:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:06 smithi047 ceph-mon[36835]: pgmap v10724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:13:06.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:06 smithi029 ceph-mon[32610]: pgmap v10724: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:13:08.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:13:08.261+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:13:08.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:13:08.261+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:13:08.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:13:08.261+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:13:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19409.815108 secs 2024-07-27T12:13:08.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:08 smithi077 ceph-mon[37153]: pgmap v10725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:13:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19409.815108 secs 2024-07-27T12:13:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:08 smithi047 ceph-mon[36835]: pgmap v10725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:13:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19409.815108 secs 2024-07-27T12:13:08.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:08 smithi029 ceph-mon[32610]: pgmap v10725: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:13:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:10 smithi029 ceph-mon[32610]: pgmap v10726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:13:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:10 smithi077 ceph-mon[37153]: pgmap v10726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:13:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:10 smithi047 ceph-mon[36835]: pgmap v10726: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:13:12.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:12 smithi077 ceph-mon[37153]: pgmap v10727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:12 smithi047 ceph-mon[36835]: pgmap v10727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:12.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:12 smithi029 ceph-mon[32610]: pgmap v10727: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:13.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19414.815257 secs 2024-07-27T12:13:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19414.815257 secs 2024-07-27T12:13:13.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19414.815257 secs 2024-07-27T12:13:14.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:14 smithi077 ceph-mon[37153]: pgmap v10728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:14 smithi047 ceph-mon[36835]: pgmap v10728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:14 smithi029 ceph-mon[32610]: pgmap v10728: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:16.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:16 smithi077 ceph-mon[37153]: pgmap v10729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:16.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:16 smithi047 ceph-mon[36835]: pgmap v10729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:16.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:16 smithi029 ceph-mon[32610]: pgmap v10729: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:18.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19419.815394 secs 2024-07-27T12:13:18.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:18 smithi029 ceph-mon[32610]: pgmap v10730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19419.815394 secs 2024-07-27T12:13:18.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:18 smithi077 ceph-mon[37153]: pgmap v10730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19419.815394 secs 2024-07-27T12:13:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:18 smithi047 ceph-mon[36835]: pgmap v10730: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:20.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:20 smithi077 ceph-mon[37153]: pgmap v10731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:13:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:20 smithi047 ceph-mon[36835]: pgmap v10731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:13:20.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:20 smithi029 ceph-mon[32610]: pgmap v10731: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:13:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:22 smithi077 ceph-mon[37153]: pgmap v10732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:22.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:22 smithi047 ceph-mon[36835]: pgmap v10732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:22.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:22 smithi029 ceph-mon[32610]: pgmap v10732: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:13:23.277+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:13:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:13:23.277+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:13:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:13:23.277+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:13:23.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19424.815531 secs 2024-07-27T12:13:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19424.815531 secs 2024-07-27T12:13:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19424.815531 secs 2024-07-27T12:13:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:24 smithi077 ceph-mon[37153]: pgmap v10733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:24.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:24 smithi047 ceph-mon[36835]: pgmap v10733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:24.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:24 smithi029 ceph-mon[32610]: pgmap v10733: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:26 smithi077 ceph-mon[37153]: pgmap v10734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:26.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:26 smithi047 ceph-mon[36835]: pgmap v10734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:26.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:26 smithi029 ceph-mon[32610]: pgmap v10734: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19429.815660 secs 2024-07-27T12:13:28.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:28 smithi077 ceph-mon[37153]: pgmap v10735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19429.815660 secs 2024-07-27T12:13:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:28 smithi047 ceph-mon[36835]: pgmap v10735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19429.815660 secs 2024-07-27T12:13:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:28 smithi029 ceph-mon[32610]: pgmap v10735: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:30 smithi077 ceph-mon[37153]: pgmap v10736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:13:30.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:30 smithi047 ceph-mon[36835]: pgmap v10736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:13:30.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:30 smithi029 ceph-mon[32610]: pgmap v10736: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:13:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:32 smithi077 ceph-mon[37153]: pgmap v10737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:32.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:32 smithi047 ceph-mon[36835]: pgmap v10737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:32.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:32 smithi029 ceph-mon[32610]: pgmap v10737: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:33.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19434.815813 secs 2024-07-27T12:13:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19434.815813 secs 2024-07-27T12:13:33.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19434.815813 secs 2024-07-27T12:13:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:34 smithi077 ceph-mon[37153]: pgmap v10738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:34.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:34 smithi047 ceph-mon[36835]: pgmap v10738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:34.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:34 smithi029 ceph-mon[32610]: pgmap v10738: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:36 smithi077 ceph-mon[37153]: pgmap v10739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:36.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:36 smithi047 ceph-mon[36835]: pgmap v10739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:36.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:36 smithi029 ceph-mon[32610]: pgmap v10739: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:38.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:13:38.279+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:13:38.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:13:38.279+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:13:38.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:13:38.279+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:13:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19439.815944 secs 2024-07-27T12:13:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:38 smithi077 ceph-mon[37153]: pgmap v10740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19439.815944 secs 2024-07-27T12:13:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:38 smithi047 ceph-mon[36835]: pgmap v10740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:38.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19439.815944 secs 2024-07-27T12:13:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:38 smithi029 ceph-mon[32610]: pgmap v10740: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:40 smithi077 ceph-mon[37153]: pgmap v10741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:13:40.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:40 smithi047 ceph-mon[36835]: pgmap v10741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:13:40.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:40 smithi029 ceph-mon[32610]: pgmap v10741: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:13:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:42 smithi077 ceph-mon[37153]: pgmap v10742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:42.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:42 smithi047 ceph-mon[36835]: pgmap v10742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:42.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:42 smithi029 ceph-mon[32610]: pgmap v10742: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19444.816075 secs 2024-07-27T12:13:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19444.816075 secs 2024-07-27T12:13:43.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19444.816075 secs 2024-07-27T12:13:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:44 smithi077 ceph-mon[37153]: pgmap v10743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:44.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:44 smithi047 ceph-mon[36835]: pgmap v10743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:44.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:44 smithi029 ceph-mon[32610]: pgmap v10743: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:46 smithi077 ceph-mon[37153]: pgmap v10744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:46.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:46 smithi047 ceph-mon[36835]: pgmap v10744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:46.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:46 smithi029 ceph-mon[32610]: pgmap v10744: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19449.816190 secs 2024-07-27T12:13:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:48 smithi077 ceph-mon[37153]: pgmap v10745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19449.816190 secs 2024-07-27T12:13:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:48 smithi047 ceph-mon[36835]: pgmap v10745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19449.816190 secs 2024-07-27T12:13:48.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:48 smithi029 ceph-mon[32610]: pgmap v10745: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:50 smithi077 ceph-mon[37153]: pgmap v10746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:13:50.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:50 smithi047 ceph-mon[36835]: pgmap v10746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:13:50.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:50 smithi029 ceph-mon[32610]: pgmap v10746: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:13:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:52 smithi077 ceph-mon[37153]: pgmap v10747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:52.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:52 smithi047 ceph-mon[36835]: pgmap v10747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:52.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:52 smithi029 ceph-mon[32610]: pgmap v10747: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:13:53.295+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:13:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:13:53.295+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:13:53.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:13:53.295+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:13:53.597 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19454.816258 secs 2024-07-27T12:13:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19454.816258 secs 2024-07-27T12:13:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19454.816258 secs 2024-07-27T12:13:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:54 smithi077 ceph-mon[37153]: pgmap v10748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:54.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:54 smithi047 ceph-mon[36835]: pgmap v10748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:54.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:54 smithi029 ceph-mon[32610]: pgmap v10748: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:13:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:56 smithi077 ceph-mon[37153]: pgmap v10749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:56.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:56 smithi047 ceph-mon[36835]: pgmap v10749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:56.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:56 smithi029 ceph-mon[32610]: pgmap v10749: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:13:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19459.816325 secs 2024-07-27T12:13:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:58 smithi077 ceph-mon[37153]: pgmap v10750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19459.816325 secs 2024-07-27T12:13:58.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:58 smithi047 ceph-mon[36835]: pgmap v10750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19459.816325 secs 2024-07-27T12:13:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:58 smithi029 ceph-mon[32610]: pgmap v10750: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:13:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:13:59 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:13:59.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:13:59 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:13:59.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:13:59 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:14:00.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:00.561+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:14:00.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:00.561+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:14:00.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:00.561+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:14:00.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:00.561+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:14:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:00 smithi077 ceph-mon[37153]: pgmap v10751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:14:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:14:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:14:00.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:14:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:00 smithi047 ceph-mon[36835]: pgmap v10751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:14:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:14:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:14:00.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:14:00.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:00 smithi029 ceph-mon[32610]: pgmap v10751: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:14:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:14:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:14:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:14:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:01 smithi077 ceph-mon[37153]: pgmap v10752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:14:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:01 smithi077 ceph-mon[37153]: pgmap v10753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:14:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:01 smithi047 ceph-mon[36835]: pgmap v10752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:14:01.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:01 smithi047 ceph-mon[36835]: pgmap v10753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:14:01.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:01 smithi029 ceph-mon[32610]: pgmap v10752: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:14:01.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:01 smithi029 ceph-mon[32610]: pgmap v10753: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:14:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:03 smithi077 ceph-mon[37153]: pgmap v10754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:14:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19464.816431 secs 2024-07-27T12:14:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:03 smithi047 ceph-mon[36835]: pgmap v10754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:14:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19464.816431 secs 2024-07-27T12:14:03.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:03 smithi029 ceph-mon[32610]: pgmap v10754: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:14:03.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19464.816431 secs 2024-07-27T12:14:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:05 smithi077 ceph-mon[37153]: pgmap v10755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:14:05.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:05 smithi047 ceph-mon[36835]: pgmap v10755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:14:05.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:05 smithi029 ceph-mon[32610]: pgmap v10755: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:14:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:07 smithi077 ceph-mon[37153]: pgmap v10756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:14:07.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:07 smithi047 ceph-mon[36835]: pgmap v10756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:14:07.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:07 smithi029 ceph-mon[32610]: pgmap v10756: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:14:08.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:08.309+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:14:08.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:08.309+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:14:08.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:08.309+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:14:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19469.816580 secs 2024-07-27T12:14:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19469.816580 secs 2024-07-27T12:14:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19469.816580 secs 2024-07-27T12:14:09.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:09 smithi047 ceph-mon[36835]: pgmap v10757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:14:09.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:09 smithi029 ceph-mon[32610]: pgmap v10757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:14:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:09 smithi077 ceph-mon[37153]: pgmap v10757: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:14:11.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:11 smithi047 ceph-mon[36835]: pgmap v10758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:14:11.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:11 smithi029 ceph-mon[32610]: pgmap v10758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:14:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:11 smithi077 ceph-mon[37153]: pgmap v10758: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:14:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:13 smithi047 ceph-mon[36835]: pgmap v10759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19474.816736 secs 2024-07-27T12:14:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:13 smithi029 ceph-mon[32610]: pgmap v10759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19474.816736 secs 2024-07-27T12:14:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:13 smithi077 ceph-mon[37153]: pgmap v10759: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19474.816736 secs 2024-07-27T12:14:15.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:15 smithi047 ceph-mon[36835]: pgmap v10760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:15.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:15 smithi029 ceph-mon[32610]: pgmap v10760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:15 smithi077 ceph-mon[37153]: pgmap v10760: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:17.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:17 smithi047 ceph-mon[36835]: pgmap v10761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:14:17.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:17 smithi029 ceph-mon[32610]: pgmap v10761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:14:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:17 smithi077 ceph-mon[37153]: pgmap v10761: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:14:18.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19479.816851 secs 2024-07-27T12:14:18.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19479.816851 secs 2024-07-27T12:14:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19479.816851 secs 2024-07-27T12:14:19.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:19 smithi047 ceph-mon[36835]: pgmap v10762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:19.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:19 smithi029 ceph-mon[32610]: pgmap v10762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:19 smithi077 ceph-mon[37153]: pgmap v10762: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:21.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:21 smithi047 ceph-mon[36835]: pgmap v10763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:21.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:21 smithi029 ceph-mon[32610]: pgmap v10763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:21 smithi077 ceph-mon[37153]: pgmap v10763: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:23.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:23.310+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:14:23.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:23.310+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:14:23.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:23.310+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:14:23.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:23 smithi047 ceph-mon[36835]: pgmap v10764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:23.698 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19484.816979 secs 2024-07-27T12:14:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:23 smithi029 ceph-mon[32610]: pgmap v10764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19484.816979 secs 2024-07-27T12:14:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:23 smithi077 ceph-mon[37153]: pgmap v10764: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19484.816979 secs 2024-07-27T12:14:25.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:25 smithi047 ceph-mon[36835]: pgmap v10765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:25 smithi029 ceph-mon[32610]: pgmap v10765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:25 smithi077 ceph-mon[37153]: pgmap v10765: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:27.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:27 smithi047 ceph-mon[36835]: pgmap v10766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:14:27.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:27 smithi029 ceph-mon[32610]: pgmap v10766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:14:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:27 smithi077 ceph-mon[37153]: pgmap v10766: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:14:28.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19489.817112 secs 2024-07-27T12:14:28.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19489.817112 secs 2024-07-27T12:14:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19489.817112 secs 2024-07-27T12:14:29.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:29 smithi047 ceph-mon[36835]: pgmap v10767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:29.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:29 smithi029 ceph-mon[32610]: pgmap v10767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:29 smithi077 ceph-mon[37153]: pgmap v10767: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:31.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:31 smithi047 ceph-mon[36835]: pgmap v10768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:31.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:31 smithi029 ceph-mon[32610]: pgmap v10768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:31 smithi077 ceph-mon[37153]: pgmap v10768: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:33.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:33 smithi077 ceph-mon[37153]: pgmap v10769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:33.659 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19494.817261 secs 2024-07-27T12:14:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:33 smithi047 ceph-mon[36835]: pgmap v10769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:33.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19494.817261 secs 2024-07-27T12:14:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:33 smithi029 ceph-mon[32610]: pgmap v10769: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:33.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19494.817261 secs 2024-07-27T12:14:35.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:35 smithi047 ceph-mon[36835]: pgmap v10770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:35.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:35 smithi029 ceph-mon[32610]: pgmap v10770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:35.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:35 smithi077 ceph-mon[37153]: pgmap v10770: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:37.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:37 smithi047 ceph-mon[36835]: pgmap v10771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:14:37.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:37 smithi029 ceph-mon[32610]: pgmap v10771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:14:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:37 smithi077 ceph-mon[37153]: pgmap v10771: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:14:38.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:38.326+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:14:38.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:38.326+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:14:38.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:38.326+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:14:38.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19499.817386 secs 2024-07-27T12:14:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19499.817386 secs 2024-07-27T12:14:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19499.817386 secs 2024-07-27T12:14:39.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:39 smithi047 ceph-mon[36835]: pgmap v10772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:39.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:39 smithi029 ceph-mon[32610]: pgmap v10772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:39 smithi077 ceph-mon[37153]: pgmap v10772: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:41.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:41 smithi047 ceph-mon[36835]: pgmap v10773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:41.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:41 smithi029 ceph-mon[32610]: pgmap v10773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:41 smithi077 ceph-mon[37153]: pgmap v10773: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:43.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:43 smithi029 ceph-mon[32610]: pgmap v10774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:43.515 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19504.817536 secs 2024-07-27T12:14:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:43 smithi047 ceph-mon[36835]: pgmap v10774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:43.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19504.817536 secs 2024-07-27T12:14:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:43 smithi077 ceph-mon[37153]: pgmap v10774: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19504.817536 secs 2024-07-27T12:14:45.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:45 smithi047 ceph-mon[36835]: pgmap v10775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:45.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:45 smithi029 ceph-mon[32610]: pgmap v10775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:45 smithi077 ceph-mon[37153]: pgmap v10775: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:47.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:47 smithi047 ceph-mon[36835]: pgmap v10776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:47.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:47 smithi029 ceph-mon[32610]: pgmap v10776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:47 smithi077 ceph-mon[37153]: pgmap v10776: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:48.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19509.817666 secs 2024-07-27T12:14:48.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19509.817666 secs 2024-07-27T12:14:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19509.817666 secs 2024-07-27T12:14:49.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:49 smithi047 ceph-mon[36835]: pgmap v10777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:49.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:49 smithi029 ceph-mon[32610]: pgmap v10777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:49 smithi077 ceph-mon[37153]: pgmap v10777: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:51.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:51 smithi047 ceph-mon[36835]: pgmap v10778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:51.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:51 smithi029 ceph-mon[32610]: pgmap v10778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:51 smithi077 ceph-mon[37153]: pgmap v10778: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:14:53.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:53.342+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:14:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:53.342+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:14:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:14:53.342+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:14:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:53 smithi047 ceph-mon[36835]: pgmap v10779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:53.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19514.817814 secs 2024-07-27T12:14:53.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:53 smithi029 ceph-mon[32610]: pgmap v10779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19514.817814 secs 2024-07-27T12:14:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:53 smithi077 ceph-mon[37153]: pgmap v10779: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:14:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19514.817814 secs 2024-07-27T12:14:55.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:55 smithi047 ceph-mon[36835]: pgmap v10780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:55.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:55 smithi029 ceph-mon[32610]: pgmap v10780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:55 smithi077 ceph-mon[37153]: pgmap v10780: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:14:57.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:57 smithi029 ceph-mon[32610]: pgmap v10781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:14:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:57 smithi077 ceph-mon[37153]: pgmap v10781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:14:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:57 smithi047 ceph-mon[36835]: pgmap v10781: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:14:58.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19519.817912 secs 2024-07-27T12:14:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19519.817912 secs 2024-07-27T12:14:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19519.817912 secs 2024-07-27T12:14:59.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:14:59 smithi029 ceph-mon[32610]: pgmap v10782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:14:59 smithi077 ceph-mon[37153]: pgmap v10782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:14:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:14:59 smithi047 ceph-mon[36835]: pgmap v10782: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:15:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:15:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:15:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:15:00.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:15:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:15:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:15:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:15:00.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:15:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:15:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:15:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:15:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:15:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:01 smithi077 ceph-mon[37153]: pgmap v10783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:15:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:01 smithi077 ceph-mon[37153]: pgmap v10784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T12:15:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:01 smithi077 ceph-mon[37153]: pgmap v10785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:15:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:01 smithi047 ceph-mon[36835]: pgmap v10783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:15:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:01 smithi047 ceph-mon[36835]: pgmap v10784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T12:15:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:01 smithi047 ceph-mon[36835]: pgmap v10785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:15:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:01 smithi029 ceph-mon[32610]: pgmap v10783: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:15:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:01 smithi029 ceph-mon[32610]: pgmap v10784: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T12:15:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:01 smithi029 ceph-mon[32610]: pgmap v10785: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:15:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19524.818073 secs 2024-07-27T12:15:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:03 smithi077 ceph-mon[37153]: pgmap v10786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:15:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19524.818073 secs 2024-07-27T12:15:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:03 smithi047 ceph-mon[36835]: pgmap v10786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:15:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19524.818073 secs 2024-07-27T12:15:03.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:03 smithi029 ceph-mon[32610]: pgmap v10786: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:15:05.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:05 smithi077 ceph-mon[37153]: pgmap v10787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:15:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:05 smithi047 ceph-mon[36835]: pgmap v10787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:15:05.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:05 smithi029 ceph-mon[32610]: pgmap v10787: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:15:07.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:07 smithi077 ceph-mon[37153]: pgmap v10788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:15:07.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:07 smithi047 ceph-mon[36835]: pgmap v10788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:15:07.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:07 smithi029 ceph-mon[32610]: pgmap v10788: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:15:08.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:15:08.356+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:15:08.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:15:08.356+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:15:08.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:15:08.356+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:15:08.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19529.818208 secs 2024-07-27T12:15:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19529.818208 secs 2024-07-27T12:15:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19529.818208 secs 2024-07-27T12:15:09.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:09 smithi077 ceph-mon[37153]: pgmap v10789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:15:09.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:09 smithi047 ceph-mon[36835]: pgmap v10789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:15:09.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:09 smithi029 ceph-mon[32610]: pgmap v10789: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:15:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:11 smithi077 ceph-mon[37153]: pgmap v10790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:15:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:11 smithi047 ceph-mon[36835]: pgmap v10790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:15:11.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:11 smithi029 ceph-mon[32610]: pgmap v10790: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:15:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19534.818357 secs 2024-07-27T12:15:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:13 smithi077 ceph-mon[37153]: pgmap v10791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19534.818357 secs 2024-07-27T12:15:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:13 smithi047 ceph-mon[36835]: pgmap v10791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19534.818357 secs 2024-07-27T12:15:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:13 smithi029 ceph-mon[32610]: pgmap v10791: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:15 smithi077 ceph-mon[37153]: pgmap v10792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:15 smithi047 ceph-mon[36835]: pgmap v10792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:15.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:15 smithi029 ceph-mon[32610]: pgmap v10792: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:17 smithi077 ceph-mon[37153]: pgmap v10793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:17 smithi047 ceph-mon[36835]: pgmap v10793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:17.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:17 smithi029 ceph-mon[32610]: pgmap v10793: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:18.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19539.818501 secs 2024-07-27T12:15:18.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19539.818501 secs 2024-07-27T12:15:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19539.818501 secs 2024-07-27T12:15:19.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:19 smithi077 ceph-mon[37153]: pgmap v10794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:19 smithi047 ceph-mon[36835]: pgmap v10794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:19.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:19 smithi029 ceph-mon[32610]: pgmap v10794: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:21.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:21 smithi077 ceph-mon[37153]: pgmap v10795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:15:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:21 smithi047 ceph-mon[36835]: pgmap v10795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:15:21.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:21 smithi029 ceph-mon[32610]: pgmap v10795: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:15:23.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:15:23.365+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:15:23.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:15:23.365+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:15:23.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:15:23.365+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:15:23.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19544.818632 secs 2024-07-27T12:15:23.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:23 smithi029 ceph-mon[32610]: pgmap v10796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19544.818632 secs 2024-07-27T12:15:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:23 smithi077 ceph-mon[37153]: pgmap v10796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19544.818632 secs 2024-07-27T12:15:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:23 smithi047 ceph-mon[36835]: pgmap v10796: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:25.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:25 smithi077 ceph-mon[37153]: pgmap v10797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:25 smithi047 ceph-mon[36835]: pgmap v10797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:25.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:25 smithi029 ceph-mon[32610]: pgmap v10797: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:27.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:27 smithi077 ceph-mon[37153]: pgmap v10798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:27 smithi047 ceph-mon[36835]: pgmap v10798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:27 smithi029 ceph-mon[32610]: pgmap v10798: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19549.818745 secs 2024-07-27T12:15:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19549.818745 secs 2024-07-27T12:15:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19549.818745 secs 2024-07-27T12:15:29.831 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:29 smithi029 ceph-mon[32610]: pgmap v10799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:29.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:29 smithi077 ceph-mon[37153]: pgmap v10799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:29 smithi047 ceph-mon[36835]: pgmap v10799: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:31.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:31 smithi077 ceph-mon[37153]: pgmap v10800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:15:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:31 smithi047 ceph-mon[36835]: pgmap v10800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:15:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:31 smithi029 ceph-mon[32610]: pgmap v10800: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:15:33.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19554.818876 secs 2024-07-27T12:15:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:33 smithi077 ceph-mon[37153]: pgmap v10801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19554.818876 secs 2024-07-27T12:15:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:33 smithi047 ceph-mon[36835]: pgmap v10801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:33.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19554.818876 secs 2024-07-27T12:15:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:33 smithi029 ceph-mon[32610]: pgmap v10801: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:35.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:35 smithi077 ceph-mon[37153]: pgmap v10802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:35 smithi047 ceph-mon[36835]: pgmap v10802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:35.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:35 smithi029 ceph-mon[32610]: pgmap v10802: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:37.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:37 smithi077 ceph-mon[37153]: pgmap v10803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:37 smithi047 ceph-mon[36835]: pgmap v10803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:37.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:37 smithi029 ceph-mon[32610]: pgmap v10803: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:38.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:15:38.381+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:15:38.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:15:38.381+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:15:38.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:15:38.381+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:15:38.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19559.819016 secs 2024-07-27T12:15:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19559.819016 secs 2024-07-27T12:15:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19559.819016 secs 2024-07-27T12:15:39.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:39 smithi077 ceph-mon[37153]: pgmap v10804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:39 smithi047 ceph-mon[36835]: pgmap v10804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:39 smithi029 ceph-mon[32610]: pgmap v10804: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:41.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:41 smithi077 ceph-mon[37153]: pgmap v10805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:15:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:41 smithi047 ceph-mon[36835]: pgmap v10805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:15:41.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:41 smithi029 ceph-mon[32610]: pgmap v10805: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:15:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19564.819099 secs 2024-07-27T12:15:43.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:43 smithi077 ceph-mon[37153]: pgmap v10806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19564.819099 secs 2024-07-27T12:15:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:43 smithi047 ceph-mon[36835]: pgmap v10806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:43.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19564.819099 secs 2024-07-27T12:15:43.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:43 smithi029 ceph-mon[32610]: pgmap v10806: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:45.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:45 smithi077 ceph-mon[37153]: pgmap v10807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:45 smithi047 ceph-mon[36835]: pgmap v10807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:45.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:45 smithi029 ceph-mon[32610]: pgmap v10807: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:47.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:47 smithi077 ceph-mon[37153]: pgmap v10808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:47 smithi047 ceph-mon[36835]: pgmap v10808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:47.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:47 smithi029 ceph-mon[32610]: pgmap v10808: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:48.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19569.819183 secs 2024-07-27T12:15:48.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19569.819183 secs 2024-07-27T12:15:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19569.819183 secs 2024-07-27T12:15:49.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:49 smithi077 ceph-mon[37153]: pgmap v10809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:49 smithi047 ceph-mon[36835]: pgmap v10809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:49.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:49 smithi029 ceph-mon[32610]: pgmap v10809: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:51.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:51 smithi077 ceph-mon[37153]: pgmap v10810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:15:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:51 smithi047 ceph-mon[36835]: pgmap v10810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:15:51.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:51 smithi029 ceph-mon[32610]: pgmap v10810: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:15:53.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:15:53.386+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:15:53.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:15:53.386+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:15:53.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:15:53.386+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:15:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19574.819335 secs 2024-07-27T12:15:53.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:53 smithi029 ceph-mon[32610]: pgmap v10811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:53.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19574.819335 secs 2024-07-27T12:15:53.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:53 smithi077 ceph-mon[37153]: pgmap v10811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19574.819335 secs 2024-07-27T12:15:53.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:53 smithi047 ceph-mon[36835]: pgmap v10811: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:55.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:55 smithi077 ceph-mon[37153]: pgmap v10812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:55 smithi047 ceph-mon[36835]: pgmap v10812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:55.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:55 smithi029 ceph-mon[32610]: pgmap v10812: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:15:57.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:57 smithi077 ceph-mon[37153]: pgmap v10813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:57 smithi047 ceph-mon[36835]: pgmap v10813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:57.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:57 smithi029 ceph-mon[32610]: pgmap v10813: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:15:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19579.819458 secs 2024-07-27T12:15:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19579.819458 secs 2024-07-27T12:15:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19579.819458 secs 2024-07-27T12:15:59.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:15:59 smithi077 ceph-mon[37153]: pgmap v10814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:15:59 smithi047 ceph-mon[36835]: pgmap v10814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:15:59.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:15:59 smithi029 ceph-mon[32610]: pgmap v10814: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:00 smithi077 ceph-mon[37153]: pgmap v10815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:16:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:00 smithi047 ceph-mon[36835]: pgmap v10815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:16:00.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:00 smithi029 ceph-mon[32610]: pgmap v10815: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:16:01.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:16:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:16:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:01 smithi077 ceph-mon[37153]: pgmap v10816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T12:16:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:01 smithi077 ceph-mon[37153]: pgmap v10817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T12:16:01.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:01 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:16:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:16:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:16:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:01 smithi047 ceph-mon[36835]: pgmap v10816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T12:16:01.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:01 smithi047 ceph-mon[36835]: pgmap v10817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T12:16:01.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:01 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:16:01.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:16:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:16:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:01 smithi029 ceph-mon[32610]: pgmap v10816: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-27T12:16:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:01 smithi029 ceph-mon[32610]: pgmap v10817: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-27T12:16:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:01 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:16:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19584.819593 secs 2024-07-27T12:16:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19584.819593 secs 2024-07-27T12:16:03.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19584.819593 secs 2024-07-27T12:16:04.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:03 smithi077 ceph-mon[37153]: pgmap v10818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:16:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:03 smithi047 ceph-mon[36835]: pgmap v10818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:16:04.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:03 smithi029 ceph-mon[32610]: pgmap v10818: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:16:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:05 smithi047 ceph-mon[36835]: pgmap v10819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:16:06.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:05 smithi029 ceph-mon[32610]: pgmap v10819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:16:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:05 smithi077 ceph-mon[37153]: pgmap v10819: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:16:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:07 smithi047 ceph-mon[36835]: pgmap v10820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T12:16:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19589.819734 secs 2024-07-27T12:16:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:07 smithi029 ceph-mon[32610]: pgmap v10820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T12:16:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19589.819734 secs 2024-07-27T12:16:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:07 smithi077 ceph-mon[37153]: pgmap v10820: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T12:16:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19589.819734 secs 2024-07-27T12:16:08.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:16:08.397+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:16:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:16:08.397+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:16:08.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:16:08.397+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:16:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:09 smithi047 ceph-mon[36835]: pgmap v10821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:10.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:09 smithi029 ceph-mon[32610]: pgmap v10821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:09 smithi077 ceph-mon[37153]: pgmap v10821: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:12.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:11 smithi047 ceph-mon[36835]: pgmap v10822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T12:16:12.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:11 smithi029 ceph-mon[32610]: pgmap v10822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T12:16:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:11 smithi077 ceph-mon[37153]: pgmap v10822: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T12:16:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19594.819872 secs 2024-07-27T12:16:13.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19594.819872 secs 2024-07-27T12:16:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19594.819872 secs 2024-07-27T12:16:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:13 smithi047 ceph-mon[36835]: pgmap v10823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:16:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:13 smithi029 ceph-mon[32610]: pgmap v10823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:16:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:13 smithi077 ceph-mon[37153]: pgmap v10823: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:16:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:15 smithi047 ceph-mon[36835]: pgmap v10824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:16.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:15 smithi029 ceph-mon[32610]: pgmap v10824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:15 smithi077 ceph-mon[37153]: pgmap v10824: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:17 smithi047 ceph-mon[36835]: pgmap v10825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19599.820007 secs 2024-07-27T12:16:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:17 smithi029 ceph-mon[32610]: pgmap v10825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:18.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19599.820007 secs 2024-07-27T12:16:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:17 smithi077 ceph-mon[37153]: pgmap v10825: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19599.820007 secs 2024-07-27T12:16:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:19 smithi047 ceph-mon[36835]: pgmap v10826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:20.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:19 smithi029 ceph-mon[32610]: pgmap v10826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:19 smithi077 ceph-mon[37153]: pgmap v10826: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:21 smithi047 ceph-mon[36835]: pgmap v10827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:22.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:21 smithi029 ceph-mon[32610]: pgmap v10827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:21 smithi077 ceph-mon[37153]: pgmap v10827: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19604.820138 secs 2024-07-27T12:16:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19604.820138 secs 2024-07-27T12:16:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19604.820138 secs 2024-07-27T12:16:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:16:23.405+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:16:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:16:23.405+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:16:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:16:23.405+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:16:24.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:23 smithi047 ceph-mon[36835]: pgmap v10828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:16:24.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:23 smithi029 ceph-mon[32610]: pgmap v10828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:16:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:23 smithi077 ceph-mon[37153]: pgmap v10828: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:16:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:25 smithi047 ceph-mon[36835]: pgmap v10829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:26.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:25 smithi029 ceph-mon[32610]: pgmap v10829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:26.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:25 smithi077 ceph-mon[37153]: pgmap v10829: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:27 smithi047 ceph-mon[36835]: pgmap v10830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19609.820234 secs 2024-07-27T12:16:28.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:27 smithi029 ceph-mon[32610]: pgmap v10830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19609.820234 secs 2024-07-27T12:16:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:27 smithi077 ceph-mon[37153]: pgmap v10830: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19609.820234 secs 2024-07-27T12:16:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:29 smithi047 ceph-mon[36835]: pgmap v10831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:30.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:29 smithi029 ceph-mon[32610]: pgmap v10831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:29 smithi077 ceph-mon[37153]: pgmap v10831: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:31 smithi047 ceph-mon[36835]: pgmap v10832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:32.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:31 smithi029 ceph-mon[32610]: pgmap v10832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:31 smithi077 ceph-mon[37153]: pgmap v10832: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19614.820359 secs 2024-07-27T12:16:33.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19614.820359 secs 2024-07-27T12:16:33.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19614.820359 secs 2024-07-27T12:16:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:33 smithi047 ceph-mon[36835]: pgmap v10833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:16:34.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:33 smithi029 ceph-mon[32610]: pgmap v10833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:16:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:33 smithi077 ceph-mon[37153]: pgmap v10833: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:16:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:35 smithi047 ceph-mon[36835]: pgmap v10834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:36.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:35 smithi029 ceph-mon[32610]: pgmap v10834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:35 smithi077 ceph-mon[37153]: pgmap v10834: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:37 smithi047 ceph-mon[36835]: pgmap v10835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19619.820478 secs 2024-07-27T12:16:38.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:37 smithi029 ceph-mon[32610]: pgmap v10835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19619.820478 secs 2024-07-27T12:16:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:37 smithi077 ceph-mon[37153]: pgmap v10835: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19619.820478 secs 2024-07-27T12:16:38.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:16:38.421+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:16:38.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:16:38.421+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:16:38.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:16:38.421+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:16:40.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:39 smithi029 ceph-mon[32610]: pgmap v10836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:39 smithi077 ceph-mon[37153]: pgmap v10836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:39 smithi047 ceph-mon[36835]: pgmap v10836: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:16:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:41 smithi029 ceph-mon[32610]: pgmap v10837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:41 smithi077 ceph-mon[37153]: pgmap v10837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:41 smithi047 ceph-mon[36835]: pgmap v10837: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19624.820606 secs 2024-07-27T12:16:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19624.820606 secs 2024-07-27T12:16:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19624.820606 secs 2024-07-27T12:16:44.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:43 smithi029 ceph-mon[32610]: pgmap v10838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:16:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:43 smithi077 ceph-mon[37153]: pgmap v10838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:16:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:43 smithi047 ceph-mon[36835]: pgmap v10838: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:16:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:45 smithi077 ceph-mon[37153]: pgmap v10839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:16:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:45 smithi047 ceph-mon[36835]: pgmap v10839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:16:46.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:45 smithi029 ceph-mon[32610]: pgmap v10839: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-27T12:16:48.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:47 smithi029 ceph-mon[32610]: pgmap v10840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:48.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19629.820726 secs 2024-07-27T12:16:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:47 smithi077 ceph-mon[37153]: pgmap v10840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:48.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19629.820726 secs 2024-07-27T12:16:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:47 smithi047 ceph-mon[36835]: pgmap v10840: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19629.820726 secs 2024-07-27T12:16:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:49 smithi077 ceph-mon[37153]: pgmap v10841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:16:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:49 smithi047 ceph-mon[36835]: pgmap v10841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:16:50.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:49 smithi029 ceph-mon[32610]: pgmap v10841: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:16:52.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:51 smithi077 ceph-mon[37153]: pgmap v10842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T12:16:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:51 smithi047 ceph-mon[36835]: pgmap v10842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T12:16:52.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:51 smithi029 ceph-mon[32610]: pgmap v10842: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T12:16:53.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19634.820852 secs 2024-07-27T12:16:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19634.820852 secs 2024-07-27T12:16:53.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:16:53.437+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:16:53.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:16:53.437+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:16:53.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:16:53.437+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:16:53.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19634.820852 secs 2024-07-27T12:16:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:53 smithi077 ceph-mon[37153]: pgmap v10843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:16:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:53 smithi047 ceph-mon[36835]: pgmap v10843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:16:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:53 smithi029 ceph-mon[32610]: pgmap v10843: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:16:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:55 smithi077 ceph-mon[37153]: pgmap v10844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:55 smithi047 ceph-mon[36835]: pgmap v10844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:56.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:55 smithi029 ceph-mon[32610]: pgmap v10844: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:16:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:57 smithi077 ceph-mon[37153]: pgmap v10845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19639.820981 secs 2024-07-27T12:16:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:57 smithi047 ceph-mon[36835]: pgmap v10845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19639.820981 secs 2024-07-27T12:16:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:57 smithi029 ceph-mon[32610]: pgmap v10845: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:16:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19639.820981 secs 2024-07-27T12:17:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:16:59 smithi077 ceph-mon[37153]: pgmap v10846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:17:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:16:59 smithi047 ceph-mon[36835]: pgmap v10846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:17:00.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:16:59 smithi029 ceph-mon[32610]: pgmap v10846: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:17:00.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:00.585+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:17:00.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:00.585+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:17:00.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:00.586+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 153 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:17:00.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:00.586+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 153 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:17:01.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:00 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:17:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:00 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:17:01.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:00 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:17:02.061 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:01 smithi047 ceph-mon[36835]: pgmap v10847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:02.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:01 smithi029 ceph-mon[32610]: pgmap v10847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:01 smithi077 ceph-mon[37153]: pgmap v10847: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:03.154 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19644.821097 secs 2024-07-27T12:17:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19644.821097 secs 2024-07-27T12:17:03.257 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19644.821097 secs 2024-07-27T12:17:04.289 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:04 smithi029 ceph-mon[32610]: pgmap v10848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:17:04.295 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:04 smithi047 ceph-mon[36835]: pgmap v10848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:17:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:04 smithi077 ceph-mon[37153]: pgmap v10848: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:17:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:05.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: pgmap v10849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:17:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:17:05.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:17:05.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:17:05.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T12:17:05.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T12:17:05.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:17:05.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:17:05.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: pgmap v10850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T12:17:05.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: pgmap v10851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T12:17:05.948 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: pgmap v10849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:05.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:05.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:05.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:17:05.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:17:05.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:17:05.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:17:05.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T12:17:05.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T12:17:05.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:17:05.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:17:05.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: pgmap v10850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T12:17:05.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: pgmap v10851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T12:17:05.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:06.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: pgmap v10849: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:17:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:17:06.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:17:06.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:17:06.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi029 to 177.1M 2024-07-27T12:17:06.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi029 to 185705676: error parsing value: Value '185705676' is below minimum 939524096 2024-07-27T12:17:06.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:17:06.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:17:06.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: pgmap v10850: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T12:17:06.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: pgmap v10851: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-27T12:17:06.109 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:17:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:08 smithi077 ceph-mon[37153]: pgmap v10852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-27T12:17:08.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19649.821224 secs 2024-07-27T12:17:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:08 smithi047 ceph-mon[36835]: pgmap v10852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-27T12:17:08.450 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19649.821224 secs 2024-07-27T12:17:08.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:08 smithi029 ceph-mon[32610]: pgmap v10852: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-27T12:17:08.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19649.821224 secs 2024-07-27T12:17:08.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:08.451+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:17:08.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:08.451+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:17:08.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:08.451+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:17:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:10 smithi077 ceph-mon[37153]: pgmap v10853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 7 op/s 2024-07-27T12:17:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:10 smithi047 ceph-mon[36835]: pgmap v10853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 7 op/s 2024-07-27T12:17:10.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:10 smithi029 ceph-mon[32610]: pgmap v10853: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 7 op/s 2024-07-27T12:17:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:12 smithi077 ceph-mon[37153]: pgmap v10854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-27T12:17:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:12 smithi047 ceph-mon[36835]: pgmap v10854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-27T12:17:12.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:12 smithi029 ceph-mon[32610]: pgmap v10854: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-27T12:17:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19654.821350 secs 2024-07-27T12:17:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19654.821350 secs 2024-07-27T12:17:13.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19654.821350 secs 2024-07-27T12:17:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:14 smithi077 ceph-mon[37153]: pgmap v10855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T12:17:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:14 smithi047 ceph-mon[36835]: pgmap v10855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T12:17:14.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:14 smithi029 ceph-mon[32610]: pgmap v10855: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-27T12:17:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:16 smithi077 ceph-mon[37153]: pgmap v10856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T12:17:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:16 smithi047 ceph-mon[36835]: pgmap v10856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T12:17:16.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:16 smithi029 ceph-mon[32610]: pgmap v10856: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-27T12:17:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:18 smithi077 ceph-mon[37153]: pgmap v10857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19659.821477 secs 2024-07-27T12:17:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:18 smithi047 ceph-mon[36835]: pgmap v10857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:18.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19659.821477 secs 2024-07-27T12:17:18.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:18 smithi029 ceph-mon[32610]: pgmap v10857: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:18.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19659.821477 secs 2024-07-27T12:17:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:20 smithi077 ceph-mon[37153]: pgmap v10858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:17:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:20 smithi047 ceph-mon[36835]: pgmap v10858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:17:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:20 smithi029 ceph-mon[32610]: pgmap v10858: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:17:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:22 smithi047 ceph-mon[36835]: pgmap v10859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T12:17:22.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:22 smithi029 ceph-mon[32610]: pgmap v10859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T12:17:22.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:22 smithi077 ceph-mon[37153]: pgmap v10859: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-27T12:17:23.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19664.821547 secs 2024-07-27T12:17:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19664.821547 secs 2024-07-27T12:17:23.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:23.466+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:17:23.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:23.466+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:17:23.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:23.466+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:17:23.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19664.821547 secs 2024-07-27T12:17:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:24 smithi047 ceph-mon[36835]: pgmap v10860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:17:24.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:24 smithi029 ceph-mon[32610]: pgmap v10860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:17:24.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:24 smithi077 ceph-mon[37153]: pgmap v10860: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-27T12:17:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:26 smithi047 ceph-mon[36835]: pgmap v10861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:26.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:26 smithi029 ceph-mon[32610]: pgmap v10861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:26.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:26 smithi077 ceph-mon[37153]: pgmap v10861: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:28 smithi047 ceph-mon[36835]: pgmap v10862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19669.821674 secs 2024-07-27T12:17:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:28 smithi029 ceph-mon[32610]: pgmap v10862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:28.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19669.821674 secs 2024-07-27T12:17:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:28 smithi077 ceph-mon[37153]: pgmap v10862: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19669.821674 secs 2024-07-27T12:17:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:30 smithi047 ceph-mon[36835]: pgmap v10863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:17:30.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:30 smithi029 ceph-mon[32610]: pgmap v10863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:17:30.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:30 smithi077 ceph-mon[37153]: pgmap v10863: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:17:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:32 smithi047 ceph-mon[36835]: pgmap v10864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:32.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:32 smithi029 ceph-mon[32610]: pgmap v10864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:32.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:32 smithi077 ceph-mon[37153]: pgmap v10864: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19674.821826 secs 2024-07-27T12:17:33.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19674.821826 secs 2024-07-27T12:17:33.507 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19674.821826 secs 2024-07-27T12:17:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:34 smithi047 ceph-mon[36835]: pgmap v10865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:17:34.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:34 smithi029 ceph-mon[32610]: pgmap v10865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:17:34.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:34 smithi077 ceph-mon[37153]: pgmap v10865: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:17:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:36 smithi047 ceph-mon[36835]: pgmap v10866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:36.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:36 smithi029 ceph-mon[32610]: pgmap v10866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:36.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:36 smithi077 ceph-mon[37153]: pgmap v10866: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:38 smithi047 ceph-mon[36835]: pgmap v10867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19679.821895 secs 2024-07-27T12:17:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:38 smithi029 ceph-mon[32610]: pgmap v10867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:38.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19679.821895 secs 2024-07-27T12:17:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:38.471+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:17:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:38.471+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:17:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:38.471+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:17:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:38 smithi077 ceph-mon[37153]: pgmap v10867: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:38.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19679.821895 secs 2024-07-27T12:17:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:40 smithi047 ceph-mon[36835]: pgmap v10868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:17:40.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:40 smithi029 ceph-mon[32610]: pgmap v10868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:17:40.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:40 smithi077 ceph-mon[37153]: pgmap v10868: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:17:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:42 smithi047 ceph-mon[36835]: pgmap v10869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:42.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:42 smithi029 ceph-mon[32610]: pgmap v10869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:42.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:42 smithi077 ceph-mon[37153]: pgmap v10869: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19684.822018 secs 2024-07-27T12:17:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19684.822018 secs 2024-07-27T12:17:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19684.822018 secs 2024-07-27T12:17:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:44 smithi047 ceph-mon[36835]: pgmap v10870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:17:44.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:44 smithi029 ceph-mon[32610]: pgmap v10870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:17:44.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:44 smithi077 ceph-mon[37153]: pgmap v10870: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:17:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:46 smithi047 ceph-mon[36835]: pgmap v10871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:46.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:46 smithi029 ceph-mon[32610]: pgmap v10871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:46.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:46 smithi077 ceph-mon[37153]: pgmap v10871: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:48 smithi047 ceph-mon[36835]: pgmap v10872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19689.822143 secs 2024-07-27T12:17:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:48 smithi029 ceph-mon[32610]: pgmap v10872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19689.822143 secs 2024-07-27T12:17:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:48 smithi077 ceph-mon[37153]: pgmap v10872: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:48.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19689.822143 secs 2024-07-27T12:17:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:50 smithi047 ceph-mon[36835]: pgmap v10873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:17:50.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:50 smithi029 ceph-mon[32610]: pgmap v10873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:17:50.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:50 smithi077 ceph-mon[37153]: pgmap v10873: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:17:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:52 smithi047 ceph-mon[36835]: pgmap v10874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:52.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:52 smithi029 ceph-mon[32610]: pgmap v10874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:52.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:52 smithi077 ceph-mon[37153]: pgmap v10874: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19694.822244 secs 2024-07-27T12:17:53.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19694.822244 secs 2024-07-27T12:17:53.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:53.486+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:17:53.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:53.486+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:17:53.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:17:53.486+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:17:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19694.822244 secs 2024-07-27T12:17:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:54 smithi047 ceph-mon[36835]: pgmap v10875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:17:54.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:54 smithi029 ceph-mon[32610]: pgmap v10875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:17:54.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:54 smithi077 ceph-mon[37153]: pgmap v10875: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:17:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:56 smithi047 ceph-mon[36835]: pgmap v10876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:56.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:56 smithi029 ceph-mon[32610]: pgmap v10876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:56.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:56 smithi077 ceph-mon[37153]: pgmap v10876: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:17:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:58 smithi047 ceph-mon[36835]: pgmap v10877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:17:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19699.822366 secs 2024-07-27T12:17:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:58 smithi029 ceph-mon[32610]: pgmap v10877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:17:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19699.822366 secs 2024-07-27T12:17:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:58 smithi077 ceph-mon[37153]: pgmap v10877: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:17:58.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:17:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19699.822366 secs 2024-07-27T12:18:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:00 smithi047 ceph-mon[36835]: pgmap v10878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:00.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:00 smithi029 ceph-mon[32610]: pgmap v10878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:00.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:00 smithi077 ceph-mon[37153]: pgmap v10878: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:02 smithi047 ceph-mon[36835]: pgmap v10879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:02.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:02 smithi029 ceph-mon[32610]: pgmap v10879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:02.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:02 smithi077 ceph-mon[37153]: pgmap v10879: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19704.822495 secs 2024-07-27T12:18:03.507 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19704.822495 secs 2024-07-27T12:18:03.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19704.822495 secs 2024-07-27T12:18:04.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:04 smithi029 ceph-mon[32610]: pgmap v10880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:04.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:04 smithi077 ceph-mon[37153]: pgmap v10880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:04.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:04 smithi047 ceph-mon[36835]: pgmap v10880: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:05.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:05 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:18:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:05 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:18:05.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:05 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:18:06.237 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:06 smithi077 ceph-mon[37153]: pgmap v10881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:06.237 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:06.237 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:06.237 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:06.237 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:06 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:06.238 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:06 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T12:18:06.238 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:06 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T12:18:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:06 smithi029 ceph-mon[32610]: pgmap v10881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:06.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:06.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:06 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:06.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:06 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T12:18:06.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:06 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T12:18:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:06 smithi047 ceph-mon[36835]: pgmap v10881: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:06.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:06 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:06 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi047 to 177.2M 2024-07-27T12:18:06.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:06 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi047 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-07-27T12:18:08.438 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:08 smithi077 ceph-mon[37153]: pgmap v10882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:18:08.438 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19709.822591 secs 2024-07-27T12:18:08.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:08 smithi029 ceph-mon[32610]: pgmap v10882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:18:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19709.822591 secs 2024-07-27T12:18:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:18:08.500+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:18:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:18:08.500+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:18:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:18:08.500+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:18:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:08 smithi047 ceph-mon[36835]: pgmap v10882: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:18:08.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19709.822591 secs 2024-07-27T12:18:10.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:18:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:18:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T12:18:10.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T12:18:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:18:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:18:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: pgmap v10883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: pgmap v10884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T12:18:10.198 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:18:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:18:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:18:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:10.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:10.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:10.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:10.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T12:18:10.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T12:18:10.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:18:10.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:18:10.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: pgmap v10883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:10.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: pgmap v10884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T12:18:10.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:18:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:18:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:18:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-27T12:18:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: Adjusting osd_memory_target on smithi077 to 176.2M 2024-07-27T12:18:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: Unable to set osd_memory_target on smithi077 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-07-27T12:18:10.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:18:10.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:18:10.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: pgmap v10883: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:10.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: pgmap v10884: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T12:18:10.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:18:12.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:11 smithi029 ceph-mon[32610]: pgmap v10885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T12:18:12.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:11 smithi077 ceph-mon[37153]: pgmap v10885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T12:18:12.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:11 smithi047 ceph-mon[36835]: pgmap v10885: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T12:18:13.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:12 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19714.822712 secs 2024-07-27T12:18:13.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:12 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19714.822712 secs 2024-07-27T12:18:13.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:12 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19714.822712 secs 2024-07-27T12:18:14.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:13 smithi077 ceph-mon[37153]: pgmap v10886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T12:18:14.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:13 smithi047 ceph-mon[36835]: pgmap v10886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T12:18:14.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:13 smithi029 ceph-mon[32610]: pgmap v10886: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T12:18:16.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:15 smithi077 ceph-mon[37153]: pgmap v10887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T12:18:16.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:15 smithi047 ceph-mon[36835]: pgmap v10887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T12:18:16.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:15 smithi029 ceph-mon[32610]: pgmap v10887: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-27T12:18:18.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:17 smithi029 ceph-mon[32610]: pgmap v10888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:18:18.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19719.822840 secs 2024-07-27T12:18:18.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:17 smithi077 ceph-mon[37153]: pgmap v10888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:18:18.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19719.822840 secs 2024-07-27T12:18:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:17 smithi047 ceph-mon[36835]: pgmap v10888: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:18:18.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19719.822840 secs 2024-07-27T12:18:20.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:19 smithi077 ceph-mon[37153]: pgmap v10889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:18:20.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:19 smithi047 ceph-mon[36835]: pgmap v10889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:18:20.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:19 smithi029 ceph-mon[32610]: pgmap v10889: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:18:22.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:21 smithi077 ceph-mon[37153]: pgmap v10890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:22.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:21 smithi047 ceph-mon[36835]: pgmap v10890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:22.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:21 smithi029 ceph-mon[32610]: pgmap v10890: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:23.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19724.822969 secs 2024-07-27T12:18:23.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19724.822969 secs 2024-07-27T12:18:23.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19724.822969 secs 2024-07-27T12:18:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:18:23.516+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:18:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:18:23.516+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:18:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:18:23.516+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:18:24.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:23 smithi077 ceph-mon[37153]: pgmap v10891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:24.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:23 smithi047 ceph-mon[36835]: pgmap v10891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:24.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:23 smithi029 ceph-mon[32610]: pgmap v10891: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:26.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:25 smithi077 ceph-mon[37153]: pgmap v10892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:26.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:25 smithi047 ceph-mon[36835]: pgmap v10892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:26.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:25 smithi029 ceph-mon[32610]: pgmap v10892: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:28.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:27 smithi077 ceph-mon[37153]: pgmap v10893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:18:28.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19729.823048 secs 2024-07-27T12:18:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:27 smithi047 ceph-mon[36835]: pgmap v10893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:18:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19729.823048 secs 2024-07-27T12:18:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:27 smithi029 ceph-mon[32610]: pgmap v10893: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:18:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19729.823048 secs 2024-07-27T12:18:30.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:29 smithi077 ceph-mon[37153]: pgmap v10894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:30.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:29 smithi047 ceph-mon[36835]: pgmap v10894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:30.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:29 smithi029 ceph-mon[32610]: pgmap v10894: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:32.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:32 smithi077 ceph-mon[37153]: pgmap v10895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:32.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:32 smithi047 ceph-mon[36835]: pgmap v10895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:32.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:32 smithi029 ceph-mon[32610]: pgmap v10895: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:33.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19734.823178 secs 2024-07-27T12:18:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19734.823178 secs 2024-07-27T12:18:33.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19734.823178 secs 2024-07-27T12:18:34.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:34 smithi077 ceph-mon[37153]: pgmap v10896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:34.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:34 smithi047 ceph-mon[36835]: pgmap v10896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:34.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:34 smithi029 ceph-mon[32610]: pgmap v10896: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:36 smithi077 ceph-mon[37153]: pgmap v10897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:36.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:36 smithi047 ceph-mon[36835]: pgmap v10897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:36.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:36 smithi029 ceph-mon[32610]: pgmap v10897: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:38 smithi077 ceph-mon[37153]: pgmap v10898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:18:38.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19739.823301 secs 2024-07-27T12:18:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:38 smithi047 ceph-mon[36835]: pgmap v10898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:18:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19739.823301 secs 2024-07-27T12:18:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:38 smithi029 ceph-mon[32610]: pgmap v10898: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:18:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19739.823301 secs 2024-07-27T12:18:38.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:18:38.532+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:18:38.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:18:38.532+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:18:38.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:18:38.532+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:18:40.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:40 smithi077 ceph-mon[37153]: pgmap v10899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:40.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:40 smithi047 ceph-mon[36835]: pgmap v10899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:40.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:40 smithi029 ceph-mon[32610]: pgmap v10899: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:42 smithi077 ceph-mon[37153]: pgmap v10900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:42.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:42 smithi047 ceph-mon[36835]: pgmap v10900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:42.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:42 smithi029 ceph-mon[32610]: pgmap v10900: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19744.823432 secs 2024-07-27T12:18:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19744.823432 secs 2024-07-27T12:18:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19744.823432 secs 2024-07-27T12:18:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:44 smithi077 ceph-mon[37153]: pgmap v10901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:44.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:44 smithi047 ceph-mon[36835]: pgmap v10901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:44.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:44 smithi029 ceph-mon[32610]: pgmap v10901: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:46.358 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:46 smithi077 ceph-mon[37153]: pgmap v10902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:46.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:46 smithi047 ceph-mon[36835]: pgmap v10902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:46.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:46 smithi029 ceph-mon[32610]: pgmap v10902: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:48 smithi077 ceph-mon[37153]: pgmap v10903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:18:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19749.823560 secs 2024-07-27T12:18:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:48 smithi047 ceph-mon[36835]: pgmap v10903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:18:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19749.823560 secs 2024-07-27T12:18:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:48 smithi029 ceph-mon[32610]: pgmap v10903: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:18:48.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19749.823560 secs 2024-07-27T12:18:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:50 smithi077 ceph-mon[37153]: pgmap v10904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:50.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:50 smithi047 ceph-mon[36835]: pgmap v10904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:50.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:50 smithi029 ceph-mon[32610]: pgmap v10904: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:18:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:52 smithi077 ceph-mon[37153]: pgmap v10905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:52.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:52 smithi047 ceph-mon[36835]: pgmap v10905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:52.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:52 smithi029 ceph-mon[32610]: pgmap v10905: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:53.347 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19754.823702 secs 2024-07-27T12:18:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19754.823702 secs 2024-07-27T12:18:53.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19754.823702 secs 2024-07-27T12:18:53.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:18:53.547+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:18:53.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:18:53.547+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:18:53.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:18:53.547+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:18:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:54 smithi077 ceph-mon[37153]: pgmap v10906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:54.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:54 smithi047 ceph-mon[36835]: pgmap v10906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:54.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:54 smithi029 ceph-mon[32610]: pgmap v10906: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:18:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:56 smithi077 ceph-mon[37153]: pgmap v10907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:56.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:56 smithi047 ceph-mon[36835]: pgmap v10907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:56.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:56 smithi029 ceph-mon[32610]: pgmap v10907: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:18:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:58 smithi077 ceph-mon[37153]: pgmap v10908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:18:58.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:18:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19759.823836 secs 2024-07-27T12:18:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:58 smithi047 ceph-mon[36835]: pgmap v10908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:18:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:18:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19759.823836 secs 2024-07-27T12:18:58.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:58 smithi029 ceph-mon[32610]: pgmap v10908: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:18:58.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:18:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19759.823836 secs 2024-07-27T12:19:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:00 smithi077 ceph-mon[37153]: pgmap v10909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:00.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:00 smithi047 ceph-mon[36835]: pgmap v10909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:00.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:00 smithi029 ceph-mon[32610]: pgmap v10909: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:02 smithi077 ceph-mon[37153]: pgmap v10910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:02.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:02 smithi047 ceph-mon[36835]: pgmap v10910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:02.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:02 smithi029 ceph-mon[32610]: pgmap v10910: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:03.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19764.823960 secs 2024-07-27T12:19:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19764.823960 secs 2024-07-27T12:19:03.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19764.823960 secs 2024-07-27T12:19:04.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:04 smithi077 ceph-mon[37153]: pgmap v10911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:19:04.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:04 smithi047 ceph-mon[36835]: pgmap v10911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:19:04.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:04 smithi029 ceph-mon[32610]: pgmap v10911: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:19:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:06 smithi077 ceph-mon[37153]: pgmap v10912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:06.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:06 smithi047 ceph-mon[36835]: pgmap v10912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:06.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:06 smithi029 ceph-mon[32610]: pgmap v10912: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:08 smithi029 ceph-mon[32610]: pgmap v10913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:19:08.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19769.824056 secs 2024-07-27T12:19:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:08 smithi077 ceph-mon[37153]: pgmap v10913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:19:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19769.824056 secs 2024-07-27T12:19:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:08 smithi047 ceph-mon[36835]: pgmap v10913: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:19:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19769.824056 secs 2024-07-27T12:19:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:08.554+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:19:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:08.554+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:19:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:08.554+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:19:09.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:09 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:19:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:09 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:19:09.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:09 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:19:10.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:10 smithi077 ceph-mon[37153]: pgmap v10914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:19:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:19:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:10 smithi077 ceph-mon[37153]: pgmap v10915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:19:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:10 smithi077 ceph-mon[37153]: pgmap v10916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:19:10.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:19:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:10 smithi047 ceph-mon[36835]: pgmap v10914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:10.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:19:10.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:19:10.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:10 smithi047 ceph-mon[36835]: pgmap v10915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:19:10.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:10 smithi047 ceph-mon[36835]: pgmap v10916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:19:10.447 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:19:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:10 smithi029 ceph-mon[32610]: pgmap v10914: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:19:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:19:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:10 smithi029 ceph-mon[32610]: pgmap v10915: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:19:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:10 smithi029 ceph-mon[32610]: pgmap v10916: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2024-07-27T12:19:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:19:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:12 smithi077 ceph-mon[37153]: pgmap v10917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:19:12.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:12 smithi047 ceph-mon[36835]: pgmap v10917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:19:12.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:12 smithi029 ceph-mon[32610]: pgmap v10917: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-27T12:19:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19774.824160 secs 2024-07-27T12:19:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19774.824160 secs 2024-07-27T12:19:13.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19774.824160 secs 2024-07-27T12:19:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:14 smithi077 ceph-mon[37153]: pgmap v10918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:19:14.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:14 smithi047 ceph-mon[36835]: pgmap v10918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:19:14.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:14 smithi029 ceph-mon[32610]: pgmap v10918: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:19:16.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:16 smithi077 ceph-mon[37153]: pgmap v10919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:19:16.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:16 smithi047 ceph-mon[36835]: pgmap v10919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:19:16.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:16 smithi029 ceph-mon[32610]: pgmap v10919: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:19:18.858 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19779.824285 secs 2024-07-27T12:19:18.859 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:18 smithi077 ceph-mon[37153]: pgmap v10920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T12:19:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19779.824285 secs 2024-07-27T12:19:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:18 smithi047 ceph-mon[36835]: pgmap v10920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T12:19:18.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19779.824285 secs 2024-07-27T12:19:18.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:18 smithi029 ceph-mon[32610]: pgmap v10920: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-27T12:19:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:20 smithi077 ceph-mon[37153]: pgmap v10921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:19:20.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:20 smithi047 ceph-mon[36835]: pgmap v10921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:19:20.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:20 smithi029 ceph-mon[32610]: pgmap v10921: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:19:22.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:22 smithi077 ceph-mon[37153]: pgmap v10922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:19:22.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:22 smithi047 ceph-mon[36835]: pgmap v10922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:19:22.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:22 smithi029 ceph-mon[32610]: pgmap v10922: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:19:23.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:23.569+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:19:23.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:23.569+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:19:23.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:23.569+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:19:23.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19784.824414 secs 2024-07-27T12:19:23.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19784.824414 secs 2024-07-27T12:19:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19784.824414 secs 2024-07-27T12:19:24.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:24 smithi077 ceph-mon[37153]: pgmap v10923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:24 smithi047 ceph-mon[36835]: pgmap v10923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:24.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:24 smithi029 ceph-mon[32610]: pgmap v10923: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:26.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:26 smithi077 ceph-mon[37153]: pgmap v10924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:26.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:26 smithi047 ceph-mon[36835]: pgmap v10924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:26.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:26 smithi029 ceph-mon[32610]: pgmap v10924: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:28.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19789.824537 secs 2024-07-27T12:19:28.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:28 smithi077 ceph-mon[37153]: pgmap v10925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:19:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19789.824537 secs 2024-07-27T12:19:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:28 smithi047 ceph-mon[36835]: pgmap v10925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:19:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19789.824537 secs 2024-07-27T12:19:28.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:28 smithi029 ceph-mon[32610]: pgmap v10925: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:19:30.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:30 smithi077 ceph-mon[37153]: pgmap v10926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:19:30.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:30 smithi047 ceph-mon[36835]: pgmap v10926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:19:30.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:30 smithi029 ceph-mon[32610]: pgmap v10926: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:19:32.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:32 smithi077 ceph-mon[37153]: pgmap v10927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:19:32.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:32 smithi047 ceph-mon[36835]: pgmap v10927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:19:32.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:32 smithi029 ceph-mon[32610]: pgmap v10927: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:19:33.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19794.824658 secs 2024-07-27T12:19:33.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19794.824658 secs 2024-07-27T12:19:33.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19794.824658 secs 2024-07-27T12:19:34.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:34 smithi077 ceph-mon[37153]: pgmap v10928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:34.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:34 smithi047 ceph-mon[36835]: pgmap v10928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:34.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:34 smithi029 ceph-mon[32610]: pgmap v10928: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:36.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:36 smithi077 ceph-mon[37153]: pgmap v10929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:19:36.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:36 smithi047 ceph-mon[36835]: pgmap v10929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:19:36.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:36 smithi029 ceph-mon[32610]: pgmap v10929: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:19:38.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:38.573+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:19:38.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:38.573+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:19:38.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:38.573+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:19:38.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19799.824814 secs 2024-07-27T12:19:38.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:38 smithi077 ceph-mon[37153]: pgmap v10930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:19:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19799.824814 secs 2024-07-27T12:19:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:38 smithi047 ceph-mon[36835]: pgmap v10930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:19:38.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19799.824814 secs 2024-07-27T12:19:38.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:38 smithi029 ceph-mon[32610]: pgmap v10930: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:19:40.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:40 smithi077 ceph-mon[37153]: pgmap v10931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:19:40.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:40 smithi047 ceph-mon[36835]: pgmap v10931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:19:40.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:40 smithi029 ceph-mon[32610]: pgmap v10931: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:19:42.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:42 smithi077 ceph-mon[37153]: pgmap v10932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:19:42.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:42 smithi047 ceph-mon[36835]: pgmap v10932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:19:42.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:42 smithi029 ceph-mon[32610]: pgmap v10932: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:19:43.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19804.824940 secs 2024-07-27T12:19:43.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19804.824940 secs 2024-07-27T12:19:43.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19804.824940 secs 2024-07-27T12:19:44.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:44 smithi077 ceph-mon[37153]: pgmap v10933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:44.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:44 smithi047 ceph-mon[36835]: pgmap v10933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:44.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:44 smithi029 ceph-mon[32610]: pgmap v10933: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:46.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:46 smithi047 ceph-mon[36835]: pgmap v10934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:46.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:46 smithi029 ceph-mon[32610]: pgmap v10934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:47.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:46 smithi077 ceph-mon[37153]: pgmap v10934: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:47 smithi047 ceph-mon[36835]: pgmap v10935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:19:47.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:47 smithi029 ceph-mon[32610]: pgmap v10935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:19:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:47 smithi077 ceph-mon[37153]: pgmap v10935: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-27T12:19:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19809.825010 secs 2024-07-27T12:19:48.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19809.825010 secs 2024-07-27T12:19:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19809.825010 secs 2024-07-27T12:19:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:49 smithi047 ceph-mon[36835]: pgmap v10936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:19:49.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:49 smithi029 ceph-mon[32610]: pgmap v10936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:19:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:49 smithi077 ceph-mon[37153]: pgmap v10936: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:19:52.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:52 smithi077 ceph-mon[37153]: pgmap v10937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:19:52.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:52 smithi047 ceph-mon[36835]: pgmap v10937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:19:52.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:52 smithi029 ceph-mon[32610]: pgmap v10937: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:19:53.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:53.589+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:19:53.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:53.589+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:19:53.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:53.589+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:19:53.847 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19814.825143 secs 2024-07-27T12:19:53.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19814.825143 secs 2024-07-27T12:19:53.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19814.825143 secs 2024-07-27T12:19:54.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:54 smithi077 ceph-mon[37153]: pgmap v10938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:54.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:54 smithi047 ceph-mon[36835]: pgmap v10938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:54.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:54 smithi029 ceph-mon[32610]: pgmap v10938: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:19:56.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:56 smithi077 ceph-mon[37153]: pgmap v10939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:56.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:56 smithi047 ceph-mon[36835]: pgmap v10939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:56.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:56 smithi029 ceph-mon[32610]: pgmap v10939: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:19:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:57.673+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0be2c0 msgr2=0x7eff3c0c2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-27T12:19:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:57.673+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0be2c0 msgr2=0x7eff3c0c2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-27T12:19:57.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:57.673+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0be2c0 0x7eff3c0c2820 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7eff400047b0 tx=0x7eff4002b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-27T12:19:57.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:57.673+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0be2c0 0x7eff3c0c2820 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7eff400047b0 tx=0x7eff4002b040 comp rx=0 tx=0).stop 2024-07-27T12:19:57.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:57.673+0000 7eff4ed9d640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0be2c0 msgr2=0x7eff3c0c2820 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.29:6800/2980069896, fault. 2024-07-27T12:19:57.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:57.673+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0be2c0 msgr2=0x7eff3c0c2820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-27T12:19:57.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:57.673+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0be2c0 0x7eff3c0c2820 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-27T12:19:57.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:57.673+0000 7eff47fff640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-27T12:19:57.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:57.674+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-27T12:19:57.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:19:57.674+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/399112257 >> [v2:172.21.15.29:6800/2980069896,v1:172.21.15.29:6801/2980069896] conn(0x7eff3c0b5580 0x7eff3c0b5ed0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7eff400047b0 tx=0x7eff4002b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-27T12:19:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19819.825267 secs 2024-07-27T12:19:58.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:19:58 smithi077 ceph-mon[37153]: pgmap v10940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:19:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19819.825267 secs 2024-07-27T12:19:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:19:58 smithi047 ceph-mon[36835]: pgmap v10940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:19:58.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19819.825267 secs 2024-07-27T12:19:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:19:58 smithi029 ceph-mon[32610]: pgmap v10940: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:20:00.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:00.609+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:20:00.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:00.609+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:20:00.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:00.610+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 154 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7eff28006500 con 0x55a2cdf3b350 2024-07-27T12:20:00.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:00.610+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T12:20:00.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:00.610+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 154 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7eff24006570 con 0x7eff50104ab0 2024-07-27T12:20:00.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:00.610+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T12:20:00.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:00.610+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 155 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T12:20:00.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:00.610+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 155 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T12:20:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:00 smithi077 ceph-mon[37153]: pgmap v10941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:20:00.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:00 smithi077 ceph-mon[37153]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T12:20:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:00 smithi047 ceph-mon[36835]: pgmap v10941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:20:00.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:00 smithi047 ceph-mon[36835]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T12:20:00.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:00 smithi029 ceph-mon[32610]: pgmap v10941: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-27T12:20:00.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:00 smithi029 ceph-mon[32610]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-27T12:20:02.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:02 smithi077 ceph-mon[37153]: pgmap v10942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:20:02.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:02 smithi047 ceph-mon[36835]: pgmap v10942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:20:02.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:02 smithi029 ceph-mon[32610]: pgmap v10942: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:20:03.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19824.825386 secs 2024-07-27T12:20:03.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19824.825386 secs 2024-07-27T12:20:03.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19824.825386 secs 2024-07-27T12:20:04.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:04 smithi077 ceph-mon[37153]: pgmap v10943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:04.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:04 smithi047 ceph-mon[36835]: pgmap v10943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:04.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:04 smithi029 ceph-mon[32610]: pgmap v10943: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:06.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:06 smithi077 ceph-mon[37153]: pgmap v10944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:06.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:06 smithi047 ceph-mon[36835]: pgmap v10944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:06.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:06 smithi029 ceph-mon[32610]: pgmap v10944: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:08.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:08.595+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:20:08.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:08.595+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:20:08.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:08.595+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:20:08.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19829.825478 secs 2024-07-27T12:20:08.901 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:08 smithi077 ceph-mon[37153]: pgmap v10945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:20:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19829.825478 secs 2024-07-27T12:20:08.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:08 smithi047 ceph-mon[36835]: pgmap v10945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:20:08.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19829.825478 secs 2024-07-27T12:20:08.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:08 smithi029 ceph-mon[32610]: pgmap v10945: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:20:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:10 smithi077 ceph-mon[37153]: pgmap v10946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:20:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:20:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:20:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:20:10.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:20:10.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:10 smithi029 ceph-mon[32610]: pgmap v10946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:20:10.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:20:10.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:20:10.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:20:10.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:20:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:10 smithi047 ceph-mon[36835]: pgmap v10946: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:20:10.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:20:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:20:10.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:20:10.949 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:20:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:11 smithi077 ceph-mon[37153]: pgmap v10947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:20:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:11 smithi077 ceph-mon[37153]: pgmap v10948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:20:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:11 smithi047 ceph-mon[36835]: pgmap v10947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:20:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:11 smithi047 ceph-mon[36835]: pgmap v10948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:20:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:11 smithi029 ceph-mon[32610]: pgmap v10947: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-27T12:20:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:11 smithi029 ceph-mon[32610]: pgmap v10948: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-27T12:20:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:13 smithi077 ceph-mon[37153]: pgmap v10949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T12:20:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19834.825599 secs 2024-07-27T12:20:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:13 smithi047 ceph-mon[36835]: pgmap v10949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T12:20:13.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19834.825599 secs 2024-07-27T12:20:13.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:13 smithi029 ceph-mon[32610]: pgmap v10949: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T12:20:13.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19834.825599 secs 2024-07-27T12:20:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:15 smithi077 ceph-mon[37153]: pgmap v10950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:20:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:15 smithi047 ceph-mon[36835]: pgmap v10950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:20:15.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:15 smithi029 ceph-mon[32610]: pgmap v10950: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-27T12:20:17.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:17 smithi077 ceph-mon[37153]: pgmap v10951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T12:20:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:17 smithi047 ceph-mon[36835]: pgmap v10951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T12:20:17.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:17 smithi029 ceph-mon[32610]: pgmap v10951: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-27T12:20:18.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19839.825727 secs 2024-07-27T12:20:18.947 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19839.825727 secs 2024-07-27T12:20:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19839.825727 secs 2024-07-27T12:20:19.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:19 smithi047 ceph-mon[36835]: pgmap v10952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:20:19.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:19 smithi029 ceph-mon[32610]: pgmap v10952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:20:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:19 smithi077 ceph-mon[37153]: pgmap v10952: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-27T12:20:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:21 smithi047 ceph-mon[36835]: pgmap v10953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:20:21.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:21 smithi029 ceph-mon[32610]: pgmap v10953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:20:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:21 smithi077 ceph-mon[37153]: pgmap v10953: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:20:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:23.604+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:20:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:23.604+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:20:23.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:23.604+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:20:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:23 smithi047 ceph-mon[36835]: pgmap v10954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:20:23.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:23 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19844.825883 secs 2024-07-27T12:20:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:23 smithi029 ceph-mon[32610]: pgmap v10954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:20:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:23 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19844.825883 secs 2024-07-27T12:20:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:23 smithi077 ceph-mon[37153]: pgmap v10954: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:20:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:23 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19844.825883 secs 2024-07-27T12:20:24.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:24 smithi047 ceph-mon[36835]: pgmap v10955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:24.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:24 smithi029 ceph-mon[32610]: pgmap v10955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:25.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:24 smithi077 ceph-mon[37153]: pgmap v10955: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:27.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:27 smithi077 ceph-mon[37153]: pgmap v10956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:27.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:27 smithi047 ceph-mon[36835]: pgmap v10956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:27.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:27 smithi029 ceph-mon[32610]: pgmap v10956: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:28.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19849.825968 secs 2024-07-27T12:20:28.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19849.825968 secs 2024-07-27T12:20:28.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19849.825968 secs 2024-07-27T12:20:29.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:29 smithi047 ceph-mon[36835]: pgmap v10957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:20:29.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:29 smithi029 ceph-mon[32610]: pgmap v10957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:20:29.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:29 smithi077 ceph-mon[37153]: pgmap v10957: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:20:31.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:31 smithi047 ceph-mon[36835]: pgmap v10958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:31.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:31 smithi029 ceph-mon[32610]: pgmap v10958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:31.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:31 smithi077 ceph-mon[37153]: pgmap v10958: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:33.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:33 smithi077 ceph-mon[37153]: pgmap v10959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:20:33.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:33 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19854.826088 secs 2024-07-27T12:20:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:33 smithi047 ceph-mon[36835]: pgmap v10959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:20:33.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:33 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19854.826088 secs 2024-07-27T12:20:33.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:33 smithi029 ceph-mon[32610]: pgmap v10959: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:20:33.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:33 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19854.826088 secs 2024-07-27T12:20:35.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:35 smithi047 ceph-mon[36835]: pgmap v10960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:35.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:35 smithi029 ceph-mon[32610]: pgmap v10960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:35.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:35 smithi077 ceph-mon[37153]: pgmap v10960: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:37.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:37 smithi047 ceph-mon[36835]: pgmap v10961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:37.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:37 smithi029 ceph-mon[32610]: pgmap v10961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:37.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:37 smithi077 ceph-mon[37153]: pgmap v10961: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:38.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19859.826211 secs 2024-07-27T12:20:38.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19859.826211 secs 2024-07-27T12:20:38.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19859.826211 secs 2024-07-27T12:20:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:38.614+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:20:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:38.614+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:20:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:38.614+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:20:39.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:39 smithi047 ceph-mon[36835]: pgmap v10962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:20:39.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:39 smithi029 ceph-mon[32610]: pgmap v10962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:20:39.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:39 smithi077 ceph-mon[37153]: pgmap v10962: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:20:41.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:41 smithi047 ceph-mon[36835]: pgmap v10963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:41.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:41 smithi029 ceph-mon[32610]: pgmap v10963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:41.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:41 smithi077 ceph-mon[37153]: pgmap v10963: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:43 smithi047 ceph-mon[36835]: pgmap v10964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:20:43.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:43 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19864.826306 secs 2024-07-27T12:20:43.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:43 smithi029 ceph-mon[32610]: pgmap v10964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:20:43.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:43 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19864.826306 secs 2024-07-27T12:20:43.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:43 smithi077 ceph-mon[37153]: pgmap v10964: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:20:43.607 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:43 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19864.826306 secs 2024-07-27T12:20:45.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:45 smithi047 ceph-mon[36835]: pgmap v10965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:45.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:45 smithi029 ceph-mon[32610]: pgmap v10965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:45.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:45 smithi077 ceph-mon[37153]: pgmap v10965: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:47.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:47 smithi047 ceph-mon[36835]: pgmap v10966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:47.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:47 smithi029 ceph-mon[32610]: pgmap v10966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:47.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:47 smithi077 ceph-mon[37153]: pgmap v10966: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:48.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19869.826428 secs 2024-07-27T12:20:48.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19869.826428 secs 2024-07-27T12:20:48.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19869.826428 secs 2024-07-27T12:20:49.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:49 smithi047 ceph-mon[36835]: pgmap v10967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:20:49.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:49 smithi029 ceph-mon[32610]: pgmap v10967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:20:49.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:49 smithi077 ceph-mon[37153]: pgmap v10967: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:20:51.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:51 smithi047 ceph-mon[36835]: pgmap v10968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:51.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:51 smithi029 ceph-mon[32610]: pgmap v10968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:51.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:51 smithi077 ceph-mon[37153]: pgmap v10968: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:53 smithi047 ceph-mon[36835]: pgmap v10969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:20:53.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:53 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19874.826552 secs 2024-07-27T12:20:53.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:53 smithi029 ceph-mon[32610]: pgmap v10969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:20:53.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:53 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19874.826552 secs 2024-07-27T12:20:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:53 smithi077 ceph-mon[37153]: pgmap v10969: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:20:53.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:53 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19874.826552 secs 2024-07-27T12:20:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:53.626+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:20:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:53.626+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:20:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:20:53.626+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:20:55.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:55 smithi047 ceph-mon[36835]: pgmap v10970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:55.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:55 smithi029 ceph-mon[32610]: pgmap v10970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:55.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:55 smithi077 ceph-mon[37153]: pgmap v10970: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:20:57.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:57 smithi047 ceph-mon[36835]: pgmap v10971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:57.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:57 smithi029 ceph-mon[32610]: pgmap v10971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:57.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:57 smithi077 ceph-mon[37153]: pgmap v10971: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:20:58.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19879.826694 secs 2024-07-27T12:20:58.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19879.826694 secs 2024-07-27T12:20:58.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19879.826694 secs 2024-07-27T12:20:59.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:20:59 smithi047 ceph-mon[36835]: pgmap v10972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:20:59.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:20:59 smithi029 ceph-mon[32610]: pgmap v10972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:20:59.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:20:59 smithi077 ceph-mon[37153]: pgmap v10972: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:21:01.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:01 smithi047 ceph-mon[36835]: pgmap v10973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:01.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:01 smithi029 ceph-mon[32610]: pgmap v10973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:01.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:01 smithi077 ceph-mon[37153]: pgmap v10973: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:03 smithi047 ceph-mon[36835]: pgmap v10974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:21:03.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:03 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19884.826851 secs 2024-07-27T12:21:03.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:03 smithi029 ceph-mon[32610]: pgmap v10974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:21:03.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:03 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19884.826851 secs 2024-07-27T12:21:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:03 smithi077 ceph-mon[37153]: pgmap v10974: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:21:03.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:03 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19884.826851 secs 2024-07-27T12:21:05.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:05 smithi047 ceph-mon[36835]: pgmap v10975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:05.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:05 smithi029 ceph-mon[32610]: pgmap v10975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:05.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:05 smithi077 ceph-mon[37153]: pgmap v10975: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:07.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:07 smithi047 ceph-mon[36835]: pgmap v10976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:07.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:07 smithi029 ceph-mon[32610]: pgmap v10976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:07.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:07 smithi077 ceph-mon[37153]: pgmap v10976: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:08.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:08 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19889.826972 secs 2024-07-27T12:21:08.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:08 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19889.826972 secs 2024-07-27T12:21:08.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:08 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19889.826972 secs 2024-07-27T12:21:08.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:21:08.634+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:21:08.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:21:08.634+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:21:08.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:21:08.634+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:21:09.446 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:09 smithi047 ceph-mon[36835]: pgmap v10977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:21:09.461 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:09 smithi029 ceph-mon[32610]: pgmap v10977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:21:09.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:09 smithi077 ceph-mon[37153]: pgmap v10977: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-27T12:21:10.462 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:10 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:21:10.606 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:10 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:21:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:10 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:21:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:11 smithi077 ceph-mon[37153]: pgmap v10978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:11.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:11 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:11 smithi047 ceph-mon[36835]: pgmap v10978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:11.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:11 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:11.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:11 smithi029 ceph-mon[32610]: pgmap v10978: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:11.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:11.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:11 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:12 smithi077 ceph-mon[37153]: pgmap v10979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:21:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:12 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:12 smithi047 ceph-mon[36835]: pgmap v10979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:21:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:12 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:12 smithi029 ceph-mon[32610]: pgmap v10979: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:21:13.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:13.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:12 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19894.827094 secs 2024-07-27T12:21:14.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:21:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:21:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:13 smithi077 ceph-mon[37153]: pgmap v10980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T12:21:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:13 smithi077 ceph-mon[37153]: pgmap v10981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:21:14.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19894.827094 secs 2024-07-27T12:21:14.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:21:14.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:21:14.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:13 smithi047 ceph-mon[36835]: pgmap v10980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T12:21:14.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:13 smithi047 ceph-mon[36835]: pgmap v10981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:21:14.197 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:14.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19894.827094 secs 2024-07-27T12:21:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:21:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:21:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:13 smithi029 ceph-mon[32610]: pgmap v10980: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-27T12:21:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:13 smithi029 ceph-mon[32610]: pgmap v10981: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-27T12:21:14.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:21:16.107 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:15 smithi077 ceph-mon[37153]: pgmap v10982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:21:16.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:15 smithi047 ceph-mon[36835]: pgmap v10982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:21:16.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:15 smithi029 ceph-mon[32610]: pgmap v10982: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-27T12:21:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:17 smithi077 ceph-mon[37153]: pgmap v10983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T12:21:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:17 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19899.827214 secs 2024-07-27T12:21:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:17 smithi047 ceph-mon[36835]: pgmap v10983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T12:21:18.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:17 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19899.827214 secs 2024-07-27T12:21:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:17 smithi029 ceph-mon[32610]: pgmap v10983: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-27T12:21:18.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:17 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19899.827214 secs 2024-07-27T12:21:20.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:19 smithi077 ceph-mon[37153]: pgmap v10984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:20.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:19 smithi047 ceph-mon[36835]: pgmap v10984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:20.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:19 smithi029 ceph-mon[32610]: pgmap v10984: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:21 smithi077 ceph-mon[37153]: pgmap v10985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:21:22.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:21 smithi047 ceph-mon[36835]: pgmap v10985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:21:22.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:21 smithi029 ceph-mon[32610]: pgmap v10985: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:21:23.098 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19904.827300 secs 2024-07-27T12:21:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19904.827300 secs 2024-07-27T12:21:23.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19904.827300 secs 2024-07-27T12:21:23.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:21:23.639+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:21:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:21:23.639+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:21:23.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:21:23.639+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:21:23.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:23 smithi029 ceph-mon[32610]: pgmap v10986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:21:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:23 smithi077 ceph-mon[37153]: pgmap v10986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:21:24.114 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:23 smithi047 ceph-mon[36835]: pgmap v10986: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:21:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:25 smithi077 ceph-mon[37153]: pgmap v10987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:21:26.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:25 smithi047 ceph-mon[36835]: pgmap v10987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:21:26.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:25 smithi029 ceph-mon[32610]: pgmap v10987: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:21:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:27 smithi077 ceph-mon[37153]: pgmap v10988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:21:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:27 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19909.827421 secs 2024-07-27T12:21:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:27 smithi047 ceph-mon[36835]: pgmap v10988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:21:28.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:27 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19909.827421 secs 2024-07-27T12:21:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:27 smithi029 ceph-mon[32610]: pgmap v10988: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:21:28.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:27 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19909.827421 secs 2024-07-27T12:21:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:29 smithi077 ceph-mon[37153]: pgmap v10989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:30.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:29 smithi047 ceph-mon[36835]: pgmap v10989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:30.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:29 smithi029 ceph-mon[32610]: pgmap v10989: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:31 smithi077 ceph-mon[37153]: pgmap v10990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:32.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:31 smithi047 ceph-mon[36835]: pgmap v10990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:32.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:31 smithi029 ceph-mon[32610]: pgmap v10990: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19914.827566 secs 2024-07-27T12:21:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19914.827566 secs 2024-07-27T12:21:33.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19914.827566 secs 2024-07-27T12:21:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:33 smithi077 ceph-mon[37153]: pgmap v10991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:21:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:33 smithi047 ceph-mon[36835]: pgmap v10991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:21:34.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:33 smithi029 ceph-mon[32610]: pgmap v10991: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:21:36.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:35 smithi047 ceph-mon[36835]: pgmap v10992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:21:36.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:35 smithi029 ceph-mon[32610]: pgmap v10992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:21:36.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:35 smithi077 ceph-mon[37153]: pgmap v10992: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:21:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:37 smithi047 ceph-mon[36835]: pgmap v10993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:21:38.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:37 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19919.827692 secs 2024-07-27T12:21:38.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:37 smithi029 ceph-mon[32610]: pgmap v10993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:21:38.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:37 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19919.827692 secs 2024-07-27T12:21:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:37 smithi077 ceph-mon[37153]: pgmap v10993: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:21:38.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:37 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19919.827692 secs 2024-07-27T12:21:38.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:21:38.642+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:21:38.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:21:38.642+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:21:38.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:21:38.642+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:21:40.199 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:39 smithi047 ceph-mon[36835]: pgmap v10994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:40.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:39 smithi029 ceph-mon[32610]: pgmap v10994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:40.357 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:39 smithi077 ceph-mon[37153]: pgmap v10994: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:42.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:41 smithi047 ceph-mon[36835]: pgmap v10995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:42.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:41 smithi029 ceph-mon[32610]: pgmap v10995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:42.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:41 smithi077 ceph-mon[37153]: pgmap v10995: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19924.827817 secs 2024-07-27T12:21:43.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19924.827817 secs 2024-07-27T12:21:43.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19924.827817 secs 2024-07-27T12:21:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:43 smithi047 ceph-mon[36835]: pgmap v10996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:21:44.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:43 smithi029 ceph-mon[32610]: pgmap v10996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:21:44.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:43 smithi077 ceph-mon[37153]: pgmap v10996: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:21:46.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:45 smithi047 ceph-mon[36835]: pgmap v10997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:21:46.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:45 smithi029 ceph-mon[32610]: pgmap v10997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:21:46.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:45 smithi077 ceph-mon[37153]: pgmap v10997: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:21:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:47 smithi047 ceph-mon[36835]: pgmap v10998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:21:48.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:47 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19929.827971 secs 2024-07-27T12:21:48.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:47 smithi029 ceph-mon[32610]: pgmap v10998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:21:48.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:47 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19929.827971 secs 2024-07-27T12:21:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:47 smithi077 ceph-mon[37153]: pgmap v10998: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:21:48.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:47 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19929.827971 secs 2024-07-27T12:21:50.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:49 smithi047 ceph-mon[36835]: pgmap v10999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:50.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:49 smithi029 ceph-mon[32610]: pgmap v10999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:50.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:49 smithi077 ceph-mon[37153]: pgmap v10999: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:21:52.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:51 smithi047 ceph-mon[36835]: pgmap v11000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:52.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:51 smithi029 ceph-mon[32610]: pgmap v11000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:52.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:51 smithi077 ceph-mon[37153]: pgmap v11000: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:21:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19934.828121 secs 2024-07-27T12:21:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19934.828121 secs 2024-07-27T12:21:53.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19934.828121 secs 2024-07-27T12:21:53.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:21:53.657+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:21:53.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:21:53.657+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:21:53.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:21:53.657+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:21:53.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:53 smithi029 ceph-mon[32610]: pgmap v11001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:21:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:53 smithi047 ceph-mon[36835]: pgmap v11001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:21:54.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:53 smithi077 ceph-mon[37153]: pgmap v11001: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:21:56.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:55 smithi047 ceph-mon[36835]: pgmap v11002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:21:56.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:55 smithi029 ceph-mon[32610]: pgmap v11002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:21:56.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:55 smithi077 ceph-mon[37153]: pgmap v11002: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:21:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:57 smithi047 ceph-mon[36835]: pgmap v11003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:21:58.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:57 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19939.828252 secs 2024-07-27T12:21:58.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:57 smithi029 ceph-mon[32610]: pgmap v11003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:21:58.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:57 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19939.828252 secs 2024-07-27T12:21:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:57 smithi077 ceph-mon[37153]: pgmap v11003: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:21:58.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:57 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19939.828252 secs 2024-07-27T12:22:00.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:21:59 smithi047 ceph-mon[36835]: pgmap v11004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:00.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:21:59 smithi029 ceph-mon[32610]: pgmap v11004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:00.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:21:59 smithi077 ceph-mon[37153]: pgmap v11004: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:02.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:01 smithi047 ceph-mon[36835]: pgmap v11005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:02.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:01 smithi029 ceph-mon[32610]: pgmap v11005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:02.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:01 smithi077 ceph-mon[37153]: pgmap v11005: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19944.828338 secs 2024-07-27T12:22:03.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19944.828338 secs 2024-07-27T12:22:03.257 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19944.828338 secs 2024-07-27T12:22:04.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:03 smithi047 ceph-mon[36835]: pgmap v11006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:04.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:03 smithi029 ceph-mon[32610]: pgmap v11006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:04.275 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:03 smithi077 ceph-mon[37153]: pgmap v11006: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:06.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:05 smithi047 ceph-mon[36835]: pgmap v11007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:06.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:05 smithi029 ceph-mon[32610]: pgmap v11007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:06.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:05 smithi077 ceph-mon[37153]: pgmap v11007: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:07 smithi047 ceph-mon[36835]: pgmap v11008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:22:08.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:07 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19949.828460 secs 2024-07-27T12:22:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:07 smithi029 ceph-mon[32610]: pgmap v11008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:22:08.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:07 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19949.828460 secs 2024-07-27T12:22:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:07 smithi077 ceph-mon[37153]: pgmap v11008: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:22:08.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:07 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19949.828460 secs 2024-07-27T12:22:08.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:22:08.672+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:22:08.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:22:08.672+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:22:08.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:22:08.672+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:22:10.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:10 smithi047 ceph-mon[36835]: pgmap v11009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:10.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:10 smithi029 ceph-mon[32610]: pgmap v11009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:10.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:10 smithi077 ceph-mon[37153]: pgmap v11009: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:12.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:12 smithi047 ceph-mon[36835]: pgmap v11010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:12.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:12 smithi029 ceph-mon[32610]: pgmap v11010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:12.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:12 smithi077 ceph-mon[37153]: pgmap v11010: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:13 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19954.828601 secs 2024-07-27T12:22:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:22:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:13.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:13.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:13 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:13.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:13 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19954.828601 secs 2024-07-27T12:22:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:22:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:13.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:13 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:13.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:13 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19954.828601 secs 2024-07-27T12:22:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-27T12:22:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:13.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:13 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:14 smithi047 ceph-mon[36835]: pgmap v11011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:14.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:22:14.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:22:14.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:14 smithi047 ceph-mon[36835]: pgmap v11012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:22:14.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:14 smithi047 ceph-mon[36835]: pgmap v11013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T12:22:14.697 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:14 smithi047 ceph-mon[36835]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:14.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:14 smithi029 ceph-mon[32610]: pgmap v11011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:22:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:22:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:14 smithi029 ceph-mon[32610]: pgmap v11012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:22:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:14 smithi029 ceph-mon[32610]: pgmap v11013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T12:22:14.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:14 smithi029 ceph-mon[32610]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:14 smithi077 ceph-mon[37153]: pgmap v11011: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:14.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-27T12:22:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-27T12:22:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:14 smithi077 ceph-mon[37153]: pgmap v11012: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-27T12:22:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:14 smithi077 ceph-mon[37153]: pgmap v11013: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T12:22:14.857 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:14 smithi077 ceph-mon[37153]: from='mgr.14208 172.21.15.29:0/3653913424' entity='mgr.x' 2024-07-27T12:22:15.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:15 smithi077 ceph-mon[37153]: pgmap v11014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:22:15.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:15 smithi047 ceph-mon[36835]: pgmap v11014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:22:15.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:15 smithi029 ceph-mon[32610]: pgmap v11014: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:22:17.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:17 smithi047 ceph-mon[36835]: pgmap v11015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T12:22:17.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:17 smithi029 ceph-mon[32610]: pgmap v11015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T12:22:18.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:17 smithi077 ceph-mon[37153]: pgmap v11015: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-27T12:22:18.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:18 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19959.828723 secs 2024-07-27T12:22:18.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:18 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19959.828723 secs 2024-07-27T12:22:19.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:18 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19959.828723 secs 2024-07-27T12:22:20.696 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:20 smithi047 ceph-mon[36835]: pgmap v11016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:22:20.711 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:20 smithi029 ceph-mon[32610]: pgmap v11016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:22:20.856 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:20 smithi077 ceph-mon[37153]: pgmap v11016: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-27T12:22:21.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:21 smithi047 ceph-mon[36835]: pgmap v11017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T12:22:21.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:21 smithi029 ceph-mon[32610]: pgmap v11017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T12:22:22.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:21 smithi077 ceph-mon[37153]: pgmap v11017: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-27T12:22:23.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:22 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19964.828871 secs 2024-07-27T12:22:23.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:22 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19964.828871 secs 2024-07-27T12:22:23.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:22 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19964.828871 secs 2024-07-27T12:22:23.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:22:23.688+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:22:23.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:22:23.688+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:22:23.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:22:23.688+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:22:23.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:23 smithi029 ceph-mon[32610]: pgmap v11018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:22:24.097 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:23 smithi047 ceph-mon[36835]: pgmap v11018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:22:24.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:23 smithi077 ceph-mon[37153]: pgmap v11018: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-27T12:22:25.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:25 smithi047 ceph-mon[36835]: pgmap v11019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:25.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:25 smithi029 ceph-mon[32610]: pgmap v11019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:26.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:25 smithi077 ceph-mon[37153]: pgmap v11019: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:27.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:27 smithi047 ceph-mon[36835]: pgmap v11020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:27.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:27 smithi029 ceph-mon[32610]: pgmap v11020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:28.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:27 smithi077 ceph-mon[37153]: pgmap v11020: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:28.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:28 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19969.828937 secs 2024-07-27T12:22:28.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:28 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19969.828937 secs 2024-07-27T12:22:29.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:28 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19969.828937 secs 2024-07-27T12:22:29.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:29 smithi047 ceph-mon[36835]: pgmap v11021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:29.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:29 smithi029 ceph-mon[32610]: pgmap v11021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:30.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:29 smithi077 ceph-mon[37153]: pgmap v11021: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:31.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:31 smithi047 ceph-mon[36835]: pgmap v11022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:22:31.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:31 smithi029 ceph-mon[32610]: pgmap v11022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:22:32.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:31 smithi077 ceph-mon[37153]: pgmap v11022: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:22:33.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:32 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19974.829010 secs 2024-07-27T12:22:33.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:32 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19974.829010 secs 2024-07-27T12:22:33.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:32 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19974.829010 secs 2024-07-27T12:22:34.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:33 smithi077 ceph-mon[37153]: pgmap v11023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:34.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:33 smithi047 ceph-mon[36835]: pgmap v11023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:34.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:33 smithi029 ceph-mon[32610]: pgmap v11023: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:35.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:35 smithi047 ceph-mon[36835]: pgmap v11024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:35.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:35 smithi029 ceph-mon[32610]: pgmap v11024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:36.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:35 smithi077 ceph-mon[37153]: pgmap v11024: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:37.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:37 smithi047 ceph-mon[36835]: pgmap v11025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:37.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:37 smithi029 ceph-mon[32610]: pgmap v11025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:38.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:37 smithi077 ceph-mon[37153]: pgmap v11025: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:38.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:22:38.703+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:22:38.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:22:38.703+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:22:38.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:22:38.703+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:22:38.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:38 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19979.829138 secs 2024-07-27T12:22:38.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:38 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19979.829138 secs 2024-07-27T12:22:39.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:38 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19979.829138 secs 2024-07-27T12:22:39.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:39 smithi047 ceph-mon[36835]: pgmap v11026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:39.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:39 smithi029 ceph-mon[32610]: pgmap v11026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:40.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:39 smithi077 ceph-mon[37153]: pgmap v11026: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:41.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:41 smithi047 ceph-mon[36835]: pgmap v11027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:22:41.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:41 smithi029 ceph-mon[32610]: pgmap v11027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:22:42.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:41 smithi077 ceph-mon[37153]: pgmap v11027: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:22:43.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:42 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19984.829221 secs 2024-07-27T12:22:43.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:42 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19984.829221 secs 2024-07-27T12:22:43.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:42 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19984.829221 secs 2024-07-27T12:22:44.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:43 smithi077 ceph-mon[37153]: pgmap v11028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:44.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:43 smithi047 ceph-mon[36835]: pgmap v11028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:44.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:43 smithi029 ceph-mon[32610]: pgmap v11028: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:45.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:45 smithi047 ceph-mon[36835]: pgmap v11029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:45.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:45 smithi029 ceph-mon[32610]: pgmap v11029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:46.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:45 smithi077 ceph-mon[37153]: pgmap v11029: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-27T12:22:47.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:47 smithi047 ceph-mon[36835]: pgmap v11030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:47.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:47 smithi029 ceph-mon[32610]: pgmap v11030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:48.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:47 smithi077 ceph-mon[37153]: pgmap v11030: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:48.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:48 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19989.829334 secs 2024-07-27T12:22:48.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:48 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19989.829334 secs 2024-07-27T12:22:49.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:48 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19989.829334 secs 2024-07-27T12:22:49.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:49 smithi047 ceph-mon[36835]: pgmap v11031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:49.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:49 smithi029 ceph-mon[32610]: pgmap v11031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:50.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:49 smithi077 ceph-mon[37153]: pgmap v11031: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:51.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:51 smithi047 ceph-mon[36835]: pgmap v11032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:22:51.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:51 smithi029 ceph-mon[32610]: pgmap v11032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:22:52.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:51 smithi077 ceph-mon[37153]: pgmap v11032: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:22:53.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:52 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19994.829473 secs 2024-07-27T12:22:53.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:52 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19994.829473 secs 2024-07-27T12:22:53.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:52 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19994.829473 secs 2024-07-27T12:22:53.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:22:53.719+0000 7eff4ed9d640 1 -- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 msgr2=0x55a2ce2fded0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:6836/2573307312 2024-07-27T12:22:53.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:22:53.719+0000 7eff4ed9d640 1 --2- 172.21.15.29:0/3150119665 >> [v2:172.21.15.47:6836/2573307312,v1:172.21.15.47:6837/2573307312] conn(0x55a2cdef9cd0 0x55a2ce2fded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-27T12:22:53.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:22:53.719+0000 7eff367fc640 1 client.16383 ms_handle_refused on v2:172.21.15.47:6836/2573307312 2024-07-27T12:22:54.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:53 smithi077 ceph-mon[37153]: pgmap v11033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:54.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:53 smithi047 ceph-mon[36835]: pgmap v11033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:54.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:53 smithi029 ceph-mon[32610]: pgmap v11033: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-27T12:22:55.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:55 smithi047 ceph-mon[36835]: pgmap v11034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:55.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:55 smithi029 ceph-mon[32610]: pgmap v11034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:56.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:55 smithi077 ceph-mon[37153]: pgmap v11034: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:57.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:57 smithi047 ceph-mon[36835]: pgmap v11035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:57.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:57 smithi029 ceph-mon[32610]: pgmap v11035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:58.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:57 smithi077 ceph-mon[37153]: pgmap v11035: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-27T12:22:58.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:58 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 19999.829615 secs 2024-07-27T12:22:58.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:58 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 19999.829615 secs 2024-07-27T12:22:59.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:58 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 19999.829615 secs 2024-07-27T12:22:59.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:22:59 smithi047 ceph-mon[36835]: pgmap v11036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:22:59.961 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:22:59 smithi029 ceph-mon[32610]: pgmap v11036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:23:00.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:22:59 smithi077 ceph-mon[37153]: pgmap v11036: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-27T12:23:00.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:23:00.633+0000 7eff357fa640 1 -- 172.21.15.29:0/3150119665 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T12:23:00.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:23:00.633+0000 7eff46ffd640 1 -- 172.21.15.29:0/399112257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T12:23:00.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:23:00.634+0000 7eff367fc640 1 -- 172.21.15.29:0/3150119665 <== mon.0 v2:172.21.15.29:3300/0 156 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7eff28003690 con 0x55a2cdf3b350 2024-07-27T12:23:00.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-07-27T12:23:00.634+0000 7eff47fff640 1 -- 172.21.15.29:0/399112257 <== mon.0 v2:172.21.15.29:3300/0 156 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7eff24003690 con 0x7eff50104ab0 2024-07-27T12:23:01.946 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:23:01 smithi047 ceph-mon[36835]: pgmap v11037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:23:01.962 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:23:01 smithi029 ceph-mon[32610]: pgmap v11037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:23:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:23:01 smithi077 ceph-mon[37153]: pgmap v11037: 97 pgs: 97 active+clean; 45 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-27T12:23:03.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:Jul 27 12:23:02 smithi077 ceph-mon[37153]: 1 slow requests, 0 included below; oldest blocked for > 20004.829777 secs 2024-07-27T12:23:03.196 INFO:journalctl@ceph.mon.b.smithi047.stdout:Jul 27 12:23:02 smithi047 ceph-mon[36835]: 1 slow requests, 0 included below; oldest blocked for > 20004.829777 secs 2024-07-27T12:23:03.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Jul 27 12:23:02 smithi029 ceph-mon[32610]: 1 slow requests, 0 included below; oldest blocked for > 20004.829777 secs 2024-07-27T12:23:03.480 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-07-27T12:23:03.481 DEBUG:teuthology.task.console_log:Killing console logger for smithi029 2024-07-27T12:23:03.482 DEBUG:teuthology.task.console_log:Killing console logger for smithi047 2024-07-27T12:23:03.482 DEBUG:teuthology.task.console_log:Killing console logger for smithi077 2024-07-27T12:23:03.482 DEBUG:teuthology.exit:Finished running handlers